Grokking the System Design Interview, Volume II
Vote

0% completed

​

Step 3: Capacity Estimation

Two answers from the interviewer drive every number in this lesson: how many people the service reaches, and how many notifications each receives in a day. The figures are rough on purpose, because they guide decisions.

Capacity estimation
Capacity estimation

Tenants and users. Suppose we serve 10 to 50 tenant applications, some with tens of millions of users and some small. Together they have about 100 million registered users, of whom 20 to 30 million are daily active users (DAUs).

Notification volume. Assume each active user triggers or receives about 5 notifications a day, more in social apps and fewer in e-commerce. That gives about 100 to 150 million notifications a day, with jumps during viral events or big sales.

Peak throughput. 150 million a day is about 1.7k notifications per second on average. Peaks run more than 10x that, so we design for 20 to 50k events per second. The typical source of such a peak is a celebrity with 10 million followers who posts at noon: that single event becomes 10 million notifications within minutes. One event becoming many notifications is called fan-out.

Fan-out also sets how long a burst lasts. If the workers sustain 30,000 sends per second, 10 million notifications take about 330 seconds, between five and six minutes, to drain. Followers at the end of the queue hear about the post several minutes after those at the front, which is acceptable for a social post but not for a passcode. That difference is why Step 7 separates urgent traffic from bulk traffic.

Writes. Every notification is a write to a queue and to a database: about 1.7k per second on average and tens of thousands per second in a burst. Each delivery also produces status updates, sent and then delivered or failed, so the history store receives two to three writes per notification.

Reads. If 20 million users open their in-app list about once a day, that is about 230 reads per second on average, with peaks in the morning and evening.

Writes outnumber reads. Most systems read far more than they write, but this one runs the other way. Notifications are pushed out to people, and a read happens only when someone opens the app or the inbox, which few people do.

Notification storage. Keeping the last 100 notifications for all 100 million users would mean up to 10 billion records. Counting only the 20 million active users at about 50 each gives about 1 billion records. At about 500 bytes each, that is about 500 GB, or a few TB with overhead and replication, which is routine for a distributed database.

Preferences and contact data. One preferences record per user, at a few hundred bytes for 100 million users, is tens of GB and fits easily in a partitioned SQL database. A user with a phone and a tablet has two tokens, and tokens change when an app is reinstalled, so the token table needs regular cleanup. Some tenants send contact data with each request, and for those we store none.

Bandwidth. At about 1 KB per notification, 150 million notifications carry about 150 GB of payload a day, which is easy for a distributed system.

Latency budget. Push and in-app should arrive within 1 to 2 seconds, so each stage, from intake to processing to the provider call, may add only tens or low hundreds of milliseconds. Email is looser: we hand a message to the email provider within a second, and it may reach the inbox seconds later.

Per-channel limits. During a major event, we may send tens of thousands of pushes per second to APNs and FCM, which means keeping connections open and staying within their rate guidance. An email provider sends thousands per second per account, so very large volumes need several accounts or providers. In practice, most notifications go in-app or by push, and email carries a smaller share. SMS volume is far lower, perhaps tens of thousands a day, because every message is billed. It is kept for urgent uses like two-factor codes. One SMS segment holds 160 characters of plain Latin text, or only 70 if the message contains other characters such as emoji, and longer messages are billed as several segments.

Two numbers size the rest of the design: about 150 million notifications a day, and a fan-out event that turns one post into tens of thousands of sends per second. Both point toward many machines working in parallel, partitioned data, and asynchronous processing.

Reading Progress

0%


Vote for new content