Design WhatsApp

3. Clarify non-functional requirements

Non-functional requirements describe how well the system must work: how big, how fast, and how reliable.

• Target Users: 1 billion daily active users (DAU), meaning 1 billion different people use the app on a normal day.
• Message Volume: 50 billion messages per day, about 50 messages per user.
• Latency: Latency is the time a user waits. When both users are online, a message should reach the recipient's phone in under 200 ms.
• Availability: 99.99%. People depend on messaging every day. For reference, 99.99% still allows about 52 minutes of downtime a year, so the design must contain failures rather than assume none.
• Ordering: Within one chat, every device must show the messages in the same order. Messages from one sender must appear in the order that sender wrote them.
• Durability: Once the sender sees one tick, the message must never be lost, even if a server crashes a moment later.
• Data Retention: The server keeps every message permanently, in encrypted form. Because the server cannot read the messages, this stored history is useful only to a device that holds the right keys. A brand new phone cannot read it on its own, so moving history to a new phone is a separate backup feature. Keeping history is a choice made for this design. The real WhatsApp deletes a message from its servers once it is delivered, and keeps an undelivered message for a limited time (about 30 days).

The 200 ms target and the durability rule pull in opposite directions. Durability needs a safe write to disk before we confirm anything, and every step we add before delivery uses part of the 200 ms.

4. Back of the envelope estimates

These estimates are rough on purpose. Their job is to show which part of the system needs the most attention.

• Traffic (messages per second):
  • Total messages: 50 billion per day.
  • Seconds in a day: about 86,400.
  • Average rate: 50,000,000,000 ÷ 86,400 ≈ 580,000 messages per second.
  • Peak rate, for example at midnight on New Year's Eve: about 3 times the average ≈ 1.7 million messages per second.
  • Database operations: each message is saved in two tables, the Messages table and the recipient's unread queue, and the queue entry is deleted when the phone confirms delivery. That is about 3 database operations per message, or roughly 5 million per second at peak, before counting receipt updates.
  • Receipts: each message also creates a "delivered" and a "read" receipt. The servers that hold connections therefore handle about 3 times as many small events, roughly 5 million per second at peak.
• Storage (Text):
  • Average message size: about 100 bytes.
  • Daily text storage: 50 billion × 100 bytes = 5 TB per day.
  • Yearly text storage: 5 TB × 365 ≈ 1.8 PB (petabytes). The database keeps 3 copies of each message for safety, so the disks hold about 5.5 PB a year.
  • Conclusion: No single machine can hold this. We need a database that spreads data across many machines (sharding).
• Storage (Media):
  • Assume 10% of messages carry media, with an average size of about 200 KB.
  • Daily media storage: 5 billion × 200 KB = 1 PB per day.
  • Conclusion: Media must go to object storage (such as Amazon S3), not into the message database.
• Bandwidth:
  • Media uploaded (ingress): 1 PB per day ÷ 86,400 seconds ≈ 11.6 GB per second.
  • Media downloaded (egress) is at least as large, because every file is downloaded by each recipient, and a photo sent to a group is downloaded many times.
  • Conclusion: Media downloads should come from a Content Delivery Network (CDN), which is a large set of cache servers placed close to users in many cities.

The text numbers look small next to the media numbers. The real pressure on the text path is not size but rate: 1.7 million messages a second, each one needing a lookup and a delivery within 200 ms.

5. API design

A normal HTTP request works one way: the app asks, and the server answers. In chat, the server must be able to send a message to the app at any moment, and asking the server again and again (polling) would waste traffic and add delay.

We therefore use WebSockets. A WebSocket is a connection that starts as an HTTP request and then stays open, so both sides can send data at any time.

1. Connect (WebSocket Handshake)

• GET /ws/connect
• Headers: Authorization: Bearer <token>
• Response: 101 Switching Protocols. The connection is now open and stays open.

2. Send Message (WebSocket Event)

• Client sends:

{
  "type": "send_message",
  "data": {
    "client_msg_id": "c-7f3a-0001",
    "to_user_id": "user-uuid-2",
    "content": "encrypted-blob",
    "type": "TEXT"
  }
}

• Note: The app creates client_msg_id before sending. If the app must send the same message again, for example after a dropped connection, the copy carries the same ID, so the receiving app can recognize and discard it. The content is already encrypted on the sender's phone, and the server cannot read it.

3. Upload Media (REST Endpoint)

• POST /v1/media
• Request: Multipart file (an image or a video), already encrypted on the sender's phone.
• Response:

{
  "media_id": "media-uuid-99",
  "url": "https://media-cdn.example.com/media-uuid-99"
}

4. Acknowledge Message (WebSocket Event)

• Client sends:

{
  "type": "ack",
  "data": { "message_id": "msg-123", "status": "DELIVERED" }
}

• Note: The recipient's app sends this acknowledgment (ACK) when the message reaches the phone, and sends it again with READ when the user opens the chat. These two events are what turn the sender's ticks gray and then blue.

6. High level architecture

We separate the servers that hold the open connections from the servers that run the message logic. The connection holders must stay up for hours, while the logic servers can be added, removed and restarted at any time.

Client → Load Balancer → Gateway Service → Message Service → Database

1. Client: The mobile app. It keeps its chat history in a local SQLite database on the phone, and it holds one open WebSocket connection to our servers.
2. Load Balancer: It spreads millions of WebSocket connections across the gateway servers, sending each new connection to a gateway with few connections.
3. Gateway Service (Connection Holder):
   • A stateful server, meaning it remembers something between requests: the open connections of the users attached to it.
   • It keeps a map of User ID → open connection, for the users connected to this one machine.
   • It works like a pipe: it passes messages from users to the Message Service, and it pushes messages down to users. It contains no message logic.
4. Session Cache (Redis): Redis is an in-memory key-value store, which answers in well under a millisecond. Here it holds one shared map that answers the question "Which gateway server is User B connected to?".
5. Message Service: A stateless service, meaning any of its machines can handle any request. It checks each message, stores it in the database, and routes it to the right gateway.
6. Group Service: It stores who belongs to each group, and it sends one group message to every member (fan-out).
7. Database (Cassandra): Cassandra is a wide-column database that spreads data across many machines and accepts a very high rate of writes. It stores chat history and the queue of messages each user has not yet received.
8. Object Storage (S3): It stores encrypted images and videos.

Could a message broker sit between the gateway and the Message Service? Yes, it could. A message broker, such as Kafka, is a server that stores messages in an ordered log until another service reads them. The gateway would write each message to the broker, and the Message Service would read from it.

A broker in that position would give us four things:

• Buffering: A sudden burst of messages waits in the broker instead of overloading the Message Service.
• Decoupling: The gateway does not need to know which Message Service machines are alive. It only writes to the broker.
• Durability: Once the broker has written a message to disk on several machines, a crash of a Message Service machine cannot lose it.
• Ordering per conversation: If the broker uses the chat_id as its partition key, all messages of one chat go into one ordered log and are read in the order they were written.

It also has a cost. Every message makes one extra network hop and one extra disk write before any work on it begins, and readers often take messages in small batches. Inside one data center, this usually adds a few milliseconds, and more under heavy load. It also adds one more large system to run. And the sender's one tick must now mean "written to the broker", or the app would have to wait for both writes.

This design calls the Message Service directly for one-to-one messages, as a synchronous request (the gateway sends the request and waits for the answer). The direct path already has most of what the broker would add. Durability comes from writing to Cassandra before any delivery attempt. Ordering comes from the message_id that the Message Service assigns (section 7). For buffering, the Message Service is stateless, so we add machines when traffic grows, and section 12 explains how the gateways slow down a sender who sends too much.

We do use a broker where the work is slow or bursty: group fan-out, in section 11. There, one message becomes up to 256 deliveries, so the sender should not wait for all of them. The extra milliseconds are acceptable there, and that is why group messages arrive slightly later than direct ones.

7. Data model

We need a database built for a very high rate of writes: about 580,000 messages a second on average, plus their receipts. A single SQL database cannot accept that many writes, and splitting one across many machines by hand is complex. We use Cassandra, which spreads rows across machines by a partition key, the column that decides which machine holds a row.

1. Messages Table (Chat History)

• Partition Key: chat_id (the ID of a one-to-one chat or a group) together with a time bucket, such as the month. All recent messages of one chat sit together on the same machines, so loading a chat is one fast read. The bucket stops a chat that is years old from growing into one huge partition, which Cassandra handles badly.
• Clustering Key: message_id, a TimeUUID, which is a unique ID that also contains a timestamp. The clustering key sorts the rows inside a partition, so the messages are stored in time order.

• Ordering: The Message Service creates the message_id when it stores the message, and every device shows a chat in message_id order. Because the server sets the ID and not a phone, a phone with a wrong clock cannot change the order.
• One sender can send two messages so quickly that two different Message Service machines store them at almost the same moment. To keep that sender's order exact, the app also numbers its own messages in each chat, inside the encrypted content. The receiving app never shows a sender's message 5 before that sender's message 4.
• The status column fits one-to-one chats. In a group, every member has their own status, so a real system keeps group receipts in a small separate table keyed by message and member.

2. User_Unread_Queue (Offline Optimization)

• Partition Key: user_id.
• Clustering Key: message_id.
• Other columns: chat_id, so the full message can be found in the Messages table.
• Purpose: This table lists, for each user, the messages that have not yet reached their phone. The Message Service adds an entry when it stores a message, and deletes it when the recipient's phone sends a DELIVERED acknowledgment. When a user comes online, one read of this table shows exactly what they missed, instead of a scan of every chat they belong to. This table costs one extra write for every message, even when the recipient is online. We accept that cost because an entry kept until the phone confirms is what guarantees that a message lost in transit is sent again.

3. Group_Members (SQL or NoSQL)

• group_id (PK) and user_id (PK). It answers the question "Who should receive this group message?".

8. Core flows end to end

This section follows a message through the system in the three situations that matter: both users online, the recipient offline, and the recipient coming back online. For each step, we describe what happens, why the step is needed, and what it produces.

Flow 1: User A sends a message to User B (Both Online)

This is the normal case: both users have the app open and are connected to the internet.

Step 1: Connections are already open

What happens:

• Both users opened a WebSocket connection to the gateway service when they started the app.
• User A is connected to Gateway Node 1, and User B is connected to Gateway Node 2.
• Redis (Session Cache) stores the mapping: User A → Gateway 1 and User B → Gateway 2.

Why: Opening a new connection takes several network round trips. If the connection is already open, a message can leave the phone at once.

Outcome: The system knows where each user can be reached.

Step 2: User A sends the message

What happens:

• User A types "Hello" and presses send.
• The app encrypts the text on the phone, creates a client_msg_id, and sends the message over its open WebSocket to Gateway Node 1.
• The app shows the message at once with a small clock, which means "not yet confirmed by the server".

Why: Showing the message immediately makes the app feel fast. The clock tells the user honestly that the server has not confirmed it yet.

Outcome: The encrypted message has reached our first server.

Step 3: The Message Service stores the message

What happens:

• Gateway Node 1 forwards the message to the Message Service with a direct request. It does not read or change the message.
• The Message Service assigns a message_id. It writes the message to the Messages table with status SENT, and it adds an entry for User B to User_Unread_Queue. Both writes go in one logged batch, a Cassandra feature that makes sure that if one write is applied, the other is too.
• The write counts as done when a majority of the copies, 2 out of 3, have it on disk.

Why: We save the message before we try to deliver it. If a server crashed after accepting the message but before saving it, a message the sender believes was sent would be lost. The queue entry means that if delivery fails for any reason, the message is still waiting for User B. We call the Message Service directly, not through a broker, because this write already makes the message durable, and a broker would add an extra hop and a few milliseconds to every message (section 6).

Outcome: The message is safely stored, and it will reach User B even if every later step fails.

Step 4: The sender gets one tick

What happens:

• After Cassandra confirms the write, the Message Service tells Gateway Node 1 to send an ACK to User A, with the new message_id.
• On User A's screen, the clock changes to one gray tick.

Why: The tick is a promise that the message is stored. That is why it waits for the write and does not appear earlier.

Outcome: User A knows the server has the message.

Step 5: Routing and delivery

What happens:

• The Message Service asks Redis: "Where is User B connected?" Redis replies: "Gateway Node 2."
• The Message Service sends the message to Gateway Node 2, which pushes it down the open WebSocket to User B's phone.
• If Gateway Node 2 replies that User B is no longer connected, because the Redis entry was out of date, the Message Service treats User B as offline and continues with Flow 2.

Why: Out of thousands of gateways, only one holds User B's connection. Redis answers that question in under a millisecond, which keeps us inside the 200 ms target.

Outcome: The message is on User B's phone. The app decrypts it and shows it.

Step 6: Receipts travel back

What happens:

• User B's app sends ACK: DELIVERED to Gateway Node 2, which forwards it to the Message Service.
• The Message Service deletes User B's entry from User_Unread_Queue and updates the status in the Messages table.
• It looks up User A in Redis, and Gateway Node 1 pushes the receipt to User A. The tick becomes two gray ticks.
• When User B opens the chat, the app sends ACK: READ along the same path, and the ticks turn blue.
• If User A is offline by then, the receipt waits for User A in the same way a message would.

Why: Only User B's phone can say that the message really arrived. Deleting the queue entry at this moment, and not earlier, is what guarantees that an undelivered message is never forgotten.

Outcome: Both users see the same status, and the server no longer has any work left for this message.