Step 8: Scalability and Performance

On a normal day, this system sends about 150 million notifications. Now a celebrity with 10 million followers posts a photo, and a social app asks us to tell every follower. That one request causes three problems, shown in the diagram below:

1. Intake: the request becomes millions of sends, arriving as a burst.
2. Delivery: the providers accept only so many messages a second.
3. Storage: every send adds a row to a history of billions.

Problem 1: A burst at intake

The post turns into tens of thousands of sends a second, and the fix is somewhere for them to wait. Kafka sits between the service and the channel workers. A large cluster handles millions of messages a second, so a peak of 50,000, or 100,000 with headroom, is comfortable. Queue depth climbing during the burst means the queue is working. Depth that keeps growing after it is the warning. Service instances sit behind a load balancer and Kafka spreads partitions across brokers, so each stage grows by adding machines.

The queue must not lose what it accepted. Each topic keeps three copies across availability zones, and the producer waits for all in-sync replicas (acks=all) before the service returns 202 Accepted. One lost broker then loses no accepted notification.

The burst must not delay other tenants, the applications that call our API for their users. Each tenant has a request quota at the gateway. A tenant over it receives 429 instead of filling the shared queues, and the largest tenants can get their own partitions.

Every send also reads the user's preferences and a template. Both change rarely, so Redis caches preferences, which keeps the relational store off the path of every send, and each instance keeps templates in memory. When a follower turns SMS off, the update clears that user's cache entry, so the next send respects it.

Problem 2: Providers set the delivery rate

Apple, Google and the SMS gateways set the delivery rate, not us, so we scale to their limit. Workers are added per channel as that queue's backlog grows. Each worker keeps several connections open to its provider, so one slow connection does not stall it. Before a large campaign, request higher quotas, and for SMS, spread the peak over a pool of sending numbers or a second provider.

Problem 3: Billions of rows of history

The history is partitioned two ways. The NoSQL history is sharded by user ID, so the writes spread out and one user's recent notifications are one read on one shard.

The SQL logs are partitioned by time, one partition per day or month. Writes still go to the newest partition, but removing old data becomes cheap: dropping a partition is one operation instead of millions of row deletes.