Notification System: Detailed Component Design

Step 7: Detailed Component Design

Step 5 drew the boxes and Step 6 chose the stores. What is left is the inside of the parts that move a notification from the service to a provider. This step goes into three of them: the scheduler, the queue, and the workers that read it.

The scheduler and batch campaigns

A queue is built to be emptied as fast as possible, not to hold work back. Workers take whatever is there, the moment it is there. So a notification due tomorrow cannot sit in the delivery queue, because it would go out now. A Scheduler holds it somewhere else and moves it into the queue only when it is due.

The simplest form is a table and a background job. A table ScheduledNotifications(id, user, content, channels, send_time) holds every notification whose send_time is in the future. A Scheduler Service scans it on a timer. Or a small cron job queries for rows where send_time <= now(). Due rows are pushed onto the Notification Queue, and from there they are ordinary notifications. A large batch may have to go out at an exact time. The scheduler can then fetch it a little early and stage it.

At high scale a table scan is not enough. One option is a delayed queue or a priority queue. Either one holds a message back until its time arrives. Some message queues do this natively, like RabbitMQ or AWS SQS with delay. Another option is to bucket scheduled jobs by time, for example one bucket per minute. Distributed workers then pick up the right bucket at the right time.

Whichever form it takes, the scheduler must not be a single point of failure. Run more than one scheduler node. Use locks or leader election, so two nodes never send the same job. Leader election is how a group of nodes agrees which one is in charge. Then decide what happens when the scheduler fails anyway. If it is down for a minute, the jobs due in that minute are missed. The system should detect that and recover, or fall back to a manual resend.

Batch campaigns are the other work the delivery queue cannot take directly. A marketing email to 10 million users should not be 10 million API calls. Offer a bulk interface or an offline load instead. An admin uploads a list of target user ids. The system fetches those users' data ahead of time and stores the list in a file or a database. When the campaign is triggered, it pushes all of them through the pipeline at once. This is how the Duolingo case handled its campaign.

Two things must be ready for that burst. The queue and the workers must be scaled up to take it. And the providers below them, like an email service, must not get more than they can handle. So the system throttles how fast workers consume, or spreads the load over several provider accounts.

Workers and the queue

The Channel Processors are the workers that call the providers. They hold no state between messages, and that is what makes them easy to add. Any number of them can run at the same time. Each one is a consumer of the queue.

For this volume, a distributed log like Kafka is a good choice for that queue. It handles very high message rates. It splits a stream into partitions, which are separate ordered sequences. Many consumers can then read at the same time. Create one topic per channel: Email, SMS, Push, and so on. Give each topic several partitions.

The partition count sets the maximum parallelism, because each partition is read by one consumer thread. Say peak is 100k notifications a second, and one consumer handles about 1k a second. Then about 100 consumers are needed across the partitions. Split them by expected load per channel: say 50 partitions for email, 20 for SMS, 30 for push. Consumers can be added up to the partition count, and no further. They can be auto-scaled on queue backlog or system load.

Every message carries a unique notification id and, where useful, a deduplication key. Those ids matter because of what the queue promises. The promise is at-least-once delivery, meaning a message is redelivered if a worker crashes before finishing it. Kafka does this by default, because a consumer commits its offset only after processing. RabbitMQ does it by requeuing a message that was never acknowledged.

Either way, a worker that fails after a partial send could send twice. So the send must be idempotent, meaning that doing it a second time has no extra effect. When a message comes back after a retry, the worker sees an id it has already handled. It skips the send, or updates instead of sending again. Workers also track a retry count and the delivery status, in memory or in the log metadata.

An exactly-once pipeline is possible, with Kafka transactions and idempotent producers and consumers. It costs a lot of complexity. Most large systems choose at-least-once and deduplicate on the consumer side. Deduplication is a check against a cache or a database of recently sent notification ids.

One thing the queue must not do is rank the channels. The requirement says no channel is favored over another, so no channel topic is weighted above the rest. Every channel topic is consumed as fast as it can be. Priority inside one channel is still allowed. An SMS topic could hold normal and high priority messages in separate queues. But across channels, all are treated equally. The next section splits each channel by urgency, and the diagram shows the result.

Priority classes. Channels are treated equally, as the requirement asks. Urgency is a different question, and the requirement says nothing about it. The API in Step 4 accepts a priority on every request, and the design must use it.

A one-time passcode and a marketing email are both notifications. That is all they have in common. A passcode is worthless thirty seconds late. A campaign email can wait an hour. In one queue, the passcode waits behind the campaign.

So each channel topic is split by class:

• transactional: passcodes, password resets, and order updates. Never delayed, never dropped.
• default: most product notifications.
• bulk: marketing and digests. First to be delayed under load, and first to be dropped.

That is three topics per channel instead of one, and consumers get capacity in that order. The cost is more topics to run. The gain is that a ten million message campaign cannot delay a passcode. One shared queue cannot promise that, however deep it is.

Ordering. Notifications to the same user on the same channel should arrive in the order they were made. Kafka keeps order inside a partition, so everything for one user must go to one partition. Use the user id, or a hash of it, as the partition key. Then "Order Shipped" cannot overtake "Order Placed" for the same person. Order across different users does not matter.

Partitioning by user id also spreads load evenly, if user ids are random. A very active user makes one partition heavier. In extreme cases, use a composite key like user plus notification type. Or accept a little imbalance to keep the ordering guarantee. Sharding, which means splitting data across machines, can differ per store: preferences by user, logs by time. For the queue, user id is the simple choice that keeps order per recipient.

Retries and failures

Every provider call can fail, and most failures are temporary. Each channel processor retries those. The common strategy is exponential backoff: wait a short time, try again, and wait longer after each failure. For example, wait 1s after the first failure, 5s after the second, then 30s. Cap the attempts, say 3 per notification per channel.

The delay grows for a reason. A provider that is failing is usually overloaded. A fixed short retry from thousands of workers keeps it overloaded, so it never recovers.

Some channels need their own rules. An SMS that fails on a carrier problem will not succeed a second later. Retry after a longer delay, or switch to a second SMS provider. Email is usually retried a few times over several minutes, for temporary SMTP problems. Push either works or it does not. An invalid device token will never work, so a retry is useless.

So the worker sorts errors into permanent and temporary. Permanent errors, like an invalid address or an unregistered device, are logged and dropped at once. Temporary errors, like a timeout or a rate limit, go into the retry logic.

Retries can run in a few places. The simplest is a timer inside the worker process. A stronger option at scale is a retry queue. Publish the failed message to a separate topic like Email_Retry, or back to the main topic with a delay. The message carries an attempt count and a not-before time, and the worker raises the count on each requeue. After N failed attempts the message goes to a dead letter queue instead. A dead letter queue is a holding place for messages that could not be delivered.

Deduplication. At-least-once delivery plus retries means a user could get the same message twice. Every notification request gets a unique notification id, from the API or generated by the system. That id travels with every channel message.

The Notification Service stores a short-lived record of each processed request, keyed by that id. A repeat request, from a client retry or a duplicated message, is ignored. On the consumer side, a worker checks a Redis set or the logs before sending. If the id was already handled, it skips the send.

Sends should be idempotent too. Many email providers detect a duplicate message id if you pass one, but do not rely on that alone. Careful commit handling in the queue plus these id checks stop duplicates in normal operation. The Duolingo case made the same point: no duplicate message, even when two triggers fired at once.

Permanent failures are not retried. Retrying is right for a provider that is briefly down. It is wrong for a recipient who is gone. A device token stops working when the app is uninstalled. An email address hard bounces when it is closed, meaning the receiving server rejects it outright. A user who unsubscribes has said no. The provider reports each of these on the first attempt, and no retry changes the answer.

Split the two on the provider's response. A timeout or a 503 is temporary, so retry with growing delays. An invalid token, a hard bounce, or an opt-out is permanent. Remove the address or token, and update the user's record so nothing else tries it. A notification whose channels have all been removed ends there.

Getting this wrong costs money. Every retried SMS is billed. A domain that keeps mailing invalid addresses gets its reputation lowered by the receiving providers. Step 9 lists what to watch so this shows up early.

Some failures are larger than one message. A whole channel or provider can be down. Then the system can fail over, meaning switch to another provider. If the primary SMS provider fails, a secondary one is used. Configure several integrations per channel, ranked by preference, and try the next when one fails. This adds reliability, at the cost of complexity and possibly money. Large systems often keep two email providers for the same reason. If one FCM region does not respond, try another region's endpoint. Fallbacks must still obey the no-duplicate rule: only one attempt may finally succeed.

When a notification fails after every retry, log the failure and move it to the dead letter queue. Dropping it silently means nobody learns why it failed. Operators can inspect it and reprocess it later, or tell the source service the message was not delivered. If an email keeps bouncing, flag that address as invalid in the user's preferences.

Under overload. Under heavy load or partial failure, the system should give up a little rather than stop. If the notification database is down, keep sending and skip the database log. Or write the log to a buffer and flush it later. If the queue grows faster than the workers empty it, something downstream is slow. So throttle new requests at the API gateway, or use backpressure, a signal that tells callers to slow down. Under overload, drop or delay the lower priority classes first. Channels are still never ranked, but classes of notification can be.

Rate limits, templates, and security

Rate limits per user and channel. A user should not get too many notifications, especially promotions. Set a limit, like no more than X promotional notifications per user per day. That means counting sends, from the logs or a counter service, and checking before each send. Over the limit, defer or drop the extra ones, or combine them into a digest. This belongs in the Notification Service, next to the preference check.

Format and personalization. Templates plus data give personalized messages. The system should also localize, meaning it picks a template in the user's language. Templates may live in a CMS the Notification Service reads. For this design, it is enough that templates exist and are fetched when a message is composed.

Security. Protect the API with OAuth tokens or API keys, so only authorized backends can send. Otherwise spammers would use it. Validate content to prevent injection, especially HTML for email or rich content. Check authorization on reads too, so one user can never fetch another user's notifications.

Next: Step 8, which scales the finished design.