Step 7: Detailed Component Design

This lesson looks closely at three parts of the delivery path, using one online store as the example. In one evening, the store sends Maria a login passcode, an "Order Placed" email and an "Order Shipped" email, while a marketing email goes to 10 million other customers. It also plans to remind her at nine tomorrow to review her purchase.

That evening raises three questions:

1. Work that is not due yet (the scheduler). Where does tomorrow's reminder wait, and how does a 10-million-user campaign enter the system?
2. Priority and order (the queues and workers). How does Maria's passcode avoid waiting behind the campaign, and how does "Order Shipped" avoid arriving before "Order Placed"?
3. Failures (the retry logic). What happens when the email provider times out or rejects a message?

1. The scheduler: work that is not due yet

Tomorrow's reminder cannot go into the delivery queue today. A queue is built to be drained, not to hold work back, so the workers would send the reminder as soon as it arrived.

Instead, the reminder becomes a row in the scheduled_notifications table from Step 6, with status scheduled and a scheduled_time. A scheduler process checks every few seconds for rows whose time has come, renders them, puts them on the delivery queue, and marks them sent. At larger scale, one-minute time buckets that workers claim as each minute arrives replace the scan.

We run several scheduler nodes, so one failure does not stop scheduling. To stop two nodes from sending the same reminder, each node claims a row with a conditional update, and only the node whose update changes the row sends it:

UPDATE scheduled_notifications
SET    status = 'sent'
WHERE  schedule_id = :id AND status = 'scheduled';

If every scheduler node is down for a while, the due rows are still scheduled when one returns, so the reminders go out late instead of being lost.

The marketing campaign is the other kind of work the queue cannot take directly. The store sends one bulk request naming a customer segment, not 10 million API calls. Before launch, a batch job turns the segment into users, removes anyone who opted out of marketing, and writes their addresses to files. At launch, a loader feeds those files into the pipeline at a rate the workers and providers can handle.

2. Queues and workers: priority and order

The channel workers keep nothing in memory between messages, so we can run many of them. They read from Kafka, a distributed log that splits each topic into partitions: separate ordered lists that different workers read at the same time. Each partition is read by one worker in a group, so the number of partitions is the most workers a topic can use.

For the design peak of 50,000 events a second, about 100 workers are enough, so partitions are created with room to grow beyond that. Workers scale automatically on consumer lag, the number of messages the group has not yet processed.

Maria's passcode is useless thirty seconds late, while the campaign email can wait an hour. If both shared one queue, the passcode would wait behind ten million emails. Each channel therefore has its own topics, split again by priority class, as the diagram below shows.

Priority classes. The priority field from Step 4 places each message in one of three classes:

• transactional: passcodes, password resets, and order updates. Never delayed, never dropped.
• default: most product notifications.
• bulk: marketing and digests. First to be delayed under load, and first to be dropped.

Four channels times three classes gives twelve topics, and workers serve the classes in that order. Separate channels mean a failing email provider cannot delay a push. Separate classes mean the campaign cannot delay Maria's passcode. The cost is more topics to operate.

Order. Kafka keeps messages in order only within one partition, so the partition key is the user ID. All of Maria's emails land in one partition, and "Order Shipped" cannot overtake "Order Placed". A very busy account, like a shared company inbox, can make its partition heavier than the rest. A key of user plus notification type spreads that load, but then order holds only within each type.

Duplicates. Kafka delivers each message at least once: a message counts as done only after the worker confirms it, so a worker that crashes after sending but before confirming receives it again. Every message carries a unique notification ID, and each worker checks a short-lived Redis set of IDs already sent on its channel and skips repeats. Repeats from the store itself are stopped earlier: the notification service saves each idempotency_key with its result, so a request resent after a timeout gets back the first notification_id and queues nothing.

3. Retries: when a send fails

Suppose the email provider times out on Maria's "Order Placed" email. Most provider failures like this are temporary, and the worker retries with exponential backoff, for example after 1 second, then 5, then 30, with at most 3 attempts. The delay grows because a failing provider is usually an overloaded one, and thousands of workers retrying every second would keep it from recovering.

Other failures are permanent, and providers say so. The worker sorts each response into one of the two kinds:

def classify(channel, response):
    if channel == "push_ios" and response.status == 410:        # APNs: token no longer active
        return PERMANENT
    if channel == "push_android" and response.error == "UNREGISTERED":   # FCM
        return PERMANENT
    if channel == "email" and response.smtp_code >= 500:         # hard bounce, e.g. 550
        return PERMANENT
    if channel == "email" and response.smtp_code >= 400:         # soft bounce: mailbox full, greylisting
        return TEMPORARY
    if response.status in (429, 500, 502, 503, 504) or response.timed_out:
        return TEMPORARY                                         # back off and retry
    return PERMANENT

A permanent failure, like an unregistered device or a hard bounce, removes that token or address from the user's record, so later notifications skip it. Getting this split wrong costs money: every retried SMS is billed, and a domain that keeps mailing dead addresses has its legitimate mail sent to spam.

A temporary failure goes to a retry topic, such as email.retry, with an attempt count and a not-before time. Retries stay out of the main topic, because a worker waiting there would hold up every message behind the failed one. After the last attempt, the message moves to the dead letter queue, where a person can see why it failed.

The retry topic can break Maria's order: while "Order Placed" waits in email.retry, "Order Shipped" can go out first. Where order matters, the worker sends later messages for the same user to the retry topic, behind the earlier one.

Provider failover. If a whole SMS or email provider fails, the worker switches to a second provider once the first one's error rate crosses a threshold. The duplicate check still applies, so only one attempt counts as the send.

Overload. If the queue grows faster than workers can drain it, the gateway slows new requests or returns 429, and the bulk class is delayed or dropped first. If the history database is down, workers keep sending and buffer the log entries until it returns. A missing log line can be repaired later; a password reset that never left cannot.

Limits and security

The store should not send Maria more than three promotions a day, so the notification service counts promotions per user per day in Redis, next to the preference check:

def allow_promotion(user_id, day):
    key = f"promo:{user_id}:{day}"
    count = redis.incr(key)            # atomic, so two sends cannot both read 2
    if count == 1:
        redis.expire(key, 86400)       # the counter disappears after a day
    return count <= 3                  # over the limit: defer, drop, or add to a digest

Transactional messages skip this check, because a password reset must still go out after the third promotion of the day. For security, the API requires tenant keys, validates content against injection, and lets users read only their own notifications.