Step 5: High-Level Design

Path 1: Accepting the request

1. The store calls our API. The store's backend is a producer, an application outside our system that sends notifications, and the business that runs it is our tenant. It calls POST /v1/notifications with Maria's user ID, the template ID order_shipped_v1, and the data the template needs, like the order number.
2. The API gateway checks the caller. The gateway authenticates the tenant and holds it to its request quota. Making these checks at the edge stops bad or excessive traffic early. A load balancer then passes the request to one copy of the notification service.
3. The notification service validates and stores it. The service checks that the request names a recipient, a template, and valid channels. It stores the request and returns 202 Accepted, which promises only that the request is stored and will be attempted. Everything after this point happens in the background. The service keeps no data between requests (it is stateless), so any copy can handle any request.
4. The service checks Maria's preferences. It reads her settings from the user preference service, normally from a cache. If Maria had turned SMS off, an SMS would be dropped here, before anything is queued. Quiet hours use her time zone: a promotion due at 2 a.m. her time is held until morning, while a transactional message like a fraud alert goes out at once. Deciding this once, before queueing, means the system never queues work Maria did not agree to receive.
5. The service renders the message. Producers send a template ID and data, not finished text, so wording stays consistent and a tenant can change a message without a code deploy. For each channel, the service fills in that channel's template: a subject and HTML body for email, a title and short body for push. The text is fixed now, so the queued message says what it said when the request was accepted, even if someone edits the template while it waits.
6. The service puts one message per channel on the queue. The queue is Apache Kafka (RabbitMQ or Amazon SQS would also work), with one topic per channel. Maria's notification becomes three messages, on the email, push, and in-app topics. With one shared topic, every worker would read past messages it does not handle, and a slow channel could delay the others.

Steps 3 to 6 are the service's whole job, and they fit in a few lines:

def dispatch(tenant, req):
    if seen := idempotency.get(tenant.id, req.idempotency_key):
        return seen                                     # a retry of a request already accepted
    prefs = preferences.get(req.user_id)                # cached; Step 9 covers an outage
    channels = [c for c in (req.channels or prefs.enabled) if prefs.allows(c, req.template_id)]
    nid = new_notification_id()
    for ch in channels:
        body = templates.render(req.template_id, ch, req.data)   # fixed now, not at send time
        queue.publish(topic=f"{ch}.{req.priority}", key=req.user_id,
                      value={"id": nid, "to": prefs.address(ch), "body": body})
    result = {"notification_id": nid, "channels": channels}
    idempotency.put(tenant.id, req.idempotency_key, result, ttl_hours=24)
    return result                                       # sent back with 202 Accepted

The topic name combines the channel and a priority class, and the key is the user ID; Step 7 explains both choices.

When a celebrity post triggers ten million notifications, the queue grows quickly. That is the queue doing its job: it absorbs the burst, and the workers drain it as fast as the providers allow.

Path 2: Delivering to Maria

Each channel has its own worker service that reads only its own topic and calls one outside provider:

• The email worker sends through an email provider such as SendGrid or Amazon SES.
• The push worker sends through APNs for iPhones and FCM for Android phones.
• The SMS worker sends through an SMS gateway such as Twilio.
• The in-app worker writes the notification to the in-app store and, if Maria has the app open, sends it over her live connection at once.

Maria's email, push, and in-app message are all in flight at the same time, and none waits for another. Because each channel has its own topic and its own workers, a dead SMS provider stops only SMS, and email and push continue at their normal speed.

Every attempt then produces a result, and the worker writes it to the notification log: notification ID, user ID, channel, time, status, and any provider error. Some results come back at once. Email providers often report a delivery or a bounce minutes later through a webhook, a call the provider makes back to our service. A retry, a bounce handler, and a support agent answering "I never got this email" all read from this log.

Path 3: Sending later

Some notifications should not go out yet, like a reminder for tomorrow or a campaign planned for next week. Putting them in the delivery queue now would be wrong, because a queue is built to be drained, not to hold work back. Instead, the notification service hands them to the scheduler, which stores each one with its send time. When the time arrives, the scheduler places it on the queue, and it continues from step 6 like any other notification. The scheduler also feeds large campaigns into the pipeline at a controlled rate.

Path 4: Maria opens the app

A push is a single moment, and people miss or dismiss them. When Maria opens the app, it asks a read API for her recent in-app notifications, with their read and unread state, from the in-app store. This read runs on a separate service or a read replica, a copy of the database that only serves reads, so a morning spike of people opening the app never slows sending.

The stores

The stores split by the kind of data they hold:

• Relational (SQL) database: users, tenants, preferences, templates, and scheduled sends. This data is small and must be exactly right. Preferences and templates are read on every send, so each sits behind a cache that keeps the database off the path of every notification.
• NoSQL store: the notification log and history. This data is enormous, written constantly, and read only by one key, so it needs a store that scales out instead.
• In-app store: each user's unread notifications, keyed by user and behind a cache, so the app can fetch them the moment it opens.

Step 6 defines these stores in detail.