Notification System: High-Level Design

Data Flow: from request to delivery

Now let's follow one notification through those parts. The path below is the real-time one, from the API call to the moment somebody reads the message. Scheduled and batch sends branch off at step 6.

1. Request ingestion. A producer calls the API to send a notification. An e-commerce platform calls POST /v1/notifications with the user id, the template id, say "order_shipped_v1", and the data that fills that template. The request reaches the API gateway. The gateway checks that the caller is allowed to send. It then forwards the request to one notification service instance.

2. Request handling. The notification service validates the payload. It checks that the required fields are present: a recipient, message content or a template, and at least one channel. It also checks that the payload is well formed. If the gateway did not authenticate the caller, the service does it here. Then it acknowledges the request right away, so the caller can move on. That is the 202 Accepted from Step 4. It means the request is stored and will be attempted, not that anybody has been notified. Everything after this point happens asynchronously.

3. Preferences and routing. Next, the service decides how to route the notification. It reads the user's settings from the user preference service, or from a cache of them. Those settings name the channels the user has enabled, and any channel overrides for this notification type. They also show any rate limits or snooze settings the user has hit. For example, if the user opted out of promotional email and this is a promotion, email is removed from the list. The service also applies system-wide rules, like a maximum of N marketing notifications per user per day. A notification can be throttled or dropped here. A channel the user has turned off never reaches the queue.

4. Message preparation. The service now knows the channels, from the request and the preferences. It builds the content for each one. If templates are used, it fetches the template for, say, email and fills in the fields, like user name and order details. Each channel needs a different payload. An email needs a subject and an HTML body. An SMS needs a short text. A push needs a title, a body, and custom data for the app. So the service turns one request into channel-specific messages. The text is fixed here, before queueing. A template edited while messages wait cannot change what a queued message says.

5. Enqueueing. The service puts the notification on the queue. Channels have no priority among them, so it enqueues a separate message for each channel. There are two ways to lay this out. One is a single queue or topic, with the channel stored inside each message. The other is a separate queue or topic per channel: an email queue, an SMS queue, and so on. A common design is a Kafka topic per channel, so each channel's workers read only their own messages. A notification going by email, SMS, and push then produces three messages, one to each topic. Each message carries what its channel needs: the content, the destination address, a notification id, and perhaps a retry count. Because the message goes to a queue, the API call finishes fast, and the slower delivery work happens later.

6. Scheduled notifications. Some requests have a future send time, or belong to a batch campaign. The service hands those to the scheduler instead of the delivery queue. The scheduler stores the request, in its own database or a delayed queue, together with the send time. It scans or waits until that time, then moves the notification into the regular delivery queue. From there it follows step 5 like any real-time notification. A batch campaign can be loaded into the scheduler or the queue in bulk. Often a separate tool builds the list of user-specific messages first. Duolingo's Super Bowl campaign did this with precomputed user lists in S3.

7. Queue processing and delivery. Each channel processor subscribes to the queue, or to its own topic, and pulls messages for its channel. An email worker reads from the email topic. When it takes a message, it connects to the delivery provider for that channel and tries to send. Each channel uses its own protocol or third-party API:

   • The email processor calls an email service or SMTP server, like SendGrid, Mailgun, or Amazon SES. It formats the email as HTML or text if that is not done yet. It handles attachments and images, and sends to the recipient's address.
   • The SMS processor sends the text through an SMS gateway API like Twilio or Nexmo. It may need to fit the text within SMS length limits, split long messages, and handle country codes.
   • The push processor uses Firebase Cloud Messaging (FCM) for Android and Apple Push Notification service (APNs) for iOS. It builds the payload with title, body, icon, and custom data, and sends it to the service. The service then delivers it to the app on the user's device.
   • The in-app processor writes the notification to the in-app store. If the user is online, it also sends it live over a persistent connection, like a WebSocket message to the user's session. So the notification shows up at once for an active user. If the user is away, it is stored for later.

   These processors run independently. The email, SMS, and push for one event can all be sending at the same time. Nothing requires them to run in sequence. That keeps latency low when one event goes out on several channels.

8. Delivery confirmation and logging. Each send attempt returns a result, success or failure. Some providers answer at once. Others, like email, report delivery or a bounce later through a callback. Either way, the first send status is known. The processor writes that outcome to a notification logs store. A log entry holds the notification id, user id, channel, timestamp, and a status like sent, delivered, or failed. If it failed, it also holds an error code or the provider's response. A successful send is logged as delivered. A failed send is logged as failed or pending, and may be marked for retry. These logs support tracking and audits. A support agent can answer "I never got this email", and operators can measure system health. The logs store must take a high write rate. It can be a relational table, a time-series store, or a log index, depending on how the logs are queried.

9. Reading notifications. For in-app notifications, the last step is letting the user read them. The client, a mobile app or a web front end, calls an API to fetch recent notifications. That API can be the notification service, or a dedicated notification query service. It reads the user's records from the notifications database, perhaps through a cache. It returns them for display with read and unread status. This read path is kept separate from the write path. Often it is a different service, or a read replica, which is a copy of the database that only serves reads. Either way, reads never slow down sending. The query service can also mark notifications as read, or search them, but those go beyond core delivery.

Two rules hold all the way through this flow. No channel is prioritized, and no channel blocks another. Each channel's delivery runs on its own through the queue, so every channel is dispatched as fast as it can go.

That isolation comes from the split. If the email provider is slow, only the email worker waits on it. Push and SMS keep sending at their normal speed.

Next: Step 6, which defines the tables this design needs.