Notification System: High-Level Design

Data Flow (Notification Request to Delivery)

Let's walk through how a notification travels through the system in real-time mode, and in batch mode:

1. Request Ingestion: An external application (producer) makes an API call to send a notification. For example, an e-commerce platform calls POST /sendNotification with the user's ID, notification type (say, "OrderShipped"), and perhaps some content or template data. This request hits the API Gateway, which authenticates it (ensuring the caller is allowed to send notifications) and forwards it to one of the Notification Service instances.
2. Request Handling in Notification Service: The Notification Service receives the request. It validates the payload - confirming required information is present (such as a recipient identifier, message content or template, and at least one channel) and that it's well-formed. It might also perform authentication/authorization checks if not done earlier. After validation, it quickly acknowledges receipt (especially if this is a synchronous API call) so the upstream caller can proceed, while the notification is handled asynchronously thereafter.
3. Preference & Routing Logic: The Notification Service then determines how to route this notification. It queries the User Preference Service (or reads from a cached preferences store) to get the user's current notification settings. This yields information such as: which channels the user has enabled, any channel overrides for this notification type, and whether the user has hit any rate limits or snooze settings. For example, if the user opted out of email for promos, and this is a promotional notification, the service would exclude email from the channels. It also checks system-wide rules (e.g., not sending more than N marketing notifications to a user per day) to decide if this notification should be throttled or dropped.
4. Message Preparation: Now, the service knows which channels to send the notification through (based on the request and preferences). It prepares the content for each channel. If templates are used, the service fetches the template for, say, email and fills in the dynamic fields (like user name, order details). Each channel may require a slightly different payload - e.g., an email needs a subject and HTML body, whereas an SMS needs a short text, and a push notification might have a title, body, and custom data for the app. The Notification Service can transform the request into channel-specific messages.
5. Enqueueing Messages: The service then enqueues the notification for delivery. Since there's no priority among channels, it will enqueue a separate message for each channel the notification should go out on. This can be done in two ways: (a) using a unified queue/topic with the channel as part of the message data, or (b) using separate queues or topics for each channel (e.g., an "Email" queue, "SMS" queue, etc.). A common design is to use topic partitioning - for instance, a Kafka topic per channel - so that channel workers can consume their respective messages independently. For example, if a notification needs to go via Email, SMS, and Push, three messages are produced: one to the Email topic, one to the SMS topic, one to the Push topic. Each message includes the content and metadata needed for that channel (like the email subject or the SMS text, destination addresses, etc., along with a notification ID and perhaps a retry count). By pushing messages to a queue, we decouple the immediate request handling from the slower delivery process - the API call can complete quickly, and actual sending happens asynchronously.
6. (Scheduled Notifications): If the request included a future send time or it's part of a batch campaign, the Notification Service would invoke the Scheduler instead of immediately enqueueing for delivery. For a scheduled notification, the service stores the request (for example, in a Scheduler Service database or a delayed-queue) with the desired send timestamp. The Scheduler continuously scans or waits until the send time is reached, then moves those pending notifications into the regular delivery queue (feeding into step 5 at the appropriate time). This way, scheduled notifications join the same pipeline as real-time ones when their time comes. Batched campaigns might be handled by inserting many notifications into the scheduler or queue in bulk (possibly with their own tools to generate a large list of user-specific messages, as was done in the Duolingo Super Bowl campaign using precomputed user lists in S3).
7. Queue Processing and Channel Delivery: The various Channel Processor services are subscribed to the Notification Queue (or specific topics). Each channel processor continuously pulls messages intended for its channel. For example, an Email Worker instance will read from the Email topic/queue. Once a message is pulled, the worker processes it: it establishes a connection to the relevant delivery provider or service and attempts to send the notification. This involves using channel-specific protocols or third-party APIs:
   • The Email Processor might call an email sending service or SMTP server (such as SendGrid, Mailgun, or Amazon SES) to send out the email. It will format the email (HTML or text) if not already formatted, handle attachments or images, and send the message to the recipient's email address.
   • The SMS Processor will send the SMS text via an SMS gateway API (like Twilio or Nexmo). It may need to ensure the text fits within SMS length limits or split long messages, handle country codes, etc.
   • The Push Notification Processor uses push services such as Firebase Cloud Messaging (FCM) for Android and Apple Push Notification service (APNs) for iOS. It prepares the payload (title, body, icon, any custom data) and sends it to the respective service, which will then deliver it to the app on the user's device.
   • The In-App Notification Processor will write the notification to the in-app notifications store (database) and if the user is currently online/connected, it can also deliver it in real-time via a persistent connection (for example, via a WebSocket message to the user's session). This ensures the notification appears inside the app immediately if the user is active, and is stored for later retrieval if the user is offline.
     Each of these processors operates independently, so an email notification could be sending at the same time as the SMS and push for the same event - there's no requirement to do them in sequence. This parallelism improves overall latency for multi-channel notifications.
8. Delivery Confirmation & Logging: After attempting to send, each channel processor gets a result - success or failure. For channels with synchronous APIs, the result is immediate; for some (like email), there might be callbacks or events for delivery/bounce later, but at least initial send status is known. The processor then logs the outcome of the notification into a Notification Logs store. A log entry typically includes: notification ID, user ID, channel, timestamp, status (sent, delivered, failed, etc.), and perhaps an error code or provider response if failed. These logs enable tracking and auditing - e.g. for customer support queries "I didn't get this email" or for system health metrics. If the send was successful, the log is marked delivered; if it failed, the system may mark it for retry (and log an initial failure with maybe a pending status). All log entries are stored in a database built to handle high write volume. This could be a relational DB table or a time-series store or even a log indexing system, depending on query needs.
9. User Notification Retrieval: For in-app notifications, the final step is allowing the user to read their notifications. The app's client (e.g., mobile app or web frontend) can call an API (possibly the Notification Service or a dedicated Notification Query Service) to fetch recent notifications for that user. The service will query the Notifications DB for that user's records (potentially using a cache for speed) and return them so the client can display, for example, a list of notifications (read/unread status, etc.). This read path is separate from the write path of sending notifications to maintain high throughput - often even separated into a different service or read replica to not impact send performance. The query service can also support features like marking notifications as read, or searching within notifications, but those are extensions beyond core delivery.

Throughout this flow, no single channel is prioritized or blocks another - each channel's delivery is handled independently via the queue. This design ensures the fastest possible dispatch on all channels and isolates any slowness (for instance, an email service lag) to that channel's worker without holding up others.