Notification Scheduler & Batch Processing

To support scheduled and batch notifications, a Scheduler component is included. This could be as simple as a database of future notifications with a background job, or a distributed scheduling system. One implementation: use a table ScheduledNotifications(id, user, content, channels, send_time) where notifications are stored if send_time is in the future. A Scheduler Service periodically scans for due notifications (or a lightweight cron job queries for items where send_time <= now()), then enqueues them to the Notification Queue for delivery. For precision (when a large batch needs to go out exactly at a specific time), the system might pre-fetch those notifications slightly before due time and stage them. In high-scale cases, a more sophisticated approach is needed: e.g., use a delayed queue or priority queue data structure. Some message queues (like RabbitMQ or AWS SQS with delay) support delayed messages natively. Another approach is to partition scheduled jobs by time (e.g. per minute) and have distributed workers pick up the correct bucket at the right time. The key is that scheduling logic runs in a fault-tolerant way: possibly multiple scheduler nodes for redundancy, using locks or leader election to avoid duplicate sends. If a scheduled notification is missed (e.g., scheduler down at that minute), the system should detect and recover or have a manual fallback.

For batch campaigns (e.g., sending a marketing email to 10 million users), it’s inefficient for a client to call the API 10 million times. Instead, we might provide a bulk interface or offline loading mechanism. For instance, an admin could upload a list of target user IDs for the campaign. The system (as described by the Duolingo case) might pre-fetch those users’ data, store the list in a file or DB, and then when triggered, rapidly push all those notifications through the pipeline. The workers and queue must be scaled up to handle this burst. To avoid overwhelming downstream providers (like an email service), we might intentionally throttle the consumption rate or use multiple provider accounts spread over the load.

Worker Architecture and Queuing Mechanisms

The Channel Processors (workers) are designed to be stateless and horizontally scalable. Each worker instance is a consumer from the queue. For a high-throughput system, a distributed log/queue like Kafka is a good choice because it can handle very high message rates and partition the stream across many consumers. We would create (for example) separate Kafka topics for Email, SMS, Push, etc., each with multiple partitions. The number of partitions defines the maximum parallelism – we can have one consumer thread per partition. If we anticipate 100k notifications/sec on peak, and a single consumer can process say 1k messages/sec, we’d need on the order of 100 consumers working in parallel across partitions. We might allocate, say, 50 partitions for email, 20 for SMS (if volume is lower), 30 for push, etc., based on expected load per channel. Each partition could be processed by one consumer instance, and we can always add more consumers up to the partition count. Workers can be auto-scaled based on queue backlog or system load. Each message includes a unique notification ID and perhaps a deduplication key (especially if using at-least-once delivery semantics, so if the same message reappears due to a retry, the worker can detect it has seen that ID before and skip or update instead of duplicate sending). Workers also maintain a retry count or delivery status in memory or via the log metadata.

Queue Semantics: The queue should guarantee at-least-once delivery of messages to the workers, meaning a message will be retried if a worker crashes mid-processing. Kafka by default provides at-least-once (consumers commit offsets after processing). We could also use a system like RabbitMQ which can requeue un-ACKed messages. In either case, if a worker fails after partially sending, we need idempotency to avoid duplicates. Alternatively, an exactly-once pipeline could be built with more complexity (e.g., Kafka transactions, idempotent producers/consumers). Many large systems choose at-least-once with de-duplication on the consumer side as needed. Deduplication could involve checking a cache or database of recently sent notification IDs.

No Prioritization: Since “no prioritization between channels” is required, we are not weighting the queue processing in favor of any channel. All channel topics are processed as fast as possible. Priorities can still be handled within a channel if needed (for example, maybe an SMS could have normal vs high priority messages in different queues), but the problem statement says no prioritization across channels, so we treat them equally.

Ordering Considerations: Generally, notifications to the same user via the same channel should be delivered in order of generation. Using a partitioning scheme where the partition key is the user ID (or some hash of it) can ensure that all notifications for a given user go to the same partition (and thus are processed in sequence). This prevents, say, an “Order Shipped” notification from overtaking the “Order Placed” notification for the same user. Cross-user ordering doesn’t matter. Partitioning by user also distributes load roughly evenly if user IDs are random. If certain users generate a lot of notifications (e.g. a very active user), that partition could see heavier load – in extreme cases we might partition by a composite key (user and maybe notification type) or just accept some imbalance for ordering guarantees. Another approach is partition by notification type or channel-specific logic (ensuring, for example, all promotional emails are spread out). The design can incorporate sharding at multiple levels: e.g., user-based sharding for preferences DB and partitioning by time for logs, but for the queue, user-based partitioning is a simple strategy to preserve order per recipient.

Retry and Failure Handling

Retry Policies: Each channel processor will implement a retry mechanism for transient failures. Common strategy is exponential backoff – if a send fails, wait a short interval and try again, increasing the wait time on each failure. For example, after 1st failure wait 1s, after 2nd wait 5s, then 30s, etc. We will configure a maximum number of retries (say 3 attempts) per notification per channel. Some channels might have specific retry rules: e.g., for SMS, if the first attempt fails due to a carrier issue, it might be pointless to retry quickly – maybe we retry after a longer delay or use an alternate SMS provider if available. For email, a common pattern is to retry a couple of times over a few minutes for temporary SMTP issues. Push notifications typically either succeed or not; if a device token is invalid, a retry won’t help (the failure is permanent for that token). So the worker might classify errors into permanent vs transient. Permanent errors (invalid address, unregistered device, etc.) will not be retried excessively; they might be logged and dropped immediately. Transient ones (server timeouts, rate limit exceeded, etc.) trigger the retry logic.

The system can implement retries in a few ways. One is in the worker process itself with an in-memory timer or loop. Another robust way at scale is to publish the failed message to a retry queue (or back onto the main queue with a delay). For instance, if using Kafka, we might have separate topics like Email_Retry or use a field in the message for attempt count and have workers requeue the message with an incremented attempt count and a timestamp to not process until a certain time. Some setups use a dead-letter queue pattern: after N failed attempts, the message goes to a DLQ instead of retrying further.

Deduplication: With at-least-once delivery and retries, deduplicating notifications is important so users don’t get spammed with duplicates. We handle this by using a unique notification ID for each notification request (the API should provide or the system generates one). This ID travels with all channel messages. Workers or the Notification Service can keep a cache of recently seen IDs or check in the database before sending. For example, the Notification Service could store a short-lived record of "notification request processed" keyed by that ID, so if the same request comes again (due to client retry or message duplication), it ignores it. On the consumer side, if a worker sees a message ID that was already processed (could check a Redis set or an entry in the logs), it will skip sending. Idempotent operations are also key: if we accidentally send the same email twice, many email providers have ways to detect duplicate IDs if we pass a message ID, but we shouldn’t rely solely on that. Within our system, a combination of careful commit handling in the queue and idempotency checks prevents duplicates from normal operation. The Duolingo example explicitly mentioned ensuring no duplicate messages even if two triggers fired, highlighting the need for idempotency in the face of concurrent events.

Failure Handling and Fallbacks: Not all failures are temporary. If a channel is down or a third-party service is having an outage, we may want to failover or fallback. For instance, if our primary SMS provider fails, a secondary provider could be used. The system can be configured with multiple integrations per channel, ranked by preference. On failure to deliver via the first, it can try the second. This improves reliability at the cost of complexity and possibly higher cost. Large systems often integrate multiple vendors for critical channels (e.g., two email service providers) and implement logic to route around outages. Similarly, if sending push notifications, if one region of FCM is unresponsive, we might try another region’s endpoint. These fallbacks should still obey the no-duplication rule (only one should ultimately succeed).

If a notification ultimately fails after retries (e.g., email bounced, or we exhausted retries for a transient error), the system should record that failure in the log and possibly move the notification to a Dead Letter Queue for manual review. Operators can inspect DLQ messages and decide to reprocess them later or alert the source service that the notification was not delivered. For example, if an email keeps bouncing, perhaps the user’s email is invalid – the system could flag that user’s email contact in the preferences as invalid to avoid future attempts.

Graceful Degradation: Under extreme load or partial failures, the system should degrade gracefully. For instance, if the notification database is down, the system might still attempt to send notifications but skip logging to the DB (or log to an in-memory buffer or fallback store to flush later). If the queue is backed up (indicating downstream slowness), the rate of accepting new requests might be throttled via the API gateway or using backpressure signals. The idea is to prevent total crashes – e.g., shed non-critical load if needed (maybe drop or delay lower priority notifications, though by requirement we aren’t prioritizing channels, an extension could be prioritizing notification types in an overload scenario).

Additional Features

Rate Limiting per User/Channel: To prevent a user from being overwhelmed by notifications (especially promotional), the system should enforce limits. For example, not more than X promotional notifications per day per user. This requires counting notifications sent (which could be done via the logs or a separate counter service) and checking before sending. If a limit is exceeded, the system can defer or drop some notifications (or combine them into a digest). This is typically part of the business logic in the Notification Service or User Preferences evaluation.

Notification Format and Personalization: The system design allows personalization by using templates and data. It also should handle localization (different languages for notifications, possibly choosing template based on user locale). Templates and content might be stored or managed through a CMS that the Notification Service can query, but at design time it suffices to note templates exist and are fetched when composing messages.

Security Considerations: Ensure that the Notification Service APIs are protected (e.g., with OAuth tokens or API keys) so that only authorized systems (like the company’s own backends) can send notifications – otherwise spammers could abuse it. Also, validate content to prevent injection (especially if content might be HTML for email, or if we allow some rich content in notifications). The system should also not leak data – e.g., one user shouldn’t be able to query another’s notifications due to proper auth checks.