Before designing the system, it’s important to gauge the scale of data and traffic we need to handle. Here are some rough estimations for a “web-scale” scenario:

• User Base: Suppose we have 10 million daily active users using the reminder service (this could be a subset of a larger registered user base). It’s web-scale, so we plan for possibly tens of millions of users eventually.

• Reminders per User: Each user can create up to 500 recurring reminders, but the average user might only create, say, 5–10. Heavy users might approach the limit. Let’s assume an average of 10 reminders/user to be safe. With 10M users, that’s about 100 million reminders stored in the system. (Even if our average is overestimated, we design for this upper bound. In reality, some users will have none or a few reminders, so the actual number might be lower.)

• Total Daily Trigger Events: The number of notifications triggered per day depends on the recurrence frequencies. Let’s assume on average each reminder triggers once per day (some hourly ones will trigger 24 times a day, but many daily or weekly ones trigger less; we’ll approximate it as ~1 for a large mix). With ~100M reminders, that’s on the order of 100 million notifications per day. This is an approximation – the actual could vary, but it gives the right order of magnitude (10^8). For comparison, one design example targets ~50 million tasks per day, leading to ~34k tasks per minute on average. Our estimate is roughly double that scale.

• Traffic Rate: 100M notifications/day breaks down to ~1.16k notifications per second on average (100M/86400s ≈ 1157/s). However, traffic won’t be uniform. Likely, there are peaks at certain times of day. For example, if many people set daily 9:00 AM reminders, we’ll see surges around top-of-hour or morning times in each region. We should design for peak bursts significantly above the average. It’s possible to have thousands or even tens of thousands of notifications in a particular second during peak hours. (For instance, a scenario of 1 million notifications in one minute has been considered in similar systems, which is ~17k/sec peak.) So our system should comfortably handle tens of thousands of notifications per second at peak.

• Read vs Write Workload:

  • Writes: Creating or updating reminders are write operations to our storage. If we have 100M total reminders over time, the creation rate might be, say, 1% of reminders created per day (just a guess for steady state) = 1M creations/updates per day, which is about 11.6 per second on average – that’s trivial compared to notification sends. Even if spikes of user activity happen (say 100k users join and set reminders in a short time), those writes might hit a few hundred per second at peak. This is manageable with a distributed database.
  • Reads: Reading user’s reminders (e.g., when a user opens the app) is also relatively low traffic compared to sends. The heavy read-like workload is actually the scheduler scanning for due reminders and the reads/writes to process those due tasks. This will be a significant portion of system activity. We will design the scheduler to efficiently query upcoming reminders (likely by indexed time), rather than full table scans. Each notification sent also incurs a read of the reminder data (if not already in memory or included in the due query). We might treat sending a notification as a “read” from the perspective of retrieving the reminder and then a “write” to the outbound channel.
  • Notification Delivery: The act of sending out notifications isn’t a traditional database read/write, but it generates outgoing requests (to email servers, SMS gateways, push services). These external calls can also be on the order of thousands/sec at peak. We must ensure our external channel integrations (e.g., email/SMS providers) and network can handle that throughput or have a queuing/retry strategy if they rate-limit us.

• Data Storage Volume:

  • Each reminder stored might include: user ID, schedule pattern, next trigger time, message content (text), chosen channels, and maybe some metadata. This could be on the order of a few hundred bytes per reminder. 100M reminders * 300 bytes each ≈ 30 billion bytes (~30 GB) of active reminder data. This is feasible for a distributed NoSQL database (or a partitioned SQL) to handle. We may also store a history of sent notifications (for audit or user reference), which could grow large – e.g., 100M notifications/day, if we keep 7 days of history that’s 700M records. If each history record is small (say 100 bytes), 700M * 100B = 70GB for a week’s history. We might not need to store all of that long-term; perhaps only recent history or aggregate stats.
  • As a baseline, storing 100M reminders and their next execution times is not a big issue with modern distributed storage. Even higher user counts (50M users with a few reminders each) are fine – e.g. another design projected 50GB for user data at 50M users. Our focus will be on efficient indexing by time rather than volume per se.

• Network Bandwidth: If we send, say, 100M notifications (emails/SMS/push) per day, and each notification payload is small (let’s assume 1KB on average including headers), that’s 100M KB = ~100 GB of outbound data per day. Spread over the day it’s about 1.16MB/s average, but at peak could be an order of magnitude higher. This is within reason for a data center, but we need to ensure our design (especially the email/SMS sending components) can handle the concurrency and volume.

• Latency Requirements: To achieve “few seconds” accuracy, our scheduler likely needs to check for due tasks very frequently (possibly every second or so). If we only checked once a minute, worst-case latency could be up to 60 seconds. So we anticipate a scheduling tick of perhaps 1–5 seconds. This increases the number of queries to the due-task datastore but is necessary for high timing precision. We will account for that in the design (e.g., making those queries extremely efficient via keys or in-memory timers).

In summary, this system will handle on the order of 10^7–10^8 reminders and notifications per day, require high throughput processing (thousands of events per second), and manage tens of gigabytes of reminder data. These numbers drive us toward a distributed, partitioned architecture with a focus on time-based data access and parallel processing of jobs.

6.1 Scheduling Algorithm and Reminder Execution Pipeline

The Scheduler Service is responsible for executing reminders on schedule. It’s essentially a distributed cron engine tailored to our data model. Here’s how it functions in detail:

• Scheduler Instances and Coordination: We will run multiple instances of the Scheduler service to handle the volume. To avoid two instances picking the same reminder, we partition the work by the keys we set up (time and segment). One approach is to have a primary coordinator for schedulers that assigns each instance a set of segments to handle. For example, if we have 8 scheduler workers and we defined segments 1–8, the coordinator can assign each worker one segment (or if fewer workers than segments, some get multiple). The coordinator can also monitor heartbeats – if a worker dies, the coordinator reassigns that worker’s segment range to another, so no scheduled tasks are missed. This dynamic segment assignment ensures fault tolerance. Alternatively, we could design it such that each scheduler instance knows its index and just takes segment = index mod N (if stable), but the explicit coordinator approach is more flexible for scaling up/down and handling failures.

• Time-based Scanning: Each scheduler instance runs a loop (or a timed trigger) to scan for due reminders in its assigned segment(s). The frequency of scanning should be high to meet our timing needs. A simple strategy is: every second (or every few seconds), check the Schedule table for entries where NextExecutionTime equals the current time (to the minute or second) and Segment ∈ (the segments this instance is responsible for). For example, at 12:00:00, scheduler instance for segment 3 will query “SELECT * FROM ScheduleTable WHERE ExecutionMinute = 2025-05-19T06:11 (in epoch) AND Segment = 3” (if minute-level). It will retrieve all reminders due in that minute for that segment. If we use second-level precision, it might include seconds in the key and query exactly a timestamp match. In practice, we might fetch a small range (<= now time) to catch any that might have been slightly overdue (for instance, if there was a slight delay, anything up to now gets picked). The result of the query is a list of due reminders that this instance should process. Because we segmented by Segment and coordinated assignments, no other scheduler will handle these same entries, avoiding double processing.

• Processing Due Reminders: For each due reminder retrieved, the Scheduler will perform the following steps:

  1. Fetch/Verify Reminder Details: If the schedule entry doesn’t contain all needed info (e.g., maybe it only has ReminderID and we need the full message or updated channels), the scheduler will fetch the corresponding record from the Reminders table (using UserID + ReminderID). This ensures we have the latest data in case the user updated the reminder after it was initially scheduled. Alternatively, if we stored the message and channels in the schedule entry (denormalized), we might trust that data for sending. But to be safe regarding updates, a quick lookup can help (this is a trade-off: extra read vs. ensuring latest info). Because this is keyed by UserID partition, it’s a fast primary-key read.

  2. Compute All Target Notifications: Determine which channels need to be notified and gather the destination addresses. For example, if channels = [Email, Push], fetch the user’s email address (maybe from the reminder or a user profile service) and the user’s device push token (perhaps stored when they logged in). This could involve calling a User/Device service or looking up a cache. We then prepare the payload for each channel (e.g., format the email subject/body, the SMS text, etc., possibly using templates if needed).

  3. Enqueue Notification(s): For each channel, create a message and push it to the Notification Queue for that channel. If using a unified queue with a channel field, publish one message with all info; channel processors will filter or ignore if not their type. If using separate queues per channel (likely simpler), push one message to each relevant queue/topic. Each message contains at least: the reminder ID, user ID, message content, and target contact (or enough to derive it). The queue will handle durable storage and delivery to consumers. We ensure the enqueue operation is robust – if the queue is down, we might have to retry or, worst case, mark the reminder as not sent. Typically, these queue systems are highly available.

  4. Schedule Next Occurrence: If the reminder is recurring (which it is, in our case), calculate the next execution time after the current one. This depends on the recurrence pattern:

     • Hourly: add the hour interval. E.g., if it’s every 4 hours, next = current time + 4 hours (taking time zone into account – likely our times are in UTC internally for schedule, so 4h is straightforward in UTC for a fixed interval).
     • Daily: we need to add 1 day in the user’s local date. This is where we use the stored TimeZone. We can take the current execution time, interpret it in the user’s timezone, then add one day (24 hours in that timezone context, which accounts for DST if applicable). For example, if it was 9:00 AM PDT on June 1, the next local 9:00 AM might be 24 hours later which on UTC could be +24 or +23/+25 hours if crossing a DST boundary. A robust way is using a library: e.g., in Java, use the ZoneId and ZonedDateTime for the user’s zone, then .plusDays(1) for daily recurrence, which auto-adjusts for DST jumps. Similarly for monthly: add one month (which handles month lengths, though if something was on the 31st and next month has 30 days, one strategy is to move to last day of month or choose 30th; these are policy decisions).
     • Monthly: likely interpreted as “same day of month each time” (if the user set 15th of every month at 10:00, do that). We must handle if a month is shorter (some systems send on the last day if so, or on the 1st of next month – but typically last day of shorter month is a common approach).
     • We update the Schedule Table: remove or mark the old entry as done and insert a new entry with the new NextExecutionTime (converted to UTC) and the same ReminderID, assigning it a new random Segment value (to keep distribution even) unless we choose to keep the same segment (random each time is fine to balance load long-term). This insertion is a simple DB write that will be picked up in the future when its time comes. If the reminder was marked as “non-recurring” or the user only wanted one occurrence (like a one-time scheduled notification), we would simply not insert a new one (thus completing the job). Here, since they are recurring by definition, we continue until the user deletes it.

  5. Acknowledge completion: Depending on implementation, we may update the Reminders table’s LastSentTime or similar for record-keeping. The history log entry can also be added now (e.g., “reminder X was executed at time Y, enqueued to channels…”).

  6. Error Handling: If any step fails (e.g., database update fails or queue push fails), the Scheduler should have a retry mechanism. For instance, it might try a few times or move on and let a future tick pick it up again. It’s crucial to avoid a scenario where a reminder is “stuck” and never gets enqueued. We might use an atomic update approach: e.g., use a lightweight transaction or a compare-and-set to mark a reminder as being processed to avoid duplicate processing by a failover node (if our coordination is robust, we may not need this). We also ensure idempotency: if, by rare chance, two scheduler nodes tried the same reminder, the reminderID and timestamp can serve as an idempotency key to avoid sending twice (channel services can check if they already sent a notification with that ID). We aim to design so that doesn’t happen, but belts-and-suspenders is good for reliability.

• Frequency of Checks: As mentioned, to achieve low latency, we might have the Scheduler check every second or every few seconds for due tasks. With the partitioning strategy, these checks are cheap (a single partition read per segment for the current time slot). If we did it every second and there are, say, up to 50k tasks per minute, that’s on average <1000 tasks per second being fetched – split among segments, each segment might only see a few hundred per second, which is doable. The scheduler can batch process all due tasks in that second’s partition. If we use minute-level partitions, the scheduler might run every 10-15 seconds and fetch the current minute’s tasks, or fetch a minute a little ahead of time and then sleep. We might even look one minute into the future and pre-fetch tasks into memory, scheduling them on an internal timer for exact second dispatch. This could further tighten accuracy but adds complexity. A simpler approach: store with second precision in DB and query frequently. Given modern databases and moderate partition sizes, querying every second is not impossible, but could cause a lot of tiny queries. Possibly querying every 5 seconds for tasks in the next 5-second window is a compromise. In any case, the design target is being within a few seconds tolerance. The exact scheduling tick can be tuned and multiple schedulers ensure parallelism.

• Scaling the Scheduler: Using the segment partitioning, we can horizontally scale. If one Scheduler instance can process, say, 1000 tasks per second, and peak we need 10,000 per second, we deploy 10 instances (and set number of segments accordingly). Because we assign segments and each scheduler only queries its segments, they operate in parallel safely. We avoid two instances hitting the same partition concurrently by design. If an instance goes down, its segments will not be queried until reassigned – the coordinator will detect the missed heartbeat and reassign those segments to others or a standby. This should happen quickly to avoid missing tasks. Even if a minute’s tasks were partially processed and the instance died, when a new scheduler picks up that segment, it might find some tasks whose time is already passed; it should still process them (perhaps marked as slightly delayed but still deliver). This guarantees at-least-once delivery even in failure scenarios, though a few notifications could be late by a few seconds or a minute during a failover.

• Avoiding Skew and Hotspots: If a very popular time (like at exactly 9:00:00) has an enormous number of reminders, we rely on segments to distribute those among multiple machines. Also, randomizing segment assignment on insertion spreads out reminders roughly evenly. If one user has 500 reminders all due at 9:00, they’ll be randomly assigned segments 1–N so they won’t all fall on one segment. This smooths the per-segment load. We choose the number of segments (N) to be >= max number of scheduler instances we might need. For example, if we foresee up to 20 instances, we might choose N=32 segments, leaving headroom. This way, we can always distribute segments without running out. The coordinator can assign roughly equal count of segments to each instance (some might have one more than others if not divisible).

• Example Flow: Suppose user Alice creates a daily 9:00 AM reminder in PST (UTC-8 standard, UTC-7 DST). She does this at 1 PM on Jan 1. The Reminder Service calculates the first occurrence: if 9 AM Jan 1 already passed, then Jan 2 9:00 AM PST is next. That in UTC is Jan 2 17:00 (5pm) UTC (assuming PST vs UTC). It stores an entry with NextExecutionTime = 2025-01-02T17:00Z. Now on Jan 2, when time approaches 16:59:xx UTC, the scheduler (responsible for that time slot) will pick up the entry. It will see it’s due at 17:00. It waits until roughly that time (maybe it queries at 16:59:55 for anything up to 17:00:00). It finds Alice’s reminder. It fetches details (message “Daily Stand-up meeting”, channel “push notification”), and sends off to the queue. The Push Notification service picks it up and sends to Alice’s device. Meanwhile, Scheduler computes next occurrence: Jan 3 9:00 AM PST which is Jan 3 17:00 UTC (if DST not changed yet). Inserts a new schedule entry for that. On March 13, when DST shift happens (clocks jump forward to UTC-7 to UTC-8 offset or vice versa), the Scheduler will compute next execution using timezone rules – on DST start, 9:00 might jump an hour in UTC. It will handle that via proper timezone arithmetic. Thus Alice never notices anything except her reminder always comes at 9:00 local.

6.2 Notification Delivery per Channel

Once the Scheduler has enqueued the notification events, the Channel Processors take over to actually deliver messages. We design each channel service to be scalable and reliable:

• Email Service: This service (likely a cluster of workers) pulls email-type notifications from the queue. For each message, it prepares an email. This could involve plugging the reminder text into an email template (for example, a subject like “Reminder: {Reminder Text}”). The service then sends the email using an SMTP server or an email sending API. At web-scale, sending email directly via SMTP might be too slow (opening connections etc.), so integration with a service like Amazon SES or SendGrid is preferred – they handle the heavy lifting of email infrastructure. We just call their API (possibly batched). The Email service should handle results – e.g., the API might immediately say “accepted” or might return an error. If an error occurs (e.g., transient network issue or provider outage), the service should retry a few times. If it ultimately fails, it logs the failure (and maybe pushes the task to a dead-letter queue or marks it in the database for later retry). We also consider quotas – if the user has too many emails going out (500 daily reminders all at once), we rely on provider limitations or perhaps implement our own rate limiting (but since the user explicitly set them, we assume it’s okay, though we might stagger them by a few seconds to not appear as spam). The service also needs to handle email bounces or invalid addresses – those could be fed back into a “do not send” list if email is bad. However, that’s more of an email system concern. For our design, main point is it sends reliably and scales horizontally (we can run many instances pulling from the email queue). If using Kafka, we can have a consumer group such that partitions of the queue are split across instances (achieving parallel consumption). The queue ensures each message goes to one consumer.

• SMS Service: Similar to email, this service reads SMS notification events. It calls an SMS gateway API (like Twilio) to send the text. Usually, you provide the phone number and message content. We likely have the user’s phone on file (perhaps in the user profile, linked by UserID). If not included in the message, the SMS service might do a quick lookup to get the phone number. After sending, we get a success or failure. SMS gateways might have rate limits (e.g., Twilio might limit X messages/second per number or account). We may need to throttle or purchase more capacity if needed. For scale, multiple SMS service instances can send concurrently; the gateway and telecom networks handle queuing on their side too. We implement retries for failures (e.g., network timeout, or gateway returned an error). If a phone is invalid or opt-out, the gateway usually tells us and we should respect that (maybe marking that user’s SMS channel as invalid). Given SMS can be costly, we also ensure we only send what’s necessary (the system might avoid duplicate SMS if the same reminder was already delivered by another channel, but since the user likely wanted both, we do both). SMS content is short, we might truncate or split if too long for one message.

• Push Notification Service: This handles push notifications for mobile/web. For mobile apps, we use Apple APNs and Google FCM. We need device tokens or registration IDs from the user’s devices. Let’s assume the user’s devices are registered in a separate Device Service – our Push service might query that by UserID to get all active device tokens to send the reminder to (e.g., user might have phone and tablet). For each device, it constructs a payload (title/body for the notification). Then it connects to APNs or FCM (usually over HTTP/2 or their APIs) to send the notification. These services are quite scalable but also have rate limits and require maintaining authentication (keys, certs). We manage that within the Push service. The Push service should also handle responses – e.g., if a device token is invalid (user uninstalled app), APNs/FCM will return an error, and we should unregister that token in our device database. For reliability, push notifications are typically fire-and-forget (APNs/FCM will handle delivery to device if online). We rely on their infrastructure for retries to the device. Our concern is just getting it to their servers successfully. We might do limited retry on failure to contact APNs/FCM. This service also scales horizontally (multiple instances can each handle subsets of notifications; if using Kafka, partition by user or by device ID to ensure ordering per device perhaps).

• In-App Notification Service: For in-app, since the user might be looking at the app or might open it later, the strategy can differ:

  • If the user is currently online (e.g., has a WebSocket connection to the server or is polling), we can push the notification in real-time. This could be done via a WebSocket server or push gateway that the app is connected to. In design terms, we might integrate with an existing real-time messaging system (for example, if the app uses something like Socket.io or a pub-sub for live updates). The In-App service would publish the reminder event to that channel (e.g., user-specific channel).
  • Additionally, or if the user is offline, we store the notification in a persistent store (maybe this is simply the same as the log table or another “notifications inbox” table keyed by user). The next time the user opens the app, they fetch unseen notifications. Many applications have a notification center in-app; we would populate that.
  • Implementation could be: the In-App service receives the event, writes a record to a UserNotifications table (UserID partition, with a list of notifications, unread flags, etc.), and also pings a real-time system to notify the user immediately if possible.
  • This service should ensure that even if the real-time push fails (user offline), the data is persisted so the user doesn’t miss it. It’s less about external integration (except the real-time channel) and more about internal data handling.
  • Scaling: It’s mostly writing to a DB and optionally sending realtime messages, which can be scaled by partitioning by user. Many app frameworks handle this as part of the main app, but we illustrate it as a service.

• Unified Notification Processing: Each channel service operates independently, but we should maintain some consistency: e.g., if a reminder had multiple channels, ideally they all go out around the same time. By pulling from the queue, generally they will. Minor differences (one channel might send faster than another) are acceptable. We just ensure none are unreasonably delayed.

• Logging and Acknowledgment: After a channel sends the notification, it can log the outcome. Successful sends might update the Notification Log (execution history) with a delivered status. Failures may either go to the log with a failed status and possibly trigger an alert if persistent. If a message fails after retries, we could escalate (maybe notify the user via an alternate channel that one channel failed? For example, if an email failed to deliver, perhaps send an in-app alert “Your email reminder could not be delivered.” This is an edge consideration and may not be necessary if they’ll see it in-app anyway). The system can also aggregate stats like delivery latency, success rates per channel, etc., for monitoring.

• Idempotency: To guard against duplicate deliveries (in case the same message ID gets processed twice due to a rare bug or retry), each channel service can use an idempotency key (for example, a combination of ReminderID and the scheduled timestamp) and keep a cache of recently sent notifications. If it sees a duplicate, it can drop it. Or the Notification Queue could be configured for exactly-once processing if using something like Kafka with careful consumer management (though exactly-once at scale is tricky, at-least-once is more common). Our design leans on at-least-once and making operations idempotent, which is simpler.

6.3 Time Zone and Daylight Savings Handling

Handling time zones correctly is crucial for a reminder system – it’s what makes “recurring at 9:00 AM local time” possible. Here’s how we address it:

• Storage of Time Zone: Each reminder created by a user will carry a TimeZone attribute (likely the IANA time zone string like “America/New_York”). The user’s client can provide this (most apps know the user’s locale or have it in profile). If not, we default to the user’s account timezone. This is stored in the Reminders table. We always keep the original intended timezone and local time, rather than converting to a fixed offset, because offsets change with DST. For example, we might store “Every day at 09:00 in America/Los_Angeles”.

• Normalization to UTC for scheduling: Internally, for the Schedule table, we use UTC timestamps for NextExecutionTime. That means when we schedule the next occurrence, we calculate the exact UTC time when it will happen. The conversion uses the timezone rules. E.g., if user’s local 9:00 AM on a given date corresponds to 16:00 UTC, we store 16:00 UTC. This UTC value is what the scheduler queries against. This decouples scheduling from time zone math at runtime – the heavy lifting is done when computing next times, not when querying.

• Calculating Next Occurrence with DST: We rely on standard libraries or algorithms for recurrences:

  • We can use a library like rrule (used in iCalendar recurrence) or simply use date-time arithmetic in the user’s timezone. For daily/weekly recurrences, the pattern is: take the last execution’s local date/time, add the interval (1 day, 7 days, etc.) in calendar terms in that timezone, then convert to UTC for storage. This automatically handles DST shifts. For example:

    • DST Spring Forward: Suppose a daily 2:30 AM reminder in a region where the clocks jump forward at 2:00 AM to 3:00 AM on March 14. On March 13, it fired at 2:30 AM. The next day, March 14, 2:30 AM local time does not exist (clocks skip from 1:59 to 3:00). Using calendar addition, what happens? Many libraries would roll such a time forward by the DST gap, effectively scheduling it at 3:00 AM (or skip to next valid time). We must decide our policy: we could say “skip that occurrence” or “fire at 3:00 AM”. Typically, calendar apps would fire at 3:00 AM in this case (so the user still gets a reminder albeit 30 min late on that day). Conversely, in Fall when clocks repeat, a daily 1:30 AM might happen twice; we likely only want one trigger (most systems fire at the first occurrence of 1:30 and then when clock repeats 1:00-2:00, they don’t fire again at the repeated hour to avoid duplicates). These nuances can be handled by leveraging library behavior or by storing an extra flag that DST changed. However, for our scope, assume using the timezone’s rules via a library will yield an acceptable outcome (most likely skip or adjust automatically). We can document that the reminder will adjust to DST transitions smoothly (maybe with minor one-time shifts).

  • For hourly intervals (say “every 4 hours”), we might treat those as fixed increments in absolute time (e.g., 4h in UTC) or as clock time increments. Usually “hourly” implies every X hours regardless of timezone shifts (so during DST changes, maybe one interval might be 3 or 5 hours as clock changes). But if a user says hourly, they probably mean “just keep coming every hour” which in practice is simpler: we can just add 3600*1000 ms to last UTC time for hourly intervals (DST doesn’t affect the interval – except the pattern relative to local time will temporarily shift by an hour during DST transitions, but that might be acceptable). Alternatively, if they specifically wanted it at :00 of each hour local time, that’s a different interpretation (less likely unless explicitly stated). We’ll assume hourly means a fixed period.

  • For monthly, we have to handle different month lengths. If someone sets a reminder on the 30th of each month, February doesn’t have 30th. We can decide to send on last day of Feb for that month. This can be an internal rule. Libraries like RRULE allow specifying “last weekday of month” etc., but we won’t go that deep. We simply note that our system would have logic for “if next occurrence falls on a non-existent date, adjust to last day of month”.

  • We will maintain a consistent approach: The original creation of the reminder will establish the pattern. For recurring tasks, it’s common to store a recurrence rule (like RRULE or Cron expression plus timezone). We can store something like “RRULE:FREQ=DAILY;INTERVAL=1;BYTIME=09:00;TZID=America/Los_Angeles”. That captures it fully. Then on each trigger, rather than naive addition, we could calculate the next occurrence via that rule (there are libraries to get “next occurrence after X”). This might be more precise and handle all cases (e.g., skip DST duplicate times elegantly). However, implementing RRULE might be overkill; a simpler custom logic as above is fine for typical patterns.

• Time Zone Database Updates: We should note that time zone definitions can change (countries change DST rules, etc.). Our system should be updated with the latest tz database. If a user’s timezone offset changes due to law changes, reminders might shift unless updated. This is rare and probably beyond our need to handle dynamically, but good to mention that we’d keep our libraries updated.

• User Timezone Changes: If a user travels or manually changes their preferred timezone in settings, how do we handle existing reminders? Possibly the reminder stays in the originally set timezone (e.g., you set daily 9am in PST, and even if you move to EST, maybe you still want it PST time). Or maybe the user expects it to now come at 9am in their new timezone. This is a product decision: perhaps tie reminders to a timezone or to “local time wherever you are”. For simplicity, assume reminders are bound to a fixed timezone chosen at creation (likely the user’s home zone). If the user updates it, we can update the stored TZ and recompute future times accordingly.

• Testing DST: We would test scenarios around DST boundaries to ensure no double-send or drop: e.g., if a reminder is daily at 1:30 AM and the clock goes back causing 1:30 twice, we ensure to send only once. That might require the scheduler tracking that it already sent at that wall-time once. This could be managed by storing last execution timestamp and checking if the next computed time is not <= last (to avoid going backwards). But since we compute from last, likely it will move to next day anyway avoiding duplicates.

In summary, by storing timezone info and using proper conversions, we ensure scheduled times align with user expectations. The system always works internally in absolute time (UTC) for coordination, but all calculations of next times go through the lens of the user’s local calendar.

6.4 Failure Handling and Reliability

A robust system must anticipate various failure scenarios and mitigate them:

• Scheduler Failover and No Single Point: As described, we run multiple scheduler instances with a coordinator. If a scheduler instance crashes mid-processing, the coordinator will detect it (missed heartbeat) and reassign its segments to another instance. The tasks that it was supposed to process for the current minute might be slightly delayed – the new instance will take over and query those partitions, possibly finding tasks that are already due (past by a minute). It should still process them (better late than never). This ensures the reminders are not lost. To minimize delay, the failover should be quick (heartbeat intervals of a few seconds). Additionally, because the state (the schedule table) is in the database, any instance can pick up any task – there is no in-memory state lost except maybe some currently processing tasks. So redundancy is inherent. We avoid single points of failure by clustering the DB (Cassandra is distributed with replication factor; Kafka for queue is also clustered; load balancers have failover; etc.). We should also have multiple instances of each channel service – if one email sender crashes, others continue, and the queue retains messages until consumed.

• Exactly-Once vs At-Least-Once Delivery: Our system as designed leans towards at-least-once delivery. This means in rare cases (like a crash at just the wrong moment) a reminder might send twice, but we prefer that over missing it. For example, if a scheduler sent a message to the queue and crashed before it could update the next execution time, another scheduler might retry that task and send again. We can mitigate duplicates:

  • Use an idempotency key for sends (the combination of ReminderID and scheduled timestamp is unique for each occurrence). Channel services or even the external providers could filter duplicates if they get the same key. Or the Notification Queue can de-duplicate if configured (Kafka exactly-once processing or a de-dup layer).
  • The scheduler can mark an item as being processed (e.g., update a status flag in the schedule table) in an atomic operation, so that another node won’t process it. This is tricky in Cassandra (where lightweight transactions exist but are costly). The segment approach largely avoids parallel processing collisions, but not the case of a crash mid-way. We assume at-least-once and accept the possibility of a duplicate notification on rare occasions (with user impact being maybe two emails instead of one). We can mention in non-functional that we aim for at-least-once delivery and will handle duplicates in logic or apologize if they occur.

• Database Failures: If the database (Cassandra) goes down or a partition becomes unavailable, the scheduler might fail to fetch due tasks or update times. Cassandra is designed for high availability; with proper replication, one node down doesn’t make data unreachable. The scheduler should handle transient query failures by retrying after a short delay. In worst-case scenario of a multi-node outage, we might temporarily not send some reminders (which is serious). That’s why multi-region or multi-cluster replication can be considered: maybe have a standby in another data center that can take over if the primary DB cluster fails. That adds complexity (multi-active clusters and ensuring no double sends across regions, etc.). At least within one region cluster, high replication (RF=3) is used so that data is durably stored.

• Queue Failures: If our message queue (Kafka/Rabbit) is down, the scheduler might not be able to enqueue messages. We should design for the queue to be a reliable cluster as well. Kafka, for example, can survive broker failures if replicas are configured for partitions. If the queue is down entirely, the system effectively can’t deliver notifications. The scheduler could either buffer them (not ideal, as memory could overflow) or mark them in DB and try later. Perhaps if enqueue fails, we leave the schedule entry as still due (so it will be picked up again next scan) or move it to a “stuck” state. We would need to alert on this. A highly available queue mitigates this risk. Alternatively, use cloud managed queues with good SLAs.

• Channel Failures: If an external service (email/SMS provider) is down or returns errors, our channel services will catch that. We implement retry policies – e.g., exponential backoff and retry up to a limit. We also have to be careful not to lose messages. If a channel service crashes while processing a batch from the queue, in at-least-once mode the message will reappear or be reprocessed by another consumer (depending on the queue mechanism). We must ensure our processing is idempotent since a retry might happen by a different instance. For example, if Email service crashes after sending the email but before acknowledging the queue, the queue might redeliver that message – resulting in potentially a duplicate email if we don’t check. Idempotency keys or storing a “sent” status in a temporary store can help. But because email might not easily deduplicate on content, we might accept a rare duplicate.

• Ensuring No Notification Lost: To not lose notifications, every piece is durable: the schedule in DB is persistent, the queue is persistent (so once scheduler enqueues, it’s stored until delivered), and each channel attempt either succeeds or is logged for retry. If all retries fail, we log the failure. Perhaps we alert admin or even notify the user through an alternate path (“We couldn’t deliver your reminder via SMS, please check your phone or settings.”). That could be done via email or in-app as a fallback, but that’s optional.

• Data Consistency on Updates: When a user updates or deletes a reminder, there’s a potential race if that reminder is about to fire. We handle this by always checking latest data at execution time. For example, if the user deletes a reminder at 10:00 and it was supposed to fire at 10:01, what happens? If the deletion removes the schedule entry before 10:01, the scheduler will simply not find it and nothing will send – good. If the timing was such that the scheduler already fetched it from DB just before deletion, we need a safeguard: perhaps mark reminders as deleted in the main table, and the scheduler double-checks the Reminders table status before sending. If it sees a deleted flag, it should skip sending. This check ensures no notification goes out after deletion. For updates (like changed time or channels), since we update DB, the next occurrence time or channel info might be different. If a reminder was due very soon and scheduler picked it up with old info, it might send the old message. This is a narrow window; to avoid it, we could implement a short lock: e.g., when updating a reminder’s time, set a flag that a change is in progress or just ensure the schedule entry is updated immediately. Possibly simpler: require that changes to a reminder that is due imminently (say in next 1 minute) might not take effect immediately – but that’s not great UX. Ideally, our system is quick enough that update operations update the schedule table atomically such that even if due within seconds, the entry reflects the new time, meaning scheduler won’t pick the old time. Using lightweight transactions or careful order of operations (delete old schedule entry, then update main) can help achieve that atomicity.

By planning for these failure scenarios, the system will gracefully handle errors: no single failure will completely stop reminders; at worst some might be delayed, but they should eventually go out. Our design aims for high reliability, borrowing techniques used in large-scale systems (redundancy, idempotent processing, partitioning to limit blast radius, and thorough monitoring).