Step 10: Failure Modes and Operations

The consumer falls behind

Kafka keeps every event until a consumer reads it. The distance between the newest event in a partition and the last one the consumer group has processed is the consumer lag, and a traffic spike, a slow database, or a worker restart all increase it.

Viewers do not see lag directly, because the cache updates on the write path; only ContentStats falls behind. A lag of several minutes becomes visible when a cache entry expires: the next read loads the older total, and the count drops.

The first repair is more consumer instances, up to one per partition, since each partition is read by only one consumer in a group. The second is a cache TTL longer than the worst lag the system will accept.

A consumer crashes during a batch

A replacement consumer resumes from the group's last committed offset, the last position it confirmed, so events that were applied but not yet committed arrive a second time. The versioned transitions from Step 8 make this safe, because the consumer skips any version it has already applied.

This depends on one ordering rule: commit the offset after the totals are written, never before. Replaying an event costs nothing, while skipping one leaves a wrong total that nobody detects until the recount runs.

A cache node fails

When a Redis node fails without a replica, its keys are gone, and the read path refills them from ContentStats. The refill must start from the durable total, because a cache that restarted a key at zero would show a video with 1.5 million likes as having 12.

Running Redis with replicas keeps the keys through a failover, the automatic switch to a standby copy. A refresh also never lowers a value already in the cache, because a lagging ContentStats read would otherwise move a visible count backward.

The counts drift

Over months, the totals in ContentStats will stop matching the rows in UserLikes. A transition lost in the small window from Step 8, a bug in one release, or a manual correction each adds a small difference, so the design treats drift as certain.

The repair is a recount job. For one item, it counts the matching rows in UserLikes, compares the result with ContentStats, and writes the counted value when they differ. It runs as a slow rolling pass over all content, and on demand, so support staff can correct one video immediately. This job is what the per-user rows are stored for: a design that kept only totals could never answer "is this number right?".

A region fails

Writes complete in their local region and replicate in the background, so losing one region does not stop the others. What is lost is the last second or so of reactions that had not yet left the failed region. Those reactions are missing until the region recovers, and the recount job then corrects the affected totals. A design that chooses availability should state this loss openly rather than claim zero loss.

What to watch

These six signals cover the failures above.

The complete design

A reaction is one conditional, versioned write to UserLikes, plus a Redis increment and a Kafka event in the same request. Workers apply each transition once to ContentStats, reads come from Redis, and a recount job corrects any drift. Redis meets the 20 ms read target, Kafka keeps writes under the 100 ms target during a spike, and the versioned write lets the design accept eventual consistency without accepting wrong numbers.

💡 In the interview: Raise a failure before the interviewer asks: "The failure I would watch most closely is the consumer falling behind. The cache still looks right while the durable count is stale, and when an entry expires, the visible number drops. I would tie the cache TTL to the lag I am willing to accept." Then mention the recount job, which answers most follow-up questions about correctness.

Key takeaway: This system is allowed to be wrong for a few seconds, and that permission is what lets it scale. It is not allowed to stay wrong without anyone noticing. The cache serves reads, the queue absorbs writes, and the recount from the per-user rows turns an eventually consistent count into one you can trust.