YouTube Likes Counter: Failure Modes and Operations

Step 10: Failure Modes and Operations

A design is finished when you can say what happens as each part of it stops. This step walks through the five failures this system really has, and what it does about each one.

The consumer falls behind

Kafka holds every event until a consumer reads it. The gap between the newest event and the last one read is called consumer lag. A traffic spike, a slow database, or a restarted worker all make it grow.

The reader sees nothing at first. The count in the cache is updated on the write path, so the visible number is still right. What falls behind is ContentStats, the durable copy.

That is fine for a few seconds and a problem after a few minutes. If the cache entry expires while ContentStats is behind, the next read serves the older number and the count appears to drop.

Two repairs. Add consumer instances, up to the partition count, because one partition feeds one consumer in a group. And keep the cache entry alive longer than the worst lag you are willing to accept, rather than a fixed five seconds.

A consumer crashes in the middle of a batch

Kafka restarts it and replays from the last committed offset. Events already applied arrive a second time.

This is the failure the versioned transitions from Step 8 exist for. The consumer has already applied version 7 for that user and item, so it drops the repeat. Nothing is double counted and no extra coordination is needed.

Commit the offset after the count update, never before. Replaying an event is harmless. Losing one leaves a wrong number that nobody notices.

A cache node fails

Redis holds the counts being served. When a node fails, its keys are gone.

The read path falls back to ContentStats and refills the cache from it. The rule that matters is that the fallback reads the durable count. A cache that refills from zero and starts counting again loses every reaction ever recorded for that video.

Run replicas so a failover keeps the keys. And never let a refill write a value lower than the one already in the cache. A stale ContentStats read would otherwise push a visible count backwards.

The counts drift

Given enough time, the total in ContentStats will not match the rows in UserLikes. A dropped event, a bug in one transition, or a manual repair will each move it.

The repair is a job that recounts. For one content item it counts the matching rows in UserLikes, compares that to ContentStats, and writes the counted value when they differ.

Run it two ways. On a rolling schedule across all content, slowly, so it costs little. And on demand for a single item, so support can correct one video without waiting for the rotation.

This job is why the per user rows are worth their storage cost. They make every count recomputable. A design that keeps only totals has no way to answer the question "is this number right?".

A region fails

Writes are local to a region and replicate in the background, so losing one region does not stop the others. The UserLikes data they hold is already there.

What is lost is the events that had not replicated yet. Those reactions are missing until the region returns, and the reconciliation job corrects the counts once it does. Say that plainly in an interview rather than claiming no data is ever lost.

What to watch

The two latency rows are the targets set in Step 2. A requirement nobody measures is not a requirement.

The whole design in one pass

A reaction arrives and becomes one conditional, versioned write to UserLikes. That write is the source of truth.

The service then updates the count in Redis, so the reader sees the change at once, and publishes the transition to Kafka. Consumers apply transitions in version order and keep ContentStats current in batches. Reads are served from Redis and fall back to ContentStats. A reconciliation job recounts from UserLikes and corrects any drift.

Every choice in that sentence comes from a requirement in Step 2. Redis serves the 20ms read target. Kafka absorbs the write burst so the 100ms write target holds during a spike. The versioned write is what makes eventual consistency safe rather than merely fast.

💡 In the interview: Volunteer a failure before you are asked one. "The interesting failure here is the consumer falling behind, because the cache still looks right while the durable count does not. If the cache expires in that window the number appears to drop, so I would tie the cache TTL to the lag I am willing to tolerate." Then offer the reconciliation job, because it is the answer to almost any follow-up about correctness: the per user rows make every count recomputable. If they ask about a lost region, say the unreplicated reactions are missing until it returns and the recount fixes them. Do not claim zero loss in an AP system.

Key takeaway: This system is allowed to be briefly wrong, and that is the whole design. What it is not allowed to do is stay wrong without noticing. The cache absorbs the reads, the queue absorbs the writes, and the recount from the per user rows is what turns an eventually consistent count into one you can defend.