YouTube Likes Counter: Failure Modes and Operations

The consumer falls behind

Kafka holds every event until a consumer reads it. The gap between the newest event and the last one read is called consumer lag. A traffic spike, a slow database, or a restarted worker all make it grow.

The reader notices nothing at first, which is what makes this failure easy to miss. The cache is updated on the write path, so the visible number is still right. What falls behind is ContentStats, the durable copy.

That is fine for a few seconds and a problem after a few minutes. If the cache entry expires while ContentStats is behind, the next read serves the older number. The count appears to drop.

We repair this in two ways. First, we add consumer instances, up to the partition count. The limit exists because one partition feeds one consumer in a group.

Second, we set the entry's TTL, the time before it expires, longer than the worst lag we will accept. Do not set a fixed five seconds, because a fixed number ignores what the lag actually is.

A consumer crashes in the middle of a batch

When a consumer crashes, Kafka restarts it. The new instance replays from the last committed offset, the marker of how far the group had read. Events already applied arrive a second time.

This is the failure the versioned transitions from Step 8 exist for. The consumer has already applied version 7 for that user and item, so it drops the repeat. Nothing is double counted, and no extra coordination is needed.

One rule follows: commit the offset after the count update, never before. The order matters because the two mistakes are not equal. Replaying an event is harmless, but losing one leaves a wrong number that nobody notices.

A cache node fails

Redis holds the counts being served. When a node fails, its keys are gone.

The read path falls back to ContentStats and refills the cache from it. The rule that matters is that the fallback reads the durable count. A cache that refills from zero and starts counting again loses every reaction ever recorded for that video.

Two rules keep this failure small. We run replicas, so a failover, the switch to a standby copy, keeps the keys.

We also never let a refill write a value lower than the one already in the cache. A stale ContentStats read would otherwise push a visible count backwards.

The counts drift

Given enough time, the total in ContentStats will not match the rows in UserLikes. A dropped event, a bug in one transition, or a manual repair will each move it. We treat that drift as certain and build the repair before we need it.

The repair is a job that recounts. For one content item, it counts the matching rows in UserLikes. It compares that number to ContentStats and writes the counted value when they differ.

We run it two ways. It covers all content on a rolling schedule, slowly, so it costs little. It also runs on demand for one item, so support can correct a single video without waiting for the rotation.

This job is why the per user rows are worth their storage cost. They make every count recomputable. A design that keeps only totals has no way to answer the question "is this number right?".

A region fails

Writes are local to a region and replicate in the background. Losing one region therefore does not stop the others, because the UserLikes data they hold is already there.

What is lost is the events that had not replicated yet. Those reactions are missing until the region returns, and the reconciliation job corrects the counts once it does.

Say that plainly in an interview rather than claiming no data is ever lost. An honest account of what is missing is stronger than a promise you cannot keep.

What to watch

A repair only helps if someone notices the failure first. The table lists the six signals we watch and the point where each one demands action.

The two latency rows are the targets we set in Step 2. They are p95 targets, meaning 95 out of every 100 requests must finish inside the limit. A requirement nobody measures is not a requirement.

The whole design in one pass

A reaction arrives and becomes one conditional, versioned write to UserLikes. That write is the source of truth, and every count in the system can be rebuilt from it.

The service then updates the count in Redis, so the reader sees the change at once. It also publishes the transition to Kafka.

Consumers apply transitions in version order and keep ContentStats current in batches. Reads are served from Redis and fall back to ContentStats. A reconciliation job recounts from UserLikes and corrects any drift.

Every choice in that summary comes from a requirement in Step 2. Redis serves the 20ms read target. Kafka absorbs the write burst so the 100ms write target holds during a spike.

The versioned write is what makes eventual consistency safe rather than merely fast. Safe means a replay or a retry can never move a count twice.

💡 In the interview: Volunteer a failure before you are asked one. "The interesting failure here is the consumer falling behind, because the cache still looks right while the durable count does not. If the cache expires in that window, the number appears to drop. So I would tie the cache TTL to the lag I am willing to tolerate." Then offer the reconciliation job. It answers almost any follow-up about correctness, because the per user rows make every count recomputable. If they ask about a lost region, say the unreplicated reactions are missing until it returns and the recount fixes them. Do not claim zero loss in an AP system, one that chooses availability over consistency.

Key takeaway: This system is allowed to be briefly wrong, and that is the whole design. What it is not allowed to do is stay wrong without noticing. The cache absorbs the reads, and the queue absorbs the writes. The recount from the per user rows is what turns an eventually consistent count into one you can defend.