0% completed
YouTube Likes Counter: Detailed Component Design
On This Page
Step 7: Detailed Component Design
- Synchronous updates: write the reaction and the count together
- Asynchronous counting with Kafka
- Hybrid: Kafka for the durable count, cache for instant reads
Which approach we choose, and why
Step 7: Detailed Component Design
There are three ways to keep a count at this scale. Each one gives something up, so we look at all three before choosing.
1. Synchronous updates: write the reaction and the count together
The first approach is the direct one. Every like or dislike goes to the database at once, and one operation, often one transaction, does two things. It stores the user's action, and it moves the total count up or down by one. Clicking "like" inserts a like row and updates the post's like count column in the same request.
The strengths follow from that single write. The count is strongly consistent: the moment an action is processed, the stored count includes it. Any read after a write sees the latest value. The user sees the new count right after clicking, because nothing waits on a later job. And it is simple to build. Writes go straight to the database, with no queues, caches, or batch jobs, so the logic is easy to follow.
The costs appear under load. The first is high write contention. When a viral post takes many likes at once, every reaction updates the same counter row. Transactions wait on one another for that row, or conflict and retry. So throughput falls as more requests arrive at the same time, which is exactly when it is needed most.
The second cost is that the database is the limit. It must take every like and dislike event and also serve every count read. As traffic grows, its load grows and it slows down. If it slows or goes down, the whole like system goes with it, because there is no buffer and no other path. A sudden surge or a very large scale needs a bigger machine or replication.
When to use it. This approach fits low to moderate traffic, or a case where an exact count matters above all else. A small community forum or an internal tool can update the database directly and stay simple. It also works when reactions arrive slowly enough for the database to keep up. At large scale, with millions of likes, the single counter update is the point where everything waits.
The diagram below shows this first approach.
2. Asynchronous counting with Kafka
The second approach records each action at once but does not update the total in the same request. Instead it sends the change through a message queue, Kafka, and workers update the counts in batches. The flow has four steps.
- Write the action at once. When a user likes or dislikes an item, the service inserts the row in
UserLikesright away, withuser_id,content_id, andaction_type. So the record of individual actions is always current. - Publish an event to Kafka. Instead of updating the item's total in the same request, the service publishes an event to a Kafka topic, like
like_events. The event holds the item id, whether it was a like or a dislike, and similar details. - Kafka consumers accumulate counts. One or more worker processes subscribe to the topic. They collect events and update the totals from time to time. A worker might keep a counter per item in memory, or hold a batch of events. It can be set to flush after every 100 events or every few seconds. Many increments become one update, which cuts the write load on the database.
- Write the batched totals. When a worker reaches its threshold, it writes the totals it has accumulated into
ContentStats. A batch of 100 events is not 100 changes to one item. It is a smaller number of changes spread over many items. So the worker groups the batch bycontent_idand writes one update per item. Batching turns a write per reaction into a write per item, which is the reason the database can keep up.
At large scale, this design has four strengths. The user's action is an insert and a publish, and both are quick. The user never waits for a count to be computed, because that work moves to background workers. So the system can take a very high rate of likes and dislikes.
Batching cuts the load. Many events become one database update per item, so the counter rows see far less contention. 100 like events become a few updates, one per item, instead of 100 separate ones. The database scales further as a result.
Kafka also separates the user's action from the counting work. We add consumer workers as event volume grows, without touching the user-facing service. The service does not need to know how counts are aggregated. It only publishes an event.
The pipeline also survives failures. Kafka keeps a replicated log, and consumers work in groups. Each like event stays in Kafka until it is processed, and it is retried after a failure. Add idempotent processing, meaning that applying an event twice has the same effect as applying it once. Then a worker crash or an outage loses no events.
There are three costs. The first is eventual consistency: the stored total is not updated at the moment of the action. There is a short delay, set by the batch frequency, of a few seconds or less. In that window a reader may not see the newest count. This is the usual trade: a slightly late count in exchange for much higher throughput. Approach 3 below adds a way around this.
The second cost is more parts to run. Someone has to run a Kafka cluster and keep the consumers healthy. The code for batching, offset commits, and recovery is harder than a plain synchronous update. More can go wrong, so monitoring and error handling matter more.
The third cost is duplicates. Processing each event exactly once takes careful design. Done wrong, a message processed twice double counts a like, and a mismanaged offset misses an update. Deduplication takes work to build and test, whether it is a store of processed message ids or Kafka transactions. That store, in Redis or an extra table, must itself be run, and it can fail or become slow.
When to use it. This approach fits a large volume of likes where the user's action must not slow down, but every action still has to be logged. Many systems accept a count that updates every few seconds or on refresh. In a social app, a count that is right "eventually" is good enough.
The diagram below shows this second approach.
3. Hybrid: Kafka for the durable count, cache for instant reads
The third approach keeps the Kafka pipeline from approach 2 and adds a cache in front of the counts. The cache gives instant feedback and fast reads. The flow has five steps.
- Write the action to the database at once. As in approach 2, each like or dislike is recorded as its own row right away. That gives durability and a record of each user's action.
- Update the count in the cache. The service then updates the cached total for that item in the same request. If a post shows 50 likes in the cache and a new like arrives, the cached count becomes 51. The next read sees 51 from the cache without waiting for the pipeline. That includes a read by the same user a moment after clicking. The cache is a fast in-memory store like Redis or Memcached. The update is an atomic operation, like Redis
INCR. Two likes arriving at the same time both count, and neither is lost. - Publish an event to Kafka. In parallel, the service still publishes the event to the likes topic. Background workers use it to bring the durable count in the database up to date.
- Kafka workers update the database. The workers run as in approach 2 and consume events in batches. In the short term their updates repeat what the cache already shows. Their job is to persist the total for the long term and to serve as the fallback. A worker takes a batch of 100 events and writes one update per item into
ContentStats. Every few seconds or minutes, the stored count catches up with the cache. If the cache was updated for each like, the two match once the batch is applied. - Cache and database converge. The cache entry can be given a short TTL (time to live), say 5 seconds. TTL is the time after which an entry expires on its own. If no update arrives in that window, the entry expires. The next read fetches the count from the database, which by then holds the workers' recent updates. That corrects any gap between cache and database. Or the worker can invalidate or refresh the cache entry when it writes the database. Then the cache and the database are written together.
The read path. Any client that wants a count reads the cache. If the entry is there and not expired, the answer comes back almost at once. So readers see the current count, including recent likes, with low latency. On a miss, because the entry expired or was never there, the service falls back to the database. It reads the stored count, which may be slightly behind, returns it, and writes it back into the cache. That is the cache-aside pattern: the application fills the cache on a miss. With a short TTL and constant updates, a hot item rarely misses. The cache is the main source for reads. The database is the backup and the long-term store.
Three strengths follow. The count moves at once: the user's like shows immediately, because the cache was updated on the write path. The interface feels instant, and reads from the cache are in-memory lookups, which is what a very high read rate needs.
Database reads drop sharply, because the cache serves the counts. That leaves the database free for other queries and keeps it from becoming the limit. The only count writes it sees are the batched ones from the workers, which are few and cheap.
And both paths scale. Writes still go through Kafka, so the system takes a very large stream of reactions. A count that is read very often, like a viral post seen by thousands of users, does not reach the database. Each part scales by adding more of it: more app servers, more Kafka partitions and workers, and more cache nodes.
The first cost is that there are now three systems that must agree: a database, Kafka, and a cache. Keeping them consistent and debugging problems across them is harder. Cache invalidation must be right. If a bug stops an invalidation after a database update, the cache serves stale data for longer than intended. The way TTL, cache updates, and Kafka timing interact needs careful design and testing.
The second cost is that the cache and the database can disagree. If the consumers fall behind, for example when the system is briefly overloaded, the cache holds increments the database has not yet confirmed. If the entry then expires before the database is updated, a reader sees the count drop and then rise again. In practice that window is small. It is the trade we take: show the new count at once, and accept a rare, brief correction downward. A very short TTL narrows the window, but too short a TTL sends more reads to the database. The TTL balances the two.
The third cost is cache memory. With millions of items, we cache the ones being liked and viewed now, not all of them. The short TTL keeps rarely read items from filling the cache. Those items cost one database read when they expire and are asked for again. That is fine, and it is the usual balance between hit rate and staleness.
When to use it. This approach combines Kafka for reliable processing at scale with a cache for instant feedback and fast reads. No single failed part, cache or worker, loses data or shows a wrong count for long. Short TTLs, cache invalidation, and idempotent message processing together handle the consistency problems. It fits large systems that need both performance and accuracy. Users get a fast experience, and the system takes the load and recovers from failures. The cost is more parts to run.
The diagram below shows this third approach.
Which approach we choose, and why
We choose approach 3. The reaction is written first, the cache is updated at once, and Kafka carries the change to the workers that keep the durable count.
Two numbers decide it.
The read to write ratio from Step 3 is about 100 to 1. That makes the read path the thing to protect. Only a cache in front of the counts meets the 20ms target at 250,000 reads per second. Approach 2 has no cache, so every count read reaches the database.
Peak writes reach 11,500 per second, with bursts above 20,000. A synchronous update cannot absorb that, because every reaction to one popular video contends on the same row. Kafka takes the burst. The workers then apply the changes in batches, at a rate the database can accept.
What we give up is a count that is always exact. For a few seconds the cache and the durable count can disagree, and a reader can watch a number correct itself. Step 2 accepted that when it chose eventual consistency for popularity metrics. Nobody makes a decision based on the exact like count of a video.
Approach 1 is still the right answer at a smaller scale. If the question sets the load at a few hundred reactions per second, use it and say why. The extra parts in approach 3 are only justified above the point where one row becomes the limit.
| 1. Synchronous | 2. Kafka only | 3. Kafka and cache | |
|---|---|---|---|
| Write path | one transaction per reaction | append to a log | append to a log |
| Read path | database | database | cache, database behind it |
| Count freshness | exact | seconds behind | current in the cache |
| Breaks at | one hot row | 250,000 reads per second | nothing at this scale |
| Operating cost | none | a queue to run | a queue and a cache |
💡 In the interview: Do not present three options and stop. Interviewers read that as an inability to decide. Name the two you are rejecting, and say why in one sentence each. "Synchronous fails because every write shares one row. Kafka alone still sends every read to the database." Then commit: "I am taking the third, and the cost is a count that can be a few seconds stale." Naming the cost is what makes it read as a decision rather than a preference. If they set the scale lower, switch to approach 1 without hesitating. Matching the design to the load is the skill being tested.
Next: Step 8, which closes the toggle and retry problem this design still leaves open.
Reading Progress
0%
On This Page
Step 7: Detailed Component Design
- Synchronous updates: write the reaction and the count together
- Asynchronous counting with Kafka
- Hybrid: Kafka for the durable count, cache for instant reads
Which approach we choose, and why