YouTube Likes Counter

We use a REST API.

1. Cast Vote

• POST /v1/votes
• Request Body:

{
  "target_id": "video_abc123",
  "target_type": "video", // "comment"
  "action": "like"        // "dislike", "none" (remove)
}

• Response: 200 OK (Empty body).

2. Get Counts & State (Batch)

• GET /v1/votes/summary?target_ids=video_abc,comment_xyz
• Headers: Authorization: Bearer <token>
• Response:

{
  "items": {
    "video_abc": {
      "likes": 1500200,
      "dislikes": 4050,
      "user_state": "like"
    },
    "comment_xyz": {
      "likes": 45,
      "dislikes": 0,
      "user_state": "none"
    }
  }
}

Architectural Approaches

When designing a scalable like/dislike system, various architectural strategies can be employed. Each approaches has its pros/cons, lets look into these:

1. Synchronous Updates (Direct Write to DB and Counters Together)

How it works: Every user like/dislike action is immediately written to the database, updating both the individual user-action record and incrementing/decrementing the total like counter in a single, synchronous operation (often within a transaction). For example, clicking "like" will insert a like record and update the post’s like count column in the same request.

Pros:

• Strong consistency: The moment a user action is processed, the stored count reflects it for all users. The count in the database is always up-to-date immediately after each action, which means any read after a write will get the latest value.
• Immediate visibility: Users can see the new like/dislike count right away after their action, as the system doesn’t delay or defer the update. There’s no waiting for back-office processes – the data is consistent in real-time.
• Simplicity in implementation: It’s straightforward – write directly to the database. There are fewer moving parts (no queues, caches, or batch jobs needed), making the logic easy to understand and implement.

Cons:

• High write contention: Under heavy load (e.g., a viral post with many concurrent likes), the single counter field becomes a hot spot. Transactions will contend on the same row/record, causing lock waits or conflicts. This can severely throttle throughput as concurrency rises.
• Database bottleneck risk: The database must handle every like/dislike event and serve reads for counts. This increases load and can slow down as traffic grows. If the DB slows or goes down, the whole like system is affected (no buffering or alternative path). In essence, it doesn’t handle sudden surges or very large scale well without vertical scaling or replication.

Use Cases: This approach is best in scenarios with low to moderate traffic or where absolute consistency is paramount. For example, a small community forum or an internal application can use direct DB updates for simplicity. It’s also acceptable when the rate of likes/dislikes is low enough that the database can easily handle it. However, it becomes problematic at large scale (millions of likes) where the single counter update is the choke point.

2. Hybrid Approach (User Action Stored Immediately, Counts Updated Asynchronously)

How it works: In this approach, the application handles like/dislike actions asynchronously using a message queue (Kafka) to update counts in batches:

1. Immediate Write of Action: When a user likes or dislikes a content item, the individual action is recorded immediately in the database (e.g. inserting a row in a Likes table with user_id, item_id, action_type). This ensures the source of truth for individual actions is always up-to-date.
2. Publish Event to Kafka: Instead of updating the aggregate like/dislike count on that content item synchronously, the service publishes an event to a Kafka topic (e.g. “like_events”). The event contains details such as the item ID, whether it was a like or dislike, etc.
3. Kafka Consumers Aggregate Counts: One or more Kafka subscriber workers listen on the topic. These workers accumulate events and periodically update the total counts. For example, a worker might keep an in-memory counter for each item or buffer a batch of events. Workers can be configured to batch updates – for instance, after every 100 events or every X seconds, they will compute the new totals. This batching dramatically reduces write load on the primary database by coalescing many increments into a single update.
4. Batched Database Update: When a worker’s threshold is met, it writes the aggregated count to the database. This could mean updating a counter field in an “Items” or “Posts” table (e.g. incrementing the like_count by 100). The update can be done with an atomic operation (like SQL UPDATE ... SET like_count = like_count + 100) to avoid race conditions between batches. By processing in batches, the system can handle high throughput of likes/dislikes while keeping database load manageable.

Pros:

This Kafka-based asynchronous approach offers several benefits at large scale:

• High Throughput & Low Latency for Users: The user’s action is recorded quickly (just an insert and a publish, both of which are fast), and they are not blocked by expensive count calculations. The heavy lifting of updating the aggregate count is offloaded to background workers, allowing the system to absorb a very high rate of likes/dislikes.
• Reduced Load via Batching: By aggregating multiple events into one database update, we dramatically reduce write contention on the main counters. For example, 100 like events might translate to a single UPDATE query, which is much more efficient than 100 separate updates. This batching improves scalability of the database.
• Scalability and Decoupling: Using Kafka decouples the frontend action from the backend processing. We can scale out multiple consumer workers to handle increasing event volume without affecting the user-facing app. The system is also more loosely coupled – the app doesn’t need to know how counts are aggregated, it just fires an event.
• Fault Tolerance: Kafka’s design (with replicated logs and consumer groups) combined with the idempotent processing means the system can recover from worker crashes or downtime without losing events. Each like event is durably stored in Kafka until processed, and will be retried if a failure happens.

Cons:

• Eventual Consistency: The total like/dislike count in the database is not updated immediately at the time of the user action. There is a small delay (depending on batch frequency – maybe a few seconds or less) before the new like is reflected in the official count. During that window, a user might not see the most up-to-date count unless additional measures are taken (as addressed in the caching approach below). This is a classic trade-off of eventual consistency for higher performance.
• Complexity of System: The introduction of Kafka and asynchronous workers adds complexity. You need to manage a Kafka cluster and ensure consumers are working correctly. The code to handle batching, offset commits, and failure recovery is more complex than a simple synchronous update. There is more that can go wrong, so thorough monitoring and error-handling logic is necessary.
• Idempotency & Duplicate Handling: Ensuring exactly-once processing requires careful design. If not done properly, you could double-count likes (if a message is processed twice) or miss updates (if offsets are mismanaged). Implementing and testing the deduplication (such as maintaining a processed message store or using Kafka transactions) adds development overhead. Also, any deduplication store (like Redis or an extra DB table) must be maintained and can itself become a point of failure or performance bottleneck.

Use Cases: The hybrid approach is useful when you need to handle a decent volume of likes and want to avoid slowing down the user’s action, but you still want the reliability of logging every action. Many systems use this pattern in combination with slightly delayed updates. For example, an application might show counts that update every few seconds or on page refresh, which is acceptable in social apps where seeing the count “eventually” is good enough.

3. Asynchronous Count Updates with Kafka and Caching (Revised Third Approach)

How it works: This approach builds on the Kafka asynchronous update mechanism but adds a caching layer to provide instant feedback and fast reads. The steps are:

1. Immediate Write of Action to DB: Just like the previous approach, each like/dislike action is recorded as a separate entry in the database right away. This ensures durability and a trace of each user’s action.
2. Update Cache’s Count: The system then updates the cache for the total like/dislike count of that item immediately. For example, if a post had 50 likes in cache, and a new like comes in, the application or a caching layer will increment the cached count to 51. This gives real-time feedback – the next time someone fetches the like count (even the same user immediately after liking), they will see “51” from the cache without waiting for the backend aggregation. This cache is typically a fast in-memory store like Redis or Memcached. The update can be done with an atomic operation (e.g., Redis INCR) to handle concurrent updates safely – ensuring two simultaneous likes both get applied without losing one.
3. Publish Event to Kafka: In parallel, the service still publishes an event to the Kafka topic for likes (so the asynchronous pipeline is informed of the new like). The event will be used by background workers to eventually reconcile the persistent count in the database.
4. Kafka Workers Update Database: Kafka consumer workers operate similarly as in Approach 1: they consume like events in batches. The difference now is that these updates to the database’s aggregate count are somewhat redundant in the short term (because the cache already has the latest count), but they serve to persist the aggregated count for long-term consistency and as a fallback. Workers might batch 100 events and then do an SQL UPDATE posts SET like_count = like_count + 100 WHERE post_id=.... Over time (every few seconds or minutes), the database’s stored count catches up with what the cache shows. If the cache was updated for each like, the DB update should match that total after processing the batch.
5. Cache and DB Convergence: The cache entry for the count can be given a short TTL (Time To Live), say 5 seconds, or some small window. This means every few seconds, if no new updates happen, the cache will expire and the next read will fetch the count from the database (which by then should include all recent updates from the Kafka consumers). This helps correct any discrepancy that might have occurred between cache and database. Alternatively, the system can invalidate the cache or refresh it when the database is updated by the worker (a mini write-through on the aggregated count update).

Read Path: For any client retrieving the like/dislike count, the application will read from the cache. If the cache has a value (not expired), it returns that almost instantly. This ensures that users always see the most up-to-date count (including recent likes) with low latency, as long as the cache is being kept in sync. If the cache entry expired or is missing (cache miss), the service can fall back to the database: fetch the persisted count from DB (which might be slightly behind), return it, and repopulate the cache with that value (cache-aside pattern). However, because of the short TTL and continuous updates, such cache misses for a hot item would be rare. Essentially, the cache acts as the primary source for reads, with the DB as backup and long-term storage.

Pros:

The Kafka + Cache approach provides the best of both worlds – responsive updates and scalable processing:

• Real-Time User Experience: Users see their like/dislike actions reflected immediately in the count, thanks to the cache update. The interface feels instant and interactive, with no waiting for the backend to catch up. Reads hitting the cache are extremely fast (in-memory lookups), which is crucial for high-traffic scenarios where many users are viewing counts.
• Reduced Database Load on Reads: With the cache as the primary source for counts, read traffic to the database for these counts drops dramatically. This frees the database to handle other queries and reduces the chance of it becoming a bottleneck. The cache shields the database from reads, and the batched writes (from Kafka workers) are infrequent and efficient.
• Scalability and Throughput: Like the previous approach, writes are still handled asynchronously via Kafka, allowing the system to handle a massive stream of likes/dislikes. The addition of caching also means that even if the like count is requested very frequently (e.g., a viral post being liked and viewed by thousands of users), those requests don’t overwhelm the DB. The architecture can scale horizontally: more app servers to handle user actions (each updating cache and sending events), more Kafka partitions/workers to handle the firehose of events, and a distributed cache cluster to handle fast reads.

Cons:

While this approach is powerful, it adds more components to manage:

• Higher Complexity: There are now three moving parts (DB, Kafka, Cache) instead of two. Ensuring all three stay in sync and debugging issues can be challenging. Developers must handle cache invalidation properly (a notoriously hard problem) – e.g., if a bug prevents invalidation on a DB update, the cache could serve stale data for longer than intended. Understanding the interplay of TTL, cache updates, and Kafka timing requires careful design and testing.
• Cache Consistency Issues: Despite our strategies, there can be edge cases of inconsistency. For instance, if the Kafka consumers are delayed (say the system is briefly overwhelmed), the cache might carry “unconfirmed” increments for longer. If the cache TTL expires before the DB is updated, users might see the count jump down then up again, which could be confusing. In practice this window is small, but it’s a trade-off: we prefer showing a possibly temporary count increase (optimistic update) for responsiveness, accepting that in rare cases it might momentarily correct downward. Keeping the TTL very short minimizes this inconsistency window but too short a TTL could increase database reads (striking a balance is key).
• Cache Memory Overhead: Storing counts for many items in cache consumes memory. If the app has millions of content items, you wouldn’t cache all of them permanently – likely you cache the hot ones that are being actively liked/viewed. The short TTL helps avoid filling the cache with rarely-accessed items, but it means those items will cause a DB read occasionally when they expire. This is generally fine, but it’s a consideration (this is a balance between cache hit rate and staleness).

Use Cases: The revised third approach combines Kafka for scalable, reliable processing with caching for instant user feedback and fast reads. It handles failures by ensuring no single point (cache or worker) being down will permanently lose data or show wrong counts for long. The use of short TTLs, cache invalidation, and idempotent message processing collectively address the consistency challenges, making this approach suitable for large-scale systems where both performance and accuracy are required. Users get a snappy experience, and the system can handle the load and recover from issues gracefully, at the cost of a more complex architecture that must be carefully managed.