Grokking the System Design Interview, Volume II
Vote

0% completed

YouTube Likes Counter: Toggles and Idempotency

Step 8: Toggles, Undo, and Idempotency

Why a plain increment is wrong

The read then write race

Three ways to close the race

The idempotency key on the API

Why exactly-once delivery in Kafka is not the answer

Step 8: Toggles, Undo, and Idempotency

A like is not an increment. It is a change of state. Watch Alice to see the difference.

Alice has not reacted to a video. She clicks like, so the like count goes up by one. Then she clicks dislike, and now two counts move at once. The like count goes down by one, and the dislike count goes up by one. She clicks dislike again to undo it. The dislike count goes down by one, and nothing replaces it.

The number the system publishes is a delta, meaning how far each count moves. The delta depends on what the reaction was before, so a service that knows only the new reaction cannot work it out. Everything in this step exists to learn that earlier state safely.

This is the part of the design an interviewer will test hardest. It is also the part Step 7 left open.

Why a plain increment is wrong

Suppose the handler does two things. It writes the new reaction to UserLikes, then it publishes {content_id, action: "like", delta: +1}. That looks reasonable, and two failures follow from it.

A double click inflates the count. Alice clicks like twice within a second. Both requests write the same row, so UserLikes is still correct. But both publish +1, so the total ends up one too high.

A retry inflates the count. The server records the reaction, and the response is lost on the way back. The client retries, and the same thing happens again.

Both failures have one cause: the delta was decided without knowing the previous state. Fix that one decision and both failures disappear.

The read then write race

The obvious repair is to read first:

  1. Read the user's current reaction for this item.
  2. Compare it to the new one and work out the delta.
  3. Write the new reaction.
  4. Publish the delta.

This is correct while one request runs at a time. It is wrong the moment two run together. Both read "no reaction", both work out +1, and both write like. The count is one too high, and UserLikes shows nothing wrong.

The window between the read and the write is small. But at 11,500 writes per second, small windows are hit often.

There is a second trap here. A write to Cassandra is an upsert, meaning it inserts the row or overwrites it without looking. It has no unique constraint, and it does not report what was there before. So the row itself does not close the race. Something has to read and write as one step. The diagram below shows the race, and the write that closes it.

The same four steps twice. Reading and then writing lets two clicks agree on the wrong delta. A conditional write hands back the old reaction, so the delta is known and the transition can be replayed safely.
The same four steps twice. Reading and then writing lets two clicks agree on the wrong delta. A conditional write hands back the old reaction, so the delta is known and the transition can be replayed safely.

Three ways to close the race

Each option below turns the read and the write into that one step. Take them in order, because the third is layered on one of the first two.

1. A conditional write that returns the old value.

A conditional write succeeds only if the row still holds what the writer expected. DynamoDB's UpdateItem does the read and the write in a single call:

UpdateItem
  Key:                 {user_id, content_id}
  UpdateExpression:    SET action_type = :new, version = version + 1
  ConditionExpression: attribute_not_exists(version) OR version = :expected
  ReturnValues:        ALL_OLD

If the condition fails, another request got there first, and you retry using the value returned with the failure. If it succeeds, the response carries the old reaction, so the delta is exact. The whole exchange costs one round trip, and no lock is held.

2. A lightweight transaction in Cassandra.

Cassandra calls a conditional write a lightweight transaction, written as an IF clause. It works, and it has a real cost. Before the write, the coordinator runs a consensus round, a vote among the replicas. That is roughly four round trips instead of one.

At 11,500 writes per second that cost matters. Choose this option when you are already on Cassandra and one user's write rate is low. If you propose it, state this cost as well.

3. Publish the transition, not the delta.

This is the option we choose, and it sits on top of one of the two above. The service writes the new reaction with a version that goes up by one on every change. It then publishes the whole transition:

{ "content_id": "video_abc123", "user_id": "u_9931", "from": "none", "to": "like", "version": 7 }

The consumer remembers the last version it applied for each user and item. An event carrying a version it has already applied is dropped. A new event is applied as from to to. Here that moves the like count by +1 and leaves the dislike count alone.

A duplicate event now changes nothing. That is not because the pipeline promises to deliver each message once. It is because applying the same transition twice gives the same result as applying it once. An operation with that property is called idempotent. Idempotency is the only kind of correctness that survives a retry.

Here is how the three compare:

OptionCostChoose it when
Conditional write returning the old valueone round tripthe store supports compare and set, such as DynamoDB
Lightweight transactionabout four round tripsyou are already on Cassandra and per user writes are rare
Versioned transitionsone extra field plus consumer statealways, layered on one of the two above

The idempotency key on the API

Step 4 puts an Idempotency-Key header on POST /v1/votes. The client makes one value per user action and reuses it on every retry of that action.

The service stores the key with its result for a short window, like 24 hours. A repeat of a key it has already seen returns the stored result and does nothing else. That stops a network retry from becoming a second state change before the version logic is ever reached.

The key is the client's statement that two requests are the same action. The version is the server's record of what actually happened. A production system wants both, because the key stops a duplicate before any write and the version catches one inside the pipeline.

Why exactly-once delivery in Kafka is not the answer

Kafka can deliver a message exactly once between a producer and a consumer inside one Kafka cluster. The guarantee holds inside that cluster and nowhere else.

It does not cover the write to UserLikes, the update to the cache, or a consumer writing to a store outside Kafka. Those are separate systems with their own failures. Idempotent consumers work across all of them. That is why they are the general answer and exactly-once delivery is not.

💡 In the interview: Say the delta out loud before drawing anything. "A like is a state change from none to like, so the count moves by the difference between them." Then name the race and close it. "I will make the reaction write conditional so it returns the previous value. I will version the row and publish the transition rather than a bare increment. The consumer applies each version once, so a retry costs nothing." Two follow-ups usually come next. If they ask what happens when the conditional write fails, say you retry with the value the failure returned. If they ask why not rely on exactly-once delivery, say it only covers the hop inside Kafka. It does not cover the stores on either side of it.

Key takeaway: The count is not the source of truth. The per user reaction row is, and every count in the system is a total derived from those rows. So the only thing that has to be exactly right is the state change itself. Make that one write conditional and versioned, and everything downstream can be replayed safely.

Next: Step 9, which scales this design to the numbers from Step 3.

Reading Progress

0%


Vote for new content

On This Page

Step 8: Toggles, Undo, and Idempotency

Why a plain increment is wrong

The read then write race

Three ways to close the race

The idempotency key on the API

Why exactly-once delivery in Kafka is not the answer