Step 8: Toggles and Idempotency

Alice has not reacted to a video. She clicks like, then dislike, then dislike again to remove it. The three clicks move the totals differently:

The change to each total, the delta, depends on the reaction before the click as well as the reaction after it, so a service that knows only the new reaction cannot compute it. The hard part is knowing that previous state when requests are duplicated, retried, or concurrent.

Why a plain increment fails

Suppose the handler writes the new reaction and then publishes {content_id, action: "like", delta: +1}. If Alice double-clicks, both requests write the same row, which is harmless, but both publish +1, and the total ends up one too high. A client retry after a lost response does the same.

The read-then-write race

The obvious repair is to read the current reaction first, compute the delta, write the new reaction, and publish. With two concurrent requests, both read "none", both compute +1, both write like, and the total is again one too high, while UserLikes shows nothing wrong. The window is a few milliseconds, but 11,500 writes per second hit it many times a day.

In Cassandra, this race is hard to see, because an upsert overwrites the row without reporting the previous value. The key prevents duplicate rows, not duplicate deltas. The read and the write must become one operation, as the diagram below shows.

Three ways to close the race

1. A conditional write that returns the old value.

A conditional write succeeds only if the row still holds what the writer expected. DynamoDB's UpdateItem reads and writes in a single call:

UpdateItem
  Key:                 {user_id, content_id}
  UpdateExpression:    SET action_type = :new, version = version + 1
  ConditionExpression: attribute_not_exists(version) OR version = :expected
  ReturnValues:        ALL_OLD
  ReturnValuesOnConditionCheckFailure: ALL_OLD

If the condition fails, another request changed the row first, and the second setting puts the current item in the failure response for the retry. If it succeeds, the response carries the old reaction, so the delta is exact, in one round trip and with no lock.

2. A lightweight transaction in Cassandra.

Cassandra offers the same guarantee as a lightweight transaction, written with an IF clause. The coordinating node runs a Paxos consensus round among the replicas, about four network round trips instead of one. It fits when the data already lives in Cassandra and each user writes rarely, and the extra latency is acceptable.

3. Publish the transition, not the delta.

We choose this option, used together with one of the two above. Each change to a UserLikes row increments its version, and the service publishes the whole transition:

{
  "content_id": "video_abc123",
  "user_id": "u_9931",
  "from": "none",
  "to": "like",
  "version": 7
}

This replaces the plain delta events from Step 7. The consumer records the last version applied for each user and item, ignores any version it has already seen, and turns each new transition into deltas:

DELTAS = {("none", "like"): (+1, 0),  ("like", "none"): (-1, 0),
          ("none", "dislike"): (0, +1), ("dislike", "none"): (0, -1),
          ("like", "dislike"): (-1, +1), ("dislike", "like"): (+1, -1)}

def apply(event):
    key = (event.user_id, event.content_id)
    if event.version <= applied_versions.get(key, 0):
        return                                # a replay: already counted
    likes, dislikes = DELTAS[(event.from_state, event.to_state)]
    pending[event.content_id].add(likes, dislikes)
    applied_versions.set(key, event.version)  # written with the batch, see below

A duplicate event now changes nothing, because the consumer skips a transition it has already applied. That makes the consumer idempotent: processing an event twice has the same result as processing it once.

The applied version and the count change must be saved together. If the worker saves the version and crashes before writing the count, the transition is lost; in the reverse order, a replay counts it twice. If the store can write both in one batch, the problem disappears; if not, the recount job in Step 10 corrects the small window that remains.

The idempotency key on the API

The Idempotency-Key header from Step 4 and the version protect different boundaries. The key is the client's statement that two requests are the same action, and the service answers the repeat with its stored result before any write happens. The version records what actually happened and stops a duplicate inside the pipeline. A production system uses both.

Why exactly-once delivery in Kafka does not solve this

Kafka can process a message exactly once, but only when the input, processing, and output all stay inside one Kafka cluster. The writes to UserLikes, Redis, and ContentStats happen outside it, so only idempotent consumers protect the whole path.

💡 In the interview: Describe the delta before drawing anything: "A like is a change of state from none to like, so the count moves by the difference between the two." Then name the race and close it: "The reaction write is conditional and returns the previous value. The row is versioned, I publish the transition, and the consumer applies each version once, so a retry has no effect." If the conditional write fails, retry with the value the failure returned.

Key takeaway: The per-user reaction row is the source of truth, and every total is derived from it. Only the state change itself has to be exactly right: make that write conditional and versioned, and every later stage can be replayed safely.