Stampede, Thundering Herd, Dogpile

3. Request Coalescing: One Recomputation, Not Two Thousand

The first mechanism attacks the pile-up directly. If 2,000 requests all need the same value, let exactly one of them compute it and have the rest wait for that result. This is usually called request coalescing or single-flight.

The shape is the same everywhere it is implemented.

1. A request misses and tries to acquire a lock on that key.
2. Whichever request wins the lock performs the recomputation and stores the result.
3. Every other request waits on that lock, then reads the value the winner stored.

Two decisions make the difference between a working implementation and a new outage.

Where the lock is held. A lock inside one application process coalesces only that process's requests. With twenty instances you have reduced 2,000 concurrent recomputations to 20, which is usually enough. A lock held in the cache itself, using an atomic set-if-absent with a short expiry, coalesces the whole fleet down to one. The cluster-wide version is stronger and adds a dependency on the cache for correctness rather than only for speed.

What the waiters do when the winner is slow. This is where coalescing creates its own failure. Two thousand requests now wait on one computation, so if it takes three seconds, all of them take three seconds, and the thread or connection each one holds is occupied for that time. Always put a timeout on the wait, and decide in advance what a timed-out waiter returns: an error, a default, or a stale value if one is still available.

What matters is that coalescing turns a load problem into a latency problem. That is almost always a good trade, and it is still a trade.

4. Refresh Before You Have To

Coalescing manages the miss. The next family of mechanisms prevents it, by refreshing the value while the old one is still valid.

Probabilistic early expiration. On every read, the reader decides whether to recompute even though the entry is still fresh. The probability is near zero early in the entry's life and rises as expiry approaches, so some reader almost always refreshes slightly early and nothing has to be coordinated. The published version of this scales the probability by how expensive the recomputation is, so costly values are refreshed earlier than cheap ones. The appeal is that it needs no lock, no background job and no list of hot keys. It is a few lines in the read path.

Serve stale while revalidating. Keep the expired value and continue serving it, while one request refreshes it in the background. Reads never block on a recomputation at all. The cost is clear: you have decided that serving a value slightly past its expiry is better than making anyone wait, which is a product decision rather than a technical one.

Scheduled refresh for a known hot set. If a small number of keys carry most of the traffic, refresh those on a timer regardless of reads. This gives the most predictable backend load of any option here, and it only works when you can identify the hot set and keep that list current.

These three differ in who pays. Probabilistic refresh charges a small number of unlucky readers. Serving stale charges correctness. Scheduled refresh charges you a background job and the risk that the list goes out of date.

5. Jitter, and What It Cannot Fix

Jitter means giving each entry a slightly different expiry rather than an identical one: instead of exactly 300 seconds, a random value between 270 and 330. It is one line of code and it is the change with the highest return in this lesson.

For a set of keys written together, the effect is straightforward. Ten percent of jitter on a 300-second expiry spreads expiries over a 60-second window, so the wave of misses becomes a trickle and backend load flattens. The same change breaks the synchronized repopulation after a deploy, and it stops a cache flush from scheduling its own repeat.

Now the part people get wrong, and it matters because the two cases look similar on a dashboard.

• Jitter solves many keys expiring together. It spreads a population of keys apart.
• Jitter does nothing for one hot key. A single key has a single expiry time. Moving that moment randomly does not change the fact that when it arrives, every concurrent request misses at once.

So jitter and coalescing are not alternatives. They address different problems that produce the same graph. A system with one very hot key needs coalescing or early refresh. A system with a hundred thousand keys written by one job needs jitter. A system with both needs both, which is the normal case.

6. The Senior Decision

These mechanisms compose, and the order to adopt them comes from cost rather than from sophistication.

1. Add jitter first. It is a one-line change, it cannot create a new failure mode, and it removes the entire class of synchronized expiry.
2. Add coalescing next, as a per-process lock before a cluster-wide one. Per-process usually reduces the pile-up by the number of instances, which is enough, and it adds no dependency.
3. Add early refresh for the small set of keys that are genuinely hot, once you know which ones those are. Probabilistic refresh if you want it automatic, scheduled refresh if you want it predictable.
4. Decide explicitly whether you may serve stale. Not as a fallback that appears during an incident, but as a documented decision with a bound on how stale.

The judgment being tested is whether you can tell the two underlying problems apart. "Add a lock" is the right answer for a hot key and the wrong answer for a hundred thousand keys expiring at once, where it would serialize your entire cache layer behind a lock that nothing contends for.

What the interviewer is scoring: this appears as "your cache is working, so why did the database stop responding at the same time every hour?" The weak answer raises the expiry time, which halves the frequency of the event and changes nothing about its size. The strong answer separates the two causes, naming jitter for synchronized expiry and coalescing or early refresh for a hot key. Then it attaches the arithmetic: at this request rate and this recomputation time, one miss window admits this many concurrent recomputations. Expect the follow-up "what happens to those waiters if the recomputation hangs?" Having a timeout and a documented answer for what a timed-out waiter returns is what distinguishes someone who has run this from someone who has read about it.