10 Must-know System Design Failure Modes

Most system design interviews reward you for explaining how a system works when everything goes right. The strongest answers explain what happens when it does not.

Distributed systems do not fail in infinite ways. They fail in a small number of recognizable patterns, and the same ones keep appearing in postmortems at every company. Once you can name a pattern, you can design against it before the interviewer has to ask, which is exactly the signal that separates a mid-level answer from a senior one.

None of these need exotic infrastructure to hit. Most of them show up the first time a system carries real traffic, which is why interviewers treat familiarity with them as a proxy for whether you have actually operated something in production rather than only drawn it.

Here are ten failure modes worth knowing cold, with the fix that experienced engineers reach for in each case.

1. Single Point of Failure

A single point of failure is any component on the request path with no redundancy: one load balancer, one primary database, one message broker, one auth service. When it dies, everything behind it becomes unreachable, even though those instances are perfectly healthy.

The fix is redundancy plus automatic failover. Run at least two instances of everything on the critical path, put health checks in front of them, and spread those instances across availability zones so one datacenter problem cannot remove an entire tier. Failover also has to be tested, because a standby that has never taken traffic is a guess, not a plan. When you sketch an architecture on the whiteboard, trace each request path and ask which single box, if deleted, stops all traffic. That box is the one to talk about.

2. Cascading Failure

One slow dependency rarely stays contained. When a database slows from 20ms to 5 seconds, the service calling it holds its threads open waiting. Those threads are a fixed pool, so it stops answering its own callers, which then block too. The slowness travels backwards up the call chain until the whole system looks down, even though only one component actually degraded.

The fix is containment. Set aggressive timeouts so a caller gives up rather than waits forever. Add circuit breakers that stop sending traffic to a dependency that is already failing. Use bulkheads to give each dependency its own connection pool, so one sick downstream cannot consume every thread the service owns.

3. Retry Storm

Retries are usually good. They stop being good when everyone retries at the same moment. A service comes back after a blip, and every client that failed during the outage retries at once, often three times each. The recovering service now sees several times its normal load, falls over again, and generates a fresh round of retries. The retry logic becomes the outage.

The fix is exponential backoff with jitter, so clients spread their attempts across a window instead of synchronizing on the same second. Add a retry budget that caps retries as a percentage of total traffic, and a circuit breaker so clients stop retrying a service that is clearly down.

4. Cache Stampede

Caches hide how much load your database really receives. When one popular key expires, every request that would have been a hit becomes a miss at the same instant. All of them run the same expensive query against the database, which suddenly takes the full read volume it has been shielded from for months.

The fix is to make sure only one caller does the work. Request coalescing (also called single-flight) lets the first miss recompute the value while the rest wait on that result. Refreshing hot keys shortly before they expire keeps them warm, and adding jitter to TTLs stops thousands of keys from expiring together.

5. Hot Partition

Sharding assumes traffic spreads evenly across shards. Real traffic does not. Pick a low-cardinality shard key such as country, tenant, or day, and one shard receives most of the writes while the others idle. That shard saturates first, and adding more shards does not help because the skew follows the key, not the machine count.

The fix starts with the key. Choose something high-cardinality and evenly distributed, such as a hash of the user ID. For celebrity entities that are hot no matter what, salt the key into several sub-partitions, or give those entities dedicated capacity so the rest of the system is not shaped around them. Track per-shard metrics rather than cluster averages, since an average hides the one node that is on fire.

6. Replication Lag

Read replicas make reads cheap, but they are always slightly behind the primary. Under load that gap grows from milliseconds to seconds. A user updates their profile, the write commits on the primary, the next read lands on a lagging replica, and the change appears to have vanished. Nothing is broken, and the bug report is still real.

The fix is to be explicit about which reads can tolerate staleness. Route read-your-writes traffic to the primary for a short window after a write, or track the replication position and only serve a read from a replica that has caught up past it. Analytics and feed reads can safely stay on replicas.

7. Duplicate Processing

Any retried request may execute twice. The first attempt reaches the server and succeeds, the response is lost to a timeout, and the client retries the same operation. For a read that is harmless. For a payment, an email send, or an inventory decrement, the user is charged twice and the ledger is wrong.

The fix is idempotency. Have the client attach a unique idempotency key to each logical operation, and have the server record that key with the result. When a request arrives carrying a key it has already processed, the server skips the work and returns the original response. Same request, same outcome, any number of attempts.

8. Queue Backlog

Queues absorb bursts, which is exactly why they hide problems. If producers publish faster than consumers process, even by a small margin, the backlog grows continuously. Every health check stays green while the age of the oldest message climbs from seconds to hours, and by the time anyone notices, the data being processed is badly out of date.

The fix is to make the imbalance visible and bounded. Alert on consumer lag and oldest-message age rather than queue length alone. Autoscale consumers on that lag. Bound the queue so it applies backpressure, and shed low-value work under sustained overload instead of accepting everything and falling further behind.

9. Poison Message

One malformed message can stop an entire queue. The consumer picks it up, throws, and never acknowledges it. The broker redelivers it, the consumer throws again, and the loop repeats forever. Throughput drops to zero while every valid message behind the bad one waits, and the crash loop can take the consumer group down with it.

The fix is to give bad messages somewhere to go. Cap delivery attempts, then move the message to a dead-letter queue so the consumer can continue. Alert when anything lands in the dead-letter queue, and keep enough context on the message to fix the bug and replay it later.

10. Split Brain

During a network partition, two nodes can both conclude they are the primary. Each accepts writes from the clients that can still reach it, and the two copies of the data diverge. When the partition heals there is no correct way to merge them, because both sides accepted writes that assumed they were authoritative.

The fix is to make leadership require a majority. Quorum-based writes ensure only the side holding more than half the nodes can commit. Leader election through a consensus protocol such as Raft prevents two leaders from being elected in the same term, and fencing tokens let storage reject writes from a leader whose lease has already expired.

How to use this in an interview

You do not need to recite all ten. You need to reach for the right one at the right moment. When you propose a cache, mention stampede protection. When you propose sharding, name your shard key and say why it will not go hot. When you add retries, say backoff and jitter in the same breath. When you add replicas, state which reads can tolerate lag.

That habit is what interviewers are listening for. Anyone can draw boxes. Senior engineers explain how the boxes behave on the worst day of the quarter.

If you want to practice this systematically, Grokking the System Design Interview walks through these trade-offs inside full design problems, and System Design Patterns covers the underlying patterns in depth. For the fundamentals behind them, start with Grokking System Design Fundamentals.