System Design Fundamentals

0% completed

Resilience and Error Handling

On This Page

Failures Are Normal

Cascading Failures

Fault Tolerance

Timeouts

Retries and Backoff

Circuit Breakers

Graceful Degradation and Fallbacks

Bulkheads

Error Handling and Reporting

Chaos Engineering

Key Takeaways

Practice Questions

An online store shows product reviews from a separate reviews service. One afternoon, the reviews service becomes slow. Each call to it now takes 30 seconds instead of 50 ms.

A few minutes later, the whole store stops loading. Even the checkout page fails, and checkout does not use reviews at all.

This lesson answers two questions. How does a system keep working when some of its parts fail? And how do we stop one failure from spreading to everything else?

Failures Are Normal

In a large system, some part is always failing. Disks break, machines restart, networks lose messages, and services become slow.

The numbers show why. Suppose one server fails about once every 1,000 days. A company with 1,000 servers then sees about one server failure every day. So a large system must be designed to expect failure, not to avoid it completely.

Resilience is the ability of a system to keep working when parts of it fail, and to recover quickly afterwards.

Cascading Failures

The online store had a cascading failure, which is a failure in one part that spreads to other parts that were healthy.

Here is how it happened. Each web server had 200 threads to handle requests. Every product page called the reviews service and waited. Each wait now took 30 seconds, so the threads filled up with waiting requests. Soon no free thread was left for anything, including checkout.

A slow reviews service keeps every web server thread waiting, so checkout fails too and the whole store goes down

The reviews service was only slow. The store failed because nothing stopped the waiting. The rest of this lesson covers the techniques that prevent this.

Fault Tolerance

Fault tolerance is the ability of a system to keep functioning correctly when faults or failures are present.

It comes from redundancy at several levels.

Data. Replication keeps copies of the data on several machines, so losing one copy loses no data.
Services. Several instances of each service run behind a load balancer, so one instance can fail without an outage.
Nodes. Spare machines can take over the work of a machine that fails.

Sharding also helps. When data is split across shards, a failure affects only the users of one shard, not everyone. The Availability lesson explains redundancy and failover in detail.

Redundancy protects against a part that stops completely. But many failures are slow, partial, or temporary. The next techniques handle those.

Timeouts

A timeout is the longest time a caller waits for a response. When the time runs out, the caller stops waiting and treats the call as failed.

Without a timeout, a caller can wait for a very long time. That is exactly what happened in the online store.

Pick a timeout based on the normal latency of the call. Suppose the reviews service's p99 latency is under 200 ms. A timeout of 1 second then allows for an occasional slow call, and it still frees the thread long before 30 seconds.

Both extremes cause problems.

Too short. Calls that would have succeeded are cut off and counted as failures.
Too long. Threads stay busy waiting, and a slow dependency can still cause a cascading failure.

Retries and Backoff

Many failures are transient, which means they last only a short time. Examples are a brief network problem, a single timeout, or a service that takes a few seconds to restart. A second attempt often succeeds.

A retry strategy automatically tries a failed operation again. Backoff increases the delay between attempts.

Exponential backoff doubles the wait after each failure. For example, the client waits 100 ms before the first retry, then 200 ms, then 400 ms. It stops after a fixed number of attempts, like 3 or 4.

Each failed attempt doubles the wait before the next one, and the fourth attempt succeeds

Why the delay matters. Suppose a service is struggling, and 10,000 clients call it. Each failed client retries 3 times immediately, without waiting. The service now receives up to 40,000 requests instead of 10,000. The retries become the extra load that stops it from recovering. Spacing the retries out spreads this extra load over time, so a failing system has time to recover.

Good retries follow three more rules.

Add jitter. Jitter is a small random amount added to each wait. Without it, thousands of clients that failed together also retry together, at the same moment.
Retry only transient errors. A timeout or an HTTP 503 (service unavailable) may succeed on the next try. An HTTP 400 (bad request) fails the same way every time, so retrying it only wastes work.
Make retried operations idempotent. An operation is idempotent when doing it twice has the same effect as doing it once. Reading a user's profile is idempotent. Charging a card is not, unless you make it so.

To make a payment idempotent, the client sends a unique idempotency key with the request. The server remembers the keys it has already processed. If the same key arrives again, the server returns the first result instead of charging the card again.

Circuit Breakers

Retries help with short failures. But when a service is down for minutes, calling it again and again only makes callers wait and adds load.

A circuit breaker watches the calls to a dependency. When too many calls fail, it stops calling the dependency for a while. During that time, calls fail immediately, without waiting. This is called failing fast.

A circuit breaker has three states.

Closed. Calls go through normally, and the breaker counts failures. For example, it opens when more than 50 percent of calls fail within 10 seconds.
Open. Calls fail at once, without contacting the dependency. After a wait, like 30 seconds, the breaker moves to half-open.
Half-open. The breaker lets a few test calls through. If they succeed, it closes again. If one fails, it opens again.

A circuit breaker opens after too many failures, waits, tests the dependency in half-open, and closes when test calls succeed

A circuit breaker helps both sides. The caller stops wasting threads on calls that will fail. The struggling service gets time to recover, because new requests stop arriving.

The name comes from the electrical circuit breaker in a building. It cuts the power when something goes wrong, so the problem does not spread.

Graceful Degradation and Fallbacks

Graceful degradation is the ability to keep providing limited functionality when some parts of a system fail. Instead of shutting down completely, the system keeps serving requests with fewer features.

By default, many systems do the opposite. One dependency fails, and the whole page returns an error.

The main tool is a fallback, which is a backup answer used when a call fails. Common fallbacks are listed below.

Hide the feature. If reviews fail, the product page loads without the reviews section.
Use a simpler answer. If personal recommendations fail, show the store's best sellers instead.
Use cached data. If a currency service fails, show the last known exchange rate, with the time it was updated.
Use a default value. If the settings service fails, use the standard settings.

When the reviews and recommendations services fail, the product page still loads, hides reviews, and shows best sellers instead

A product page without recommendations is a worse page. A product page that does not load is an outage.

Timeouts, circuit breakers, and fallbacks work together. The timeout stops the waiting. The circuit breaker stops calls that are likely to fail. The fallback decides what the user sees instead.

Bulkheads

A bulkhead is a separate pool of resources for each dependency, so that one failing dependency can use only its own share.

The name comes from ships. A ship is split into sealed sections by walls called bulkheads. If one section fills with water, the other sections stay dry, and the ship does not sink.

Here is how it would have saved the online store. The web server has 200 threads. The team allows reviews calls to use at most 20 of them. When the reviews service becomes slow, only those 20 threads wait. The other 180 threads keep serving checkout and every other page.

Bulkheads protect against one slow dependency using up a server's own resources. Rate limiting protects against too much traffic from outside, by capping how many requests a client can send. The rate limiting chapter covers it.

Error Handling and Reporting

A resilient system handles errors on purpose. It does not hide them.

Sort errors into two kinds. A transient error, like a timeout, may succeed on retry. A permanent error, like invalid input or a missing record, will fail again. Retry the first kind. Fail fast on the second, and return a clear message.

Never hide an error. Code that catches an error and does nothing makes a failure invisible. The system then looks healthy while users see wrong results.

Record errors in a consistent way. Log every error with its category, the service name, and the trace ID of the request. Count errors as metrics, and create alerts when they matter. The monitoring and observability lesson explains these tools. Consistent error records let the team diagnose a failure before a user has to report it.

Handle messages that keep failing. In a message queue, a message that fails many times can block other work. Systems move it to a dead letter queue, which is a separate queue where engineers can inspect failed messages later.

Chaos Engineering

Chaos engineering is the practice of injecting failures into a system on purpose, to test its resilience and find its weaknesses.

Every technique in this lesson is a claim about what happens during a failure. A claim that nobody tested may be wrong. Chaos engineering tests the claims under controlled conditions, before a real failure happens.

A chaos experiment usually follows four steps.

Define normal. Choose metrics that show the system is healthy, like the checkout success rate.
Predict. State what should happen. For example, "Checkout keeps working if one reviews instance stops."
Inject a failure. Stop an instance, add network delay, or make a dependency return errors. Start small, like one instance or 1 percent of traffic.
Compare and fix. Compare the metrics with normal. If the prediction was wrong, fix the weakness and test again.

A chaos experiment defines normal behavior, predicts the result, injects a small failure, and compares the metrics

Common tools are Chaos Monkey, which randomly stops server instances, and Gremlin. Teams usually start in a test environment and later run small experiments in production.

Practice	Protects against	What it costs
Fault tolerance	A part that stops completely	Extra machines and copies
Timeouts	Waiting too long for a slow call	Some slow calls fail that would have finished
Retries with backoff	Short, transient failures	A little more latency
Circuit breakers	Calling a dependency that is down	Some calls fail fast during recovery
Fallbacks	A dependency of one feature failing	A simpler or smaller answer
Bulkheads	One failure using up all resources	Resources are split, so less sharing
Chaos engineering	Weaknesses that nobody tested	Controlled risk during experiments

Key Takeaways

In a large system, failures happen every day. A cascading failure spreads from one failing part to parts that were healthy.
Fault tolerance keeps a system working correctly through failures, using redundancy at the data, service, and node levels.
Timeouts stop callers from waiting too long. Retries handle transient failures, and backoff spaces the retries so they do not add load to a struggling system.
Retry only transient errors, add jitter, and make retried operations idempotent.
A circuit breaker stops calls to a failing dependency, then tests it before calling it normally again.
Graceful degradation keeps serving requests with reduced functionality, using circuit breakers, timeouts, and fallbacks.
Bulkheads give each dependency its own share of resources, so one failure cannot use them all.
Errors should be sorted, logged, and reported, never hidden. Chaos engineering injects failures on purpose to find weaknesses before a real failure happens.

A resilient system is not one that never fails. It is one where a failure stays small, stays visible, and ends quickly. The next lesson, Fault Tolerance vs. High Availability, compares two promises that sound the same but are not.

Practice Questions

Try each question first, then open the answer.

1. A client uses exponential backoff with a first wait of 200 ms, and it retries at most 4 times. What are the waits before each retry, and what is the total waiting time?