Resilience and Error Handling | Learn System Design

System Design

Learn System Design

Introduction to System Design

How to Learn System Design?

Key Characteristics of Distributed Systems

Scalability

Availability

Latency and Performance

Concurrency and Coordination

Monitoring and Observability

Resilience and Error Handling

Fault Tolerance vs. High Availability

Flashcards Review

Chapter Assessment

Network Essentials

HTTP vs. HTTPS

TCP vs. UDP

HTTP: 1.0 vs. 1.1 vs 2.0 vs. 3.0

URL vs. URI vs. URN

What Happens When You Type a URL into the Browser

Flashcards Review

Chapter Assessment

Long-Polling vs. WebSockets vs. Server-Sent Events

Introduction to Real-Time Communication

What is Long-Polling?

What is WebSocket?

What are Server-Sent Events?

Difference Between Long-Polling, WebSockets, and Server-Sent Events

Flashcards Review

Chapter Assessment

Domain Name System (DNS)

Introduction to DNS

DNS Resolution Process

DNS Load Balancing and High Availability

Flashcards Review

Chapter Assessment

Proxies

What is a Proxy Server?

Uses of Proxies

VPN vs. Proxy Server

Flashcards Review

Chapter Assessment

Load Balancing

Introduction to Load Balancing

Load Balancing Algorithms

Uses of Load Balancing

Load Balancer Types

Stateless vs. Stateful Load Balancing

High Availability and Fault Tolerance

Scalability and Performance

Challenges of Load Balancers

Flashcards Review

Chapter Assessment

API Gateway

Introduction to API Gateway

Usage of API gateway

Advantages and disadvantages of using API gateway

Flashcards Review

Chapter Assessment

API Design

What Is an API?

What Are REST APIs?

Resources, Not Actions

HTTP Methods and Their Semantics

URL Design

Request and Response Shapes

Status Codes and Error Design

Pagination from the Consumer's View

Idempotency Keys

Versioning and Backward Compatibility

Concurrency and Conditional Requests

REST vs gRPC vs GraphQL

Flashcards Review

Chapter Assessment

Rate Limiting and Throttling

What Is Rate Limiting

Rate Limiting Algorithms

Distributed Rate Limiting

Rate Limiting in Practice

Flashcards Review

Chapter Assessment

Caching

Introduction to Caching

Why is Caching Important?

Types of Caching

Cache Replacement Policies

Cache Invalidation

Cache Read Strategies

Cache Coherence and Consistency Models

Caching Challenges

Cache Performance Metrics

Flashcards Review

Chapter Assessment

CDN

What is CDN?

Origin Server vs. Edge Server

CDN Architecture

Push CDN vs. Pull CDN

Flashcards Review

Chapter Assessment

Data Partitioning

Introduction to Data Partitioning

Partitioning Methods

Data Sharding Techniques

Benefits of Data Partitioning

Common Problems Associated with Data Partitioning

Flashcards Review

Chapter Assessment

Redundancy and Replication

What is Redundancy?

What is Replication?

Replication Methods

Data Backup vs. Disaster Recovery

Flashcards Review

Chapter Assessment

CAP & PACELC Theorems

Introduction to CAP Theorem

Components of CAP Theorem

Trade-offs in CAP Theorem

Examples of CAP Theorem in Practice

Beyond CAP Theorem

System Design Trade-offs in Interviews

Flashcards Review

Chapter Assessment

Databases (SQL vs. NoSQL)

Introduction to Databases

SQL Databases

NoSQL Databases

SQL vs. NoSQL

ACID vs BASE Properties

Real-World Examples and Case Studies

SQL Normalization and Denormalization

In-Memory Database vs. On-Disk Database

Data Replication vs. Data Mirroring

Database Federation

Flashcards Review

Chapter Assessment

Indexes

What are Indexes?

How a B-Tree Index Works

Types of Indexes

B-Tree vs. LSM Tree

Indexes in Distributed Systems

Flashcards Review

Chapter Assessment

Bloom Filters

Introduction to Bloom Filters

Benefits & Limitations of Bloom Filters

Variants and Extensions of Bloom Filters

Applications of Bloom Filters

Flashcards Review

Chapter Assessment

Quorum

Why Quorum?

What is Quorum?

Flashcards Review

Chapter Assessment

Leader and Follower

What is Leader and Follower Pattern?

Flashcards Review

Chapter Assessment

Heartbeat

What is Heartbeat?

Flashcards Review

Chapter Assessment

Checksum

What is Checksum?

Uses of Checksum

Flashcards Review

Chapter Assessment

Distributed Messaging System

Introduction to Messaging System

Introduction to Kafka

Messaging patterns

Popular Messaging Queue Systems

RabbitMQ vs. Kafka vs. ActiveMQ

Scalability and Performance

Flashcards Review

Chapter Assessment

Distributed File Systems

What is a Distributed File System?

Architecture of a Distributed File System

Key Components of a DFS

Flashcards Review

Chapter Assessment

Security

What is Security and Privacy?

What is Authentication?

What is Authorization?

Authentication vs. Authorization

OAuth vs. JWT for Authentication

What is Encryption?

What are DDoS Attacks?

Flashcards Review

Chapter Assessment

Misc Concepts

Batch Processing vs. Stream Processing

XML vs. JSON

Synchronous vs. Asynchronous Communication

Push vs. Pull Notification Systems

Microservices vs. Serverless Architecture

Message Queues vs. Service Bus

Stateful vs. Stateless Architecture

Event-Driven vs. Polling Architecture

Flashcards Review

Chapter Assessment

Quiz - System Design Fundamentals

Quiz

System Design Trade-offs

Importance of Discussing Trade-offs

Strong vs Eventual Consistency

Latency vs Throughput

ACID vs BASE Properties in Databases

Read-Through vs Write-Through Cache

Batch Processing vs Stream Processing

Load Balancer vs. API Gateway

API Gateway vs Direct Service Exposure

Proxy vs. Reverse Proxy

API Gateway vs. Reverse Proxy

SQL vs. NoSQL

Primary-Replica vs Peer-to-Peer Replication

Data Compression vs Data Deduplication

Server-Side Caching vs Client-Side Caching

REST vs RPC

Polling vs. Long-Polling vs. WebSockets vs. Webhooks

CDN Usage vs Direct Server Serving

Serverless Architecture vs Traditional Server-based

Stateful vs Stateless Architecture

Hybrid Cloud Storage vs All-Cloud Storage

Token Bucket vs Leaky Bucket

Read Heavy vs Write Heavy System

Quiz

How to Approach a System Design Interview

System Design Interviews - A step by step guide

Functional vs. Non-functional Requirements

What are Back-of-the-Envelope Estimations?

Things to Avoid During System Design Interview

System Design Master Template

Quiz

Designing a URL Shortening Service like TinyURL

Quiz - Designing URL Shortner

Designing Pastebin

Quiz - Designing Pastebin

Designing Instagram

Quiz - Designing Instagram

Designing Dropbox

Quiz - Designing Dropbox

Designing Facebook Messenger

Quiz - Designing Facebook Messenger

Designing Twitter

Quiz - Designing Twitter

Designing Youtube or Netflix

Quiz - Designing Youtube

Designing Typeahead Suggestion

Quiz - Designing Typeahead Suggestion

Designing an API Rate Limiter

Quiz - Designing an API Rate Limiter

Designing Twitter Search

Quiz - Designing Twitter Search

Designing a Web Crawler

Quiz - Designing a Web Crawler

Designing Facebook’s Newsfeed

Quiz - Designing Facebook’s Newsfeed

Designing Yelp or Nearby Friends

Quiz - Designing Yelp or Nearby Friends

Designing Uber backend

Quiz - Designing Uber backend

Designing Ticketmaster

Quiz - Designing Ticketmaster

Dynamo: How to design a key value store?

Dynamo: Introduction

High-Level Architecture

Data Partitioning

Replication

Vector Clocks and Conflicting Data

The Life of Dynamo’s put() & get() Operations

Anti-entropy Through Merkle Trees

Gossip Protocol

Dynamo Characteristics and Criticism

Summary: Dynamo

Quiz: Dynamo

Mock Interview: Dynamo

Designing YouTube Likes Counter (medium)

YouTube Likes Counter

Quiz

Cassandra: How to Design a Wide-column NoSQL Database?

Cassandra: Introduction

High-level Architecture

Replication

Cassandra Consistency Levels

Gossiper

Anatomy of Cassandra's Write Operation

Anatomy of Cassandra's Read Operation

Compaction

Tombstones

Summary: Cassandra

Quiz: Cassandra

Mock Interview: Cassandra

Kafka: How to Design a Distributed Messaging System?

Messaging Systems: Introduction

Kafka: Introduction

High-level Architecture

Kafka: Deep Dive

Consumer Groups

Kafka Workflow

Role of ZooKeeper

Controller Broker

Kafka Delivery Semantics

Kafka Characteristics

Summary: Kafka

Quiz: Kafka

Mock Interview: Kafka

Chubby: How to Design a Distributed Locking Service?

Chubby: Introduction

High-level Architecture

Design Rationale

How Chubby Works

File, Directories, and Handles

Locks, Sequencers, and Lock-delays

Sessions and Events

Master Election and Chubby Events

Caching

Database

Scaling Chubby

Summary: Chubby

Quiz: Chubby

Mock Interview: Chubby

HDFS: How to Design File Storage System?

Hadoop Distributed File System: Introduction

High-level Architecture

Deep Dive

Anatomy of a Read Operation

Anatomy of a Write Operation

Data Integrity & Caching

Fault Tolerance

HDFS High Availability (HA)

HDFS Characteristics

Summary: HDFS

Quiz: HDFS

Mock Interview: HDFS

GFS: How to Design a Distributed File System Storage?

Google File System: Introduction

High-level Architecture

Single Master and Large Chunk Size

Metadata

Master Operations

Anatomy of a Read Operation

Anatomy of a Write Operation

Anatomy of an Append Operation

GFS Consistency Model and Snapshotting

Fault Tolerance, High Availability, and Data Integrity

Garbage Collection

Criticism on GFS

Summary: GFS

Quiz: GFS

Mock Interview: GFS

BigTable: How to Design a Wide Column Storage System?

BigTable: Introduction

BigTable Data Model

System APIs

Partitioning and High-level Architecture

SSTable

GFS and Chubby

Bigtable Components

Working with Tablets

The Life of BigTable's Read & Write Operations

Fault Tolerance and Compaction

BigTable Refinements

BigTable Characteristics

Summary: BigTable

Quiz: BigTable

Mock Interview: BigTable

Designing Reddit (medium)

Design Reddit

Quiz

Designing Notification Service (medium)

Designing a Notification System

Quiz

Design Google Calendar (medium)

Design Google calendar (Medium)

Quiz

Design a Recommendation System (medium)

Design a Recommendation System for Netflix

Quiz

Designing Gmail (medium)

Design Gmail

Quiz

Designing Google News (medium)

Design Google News, a Global News Aggregator System (Medium)

Quiz

Designing Unique ID Generator (medium)

Design Unique ID Generator (Easy)

Quiz

Designing Code Judging System (medium)

Design Code Judging System like LeetCode (Medium)

Quiz

Designing Payment System (hard)

Design Payment System

Quiz

Designing Flash Sale System (hard)

Design a Flash Sale for an E-commerce Site (Hard)

Quiz

Designing Reminder Alert System (hard)

Design a Reminder Alert System

Quiz

System Design Patterns

Introduction: System Design Patterns

1. Bloom Filters

2. Consistent Hashing

3. Quorum

4. Leader and Follower

5. Write-ahead Log

6. Segmented Log

7. High-Water Mark

8. Lease

9. Heartbeat

10. Gossip Protocol

11. Phi Accrual Failure Detection

12. Split Brain

13. Fencing

14. Checksum

15. Vector Clocks

16. CAP Theorem

17. PACELC Theorem

18. Hinted Handoff

19. Read Repair

20. Merkle Trees

Quiz

Resilience and Error Handling

fault tolerance

circuit breaker pattern

idempotency

graceful degradation

hard

16 min

·Updated Sep 2026·Credit: System Design Fundamentals

An online store shows product reviews from a separate reviews service. One afternoon, the reviews service becomes slow. Each call to it now takes 30 seconds instead of 50 ms.

A few minutes later, the whole store stops loading. Even the checkout page fails, and checkout does not use reviews at all.

This lesson answers two questions. How does a system keep working when some of its parts fail? And how do we stop one failure from spreading to everything else?

Failures Are Normal

In a large system, some part is always failing. Disks break, machines restart, networks lose messages, and services become slow.

The numbers show why. Suppose one server fails about once every 1,000 days. A company with 1,000 servers then sees about one server failure every day. So a large system must be designed to expect failure, not to avoid it completely.

Resilience is the ability of a system to keep working when parts of it fail, and to recover quickly afterwards.

Cascading Failures

The online store had a cascading failure, which is a failure in one part that spreads to other parts that were healthy.

Here is how it happened. Each web server had 200 threads to handle requests. Every product page called the reviews service and waited. Each wait now took 30 seconds, so the threads filled up with waiting requests. Soon no free thread was left for anything, including checkout.

A slow reviews service keeps every web server thread waiting, so checkout fails too and the whole store goes down

The reviews service was only slow. The store failed because nothing stopped the waiting. The rest of this lesson covers the techniques that prevent this.

Fault Tolerance

Fault tolerance is the ability of a system to keep functioning correctly when faults or failures are present.

It comes from redundancy at several levels.

Data. Replication keeps copies of the data on several machines, so losing one copy loses no data.
Services. Several instances of each service run behind a load balancer, so one instance can fail without an outage.
Nodes. Spare machines can take over the work of a machine that fails.

Sharding also helps. When data is split across shards, a failure affects only the users of one shard, not everyone. The Availability lesson explains redundancy and failover in detail.

Redundancy protects against a part that stops completely. But many failures are slow, partial, or temporary. The next techniques handle those.

Timeouts

A timeout is the longest time a caller waits for a response. When the time runs out, the caller stops waiting and treats the call as failed.

Without a timeout, a caller can wait for a very long time. That is exactly what happened in the online store.

Pick a timeout based on the normal latency of the call. Suppose the reviews service's p99 latency is under 200 ms. A timeout of 1 second then allows for an occasional slow call, and it still frees the thread long before 30 seconds.

Both extremes cause problems.

Too short. Calls that would have succeeded are cut off and counted as failures.
Too long. Threads stay busy waiting, and a slow dependency can still cause a cascading failure.

Retries and Backoff

Many failures are transient, which means they last only a short time. Examples are a brief network problem, a single timeout, or a service that takes a few seconds to restart. A second attempt often succeeds.

A retry strategy automatically tries a failed operation again. Backoff increases the delay between attempts.

Exponential backoff doubles the wait after each failure. For example, the client waits 100 ms before the first retry, then 200 ms, then 400 ms. It stops after a fixed number of attempts, like 3 or 4.

Each failed attempt doubles the wait before the next one, and the fourth attempt succeeds

Why the delay matters. Suppose a service is struggling, and 10,000 clients call it. Each failed client retries 3 times immediately, without waiting. The service now receives up to 40,000 requests instead of 10,000. The retries become the extra load that stops it from recovering. Spacing the retries out spreads this extra load over time, so a failing system has time to recover.

Good retries follow three more rules.

Add jitter. Jitter is a small random amount added to each wait. Without it, thousands of clients that failed together also retry together, at the same moment.
Retry only transient errors. A timeout or an HTTP 503 (service unavailable) may succeed on the next try. An HTTP 400 (bad request) fails the same way every time, so retrying it only wastes work.
Make retried operations idempotent. An operation is idempotent when doing it twice has the same effect as doing it once. Reading a user's profile is idempotent. Charging a card is not, unless you make it so.

To make a payment idempotent, the client sends a unique idempotency key with the request. The server remembers the keys it has already processed. If the same key arrives again, the server returns the first result instead of charging the card again.

Circuit Breakers

Retries help with short failures. But when a service is down for minutes, calling it again and again only makes callers wait and adds load.

A circuit breaker watches the calls to a dependency. When too many calls fail, it stops calling the dependency for a while. During that time, calls fail immediately, without waiting. This is called failing fast.

A circuit breaker has three states.

Closed. Calls go through normally, and the breaker counts failures. For example, it opens when more than 50 percent of calls fail within 10 seconds.
Open. Calls fail at once, without contacting the dependency. After a wait, like 30 seconds, the breaker moves to half-open.
Half-open. The breaker lets a few test calls through. If they succeed, it closes again. If one fails, it opens again.

A circuit breaker opens after too many failures, waits, tests the dependency in half-open, and closes when test calls succeed

A circuit breaker helps both sides. The caller stops wasting threads on calls that will fail. The struggling service gets time to recover, because new requests stop arriving.

The name comes from the electrical circuit breaker in a building. It cuts the power when something goes wrong, so the problem does not spread.

Graceful Degradation and Fallbacks

Graceful degradation is the ability to keep providing limited functionality when some parts of a system fail. Instead of shutting down completely, the system keeps serving requests with fewer features.

By default, many systems do the opposite. One dependency fails, and the whole page returns an error.

The main tool is a fallback, which is a backup answer used when a call fails. Common fallbacks are listed below.

Hide the feature. If reviews fail, the product page loads without the reviews section.
Use a simpler answer. If personal recommendations fail, show the store's best sellers instead.
Use cached data. If a currency service fails, show the last known exchange rate, with the time it was updated.
Use a default value. If the settings service fails, use the standard settings.

When the reviews and recommendations services fail, the product page still loads, hides reviews, and shows best sellers instead

A product page without recommendations is a worse page. A product page that does not load is an outage.

Timeouts, circuit breakers, and fallbacks work together. The timeout stops the waiting. The circuit breaker stops calls that are likely to fail. The fallback decides what the user sees instead.

Bulkheads

A bulkhead is a separate pool of resources for each dependency, so that one failing dependency can use only its own share.

The name comes from ships. A ship is split into sealed sections by walls called bulkheads. If one section fills with water, the other sections stay dry, and the ship does not sink.

Here is how it would have saved the online store. The web server has 200 threads. The team allows reviews calls to use at most 20 of them. When the reviews service becomes slow, only those 20 threads wait. The other 180 threads keep serving checkout and every other page.

Bulkheads protect against one slow dependency using up a server's own resources. Rate limiting protects against too much traffic from outside, by capping how many requests a client can send. The rate limiting chapter covers it.

Error Handling and Reporting

A resilient system handles errors on purpose. It does not hide them.

Sort errors into two kinds. A transient error, like a timeout, may succeed on retry. A permanent error, like invalid input or a missing record, will fail again. Retry the first kind. Fail fast on the second, and return a clear message.

Never hide an error. Code that catches an error and does nothing makes a failure invisible. The system then looks healthy while users see wrong results.

Record errors in a consistent way. Log every error with its category, the service name, and the trace ID of the request. Count errors as metrics, and create alerts when they matter. The monitoring and observability lesson explains these tools. Consistent error records let the team diagnose a failure before a user has to report it.

Handle messages that keep failing. In a message queue, a message that fails many times can block other work. Systems move it to a dead letter queue, which is a separate queue where engineers can inspect failed messages later.

Chaos Engineering

Chaos engineering is the practice of injecting failures into a system on purpose, to test its resilience and find its weaknesses.

Every technique in this lesson is a claim about what happens during a failure. A claim that nobody tested may be wrong. Chaos engineering tests the claims under controlled conditions, before a real failure happens.

A chaos experiment usually follows four steps.

Define normal. Choose metrics that show the system is healthy, like the checkout success rate.
Predict. State what should happen. For example, "Checkout keeps working if one reviews instance stops."
Inject a failure. Stop an instance, add network delay, or make a dependency return errors. Start small, like one instance or 1 percent of traffic.
Compare and fix. Compare the metrics with normal. If the prediction was wrong, fix the weakness and test again.

A chaos experiment defines normal behavior, predicts the result, injects a small failure, and compares the metrics

Common tools are Chaos Monkey, which randomly stops server instances, and Gremlin. Teams usually start in a test environment and later run small experiments in production.

Practice	Protects against	What it costs
Fault tolerance	A part that stops completely	Extra machines and copies
Timeouts	Waiting too long for a slow call	Some slow calls fail that would have finished
Retries with backoff	Short, transient failures	A little more latency
Circuit breakers	Calling a dependency that is down	Some calls fail fast during recovery
Fallbacks	A dependency of one feature failing	A simpler or smaller answer
Bulkheads	One failure using up all resources	Resources are split, so less sharing
Chaos engineering	Weaknesses that nobody tested	Controlled risk during experiments

Key Takeaways

In a large system, failures happen every day. A cascading failure spreads from one failing part to parts that were healthy.
Fault tolerance keeps a system working correctly through failures, using redundancy at the data, service, and node levels.
Timeouts stop callers from waiting too long. Retries handle transient failures, and backoff spaces the retries so they do not add load to a struggling system.
Retry only transient errors, add jitter, and make retried operations idempotent.
A circuit breaker stops calls to a failing dependency, then tests it before calling it normally again.
Graceful degradation keeps serving requests with reduced functionality, using circuit breakers, timeouts, and fallbacks.
Bulkheads give each dependency its own share of resources, so one failure cannot use them all.
Errors should be sorted, logged, and reported, never hidden. Chaos engineering injects failures on purpose to find weaknesses before a real failure happens.

A resilient system is not one that never fails. It is one where a failure stays small, stays visible, and ends quickly. The next lesson, Fault Tolerance vs. High Availability, compares two promises that sound the same but are not.

Practice Questions

Try each question first, then open the answer.

1. A client uses exponential backoff with a first wait of 200 ms, and it retries at most 4 times. What are the waits before each retry, and what is the total waiting time?