System Design
Learn System Design
Introduction to System Design
How to Learn System Design?
Network Essentials
HTTP vs. HTTPS
TCP vs. UDP
HTTP: 1.0 vs. 1.1 vs 2.0 vs. 3.0
URL vs. URI vs. URN
What Happens When You Type a URL into the Browser
Flashcards Review
Chapter Assessment
Long-Polling vs. WebSockets vs. Server-Sent Events
Introduction to Real-Time Communication
What is Long-Polling?
What is WebSocket?
What are Server-Sent Events?
Difference Between Long-Polling, WebSockets, and Server-Sent Events
Flashcards Review
Chapter Assessment
Domain Name System (DNS)
Introduction to DNS
DNS Resolution Process
DNS Load Balancing and High Availability
Flashcards Review
Chapter Assessment
Proxies
What is a Proxy Server?
Uses of Proxies
VPN vs. Proxy Server
Flashcards Review
Chapter Assessment
Load Balancing
Introduction to Load Balancing
Load Balancing Algorithms
Uses of Load Balancing
Load Balancer Types
Stateless vs. Stateful Load Balancing
High Availability and Fault Tolerance
Scalability and Performance
Challenges of Load Balancers
Flashcards Review
Chapter Assessment
API Gateway
Introduction to API Gateway
Usage of API gateway
Advantages and disadvantages of using API gateway
Flashcards Review
Chapter Assessment
API Design
What Is an API?
What Are REST APIs?
Resources, Not Actions
HTTP Methods and Their Semantics
URL Design
Request and Response Shapes
Status Codes and Error Design
Pagination from the Consumer's View
Idempotency Keys
Versioning and Backward Compatibility
Concurrency and Conditional Requests
REST vs gRPC vs GraphQL
Flashcards Review
Chapter Assessment
Rate Limiting and Throttling
What Is Rate Limiting
Rate Limiting Algorithms
Distributed Rate Limiting
Rate Limiting in Practice
Flashcards Review
Chapter Assessment
Caching
Introduction to Caching
Why is Caching Important?
Types of Caching
Cache Replacement Policies
Cache Invalidation
Cache Read Strategies
Cache Coherence and Consistency Models
Caching Challenges
Cache Performance Metrics
Flashcards Review
Chapter Assessment
CDN
What is CDN?
Origin Server vs. Edge Server
CDN Architecture
Push CDN vs. Pull CDN
Flashcards Review
Chapter Assessment
Data Partitioning
Introduction to Data Partitioning
Partitioning Methods
Data Sharding Techniques
Benefits of Data Partitioning
Common Problems Associated with Data Partitioning
Flashcards Review
Chapter Assessment
Redundancy and Replication
What is Redundancy?
What is Replication?
Replication Methods
Data Backup vs. Disaster Recovery
Flashcards Review
Chapter Assessment
CAP & PACELC Theorems
Introduction to CAP Theorem
Components of CAP Theorem
Trade-offs in CAP Theorem
Examples of CAP Theorem in Practice
Beyond CAP Theorem
System Design Trade-offs in Interviews
Flashcards Review
Chapter Assessment
Databases (SQL vs. NoSQL)
Introduction to Databases
SQL Databases
NoSQL Databases
SQL vs. NoSQL
ACID vs BASE Properties
Real-World Examples and Case Studies
SQL Normalization and Denormalization
In-Memory Database vs. On-Disk Database
Data Replication vs. Data Mirroring
Database Federation
Flashcards Review
Chapter Assessment
Indexes
What are Indexes?
How a B-Tree Index Works
Types of Indexes
B-Tree vs. LSM Tree
Indexes in Distributed Systems
Flashcards Review
Chapter Assessment
Bloom Filters
Introduction to Bloom Filters
Benefits & Limitations of Bloom Filters
Variants and Extensions of Bloom Filters
Applications of Bloom Filters
Flashcards Review
Chapter Assessment
Quorum
Why Quorum?
What is Quorum?
Flashcards Review
Chapter Assessment
Leader and Follower
What is Leader and Follower Pattern?
Flashcards Review
Chapter Assessment
Heartbeat
What is Heartbeat?
Flashcards Review
Chapter Assessment
Checksum
What is Checksum?
Uses of Checksum
Flashcards Review
Chapter Assessment
Distributed Messaging System
Introduction to Messaging System
Introduction to Kafka
Messaging patterns
Popular Messaging Queue Systems
RabbitMQ vs. Kafka vs. ActiveMQ
Scalability and Performance
Flashcards Review
Chapter Assessment
Distributed File Systems
What is a Distributed File System?
Architecture of a Distributed File System
Key Components of a DFS
Flashcards Review
Chapter Assessment
Security
What is Security and Privacy?
What is Authentication?
What is Authorization?
Authentication vs. Authorization
OAuth vs. JWT for Authentication
What is Encryption?
What are DDoS Attacks?
Flashcards Review
Chapter Assessment
Misc Concepts
Batch Processing vs. Stream Processing
XML vs. JSON
Synchronous vs. Asynchronous Communication
Push vs. Pull Notification Systems
Microservices vs. Serverless Architecture
Message Queues vs. Service Bus
Stateful vs. Stateless Architecture
Event-Driven vs. Polling Architecture
Flashcards Review
Chapter Assessment
Quiz - System Design Fundamentals
Quiz
System Design Trade-offs
Importance of Discussing Trade-offs
Strong vs Eventual Consistency
Latency vs Throughput
ACID vs BASE Properties in Databases
Read-Through vs Write-Through Cache
Batch Processing vs Stream Processing
Load Balancer vs. API Gateway
API Gateway vs Direct Service Exposure
Proxy vs. Reverse Proxy
API Gateway vs. Reverse Proxy
SQL vs. NoSQL
Primary-Replica vs Peer-to-Peer Replication
Data Compression vs Data Deduplication
Server-Side Caching vs Client-Side Caching
REST vs RPC
Polling vs. Long-Polling vs. WebSockets vs. Webhooks
CDN Usage vs Direct Server Serving
Serverless Architecture vs Traditional Server-based
Stateful vs Stateless Architecture
Hybrid Cloud Storage vs All-Cloud Storage
Token Bucket vs Leaky Bucket
Read Heavy vs Write Heavy System
Quiz
How to Approach a System Design Interview
System Design Interviews - A step by step guide
Functional vs. Non-functional Requirements
What are Back-of-the-Envelope Estimations?
Things to Avoid During System Design Interview
System Design Master Template
System Design Master Template
Quiz
Designing a URL Shortening Service like TinyURL
Designing a URL Shortening Service like TinyURL
Quiz - Designing URL Shortner
Designing Pastebin
Designing Pastebin
Quiz - Designing Pastebin
Designing Instagram
Designing Instagram
Quiz - Designing Instagram
Designing Dropbox
Designing Dropbox
Quiz - Designing Dropbox
Designing Facebook Messenger
Designing Facebook Messenger
Quiz - Designing Facebook Messenger
Designing Twitter
Designing Twitter
Quiz - Designing Twitter
Designing Youtube or Netflix
Designing Youtube or Netflix
Quiz - Designing Youtube
Designing Typeahead Suggestion
Designing Typeahead Suggestion
Quiz - Designing Typeahead Suggestion
Designing an API Rate Limiter
Designing an API Rate Limiter
Quiz - Designing an API Rate Limiter
Designing Twitter Search
Designing Twitter Search
Quiz - Designing Twitter Search
Designing a Web Crawler
Designing a Web Crawler
Quiz - Designing a Web Crawler
Designing Facebook’s Newsfeed
Designing Facebook’s Newsfeed
Quiz - Designing Facebook’s Newsfeed
Designing Yelp or Nearby Friends
Designing Yelp or Nearby Friends
Quiz - Designing Yelp or Nearby Friends
Designing Uber backend
Designing Uber backend
Quiz - Designing Uber backend
Designing Ticketmaster
Designing Ticketmaster
Quiz - Designing Ticketmaster
Dynamo: How to design a key value store?
Dynamo: Introduction
High-Level Architecture
Data Partitioning
Replication
Vector Clocks and Conflicting Data
The Life of Dynamo’s put() & get() Operations
Anti-entropy Through Merkle Trees
Gossip Protocol
Dynamo Characteristics and Criticism
Summary: Dynamo
Quiz: Dynamo
Mock Interview: Dynamo
Designing YouTube Likes Counter (medium)
YouTube Likes Counter
Quiz
Cassandra: How to Design a Wide-column NoSQL Database?
Cassandra: Introduction
High-level Architecture
Replication
Cassandra Consistency Levels
Gossiper
Anatomy of Cassandra's Write Operation
Anatomy of Cassandra's Read Operation
Compaction
Tombstones
Summary: Cassandra
Quiz: Cassandra
Mock Interview: Cassandra
Kafka: How to Design a Distributed Messaging System?
Messaging Systems: Introduction
Kafka: Introduction
High-level Architecture
Kafka: Deep Dive
Consumer Groups
Kafka Workflow
Role of ZooKeeper
Controller Broker
Kafka Delivery Semantics
Kafka Characteristics
Summary: Kafka
Quiz: Kafka
Mock Interview: Kafka
Chubby: How to Design a Distributed Locking Service?
Chubby: Introduction
High-level Architecture
Design Rationale
How Chubby Works
File, Directories, and Handles
Locks, Sequencers, and Lock-delays
Sessions and Events
Master Election and Chubby Events
Caching
Database
Scaling Chubby
Summary: Chubby
Quiz: Chubby
Mock Interview: Chubby
HDFS: How to Design File Storage System?
Hadoop Distributed File System: Introduction
High-level Architecture
Deep Dive
Anatomy of a Read Operation
Anatomy of a Write Operation
Data Integrity & Caching
Fault Tolerance
HDFS High Availability (HA)
HDFS Characteristics
Summary: HDFS
Quiz: HDFS
Mock Interview: HDFS
GFS: How to Design a Distributed File System Storage?
Google File System: Introduction
High-level Architecture
Single Master and Large Chunk Size
Metadata
Master Operations
Anatomy of a Read Operation
Anatomy of a Write Operation
Anatomy of an Append Operation
GFS Consistency Model and Snapshotting
Fault Tolerance, High Availability, and Data Integrity
Garbage Collection
Criticism on GFS
Summary: GFS
Quiz: GFS
Mock Interview: GFS
BigTable: How to Design a Wide Column Storage System?
BigTable: Introduction
BigTable Data Model
System APIs
Partitioning and High-level Architecture
SSTable
GFS and Chubby
Bigtable Components
Working with Tablets
The Life of BigTable's Read & Write Operations
Fault Tolerance and Compaction
BigTable Refinements
BigTable Characteristics
Summary: BigTable
Quiz: BigTable
Mock Interview: BigTable
Designing Reddit (medium)
Design Reddit
Quiz
Designing Notification Service (medium)
Designing a Notification System
Quiz
Design Google Calendar (medium)
Design Google calendar (Medium)
Quiz
Design a Recommendation System (medium)
Design a Recommendation System for Netflix
Quiz
Designing Gmail (medium)
Design Gmail
Quiz
Designing Google News (medium)
Design Google News, a Global News Aggregator System (Medium)
Quiz
Designing Unique ID Generator (medium)
Design Unique ID Generator (Easy)
Quiz
Designing Code Judging System (medium)
Design Code Judging System like LeetCode (Medium)
Quiz
Designing Payment System (hard)
Design Payment System
Quiz
Designing Flash Sale System (hard)
Design a Flash Sale for an E-commerce Site (Hard)
Quiz
Designing Reminder Alert System (hard)
Design a Reminder Alert System
Quiz
System Design Patterns
Introduction: System Design Patterns
1. Bloom Filters
2. Consistent Hashing
3. Quorum
4. Leader and Follower
5. Write-ahead Log
6. Segmented Log
7. High-Water Mark
8. Lease
9. Heartbeat
10. Gossip Protocol
11. Phi Accrual Failure Detection
12. Split Brain
13. Fencing
14. Checksum
15. Vector Clocks
16. CAP Theorem
17. PACELC Theorem
18. Hinted Handoff
19. Read Repair
20. Merkle Trees
Quiz
Latency and Performance
latency
throughput
performance
caching
+2
You tap "Pay" in a shopping app. Nothing happens for three seconds. You are not sure the payment worked, so you tap again. Now you worry that you paid twice.
The app did not crash. It was only slow. For the user, that can feel almost as bad as an error.
This lesson answers three questions. How do we measure how fast a system is? Where does the time in a request go? And what can we change to make it faster?
Latency, Throughput, and Performance
Latency is the time between sending a request and receiving the response. It is usually measured in milliseconds (ms). If you tap a button and the result appears 200 ms later, the latency is 200 ms.
Throughput is the number of requests a system completes in a period of time. It is usually measured in requests per second. A service that finishes 5,000 requests each second has a throughput of 5,000 requests per second.
Performance is how well a system does its work. We usually describe it with latency and throughput together.
The two numbers measure different things. Think of a checkout counter in a shop. Latency is how long one customer waits. Throughput is how many customers are served in an hour. Opening a second counter doubles the throughput, but each checkout does not become faster.
A system can have high throughput and poor latency at the same time. So teams watch both numbers.
Where the Time Goes
A request does not spend its time in one place. It passes through several steps, and each step adds latency.
- Setup. The browser finds the server's address with a DNS lookup. DNS is the system that turns a name like example.com into an IP address. Then the browser opens a connection with TCP, and with TLS for HTTPS.
- Network travel. The request travels to the server, and later the response travels back.
- Server work. The request may wait in a queue until the server is free. Then the app code runs.
- Database and other services. The app reads or writes data, or calls other services.
Here is an example with real numbers. Setup takes 80 ms, and network travel takes 80 ms. Server work takes 40 ms, of which the app code is 30 ms. One database query takes 150 ms. The total is 350 ms.
Always measure before you optimize. In this example, making the app code twice as fast saves only 15 ms. Fixing the slow query could save more than 100 ms. Tools that record how long each step takes show where to start. The monitoring and observability lesson covers these tools.
Distance often matters more than hardware. Reading from memory takes about 100 nanoseconds, where a nanosecond is one billionth of a second. A round trip inside one data center takes about 0.5 ms. A round trip across the world takes about 150 ms. No faster server can remove the time data spends traveling. The numbers you should know lesson lists more of these values.
Averages and Percentiles
Suppose you measure 100 requests. Of those, 98 take 100 ms and 2 take 5,000 ms. The average is 198 ms. That number looks fine, but two users waited five seconds.
An average hides slow requests. So teams use percentiles instead.
A percentile tells you how long a given share of requests take, at most.
- The p50, also called the median, is the latency that half of the requests stay at or below.
- The p95 is the latency that 95 percent of requests stay at or below.
- The p99 is the latency that 99 percent of requests stay at or below.
In the example, the p50 and the p95 are both 100 ms. The p99 is 5,000 ms. The p99 shows the problem that the average hides.
The latency of the slowest requests is called tail latency. Tail latency matters more than it seems, for two reasons.
- Active users make many requests. Suppose a user makes 100 requests in a day. There is about a 63 percent chance that at least one of them is slower than the p99.
- One page often makes many calls. Suppose a page calls 10 backend services, and each call has a 1 percent chance of being slow. The page is slow when any one call is slow. That happens about 10 percent of the time.
So a latency target is usually written as a percentile. For example, "the p99 latency of the checkout API stays under 300 ms".
How fast is fast enough? A common guideline from usability research gives three limits. A response under about 100 ms feels instant. Around 1 second, users notice the delay but stay focused. After about 10 seconds, many users give up or switch to another task.
How to Reduce Latency
Most techniques do one of three things. They avoid repeated work, they shorten the distance data travels, or they stop requests from waiting in a queue.
Caching
Caching stores frequently accessed data or computed results temporarily. The system can then return them quickly, instead of recalculating them or fetching them again from the primary data source.
A cache is usually fast memory, like Redis. When the requested data is in the cache, the lookup is a cache hit. When it is not, the lookup is a cache miss. On a miss, the app reads the database and saves a copy in the cache for the next request.
Here is the effect in numbers. A cache read takes about 1 ms, and the database query takes 150 ms. A miss costs 151 ms because the app checks the cache first. Suppose 90 percent of reads are hits. The average is 0.9 x 1 + 0.1 x 151, which is about 16 ms. That is almost 10 times faster than 150 ms.
There are three common caching strategies.
- In-memory caching keeps data in the memory of the app server itself. It is the fastest, but each server has its own copy.
- Distributed caching keeps data on a separate group of cache servers, like a Redis cluster. All app servers share it.
- Content delivery networks (CDNs) keep copies of static files, like images and videos, on servers in many cities near the users.
The cost of caching is stale data, which means the cache may still hold an old value after the database changes. The caching chapter explains how to handle this.
Data Locality
Data locality means organizing and placing data so that as little data as possible moves between nodes. Related data is stored together. It is also stored near the nodes and users that read it most often.
Distance adds latency that faster hardware cannot remove. Suppose a user in London reads from a database in California. Each round trip takes about 150 ms. A copy of the data in a London data center answers in under 10 ms.
Locality matters inside one data center too. Suppose a page needs a user and that user's orders. If both are stored on the same database server, one query returns them. If they are stored on different servers, the app makes several network calls instead.
Three techniques create data locality.
- Data partitioning splits a large dataset into smaller parts, based on a key like the user's region.
- Sharding places those parts on different servers, so each server holds the data its users need.
- Data replication keeps copies of the data in several places, so reads can use the nearest copy.
Load Balancing
Load balancing spreads incoming requests across several servers, so that no single server is overwhelmed.
This matters for latency because an overloaded server makes requests wait. Suppose one server has 50 requests in its queue, and another server has none. A new request sent to the busy server waits for all 50 to finish. The same request on the idle server starts at once.
A load balancer uses an algorithm to choose a server. Three common algorithms are below.
- Round-robin sends each request to the next server in turn. It is simple, but it ignores how busy each server is.
- Least connections sends each request to the server with the fewest active requests. It reacts to how busy each server really is.
- Consistent hashing sends requests with the same key, like the same user ID, to the same server. When a server is added or removed, only a small share of keys move to a different server. It also keeps each server's cache useful because the same user keeps reaching the same server.
The load balancing algorithms lesson covers more of them.
Other Techniques
Add an index. A database index is a sorted structure that lets the database find rows without reading the whole table. A query that scans 10 million rows can take seconds. With the right index, the same query may take a few milliseconds. The indexes chapter explains how they work.
Make calls at the same time. Suppose a page needs data from three services that take 50 ms, 80 ms, and 120 ms. Called one after another, they take 250 ms. Called at the same time, they take about 120 ms, which is the time of the slowest call.
Reuse connections. Opening a new connection with TCP and TLS can take tens of milliseconds. A connection pool keeps connections open and reuses them for many requests, so most requests skip the setup.
Send less data. Smaller responses travel faster. Return only the fields the client needs, and compress text responses like JSON. Compression often makes JSON several times smaller.
Move slow work out of the request. Work like sending an email can go into a message queue and run after the response is sent. The user waits only for the part that matters.
| Technique | What it reduces | Main cost |
|---|---|---|
| Caching | Repeated work | Stale data |
| Data locality | Distance data travels | More copies to keep in sync |
| Load balancing | Time waiting in queues | One more part to run |
| Indexes | Rows the database reads | Slower writes, more storage |
| Parallel calls | Waiting for calls one by one | More complex code |
Latency and Throughput Trade-offs
Improving one number can hurt the other.
Batching is one example. A system can save 100 writes together in one batch, instead of one at a time. This raises throughput because one large write costs less than 100 small ones. But the first write in the batch must wait until the batch is full or a timer ends. So its latency goes up.
Busy servers are another example. Running servers close to 100 percent CPU gets the most throughput from each machine. But requests then wait longer in the queue. In a simple queue model, the average wait at 90 percent busy is about 9 times the wait at 50 percent busy. So teams keep spare capacity to protect latency.
Key Takeaways
- Latency is how long one request takes. Throughput is how many requests the system completes per second.
- Measure where the time goes before you optimize, and fix the largest part first.
- Averages hide slow requests. Use percentiles like p95 and p99 to see tail latency.
- Caching stores frequently accessed data temporarily, so repeated reads skip the slow source. Common strategies are in-memory caching, distributed caching, and CDNs.
- Data locality keeps related data together and near its users, through partitioning, sharding, and replication.
- Load balancing spreads requests so that no server's queue grows long. Common algorithms are round-robin, least connections, and consistent hashing.
- Raising throughput, for example with batching or very busy servers, can make latency worse.
Fast systems are rarely fast because of one trick. They avoid repeated work, keep data close to where it is used, and stop any one server's queue from growing long. The next lesson, Concurrency and Coordination, covers what happens when many processes work on the same data at once.
Practice Questions
Try each question first, then open the answer.
1. You measure 100 requests. Of those, 95 take 80 ms and 5 take 2,000 ms. What are the average and the p99 latency?
<details> <summary>Show answer</summary>The average is 176 ms, and the p99 is 2,000 ms. The total time is 95 x 80 + 5 x 2,000 = 7,600 + 10,000 = 17,600 ms. Divide by 100 to get 176 ms. For the p99, sort the requests from fastest to slowest. The 99th request is one of the five slow ones, so the p99 is 2,000 ms. The average looks acceptable, but 5 users waited 2 seconds.
</details>2. A page calls four services one after another. They take 40 ms, 60 ms, 30 ms, and 120 ms, and no call needs the result of another. What is the total latency now, and what would it be if all calls ran at the same time?
<details> <summary>Show answer</summary>250 ms now, and about 120 ms at the same time. One after another, the times add up: 40 + 60 + 30 + 120 = 250 ms. At the same time, the page waits only for the slowest call, which takes 120 ms.
</details>3. A database query takes 200 ms, and a cache read takes 2 ms. 80 percent of reads are cache hits. What is the average read latency?
<details> <summary>Show answer</summary>About 42 ms. A hit takes 2 ms. A miss checks the cache first and then reads the database, so it takes 202 ms. The average is 0.8 x 2 + 0.2 x 202 = 1.6 + 40.4 = 42 ms. That is almost 5 times faster than 200 ms.
</details>4. Users in Europe say the app is slow. The servers and the database are in California, and the server work takes only 30 ms. What is the likely cause, and what would help?
<details> <summary>Show answer</summary>Network distance. Each round trip between Europe and California takes about 150 ms, and one page often needs several round trips. Faster servers cannot fix this. Data locality helps: serve static files from a CDN, and run app servers and read replicas in a European region.
</details>5. A load balancer uses round-robin. Most requests take 50 ms, but some report requests take 5 seconds. When one server receives several reports in a row, its other users wait a long time. Which algorithm helps, and why?
<details> <summary>Show answer</summary>Least connections. Round-robin keeps sending new requests to the busy server in turn, even while it is still working on the slow reports. Least connections sends each new request to the server with the fewest active requests. So the busy server gets fewer new requests until it finishes its current work.
</details>Discussion
On This Page