System Design
Learn System Design
Introduction to System Design
How to Learn System Design?
Key Characteristics of Distributed Systems
Scalability
Availability
Latency and Performance
Concurrency and Coordination
Monitoring and Observability
Resilience and Error Handling
Fault Tolerance vs. High Availability
Flashcards Review
Chapter Assessment
Network Essentials
HTTP vs. HTTPS
TCP vs. UDP
HTTP: 1.0 vs. 1.1 vs 2.0 vs. 3.0
URL vs. URI vs. URN
What Happens When You Type a URL into the Browser
Flashcards Review
Chapter Assessment
Long-Polling vs. WebSockets vs. Server-Sent Events
Introduction to Real-Time Communication
What is Long-Polling?
What is WebSocket?
What are Server-Sent Events?
Difference Between Long-Polling, WebSockets, and Server-Sent Events
Flashcards Review
Chapter Assessment
Domain Name System (DNS)
Introduction to DNS
DNS Resolution Process
DNS Load Balancing and High Availability
Flashcards Review
Chapter Assessment
Proxies
What is a Proxy Server?
Uses of Proxies
VPN vs. Proxy Server
Flashcards Review
Chapter Assessment
API Gateway
Introduction to API Gateway
Usage of API gateway
Advantages and disadvantages of using API gateway
Flashcards Review
Chapter Assessment
API Design
What Is an API?
What Are REST APIs?
Resources, Not Actions
HTTP Methods and Their Semantics
URL Design
Request and Response Shapes
Status Codes and Error Design
Pagination from the Consumer's View
Idempotency Keys
Versioning and Backward Compatibility
Concurrency and Conditional Requests
REST vs gRPC vs GraphQL
Flashcards Review
Chapter Assessment
Rate Limiting and Throttling
What Is Rate Limiting
Rate Limiting Algorithms
Distributed Rate Limiting
Rate Limiting in Practice
Flashcards Review
Chapter Assessment
Caching
Introduction to Caching
Why is Caching Important?
Types of Caching
Cache Replacement Policies
Cache Invalidation
Cache Read Strategies
Cache Coherence and Consistency Models
Caching Challenges
Cache Performance Metrics
Flashcards Review
Chapter Assessment
CDN
What is CDN?
Origin Server vs. Edge Server
CDN Architecture
Push CDN vs. Pull CDN
Flashcards Review
Chapter Assessment
Data Partitioning
Introduction to Data Partitioning
Partitioning Methods
Data Sharding Techniques
Benefits of Data Partitioning
Common Problems Associated with Data Partitioning
Flashcards Review
Chapter Assessment
Redundancy and Replication
What is Redundancy?
What is Replication?
Replication Methods
Data Backup vs. Disaster Recovery
Flashcards Review
Chapter Assessment
CAP & PACELC Theorems
Introduction to CAP Theorem
Components of CAP Theorem
Trade-offs in CAP Theorem
Examples of CAP Theorem in Practice
Beyond CAP Theorem
System Design Trade-offs in Interviews
Flashcards Review
Chapter Assessment
Databases (SQL vs. NoSQL)
Introduction to Databases
SQL Databases
NoSQL Databases
SQL vs. NoSQL
ACID vs BASE Properties
Real-World Examples and Case Studies
SQL Normalization and Denormalization
In-Memory Database vs. On-Disk Database
Data Replication vs. Data Mirroring
Database Federation
Flashcards Review
Chapter Assessment
Indexes
What are Indexes?
How a B-Tree Index Works
Types of Indexes
B-Tree vs. LSM Tree
Indexes in Distributed Systems
Flashcards Review
Chapter Assessment
Bloom Filters
Introduction to Bloom Filters
Benefits & Limitations of Bloom Filters
Variants and Extensions of Bloom Filters
Applications of Bloom Filters
Flashcards Review
Chapter Assessment
Quorum
Why Quorum?
What is Quorum?
Flashcards Review
Chapter Assessment
Leader and Follower
What is Leader and Follower Pattern?
Flashcards Review
Chapter Assessment
Heartbeat
What is Heartbeat?
Flashcards Review
Chapter Assessment
Checksum
What is Checksum?
Uses of Checksum
Flashcards Review
Chapter Assessment
Distributed Messaging System
Introduction to Messaging System
Introduction to Kafka
Messaging patterns
Popular Messaging Queue Systems
RabbitMQ vs. Kafka vs. ActiveMQ
Scalability and Performance
Flashcards Review
Chapter Assessment
Distributed File Systems
What is a Distributed File System?
Architecture of a Distributed File System
Key Components of a DFS
Flashcards Review
Chapter Assessment
Security
What is Security and Privacy?
What is Authentication?
What is Authorization?
Authentication vs. Authorization
OAuth vs. JWT for Authentication
What is Encryption?
What are DDoS Attacks?
Flashcards Review
Chapter Assessment
Misc Concepts
Batch Processing vs. Stream Processing
XML vs. JSON
Synchronous vs. Asynchronous Communication
Push vs. Pull Notification Systems
Microservices vs. Serverless Architecture
Message Queues vs. Service Bus
Stateful vs. Stateless Architecture
Event-Driven vs. Polling Architecture
Flashcards Review
Chapter Assessment
Quiz - System Design Fundamentals
Quiz
System Design Trade-offs
Importance of Discussing Trade-offs
Strong vs Eventual Consistency
Latency vs Throughput
ACID vs BASE Properties in Databases
Read-Through vs Write-Through Cache
Batch Processing vs Stream Processing
Load Balancer vs. API Gateway
API Gateway vs Direct Service Exposure
Proxy vs. Reverse Proxy
API Gateway vs. Reverse Proxy
SQL vs. NoSQL
Primary-Replica vs Peer-to-Peer Replication
Data Compression vs Data Deduplication
Server-Side Caching vs Client-Side Caching
REST vs RPC
Polling vs. Long-Polling vs. WebSockets vs. Webhooks
CDN Usage vs Direct Server Serving
Serverless Architecture vs Traditional Server-based
Stateful vs Stateless Architecture
Hybrid Cloud Storage vs All-Cloud Storage
Token Bucket vs Leaky Bucket
Read Heavy vs Write Heavy System
Quiz
How to Approach a System Design Interview
System Design Interviews - A step by step guide
Functional vs. Non-functional Requirements
What are Back-of-the-Envelope Estimations?
Things to Avoid During System Design Interview
System Design Master Template
System Design Master Template
Quiz
Designing a URL Shortening Service like TinyURL
Designing a URL Shortening Service like TinyURL
Quiz - Designing URL Shortner
Designing Pastebin
Designing Pastebin
Quiz - Designing Pastebin
Designing Instagram
Designing Instagram
Quiz - Designing Instagram
Designing Dropbox
Designing Dropbox
Quiz - Designing Dropbox
Designing Facebook Messenger
Designing Facebook Messenger
Quiz - Designing Facebook Messenger
Designing Twitter
Designing Twitter
Quiz - Designing Twitter
Designing Youtube or Netflix
Designing Youtube or Netflix
Quiz - Designing Youtube
Designing Typeahead Suggestion
Designing Typeahead Suggestion
Quiz - Designing Typeahead Suggestion
Designing an API Rate Limiter
Designing an API Rate Limiter
Quiz - Designing an API Rate Limiter
Designing Twitter Search
Designing Twitter Search
Quiz - Designing Twitter Search
Designing a Web Crawler
Designing a Web Crawler
Quiz - Designing a Web Crawler
Designing Facebook’s Newsfeed
Designing Facebook’s Newsfeed
Quiz - Designing Facebook’s Newsfeed
Designing Yelp or Nearby Friends
Designing Yelp or Nearby Friends
Quiz - Designing Yelp or Nearby Friends
Designing Uber backend
Designing Uber backend
Quiz - Designing Uber backend
Designing Ticketmaster
Designing Ticketmaster
Quiz - Designing Ticketmaster
Dynamo: How to design a key value store?
Dynamo: Introduction
High-Level Architecture
Data Partitioning
Replication
Vector Clocks and Conflicting Data
The Life of Dynamo’s put() & get() Operations
Anti-entropy Through Merkle Trees
Gossip Protocol
Dynamo Characteristics and Criticism
Summary: Dynamo
Quiz: Dynamo
Mock Interview: Dynamo
Designing YouTube Likes Counter (medium)
YouTube Likes Counter
Quiz
Cassandra: How to Design a Wide-column NoSQL Database?
Cassandra: Introduction
High-level Architecture
Replication
Cassandra Consistency Levels
Gossiper
Anatomy of Cassandra's Write Operation
Anatomy of Cassandra's Read Operation
Compaction
Tombstones
Summary: Cassandra
Quiz: Cassandra
Mock Interview: Cassandra
Kafka: How to Design a Distributed Messaging System?
Messaging Systems: Introduction
Kafka: Introduction
High-level Architecture
Kafka: Deep Dive
Consumer Groups
Kafka Workflow
Role of ZooKeeper
Controller Broker
Kafka Delivery Semantics
Kafka Characteristics
Summary: Kafka
Quiz: Kafka
Mock Interview: Kafka
Chubby: How to Design a Distributed Locking Service?
Chubby: Introduction
High-level Architecture
Design Rationale
How Chubby Works
File, Directories, and Handles
Locks, Sequencers, and Lock-delays
Sessions and Events
Master Election and Chubby Events
Caching
Database
Scaling Chubby
Summary: Chubby
Quiz: Chubby
Mock Interview: Chubby
HDFS: How to Design File Storage System?
Hadoop Distributed File System: Introduction
High-level Architecture
Deep Dive
Anatomy of a Read Operation
Anatomy of a Write Operation
Data Integrity & Caching
Fault Tolerance
HDFS High Availability (HA)
HDFS Characteristics
Summary: HDFS
Quiz: HDFS
Mock Interview: HDFS
GFS: How to Design a Distributed File System Storage?
Google File System: Introduction
High-level Architecture
Single Master and Large Chunk Size
Metadata
Master Operations
Anatomy of a Read Operation
Anatomy of a Write Operation
Anatomy of an Append Operation
GFS Consistency Model and Snapshotting
Fault Tolerance, High Availability, and Data Integrity
Garbage Collection
Criticism on GFS
Summary: GFS
Quiz: GFS
Mock Interview: GFS
BigTable: How to Design a Wide Column Storage System?
BigTable: Introduction
BigTable Data Model
System APIs
Partitioning and High-level Architecture
SSTable
GFS and Chubby
Bigtable Components
Working with Tablets
The Life of BigTable's Read & Write Operations
Fault Tolerance and Compaction
BigTable Refinements
BigTable Characteristics
Summary: BigTable
Quiz: BigTable
Mock Interview: BigTable
Designing Reddit (medium)
Design Reddit
Quiz
Designing Notification Service (medium)
Designing a Notification System
Quiz
Design Google Calendar (medium)
Design Google calendar (Medium)
Quiz
Design a Recommendation System (medium)
Design a Recommendation System for Netflix
Quiz
Designing Gmail (medium)
Design Gmail
Quiz
Designing Google News (medium)
Design Google News, a Global News Aggregator System (Medium)
Quiz
Designing Unique ID Generator (medium)
Design Unique ID Generator (Easy)
Quiz
Designing Code Judging System (medium)
Design Code Judging System like LeetCode (Medium)
Quiz
Designing Payment System (hard)
Design Payment System
Quiz
Designing Flash Sale System (hard)
Design a Flash Sale for an E-commerce Site (Hard)
Quiz
Designing Reminder Alert System (hard)
Design a Reminder Alert System
Quiz
System Design Patterns
Introduction: System Design Patterns
1. Bloom Filters
2. Consistent Hashing
3. Quorum
4. Leader and Follower
5. Write-ahead Log
6. Segmented Log
7. High-Water Mark
8. Lease
9. Heartbeat
10. Gossip Protocol
11. Phi Accrual Failure Detection
12. Split Brain
13. Fencing
14. Checksum
15. Vector Clocks
16. CAP Theorem
17. PACELC Theorem
18. Hinted Handoff
19. Read Repair
20. Merkle Trees
Quiz
Introduction to Load Balancing
load balancing
availability
performance
distributed systems
+2
A concert ticket website runs on three servers. When tickets go on sale, thousands of fans arrive in the same minute. But every request goes to the first server because that is the only address the website gives out.
The first server slows down, then crashes. The other two servers sit almost unused the whole time.
The website had enough servers. What it lacked was a way to spread the work across them. This lesson explains the part that does that job, how it works, and where it belongs in a system.
What Load Balancing Is
Load balancing means spreading incoming requests and traffic across multiple servers. Its main goal is to keep any single server from being overloaded or becoming a single point of failure. That improves the availability, reliability, and performance of the whole system.
A load balancer is a device or software that does this job. It sits between the clients and the backend servers. It accepts incoming network and application traffic, and it forwards each request to one of the servers, based on predefined rules or algorithms.
The servers behind a load balancer are called backend servers. As a group, they are also called the server pool or the server farm.
A load balancer brings three main benefits.
- Scalability. To handle more traffic, you add servers to the pool. Clients keep using the same address.
- Availability. If one server fails, the load balancer stops sending it traffic, and the other servers keep serving users.
- Performance. No server gets overloaded while others have little work, so response times stay low.
How a Load Balancer Works
A load balancer follows the same steps for every request.
- It receives a request from a client or user.
- It chooses a server. It uses a load balancing algorithm, which can consider things like server capacity, response time, the number of active connections, and geographic location.
- It forwards the request to the chosen server.
- The server processes the request and sends a response back to the load balancer.
- The load balancer returns the response to the client that made the request.
Clients never need to know how many servers exist or which one answered. They connect to one address, often called a virtual IP address, that belongs to the load balancer.
Key Concepts
Load Balancing Algorithm
A load balancing algorithm is the method the load balancer uses to decide which backend server gets each request. For example, round robin sends requests to the servers in turn. Least connections sends each request to the server with the fewest active connections. The load balancing algorithms lesson compares the common algorithms.
Health Checks
Health checks are periodic tests that the load balancer runs to find out whether each backend server is available and working well. Unhealthy servers are removed from the server pool until they recover.
Here is a typical setup.
- Every 10 seconds, the load balancer sends a small request, like
GET /health, to each server. - If a server fails 3 checks in a row, the load balancer stops sending it traffic.
- If the server then passes 2 checks in a row, the load balancer adds it back.
These are active health checks, because the load balancer sends test requests on a timer. Many load balancers also use passive health checks. They watch the real requests and stop using a server that keeps returning errors or timing out.
Session Persistence
Session persistence, also called sticky sessions, makes sure that later requests from the same client go to the same backend server. The load balancer usually does this with a cookie that records which server the client used.
This helps when a server keeps a user's session data, like a shopping cart, in its own memory. But it has costs. Load can become uneven, and users lose their session if their server fails. A better long-term design keeps session data in a shared store, so any server can handle any request. The stateless vs. stateful load balancing lesson explains this choice.
SSL/TLS Termination
SSL (Secure Sockets Layer) and TLS (Transport Layer Security) are the protocols that encrypt HTTPS traffic. TLS is the modern version of SSL.
SSL/TLS termination means decrypting HTTPS traffic at the load balancer, instead of on each backend server. This removes the decryption work from the backend servers, so they have more CPU for application work. It also keeps certificates in one central place. The HTTP vs. HTTPS lesson explains how TLS works.
Where to Place Load Balancers
A system can use a load balancer at each layer, not only at the front. To get full scalability and redundancy, you can add load balancers in three places.
- Between the users and the web servers. This is the most common place.
- Between the web servers and the internal platform layer, like application servers or cache servers.
- Between the internal platform layer and the database servers.
Each layer can then grow and survive a failed server on its own.
The database layer needs more care. Usually only the primary database accepts writes. So a load balancer in front of databases normally spreads read requests across read replicas, while writes still go to the primary.
Kinds of Load Balancers
Load balancers come in three main forms.
- Hardware load balancers are dedicated physical devices. They are very fast, but expensive and less flexible.
- Software load balancers run on ordinary servers. Common examples are Nginx, HAProxy, and Envoy.
- Cloud load balancers are managed services from cloud providers. The provider runs and scales them for you.
Load balancers also differ in how much of each request they read. A Layer 4 load balancer routes by IP address and port. A Layer 7 load balancer can read HTTP details, like the URL path and cookies. The load balancer types lesson covers these differences.
The Load Balancer Must Not Fail
If every request passes through one load balancer, that load balancer becomes a new single point of failure. So production systems run at least two.
A common setup is active-passive. One load balancer handles all traffic, and a second one waits as a standby. If the active one fails, the standby takes over the same virtual IP address, and clients keep using the same address. The challenges of load balancers lesson covers this and other problems.
Key Takeaways
- A load balancer is a device or software that sits between clients and backend servers, and distributes incoming traffic across the servers.
- Its main goal is to prevent any server from being overloaded or becoming a single point of failure, which improves availability, reliability, and performance.
- The backend servers are also called the server pool or server farm.
- A load balancer receives a request, chooses a server with an algorithm, forwards the request, and returns the server's response to the client.
- Health checks remove unhealthy servers from the pool until they recover.
- Session persistence sends a client's requests to the same server. SSL/TLS termination decrypts HTTPS traffic at the load balancer.
- Load balancers can sit at each layer: before the web servers, before the application layer, and before the databases.
- Run at least two load balancers, so the load balancer is not a single point of failure.
A load balancer lets a group of servers work together and serve users through one reliable address. The next lesson, Load Balancing Algorithms, explains the different ways it can choose a server for each request.
Practice Questions
Try each question first, then open the answer.
1. Three servers sit behind a load balancer, and each server can handle 1,000 requests per second. Peak traffic is 2,400 requests per second. Can the system still handle the peak if one server fails?
<details> <summary>Show answer</summary>No. With one server down, two servers can handle 2 x 1,000 = 2,000 requests per second, which is less than 2,400. To survive one failure at peak, the pool needs at least four servers. Then the three remaining servers can handle 3,000 requests per second.
</details>2. A load balancer runs a health check every 10 seconds, and removes a server after 3 failed checks in a row. A server crashes. About how long does it keep receiving traffic?
<details> <summary>Show answer</summary>About 20 to 30 seconds. The first failed check happens within 10 seconds of the crash. Two more checks follow, 10 seconds apart. So the server is removed after the third failure, about 20 to 30 seconds after the crash. Passive health checks can detect the problem sooner, because they watch real requests.
</details>3. A shopping app stores each user's cart in the memory of its server. After a load balancer is added, users say their carts keep disappearing. Which load balancer feature helps right away, and what is the better long-term fix?
<details> <summary>Show answer</summary>Session persistence helps right away, and a shared store is the better fix. Sticky sessions send each user back to the same server, so the cart stays visible. But load can become uneven, and carts are lost when a server fails. Storing carts in a shared store, like Redis or a database, lets any server handle any request.
</details>4. The backend servers spend a lot of CPU time decrypting HTTPS traffic. Which load balancer feature reduces this work?
<details> <summary>Show answer</summary>SSL/TLS termination. The load balancer decrypts the HTTPS traffic itself, and forwards the requests to the backend servers. The servers no longer do the decryption, so they have more CPU for application work. Certificates are also managed in one place.
</details>5. An app has web servers, application servers, and database replicas. Where can load balancers be placed? Why does a load balancer in front of the databases usually spread only read requests?
<details> <summary>Show answer</summary>At three places: before the web servers, before the application servers, and before the database servers. In most database setups, only the primary accepts writes, and the replicas hold copies for reading. So the load balancer spreads read requests across the replicas, while writes still go to the primary.
</details>Discussion
On This Page