System Design Fundamentals

0% completed

Scalability

On This Page

What Scalability Means

Vertical Scaling

Horizontal Scaling

Stateless Servers

Scaling the Database

Other Ways to Scale

Finding the Bottleneck

Key Takeaways

Practice Questions

Your team runs a website that sells tickets for soccer matches. On a normal day, about 2,000 people visit it. Then tickets for a cup final go on sale. In the first ten minutes, 200,000 people open the site at the same time.

The pages slow down. Then the server stops answering, and most people never get a ticket.

The code did not change. Only the amount of work changed. This lesson answers two questions. How does a system handle more work as it grows? And how should we design it so that adding machines actually helps?

What Scalability Means

Scalability is the ability of a system to handle a growing workload by adding resources. A scalable system keeps working well as users, requests, and data grow.

A workload can grow in several ways.

More requests. Traffic is often measured in requests per second (RPS), the number of requests the system receives each second.
More data. The database grows from 10 GB to 10 TB.
More users in more places. Users in Chennai and users in London both expect a fast response.

Here is a simple example. One app server can handle 500 requests per second. At peak time, the site gets 2,000 requests per second. If the system scales well, four servers can handle that peak.

Scalability is not the same as speed. Performance is how fast the system answers one request. Scalability is whether it stays fast when the number of requests grows. A system can answer one user in 50 ms and still fail when 10,000 users arrive together.

There are two ways to add resources. You can make one machine bigger, or you can add more machines.

Vertical Scaling

Vertical scaling, also called scaling up, means increasing the capacity of one machine by upgrading its hardware. You give it more CPU, more memory, or more storage.

For example, a database server has 8 CPU cores and 32 GB of memory. The team moves it to a machine with 64 cores and 512 GB. The same single machine now handles more work. This is a common way to grow a relational database like MySQL.

Vertical scaling has real benefits.

It is simple. The application code usually does not change.
There is still only one machine to run, watch, and back up.
All data stays in one place, so every read sees the latest write.

It also has hard limits.

There is an upper limit. You cannot buy a machine bigger than the biggest machine available.
The biggest machines are expensive. Their price usually grows faster than the capacity they add.
Upgrading usually needs downtime. The machine often has to restart on the new hardware.
It is still one machine. If it fails, everything on it stops. It is a single point of failure.

Horizontal Scaling

Horizontal scaling, also called scaling out, means adding more machines, called nodes, so the workload is spread evenly across them. A load balancer sits in front of the nodes. It is a server that sends each incoming request to one of them.

No single machine has to handle all the growth. When more requests arrive, more machines share them.

Horizontal scaling has strong benefits.

There is no fixed limit. You can keep adding machines.
Capacity can grow while the system runs. New machines join the pool without downtime.
It is cost-effective when traffic goes up and down. You add machines for a sale day and remove them afterwards.
One failure is not an outage. If one machine fails, the others keep serving requests.

Databases like Cassandra and MongoDB are built to scale this way. You add nodes as the data and traffic grow.

Horizontal scaling also has costs. There are more machines to deploy and watch. Machines talk over the network, which adds delay and new ways to fail. Also, the application must be designed so that any machine can handle any request.

Scaling up makes one machine bigger, while scaling out adds more machines behind a load balancer

	Vertical (scaling up)	Horizontal (scaling out)
What changes	Size of one machine	Number of machines
Upper limit	The biggest machine available	No fixed limit
Adding capacity	Usually needs downtime	Add nodes while running
One machine fails	Everything on it stops	The others keep working
Code changes	Usually none	The app must be designed for it
Examples	MySQL	Cassandra, MongoDB

Most real systems use both. Teams often scale up first because it is simple. They scale out when they reach the limit of one machine, or when the system must keep working after a machine fails.

Stateless Servers

Horizontal scaling works only when any server can handle any request. The main thing that breaks this is state.

State is data a server remembers between requests, like a user's login session or shopping cart. A server that keeps this data in its own memory is called stateful.

Here is the problem. A user adds a phone to the cart, and the load balancer sends that request to Server A. Server A stores the cart in its memory. The next request goes to Server B, and Server B has no cart for this user. The user sees an empty cart.

The fix is to move state out of the servers. Every server stores sessions and carts in one shared store, like Redis or a database. A server that keeps no user data between requests is called stateless.

Now every server reads the same cart. The load balancer can send any request to any server. Adding a server is as simple as starting it and adding it to the load balancer.

A stateful server loses the cart when the next request goes elsewhere, while stateless servers all read the cart from one shared store

Some load balancers can send a user back to the same server every time. This is called a sticky session. It hides the problem, but it does not fix it. If that server fails, its users lose their sessions. Many busy users can also end up on one server and overload it. The stateful vs. stateless architecture lesson covers this in more detail.

Scaling the Database

Stateless app servers are easy to scale. The database is usually harder to scale because it now holds all the state. It often becomes the bottleneck, which is the one part that limits the capacity of the whole system.

Teams usually scale a database in steps, starting with the simplest one.

1. Add a cache. A cache is fast storage in memory, like Redis, that keeps copies of data that is read often. Suppose 80 percent of reads ask for the same popular items. The cache answers those reads, and the database handles only the rest. The caching chapter covers this in detail.

2. Add read replicas. A read replica is a copy of the database that serves read requests. The primary database accepts all writes and copies them to the replicas. This helps most when reads are much more common than writes, which is true for many apps. A replica can be a little behind the primary, so a read may briefly return old data.

3. Split the data into shards. When writes or data size outgrow one machine, the data is split across several databases. Each part is called a shard, and the method is called sharding or partitioning. For example, users with IDs from 1 to 1,000,000 go to shard 1, and the next million go to shard 2. Each shard holds part of the data and takes part of the writes.

Sharding adds the most work. A query that needs data from many shards is slower and harder to write. Moving data when you add a shard is also hard. So teams shard only when the simpler steps are not enough. The data partitioning chapter explains the methods.

Teams scale a database in steps, from one machine to a cache, then read replicas, and finally shards

Other Ways to Scale

Content delivery network. A CDN is a group of servers in many cities that stores copies of static files, like images, videos, and scripts. Users download these files from a nearby CDN server instead of from your servers. This removes a large share of traffic from your system. The CDN chapter explains how it works.

Message queues. Some work does not need to finish before the user gets a response. Examples are sending an email, creating an invoice, or resizing a photo. A message queue stores these jobs until a worker is ready. Workers are separate machines that take jobs from the queue and process them. When the queue grows, you add more workers. The user gets a fast response, and the slow work happens in the background.

Autoscaling. Autoscaling means adding or removing servers automatically, based on a measured number like CPU use. For example, one rule adds a server when average CPU stays above 70 percent for 5 minutes. Another rule removes a server when average CPU stays below 30 percent. A food delivery app can then run 20 servers at lunch and 5 servers at night. Autoscaling needs stateless servers because servers start and stop all the time.

A web application built to scale, with a CDN, stateless app servers, a cache, a database, and a queue with workers

Finding the Bottleneck

Adding machines helps only if the machines were the problem. Suppose a team doubles its app servers from 10 to 20. If every request still waits on one busy database, capacity hardly grows.

Linear scaling is the ideal case, where doubling the machines doubles the capacity. Real systems get less than that. Shared parts, like one database, limit the gain. Machines also spend time talking to each other.

So teams measure before they scale. They use load testing, which means sending a large amount of test traffic to a copy of the system. They raise the traffic step by step and watch each part. The first part to reach its limit is the bottleneck. For example, database CPU reaches 100 percent, or queries start taking seconds instead of milliseconds.

The team fixes that part and tests again. After each fix, the next bottleneck appears in a different part.

Key Takeaways

Scalability is the ability to handle a growing workload by adding resources, while the system stays fast.
Vertical scaling (scaling up) makes one machine bigger. It is simple, but it has an upper limit, usually needs downtime, and is a single point of failure.
Horizontal scaling (scaling out) adds machines and spreads the work across them. It has no fixed limit and keeps working when one machine fails, but the app must be designed for it.
Horizontal scaling needs stateless servers. Keep sessions and carts in a shared store.
The database is often the bottleneck. Scale it in steps: a cache, then read replicas, then shards.
CDNs and message queues move work away from the core system. Autoscaling matches the number of servers to the traffic.
Load test before you scale. Adding machines does not help when the bottleneck is somewhere else.

A scalable design does not need every technique in its first version. It needs parts that can grow without a rewrite, like stateless servers and state kept in shared stores. It also needs a clear idea of which part will become the bottleneck next. The next lesson, Availability, covers how to keep such a system running when some of its machines fail.

Practice Questions

Try each question first, then open the answer.

1. One app server handles 400 requests per second. Peak traffic is 3,000 requests per second, and you want to handle 25 percent more than the peak. How many servers do you need?