Scalability in Distributed Systems: What It Means and How to Get It

Scalability is the ability of a system to handle more work by adding more resources. The test is simple. Double the machines, and see whether throughput nearly doubles while response time stays flat.

Here is the concrete version. A service runs on 10 servers and handles 1,000 requests per second at 100 ms. Add 10 more servers. If it now handles close to 2,000 requests per second at 100 ms, it scales well.

Real systems rarely reach that. Going from 10 to 20 servers often gives 1,600 requests per second, not 2,000. The gap is the cost of coordination between nodes.

The Two Ways to Scale

Vertical scaling means making one machine bigger. You add CPU cores, memory or faster disks.

Horizontal scaling means adding more machines and splitting the work between them. Distributed systems are built for this.

	Vertical scaling	Horizontal scaling
What you add	A bigger machine	More machines
Upper limit	The largest machine you can buy	No hard limit
Code changes	Usually none	Sharding, routing, coordination
Failure effect	One machine down means full outage	One node down means reduced capacity
Cost curve	Rises steeply at the top end	Close to linear

Vertical scaling is easier and it is often the right first step. It stops working when the biggest machine is still too small.

What Actually Limits Scaling

Two things stop a system from scaling in a straight line.

The first is serial work. Any part that must run on one node caps the whole system. If 5 percent of the work must run on one node, the best speedup is 20 times.

The second is crosstalk. Nodes must agree on state, so they exchange messages. The number of pairs grows fast. Ten nodes have 45 pairs, and 100 nodes have 4,950.

Past a point, extra nodes spend more time coordinating than working. Throughput stops rising and can fall.

Why Databases Are the Hard Part

Stateless web servers are easy to scale. You run more copies behind a load balancer and the work splits itself.

Databases hold state, so a copy is not free. Every write must reach every replica that serves it.

Reads scale first. You add read replicas and send queries to them. This works because reads do not conflict with each other.

Writes scale by sharding. You split rows across machines by a key, such as user ID. Each shard owns part of the data.

Sharding creates three new problems. A query that spans shards must contact many machines. A popular key makes one shard hot. A transaction across two shards needs a distributed protocol.

A sharded database is called scalable only for the access patterns it was sharded for. The same logic applies to a distributed file system such as HDFS.

The Trade-Off Scaling Creates

More nodes means more failures. With 1,000 machines, something is broken every day.

So a scalable system must also be a fault tolerant system. That usually means replicating data, which then means choosing between strong consistency and low latency during a network fault.

You also pay in tail latency. A request that touches 50 shards waits for the slowest of the 50. If each shard is slow 1 percent of the time, that request is slow about 39 percent of the time.

Coordination between nodes often needs a distributed lock, which adds one more round trip to every write.

How to Measure It

Do not report a single number. Measure three things as load rises.

Throughput is requests per second. Latency is the p99 response time, not the average. Cost is dollars per million requests.

A system scales well when throughput rises with machines, p99 stays flat, and cost per request does not climb.

How to Prepare

Start with the definition and one number. Say what scalability means, then give a before and after figure. Vague answers lose marks in a system design interview.
Know vertical and horizontal cold. Read the deeper comparison of ways to scale a system.
Learn the failure side too. Scaling and reliability are the same topic, covered in distributed system design challenges.
Practice the building blocks. Grokking System Design Fundamentals walks through load balancing, caching, replication and sharding.
Then apply them to full problems. Grokking the System Design Interview uses these parts in end to end designs.