Latency vs Throughput

Read that curve carefully, because it explains a lot of production behaviour:

• Up to roughly 70 percent utilisation, latency is nearly flat. Adding load costs almost nothing.
• Between 70 and 90 percent, latency starts climbing noticeably while throughput gains get smaller.
• Past about 90 percent, throughput flattens out entirely and latency goes vertical. You are getting no extra work done and every user is waiting.

This is why teams run servers at 50 to 70 percent utilisation rather than 95 percent. That apparently wasted headroom is what keeps response times stable when traffic spikes. Running "efficiently" at 95 percent means a ten percent traffic increase makes the system feel broken.

It is also the honest version of the trade-off: you can convert spare capacity into throughput, and you pay for it in latency. Batching is the clearest example. Collecting 100 records and writing them in one database call is far more efficient per record, so throughput rises. But the first record in the batch now waits for the other 99 to arrive before anything happens, so its latency gets worse.

Averages Lie

Here is the mistake that separates a junior answer from a senior one. If someone tells you "our average latency is 200 ms", you have learned almost nothing.

An average hides the shape of the distribution. Latency distributions are not symmetric. Most requests are fast and a small number are very slow, because of cache misses, garbage collection pauses, retries, lock contention, or a cold connection. That long tail barely moves the average and is exactly what users complain about.

So real systems are measured in percentiles:

• p50 (the median): half of requests are faster than this. It describes the typical experience.
• p95: 95 percent of requests are faster. One in twenty is slower.
• p99: 99 percent are faster. This is the tail, and it is usually the number in a service level objective.

A system with a p50 of 100 ms and a p99 of 4 seconds is a system where one request in a hundred is unusably slow.

Why the Tail Matters More Than It Looks

A p99 of one second sounds tolerable. One user in a hundred waits a second. Not ideal, but survivable.

It is worse than that, because one page view is rarely one request. If loading a screen makes 100 backend calls and each has a p99 of one second, the chance that all 100 come back fast is 0.99 to the power of 100, which is about 0.37. So roughly 63 percent of page loads hit at least one slow call.

The tail is not a rare edge case. Once you fan out, it becomes the typical experience. This is why large systems invest so heavily in tail latency, and why quoting an average in an interview invites a follow-up you may not want.

When you state a latency requirement, state it as a percentile: "p99 under 200 ms for the timeline endpoint" is a real requirement. "Fast" is not.

How to Improve Latency

1. Move the data closer. A CDN serves users from a nearby edge location, which removes physical distance from the round trip.
2. Cache. A cache hit skips the slow work entirely.
3. Avoid extra round trips. Every sequential network call adds its full latency to the total. Batch related calls, or fetch them in parallel instead of one after another.
4. Index and tune queries. Most surprising latency in a backend is a query doing more work than it needs to.
5. Do less on the request path. Push anything the user does not need immediately into a background job.
6. Keep utilisation moderate. As the curve above shows, this is often the cheapest latency fix available.

How to Improve Throughput

1. Scale horizontally. Add instances and put them behind a load balancer. This is the main lever.
2. Increase concurrency. More workers, threads or connections, so more requests are in flight at once. Little's Law tells you how many you need.
3. Batch. Group work to reduce per-item overhead, accepting some latency in exchange.
4. Process asynchronously. Put work on a queue and let consumers drain it at their own rate, which also absorbs spikes.
5. Partition the data. Sharding spreads load across machines so no single database is the ceiling.
6. Cache. Yes, again.

Why Caching Appears in Both Lists

It is worth being explicit about this, because it looks like a contradiction. A cache helps latency and throughput, through two different mechanisms.

A cache hit is faster than the work it replaced, which is the latency win. It also means the database never did that work, so the capacity it would have consumed is now free for other requests, which is the throughput win. One video transcoded once and served from cache to a thousand viewers is a thousand times less work for the origin.

Most good techniques help both. The ones that genuinely trade are the ones that add waiting on purpose: batching, buffering, and queueing all raise throughput by making individual requests wait longer.

💡 In the interview: when you are given a scale requirement, convert it into both numbers before you design. "One million daily active users, each loading the feed five times a day" becomes roughly 58 requests per second average, and you should assume peak is several times that. Then apply Little's Law to size the fleet. When you state a latency target, always attach a percentile. Saying "p99 under 300 ms, and I'm sizing for 3x average traffic so we sit near 60 percent utilisation at peak" tells an interviewer more than a paragraph of description.

Key takeaway: latency is the time for one request, throughput is requests completed per second, and they are independent axes rather than opposites. Concurrency links them: concurrency = throughput × latency. They trade off because latency climbs steeply as utilisation approaches capacity, which is why systems run with headroom, and because techniques like batching buy throughput by making requests wait. Always measure latency in percentiles, since averages hide the tail that users actually notice.