Throughput vs Latency: What Each Measures and the Trade-Off

Latency is how long one request takes. Throughput is how many requests finish per second. They measure different things, and raising one often lowers the other.

Most systems cannot maximize both. Batching and queuing raise throughput and add waiting time to each request. Interviewers ask about this because your answer shows whether you know what the system is for.

This article defines both numbers, shows the trade with a worked example, and gives the rule to state in an interview.

What is latency?

Latency is the time one request takes from start to finish. You measure it in milliseconds. A search that returns in 40 ms has a latency of 40 ms.

Latency is a per-request number, so every request carries its own value.

Say 99 requests return in 20 ms and one takes 3 seconds. The average is about 50 ms, but the one slow user still waited 3 seconds.

What is throughput?

Throughput is how much work the system finishes per unit of time. You measure it in requests per second (RPS), transactions per second, or megabytes per second.

Throughput is a whole-system number. It describes capacity, not the experience of one user.

A service running at 5,000 RPS finishes 5,000 requests every second. That number says nothing about how long any one of them waited.

Throughput vs latency compared

The last two rows matter most. The same technique often appears on both sides.

Why improving one can hurt the other

Batching is the clearest case. Batching means holding several requests and processing them together.

A database that writes one row at a time might handle 2,000 writes per second. Group the writes into batches of 100 and it might handle 20,000 per second.

Throughput went up 10 times. But each write now waits for its batch to fill. A write that took 5 ms may now take 50 ms.

The same pattern appears in queues. A queue holds requests during a traffic spike and keeps servers busy, which raises throughput. Every message also waits in line, which raises latency.

Retries follow the same pattern. A retry recovers one failed request, but it adds load for every other request in the system.

If you want to practice these trade-offs on real case studies, Grokking the System Design Interview walks through the designs where this choice decides the architecture.

How the two are connected

Little's Law links them. Little's Law says the number of requests inside the system equals throughput times latency.

Write it as L = W x T. L is requests in flight, W is throughput, and T is average latency.

The useful reading is this. If requests in flight stay fixed and latency doubles, throughput halves.

That is why one slow dependency lowers the capacity of the whole service. Threads wait instead of finishing work.

Which one should you optimize?

Optimize the number your users notice. Add servers for the other one.

On a checkout page or a search box, the user is waiting on latency. Someone who waits 3 seconds often leaves. Optimize latency first.

For a nightly report, a log pipeline, or a video encoder, nobody watches a single item. Optimize throughput first.

Name your choice out loud in an interview. "This is a user-facing read path, so I am optimizing for p99 latency and accepting lower throughput per server." That one sentence does more for your score than another box on the diagram.

Where the trade-off appears in a design

The choice is not abstract. It appears at four places in almost every design.

Load balancer. Spreading requests across more servers raises throughput. The extra hop adds a few milliseconds of latency.

Cache. A cache hit cuts latency and raises throughput at the same time. This is the rare case where both numbers improve, which is why caching is usually the first optimization to consider.

Queue. A queue protects the database during a traffic spike and raises throughput. The work behind it finishes later, so the time from request to result rises.

Read replica. Extra copies of the database raise read throughput. Replication lag means a reader can see stale data, which is a correctness cost rather than a latency cost.

Name the part and the direction when you talk through a design. "I am adding a queue here, which raises write throughput and means the user gets a confirmation before the work is finished."

How to measure latency correctly

Use percentiles, not averages. A percentile is the value below which a given share of requests fall.

p50 is the middle request. p99 means 99 percent of requests were faster than this number.

Quote p99. It describes the worst common experience, and it is the number most teams write into a service level objective.

An average of 50 ms with a p99 of 3 seconds is a broken system. The average alone does not show that.

What interviewers ask about this

Three questions are common.

"What is the difference?" Answer in two sentences, then give the unit for each.

"Which are you optimizing here?" Name one, and name what you give up.

"How would you raise throughput without making users wait?" Good answers separate the paths. Keep the read path small and cached. Move the heavy work to a queue the user does not wait on.

The mistake to avoid is treating both as goals. A candidate who says "we want low latency and high throughput" has not made a decision yet.

Frequently asked questions

Is high throughput the same as low latency? No. A system can finish 50,000 requests per second while each one waits 2 seconds. Batching produces exactly that result.

Can you improve both at the same time? Sometimes. Removing wasted work helps both numbers. A cache that answers in 1 ms cuts latency and frees servers to handle more requests.

What is a good latency number? It depends on the path. Under 100 ms feels instant for a web request. A background job can take minutes without anyone noticing.

Is bandwidth the same as throughput? No. Bandwidth is the maximum a link can carry. Throughput is what you actually get after overhead, errors, and retries.

Why does latency rise when a system gets busy? Requests start waiting for a free thread, connection, or disk. That queue time adds to the service time.

Which one matters more in a system design interview? Neither by default. You are scored on naming the right one for the case study and saying what you trade for it.

Related reading

• Learn System Design: a free roadmap covering the foundations, the patterns, and how to practice.
• Demystifying Long-Tail Latency: why p99 behaves so differently from the average.
• Cache Invalidation Strategies: the main tool for cutting latency, and the cost that comes with it.
• Content Delivery Networks (CDN) in System Design: how moving data closer lowers latency.
• The Load Balancer Mistakes That Quietly Kill Your Interview Score: where queue time hides in a design.

Ready to practice this on real designs? Grokking the System Design Interview covers the case studies where you have to choose between latency and throughput.