Designing LLM Inference Systems: Batching, Memory, and GPUs

Slow software is frustrating.

We have all experienced the irritation of typing a query into a chatbot and staring at a blinking cursor for seconds on end.

When the text finally appears, it might trickle out so slowly that you lose your train of thought. This sluggishness is a major user experience problem.

On the other side of the screen, the engineers running that chatbot face a different problem.

The hardware required to run Large Language Models (LLMs) is incredibly expensive. Graphics Processing Units (GPUs) with high memory bandwidth cost thousands of dollars.

If an engineer dedicates a whole GPU to a single user to make it fast, the operational costs become unsustainable.

This creates the central conflict of AI system design. You must balance latency (speed for the user) against throughput (cost efficiency for the system).

For junior developers and computer science students, understanding this trade-off is essential. It is the difference between a toy project and a scalable production system.

In this post, we will tear down the architecture of a high-performance inference platform. We will look at how modern tools like vLLM use clever memory management and scheduling to get the best of both worlds.

Defining the Core Metrics

Before we design the system, we need to define exactly what we are measuring.

In standard web development, we often look at simple response times.

In the world of Generative AI, metrics are more nuanced because the response is generated over time.

Latency

Latency is the speed of the system from the perspective of a single user. We break this down into two critical numbers.

Time to First Token (TTFT): This is the time elapsed between the user hitting "Enter" and the first character appearing on the screen. It measures how responsive the system feels. If TTFT is high, the user wonders if the app is broken.

Time Per Output Token (TPOT): Once the generation starts, this metric measures the time it takes to generate each subsequent token. A "token" is roughly 0.75 words. If TPOT is high, the text appears to stutter or stream slowly.

Throughput

Throughput is the measure of total system capacity. We measure this in Tokens Per Second (TPS) across all concurrent users.

High throughput means your hardware is working hard. It means you are serving more customers with fewer GPUs.

This is the metric that business teams care about because it directly correlates to profit margins.

The Trade-off: Here is the hard truth. To get the highest throughput, you generally need to process many requests at once. But processing many requests at once usually slows down the specific response time for each individual user. Your job as a system architect is to find the "sweet spot" where the system is cheap enough to run but fast enough to use.

The Anatomy of an LLM Request

To understand why this trade-off exists, we have to look at how an LLM actually works under the hood. It is not a uniform process.

Every request goes through two distinct phases that place different demands on your hardware.

Phase 1: The Prefill (Compute-Bound)

When a request first arrives, the model receives your prompt. It processes all the input tokens in parallel. It does this to build the initial "understanding" or state of the conversation.

This phase requires a massive amount of raw calculation. The GPU cores are crunching numbers as fast as they can. We call this a Compute-Bound task. The speed limit here is how many floating-point operations the GPU can do per second.

Phase 2: The Decode (Memory-Bound)

This is where the model generates the answer.

LLMs are auto-regressive. This means they generate one token, add it to the list, and then use the new list to generate the next token. They do this one by one.

This phase behaves very differently. To generate just one single token, the GPU has to read the entire model from its memory.

For a large model, this could be over 100 gigabytes of data. It moves all that data to the processor, does a tiny bit of math to find one word, and then has to do it all over again for the next word.