Training vs Inference

Notice that the two pipelines share a single link. Training produces one output: a file of parameters. That file is copied to servers, and every later request uses it. The training side runs once per model version, and the inference side runs again for every request.

Part 2: A Recipe Analogy

A bakery that develops a new cake recipe goes through the same two phases. Each part of the bakery matches one part of a model:

• Developing the recipe is training. The baker bakes many test cakes, tastes each one, and adjusts the amounts. This takes months and a great deal of flour.
• The finished recipe card is the trained model. It is a fixed list of amounts, just as the model is a fixed list of parameters.
• Baking one cake for a customer is inference. The baker follows the card and does not change it.
• The ovens are the GPUs, the processors that do the math. Both jobs need them, but developing the recipe needs far more oven time.
• Filling one oven with several trays at once is batching, which Part 5 explains.

You pay for the test cakes once. After that, each cake costs only its own ingredients and oven time.

Part 3: Why Training Costs So Much

Training a large model is expensive for three reasons, and each reason multiplies the others.

The data is huge. A large language model learns from trillions of tokens. A token is a small piece of text, often a whole word and sometimes part of one.

The model is huge. A large model has billions of parameters, and often far more. Each one participates in every training step.

Each training step does more work than inference. Training repeats the four steps from How a Model Learns: predict, measure the loss, compute the gradient, and update the parameters. Inference does only the first step.

The diagram below compares one training step with one inference step.

The predict step is the forward pass. Computing the gradient is called the backward pass, because it works back through the layers, and it takes about twice the work of the forward pass. Training also needs much more memory, because it must hold the gradients and other working numbers as well as the parameters.

A common estimate quantifies this difference. Training takes about six calculations per parameter for each training token. Producing one token at inference takes about two calculations per parameter.

Here is a worked example with illustrative numbers. Suppose a model has 10 billion parameters and trains on 1 trillion tokens. Compare that to a 500-token reply, not counting the question. By this estimate, training the model takes as much computing as about six billion such replies.

The largest models use far more parameters and tokens than this example. Their training runs on thousands of GPUs for weeks or months. Teams also run many smaller experiments, and some failed runs, before the final one. For these models, public estimates put the total cost in the tens or hundreds of millions of dollars.

Part 4: Why GPUs Do the Work

Most of the work inside a neural network is multiplying large tables of numbers. A table of numbers like this is called a matrix. Each multiplication is simple, but there are billions of them, and many of them can run at the same time.

A CPU is a computer's main processor. It has a small number of powerful cores, each built to finish one task quickly. A GPU, or graphics processor, has thousands of simpler cores that run in parallel. GPUs were first built to color millions of screen pixels at once, and the same design suits matrix math.

GPUs do the work in both phases, but at very different scales. A large training run links thousands of GPUs with fast networks, all working on one job. Running a trained model so that applications can send it requests is called serving, and one copy of a model needs only one GPU or a few. As Neural Networks in One Lesson showed, the parameters must fit in memory, usually GPU memory, while the model runs.

Part 5: Batches in Training and in Serving

A group of inputs that a model processes together, in one pass, is called a batch. Batches appear in both phases.

In training, the model reads a batch of examples and then updates its parameters once for the whole batch. This keeps the GPUs busy, and it makes each update steadier.

In serving, a batch holds requests from different users. To produce each new token, the GPU must read every parameter from memory, and this reading is often slower than the math. If a server handles 32 requests in one batch, it reads the parameters once and uses them for all 32.

Serving systems also store part of the attention work for earlier tokens, so it is not computed again for each new token. This store is called the KV cache, and The Transformer at Box Level explains it.

The diagram below shows a server batching requests.

Sharing one read of the parameters is a major reason a busy server can answer each request so cheaply.

Batching has a trade-off. The number of tokens a server produces per second, across all users, is called throughput. Latency is the time one user waits for an answer. Larger batches raise throughput, but each request may wait a little longer. Many modern serving systems reduce this wait by adding new requests to a running batch as other requests finish.

Part 6: One Request, Step by Step

A request to a hosted model goes through four steps.

Step 1: Wait and join a batch. The request waits briefly in a queue, and then it joins a batch with other requests.

Step 2: Read the input. The model processes all the input tokens together. The model knows them all in advance, so the GPU can process them at the same time. This phase is called prefill.

Step 3: Write the reply. The model produces the reply one token at a time. Each new token needs another forward pass, because it depends on the tokens before it. This phase is called decode.

Step 4: Return the tokens. The tokens go back to your application, often as a stream, so the user sees the text appear as it is written.

Step 3 explains two patterns you will see. First, many providers charge more for output tokens than for input tokens, partly because output is written one token at a time. Second, long replies take longer than long questions. The lesson How Text Gets Generated examines this loop closely.

Notice what does not happen. The parameters don't change, so the model doesn't learn from your request. The model also learns nothing about events after it collects its training data. That limit is the knowledge cutoff from What GenAI Can and Can't Do Today. A later training run produces a new model, not a change to the model you called.

Part 7: What This Changes for an Engineer

The table below sums up the two phases.

Your costs are inference costs. Most teams never train a large model. They pay a provider per token, or they pay for GPU time to run an open-weight model. Prices vary widely by model. For example, suppose input costs 1 per million tokens, and output costs 4 per million tokens. A request with 1,000 input tokens and 300 output tokens then costs about 0.0022. At one million requests a month, that becomes about 2,200. The later lesson "Cost, Latency and Rate Limits" covers this math in depth.

Changing behavior rarely means training. Better instructions, examples, and your own documents in the request all change the output, and none of them touches the parameters. Fine-tuning is a smaller, extra training run on top of a trained model. The lesson Prompting vs RAG vs Fine-Tuning compares these options.

Self-hosting changes the shape of the cost. On your own GPUs, you pay for every hour the GPU is on, whether it is busy or idle. A busy server batches many requests, so each one is cheap. A quiet server spreads the same hourly cost over very few requests.

A new model version is a new training run. Its parameters differ, so its answers do too. Name the exact model version in your code, and test before you switch.

Common Mistakes

Mistake 1: Assuming a custom feature needs a custom-trained model. Most changes in behavior come from clearer instructions or from your own data in the request. Treat fine-tuning as a later option, not the first one.

Mistake 2: Treating a fraction of a cent as free. Small costs grow with traffic and with reply length. Estimate tokens and volume before launch.

Mistake 3: Switching to a new model version without testing. A new version comes from a new training run, so the same request can get a different answer. Run your test cases on it before users see it.

Key Takeaways

• Training builds the model once, and inference uses it on every request. Training sets the parameters, and inference only reads them.
• Training is expensive because of scale and extra work. Huge data, billions of parameters, and a backward pass on every step need thousands of GPUs for weeks.
• Inference is cheap per request because the work is small and shared. It needs one forward pass per new token, and batching lets many users share each GPU.
• Your bill comes from inference. The cost grows with the tokens in each request and with the number of requests.

The next lesson, Models Are Probability Machines, looks at what each forward pass returns actually. It explains why the output is a list of probabilities, and why the same request can get different answers.