Introduction: LLM Serving & Inference

The Cost Flips Over

Up to now, every system in this course predicted something. A score, a label, a yes or no. Will this user click? Is this payment fraud?

Starting now, the system writes something instead. An answer, a summary, a first draft.

That one shift, from predicting to generating, changes the economics of the whole design. Inference is the word for running a trained model to get an answer.

In earlier examples, a small prediction model could be inexpensive to run, while data processing contributed much of the cost. That balance depends on the workload.