LLM Inference at Scale

1. A Workload With Unusual Properties

Serving a language model is a capacity problem with a shape that breaks most of the instincts the rest of this course builds.

Five properties account for it.

• A request is not one unit of work. It is a prefill phase that processes the whole input at once, then a decode phase that produces one token at a time. The two have completely different resource profiles.
• Duration is unknown in advance. You do not know how many tokens the answer will be until it ends, which makes every queueing and timeout assumption shakier than usual.