Serving Optimizations

Know the Vocabulary

This lesson uses the terms below many times. Read them once now, and come back here when a later section needs one.

• Token: The unit of text a language model reads and writes. One token is about three quarters of an English word.
• Frontier model: The most capable model a team can use. It is usually also slower and more expensive per token than smaller models.
• Small model: A model with far fewer learned numbers. It is faster and cheaper, and it handles many easy requests well.
• Prefill: The first phase of a request