LLMs at Interview Altitude

1. The Component View

For system design purposes, think of a large language model as a function. You put tokens in, you get tokens out, one token at a time. A token is the unit of text the model reads and writes, roughly three-quarters of an English word. You send the model a sequence called the prompt: your instructions, the conversation so far, maybe some documents. The model then writes a continuation by predicting the next token, tacking it on, and doing it again.

Hold on to that one sentence: it writes one token per step