LLMs at Interview Altitude

The Component View

For system design purposes, think of a large language model as a function. Tokens go in, tokens come out, one token at a time.

A token is the unit of text the model reads and writes, roughly three-quarters of an English word.

You send the model a sequence called the prompt: your instructions, the conversation so far, maybe some documents. The model then writes a continuation. It predicts the next token, adds it to the text, and does it again.

In ordinary autoregressive decoding, the model generates one token per step