How Large Language Models Work: Tokens, Embeddings, Attention

A large language model (LLM) does one thing: it reads a sequence of text and predicts the next small piece of text. It splits the input into pieces called tokens, turns each token into a list of numbers, lets every token gather information from the tokens before it through a step called attention, and then scores every possible next token. It picks one, adds it to the input, and repeats the whole process until the answer is complete.

Everything an LLM appears to do, from answering questions to writing code, comes from repeating that single prediction thousands of times. The model has no separate module for facts, grammar, or reasoning. Those abilities are patterns stored in its numbers, learned by predicting the next token across a very large amount of text.

This article follows one short sentence through the whole pipeline: tokenization, embeddings, positions, attention, the transformer block, the output step, and generation. It then explains how the model is trained, why it sometimes states false things with confidence, and how the parts compare.

The running example

We will trace one input from start to finish:

The capital of France is

A trained model should continue this with " Paris". The question this article answers is how a program made of matrix multiplication arrives at that word. Every section below takes the same five words one step further through the model.

What is a token?

A model cannot read letters or words directly. It reads whole numbers. The first step, then, is to cut the text into small pieces from a fixed list and replace each piece with its number in that list (a token). The fixed list is called the vocabulary, and a modern vocabulary holds roughly 50,000 to 200,000 entries.

Common words usually become one token each. Our sentence becomes five tokens, and the leading space is part of each token:

"The"  " capital"  " of"  " France"  " is"
 464     3139      286     4881     318      <- token ids (illustrative)

Rare words are split into smaller parts. A tokenizer might cut "tokenization" into "token" and "ization", and an unusual name into three or four fragments. The exact split depends on the tokenizer, so treat these examples as illustrative.

How the vocabulary is built: byte pair encoding

Most LLMs build their vocabulary with a method that starts from single characters and repeatedly merges the pair that appears most often in the training text (byte pair encoding, or BPE). If "t" followed by "h" is the most common pair, "th" becomes a new entry. If "th" followed by "e" is next, "the" becomes an entry, and so on.

After tens of thousands of merges, frequent words are single tokens and rare words are a few common fragments. This gives the model a fixed vocabulary size while still letting it read any text, including words it has never seen. A word the vocabulary lacks is simply spelled out from smaller pieces.

Why token counts matter

Tokens are the unit of cost and of limits. API providers charge per token, and every model has a maximum number of tokens it can read at once. OpenAI's help center gives a common estimate for English: one token is about four characters, or about three quarters of a word. So 1,000 English words cost roughly 1,300 tokens.

That estimate changes with the content. Code, numbers, and languages other than English usually need more tokens per word. A sentence in a language the tokenizer saw less often during training can cost two or three times as many tokens as the same sentence in English, which makes it slower and more expensive to process.

What is an embedding?

A token id such as 4881 says nothing about meaning. The number for " France" is not close to the number for " Germany" in any useful way. So the model replaces each token id with a long list of numbers that it learned during training (an embedding). A list of numbers like this is called a vector, and its length is called the dimension.

The model stores one vector per vocabulary entry in a large table, and looking up a token simply returns its row. The original transformer described below used vectors of 512 numbers. Large modern models use several thousand.

Similar meaning, nearby vectors

Training arranges these vectors so that tokens used in similar ways end up close together. " France", " Germany", and " Italy" sit near each other, because they appear in similar sentences. " Paris" sits near " Berlin" and " Rome". Closeness is usually measured by the angle between two vectors (cosine similarity), where a value near 1 means the two point in almost the same direction.

Directions in this space can also carry meaning. The often-cited example is that the vector from " France" to " Paris" is similar to the vector from " Germany" to " Berlin". Real embedding spaces are not that tidy, but the idea is correct: relationships between concepts become geometric relationships between vectors.

How does the model know word order?

The steps that follow treat the tokens as a set, so without extra information the model could not tell "dog bites man" from "man bites dog". To fix this, the model adds information about each token's position to its vector before any other processing (positional encoding).

The original transformer added a fixed pattern of sine and cosine waves, a different pattern for each position. Many current models instead rotate parts of each vector by an angle that depends on the position (rotary position embeddings). The method varies, but the purpose is the same: after this step, " France" in position 4 has a slightly different vector from " France" in position 1.

What is self-attention?

Self-attention is the step that lets each token collect information from the other tokens in the input. Without it, the vector for " is" would know nothing about " France", and the model could not tell which capital the sentence asks about.

The idea comes from the paper "Attention Is All You Need" by Ashish Vaswani and colleagues at Google, which introduced the transformer architecture. Almost every current LLM is built on that design.

Queries, keys, and values

For every token, the model computes three new vectors by multiplying the token's vector by three learned matrices. Each one has a plain job:

• The query describes what this token is looking for.
• The key describes what this token contains, so other tokens can find it.
• The value is the information this token passes on when another token pays attention to it.

A useful comparison is a search: the query is what you type, the keys are the titles of the documents, and the values are the documents' contents. A token compares its query with every key, decides how much each match matters, and takes a weighted mix of the values.

A worked example with three tokens

Take the last three tokens, " capital", " France", and " is", and follow the token " is". To keep the arithmetic readable, the vectors below have two numbers each. These numbers are illustrative, not taken from a real model.

Suppose the query for " is" is [1, 0]. The model computes a score for each token by multiplying matching positions and adding them (the dot product). " France" scores 2, " capital" scores 1, and " is" scores 0. The scores are then divided by the square root of the vector length, here about 1.41, which keeps them from growing too large in big models. That gives 1.41, 0.71, and 0.

Next, a function called softmax turns the scores into weights that are positive and add up to 1. It raises e, about 2.72, to the power of each score and divides each result by their total. The weights come out to about 0.58 for " France", 0.28 for " capital", and 0.14 for " is". So the token " is" will take most of its new information from " France".

The new vector for " is" is the weighted sum of the values: 0.58 times [0.9, 0.1], plus 0.28 times [0.2, 0.8], plus 0.14 times [0.1, 0.1]. The result is about [0.59, 0.30]. The vector for " is" now carries information about France and, to a lesser degree, about capitals.

Why a model cannot look ahead

An LLM generates text left to right, so during training it must not see the token it is trying to predict. The model therefore blocks every token from attending to tokens after it (a causal mask). In practice the blocked scores are set to a very large negative number, so softmax gives them a weight of zero.

Multi-head attention

One set of queries, keys, and values can only capture one kind of relationship at a time. The model therefore runs several attention computations side by side, each with its own learned matrices (multi-head attention). The original transformer used 8 heads, each working on 64 of the 512 numbers.

Different heads tend to learn different jobs. One head might link a pronoun to the noun it refers to, another might track which word a verb belongs to, and another might simply look at the previous token. The outputs of all heads are joined together and multiplied by one more matrix to form the final result.

Want to build a solid base before going deeper into models like this? Grokking Modern AI Fundamentals explains the core ideas behind modern AI systems in plain language, from data and training to how models are used in products.

What happens inside a transformer block?

Attention is one half of a repeating unit (a transformer block). The other half is a small neural network applied to each token separately (the feed-forward layer). Attention moves information between tokens, and the feed-forward layer processes the information inside each token. Research suggests that much of a model's stored factual knowledge sits in these feed-forward weights.

Two more pieces keep the block stable during training:

• A residual connection adds the block's input back to its output. Each block therefore makes an adjustment to the vector instead of replacing it, which lets information and training signals pass through many layers without fading.
• Layer normalization rescales each vector so its numbers keep a steady average and spread. Without it, values can grow or shrink layer after layer until training fails.

Stacking the blocks

A model is a stack of many identical blocks, each with its own learned weights. The original transformer used 6 blocks in its encoder. Large modern LLMs use dozens, and the largest use more than one hundred.

Each layer works on the output of the layer before it, so early layers tend to handle surface features such as spelling and grammar, and later layers handle meaning and the relationships between ideas. By the last block in our example, the vector at the position of " is" holds a compressed summary of the whole sentence that points strongly toward a city name.

How does the model choose the next token?

After the last block, the model takes the vector at the final position and multiplies it by one large matrix. The result is one score per vocabulary entry, so a vocabulary of 100,000 tokens gives 100,000 scores (logits). Softmax then turns the scores into probabilities that add up to 1.

For our sentence, the top of that list might look like this, with illustrative numbers:

Sampling, temperature, and top-p

The model does not always pick the most likely token. Picking the top token every time (greedy decoding) often produces repetitive text, so most systems draw a token at random according to the probabilities (sampling).

Two settings control that draw:

• Temperature divides every score before softmax. A temperature of 2 halves the scores, which flattens the probabilities: " Paris" falls from 0.91 to about 0.70 and " Lyon" rises to about 0.20. A temperature of 0.5 doubles them, and " Paris" rises above 0.99. Low temperature gives focused, predictable text, while high temperature gives more varied text with more mistakes.
• Top-p keeps only the smallest group of top tokens whose probabilities add up to p, and samples from that group. With top-p set to 0.9, our example keeps " Paris" alone, because it already reaches 0.91. This removes the long tail of unlikely tokens that would otherwise be chosen now and then.

Generating a full answer

Once a token is chosen, the model appends it to the input and runs again to predict the token after it (autoregressive generation). A 300-token answer therefore needs 300 passes through the model. Generation stops when the model produces a special end token or reaches a length limit set by the caller.

What is the context window?

The maximum number of tokens a model can consider at once, counting both the input and the generated output, is its context window. Current models range from tens of thousands of tokens to more than a million. Any text beyond the window is invisible to the model, which is why long conversations are often summarized or trimmed.

Attention is the reason long contexts are expensive. Every token compares its query with the keys of every earlier token, so the work grows with the square of the length. Doubling the input from 10,000 to 20,000 tokens roughly quadruples the attention work.

The KV cache

During generation, the keys and values of earlier tokens do not change when a new token is added. So instead of computing them again on every pass, the model stores them in memory and reuses them (the KV cache). Each new token then computes only its own query, key, and value and attends to the stored ones.

The cache makes generation far faster, but it uses a lot of memory. Its size grows with the number of layers, the number of heads, the vector size, and the context length, and it often decides how many requests a GPU can serve at once. Serving systems manage it carefully, which is covered in Designing LLM Inference Systems.

How is a large language model trained?

Pretraining: predicting the next token

The model starts with random weights. It is then shown a very large collection of text, often trillions of tokens of web pages, books, and code, and asked to predict each next token (pretraining). After every prediction, the training process measures how much probability the model gave to the correct token (the loss). It then adjusts every weight slightly to raise that probability (gradient descent, using a method called backpropagation).

No one labels this data, because the correct answer is simply the next token in the text. That is why the method scales to so much data. Repeated over trillions of examples, next-token prediction forces the model to learn spelling, grammar, facts, and patterns of reasoning, because all of them make the next token easier to predict.

Fine-tuning and RLHF

A pretrained model continues text, but it does not follow instructions well. Ask it a question and it may answer with more questions, because that is a common pattern on the web. So the model is trained further on examples of instructions paired with good responses written by people (supervised fine-tuning).

A further step uses human preferences. People compare pairs of model answers and pick the better one, a second model learns to predict those choices, and the LLM is trained to produce answers that score well (reinforcement learning from human feedback, or RLHF). The InstructGPT paper by Long Ouyang and colleagues at OpenAI reported that human raters preferred the outputs of a 1.3 billion parameter model trained this way over the outputs of the 175 billion parameter GPT-3. Training on what people wanted mattered more than a model more than 100 times larger.

Why do LLMs hallucinate?

A model sometimes states false information with complete confidence (hallucination). The mechanism explains why. The model's only objective is to produce a likely next token, and nothing in that process checks the output against a source of truth.

When the model has seen a fact many times, like the capital of France, the correct token gets a high probability. When it has seen a fact rarely or never, such as the birth date of an obscure person, the probabilities spread across many plausible tokens. Sampling still picks one, and the result reads as fluently as a true answer, because fluency and truth come from different parts of the training signal.

Several practices reduce the problem. Retrieval systems place relevant documents in the context so the answer can be copied from a source, as described in System Design for RAG. Lower temperature reduces random choices, and asking the model to say "I do not know" when the context lacks an answer helps further. None of these remove the cause, so production systems also measure accuracy with tests, as explained in LLM Evaluation.

The parts of an LLM compared

Each part has one narrow job, and none of them understands language on its own. The ability to answer comes from stacking these simple operations many times and training all of their weights together on one objective: predict the next token well.

Frequently asked questions

How do large language models work in simple terms? An LLM splits text into tokens, turns each token into a vector of numbers, and uses attention layers to let every token gather context from the tokens before it. It then scores every possible next token, picks one, and repeats. A full answer is many single-token predictions in a row.

What is a token in an LLM? A token is a piece of text from the model's fixed vocabulary, often a whole common word or a fragment of a rare one. In English, one token averages about four characters, or about three quarters of a word. Providers charge and set limits per token.

What is the difference between a token and an embedding? A token is a number that identifies a piece of text. An embedding is the learned vector the model looks up for that number. The token says which piece it is, while the embedding carries what the model has learned about its meaning.

What does attention do in a transformer? Attention lets each token collect information from other tokens. Each token compares its query with the keys of earlier tokens, turns the scores into weights, and takes a weighted mix of their values. This is how " is" learns that the sentence is about France.

Why do LLMs make things up? The model is trained to produce likely text, not true text. When it has seen a fact rarely, the probabilities spread across several plausible answers, and sampling still picks one that reads well. Retrieval, lower temperature, and evaluation reduce the problem but do not remove it.

What is the KV cache? It is memory that stores the keys and values of tokens the model has already processed. Each new token reuses them instead of computing them again, which makes generation much faster. The cost is GPU memory, which grows with context length.

What does temperature do? Temperature divides the scores before softmax. A low value sharpens the probabilities, so the model almost always picks the top token. A high value flattens them, so less likely tokens are chosen more often.

Related reading

• What Is Generative AI? How Models Create Text, Images and Code
• Generative AI vs Machine Learning vs Deep Learning: How They Relate
• Designing LLM Inference Systems: Batching, Memory, and GPUs
• System Design for RAG: Vector Databases, Chunking, and Re-ranking
• LLM Evaluation: How to Test an LLM App Before It Ships

Ready to understand the ideas behind modern AI, not just the terms? Grokking Modern AI Fundamentals covers how models learn, how they are built into products, and how to reason about their limits.