What Is Generative AI? How Models Create Text, Images and Code

Generative AI is software that creates new content, such as text, images, audio, or code, after learning patterns from a very large number of examples. Most older AI systems choose a label for an input, for example "spam" or "not spam". A generative system produces something that did not exist before, for example a reply to an email or a picture of a red bicycle on a beach.

The idea behind every generative model is the same. During training, the model studies millions or billions of examples and learns which pieces tend to appear together. When you use it, the model builds new output one small piece at a time, and at each step it picks a piece that fits the patterns it learned. For text, the pieces are fragments of words. For images, the model starts from random noise and removes it in many small steps.

This article explains what generative AI is, how it creates text, images, and code, how the models are trained, what they do well and badly, and how real products use them.

What is generative AI?

A generative model is a program that has learned the patterns in a type of data well enough to produce new data of the same type. A text model has read a large part of the public internet, so it can write a paragraph that reads like human writing. An image model has seen hundreds of millions of pictures with captions, so it can draw a new picture that matches a caption you write.

Three words come up in every discussion of generative AI, so it helps to define them first:

• Model. A large mathematical function with billions of adjustable numbers inside it. Training sets those numbers. When people say "the model", they mean this trained function.
• Prompt. The input you give the model, such as a question, an instruction, or a description of an image. The model's output is a continuation of, or a response to, this input.
• Parameters. The adjustable numbers inside the model. A model with 7 billion parameters has 7 billion of them. More parameters usually means the model can learn more patterns, and it also means each answer costs more to compute.

The word "generative" separates these systems from discriminative ones. A discriminative model (one that tells categories apart) answers a narrow question about an input: is this transaction fraud, which digit is in this image, will this user click. A generative model answers with new content. Both kinds learn from data, and both are part of machine learning. For a full map of how generative AI, machine learning, and deep learning relate, see Generative AI vs Machine Learning vs Deep Learning.

How does generative AI create text?

A text model writes one small piece at a time and predicts each piece from everything before it. That one sentence describes almost every chatbot and writing assistant in use today.

Text is split into tokens

The model does not read whole words. It reads tokens, which are short chunks of text: a common word, part of a longer word, a space, or a punctuation mark. "Unbelievable" might become three tokens, "un", "believ", and "able". OpenAI's documentation gives a useful estimate for English: one token is about four characters, or about three quarters of a word, so 100 tokens is roughly 75 words.

Each token has a number. The model converts each number into a list of numbers that represents its meaning (an embedding), so that tokens with similar meanings have similar lists. The deeper mechanics of tokens, embeddings, and attention are covered in How Large Language Models Work. For this article, the short version is enough: text goes in as numbers, and the model works on those numbers.

The model predicts the next token

Given the tokens so far, the model outputs a probability for every token in its vocabulary. A vocabulary usually holds tens of thousands of tokens, often between 50,000 and 200,000. Suppose the text so far is "The capital of France is". The model might assign these probabilities to the next token (illustrative numbers):

• " Paris": 0.92
• " a": 0.03
• " located": 0.02
• every other token: 0.03 in total

The program running the model then picks one token, adds it to the text, and asks the model again. This repeats until the model produces a special token that means "stop", or until a length limit is reached. A 300-word answer is about 400 tokens, so it takes about 400 of these prediction steps. That is why answers appear word by word in a chat window: each token is shown as soon as it is chosen.

Sampling and temperature

The step that picks a token from the probabilities is called sampling. The simplest rule is to always take the most likely token, which is called greedy decoding. Greedy decoding is predictable, but it tends to produce flat, repetitive text.

Most systems instead pick at random in proportion to the probabilities. In the example above, " Paris" would be chosen 92 times out of 100. A setting called temperature controls how much the choice is spread out. A temperature near 0 makes the model almost always take the top token, which suits factual answers and code. A temperature near 1 keeps the original probabilities, which gives more varied writing. Higher values make unusual tokens more likely and can make the output incoherent.

This is also why the same question can produce two different answers. The model is the same, but sampling made different choices along the way, and each choice changes everything that follows.

Why next-token prediction produces useful answers

It can seem surprising that predicting the next token produces answers that are relevant and well organized. The reason is scale. To predict the next token well across billions of documents, the model has to learn grammar, facts, the structure of arguments, the format of code, and the patterns of question and answer. None of these is programmed in. They are side effects of becoming very good at one prediction task.

The same fact explains the main weakness. The model learned what text usually looks like, not what is true. When the training data contains the answer many times, the most likely continuation is usually correct. When it does not, the model still produces the most likely-looking continuation, which can be confident and wrong.

How does generative AI create images?

Most modern image generators use a method called diffusion. Instead of placing pixels one at a time, a diffusion model starts from an image made entirely of random noise and removes that noise gradually until a picture appears.

Training: learning to remove noise

Training works backward from real images. The training program takes a real photo and adds a small amount of random noise to it. It repeats this many times until the photo becomes pure static. In the original Denoising Diffusion Probabilistic Models paper by Ho, Jain, and Abbeel, this forward process used 1,000 steps.

The model is then trained on one task: given a noisy image and the step number, predict the noise that was added. If it can predict the noise, the program can subtract it and get a slightly cleaner image. After training on hundreds of millions of images, the model becomes very good at this cleaning step for every level of noise.

Generation: running the process in reverse

To create a new image, the program starts with pure random noise and asks the model to remove a little noise, then repeats. Each pass makes the image a bit clearer. Early passes decide the overall layout and colors. Later passes add edges, textures, and fine detail.

The original method needed about 1,000 passes through the model for each image, which was slow. Later sampling methods reduced this to roughly 20 to 50 passes for good results, which is one reason image generation became practical for everyday products.

How the text description steers the image

A diffusion model on its own would produce a random but realistic image. To follow your description, the model also receives an embedding of your text at every step. During training, it learned which noise to remove so that the result matches a caption. During generation, your description plays the role of the caption, so the noise removal moves toward a picture of a red bicycle on a beach rather than any other picture.

Latent diffusion: working on a smaller version

Running diffusion on full-size pixels is expensive, because a 512 by 512 color image holds 786,432 numbers. Latent diffusion, introduced by Rombach and colleagues and used by Stable Diffusion, first compresses the image into a much smaller representation (a latent). In Stable Diffusion, a 512 by 512 image becomes a 64 by 64 grid with 4 values per position, which is 16,384 numbers, about 48 times fewer. All the noise removal happens in this small space, and a separate decoder expands the final result back into a full image. The output looks the same to a person, and each step costs far less to compute.

Video generators extend the same idea across time. They remove noise from a stack of frames together, so that objects stay consistent from one frame to the next. Audio and speech models often use either diffusion or next-token prediction over short chunks of sound.

How does generative AI write code?

Code generation uses the same next-token method as text. Code is text with strict rules, and large public collections of source code are part of the training data. A model that has seen millions of functions learns the patterns of each language: how loops are written, which library calls usually follow each other, and how a function name relates to what it does.

When you ask for a function, the model predicts the code token by token, exactly as it predicts a sentence. Its context (everything the model can read at once) can include your instruction, the file you are editing, and other relevant files. Coding assistants work well when they can see the surrounding code, because the most likely continuation then matches your names, your types, and your style.

Why tests matter more than the model

A code model optimizes for code that looks right, which is not the same as code that is right. An off-by-one error, a wrong boundary check, or a call to a library function that does not exist can all look plausible. For this reason, the most reliable coding tools do not stop after generating code. They run it.

A common loop looks like this:

1. The model writes a function and a set of tests.
2. The tool runs the tests.
3. If a test fails, the error message goes back to the model as new input.
4. The model edits the code, and the tool runs the tests again.

This loop turns a guess into a checked result, because the tests give the model information it could not get from its training data. It is the same pattern an AI agent uses for any task: act, observe the result, and decide again. Design an AI Agent explains that loop and the controls it needs.

The practical rule for engineers follows directly. Treat generated code like code from a fast new teammate. Read it, run it, and test the edge cases before you merge it.

How are generative models trained?

A generative model goes through two very different phases: training, where it learns, and inference, where it is used. Training a large model takes thousands of specialized chips (GPUs, graphics processors designed for parallel math) running for weeks. Inference is each individual use of the finished model, such as answering one question, and it runs on far fewer chips for seconds.

Training a text model usually happens in three stages:

• Pretraining. The model learns to predict the next token across a huge collection of text and code. This stage gives it language, facts, and general skills, and it is by far the most expensive stage. The result is a base model, which continues text well but does not reliably follow instructions.
• Fine-tuning. The base model trains further on a smaller, curated set of examples, such as questions paired with good answers. This teaches the format of a helpful assistant. Companies also fine-tune models on their own data to specialize them for a task.
• Reinforcement learning from human feedback (RLHF). People compare pairs of model answers and mark which one is better. A second model learns to predict those preferences, and the main model is then trained to produce answers that score well. This stage makes answers more helpful and more likely to refuse harmful requests.

RLHF has a large effect for its cost. In OpenAI's InstructGPT paper, human raters preferred answers from a 1.3 billion parameter model trained this way over answers from the 175 billion parameter GPT-3 base model, which has more than 100 times as many parameters. Training on what people actually want mattered more than size alone.

Want to understand these models well enough to build with them? Grokking Modern AI Fundamentals explains how modern AI systems are trained and used, in plain language and with worked examples.

One more term is worth knowing. The training data has an end point, called the knowledge cutoff. A model knows nothing about events after its cutoff unless the product gives it that information at the time of the question. This limit is one of the main reasons products connect models to search and to company documents, as described later in this article.

What is generative AI good at?

Generative models are strongest on tasks where many answers are acceptable and a person can check the result quickly:

• Drafting and rewriting text. Emails, summaries, translations, and first drafts of documents.
• Answering questions over provided material. When the relevant documents are included in the input, models are good at finding and restating the answer.
• Writing and explaining code. Boilerplate, tests, small functions, and explanations of unfamiliar code.
• Creating images for concepts and drafts. Illustrations, mockups, and variations on a design.
• Changing the format of information. Turning notes into a table, a table into prose, or free text into structured data.

These tasks share one property: the output is useful even when it is not perfect, because a person reviews it or a program checks it.

Where does generative AI fail?

The same design that makes these models flexible also creates predictable failures. Knowing them is more useful than knowing any single model.

Hallucination

A hallucination is output that is fluent and confident but false, for example a made-up citation or a function that does not exist. It happens because the model generates the most likely-looking continuation, and a false answer can look as likely as a true one. Hallucination becomes more common for rare facts, precise numbers, recent events, and questions with no answer in the training data. The main defenses are to give the model the source material at the time of the question, to ask it to cite that material, and to check the output with tests or a second step.

Cost

Every token costs computation, so long inputs and long outputs cost more. A product that sends a 10-page document with every question pays for those tokens on every call. At the scale of millions of requests a day, the choice of model size, the length of the input, and caching of repeated work decide whether a feature is affordable.

Latency

Text is produced one token at a time, so a long answer takes longer. A model that produces 50 tokens per second needs about 8 seconds for a 400-token answer (illustrative numbers). Products hide part of this by streaming tokens to the screen as they are produced. Image generation also takes several seconds, because it runs many denoising passes.

Inconsistency

Because sampling involves randomness, the same input can produce different outputs. This is useful for creative tasks and harmful for tasks that must be repeatable. Lower temperature, strict output formats, and validation of every response all reduce the problem, but none removes it completely.

Security

A model reads instructions and data through the same input. If a document it reads contains text such as "ignore your previous instructions", the model may follow it. This attack is called prompt injection. Defenses include limiting what actions the model can take, filtering inputs and outputs, and requiring human approval before any action that cannot be undone.

How do real products use generative AI?

Very few useful products send a question straight to a model and show the answer. Most wrap the model in a system that supplies the right information before the call and checks the output after it. Two patterns appear again and again.

Retrieval-augmented generation

Retrieval-augmented generation (RAG) answers a question in two steps. First, the system searches a collection of documents, such as a help center or internal wiki, for the passages most related to the question. Second, it sends those passages to the model together with the question and asks for an answer based only on them.

RAG addresses three of the problems above at once. The model can answer about private documents it never saw in training. It can answer about events after its knowledge cutoff. It also hallucinates less, because the answer is written from text in front of it rather than from memory. The design decisions behind RAG, including how to split documents and how to search them, are covered in System Design for RAG.

AI agents

An AI agent is a system where the model decides on the next step itself and can take actions through tools, such as running a search, querying a database, or sending a message. The agent repeats a loop: decide on an action, run it, read the result, and decide again, until the task is done or a limit stops it. The coding loop described earlier is one example.

Agents can complete multistep tasks that a single model call cannot, and they also need more control, because their actions have real effects. Step limits, cost limits, narrow permissions, and human approval for risky actions are standard parts of the design. Design an AI Agent walks through the full architecture, layer by layer.

Checking the output

Products that use generative AI well also measure it. A team builds a set of test questions with known good answers and runs every new version of the system against them before release. This step catches regressions that no single manual test would find, and it is the main difference between a demo and a product. LLM Evaluation explains how to build those tests.

Generative AI by type of output

The same two methods, next-token prediction and diffusion, cover most of the field. The table below shows how they map to each type of output.

The last row is included because embeddings appear in almost every generative AI product, even though they do not generate content. They are how a RAG system finds the right documents before the model writes anything.

Frequently asked questions

What is generative AI in simple terms? Generative AI is software that creates new content, such as text, images, or code, after learning patterns from a very large number of examples. It builds its output one small piece at a time, choosing each piece so that it fits the patterns it learned.

How is generative AI different from traditional AI? Traditional AI systems usually choose a label or a number for an input, such as "fraud" or "not fraud". Generative AI produces new content instead. Both learn from data, and generative AI is one part of the wider field of machine learning.

How does ChatGPT generate text? ChatGPT is a large language model that predicts the next token, a short chunk of text, from everything written so far. It repeats this hundreds of times to build an answer. Training with human feedback taught it to produce answers that people find helpful.

How does AI generate images from text? Most image generators use diffusion. The model starts from random noise and removes a little noise in each step, guided by an embedding of your description, until a clear picture appears. Latent diffusion does this work on a compressed version of the image to save computation.

Why does generative AI make things up? The model produces the most likely-looking continuation, not a checked fact. When the answer is rare or missing from its training data, a false answer can look as likely as a true one. Giving the model source documents and checking its output reduce this problem.

Can generative AI write working code? It often writes correct code for common tasks, but it can also produce plausible code with subtle bugs. The most reliable approach is to run the generated code against tests and feed any errors back to the model. Treat its output as a draft that needs review.

Is generative AI the same as a large language model? No. A large language model is one kind of generative model, the kind that works on text and code. Image, audio, and video generators are also generative AI, and many of them use diffusion rather than next-token prediction.

Related reading

• How Large Language Models Work: Tokens, Embeddings, Attention
• Generative AI vs Machine Learning vs Deep Learning: How They Relate
• System Design for RAG: Vector Databases, Chunking, and Re-ranking
• Design an AI Agent: The 7-Layer Blueprint
• LLM Evaluation: How to Test an LLM App Before It Ships

Ready to go from using AI tools to understanding them? Grokking Modern AI Fundamentals covers how modern models learn, generate, and fail, with clear explanations you can apply in your own work.