0% completed
What GenAI Can and Can't Do Today
On This Page
Part 1: A Map of Strengths and Weaknesses
Part 2: Where Generative Models Are Strong
Part 3: The Confident but Wrong Problem
Part 4: Other Common Failures
Part 5: Designing Around the Limits
Part 6: What This Changes for an Engineer
Common Mistakes
Key Takeaways
import requests response = requests.get("https://api.example.com/orders", timeout=5, retries=3)
An engineer asked an AI assistant how to add retries to a Python HTTP call, and this line was the core of the answer. The answer arrived in two seconds, with a clear explanation, and it sounded completely sure. But the line fails the first time it runs. The requests library is real, but its get() function has no retries argument, so Python raises a TypeError.
Nothing in the answer looked wrong. This is the central fact about working with generative AI. These models are very good at many tasks, and their mistakes often look exactly like their successes. So the useful skill is knowing which tasks to trust, which tasks to check, and where the checks belong.
Part 1: A Map of Strengths and Weaknesses
Generative models now produce five main kinds of output: text, code, images, audio, and video. For each kind, some tasks are reliable, some need checking, and some should never be used without human review or testing.
The diagram below sorts common tasks by color. Green means the model usually does the task well. Yellow means the model is useful for the task, but its output needs checking. Red means the output should not be used without a check.
Green doesn't mean perfect, and red doesn't mean impossible. The colors show how much checking a task usually needs. Models improve quickly, so some tasks will change color over time. One question stays the same for every task: if this is wrong, what would catch it?
Part 2: Where Generative Models Are Strong
Generative models are strongest at tasks built from patterns that appear many times in their training data. Language and code are full of such patterns.
- Drafting and rewriting. Turning notes into an email, or making a message shorter or more polite.
- Summarizing. Reducing a long document to its main points.
- Translating. Moving text between common languages, or between styles such as formal and casual.
- Extracting structure. Pulling names, dates, and amounts out of messy text into a clean format.
- Writing common code. Boilerplate, small functions, and code in well-known languages and libraries.
- Explaining. Describing in plain words what a piece of code or a paragraph of legal text does.
These tasks have one thing in common. A person can usually check the result quickly by reading it. That makes them a good fit for a model that is fast and fluent but sometimes wrong.
Part 3: The Confident but Wrong Problem
An output that sounds correct but is false or made up is called a hallucination. It might be a wrong fact, a source that does not exist, or a function argument that no library provides, like retries in the example above.
Hallucinations come from the way the model works. A generative model predicts likely text, one piece at a time. It does not look up facts, and nothing in its text signals "I am unsure here." When the model lacks the right information, it still produces the most likely-sounding text. That text is often fluent and specific, because fluent and specific text is what the model learned to produce.
The diagram below takes the retry answer apart, claim by claim.
Look at how the three claims are written. All three sound equally confident, and the true claim looks exactly like the invented one. This is why the tone of an answer tells you nothing about whether it is correct.
Part 4: Other Common Failures
Hallucination is the best-known failure, but it is not the only one. Each of the following failures comes up often in real projects.
- Knowledge cutoff. A model learns from data collected up to a certain date, called its knowledge cutoff. It knows nothing after that date unless your application gives it the new information.
- Exact counting and arithmetic. A model reads text in chunks called tokens, not letter by letter. So counting letters or doing long calculations can go wrong. A later lesson, Tokens and Tokenization, explains why.
- Answers that vary. The same question can get a different answer each time. This helps with writing, but it is a problem for anything that must give the same result every time.
- Drift on long tasks. Over a long document or a long chain of steps, a model can lose track of earlier details or contradict itself.
- Following hidden instructions. Text that a model reads, such as a web page, can contain instructions that change what the model does. This is called prompt injection, and a later lesson covers it in depth.
- Bias. A model can repeat unfair patterns that exist in its training data.
None of these failures makes a model useless. Each one tells you which safeguard a feature needs.
Part 5: Designing Around the Limits
Engineers do not wait for perfect models. They build features that work with today's models, and they add checks where the models are weak. The diagram below shows where these checks usually go.
Step 1: Give the model the facts. The application does not rely on what the model remembers. Instead, it adds the relevant documents or data to the request. This is called grounding. A later chapter, Grounding Models in Your Data, shows how it works.
Step 2: Let the model draft. The model does what it does best. It turns facts and instructions into fluent text or code.
Step 3: Check automatically. Code can be compiled and tested. You can check structured output against a schema, a formal description of the expected shape. Quoted facts can be compared with the source documents. Do any math in ordinary code instead of having the model do it.
Step 4: Add a person when the stakes are high. A refund decision, a medical summary, or a legal reply goes to a person for review before anyone receives it.
Notice that the model is only one box in this flow. A feature is reliable because of the whole flow, not because of the model alone.
Part 6: What This Changes for an Engineer
How much checking a feature needs depends on the cost of a mistake. The table below matches common situations to the checking they usually need.
| Situation | Cost of a mistake | Typical safeguard |
|---|---|---|
| Brainstorming names for a feature | Very low | None beyond a quick read |
| A draft email a person will edit | Low | The person reviews before sending |
| Code that goes into your product | Medium | Tests, code review, and running it |
| An answer about a customer's account | High | Grounding in the account data, plus source checks |
| A decision about money, health, or law | Very high | A human makes or approves the final decision |
Before you ship any generated output, ask one question: "If this is wrong, what would catch it?" If the answer is "nothing," the feature needs another check.
Common Mistakes
Mistake 1: Trusting fluent output. Polished wording says nothing about accuracy. Check facts, names, and numbers against a source.
Mistake 2: Asking the model to do exact math or counting. Do the calculation in code, and let the model explain the result.
Mistake 3: Using the same safeguards for every task. A brainstorm needs no checks, and a refund decision needs strong ones. Match the checking to the cost of a mistake.
Key Takeaways
- Generative models are strongest at work built from common patterns. Drafting, summarizing, translating, extracting, and writing common code are their best tasks.
- A hallucination looks exactly like a correct answer. Models predict likely text, so false claims come out just as fluent as true ones.
- Each failure needs a matching safeguard. Grounding, automatic checks, code for math, and human review each cover a different weakness.
- Reliability comes from the whole feature, not the model alone. The model is one box in a flow of checks.
This lesson ends the first chapter. The next chapter starts with How a Model Learns, which shows how training turns data into a working model.
Reading Progress
0%
On This Page
Part 1: A Map of Strengths and Weaknesses
Part 2: Where Generative Models Are Strong
Part 3: The Confident but Wrong Problem
Part 4: Other Common Failures
Part 5: Designing Around the Limits
Part 6: What This Changes for an Engineer
Common Mistakes
Key Takeaways