0% completed
Welcome to Grokking LLM Evals
On This Page
Who This Course Is For
What the Course Covers
The Running Examples
How the Code Runs
How the Lessons Work
Key Takeaways
Practice Questions
This course teaches you to measure whether an LLM feature does its job, and to decide from those measurements whether a change should ship.
That skill is now part of ordinary software work. A feature built on a large language model can pass every demo and still fail the people who depend on it. Its outputs vary, its inputs are open text, and a small change to a prompt or a model can fix one case while breaking another. The tests an engineer already knows were not built for any of that. An eval was. It is a repeatable test that runs the feature on fixed inputs, grades every output, and turns the grades into a decision.
This introduction explains who the course is for, what it assumes you know, the examples it uses, how its code runs, and how its lessons are arranged.
Who This Course Is For
The course is written for software engineers who build, maintain or review features that call a language model. You may be adding a model to an existing product, owning a feature that already uses one, or deciding whether to switch models. Product managers and data scientists on such teams can use every method too, because each one is explained from the beginning.
You need basic Python and nothing else. Every term is defined where it first appears. The course teaches the statistics it uses, starting from what a pass rate means, so no earlier statistics course is needed. Nothing depends on a particular model provider, framework or eval tool. When a tool is mentioned, it is only an example of a general method.
What the Course Covers
The lessons follow the order in which a team builds its evals:
- What an eval is, and why LLM features need one.
- Error analysis: reading real outputs and naming the ways they fail, before any metric exists.
- Test sets: sampling real traffic, generating cases from dimensions, and labeling what a good output looks like.
- Graders: checks written in code, comparisons with a reference answer, and LLM judges, including how to test a judge for bias, validate it against human labels, and correct a pass rate for its errors.
- Retrieval-augmented generation (RAG) and agents, which need measurements of their own.
- Safety and robustness: prompt injection, harmful requests, refusals of safe requests, and small changes to the input.
- Statistics for evals: intervals for a pass rate, how many cases a decision needs, comparing two versions on the same cases, and noise from repeated runs.
- Evals in production: release gates, traces, grading sampled live traffic, A/B tests and drift.
- Public benchmarks, and why they cannot replace an eval built from your own data.
- Two complete eval designs, one for a help center bot and one for a research agent, that put every earlier method together.
Notice that measuring comes after reading. The failures that matter most are usually ones nobody predicted, so the course finds them first and only then decides how to grade them.
The Running Examples
Most lessons use one product, a home repair booking app where customers describe a problem in their own words. Its main LLM feature, the job ticket assistant, turns each request into a ticket for a technician. The ticket has a trade, an urgency level, a summary and a reply to the customer. A request about a smell of gas must become an emergency, and a summary must not invent a detail the customer never gave. Promises like these run through the whole course.
The same app supplies two more features. The repair guide assistant answers technicians' questions from manufacturer manuals, and it is the example for RAG. The scheduling agent books technicians through several tools, and it is the example for agents. The two final designs use different products. One is a help center bot for payroll software. The other is a research agent that answers questions about upgrading a software library.
All five examples are fictional, and so are their numbers unless a lesson names a source. The numbers stay consistent from lesson to lesson, so a result found in one lesson can be used in the next.
How the Code Runs
Most lessons include Python code that you can run here, in the course player, with the Run button. It uses only Python's standard library, so it needs no installation and no account with any model provider. Each program holds its own data, such as requests, outputs, verdicts and human labels. The model outputs in these programs were written for teaching. They show realistic behavior, but they were not produced by a model run.
A few lessons show code that calls a model, for example to run an LLM judge or to generate test cases. That code cannot run in the player, because it needs an API key from a model provider. It is shown read-only, with every model call inside one small function named call_model. You can connect that function to any provider and run the code on your own machine.
Notice that both kinds of code share the same grading and statistics. Only the step that produces the model's outputs differs.
How the Lessons Work
Each lesson teaches one idea with one running example, worked through with real numbers. Diagrams are animated to show the order in which things happen. Each one is also complete as a still picture. A lesson ends with its key takeaways, a link to the next lesson, and four practice questions whose answers are folded until you open them.
Each chapter ends with an assessment. Every attempt draws ten questions from a larger bank, so a second attempt covers different ground, and every option comes with an explanation. A final exam covers the whole course.
Key Takeaways
- The course teaches how to measure whether an LLM feature does its job and how to decide whether a change should ship.
- It assumes basic Python only, and it teaches the statistics it uses.
- It follows the order a team builds evals in: read outputs, build test sets, grade, measure carefully, and watch production.
- One home repair app supplies most examples, and two complete designs close the course.
- Most code runs in the player with the standard library; code that calls a model runs on your own machine.
The first idea to settle is what an eval actually contains. The next lesson, What an Eval Is, builds one in a few lines of Python.
Practice Questions
1. Do you need a statistics background to take this course?
<details><summary>Show answer</summary>No. The course teaches every statistical idea it uses, starting from what a pass rate means. Basic Python is the only requirement.
</details>2. Why can most code in the course run in the player without an API key?
<details><summary>Show answer</summary>Because each program carries its own data, including the model outputs. Grading and statistics do not need a live model. Only code that produces new model outputs, such as an LLM judge, needs a key, and that code is shown for running on your own machine.
</details>3. Are the model outputs in the course's programs real model runs?
<details><summary>Show answer</summary>No, they were written for teaching. They show realistic behavior and stay consistent across lessons, but they were not produced by running a model.
</details>4. Why does the course read real outputs before it chooses any metric?
<details><summary>Show answer</summary>Because the failures that matter most are usually ones nobody predicted. Reading outputs first finds those failures, and the metrics are then built to measure them.
</details>Reading Progress
0%
On This Page
Who This Course Is For
What the Course Covers
The Running Examples
How the Code Runs
How the Lessons Work
Key Takeaways
Practice Questions