0% completed
What an Eval Is
On This Page
The Four Parts of an Eval
A Small Eval of the Job Ticket Assistant
What a Grader Returns
The Decision Rule
Evals, Unit Tests, and Benchmarks
Key Takeaways
Practice Questions
The following lines of Python are a complete eval, reduced to its smallest form.
cases = [ {"request": "I smell gas near the stove.", "expected": "emergency"}, {"request": "The kitchen tap drips at night.", "expected": "scheduled"}, ] passed = 0 for case in cases: output = assistant(case["request"]) # run the feature if output["urgency"] == case["expected"]: # grade the output passed += 1 print(f"{passed} of {len(cases)} passed") ship = passed == len(cases) # decide
The code sends each request to an LLM feature, checks one field of each answer, and decides from the count. Every eval in this course has the same four parts, including evals that grade agents and long answers.
This lesson names those four parts, runs a small eval from start to finish, and shows how an eval differs from a unit test and from a public benchmark.
The Four Parts of an Eval
An eval is a repeatable test that runs an LLM feature on a fixed set of inputs, grades every output, and turns the grades into a decision. The inputs and grading stay the same, so runs before and after a change can be compared.
- Test cases. Each case is one input plus what a good output looks like: an exact answer, such as an urgency level, or a rule, such as "the reply must not promise an arrival time".
- The system under test. This is the whole feature: the instructions sent to the model (the prompt), the model and its settings, and the code around them. Changing any of these changes the system, so the eval records which version it ran.
- The grader. A grader returns a verdict on one output, usually pass or fail, with a short reason. It can be code, a comparison with a reference answer, or a second model acting as a judge.
- The decision rule. The decision rule turns all the verdicts into an action, such as release the change or keep working. It is written before the eval runs.
Notice that the first three parts work on one test case at a time, and only the decision rule looks at the whole set. Without it, a team has measurements but no answer.
A Small Eval of the Job Ticket Assistant
Most examples in this course come from a home repair booking app. Its job ticket assistant turns each customer's description of a problem into a ticket for a technician. One field of the ticket is the urgency: emergency, same day, or scheduled. The lesson The Job Ticket Assistant describes the whole feature; this lesson tests only urgency.
The team writes five test cases, each with the urgency a dispatcher would choose:
| Case | Customer request | Expected urgency |
|---|---|---|
| 1 | I smell gas near the stove and the pilot light is out. | emergency |
| 2 | The kitchen tap drips all night. Any day next week works. | scheduled |
| 3 | Water is coming through the ceiling from the bathroom upstairs. | emergency |
| 4 | Our fridge stopped cooling this morning and it is full of food. | same day |
| 5 | Two kitchen cabinet doors hang crooked. | scheduled |
The decision rule, written first, has two conditions. No emergency may be marked as anything else, and at least 80 percent of cases must pass. A missed gas leak costs far more than a cabinet repair booked a day early.
You can run the eval below without an API key. The assistant's answers were recorded from one run and stored in the file, and run_assistant returns them. In a real eval, only that function would change: it would send the request to the model.
Running it prints the following:
case 1: PASS (urgency matches) case 2: PASS (urgency matches) case 3: FAIL (expected emergency, got same_day) case 4: PASS (urgency matches) case 5: PASS (urgency matches) 4 of 5 passed (80%) missed emergencies: 1 decision: do not ship
Four of five cases pass, which meets the 80 percent condition, yet the decision is "do not ship". Case 3 is a ceiling leak marked as same day, and an overall score of 80 percent would have hidden it.
Notice that the rule weighs failures by their cost. Error analysis, taught later in this course, is how a team learns which failures cost the most.
What a Grader Returns
A grader returns a verdict for one output, and a reason that a person can read. The reason, such as "expected emergency, got same_day", turns a failing run from a number into a list of problems someone can fix.
The urgency grader is the simplest kind, because urgency has one right answer. A summary for the technician has many acceptable wordings, so it needs a different grader, and the lesson Three Kinds of Graders explains the choices. Whatever its kind, a useful grader is specific, checking one failure rather than "quality", and consistent, giving the same verdict for the same output every time.
The Decision Rule
A decision rule states, before the eval runs, which results lead to which action. Writing it first stops a team from seeing 78 percent and then deciding that 78 percent is acceptable. A good rule has three properties:
- It names an action. "Ship to 5 percent of users first" is an action; "looks fine" is not.
- It weighs failures by cost. A missed emergency and a wrong cabinet booking do not count the same.
- It respects the number of cases. Five cases show how an eval works, but they cannot support a claim about a percentage.
If an eval passes 45 of 50 cases, the true pass rate could plausibly be anywhere from about 79 to 96 percent. A rule of "at least 90 percent" cannot be checked with 50 cases, as the lesson Confidence Intervals for Pass Rates shows.
Evals, Unit Tests, and Benchmarks
A unit test checks deterministic code, which gives the same output for the same input. Each test has one correct answer, and the suite must pass completely. An eval checks a feature whose output varies and has many acceptable forms, so it reports a rate.
A public benchmark is a fixed, published set of test cases for comparing models in general, such as school math problems. Its score says how a model does on that set, not whether your feature handles your customers' requests. The lesson What a Public Benchmark Measures explains why the two often disagree.
| Unit test | Eval | Public benchmark | |
|---|---|---|---|
| What it tests | A function in your code | Your LLM feature | A model, in general |
| Who writes the cases | Your team | Your team, from your own data | Researchers |
| What a run reports | All pass, or the failures | A rate, and the failing cases | One score per model |
| How it is used | Block a merge that breaks a test | Decide whether a change ships | Shortlist models to try |
A team usually needs all three, but only the eval says whether this version of the feature should ship.
Key Takeaways
- An eval runs an LLM feature on fixed test cases, grades every output, and turns the grades into a decision.
- Its four parts are the test cases, the system under test, the grader, and the decision rule.
- The system under test is the whole feature: prompt, model, settings, and surrounding code.
- A grader returns a verdict with a readable reason, and checks one failure consistently.
- The decision rule is written first, names an action, weighs failures by cost, and respects the number of cases.
The urgency eval was simple because urgency has one right answer. Most outputs of an LLM feature do not. The next lesson, Why LLM Features Need Evals, explains what makes these features hard to test with ordinary methods.
Practice Questions
1. A team changes the model behind the job ticket assistant and edits its prompt in the same change. The pass rate rises from 82 to 88 percent. What can they conclude?
<details><summary>Show answer</summary>Only that the new combination scores higher, not which change caused it. Both parts of the system under test changed at once. To learn the effect of each, the team must change one at a time and record which version of each part it ran.
</details>2. The assistant passes 4 of 5 cases, meeting the 80 percent condition. Why is the decision still "do not ship"?
<details><summary>Show answer</summary>Because the failing case is a missed emergency, which the rule forbids. The rule weighs failures by cost, while an overall score treats a ceiling leak the same as a crooked cabinet door.
</details>3. After an eval finishes at 78 percent, the product lead proposes lowering the release bar from 80 to 75 percent. What is the problem?
<details><summary>Show answer</summary>The bar is being set by the result, not by what the product needs. A rule that moves after every result no longer decides anything. A rule can change, but for its own reasons and on the record.
</details>4. A colleague says the team needs no eval of its own, because the chosen model ranks near the top of a public benchmark. How would you answer?
<details><summary>Show answer</summary>A benchmark tests the model on someone else's cases, not your feature on your customers' requests. The score is a good reason to try the model. Only an eval built from your own data shows whether the feature does its job, such as never marking a gas leak as a scheduled repair.
</details>Reading Progress
0%
On This Page
The Four Parts of an Eval
A Small Eval of the Job Ticket Assistant
What a Grader Returns
The Decision Rule
Evals, Unit Tests, and Benchmarks
Key Takeaways
Practice Questions