0% completed
Why LLM Features Need Evals
On This Page
Many Correct Outputs
Different Outputs from the Same Input
Changes With Side Effects
Inputs Nobody Listed
Why Trying a Few Examples Is Not Enough
Key Takeaways
Practice Questions
A customer writes: "The water heater in the garage is leaking a little onto the floor." The job ticket assistant reads it five times, with the same prompt, the same model and the same settings each time. Three runs mark the request as same day, one marks it as scheduled, and one marks it as an emergency.
Ordinary code cannot do this. A function that sorts a list returns the same list every time, so one test run settles the question for good. The assistant gave three different answers to one question, and nothing in the code changed between the runs.
This lesson explains the four properties that make LLM features hard to test with ordinary methods, and what an eval does about each one.
Many Correct Outputs
Most outputs of an LLM feature have many acceptable forms, so they cannot be checked by comparing them with one expected string. Urgency is the exception: it has three possible values, and one is right. The summary the assistant writes for the technician is the usual case. Here are two summaries of the same request:
- "Water heater in garage leaking slowly onto the floor. Customer reports a small amount of water."
- "Small leak from the garage water heater; water is pooling on the floor."
Both are correct, and they share only a few words. A test that demands an exact match fails both, and a test that accepts anything passes a summary that invents a model number. An eval therefore grades properties of an output, such as "mentions the appliance and its location" or "adds no detail the customer did not give", rather than the exact text.
Different Outputs from the Same Input
An LLM feature can return a different output each time it receives the same input. Part of the reason is sampling: the model picks each word from a list of likely words, and a setting called temperature controls how much randomness that choice allows. Many teams set temperature to 0, which tells the model to pick the most likely word every time.
That does not make the output fixed. Anthropic and OpenAI both state in their documentation that temperature 0 does not guarantee identical outputs. One research lab sent the same request 1,000 times at temperature 0 and received 80 different completions. The main cause it found was in the inference servers, which group requests into batches. The batch size changes with load, and that slightly changes the arithmetic for each request.
Notice that the request, the prompt and the settings were identical in all five runs. Only the output changed.
The consequence for testing is direct. A single run of one test case is a sample, not a fact, so an eval reports a rate over many cases. For some decisions it also runs each case several times, which the lesson Noise from Repeated Runs explains.
Changes With Side Effects
A change that fixes one case can break another case that nobody was watching. Suppose the team sees the water heater request marked as scheduled and adds one sentence to the prompt: "Treat any water leak as same day or faster." The water heater case is now right. A dripping kitchen tap, whose owner said any day next week was fine, is now marked same day as well.
The program below grades six cases with the old prompt and with the new one. It also counts the five runs of the water heater request from the opening. As in the previous lesson, the outputs were recorded and stored in the file, so it runs without an API key.
Running it prints the following:
The water heater in the garage is leaking a little onto the floor. same_day: 3 of 5 runs scheduled: 1 of 5 runs emergency: 1 of 5 runs old prompt: 5 of 6 passed new prompt: 5 of 6 passed case 2 broken: The kitchen tap drips all night. Any day next week works. case 3 fixed: The water heater in the garage is leaking a little onto the floor.
Both prompts pass 5 of 6 cases, yet they fail different cases. A team that compared only the two totals would conclude that the change did nothing. A team that compared the cases one by one sees that it fixed one request and broke another.
Notice that the score stays at 5 of 6 while the failing case moves. The same thing happens when a provider updates a model or when someone edits the code that prepares the input, which is why the whole test set is run again after every change. The lesson Comparing Two Versions on the Same Cases shows how to compare two runs case by case.
Inputs Nobody Listed
An LLM feature accepts open text, so its possible inputs cannot be listed in advance. A function that takes a date can be tested at its boundaries. Customers of the home repair app write in any length and style. They describe two problems in one message, mix languages, leave out the address, or paste a long story with the problem in its last line.
Each of these can fail in its own way, and nobody can predict all of them without reading real requests. Test cases therefore come from reading what real users send and what the feature returned, which is the subject of the lesson Why Error Analysis Comes First.
Why Trying a Few Examples Is Not Enough
Most teams first test an LLM feature by typing a few requests and reading the answers. This check is useful, but it gives a false sense of safety for all four reasons above. It checks a handful of outputs whose wording varies, runs each one once, never sees what a change broke elsewhere, and only covers inputs that the developer thought of.
| Property | What ordinary testing assumes | What an eval does instead |
|---|---|---|
| Many correct outputs | One expected output | Grades properties of the output |
| Different outputs from one input | One run settles it | Reports a rate, and may run each case several times |
| Changes with side effects | A change affects what it targets | Reruns every case after every change |
| Inputs nobody listed | Inputs can be enumerated | Builds cases from real traffic and reading outputs |
Key Takeaways
- Most LLM outputs have many correct forms, so an eval grades properties rather than exact text.
- The same input can produce different outputs, even at temperature 0, so one run of one case is a sample.
- A change can fix one case and break another, so every case is rerun and compared one by one.
- Open text inputs cannot be listed in advance, so test cases come from real traffic.
- Trying a few examples by hand is a useful start, but it is not an eval.
Every one of these properties depends on knowing exactly what the feature is supposed to do. The next lesson, The Job Ticket Assistant, writes that down for the feature this course tests.
Practice Questions
1. A developer reruns a failing test case and it passes. The developer marks the bug as fixed. What is wrong with this?
<details><summary>Show answer</summary>One passing run proves little about a feature whose output varies. The same input can return different outputs, even at temperature 0. To know whether the bug is fixed, the case must be run several times, and the fix must be checked against the whole test set.
</details>2. Why can an exact string comparison not grade the technician summary?
<details><summary>Show answer</summary>Because many different summaries are correct. Two good summaries of the same request can share only a few words. An exact comparison would fail correct summaries, so the grader checks properties such as whether the summary mentions the appliance and adds no invented detail.
</details>3. A prompt change leaves the pass rate at exactly the same number. Can the team assume nothing changed?
<details><summary>Show answer</summary>No. The change may have fixed some cases and broken others in equal numbers, as the six-case example shows. The team must compare the two runs case by case to see which cases moved.
</details>4. A team sets temperature to 0 and stops worrying about variation between runs. What does the documentation say?
<details><summary>Show answer</summary>Temperature 0 makes outputs more repeatable, not identical. Anthropic and OpenAI both state that identical outputs are not guaranteed. One lab received 80 different completions from 1,000 identical requests at temperature 0.
</details>Reading Progress
0%
On This Page
Many Correct Outputs
Different Outputs from the Same Input
Changes With Side Effects
Inputs Nobody Listed
Why Trying a Few Examples Is Not Enough
Key Takeaways
Practice Questions