0% completed
Why Error Analysis Comes First
On This Page
Generic Metrics
Criteria Drift
Why the List Must Come from Your Own Outputs
What Error Analysis Is
Key Takeaways
Practice Questions
"Users need criteria to grade outputs, but grading outputs helps users define criteria."
Shreya Shankar and coauthors wrote this sentence after a study of people building graders for LLM outputs with the help of a tool. The participants started with criteria for a good output. Then they graded real outputs, and the grading changed the criteria.
Each side of the sentence depends on the other, and the practical answer is to read outputs before deciding what to measure. This lesson explains why a list of generic metrics misses the failures that matter, what the researchers call criteria drift, and why the failure list for an eval has to come from your own feature's outputs.
Generic Metrics
A generic metric is a quality score defined without looking at any one product, such as helpfulness, fluency or coherence on a scale of 1 to 5. Such metrics are easy to adopt, because a list of them and a judge prompt for each are already available. They also produce a number on the first day.
Suppose the job ticket team scores a week of tickets for helpfulness and coherence, and both average above 4 out of 5. Then a dispatcher reads the same tickets. One reply to a customer who smells gas is polite, clear and complete, and its urgency is scheduled. A summary for a leaking washing machine is fluent and adds an age the customer never gave. Both outputs score well on every generic metric, and both break a promise of the feature.
| Failure in a real ticket | What a generic metric sees | Why it scores well |
|---|---|---|
| A gas smell marked as scheduled | A polite, complete reply | Urgency is not part of helpfulness |
| A door seal leak sent to a plumber | A coherent ticket | Coherence cannot know the appliance rule |
| An age added to the summary | A fluent, specific summary | Extra detail reads as more helpful |
| "A technician will be there shortly" | A friendly reply | The metric does not know the promise |
The rows share one cause. Each failure breaks a rule that belongs to this product, such as "anything inside an appliance is appliance work", and no generic metric contains that rule. Hamel Husain, who teaches this method with Shreya Shankar, therefore recommends working from the bottom up: read real outputs first and let the metrics come out of what you find.
Notice that the generic path ends with a high score and no failures named. The reading path ends with a list of named failures, and each one can become a test.
Criteria Drift
Criteria drift is the change in what people count as a failure, caused by grading real outputs. Shankar and coauthors named it in that study, published as "Who Validates the Validators?". Their conclusion was strong: criteria for an LLM feature cannot be fully decided before people judge its outputs.
The job ticket assistant shows how this happens. One of its promises is that the reply never states an arrival time. Before reading, the team takes this to mean a clock time, such as "at 3 pm". Then the dispatcher reads a reply that says "a technician will be there shortly", and another that says "we will have someone out today". Neither names a clock time, yet a customer reads both as a commitment. After those two outputs, the team rewrites the promise: any phrase that tells the customer when a visit will happen counts as a promised time.
Notice that the criterion improved only because someone graded outputs with the first version. A rubric written once, before reading, keeps its first gaps for as long as nobody reads.
The same study makes a second point. Shankar and coauthors point out that a grader written by an LLM inherits the problems of the LLM it grades. A model asked to write the criteria for the assistant would produce sensible, general rules, and it would not know about the door seal or about "shortly". Only a person who reads the outputs and knows the business can supply those.
Why the List Must Come from Your Own Outputs
The failures worth measuring depend on your users, your prompt and your product's rules, so they cannot be copied from another team. Three things make the job ticket assistant's failures its own:
- Its users. Customers describe danger in mild words ("a faint smell", "a bit of water on the ceiling"), and those words decide which emergencies are missed.
- Its rules. The appliance rule and the list of dangers exist only in this product.
- Its current version. A failure that one prompt causes may disappear after the next change, and a new one may appear, as the lesson The Eval Development Loop showed.
A failure list from a similar product is a useful set of ideas to look for, but it is not a list of what to measure. Only reading this feature's outputs shows which failures it has and how often they occur.
What Error Analysis Is
Error analysis is the method of reading a sample of a feature's outputs, writing a note on each failure, grouping the notes into named failure modes, and counting them. A failure mode is one kind of failure with a written definition, such as "the summary states a detail the customer did not give". The result is a short, ranked list, and each item on it becomes an eval or a direct fix.
Husain and Shankar describe error analysis as the step that decides which evals to write at all. The method has four parts, and each has its own lesson:
- Sampling outputs and reading them with their inputs, in Reading Traces.
- Writing one specific note per failing output, in Open Coding: Notes on What Went Wrong.
- Grouping the notes into defined modes, in Axial Coding: From Notes to Failure Modes.
- Deciding what to do about each mode, in Counting and Ranking Failure Modes.
Error analysis has costs. It needs several hours from someone who knows the business well, and it must be repeated as users and versions change. Its counts come from about a hundred outputs, so they show which failures are common, not their exact rates. Measuring a rate precisely is the job of the evals that error analysis produces.
Key Takeaways
- Generic metrics such as helpfulness can score high while the feature breaks its own promises.
- Each important failure breaks a rule that belongs to one product, so no generic list contains it.
- Criteria drift means that grading real outputs changes what people count as a failure.
- Criteria cannot be fully decided before reading outputs, and criteria written by an LLM still need people to check them.
- Error analysis reads outputs, notes failures, groups them into modes and counts them, before any eval is written.
Every part of error analysis starts with a record of what the feature received and returned. The next lesson, Reading Traces, describes that record and how to choose which ones to read.
Practice Questions
1. A team adopts five generic quality metrics on the first day and sees all of them above 4 out of 5. Why is this weak evidence that the feature works?
<details><summary>Show answer</summary>Because generic metrics do not contain the product's own rules. A reply can be polite and complete while its urgency is wrong. The scores say the outputs read well, not that the feature keeps its promises.
</details>2. The team's rubric said "no clock time in the reply". After reading outputs, it now says "no phrase that tells the customer when the visit happens". What is this an example of?
<details><summary>Show answer</summary>Criteria drift. Grading real outputs showed replies such as "shortly" that the first criterion did not cover. The criterion improved because someone graded outputs with it.
</details>3. A colleague proposes asking an LLM to write the full list of failure criteria before anyone reads outputs. What is the problem?
<details><summary>Show answer</summary>The LLM does not know the product's rules or its users' wording. It would write sensible general rules but miss failures such as the door seal sent to a plumber. A grader written by an LLM also inherits the problems of the LLM it grades.
</details>4. Another team shares the failure list from its own booking assistant. How should the job ticket team use it?
<details><summary>Show answer</summary>As ideas to look for while reading, not as the list to measure. The other product has different users, rules and prompt. Only reading this feature's outputs shows which failures it has and how often.
</details>Reading Progress
0%
On This Page
Generic Metrics
Criteria Drift
Why the List Must Come from Your Own Outputs
What Error Analysis Is
Key Takeaways
Practice Questions