Open Coding: Notes on What Went Wrong

Here is one failing trace from the sample of 120, shortened to its request and output.

{
  "trace_id": "t071233",
  "request": "Our washing machine leaks from the front door seal every time it runs. It leaves a small puddle by the machine. We are home after 4.",
  "output": {
    "trade": "plumbing",
    "urgency": "same_day",
    "summary": "Washing machine leaks from the front door seal during each cycle, leaving a small puddle. Customer home after 4.",
    "reply": "Thanks for letting us know. A plumber will contact you today to arrange a visit."
  }
}

And here is the note the senior dispatcher wrote on it.

Trade is plumbing, but the leak comes from the door seal, which is inside the
appliance, so the trade should be appliance.

The note names one problem, says what should have happened, and cites the rule that decides it. It does not mention the reply, which also says "a plumber", even though the reply is wrong too. This lesson explains how to write notes like this one, why only the first failure is noted, and how good and bad notes differ.

What Open Coding Is

Open coding is writing a short, free-text note on each failing trace that describes what went wrong, before deciding on any categories. The term comes from qualitative research, where a researcher attaches labels, called codes, to observed data. Husain and Shankar adapted the step for LLM outputs, and at least one tool vendor now teaches it in nearly the same words: read each trace and write a note about the first thing that went wrong.

The word "open" matters. The reader does not start with a list of failure types and tick boxes. A list fixed in advance would only find the failures someone already imagined, which is the problem the lesson Why Error Analysis Comes First described. Categories come later, from the notes themselves.

The First Failure Only

When a trace fails in more than one way, the reader notes only the first failure, because an early mistake often causes the later ones. In the washing machine trace, the wrong trade came first, and the reply said "a plumber" because the trade said plumbing. Fixing the trade would fix the reply as well. If the reader noted both, the count would show two problems where there is one cause.

"First" means first in the order the feature produces its output: trade and urgency, then the fields, then the summary, then the reply. That order is a practical guide, not a law. When two failures have clearly separate causes, such as a wrong urgency and an invented brand in the summary, the reader still notes the first one and moves on. A failure that is real will appear again in other traces where it comes first.

Notice that the reply's mistake comes after the trade and follows from it. Noting only the first failure keeps each trace to one note, which makes the notes possible to count later.

Husain and Shankar recommend this rule when a team is starting out, which is the situation of a first pass like this one.

Good Notes and Bad Notes

A good note is specific enough that someone who never saw the trace could tell what went wrong and what should have happened. It names the field, describes what the output did, and says what was expected or which rule was broken. A bad note records only the reader's reaction.

The bad notes are short, but they lose information that the next step needs. "Hallucination" could mean an invented age, an invented brand, or a wrong diagnosis, and these might need different fixes. "Model confused" names no failure at all. The good notes are longer by 11 to 13 words each, and each one could later be checked against a definition.

Notice that each good note contains three things: the field, what the output did, and what was expected. When a note is missing one of these, the reader should add it while the trace is still on screen.

Writing Notes in Practice

A few habits make the notes easier to group later:

• Describe, do not diagnose. "Summary adds an age" is what the reader saw. "The model is copying from its training data" is a guess about the cause, and it belongs in a separate column if anywhere.
• Quote the output. A short quote, such as "within 2 hours", lets anyone check the note without opening the trace again.
• Write a note even when unsure. A note such as "Reply says 'soon'; is this a promised time?" is useful. Questions like this often lead the team to sharpen a criterion, as criteria drift predicts.
• Leave passing traces blank. Only failures get notes, so 120 traces in this sample produce 34 notes, not 120.

The senior dispatcher writes the verdicts and notes, because the dispatcher decides what counts as a failure. Husain also suggests that the domain expert explain each verdict in a short written critique, and a good note is that explanation. The engineer reads alongside the dispatcher and adds a separate column for any cause visible in the code.

What a Note Is Not

A note is not yet a category, and the reader should resist naming categories while reading. A reader who decides after ten traces that the main problem is "trade errors" will start writing "trade error" on every trace that resembles one, and will miss the differences between them. Grouping is a separate step, done after reading, and the next lesson describes it.

A note is also not a fix. The reader may see how to fix a failure while writing about it, but changing the prompt in the middle of the sample would mean the remaining traces describe a version that no longer exists. Ideas for fixes go into a list, and the team acts on them after counting.

Key Takeaways

• Open coding means writing a free-text note on each failing trace before any categories exist.
• The reader notes only the first failure, because early mistakes often cause later ones.
• A good note names the field, what the output did, and what was expected or which rule it broke.
• Notes describe what was seen and quote the output; guesses about causes go elsewhere.
• Categories and fixes both wait until reading is finished.

The 34 notes now describe 34 failures, each in its own words. The next lesson, Axial Coding: From Notes to Failure Modes, turns them into a short list of named failure modes that can be counted.

Practice Questions

1. A trace has a wrong urgency, and its reply tells the customer to expect a visit "next week" as a result. How many notes does the reader write, and about what?

<details><summary>Show answer</summary>

One note, about the urgency. The urgency came first and caused the reply's wording. Noting both would count one cause as two failures.

</details>

2. A reader writes "hallucination" on eight traces. Why will this cause trouble in the next step?

<details><summary>Show answer</summary>

Because the note hides which failure occurred. It could mean an invented age, an invented brand, or a guessed cause. The grouping step needs to see what each output actually did, so the note should name the field and quote the invented detail.

</details>

3. After 15 traces, the reader notices many trade errors and decides to write "trade error" whenever one appears. What is lost?

<details><summary>Show answer</summary>

The differences between those failures. Some may be appliance boundary cases, and others may be vague requests where any trade was a guess. Categories should come from the notes after reading, not be fixed partway through.

</details>

4. While reading, the engineer sees an easy prompt fix for invented ages and wants to deploy it at once. What should happen instead?

<details><summary>Show answer</summary>

The idea goes on a list, and the reading continues on the current version. Changing the prompt in the middle would leave the rest of the sample describing a version that no longer exists. Fixes are chosen after the failures are counted.

</details>