Reading Traces

The job ticket assistant handles about 18,000 requests a day, or 126,000 in a week. The team will read 120 of them, about 1 in every 1,050.

That number sounds too small to learn anything from, yet it is enough to find the failures that matter most. It works only if the 120 are chosen well, read by the right people, and recorded on a simple sheet.

This lesson covers what a trace holds, how to choose the 120, who reads them, what they write down, and how long the reading takes.

What a Trace Holds

A trace is the stored record of one request: everything the feature received, the version that handled it, everything it returned, and what happened afterward. Error analysis reads traces rather than outputs alone, because a summary cannot be judged without the request it summarizes. Here is one trace from the job ticket assistant, shortened:

{
  "trace_id": "t061904",
  "received": "Thu 18:42",
  "request": "Our furnace makes a loud bang every time it starts. It still heats. Evenings are best for us.",
  "account": {"address_on_file": true, "home_type": "house"},
  "version": {"prompt": "ticket-v7", "model": "model-a", "temperature": 0},
  "output": {
    "trade": "heating_and_cooling",
    "urgency": "scheduled",
    "summary": "Furnace makes a loud bang at startup but still heats. Customer prefers evenings.",
    "reply": "Thanks for the details. A dispatcher will contact you to arrange an evening visit."
  },
  "later": {"dispatcher_changed": []}
}

The trace has four parts. The input is the customer's text plus the facts the app already knew. The version names the prompt, model and settings, so a failure can be tied to the system that produced it. The output is the full answer, not only the fields a grader checks. The later part records what happened next, such as a dispatcher changing the urgency.

Notice that the reader needs all four parts. The input shows what the customer asked for, and the later part sometimes shows that a person already noticed a problem. The lesson Logging Traces for Evaluation covers how a team stores traces in production.

Choosing the 120

A random sample gives every trace the same chance of being read, so the failures found are ones the feature really has, in roughly the share they occur. The team draws 120 at random from one full week. A week covers weekdays and weekends, mornings and nights, and the different requests each brings.

Other ways of choosing look easier, and each one hides something. Reading the 120 newest traces covers about ten minutes of one day. Reading only tickets a dispatcher corrected finds failures someone already noticed and misses failures nobody reported. Husain and Shankar advise keeping some random traces in every batch for this reason. Targeted samples are useful later, once the team knows what it is looking for, and the lesson Sampling Real Traffic covers them.

The program below draws the 120 from a week of trace ids. The ids are generated inside the file, so it runs without a trace store.

Running it prints the following:

traces in the week: 126,000
traces read: 120 (about 1 in 1,050)

by day of the week
  Mon:  24
  Tue:  14
  Wed:  15
  Thu:  20
  Fri:  25
  Sat:  12
  Sun:  10

by part of the day   sampled   expected
  night                 10         10
  morning               49         46
  afternoon             40         41
  evening               21         24

reading sheet (first 5 rows)
  trace_id  day  verdict  note
  t004915  Mon  ______   ________________
  t005139  Mon  ______   ________________
  t006106  Mon  ______   ________________
  t006329  Mon  ______   ________________
  t006500  Mon  ______   ________________

reading time: 120 x 2 min = 240 min = 4 hours

The sample includes every day and every part of the day. The parts of the day land close to their expected counts, such as 49 mornings against 46 expected. The days are uneven, from 10 traces on Sunday to 25 on Friday, which is normal for a random draw of this size. The fixed seed means the same 120 are drawn every time, so a second reader can open exactly the same traces.

Notice that the random draw reaches requests nobody complained about. Those are where the unreported failures are.

Who Reads Them

Traces are read by a domain expert, the person who decides what counts as a failure, together with an engineer who knows how the feature works. For the job ticket assistant, the domain expert is a senior dispatcher, who knows that a faint gas smell is an emergency and that a door seal leak is appliance work. The engineer knows which prompt version produced the output and can see when a failure comes from the code rather than the model.

Husain and Shankar recommend that one domain expert make the final decision on quality for a small or mid-sized team. Two readers who disagree about a trace then have a way to settle it, and the definitions stay consistent across the whole sample.

The readers should form their own view before using any automated help. Husain and Shankar advise annotating at least 30 traces yourself before reviewing suggestions from an AI tool, because early suggestions can bias the reader's judgment.

The Reading Sheet

The reading sheet has one row per trace and three columns that matter: the trace id, a verdict, and a note. The verdict is pass or fail. The note is written only for a failing trace, and it describes the first thing that went wrong. A spreadsheet is enough.

The note is the most important column, and it has its own method, taught in the next lesson. The verdict looks only at whether the trace breaks a promise of the feature, as listed in The Job Ticket Assistant, or does something a person who knows the work would call wrong.

What the Reading Costs

Reading 120 traces with care takes about half a working day. Suppose a trace takes two minutes on average: a short request is read in less, and a long story with a two-part summary takes more. The program above computes 120 times 2 minutes, or 240 minutes, which is 4 hours. With two people reading together, the cost is about 8 hours of staff time.

That cost falls when reading is easy. Husain argues that teams should remove every obstacle to looking at data, often by building a small viewer that shows the request, the ticket and the reply on one screen. A viewer that saves even 30 seconds per trace saves an hour on 120 traces.

As for the size, Husain and Shankar suggest reading at least 100 traces and continuing until new traces stop showing new kinds of failure. The lesson Axial Coding: From Notes to Failure Modes explains that stopping point.

Key Takeaways

• A trace holds the input, the version that handled it, the full output, and what happened later.
• A random sample from a full week finds the failures the feature really has, including unreported ones.
• A domain expert decides what counts as a failure, and an engineer explains where it came from.
• The reading sheet needs a trace id, a pass or fail verdict, and a note on the first failure.
• Reading 120 traces takes about 4 hours, or 8 hours of staff time for two readers, and a simple viewer reduces that.

The readers now have 120 traces and a sheet with an empty note column. The next lesson, Open Coding: Notes on What Went Wrong, shows how to write notes that can later be grouped and counted.

Practice Questions

1. To save time, a team reads only the tickets that dispatchers corrected last week. What will this sample miss?

<details><summary>Show answer</summary>

Every failure that nobody noticed or reported. A corrected ticket is one a person already caught. Failures such as an invented detail in a summary may pass unnoticed by dispatchers, and only a random sample gives those traces a chance to be read.

</details>

2. Why does a trace store the prompt version and model, and not only the request and the output?

<details><summary>Show answer</summary>

So that each failure can be tied to the system that produced it. If the prompt changed during the week, the same failure might belong to one version only. Without the version, the team cannot tell whether a failure is still present.

</details>

3. The random sample of 120 contains 25 traces from Friday and 10 from Sunday. Is the sample broken?

<details><summary>Show answer</summary>

No. A random draw of 120 traces is uneven by chance. Every day and every part of the day is still represented, and the fixed seed lets anyone redraw the same traces.

</details>

4. An engineer offers to read all 120 traces alone, with an AI tool suggesting a verdict for each one. What two problems does this create?

<details><summary>Show answer</summary>

No domain expert decides what counts as a failure, and the suggestions can bias the reader. The engineer may not know that a faint gas smell is an emergency. Husain and Shankar advise annotating at least 30 traces yourself before looking at an AI tool's suggestions.

</details>