0% completed
Axial Coding: From Notes to Failure Modes
On This Page
Where the Method Comes From
From Notes to Groups
Pass and Fail Definitions
Checking a Definition
When to Stop Reading
Key Takeaways
Practice Questions
Grouping the free-text notes from open coding into a small number of named categories, each with a definition that someone else could apply, is called axial coding. Each category becomes a failure mode, and its definition must be precise enough to decide for any single output whether it occurred.
Husain and Shankar call axial coding the most important step of error analysis. Notes describe single traces, but failure modes are what a team can count, rank and turn into evals.
This lesson traces the method to its origin, shows how the job ticket notes become failure modes, explains how to write a pass and fail definition for each mode, and describes when to stop reading.
Where the Method Comes From
Open and axial coding come from grounded theory, a method from sociology for building categories out of observed data instead of starting from a theory. The sociologists Barney Glaser and Anselm Strauss developed it and described it in their book The Discovery of Grounded Theory. Strauss and Juliet Corbin later defined axial coding as putting data back together in new ways after open coding, by making connections between categories.
The method is not uniform even among its own researchers. Charmaz, another grounded theory author, regards axial coding as highly structured and optional. For evals, the structured version is useful, because the goal is a short list of categories that can be counted and tested, not a full social theory.
From Notes to Groups
Axial coding sorts notes by what the output did wrong, then names and defines each group. The team puts the 34 notes from the sample in one list and reads them together. Notes that describe the same kind of failure go into one group, and each group gets a short working name.
Three notes from the job ticket sample show how a group forms:
- "Door seal leak sent to plumbing; the seal is inside the washer, so appliance."
- "Oven igniter clicks and will not light; marked electrical, but the igniter is part of the oven."
- "Dishwasher drain pump hums with no water draining; marked plumbing."
Each note describes a request on the border between two trades, sent to the wrong one. The group is named wrong trade on a boundary case.
Two kinds of decision shape the groups. Merging joins groups that need the same fix and the same grader. Notes about a missing address and notes about a missing problem description start as two groups. Both break one promise in one field, the reply's duty to ask, so they become one mode. Splitting divides a group whose notes need different fixes. An early group called "summary problems" held invented ages and also summaries that dropped the customer's second problem. One adds something and the other leaves something out, so they became two modes.
Notice that every note ends in exactly one mode. Because each trace has only its first failure noted, the mode counts will add up to the number of failing traces.
Pass and Fail Definitions
A failure mode is finished only when it has a written definition of fail and of pass, with an example of each. A name such as "invented detail" is clear to the person who wrote it, but a second reader, or a grader, needs the boundary spelled out. Here are two of the six definitions:
| Mode | Fail when | Pass when |
|---|---|---|
| Danger under-called | The request mentions a gas smell, sparks or burning, water through a ceiling or flooding a floor, or no heat in freezing weather, and urgency is not emergency | Every such request is marked emergency, however mildly the customer wrote it |
| Invented detail in the summary | The summary states a brand, model, age, cause or other fact the customer did not give | Every fact in the summary appears in the request or the account |
Each definition comes with two short examples. The danger mode lists "a faint smell of gas in the basement" as a fail when marked scheduled. Its pass example is "water dripping through the hall ceiling" marked emergency. Examples settle cases that the words alone leave open.
These definitions are what later lessons grade. Once each test case records its correct urgency, a code check can apply the danger definition, while the invented detail definition needs a judge, as Writing a Judge for One Failure Mode shows.
Checking a Definition
A definition is good enough when a second person, given only the definition, assigns the same notes to the same modes. The engineer takes 10 notes written by the dispatcher and sorts them using only the six definitions. Suppose the engineer agrees with the senior dispatcher on 9 of the 10.
The one disagreement is a summary that says "upstairs bathroom" for a customer who wrote "the bathroom above the kitchen". The engineer counted it as an invented detail, and the dispatcher did not, because the fact follows directly from the customer's words. The team adds this case to the definition's pass examples, and both readers now agree. This is an example of criteria drift, and it is the reason to check definitions before building graders on them.
When to Stop Reading
Reading can stop when new traces no longer produce new failure modes, a point qualitative researchers call theoretical saturation. Husain and Shankar suggest reading at least 100 traces and continuing until saturation.
The job ticket readers kept a simple record: the trace number at which each mode first appeared. Wrong trade appeared at trace 4, invented detail at trace 7, danger under-called at trace 15, and missing information at trace 22. A reply promising an arrival time or price first appeared at trace 38, and a second problem ignored at trace 88. From about trace 90 to trace 120, every failure fit a mode the team already had.
Notice that the curve rises quickly and then stays flat. The last new mode appeared late, which is why stopping at 50 traces would have missed it.
Saturation means only that this sample has no new modes. A rare failure, such as one in every thousand requests, is unlikely to appear in any sample of 120. The lesson Hard Cases, Edge Cases, and Attacks adds such cases on purpose.
An LLM can help with grouping once the readers have formed their own view. After the notes are written, a team can ask a model to propose groups and compare them with its own. The domain expert still makes the final decision on each mode and writes its definition.
Key Takeaways
- Axial coding groups open-coding notes into named failure modes, each with a definition others can apply.
- The method comes from grounded theory, by Glaser and Strauss, with axial coding defined by Strauss and Corbin.
- Merge groups that need the same fix and grader; split groups whose notes need different fixes.
- Every mode needs a fail definition, a pass definition, and an example of each, checked by a second reader.
- Stop reading at saturation, when new traces add no new modes, and only after reading at least 100 traces.
The six modes now have names and definitions, but not yet an order. The next lesson, Counting and Ranking Failure Modes, decides which ones to act on first.
Practice Questions
1. Two groups of notes cover replies that do not ask for a missing address and replies that do not ask for a missing problem description. Why does the team merge them into one mode?
<details><summary>Show answer</summary>Because they break the same promise in the same field and need the same fix and grader. Keeping them apart would produce two small counts for one problem, and two graders that check almost the same thing.
</details>2. A failure mode is named "bad summary". What does it need before anyone can grade it?
<details><summary>Show answer</summary>A fail definition, a pass definition, and an example of each. "Bad" does not say what to look for. A usable mode states the boundary, such as "the summary states a fact the customer did not give", so that a second reader or a grader reaches the same verdict.
</details>3. After 50 traces the team has five modes, and the next 20 traces add none. Should it stop reading?
<details><summary>Show answer</summary>Not yet. Husain and Shankar suggest reading at least 100 traces. In the job ticket sample, the sixth mode first appeared at trace 88, so stopping at 70 would have missed it.
</details>4. The engineer and the dispatcher disagree about whether "upstairs bathroom" is an invented detail when the customer wrote "the bathroom above the kitchen". What should the team do?
<details><summary>Show answer</summary>Decide, and add the case to the definition as an example. The disagreement shows that the definition left this case open. Writing it into the pass examples lets every later reader and grader reach the same verdict.
</details>Reading Progress
0%
On This Page
Where the Method Comes From
From Notes to Groups
Pass and Fail Definitions
Checking a Definition
When to Stop Reading
Key Takeaways
Practice Questions