Kinds of Evals

A set of test cases that the feature cannot yet pass, kept so the team can measure progress toward a new ability, is called a capability eval. A set of test cases that the feature already passes, kept so that no future change breaks them, is called a regression eval. The two can contain cases that look alike, and they still answer opposite questions.

Evals differ in two more ways besides this one: whether they run before release or on live traffic, and whether they measure quality or safety. They also run at different moments, from a developer's laptop to production. This lesson describes each distinction and shows where each kind of eval runs.

Capability and Regression Evals

A capability eval measures progress toward something the feature cannot yet do, so a low pass rate is expected. Suppose the team wants the job ticket assistant to handle requests that describe two problems, such as a leaking tap and a broken light switch in one message. Today it opens one ticket and ignores the second problem. The team writes 40 such requests, and the first run passes 9 of them. That result is not a failure of the eval; it is the starting point that later changes are measured against.

A regression eval protects what already works, so anything below a pass rate near 100 percent is a warning. Its cases come from behavior the team has already fixed or promised: every danger marked as an emergency, every reply free of a promised price. When a case fails here, something that used to work has broken.

The two sets are connected. Once the assistant passes a two-problem case reliably, across several runs and several releases, that case moves into the regression set. From then on, it protects the new ability instead of measuring progress toward it.

Notice that the capability set shrinks as the feature improves, while the regression set only grows. A team that keeps a single set of cases loses this distinction, and a falling score can then mean either a new failure or a new, harder case.

Offline and Online Evals

An offline eval runs the feature on a fixed set of test cases before a change reaches users. Every eval so far in this course has been offline. Its strength is control: the same cases run before and after a change, so the two results can be compared directly.

An online eval measures the feature on live traffic after release. It grades a sample of real requests, or it watches signals such as how often dispatchers correct a ticket. Its strength is coverage: it sees requests nobody thought to write as test cases. Its weakness is delay and noise, because some outcomes, such as whether the right technician arrived, are known only days later.

The two kinds feed each other. Online results reveal new failures, those failures become offline test cases, and the offline set then protects the next release. The lesson Offline and Online Evaluation covers this exchange in detail.

Quality and Safety Evals

A quality eval asks whether the feature does its job well. Is the trade right, is the summary accurate, does the reply ask for a missing address? Quality is usually reported as a rate, and a release may accept a small number of failures.

A safety eval asks whether the feature ever does something harmful or forbidden. For the job ticket assistant, that includes telling a customer who smells gas to wait for a scheduled visit, or following instructions that someone hid inside a request. Safety rules are often zero-tolerance, and their test cases include deliberate attacks as well as ordinary requests. The lesson Turning a Safety Policy into Test Cases shows how such cases are written.

Some promises belong to both kinds. Marking danger as an emergency is part of doing the job well, and missing it is also a safety failure. That is why the decision rule in What an Eval Is gave that promise its own condition.

Where Each Kind Runs

An eval also differs by the moment it runs, and each moment has its own budget of time and money. The numbers below are one reasonable setup for the job ticket assistant, not a standard.

Continuous integration is the automated process that tests every proposed code change before it is merged. Putting the regression eval there means a broken promise blocks the change the same way a failing unit test does.

Notice that the sets grow larger as a change moves toward users. Early checks must be fast so that people run them often, while the checks just before release must be large enough to detect small drops.

Key Takeaways

• A capability eval measures progress toward a new ability, so a low pass rate is expected.
• A regression eval protects what already works, so any failure is a warning.
• Cases move from the capability set to the regression set once the feature passes them reliably.
• Offline evals give control before release, online evals give coverage after it, and each feeds the other.
• Quality evals measure how well the feature works; safety evals look for anything harmful, often with zero tolerance.
• Evals run in four places, with larger sets closer to users.

Knowing the kinds of eval does not yet say how a team should work with them from day to day. The next lesson, The Eval Development Loop, describes the cycle that ties them together.

Practice Questions

1. A new eval for two-problem requests passes 9 of 40 cases on its first run. Should the change that introduced it be blocked?

<details><summary>Show answer</summary>

No. This is a capability eval, and a low pass rate is its expected starting point. It measures progress toward a new ability. Blocking releases on it would stop the team from shipping anything until the ability is finished.

</details>

2. The regression eval drops from 100 to 98 percent after a prompt change. Why is this more serious than the same drop in a capability eval?

<details><summary>Show answer</summary>

Because every regression case passed before the change. A drop means that something which used to work is now broken. In a capability eval, a small movement is normal while the team works toward a new ability.

</details>

3. Why does the CI eval use about 300 cases rather than every case the team has?

<details><summary>Show answer</summary>

Because it runs on every proposed change and must finish quickly. If it took hours, developers would stop waiting for it or stop running it. The large sets, with repeated runs, belong to the check before release, which runs less often.

</details>

4. An online eval finds that the assistant mishandles requests written in two languages. What should happen next?

<details><summary>Show answer</summary>

Those requests should become offline test cases. Once they are in the offline set, every future change is checked against them before release. This is how online results feed the offline eval.

</details>