Keeping a Test Set Useful

Before the job ticket team split its test set, a developer spent two weeks improving the assistant's prompt. After each of 23 edits, the developer reran all 400 cases and read the failures. The number of passing cases climbed from 360 to 394, and the change was released.

Within a week, dispatchers were correcting more tickets than before. Suppose the share of tickets they corrected rose from 6 percent to 9 percent