Comparing Two Versions on the Same Cases

A new prompt for the job ticket assistant passes 255 of the 300 development cases, and the old prompt passed 240. That is 85 percent against 80 percent, a gain of 5 points.

Read as two separate pass rates, the result is inconclusive: the 95 percent interval for the gain runs from minus 1.1 to plus 11.1 points, so the new prompt might even be worse. Read case by case, the same verdicts give an interval from plus 1.2 to plus 8.8 points, and the gain is clear. Nothing about the eval changed between the two readings except the method.