Correcting a Pass Rate for Judge Error

The team runs a new prompt version of the job ticket assistant on the 300 cases of its development set. The invented detail judge reads all 300 summaries and passes 240 of them, so the judge reports a pass rate of 80 percent.

The judge was validated in the previous lesson, so its errors are known. It passes 92 percent of good summaries and fails 88 percent of bad ones. Taking those errors into account, the best estimate of the true pass rate is not 80 percent but 85 percent. The judge's own mistakes moved the number by 5 points.