Noise from Repeated Runs

The release gate for the job ticket assistant had one rule: block any change that lowers the pass rate on the 300 development cases by more than 2 points. A change that touched only the wording of the customer reply scored 78.7 percent, against 81.7 percent for the current version. The gate blocked it for a 3 point drop.

An engineer then ran the current version ten times, with nothing changed at all. Its scores ranged from 79.7 to 82.0 percent. The 3 point drop came from running the eval once. When the two versions were later compared with several runs per case, the change showed no loss.