Grading Sampled Production Traffic

The job ticket assistant receives about 18,000 requests a day. Online, each traced request can be graded by three judges, one for each of three promises. They check that danger is marked as an emergency, that the summary invents nothing, and that the reply promises no arrival time or price. Suppose each judge call costs 0.4 cents. Grading every request would take 54,000 calls and cost 216 dollars a day.

The team grades far fewer. It takes a random 2 percent of requests, about 360 a day, and adds every request that contains a danger word, about 540 a day. That costs about 10.67 dollars a day