Sampling Real Traffic

The job ticket assistant receives about 18,000 requests a day, or 126,000 in a week. Suppose that 3,700 of a week's requests are emergencies, and only 300 of those are appliance emergencies, such as a stove that smells of gas. Now draw 200 requests at random from that week. The sample would contain about 6 emergencies.

The rarer combinations of trade and urgency are worse. Such a sample would contain about half an appliance emergency on average, and in 62 percent of draws it would contain none at all. An eval built on those 200 cases could report a high pass rate