Validating a Judge Against Human Labels

If a judge agrees with your domain expert on 91 of 100 summaries, is it good enough to grade every summary your feature writes?

The question sounds as if it has an obvious answer. It does not, because 91 percent agreement can come from a judge that catches most failures, or from one that catches far fewer. A judge that passes every summary without reading it would still agree with the expert on every passing summary.

The measurement that separates these cases looks at passes and failures separately