How Many Test Cases You Need

The function below answers a planning question before any eval runs. How precise will a pass rate be on a test set of a given size?

import math

def half_width(p, n):
    """Half the width of a 95 percent interval for a pass rate p on n cases."""
    return 1.96 * math.sqrt(p * (1 - p) / n)

print(round(100 * half_width(0.80, 100), 1))   # 7.8 points
print(round(100 * half_width(0.80, 400), 1))   # 3.9 points

Four times as many cases give an interval only half as wide. That one fact explains why each gain in precision costs more cases than the one before it.