Evals as Release Gates

The following function decides whether a change to the job ticket assistant may be merged.

def release_gate(passed, total, floor=0.95):
    low, high = wilson(passed, total)   # 95 percent interval for the pass rate
    if low >= floor:
        return "pass"
    if high < floor:
        return "fail"
    return "too close to call"

Most gates compare a pass rate with a number. This one compares a range of plausible pass rates with the number. It also has a third answer, for when the range sits on both sides of the number.