Grokking LLM Evals
Vote

0% completed

​

A Pass Rate Is an Estimate

Suppose the job ticket assistant passes 90 of 100 test cases. What is its pass rate?

The obvious answer is 90 percent, and it is only partly right. Ninety percent is the rate on those 100 cases. The rate a team actually cares about is the one on the requests customers will send next month, and no test set contains those requests.

The sections below treat a pass rate as a measurement taken on a sample, and show how to state how far that measurement might be from the value it estimates.

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content