Grokking LLM Evals
0% completed
Repeated Runs and Agent Reliability
Suppose the scheduling agent succeeds on 75 percent of its runs, and each task is run three times. One common way of counting gives it a reliability of 98.4 percent. Another common way gives it 42.2 percent.
Both numbers come from the same 75 percent, and both are correct answers to different questions. The first asks whether at least one of the three runs succeeded. The second asks whether all three did.
This lesson defines the two measures, shows how to estimate them from real repeated runs, and explains which one a product should report and how many runs each task needs.
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%