Grokking LLM Evals
0% completed
Writing an Eval Report
Two engineers write up the same eval run of the job ticket assistant. The first report is three lines long.
Version 2 passes 85% of test cases, up from 80%. The judge says summaries improved from 80% to 83%. Recommendation: ship.
The second report is longer. These are its central lines.
Paired gain: +5.0 points (95 percent interval +1.2 to +8.8), 25 fixed, 10 broken Emergencies caught: 36 of 40 before, 37 of 40 after Summaries, corrected for judge error: 85.0 and 88.8 percent, intervals overlap widely
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%