Grokking LLM Evals
Vote

0% completed

​

Writing an Eval Report

Two engineers write up the same eval run of the job ticket assistant. The first report is three lines long.

Version 2 passes 85% of test cases, up from 80%. The judge says summaries improved from 80% to 83%. Recommendation: ship.

The second report is longer. These are its central lines.

Paired gain: +5.0 points (95 percent interval +1.2 to +8.8), 25 fixed, 10 broken Emergencies caught: 36 of 40 before, 37 of 40 after Summaries, corrected for judge error: 85.0 and 88.8 percent, intervals overlap widely

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%


Vote for new content