Grokking LLM Evals
0% completed
Retrieval and Generation as Two Measurements
A technician standing at a customer's dishwasher asks: "On the DW-300, which error code means the drain is blocked?" The manual says E24. Two versions of the assistant both answer "E15", and an eval that checks only the final answer marks both as wrong.
The two failures have nothing else in common. The first version found the right page, the DW-300 error code table, and then read the wrong row. The second version found the error table for a different model, the DW-200, and copied its answer exactly
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%