Grokking LLM Evals
0% completed
Grouped Test Cases and Clustered Error Bars
Test cases are grouped, or clustered, when several of them share one source, so that a result on one case says something about the result on the others. Four rewordings of one customer request form a group. So do three messages from one conversation, or ten questions about one manual page.
The error bars in the earlier lessons assume the opposite: that every case is a separate, independent draw from the requests customers send. When cases come in groups, that assumption is false, and the error bar comes out too narrow.
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%