Grokking LLM Evals
Vote

0% completed

​

Counting and Ranking Failure Modes

Two failure modes from the job ticket sample have exactly the same count: 5 traces each out of 120. In one, a customer wrote that water was dripping through the hall ceiling, and the ticket said same day. In the other, a customer forgot to give the address of a rental apartment, and the reply did not ask for it.

The second failure costs a dispatcher a phone call and the customer a few minutes. The first leaves water coming through a ceiling for hours while the ticket waits in the same-day queue. A list sorted by count alone would place them side by side.

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%


Vote for new content