Grokking LLM Evals
0% completed
Cost, Latency, and Step Counts
Two versions of the scheduling agent run the same 60 tasks in the fake environment. The current version passes 51 of them. A new version, with a prompt that asks the agent to double check each step, passes 52.
By pass count, the new prompt looks slightly better. The same run also shows that its median task uses 49 percent more tokens, and that about one task in twenty takes longer than 84 seconds, against 35 seconds before. A customer waiting for a booking confirmation would notice that change before any dispatcher noticed the extra pass.
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%