Grokking LLM Evals
Vote

0% completed

​

The Help Center Bot and Its First Error Analysis

A help center bot answers about 9,000 questions a day from about 600 help articles. Two people read 140 of its traces over two days, found a failure in 41 of them, and named six kinds of failure.

Those numbers describe the first complete eval design in this course. The feature is a help center bot for a payroll software product. Every later step of the design depends on the six failure modes at the end of this lesson.

What follows describes the bot and its traces. It then works through the error analysis behind the six modes and ranks them by cost.

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content