Grokking Agentic Design Patterns
Vote

0% completed

​

Capability Evals vs Regression Evals

Suppose Desk runs two suites after a change. The first passes 13 of 40 tasks, which is 32.5 percent. The second passes 38 of 40, which is 95 percent.

The first result may be good news, while the second should stop the release until someone reads its 2 failures. The two suites answer different questions, so neither score means anything until you know what its suite is for.

Two Questions

A capability suite asks what the agent can do well, and a regression suite asks whether it still does what it did before. Anthropic's article on evals draws this distinction

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content