The First Eval Set From Real Failures

What should be in the first test set for a new agent, and how large should it be? The natural plan is to have a model generate several hundred plausible customer messages. That list grows quickly, but it shows only what a model predicted would be difficult. A better source exists, because every team that has tried its agent has seen it fail. The work is to turn those failures into repeatable tasks, each run in an environment that no other trial can affect.

Where the Tasks Come From

**The first eval set is a small list of test tasks taken from failures that have already happened