Case Study: The Eval Set for Desk

Before designing Desk's eval set, state what a passing result will and will not mean.

A passing result promises three things:

• On these tasks, in a clean environment, Desk ended in the required state at the measured rate.
• This version is no worse than the last version on the behaviors recorded as tasks.
• When a task fails, a transcript shows what happened.

It does not promise three others:

• That Desk will handle a kind of message that no task contains.
• That Desk resists a determined attacker, which a separate audit of the guardrails must test.