The Help Center Eval Plan

A weak eval plan for the help center bot fits in three lines. Test 50 questions by hand before launch. Require 90 percent of answers to look correct. After launch, watch the thumbs-up rate.

Each line sounds reasonable, yet this plan could easily have shipped version B, which answered unanswerable questions, and it would not have noticed the stale deadline for a week. It also could not tell a retrieval problem from a writing problem.

The plan this design produced also fits on one page, but every line names a number, an owner or a decision