Evals: Knowing Your Feature Works

The computer scientist Edsger W. Dijkstra made a famous point in his "Notes on Structured Programming." Program testing, he wrote, "can be used to show the presence of bugs, but never to show their absence." He wrote about ordinary programs, but the same is true of AI features.

Take a support assistant that drafts replies for an online store. This lesson uses it as the example throughout. A developer shortens its system prompt and tests it with five customer messages. All five replies read better than before, so the change ships