Offline and Online Evaluation

Offline evaluation runs a feature on a fixed set of test cases, whose good outputs are known, before a change reaches users. Online evaluation measures the feature on live requests after release, where nobody has written down the good output in advance. Both grade the same feature, yet they catch different failures, because an offline eval knows the right answer for a few cases and an online eval sees every request but knows few right answers.

The lesson Kinds of Evals introduced the two