Contamination, Label Errors, and Saturation

Up to 8 points: the drop some leading models showed on 1,205 new math problems, compared with a famous public set. 6.49 percent: the estimated share of MMLU questions that contain errors. About 5 points: the change in MMLU accuracy that one lab saw after relabeling the answer options or adding one space.

Each number describes one way a benchmark score can be wrong about its own tasks. The previous lesson showed that a benchmark is weak evidence about your product. These problems mean it can also be weak evidence about its own subject.