What a Public Benchmark Measures

A public benchmark is a fixed set of tasks, published with a fixed way of presenting each task to a model and a fixed way of scoring the answers. Anyone can run many models on it and compare their scores. Researchers build it once and release it, and model makers then report their results on it.

That definition has the same parts as an eval: test cases, a system under test, a grader, and a number at the end. The difference is whose cases they are, and what the number is for