Choosing a Model with Your Own Evals

If a new model reached the top of a public leaderboard tomorrow, would you switch the job ticket assistant to it? And if a smaller model cost a third as much, what would you need to see before trusting it with gas leaks?

The previous three lessons answer half of those questions. A leaderboard rank measures general preference, a benchmark score can carry contamination and label errors, and neither one contains your customers' requests. The other half needs a method that ends in a decision.