Online Experiments and A/B Testing

Why Offline Numbers Are Not Enough

A model that wins offline can still lose in production. Interviewers probe whether you know why.

Offline evaluation asks one question: does the new model predict the held-out labels better than the old one? Production asks a different question: does the new model make the business better?

Those are not the same question. The gap between them is exactly where careful candidates separate from careless ones.

There are three reasons the offline number can mislead you.

1. The label is a proxy, meaning a stand-in for what you actually care about