Grokking the AI System Design Interview

0% completed

​

Online Experiments and A/B Testing

1. Why Offline Numbers Are Not Enough

A model that wins offline can still lose in production. Interviewers probe whether you know why. Offline evaluation asks one question: does the new model predict the held-out labels better than the old one? Production asks a different question entirely: does the new model make the business better? Those are not the same question. The gap between them is exactly where careful candidates separate from careless ones.

Three reasons the offline number can lie to you:

  1. The label is a proxy, meaning a stand-in for what you actually care about

.....

.....

.....

Like the course? Get enrolled and start learning!