Offline Evaluation: Choosing the Metric

The Map That Answers "Which Metric?"

"Which metric would you use?" is about the closest thing to a guaranteed question in an AI system design interview.

It is not a matter of taste. Two facts decide the answer: the problem family, and the distribution of the thing you are predicting.

Classification questions draw from the precision-recall family. Ranking questions draw from the funnel metrics, recall@k and NDCG. Regression questions draw from the error magnitudes, MAE and RMSE.

Then the base rate picks the specific metric within that family