Offline Evaluation: Choosing the Metric

1. The Map That Answers "Which Metric?"

"Which metric would you use?" is about the closest thing to a guaranteed question in an AI system design interview. It is not a matter of taste. Two facts decide the answer: the problem family, and the distribution of the thing you are predicting. Classification questions draw from the precision-recall family. Ranking questions draw from the funnel metrics, recall@k and NDCG. Regression questions draw from the error magnitudes, MAE and RMSE