Offline Evaluation: Choosing the Metric

The Map That Answers "Which Metric?"

Be ready to explain which metric fits the task and what it does not measure.

Choose metrics using the product goal, task, class distribution, and cost of different errors.

Classification questions draw from the precision-recall family. Ranking questions draw from the funnel metrics, recall@k and NDCG. Regression questions draw from the error magnitudes, MAE and RMSE.

The base rate, or frequency of the target event, helps interpret the metric. It does not determine the choice by itself.