Monitoring, Drift, and Retraining

When a Deployed Model Loses Quality

A deployed model can lose quality when data or input-outcome relationships change. This is not inevitable for every model, so measure the effect instead of assuming a fixed rate of decline.

Fraudsters test the fraud model and adapt around it. A pandemic changes delivery times overnight. A new slang word makes a toxicity classifier wrong.

These changes are described as drift. Monitoring helps distinguish changing data from implementation errors and other causes of poor results.

There are two kinds