Drift and Model Updates

On a Monday morning, the dispatch lead reports that dispatchers raised many more tickets to emergency over the weekend than usual. Nobody on the team has changed the prompt, the code or the settings for nine days.

The nightly eval runs 60 fixed emergency cases against the live feature, and it tells the same story. From Monday to Friday it marked 59 of the 60 as emergency. That is a recall of 98.3 percent, where recall is the share of true emergencies the feature catches. On Saturday and Sunday it marked only 52.