Batch vs Online Inference

The Cheapest Model Call Is the One You Made Last Night

Inference means running the model to get a prediction. It does not have to happen the moment the user makes a request.

For many products, predictions do not need to happen live at all. You can compute them in batch, offline, for every user or item at once. The results go into a fast key-value store, which is a simple lookup-by-id database.

The "model serving" path at request time then becomes a plain cache read, and nothing more.

Netflix has famously precomputed large parts of each member's homepage overnight