Batch vs Online Inference

1. The Cheapest Model Call Is the One You Made Last Night

Inference, meaning the act of running the model to get a prediction, does not have to happen the moment the user shows up. For many products, predictions do not need to happen live at all. You can compute them in batch, offline, for every user or item at once, and store the results in a fast key-value store, a simple lookup-by-id database. The "model serving" path at request time then becomes a plain cache read, nothing fancier. Netflix has famously precomputed large parts of each member's homepage overnight