Batch vs Online Inference

The Cheapest Model Call Is the One You Made Last Night

Inference means running the model to get a prediction. It does not have to happen the moment the user makes a request.

For many products, predictions do not need to happen live at all. You can compute them in batch, offline, for every user or item at once. The results go into a fast key-value store, which is a simple lookup-by-id database.

The request path can then read stored predictions, while still applying permissions, availability checks, and other product rules.