Model Serving Basics

A Model in Production Is a Stateless Service

Start with the simple part. A deployed predictive model is a stateless service, meaning it keeps no memory of past requests between calls.

It loads a file into memory at startup and answers one kind of call: "here are the features, give me a score."

Because it is stateless, everything you already know about scaling stateless services applies unchanged. Replicas behind a load balancer. Horizontal autoscaling on CPU or queue depth. Rolling deploys. Health checks.