Model Serving Basics

A Model in Production Is a Stateless Service

A common predictive model server is stateless between requests: each request supplies the inputs needed for a prediction. Models that require session state need an explicit state-management design.

It loads a file into memory at startup and answers one kind of call: "here are the features, give me a score."

Because it is stateless, everything you already know about scaling stateless services applies unchanged. Replicas behind a load balancer. Horizontal autoscaling on CPU or queue depth. Rolling deploys. Health checks.