Model Serving Basics

Know the Vocabulary

This lesson uses the terms below many times. Read them once now, and come back here when a later section needs one.

• Model server: A service that loads a trained model and returns its predictions over the network.
• Stateless service: A service that keeps no memory of past requests between calls. Each answer depends only on the current request and the data it reads.
• Replica: One running copy of a service. A load balancer spreads requests across many replicas.