Training vs Serving

Know the Vocabulary

This lesson uses the terms below many times. Read them once now, and come back here when a later section needs one.

• Training plane: The part of an ML system that builds models from past data. It runs as background jobs, such as a nightly training run.
• Serving plane: The part of an ML system that answers live requests with predictions. It runs all the time, while users wait.
• Throughput: How much work a system finishes in a given time, such as the number of rows it processes per hour.
• Latency: How long one request takes, from the moment it arrives until the answer goes back.
• Training pipeline: The chain of jobs that turns raw data into an approved model. It has four stages: data preparation, training, evaluation, and registration.
• Held-out data: Past examples that the model did not train on. They are kept aside so the model can be tested fairly.
• Model registry: A store that keeps every approved model version, with its quality results and its lineage.
• Lineage: The record of what produced a model: which data, which code, and which settings.
• Rollback: Switching serving back to an earlier model version when a new version causes problems.
• Fallback: The answer the system gives when the model cannot answer in time, such as a cached or default prediction.
• Training-serving skew: Any gap between the features a model saw in training and the features it sees in serving.
• Label leakage: Training with a feature that contains part of the answer, because it uses information that would not exist yet at prediction time. The model looks excellent on past data and then does poorly on live requests.
• Point-in-time join: A way to build training examples so that each example sees only the data that existed at its prediction time.
• Distribution shift: A change over time in the data the model receives, such as new restaurants or new fraud tricks. It is also called drift.

Part 1: Two Systems That Share One Model

Suppose an interviewer says, "Design a fraud detection system." Or suppose the question is this module's example: "Design delivery-time prediction for a food-delivery app." The first useful thing to say is the same for both. This is two systems that share one model.

The first system is the training plane. It reads months of past orders, builds a model, tests it, and stores it. The second system is the serving plane. It receives a live request, gathers the model's inputs, runs the model, and returns a prediction in milliseconds.

The Module 2 introduction called these the offline plane and the online plane. This lesson calls them the training plane and the serving plane. The names mean the same thing.

The Most Common Whiteboard Error

The most common structural error in whiteboard ML designs is drawing training and serving as one box. The box is often labeled "ML service." It receives requests, and somehow it also learns from them.

That box leaves every hard question open. When does training run, and on what hardware? What happens to live requests while a new model trains? Which model version answered a given request? A single box has no answers, so the interviewer starts asking.

Two boxes with clear links between them answer those questions before anyone asks. The rest of this lesson explains what goes in each box and what crosses between them.

What the Split Is Not

The split is not about two different models. Both planes use the same model file. Training produces it, and serving runs it.

The split is not only a drawing habit. The two planes usually run on different machines, on different schedules, and often in different programming languages. That last point causes a real failure, which Part 6 covers.

Part 2: Different Jobs, Different Rules

The two planes are separate because their jobs need opposite things.

Training optimizes for throughput. It must process millions of past orders, and nobody is waiting for any single row. A training run that takes two hours is fine if it finishes before its deadline.

Serving optimizes for latency. A customer is looking at the checkout screen. Each request must finish in milliseconds, and the service must stay up all day.

The table below compares the two planes for the delivery-time example. The numbers are illustrative.

A replica is one identical copy of the serving service. A load balancer spreads incoming requests across the replicas.

Why Spot Instances Suit Training

A spot instance is a cloud machine sold at a large discount. The drawback is that the cloud provider can take it back at short notice.

Training can use spot instances because training is a batch job. A batch job processes a fixed set of data and then stops. If a machine disappears, the job restarts from its last checkpoint. A checkpoint is a saved copy of the work done so far. The run finishes later, but no user notices.

Serving cannot accept that risk. If a serving machine disappears during the busy dinner hours, live requests fail. So serving runs on machines that stay up, with spare replicas ready.

A Late Model Is a Soft Failure

Suppose the nightly training job fails, and no new model appears. The immediate effect on users is nothing. Yesterday's model keeps serving, and an alert tells the team to fix the job. This is a soft failure: something is wrong, but the product still works.

Suppose instead that the serving plane fails. Every checkout that needs an estimate is affected at once. This is an incident that users see.

This difference shapes the design. Training can be cheap, slow, and retried. Serving must be fast, redundant, and protected by a fallback.

A soft failure still needs attention. If the training job keeps failing for weeks, the old model slowly loses accuracy as the world changes. So teams also alert on the age of the serving model.

Part 3: What Crosses Between the Planes

If the planes are separate, something must connect them. Exactly two artifacts cross between them. An artifact here means a stored output that one plane produces and the other plane uses.

1. The model flows forward. The training plane stores each approved model in the model registry. The serving plane loads a chosen version from the registry. Training never pushes a model straight into a running server.

2. The logs flow back. The serving plane records every prediction together with the exact features it used and the model version that made it. Later, the real outcome is joined to that record. The next training run learns from these logged predictions and their outcomes.

The Module 2 introduction also placed a feature store between the planes. A feature store is a system that keeps feature values, with a full history for training and the latest values for serving. It is shared infrastructure that both planes read. It is not an output that one plane produces for the other. So the two artifacts that cross are still the model and the logs.

The diagram below shows the two planes, the four stages inside training, the request path inside serving, and the two artifacts that cross.

Notice that the two arrows form a loop. The model goes forward, its predictions come back as logs, and those logs train the next model. Notice also that nothing else crosses. A live request never reaches the training plane, and the raw training data never reaches the serving plane.

Part 4: Inside the Training Plane

A production training pipeline has four stages, and they run in a fixed order. The steps below follow one nightly run for the delivery-time model.

Step 1: Data preparation. The job collects past orders and their features. It then joins each order to its label, which is the real delivery time. The label arrives later than the features, often 30 to 60 minutes after checkout. So the join must match each label to the features as they were at checkout, not as they are now. This stage is most of the code, and it is where most of the bugs are. Part 6 shows the worst of them.

Step 2: Training. The job fits a model to the prepared examples. For most tabular business problems, such as fraud, delivery time, and churn, the typical default is a gradient-boosted tree. Churn prediction means predicting which customers will stop using a product. Tabular means the data comes as rows and columns, like a spreadsheet. A gradient-boosted tree is a model built from many small decision trees. Each new tree corrects the mistakes of the trees before it. It trains quickly on ordinary CPUs and often does well on this kind of data.

Step 3: Evaluation. The job tests the new model, called the candidate, on held-out data. For delivery time, that might be the most recent week of orders, which the candidate never trained on. The job also scores the model now in production on the same data. The test is fair only if neither model trained on those orders. The candidate is promoted only when it beats the production model. A newer model is not automatically a better one.

Step 4: Registration. A candidate that passes is stored in the model registry. The registry keeps the model file, its version number, its evaluation metrics, and its lineage. The lineage names the data snapshot, the code version, and the settings that produced the model.

The registry is what makes rollback possible. Suppose version 42 misbehaves in production. Serving switches back to version 41, which the registry still holds. Without the registry, nobody would know exactly what version 41 was or how to rebuild it.

Passing evaluation makes a candidate ready to meet real traffic. It does not yet make it the model every user gets. First the candidate runs in shadow mode: it scores live requests, but nobody sees its results. Then a canary release sends it a small share of traffic, and then more. Model Serving Basics explains both steps.

When One Machine Is Not Enough

A gradient-boosted tree for delivery time usually trains on one machine or a small cluster. Large deep models, such as image or language models, are different. They need many GPUs working together. This is called distributed training, and it comes in two main forms.

A batch is the group of examples a model learns from in one training step. The two forms differ in what they split.

• Data parallelism splits the batch. Each GPU holds a full copy of the model. Each GPU trains on a different slice of the same batch. The GPUs then combine what they learned, so all the copies stay the same.
• Model parallelism splits the model. The model is too large to fit on one GPU. So its parts are spread across several GPUs. For example, each GPU can hold a different group of layers, and each example passes through all of them.

Very large models often use both forms at once. A tabular model for fraud or delivery time usually needs neither.

How Often to Retrain

Every team chooses a retraining cadence, meaning how often the pipeline runs. A good answer gives a cadence and a reason for it.

For example, a team might retrain the fraud model daily, because fraudsters change their patterns quickly. The same team might start the delivery-time model weekly, because kitchens and roads change more slowly. Its labels arrive within an hour and it trains cheaply, so a nightly run is also a reasonable start. This lesson's walkthrough uses nightly runs. These are example starting points, not rules.

The real cadence comes from measuring how fast quality drops between retrains. Monitoring, Drift, and Retraining shows how to measure that drop. A weak answer gives a cadence with no reason, such as "once a day, because that is the standard."

Why Not Train Continuously?

Interviewers often ask, "Why not just train continuously?" The strongest answer has two parts: cost and safety.

Cost. Each training run and each evaluation uses compute. Most data does not change in a meaningful way from one hour to the next. So hourly runs would spend money to produce models that are almost the same.

Safety. Every new model must pass the evaluation gate before it serves users. A model that changes every few minutes cannot be checked, compared, and rolled back in an orderly way. One bad stream of data could reach users within minutes.

Some systems do update a model continuously, a method called online learning. They are the exception, and the monitoring lesson describes the extra controls they need.

Part 5: Inside the Serving Plane

The Latency Budget

The serving plane works to a latency budget: the most time one request may take. For a predictive model, a common budget for the full request path is about 10 to 100 ms. The delivery-time example uses about 100 ms. A payment fraud check might get about 50 ms in total, with about 10 ms for the model.

The table below shows where the time goes in one illustrative delivery-time request.

Notice that the model is one of the smallest lines. On the serving path, most of the latency usually goes into fetching or computing features. Only a small part goes into inference, which means running the model on one input. So when a request is slow, look at the features first.

The logging step can run asynchronously. That means the response goes back without waiting for the log write to finish.

The Fallback

Sometimes the model will not answer in time. A replica may be overloaded, or a feature read may stall. The serving plane needs a plan for that moment.

A fallback is a default or cached prediction that the system uses when the model times out. For delivery time, the fallback might be the last estimate cached for that restaurant and area. It might also be a simple rule: the restaurant's average delivery time plus a term for distance. Either way, the customer still sees a reasonable number.

A fallback is not the same as a rollback. A fallback answers one request when the model is too slow. A rollback replaces a bad model version for all requests.

Fail Closed or Fail Open

For some models, choosing the fallback is a hard decision. Consider a fraud model that times out while a customer is paying.

• Fail closed means the system blocks the payment when it is unsure. This stops fraud, but it also blocks some good customers.
• Fail open means the system allows the payment when it is unsure. This lets good customers finish paying, but some fraud gets through.

This is mainly a product and risk decision, not a modeling decision. The answer depends on what a blocked good customer costs and what a missed fraud costs. A strong answer states the choice and the reason. For example: "Fail open for payments under a small limit, because the possible loss is small. Decline larger payments, or ask for an extra check that a real cardholder can pass." Design Fraud Detection develops this choice further.

Part 6: Training-Serving Skew

What Skew Is

Training-serving skew is any gap between training-time features and serving-time features. The model learned from inputs that looked one way. In production, it receives inputs that look a different way. So its predictions get worse.

Some engineers call skew "the silent killer" of ML systems. The name describes how skew fails. Quality quietly drops, while nothing crashes and nothing pages the on-call engineer. To page someone means to send them an urgent alert. Every request still returns a number, so every health check still passes. A health check is a routine test that a service is up and answering. The numbers are simply less accurate.

Skew has two common causes, and each has a cure.

Cause 1: Two Code Paths

Suppose the data team computes features in Python and Spark for training. Spark is a tool that processes large data sets across many machines. The serving team then rewrites the same features in Java, because the serving service is written in Java.

Now each feature exists twice, and the two versions drift apart over time. For example, the training code computes "hour of day" in the restaurant's local time. The Java code computes it in UTC, a single clock used worldwide. For a New York restaurant in summer, 7 PM local time is hour 23 in UTC. The model learned the dinner rush at hour 19. Serving now tells it hour 23.

The cure is to compute each feature once, from one shared definition, and use the result in both planes. A feature store helps with this, as Features and Feature Stores explains. Teams also run parity tests. A parity test computes features through both paths for the same inputs and checks that the values match.

Cause 2: Time Travel in the Training Data

The second cause is inside data preparation. A training feature can accidentally use information from the future.

Here is an example. A training job builds a feature called "the restaurant's average delivery time today." It computes the average over the whole day. For an order placed at 6:02 PM, that average includes deliveries from later in the evening. It even includes the 6:02 PM order itself, which is the very delivery the model is trying to predict.

This is label leakage, sometimes called time travel. The feature contains part of the answer. The model learns to rely on it, so it scores very well offline.

Then the model goes live. At 6:02 PM, the serving plane cannot know how the rest of the evening will go. The value it computes covers only the deliveries finished so far. The model gets a different input from the one it learned from, and its live quality is poor.

The cure is a point-in-time join. When the job builds the training example for the 6:02 PM order, it reads each feature as it was at 6:02 PM. Data that arrived later is left out. Each example sees only data that existed at prediction time, which is exactly what serving will see.

The diagram below shows the same day twice. The top row uses the leaky feature, and the bottom row uses the point-in-time feature.