0% completed
Training vs Serving
On This Page
Two Systems, Not One
The Training Pipeline, Piece by Piece
The Serving Path, Piece by Piece
Training/Serving Skew
In the Interview
TL;DR
Two Systems, Not One
An interviewer says "design a fraud detection system."
Explain two responsibilities: training the model and serving predictions. They share model artifacts, feature definitions, and operational metadata.
The training system turns historical data into a model file. The serving system loads that file and answers live requests.
Their hardware, schedules, reliability needs, and ownership may differ. Decide these from the workload rather than assuming separate teams are required.
| Training | Serving | |
|---|---|---|
| Runs | On a schedule (nightly, weekly) or triggered | Continuously, per request |
| Optimizes for | Throughput over huge datasets | Latency and availability |
| Hardware | GPU/large-memory clusters, interruptible capacity if recoverable | CPU or small GPU fleets, always on |
| Failure means | A late model; yesterday's keeps serving | User-facing errors; on-call alerts fire |
| Data | Months or years of history | This request's inputs, right now |
Two rows in that table deserve a sentence each.
Hardware. Training can use interruptible instances when jobs can checkpoint and resume safely. Serving needs enough reliable capacity to meet availability targets, even if some capacity is interruptible.
Failure. If training fails, the previous model may continue serving. The urgency depends on how long that model remains useful. A serving failure may affect a user immediately, so define a safe fallback.
Showing training and serving separately makes their different responsibilities easier to explain.
Draw two separate groups of components from the start. The offline plane trains. The online plane serves.
Show the model moving from training to serving and prediction records returning for evaluation and future training. Schemas, preprocessing code, and configuration may also be shared.
The Training Pipeline, Piece by Piece
A production training pipeline is a scheduled workflow, often built on a workflow scheduler such as Airflow.
Four common stages provide a useful overview. Add detail where the role and question require it.
1. Data preparation. Pull raw events from the warehouse or lake. Join them into training examples.
Each example pairs features as they looked at prediction time with labels that arrived later.
Data preparation needs careful testing because incorrect joins or features can invalidate the training results.
2. Training. Fit the model to the data.
A gradient-boosted tree is a model built from many small decision trees, where each tree corrects the mistakes of the ones before it. It is a useful baseline for many tabular tasks, meaning tasks whose data is organized in rows and columns.
Compare it with other models using the task's quality and cost requirements. A modest neural model is the other common choice.
Training time depends on the data, model, hardware, and tuning process. Distributed training is useful when a single machine cannot meet memory or completion-time requirements.
Data parallelism distributes training examples across workers. Model parallelism distributes parts of the model. State which you need and why.
3. Evaluation. Score the new model on held-out data. Held-out data is data the model never saw while training.
Compare it with the current version using the required quality, latency, cost, and safety criteria. A release may be justified by lower cost at similar quality, not only higher accuracy.
4. Registration. Push the approved model into a model registry.
The registry is a system of record for models. It stores the model file, its version, its metrics, and its lineage. Lineage means the data and the code that produced it.
The registry records versions and supports rollback. Restoring a version also requires compatible feature definitions, dependencies, and deployment configuration.
💡 Propose a retraining schedule with a reason, then confirm it using label availability, measured quality changes, and operational cost. Daily or weekly retraining is an assumption to test, not a fixed rule for a product category.
The Serving Path, Piece by Piece
The online plane is a low-latency read path. A request comes in, and something must answer fast.
The path has four steps. A request arrives. Features are fetched or computed, which the next lesson covers. The model runs. The prediction returns, and the whole exchange is logged.
Set a response-time target for the full path, including feature reads, preprocessing, inference, and networking. Use measurements or clearly stated estimates.
Feature fetching can take longer than inference for small models. Measure each stage before choosing an optimization.
Serving also has to meet the classic system design requirements. These get forgotten once there is a model in the design.
- Timeouts.
- Fallbacks. A fallback is a default score or a cached prediction, returned when the model times out.
- Horizontal scaling behind a load balancer.
- Versioned rollout.
One more question is about the product, not the model. Does a fraud model fail closed, blocking everything when it is unsure? Or does it fail open, allowing everything through? Name which one, and say why.
Training/Serving Skew
This is the classic silent failure, and a common interview question.
Skew is any difference between what the model saw in training and what it sees in serving. The model trained on features computed one way. In production it meets features computed a slightly different way.
Quality may decline without a service error. Data and quality monitoring are needed to detect this.
There are three common sources.
1. Different code paths. Features get computed in Python or Spark for training. Then they are written a second time in Java for serving. The two versions drift apart.
A different default for missing values. A different rounding rule. A different time zone. Any one of them is enough.
Use shared feature definitions and tested transformations for training and serving. A feature store can help maintain consistency, but it does not automatically eliminate differences.
2. Information leakage. A training example accidentally includes information that would not have been available when the prediction was made.
Take "restaurant's average delivery time today", computed over the whole day. That average includes the very delivery being predicted.
Offline results may look better than production results because the model used information that was unavailable at serving time.
The cure is a point-in-time-correct join during data preparation. Each example then sees only data that existed at that exact moment.
3. Distribution shift between training data and live traffic. You trained on last month's traffic, and you are serving this month's.
This is not a bug. But it has the same symptom. Monitoring, Drift, and Retraining treats it.
💡 To check feature consistency, use shared definitions, point-in-time joins, and tests that compare training and serving transformations. Logging the features actually used can help, subject to privacy controls. Logs can still be incomplete or processed incorrectly, so this does not make skew impossible.
In the Interview
The 30-second answer (to "walk me through the ML architecture"): "The training pipeline prepares point-in-time-correct examples, trains a candidate, evaluates it, and records its version. The serving path fetches compatible features and runs inference within a response-time limit, with timeouts and a fallback. Model artifacts move toward serving; permitted prediction records and outcomes support evaluation and future training. I would test feature consistency and define a safe release and rollback process."
Likely follow-ups:
- "Why not train continuously?" Compare the value of fresh models with label delay, compute cost, and evaluation requirements. Online learning can be appropriate, but it still needs controls and monitoring.
- "Where does the first model come from, before there are logs?" Use suitable historical data when available. Otherwise begin with a rule-based baseline and collect permitted interaction data, while accounting for the selection bias that baseline creates.
TL;DR
| Two planes | Offline training (throughput, scheduled) and online serving (latency, always-on). |
| Two connections | Model registry forward; prediction/outcome logs back. Other schemas and configuration may also be shared. |
| Training pipeline | Data prep (point-in-time), then train, then evaluate against the current model, then register. |
| Serving path | Features, then inference, then response, under timeouts with a fallback; log the permitted data needed for evaluation. |
| Skew | One feature code path, point-in-time joins, and tests that compare training with serving. |
Reading Progress
0%
On This Page
Two Systems, Not One
The Training Pipeline, Piece by Piece
The Serving Path, Piece by Piece
Training/Serving Skew
In the Interview
TL;DR