Training vs Serving

Two Systems, Not One

When an interviewer says "design a fraud detection system," they are asking for two systems, not one.

The two share a single model, but that is where the overlap ends. The training system turns historical data into a model file. The serving system loads that file and answers live requests.

They run on different hardware. They run on different schedules. They fail with different consequences, and usually different teams are on call for each.

Training can run on spot instances, which are cheap machines the cloud provider can take back at any time. A training job is a batch job, so it can simply restart. Serving cannot, because it has to stay up.

Drawing these as one box is the most common structural error in whiteboard ML designs.

Draw them from the start as two separate groups of components: the offline plane, which trains, and the online plane, which serves. Connect them with exactly two artifacts. The model flows from training to serving. Logs flow from serving back to training.

The Training Pipeline, Piece by Piece

A production training pipeline is a scheduled workflow, usually built on Airflow or a similar tool. It has four stages. Naming them in order is usually all the detail an SWE loop wants.

1. Data preparation. Pull raw events from the warehouse or lake. Join them into training examples.

Each example pairs features as they looked at prediction time with labels that arrived later. This stage is most of the code, and it is where most of the bugs are.

2. Training. Fit the model to the data.

Most tabular business problems, like fraud, ETA, and churn, use a gradient-boosted tree. That is a model built from many small decision trees that each correct the last one's mistakes. A modest neural model is the other common choice.

Training takes minutes to hours, not weeks. Only large deep models need distributed multi-GPU training. Two terms cover that case. Data parallelism splits the batch across GPUs, which is the common case. Model parallelism splits the model itself, which is needed only for the largest models. In an interview, say which of the two you mean.

3. Evaluation. Score the candidate model on held-out data, meaning data the model never saw while training.

Compare it against the model currently in production, on the metrics from your requirements step. No improvement, no promotion.

4. Registration. Push the approved model into a model registry.

Think of the registry as a system of record. It stores the model file, its version, its metrics, and its lineage, meaning what data and code produced it.

The registry is the handoff point between the planes. It is also what makes rollback possible: point back at an older version in minutes.

💡 Say the retraining schedule out loud and justify it: "Fraud patterns shift fast, so retrain daily; the ETA model can retrain weekly." A schedule with a reason is a small sentence that sounds like production experience.

The Serving Path, Piece by Piece

The online plane is a low-latency read path. A request comes in, and something must answer fast.

A request arrives. Features are fetched or computed, which the next lesson covers. The model runs. The prediction returns, and the whole exchange is logged.

The full path commonly gets a budget of 10 to 100 ms for predictive systems. The model itself may take only single-digit milliseconds. Feature fetching, not inference, is usually the real latency problem.

Serving also has to meet the classic system design requirements, and these get forgotten once there is a model in the design:

• Timeouts.
• Fallbacks, meaning a default score or a cached prediction returned when the model times out.
• Horizontal scaling behind a load balancer.
• Versioned rollout.

One more question is really about the product, not the model. Does a fraud model fail closed, blocking everything when unsure, or fail open, allowing everything through? Name which one, and say why.

Training/Serving Skew

This is the classic silent failure, and a favorite interview question.

Skew is any difference between what the model saw in training and what it sees in serving. The model trained on features computed one way. In production, it meets features computed a slightly different way.

Quality drops, and no error appears anywhere. Nothing crashes, no alert fires, no page goes out. The numbers just get worse.

There are three common sources.

1. Different code paths. Features get computed in Python or Spark for training, then reimplemented in Java for serving. The two versions drift apart.

A different null default. A different rounding. A different time zone. The fix is to compute features once, in one system, used by both planes. That is what a feature store is for.

2. Time travel, also called label leakage. Training examples accidentally include information from after the prediction moment.

Think of "restaurant's average delivery time today", computed over the whole day, including the delivery being predicted. The model looks brilliant offline and mediocre online, because that future information does not exist yet at serving time.

The fix is point-in-time-correct joins in data preparation. Each example then sees only data that existed at that exact moment.

3. Distribution shift between training data and live traffic. You trained on last month's traffic and you are serving this month's. This is not a bug, but it has the same symptom. Monitoring, Drift, and Retraining treats it.

💡 "How would you make sure training and serving features match?" is a standard follow-up. The strong answer names one shared computation path and point-in-time joins. Then it adds logging the features actually used at serving time and training on those logs, which makes skew structurally impossible for logged features.

In the Interview

The 30-second answer (to "walk me through the ML architecture"): "Two planes. Offline: a scheduled pipeline builds point-in-time-correct training data from the warehouse, trains, evaluates against the model currently in production, and registers the model. Online: requests fetch features, run inference inside a tight latency budget behind timeouts and a fallback, and every prediction is logged. The planes connect in exactly two places: the registry pushes models forward, and serving logs flow back to become the next training set. The classic failure between them is skew, so features are computed by one shared path and I train on logged serving features."

Likely follow-ups:

1. "Why not train continuously?" Cost and safety. Each retrain needs evaluation gates before promotion, and most business data does not shift hourly. Streaming or online learning exists for the few domains that do, like ad ranking at extreme scale. Say it is the exception, not the default.
2. "Where does the first model come from, before there are logs?" Bootstrap from historical data if it exists. Otherwise, ship a heuristic, meaning rules or averages, to generate traffic and labels. Then train v1 on those logs. Naming the cold-start plan shows senior judgment.

TL;DR