Training vs Serving

2. The Training Pipeline, Piece by Piece

A production training pipeline is a scheduled workflow, usually built on Airflow or a similar tool. It has four stages. Naming them in order is usually all the detail an SWE loop wants:

1. Data preparation. Pull raw events from the warehouse or lake. Join them into training examples. Each example pairs features as they looked at prediction time with labels that arrived later. Here is the tricky part: this stage is most of the code, and it is where most of the bugs hide.
2. Training. Fit the model to the data. Most tabular business problems (fraud, ETA, churn) use a gradient-boosted tree, a model built from many small decision trees that each correct the last one's mistakes, or a modest neural model. Training takes minutes to hours, not weeks. Only large deep models need distributed multi-GPU training. If that comes up, two terms cover it: data parallelism splits the batch across GPUs (the common case), and model parallelism splits the model itself (frontier-scale training only). In an interview, say which regime you are in.
3. Evaluation. Score the candidate model on held-out data, meaning data the model never saw while training. Compare it against the current production model on the metrics from your requirements step. No improvement, no promotion.
4. Registration. Push the approved model into a model registry. Think of it as a system of record: it stores the model file, its version, its metrics, and its lineage, meaning what data and code produced it. The registry is the handoff point between the planes. It is also what makes rollback possible: point back at an older version in minutes.

💡 Say the cadence out loud and justify it: "Fraud patterns shift fast, so retrain daily; the ETA model can retrain weekly." Cadence-with-reason is a small sentence that reads as production experience.

3. The Serving Path, Piece by Piece

The online plane is a low-latency read path: a request comes in, and something must answer fast. A request arrives. Features are fetched or computed (the next lesson covers how). The model runs. The prediction returns, and the whole exchange is logged. The full path commonly gets a budget of 10 to 100 ms for predictive systems. The model itself may take only single-digit milliseconds. Feature fetching, not inference, is usually the real latency problem.

Serving also has to meet the classic system design requirements. Candidates forget these once a model enters the picture. Timeouts. Fallbacks, meaning a default score or a cached prediction fires when the model times out. Horizontal scaling behind a load balancer. Versioned rollout. One more question is really about the product, not the model: does a fraud model fail closed (block everything when unsure) or fail open (allow everything through)? Name which one, and say why.

4. Training/Serving Skew

This is the classic silent killer, and a favorite interview probe. Skew is any difference between what the model saw in training and what it sees in serving. The model trained on features computed one way. In production, it meets features computed a slightly different way. Here is the tricky part: quality drops, and no error shows up anywhere. Nothing crashes, nothing pages anyone, the numbers just quietly get worse.

The three common sources:

1. Different code paths. Features get computed in Python/Spark for training, then reimplemented in Java for serving. The two versions drift apart. A different null default. A different rounding. A different time zone. The cure: compute features once, in one system, consumed by both planes. That is the feature-store promise.
2. Time travel (label leakage). Training examples accidentally include information from after the prediction moment. Think of "restaurant's average delivery time today" computed over the whole day, including the delivery being predicted. The model looks brilliant offline and mediocre online, because that future information does not exist yet at serving time. The cure: point-in-time-correct joins in data prep, meaning each example only ever sees data that existed at that exact moment.
3. Distribution shift between training data and live traffic. You trained on last month's traffic and you are serving this month's. Not a bug, but the same symptom. Monitoring, Drift, and Retraining treats it.

💡 "How would you make sure training and serving features match?" is a standard follow-up. The strong answer names one shared computation path, point-in-time joins, plus logging the features actually used at serving time and training on those logs, which makes skew structurally impossible for logged features.

5. In the Interview

The 30-second answer (to "walk me through the ML architecture"): "Two planes. Offline: a scheduled pipeline builds point-in-time-correct training data from the warehouse, trains, evaluates against the incumbent, and registers the model. Online: requests fetch features, run inference inside a tight latency budget behind timeouts and a fallback, and every prediction is logged. The planes connect in exactly two places: the registry pushes models forward, and serving logs flow back to become the next training set. The classic failure between them is skew, so features are computed by one shared path and I train on logged serving features."

Likely follow-ups:

1. "Why not train continuously?" Cost and safety. Each retrain needs evaluation gates before promotion, and most business data does not shift hourly. Streaming or online learning exists for the few domains that do, like ad ranking at extreme scale. Say it is the exception, not the default.
2. "Where does the first model come from, before there are logs?" Bootstrap from historical data if it exists. Otherwise, ship a heuristic (rules, averages) to generate traffic and labels. Then train v1 on those logs. Naming the cold-start plan is a senior mark.

6. TL;DR