0% completed
Introduction: ML Systems Foundations
On This Page
Know the Vocabulary
Part 1: The Model Is the Smallest Part
What Interviewers Expect
Part 2: What an ML System Is
The Definition
What an ML System Is Not
Part 3: Follow One Prediction
Part 4: The Lessons in This Module
Part 5: Where This Module Fits in the Framework
Part 6: What the Rest of the Course Reuses
Part 7: Common Mistakes This Module Prevents
Mistake 1: Drawing Only the Model
Mistake 2: Assuming Labels Already Exist
Mistake 3: Computing Features Twice
Mistake 4: Reaching for the Biggest Model First
Mistake 5: Shipping on an Offline Win Alone
Mistake 6: Forgetting the Fallback
FAQs
Key Takeaways
Check Your Understanding
Know the Vocabulary
This lesson uses the terms below many times. Read them once now, and come back here when a later section needs one.
- Model: A function whose behavior is learned from data instead of written by a developer. It takes an input and returns an output, such as a score or an estimate.
- Prediction: The output a model returns for one input, such as "this delivery will take 32 minutes."
- Feature: One input value the model uses, such as the restaurant's average prep time or the distance to the customer.
- Label: The answer the model should learn for one past example, such as the delivery time that really happened. A model learns from examples that pair features with labels.
- Training: The process that builds a model from past examples. It usually runs in background jobs.
- Serving: Delivering a trained model's predictions to users. Running a model on an input to get a prediction is called inference.
- Online inference: Running the model when a request arrives, while the user waits for the result.
- Batch inference: Running the model ahead of time in a background job, and storing the results so a request only looks them up.
- Offline plane: The part of the system that works in the background on historical data, such as training jobs.
- Online plane: The part of the system that answers live requests while a user waits.
- Feature store: A system that keeps feature values, with a full history for training and the latest values for fast lookups during serving.
- Model registry: A store that keeps every approved model version, with its quality results and the data and code that produced it.
- Training-serving skew: A difference between the features a model saw in training and the features it sees in serving.
- Drift: A change in the incoming data, or in how that data relates to the outcome, that can make a model less accurate over time.
- Baseline: The simplest approach that could work, such as a historical average. Every new model must beat it.
- Embedding: A list of numbers that captures what an item or a user is about. Items the model treats as similar get similar numbers.
Part 1: The Model Is the Smallest Part
The Module 1 capstone looked at a food-delivery company and listed several AI features it could build. One of them was delivery-time prediction: when a customer opens checkout, the app shows how long the delivery will take. The capstone chose a different feature to scope. This module picks up delivery-time prediction and turns it into a working design.
Start with one number: the estimate must appear within about 100 milliseconds (ms). Now look at what must happen inside that limit. Running a small model often takes only 1 to 10 ms. Fetching the model's inputs can take longer than the model itself. Routing, logging, and the network add more time.
So the model is one small box in the request. And the request is only half of the system. Behind it sit the data that taught the model and the jobs that retrain it. Behind it also sit the checks that decide whether a new version is safe to ship.
What Interviewers Expect
Many candidates draw one box labeled "model" and stop there. Interviewers then ask the questions this module answers. Where do the labels come from? How are the features computed, and are they the same in training and serving? What happens when the model is slow? How do you know the model is still accurate next month?
These questions are not about model math. They are about the system around the model. A software engineer who knows that system can design many AI features, even without training models every day.
This lesson gives you a map of that system. The ten lessons after it fill in each part of the map.
Part 2: What an ML System Is
The Definition
An ML system is everything needed to produce a model's predictions, deliver them to users, and keep them accurate over time. It has three parts.
-
The offline plane builds the model. It collects data and labels, trains a candidate model, evaluates it, and stores approved versions in the model registry. It runs on a schedule or when triggered, and it cares more about processing large amounts of data than about quick responses.
-
The online plane uses the model. It receives a request, looks up the features, runs the model, and returns a prediction. It runs all the time, and it cares about speed and availability, because a user is waiting.
-
The feedback loop keeps the model accurate. Every prediction is logged with the features it used. Later, the real outcome arrives and becomes a new label. Monitoring watches the service, the inputs, and the predictions right away. It compares predictions with outcomes once the outcomes arrive, and the next training run learns from them.
Three things link the planes. The model registry sends a versioned model from the offline plane to the online plane. The logs send predictions and outcomes back the other way. The feature store gives both planes the same feature definitions, with history for training and fresh values for serving.
The diagram below shows the three parts and how they connect.
Not every prediction has to run while the user waits. With batch inference, a background job computes predictions ahead of time, and the online plane only looks them up. With online inference, the model runs when the request arrives, as the diagram shows. Delivery-time prediction needs online inference, as Part 3 explains.
What an ML System Is Not
An ML system is not just a model file. A model with no data pipeline, no serving path, and no monitoring cannot run in production for long.
An ML system is not built once. Its data keeps changing, so its quality can change too, even when no code changes. The feedback loop is part of the design, not an extra.
An ML system is not only for ML specialists. Most of its parts are familiar to any backend engineer: scheduled jobs, key-value stores, stateless services, load balancers, logs, and dashboards. What is new is how they fit together around a model.
Part 3: Follow One Prediction
The fastest way to understand the map is to follow one prediction through it. The steps below use the delivery-time example. The times are illustrative.
Step 1: Checkout opens. At 6:02 PM, a customer opens checkout. The app sends a request to the prediction service, which must answer within about 100 ms.
Step 2: Features are fetched. The service reads the restaurant's average prep time, the number of orders in its kitchen right now, and the route distance. Some of these values were computed hours ago, some seconds ago, and one is computed now. Features and Feature Stores explains how each kind is kept fresh.
Step 3: The model predicts. The model server runs the model and returns 32 minutes. The app may add a small safety buffer before it shows the number. Because the route and the kitchen queue exist only at request time, this prediction must run online. Batch vs Online Inference gives the test for that choice.
Step 4: The prediction is logged. The service records the prediction together with the exact features it used. Without this record, nobody can later check what the model saw.
Step 5: The outcome arrives. At 6:40 PM, the order is delivered. The real time, 38 minutes, becomes a label. The prediction was 6 minutes too short. Where Training Data Comes From covers labels like this one, including labels that take weeks to arrive.
Step 6: Training runs. That night, a training job joins the logged predictions with their labels. It uses only the feature values that existed at prediction time, so the model never learns from information that arrived later. It trains a candidate model and compares it with the current one.
Step 7: The new model ships safely. If the candidate is better, it moves through careful release steps. The Model Serving Basics lesson describes them. First the new model runs in shadow mode: it scores live requests next to the old model, but nobody sees its results. Then a canary release sends it a small share of traffic, and then more.
The diagram below shows the same seven steps. In the top row, each time after 6:02 PM counts from the moment the request arrives.
Notice that the user saw only the result of Steps 1 to 3. Steps 4 to 7 happened out of sight, and most of them happened after the user had left the screen. Yet those hidden steps decide whether the next estimate is better or worse.
Part 4: The Lessons in This Module
The ten lessons after this one follow the map in order. The first three build the model, the next three serve it, the next three keep it working, and the capstone puts it all together.
| Question the lesson answers | Lesson |
|---|---|
| Why are training and serving two separate systems, and what goes wrong when they disagree? | Training vs Serving |
| Where do labels come from, and what makes training data trustworthy? | Where Training Data Comes From |
| Which inputs does the model need, and how do they stay fresh and consistent? | Features and Feature Stores |
| Should a prediction be computed ahead of time or when the request arrives? | Batch vs Online Inference |
| How do you find a few hundred good candidates out of 100 million items? | Embeddings and Two-Tower Retrieval |
| How does a model run as a service, and how do new versions ship safely? | Model Serving Basics |
| Which number tells you that a model is good before it reaches users? | Offline Evaluation: Choosing the Metric |
| How do you notice when a model gets worse, and when should you retrain it? | Monitoring, Drift, and Retraining |
| How do you test whether a new model is better for real users? | Online Experiments and A/B Testing |
| How do all these parts form one design? | Capstone: An ML Platform for ETA Prediction |
Part 5: Where This Module Fits in the Framework
The framework from Module 1 has seven steps. This module supplies the technical content for most of them. The table shows which lessons support which step.
| Framework step | What this module adds | Lessons |
|---|---|---|
| 2. Define success | Offline metrics, and the business metric an experiment decides on | Offline Evaluation; Online Experiments |
| 3. Map the data | Label sources, label delay, features, and freshness | Where Training Data Comes From; Features and Feature Stores |
| 4. Pick the model | A baseline first, and retrieval for very large catalogs | Training vs Serving; Offline Evaluation; Embeddings and Two-Tower Retrieval |
| 5. Design the serving path | Batch or online inference, the latency budget, and fallbacks | Batch vs Online Inference; Model Serving Basics |
| 6. Keep it working | Monitoring, retraining, and safe rollout | Monitoring, Drift, and Retraining; Model Serving Basics |
| 7. Count the costs | What serving a small model costs, compared with its data pipelines | Model Serving Basics; the capstone |
Step 1, framing the problem, stays with Module 1. Everything after it gets deeper here.
Part 6: What the Rest of the Course Reuses
This module uses classic, predictive models: models that score, rank, or estimate, such as a fraud score or a delivery time. Later in the course come large language models (LLMs) and agents. It is tempting to skip ahead to them. That would be a mistake, because the rest of the course reuses this module constantly.
The classic design questions use all of it. Lessons such as Design a Recommendation System and Design Fraud Detection are built from these parts. Between them, they use labels, features, retrieval with embeddings, offline metrics, monitoring, and A/B tests. They expect you to know features, batch and online inference, evaluation, and monitoring before you start.
The LLM lessons reuse the serving and evaluation ideas. Running a large language model is still a serving problem with a latency budget, a fallback, and a safe rollout. Evaluating it still needs the right metrics, a comparison with the current version, and an online check, as Evaluating LLM Systems shows. The numbers change, though. A small model can be cheap to run for each prediction. A large language model can cost far more per request, so cost becomes a main design concern.
Retrieval for LLMs builds on the same embedding idea. The text embeddings in Embeddings and Vector Databases are trained differently. But the idea of turning things into embeddings and searching for the nearest ones starts in this module.
Part 7: Common Mistakes This Module Prevents
Mistake 1: Drawing Only the Model
A design with one "model" box has no labels, no features, no monitoring, and no way to improve. Draw the offline plane, the online plane, and the loop from the start, even as three simple boxes.
Mistake 2: Assuming Labels Already Exist
Labels must come from somewhere: an outcome, a user action, or a human reviewer. Each source has a cost and a delay. A fraud label may take weeks to arrive, which changes how fast you can measure quality.
Mistake 3: Computing Features Twice
If training computes a feature one way and serving computes it another way, the model sees different inputs in production. Quality drops, and no error appears. This is training-serving skew, and shared feature definitions help prevent it.
Mistake 4: Reaching for the Biggest Model First
A historical average or a simple tree model is often the right starting point. It is cheap, it usually runs well on ordinary CPUs, and it gives every later model a number to beat.
Mistake 5: Shipping on an Offline Win Alone
A model can score better on old data and still make the product worse. Offline results decide which models are worth testing. A careful rollout, and an A/B test where it is practical, decides which model ships.
Mistake 6: Forgetting the Fallback
The model will sometimes be slow or unavailable. Decide in advance what the user sees instead, such as the previous model, a cached estimate, or a simple average.
FAQs
Q: Do I need to know the math behind ML models for this module?
No. This module treats a model as a component with inputs, outputs, a response time, and a cost. You need to know what each part of the system does and why, not how to derive a training algorithm.
Q: My interview will be about LLMs. Should I skip this module?
No. The later modules on LLMs and agents reuse this module's ideas: serving, evaluation, monitoring, and safe rollout. Part 6 lists the main connections.
Q: Should I go deep on every topic in an interview?
No. Cover every part of the map briefly, then go deeper where the interviewer asks. A short answer that names the labels, features, serving path, monitoring, and rollout usually beats a long answer about one part.
Q: Should I read the lessons in order?
Yes, if you can. Each lesson builds on the ones before it, and the capstone uses all of them.
Key Takeaways
- An ML system is much more than its model. Within a 100 ms limit, the model may use only a few milliseconds. Data, features, logs, and checks make up the rest of the work.
- The system has three parts. The offline plane builds the model, the online plane serves it, and the feedback loop keeps it accurate.
- Three things link the planes. The model registry moves versioned models forward, the logs carry outcomes back, and the feature store gives both planes the same feature definitions.
- Every prediction should be logged with its features. Its real outcome later becomes a label that teaches the next model.
- This module supplies the technical content for framework steps 2 to 7, and the later modules reuse it.
- In a system design interview, draw both planes and the loop at the start. Then name where the labels come from, how features stay consistent, what the fallback is, and how a new model ships safely.
The next lesson, Training vs Serving, looks closely at the two planes and at the silent failure that happens when they disagree.
Check Your Understanding
1. A delivery-time model takes 5 ms to run, but the whole request takes 40 ms. Where do the other 35 ms go?
<details> <summary>Show answer</summary>Mostly to everything around the model. Fetching features, routing, logging, and the network all take time. Fetching features often takes longer than running a small model.
</details>2. What are the three parts of an ML system, and what does each one do?
<details> <summary>Show answer</summary>The offline plane, the online plane, and the feedback loop. The offline plane builds and evaluates the model. The online plane answers live requests. The feedback loop logs predictions, collects real outcomes, and uses them to monitor and retrain the model.
</details>3. Why must a prediction be logged together with the features it used?
<details> <summary>Show answer</summary>So that it can be checked and learned from later. When the real outcome arrives, the log shows exactly what the model saw. That record helps find training-serving skew and turns the outcome into a correct training example.
</details>4. A new model scores better than the current one on last month's data. Is it ready to ship to all users?
<details> <summary>Show answer</summary>No. A better offline score makes it a candidate, not a winner. It still needs a careful rollout, and an A/B test where practical, to show that it improves the product for real users.
</details>Reading Progress
0%
On This Page
Know the Vocabulary
Part 1: The Model Is the Smallest Part
What Interviewers Expect
Part 2: What an ML System Is
The Definition
What an ML System Is Not
Part 3: Follow One Prediction
Part 4: The Lessons in This Module
Part 5: Where This Module Fits in the Framework
Part 6: What the Rest of the Course Reuses
Part 7: Common Mistakes This Module Prevents
Mistake 1: Drawing Only the Model
Mistake 2: Assuming Labels Already Exist
Mistake 3: Computing Features Twice
Mistake 4: Reaching for the Biggest Model First
Mistake 5: Shipping on an Offline Win Alone
Mistake 6: Forgetting the Fallback
FAQs
Key Takeaways
Check Your Understanding