0% completed
Introduction: Classic ML Design Questions
On This Page
Know the Vocabulary
Part 1: The Questions Repeat
What Interviewers Expect
Part 2: Two Shapes Behind Six Questions
The Definition
The Ranking Shape: A Funnel
The Judging Shape: Score, Then Decide
What the Shapes Are Not
Part 3: Follow One Home Page Request
Part 4: What Each Design Adds
Part 5: The Lessons in This Module
Part 6: How Each Design Lesson Follows the Framework
Part 7: What the Rest of the Course Reuses
Part 8: Common Mistakes This Module Prevents
Mistake 1: Designing From Scratch Every Time
Mistake 2: Running One Model Over Every Item
Mistake 3: Judging Every Stage by One Number
Mistake 4: Treating Clicks as the Truth
Mistake 5: Turning Every Decision Into Yes or No
Mistake 6: Quoting Accuracy on Rare Events
FAQs
Key Takeaways
Check Your Understanding
Know the Vocabulary
This lesson uses the terms below many times. Read them once now, and come back here when a later section needs one.
- Predictive system: A system whose model returns a score, a rank, or a label, and the product acts on that output. Every design in this module is a predictive system.
- Candidate: One item the system might show or act on, such as a video, a post, a product, or an ad.
- Funnel: A sequence of stages that cuts down the number of candidates. Each stage handles fewer items, so it can spend more work on each one.
- Candidate generation: The first funnel stage, also called retrieval. It quickly pulls a few hundred possibly good candidates out of millions.
- Ranking: The funnel stage that scores each remaining candidate with a richer model and puts the candidates in order.
- Rule layer: The last funnel stage, which applies business rules, such as variety, freshness, and content policy, to the ranked list.
- Recall: Of all the items that would be good for a request, the share that a stage keeps.
- Precision at the top: Of the first few items shown, the share that are truly good.
- Embedding: A list of numbers that captures what an item or a user is about. Items the model treats as similar get similar numbers.
- Approximate nearest neighbor (ANN) index: A search structure that quickly finds the embeddings closest to a given one. It is fast because it skips most comparisons, so it can miss a few good matches.
- Impression: One item shown to one user, one time.
- Label: The answer the model should learn for one past example, such as "the user watched this video."
- Threshold: A score cutoff where the action changes, so items above it get one action and items below it get another.
- Position bias: Items shown near the top get more clicks partly because of where they appear, not only because they are better.
- Adversary: A person who studies the system and changes their behavior to get past it, such as a fraudster.
- Calibration: How well predicted probabilities match real rates. If a model gives many similar ad impressions a 2 percent click chance, about 2 percent of those impressions should get a click.
Part 1: The Questions Repeat
Interviewers ask a small set of classic ML design questions again and again. The common ones cover recommendations, news feeds, search for an online store, ad click prediction, fraud detection, and harmful content.
Each of these is a predictive system, in which a model returns a score or a label and the product acts on it.
Go back to the food-delivery company from Module 2, whose delivery-time prediction was only one of its AI projects. The same company could face almost every question in this module.
- A recommendation question asks which restaurants the home screen should show first.
- A search question asks what should appear when a customer types the word "pizza" into the app.
- An ad question asks which restaurants get the paid spots at the top of the list.
- A fraud question asks whether this order was paid with a stolen card.
- A harmful content question asks whether this restaurant review is abusive.
The products differ, but the designs share most of their parts. So you do not need a new design for every question. You need a few known designs and the skill to adapt them.
What Interviewers Expect
An interviewer does not expect you to invent a new architecture in 45 minutes. They expect you to recognize the question quickly and adapt a known design with care.
Weak answers start with a model name. Strong answers start with the scale and the time limit, because those two numbers limit how much work each item can get. Then they name what makes this question harder than the last one.
This lesson gives you the two shapes behind the six designs. It then shows what each design in this module adds.
Part 2: Two Shapes Behind Six Questions
The Definition
Every design in this module has one of two shapes. The shape depends on what arrives with each request.
-
Ranking questions choose a few items out of many. One request arrives, such as a user opening the home screen. The system must pick a short list, such as 20 items, out of thousands or millions. Recommendations, feeds, search, and ads have this shape.
-
Judging questions decide about one item. One event arrives, such as a payment or a new post. The system must score that one item and choose an action. Fraud detection and harmful content detection have this shape.
The Ranking Shape: A Funnel
A ranking question cannot run its best model on every item. For example, a video platform may hold 50 million videos. It may need to load the home page in about 200 milliseconds (ms). Running a detailed model 50 million times per request would far exceed that time limit and the budget.
So the system uses a funnel with three stages.
- Retrieve. Candidate generation uses cheap methods to cut millions of items down to a few hundred.
- Rank. A richer model scores those few hundred candidates and puts them in order.
- Apply the rules. The rule layer adds variety, freshness, and policy checks, and picks what fits on the screen.
Each stage is judged on its own job. Retrieval is judged on recall, because its job is to keep the good items. Ranking is judged on precision at the top, because users look mostly at the first few results.
The two kinds of failure are not equal. Suppose retrieval drops a good video. The ranker never sees it, and no later stage can bring it back. Now suppose retrieval keeps the video, but the ranker places it 40th. A better ranker can fix that.
The Judging Shape: Score, Then Decide
A judging question has no catalog to narrow down. It gets a stream of events, and each event needs a decision. A payment, for example, may need an answer within about 50 ms while the purchase waits.
The judging shape also has three parts.
- Cheap checks first. Rules and simple lookups handle clear cases fast. Examples are a card on a block list, or an image that matches a known harmful image.
- A model score. A model estimates the risk of this one item.
- An action policy. Thresholds turn the score into an action. A fraud system may approve a payment, ask for extra verification, hold it for human review, or block it. A review takes far longer than 50 ms, so the payment must wait in a pending state.
Harmful content detection stretches the first two parts into a longer chain. A fast model checks every post. A heavier model checks only the small share that looks risky. Human reviewers handle the hardest cases. This chain saves cost for the same reason the funnel does: each later step sees fewer items.
The diagram below shows both shapes side by side.
Notice that both shapes put cheap work first and expensive work last. The difference is the input: a ranking system starts from many items, while a judging system starts from one.
What the Shapes Are Not
The funnel is not one big model. Each stage can use a different method, and each stage is tested on its own job.
The rule layer is not hidden inside the ranker. Business rules stay separate, so a policy change does not always need a new model.
A judging system is not one yes-or-no cutoff. The model returns a score, and a separate policy, set by the cost of each kind of mistake, decides the action.
Neither shape is a fixed recipe. The numbers in this module are illustrative assumptions. When your interviewer gives different numbers, the design may change.
Part 3: Follow One Home Page Request
The fastest way to understand the funnel is to follow one request through it. The steps below use the video platform from the first design lesson. Assume 50 million videos, about 200 ms for the whole page, and about 20 videos on screen. The numbers and times are illustrative assumptions.
Step 1: The home screen opens. A user opens the video app. The app sends a request to the recommendation service, and the clock starts at 0 ms.
Step 2: Candidate generation pulls about 500 videos. Three cheap sources run at the same time. The first looks up the user's embedding and asks the ANN index for the nearest video embeddings. The second looks up videos often watched together with the user's recent videos. The third adds trending and new videos. The service merges the three lists and removes duplicates. By about 35 ms, about 500 candidates remain.
Step 3: The ranker orders them. The service fetches richer features for the 500 candidates. Examples are the user's watch history, each video's recent statistics, the device, and the hour. The ranker scores all 500 in one batched call and predicts engagement, such as a click or a long watch. By about 80 ms, the top 50 or so remain.
Step 4: Rules shape the page. The rule layer removes videos the user just saw. It stops one channel from filling the screen. It applies policy demotions and gives new videos a small boost. By about 85 ms, about 20 videos remain. The rest of the 200 ms goes to the network and to building the page.
Step 5: The impressions are logged. Each of the 20 videos shown is logged as an impression. The log records the features and the model versions used. Without this record, nobody can later train on what the user really saw.
Step 6: The outcomes arrive. A few minutes later, the user has watched two videos and skipped the rest. A watched video becomes a positive label. A skipped video is a weaker negative label, because the user may not have noticed it. Position bias also enters here: the video in the first slot got the most attention.
Step 7: Training runs. That night, training jobs join the impressions with their outcomes. They retrain the ranker and rebuild the video embeddings and the ANN index. The new versions serve the next day, but only after they pass offline checks and a careful release.
The diagram below shows the same seven steps. In the top row, each time counts from the moment the request arrives.
Notice that the ranker saw only about 500 of 50 million videos. That is 0.001 percent of the catalog. This is why the funnel can afford a rich model at all.
Notice also where the risks sit. A good video that Step 2 missed is gone for this request. A label from Step 6 carries the bias of the slot it appeared in. The design lessons return to both problems.
Part 4: What Each Design Adds
Each design in this module starts from one of the two shapes. Then it adds one new requirement that makes the question harder. The list below names that requirement for each lesson.
Recommendations add the funnel itself and two classic problems. That lesson builds the full funnel from retrieval to rules. It also covers cold start, which means the system has no history for a new user or a new item. And it shows how position bias misleads a model trained on raw clicks.
Feed ranking adds many goals at once. A post can earn a like, a comment, a share, or a long view. It can also earn a hide or a report. A common design uses one model to predict all of these events. A weighted formula combines them into one score, with the negative events, such as a hide or a report, subtracted.
Search ranking adds the query that the user types. The user types what they want, so a new first stage reads the query. It fixes spelling, adds synonyms, and pulls out details such as size or brand. Retrieval then runs two ways at once. Lexical search matches the exact words, and semantic search matches the meaning.
Ad click prediction adds a price. The predicted click chance feeds an auction that decides which ad appears. In a simplified auction, each ad's value is its bid multiplied by its predicted click chance. So the order of the scores is not enough. The probabilities themselves must be accurate, which is calibration. A model can order ads well and still give every ad the wrong probability. Ads also apply hard filters first: an ad that misses its targeting or has spent its budget never enters the auction.
Fraud detection adds an adversary who adapts to the system. Fraudsters change their methods to get past the system. Fraud is rare, so a model that approves everything can look more than 99 percent accurate and still catch nothing. Labels such as chargebacks can arrive weeks later. A blocked payment produces no chargeback, so the system never learns what would have happened if it had been approved.
Harmful content detection adds scale and harm in two directions. A missed post can expose many people to harm. A wrong removal can silence a legitimate user. Neither cost has a simple dollar value, so policy teams help set the thresholds. The decision becomes a ladder of actions, from allow, to reduce reach, to human review, to removal.
The capstone adds a goal with two sides. "People You May Know" suggests connections on a professional network. A good suggestion needs both members to want the connection, not only the one who sees it. Privacy rules must hold even when parts of the system fail.
Part 5: The Lessons in This Module
The seven lessons after this one follow the order above. Each design lesson defines its own new terms, so you can read it on its own. Reading them in order is still easier, because each one reuses parts of the one before.
| Question the lesson answers | Lesson |
|---|---|
| How do you pick 20 videos out of 50 million in about 200 ms? | Design a Recommendation System |
| How do you rank posts when likes, comments, hides, and reports all matter? | Design Feed Ranking |
| How does a typed query change retrieval, ranking, and evaluation? | Design Search Ranking |
| Why must click predictions be calibrated when they set prices? | Design Ad Click Prediction |
| How do you score payments when labels are late and fraudsters adapt? | Design Fraud Detection |
| How do you find rare harm across millions of posts without silencing good ones? | Design Harmful Content Detection |
| Can you combine the six designs to solve a new problem? | Capstone: People You May Know |
Part 6: How Each Design Lesson Follows the Framework
The framework from Module 1 has seven steps. Each design lesson in this module walks through those seven steps in order, one Part per step. It then shows how to give the answer in an interview, with likely follow-up questions. The capstone works differently: it gives you the question first and asks you to try it yourself.
The table shows how the same step looks in a ranking question and in a judging question. The examples come from the recommendation and fraud lessons.
| Framework step | Ranking example (recommendations) | Judging example (fraud) |
|---|---|---|
| 1. Frame the problem | A funnel over 50 million videos, within about 200 ms | One payment at a time, within about 50 ms, against an adversary |
| 2. Define success | Retention and watch time online; recall and ranking quality offline | The cost of missed fraud and of declined good customers |
| 3. Map the data | Impressions joined with clicks, which arrive within minutes | Chargebacks, which can arrive weeks later |
| 4. Pick the model | "Most popular plus recent" as the baseline | Hand-written rules plus a simple model |
| 5. Design the serving path | Retrieve, rank, apply the rules | Cheap checks, features, score, thresholds |
| 6. Keep it working | Correct for position bias; retrain the ranker daily | Watch score changes early, because labels come late |
| 7. Count the costs | Do heavy work ahead of time; limit the ranker to about 500 items | Missed fraud, lost sales, and reviewer hours |
Module 2 supplied the parts behind every row: labels, features, batch and online inference, evaluation, monitoring, and safe rollout. This module assumes you know them. If one feels unfamiliar, the Module 2 introduction shows where each part fits and which lesson teaches it.
Part 7: What the Rest of the Course Reuses
Later lessons build products around large language models. The model changes, but many of this module's parts return.
The funnel returns in retrieval for language models. A system first pulls many text passages with a cheap search. A slower, more accurate model then reranks a short list. Hybrid Search and Reranking uses this pattern. It also combines exact-word search with meaning-based search, as search ranking does here.
The ANN index returns for text. Embeddings and Vector Databases stores text embeddings in the same kind of index that serves recommendations in this module.
Checks around a model return in safety. Safety and Guardrails places checks before and after a language model, including a classifier that screens user input. The idea of layered checks with clear actions starts with fraud and harmful content here.
The six designs return as practice. The case studies run these designs at interview speed. System Design of Spotify-Style Music Discovery reuses the recommendation funnel. System Design of Harmful Content at Platform Scale reuses the risk-based chain of checks from the harmful content design.
Part 8: Common Mistakes This Module Prevents
Mistake 1: Designing From Scratch Every Time
A new question is often a known design with one change. Say which design it is based on, then spend your time on the change. For example, job search is search ranking with a second side: the employer must also want the job seeker.
Mistake 2: Running One Model Over Every Item
A model rich enough to rank well is usually far too expensive to run 50 million times per request. Start with the scale and the time limit, and let them force the funnel.
Mistake 3: Judging Every Stage by One Number
Retrieval is judged on recall, and ranking is judged on precision at the top. A strong ranker cannot fix a retrieval stage that drops good items.
Mistake 4: Treating Clicks as the Truth
Users click what they see, and they see the top slots most. A model trained on raw clicks can learn to repeat the old ranking. Name position bias before the interviewer asks.
Mistake 5: Turning Every Decision Into Yes or No
Fraud and harmful content need more than two outcomes. Middle actions, such as extra verification, reduced reach, or human review, handle the uncertain cases.
Mistake 6: Quoting Accuracy on Rare Events
For example, when fraud is 0.2 percent of payments, approving everything is 99.8 percent accurate. Quote precision and recall at the thresholds you chose, or the expected cost of errors. Here, precision is the share of flagged payments that are really fraud. Recall is the share of all fraud that the system catches.
FAQs
Q: Do I need to study all six designs before an interview?
Yes, if you can. They share most of their parts, so each new one is faster to learn than the last. Interviewers often change a question slightly, and knowing all six helps you spot which design a new question is based on.
Q: Does every classic ML question use the funnel?
No. The funnel fits questions that choose a few items out of many, such as recommendations, feeds, search, and ads. Fraud detection scores one payment at a time. Harmful content detection uses a chain of checks that grow more expensive, which follows the same cost logic as the funnel.
Q: Do I need to know the newest model architectures?
No. These lessons treat each model as a component with inputs, outputs, a time limit, and a cost. Interviewers grade the system around the model more than the model's internal design.
Q: Should I memorize the numbers in these lessons?
No. Numbers such as 50 million videos or a 200 ms page are illustrative assumptions. Learn how the numbers shape the design, then work with the numbers your interviewer gives you.
Key Takeaways
- Classic ML design questions repeat, and they share most of their parts. Recommendations, feeds, search, ads, fraud, and harmful content cover most of the questions you will meet.
- Every design in this module has one of two shapes. Ranking questions choose a few items out of many with a funnel. Judging questions score one item and turn the score into an action.
- The funnel has three stages, each judged on its own job. Retrieval keeps the good items and is judged on recall. Ranking orders them and is judged on precision at the top. The rule layer applies business rules.
- An item that retrieval misses is gone. No later stage can bring it back, so retrieval quality matters as much as ranking quality.
- Each design adds one new requirement: many goals, a query, a price, an adversary, or harm in two directions.
- In a system design interview, say which known design the question is based on. Then give the scale and the time limit, name the shape, and spend most of your time on what is new.
The next lesson, Design a Recommendation System, builds the full funnel for the video platform from Part 3.
Check Your Understanding
1. Which shape fits a question that asks for 10 products out of 100 million, and why?
<details> <summary>Show answer</summary>The ranking shape, which uses a funnel. One request must choose a few items out of many. Running a rich model on 100 million products per request would exceed the time limit and the budget. So cheap retrieval narrows the set, a ranker orders it, and rules shape the final list.
</details>2. Retrieval keeps 500 candidates, but the item the user would love is not among them. Can a better ranker fix this?
<details> <summary>Show answer</summary>No. The ranker only sees what retrieval passes on. An item dropped at retrieval is gone for that request. That is why retrieval is judged on recall.
</details>3. Why is fraud detection a judging question and not a ranking question?
<details> <summary>Show answer</summary>Because each event is one item that needs a decision. There is no catalog to narrow down. The system scores one payment, then thresholds turn the score into an action, such as approve, verify, review, or block.
</details>4. Why is it a problem if an ad model orders ads correctly but predicts click chances twice as high as the real rates?
<details> <summary>Show answer</summary>Because the auction uses the probabilities, not only the order. Each ad's value is its bid multiplied by its predicted click chance. A model that is twice too high doubles what every opportunity seems to be worth, which can distort prices, spend, and reports to advertisers. If the error differs between groups of ads, it can also change which ad wins. Ad systems need both good order and calibration.
</details>Reading Progress
0%
On This Page
Know the Vocabulary
Part 1: The Questions Repeat
What Interviewers Expect
Part 2: Two Shapes Behind Six Questions
The Definition
The Ranking Shape: A Funnel
The Judging Shape: Score, Then Decide
What the Shapes Are Not
Part 3: Follow One Home Page Request
Part 4: What Each Design Adds
Part 5: The Lessons in This Module
Part 6: How Each Design Lesson Follows the Framework
Part 7: What the Rest of the Course Reuses
Part 8: Common Mistakes This Module Prevents
Mistake 1: Designing From Scratch Every Time
Mistake 2: Running One Model Over Every Item
Mistake 3: Judging Every Stage by One Number
Mistake 4: Treating Clicks as the Truth
Mistake 5: Turning Every Decision Into Yes or No
Mistake 6: Quoting Accuracy on Rare Events
FAQs
Key Takeaways
Check Your Understanding