Design a Recommendation System

Know the Vocabulary

This lesson uses the terms below many times. Read them once now, and come back here when a later section needs one.

• Recommendation system: A system that picks items for one user from a large catalog, such as the videos on a homepage.
• Catalog: Every item the system could show. In this lesson, the catalog is 50 million videos.
• Impression: One video shown to one user, in one slot on the screen, at one time.
• Engagement: What a user does with an impression, such as clicking it and watching for a while.
• Funnel: A series of stages that turns a very large set of items into a short list. Each stage keeps fewer items and spends more effort on each one.
• Candidate generation: The first stage of the funnel. It quickly pulls a few hundred possible videos out of the whole catalog. It is also called retrieval.
• Ranking: The second stage. A heavier model scores each candidate and sorts them.
• Re-ranking: The last stage. Rules adjust the final list for reasons such as variety, freshness, and policy.
• Embedding: A short list of numbers that describes a user or an item. Items the model treats as similar get similar numbers.
• Two-tower model: A model with two parts. One part turns a user into an embedding, and the other turns an item into an embedding.
• ANN index: ANN stands for approximate nearest neighbor. An ANN index is a search structure that finds the closest embeddings quickly, but not always the exact closest ones.
• Co-occurrence: How often two items are used by the same people within a short time. "People who watched this also watched that" is co-occurrence.
• Recall@K: Of the videos a user really engaged with, the share that appears among the top K candidates.
• AUC: A score that measures ordering. It is the chance that the model scores a random positive example above a random negative one. A score of 0.5 is chance, and 1.0 is a perfect order.
• Cold start: The problem of recommending for a new user, or recommending a new item, when there is no history yet.
• Position bias: The effect of a slot on engagement. Videos near the top get more clicks partly because more people look there.

Part 1: Frame the Problem

Start With the Numbers

An interviewer says, "Design a recommendation system for a video platform." Do not start with a model. Start with the scale, because the scale decides the shape of the design.

Ask for three numbers, or state them as assumptions. The numbers below are illustrative assumptions for this lesson.

• Items: 50 million videos in the catalog.
• Users: 100 million daily users.
• Latency budget: the homepage must load its recommendations within about 200 ms.

The screen shows about 20 videos, so the system must choose 20 out of 50 million for each user in about 200 ms.

Then name the known system and the scope. "This is a recommendation system, so I will use a retrieve-then-rank funnel. I will design the homepage feed for signed-in users. Search, ads, and creator tools are out of scope." Then check that the scope matches what the interviewer wants.

Why One Model Cannot Do It

Suppose one accurate model scored every video for every request. Even at one microsecond per video, 50 million videos take 50 seconds, against a budget of 200 ms.

A model that is cheap enough to score 50 million videos is too simple to rank them well. A model that ranks well is too slow to score them all. No single model is both good enough and cheap enough at this scale. So the design splits the work into stages.

The Funnel

The funnel has three stages. The counts are examples.

1. Candidate generation narrows 50 million videos to about 500. It runs three cheap sources at the same time, which Part 4 explains.
2. Ranking scores those 500 with a heavier model and keeps the best 50, in order.
3. Re-ranking applies rules to those 50 and picks the 20 that appear on screen.

The funnel works like hiring for one job. A recruiter skims thousands of resumes quickly and keeps a few hundred. That is candidate generation. A hiring team interviews a short list in depth and orders them. That is ranking. The final choice also follows company rules, such as budget and team balance. That is re-ranking. In both cases, fast checks come first and careful checks are saved for the few that remain.

The diagram below shows the three stages, the count after each one, and the metric that judges each stage.

Notice one rule of the funnel. A video that candidate generation drops can never come back. Ranking can only sort what it receives. The module introduction explains why a dropped video costs more than a video that is ranked too low.

Part 2: Define Success

Online Truth and Offline Proxies

The business cares about long-term outcomes. Two common ones are the daily return rate and watch time per session. The daily return rate is the share of users who come back the next day. It measures retention, which means users keep using the product.

These outcomes are the online truth, but a model cannot train on them directly. Retention is slow, and it does not belong to any one impression. Nobody can say which video made a user return tomorrow.

So the model trains on a faster signal: engagement with each impression. Did the user click the video and watch it for a while? That signal is a proxy, meaning a stand-in for the real goal. It is useful because it arrives within minutes. It is risky because it can rise while the real goal falls, as Part 6 shows.

One Metric per Stage

Each stage has a different job, so each stage gets its own offline metric.

Candidate generation is judged on recall. Take a later week of logs that training never saw. For each user, list the videos they really watched. Recall@K asks what share of those videos appears among the top K candidates, such as the top 500. Low recall means good videos were dropped before ranking.

Ranking is judged on its ordering, and most of all on precision near the top. AUC checks the overall ordering. An AUC of 0.5 means the order is no better than chance, and 1.0 means every positive is above every negative. Precision near the top asks a narrower question. Of the 20 videos at the top, how many did the user engage with? The user sees only that top part, so it matters most.

Offline Evaluation: Choosing the Metric explains these metrics in more depth.

What Offline Numbers Decide

Good offline numbers do not launch a model. They earn the model a live A/B test. An A/B test shows the new model to a random group of users and the current model to another group. The test then compares the online metrics, such as daily return rate and watch time per session.

Name guardrail metrics too. A guardrail metric is a number that must not get worse, such as the rate of "not interested" clicks or page latency. Online Experiments and A/B Testing covers how such a test is run.

Finally, state the system targets. Recommendations load within about 200 ms at a high percentile. If any stage fails, the page still shows videos. These are example targets, not guarantees.

Part 3: Map the Data

Impressions Become Labels

A label is the answer a model should learn for one past example. Here, the labels come from the impression logs.

Every time the homepage shows a video, the system logs an impression. The log row records the user, the video, the slot, the time, and the features used to rank it. Later events join that row: a click, the watch time, a "not interested" click.

The rule that turns logs into labels is simple.

• Shown and engaged is a positive example. For example, the user clicked and watched at least 30 seconds.
• Shown and skipped is a negative example. The video was on screen, and the user did not engage with it.

A skip is a weaker signal than a watch. The user may have scrolled past without looking at the video. The 30-second threshold is an example. Each product chooses its own definition of engagement and checks it against the online goals.

Two Flaws in Impression Labels

Labels built only from impressions have two flaws. Name both in an interview.

1. They cover only shown videos. The system has labels only for videos it chose to show. It knows nothing about the millions it never showed. So the model learns from its own past choices.

2. They carry position bias. A video in the first slot gets more clicks than the same video in the tenth slot. Part of that difference comes from the slot, not the video. A model trained on raw clicks partly learns "the top slot is good" and repeats yesterday's order.

Part 6 gives the standard fixes. Where Training Data Comes From explains label sources in general.

Features and Freshness

The ranker reads three groups of features. A feature is one input value that the model uses.

Notice that freshness differs by group. A video's topic rarely changes, but a user's interests in this session can change with every click.

Part 4: Pick the Model

Start With a Baseline

The simplest working system shows the most-watched videos of the last day in the user's region. It is the same for everyone in that region. It is cheap and reliable, and it gives every later model a number to beat. It also becomes a fallback in Part 5.

Three Candidate Sources

Candidate generation does not use one method. It runs three sources in parallel, then merges their results. Each source can find videos that the others miss.

1. Two-tower embeddings. A two-tower model learns a short embedding vector for each user and each video. Training places them so that similar ones sit close together, and a user sits close to the videos that user is likely to enjoy. The two towers meet only at the final comparison, so a video's vector never depends on the user. That is why every video vector can be computed offline and stored in an ANN index. At request time, the system computes one user vector and asks the index for the nearest few hundred videos.

An ANN index trades exactness for speed. Comparing one user vector with all 50 million video vectors would take too long. The index finds near matches fast, even though it sometimes misses the exact closest video. One common index type is HNSW, which links each vector to some of its near neighbors. Embeddings and Two-Tower Retrieval explains how the towers train and serve.

2. Co-occurrence. A nightly job counts which videos the same users watch within a short time. For each video, it stores a list of its most frequent partners. At request time, the system looks up the lists for the user's last few watched videos.

3. Non-personalized sources. These lists are the same for everyone in a group. Examples are trending videos in the user's region, new uploads, and popular videos in the user's language. They give new users something to see, and they let new videos enter the funnel.

The merge step removes duplicates, removes videos the user already watched, and removes videos the user is not allowed to see. About 500 candidates remain.

The Ranker

The ranker scores each of the 500 candidates with the features from Part 3. A good first ranker is a gradient-boosted tree model, a model built from many small decision trees. It is fast and works well with mixed features. Move to a neural network when the data is large and an experiment shows a real gain.

The ranker predicts engagement, such as the chance of a click followed by a meaningful watch. Many rankers also predict expected watch time and combine the predictions in one weighted score. Design Feed Ranking shows how to combine many predictions.

The Re-Ranking Rules

The last stage is a layer of rules, kept outside the ranking model. Common rules include these.

• Diversity: no more than two videos from one creator on a page.
• Freshness: a few slots for recent uploads.
• Policy: remove videos that are not suitable for this user, such as age-restricted content.
• Editorial pins: a video the product team chooses for a fixed slot, such as coverage of a major live event.

Why keep these as rules? Product teams can change a rule without retraining the model. A rule change takes effect at once, and a person can read exactly what it does. If diversity lived inside the model, every change would need new training data, a new model, and a new test.

Part 5: Design the Serving Path

Two Planes

The design has an offline plane and an online plane. The offline plane runs background jobs. It builds the video embeddings, the ANN index, the co-occurrence lists, and the ranker. The online plane answers each homepage request while the user waits.

Follow One Request

The steps below follow one homepage request. The times are illustrative and count from the moment the request arrives.

Step 1: The homepage opens. At 8:00 PM, a user opens the app. This evening, the user already watched three cooking videos. The recommendation service receives the request.

Step 2: The user state is read (about 10 ms). The service reads the user's features, including the three videos watched this session.

Step 3: The user vector is computed (about 15 ms). The user tower turns those features into one embedding.

Step 4: Three sources run in parallel (about 35 ms). The ANN index returns its nearest videos. Co-occurrence returns partners of the three cooking videos. The non-personalized source returns trending videos in the user's region. The merge step keeps about 500 candidates.

Step 5: The ranker scores (about 80 ms). The service fetches features for all 500 candidates. The ranker scores them in one batched call, which takes roughly 20 to 40 ms. Scoring them one at a time would cost far more. The top 50 stay, in order.

Step 6: The rules pick 20 (about 85 ms). The re-ranking rules apply diversity, freshness, and policy. Twenty videos remain.

Step 7: The page is sent and logged. The response leaves well inside 200 ms. The remaining time covers the network and slow cases. When the page appears, each of the 20 videos is logged as an impression.

Step 8: Labels form later. Within minutes, each impression gets its label: shown and engaged, or shown and skipped. That night, the video embeddings and the ANN index are rebuilt. Each day, the ranker is retrained on the new labels.

The diagram below shows the same steps. In the online band, the first box shows the clock time, and each later time counts from the moment the request arrives.