Design a Recommendation System

Stage 1: candidate generation. 50 million down to about 500. The sources run at the same time, and you merge their results. The item vectors are computed ahead of time and loaded into an approximate nearest neighbor (ANN) index. In plain terms: instead of comparing the user against all 50 million items, the index uses a clever shortcut structure (HNSW is the usual one) to find the closest matches fast. The trade is that it is approximately right instead of exactly right, which is fine here. At request time, you grab the user's vector and pull the nearest few hundred items in a few milliseconds. That is how "50 million to 500" is even possible.

Stage 2: ranking. About 500 down to about 50. Here is a latency detail that lands well in interviews: scoring 500 items in one batched call takes 20 to 40 ms on a GPU, or on a well-tuned CPU fleet.

Stage 3: re-ranking. Business judgment on top. The score-ordered list is not what goes on screen yet. First you apply the rules: diversity (not 20 videos from one channel), freshness boosts, policy demotions, dropping items you just showed the user, and editorial pins. Keep this layer outside the model, as plain rules. The reason is practical, and worth saying out loud: product teams can turn these knobs without retraining anything.

Deep dive: cold start, both kinds

Cold start means the system has no history to work with. It comes in two flavors, and the interviewer will ask about both. Bring them up before they do.

• A new user. No history, so the personalized sources come up empty. Fall back to popularity and regional trends. Use whatever the signup gave you (language, declared interests, platform). Then treat that first session like a fast interview: weight their early clicks heavily and update their state during the session, not overnight.
• A new item. No engagement yet, so the collaborative sources cannot find it. Bridge the gap with content features (title, tags, audio and visual embeddings) so the item can enter the candidate pool by looking similar to items that are already established. Add a small exploration boost, since a slice of traffic buys honest labels. Skip this step and you get rich-get-richer: the popular videos keep winning and new ones never get a chance.

6. Keep It Working: The Loop and Its Traps

The rhythm of keeping this alive:

• Retrain the ranker every day.
• Rebuild item embeddings and the ANN index nightly, with quick incremental updates for new items.
• Update each user's short-term state during their session, through the streaming path.

Now the two traps. Here is the tricky part of the whole design, and interviewers know it, so raise them yourself.

• Position bias. Items shown first get clicked more, no matter how good they are. It is like judging books by which shelf the store happened to put them on. Train naively on those clicks and the model learns "whatever we ranked first must be best," which just freezes yesterday's ranking in place forever. The standard fixes: feed the display position in as a training feature and then zero it out at serving time, or weight each example by how unlikely a click was at that position (the technical name is inverse propensity weighting).
• The proxy trap. A proxy is a stand-in metric, one you optimize because the real goal is too slow to measure. Optimize for clicks and you get clickbait. Optimize for watch time and you get videos padded to run long. The grown-up answer is to weigh several objectives at once, and to admit that the real target (long-term retention) only shows up in A/B tests. This is a preview of feed ranking, where the objectives multiply.

7. Count the Costs: Precompute Heavy, Serve Cheap

The cost shape is simple: almost everything expensive happens offline, ahead of time. Nightly jobs embed and index millions of videos that nobody will even ask for tomorrow. That waste is on purpose. It is the price you pay to keep the request path cheap enough for 100 million users. The only real per-request cost is the ranker's one batched call. That is exactly why stage 2 scores 500 items and not 5,000.

Failures follow the same shape, and the good news is they fail soft:

• Nightly build breaks? Serve yesterday's index. Recommendations get a little stale, and nobody's homepage throws an error.
• Ranker times out? Fall back to candidate-generation order, or to trending.

So you alert on index age and fallback rate, not on user-visible errors, because there should not be any user-visible errors to alert on.

💡 If you are asked to estimate infrastructure: item embeddings are 50M x 128 floats x 4 bytes, which is about 25 GB. That fits in memory across a few ANN replicas. Saying that arithmetic out loud turns an abstract design into an engineered one.

8. In the Interview

The 30-second answer: "It is a three-stage funnel. First, candidate generation cuts 50 million videos down to about 500, using a few cheap sources in parallel: a two-tower embedding model served from an ANN index, item-to-item co-occurrence, and a trending-plus-new source for cold start and freshness. Second, a feature-rich ranker scores those 500 for predicted engagement in one batched call. Third, a rule layer adds diversity, freshness, and policy before anything hits the screen. I log every impression and its outcome, retrain the ranker daily, refresh embeddings nightly, and correct for position bias during training. Offline recall and AUC decide which models get an A/B test; long-term engagement decides what actually ships."

Likely follow-ups:

1. "Why not one model end to end?" Money and physics. No model good enough to rank well is cheap enough to run 50 million times on every request. The funnel is the compromise. And each stage gets judged on its own job: retrieval on recall, ranking on precision at the top.
2. "Recall looks fine, but users seem bored. Where do you look?" At the re-rank layer and the objective, not retrieval. Maybe diversity collapsed and every candidate comes from one taste cluster. Or your stand-in metric is over-serving short-term clicks. Break watch time down by content category over time and look for it narrowing. This is a feedback-loop symptom, not a model bug.
3. "How do you do real-time personalization?" Session events stream into the user's short-term features within seconds. The user embedding gets a quick in-session update, or a fast recency-weighted stand-in. Full retraining stays nightly. Two tiers of freshness, and you pay for each.
4. "Your nightly embedding build fails. What do users see?" Nothing, if the design is right. The ANN replicas keep serving yesterday's index. New items lean on content features and the trending source. An alert fires on index age. A second failure in a row starts to hurt new-item discovery, so the pager severity should climb with staleness.
5. "Someone proposes a new candidate source. How do you decide it earns a slot?" Measure its marginal value, not its standalone recall. What share of engaged impressions came from candidates only this source found? Run it in shadow first, then a small A/B. A source that just re-finds what the two-tower already returns adds latency, not lift.

9. Common Mistakes

• Weak: starts by picking a model architecture. Strong: starts with 50 million items against a 200 ms budget, which forces the funnel before any model gets named.
• Weak: treats clicks as the plain truth. Strong: brings up position bias unprompted and offers a fix (position as a training feature, or inverse propensity weighting).
• Weak: buries diversity and policy inside the ranking model. Strong: keeps re-ranking as a separate rule layer, so product teams can turn knobs without retraining.
• Weak: waits to be asked about cold start. Strong: brings up both kinds, with different fixes for new users (popularity plus in-session learning) and new items (content features plus exploration).

10. Same Design, Other Questions

• "Design 'customers also bought' for an online store." What changes: co-occurrence becomes the main source, and purchases replace watch time as the label (rarer, but cleaner). The funnel, cold start, and bias story all carry over unchanged.
• "Design podcast discovery for an audio app." What changes: episodes are long, so "did they finish it" labels are slow and weak. Content embeddings from the audio and transcript now do more of the retrieval work, and the new-item path dominates because new episodes ship every week.

11. TL;DR