Design a Recommendation System

Position bias. Items near the top are more likely to be seen and clicked. Click counts therefore reflect both placement and relevance.

Training directly on these clicks can reinforce the previous ranking rather than improve relevance.

One approach explicitly models how position affects examination, separately from relevance. Another uses inverse propensity weighting: examples are weighted using the estimated probability that an item was examined or exposed. Reliable estimates and controls on large weights matter. Simply adding position as a feature and setting it to zero at serving time does not guarantee unbiased scores.

The proxy trap. A proxy is a stand-in metric, one you optimize because the real goal is too slow to measure.

Optimizing clicks alone can favor misleading titles. Optimizing watch time alone can favor unnecessarily long videos.

The mature answer is to weigh several objectives at once, and to admit that long-term retention needs a sufficiently long evaluation window. This is a preview of feed ranking, where the objectives multiply.

Count the Costs: Precompute Heavy, Serve Cheap

The design moves substantial computation offline, but both batch processing and online serving contribute to cost.

Batch jobs compute and index item vectors in advance. Incremental updates can avoid recomputing unchanged items. This reduces request-time work at the cost of storage and background processing.

Each request pays for retrieval, feature lookups, ranking, rules, and data transfer. Limiting the expensive ranker to about 500 candidates helps control this cost.

Plan fallbacks for component failures:

• If the index build fails, use the last validated index within a defined age limit. Continue enforcing current availability and policy restrictions.
• If the ranker times out, use a tested fallback such as candidate order or trending items, with the same required filters.

Monitor index age, fallback rate, latency, quality, and user-visible errors. Fallbacks reduce failures but do not guarantee that every request succeeds.

💡 Raw item vectors require 50M x 128 x 4 bytes = 25.6 GB. This excludes index structures, identifiers, metadata, replicas, and runtime memory. Estimate those separately before choosing the number of machines.

In the Interview

The 30-second answer: "It is a three-stage funnel. First, candidate generation cuts 50 million videos down to about 500. It uses a few cheap sources in parallel. A two-tower embedding model served from an ANN index. Item-to-item co-occurrence. And a trending-plus-new source for cold start and freshness. Second, a feature-rich ranker scores those 500 for predicted engagement in one batched call. Third, a rule layer adds diversity, freshness, and policy before anything appears on screen. I log every impression and its outcome, retrain the ranker daily, refresh embeddings nightly, and correct for position bias during training. Offline recall and AUC decide which models get an A/B test. Long-term engagement decides what actually ships."

Likely follow-ups:

1. "Why not one model end to end?" Cost and latency. No model good enough to rank well is cheap enough to run 50 million times on every request. The funnel is the compromise. And each stage gets judged on its own job: retrieval on recall, ranking on precision at the top.
2. "Recall looks fine, but users seem bored. Where do you look?" Inspect the re-rank layer, objective, and retrieval performance by user group. Maybe diversity collapsed and every candidate comes from one taste cluster. Or your stand-in metric is favoring short-term clicks too much. Break watch time down by content category over time and look for it narrowing. Investigate feedback effects, segment-level retrieval quality, and model behavior.
3. "How do you do real-time personalization?" Session events stream into the user's short-term features within seconds. The user embedding gets a quick in-session update, or a fast recency-weighted stand-in. Full retraining stays nightly. Two tiers of freshness, and you pay for each.
4. "Your nightly embedding build fails. What do users see?" Usually a slightly older set of recommendations, if the fallback works. The ANN replicas keep serving yesterday's index. New items lean on content features and the trending source. An alert fires on index age. A second failure in a row starts to hurt new-item discovery, so the alert severity should rise with staleness.
5. "Someone proposes a new candidate source. How do you decide whether to include it?" Measure its marginal value, not its standalone recall. What share of engaged impressions came from candidates only this source found? Run it in shadow first, then a small A/B. A source that only finds what the two-tower already returns adds latency without improving results.

Common Mistakes

• Weak: starts by picking a model architecture. Strong: starts with 50 million items against a 200 ms budget, which forces the funnel before any model gets named.
• Weak: treats clicks as the plain truth. Strong: brings up position bias unprompted and offers a fix (an explicit position-bias model or validated propensity weighting).
• Weak: buries diversity and policy inside the ranking model. Strong: keeps re-ranking as a separate rule layer, so product teams can change rules without retraining.
• Weak: waits to be asked about cold start. Strong: brings up both kinds, with different fixes for new users (popularity plus in-session learning) and new items (content features plus exploration).

Same Design, Other Questions

• "Design 'customers also bought' for an online store." What changes: co-occurrence becomes the main source, and purchases replace watch time as the label (rarer, but cleaner). The funnel, the cold start, and the bias story are all the same.
• "Design podcast discovery for an audio app." What changes: episodes are long, so "did they finish it" labels are slow and weak. Content embeddings from the audio and transcript now do more of the retrieval work, and the new-item path dominates because new episodes ship every week.

TL;DR