Step 3: Capacity Estimations

Before finalizing the design, we estimate the scale to ensure components can handle the load:

• User Base: Assume on the order of 200 million registered users (for a global service). Let’s say 100 million are active in a given day (just as a rough figure). Peak concurrent usage might be tens of millions of users online during prime hours.
• Content Catalog Size: The number of content items (movies, series, episodes) might be around tens of thousands. For example, 50,000–100,000 total titles (Netflix’s library is in that range, excluding individual episodes). This is relatively small compared to user count, which means any algorithm that scales with content size (like scanning all items) is feasible, but anything scaling with user count needs to be distributed. Each content item’s metadata might be a few kilobytes (title, description, images, etc.), so storing the catalog might be on the order of a few hundred MB to a few GB.
• User Watch History Data: Each user might have a watch history of dozens or hundreds of items. If we assume on average 100 items watched per user, for 100M users that’s 10 billion watch events recorded in total. Storing all these events (as records with userID, itemID, timestamp, rating, etc.) is a big data problem – potentially several terabytes of data. However, we might not need all history online; recent history could be in a fast database and older logs in archival storage for batch processing. Daily new events could be tens of millions: e.g. if 100M users watch on average 1 item every 2 days, that’s ~50 million watch events per day.
• Recommendation Outputs: If we precompute, say, a top 50 recommendations for each active user, that would be 50 * 100M = 5 billion recommendation entries. Storing these might require on the order of tens of gigabytes (if each entry is just an item ID or a small structure). This suggests using a distributed store (like a key-value store partitioned by user) to hold these lists in memory or fast storage. Not all users are active daily, so we might prioritize storing for active users; still, in worst case it’s large but manageable with horizontal scaling.
• Traffic – Reads: For recommendation queries, assume each active user triggers ~2–3 recommendation fetches on average per day (e.g. opening app and maybe refreshing or exploring). That’s about 200–300 million recommendation requests per day. In peak hours, this could be roughly 5–10k requests per second hitting the recommendation service. We should design for perhaps >10k rps capacity on the recommendation API. Additionally, each of those requests might fetch multiple items (e.g. 20 recommendations), but that’s handled in one API call. The content images and details would be fetched via CDN or content service separately, but those are read-heavy too (outside our immediate scope).
• Traffic – Writes: User interaction events (views, ratings) are writes. As estimated, tens of millions of events per day (hundreds per second on average, spikes when many people watch popular new show episodes, etc.). These writes initially land in a logging/stream system or a database. There’s also occasional writes for new content added (not as frequent, maybe a few hundred titles a week) and updates to models or caches. The read-to-write ratio is therefore very high – possibly 100:1 or 1000:1 (for every write event, there are hundreds of read requests for recommendations). This implies we can afford to handle writes in a slightly slower, buffered manner (e.g. batching them) as long as reads remain fast.
• Latency Targets: As noted, each recommendation API call should ideally respond within ~50–100ms from the service (so that with network and client rendering the total perceived time is small). The batch jobs have a different “latency” – e.g. the daily batch might take several hours to run through all users. If we do a full re-computation of recommendations once a day, we might allow, say, a 4-6 hour window with a big cluster to finish it. If using incremental updates, the near-real-time component might update a user’s recs within minutes of an event.
• Storage and Memory: We anticipate using scalable storage solutions: e.g. a user profile database of size on the order of hundreds of GB, a content DB few GB, a large distributed file system for logs in the order of TBs. Caches will consume a good amount of memory – for instance, caching top recommendations or popular items might use tens of GB of RAM across a cluster. Each server in the recommendation service might also have some in-memory cache for fast access. We should ensure storage can scale horizontally (adding nodes/disks as data grows).

These estimates guide the choice of technologies (NoSQL databases, distributed caches, etc.) and ensure the architecture can scale to these volumes.

Step 6: Detailed Component Design

Recommendation Computation Workflow

1. Offline Batch Recommendation Generation

The offline pipeline is where the heavy lifting is done to compute personalized recommendations using the full history of data. Here’s how it typically works:

• The pipeline job (running on a cluster) retrieves aggregated data. For example, it might query the data warehouse for a table of user-item interactions (this could be built from the event logs). This is a large-scale data operation. Using a distributed framework (Hive/Spark), the system might get a matrix of which users watched which items, or a list of each user’s recent watches, and item popularity metrics.
• Using this data, the job applies a recommendation algorithm. Going with simple ML complexity, we would choose algorithms that are well-understood and relatively straightforward to implement: for instance, item-based Collaborative Filtering or a form of matrix factorization for collaborative filtering (many libraries exist for MF). We could also incorporate simple content-based filtering (like if a user has watched a lot of Sci-Fi, include some unseen Sci-Fi titles). In practice, a hybrid is often best. For example:
  • Compute similarity between items based on user co-watch patterns (if many users who watched item X also watched Y, then Y is similar to X). Store top similarities in the Item-to-Item Similarity Index.
  • For each user, look at their watched items and pick top recommendations by looking at similar items to those they liked. This is an item-based CF approach: for each item the user liked, take the similar items (from the index) that the user hasn’t seen, score them (perhaps sum of similarity from multiple liked items), and pick the highest scoring ones as recommendations.
  • Additionally, factor in item popularity or newness: e.g. the algorithm can give a slight boost to newer or trending items so that they appear for users who have vaguely relevant taste, even if pure similarity wouldn’t surface them.
• The output of the batch algorithm is typically a list of recommended item IDs for each user (and possibly a score or ranking). This can be huge to materialize (hundreds of millions of users * tens of items), but it’s partitionable. The job might directly write this into a partitioned key-value store, or write to files which are then used to load a database. Alternatively, the job might output model parameters (like user factors and item factors). In that case, the serving system would use those factors to compute recommendations on the fly. However, computing on the fly is complex and slow without ML infra, so in our design we lean towards precomputing the final results to make serving simple.
• Once computed, these results are stored in the Recommendation Store. For example, for each user U, store the top 50 items (IDs) in sorted order of relevance. If using multiple categories of recs (like “top picks” vs “because you watched X”), the batch process might actually produce multiple lists or label them appropriately. We might also store the top similar items for each item (the Item-to-Item index) as a byproduct if we did item-similarity.
• The batch process also can update global data like item popularity counts, user cluster segments, etc., which can be used in recommendations. Because the batch job runs with relaxed time constraints, we can incorporate complex logic here: combining collaborative filtering with content features, ensuring diversity by rule (e.g. don’t put more than 3 items of the same series), etc. These rules are easier to implement in batch than in a real-time service.

Real-Time Recommendation Serving and Updates

The online part of the system uses the precomputed data but also reacts to new information in real time to fine-tune recommendations for immediacy:

• When a request comes in, the Recommendation Service uses primarily the precomputed recommendation list for that user (from the latest batch). This gives a strong personalized baseline with minimal computation. However, that list might need adjustment. The service will typically perform filtering: remove any items the user has already watched (in case the batch data was slightly outdated or the user watched something on another device after the batch run). It also might filter by availability (if a title got removed or is not in the user’s region).
• The service can then incorporate real-time context. For instance, if the user is currently looking at a particular genre or just finished a show, the service might want to bias recommendations towards that context. It could do this by merging results from the Item-to-Item Similarity Index: e.g., “because you watched that comedy, here are similar comedies” – pulling a few items from the similar-items list and mixing them into the overall recommendations. These on-the-fly similar items come from precomputed similarity data, so the computation is just a lookup and maybe a score merge, which is fast.
• Trending/New Content injection: The service may also call the Trending Service or look at a cached trending list. If the user’s own recommended list is short or the user is new (cold start), trending items will be used. Even for existing users, we might include 1–2 trending or editorial picks to promote popular content. This is often done by simply taking the top of the trending list (that the user hasn’t seen yet) and inserting it into their results (perhaps in a separate section like “Trending Now” on the UI).
• The Recommendation Service then scores or ranks the combined candidates. If using a very simple approach, we might just have a priority (e.g. personalized list items first, unless trending is being forced into top slots for business reasons). Or we could have a simple linear combination: for example, a candidate item might have a base score from the personal model and a small bonus if it’s currently trending or very new.
• Finally, the service produces the list of item IDs in the desired order. It then either returns those or fetches the necessary metadata for each (like title, image URL) from the Content Metadata store (which likely is cached heavily). The results are sent back to the client.
• Cold Start Solutions: For new users, since we have no interaction data, the system must rely on other strategies initially. One common approach (used by Netflix) is to perform an onboarding survey – ask new users to pick a few titles or genres they like. This explicit input immediately gives some data to the content-based part of the model (e.g. user likes “Stranger Things” implies they enjoy sci-fi thriller content). With this, we can generate initial recommendations using content similarity and popularity. Additionally, even without a survey, we can use default popular/trending items, possibly localized by region or globally, as a safe bet for first recommendations (e.g. show the currently popular series since many new users might also like them). As the new user starts interacting (clicking or watching something), those events are fed into the nearline pipeline to update their profile quickly.
• Implicit and Explicit Feedback Utilization: The model takes into account both implicit behavior (e.g. a user watching 80% of a movie implies they liked it somewhat, or fast-forwarding a lot might imply disinterest) and explicit signals (like a thumbs-up). Explicit ratings are straightforward indications of preference, so they can directly influence the user’s profile (e.g. if a user rates 5 stars for a comedy, boost their affinity for comedy genre). However, explicit feedback is typically sparse and sometimes biased (not all users rate content, or they might rate only extremes). Implicit feedback is abundant – every play, pause, search, or even hover can be logged – but it’s noisy (watching something fully could mean enjoyment or just inertia). Our system weights these signals appropriately. For example, we might convert implicit feedback into numeric “preference” scores (watching >80% of a movie = +1 implicit rating; aborting after 5 minutes = -0.5, etc.). The collaborative filtering model training will use a combination of actual ratings and derived implicit ratings. The content-based profile of a user is also updated by both: explicit likes might directly add weight to certain attributes, while implicit watch data slowly shifts the profile. All feedback is fed into model updates: the batch retraining will use the cumulative data (e.g., all implicit feedback over months), whereas the real-time system might react instantly to certain signals (e.g. if user thumbs-down a recommendation, immediately remove similar items from the candidate list in this session). Logging is comprehensive: Netflix, for instance, logs billions of rating events and detailed viewing statistics (duration, clicks, etc.) to fuel its algorithms. Our design ensures this feedback loop is robust so that every interaction can potentially refine future recommendations.

Nearline Processing

In between, we have a nearline or streaming layer, as mentioned. This operates in quasi real-time but off the request path (asynchronous). For example, when a user finishes watching something, a nearline processor might within a minute update some of that user’s recommendations or trigger partial model updates (like updating that user’s factor vector with a simple formula rather than waiting for tonight’s full retraining). This gives a compromise: it can handle event-by-event updates with a bit more computation than the strict online path, but since it doesn’t need to respond to the user immediately, it can take a few seconds and aggregate work. The nearline system often writes to the same storage that the online system reads from (like updating the user’s profile store or a cache of their top picks). For instance, Netflix’s architecture uses a nearline pipeline to update “recently watched” and similar signals via a distributed computation framework.

Interaction of Real-Time and Batch Components

The batch and real-time parts work together to provide a balanced solution. The batch system ensures deep analysis of large data (for accuracy and breadth), while the real-time system ensures freshness and context-awareness. Key interaction points and strategies include:

• Data Handoff: The batch job outputs feed the online service. We ensure a reliable handoff: e.g. after computing, the new recommendations are in a database that the online service queries. We might version these (to avoid partial updates issues, possibly update in a staging table and then atomically switch). The online service always uses the latest available batch data but if it’s mid-update, it can temporarily serve slightly older data rather than nothing.
• Real-Time Updates to Batch Data: If a user performs an action that significantly changes their profile (like watching something in a new genre), waiting for the next day’s batch might be too slow to reflect this. One approach is nearline updates: as soon as the event happens, trigger a targeted update for that user. For example, a lightweight job could recompute that one user’s top recommendations by leveraging the item similarity index or by adjusting their existing list (dropping items too similar to what they just watched and adding some new ones). This could be done via a message queue: the event “U watched M” goes to a small service that recalculates U’s recommendations (perhaps only if U is currently active). The recalculated list could then be written to the Recommendations store, updating it out-of-cycle. This provides a quasi real-time refresh for active users. The complexity here is ensuring this doesn’t overwhelm the system (we might do it for key events or a subset of most active users). If such complexity is not desired, an alternative is to simply rely on the item-sim lookup at request time (which we already do) to handle immediate needs, and let the batch catch up later.
• Fallback Mechanism: Because the online service depends on the batch data for heavy lifting, a failure of the batch pipeline or the unavailability of fresh data should not take down recommendations. The system should have fallbacks: if the latest personalized list is not present, fall back to a slightly older list (perhaps stored with a timestamp or kept in cache). If even that’s not available (e.g. new user or something went wrong), fall back to a generic set of popular content. This ensures the service is robust.
• Model Simplicity vs. Freshness Trade-off: We intentionally choose to precompute as much as possible offline. The trade-off is that those results can become stale with respect to a user’s latest actions or the newest content. The real-time component addresses this by overlaying fresh info (like just-watched item influences, or adding new content). This hybrid approach (sometimes called Lambda architecture in big data) leverages both worlds. All approaches (offline, nearline, online) are combined for the best result. We avoid having to do heavy computations during the user request, which keeps latency low, but we still respond to recent events in a timely way.

In summary, the detailed design ensures that all user and content data is organized in the right storage, the batch algorithms can efficiently crunch through the big picture and store results, and the online service can quickly assemble a recommendation list by combining precomputed insights with real-time context. Simpler algorithms are used, but smart use of data (like item similarities and trending info) ensures recommendations remain relevant and timely.

Caching Layers

Caching is everywhere to ensure speed. We specifically use:

• In-memory cache for user profile: When a user is active (online), their profile data can be cached in memory on the recommendation service node or in a distributed cache. This saves repeated DB hits for each request (especially if the user navigates through multiple pages). The cache entry might store the user’s feature vector, recent watch list, etc. If an update comes (via event), we invalidate or update the cache.
• Cache for precomputed recommendations: If we precompute a list of recommendations offline for each user, those lists can be kept in a fast key-value cache (like Redis). The online service simply does GET user_recs:userID to retrieve it. This is very fast (sub-millisecond). We might store multiple lists per user (e.g., one list of “top picks overall”, another of “because you watched X”), each as a sorted list of item IDs.
• Content cache: Frequent item metadata (like the details of the top 1000 popular items) can be cached in memory so that the service doesn’t query the metadata DB repeatedly. Also, things like “list of trending now items” or “new releases this week” might be cached results that get periodically refreshed.
• CDN: While not a database, using a Content Delivery Network is crucial for delivering images (thumbnails, cover art) and maybe even JSON for static rows. For instance, artwork and previews are cached at edge servers near the users. This offloads a huge bandwidth from the core system. The recommendation system might instruct the client what images to fetch (via URLs), which the CDN handles.

Recommendation Request Workflow (End-to-End)

To illustrate how everything comes together, consider what happens when a user opens their Netflix home page (i.e., a recommendation request):

1. Request Arrival: The user’s device (client) hits the API Gateway with an authenticated request like “GET /recommendations/home” with the user’s ID or token. The API gateway authenticates and forwards this to the Recommendation Service (which is part of our Recommendation Engine). Suppose the user is in region US-EAST, the DNS/load balancer directs it to a nearby cluster of servers for low latency.
2. Fetching User Context: The recommendation service instance handling this request will first gather the necessary data about the user. It calls the User Profile Service (or reads from a cache) to get the user’s profile info – e.g. their viewing history, explicit ratings, maybe demographic info if available, and any precomputed user features (like their embedding or genre affinity scores). This is typically a quick lookup (thanks to caching or a fast NoSQL query).
3. Fetching Candidates: Based on the user’s profile, the service generates recommendation candidates:
   • It might query the precomputed recommendations store: e.g., check if we have a ready-to-use list of top items for this user from the last batch job. If so, those items form an initial candidate set.
   • It will likely augment this with fresh candidates from algorithms. For collaborative filtering: it could take the user’s top N similar users (which might be stored from an offline job) and gather items those users rated highly. Or use the user’s embedding to do a nearest-neighbor search in the item embedding space (via the ANN index). For content-based: it could take the top 2–3 genres the user watches and fetch a few popular new items in those genres, or find items with high similarity to the last thing the user watched. In practice, we might have separate modules, e.g., UserCF module yields 50 items, ItemCF module yields 50 items, Content module yields 50, Popularity module yields 10, etc. We union them (possibly weighting some more). If there are duplicates (the same item came from two methods), we handle that by merging or increasing its score.
   • This candidate retrieval phase might involve calls to an Item Similarity Service (to get similar items by content or collaborative similarity) and to a Trending Service (to get currently popular items). Those services either compute quickly or look up cached lists.
   • Also, some candidates are rule-based: e.g., “Continue Watching” items (if the user has incomplete videos) are directly pulled from the user’s profile; these will be placed in a special row on the UI rather than the algorithmic list, but it’s part of the overall logic to decide what rows to show.
4. Feature Preparation: For each candidate item, the service gathers features needed for ranking. This could include:
   • Item features: from the Content Metadata Service (like item genre, year, etc. if not already present in an earlier lookup).
   • User-item interaction features: e.g., has the user watched other seasons of this show (for a series recommendation)? Did the user search for this title recently? Some of these features may come from the user’s profile (which might store that info). Others might require a quick query (though we try to avoid any heavy per-item queries at request time).
   • Context features: current time, device type, etc., which are straightforward. These features might be assembled into a vector per (user, item) pair for input to the ranking model. The system likely does this in-memory for speed, since the data needed is mostly local or already fetched.
5. Ranking & Filtering: Now the Ranking module applies the machine learning model to score each candidate. It runs the model for each candidate and sorts the candidates by score.
   • We also incorporate any business rules at this stage: e.g., diversity constraints. If the top 10 all happen to be very similar (say all are from the same franchise), the system might enforce some spreading out (demote some and promote the next best different item). This can be done via reranking algorithms or by adding diversity as part of the scoring function.
   • Filtering: Ensure we remove any content that should not be shown. For example, items that the user has “thumbs-downed” in the past might be filtered out regardless of score. Or if the user is a kid profile, filter out R-rated content. These rules are applied post-scoring or by setting their score to -inf effectively.
6. Return Results: The top K items (say top 20 for the first row of the homepage) are selected. The service formats the response, which typically includes item IDs and possibly some metadata like the title or a synopsis snippet. However, often the client will retrieve the detailed info (like images) separately or from a CDN. The response goes back through the API Gateway to the user’s device. Now the user sees the recommendations.
7. Caching and Logging: The service might cache the result of this computation for a short time (couple of minutes) keyed by (user, context) so that if the user refreshes quickly, it doesn’t redo all steps. But if any new event comes in (like user watched something), that cache would be invalidated. All the details of what was recommended are also logged (which items in what order) to the event pipeline. This is important for training data – we need to know “we recommended item X, and the user did/didn’t click it” to learn from implicit feedback which recommendations worked. So a “impression” or recommendation-served event is generated.
8. Continuous Feedback: As the user interacts (maybe clicks one of the recommended items), that goes into the feedback loop as described. Suppose they watch one of the recommended movies – an event for “play start” and later “play complete” will fire. The nearline system might pick that up and update their profile immediately (so if they go back to the home screen after watching, the recommendations account for it). The next day’s batch retraining will also include that data point in refining the models.

The above workflow ensures real-time personalization while leveraging precomputed data. Multi-device consistency is naturally handled because no matter which device makes the request, it hits the same backend and the profile is centralized. If a user starts a show on their TV, a “continue watching” candidate will be added to their profile; later if they open the app on mobile, the workflow will include that candidate because the profile knows the show is in progress. Consistency is maintained by updating the central profile and caches on each event, rather than storing state on the device.

Also, multi-profile support (like Netflix’s separate profiles on one account) is easily handled by treating each profile as a separate user in the recommendation system (with its own profile ID and data).

Throughout the workflow, performance optimizations like caching, parallelism (fetching user profile and item features in parallel), and not blocking on non-essential data (e.g., if an external metadata service is slow, we might proceed with whatever data we have and fill that item later or exclude it) are applied to meet the latency target.