What Is a Machine Learning System Design Interview?

A machine learning system design interview asks you to design a working ML product, not to train a model. You get an open question like "design a video recommendation system" and about 45 to 60 minutes. The interviewer scores the whole pipeline. That means the product goal, the data, the features, the model choice, how predictions are served, and how the system is retrained.

The round exists because most ML failures in production are not modeling failures. They are data problems, latency problems, and feedback problems. So the interviewer wants to see whether you can build a system that still works six months after launch.

Here is the structure most companies use, and what each step is worth.

Quick Overview

Step 1: Turn the Request Into an ML Problem

"Design a video recommendation system" is a product request, not an ML task. Your first job is to convert it.

A good conversion names three things. The prediction target, for example the chance that a user watches a video for 30 seconds. The input, for example one user and 500 candidate videos. The business metric, for example watch time per session.

Then name the offline metric you will use while training, such as AUC or recall at 50. Interviewers want both numbers. One you can measure daily during development, and one the business actually cares about.

Step 2: Data and Labels

Say where the training data comes from before you name any model. Many candidates skip this and lose the round here.

Cover the source, such as click logs or purchase events. Cover the label, meaning the value you treat as the correct answer. Cover the volume, such as 50 million events a day.

Labels create the hardest questions. If you train on clicks, the model learns what people click, not what they value. Say that out loud, then give a fix, such as weighting each label by watch time.

Step 3: Features and the Feature Store

A feature is one input signal, like the number of videos a user watched this week. Group your features into user features, item features, and context features.

The trap here is training and serving skew. That means a feature computed during training does not match the same feature computed when serving live traffic. A feature store, which is a shared service that computes and stores each feature once, is the standard answer.

Step 4: Model Choice

Name a simple baseline first. Logistic regression or gradient boosted trees are both fine, and starting there shows experience.

Then name one stronger option and say what it costs. A two-tower neural network finds candidates quickly but needs more infrastructure and more data.

Large systems split this into two stages. Retrieval narrows millions of items to a few hundred. Ranking then scores those few hundred carefully. Describing both stages is the clearest seniority signal in this round.

Step 5: Serving and Latency

State a latency budget early, such as 100 milliseconds for the full request. Then show how your design fits inside it.

Decide what is precomputed and what runs live. Embeddings and candidate lists are usually built in batch jobs. Final ranking usually runs in real time. Cache results for repeat requests inside the same session.

Step 6: Monitoring and Retraining

Say how you will know the model is failing. Track the prediction distribution, the input data, and the online metric, then alert on each one.

Drift means live data no longer matches training data. Give a retraining cadence, such as daily features and weekly full retraining. Finish with the rollout plan: shadow traffic first, then an A/B test on a small share of users.

Questions That Get Asked

Design a feed ranking system. Design a search ranking system. Design fraud detection. Design ad click prediction. Design content moderation. Nearly all of them map onto the same six steps.

How to Prepare

• Learn the standard shape. Practice the six steps until you can run them without notes. Grokking the AI System Design Interview works through real designs in this format.
• Fix the fundamentals first. Sharding, caching, and queues still decide half of this round. Grokking System Design Fundamentals covers them.
• Know the AI building blocks. Embeddings, vector search, and inference cost come up constantly. Grokking Modern AI Fundamentals is a fast introduction.
• Read two adjacent answers. See machine learning infrastructure design in interviews and how to scale an LLM application.
• Say the trade-off, then move. One sentence per choice is enough. Spending five minutes on model math is the most common way to run out of time.