AI System Design Interview: The Complete Guide

A short disclosure: I created Grokking the System Design Interview and, more recently, Grokking the AI System Design Interview, so I have a bias toward "take a course" as the answer. Everything below is complete and useful on its own. It comes from building those courses and from running hundreds of system design interviews.

A new round has quietly entered the loop. Next to "Design Twitter" and "Design a URL shortener," interviewers now ask "Design a recommendation system," "Design a ChatGPT-style assistant," or "Design an agent that resolves support tickets." This is the AI system design interview, and it trips up strong engineers who have never had to design a system with a machine learning model at its core.

The good news: it is more learnable than it looks. Under all the product names, the same handful of shapes keep coming back, and one framework answers all of them. This guide covers what the interview is, how it differs from classic system design, the three families every question falls into, a seven-step method for answering any of them, the questions you are most likely to get, and how to prepare.

What Is an AI System Design Interview?

An AI system design interview (often called an ML system design interview or machine learning system design interview) asks you to design a production system whose central component is a learned model, not hand-written logic. Think a recommendation feed, a search ranker, a fraud detector, a retrieval-augmented chatbot, or an autonomous agent.

It sits on top of everything you already know about system design: requirements, APIs, databases, caching, load balancing, and scale. Then it adds the parts that only appear once a model is in the picture: where the training data comes from, how training differs from serving, how you evaluate quality, what the model costs per request, and the failure modes that no traditional system has.

Here is the key thing interviewers are grading, and it surprises people: they are almost never testing whether you can derive backpropagation or recite a transformer's internals. They are testing whether you can turn a vague product into a working, affordable, reliable system, and defend the trade-offs out loud. The models are new. The interview discipline is not.

Who Gets Asked AI System Design Questions?

The role of "ML engineer" is poorly standardized, so the interview varies. But AI design questions now show up for a wide range of candidates:

• Software engineers moving toward AI-heavy teams, who are increasingly asked how they would add a model, a RAG pipeline, or an agent to a product.
• Machine learning and applied-science engineers, for whom this is the core round.
• Data engineers, especially where the design leans on feature pipelines, streaming, and evaluation.

AI-first companies build entire interview loops around these questions. If you are preparing for a specific lab, it helps to read up on the format first: see our guides on the OpenAI system design interview and the broader set of OpenAI system design interview questions. Beyond AI labs, the questions are quietly joining the standard loop at many product companies.

Classic System Design vs AI System Design

If you have prepared for a traditional system design interview, you are most of the way there. The framing, the emphasis on requirements, and the focus on trade-offs all carry over. What changes is where the hard parts live.

The single most important mental shift: in classic system design a bug usually crashes something loudly. In AI system design the scariest failures are silent. Quality drifts, nobody gets paged, and the numbers just quietly get worse. A candidate who names that risk unprompted stands out immediately. If you want to shore up the classic foundations first, Grokking the System Design Interview is the place to do it, and our system design interview fundamentals guide is a free starting point.

The Three Families of AI Design Questions

Almost every AI design question belongs to one of three families, sorted by one simple thing: what does the model output? Learn to recognize the family in the first minute, and you know which architecture skeleton to reach for.

1. Predictive (the model outputs a score)

Recommendations, feed and search ranking, ad click prediction, fraud detection, content moderation. The model scores items, and you show or act on the best ones. The skeleton is a funnel: a cheap retrieval stage narrows millions of candidates down to hundreds, an expensive ranking stage scores those hundreds, and a final rules layer applies business judgment. Every predictive design is this funnel with the dials set differently.

2. Generative (the model outputs content)

A chat assistant, an AI-powered search answer, a code assistant, a text-to-image product. The model writes something. The skeleton is a retrieve-and-generate pipeline: assemble the right context (often with retrieval, or RAG), prompt the model, generate, and guard the output for safety and grounding. The defining challenge is that there is no free "clicks" signal, so you have to design how you measure quality.

3. Agentic (the model chooses actions)

An agent that resolves tickets, an autonomous coding assistant, a research agent. The model decides the next step at runtime and can take real actions through tools. The skeleton is a bounded loop: reason, act, observe, repeat, with hard limits on steps, cost, and blast radius. The defining challenge is that errors compound across steps and a confused agent fails quietly rather than loudly.

Naming the family out loud ("this is a predictive ranking problem, so I will use a retrieval-and-ranking funnel") is the fastest way to sound like you have done this before.

A Seven-Step Framework for Any AI Design Question

The reason these interviews feel unpredictable is that candidates improvise a different structure every time. Do not. Walk the same seven steps in order for every question. It keeps you from freezing, and it hits exactly what interviewers grade.

1. Frame the problem. State the scale in numbers (how many items, how many users, the latency budget), because the numbers decide the design. Name the question family and the skeleton it implies. Say what is out of scope.
2. Define success. Separate the business truth (retention, revenue, resolution rate) from the trainable stand-in you actually optimize (clicks, watch time, a graded relevance score), and the offline metrics that gate a change before it ships. Say how they connect: offline filters, online decides.
3. Map the data. Where do the labels come from? Interaction logs are the luxury of predictive systems: cheap and fast, but biased toward whatever the system already showed. Generative and agentic systems usually have to manufacture their labels through evaluation.
4. Pick the model. At the altitude an interviewer cares about: which family of model and why, not a proof. A gradient-boosted tree for tabular prediction, a two-tower model for retrieval, a frontier LLM you buy rather than build for generation. Always offer a simple baseline first.
5. Design the serving path. This is where most of the interview lives. Draw the request path with real latency budgets, name what is precomputed offline versus computed live, and keep business rules outside the model so product teams can change them without retraining.
6. Keep it working. The training-and-serving loop and its traps: training/serving skew, position bias in click data, the proxy trap where you optimize a stand-in and get clickbait. Name the monitoring signals and the retraining cadence, with a reason for the cadence.
7. Count the costs. Unit economics said out loud: cost per request or per conversation, times volume. Then the failure ledger: the top failure modes and the fallback for each. The best answers fail soft, never silent.

This framework is the spine of Grokking the AI System Design Interview, where each design lesson walks these exact seven steps so the path you study is the path you walk on the whiteboard.

The Questions You Are Most Likely to Get

You cannot memorize every prompt, but you can learn the canon. Master these and you can improvise the rest by recombining their parts.

• Design a recommendation system (video, e-commerce, or "customers also bought"). The base case for the funnel: multi-source retrieval, a feature-rich ranker, a rules layer, and cold start for new users and new items.
• Design feed ranking. The twist is many objectives at once: a good post is one you like, comment on, and do not report. Balancing them is the whole game.
• Design search ranking. Adds a query and a trap: people click whatever sits at the top, so your click logs lie by position. Query understanding, hybrid lexical-plus-semantic retrieval, and two-level ranking.
• Design ad click prediction. The score sets a price in an auction, so it has to be a calibrated probability, not just correctly ordered. Calibration is the word that wins this one.
• Design fraud or abuse detection. Adds an adversary who adapts to your model, and asymmetric costs where a miss is far more expensive than a false alarm.
• Design a RAG system (document Q&A). Retrieve relevant chunks, ground the answer in them, cite sources, and enforce permissions inside the retrieval query so the system never leaks a document.
• Design a ChatGPT-style assistant. The model is bought, not built. The design is everything around it: conversation state, context assembly under a token budget, model routing, streaming, and the moderation sandwich.
• Design a text-to-image or agent system. GPU fleet economics for image generation; bounded loops, scoped tools, and safety-by-construction for agents.

For the machine learning infrastructure angle that often comes up in these rounds, our answer on ML infrastructure design in interviews is a useful companion.

The Concepts You Have to Know

You do not need a research background, but a handful of terms come up again and again. Be able to define each in one plain sentence and say why it matters.

• Training vs serving skew. Any difference between the features a model saw in training and the features it sees live. It degrades quality silently, and the cure is to compute features once, in one shared path.
• Feature store. The shared system that computes and serves features to both training and serving, so the two never drift apart.
• Embeddings and ANN search. An embedding is a list of numbers that captures meaning; an approximate nearest neighbor index finds the closest ones fast. This is the machinery behind retrieval and RAG alike.
• Calibration. Predicted probabilities that match reality. Essential when the score sets a price or a threshold, not just a ranking.
• Position bias. Items shown first get clicked more regardless of quality, which poisons naive training on click logs.
• Prefill and decode. The two phases of LLM inference. Reading the prompt is a fast parallel pass; writing each output token is a slower, one-at-a-time step, and it dominates cost.
• RAG and hybrid retrieval. Retrieval-augmented generation fetches relevant text and pastes it into the prompt as evidence. Hybrid retrieval runs keyword and semantic search together and merges them.
• Bounded agent loops. Every agent ships with budgets: max steps, a cost cap, a timeout, and a blast-radius limit, plus a plan for what happens when a budget runs out.
• Guardrails and prompt injection. LLMs read instructions and data through the same channel, so attacker text hidden in retrieved content can hijack the model. Defense is layered input and output gates, not a single clever instruction.

Red Flags That Fail Candidates

Interviewers see the same avoidable mistakes over and over. Avoiding them is half the battle:

• Opening with a model architecture instead of the requirements and the scale. The numbers should force the design before any model is named.
• Treating clicks as ground truth. Strong candidates name position bias unprompted and correct for it.
• Forgetting classic system design once a model appears. Timeouts, fallbacks, and horizontal scaling still matter.
• Skipping evaluation and cost. If you never say how you measure quality or what a request costs, you sound like you have never operated one of these systems.
• Designing an unbounded agent. An agent loop with no step or cost limit is the same red flag as an unbounded retry loop.
• Handling permissions "in the prompt." For any system over private data, access control belongs in the retrieval query and the tools, enforced fail-closed, never in the prompt.

How to Prepare

A focused plan beats grinding random questions:

1. Learn the one framework and make the seven steps muscle memory, so you never open an answer facing the wrong direction.
2. Work the canon (recommendation, ranking, RAG, an assistant, an agent) until you can walk each in the framework without notes.
3. Drill the concepts above until you can define each in a sentence and defend the trade-off it implies.
4. Practice out loud and timed. These interviews are interactive, and how you respond to a probe matters as much as the overall design. Rehearse a 30-second answer for each canonical design.
5. Read the company's format if you know where you are interviewing, so the round holds no surprises.

If you want a structured path through all of this, Grokking the AI System Design Interview covers every question family and concept above with worked numbers, a 30-second answer and common mistakes for each design, and full timed case studies.

Frequently Asked Questions

How is an AI system design interview different from a regular system design interview?

The framing is the same, but the hard parts move. A regular interview centers on databases, caching, and availability. An AI interview centers on data, training versus serving, evaluation, model cost, and silent quality drift. You still need the classic foundations; you add a model-shaped layer on top.

Do I need a machine learning background to pass?

No. Interviewers grade whether you can turn a product into a working, affordable, reliable system and defend the trade-offs, not whether you can derive the math. You do need to reason about a model as a component with a latency, a cost, and a failure mode, and to know a handful of concepts like training/serving skew, calibration, RAG, and bounded agent loops.

Is "AI system design" the same as "ML system design"?

In practice, yes. The terms are used interchangeably. "ML system design" leans slightly toward predictive systems like recommendation and ranking, while "AI system design" now also covers generative and agentic systems built on large language models. The framework in this guide handles all of them.

What companies ask AI system design questions?

AI labs such as OpenAI, Anthropic, and their peers, along with big tech and any product company adding machine learning or generative AI to its stack. The round is most common for ML and applied-science roles, but software engineers increasingly see it too.

How long does it take to prepare?

If you already know classic system design, a few focused weeks is usually enough to learn the framework, work the canonical designs, and drill the concepts. Starting from less system design background, budget longer and build the foundations in parallel.

What is the best way to practice?

Out loud and timed, ideally with a friend or mentor who can probe your answers, because responding well to follow-ups is graded as heavily as the design itself. Pick a canonical question, run the seven steps in forty-five minutes, and compare what you said against a worked walkthrough.

The AI system design interview looks intimidating from the outside and turns out to be one of the most learnable rounds once you see the structure underneath it. Learn the three families, internalize the seven-step framework, work the canon, and practice out loud. If you want the whole thing in one place, that is exactly what Grokking the AI System Design Interview was built to give you.