How AI System Design Shows Up in Interviews

The Five Forms of the Question

AI system design questions come in five forms. Spot the form in the first thirty seconds. It tells you what the interviewer wants to spend the hour on.

1. Design a known ML product. "Design YouTube recommendations." "Design Instagram's feed ranking." "Design Google's search autocomplete."

These are the classics. ML engineer loops have asked them for a decade. A loop is the full set of interview rounds a company runs for one candidate. Senior software engineer loops now ask them too.

Here the interviewer wants the whole system, end to end. That means data collection, the feature pipeline, model serving, evaluation, and the feedback loop.

Three quick definitions, because you will use these words constantly. Features are the input signals a model reads, like a user's watch history or the hour of day. The feature pipeline is the system that computes and delivers them. The feedback loop is the path that carries user reactions back into training.

2. Design a GenAI product. "Design ChatGPT." "Design a coding assistant like GitHub Copilot." "Design an assistant that answers questions over our company's documents."

GenAI is short for generative AI: models that produce content, like text, code, or images. These questions went from novelty to standard within about two years of ChatGPT's launch.

The focus shifts here. There is less feature engineering. There is more of this:

• Retrieval: fetching the right documents for the model to read.
• Prompt assembly: packing instructions and context into the model's input.
• Inference cost: what each model call actually costs to run.
• Safety.

3. Add AI to an existing system. "Here is our e-commerce platform. Where would you add AI, and how?" "Our support team receives more tickets than it can handle. Design something."

This form tests judgment, not technical detail. The strongest answers spend real time on whether AI is even needed. Then they pick the smallest system that could work.

4. Design an agent. "Design an AI agent that books travel." "Design a system where an LLM can take actions on a user's behalf."

An LLM is a large language model: the general-purpose text model behind products like ChatGPT. This is the newest form, and it is all about control. What is the agent allowed to do? How do you stop it from doing damage? How do you know whether it is doing a good job?

5. Critique or scale an existing AI design. "Here is our RAG architecture. It is slow, and users say answers are wrong. Fix it."

RAG stands for retrieval-augmented generation. The system fetches relevant documents and hands them to the model, so answers come from facts rather than memory. This form is less common, but it is growing. It rewards operational experience more than memorized theory.

Who Gets Asked What

The same question is graded differently depending on your role. Know what your role is expected to cover. That keeps you from going too deep or not deep enough.

The most common mistake is answering an SWE-loop question like an ML research defense. The question is "Design a documents assistant." The first fifteen minutes go to fine-tuning objectives.

Fine-tuning means giving an existing model extra training on your own data. Those fifteen minutes are fifteen minutes of system design time lost, not gained.

In an SWE loop, the model is one component. The system around it is the actual interview.

💡 Ask directly: "Should I go deep on the modeling side, or focus on the serving architecture?" Interviewers are happy to answer. And asking shows you know the two sides exist.

What Interviewers Actually Grade

Across all five forms, the rubric is more consistent than candidates expect. Interviewers report grading four things.

1. Requirements discipline. Did you turn "design a chatbot" into concrete requirements with numbers?

How many users? What latency? What does a good answer even mean? What is out of scope?

AI questions arrive vaguer than classic system design questions. That makes this step worth more, not less.

2. A complete loop, not just a request path. Classic answers end when the response reaches the user. AI answers cannot end there.

The system must also capture signals about whether the output was any good. Those signals must flow back into improvement. Candidates who draw the feedback loop without being asked stand out immediately.

3. Cost and latency realism. Here is a number worth memorizing. A single LLM call can cost ten thousand times what a database read costs, and take two seconds instead of two milliseconds.

Compare a database read to taking a file off a shelf. Compare an LLM call to ordering a custom report.

Answers that treat model calls like cheap database reads lose points. Answers that budget for them sound senior. "We cache, and we route the easy 80% to a small model" is exactly that kind of answer.

4. Failure honesty. Models produce wrong answers with full confidence. Interviewers probe what happens next.

Who catches it? What is the blast radius, meaning how much damage one wrong answer can do before someone stops it? What is the fallback?

Say "the model might be wrong, so we..." before they have to ask.

What Interviewers Do Not Grade

It helps just as much to know what is not on the rubric. Then you can stop preparing for it.

Deriving the transformer. The transformer is the neural network design inside modern language models, and attention is its core math.

Nobody asks you to write attention equations in a system design round. You need transformers at the level used in What Changes When the System Learns: inputs, outputs, latency, and cost. Nothing deeper.

Naming the newest model. Model names go out of date within months.

"A large hosted LLM for quality, a small open model for cheap high-volume paths" is a better answer than any specific model name. It will still be a good answer later.

Framework trivia. Kafka or Kinesis, Pinecone or pgvector. Kafka and Kinesis are two queue systems, and a queue passes work between services. Pinecone and pgvector are two vector indexes, and a vector index finds items by similarity of meaning instead of by exact key.

Knowing why a queue or a vector index is there matters far more than knowing its name.

In the Interview

The 30-second read: In the first minute, classify the form: known ML product, GenAI product, add-AI, agent, or critique. Then confirm the expected depth by asking "modeling depth or system depth?" Then say your plan out loud: "I'll clarify requirements and success metrics, then data, then the serving path, then evaluation and the feedback loop, then risks." Interviewers relax the moment they hear a plan. And that plan gives you room to spend time where you are strongest.

Likely follow-ups this early in the interview:

1. "Whatever you'd like to design." You may hear this right after your own clarifying question. It is form 3 asked in a different way. They are testing your scoping skill. Pick something small and concrete, and say why you are keeping it small.
2. "Assume the model already exists." This is useful information. The interviewer just told you the focus is serving, cost, and the feedback loop. Use it. Do not return to training.

TL;DR