How AI System Design Shows Up in Interviews

1. The Five Forms of the Question

AI system design questions come in five forms. Learn to spot the form in the first thirty seconds. It tells you what the interviewer wants to spend the hour on.

1. Design a known ML product. "Design YouTube recommendations." "Design Instagram's feed ranking." "Design Google's search autocomplete." These are the classics. ML engineer loops (a loop is the full set of interview rounds a company runs for one candidate) have asked them for a decade. Now senior software engineer loops ask them too. Here the interviewer wants the whole system, end to end: data collection, the feature pipeline, model serving, evaluation, and the feedback loop. Quick definitions, since you will use these words constantly. Features are the input signals a model reads, things like a user's watch history or the hour of day. The feature pipeline is just the plumbing that computes and delivers them. The feedback loop is the path that carries user reactions back into training.

2. Design a GenAI product. "Design ChatGPT." "Design a coding assistant like GitHub Copilot." "Design an assistant that answers questions over our company's documents." These went from novelty to standard interview question within about two years of ChatGPT's launch. GenAI is short for generative AI: models that produce content, like text, code, or images. The center of gravity shifts here. There is less feature engineering. There is more of this:

• Retrieval: fetching the right documents for the model to read.
• Prompt assembly: packing instructions and context into the model's input.
• Inference cost: what each model call actually costs to run.
• Safety.

3. Add AI to an existing system. "Here is our e-commerce platform. Where would you add AI, and how?" "Our support team is drowning in tickets. Design something." This form tests judgment, not machinery. The strongest answers spend real time on whether AI is even needed. Then they pick the smallest system that could work.

4. Design an agent. "Design an AI agent that books travel." "Design a system where an LLM can take actions on a user's behalf." An LLM is a large language model: the general-purpose text model behind products like ChatGPT. This is the newest form, and it is all about control. What is the agent allowed to do? How do you stop it from doing damage? How do you even know whether it is doing a good job?

5. Critique or scale an existing AI design. "Here is our RAG architecture. It is slow, and users say answers are wrong. Fix it." RAG stands for retrieval-augmented generation: the system fetches relevant documents and hands them to the model, so answers come from facts rather than memory. This form is less common, but it is growing. It rewards operational instincts over textbook recall.

2. Who Gets Asked What

The same question gets graded differently depending on your role. Know your lane. It keeps you from overshooting or undershooting.

Here is the most common mistake: a software engineer answers an SWE-loop question like they are defending an ML research paper. Picture the interviewer asking "Design a documents assistant." Now picture spending the first fifteen minutes on fine-tuning objectives (fine-tuning means giving an existing model extra training on your own data). That is fifteen minutes you did not spend designing the system. In an SWE loop, the model is just a component. The system around it is the actual interview.

💡 Ask directly: "Should I go deep on the modeling side, or focus on the serving architecture?" Interviewers are happy to answer. And asking shows you know the two sides exist.

3. What Interviewers Actually Grade

Across all five forms, the rubric is more consistent than candidates expect. Interviewers report grading four things:

1. Requirements discipline. Did you turn "design a chatbot" into concrete requirements with numbers? How many users? What latency? What does a good answer even mean? What is out of scope? AI questions arrive vaguer than classic SD questions. That makes this step worth more, not less.
2. A complete loop, not just a request path. Classic SD answers end when the response reaches the user. AI answers cannot end there. The system must also capture signals about whether the output was any good, and those signals must flow back into improvement. Candidates who draw the feedback loop unprompted stand out immediately.
3. Cost and latency realism. Here is a number worth knowing cold: a single LLM call can cost ten thousand times what a database read costs, and take two seconds instead of two milliseconds. Think of a database read as grabbing a file off a shelf, and an LLM call as commissioning a custom report. Answers that treat model calls as free reads fail quietly. Answers that budget for them read as senior. "We cache, and we route the easy 80% to a small model" is exactly that kind of answer.
4. Failure honesty. Models produce wrong answers with full confidence. Interviewers probe what happens next. Who catches it? What is the blast radius? What is the fallback? Say "the model might be wrong, so we..." before they have to ask.

4. What Interviewers Do Not Grade

It helps just as much to know what is not on the rubric. Then you can stop preparing for it.

• Deriving the transformer. The transformer is the neural network design inside modern language models, and attention is its core math. Nobody asks you to write attention equations in a system design round. You need transformers at the altitude of What Changes When the System Learns: inputs, outputs, latency, and cost. Nothing deeper.
• Naming the newest model. Model names age in months. "A large hosted LLM for quality, a small open model for cheap high-volume paths" beats any specific model name, and it will still be a good answer down the road.
• Framework trivia. Kafka or Kinesis (two queue systems), Pinecone or pgvector (two vector indexes). A queue passes work between services. A vector index finds items by similarity of meaning instead of by exact key. Knowing why a queue or a vector index is there matters far more than knowing its name.

5. In the Interview

The 30-second read: In the first minute, classify the form: known ML product, GenAI product, add-AI, agent, or critique. Then confirm the lane by asking "modeling depth or system depth?" Then say your plan out loud: "I'll clarify requirements and success metrics, then data, then the serving path, then evaluation and the feedback loop, then risks." Interviewers relax the moment they hear a plan. And that plan buys you the right to spend time where you are strongest.

Likely follow-ups this early in the interview:

1. "Whatever you'd like to design." You may hear this right after your own clarifying question. It is form 3 in disguise. They are testing your scoping instinct. Pick something small and concrete, and say why you are keeping it small.
2. "Assume the model already exists." This is a gift. The interviewer just told you the lane is serving, cost, and the feedback loop. Take the gift. Do not wander back into training.

6. TL;DR