0% completed
How AI System Design Shows Up in Interviews
On This Page
The Five Forms of the Question
Who Gets Asked What
What Interviewers Focus On
What Usually Matters Less
In the Interview
TL;DR
The Five Forms of the Question
This course groups AI system design questions into five common forms.
Try to identify the form early, then confirm your understanding with the interviewer. This helps you decide which parts of the design need the most attention.
1. Design a known ML product.
"Design YouTube recommendations." "Design Instagram's feed ranking." "Design Google's search autocomplete."
These questions can appear in machine learning (ML) engineering interviews and software engineering interviews involving AI features.
The task is to explain the complete system: data collection, the feature pipeline, model serving, evaluation, and the feedback loop.
Three definitions will help. Features are input signals a model uses, such as a user's watch history or the hour of the day. The feature pipeline computes and delivers those signals. The feedback loop collects outcomes, such as user reactions, to evaluate and improve the system.
2. Design a GenAI product.
"Design ChatGPT." "Design a coding assistant." "Design an assistant that answers questions using our company's documents."
GenAI is short for generative AI. These models produce content such as text, code, or images.
Compared with a predictive ML question, the discussion may focus less on feature engineering and more on:
- Retrieval: Finding relevant documents for the model to use.
- Context assembly: Combining instructions, documents, and other information into the model's input.
- Inference cost: The cost of running the model to produce an output.
- Safety: Preventing unauthorized actions and reducing harmful or incorrect outputs.
3. Add AI to an existing system.
"Here is our e-commerce platform. Where would you add AI, and how?" "Our support team receives more tickets than it can handle. Design something."
This form tests your ability to choose a useful feature, not only to design one.
First, consider whether AI is needed. Then choose the simplest system that could meet the product goal. Explain why you chose it.
4. Design an agent.
"Design an AI agent that books travel." "Design a system where a model can take actions on a user's behalf."
A large language model (LLM) processes and generates language. An agent can use a model to select tools and decide what to do next.
These questions require clear controls. What may the agent do? Which actions need approval? When must it stop? How will you measure whether it completed the task correctly?
5. Critique or scale an existing AI design.
"Here is our RAG architecture. It is slow, and users say the answers are wrong. Fix it."
RAG stands for retrieval-augmented generation. The system retrieves relevant information and includes it in the model's input. This can help support answers with evidence, but it does not guarantee that every answer is correct.
For this form, explain how you would find the cause of a problem before changing the design. Use measurements, logs, and examples of failed requests to support your decisions.
Who Gets Asked What
The expected depth depends on the role and company. The table below is a preparation guide, not a fixed rule about which questions each role receives.
Confirm the expected focus so you do not spend too much or too little time on model details.
| Role | Forms to prepare for | Expected depth |
|---|---|---|
| Software engineer, senior and above | 2, 3, and sometimes 1 | Architecture, model serving, cost, and failure handling. Explain the model's inputs, outputs, and limits. |
| ML engineer | 1, 2, 5 | System design plus model choices, features, training pipelines, and evaluation metrics. |
| AI or platform engineer | 2, 4, 5 | Inference infrastructure: GPU use, batching, caching, and response times under load. |
| Engineering manager, staff and above | 3, 5 | Build-versus-buy decisions, cost limits, team responsibilities, risk, and gradual rollout. Technical depth still depends on the role. |
A common mistake is answering a software engineering question as if it were a discussion about model research.
For example, suppose the interviewer asks you to design a document assistant. Spending the first fifteen minutes on fine-tuning objectives leaves less time for retrieval, serving, evaluation, and failure handling. Fine-tuning means giving an existing model additional training.
Model details can matter, but they should support the system design rather than replace it.
💡 Ask directly: "Should I focus on model development or on the architecture that serves the model?" Use the answer to decide how much detail to include.
What Interviewers Focus On
The exact evaluation criteria vary, but the following areas are useful to cover across all five forms.
1. Requirements discipline.
Turn "design a chatbot" into concrete requirements. How many users will it serve? How quickly must it respond? How will you measure answer quality? What is out of scope?
AI questions can be broad. Clarify the goal and state any numerical assumptions before choosing an architecture.
2. A complete loop, not just a request path.
Explain what happens after the response reaches the user.
The system needs signals about output quality, such as ratings, task outcomes, or reviewed examples. Explain how you will use these signals to evaluate and improve it. Improvements might involve data, retrieval, instructions, or the model itself.
Include this feedback process in your diagram when it is relevant. Do not assume that user feedback should automatically become training data.
3. Cost and latency realism.
A model call may be much slower and more expensive than a database read. The difference depends on the model, input and output length, hardware, and serving setup. There is no single cost ratio to memorize.
State your assumptions and estimate the cost. Then explain where caching, smaller models, or background processing could help.
For example: "We will test whether a smaller model meets the quality target for common requests, and use the larger model when needed." Do not claim that a fixed percentage of requests can use the smaller model without evidence.
4. Failure honesty.
A model can produce an incorrect answer even when the service returns successfully.
Explain how you would detect the problem, limit its impact, and provide a fallback. A confident writing style is not proof that an answer is correct.
Discuss these risks before the interviewer has to ask about them.
What Usually Matters Less
For a system-design-focused round, the following topics are usually secondary. A model-development or research role may require more detail, so confirm the scope.
Deriving the transformer. A transformer is a neural network architecture used by many language models. Attention is one of its main mechanisms.
You usually need to explain the model's inputs, outputs, quality, latency, and cost. Detailed equations matter only when they are relevant to the question or expected for the role.
Naming the newest model. Model names and capabilities change. Explain the requirements your model must meet before choosing a specific product.
For example: "We will compare a larger hosted model with a smaller self-hosted model using our evaluation set and cost target." Size and hosting choice alone do not determine which model is better.
Framework trivia. Explain why you need a component before naming a tool.
A queue passes work between services. A vector index finds the stored vectors closest to a query vector. A vector is a list of numbers representing a piece of text, an image, or other data, so closeness between two vectors stands for similarity between the two items. Explain why each component is needed and what limitations it introduces.
In the Interview
The 30-second read: Identify whether the question concerns a known ML product, a GenAI product, an AI addition, an agent, or a design review. Confirm whether the interviewer wants model-development detail or system-design detail. Then state your plan: "I'll clarify requirements and success metrics, discuss the data and serving path, and finish with evaluation, feedback, and risks."
Likely follow-ups this early in the interview:
- "Whatever you'd like to design." The interviewer is asking you to choose and define a problem. If you are adding AI to an existing product, this is form 3. Pick a specific feature and explain why its scope is manageable.
- "Assume the model already exists." Focus on integration, serving, cost, evaluation, and feedback. Discuss training only if a requirement makes it relevant or the interviewer asks for it.
TL;DR
| Five forms | Known ML product, GenAI product, adding AI, agent, and design review. |
| What to confirm | How much model-development and system-design detail is expected. |
| Focus on | Clear requirements, evaluation and feedback, realistic cost and latency, and failure handling. |
| Matters less | Model equations and tool trivia, unless the role or question requires them. |
| First minute | Identify the form, confirm the focus, and state your plan. |
Reading Progress
0%
On This Page
The Five Forms of the Question
Who Gets Asked What
What Interviewers Focus On
What Usually Matters Less
In the Interview
TL;DR