How AI System Design Shows Up in Interviews

Notice that one company produced all five questions. The topic stays the same, but the form changes what a strong answer contains.

A Form Can Be in Disguise

Some questions hide their form. An interviewer might say, "Here is our product. Design whatever AI feature you'd like." That sounds open, but it is Form 3 in disguise. The product already exists, and you choose where AI goes. The question tests your scoping instinct, not your modeling skill. A strong answer lists a few options, picks one, and says why. Capstone: Scoping an AI Design Question works through a question of this kind for the food-delivery company.

What the Five Forms Are Not

The forms are not the same as the three question families. A family describes what the model mainly outputs: a number or a label, content, or actions. A form describes how the question is handed to you. Forms 1, 2, and 4 line up with the predictive, generative, and agentic families. Forms 3 and 5 can hold a design from any family. The Three Question Families explains the families.

The forms are not a ranking of difficulty either. A Form 1 question can be as hard as a Form 4 question. The form changes where the effort goes, not how much effort is needed.

Part 3: The Lane Your Role Sets

The Same Question at Four Depths

The same question can be asked in four kinds of loops. Each loop expects depth in a different lane. The table below shows a common pattern. Loops differ by company, so treat it as a guide, not a rule.

Several terms in that table need a plain definition.

• A GPU is a chip that runs model math very fast. Large models usually run on GPUs, which are expensive to rent. GPU utilization is the share of that paid time the GPU spends doing useful work.
• Batching means running several requests through the model together, so the GPU does more useful work at once.
• Caching means keeping a result so that a repeat request can reuse it instead of calling the model again.
• A latency budget is the most time each part of a request may take, so that the whole request meets its target.
• Build-vs-buy is the choice between building a part yourself and paying for an existing service, such as a hosted model.
• A phased rollout releases a change to a small share of users first, then to more users while the results stay good.

Notice what the software engineer row leaves out. The model can stay a box. You do not derive its insides. But you must still say what goes into it and what comes out of it. You must also state its limits, such as how slow it is and how often it is wrong.

A Common Lane Mistake

One of the most common lane mistakes is going too deep on the model in a software engineer loop.

Here is how it looks. A senior software engineer hears "Design a documents assistant." They spend the first fifteen minutes on fine-tuning objectives: which data to train on, and what the training should optimize. That knowledge is real. But the loop grades the system around the model.

Those fifteen minutes were lost, not gained. They were taken from the serving path, the feedback loop, and the cost, which are the parts this loop grades. Depth in the wrong lane earns little credit, and it uses up time that the graded parts needed.

Ask When the Lane Is Unclear

Sometimes you cannot tell which lane the interviewer wants. Then ask directly:

"Should I focus on modeling depth, or on serving architecture?"

The question does two jobs. It gets you an answer. It also shows that you know both sides exist, which interviewers often look for.

"Assume the Model Already Exists"

Sometimes the interviewer answers the lane question before you ask it. They say, "Assume the model already exists." That sentence saves you time, because it removes training from the scope.

So spend the hour on serving the model, on its cost, and on the feedback loop. Do not use the freed time to discuss fine-tuning or to pick the ideal model. The interviewer just told you that those topics are out of scope.

Part 4: Four Areas Worth Covering in Every Form

The Same Four Areas

Exact grading criteria vary by company. But the same four areas are worth covering in all five forms. A strong AI design answer shows each of them, and a classic system design answer often misses at least one.

1. Requirements discipline, with concrete numbers.

2. The complete loop, from the request back into training and evaluation.

3. Cost and latency realism about every model call.

4. Failure honesty: saying how the model fails before anyone asks.

Area 1: Requirements Discipline, With Concrete Numbers

This area comes first because AI questions usually arrive vaguer than classic ones. "Make support smarter" is not a requirement. A model's quality is not simply right or wrong, so you must choose what good enough means.

Turn the vague request into numbers. For the support assistant, the numbers might be these illustrative assumptions:

• Resolve at least half of routine questions without a person.
• Show the first words of an answer within about one second.
• Spend about 13 cents or less per conversation.

Each number can be argued with, and that is the point. A number gives every later choice something to be measured against.

Area 2: The Complete Loop

A classic system design answer can often stop when the response reaches the user. An AI design answer cannot.

The reason is that a model's answer can be wrong while the service reports success and no error appears. So the system must capture quality signals and feed them back. For delivery-time prediction, the signal is the real delivery time, which arrives later. For the support assistant, the signals include whether the customer still contacted a person, and their rating of the answer.

Those signals flow back into evaluation. They also drive the next change. The delivery-time model is retrained on the real times. The assistant's prompt, documents, or model are improved where the ratings show weak answers. Without these signals, nobody knows whether the system is getting better or worse. What Changes When the System Learns explains why this loop is part of every AI design.

Area 3: Cost and Latency Realism

Many candidates treat a model call like a database read. It is not.

Compare the two with rough, illustrative numbers. A read from a managed key-value database might cost about a millionth of a dollar and take about 2 milliseconds (ms). One chat turn on a large hosted LLM might cost about a cent and take about 2 seconds.

So the LLM call costs about 10,000 times as much as the database read. It is also about 1,000 times slower. These are orders of magnitude, not prices to quote. An order of magnitude is a factor of ten. The gap is worth remembering, because it changes the design.

A senior-sounding answer budgets for that gap in one sentence:

"We cache, and we route about 80 percent of traffic to a small model."

A small model is a model with far fewer learned numbers inside it, so it is cheaper and faster. The cache answers repeated questions that are not personal, such as a refund policy question, without any model call. A router is a small component that picks the model for each request. It sends easy requests to the small model and hard ones to the large model. The 80 percent is an assumption. Tests on real traffic must confirm that the small model answers that share well. Inference Economics shows how to price these choices, and Serving Optimizations covers caching and routing in detail.

Area 4: Failure Honesty

A model can be confidently wrong. It can return a delivery estimate that is 20 minutes too short, or an answer that invents a refund rule. The service still reports success.

So name the failure before the interviewer has to ask. The sentence has a fixed shape:

"The model might be wrong, so we detect it and fall back to something safer."

Then fill in the specifics. If the delivery-time model is down or too slow, the app shows the restaurant's historical average time. If its estimates start to drift away from the real times, monitoring compares the two and raises an alert. If the assistant cannot support its answer with the help articles it found, it does not show that answer. It shows the articles and offers a human agent.

Waiting to be asked reads as if you had not thought about it. Saying it first reads as experience.

What Is Not Graded

Three things take time in many interviews, but they earn little credit.

The math inside the model. Most LLMs are built on a model design called the transformer. In a design round, nobody expects you to derive the transformer's attention equations on a whiteboard. Those equations are part of the math inside the model.

The newest model's name. Model names age in months, which means a newer model replaces the newest one within months. "We use the newest model from one provider" will be out of date soon. A role-based answer stays true: "a hosted large model for hard cases, plus a small, cheap model for easy ones."

Framework trivia. Kafka and Kinesis are two services that carry streams of messages between parts of a system. Pinecone and pgvector are two tools that provide a vector index. Comparing their features is trivia. Knowing why the design needs a message stream or a vector index matters far more than knowing its brand name.

Part 5: Follow One Question

The steps below follow the first minute of one interview. The loop is for a senior software engineer. The question is: "Add AI to our support ticket system."

Step 1: Classify the form. The ticket system already exists, and the question asks where AI goes. This is Form 3. Say it: "This is about adding AI to a working system, so I'll start by choosing where AI helps most."

Step 2: Confirm the lane. The candidate asks, "Should I focus on modeling depth, or on serving architecture?" The interviewer replies, "Assume the model already exists." Training is now out of scope. The first line set the form, and the reply set the lane. The hour belongs to serving, cost, and the feedback loop.

Step 3: State the plan. "I'll pick the first AI feature for the ticket system and set its targets. Then I'll design how the model is served, what it costs, and how we learn whether it helps. I'll finish with how it fails."

The diagram below shows the same three moves, with what is said at each one.