A Framework for AI Design Answers

Notice where each shape stops. The classic shape stops at the user, while the AI shape passes the user and returns to the data.

What Interviewers Expect

Interviewers expect a complete answer in about 45 minutes. Complete means the answer reaches how the system is measured, how it stays accurate, what breaks, and what it costs.

They do not expect a new invention. They expect you to cover every step in a sensible order, at the right depth. A fixed framework makes that possible under pressure.

Part 2: The Seven Steps

The Definition

The framework is a fixed order of seven steps. You use the same seven steps for every AI design answer, whatever the family.

1. Frame the problem. Name the family and the known system, then agree on the scope.

2. Define success. Put a number on quality, latency, and scale and cost.

3. Map the data. Say where the data and labels come from, how late the labels arrive, and how fresh the inputs must be.

4. Pick the model. Start with a baseline, and name the condition that would justify something more complex.

5. Design the serving path. Draw the live request path. Every arrow gets a time limit or a cost, and a failure behavior.

6. Keep it working. Close the feedback loop, and ship every change safely.

7. Count the costs. Say the failure ledger and the cost ledger.

The table below gives the question each step answers.

Notice that Steps 2 and 6 fix the two breaks from Part 1. Step 2 pins down quality early, and Step 6 closes the loop.

What the Framework Is Not

The framework is not a script to read aloud. It is an order for your thinking. The interviewer can still steer you, and you follow them.

It is not a reason to spend equal time on every step. Part 5 gives each step its own share of the 45 minutes.

It is not only for new designs. When an interviewer shows you a finished design and asks what is wrong, the seven steps work as a checklist. Critiquing a System Design shows that use.

Part 3: Steps 1 to 4, Before the Diagram

The first four steps decide what you are building and from what. The examples come from the food-delivery company that runs through this course. Its delivery-time estimate is predictive, and its support assistant is generative. An agent that issues small refunds would be agentic. All numbers in this lesson are illustrative assumptions.

Step 1: Frame the Problem

Step 1 has two moves that come before any scoping.

Move 1: Name the family. Ask what the system mainly outputs. A number or a label, such as a score or an estimate, points to the predictive family. Content points to the generative family. Actions point to the agentic family. The Three Question Families gives this test in full.

Move 2: Name the known system. Say which design the question is closest to, and what changes. For example: "This is a timing estimate on a fast request path, like the arrival estimate in a ride-hailing app. What changes is that the estimate must also cover the kitchen, and its prep time changes every minute."

Naming the known system helps in two ways. First, it gives you a proven architecture to adapt, so you do not start from a blank page. Second, it changes the interviewer's role from examiner to collaborator. They now know your route, so they can work with you instead of only testing you.

Then state the scope out loud, saying what is in and what is out. For the estimate, the number shown when checkout opens is in scope. Live updates after the order is placed are out of scope for now.

Close Step 1 with a one-line checkpoint question: "Shall I proceed on that scope?" A wrong scope found at minute 3 costs little. One found at minute 25 costs most of the interview.

Step 2: Define Success

Step 2 turns the product goal into requirements on three axes, and each one gets a number.

1. Quality is the metric that says the model is good, with a target. For the estimate, the average miss might need to stay under 5 minutes. Many answers also name a guardrail metric, a second metric that must not get worse. Here, the share of orders that arrive far later than promised is a good guardrail metric.

2. Latency is the time budget for one request. The estimate must appear within about 100 ms, because the customer is waiting at checkout.

3. Scale and cost are the volume and what each request may cost. The company might handle about 500,000 orders a day.

For a generative system, the same three axes apply with different units. Quality might be the share of support questions answered correctly. Latency might be the time until the first words appear, and cost is often counted per conversation.

These numbers are the measuring stick for every later step. Without them, no choice in the design can be called good or bad.

Step 3: Map the Data

Step 3 answers three questions. Where does the data come from? Where do the labels come from, and how late do they arrive? How fresh must the inputs be when the model reads them?

The answers differ a lot from system to system, as the four examples in the table show.

Two rows need a closer look.

Fraud labels arrive weeks late. A chargeback is a payment that the card owner's bank reverses, often because the owner says they did not make it. It is the main label for fraud, and it can arrive weeks after the payment. Meanwhile, the inputs must be very fresh. A stolen card can be used many times within a minute. So the model's counts of recent payments must include the last few seconds of activity.

A support assistant often has no training labels at all. Nobody has marked past conversations as good or bad answers. Until the product collects signals, there is nothing to learn from. Such signals include conversations that people reviewed, problems that were confirmed as solved, and feedback from users. So the design must include the ways to collect them.

Where Training Data Comes From covers label sources and label delay in depth.

Step 4: Pick the Model

Step 4 always starts with a baseline. Then it names the condition that would justify an upgrade. Each family has its own usual baseline.

Predictive: a simple rule or average. For recommendations, the baseline shows popular items, the same list for everyone in a region. The upgrade is a learned ranker that scores items for each user. For the delivery estimate, the baseline is the restaurant's historical average at that hour. The upgrade is a tree model that also reads the kitchen load and the route. A tree model predicts by asking a series of simple yes-or-no questions about the inputs, such as whether the kitchen is busy.

Generative: an existing model with clear instructions. The baseline is a hosted LLM with clear instructions. When answers must come from the company's own documents, it also gets good retrieval. Prompting means writing the model's instructions and examples, and it comes first. Fine-tuning is not the starting point, so it enters the design only with a stated trigger. One example trigger is "if prompting cannot reach the quality bar on tone or format." Missing facts are not a trigger, because retrieval supplies facts better than training does. The model choice also covers who runs the model. A hosted API comes first, and running your own GPUs is an upgrade that needs high, steady volume as its trigger.

Agentic: one model with a few tools. A tool is a function the model may call, such as looking up an order or issuing a refund. The model works in a loop: it picks a tool, reads the result, and decides the next step. The baseline is a single model with two or three tools, inside a loop with limits on steps, time, and cost. If the steps are always the same, a fixed workflow is even simpler. A design with several agents adds coordination and operating costs. It must earn those costs with a measured reason, such as independent subtasks that can run in parallel.

Notice that every upgrade comes with its condition. "I would use a tree model" is a preference. "I would use a tree model if it beats the average by a clear margin on held-out orders" is a decision. Held-out orders are past orders that the model did not train on.

Part 4: Steps 5 to 7, the Design and Its Life

Step 5: Design the Serving Path

Step 5 is the diagram of the live request path. Three questions shape it, in this order.

Question 1: What can be computed ahead of time? Everything that can be precomputed should be precomputed, before the request arrives. A nightly job can compute restaurant averages, user histories, or lists of candidates. The live path then only looks them up, which saves time on every request.

Question 2: Which requests can take the cheap path? Many requests do not need the most expensive model. A cache, a heuristic, or a small model may be good enough for them. A heuristic is a simple fixed rule, such as showing the restaurant's average time. A router sends each request to the cheapest path that is good enough. Only the rest takes the expensive path, such as a large LLM.

Question 3: What happens when a model is slow or down? Every arrow that points into a model needs a timeout and a fallback, or a clear failure response. For the estimate, a timeout well inside the 100 ms budget falls back to the historical average. The customer still sees a number.

Agentic designs add one more question: what may each tool do? Tool permissions and the limits on the loop belong in this step. A tool that can move money needs stricter rules than a tool that only reads an order.

Step 5 is done when every arrow carries its time limit or cost and its failure behavior. Naming every box is not enough to finish the diagram.

Step 6: Keep It Working

Step 6 closes the loop from Part 1, in four parts.

1. Log the model's inputs and outputs for every request.

2. Capture outcome signals, such as the real delivery time or whether a customer's problem was solved.

3. Compute the quality metrics from Step 2, using those outcomes.

4. Feed it back. The results drive retraining for a predictive model, or changes to the instructions for an LLM.

The same step also covers how every change ships, whether it is a new model, a new prompt, or new rules. The safe order has three stages.

Stage 1: Evaluate offline. Test the change on past data first. If it does not beat the current version, it stops here.

Stage 2: Test on live traffic. First, a shadow deployment runs the new version on live requests while only the old version answers users. It catches crashes and slow responses before any user sees them. Then, where practical, an A/B test shows the new version to some users and the old one to others, and compares the results.

Stage 3: Roll out gradually. Give the new version a small share of users first, then more. Keep a number that triggers a rollback to the old version.

Monitoring watches for drift: a loss of accuracy as the world changes, often slow but sometimes sudden. The inputs can shift, or the way they relate to the outcome can change. Monitoring, Drift, and Retraining covers this step in depth.

Step 7: Count the Costs

Step 7 closes the answer with two ledgers.

The failure ledger names the top failure modes, often three, with a response for each. Good entries are specific, such as an outage of the route service. Another is drift after a city adds a new delivery zone.

One failure type deserves a place in many ledgers: a feedback loop that amplifies bias. A feed shows clickbait, and users click it, because clickbait is made to be clicked. The model learns from those clicks and shows even more clickbait. Each round makes the problem larger than the round before. The response is a guard outside the model, such as rules that push such posts down.

For agentic designs, the failure ledger also states the most damage one runaway task could do. A runaway task is one that keeps acting after it has gone wrong. A limit on refunds or on steps caps that damage.

The cost ledger multiplies the cost of one unit by the volume. Suppose a product gets 10 million requests a day, and a router sends 20 percent of them to an LLM. Each LLM call costs about one cent.

• 10 million requests x 20 percent = 2 million LLM calls a day.
• 2 million calls x 0.01 = about 20,000 a day.

That is about $600,000 a month, and it would be five times larger if every request went to the LLM. So for this product, routing is not optional. It decides whether the product can afford to run.

For a small predictive model, the result is often the opposite. The model itself is cheap to run. The data pipelines and the services around it can cost more.

Notice that the two ledgers answer the two questions every interviewer has at the end. What breaks, and what does it cost?

Part 5: The 45-Minute Clock

The Split

A good answer reaches all seven steps. That only happens if each step gets its own share of the time. The table gives the default split for a 45-minute interview.

The diagram below shows the same split as one timeline, with the question each step answers.