Capstone: Scoping an AI Design Question

Know the Vocabulary

This lesson uses the terms below many times. Read them once now, and come back here when a later section needs one.

• Scoping: Turning a vague question into one system with a clear goal, clear numbers, and clear limits. It happens before you draw any architecture.
• Underspecified question: A question that leaves out important facts on purpose. The interviewer wants to see how you fill them in.
• Question family: One of three kinds of AI system, sorted by what the model outputs. Predictive systems output a number or a label, such as a score. Generative systems output content, and agentic systems take actions.
• Family test: The question "What does the model mainly output: a number or a label, content, or actions?" The answer names the family.
• Agent: A system in which a model chooses some of its own next steps and can take actions through tools, such as placing an order.
• Label: The correct answer for one past example, which a model learns from. For delivery times, the label is the time the delivery really took.
• Impact: How much a system would matter to the business or its users if it worked well.
• Tractability: How realistic it is to build the system well with the data and the time available.
• Tier-1 ticket: A routine, low-stakes support question, such as "Where is my order?"
• Escalation: Handing a conversation from the assistant to a human support agent.
• Resolution rate without escalation: The share of conversations that the assistant fully resolves, with no person taking over.
• Customer satisfaction: A score that customers give after a conversation, usually through a short survey.
• Time to first token: The time from sending a message to seeing the first piece of the answer. A token is a small chunk of text, often part of a word.
• Streaming: Showing the answer piece by piece while the model is still writing it.
• Human baseline: What the current process costs when people do the work. Here, it is the cost of one ticket handled by a person.
• Order of magnitude: A factor of 10. Two orders of magnitude is a factor of 100.
• Scope cut: A part of the problem that you say, out loud, you will not build in this version.
• Design center: One of the few parts of a design where the hardest decisions sit. The rest of the interview goes into the most detail there.
• Large language model (LLM): A model trained on a very large amount of text, which writes text in reply to the text it receives.
• Prompt: All the text sent to the model in one call. It includes the instructions, the facts found for the question, and the conversation so far.
• Retrieval-augmented generation (RAG): A design that first finds the relevant documents and records, then has the model answer from them.
• Model routing: Sending each request to one of several models, such as a cheap small model for easy requests and a larger model for hard ones.
• Guardrail: A check that runs outside the model and limits what the system can say or do.

The Exercise

Read the question below, then stop and answer it yourself before you read on.

"You work at a food-delivery company. The app has about 25 million monthly users and handles about 500,000 orders a day. Leadership wants to use AI. What would you build?"

Monthly users are the people who open the app at least once in a month. All numbers in this lesson are illustrative assumptions, not facts about a real company.

Give yourself about 8 minutes, and say your answer out loud. That is the share of a 45-minute interview that the framework gives to its first two steps, frame the problem and define success. By minute 8, you should be able to say five things:

1. the options you considered, grouped by family
2. the one option you picked, and why
3. the family of your pick, and its design centers
4. what you cut from scope
5. the numbers for quality, latency, scale, and cost

Write your five things down before you read the reference answer.

The skill this capstone drills is scoping a vague AI question quickly. It does not draw the architecture. The rest of the course builds the architecture, and much of it builds on the system this lesson picks.

Part 1: What the Question Leaves Out

Three Missing Pieces

Read the question again, and notice that it gives a company, an industry, and a scale. But it leaves out three things that every design needs.

1. No feature is named. "Use AI" could mean a hundred different systems, so you must choose one.
2. No metric is given. Nothing says what "better" means, so nothing yet decides whether the system works.
3. No family is given. The question does not say whether the model should predict, generate, or act.

These gaps are on purpose. A question with a named feature and a named metric tests whether you know a design. A question without them also tests whether you can decide what to design. Senior engineers are asked to do this at work. A manager says "we should use AI," and someone must turn that into one buildable system.

This question also takes one of the common forms of AI design question: add AI to an existing system. How AI System Design Shows Up in Interviews describes all five forms. In this form, the business already exists, and you must find where AI helps it most.

What the Interviewer Listens For

In the first minutes, the interviewer listens for two signals.

• Range. Do you see more than one kind of AI system? A candidate who only thinks of chatbots misses most of the options.
• Decision. Can you choose one option and defend it? A candidate who lists ten ideas and picks none has not started the design.

You need both, and the reference answer below shows them in five moves. Each move is one of the five things from the exercise.

The five moves are this lesson's own labels, not new framework steps. Moves 1 to 4 make up Step 1 of the framework, frame the problem. Move 5 is Step 2, define success. The framework names the family and the known system inside Step 1, and here both happen in Move 3. An open question needs a pick before it has a family, so the menu and the pick come first.

What Scoping Is Not

Scoping is not asking the interviewer to choose for you. You may ask a few questions, but you are expected to propose the options and make the pick yourself. Then the interviewer can correct you.

Scoping is not a long list of requirements. It should take minutes, not half the interview. The rest of the framework still needs its time.

Scoping is not architecture, so no boxes or arrows appear yet. The output of scoping is one system, its limits, and its numbers.

Part 2: Move 1, Show the Menu

Options by Family

Start by listing a few options, grouped by family, in about 30 seconds. One analogy helps here: show the menu before you order. The menu is your short list of options, grouped by family. Ordering is choosing one of them and saying why. Reading the menu should be quick, and so should the order.

For the food-delivery company, a menu might look like this.

ETA stands for estimated time of arrival. Notice that the menu covers all three families, and that alone shows range.

The diagram below shows the same menu as three columns. Three of the options carry a short verdict, which the next move explains.

Why Both Halves Matter

Listing options shows breadth: you see the whole space. Picking one quickly shows decisiveness: you can commit, and the interviewer needs to see that too.

A candidate who skips the menu looks narrow. A candidate who reads the menu for five minutes looks unable to decide. So keep the menu short, and move to the pick right after it.

Part 3: Move 2, Pick One and Say Why

Impact Plus Tractability

Pick the option with the best mix of impact and tractability. Impact is how much the system would matter. Tractability is how realistic it is to build it well, with the data and the time available. A high-impact option that cannot be built well is a poor first choice. So is an easy option that nobody needs.

For tractability, ask three questions about each option.

• Do labels already exist? A predictive model learns from past examples with correct answers. A generative system can start from a pretrained model, but it still needs examples to test its answers against.
• Is the knowledge available? The system must be able to reach the facts it needs.
• What does a mistake cost? Lower stakes make a first version safer to ship.

The Strongest Option in Each Family

ETA prediction is the strongest predictive pick. It touches every order, so its impact is wide. And its labels already exist: every finished delivery records how long it really took. At 500,000 orders a day, the company gains 500,000 new labels a day without any extra work.

The support assistant is the strongest generative pick. Support cost is a known pain, with a known price per ticket. And the knowledge it needs is internal. Order status lives in the order system as structured records. The refund and coupon rules live in policy documents, which the company writes and controls.

The ordering agent is the weakest opening pick. It would place orders and pay for them. It would run as an autonomous loop: the model decides each next step and acts, with no person checking each step. That puts payment actions on top of trust questions that are not yet solved. What if it orders the wrong meal or orders twice, and who pays when it does? An agent can be the right system later, but it is a risky place to start.

The Pick

Both strong options are good answers, and this reference answer picks the support assistant for three reasons.

1. Its impact is easy to measure in dollars. Every ticket a person handles has a cost, as Part 6 shows.
2. Its first version can be small. It can answer questions before it takes any actions.
3. It exercises the newer parts of the stack. Retrieval, model cost, and guardrails are the parts many AI interviews probe.

Then say what the pick leaves for later. ETA is a strong second choice, and the right pick when the interviewer wants a predictive design. Introduction: ML Systems Foundations picks up delivery-time prediction and builds it.

The pick itself is less important than the stated reason. If the interviewer prefers ETA and you gave a clear reason, you can switch easily, because the method stays the same.

A Sample Dialogue

Here is how Moves 1 and 2 might sound in a real interview.

Interviewer: "So, what would you build?"

Candidate: "Let me lay out a few options by family, then pick one. Predictive options are delivery-time prediction, restaurant recommendations, and promotion abuse detection. Generative options are a support assistant, dish descriptions, and review summaries. The agentic option is an assistant that places orders."

Candidate: "I would not start with the ordering agent. It puts payments behind an autonomous loop, and we have not yet solved the trust questions. ETA is the strongest predictive option, because it touches every order and the labels already exist."

Candidate: "I'll pick the support assistant. Support cost is a known pain, and the knowledge it needs is internal. If you would rather see the ETA model, I'm happy to switch."

Interviewer: "The support assistant is fine. Go on."

Notice the last sentence of the candidate. It gives the interviewer a clear chance to steer, without handing over the decision.

Part 4: Move 3, Name the Family and the Design Centers

Run the Family Test

Ask what the model mainly outputs: for the support assistant, it is answers, which are content. So it belongs to the generative family. The Three Question Families explains the test in full.

More exactly, it is generative with retrieval, because a model alone does not know this customer's order or this company's refund rules. So the system first finds the order record and the right policy text, then writes the answer from them, and that is RAG. Naming RAG also names the known system that the design starts from. The RAG Pipeline builds it step by step.

One feature pushes the design slightly toward the agentic family. The first version can issue small refunds. A refund is an action, not text. Once the assistant can issue a refund, the system gains an agentic part.

So the full classification is "generative with retrieval, pushed slightly toward agentic by one action." Say it in one sentence. It tells the interviewer what kind of design is coming.

Name the Three Design Centers

A design center is where the hard decisions sit, and this assistant has three of them.

1. Retrieval and context assembly. The answer is only as good as the facts given to the model. The system must find the right order record and the right policy text for each question. Context assembly is building the prompt from those facts and the conversation so far.
2. Cost control through model routing. At tens of thousands of conversations a day, model charges add up. Easy questions can go to a small, cheap model, and hard ones to a larger model.
3. Guardrails around the refund action. A refund moves money, so its limits must live in code that the model cannot change.

Naming the design centers now tells the interviewer where you plan to spend most of the remaining time. The later steps go into the most detail on these three.

Part 5: Move 4, Cut the Scope

In and Out

Next, say what this version covers and what it does not. Each cut makes the design smaller and safer.

• In scope: text chat in English, and tier-1 tickets. Examples are "Where is my order?", questions about why a coupon did not work, and questions about how to cancel. The assistant answers from the order record and the policy documents. It can also issue a refund up to an assumed $10 limit.
• Out of scope for now: voice support, and languages other than English. Refunds above $10 are out of scope for the assistant, so they go to a person.

Why Each Cut

Voice is cut because speech adds its own systems: turning speech into text, turning text into speech, and much tighter time limits. It can come later, on top of a text assistant that works.

Other languages are cut because each language needs its own testing. A policy answer that is correct in English must also be correct in every other language.

**Refunds above 10 are cut** because they are the riskiest action. A wrong 8 refund is a small, bounded loss. A wrong 200 refund is real money, and abusers will look for it. So the refund tool checks the amount in its own code. Anything above 10 goes to a human support agent. Many small refunds also add up, so the tool can also limit how many refunds one customer gets in a set period. Safety and Guardrails shows how a limit like this works in code.

The $10 limit is an example for the first version. A team raises a limit like this only when evidence shows the lower one is safe. Tool Use and Function Calling shows a later version with a higher cap.

Confirm the Scope

End the move with one short question: "Should I go ahead on that scope?" It invites the interviewer to change a cut before you spend time on it.

Part 6: Move 5, Put Numbers on It

The numbers come last, because they depend on the scope. They cover four things: scale, quality, latency, and cost.

Scale

Derive the volume from the numbers in the question. Assume that about 8 percent of orders lead to a support contact.

Step 1: Start with orders. The question gives 500,000 orders a day.

Step 2: Apply the contact rate. 8 percent of 500,000 is 40,000 conversations a day.

Step 3: Turn it into a rate. A day has 86,400 seconds, so 40,000 conversations a day is about one new conversation every 2 seconds on average. Dinner time may bring several times that rate.

Step 4: Find the tier-1 share. Assume about 60 percent of contacts are tier-1 tickets. That is about 24,000 tier-1 tickets a day.

Notice that this is not a hard throughput problem. Throughput is the amount of work the system must handle each second. With about 10 model calls per conversation, the average is about 5 calls a second. That is a modest load, even at several times the average. The hard parts are quality, cost, and safety.

Quality

The primary quality metric is the resolution rate without escalation. It counts the conversations the assistant fully resolves, with no person taking over.

That metric has a weakness: a conversation can look resolved because the customer gave up and left. So pair it with a guardrail metric, a second number that must not get worse: customer satisfaction on the tickets the assistant handled. If resolution rises while satisfaction falls, something is wrong.

An example target: the assistant fully resolves at least 50 percent of tier-1 tickets, with customer satisfaction no worse than human agents get. Both halves are part of the target. A high resolution rate with unhappy customers does not pass.

At 24,000 tier-1 tickets a day, a 50 percent rate means about 12,000 tickets a day never reach a person.

Latency

A chat answer has two latency numbers, not one.

• Time to first token: about 1 second. The customer should see the answer start within about a second.
• Time to finish: about 5 seconds for a short answer. Streaming shows the answer as it is written, so the customer reads along while the rest arrives.

Both are example targets, and each has its own cause. A long prompt slows the start, and a long answer slows the finish. Introduction: LLM Serving & Inference explains why.

Cost

Start with the human baseline, and assume that a ticket handled by a person costs about $4.

At example prices, one model call costs about a cent. A cent against $4 is a factor of 400. That is more than two orders of magnitude under the human cost. A whole conversation has about 10 turns, with one model call per turn. Later turns cost a bit more, because each call resends the conversation so far. So the conversation costs about 13 cents, not 10. That is still about 30 times less than one human ticket.

Now put it together for one day, using the illustrative numbers above. Every conversation starts with the assistant, even one it later hands to a person. So the model charges count all 40,000 conversations.

So the model bill is roughly a tenth of the human work it removes, and that ratio is the business case. The model charges leave out retrieval, retries, monitoring, and the team that runs the system, so the real gap is smaller.

There is one caveat: a conversation that the assistant cannot resolve still goes to a person, so it costs the 13 cents and the $4. So compare the total support cost per correctly resolved issue, not one bot conversation against one human ticket. Inference Economics works through this arithmetic step by step.

The diagram below shows how the numbers chain together, from the company's size to the tickets resolved.

Notice that every number after the first rests on one stated assumption. Say each assumption out loud, so the interviewer can change it.

The Whole Scope on One Page

By minute 8, the board should hold the whole scope. The diagram below shows it as one sheet.