Capstone: Scoping an AI Design Question

1. The Exercise

This capstone practices the exact skill the module just built. Take a vague AI question. In the first ten minutes, turn it into a designable problem. No architecture yet, that is the rest of the course. Here we practice only the opening, because interviews get lost in the opening far more often than in the deep dive.

Here is the question, delivered the way an interviewer actually would:

"We're a food-delivery company, think DoorDash scale: about 25 million monthly users, 500,000 orders a day. Leadership wants us to 'use AI to improve the customer experience.' Design something."

Read it again. It is deliberately underspecified in every direction: no feature named, no metric, no family. That is the test. Work through the next three sections yourself before you peek at the sample dialogue in section 5.

2. Step One: Generate Options, Then Choose Deliberately

"Design something" questions reward candidates who show a menu before they order. Thirty seconds spent laying out options signals breadth. Picking one quickly right after signals decisiveness. You need both. Here are the candidate options, sorted by family (per The Three Question Families):

1. Predictive: delivery-time (ETA) prediction, restaurant and dish recommendations on the home screen, dynamic courier assignment, and fraud detection on refund claims.
2. Generative: a support assistant for "where is my order?" and refund questions, AI-written dish descriptions, and review summarization ("what do people say about this place?").
3. Agentic: an ordering assistant that takes "get me dinner for four under $60" and places the order.

Any of these could fill an hour well. The real scoping skill is picking one and saying why. Pick for impact plus tractability, meaning how realistic it actually is to build well with the data and time you have. ETA prediction touches every single order, and it comes with abundant labels already sitting in the actual delivery times. That makes it the strongest predictive pick. The support assistant is the strongest generative pick: support cost is a known pain, the knowledge it needs (order status and policies) is structured and internal, and the stakes stay manageable. The agent is the weakest opening pick. It stacks payment actions on top of unsolved trust questions. A strong candidate says exactly that, in one sentence. Saying it actually works in your favor: it shows the interviewer you can see where the real risk sits, not just where the excitement is.

3. Step Two: Run the Framework Opening

Suppose we go with the support assistant. Before a single box gets drawn, the framework demands numbers. Practice producing them:

1. Quality. What is the actual metric? Resolution rate without human escalation is the primary one. Customer satisfaction on bot-handled tickets guards against "resolved" secretly meaning "the customer gave up." Then put a number on it. The current human baseline handles tickets at $4 each. The bot should fully resolve 50%+ of tier-1 tickets ("where is my order," "apply my coupon," "cancel this") at a satisfaction score no worse than humans get.
2. Latency. This is conversational, not a feed. Replies stream out word by word, so the number that actually matters is time to first token, the first small piece of text the user sees. That should arrive within about a second. Two seconds to a complete answer is fine, as long as it streams.
3. Scale and cost. Start from 500K orders/day. If 8% of those generate a support contact, that is 40K conversations/day. Model calls cost a few dollars per thousand, versus $4 per human ticket, so the economics are loudly in your favor. Say so out loud: "even at a cent per message we are two orders of magnitude under the human cost." That is exactly the kind of sentence interviewers remember.
4. Out of scope. Voice support, languages beyond English, and any bot-initiated refunds above a threshold (those escalate to a human). Scope cuts are design decisions. State them out loud, do not just quietly assume them.

4. Step Three: Classify, and Preview the Shape

Close the opening by naming the family and what it implies: "This is a generative system with retrieval over structured order data and policy documents, plus a small set of privileged actions (order lookup, refund under $10), which pushes it slightly toward agentic. The design centers will be retrieval and context assembly, cost control through model routing, and hard guardrails around the refund action." Ten minutes in, and the interviewer already knows you will not wander. You have also just named the exact modules this course spends the rest of its time teaching.

5. Sample Opening Dialogue

Here is how the first minutes sound when it goes well.

Interviewer: "Use AI to improve the customer experience. Design something."

Candidate: "Let me scope this, since the question is open. I see three directions: predictive systems like ETA prediction or restaurant recommendations, which train on our own behavioral data; a generative support assistant answering order and refund questions; or a full ordering agent. The agent is the most exciting and the least ready, it puts payments behind an autonomous loop, so I wouldn't lead with it. Between ETA and support, I'd pick the support assistant: support is a direct cost line we can measure against, and the knowledge it needs, order status and policy docs, is data we already control. Should I proceed with that, or would you rather see the ETA system?"

Interviewer: "Support assistant works."

Candidate: "Then success means: resolve at least half of tier-1 tickets end to end without human escalation, at customer satisfaction no worse than human agents, and at a cost per resolved ticket well under the roughly $4 a human contact costs. Latency is conversational: streaming, first token in about a second. Roughly 40K conversations a day if 8% of 500K orders generate contact. Out of scope: voice, non-English, and refunds above a small threshold, which escalate to humans. The system is generative with retrieval over order data and policies, so the core design problems are retrieval, cost routing, and guardrails around the refund action. Drawing the high level now..."

That is a complete, graded, ten-minute opening. Every sentence in it came from this module.

6. Now You

Now run the same three steps yourself, on these questions. Say it out loud, and time yourself at ten minutes. For each question, produce five things: options by family, a deliberate choice with a stated why, the numbers for quality, latency, scale, and cost, the scope cuts, and the family classification with its design centers.

1. "We're a music streaming service. Use AI to improve discovery."
2. "We're a bank. Leadership wants an AI feature for the mobile app."
3. "We run a job board. Make matching better with AI."

💡 The bank question is the trap of the three. The obvious pick is a financial chatbot. But it carries regulatory and hallucination stakes heavy enough to make it a worse opening choice than the predictive picks: fraud alerts, transaction categorization, and spend forecasting. If your own scoping step surfaced that trade-off without being told to, this module has done its job.

7. TL;DR