Capstone: Scoping an AI Design Question

The Exercise

In this capstone you practice turning a broad AI question into a clearly defined design problem, in the first ten minutes of an interview.

Do not draw the detailed architecture yet. Focus on choosing a useful feature, defining success, and agreeing on the scope.

Use this hypothetical question. The numbers are assumptions for the exercise, not reported figures for a particular company.

"We're a food-delivery company with about 25 million monthly users and 500,000 orders a day. Leadership wants us to use AI to improve the customer experience. Design something."

The question leaves three things unspecified: the feature, the success metric, and the type of AI system.

Work through the next three sections before reading the sample dialogue.

Step One: Generate Options, Then Choose

For an open question, briefly list several options before choosing one.

Spend about thirty seconds identifying useful possibilities. Then choose one and explain your reasons. This shows both that you considered alternatives and that you can define a manageable task.

Here are possible options, grouped using The Three Question Families.

Predictive: delivery-time prediction, restaurant and dish recommendations, fraud detection on refund claims, and predictions that support courier assignment. The final courier assignment may also require rules or optimization.

Generative: a support assistant that explains order status and refund policies, AI-written dish descriptions, and review summaries.

Agentic: an ordering assistant that handles a request such as "get me dinner for four under $60" and can place an order after the required checks and confirmation.

Each option could support a design discussion. Choose one based on its expected benefit, available data, and risks, and explain why.

Consider both impact and feasibility. Feasibility means whether the team can build and operate the feature with the available data, time, and resources.

Delivery-time prediction is a strong predictive option. It affects many orders, and recorded delivery times can provide training labels. Check whether those records are complete and reliable.

The support assistant is a strong generative option. Support cost is measurable. Order data and policy documents may already be available internally. Risks can be limited by restricting actions and transferring uncertain cases to a person.

The ordering agent needs more controls. It can spend money and place unwanted orders. It may be a suitable project, but it introduces payment authorization, confirmation, and cancellation requirements.

There is no universal best choice. Explain which option you prefer under the exercise assumptions and what information could change your decision.

Step Two: Run the Framework Opening

Suppose we choose the support assistant.

Before drawing the architecture, use the framework to state measurable requirements. Treat the following numbers as assumptions to confirm with the interviewer.

1. Quality. Use correct resolution without human escalation as the primary metric.

Also measure customer satisfaction and repeated contacts about the same issue. A ticket being closed does not prove that the problem was solved.

Set an initial target of correctly resolving at least 50 percent of eligible tier-1 tickets, with satisfaction no worse than the human baseline.

Tier-1 means routine issues, such as order-status questions, coupon questions, and simple cancellation requests. Answering a policy question is different from applying a coupon or cancelling an order; actions need separate permission and policy checks.

2. Latency. The assistant can stream its response, showing text as it is generated.

For this exercise, assume a p95 time to first token of one second and a p95 completion time of five seconds for short answers. Time to first token is the delay before the first visible output. A p95 target means 95 percent of measured responses meet the limit. Tool calls and longer answers may need separate targets.

3. Scale and cost. If 8 percent of 500,000 daily orders generate one support conversation, that is:

500,000 * 0.08 = 40,000 conversations per day

This is an average-volume estimate. Peak traffic and repeat contacts need separate estimates.

Compare costs using the same unit. If a conversation requires ten model calls and each call costs an assumed 0.01, model usage costs 0.10 per conversation. At 40,000 conversations per day, that is $4,000 per day in model costs alone.

Assume a human-handled ticket costs $4. Do not compare one model call directly with a complete human-handled ticket. Include retrieval, other infrastructure, retries, and human escalation. Measure total support cost per correctly resolved issue before claiming savings.

4. Out of scope. Voice support and languages other than English are excluded from the first version. Refunds above an assumed $10 limit require a human. Refunds within the limit still require identity, eligibility, and duplicate-payment checks outside the model.

State these limits explicitly and confirm them with the interviewer.

Step Three: Classify, and Preview the Shape

Finish the opening by naming the system family and the main design questions.

"This is mainly a generative assistant using authorized order lookups and policy retrieval. If the model can select a refund tool, it also has an agentic component. Read-only lookups and payment actions need different controls. I will focus on accurate retrieval, output evaluation, inference cost, and server-side checks for refunds up to the agreed $10 limit."

The interviewer now knows what you plan to design and which decisions need the most attention.

Sample Opening Dialogue