What Changes When the System Learns

A candidate who draws only the top row is answering last decade's question.

4. Broken Assumption 3: Compute Is Cheap

A database read costs a fraction of a microcent, and returns in single-digit milliseconds. A large LLM call (a call to a large language model like the one behind ChatGPT) is a different animal: it can cost around a cent and take 1 to 5 seconds. Even a modest self-hosted model needs GPU time, and GPU time runs orders of magnitude more expensive than CPU serving. (An order of magnitude is just a factor of ten.) Do the multiplication: at 10 million requests a day, "just call the model every time" can turn into a $100,000-a-month line item, or worse.

The architectural consequences fill an entire module of this course, but the pattern behind all of it fits in one sentence: spend the expensive model only where it earns its cost. Think of the large model as a senior specialist. You send it only the cases that truly need one. In practice that means:

• Cache aggressively, for identical and near-identical requests.
• Route easy cases to small, cheap models and hard cases to large ones.
• Precompute offline whatever does not need to run per request. (Offline just means ahead of time, in background jobs, away from live traffic.)

For many products, a nightly batch of recommendations beats a per-request model call outright. And interviewers increasingly ask you to say a rough cost estimate out loud. We will practice that exact math throughout the course.

5. Broken Assumption 4: Failures Are Loud

A classic failure is visible. You see a 500 (the standard server-error code), a timeout, or a queue backing up. Model failures do not announce themselves that way. The service happily returns 200 OK, the all-good status code, sitting right next to a confidently wrong answer. Maybe it is a hallucinated citation: a reference the model just invented, delivered as if it were real. Maybe it is a fraud score that waves through a brand-new attack pattern. Maybe it is a toxic completion. No alert fires, because nothing crashed. Here is the image: a classic failure is a fire alarm. A model failure is a slow gas leak. Nothing rings, and the damage just builds.

The architectural consequence is a safety and control layer that classic systems never needed. It includes:

• Input validation against prompt injection: attacker text hidden in the input that tricks the model into following the attacker's instructions instead of yours.
• Output filters and guardrails: automatic checks that catch bad outputs before users see them.
• Human review queues for high-stakes actions.
• Graceful degradation: if the model is down or too slow, fall back to the non-personalized version instead of erroring out.

Make a habit of asking one question: "what is the blast radius when the model is wrong?" That single instinct is among the strongest senior signals you can show in this interview.

6. Build the Model or Call an API?

One more difference deserves its own section, because it reshapes the entire design. For GenAI systems (generative AI products that produce text, code, or images) you usually do not build the model at all. The real decision comes down to three options:

1. Call a hosted API from OpenAI, Anthropic, or Google. A hosted API means you send requests to the provider's model over the internet and pay per call. It ships the fastest, and it needs no ML team. But it carries the highest per-call cost, your data leaves your infrastructure, and you inherit the provider's rate limits and outages.
2. Self-host an open-weights model, something like the Llama or Mistral families. Open weights means the model's learned parameters are published, so anyone can download and run it. You get full control, and your data stays home. The marginal cost (the cost of each additional call) drops at high volume. But now you are running a GPU fleet, and you own its utilization, upgrades, and on-call. Think of it as renting a car versus buying one: renting costs more per trip, but someone else handles the maintenance.
3. Train or fine-tune your own. Fine-tuning means giving an existing model extra training on your own data. This is justified for classic ML systems: nobody sells a hosted "your feed ranking" API, and your data is the moat, the advantage nobody else can copy. It is also justified for narrow, high-volume GenAI tasks, where a small tuned model beats a large general one on cost.

💡 Here is a one-line rule that serves well in interviews: classic ML systems like ranking, fraud, and recommendations are almost always built in-house, because they train on proprietary data. GenAI systems almost always start on a hosted API and revisit at scale. Say the rule, then apply it to the question in front of you.

7. In the Interview

The 30-second answer (to "how is designing this different from a normal service?"): "Four things change. Quality becomes a first-class requirement next to latency and availability, so the system must measure its own output quality in production. Behavior degrades over time as data drifts, so I need a feedback loop that retrains and safely promotes models. Inference is expensive, seconds and real money per call, so I cache, route to cheap models, and precompute where possible. And failures are silent, a confident wrong answer returns 200 OK, so I add guardrails, fallbacks, and human review where stakes are high. Everything else is classic distributed systems."

Likely follow-ups:

1. "What do you mean by quality, concretely?" Have one metric ready per system type. Use CTR, click-through rate, or long-term engagement for feeds. Use precision and recall at a threshold for fraud: precision asks how much of what you flagged was truly fraud, and recall asks how much of the real fraud you caught. Use resolution rate without escalation for a support bot. The next module treats these properly.
2. "Would you fine-tune a model for this?" Apply the build-vs-buy rule above. Then name the deciding variables: volume, task narrowness, data sensitivity, and whether prompting a hosted model already meets the quality bar.

8. TL;DR