What Changes When the System Learns

The request path explains how the system responds. The feedback process explains how you check and improve those responses.

Broken Assumption 3: Compute Is Cheap

Do not assume that a model call has the same cost or response time as a database read.

Inference cost depends on model size, input and output length, hardware, utilization, and the hosting provider. Some models run efficiently on CPUs; others need substantial GPU capacity. Measure or state your assumptions.

For a simple example, assume 10 million requests per day and a model cost of $0.01 per request:

10,000,000 * $0.01 = $100,000 per day

Over a 30-day month, that is $3 million for model calls alone. This is an illustrative calculation, not a quoted price.

Use a more expensive model only when the improvement justifies its cost.

Three options to evaluate are:

• Cache reusable results when the user's permissions, data freshness, and request context allow it. Similar wording does not always mean the same answer is safe to reuse.
• Send requests to a smaller model when it meets the quality target, and use a larger model when necessary.
• Compute results in background jobs when users do not need a fresh result for every request.

For example, some recommendations can be prepared overnight. Other products need more recent information. Choose based on the freshness requirement.

In the interview, explain the calculation and the conditions under which each optimization helps.

Broken Assumption 4: Failures Are Loud

Operational failures can be easy to detect: an HTTP 500 server error, a timeout, or a growing queue.

An incorrect model output may still arrive with HTTP 200 OK. That status means the request succeeded, not that the answer is correct.

Examples include an invented citation, a fraud score that misses a new attack, or harmful generated text. Error-rate monitoring alone may not detect these problems. Non-AI systems can also fail silently, so this is an additional concern rather than a completely new one.

Use several safety and control measures:

• Treat user input and retrieved content as untrusted. Prompt injection is an attempt to make a model follow unauthorized instructions. Input checks alone cannot reliably prevent it; enforce permissions outside the model.
• Check outputs against relevant rules and evidence. Filters can reduce risk, but they do not catch every mistake.
• Require human approval for actions whose risk or impact justifies it.
• Provide a fallback when the model is unavailable, too slow, or unable to answer reliably. Examples include non-personalized results or transfer to a support agent.

Ask: "Who or what could be affected by an incorrect output, and what limits that effect?"

Build the Model or Call an API?

Another important decision is how you will obtain and run the model.

For generative products, such as text or code assistants, an existing model may meet the requirements without training a new one from the beginning.

Consider three options. They can overlap: for example, you can fine-tune a model and then use managed hosting.

1. Call a hosted API. You send requests to a provider's model and pay according to its pricing, often based on usage.

This can reduce launch time and infrastructure work. You still need engineering work for integration, evaluation, and monitoring.

Check data handling, rate limits, availability, and total cost. A hosted API is not always the most expensive option; self-hosting also has hardware and staffing costs.

2. Self-host an open-weights model. Open weights means the learned parameters are available for download, subject to the model's license and access terms.

Self-hosting gives you more control over deployment and data handling. It can be cost-effective at sufficient utilization, but savings are not guaranteed.

You are responsible for capacity, security, model updates, monitoring, and operational support. Depending on the model, this may include running GPUs.

3. Train or fine-tune your own. Training develops a model from data. Fine-tuning gives an existing model additional training for a particular task or behavior.

For ranking, fraud detection, and recommendations, your own interaction data can be important. You can use that data with an in-house system or a managed service; building everything yourself is not the only option.

Fine-tuning may also help a narrow generative task when evaluation shows a quality or cost benefit. It adds data preparation, training, and maintenance work.

💡 Compare the options using quality, data sensitivity, expected volume, latency, total cost, and team capability. Start with the simplest option that meets the requirements, and explain what evidence would make you change it.

In the Interview

The 30-second answer (to "how is designing this different from a normal service?"): "I would add explicit output-quality measurements, a feedback process for detecting and addressing quality changes, an inference cost and latency budget, and controls for incorrect outputs. I would also explain how to evaluate and release changes safely. The usual distributed-system concerns still apply."

Likely follow-ups:

1. "What do you mean by quality, concretely?" Give a metric suited to the task. For feeds, consider clicks together with longer-term satisfaction. For fraud, consider precision and recall at a chosen threshold. Precision measures the share of flagged cases that are actually fraud; recall measures the share of actual fraud cases detected. For support, measure correct resolution and customer satisfaction together.
2. "Would you fine-tune a model for this?" Compare alternatives first. Consider task requirements, available data, privacy, volume, and whether an existing model already meets the quality and cost targets.

TL;DR