What to Expect in the Baseten System Design Interview

Expect a design round of about 60 minutes based on Baseten's own product. Baseten runs AI models in production for other companies. Running a trained model to produce outputs is called inference. Candidates report design questions taken from this work. Reported topics include serving a large language model under bursty traffic, scaling GPUs from queue depth, and finding the cause of a p99 latency spike. Do not expect a generic social network question. Expect a serving problem, where cost judgment is graded with the architecture.

The Question Types

Model serving under load. Design a service that serves a large language model with low latency during traffic spikes. The hard parts are batching and the KV cache. Batching groups several requests into one GPU call. The KV cache stores past computation so the model does not repeat it for each new token. Expect to discuss the trade-off between throughput and latency.

GPU autoscaling. Design the logic that adds and removes GPUs. A GPU is the processor that runs the model. Reported questions ask you to scale from queue depth and latency, not from CPU use. The hard parts are cold starts and cost. A cold start is the time to load a model onto a new GPU before it can serve.

Multi-model workflows. Baseten's Chains product links several models into one pipeline. Interview guides list a question that asks for a clean API and SDK for chaining model calls into one workflow. The hard parts are streaming between steps and independent scaling per step.

Deployments and rollbacks. Interview guides list a question that asks for the database schema and API that support zero-downtime model updates and rollbacks. Zero-downtime means users see no failed requests during the switch.

Latency debugging. Candidates report a question about a p99 spike while the median stays flat. Interview guides also list a latency spike between a client API and an asynchronous inference worker. Asynchronous means the request is queued and answered later. You walk through how you would find the cause.

Backend endpoints. Streaming responses, versioning, timeouts, and retries. These are the practical details Baseten's own API exposes.

What the Interviewer Grades

Practical judgment matters more than exotic parts. State requirements first, including traffic shape, latency target, and cost limit. Name trade-offs out loud. Connect each choice to money: an idle GPU costs real dollars every minute. Candidates report interviewers who ask follow-up questions until they reach real depth. Defend each choice with a reason and a number.

A Walkthrough: Design an LLM Serving Service

Here is a high level plan for the signature question.

1. Requirements (5 minutes). Many customers, each with private models and data. Traffic is bursty, with quiet hours and sudden spikes. Target: p99 latency under a set limit, at the lowest GPU cost.

2. Request path. A gateway receives requests, checks the API key, and applies rate limits. It places the request on a queue for the right model. A router sends work to a GPU replica that already has the model loaded.

3. Batching. The serving worker collects requests for a few milliseconds and runs them as one batch. Continuous batching adds new requests to a running batch as old ones finish. Stream tokens back to the client as they are produced.

4. Autoscaling. Watch queue depth and latency per model. Add replicas when the queue grows past a threshold. Remove them after a waiting period, so brief dips do not cause repeated scaling up and down. Keep model weights in a fast cache near the GPUs to shorten cold starts.

5. Deployments. Give each model version its own identity. Send a small share of traffic to a new version first. Keep the old version running until the new one is healthy. Rollback is a routing change, not a redeploy.

6. Observability. Record latency per stage: queue wait, model load, and generation. Alert on p99, not on averages. Log every request for cost tracking per customer.

7. Isolation. Separate each customer's models and data. Set a GPU budget per customer, so one customer's spike cannot take all the GPUs from another.

Common Mistakes in This Round

• Starting with the model. The model is one box in the diagram. The system around it, queues, batching, scaling, and rollout, is the interview.
• Ignoring cost. GPUs are the most expensive part. A design with no scale-down plan is unfinished.
• Scaling on the wrong signal. CPU use says little about a GPU service. Scale on queue depth and latency.
• No rollback story. A model update that cannot be reversed in seconds fails the deployment question.
• Averages instead of tails. Interviewers ask about p99. Have a plan for the slowest requests.

How to Prepare

• Learn the building blocks. Grokking the System Design Interview covers queues, caches, load balancers, and rate limiters, which every serving design uses.
• Go deeper on failure handling. Grokking the Advanced System Design Interview helps with replication, isolation, and partial failure.
• Rehearse the walkthrough. Practice the seven steps above out loud in under 40 minutes.
• See the full loop. The design round is one part of the Baseten interview process, next to the motivation question and the timeline for hearing back.