What to Expect in the Gimlet Labs System Design Interview

Expect a design question about running AI inference across many kinds of hardware. Gimlet Labs does not publish its design questions, and candidate reports are scarce. The topics below come from the company's product and its job postings.

The product splits an AI workload into stages and runs each stage on the chip that suits it best. The postings ask for scheduling, batching, memory placement, queues, and failure modes. A design round of about 60 minutes is typical for infrastructure startups, not confirmed here.

You will not get a generic social network question. You will get a serving or scheduling problem, and performance judgment is graded with the architecture.

The Question Types

A model serving system. Design a service that takes requests, batches them, runs a large model, and returns tokens. A token is a piece of text the model produces one at a time. The hard parts are batching under mixed request sizes, the KV cache, and tail latency.

Tail latency is the time the slowest few percent of requests take. The KV cache holds the model's working memory for each active request and fills GPU memory fast.

A scheduler for mixed hardware. Design the system that decides which chip runs which stage. Prefill reads the input and needs raw compute. Decode produces output and needs memory bandwidth.

Tool calls, where an agent calls an external service, need fast networking. The hard parts are the cost of moving data between chips and what to do when one chip type is full.

A control plane for deployments. A control plane is the part that accepts a workload, places it, and tracks it. Design the API a customer uses to deploy a model, plus the state store behind it. The hard parts are consistency, retries, and safe rollout of a new model version.

Failure and recovery. Design for a GPU node dying in the middle of a long request. Expect follow-up questions on checkpoints, request replay, and how the scheduler learns a node is gone.

Measurement. The company sells speed per unit of power. Expect a question on how you would prove a change made the system faster without breaking correctness.

What the Interviewer Grades

Practical judgment matters more than unusual parts. State requirements first, including request mix, latency target, and hardware available. Name the bottleneck of each stage out loud: compute, memory, or network.

Say what moving data between chips costs. Connect choices to money: an idle accelerator is wasted spend. Defend each choice with a measurement you would take.

A Walkthrough: Design a Split Inference Service

Here is a high level plan for the likely signature question.

1. Requirements (5 minutes). Many customers send chat and agent requests. Target: low latency for short answers, high throughput for long ones. Two hardware pools: high-compute GPUs and high-bandwidth accelerators.

2. Split the work. Route prefill to the compute pool and decode to the bandwidth pool. Move the KV cache from one to the other after prefill ends. State the transfer cost and the request size below which the split no longer saves time.

3. Batching. Group requests by expected length. Add new requests to a running batch between decode steps, so short requests never wait for long ones.

4. The scheduler. Keep a live view of free memory on every node. Place each request where its KV cache fits. When the bandwidth pool is full, decode on the compute pool instead and record the cost.

5. Failure handling. Regular check-in messages, called heartbeats, detect a dead node. Requests that were running restart from the last saved state, or from prefill if none exists. The control plane marks the node out and rebalances.

6. Measurement. Track time to first token, tokens per second, and tokens per watt. Test every scheduler change against a replay of real traffic before rollout.

Common Mistakes in This Round

• Starting with the model. The model is one box in the diagram. The system around it, batching, placement, and recovery, is the interview.
• Ignoring transfer cost. Splitting a workload across chips is only worth it when the gain exceeds the cost of moving data. Say so early.
• Treating hardware as one pool. The whole product exists because chips differ. Name the bottleneck each chip type handles best.
• No measurement story. If you cannot say how you would prove the design is faster, the design is unfinished.
• Skipping multi-tenant isolation. Multi-tenant means many customers share one system. Their requests and caches must stay separate.

How to Prepare

• Learn the building blocks. Grokking the System Design Interview covers queues, caches, and schedulers, which every serving design uses.
• Go deeper on failure handling. Grokking the Advanced System Design Interview covers replication, consistency, and recovery under load.
• Rehearse the walkthrough. Practice the six steps above out loud in under 40 minutes, with a transfer cost estimate in step 2.
• See the full loop. The design round is one stage of the Gimlet Labs interview process, next to the motivation question. Wait times between rounds are in How long it takes to hear back.