How Do You Design a Multi-Agent System?
Short answer: start by not building one. One competent agent with good tools and a bounded context solves most problems. Every agent you add multiplies cost, latency, and the number of places things can go wrong. Add agents for three reasons only: context isolation, parallelism, and specialization by configuration. When you do, use one of two shapes, orchestrator-workers or a pipeline of agentic stages, and treat every handoff as a contract.
When an interviewer asks whether you would use multiple agents, they are usually testing whether you can say no with real reasons.
Quick Overview
| Question | The answer that holds up |
|---|---|
| Default choice | One agent, good tools, bounded context |
| Reasons to add agents | Context isolation, parallelism, specialization |
| Shapes that ship | Orchestrator-workers, pipeline of agentic stages |
| Handoff carries | Task, minimal context, tool scope, budget, report schema |
| Where state lives | An external store the orchestrator owns, never inside agents |
| Typical cost | 10 to 20 times a single-agent task |
Why the default is one agent
Errors compound. A step that works 95 percent of the time fails often across twenty steps. Spreading those steps across agents makes it worse, not better. A coordinator that misreads one worker's report corrupts everything after it.
Debugging gets harder in the same proportion. A five-agent run means five times the trace to read, with none of the determinism you would want for that job.
The only three reasons that justify it
Context isolation. On a long task, accumulated history makes the model's judgment worse and burns tokens on every step. Delegating a subtask to a fresh agent with a clean, minimal context keeps each context focused. This, not raw capability, is the strongest argument for sub-agents.
Parallelism. Independent subtasks can fan out to concurrent workers and fan back in. Research five competitors. Review eight files. This is scatter-gather, with tokens instead of shards.
Specialization by configuration. Different subtasks want different system prompts, tool menus, and model tiers. A cheap model for summarizing, a stronger one for planning. In most production systems, "a different agent" honestly means "a different configuration".
The two shapes that cover production
Orchestrator-workers is the one to draw. A lead agent breaks the task into subtasks, hands each out with a defined scope, integrates the results, and owns the final outcome. Workers stay mostly stateless and bounded, with no memory carried between subtasks and a strict budget on each. They report back in a structured format: findings, confidence, artifacts. Free-prose reports are where integration quality gets lost.
A pipeline of agentic stages is the other. This is a workflow whose individual steps are small, bounded agents. A triage agent, then a resolution agent, then a check agent. It keeps a workflow's readability while still allowing judgment inside each stage.
Most production systems described as multi-agent are this second shape. Sequential handoffs, not a group of agents debating each other.
Two disciplines that make either shape work
Handoffs are contracts. Each delegation carries the task, the context the worker needs and nothing more, a tool scope, a budget, and an expected report schema. That is an API boundary, so engineer it like one.
State sits outside the agents. Task progress, artifacts, and spent budgets belong in a store the orchestrator owns. Then a crashed worker is retryable and the whole task is resumable instead of lost. This is ordinary durable-workflow discipline applied to agents.
What to reject by name
Free-form agent debate. Emergent role negotiation. Agent-to-agent chat used as the coordination mechanism. These make good demos, produce unbounded costs, and have no production evidence worth quoting. Saying that plainly is a senior signal, because the hype says otherwise.
One cost sentence is worth remembering. Orchestrator context, plus the context of every worker, plus the integration pass, routinely lands at 10 to 20 times a single-agent task. So parallel workers have to be buying real elapsed-time savings or real context freshness, something you can point to. Otherwise they are only extra cost.
How to Prepare
- Get the single-agent design right first. Everything here assumes it. Start with what is agentic system design.
- Practice the 30 second answer. Default to one agent, name the three reasons, pick a shape, describe the handoff contract, put state in an external store. Say it out loud until it is compact.
- Design the tool layer before the agent count. Most teams add agents to fix problems that were really tool design problems. See what is MCP (Model Context Protocol).
- Be ready for the cost question. Practice with how to scale an LLM-based application and designing LLM deployment architecture.
- Work the full chapter. Grokking the AI System Design Interview covers orchestration, memory, and agent evaluation, with a customer support agent capstone. The orchestration and multi-agent lesson goes deeper than this page can.
GET YOUR FREE
Coding Questions Catalog
$123
$197
$72
