What to Expect in the Factory AI System Design Interview

Expect a design round based on Factory's own product, not a generic social network question. Factory builds Droids, AI agents that take a ticket, change the code, run the tests, and open a pull request. Factory does not publish its design questions, and candidates have not reported them in public.

So the topics below come from the product itself. The likely areas are agent orchestration, code retrieval over a large repository, safe execution of agent-written code, evaluation of agents, and enterprise integration. Plan for about 60 minutes, which is typical for companies of this size.

The Question Types

Agent orchestration. Design the loop that turns a task into finished work: it plans, calls tools, reads results, and updates the plan. Factory's own Droid keeps a small set of tools and a written plan it updates as steps finish. The hard parts are long tasks, tool failures, and knowing when to stop.

Code retrieval. Design the system that finds the right files in a repository with millions of lines. Factory has said that it prefers precise retrieval over putting the whole codebase into the model at once. Expect to discuss indexing, search, and keeping the index up to date as code changes.

Safe execution. An agent runs commands and tests, so design the sandbox, the permissions, and the limits. A sandbox is an isolated environment where code runs with no access to production systems. Factory supports both local runs and remote runs, so the interviewer may ask how those differ.

Evaluation. How do you know a Droid did the task correctly? Factory has said it grades agents on task-based tests with written rubrics, not only on public benchmarks. Expect questions about test sets, regression checks before rollout, and measuring quality in production.

Enterprise integration. Droids receive work from tickets, Slack messages, and on-call alerts, through GitHub, Jira, Slack, and PagerDuty. Design the event flow, the queue, and the isolation between customer companies. Multi-tenant means many customers share one system safely.

What the Interviewer Grades

Practical judgment matters more than unusual parts. State requirements first, including scale and failure cases. Name your trade-offs out loud.

Connect each choice to the customer, because a wrong merge into a bank's codebase costs real money. Say how you would test the system. Factory itself will not let a Droid finish until the developer's tests pass.

A Walkthrough: From Ticket to Tested Pull Request

Here is a high level plan for the signature question.

1. Requirements (5 minutes). Many enterprise customers, each with private repositories, send tickets with a title and description. Output: a pull request whose tests pass. Targets: safe, traceable, and cheap enough to run on every ticket.

2. Intake. A webhook from the ticket system publishes an event, and a queue holds events so a sudden burst does not overload the workers. Each event carries the customer, the repository, and the permissions of the person who created the ticket.

3. Context retrieval. Index each repository: symbols, file summaries, and embeddings. An embedding is a list of numbers that represents the meaning of a text. Re-index on each merge so the agent never reads out-of-date code.

For a task, retrieve only the files that matter, and keep the token budget small.

4. The agent loop. Start a sandbox with the repository checked out. The agent writes a short plan, then edits files, runs the tests, reads failures, and repeats. Every command runs inside the sandbox with network and file limits.

Log every model input and output for audit.

5. Verification. The task is not done until the customer's own tests pass. Add a review step, where a second model call reads the diff against the ticket and flags risky changes. Then open the pull request with the plan and the test results attached.

6. Evaluation and rollout. Keep a set of real past tickets with known good fixes, and score every agent change against it before release. Track the merge rate and the human edit rate in production.

7. Scale and isolation. One sandbox per task, and separate storage per customer. Route each task to the model that fits its size and cost. Set a time and cost cap per task so a stuck agent stops.

Common Mistakes in This Round

• Starting with the model. The model is one box in the diagram. The retrieval, sandbox, verification, and audit around it are what the interview tests.
• Ignoring safety. An agent that runs commands can delete data or leak secrets. Permissions and sandboxing must appear early, before the interviewer asks.
• No evaluation story. If you cannot say how you measure whether the agent succeeded, the design is unfinished.
• Unbounded loops. An agent without a stop rule runs forever and spends money. Name the cap.
• Mixing tenants. Enterprise customers demand isolation. Say it in the requirements.

How to Prepare

• Learn the building blocks. Grokking the System Design Interview covers queues, caches, and databases, which every agent platform uses.
• Go deeper on hard cases. Grokking the Advanced System Design Interview helps with isolation, replication, and failure handling.
• Rehearse the walkthrough. Practice the seven steps above out loud in under 40 minutes.
• See the full loop. The design round is one part of the Factory AI interview process, next to the motivation question and the waiting and follow-up rules.