0% completed
Parallelization: Sectioning and Voting
On This Page
Two Forms of One Pattern
Sectioning: Three Guardrail Checks
Voting: Five Calls on One Question
Why Votes Are Not Independent
Related Parallel Patterns
When Parallel Calls Do Not Help
Key Takeaways
Practice Questions
Suppose a model call reads a code change and says whether it contains a security flaw. It flags a real flaw 70 percent of the time, and it wrongly flags a safe change 10 percent of the time. Now run the same call five times on the same change, and treat the change as flawed if at least a chosen number of the five calls flag it.
| Flags needed | Real flaw caught | Safe change wrongly flagged |
|---|---|---|
| One call alone | 70 percent | 10 percent |
| 1 of 5 | 99.8 percent | 41.0 percent |
| 2 of 5 | 96.9 percent | 8.1 percent |
| 3 of 5 | 83.7 percent | 0.9 percent |
| 5 of 5 | 16.8 percent | Less than 0.1 percent |
At 2 of 5, the design beats one call on both measures, for 5 times the tokens. However, the figures treat the five calls as independent, which is usually false. When do calls run at the same time improve a design, and what do the figures become without that assumption?
Two Forms of One Pattern
Parallelization runs several model calls at the same time, and your code combines their results. Anthropic's article on building effective agents names two forms. Sectioning divides a task into independent parts, and each call does a different part. Voting gives every call the same task and compares the answers.
Sectioning saves time and gives each job its own call, while voting raises confidence in one decision. The running example is Forge, the coding agent, after it has edited some files: three questions must be answered before your code allows the commit.
Notice the bottom of each side. On the left your code requires that every job pass, and on the right it compares the count of flags with a required number.
Sectioning: Three Guardrail Checks
Sectioning suits work whose parts do not need one another's results. Forge's three questions each read the same change, and none needs another's answer:
- Does the change write a password or a key into the code?
- Does the change remove or weaken an existing test?
- Does the change contain a security flaw?
Each is a guardrail, because a failure stops the commit. The pseudocode shows the two rules that a guardrail design must get right.
const results = await Promise.all([ callModel(SECRETS_CHECK, change), callModel(TESTS_CHECK, change), callModel(FLAW_CHECK, change), ]) // A call that timed out has no "pass" status, so it blocks the commit const blocked = results.filter((r) => r.status !== "pass") if (blocked.length > 0) return holdCommit(blocked) return commit(change)
First, the commit waits for all three results, because a check that finishes after the write cannot stop it. Second, a missing result counts as a block; otherwise a slow provider would silently turn off the guardrail.
The time saved equals the difference between the sum and the slowest call. Suppose the secrets check takes 2 seconds, the tests check 3, and the flaw check 6. Suppose also that the change is 2,000 tokens, and each check has 300 tokens of instructions.
| Design | Time | Tokens sent |
|---|---|---|
| Three calls, one after another | 2 + 3 + 6 = 11 seconds | 3 x 2,300 = 6,900 |
| Three calls at the same time | 6 seconds, the slowest call | 3 x 2,300 = 6,900 |
| One call asks all three questions | Suppose 6 seconds | 2,000 + 900 = 2,900 |
The token total does not change, so sectioning costs more than one combined call and needs a reason. Each separate call has one job, returns one clear result, and can be tested and changed alone. On keeping a guardrail apart from the main work, Anthropic's article says: "This tends to perform better than having the same LLM call handle both guardrails and the core response."
Each added guardrail raises the share of good work that is blocked. A commit proceeds only if all three checks pass. Suppose each check wrongly blocks a good change 5 percent of the time.
| Checks that must all pass | Calculation | Good changes wrongly held |
|---|---|---|
| 1 | 1 - 0.95 | 5 percent |
| 3 | 1 - 0.95^3 | About 14 percent |
| 6 | 1 - 0.95^6 | About 26 percent |
This is the compounding error multiplication, applied to false alarms. If alarms are frequent, people approve held changes without reading them, and the guardrail stops protecting anything. Each new check should state what it adds and how often it blocks good work.
Voting: Five Calls on One Question
Voting asks one question several times, then compares the number of flags with a threshold, which is the number of flags needed to treat the change as flawed. The flaw check is the hardest of Forge's questions, and a missed flaw is the most expensive failure, so votes are most useful there. Suppose Forge produces 1,000 changes, of which 10 have a real flaw and 990 are safe, and the voting calls are independent.
| Design | Real flaws caught | Safe changes wrongly flagged | Changes an employee must read | Tokens sent |
|---|---|---|---|---|
| One call | 7.0 | 99.0 | 106 | 2,300,000 |
| 5 calls, 2 flags needed | 9.7 | 80.6 | 90 | 11,500,000 |
| 5 calls, 3 flags needed | 8.4 | 8.5 | 17 | 11,500,000 |
Five votes send five times the tokens, while the time stays near that of the slowest call. The threshold, which your code states exactly, balances the two errors by their cost. If a missed flaw could expose customer data, 2 flags may be worth 90 readings. If the code also passes a human review later, 3 flags may be enough.
Votes need not run on every change. Suppose code picks out, from the file paths, the 100 changes that touch login or payment code, and runs five votes on those and one call on the other 900. The total is 900 x 2,300 + 100 x 11,500, which is 3,220,000 tokens, or 1.4 times the single-call design rather than 5 times.
Why Votes Are Not Independent
Five calls to the same model with the same instructions tend to make the same mistakes. The opening table treated them as five separate coin tosses. Real calls read the same text with the same trained model, so a flaw that one call cannot recognize, the other four often cannot either.
The calculation below changes only that assumption. Suppose each call flags 75 percent of real flaws 90 percent of the time, and the other 25 percent only 10 percent of the time. One call still catches 70 percent overall (0.75 x 0.9 + 0.25 x 0.1).
| Flags needed | Calls independent | Two kinds of flaw | Calculation for the last column |
|---|---|---|---|
| One call alone | 70 percent | 70 percent | 0.75 x 0.9 + 0.25 x 0.1 |
| 1 of 5 | 99.8 percent | 85.2 percent | 0.75 x 1.000 + 0.25 x 0.410 |
| 2 of 5 | 96.9 percent | 77.0 percent | 0.75 x 0.9995 + 0.25 x 0.081 |
| 3 of 5 | 83.7 percent | 74.6 percent | 0.75 x 0.991 + 0.25 x 0.009 |
The gain at 2 of 5 falls from 27 points to 7. Voting reduces random errors, and it does little for errors that come from what the model does not recognize. Neither column is a forecast; run the design on changes whose flaws you already know.
Notice that at 2 of 5, the votes raise the visible kind from 90 percent to 99.95 percent, while the hidden kind falls from 10 percent to 8.1 percent. No number of votes can catch a flaw that each call rarely recognizes.
Different instructions per call reduce the shared errors, as in Anthropic's example of several different prompts reviewing code. One call might look only at how user input reaches a database query, and another only at access checks. The design is then closer to sectioning, and often any single flag should count. A different model for some votes may also help, but test that on your own changes.
Related Parallel Patterns
Parallelization runs model calls at the same time, fork join runs agents, and parallel tool calls run tools. Azure's architecture guide calls this group concurrent orchestration and describes it with agents working on the same task.
When Parallel Calls Do Not Help
Use code for any check that code can perform. A search for strings that look like keys needs no model, costs no tokens, and always returns the same answer. Forge's secrets check should begin there, with a model call only for what the search cannot decide.
Run the checks in sequence when an early one often fails. Calls at the same time spend all their tokens even if the first result blocks the commit. If a cheap check blocks half of all changes, running it first saves the other calls on those changes, which may be worth the extra seconds.
Do not section work whose parts depend on one another. If the second call needs the first call's result, the design is a chain, and running such calls at once forces each to guess what the other will find. Many calls at once also count against your rate limit, the number of requests that a provider allows per minute.
Do not add votes when the call lacks information. If a reviewer call cannot see the function that a change depends on, five such calls share that gap. Giving one call the missing code helps more, for far fewer tokens.
Key Takeaways
- Parallelization runs model calls at the same time and combines the results in code. Sectioning gives each call a different job, and voting gives every call the same job.
- Running calls at the same time reduces time to that of the slowest call, but it does not reduce tokens.
- A guardrail design must wait for every result before the write, and it must treat a missing result as a block.
- Each added guardrail raises the share of good work that is blocked: three checks at 5 percent wrongly hold about 14 percent.
- Voting costs n times the tokens, and its threshold sets the balance between missed cases and false alarms.
- Votes share blind spots, so the independent arithmetic overstates the gain. Different instructions per call can reduce the shared errors.
Votes improve one decision, but they cannot explain what is wrong with a poor output or ask for a better one. The evaluator optimizer adds that feedback by repeating a generate and score cycle.
Practice Questions
1. Three guardrail checks take 2, 3, and 6 seconds and run at the same time. If you remove the 2-second check, how much time do you save?
<details><summary>Show answer</summary>None. Calls at the same time finish when the slowest one finishes, so the design still takes 6 seconds. Removing the 6-second check would reduce it to 3. Either removal saves tokens, because each call's tokens are spent whatever its duration.
</details>2. A team treats a timeout from a guardrail check as a pass, so that a slow provider does not delay its releases. What is the problem?
<details><summary>Show answer</summary>A provider failure silently removes the guardrail. During an outage, every change would pass that check unnoticed. A missing result should block the action or send it to an employee, because a guardrail stops the action unless the check has passed.
</details>3. Using this lesson's inputs, what share of good changes is wrongly held if four checks must all pass and each wrongly blocks 5 percent?
<details><summary>Show answer</summary>About 19 percent. A good change passes all four with a chance of 0.95^4, about 0.81. False alarms multiply like step failures, so each new guardrail needs a stated reason.
</details>4. Five identical calls catch a known set of flaws only slightly better than one call does. What should the team try before adding more votes?
<details><summary>Show answer</summary>Change what the calls can see or what they look for. A small gain suggests shared errors, which more identical votes would share too. Give the calls the missing code, or give each different instructions, then measure again on the same flaws.
</details>Reading Progress
0%
On This Page
Two Forms of One Pattern
Sectioning: Three Guardrail Checks
Voting: Five Calls on One Question
Why Votes Are Not Independent
Related Parallel Patterns
When Parallel Calls Do Not Help
Key Takeaways
Practice Questions