Compounding Error: The Arithmetic of Reliability

Suppose each step of an agent succeeds 95 percent of the time, and suppose a run completes only if every step succeeds. The chance of completing a run is then 0.95 multiplied by itself once per step.

A step that fails once in 20 attempts sounds acceptable. A 10-step run built from such steps fails about 4 times in 10, and no step has become worse. A short formula shows how much of that loss a test and a retry can recover.

The Running Case

In the running Desk case, a customer reports a damaged item on one order and a duplicate charge on another. Suppose Desk needs 12 steps, well inside the limit of 25. At 95 percent per step, those 12 steps complete only about 54 percent of the time.

Detectors and Retries

A detector is any test that reports whether a step failed. It can be a code test that a tool result has the expected fields, a guardrail, verification against the payment system, or a second model call that reviews the first. The formula describes a detector by d, the share of failures it catches. When it catches one, the harness can run the step again. That second attempt is a retry, and r is the number of retries allowed.

// p: chance that one attempt succeeds     d: share of failures the detector catches
// r: retries allowed per step             n: steps in the run
stepSuccess = p * (1 + q + q ** 2 + ... + q ** r)   // where q = (1 - p) * d
runSuccess = stepSuccess ** n

The value q is the chance that an attempt fails and is noticed, the only path to another attempt. With one retry, the first line reduces to p + (1 - p) x d x p. The diagram follows one step with d of 0.8.

The two green endings add up to the step's success, 0.95 + 0.038 = 0.988. The larger red ending is the failure that nobody saw.

The gaps between designs grow with run length.

Where the Remaining Failures Go

Most of what remains after a retry is failure that the detector missed. With d of 0.8 and one retry, a step fails 0.012 of the time. Of that, 0.010 is failure the detector never saw (0.05 x 0.2). Only 0.002 is failure it saw that the retry did not repair (0.05 x 0.8 x 0.05). Five in six remaining failures are therefore silent, which means that nothing reports them.

A silent failure is also the worse kind. A noticed failure can go to an employee with the history attached, while a silent one continues through the remaining steps and may end in a confident reply to the customer. Retries can only repair what the detector reports, so more retries soon stop helping.

Unlimited retries add about one point, while raising d from 0.8 to 0.9 adds about four. Improve what notices the failure before you add more attempts.

Why the Formula Is Not a Forecast

The formula assumes that steps are independent, and the steps of an agent are not. Each step reads the earlier history, so an early mistake changes later ones. Suppose that at step 2 Desk opens the wrong one of the customer's two orders. The formula counts that as one failure with a chance of 5 percent. In the real run, each later tool call works, each step's own detector reports success, and the run refunds the wrong order.

Retries are not independent either. A retry with the same history may repeat the same wrong choice, so it is not a fresh 95 percent. An outage of the order system fails every lookup at once, which no per-step percentage describes. The formula can also err in the other direction, because the model can notice a contradiction at step 6 and repair a mistake from step 3.

So do not report the formula's output as an expected success rate. Use it to show how quickly a long run loses reliability and to compare designs under stated inputs. The real rate comes from running trials and counting, as the first eval set lesson shows.

What a Detector Costs

Verification is a detector, so it has its own miss rate and its own token cost. A read of the payment system can confirm that a refund exists, but not that it went to the right order.

Its token cost depends on where it runs. Under the standard inputs from the agent loop lesson, a 12-step run sends 190,800 tokens.

Where possible, test in code, because it uses no tokens and behaves the same on every run. At worst, one retry on each of the 12 steps makes 24 steps and 640,800 tokens, still inside the limit of 25.

A retry has one more cost when the step is a write. A repeated read is harmless, but a repeated refund can pay the customer twice unless the receiving system rejects the duplicate. Do not retry a write until the design can prevent a duplicate refund, which the durable state lesson covers.

A Reliability Budget for the 12-Step Case

A reliability budget is a target for the whole run, divided among the steps that must meet it. Suppose the team wants this case to complete 95 percent of the time. A uniform target would be 0.95^(1/12), or about 0.9957 per step. With p of 0.95 and one retry, every step would need a detector that catches about 96 percent of failures.

Not every step can have such a detector, so suppose the steps fall into three groups by how testable they are.

The run completes about 90.3 percent of the time (0.983 x 0.923 x 0.995), which misses the target. The three conclusion steps lose 7.7 points on their own, though the reads are more than twice as numerous.

Two inputs are optimistic. A d of 1 for the reads and writes ignores the wrong but valid order, which no test of a single step can see, so the real figures for those groups are lower. The retry on a write also assumes that the payment system rejects a repeated refund.

The longest bar belongs to the conclusions, a group of only 3 steps with the weakest detector. A group's d predicts its loss better than its number of steps.

Shortening the run helps little when the removed steps were already well tested. Replacing a conclusion with code helps more than either a retry or a shorter run, although a customer reply cannot be tested that simply. The table cannot say whether a d of 0.9 is achievable, so that is the next thing to measure.

When a Simpler Design Is Better

Each unneeded step removes one multiplication from the formula. A 3-step task with no detector completes about 86 percent of the time under the same inputs, against 54 percent for the 12-step case. A long run often improves most when part of it goes to code that needs no model.

A detector is also not worth adding when its failure is cheap and visible. Suppose an employee reads every draft before sending. That employee is already a detector, and a second model call adds tokens while catching little that the employee would miss.

Key Takeaways

• Steps multiply. Suppose 95 percent per step: 10 steps complete about 60 percent of the time, and 12 steps about 54 percent.
• With one retry, per-step success is p + (1 - p) x d x p. A detector that catches everything gives about 98 percent over 10 steps, and one that catches 80 percent gives about 89 percent.
• Retries repair only what the detector reports, so an improved detector usually adds more than a second retry.
• Steps are not independent, so the formula compares designs and does not forecast a real system.
• Verification is a detector with its own miss rate and token cost, and tests in harness code use no tokens.
• A reliability budget divides a target across groups of steps and shows which group loses the most.

Replacing a model decision with code helped more than adding attempts, so a larger question follows: should a task be an agent at all? Workflows vs Agents answers it next.

Practice Questions

1. Suppose each step succeeds 95 percent of the time. A team adds a detector that catches every failure, but it allows no retries. What is the chance that a 10-step run completes?

<details><summary>Show answer</summary>

It remains about 60 percent. A detector repairs nothing. Failed runs become visible and can go to an employee, but completion does not rise.

</details>

2. With a detector that catches 80 percent of failures and one retry, why does a second retry add only about one point over 10 steps?

<details><summary>Show answer</summary>

Most remaining failures are never reported. Of the 0.012 per-step failure, 0.010 is missed, and no retry follows a missed failure. Only the 0.002 can benefit from another attempt.

</details>

3. Each step of a 10-step Desk run has a code test that catches every wrong order number, and a run still refunds the wrong order. How can this happen when the formula predicts about 98 percent?

<details><summary>Show answer</summary>

The steps are not independent, and each test checks only its own step. If Desk chose the wrong one of two valid orders at step 2, that order number is correct, so each test passes and later steps build on the mistake. Only a test of the final outcome against the customer's request catches it.

</details>

4. Using this lesson's inputs, a 12-step run sends 190,800 tokens. Why does asking the model to verify 2 writes cost 57,000 more tokens, when harness code verifying the same writes costs none?

<details><summary>Show answer</summary>

Steps taken by the model join the history, and code outside the loop does not. The two extra steps make a 14-step run of 247,800 tokens, and each adds to what every later step sends. Harness code reads the payment system without calling the model.

</details>