Research Agent Reliability and Cost

60 tasks, 5 runs of each, 300 runs in all. On a single run, the upgrade research agent passes 71 percent of the time. It passed all five runs on 28 tasks and failed all five on 3. On the other 29 tasks, the result changed from run to run.

Each answer used a median of 62 thousand tokens and took a median of 81 seconds. The slowest 5 percent took more than 142 seconds. In 35 runs, the agent used its whole budget of 20 tool calls and still failed.

Those numbers describe one version of the agent on the 60 development tasks