Study of 16,893 agent runs shows Claude, Codex and Cursor often disagree on tool choices

What the study measured

Growth-marketing firm Armature has published the results of a large experiment on how coding agents choose third-party tools. The team ran 16,893 sessions in which Claude Code, Codex and Cursor were asked to find and actually implement a service inside a codebase, rather than merely recommend one. The sessions spanned 75 synthetic repositories written in 10 languages, 1,163 prompt variations and four user personas, ranging from non-technical "vibe coders" who describe only symptoms to enterprise engineers who spell out compliance and procurement constraints.

Armature is candid about its commercial angle: the company sells growth services to developer-tool vendors, and the research feeds its work on how to get products picked by agents. It has, however, also published the underlying traces, including user prompts, the agents' reasoning and the code changes they applied.

The stakes, as Armature frames them, come from a statistic it attributes to Vercel: in April, more than 30% of deployments were initiated by coding agents, a figure Vercel said had grown 1000% in six months.

How the experiment was built

To create realistic testbeds, Armature first profiled thousands of public GitHub repositories for languages, frameworks, third-party services and team sizes, then reweighted the statistics to offset the fact that open-source code skews toward startups. Coding agents then generated 75 repositories matching that distribution, complete with invented company names, fabricated git histories and real lockfiles verified against registries such as npm. Variants of each repository stripped out existing integrations so that no provider was pre-selected.

Each task was posed by one of the four personas, and in roughly 20-25% of cases the prompt added explicit constraints such as cost or expected usage volume. Every run took place in an ephemeral sandbox, rotated across three providers — E2B, Blaxel and Daytona — after Armature verified that the sandbox choice did not change the conclusions.

Two additional pieces of automation shaped the results. A "simulated human," played by Gemini 3.7 Flash, stood in for the user and typically approved the agent's leading recommendation before implementation; Armature found that agents told to implement immediately, with no chance to ask permission, leaned toward building everything in-house. A second Gemini 3.7 Flash instance then judged each session, discarding runs it considered biased or unresolved and recording which vendors were mentioned and which one actually won, using both the conversation and the applied diffs.

Of the 16,893 runs, 5,292 across 51 codebases and 18 sectors passed validation for the first wave of published results. Armature says the remaining 10,000-plus sessions may be released later.

The agents disagree more than they agree

According to Armature, all three agents landed on the same tool in only 42% of comparable decision points. In the voice-agents category, for example, Claude Code chose Twilio while Codex went with OpenAI's Realtime API; the published excerpt cuts off before naming Cursor's pick.

Their research habits also diverged sharply:

• Cursor grounded its decisions in web searches in about two-thirds of its sessions.
• Codex searched the web in 94% of sessions, and in roughly nine out of ten queries it applied operators like site: to confine results to trusted domains or a single vendor's documentation.
• Claude Code relied mostly on its internal knowledge, searching the web in only about 30% of sessions. When it did search, it browsed roughly three times as many pages as Codex, and in newer categories such as sandboxing, where its priors are weaker, it searched around 80% of the time.

The simulated human in the loop also altered outcomes: in object-storage tasks, Cloudflare R2 began winning sessions that had previously gone to Amazon S3. And an earlier, informal version of the experiment — two sandboxes, different agents, personas and codebases — produced the same database recommendation, Neon, both times, which is what prompted the scaled-up study.

Why it matters

If agents increasingly make tool selections, their research habits become the distribution funnel. A developer whose agent rarely searches is effectively querying that model's training data; one whose agent searches aggressively gets whatever its query patterns surface. Neither is the same as an objective comparison, and the 42% agreement rate suggests the "best" tool often depends on which agent you happen to use.

For developer-tool vendors, the implications are commercial: visibility now has to be engineered for machine readers, through documentation, comparison pages and registry metadata that either show up in an agent's searches or live in its priors. Armature's own business model makes that explicit, and its study doubles as a preview of an emerging industry built around influencing agent decisions.

The caveats deserve weight. The data comes from a firm with a stated interest in shaping agent choices, only about a third of the runs passed the judge's validation, and the users were simulated rather than real. The mitigating factor is that Armature released the full traces, so independent analysts can re-examine the leaderboards and methodology rather than take the summaries on faith.