Brood War Bench: frontier AI agents still play StarCraft at beginner level

A new public benchmark has put frontier AI models in control of full StarCraft: Brood War games, and the headline finding is humbling: according to the Brood War Bench report, none of the models played beyond a beginner level. The report, dated September 19 and surfaced on the Hacker News front page, has agents play complete matches by issuing command batches. One configuration won every game it entered, yet the author's assessment is blunt: a human beginner executing a simple photon rush would win every single one of these games.

The benchmark grew out of a side project. The author built a version of Brood War that can only be played through agents, and noticed that friends with almost no StarCraft experience did surprisingly well simply by asking their agent to attack. That raised the question of how far the agents could go on their own, and the report is the answer.

What the leaderboard shows

Codex Astra at its highest effort setting topped the table with 18 wins and no losses, a 100% win rate, at an average of 12.6 actions per minute and $10.54 per game. Codex Astra at medium effort followed at 16-2, with Claude Fable third at 15-3. The rest of the top ten consisted of a lower-effort Codex Astra run, three variants of Codex 5.6, and Claude Opus 5 at 12-6, which at $20.78 per game was one of the costliest entrants.

Grok 4.6 sat at the bottom of the field; the report concludes that current Grok models are not yet capable of playing the game. Notably, within the Codex 5.6 Sol family, medium effort won 72.2% of games versus 61.1% for the maximum reasoning setting, which the author reads as evidence that in a real-time game, time spent thinking is time the opponent spends acting.

Games were short. Median game length was 8:10 for Codex Astra, 9:15 for Grok 4.6 and 10:37 for Claude Fable, with the largest share of Astra games, 54.9%, ending between the five- and ten-minute marks.

The tax on thinking

The report treats deliberation latency as the central weakness. Older models played the real-time strategy game as if it were turn-based and were destroyed while mid-thought. Newer models are far more conscious of the cost of reasoning, but still fall into the same trap.

Grok 4.6 is the extreme case. In one 43-minute game, its highest-effort run logged 11,138 reasoning tokens but issued just six command batches and never fielded a combat unit. In other games it produced three Marines, or two Zealots, and never crossed the map. The aggregate charts tell the same story: across pooled runs, Grok 4.6 averaged about seven workers alive, an army of roughly two units and more than 500 unspent minerals, while Codex Astra and Claude Fable sustained 15 to 17 workers and supply counts around 26 to 27.

Three models, three profiles

Codex models found harassment before they found macro. Their strongest recurring idea was sending a lone Probe across the map to attack workers, which worked largely because opposing agents froze for dozens of seconds deliberating about it. Sustained production was much weaker: delayed tech, basic units trickled into defended bases, and workers thrown into last stands. Codex also tended to spawn separate subagents for the economy, army production and army control, and these barely communicated, so the army agent fed units into attacks one at a time rather than massing for a planned timing, a classic beginner mistake. There were flashes of grit: after losing its army and main base in one game, Codex 5.6 Terra lifted its last Command Center to the far corner of the map and survived another six minutes.

Claude Fable came closest to simply playing the game. It built an economy and climbed the tech tree rather than stopping at the first unit available, winning one game after reaching Mutalisks and another after constructing a Robotics Facility, Citadel of Adun, Observatory and Templar Archives. Ambition did not guarantee execution, though: in a third game it reached a Factory and Academy before Claude Opus 5 overran it.

Why it matters

Real-time environments expose a failure mode that static benchmarks miss: the cost of thinking. In Brood War, latency is a first-class capability constraint, and a model that reasons brilliantly but acts sparsely loses to one that acts.

The results are also a caution about win rates as a signal. A 100% record here measures relative strength among weak agents, not competence; the author states that even the leaders could not build complex armies, defend simple attacks or execute concrete strategies.

For agent economics, the finding that cheaper, lower-effort settings sometimes beat maximum reasoning inverts the usual assumption that more deliberation is better. And the author considers the benchmark far from exhausted: general-purpose LLM agents still cannot sustain the observe-decide-act loop that real-time strategy demands, which leaves a great deal of headroom for the next generation of models.