SWE-Bench ProMax: best model solves 41.2% of large multilingual refactors

A paper published on arXiv introduces SWE-Bench ProMax, a benchmark built around large, multi-file, multilingual refactoring work, and the scores it produces are dramatically lower than the numbers vendors publicise. According to a dev.to analysis of the paper, the strongest model tested, GPT-5.2, resolves just 41.2% of the tasks — far from the roughly 90% figures that circulate on benchmarks such as SWE-bench Verified.

Why the old scores looked so good

The paper's audit of SWE-bench Verified, as summarised in the dev.to write-up, identifies three compounding problems with the industry-standard benchmark.

First, the tests themselves are unreliable. Among tasks that models failed to solve, nearly 60% had defects in the test suite: 35.5% were too strict, rejecting correct solutions because they differed from the original developer's approach, and 18.8% checked behaviour the task never asked for. The write-up also reports that OpenAI has already dropped the benchmark from its own evaluations.

Second, data contamination. SWE-bench Verified is drawn from real GitHub issues in Python projects, and the reference solutions — the gold patches committed by human developers — were public long before today's models were trained. The paper reportedly demonstrates that frontier models can reproduce those patches verbatim, which means some high scores reflect memorisation rather than reasoning.

Third, task size. 86% of the original benchmark's tasks involve editing a single file, whereas real engineering work routinely requires coordinated changes across a dozen or more files.

How ProMax differs

The new benchmark takes a heavier construction approach. It contains 170 tasks spanning seven languages — Python, Java, TypeScript, Go, C, C++ and Rust — rewritten from real commits rather than lifted from raw GitHub issues. The average task touches 11.4 files and 261.6 lines of code, and every test suite was reviewed by humans to weed out tests that were too narrow or too broad.

The hardest task in the set illustrates the scale being tested: an agent must migrate NASA's F' Prime flight-software framework from a monolithic header to a consolidated entry point, a change spanning 244 files across the codebase, while keeping runtime behaviour exactly the same.

What the models actually scored

When proprietary and open-weight frontier models were run against the benchmark, the results, as reported by dev.to, were:

GPT-5.2 topped the table at 41.2% resolved, at an average cost of $3.60 per task. Claude Sonnet 4.6 followed at 38.8%, costing $4.77 per task. GLM-5 and Qwen3.5 both resolved 36.5% of tasks, at $0.24 and $0.78 per task respectively.

Two findings stand out. Price does not buy proportional capability: Claude Sonnet 4.6 spends roughly twenty times more per task than the open-weight GLM-5 for about two percentage points of additional performance. The paper's conclusion, per the write-up, is that open models are closing in on frontier quality at a fraction of the cost.

The failure analysis is equally telling. Trajectory inspection found that when models fail, they tend to fail the same way: they edit fewer files than the reference solution requires, while running additional rounds of self-directed reasoning. The bottleneck is not intelligence per reasoning step but the ability to hold a coherent picture of an entire codebase across a long task.

Caveats before quoting 41.2%

The benchmark deliberately filters for hard work, so it measures a ceiling for difficult refactoring rather than everyday bug fixing — most developers still encounter single-file fixes far more often than 244-file migrations. At 170 tasks, the set is also small compared with other benchmarks, and the researchers themselves describe it as a starting point that needs expansion.

One further note of caution: the dev.to article states it was drafted by an AI system with human review, summarising the arXiv paper. Readers making decisions based on these numbers should consult the paper directly.

Why it matters

This benchmark lands as a reality check on AI coding claims at a moment when headline scores are routinely produced by the vendors themselves. It suggests that memorised public solutions and small, single-file tasks have flattered model performance, and that genuine long-horizon, multi-file engineering remains largely unsolved — the best model fails more often than it succeeds.

For teams choosing models, the cost-performance inversion is the practical signal: an open-weight option at $0.24 per task performs within a few points of models costing twenty times more, which makes benchmarking against your own codebase, rather than trusting vendor numbers, the sensible default. And for anyone tracking AI progress, the paper offers a simple three-question filter for any benchmark claim: is the test data fresh or at risk of contamination, who ran the measurement, and how large is the work compared with real engineering tasks.