Real-SWE benchmark: top coding agent succeeds on 38.8% of enterprise tasks

What Real-SWE measures

According to a post on dev.to, Specific Labs released Real-SWE in September 2026, a benchmark that evaluates frontier coding agents on private, licensed enterprise codebases rather than the public repositories that dominate most evaluations. The tasks model genuine production work — changes to billing logic, tax handling and migrations spanning multiple services — and scoring uses pass@1 averaged across eight runs per task.

Scale is a key differentiator, the post notes: reference solutions for Real-SWE tasks modify a median of 11 files, versus six for both FrontierCode and DeepSWE, pointing to broader and more entangled changes than public benchmarks typically capture.

How the agents scored

The strongest pairing of model and harness resolved 38.8% of tasks, the dev.to article reports, while every other configuration finished lower, with the weakest at 16.2%. Expressed the other way, even the best-performing setup delivers an incorrect result on roughly six out of ten assignments, and the rest fail more often.

Where the review burden lands

The author's central argument is that this failure rate, not pull-request length, determines how much review an agent-driven workflow requires. Producing a change may take minutes, but judging whether that change is correct — whether behavior holds across the dozen files it touched — still depends on a person who knows the system. With the leading setup wrong around 60% of the time, most agent output needs substantive correction, and corrections are not free. That dynamic explains why total review time can climb even as code arrives faster.

Suggested workflow adjustments

The post draws two practical recommendations from the numbers.

First, treat agent-generated pull requests as drafts rather than near-finished work. Reviewing a draft means actively hunting for assumptions the agent should never have made; reviewing something framed as complete invites skipping that scrutiny.

Second, track acceptance rates separately for each agent and each region of the codebase, then match review depth to those figures. Areas that consistently show low pass rates are where reviewer attention should concentrate, rather than spreading effort uniformly across every incoming change. The structure of the codebase, the author argues, matters more than raw pull-request volume for predicting where reviewer hours will burn.

Why it matters

Benchmarks built on public repositories tend to flatter coding agents because they rarely reflect the size and interconnectedness of real enterprise changes. Real-SWE, running against licensed private codebases with reference solutions touching nearly twice as many files, offers a sterner and more realistic success rate — and at 38.8% for the best setup, it quantifies the verification work that automation has not absorbed.

Teams adopting agents often assume the bottleneck shifts toward prompting or orchestration. This data suggests it shifts toward the senior reviewers who must catch the majority of wrong changes before merge. The value of the benchmark is less about ranking models than about sizing a cost: the human review load attached to every agent-generated change. For engineering leaders planning capacity, that single failure-rate figure may matter more than any leaderboard position.