deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

Meta's SWE-sweep benchmark makes coding agents find bugs without hints, and they mostly fail

Meta researchers released SWE-sweep, a benchmark where coding agents must locate and fix bugs in real repositories without being told what is broken. The best model tested resolved 4.7 percent of 4,068 bugs.

Meta's SWE-sweep benchmark makes coding agents find bugs without hints, and they mostly fail

Meta researchers have released SWE-sweep, a benchmark that inverts the usual way coding agents are graded. Instead of supplying the agent with a bug report and asking for a fix, it puts the agent inside a real repository and leaves it to work out what is broken, where the defect lives, and what category of problem it even is. According to a dev.to write-up, the team behind it includes two of the original creators of SWE-bench, the benchmark SWE-sweep is explicitly built to go beyond.

How the benchmark works

Most existing evaluations, SWE-bench included, give the model an issue description that effectively specifies the failure. The agent's job is then a slimmed-down localisation step plus repair. SWE-sweep strips that scaffolding away. Per the dev.to post, the agent receives a repository and nothing else: no indication of which file is at fault, no symptom description, no bug category. It has to surface defects on its own and then patch them. Repairs are checked against hidden tests, so a submission only counts when the fixed code passes verification the agent cannot see.

The results so far

The benchmark spans 4,068 known bugs across 100 repositories written in 22 programming languages, as reported by dev.to. The strongest model tested resolved just 4.7 percent of them. That figure is the headline: models that look competent when a bug report is provided struggle badly once discovering the defect becomes their responsibility. The write-up's reading is that locating problems, rather than writing patches, is the dominant failure mode, which reframes where the remaining difficulty in autonomous coding actually sits.

Availability and caveats

The accompanying paper and project site went live on 1 October 2026, according to the dev.to post, which says the full results are published there. The code lives in a public GitHub repository under Meta's facebookresearch organization. At the time of the write-up the repository had drawn roughly 246 stars but no tagged release, and no licence had been declared. The absent licence is worth flagging for anyone hoping to integrate the benchmark into a CI pipeline or evaluation harness, since it leaves the terms of reuse unclear until Meta clarifies them.

Why it matters

Agentic coding tools are increasingly positioned as autonomous maintainers that can triage and repair issues across large codebases, yet most public benchmarks still quietly perform the triage step for the model by handing over a description of the defect. SWE-sweep closes that gap by measuring the half of the job that current evaluations skip, and the early numbers suggest it is the harder half by a wide margin. For teams building or evaluating coding agents, a 4.7 percent success rate is a useful corrective to demo-driven optimism: an agent that can only repair problems it has been told about is closer to a patch generator than an engineer. The benchmark also gives the field a cleaner yardstick for progress, because scores on it can only improve through genuinely better discovery and debugging behaviour rather than sharper reading of hints. Watch this one if you care about how capable coding agents really are without training wheels.

  • #meta
  • #ai-agents
  • #coding-agents
  • #benchmark
  • #software-testing

Related posts