· via dev.to (home feed)
SWE-CoWork benchmark tests coding agents as collaborators across 101 real repositories
SWE-CoWork is a new benchmark that puts coding agents inside simulated organisations with 26 colleagues and 101 assignments across real repositories, testing collaboration and verification skills.

A new benchmark named SWE-CoWork sets out to measure something most coding-agent evaluations skip: how an agent behaves when the work is collaborative and the requirements are not written down anywhere convenient. According to a dev.to post summarising a paper on arXiv, the benchmark contains 101 development assignments spread across 101 real repositories, written in 11 programming languages and covering 12 domains.
What the benchmark contains
Agents work inside a simulated organisation rather than an empty repository. The organisation is staffed by 26 colleagues, and the information needed to do the job is deliberately distributed: instead of a single tidy issue description, the agent has to piece together what is wanted from conversations with its simulated coworkers and from whatever the organisation's records happen to contain.
That structure is the point. According to dev.to, the tasks come from real codebases rather than synthetic exercises, and an agent is expected not only to write code but to collaborate with the people around it and to verify its own work before treating a task as finished.
Why a benchmark like this was needed
Most established coding-agent benchmarks follow a familiar pattern, in the mould of SWE-bench: hand the agent an issue report in one repository, wait for a patch, and check whether the patch passes tests. That format is easy to score and has driven real progress, but it models software development as a solitary activity aimed at a fully specified problem.
Actual software work rarely looks like that. Requirements live in chat threads, in the memories of teammates, in half-updated documentation and in decisions nobody recorded. The dev.to summary argues that benchmarks testing only isolated code generation may overstate how prepared these systems are for workplace use, and that SWE-CoWork is built to close that gap by distributing requirements across people and records.
The design should also make visible a class of failure that patch-based benchmarks cannot see. An agent can produce technically correct code from a wrong or incomplete understanding of the task; it can fail to ask a clarifying question at the right moment; it can pass shallow checks without noticing that its verification proves little. According to dev.to, the expectation is that the benchmark exposes weaknesses in planning, communication and verification — precisely the skills that determine whether an agent is useful to a team rather than only to a test harness.
The realism question
Any simulated workplace stands or falls on how believable its inhabitants are. The dev.to analysis identifies the benchmark's main uncertainty: whether the 26 simulated colleagues behave realistically enough that an agent's score predicts how it would perform in a genuine organisation. Coworkers who are too cooperative, too available or too internally consistent would let agents succeed with habits that would not survive contact with real, distracted, occasionally contradictory humans.
The provenance of the details is also worth noting. The dev.to post carries a disclosure that it was produced with AI assistance and reviewed for accuracy, and it attributes the underlying work to a paper on arXiv, so the specifics are secondhand until checked against the paper itself.
Why it matters
If SWE-CoWork catches on, it changes what a good score means for a coding agent. Existing leaderboards largely reward solving well-specified problems alone; this benchmark rewards finding out what the problem actually is, working with others to solve it, and checking the result. For teams weighing agents for real deployment, that is the closer analogue to the job, and a plausible explanation for any gap between benchmark results and day-to-day usefulness. The outstanding question is whether the simulation can be kept faithful enough that the scores mean what they appear to mean.
- #ai
- #benchmark
- #coding-agents
- #llms
- #software-engineering