Five Qwen 27B agents in a file-passing workflow edge Fable 5 on a LiveCodeBench Hard subset

What happened

A GitHub project called Memdoor has published a multi-agent workflow for competitive programming in which five Qwen3.8-27B models — a planner, two ideators, a solver and a manager, plus a repair worker — collaborate by passing files on disk. According to the project's README, the workflow reimplements the GVS5H method (Lee, 2026), in which five Qwen3.8-27B models matched a frontier model on LiveCodeBench Hard.

Rebuilt with the paper authors' exact prompts and loop, the Memdoor version scored a mean of 9.8 out of 10 over five passes on the first ten LiveCodeBench Hard problems, against 9.6 for the authors' own harness on the same model and 8.0 for Fable 5 called once per problem. The README adds that an AI coder wrote the workflow files from a description of the method; nobody hand-wrote them.

How the team works

The pipeline runs plan, two parallel ideators, solve, test, review, repair and verify. Every agent starts from an empty context and is told to work from the files rather than chat history. The planner reads the task and writes a plan plus a notes.md ledger with one anchor per ideator; ideator A must propose the most direct provable algorithm, ideator B one from a different family of techniques. The solver picks between the proposals, records the decision in the ledger and writes solution.py. A command step runs the sample tests and captures the output; the manager reviews that captured output — not a model's claim about it — and either finds nothing to repair or leaves instructions. A final verify step re-runs the samples and fails the run on a nonzero exit code.

There is one repair round and no human gate. Success is decided by a shell script, never by a model. Following the paper, the README credits the ledger discipline for the gain. Each step is a single YAML file declaring what it reads and writes, executed by an engine called mario, and the steps name no model — any pinned endpoint works, including local ones.

Rebuilding the paper's harness

A parity test replays both the authors' original solver and the Memdoor workflow against a scripted model and compares every request; the README reports four cases, all identical, with a one-character change breaking the match. Sampling settings match the paper: temperatures of 0.3 for planning, 0.4 for ideation and 0.2 for manager and worker, top_p 0.95, top_k 20, a 128K reply cap and up to ten rounds.

The host differs — the authors served the model on their own vLLM in FP8, the rebuild through OpenRouter across FP8 providers. One mismatch mattered more: the first version wrapped each problem in its own task file, and the 27B model planned worse from it; on one problem the plan went from 0 of 2 to 2 of 2 once the exact LiveCodeBench text was used. Four engine changes were needed before the workflow could ask exactly what the authors' harness asks.

Results, cost and where it lost

49 of the 50 runs solved their problem, and all 50 are published with every request and reply, grader output, per-call cost, serving host and discarded attempt. The runs cost $35.79 in total — 15.3 million output tokens — or $38.77 counting sixteen attempts cut by a clock or stopped for a fix. A median run took 40 minutes of model time and $0.39.

The single miss is diagnosed: the manager invented a counterexample with a wrong expected answer, and the team rewrote a correct program into one that still passed the samples, which do not cover that case. In two other runs a program passing every hidden test was overwritten when the manager failed to declare done; both recovered before the loop ended, at $3.44 and $4.24, the two most expensive runs. The README argues both failure modes are fixable without changing the method — check a counterexample by brute force on small inputs, keep the last program that passed the samples — and neither fix is in the published numbers.

Caveats

The evaluation covers ten problems, five passes each, not the paper's hundred. The contests date from 29 March to 6 April 2025, so contamination from training data cannot be ruled out. Fable 5's 8.0 comes from the authors' own regraded result files as a single call per problem; it was not run inside this harness.

Why it matters

This is a small, checkable data point for agentic coding: a modest open-weights model, arranged as a file-passing team with a script as the arbiter of truth, beat a frontier model's single-call score on a hard benchmark subset, with every run published so the claim can be audited. The failure analysis is as useful as the score — the system broke in review, not in writing code, by hallucinating counterexamples and discarding correct solutions. And because the harness is prompts plus YAML steps that accept any model endpoint, it is a reproducible template, not just a benchmark claim.