IDKMesh experiment: 25-verifier panel measured at effective size of 1.00

An open-source research project has published a measurement that challenges a core assumption behind AI-assisted code review: that adding more verifiers to a gate adds proportionally more confidence. In the project's headline experiment, a panel of 25 program-based verifiers performed no better than a single one of its members, because their errors turned out to be heavily correlated.

The project, IDKMesh, is described in a dev.to post by its author as an Apache-2.0 research effort, built for Python 3.11 and above, that studies how humans, AI agents, tools and heterogeneous compute can coordinate on uncertain goals. Its one-line thesis is that reviewer count is not independent evidence count. To make that testable, it ships a gate-audit command that takes verdict logs a team has already collected and reports what the panel is actually worth. A bundled example is explicitly labelled as synthetic, with five invented reviewers producing an effective vote count of 1.69, so that demonstrations of the arithmetic are never mistaken for measurements.

How the experiment worked

The observed result comes from experiment E017, which used no language models at all. Each of its 25 verifiers is a small program: a partial test oracle that draws inputs from one named region of a problem's input domain (tiny, small, large, extreme or duplicate) and accepts a candidate only if it matches a reference implementation on all of them. Five regions crossed with five seeds produced the 25 verifiers, which were run over a 72-candidate corpus whose ground truth came from executing hidden tests.

Individually, the verifiers were sound. Every one showed a significantly positive Youden J statistic after Bonferroni correction, with mean accuracy of 0.7956. The trouble appeared in how their errors lined up with one another.

What the audit found

Verifiers drawn from the same input region shared errors at a mean correlation of +0.8924, a dependency the project had already declared. The striking number sat elsewhere: verifiers from different regions, declared independent by design, still shared errors at a mean correlation of +0.5263. Across all pairs the figure was +0.5873. In other words, how verifiers are grouped on paper says little about whether their failures are actually independent.

The consequence showed up in the bottom line. The 25-verifier majority posted an error rate of 0.2083, marginally worse than a single verifier at 0.2044. The measured effective size of the panel was 1.00 out of 25 nominal votes. The standard correlation heuristic estimated 1.66, overstating the panel's worth by a factor of 1.66. According to the post, the entire measurement ran in roughly five seconds on one laptop.

Why the usual correction does not rescue it

The familiar fix is to discount for correlation using N_eff = N / (1 + (N-1)rho). A companion experiment, E015, tested that formula across a parameter grid and found it exact at correlations of 0 and 1, wrong in between, and inconsistent in the direction of its error. It errs on the conservative side where verifiers are weak, but on the optimistic side where verifiers are accurate and share modest dependence, which is precisely the regime teams care about. At accuracy 0.90 and correlation 0.125, the measured effective size saturates at 4.60 against the heuristic's asymptote of 8.00. Where a genuinely independent 9-verifier panel would reach a balanced error of 0.000891, the real panel delivered 0.0125, fourteen times worse.

What the study does not claim

The post is explicit about scope. No AI review panel was measured: E017's verifiers are programs, and whether LLM reviewers exhibit the same correlation structure is described as an open question, and the one the author most wants data on. IDKMesh is a research preview rather than production software, is not yet on PyPI, and must be installed from a repository clone. The author invites anyone holding verdict logs from a real review gate, whether human reviewers, LLM judges or CI checks, to run gate-audit on them, especially if the results disagree.

Why it matters

Teams increasingly respond to cheap code generation by stacking reviewers: several LLM judges on a pull request, more checks in the gate. This result suggests the arithmetic behind that stacking deserves suspicion. If LLM reviewers share blind spots through overlapping training data, similar prompts or common tooling, a large panel could deliver the effective confidence of a single reviewer while looking like twenty-five. The practical move is to measure panels rather than assume their independence, and this work argues that the standard correlation discount is least reliable exactly where accurate, modestly dependent verifiers live. The cost of checking is low: the audit itself runs in seconds.