LLM judges miss omissions in AI clinical notes unless the task is restructured

Omissions are the error that matters

Ambient AI scribes — systems that draft clinical notes from a visit automatically — are already in day-to-day use, and published audits converge on their most common defect: omission. Facts established during the encounter simply never appear in the note. The standard safeguard is an LLM judge: a second model reads the draft against the encounter transcript and flags problems. An arXiv preprint submitted on 31 August 2026 and surfaced on Hacker News's front page asks whether those judges can actually catch omissions, and the answer it reports is no — not with the task as it is usually posed.

A benchmark with a real answer key

Existing public corpora could not answer the question, the paper argues, because their clinician-written reference notes and their transcripts diverge enough that there is no clean ground truth. The authors instead built 500 single-error note pairs from audited fact sheets: 298 notes in which a specific fact is definitively missing, and 202 controls containing added or altered content. They then evaluated eight judge designs using paired discrimination — whether the judge ranks the flawed note below its clean twin, with 0.5 equivalent to a coin flip.

Judges verify presence, not absence

The gap the study documents is stark. On added or altered content, the eight designs score between 0.79 and 0.94. On omissions, they drop to between 0.50 and 0.63 — barely better than chance at best. In single-note mode, where a judge sees one note rather than a pair, no design flagged omissions any more reliably than it flagged problems in perfect notes, meaning the detection signal was indistinguishable from the false-alarm noise.

The obvious remedies did not help. Rewording prompts, voting across multiple runs and GEPA prompt optimisation shifted each judge's operating point along the sensitivity-versus-false-alarm curve, according to the paper, but none produced usable detection of missing facts.

Restructuring the task works

What recovered detection was changing the shape of the task: first list the facts the transcript establishes, then check the note for each one. Two independent implementations of that structure trade off against each other:

• A per-fact pipeline that verifies each established fact individually. Each flag identifies which fact is missing and how severe the gap is, at a 2.7% false-alarm rate.
• A GEPA-evolved prompt that performs both steps in a single call. It detects more omissions than the pipeline, 36.9% against 24.6% (p=0.002), at a 6.2% false-alarm rate and one-tenth the per-note cost.

A physician who is one of the authors validated 70 items and, in the ten cases where the two routes disagreed, sided with the pipeline every time (p=0.002). A second clinician, not an author, graded the severity rubric blind and matched the original gradings to within one severity grade.

Vendor notes and a stubborn edge case

The paper also tested the methods on real vendor notes drawn from a companion census. No threshold calibrated on the benchmark transferred directly, but a re-calibrated version of the single-call method still detected more omissions than the best of the eight original designs at half its false-alarm rate. One failure persisted across both recovered routes: omissions whose fact is restated somewhere else in the note. The authors have released the benchmark, prompts and judgements.

Why it matters

If omission is the most frequent error ambient scribes make, and LLM judges are the default quality gate for their output, then a systematic class of clinically relevant errors is passing review essentially unnoticed — not because the judges are noisy, but because the audit as usually constructed cannot register absence at all. The paper's practical lesson generalises: derive an independent enumeration of what should be present — here, the facts the transcript establishes — and audit against that list. Any pipeline that uses one model to check another's output, whether for summarisation, translation or code review, inherits the same blind spot unless the check is anchored to something other than the text being checked.