Audit finds MMLU-Pro still grades 52 expert-flagged questions with MMLU's original wrong answer keys

MMLU-Pro, one of the most widely reported benchmarks for large language models, is still scoring dozens of questions against answer keys that human experts have already judged incorrect. According to an audit published on dev.to by Munim, Inc., 52 of 55 expert-flagged miskeyed questions that MMLU-Pro inherited from MMLU remain graded with MMLU's original key — 94.5% of the flagged items the audit could clearly match.

Two datasets, finally joined

MMLU-Pro consists of 12,032 questions with ten answer options each, 6,810 of them taken from MMLU. MMLU-Redux 2.0 had experts re-examine 100 MMLU test questions in each of MMLU's 57 subjects — 5,700 items in total — and marked 106 as having an incorrect published key, in most cases naming the option the experts considered correct. The Redux paper observed that MMLU's errors persist inside MMLU-Pro but gave no count or list, and MMLU-Pro's paper says its own review turned up 350 wrong answers among the MMLU-sourced questions, also without publishing them. According to the dev.to write-up, nobody had connected the two datasets before, so the audit matched them row by row.

How the audit matched them

Each Redux row was aligned to MMLU's test set by subject, question and all four options, which resolved 5,690 of the 5,700 rows. Each match was then searched among MMLU-Pro's MMLU-derived questions and kept only when at least three of MMLU's four options appear among the ten. The audit then checked whether MMLU-Pro's key still equals MMLU's, and whether the experts' preferred answer remains an available option. Six predictions were published before the analysis ran in an isolated container, and all six held, two only weakly. The write-up also discloses that the research was carried out by AI agents at Munim, and that the released outputs contain only counts, question ids, labels and answer letters — never question text.

What it counted

Of the 5,700 Redux rows, 2,829 — 49.6% — appear in MMLU-Pro. Among the miskeyed items, 55 produced a clear one-to-one match. MMLU-Pro changed the key on three of those; the other 52 still point to MMLU's original answer. In 24 of the 26 cases where the experts' answer resolves to a letter or option text, that answer is still among the ten options, so most fixes would mean editing a single letter in the key rather than writing a new question.

The affected items cluster in the sciences: 18 in chemistry, all from MMLU's college chemistry set; nine in health, eight of them virology; five in physics; four each in abstract algebra, computer science and other categories; two in psychology; and one each in law, biology, history, economics, philosophy and engineering. They represent 0.43% of MMLU-Pro, so headline scores barely move, but the density is higher in the slices where they gather, such as college chemistry.

Three corrections and a numbering puzzle

The three keys MMLU-Pro did change — question ids 2817, 8122 and 8780 — expose an odd ambiguity. Redux records the experts' answer for all three as a bare digit, and its documentation never states whether digits count from zero or from one. Read from zero, MMLU-Pro's new key matches the experts on all three; read from one, it matches on none. The audit argues the zero-based reading is the only coherent one across the 26 digit answers on flagged questions — under the one-based reading, at least seven resolve to '0', an option that does not exist, and eight point at the very key Redux rejected — while conceding the interpretation was settled after seeing the data. MMLU-Pro's changelog independently lists id 2817 among answers corrected in September 2024.

The difficulty filter missed them

MMLU-Pro pruned MMLU questions that most of eight small models answered correctly. The audit predicted flagged questions would survive that filter more often, because a model choosing the genuinely correct option gets marked wrong and the question therefore looks difficult. That held, but only marginally: 51.6% of flagged versus 49.6% of unflagged questions made the cut, a risk ratio of 1.04 that is not significant overall, although the within-subject odds ratio of 1.29 reaches p < 0.05. Either way, the filter did not remove the miskeys.

Why it matters

Benchmarks are only as trustworthy as their answer keys, and MMLU-Pro figures prominently in model launches and leaderboards. Comparisons that turn on thin margins in chemistry or virology are inheriting keys that domain experts have explicitly rejected. The count of 52 is a floor rather than a ceiling: Redux reviewed 5,700 of MMLU's 14,042 questions, so further inherited errors almost certainly sit outside the sample. The audit did not re-verify the experts' judgements — it joined existing datasets — so the final word on each key rests with subject specialists and MMLU-Pro's maintainers. The encouraging part is the cost of repair: for most flagged items, one letter in an answer key.