Two MMLU scores for one model family show why benchmark names hide implementation gaps

Two numbers, one benchmark name

A blog post by Dmitrii Zatona, featured on the Hacker News front page, takes apart a deceptively ordinary table: two evaluation records sitting side by side, one reporting an MMLU accuracy of 0.781 for build 42 of a model family, the other 0.79 for build 44 of the same family. Both claims declare the same provider, benchmark identifier, metric and unit, and the arithmetic difference is +0.009. The pair is drawn from the test vectors of a verification crate, but every ingredient will be recognisable to anyone who reads model leaderboards.

Both records are structurally valid. Each, however, carries a pointer to an evaluation "frame" — a hashed object describing how the measurement was run — and the two frames do not match. They declare different runners, graders and dataset splits. When Zatona asks a verifier from the apl-ai-eval crate to compute the delta between the two scores, it returns "incomparable".

What the MMLU label leaves open

According to the post, the benchmark identifier fixes a name and almost nothing else. Five variables stay open, and where published measurements exist, each one is worth points of accuracy, not thousandths.

The first is the dataset split. The original MMLU paper by Hendrycks et al. defines a development set of 5 questions per subject across 57 subjects, a validation set of 1,540 questions and a test set of 14,079. The Hugging Face cais/mmlu dataset that most runners actually load reports a test set of 14,042, validation of 1,531 and dev of 285. "The MMLU test set" therefore names two differently sized objects, and Zatona found no reviewed document explaining the difference. His example is starker still: frame A scores on the 285-question dev set, which the paper defines as the source of its few-shot examples, while frame B scores on a split labelled "test-lite" — a name Zatona could not trace to any published artifact, and which is not tinyMMLU. The label signals that a private slice was used; it does not say which questions.

The second is the implementation. A June 2023 Hugging Face post on the Open LLM Leaderboard measured the cost directly. Three harnesses — HELM, the Eleuther harness and the original code — ran the same dataset, all 5-shot, and scored llama-65b at 0.637, 0.488 and 0.636, and falcon-40b at 0.571, 0.527 and 0.558. The mechanism is scoring: one harness compares the probabilities of the four answer letters, another generates the next token and compares it to expected text, the third scores the full answer sequence. Rank order moves with it, placing falcon-40b above llama-65b under one harness and below it under the other two. Zatona adds that lm-evaluation-harness treats mmlu, mmlu_continuation and mmlu_generative as three distinct tasks over the same data, and that "5-shot MMLU" is a command-line flag rather than a property of the task, since the few-shot default is zero.

The third is the prompt format. Anthropic reported in 2023 that formatting alone — option labels, parentheses, an extra space before the answer — moves MMLU accuracy by roughly 5%. Answer position moves more: in a study by Zheng et al., shifting correct answers to position D dropped gpt-3.5-turbo from 67.2 to 60.9, while shifting them to position A lifted llama-30b by 15.2 points to 68.2, past gpt-3.5-turbo's 65.3 and reversing an original order of 53.1 to 67.2. In Zatona's pair both frames declare the same prompt protocol, zero-shot-mcq-v1 — a variable that is held, but only visibly held because it is written down.

The fourth is the grader. Frame A grades with exact-match-v1; frame B with llm-judge-v3. Those are not two implementations of one function. Zheng et al. found that judge models exhibit strong position bias, with only GPT-4 consistent in more than 60% of cases — 65.0%, against 46.2% for GPT-3.5 and 23.8% for Claude-v1 — and documented judgments that flip when two responses swap positions.

The fifth open variable, which the post lists without a published measurement, is the runner's network access.

Comparability lives in the reference, not the number

Zatona's central argument is that comparability is a property of the reference a result is traceable to, not of the number itself. Under the APL AI-Eval profile, the frame is a content-addressed object and every claim carries its hash; the two claims here point at different hashes, and under canonical serialisation even a subset value of "all" and an omitted key count as different scopes. A verdict of structural validity, he notes, says nothing about whether either score is actually correct.

Why it matters

Anyone subtracting benchmark scores across papers, leaderboards or model cards is often comparing numbers produced under different splits, harnesses, prompt protocols and graders — and the published evidence cited above says those differences are measured in whole points. The +0.009 delta in the post implies a precision the underlying records do not support: it pits dev-set exact-match scoring against a private slice graded by an LLM judge. The remedy Zatona points toward is provenance: pin the split, implementation, prompt protocol, grader and code commit into a hashed, resolvable frame, so that a claim of "same benchmark" can be checked rather than assumed.