Five AI models largely agree on facts but split on two-thirds of judgment calls, study finds

What the study found

A controlled comparison of five leading AI models found broad agreement on factual questions and frequent substantive disagreement on questions requiring judgment, according to a write-up published by MultiO on dev.to. Of 24 identical questions sent to GPT, Claude, Gemini, Grok and Llama, the models materially agreed on only 8 (33%). The remaining 16 questions each contained at least one material divergence, 52 in total, and on 15 questions (63%) a single model stood completely alone — either making a claim no other model made or dissenting against all four others.

How it was run

MultiO assembled 24 questions in four groups: pricing and strategy decisions, factual questions containing a built-in trap, coding-approach choices, and recommendations involving tradeoffs. Each question went out unchanged to a mid-tier model from each of the five families. The analysis focused on substantive disagreement rather than differences in phrasing or style.

The splits followed a clear pattern by category. Factual questions were near-unanimous, averaging 0.33 divergences per question. Judgment-heavy categories split hard: recommendation questions averaged 3.17 divergences each, coding defaults 2.67, and pricing 2.50.

Two examples that capture the pattern

Asked whether an indie app earning $3,000 a month from a $30 one-time unlock should switch to a $5 monthly subscription, one model said yes unconditionally, two said no unless retention exceeded six months, and only one model ran the numbers: roughly 600 active subscribers would be needed just to match the old revenue, with a dip and backlash risk along the way. MultiO considers that lone calculation the most useful contribution among the five answers.

The second example cuts the other way. On the folk claim about swallowing spiders in your sleep, every model correctly called it false — but one added a confident origin story for the statistic. That story traces to a source that, per the authors, has never been shown to exist, making the lone embellishment the only wrong line in that group.

Across the full study there were 14 sole-source claims. MultiO reports that 8 of them (57%) turned out to be the highest-value contribution in their group, while 3 were wrong or invented. The outlier is sometimes the insight and sometimes the fabrication, and a single model, according to the authors, never flags which of its claims no other model would support.

As a control, the team re-ran two questions three times each through one model and saw no change in verdict, suggesting the disagreement lives between models rather than inside one model's sampling variance.

Caveats

This is a small, vendor-published experiment, and the authors disclose the relevant conflicts themselves. MultiO builds a multi-model subscription app, so it commercially benefits from a finding that models disagree often enough to justify querying several at once. The methodology page acknowledges that the judge of the answers was also a panel participant, and the authors disclose an incident during testing in which memory bled between models. With 24 questions, five models and effectively one run per question, the percentages describe this experiment rather than establishing stable rates.

Why it matters

For anyone picking a model or relying on one for decisions, the study points to a practical asymmetry. Factual recall across the leading model families appears to be converging — ask a checkable question and most models will return the same answer. The divergences cluster precisely where users cannot verify by lookup: architecture choices, pricing strategy, product recommendations. Those are also the areas where a single confident answer is most often taken at face value.

The actionable takeaway, even at this sample size, is twofold. High-stakes judgment calls are the place to consult a second model, and a claim that appears in only one model's answer should be treated as a prompt to verify — not as bonus insight.