Artificial Analysis Intelligence Index v5 now relies mostly on private test sets

Artificial Analysis overhauls its flagship model ranking

Artificial Analysis has published version 5 of its Intelligence Index, the composite score it uses to compare language model capabilities. The company describes it as the largest revision since v4 launched in January, and for the first time the majority of the index — 60% of its weight — sits on evaluations with private test sets or held-out answers, a deliberate move to keep benchmark data out of training corpora.

Alongside v5, Artificial Analysis is introducing a separate Progress Index v1, which places models released since November 2022 on a single continuous capability scale for tracking improvement over time. The Intelligence Index remains the company's primary measure of the current frontier.

Two new benchmarks and three upgrades

The index still holds nine evaluations, split evenly across knowledge work, coding, science and general categories at 25% each.

The coding category gains FrontierCode 2.0, a private benchmark from Cognition announced the same day. It contains 132 tasks drawn from real open-source repositories, and an attempt scores zero if it fails any blocking check — grading is meant to mirror whether a repository maintainer would actually merge the agent's change. Each task is run three times in Stirrup, Artificial Analysis's agentic harness. The science category adds Terminal-Bench-Science, 70 expert-curated research tasks executed in a terminal, which pass only when every test in a task passes.

Three existing evaluations were reworked. HLE-Diamond replaces Humanity's Last Exam with a refined 1,000-question set from the Center for AI Safety and Scale AI, split evenly between reasoning and knowledge. AA-Omniscience was cleaned up, with saturated and public questions removed and a recalibrated public subset released on Hugging Face. CritPt v2 is now scored on a private 40-question hard subset — the original 70 research-level physics questions minus the 30 that seven frontier models solved on all five attempts at launch — run with a code-execution tool.

GDPval-AA v2.1, SciCode and AA-LCR v1.1 were dropped. Per the methodology page, scoring is generally pass@1, with temperature set to 0 for non-reasoning models and 0.6 for reasoning models.

The results: a narrow lead at the top

According to Artificial Analysis, Claude Opus 5.5 (max) leads at 58, one point ahead of GPT-6 Astra (max) at 57, with GPT-6.1 Sol (max) at 55. Anthropic and OpenAI hold the entire top five, and Gemini 4 Argon (high) sits next at 50.

No single model dominates: Claude Opus 5.5 tops three evaluations (FrontierCode 2.0 at 46%, HLE-Diamond at 68% and AA-Omniscience v1.1), GPT-6 Astra leads two (Terminal-Bench-Science at 63% and the CritPt hard subset at 72%), and Claude Sonnet 5.5, Kimi K3.1 Pro, Gemini 4 Argon and Muse Wonder Full each lead one of the remaining evaluations.

Open weights models fall further behind

The gap between closed and open weights models widened from 3 points under v4.3 to 12 in v5. The strongest open weights entrant is Kimi K3.1 Pro at 46, though its weights have not yet been released. Among models with published weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 — 26 points off the top and below Claude Haiku 5.5 (max) at 36. The divide is starkest on HLE-Diamond and Terminal-Bench-Science, where Kimi K3.1 Pro scores 37% and 34% against 68% and 63% for the respective leaders.

Cost and effort settings now matter more

Among the ten highest-scoring models measured, cost per Intelligence Index task varies by more than 13x: GPT-6.1 Terra (max) is cheapest at $0.96 per task while Claude Fable 5.1 (max) runs $13.04. GPT-6.1 Sol (max) lands within 4 points of Claude Opus 5.5 at roughly an eighth of the leader's $10.04 per task.

Reasoning-effort settings also swing scores more than before: Claude Opus 5.5 spans 35 points at low effort to 58 at max, at 10x the per-task cost, while GPT-6.1 Sol's low-to-max spread doubled from 10 to 20 points. Tasks are also heavier — median output tokens per task rose two-thirds, from 35k to 59k, and the four agentic evaluations, which carry 55% of the index weight, account for roughly 90% of output tokens.

Why it matters

The shift toward private datasets is a response to contamination, the practice or accident of benchmark questions leaking into training data and inflating scores. But it also means the community must trust Artificial Analysis's harness, prompts and grading, since most of the underlying data can no longer be independently inspected — which makes the released public subset for AA-Omniscience a notable gesture toward verifiability. For teams choosing models, v5 pushes two practical questions to the foreground: the closed-versus-open weights gap has widened sharply, and headline intelligence now hides large differences in price and effort configuration, where a near-frontier model can cost an eighth of the leader per task.