OpenAI runs show GPT-5.6 Sol at 13.3% or 38.3% on ARC-AGI-3 depending on harness settings

OpenAI has published benchmark runs in which the same model, GPT-5.6 Sol, scored 13.3% or 38.3% on ARC-AGI-3 depending only on how the evaluation harness and API were configured, according to a dev.to write-up. The higher score used about one sixth as many output tokens per game as the lower one. Nothing about the model itself changed.

What was measured

The score is Relative Human Action Efficiency (RHAE), a measure of how efficiently an agent acts relative to human players rather than a simple pass rate. The dev.to post, citing an OpenAI estimate, puts the average human tester at 48% on the same task set. On that scale, GPT-5.6 Sol moved from roughly 34.7 points behind the human average to 9.7 points behind; by the post's arithmetic, configuration alone closed about 72% of the original gap.

The write-up also points to a starker comparison: on the public leaderboard for one of the benchmark's games, no frontier model gets past the first level. With the reconfigured harness, GPT-5.6 Sol cleared all six.

The two harnesses

Per the post, the differences sit in two places:

• Reasoning between turns. The official harness discarded the model's private reasoning after every action. The reconfigured run kept it.
• Context management. The official harness applied rolling truncation, dropping the oldest messages once a conversation exceeded 175,000 characters. The reconfigured run compacted the history instead of cutting it.

The author's observation is that these are not exotic choices. Most agent frameworks behave the same way by default: reasoning arrives as a separate output that has to be deliberately persisted, and trimming the oldest turns is the simplest form of context management. On a benchmark built around learning a game's rules through interaction, deleting the model's working theory after each move forces it to reconstruct the situation from scratch every turn, which shows up both as weaker play and as extra output tokens spent rebuilding reasoning it had already done.

As the post notes, OpenAI says these are the same settings already running in its production systems, including ChatGPT and Codex. The benchmark harness simply did not have them enabled.

The uncomfortable part for benchmarks

The dev.to piece cites OpenAI's own framing that benchmarks rarely measure models in isolation, and also reflect less visible choices about API settings, harness design and prompting. OpenAI reportedly added that this was not the first time a surprising low score turned out to trace back to a generic harness that dropped reasoning messages, which implies published numbers may still be in circulation that measure a harness defect while being read as a fact about a model.

The post gives ARC's rationale a fair hearing: a simple, generic harness makes model shortcomings more visible and keeps cross-model comparisons fair, since vendors would otherwise tune scaffolding to each model's quirks. Its counterargument is that a neutral-looking harness still embodies assumptions, about whether reasoning persists and how history is trimmed, and those assumptions happen to penalize models trained to reason across turns.

The consequence extends to leaderboards. When a fixed harness ranks model A above model B on an agentic task, the defensible reading is that model A paired with that harness beat model B paired with the same harness. The post argues the harness should be treated as a required disclosure, the way benchmark suites already disclose temperature and prompts.

Why it matters

For anyone building on LLM APIs, this is a controlled reminder that an eval number belongs to a system, not a model. Before comparing models, or before accepting a low score as a capability ceiling, check whether reasoning is persisted between turns and how history is handled at the context limit. The economics point the same way: the configuration that kept reasoning across turns was both cheaper and better, because the model stopped paying output-token prices to regenerate thinking it had already done. And for anyone consuming benchmark results, attribution matters as much as the number itself. Here, the gap between a fact about a system and a fact about a model was worth 25 points.