LongChang benchmark: 11 of 12 frontier LLMs score zero at anomaly discovery

A benchmark named LongChang asked a blunt question of today's leading language models: if you drop them into a simulated world that quietly breaks its own rules, will they notice? According to a post by Bohdan Hordii on dev.to, the answer for 11 of 12 state-of-the-art models was no — each scored exactly 0% at discovering hidden anomalies through unprompted exploration, and the sole model that beat zero reached just 16.7%.

How the test works

Per the dev.to write-up, LongChang places each model inside simulated environments whose physical laws have been altered, with no hints and no predefined objectives handed to the agent. Success depends entirely on self-directed curiosity: the model has to notice that something in the world behaves wrongly, form a suspicion and investigate it. That setup directly probes a common industry claim — that autonomous agents will soon derive new scientific laws, debug distributed systems without human guidance, and resolve infrastructure incidents on their own.

What the numbers show

The headline result is stark. Eleven of the twelve frontier models identified none of the hidden anomalies or modified laws through open-ended exploration. The lone exception, listed as GPT-6-Astra, achieved a 16.7% discovery rate while still leaving most of the planted anomalies unexamined.

The more diagnostic finding sits one layer deeper. In over 80% of runs, agents registered anomalous behavior while manipulating tools in the environment — the discrepancy made it into their records — but they then reverted to routine actions instead of probing what they had seen. In Hordii's reading, the bottleneck is therefore not perception but follow-through: models notice the surprise and then abandon any hypothesis testing about it.

The proposed architectural fix

Hordii frames the result as an argument about architecture rather than raw model capability. Pure, unconstrained agent loops, he argues, spend tokens and compute wandering without converging on an explanation. His proposed alternative is to stop relying on emergent model curiosity and instead enforce deterministic exploration — a structured graph of hypotheses in which every detected discrepancy automatically triggers an isolation run to confirm or reject it. Under that design, the model supplies the reasoning steps while the surrounding code supplies the discipline.

A single-source caveat

The figures come from one practitioner's post on dev.to rather than a peer-reviewed paper or a public leaderboard. The write-up does not name all twelve models or detail the scoring methodology, so the results are worth tracking but should be treated as one reported evaluation rather than settled fact until the benchmark is independently verifiable.

Why it matters

The gap LongChang claims to expose is between two very different abilities: executing tasks you are told about, and noticing the thing nobody pointed you at. Autonomy pitches in production — self-healing infrastructure, zero-shot debugging, automated scientific discovery — depend on the second ability, and this benchmark suggests it is largely absent even in frontier models. For engineering teams, the practical takeaways are twofold: assume free-form agents will not reliably surface anomalies they were not asked to find, and consider building the follow-through into the scaffolding, with deterministic checks that force investigation whenever a delta appears, rather than trusting the model to loop back on its own. The benchmark also offers a cheap eval pattern worth copying — inject a fault into a sandboxed environment and measure whether your agent notices, isolates and explains it.