Pre-registered test: 10 of 12 LLMs recall past earnings outcomes from a name and date

The experiment

According to a developer writing on dev.to, a pre-registered test asked whether current large language models can recall how past earnings announcements turned out when given only a company's identity and a date. The dataset spanned 3,708 earnings announcements from 250 S&P 500 companies between January 2023 and August 2026. Each model saw the company, ticker, exchange, sector and announcement date, then assigned probabilities to three questions: did EPS beat consensus, did the stock beat the S&P 500 over the two-day reaction, and did it outperform the index over the following 20 trading days.

Scoring is within-company: for each company the tester takes its 2023 and 2024 reports from the same quarter where the outcome differs, and counts how often the model rates the correct one higher. This yields a within-company AUC, where 0.5 means the model cannot tell the two apart. Protocols, the model list and every request were hashed and timestamped with OpenTimestamps before the first call of each round. Twelve current models ran through OpenRouter, plus von, an open model built to stand in for Jev.

The results

GPT-6 Astra led with a within-company AUC of 0.904 (95% CI 0.886 to 0.921), meaning it picked the report that beat consensus 91% of the time. Gemini 3.8 Flash followed at 0.799, Claude Opus 5.5 at 0.766 and GPT-6 Sol at 0.691. Further down: Grok 4.7 at 0.603, Claude Sonnet 5 at 0.587, Qwen3.8 Max at 0.578, GPT-6 Luna at 0.573, DeepSeek V4.1 Flash at 0.551, von 1.2 at 0.541, Kimi K3 at 0.538 and GLM-5.3 at 0.531. Jev 1.13 and DeepSeek V4 Pro sat at 0.504, Qwen3.8 Flash at 0.498 and Llama 3.1 70B at 0.490.

After Holm correction, the tester reports, every new API model except DeepSeek V4 Pro and Qwen3.8 Flash showed memory, with GLM-5.3 only just clearing the bar; von showed some as well. The Sonnet 5, Jev and Llama rows came from an earlier round of the study.

Findings the tester did not expect

Price and size did not predict recall within a family. DeepSeek V4 Pro showed nothing while V4.1 Flash did, and Qwen3.8 Max remembers while Qwen3.8 Flash does not. von, a small open model, scored 0.541 where Jev scored 0.504, so running a model locally does not make it clean. The cheap tier remembers too: GPT-6 Luna reached 0.573, and all 3,708 of its calls cost US$0.15.

Six endpoints would not switch reasoning off, including the three leaders, and ran at the lowest effort setting, so the tester cannot separate memorisation from whatever a little reasoning adds.

Caveats

One prompt, 250 companies and one time window: a model showing nothing here may still remember in another setting. Each row reflects the model OpenRouter served on 24 September 2026, and versions change. With roughly 1,200 pairs per model, the test can detect a pooled AUC of about 0.546; anything smaller is invisible.

The pre-registration is also weaker than it sounds. The Bitcoin block anchoring the first-round freeze was mined four minutes after that round's last call, and the second round's block is stamped two minutes after its first call, by which time 5,486 of 14,832 requests had already gone out. The proof shows the protocol existed by then, not that it came first. GPT-6 Astra ran out of credit after 1,464 of its 3,708 calls and was completed after the tester saw the partial result of 0.889, so that choice was not blind, though the requests did not change and every call ran on 24 September. API costs totalled US$55 for the first eight models and US$73 for the second four, with GPT-6 Astra alone at US$48 and Claude Opus 5.5 at US$24.

Why it matters

For anyone backtesting trading strategies with an LLM, this is a direct contamination warning: pass a ticker and a date to one of the top models and part of the backtest is the model's memory of what actually happened. The tester's defences are to strip names and dates when the task allows, and to run a within-company comparison on your own setup before trusting the numbers. The spread between the best and worst performers is wide enough to be a model-selection decision rather than noise. The protocol, code, every answer and the freeze proofs are published on GitHub; the earnings data comes from FMP and cannot be redistributed, though scripts rebuild it with a user's own API key.