· via OpenAI blog/news
OpenAI publishes its perplexity figures and proposes shared pretraining limits
OpenAI argues that pretraining perplexity tracks the emergence of new AI capabilities, discloses its own loss figures, and proposes industry-wide development limits tied to the metric.

OpenAI makes the case for a single number
OpenAI has published a blog post arguing that the most informative measure of progress in modern AI is one most users never see: perplexity, the cross-entropy loss a model achieves when predicting the next token on held-out data. Lower numbers mean better prediction. Alongside the argument, the company disclosed internal perplexity figures, a move the post admits gives competitors a clearer view of where OpenAI stands, but one it says is outweighed by the urgency of shared safety standards across the industry.
Compression as the substrate
According to the post, modern AI can resemble traditional software development, with varied techniques targeting specific capabilities. In practice, the author writes, most of those techniques are harvesting intelligence from a much simpler, less interpretable process: training a predictor, equivalently a compressor, on a very large corpus. Development is sketched as three stages: build a large computer, pretrain a compressor on unstructured data, then apply prompting, finetuning and reinforcement learning to elicit the model's latent capabilities. The post separates the creation of what the author previously called an "alien mind", whose capabilities are hard to predict in advance, from the later, targeted work of drawing those capabilities out.
Capability milestones track the loss
OpenAI says it has tracked pretraining this way for years, and the post maps capability jumps to loss figures on its main evaluation dataset. GPT-2, which set a perplexity record on older datasets, was arguably the first model to consistently write coherent paragraphs. GPT-3 could pick up a task from a few in-context examples. GPT-3.5, at roughly 0.99 log-perplexity, carried fluent conversation and powered the original ChatGPT. GPT-4, at roughly 0.91, made productive use of chain-of-thought reasoning, which later motivated today's reasoning models. Below 0.8, OpenAI says, transformative capabilities emerged in fields such as cybersecurity and mathematics.
The author stresses that nothing made these capabilities predictable at those thresholds; they arrived as emergent consequences of progress on one number. Benchmarks churn in comparison: the post notes that older evaluations would make little sense for GPT-2 or GPT-3, and that the benchmarks of that era were largely saturated by GPT-4. Pretraining loss, the argument goes, should stay informative. Extrapolating, the author expects further transformative capabilities, of unspecified kind and possibly not tested for today, as models enter the 0.68 to 0.7 window.
AI doing pretraining research
The post also addresses AI's role in its own development. The author treats "full recursive self-improvement" as a state to avoid in order to keep human control, but sees a narrower potential step change in AI doing algorithmic research on pretraining. OpenAI says it has posed open mathematics problems to its models for more than a year as one evaluation method, and that after upgrading to a model at roughly 0.76 log-perplexity, the model began producing what the author describes as groundbreaking results, with long-standing questions resolved one after another in a single Slack thread. By extension, the author finds it conceivable that a similar wave of fundamental pretraining advances arrives as soon as 2027. Alignment and monitoring progress is expected to lag, with further degradation of currently dominant auditing methods and further increases in what the post calls evaluation and training awareness.
The proposal on the table
The concrete ask is an industry standard. The author calls on frontier developers to jointly establish norms for evaluating models, red-team safety cases before first use, and adopt shared transparency standards. On top of that, the post proposes common development limits, based on perplexity and other measurements, that developers commit not to cross until the ability to understand and monitor AI behavior reaches a concrete milestone, such as a successor to chain-of-thought monitors. The intent is a repeatable framework: set a first limit, define what must be achieved before passing it, then pace later development the same way.
Stated limitations
The post is candid about the metric's limits. Comparisons across labs are hard because developers use different evaluation datasets and train on different data mixtures; OpenAI reports the dataset it has tracked internally and argues that transferable comparisons are possible if transparency becomes a shared goal. Perplexity corresponds to no task anyone directly cares about; its value is that advances on it tend to produce entirely new capabilities. It also misses progress outside pretraining, since chain-of-thought reasoning sharply raises capability without lowering the model's perplexity. And the floor is unknown: a loss of zero would mean predicting the dataset deterministically, which the post does not expect to be reachable, but the inherent entropy of the data has not been established.
Why it matters
This is a frontier lab arguing that general capability progress reduces, to a first approximation, to one measurable number, and then publishing its own numbers to make the point. If other developers follow, the industry would have a progress measure that outlives any benchmark suite, plus a concrete basis for coordination: limits expressed in measured loss rather than model names or release dates. The post also puts specific, checkable predictions on record, from new capabilities in the 0.68 to 0.7 range to a possible burst of AI-driven pretraining research by 2027. The open question is adoption: a shared metric currently defined by one lab's internal dataset will need work before it can serve as common ground, a challenge the post itself acknowledges.
- #openai
- #language-models
- #perplexity
- #ai-safety
- #pretraining