· via MIT Technology Review – AI topic
METR's updated time-horizon report: exponential gains in task length, persistent weakness in judgment
METR's refreshed time-horizon analysis shows frontier models still doubling the task length they can handle roughly every four months, while new open-ended evaluations expose a gap between coding ability and judgment.

An exponential curve, refreshed
METR, the nonprofit that evaluates frontier AI systems, has released a major update to its "time horizon" plot, a chart that became one of the most-shared — and most-misread — visualizations in AI, according to MIT Technology Review. The original graph scored each task's difficulty by how long human testers needed to finish it, then rated models by the length of task they could complete. Its central finding: the task length that frontier models can handle has doubled roughly every four months.
The update expands the suite with tasks estimated to take humans far longer than the originals, plus more open-ended ones, and progress on measurable, verifiable tasks does not appear to be slowing, MIT Technology Review reports. But the report also introduces a new class of judgment-heavy evaluations, and there the picture changes.
Coding strength, judgment weakness
The gap comes down to how models are trained. Reinforcement learning — having a system attempt a task repeatedly and strengthening the behaviors that lead to success — works well when success can be checked automatically, which is why coding and math are where AI is strongest. The technique is much harder to apply to tasks with fuzzy success criteria, such as choosing a promising research direction.
METR's new suite tries to test exactly those capabilities. In one task, models were asked to design new AI evaluations for METR's own research, with METR staff grading the results. Even though the most advanced models now outperform almost every human at software engineering, not a single AI-designed evaluation was judged useful enough for METR's research. In a second task, scoring philosophical arguments, models also fell short of human performance — though METR notes in the report that its own employees disagree about how relevant that task is to automating AI R&D.
There are signs of movement. Some of the newest models, such as Opus 5.5, land in what METR calls "borderline above the bar" territory on the evaluation-design task; a year earlier, every model scored "clearly below the bar."
Outside corroboration
Independent evidence points the same way. In a blog post that week, OpenAI reported that its models perform worse on real-world tasks assigned by its own employees than on the easier, verifiable tasks in METR's original suite. Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, told MIT Technology Review that an August preprint he co-authored found AI models cannot yet reproduce human computer science research on their own. "We're at the point where AI evaluations need to get more creative," he said, calling METR's push into new evaluation types a welcome change.
The measurement problem
The report's biggest limitation is measurement itself. The original plot timed real humans on every task; for the new, harder tasks, completion times are mostly estimates, because paying people for work that could take hundreds of hours — or that only a handful of experts can do — is impractical. The philosophical-argument task was mostly rated by a single person, so scores partly reflect agreement with that one individual.
Najoung Kim, a linguist and computer scientist at Boston University, said the observed improvement tells us something about the current moment but is hard to extrapolate. METR's Matt Putz was blunter, saying in an email that trends for hard-to-verify tasks are unlikely ever to be as clean and robust as those for easy-to-specify ones.
Why it matters
The report lands amid debate over automated AI research. Earlier in November, according to MIT Technology Review, a heavily redacted METR report claimed AI could be accelerating the pace of AI progress at Anthropic — a company that has just completed a blockbuster IPO — by a factor of two, stoking fears of recursive self-improvement. The new data suggests the capabilities that would matter most for that scenario are precisely the ones models lack and that evaluators struggle hardest to measure.
For workforce planning, the message is similarly concrete: AI competence is not a single dial. In software engineering, frontier models are already better than almost every human tested; in open-ended judgment work, they are below the bar. Anyone budgeting for automation over the next few years should expect very different timelines for these two categories of work — and should treat the judgment-task trendlines, sparse as they are, as far less settled than the coding ones.
- #ai-evaluations
- #ai-benchmarks
- #frontier-models
- #reinforcement-learning
- #ai-progress