Schulman, Millidge and O'Neill debate what could stall AI recursive self-improvement

Dwarkesh Patel has released a long-form conversation with three frontier researchers — John Schulman, Beren Millidge and Charlie O'Neill — built around a single hypothetical: if 2036 arrives and superintelligent systems have not radically transformed the world, what is the most likely technical reason why? The episode, which reached the Hacker News front page (the listing is dated September 11, 2026), opens by steelmanning the case against recursive self-improvement (RSI), the scenario in which AI systems meaningfully accelerate AI research itself and compound their own capabilities.

The setup

Patel deliberately excluded non-technical explanations — political shocks, wars, AI bans — and asked his guests for the strongest technical argument that the superintelligence world never materialises. The three sit at what Patel called open-ish organisations, which allowed an on-record discussion: Schulman is chief scientist at Thinking Machines, previously co-founded OpenAI and led the RLHF work behind ChatGPT; Millidge is CTO of Zyphra, a developer of open-source models; O'Neill is head of model training at Baseten.

A generalization gap that never closes

Millidge's candidate failure mode echoes Moravec's paradox: systems keep conquering whatever humans put in front of them — chess, hard mathematics, benchmark environments — without that mastery translating into broad, transformative impact. If a genuine spark of generalization never arrives and continual learning stays unsolved, AI could become extraordinary inside sandboxes while a persistent sim-to-real gap blocks everything else. He named this as his default explanation if the superintelligence scenario fails, while adding that he considers such an outcome unlikely, since generalization from reinforcement learning is already observable in practice.

The hype cycle that keeps repeating

Schulman agreed with the framing but pointed at bottlenecks. Humans still hold advantages, and although each new model release closes some of them, work stalls wherever the model's judgment is weaker or where it cannot check its own output well enough. He described a loop that has already run several times: a model launches, observers declare it AGI, and after roughly a month of real use it, in his words, "starts to feel dumb." Explosive capability growth has not appeared so far, according to Schulman, because research and engineering work remains bottlenecked enough that a model writing far more code than a person still does not make anyone 100 times more productive. That loop, he suggested, may simply repeat more times than people expect.

Is the current recipe near the optimum?

O'Neill posed the question differently: how far is the transformer-plus-reinforcement-learning recipe from the global optimum of "a learner you could have on a chip"? The standard fast-takeoff argument holds that an agent even fractionally better than every human at AI research still wins, because hundreds of thousands or millions of copies can run in parallel on ever-faster hardware. His counterpoint invoked Moore's law: the smooth line on the chart concealed a series of discrete innovations that each had to be invented to keep it going. Large language models followed a similar pattern, with pre-training scaling hitting diminishing returns before RL opened a new curve.

If progress now requires another discontinuity of that kind, O'Neill doubts that scaling the existing paradigm — even with RL environments explicitly targeting RSI — will find it. He distinguished a cumulative breakthrough that extends the current recipe, which scaled models might yet connect the dots on, from a replacement that means abandoning gradient descent and neural networks entirely; if the optimum lies far enough away, simply running more LLMs may not be enough to reach it. Asked by Patel whether the next discontinuity would be harder than anything since 2012, O'Neill replied that knowing the answer would amount to being able to implement it.

Why it matters

The conversation is a marker of where frontier debate currently stands: senior researchers at or near leading labs are publicly stress-testing the recursive self-improvement hypothesis rather than treating it as settled in either direction. The three brakes they name — an unclosed generalization gap, unreliable self-verification, and a ceiling on the current paradigm — are concrete enough to watch across the next few model cycles, which makes them more useful than abstract timeline arguments. According to the published show notes, the same episode also covers what is driving Chinese labs' progress, how automated AI researchers would be trained, and why RL is working as well as it is.