Schulman, Millidge on Superintelligence by 2036 — AI Digest

Schulman, Millidge on Superintelligence by 2036 — AI Digest

Dwarkesh Patel asked three researchers what would stop superintelligence from arriving by 2036, and the first answer was Moravec’s paradox. Hard Fork gave its slot to the data-centre backlash, and the window’s two most visible open-source projects turned out to be about oversight of processes and agents rather than generation.

Today's highlights

Schulman, Millidge and O'Neill on what would stop superintelligence by 2036

Dwarkesh Patel gathered three researchers from the more open labs and opened by inverting the question: if the world is not full of superintelligent systems by 2036, what is the most likely technical reason why not? Recursive self-improvement is the scenario in which an AI system develops the next, stronger generation of AI itself, and the cycle repeats. His guests are Beren Millidge, CTO of Zyphra, which builds open-weight models; John Schulman, chief scientist at Thinking Machines, an OpenAI co-founder who led the RLHF work behind ChatGPT; and Charlie O'Neill, head of model training at Baseten. Millidge answers first and reaches for Moravec's paradox: again and again AI takes a task that looked like the summit of intelligence — hard mathematics, chess — and the world changes less than everyone expected. If that continues and the spark of generalization never arrives, the model stays extraordinarily strong at exactly what somebody wrote into a benchmark or a training environment. The episode and transcript.

Nine chapters: Chinese labs, long-horizon RL and "move 37"

The conversation runs to nine chapters, and one of them is named after the second candidate reason for failure — the sim-to-real gap. The sim-to-real gap is the distance between the environment a system was trained in and the world it has to work in. The remaining chapters of the 11 September 2026 episode list what the participants consider live questions today: what is driving the progress of Chinese labs, how automated AI researchers will be trained, whether long-horizon reinforcement learning will elicit general intelligence, how much of the progress is explained by data, why reinforcement learning works as well as it does, and "move 37" together with entropy collapse. The final chapter, a rapid-fire round on timelines, starts at 1:28:32 — so the conversation runs well past an hour and a half. The episode and transcript.

Hard Fork gave its slot to the backlash against data centres

On 11 September 2026 Hard Fork handed its feed to an episode of The Ezra Klein Show about what resistance to data centre construction looks like across the American Midwest. The guest is Jasmine Sun, a journalist who travelled through the region and reported what is happening on the ground. Ezra Klein and Jasmine Sun discuss the strange political coalitions forming against the AI infrastructure build-out, and why people do not believe what they are told about it. The episode's additional reading is a piece titled "No Data Centers In My Backyard". The story is worth attention as one of the rare cases where the binding constraint on the industry is neither chips nor megawatts but the consent of the residents of a particular county. The Hard Fork episode.

AI Daily Brief: ten ways to widen the work rather than speed it up

On 13 September 2026 the AI Daily Brief argued that AI is not only useful for speed: NLW walks through ten ways to think bigger and widen the range of what a person is capable of doing in the first place. The list includes building your own video production pipeline, making interactive proposals for clients, and turning your expertise into a product. A companion exercise ships with the episode to help find those possibilities in your own work. The reframing is useful for anyone who has already automated the routine and does not know what comes next: "make the existing thing faster" hits a ceiling quickly, while "take on what used to be out of reach" does not. Step-by-step courses for that kind of shift are in the AI SKILLS education section. The AI Daily Brief episode.

The window's two most visible open-source projects are about oversight, not generation

In our daily open-source sweep the two most visible tools of the window turned out not to be generative: witr (22.3K stars) traces what actually started a process, port, container or file, while agent-orchestrator (12K stars) carries a team of coding agents from planning through to the merge. Both are written in Go. witr offers plain commands for scripts as well as a full-screen terminal interface, runs on Linux, macOS and FreeBSD, and understands Docker. agent-orchestrator works on top of any harness — Claude Code, Codex and more than two dozen others — gives every task its own git worktree and its own feedback loop, and is reachable from a desktop, a browser and a phone. What they share is one thing: once the number of runs grows, the bottleneck stops being generation and becomes the answer to "who started this, and what is it doing now". witr, agent-orchestrator.

Numbers and facts

Different perspectives: how close is recursive self-improvement?

There is no argument about the numbers here — the argument is about what counts as evidence. The 11 September 2026 episode opens with a chapter devoted entirely to steelmanning the case against recursive self-improvement, and that framing is itself a position.

For. The host's question is built on the fast scenario: Dwarkesh Patel asks not whether billions of superintelligent systems will appear by 2036 but what would prevent them, and rules out wars, bans and other exogenous shocks — that is, he asks for a reason from inside the technology (episode).

Neutral. The line-up is picked so that people from companies with an open part of their work can speak on the record — Zyphra, Thinking Machines, Baseten. The agenda bears that out: half the chapters are not about self-improvement at all but about where progress comes from — data, reinforcement learning, long horizons, Chinese labs (episode).

Against. Beren Millidge answers with Moravec's paradox: the solved "summit" task turns out, over and over, to matter less than expected. His failure scenario is a model flawless at everything written into a benchmark or a training environment and stuck at the sim-to-real gap (episode).

Tools and techniques

In brief