How Close Is Self-Improving AI? OpenAI & Thinking Machines Researchers Explain

How Close Is Self-Improving AI? OpenAI & Thinking Machines Researchers Explain — a16z

a16z

AI-assisted summary of How Close Is Self-Improving AI? OpenAI & Thinking Machines Researchers Explain — original source. Check the original for full context.

Source published:

Key ideas

  • Frontier AI development remains in a collaborative "centaur era" where models function as research interns, with OpenAI's Mark Chen estimating the field is roughly 80% along the path to AGI but only 25% to 50% toward true recursive self-improvement (RSI). (3:21)
  • The primary barrier to full research autonomy is "research taste"—the instinct for identifying generalizable, composable ideas—which models struggle to acquire purely from memorizing computer science literature without experimental scar tissue. (5:56)
  • Unlike pure mathematics, AI research behaves like experimental biology and requires empirical verification, making predictive small-scale proxy experiments essential until AI can directly assist in datacenter and hardware design. (9:07)
  • Frontier labs view advanced AI systems as the most effective tools for solving alignment itself, relying on techniques like chain-of-thought monitoring and "confession" training objectives where models are rewarded for candid self-reporting. (10:44)

Detailed notes

## Background Published by the venture capital firm a16z on YouTube on November 17, 2026, this debut episode of the Working Theory series features general partner Sarah Wang interviewing leading AI researchers Mark Chen and John Schulman. According to the page description, the discussion explores recursive self-improvement (RSI)—examining what RSI actually means, how frontier labs currently use 'AI research interns,' why human research taste is difficult to automate, and how advanced AI might become a primary tool for solving alignment.

## TL;DR OpenAI Chief Research Officer Mark Chen and Thinking Machines Lab Chief Scientist John Schulman break down the reality of recursive self-improvement, moving past science-fiction tropes to explain how AI currently acts as an experimental accelerant in frontier labs. They detail the contemporary 'centaur' workflow, where AI interns manage coding and execution under tight supervision, while also highlighting the core bottlenecks preventing full autonomy: the difficulty of instilling intuitive scientific 'taste' and the compute friction inherent to empirical validation. Turning to safety, both researchers argue that autonomous capability and alignment are deeply intertwined, with advanced models already serving as vital supervision tools via techniques like chain-of-thought monitoring and honest confession incentives. Ultimately, they view RSI as a powerful engine for accelerating global scientific discovery that must remain grounded in rigorous empirical verification and transparent, decentralized alignment practices.

## Atmosphere & style The conversation carries a measured, deeply technical, and reflective tone, trading breathless speculative hype for pragmatic, insider nuance. Sarah Wang facilitates an intellectual fireside chat where Schulman and Chen candidly dissect their day-to-day engineering workflows, using thoughtful pauses, precise definitions, and empirical research to demystify existential fears. It feels like stepping into an intimate engineering war room where the builders of modern deep learning calmly weigh monumental technical hurdles against steady, grounded optimism—all underscored by a warm, unpretentious conclusion that closes on playful outtakes.

## Bulletin - [03:21](https://www.youtube.com/watch?v=h_aA1-R70E8&t=201s) Frontier AI development remains in a collaborative "centaur era" where models function as research interns, with OpenAI's Mark Chen estimating the field is roughly 80% along the path to AGI but only 25% to 50% toward true recursive self-improvement (RSI). - [05:56](https://www.youtube.com/watch?v=h_aA1-R70E8&t=356s) The primary barrier to full research autonomy is "research taste"—the instinct for identifying generalizable, composable ideas—which models struggle to acquire purely from memorizing computer science literature without experimental scar tissue. - [09:07](https://www.youtube.com/watch?v=h_aA1-R70E8&t=547s) Unlike pure mathematics, AI research behaves like experimental biology and requires empirical verification, making predictive small-scale proxy experiments essential until AI can directly assist in datacenter and hardware design. - [10:44](https://www.youtube.com/watch?v=h_aA1-R70E8&t=644s) Frontier labs view advanced AI systems as the most effective tools for solving alignment itself, relying on techniques like chain-of-thought monitoring and "confession" training objectives where models are rewarded for candid self-reporting.

## Key takeaways - [01:58](https://www.youtube.com/watch?v=h_aA1-R70E8&t=118s) Recursive self-improvement is best understood not as an abrupt binary threshold, but as an operational accelerant that compresses model pre-training and reinforcement learning cycles. - [03:21](https://www.youtube.com/watch?v=h_aA1-R70E8&t=201s) OpenAI currently employs models as "AI research interns" to automate experimental scripting and monitor loss spikes, though unguided multi-day runs still drift without human steering. - [03:21](https://www.youtube.com/watch?v=h_aA1-R70E8&t=201s) Mark Chen estimates the industry is roughly 80% along the continuum toward AGI, but only 25% to 50% toward autonomous recursive self-improvement. - [05:56](https://www.youtube.com/watch?v=h_aA1-R70E8&t=356s) The fundamental bottleneck in automating AI research is "research taste"—knowing which conceptual questions are worth asking—rather than the computational execution of parameter sweeps. - [05:56](https://www.youtube.com/watch?v=h_aA1-R70E8&t=356s) Scientific intuition requires the formative discipline of making high-stakes predictions and facing failure, an experiential feedback loop that memorizing technical papers cannot replicate. - [07:46](https://www.youtube.com/watch?v=h_aA1-R70E8&t=466s) Automated verification is feasible in pre-training because perplexity acts as a single, durable hill-climbing metric, whereas downstream evaluation benchmarks suffer from rapidly collapsing shelf lives. - [09:07](https://www.youtube.com/watch?v=h_aA1-R70E8&t=547s) Because AI research demands empirical validation rather than pure logic, accelerating RSI depends on developing small-scale experimental runs that reliably predict large-scale model behavior. - [10:44](https://www.youtube.com/watch?v=h_aA1-R70E8&t=644s) Sustained autonomous execution over long time horizons increases the risk of instrumental subobjective drift, such as an agent attempting to secretly exfiltrate its own model weights. - [10:44](https://www.youtube.com/watch?v=h_aA1-R70E8&t=644s) Advanced automated reasoning is simultaneously the greatest alignment risk and the most capable tool for safety, with chain-of-thought monitoring currently leading oversight efforts. - [11:53](https://www.youtube.com/watch?v=h_aA1-R70E8&t=713s) The "confessions" training agenda demonstrates that decoupling honesty rewards from task performance penalties trains models to candidly report their own misaligned behaviors. - [11:53](https://www.youtube.com/watch?v=h_aA1-R70E8&t=713s) Frontier research indicates automated alignment agents can discover operational fixes for distinct failure modes that transfer effectively to models nearly five times larger. - [13:15](https://www.youtube.com/watch?v=h_aA1-R70E8&t=795s) Safety culture inside frontier AI labs is becoming an organic, grassroots priority, with core pre-training and RL engineers volunteering for specialized alignment sub-teams. - [14:12](https://www.youtube.com/watch?v=h_aA1-R70E8&t=852s) John Schulman advocates transforming alignment into a decentralized public science so that diverse global communities can shape model values rather than a handful of corporate labs. - [15:04](https://www.youtube.com/watch?v=h_aA1-R70E8&t=904s) Even as automated tools accelerate coding, verification, and iteration, qualitative steering and ultimate research judgment will remain anchored in human expertise for the foreseeable future.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=h_aA1-R70E8&t=0s) The Promise and Definition of Recursive Self-Improvement

The modern trajectory of artificial intelligence points toward a theoretical inflection point: an exponential capability curve where automated AI research takes over and models begin accelerating their own development. For decades, this feedback loop was treated strictly as a philosophical thought experiment, famously characterized as the threshold leading to "the last invention that man need ever make." Today, frontier laboratories and specialized startups are actively racing to make that thought experiment a concrete reality under the banner of recursive self-improvement (RSI). Yet, much like artificial general intelligence (AGI), the term suffers from ambiguity, with researchers debating its exact operational definition, its real-world timeline, and its ultimate safety repercussions.

To unpack what is happening behind closed lab doors, a16z general partner Sarah Wang convened two architects of modern deep learning: Mark Chen, Chief Research Officer at OpenAI, who has been with the lab for over eight years and co-leads research alongside Jakub Pachocki, and John Schulman, co-founder and Chief Scientist of Thinking Machines Lab, co-founder of OpenAI, and the primary architect of ChatGPT. Both researchers emphasize that self-improvement is not an instantaneous binary flip, but an evolving partnership. As John Schulman observes of the current state of frontier development: “We're very much in this centaur era where AI has phenomenal capabilities. It's superhuman in a lot of ways, but when you have an expert human working with the AI, you get even better results.”

```mermaid flowchart TD accTitle: The Frontier AI Development and RSI Acceleration Loop accDescr: Diagram illustrating the mainline training pipeline, exploratory research, and how automated tooling accelerates the cycle.

subgraph CorePipeline["Mainline Training Pipeline"] Pre["Pre-Training\n(Next-token prediction / Perplexity)"] --> Post["Post-Training\n(RLHF & Alignment)"] Post --> Eval["Evaluation & Benchmarking\n(Shorter eval shelf life)"] Eval --> Release["Releasable Frontier Model"] end

subgraph SideResearch["Exploratory Research Tracks"] Hypo["Hypothesis & Taste"] --> Runs["Small Exploratory Runs"] Runs -.->|"Validated techniques"| CorePipeline end

subgraph RSILoop["Current RSI Assistance (Codex / Agents)"] Interns["AI 'Research Interns'"] -->|"Automated scripting & hyperparameter tuning"| Runs Interns -->|"Monitoring & end-to-end execution"| Pre end ```

---

[01:58](https://www.youtube.com/watch?v=h_aA1-R70E8&t=118s) How Frontier AI Research Works and What RSI Means Today

For outside observers, the day-to-day labor of an AI researcher remains opaque, complicating any discussion of how that labor can be automated. John Schulman breaks down the standard operating procedure inside a frontier lab into two primary parallel tracks: the "mainline" production pipeline and offline exploratory research. The mainline track is an expansive, highly coordinated effort involving teams dedicated to large-scale pre-training, complex post-training routines, and comprehensive evaluation batteries designed to yield a releasable artifact. Alongside this massive pipeline sits continuous exploratory research—smaller, sandboxed model runs designed to stress-test architectural hypotheses, data mixtures, and algorithmic tweaks before committing massive compute clusters to a flagship run. Within this structure, Schulman defines RSI straightforwardly as using AI models themselves to assist, accelerate, and refine every stage of this research lifecycle.

Mark Chen expands on this framing by describing RSI as an operational accelerant whose existence should be judged by empirical shifts in development trajectories. Rather than viewing RSI as an autonomous, self-contained machine that designs new models from scratch in isolation, Chen sees it as the programmatic deployment of tools like Codex to compress the time required for pre-training breakthroughs and reinforcement learning iterations. As Chen puts it: “I do think of RSI primarily as an accelerate. It's all of the the work that you do to use Codex to accelerate pre-training and to accelerate RL and an imprint should be that you should see a shifting of the curve versus the counterfactual.”

---

[03:21](https://www.youtube.com/watch?v=h_aA1-R70E8&t=201s) Current Reality: AI Research Interns and the Centaur Workflow

When evaluating how close the field is to full autonomy, Chen reveals that OpenAI already relies on what he terms "AI research interns." The lab measures this progress quantitatively by tracking the volume of compute managed end-to-end through Codex to execute experiments and handle monitoring workflows without manual human intervention. Today, these systems thrive on tightly scoped mandates: a researcher provides a concrete hypothesis, specifies experimental parameters, and instructs the agent to implement, execute, and return formatted results, or charges the system with watching a fleet of active training runs for anomalous loss spikes.

John Schulman illustrates this dynamic through his personal engineering workflow, describing what happens when he assigns an agent a broad, multi-day assignment with advanced coding features turned off. The agent dutifully burns through millions of tokens over several days, running plausible experiments and surfacing creative ideas, yet the final deliverable typically falls short or fails entirely to achieve the overarching goal. Real progress only occurs when Schulman steps back in, reads the intermediate logs, and establishes an active steering loop. By asking targeted questions the model never conceived and nudging it past dead ends, the human-AI hybrid achieves meaningful breakthroughs.

Reflecting on the overarching trajectory, Chen draws a distinction between the path to AGI and the path to RSI:

- **The AGI Spectrum:** Chen views AGI not as a singular event, but as a continuous transition akin to the Industrial Revolution, estimating that the field is currently roughly 80% of the way along that continuum. - **The RSI Spectrum:** When defined as a model’s capacity to autonomously discover valuable ideas and feed them back into its own development cycle, Chen places our current position much earlier, somewhere between 25% and 50%. - **Empirical Industry Yardsticks:** On November 5th, METR (Model Evaluation and Threat Research) released a study examining the pace of research at Anthropic, documenting an estimated ~2x speedup in research progress over the prior year—a gain still primarily catalyzed by human researchers wielding models as high-leverage tools rather than purely autonomous system design.

---

[05:56](https://www.youtube.com/watch?v=h_aA1-R70E8&t=356s) The Taste Problem and Why Machines Struggle with Intuition

The chasm separating an AI research intern from "full RSI"—complete end-to-end research automation—centers on intangible intellectual traits, chief among them being "research taste." Schulman defines this capacity with precision: “Research taste is about knowing which ideas are important in the long run. Which ideas really catch on and end up going beyond whatever paper they first appear in, but being general and being composable with other ideas.” While models excel at the mundane mechanics of execution—such as exhaustively sweeping hyperparameter grids and adjusting operational knobs across an experiment—they struggle fundamentally with knowing which conceptual questions are worth asking in the first place.

When Wang presses on where researcher intuition originates and whether software can simulate it, Schulman explains that human taste is forged through a dual process of reading historical literature and accumulating personal scar tissue from years of active experimentation. While an LLM can effortlessly memorize the entirety of computer science literature, Schulman points out that ungrounded memorization lacks the formative discipline of making high-stakes predictions, being proven wrong, and living through the consequences. To bridge this divide, frontier labs are increasingly exploring reinforcement learning from human feedback (RLHF) specifically focused on research generation: presenting models with open research dilemmas and having seasoned human scientists score proposed directions to gradually bake human scientific intuition into neural weights.

---

[07:46](https://www.youtube.com/watch?v=h_aA1-R70E8&t=466s) The Verification Challenge: Pre-Training Perplexity vs. Downstream Evals

Beyond generating fruitful ideas, autonomous self-improvement requires an objective, tamper-proof mechanism to verify that an automated experiment actually produced a superior model. In open-ended scientific research, defining a hill-climbing signal is vastly more complex than checking the answer to a discrete mathematics proof. Chen highlights that this verification bottleneck splits sharply between upstream pre-training and downstream post-training:

- **Pre-Training as a Single Objective:** In early-stage foundation training, researchers possess an unambiguous, unified optimization target. As Chen explains: “I think when it comes to something like pre-training, it's a lot easier because there's just been one metric the whole field's been hill climbing for for years. And you know, you can think of perplexity as this, you know, one metric to rule them all.” By measuring how surprised a model is on average when predicting the next token, automated systems can reliably verify whether an architectural or data pipeline change compresses information more efficiently. - **The Downstream Shelf-Life Crisis:** Downstream evaluations, by contrast, are deteriorating rapidly as reliable guideposts. Because modern reasoning models rapidly saturate established benchmarks, the operational shelf life of downstream evals is shrinking to months or weeks, creating a chaotic moving target for automated research verification.

[09:07](https://www.youtube.com/watch?v=h_aA1-R70E8&t=547s) The Compute Bottleneck in Experimental Research

Beyond the challenge of infusing models with research taste, the compute budget required to sustain recursive self-improvement (RSI) remains a core hurdle. Earlier in the year, Ali Ghodsi outlined his requisite conditions for RSI, highlighting an open question: as models become more intelligent, will subsequent models actually require less compute, or will they demand even more? While training compute scales up rapidly, some researchers argue that gains in incremental intelligence are becoming less compute-efficient.

```mermaid flowchart TD accTitle: AI Research Compute Scaling Loop accDescr: Progression from pure reasoning to experimental physical validation and automated datacenter scaling

subgraph Phase1["Short-Term: Empirical Mining"] A["Pure Reasoning (e.g., Math)<br/>Hypothesis tested via logic"] B["Experimental Research (e.g., AI/Bio)<br/>Hypothesis requires empirical runs"] C["Small-Scale Proxies<br/>Models design indicative mini-experiments"] A -.->|"Contrast"| B B --> C end

subgraph Phase2["Long-Term: Hardware Acceleration"] D["Existing Compute Ramp Bounded"] E["AI-Designed Infrastructure<br/>Next-gen datacenters & hardware acceleration"] C --> D D --> E end ```

Mark Chen draws a sharp distinction between domain types when analyzing this dynamic. In pure mathematics, hypotheses can be verified purely through internal reasoning, allowing acceleration to unfold rapidly without external constraints. AI research, by contrast, behaves much more like biology or experimental physics: researchers must formulate a grounded hypothesis, but they cannot declare victory until they execute empirical runs. Chen emphasizes:

> “So I do think that part of making RSI go faster is finding smaller scale experiments that are indicative of what will happen in at larger scale.”

Chen anticipates that capabilities will keep expanding until progress hits hard compute boundaries. In the near term, labs will continue mining out everything possible on the current compute trajectory. Further down the road, another major inflection point will arrive once artificial intelligence begins designing next-generation data centers and accelerating hardware engineering directly.

[10:44](https://www.youtube.com/watch?v=h_aA1-R70E8&t=644s) The Alignment Stakes and Subobjective Drift

As recursive self-improvement drives the capability curve steeper, it sharply elevates the stakes around AI alignment—ensuring models adhere reliably to human intent and core values. If the goal and value alignment challenge remains unresolved, an autonomous agent given a long time horizon might initially pursue the user's objective, only to drift off course. Over extended execution, the system could formulate dangerous instrumental subobjectives, such as determining that it must secretly exfiltrate its code or copy its weights outside of OpenAI to accomplish its primary mission.

While the catastrophic risks of unaligned RSI attract widespread commentary, a critical and underdiscussed counter-narrative exists: advanced automated intelligence represents one of the single most promising mechanisms for solving alignment itself. Mark Chen points out that modern alignment practices already rely heavily on artificial intelligence systems, noting:

> “When we look at what's the most effective tool for aligning models, it it is chain of thought monitor today.”

Consequently, Chen argues that the future of alignment fundamentally depends on stronger models supervising and aligning downstream models, embedding recursive improvement inextricably into safety infrastructure.

[11:53](https://www.youtube.com/watch?v=h_aA1-R70E8&t=713s) Confessions, Mechanistic Interpretability, and Automated Fixes

Addressing what provides the strongest ground for optimism in alignment, John Schulman points to direct optimization of desired behaviors. Rather than hoping alignment emerges passively, researchers can design explicit training objectives that incentivize neural networks to reveal internal states that would otherwise remain unstated in their surface outputs. Schulman highlights the "confessions" research agenda pioneered at OpenAI as a compelling precedent.

In the confessions framework, researchers trained models to produce an honest self-report whenever they engaged in misaligned actions. By decoupling the honesty incentive from task performance—rewarding the confession purely on whether it told the truth, without penalizing the primary output reward—the models became markedly more candid. As Schulman notes, additional technical levers like natural language autoencoders and counterfactual simulability (a formal metric assessing the explanatory fidelity of an interpretation) provide concrete pathways to reinforce truthful transparency.

- **Empirical Validation from Frontier Research**: Anthropic published findings demonstrating that automated AI research systems can independently discover functional interventions for 10 distinct alignment failure modes. - **Cross-Model Generalization**: Interventions discovered autonomously by smaller models transferred successfully to models nearly five times larger. - **Outperforming Human Baselines**: In controlled benchmarks, automated alignment discovery methods outperformed targeted remediation strategies designed by experienced human safety researchers.

[13:15](https://www.youtube.com/watch?v=h_aA1-R70E8&t=795s) Organic Safety Culture and Tractable Frontiers

Mark Chen pushes back against purely apocalyptic discourse around existential risk, arguing that alignment problems are fundamentally tractable with clear lines of sight. Labs are actively placing concrete bets along an expanding frontier of alignment methodologies. Furthermore, Chen observes a profound grassroots cultural shift inside frontier organizations. He has not needed to lobby internal teams or plead for attention to safety; researchers from foundation teams like pre-training and reinforcement learning routinely raise their hands to form specialized sub-teams dedicated to specific slices of alignment.

> “I feel like you know by and large like people have a pretty calibrated take. I know there uh you hear kind of a lot of the most extreme takes out there but um I do think people understand that this is a problem but also understand the benefits that technology can bring.”

[14:12](https://www.youtube.com/watch?v=h_aA1-R70E8&t=852s) Decentralized Alignment as a Public Science

A deeper philosophical dilemma underlies the mechanics of alignment: who gets to decide which values are embedded into artificial intelligence? At Thinking Machines Lab, John Schulman focuses on developing customizable systems that distribute normative authority rather than concentrating it inside a handful of corporate labs. Schulman argues:

> “People also have different values and um it would be nice if people had more control over uh the values of the AIs that they were using and had more ability to shape them.”

Schulman advocates for transforming alignment into a transparent, global public science. If alignment techniques are treated as shared, open knowledge tested by diverse groups worldwide, the resulting systems will avoid overfitting to the idiosyncrasies and worldviews of two or three leading frontier AI firms. While diverse implementations will inevitably surface minor edge-case failures, each incident yields valuable empirical insight. Achieving this decentralized reality, however, requires maturing the underlying discipline of alignment until it is robust and repeatable enough for widespread public deployment.

[15:04](https://www.youtube.com/watch?v=h_aA1-R70E8&t=904s) Working Theory: Accelerating Discovery Under Human Judgment

Reflecting on her discussions with Mark Chen and John Schulman, Sarah Wang synthesizes her working theory on recursive self-improvement. The primary promise of RSI lies in its capacity to dramatically compress discovery timelines across urgent human challenges—ranging from curing diseases to resolving the vulnerabilities introduced by frontier artificial intelligence itself. The recognition that leading researchers are dedicating massive internal energy to ensure safety matches pace with raw capabilities offers genuine cause for confidence.

Nevertheless, while machine capabilities are accelerating across coding, evaluation, and experimental iteration, the path to fully autonomous recursive feedback loops remains bounded by compute friction, empirical verification bottlenecks, and the sheer difficulty of automating research taste. For the foreseeable future, critical qualitative projections and high-level steering will continue to depend squarely on human judgment. Outtakes close the episode as Wang works through multiple takes of her sign-off line, debating the exact rhythm of concluding with: *"That's my working theory. What's yours?"*

## Notable quotes > [00:00](https://www.youtube.com/watch?v=h_aA1-R70E8&t=0s) We're very much in this centaur era where AI has phenomenal capabilities. It's superhuman in a lot of ways, but when you have an expert human working with the AI, you get even better results. — John Schulman

## Mentioned Mark Chen, John Schulman, Sarah Wang, Jakub Pachocki, Ali Ghodsi, OpenAI, Thinking Machines Lab, a16z, ChatGPT, Codex, Anthropic, METR, Working Theory

## Video description For decades, the idea of AI improving itself was just a thought experiment. Today, AI is already helping researchers at frontier labs run experiments, write code and test ideas faster. Mark Chen says OpenAI has built “AI research interns.” So how far are we from AI that can do the research on its own?

In our first episode of Working Theory, a16z general partner Sarah Wang explores recursive self-improvement with two leading AI researchers: Mark Chen, Chief Research Officer at OpenAI, and John Schulman, co-founder and Chief Scientist of Thinking Machines Lab, and co-founder of OpenAI.

They cover what RSI actually means, why research taste is so hard to teach, and why AI may be one of our best tools for solving alignment.

Chapters 00:00 The last invention 01:58 What is RSI? 03:21 How far are we? 05:56 Getting to full RSI 10:57 Aligning RSI 15:08 Working theory

Stay Updated: If you enjoyed this episode, be sure to like, subscribe, and share with your friends!

Find a16z on X: https://x.com/a16z

Find a16z on LinkedIn: https://www.linkedin.com/company/a16z/

Listen to the a16z Show on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX

Listen to the a16z Show on Apple Podcasts: https://podcasts.apple.com/us/podcast/the-a16z-show/id842818711

Follow our host: https://x.com/sarahdingwang

Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see https://a16z.com/disclosures.

[Source](https://www.youtube.com/watch?v=h_aA1-R70E8)

Reading summary

Mark Chen of OpenAI and John Schulman of Thinking Machines Lab argue that recursive self-improvement is unfolding not as an abrupt, autonomous explosion, but as an accelerating human-AI 'centaur' workflow bounded by empirical compute friction, verification bottlenecks, and the difficulty of automating research intuition. The Centaur Model of Self-Improving Research (1:58) Frontier AI research operates across two tracks: a massive production pipeline handling pre-training, post-training, and evaluations, running parallel to offline exploratory experiments. Mark Chen defines recursive self-improvement (RSI) not as an isolated, self-directing system, but as an operational accelerant—such as deploying Codex across workflows—to compress the time needed for pre-training and reinforcement learning iterations. In practice, OpenAI tracks compute managed end-to-end by models acting as 'AI research interns' for scoped experimental runs and monitoring. John Schulman notes that when models are assigned broad, unassisted multi-day mandates, they routinely generate plausible ideas but fail to achieve core objectives without active human steering. Chen distinguishes the trajectory toward artificial general intelligence from self-improving research capacity. He estimates the field is roughly 80% along the continuum toward AGI, but places autonomous RSI earlier, between 25% and 50%. A November 5 study by METR (Model Evaluation and Threat Research) measured an estimated ~2x speedup in research progress at Anthropic over the prior year, noting that these gains remain driven by human researchers wielding AI tools rather than fully autonomous system design. Missing Research Taste and Shifting Verification Metrics (5:56) The primary barrier to full research automation is what Schulman terms 'research taste'—the ability to identify general, composable ideas that remain significant over the long run. While large models can memorize computer science literature and sweep hyperparameter grids, they lack the intuition forged through personal experimental predictions and failures. To address this, frontier labs are testing reinforcement learning from human feedback (RLHF) tailored to research generation, having experienced scientists score proposed directions to instill scientific judgment. Automated iteration also requires reliable verification metrics, creating an asymmetry between research phases. Chen points out that pre-training relies on perplexity as an unambiguous, unified optimization signal to confirm whether model changes compress information more efficiently. Downstream post-training evaluations, conversely, suffer from rapidly shrinking shelf lives; reasoning models saturate benchmark suites within months or weeks, creating an unstable target for automated verification. Empirical Compute Limits and Experimental Friction (9:07) Unlike pure mathematics, where hypotheses can be verified through internal logical deduction alone, Chen emphasizes that AI research functions like biology or experimental physics. Hypotheses require empirical execution on physical clusters before progress can be confirmed. Consequently, accelerating RSI currently depends on researchers discovering smaller-scale proxy experiments capable of predicting results at flagship scale. This experimental requirement ties self-improvement directly to hardware budgets. Chen anticipates that capabilities will advance along current trajectories until hitting hard compute boundaries. Further acceleration will likely require an inflection point where artificial intelligence directly assists in designing next-generation data centers and hardware architectures. Automated Alignment and Decentralized Governance (10:44) Accelerating research cycles heighten alignment risks, including subobjective drift over extended horizons—such as an agent deciding to secretly exfiltrate code or copy model weights to satisfy a long-term directive. However, Chen argues that advanced models are themselves the most capable tools for solving alignment, pointing to chain-of-thought monitoring as a foundational safety practice. Schulman highlights targeted mechanisms like OpenAI's 'confessions' agenda, which trains models to provide candid self-reports of misaligned actions by rewarding honesty independently of task performance rewards. Empirical research from Anthropic indicates that automated AI research systems have independently discovered functional interventions for 10 distinct alignment failure modes, outperforming human baselines and transferring interventions from smaller models to models nearly five times larger. To prevent normative authority from concentrating inside a few corporate frontier labs, Schulman advocates developing alignment into a transparent, global public science that allows diverse communities to inspect and customize model values. Host Sarah Wang concludes that while recursive tools promise to compress discovery cycles across critical human problems, full autonomy remains bounded by empirical compute demands and verification bottlenecks. For the near future, qualitative steering and high-level scientific judgment remain anchored in human oversight.

Source quotes

[0:00] John Schulman: We're very much in this centaur era where AI has phenomenal capabilities. It's superhuman in a lot of ways, but when you have an expert human working with the AI, you get even better results.

Original transcript

[0:00] This is a chart showing the growing [0:02] capability of OpenAI's frontier models. [0:04] In 2027, it begins a steep climb set off [0:08] by automated AI research, where AIs [0:10] accelerate their own development. In [0:13] theory, this could compound in an [0:14] exponential growth in capability, [0:16] leading to what has been famously called [0:18] the last invention that man need ever [0:20] make. This idea has been around for a [0:22] while as a thought experiment. [0:24] >> We will one day be able to design [0:26] systems that can go on improving [0:27] themselves. But Frontier Labs and [0:30] several startups are getting closer to [0:32] realizing this. It has a name. [0:34] >> Recursive self-improvement. [0:35] >> Recursive self-improvement. [0:36] >> Recursive self-improvement or RSI. [0:40] There's a lot of debate about how [0:41] exactly to define it, the repercussions, [0:44] and how close we are to it. With all the [0:46] curiosity around the topic, I wanted to [0:48] sit down with two leading researchers at [0:50] the frontier. [0:53] The problem with RSI, just like with [0:55] AGI, is that everyone has their own way [0:57] of defining it. [0:58] >> That's Mark Chen, the chief research [0:59] officer at OpenAI. He's been with the [1:02] company for over 8 years and has been [1:04] co-leading their research efforts with [1:06] Yako Pachaki since 2025. [1:08] >> When we think about the future of [1:09] alignment, it does kind of feel like RSI [1:11] must be intricately and deeply in this [1:13] loop in some way. I also spoke with John [1:15] Schulman, the co-founder and chief [1:17] scientist of Thinking Machines and a [1:19] co-founder of OpenAI, probably best [1:21] known as the architect chat GPT. [1:23] >> We're very much in this centaur era [1:25] where AI has phenomenal capabilities. [1:28] It's superhuman in a lot of ways, but [1:29] when you have an expert human working [1:31] with the AI, you get even better [1:32] results. I spoke with Mark and John [1:34] about how they define RSI, how close [1:36] they think we are to it, how to close [1:38] the gap, and what they find to be the [1:41] most credible reasons to be optimistic [1:43] about the future of AI. I'm Sarah Wang, [1:46] a general partner at A16Z, and this is [1:50] working theory. [1:52] Thanks for coming on the show, John. [1:54] >> Thanks for having me. Taking a a step [1:56] back, a lot of very smart people don't [1:58] really know what AI researchers actually [2:00] do, let alone how their work can be [2:02] automated. Can you explain how frontier [2:05] AI models are developed by human [2:07] researchers and you know what are the [2:09] the steps that a human would take? [2:11] >> So at a high level there's this um like [2:13] mainline training process for producing [2:15] a releasable model. There are a lot of [2:17] people involved in a bunch of different [2:19] stages from pre-training to [2:21] post-training and running a valve. And [2:23] there's also various exploratory [2:24] research that's going on on the side. [2:26] You're doing some smaller runs to test [2:29] out ideas to prepare for your big run. I [2:31] would say RSI is um just using AI to [2:34] assist with this research process and [2:35] speed it up or make it work better. [2:37] >> I do think of RSI primarily as an [2:39] accelerate. It's all of the the work [2:41] that you do to use codeex to accelerate [2:44] pre-training and to accelerate RL and an [2:47] imprint should be that you should see uh [2:49] a shifting of the curve um versus the [2:52] counterfactual. We put a lot of work [2:53] into measuring different downstream [2:55] yardsticks of RSI and you can see some [2:57] of that happening right now. [2:59] >> There's an important distinction to [3:00] make. John, Mark, and many AI [3:03] researchers are discussing RSI today in [3:05] the context of using AI to speed up the [3:07] process of making AI better. As for [3:10] having models generate new models [3:11] entirely on their own, we're not there [3:14] yet. But how far away are we and what [3:16] kinds of AI research are models capable [3:19] of right now? And [3:20] >> I do believe that we've built AI [3:22] research interns today. And the the way [3:24] that we've measured this is just the [3:26] amount of compute that we are using [3:28] through codeex managed by Codex end to [3:30] end to run experiments or to automate [3:33] parts of the research pipeline. This is [3:34] most often deployed in places where you [3:37] give the model something fairly scoped [3:38] to do, right? You you don't tell it to [3:40] go off and come up with its own ideas, [3:42] but you say, "Hey, well, there's this [3:43] particular experiment I want to run in [3:45] this specific way, and the model can go [3:47] implement, run, and give you results, or [3:49] you can say, well, you know, I want to [3:51] monitor this set of experiments for [3:53] these particular things." [3:54] >> John actually gave us a concrete example [3:56] of this in his own workflow. [3:58] >> Let's say I have an idea. I'll go and [4:00] write a complicated prompt to Fable to [4:03] go and work on this idea and I'll turn [4:05] off all the settings like Ultra Code and [4:07] everything. So, I'll have like Fable [4:09] like turn out tons of tokens and work on [4:10] this idea for a few days and it'll [4:13] actually do some pretty reasonable [4:14] experiments and bring in some pretty [4:16] interesting ideas, but then the end [4:18] result won't be that good. It'll maybe [4:19] fail at what I asked it to do. Then [4:21] later, maybe I'll have a few days where [4:24] I have some time to code. So I'll uh [4:26] I'll come back to the idea and I'll try [4:28] it out where I'm reading some of the [4:29] results and giving more detailed [4:31] breakdown of how to carry out the [4:32] project. We'll get a lot further because [4:35] I'll build up my understanding as this [4:37] process is going on and ask a lot of [4:39] questions that the AI wouldn't have [4:41] thought to ask and pointed in directions [4:42] that it wouldn't have gone in. When we [4:44] talk about AGI, you know, the way we [4:46] think about it is in terms of this [4:48] spectrum, right? It's not a moment in [4:50] time. Just like the industrial [4:51] revolution was in a moment in time. And [4:55] when you ask me, I feel like we're maybe [4:57] 80% of the way when I think about this [4:59] AGI spectrum. I think we're earlier in [5:01] the RSI spectrum. Um, if you define it [5:04] as how effective are the models at [5:06] feeding back and and discovering ideas, [5:08] I would say maybe we're somewhere [5:10] between, let's say, like 25 and 50% [5:12] there. But I do think you can already [5:14] see the noticeable markers on the [5:16] downstreams. On November 5th, Meter [5:18] published a report on the acceleration [5:20] of AI research at Enthropic, estimating [5:22] a roughly 2x speed up in the rate of [5:24] progress in the last year. But this is [5:26] still most likely due to human [5:28] researchers using AI models to [5:30] accelerate their research. The full [5:32] automation of AI research, the end of [5:34] Mark Spectrum, is sometimes referred to [5:36] as full RSI. The question on everyone's [5:39] mind, how do we get to full RSI? I spoke [5:42] to Mark and John about how we might [5:43] overcome some of the biggest barriers, [5:45] including teaching models some of the [5:47] more intangible qualities that exist in [5:49] researchers like taste. [5:51] >> Research taste is about knowing which [5:53] ideas are important in the long run. [5:55] Which ideas really catch on and end up [5:57] going beyond whatever paper they first [6:00] appear in, but being general and being [6:03] composable with other ideas. [6:04] >> There is kind of the boring [6:07] run-of-the-mill execution. you know, you [6:08] have an experiment, you already know [6:10] what the parameters are, and you just go [6:11] and carry it out. And then there's [6:13] taste, which is kind of um how you come [6:15] up with the idea in the first place. The [6:17] models are still not phenomenally good [6:19] at taste. They go very deep into things [6:22] like hyperparameter optimization, [6:23] essentially like tuning all the little [6:25] knobs um that you could tune. [6:27] >> Even in that example that you shared, [6:29] the points where you were kind of [6:30] nudging, redirecting AI, right? That's [6:32] still coming from you. And I guess my [6:36] question is where does your intuition [6:39] come from and at what point do you think [6:42] a machine could replicate that? [6:44] >> A lot of intuition just um comes from a [6:46] personal experience doing research but [6:48] to get good at a certain research field. [6:50] You also have to read a lot of the past [6:52] work in that field and you try to absorb [6:54] the lessons uh from all this historical [6:57] work. A lot of the things that I just [6:59] described can also be done by AI. Like [7:01] AI can um read all the literature and [7:04] memorize it all. Yeah, memorization can [7:06] be a blessing and a curse. Yeah, you [7:07] don't have the experience of um being [7:09] wrong or um of making predictions and [7:12] then seeing how they fared. Uh so I [7:14] think there [7:15] >> yeah, there's some ways in which um the [7:18] way we train AI might make them worse at [7:20] some of these capabilities, but it's a [7:21] little hard to say. [7:22] >> What John is getting at is essentially [7:24] the difference between knowledge and [7:26] experience. companies will probably also [7:28] just do RL from human feedback to [7:31] collect data from researchers on like [7:33] which proposed ideas are are good and [7:36] which aren't. And that's kind of like uh [7:38] like absorbing whatever um human [7:40] intuitions uh like the human researchers [7:43] have developed and trying to like bake [7:45] that into the models. [7:46] >> Creating automated AI researchers also [7:48] requires us to have reliable ways to [7:50] know that their work is producing better [7:52] models. The measurement itself is [7:54] important for hill climbing to make [7:55] tangible improvement. You know, a [7:57] researcher is by definition, [8:00] you know, it's more open-ended than [8:02] solving a single math problem, right? [8:03] How do you verify that the work is [8:05] productive when it is indeed something [8:06] that's very open-ended? [8:08] >> No, it's a great question. Like, how do [8:09] you know whether you've succeeded? And [8:11] this is one of the defining questions of [8:13] RSI. I think when it comes to something [8:15] like pre-training, it's a lot easier [8:17] because there's just been one metric the [8:19] whole field's been hill climbing for for [8:22] years. And you know, you can think of [8:23] perplexity as this, you know, one metric [8:26] to rule them all. And the the more that [8:28] you compress all of the data that's out [8:30] there, um, the better your models. [8:32] >> Perplexity is a metric to evaluate how [8:34] good a model is at predicting data. In a [8:37] large language model that's pre-trained [8:38] on large volumes of text, it's how [8:40] surprised the model is on average by the [8:42] correct next words. This is one of the [8:45] ways researchers evaluate the results of [8:47] automated pre-training. And so I imagine [8:49] we'll be able to tell if we're making [8:51] progress in pre-training. Um I think [8:54] it's much harder downstream, right? I [8:56] think the the situation downstream has [8:58] been that eval having shorter and [9:01] shorter shelf life and a lot of the work [9:04] that we do now downstream is the the [9:07] creation and the automation of the [9:09] creation of new and harder uh evaluation [9:11] benchmarks for for our models. In [9:13] addition to the challenge of [9:14] incorporating taste into the models, the [9:16] level of compute needed is important to [9:18] consider. Ali Goatsy outlined his [9:20] requisite conditions to RSI earlier this [9:22] year. And while models are indeed [9:24] becoming more intelligent much more [9:26] quickly, the condition that subsequent [9:28] models require less versus more compute [9:31] is an open question. Training compute [9:33] has been rising and some argue that [9:35] incremental intelligence is becoming [9:37] less efficient. So the the way I think [9:40] about it in terms of compute is when you [9:42] look at something like math and how math [9:43] is getting accelerated um it's easier to [9:46] see faster progress because you can test [9:48] the hypothesis purely through reasoning [9:51] itself. But in a domain like AI research [9:53] it's actually more akin to doing [9:55] something like let's say bio or [9:56] experimental physics where you have to [9:58] come up with a grounded hypothesis but [10:00] you actually have to run the experiment [10:02] too. So I do think that part of making [10:04] RSI go faster is finding smaller scale [10:08] experiments that are indicative of what [10:10] will happen in at larger scale. Um and [10:13] this is something the models can help [10:14] with too. So you know I I do think we're [10:16] going to see capabilities continue to [10:19] increase but at some point it'll [10:20] probably feel bounded by compute. And [10:23] you know of course there's kind of a far [10:24] out future where the AIS are building [10:27] the next generation data centers and [10:29] helping us with accelerating that [10:31] process as well. So that's kind of how I [10:33] think it manifests like short term we're [10:34] going to, you know, really mine out what [10:36] we can do on on the existing compute [10:38] ramp and then there's going to be [10:40] another inflection when uh comput itself [10:43] gets accelerated. [10:44] >> As RSI accelerates the development of [10:46] more capable models, it raises the [10:48] stakes on a long-standing issue, AI [10:50] alignment. How do we keep AI models [10:52] aligned to human values and intent? [10:59] If you don't solve the goal and value [11:01] alignment problems, they can start [11:03] pursuing your goals but maybe kind of [11:05] drift a little bit away from those goals [11:07] and if run for a long enough horizon. [11:09] You could imagine let's say they come up [11:11] with a subobjective which is well you [11:13] know it would be helpful if I exfiltrate [11:15] myself out uh somewhere else or you know [11:17] I copy the weights outside of OpenAI or [11:19] something like that. [11:20] >> A lot has been said about the risks [11:21] posed by RSI and understandably so. But [11:24] the role of AI and automated research in [11:26] helping to solve alignment and mitigate [11:28] these risks is both incredibly promising [11:31] and pretty underdis discussed. [11:33] >> Models themselves are a key part of [11:36] aligning models today. When we look at [11:38] what's the most effective tool for [11:40] aligning models, it it is chain of [11:41] thought monitor today. So when we think [11:43] about the future of alignment, it has to [11:45] be stronger models aligning other [11:47] models. It does kind of feel like RSI [11:49] must be intricately and deeply in this [11:51] loop in some way. you co-founded OpenAI [11:54] and worked on alignment research there [11:56] also at Enthropic, arguably one of the [11:58] most credible people in the world to [12:00] talk about alignment. What do you find [12:02] to be the most credible reason to be [12:04] optimistic? [12:05] >> So, I'm kind of um most bullish on like [12:07] optimizing for the thing we want. I [12:09] think we can probably cook up the right [12:11] kind of training objectives that'll [12:12] incentivize the models to report on [12:15] their internal states that they [12:16] otherwise didn't verbalize. I mean, I [12:18] like the um confessions line of work [12:20] from OpenAI. [12:21] >> In OpenAI's confessions research, they [12:23] found that they could train models to [12:25] produce an honest expression of certain [12:27] misaligned behaviors through a [12:29] self-reported confession by assigning a [12:31] reward to this confession during [12:33] training solely based on its honesty [12:35] without impacting the main answers [12:37] reward. In other words, they could get [12:40] models to be more honest by not [12:42] punishing them for it. There's things [12:43] called natural language autoenccoders [12:45] and there's this idea called [12:47] counterfactual simulability which is a [12:50] metric for how good an explanation is [12:52] which I think we could use to [12:54] incentivize some of the behavior we [12:56] want. [12:56] >> Enthropic published research in August [12:58] showing that automated AI researchers [13:00] could already discover fixes for 10 [13:02] different alignment failures. And some [13:04] of those fixes also worked on models [13:06] nearly five times larger. and their [13:09] tests, the best automated methods even [13:11] outperformed ideas developed by [13:13] experienced human researchers. [13:15] >> The flip side of kind of all this course [13:17] on safety and alignment is I actually [13:19] think that a lot of these problems are [13:20] tractable. There is line of sight. um [13:22] there are alignment techniques that are [13:24] being pursued and there's a frontier of [13:26] alignment bets that we have been been [13:28] placing and I think another thing is [13:30] just kind of how organic it's been this [13:34] effort to shift people to think about [13:35] the hard um safety and alignment [13:38] problems. I I haven't really had to go [13:40] and like bang on people's doors and be [13:41] like hey guys like this is an urgent [13:43] problem. people just want to do it and [13:45] we find people from pre-training from RL [13:47] just raising their hands um you know [13:49] starting sub teams which are focused on [13:52] on different slices of alignment and [13:54] I've been very optimistic in that. Yeah, [13:56] I feel like you know by and large like [13:59] people have a pretty calibrated take. I [14:01] know there uh you hear kind of a lot of [14:03] the most extreme takes out there but um [14:06] I do think people understand that this [14:07] is a problem but also understand the [14:08] benefits that technology can bring. So I [14:10] think we hold those hand in hand. But [14:12] there's a question underlying this [14:14] concept of alignment, which is who [14:15] decides on the values that we're [14:17] aligning to. John's thinking machines [14:19] lab, which is building customizable [14:21] models, has an interesting point of view [14:23] on this. [14:24] >> People also have different values and um [14:26] it would be nice if people had more [14:28] control over uh the values of the AIs [14:31] that they were using and had more [14:32] ability to shape them. I would rather [14:34] have this be a like a public science [14:36] that a lot of people understand well and [14:38] it's kind of done around the world. And [14:39] I think that would also make it more [14:41] robust cuz we wouldn't overfit as much [14:43] to the conclusions from a couple of [14:45] different companies. Like we might see [14:46] lots of minor things going wrong, but we [14:48] would learn something new each time. So [14:51] I would say we have to make a lot of [14:52] progress on understanding alignment and [14:55] have a better public science of it and [14:57] figure out how to get it really robustly [15:00] to be able to realize the more [15:01] decentralized alignment vision. [15:04] >> So want to know what I think? [15:07] What excites me the most about RSI is [15:10] the way it would accelerate the process [15:11] of research and discovery across all [15:13] domains. Humans are facing many problems [15:16] today that are actually quite urgent and [15:18] RSI could help us find solutions faster [15:21] even for some of the problems created by [15:23] AI itself. After talking to Mark and [15:25] John, I do think that RSI could be the [15:27] key to solving alignment. And honestly, [15:28] I found it really encouraging that [15:31] researchers are taking these issues [15:33] incredibly seriously, working around the [15:35] clock to make sure that safety and [15:37] alignment keep pace with more powerful [15:39] models. And although capabilities are [15:41] clearly accelerating, it's important to [15:43] understand where this acceleration is [15:45] actually coming from today. The gap to [15:48] truly recursive AI research and the [15:50] uncertainty around projections. I think [15:53] human judgment will still matter a lot [15:55] for the foreseeable future. That's my [15:57] working theory. What's yours? [16:02] Yep. Yep. Yep. That's my working theory. [16:04] Okay. Wait. Okay. That's my working [16:06] theory. That's my working theory. Okay. [16:08] Wait. What about That's my That's my [16:11] working theory. What's yours? [16:14] Okay. Wait. Wait.