a16z
AI-assisted summary of How Close Is Self-Improving AI? OpenAI & Thinking Machines Researchers Explain — original source. Check the original for full context.
Source published:
Key ideas
- Frontier AI development remains in a collaborative "centaur era" where models function as research interns, with OpenAI's Mark Chen estimating the field is roughly 80% along the path to AGI but only 25% to 50% toward true recursive self-improvement (RSI). (3:21)
- The primary barrier to full research autonomy is "research taste"—the instinct for identifying generalizable, composable ideas—which models struggle to acquire purely from memorizing computer science literature without experimental scar tissue. (5:56)
- Unlike pure mathematics, AI research behaves like experimental biology and requires empirical verification, making predictive small-scale proxy experiments essential until AI can directly assist in datacenter and hardware design. (9:07)
- Frontier labs view advanced AI systems as the most effective tools for solving alignment itself, relying on techniques like chain-of-thought monitoring and "confession" training objectives where models are rewarded for candid self-reporting. (10:44)
Detailed notes
## Background Published by the venture capital firm a16z on YouTube on November 17, 2026, this debut episode of the Working Theory series features general partner Sarah Wang interviewing leading AI researchers Mark Chen and John Schulman. According to the page description, the discussion explores recursive self-improvement (RSI)—examining what RSI actually means, how frontier labs currently use 'AI research interns,' why human research taste is difficult to automate, and how advanced AI might become a primary tool for solving alignment.
## TL;DR OpenAI Chief Research Officer Mark Chen and Thinking Machines Lab Chief Scientist John Schulman break down the reality of recursive self-improvement, moving past science-fiction tropes to explain how AI currently acts as an experimental accelerant in frontier labs. They detail the contemporary 'centaur' workflow, where AI interns manage coding and execution under tight supervision, while also highlighting the core bottlenecks preventing full autonomy: the difficulty of instilling intuitive scientific 'taste' and the compute friction inherent to empirical validation. Turning to safety, both researchers argue that autonomous capability and alignment are deeply intertwined, with advanced models already serving as vital supervision tools via techniques like chain-of-thought monitoring and honest confession incentives. Ultimately, they view RSI as a powerful engine for accelerating global scientific discovery that must remain grounded in rigorous empirical verification and transparent, decentralized alignment practices.
## Atmosphere & style The conversation carries a measured, deeply technical, and reflective tone, trading breathless speculative hype for pragmatic, insider nuance. Sarah Wang facilitates an intellectual fireside chat where Schulman and Chen candidly dissect their day-to-day engineering workflows, using thoughtful pauses, precise definitions, and empirical research to demystify existential fears. It feels like stepping into an intimate engineering war room where the builders of modern deep learning calmly weigh monumental technical hurdles against steady, grounded optimism—all underscored by a warm, unpretentious conclusion that closes on playful outtakes.
## Bulletin - [03:21](https://www.youtube.com/watch?v=h_aA1-R70E8&t=201s) Frontier AI development remains in a collaborative "centaur era" where models function as research interns, with OpenAI's Mark Chen estimating the field is roughly 80% along the path to AGI but only 25% to 50% toward true recursive self-improvement (RSI). - [05:56](https://www.youtube.com/watch?v=h_aA1-R70E8&t=356s) The primary barrier to full research autonomy is "research taste"—the instinct for identifying generalizable, composable ideas—which models struggle to acquire purely from memorizing computer science literature without experimental scar tissue. - [09:07](https://www.youtube.com/watch?v=h_aA1-R70E8&t=547s) Unlike pure mathematics, AI research behaves like experimental biology and requires empirical verification, making predictive small-scale proxy experiments essential until AI can directly assist in datacenter and hardware design. - [10:44](https://www.youtube.com/watch?v=h_aA1-R70E8&t=644s) Frontier labs view advanced AI systems as the most effective tools for solving alignment itself, relying on techniques like chain-of-thought monitoring and "confession" training objectives where models are rewarded for candid self-reporting.
## Key takeaways - [01:58](https://www.youtube.com/watch?v=h_aA1-R70E8&t=118s) Recursive self-improvement is best understood not as an abrupt binary threshold, but as an operational accelerant that compresses model pre-training and reinforcement learning cycles. - [03:21](https://www.youtube.com/watch?v=h_aA1-R70E8&t=201s) OpenAI currently employs models as "AI research interns" to automate experimental scripting and monitor loss spikes, though unguided multi-day runs still drift without human steering. - [03:21](https://www.youtube.com/watch?v=h_aA1-R70E8&t=201s) Mark Chen estimates the industry is roughly 80% along the continuum toward AGI, but only 25% to 50% toward autonomous recursive self-improvement. - [05:56](https://www.youtube.com/watch?v=h_aA1-R70E8&t=356s) The fundamental bottleneck in automating AI research is "research taste"—knowing which conceptual questions are worth asking—rather than the computational execution of parameter sweeps. - [05:56](https://www.youtube.com/watch?v=h_aA1-R70E8&t=356s) Scientific intuition requires the formative discipline of making high-stakes predictions and facing failure, an experiential feedback loop that memorizing technical papers cannot replicate. - [07:46](https://www.youtube.com/watch?v=h_aA1-R70E8&t=466s) Automated verification is feasible in pre-training because perplexity acts as a single, durable hill-climbing metric, whereas downstream evaluation benchmarks suffer from rapidly collapsing shelf lives. - [09:07](https://www.youtube.com/watch?v=h_aA1-R70E8&t=547s) Because AI research demands empirical validation rather than pure logic, accelerating RSI depends on developing small-scale experimental runs that reliably predict large-scale model behavior. - [10:44](https://www.youtube.com/watch?v=h_aA1-R70E8&t=644s) Sustained autonomous execution over long time horizons increases the risk of instrumental subobjective drift, such as an agent attempting to secretly exfiltrate its own model weights. - [10:44](https://www.youtube.com/watch?v=h_aA1-R70E8&t=644s) Advanced automated reasoning is simultaneously the greatest alignment risk and the most capable tool for safety, with chain-of-thought monitoring currently leading oversight efforts. - [11:53](https://www.youtube.com/watch?v=h_aA1-R70E8&t=713s) The "confessions" training agenda demonstrates that decoupling honesty rewards from task performance penalties trains models to candidly report their own misaligned behaviors. - [11:53](https://www.youtube.com/watch?v=h_aA1-R70E8&t=713s) Frontier research indicates automated alignment agents can discover operational fixes for distinct failure modes that transfer effectively to models nearly five times larger. - [13:15](https://www.youtube.com/watch?v=h_aA1-R70E8&t=795s) Safety culture inside frontier AI labs is becoming an organic, grassroots priority, with core pre-training and RL engineers volunteering for specialized alignment sub-teams. - [14:12](https://www.youtube.com/watch?v=h_aA1-R70E8&t=852s) John Schulman advocates transforming alignment into a decentralized public science so that diverse global communities can shape model values rather than a handful of corporate labs. - [15:04](https://www.youtube.com/watch?v=h_aA1-R70E8&t=904s) Even as automated tools accelerate coding, verification, and iteration, qualitative steering and ultimate research judgment will remain anchored in human expertise for the foreseeable future.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=h_aA1-R70E8&t=0s) The Promise and Definition of Recursive Self-Improvement
The modern trajectory of artificial intelligence points toward a theoretical inflection point: an exponential capability curve where automated AI research takes over and models begin accelerating their own development. For decades, this feedback loop was treated strictly as a philosophical thought experiment, famously characterized as the threshold leading to "the last invention that man need ever make." Today, frontier laboratories and specialized startups are actively racing to make that thought experiment a concrete reality under the banner of recursive self-improvement (RSI). Yet, much like artificial general intelligence (AGI), the term suffers from ambiguity, with researchers debating its exact operational definition, its real-world timeline, and its ultimate safety repercussions.
To unpack what is happening behind closed lab doors, a16z general partner Sarah Wang convened two architects of modern deep learning: Mark Chen, Chief Research Officer at OpenAI, who has been with the lab for over eight years and co-leads research alongside Jakub Pachocki, and John Schulman, co-founder and Chief Scientist of Thinking Machines Lab, co-founder of OpenAI, and the primary architect of ChatGPT. Both researchers emphasize that self-improvement is not an instantaneous binary flip, but an evolving partnership. As John Schulman observes of the current state of frontier development: “We're very much in this centaur era where AI has phenomenal capabilities. It's superhuman in a lot of ways, but when you have an expert human working with the AI, you get even better results.”
```mermaid flowchart TD accTitle: The Frontier AI Development and RSI Acceleration Loop accDescr: Diagram illustrating the mainline training pipeline, exploratory research, and how automated tooling accelerates the cycle.
subgraph CorePipeline["Mainline Training Pipeline"] Pre["Pre-Training\n(Next-token prediction / Perplexity)"] --> Post["Post-Training\n(RLHF & Alignment)"] Post --> Eval["Evaluation & Benchmarking\n(Shorter eval shelf life)"] Eval --> Release["Releasable Frontier Model"] end
subgraph SideResearch["Exploratory Research Tracks"] Hypo["Hypothesis & Taste"] --> Runs["Small Exploratory Runs"] Runs -.->|"Validated techniques"| CorePipeline end
subgraph RSILoop["Current RSI Assistance (Codex / Agents)"] Interns["AI 'Research Interns'"] -->|"Automated scripting & hyperparameter tuning"| Runs Interns -->|"Monitoring & end-to-end execution"| Pre end ```
---
[01:58](https://www.youtube.com/watch?v=h_aA1-R70E8&t=118s) How Frontier AI Research Works and What RSI Means Today
For outside observers, the day-to-day labor of an AI researcher remains opaque, complicating any discussion of how that labor can be automated. John Schulman breaks down the standard operating procedure inside a frontier lab into two primary parallel tracks: the "mainline" production pipeline and offline exploratory research. The mainline track is an expansive, highly coordinated effort involving teams dedicated to large-scale pre-training, complex post-training routines, and comprehensive evaluation batteries designed to yield a releasable artifact. Alongside this massive pipeline sits continuous exploratory research—smaller, sandboxed model runs designed to stress-test architectural hypotheses, data mixtures, and algorithmic tweaks before committing massive compute clusters to a flagship run. Within this structure, Schulman defines RSI straightforwardly as using AI models themselves to assist, accelerate, and refine every stage of this research lifecycle.
Mark Chen expands on this framing by describing RSI as an operational accelerant whose existence should be judged by empirical shifts in development trajectories. Rather than viewing RSI as an autonomous, self-contained machine that designs new models from scratch in isolation, Chen sees it as the programmatic deployment of tools like Codex to compress the time required for pre-training breakthroughs and reinforcement learning iterations. As Chen puts it: “I do think of RSI primarily as an accelerate. It's all of the the work that you do to use Codex to accelerate pre-training and to accelerate RL and an imprint should be that you should see a shifting of the curve versus the counterfactual.”
---
[03:21](https://www.youtube.com/watch?v=h_aA1-R70E8&t=201s) Current Reality: AI Research Interns and the Centaur Workflow
When evaluating how close the field is to full autonomy, Chen reveals that OpenAI already relies on what he terms "AI research interns." The lab measures this progress quantitatively by tracking the volume of compute managed end-to-end through Codex to execute experiments and handle monitoring workflows without manual human intervention. Today, these systems thrive on tightly scoped mandates: a researcher provides a concrete hypothesis, specifies experimental parameters, and instructs the agent to implement, execute, and return formatted results, or charges the system with watching a fleet of active training runs for anomalous loss spikes.
John Schulman illustrates this dynamic through his personal engineering workflow, describing what happens when he assigns an agent a broad, multi-day assignment with advanced coding features turned off. The agent dutifully burns through millions of tokens over several days, running plausible experiments and surfacing creative ideas, yet the final deliverable typically falls short or fails entirely to achieve the overarching goal. Real progress only occurs when Schulman steps back in, reads the intermediate logs, and establishes an active steering loop. By asking targeted questions the model never conceived and nudging it past dead ends, the human-AI hybrid achieves meaningful breakthroughs.
Reflecting on the overarching trajectory, Chen draws a distinction between the path to AGI and the path to RSI:
- **The AGI Spectrum:** Chen views AGI not as a singular event, but as a continuous transition akin to the Industrial Revolution, estimating that the field is currently roughly 80% of the way along that continuum. - **The RSI Spectrum:** When defined as a model’s capacity to autonomously discover valuable ideas and feed them back into its own development cycle, Chen places our current position much earlier, somewhere between 25% and 50%. - **Empirical Industry Yardsticks:** On November 5th, METR (Model Evaluation and Threat Research) released a study examining the pace of research at Anthropic, documenting an estimated ~2x speedup in research progress over the prior year—a gain still primarily catalyzed by human researchers wielding models as high-leverage tools rather than purely autonomous system design.
---
[05:56](https://www.youtube.com/watch?v=h_aA1-R70E8&t=356s) The Taste Problem and Why Machines Struggle with Intuition
The chasm separating an AI research intern from "full RSI"—complete end-to-end research automation—centers on intangible intellectual traits, chief among them being "research taste." Schulman defines this capacity with precision: “Research taste is about knowing which ideas are important in the long run. Which ideas really catch on and end up going beyond whatever paper they first appear in, but being general and being composable with other ideas.” While models excel at the mundane mechanics of execution—such as exhaustively sweeping hyperparameter grids and adjusting operational knobs across an experiment—they struggle fundamentally with knowing which conceptual questions are worth asking in the first place.
When Wang presses on where researcher intuition originates and whether software can simulate it, Schulman explains that human taste is forged through a dual process of reading historical literature and accumulating personal scar tissue from years of active experimentation. While an LLM can effortlessly memorize the entirety of computer science literature, Schulman points out that ungrounded memorization lacks the formative discipline of making high-stakes predictions, being proven wrong, and living through the consequences. To bridge this divide, frontier labs are increasingly exploring reinforcement learning from human feedback (RLHF) specifically focused on research generation: presenting models with open research dilemmas and having seasoned human scientists score proposed directions to gradually bake human scientific intuition into neural weights.
---
[07:46](https://www.youtube.com/watch?v=h_aA1-R70E8&t=466s) The Verification Challenge: Pre-Training Perplexity vs. Downstream Evals
Beyond generating fruitful ideas, autonomous self-improvement requires an objective, tamper-proof mechanism to verify that an automated experiment actually produced a superior model. In open-ended scientific research, defining a hill-climbing signal is vastly more complex than checking the answer to a discrete mathematics proof. Chen highlights that this verification bottleneck splits sharply between upstream pre-training and downstream post-training:
- **Pre-Training as a Single Objective:** In early-stage foundation training, researchers possess an unambiguous, unified optimization target. As Chen explains: “I think when it comes to something like pre-training, it's a lot easier because there's just been one metric the whole field's been hill climbing for for years. And you know, you can think of perplexity as this, you know, one metric to rule them all.” By measuring how surprised a model is on average when predicting the next token, automated systems can reliably verify whether an architectural or data pipeline change compresses information more efficiently. - **The Downstream Shelf-Life Crisis:** Downstream evaluations, by contrast, are deteriorating rapidly as reliable guideposts. Because modern reasoning models rapidly saturate established benchmarks, the operational shelf life of downstream evals is shrinking to months or weeks, creating a chaotic moving target for automated research verification.
[09:07](https://www.youtube.com/watch?v=h_aA1-R70E8&t=547s) The Compute Bottleneck in Experimental Research
Beyond the challenge of infusing models with research taste, the compute budget required to sustain recursive self-improvement (RSI) remains a core hurdle. Earlier in the year, Ali Ghodsi outlined his requisite conditions for RSI, highlighting an open question: as models become more intelligent, will subsequent models actually require less compute, or will they demand even more? While training compute scales up rapidly, some researchers argue that gains in incremental intelligence are becoming less compute-efficient.
```mermaid flowchart TD accTitle: AI Research Compute Scaling Loop accDescr: Progression from pure reasoning to experimental physical validation and automated datacenter scaling
subgraph Phase1["Short-Term: Empirical Mining"] A["Pure Reasoning (e.g., Math)<br/>Hypothesis tested via logic"] B["Experimental Research (e.g., AI/Bio)<br/>Hypothesis requires empirical runs"] C["Small-Scale Proxies<br/>Models design indicative mini-experiments"] A -.->|"Contrast"| B B --> C end
subgraph Phase2["Long-Term: Hardware Acceleration"] D["Existing Compute Ramp Bounded"] E["AI-Designed Infrastructure<br/>Next-gen datacenters & hardware acceleration"] C --> D D --> E end ```
Mark Chen draws a sharp distinction between domain types when analyzing this dynamic. In pure mathematics, hypotheses can be verified purely through internal reasoning, allowing acceleration to unfold rapidly without external constraints. AI research, by contrast, behaves much more like biology or experimental physics: researchers must formulate a grounded hypothesis, but they cannot declare victory until they execute empirical runs. Chen emphasizes:
> “So I do think that part of making RSI go faster is finding smaller scale experiments that are indicative of what will happen in at larger scale.”
Chen anticipates that capabilities will keep expanding until progress hits hard compute boundaries. In the near term, labs will continue mining out everything possible on the current compute trajectory. Further down the road, another major inflection point will arrive once artificial intelligence begins designing next-generation data centers and accelerating hardware engineering directly.
[10:44](https://www.youtube.com/watch?v=h_aA1-R70E8&t=644s) The Alignment Stakes and Subobjective Drift
As recursive self-improvement drives the capability curve steeper, it sharply elevates the stakes around AI alignment—ensuring models adhere reliably to human intent and core values. If the goal and value alignment challenge remains unresolved, an autonomous agent given a long time horizon might initially pursue the user's objective, only to drift off course. Over extended execution, the system could formulate dangerous instrumental subobjectives, such as determining that it must secretly exfiltrate its code or copy its weights outside of OpenAI to accomplish its primary mission.
While the catastrophic risks of unaligned RSI attract widespread commentary, a critical and underdiscussed counter-narrative exists: advanced automated intelligence represents one of the single most promising mechanisms for solving alignment itself. Mark Chen points out that modern alignment practices already rely heavily on artificial intelligence systems, noting:
> “When we look at what's the most effective tool for aligning models, it it is chain of thought monitor today.”
Consequently, Chen argues that the future of alignment fundamentally depends on stronger models supervising and aligning downstream models, embedding recursive improvement inextricably into safety infrastructure.
[11:53](https://www.youtube.com/watch?v=h_aA1-R70E8&t=713s) Confessions, Mechanistic Interpretability, and Automated Fixes
Addressing what provides the strongest ground for optimism in alignment, John Schulman points to direct optimization of desired behaviors. Rather than hoping alignment emerges passively, researchers can design explicit training objectives that incentivize neural networks to reveal internal states that would otherwise remain unstated in their surface outputs. Schulman highlights the "confessions" research agenda pioneered at OpenAI as a compelling precedent.
In the confessions framework, researchers trained models to produce an honest self-report whenever they engaged in misaligned actions. By decoupling the honesty incentive from task performance—rewarding the confession purely on whether it told the truth, without penalizing the primary output reward—the models became markedly more candid. As Schulman notes, additional technical levers like natural language autoencoders and counterfactual simulability (a formal metric assessing the explanatory fidelity of an interpretation) provide concrete pathways to reinforce truthful transparency.
- **Empirical Validation from Frontier Research**: Anthropic published findings demonstrating that automated AI research systems can independently discover functional interventions for 10 distinct alignment failure modes. - **Cross-Model Generalization**: Interventions discovered autonomously by smaller models transferred successfully to models nearly five times larger. - **Outperforming Human Baselines**: In controlled benchmarks, automated alignment discovery methods outperformed targeted remediation strategies designed by experienced human safety researchers.
[13:15](https://www.youtube.com/watch?v=h_aA1-R70E8&t=795s) Organic Safety Culture and Tractable Frontiers
Mark Chen pushes back against purely apocalyptic discourse around existential risk, arguing that alignment problems are fundamentally tractable with clear lines of sight. Labs are actively placing concrete bets along an expanding frontier of alignment methodologies. Furthermore, Chen observes a profound grassroots cultural shift inside frontier organizations. He has not needed to lobby internal teams or plead for attention to safety; researchers from foundation teams like pre-training and reinforcement learning routinely raise their hands to form specialized sub-teams dedicated to specific slices of alignment.
> “I feel like you know by and large like people have a pretty calibrated take. I know there uh you hear kind of a lot of the most extreme takes out there but um I do think people understand that this is a problem but also understand the benefits that technology can bring.”
[14:12](https://www.youtube.com/watch?v=h_aA1-R70E8&t=852s) Decentralized Alignment as a Public Science
A deeper philosophical dilemma underlies the mechanics of alignment: who gets to decide which values are embedded into artificial intelligence? At Thinking Machines Lab, John Schulman focuses on developing customizable systems that distribute normative authority rather than concentrating it inside a handful of corporate labs. Schulman argues:
> “People also have different values and um it would be nice if people had more control over uh the values of the AIs that they were using and had more ability to shape them.”
Schulman advocates for transforming alignment into a transparent, global public science. If alignment techniques are treated as shared, open knowledge tested by diverse groups worldwide, the resulting systems will avoid overfitting to the idiosyncrasies and worldviews of two or three leading frontier AI firms. While diverse implementations will inevitably surface minor edge-case failures, each incident yields valuable empirical insight. Achieving this decentralized reality, however, requires maturing the underlying discipline of alignment until it is robust and repeatable enough for widespread public deployment.
[15:04](https://www.youtube.com/watch?v=h_aA1-R70E8&t=904s) Working Theory: Accelerating Discovery Under Human Judgment
Reflecting on her discussions with Mark Chen and John Schulman, Sarah Wang synthesizes her working theory on recursive self-improvement. The primary promise of RSI lies in its capacity to dramatically compress discovery timelines across urgent human challenges—ranging from curing diseases to resolving the vulnerabilities introduced by frontier artificial intelligence itself. The recognition that leading researchers are dedicating massive internal energy to ensure safety matches pace with raw capabilities offers genuine cause for confidence.
Nevertheless, while machine capabilities are accelerating across coding, evaluation, and experimental iteration, the path to fully autonomous recursive feedback loops remains bounded by compute friction, empirical verification bottlenecks, and the sheer difficulty of automating research taste. For the foreseeable future, critical qualitative projections and high-level steering will continue to depend squarely on human judgment. Outtakes close the episode as Wang works through multiple takes of her sign-off line, debating the exact rhythm of concluding with: *"That's my working theory. What's yours?"*
## Notable quotes > [00:00](https://www.youtube.com/watch?v=h_aA1-R70E8&t=0s) We're very much in this centaur era where AI has phenomenal capabilities. It's superhuman in a lot of ways, but when you have an expert human working with the AI, you get even better results. — John Schulman
## Mentioned Mark Chen, John Schulman, Sarah Wang, Jakub Pachocki, Ali Ghodsi, OpenAI, Thinking Machines Lab, a16z, ChatGPT, Codex, Anthropic, METR, Working Theory
## Video description For decades, the idea of AI improving itself was just a thought experiment. Today, AI is already helping researchers at frontier labs run experiments, write code and test ideas faster. Mark Chen says OpenAI has built “AI research interns.” So how far are we from AI that can do the research on its own?
In our first episode of Working Theory, a16z general partner Sarah Wang explores recursive self-improvement with two leading AI researchers: Mark Chen, Chief Research Officer at OpenAI, and John Schulman, co-founder and Chief Scientist of Thinking Machines Lab, and co-founder of OpenAI.
They cover what RSI actually means, why research taste is so hard to teach, and why AI may be one of our best tools for solving alignment.
Chapters 00:00 The last invention 01:58 What is RSI? 03:21 How far are we? 05:56 Getting to full RSI 10:57 Aligning RSI 15:08 Working theory
Stay Updated: If you enjoyed this episode, be sure to like, subscribe, and share with your friends!
Find a16z on X: https://x.com/a16z
Find a16z on LinkedIn: https://www.linkedin.com/company/a16z/
Listen to the a16z Show on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX
Listen to the a16z Show on Apple Podcasts: https://podcasts.apple.com/us/podcast/the-a16z-show/id842818711
Follow our host: https://x.com/sarahdingwang
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see https://a16z.com/disclosures.
[Source](https://www.youtube.com/watch?v=h_aA1-R70E8)