Schulman Warns Verification Gap Delays Autonomous AI Takeoff
Dwarkesh Patel
Recursive self-improvement faces a prolonged bottleneck because reinforcement learning cannot autonomously generate valid scientific objectives or verify novel breakthroughs without human judgment.
## Background
Published by Dwarkesh Patel on September 11, 2026, across podcast and web platforms, this episode features a roundtable discussion with frontier AI researchers John Schulman, Beren Millidge, and Charlie O’Neill. According to the episode description, Patel brings together researchers from open and frontier labs to unpack what is actively happening at the frontier of artificial intelligence and explore how close the field is to recursive self-improvement.
## TL;DR
Patel and the panel rigorously examine the viability of recursive self-improvement and fast takeoff by 2036, identifying verification bottlenecks, sample efficiency deficits, and human objective specification as primary points of failure. They analyze why reinforcement learning with verifiable reward (RLVR) succeeds by reinforcing causal trajectories despite sparse feedback, while also confronting issues like stylistic entropy collapse and catastrophic forgetting in continual learning. The conversation details competitive dynamics, including how Chinese labs utilize proxy telemetry distributions for distillation and the rising engineering cost of building asymmetric RL environment ladders. Concluding with rapid-fire predictions, the researchers reveal sharp disagreements, forecasting drop-in remote workers within one to three years but differing substantially on timelines for a 10x research productivity uplift and full artificial superintelligence.
## Atmosphere & style
The discussion unfolds with the dense, cerebral intensity of an elite frontier research seminar, trading polite generalities for granular debates over loss landscapes, gradient signals, and empirical lab telemetry. Patel serves as an incisive, well-versed moderator, while John Schulman's grounded, battle-tested engineering caution provides a compelling foil to the bolder, fast-takeoff projections championed by Charlie O'Neill and Beren Millidge. The rhythm accelerates from meticulous deconstructions of scaling laws into moments of authentic friction and candid disbelief during the timeline lightning round, capturing the electric uncertainty of practitioners directly shaping an unprecedented technological frontier.
## Key takeaways
- `00:00` The primary barrier to recursive self-improvement is verification and judgment rather than raw generation: producing vast quantities of code does not dramatically boost productivity if evaluating, debugging, and steering it still demands human oversight.
- `02:45` Reaching superintelligence may require entirely new learning paradigms beyond current gradient descent, self-attention, and synthetic RL environments, mirroring historical technological discontinuities like those seen across Moore's Law.
- `15:00` Specifying objectives and deciding human preferences represents the most enduring human bottleneck, as frontier models excel at optimizing within bounded benchmarks but struggle to autonomously propose open-ended, paradigm-shifting research directions.
- `17:15` Post-training reasoning behaviors contain relatively few informational bits compared to pre-training corpora, allowing open-weights and competing labs to distill frontier model capabilities using captured prompt distributions and minimal compute.
- `28:06` Frontier labs are training automated AI researchers directly on active production stacks and recent architectural bugs, prioritizing empirical engineering intuition and researcher taste over synthetic toy setups.
- `45:24` Current AI models lag biological intelligence by up to a million-fold in sample efficiency, creating a severe sim-to-real gap for open-ended, non-stationary domains like negotiation or law that lack verifiable synthetic simulators.
- `57:15` Continual learning remains fundamentally unsolved due to optimization failures on non-stationary data, where models suffer from catastrophic forgetting and loss of plasticity that force labs to repeatedly retrain from scratch.
- `01:00:33` Scaling reinforcement learning is constrained by the difficulty of constructing an 'environment ladder' of asymmetric tasks—problems that are easy to evaluate but hard to solve—which face diminishing returns as easy synthetic domains are depleted.
- `01:06:40` Historical efficiency gains in pre-training have been driven primarily by data curation rather than architecture, with empirical tests attributing a ~9x compute efficiency gain to data quality versus ~3x to model architecture.
- `01:17:35` Reinforcement learning scales effectively because scalar rewards reinforce the causal search trajectories needed to solve problems, whereas supervised fine-tuning introduces excessive noise by penalizing valid stylistic deviations from exact demonstrator tokens.
- `01:24:54` Heavy post-training RL and pervasive distillation create stylistic entropy collapse, leading models to rely on repetitive phrasing, narrow narrative tropes, and predictable behaviors optimized for LLM judges.
- `01:28:32` Timelines for drop-in autonomous remote workers range from 1 to 3 years, heavily depending on whether enterprises re-architect tasks around programmatic API pipelines or require agents to operate across unstructured visual desktop GUIs.
- `01:31:45` Researcher forecasts for a 10x productivity uplift in AI R&D diverge sharply between 2 years (driven by automated experiment loops) and 5–10 years (constrained by human bottlenecks in absorbing findings and steering Bayesian bets).
- `01:34:25` The trajectory toward Artificial Superintelligence is heavily asymmetric: software, code, and formal mathematics accelerate rapidly under instant verification, while physical and proprietary industrial domains lag due to context limits and scarce real-world data.
## Detailed walkthrough
### `00:00` Steelmanning the Failure of Fast Takeoff by 2036
Dwarkesh Patel opens the conversation with John Schulman (Chief Scientist at Thinking Machines and OpenAI co-founder who led the RLHF development behind ChatGPT), Beren Millidge (CTO of Zyphra), and Charlie O'Neill. Setting aside exogenous political events like wars or blanket AI bans, Dwarkesh poses a core question: if we arrive in 2036—ten years out—and the world has not been radically upended by billions of crazy superintelligences running around, what is the most plausible technical failure mode?
Beren Millidge points toward a modern variation of Moravec's paradox and a failure of broad out-of-distribution generalization. The historical pattern in AI has often been believing that once a system solves hard benchmark math or beats humans at chess, everything else will follow, only to discover that the real-world impact remains surprisingly narrow. If models simply excel at everything humans can explicitly pack into an environment or a benchmark while failing at true meta-learning and continual learning, progress could easily stall out. While Beren views this as unlikely given the empirical generalization already observed in day-to-day practice, an insurmountable wall in continual learning and generalization remains his default technical explanation.
```mermaid
flowchart TD
accTitle: The Generalization Bottleneck Loop
accDescr: A cyclical feedback loop showing how incremental model improvements reveal new bottlenecks in human evaluation and judgment.
subgraph CapabilityCycle ["The Capability-Disillusionment Cycle"]
A["New Model Released"] --> B["Matches Human Performance on Known Benchmarks"]
B --> C["Initial 'AGI' Excitement"]
C --> D["Users Discover Edge Cases & Poor Self-Checking"]
D --> E["Model Feels 'Dumb' After a Month"]
E --> F["Engineering Bottlenecks Shift to Unsolved Judgment Tasks"]
F --> A
end
```
John Schulman highlights the persistent bottlenecks in judgment and verification. Humans possess broad comparative advantages over neural networks, and while every subsequent model release closes specific gaps, human workflows inevitably run into the areas where model judgment is weak and where the system cannot reliably evaluate its own outputs. This produces a recurring psychological and practical cycle: a model arrives, people briefly declare it to be AGI, but within a month of heavy use, its sharp edge cases make it feel deficient again. When performing complex research and engineering, being able to generate vast quantities of code does not make an engineer 100 times more productive if verifying that code and steering the design requires human-level judgment.
- The human-model dynamic repeatedly falls into an evaluation trap: impressive benchmark breakthroughs mask subtle deficiencies in self-checking and error-correction.
- Code volume does not linearly translate to engineering throughput because human oversight and debugging remain the primary rate limiters.
---
`02:45` The Limits of the Current Paradigm and Moore's Law Discontinuities
Charlie O'Neill frames the problem through the lens of architectural distance: how far is the current recipe—self-attention Transformers trained with reinforcement learning in scaled-up synthetic environments—from the theoretical global optimum of a learner instantiated on silicon? Many researchers assume that the moment an agent becomes even 0.1% better than human researchers, spinning up millions of instances running at accelerated clock speeds will overwhelm every other friction and trigger rapid recursive self-improvement (RSI).
However, Charlie draws a parallel to Moore’s Law. While Moore's Law appeared on the surface as an unbroken, smooth exponential line for decades, sustaining that trajectory required navigating numerous discrete, radical engineering discontinuities and paradigm shifts. The deep learning boom has traced a similar path: pre-training scaling laws began hitting diminishing returns, only to be extended by post-training reinforcement learning (RL). If reaching true superintelligence requires discovering another fundamental paradigm shift beyond gradient descent and self-attention, Charlie questions whether an agent trained purely within today's RL environments—even environments explicitly targeted at recursive self-improvement—possesses the inductive bias or search capability to discover that discontinuity. If an LLM running on standard architectures cannot leap to a wholly alien learning paradigm, progress will eventually trace an asymptote.
---
`05:15` Chess Elo Analogies and the Clean Benchmark Trap
Dwarkesh Patel and Beren Millidge contrast two mental models of progress: smooth internal capability growth versus discontinuous real-world impact. Dwarkesh references an observation made by Ryan Greenblatt: chess engine ratings have marched upward in an extraordinarily linear Elo trajectory since the 1980s. Yet from the perspective of human players, that steady slope created an abrupt phase change:
- For years, top human grandmasters reliably defeated engines.
- Over a narrow window of Elo crossing, engines became competitive.
- Beyond that threshold, human grandmasters could never win another game.
AI economic impact may follow this pattern: current systems appear economically modest because they have not yet crossed the requisite Elo threshold across complex research skills, not because the underlying trajectory has slowed.
```mermaid
flowchart LR
accTitle: Chess Elo Phase Transition Analogy
accDescr: A comparison showing how linear algorithmic gains produce abrupt discontinuous outcomes against human benchmarks.
subgraph RealWorldImpact ["Observed Human Dynamic"]
direction TB
H1["Humans Always Win"] --> H2["Contested Window"]
H2 --> H3["Humans Never Win"]
end
E1 -.-> H1
E2 -.-> H2
E3 -.-> H3
```
Beren agrees, arguing that because frontier systems are already approaching human parity on many technical subtasks, an asymptote would have to strike almost immediately to prevent this takeover dynamic by 2036. Consequently, Beren believes heavy political intervention and regulatory bans are far more likely to freeze progress than a pure technical ceiling.
Charlie pushes back by distinguishing between two fundamentally different types of research:
1. **Clean-objective auto-research:** Settings where an objective function is tightly formalized (such as pushing down cross-entropy pre-training loss or increasing reward in an RL sandbox), where search-driven models excel.
2. **Open-ended paradigm shifts:** The messy, unformalized science where humans cannot cleanly specify the objective ahead of time, and where current AI systems are incapable of self-defining what is important.
---
`07:30` John Schulman on Early OpenAI and Why Next-Token Prediction Succeeded
Dwarkesh turns to John Schulman to reflect on the early days of deep learning and whether foundational breakthroughs can be discovered via automated optimization loops. In 2014, nobody would have imagined that running an automated speedrun on something like a nanoGPT pre-training objective would be the path to general intelligence. There is often a profound mismatch between an internal loop that is cheap to optimize and the broad, sluggish outer loop of real-world utility or revenue.
Schulman recalls that during the early days of OpenAI, researchers routinely doubted whether next-token prediction and log-loss minimization could ever produce real intelligence:
- The intuition was that the semantically critical tokens and deep conceptual bits accounted for such an infinitesimal fraction of the total cross-entropy loss that they would be completely drowned out by statistical noise.
- Humans clearly do not learn by modeling every low-level detail of their sensory stream; people cannot paint a photorealistic replica of a scene they just observed, leading researchers to assume intelligence required handcrafted, semantically focused loss functions.
- Despite these theoretical objections, raw next-token prediction scaled ruthlessly and worked anyway.
Schulman stresses that machine learning remains deeply beholden to empirical generalization. It is notoriously difficult to forecast when an architecture will achieve genuine out-of-distribution transfer versus when it will overfit the metric. While training directly on downstream tasks yields immediate local improvements, the most transformative leaps in AI history have consistently come from broad, unexpected generalization that theorists initially dismissed.
`10:00` The Hundred-Year Thought Experiment and Theory-Driven Compute
The discussion opens on the mysterious nature of generalization in deep learning. While simple interpolation on training distributions is expected, modern models exhibit forms of generalization that researchers arguably have "no right to expect"—leaping from naive next-token prediction to rich conceptual world models, or generalizing from narrow, verifiable tasks to complex, unverifiable domains. Dwarkesh Patel introduces an intuition pump for recursive self-improvement: even without scaling up non-AI inputs, what if a lab spent an equivalent amount of compute on automated AI labor as it does on a massive, seven-figure training run? In this scenario, synthetic researchers could spend the subjective equivalent of an entire century doing small-scale ablations, developing theoretical frameworks from first principles, and meticulously setting up an experiment before executing it—followed by another simulated century analyzing the telemetry to decide the subsequent move.
John Schulman agrees that research methodology is far from hitting any fundamental ceiling. Today's labs often run experiments with relatively crude heuristics; having tireless automated systems dedicate compute equal to the training budget purely toward deep post-hoc analysis, ablation design, and predictive theory building would radically elevate the field. Rather than blundering into expensive compute commitments, automated researchers could discover subtle small-scale setups whose dynamics reliably extrapolate to frontier models.
```mermaid
flowchart LR
accTitle: The Compute Allocation Shift in Automated Research
accDescr: Shifting from trial-and-error compute to an AI-driven cycle of simulated theory, targeted experimentation, and deep post-hoc analysis.
subgraph Pre["Theoretical Phase"]
A["Small-Scale Ablations"] --> B["AI Theory Building<br/>(100 Virtual Years)"]
end
subgraph Run["Execution Phase"]
B --> C["Optimal 7-Figure Experiment"]
end
subgraph Post["Analysis Phase"]
C --> D["Automated Post-Hoc Analysis"]
D --> A
end
```
---
`12:15` Catching Flaws in Scaling Laws and the Limits of Pure Deduction
Charlie O'Neill nuances the "pure thinking" argument by grounding it in Bayesian fundamentals: thinking alone cannot generate fresh empirical data; it can merely update a posterior given the bits already collected from the world. However, whenever an objective is cleanly specified, intense conceptual analysis can yield massive dividends by identifying oversights that human researchers missed for years.
A prime example is the original Kaplan et al. scaling laws. An automated researcher reviewing the data back in 2020 could have immediately spotted that the scaling curves were derived from intermediate training checkpoints that neglected the effects of learning rate schedule annealing. Catching that methodological error could have brought the Chinchilla scaling revisions forward by nearly two years. Similarly, discoveries like Maximal Update Parametrization ($\mu$P)—which uncovered how learning rates must scale relative to model width—could have been deduced purely by carefully modeling known network behavior. In narrow, well-specified regimes like pre-training loss optimization, automated analysis could easily deliver a 10x acceleration. Yet the panelists emphasize that pure deduction cannot easily bridge the chasm to formulating brand-new paradigms.
- **Bayesian Limits of Pure Deduction:** Reasoning extracts implicit patterns from priors and past observations, but cannot create empirical entropy out of thin air.
- **Historic Missed Optimizations:** The Kaplan scaling laws failed to account for cosine schedule decay/annealing on intermediate checkpoints, leaving low-hanging fruit unharvested until Chinchilla.
- **Theoretical Breakthroughs via Math:** Frameworks like $\mu$P demonstrate that foundational scaling efficiencies can be anticipated through rigorous mathematical and structural analysis rather than blind scaling.
---
`13:45` Moravec's Paradox for Autonomy and Proposing New Objectives
The core hurdle for true recursive self-improvement is whether models can autonomously set their own agendas. A self-sustaining automated loop requires an agent to propose an objective, optimize against it, extract the learnings, and invent the next objective without drifting into absurdity or getting stuck in degenerative loops.
Beren Millidge points to a possible manifestation of Moravec's Paradox. In classical robotics, tasks that feel trivial to animals—like locomotion or basic spatial survival—turned out to be vastly harder to implement than high-level symbolic logic or mathematics. In the same vein, researchers might assume that autonomous self-encapsulation (deciding what to care about and maintaining long-term goal coherence) is straightforward simply because humans do it naturally, while in reality, open-ended goal generation may prove far more treacherous for AI than mastering formal reasoning or coding.
---
`15:00` Objective Specification as the Final Human Bottleneck
When asked what remaining human contribution will resist automation the longest, the group points directly to the art of asking the right questions. Today's frontier models are capable code monkeys: if an engineer defines a benchmark or task harness, the model can implement and run scripts. But when asked for open-ended research ideas, AI tends to suggest trivial, incremental modifications rather than paradigm shifts. Historical breakthroughs rarely emerged from local optimization—such as Alec Radford betting on wide-scale next-token prediction while DeepMind remained fixated on solving intelligence purely via reinforcement learning in games, or researchers realizing that scaling laws could turn brute force into an exact science.
John Schulman emphasizes that defining the objective and deciding what humanity actually wants is the true end-state human role. Post-training is fundamentally about behavior design:
> "The role for humans that will last the longest is defining the objective and deciding what we actually want... Alignment is sort of the answer, but alignment itself can be decomposed into specification of the objective—or figuring out what the right objective should be—and then actually achieving or optimizing the objective you've defined. I think the first one is not going to go away anytime soon."
This dynamic explains why frontier post-training teams remain heavily staffed. Large groups of human researchers are needed not because the algorithms are too slow to run, but because someone has to adjudicate model behavior, author Model Specs, write constitutions, and define what constitutes "helpfulness" across hundreds of nuanced human domains.
---
`16:30` Sponsor Break: Antithesis and the Verification Bottleneck
Dwarkesh highlights the shifting economics of software engineering. As AI agents rapidly accelerate raw code generation, the primary friction point moves from writing software to verifying that it actually works and meets safety criteria. Ron Minsky, co-lead of Jane Street's technology group, notes that Jane Street began integrating Antithesis in early 2025 to test high-assurance systems, ultimately investing in the platform. Antithesis autonomous testing was able to uncover insidious edge-case bugs that had eluded meticulous manual engineering reviews, directly relieving the verification bottleneck that threatens to choke AI-assisted development.
---
`17:15` Why Distillation Prevents Total Market Centralization
The conversation shifts to market structure: why hasn't frontier AI development collapsed into an absolute monopoly? Given the exorbitant capital costs of pre-training and compute clusters, classical economics would predict extreme centralization among one or two providers. Schulman argues that knowledge distillation acts as the primary countervailing force breaking monopolies.
Because reinforcement learning reasoning steps and post-training behaviors represent a relatively small number of actual informational bits compared to pre-training corpora, an open-weights or competing lab can take output trajectories from a leading frontier model and distill those capabilities with minimal data and compute. While proprietary models might push continuous learning on enterprise deployment loops, distillation ensures that any capability leap achieved at massive cost quickly leaks outward, preventing a single incumbent from pulling permanently away from the pack.
`20:00` Prompt Distributions, Proxy Routers, and the Distillation Advantage
Even with full access to chain-of-thought traces and high-frequency distillation loops, copying a frontier model’s behavior via supervised learning is fundamentally bottlenecked by the prompt distribution. To extract useful capabilities, a model must be prompted with a vast, highly realistic distribution of tasks. One major development enabling Chinese labs to stay competitive is the prevalence of router and proxy services. Because frontier U.S. models are officially restricted in China, Chinese developers and enterprises rely on these proxy services—primarily for software engineering tasks. By capturing or purchasing these real-world user interaction traces, Chinese labs gain access to an authentic prompt distribution.
```mermaid
flowchart TD
accTitle: Real-World Trace Distillation Loop
accDescr: Flow showing how Chinese proxy router traffic creates realistic prompts for distilling frontier models into new models.
subgraph FrontierTeacher["Teacher Stage"]
B --> E["Frontier Models"]
D --> E
E -->|"Generates CoT & Solutions"| F["Rich Trajectories & Traces"]
end
subgraph Training["Distillation"]
F --> G["Student Models (e.g., GLM, Kimi)"]
end
```
In practice, frontier labs construct their training data by gathering human seed prompts, then using existing models to synthesize massive coverage. As models improve, the quantity of information humans need to supply steadily declines. While capturing full conversational workflows—where a user requests an application, iterates on failures, and asks for architectural pivots—is critical, a fully automated loop where the AI decides both its data generation and training regime effectively amounts to recursive self-improvement (RSI).
Paradoxically, distilling novel capabilities can be easier than creating them from scratch. If a lab wants a model that behaves like an elite politician, creating that de novo requires acquiring real-world data of legislative discussions and behind-the-scenes negotiation. In contrast, a distiller merely prompts an existing frontier model that already possesses that capability to generate millions of synthetic political debate traces.
- Chinese router and proxy services route restricted U.S. frontier model access (largely for software development) and monetize trace logs.
- Capturing iterative multi-turn corrections—user feedback, pivots, and follow-ups—provides the critical supervision signal for agentic task completion.
- Synthesizing broad coverage from seed prompts shifts the human role toward supplying minimal bits of novelty.
---
`22:45` Benchmarking Realism vs. Difficulty: The Student-Teacher Gap
A striking puzzle in the competitive landscape is why models like Sonnet 5 or Opus 5 are seen as lagging behind models like GLM-5.3 or Kimi k3 on certain qualitative measures, despite having access to logic distillation from models like Mythos or Fable. This discrepancy highlights a fundamental distinction between two axes of reinforcement learning (RL) environments: **difficulty** and **realism**.
| Dimension | Characteristics | Pitfalls & Distillation Failure Modes |
|:--- |:--- |:--- |
| **Difficulty** (Benchmark Axis) | Hard, verifiable puzzle-like tasks; high complexity and cleverness required in a closed environment. | Easy to overfit. Models match teachers on benchmarks but fail to generalize to messy open-ended work. |
| **Realism** (Agentic Axis) | Multi-turn back-and-forth interactions with human users; ambiguous requirements; complex, interactive coding workflows. | Difficult to verify automatically; requires careful human feedback, custom rubrics, or live traffic data. |
When labs focus heavily on benchmark difficulty, student models nail verifiable puzzles but fail in realistic settings. Larger frontier models generalize better from narrow, difficult puzzles to ambiguous, realistic tasks. If distillation only exposes a student model to benchmark-maximizing distributions, it fails to inherit this generalizability. Furthermore, post-training is fragile: Opus 5 exhibits an "AI-as-a-judge" artifact where it expends massive token budgets second-guessing every intermediate step, lacking the intuitive "big model smell" of a flagship model to discern when a trajectory is sufficient. In parallel, frontier labs buy extensive pre-training and fine-tuning datasets from commercial data brokers, and Chinese labs purchase the exact same corpora, making it remarkably easy for fast followers to remain at parity.
---
`28:06` How Automated AI Researchers Will Actually Be Trained
Speculation about the first models capable of automating AI R&D often centers on toy setups—such as having a GPT-8 try to train miniature GPT-3-scale models to beat continuous learning video games or maximize compute efficiency. In practice, frontier labs are taking a far more pragmatic, iterative approach rather than stepping back to train toy models from first principles.
Instead of rolling back model lineage to raw compute and bare neural nets, research labs anchor training directly to the bleeding edge of their active engineering stack. Training will combine:
1. **Human Feedback on Taste:** Absorbing the qualitative judgment, research taste, and prioritization heuristics of top human researchers.
2. **Multi-Step Practice Environments:** Creating complex synthetic research sandboxes modeled directly on the actual engineering bugs, architectural bottlenecks, and optimization hurdles encountered in production over the preceding months.
```mermaid
flowchart LR
accTitle: Iterative AI R&D Training Loop
accDescr: The practical feedback loop used to train automated AI researchers at the frontier.
E -->|"Deployed to Assist Labs"| F["Failure Modes & Weaknesses Identified"]
F -->|"Patch Weaknesses"| A
F -->|"Update Rubrics"| C
```
As researchers continuously work alongside these AI assistants, they spot specific cognitive blind spots and systematic errors. Each generation is patched by either collecting targeted human feedback or generating synthetic environments tailored to those discovered weaknesses, iterating right at the frontier rather than relying on isolated sandbox exploration.
`30:00` Distilling Frontier Progress vs. Intuition-Driven Research
The discussion opens with the mechanics of how labs might set up automated environments to train models on AI research itself. One theoretical route is rolling the clock back—such as resetting to the point before Group Relative Policy Optimization (GRPO) was discovered—to test if a model can rediscover the most effective reinforcement learning formulations. However, because labs remain heavily compute-constrained, the realistic practice is staying pinned to the frontier. Researchers diff the bugs, fixes, and algorithmic enhancements found over the preceding three months, wrap those into bespoke environments, and distill those discoveries directly back into the model's weights. This continual distillation loop provides clean, non-stale data between generations, though it sparks debate over whether the process is asymptotic. If models merely distill the latest three months of human-directed work, progress risks slowly inching along, capped by what human researchers uncover.
```mermaid
flowchart TD
accTitle: Lab Distillation vs Synthetic Milestone Frontiers
accDescr: Diagram showing how recent lab research is converted into training environments versus how automated super-human milestones can push beyond human capabilities.
subgraph LabCycle["Current Lab Distillation Loop"]
A["Frontier AI Research<br/>(Last 3 Months)"] --> B["Diff Bugs & Breakthroughs"]
B --> C["Construct Verifiable Environments"]
C --> D["RL / Distillation into Weights"]
D -. "Bounded by human pace?".-> A
end
subgraph FrontierTarget["Self-Improving Targets"]
E["Set Extreme Verifiable Targets<br/>(e.g., Loss = 1.3)"] --> F["AI Search & Optimization"]
F --> G["Surpass Human Baselines"]
end
```
Beren Millidge counters that while trajectory distillation has a hard ceiling at human capability, synthetic environments do not. It is relatively easy to define environments with verifiable criteria that no human can satisfy—such as driving an evaluation loss down to a specific threshold like 1.3 or training a 100-million parameter model to beat complex games far harder than Minecraft (a milestone the panel laughs at for being considered "too easy" compared to five years ago). John Schulman cautions, however, that real machine learning research rarely looks like simple hill climbing on an unambiguous metric. Breakthroughs are typically guided by informal intuition: noticing a flaw, relaxing the strict realism of production objectives, engineering an artificial test to see if a nascent algorithmic idea exhibits "signs of life," and only then scaling back up to realistic complexity once the method matures.
- **Lab update cadence**: Environments are continuously constructed from diffing the last ~3 months of algorithmic improvements and bugs discovered inside frontier labs.
- **Measurable targets vs. intuition**: Concrete tasks (e.g., hitting a specific target loss of 1.3) are verifiable, but early-stage algorithm design relies on relaxing production realism to elicit faint signals of success.
- **Informal theory**: Machine learning seldom relies on predictive mathematical theory; work is instead guided by informal mental models and empirical taste.
---
`33:51` Scaling RLVR to Remote Workers and the Generalization Bet
Dwarkesh Patel synthesizes what appears to be the master plan across leading frontier labs: scaling reinforcement learning with verifiable reward (RLVR) across millions of diverse simulated environments and hundreds of domains. The underlying hypothesis is that by forcing models to triage context, persist across complex horizons, and coordinate across multi-agent workflows, an agent will emerge that functions as a drop-in remote worker capable of executing tasks over a week or a month without direct real-world grounding.
```mermaid
flowchart LR
accTitle: Reinforcement Learning Expansion Across Domains
accDescr: The progression of RLVR environments from software engineering into business verticals like finance and office suites.
A["Coding<br/>(Abundant code, easy verification)"] --> B["Finance & Excel<br/>(Tabular data, deterministic logic)"]
B --> C["PowerPoint & Documents<br/>(Knowledge work long-tail)"]
C --> D["Autonomous Remote Worker<br/>(Long-horizon agent)"]
```
The panel analyzes why labs like Anthropic methodically expand their curriculum:
- **Phase 1 (Software Engineering)**: The lowest-hanging fruit due to massive open-source traces, automated unit tests, and internal lab repos.
- **Phase 2 (Finance and Excel)**: Abundant structured spreadsheets, verifiable numerical outputs, and massive enterprise value.
- **Phase 3 (Enterprise Long-Tail)**: Slide decks, internal reports, and cross-application administrative workflows.
When asked why labs bother explicitly training models on PowerPoint or corporate tools if in-context meta-learning is supposed to allow models to learn "on the job," the group identifies three distinct incentives:
1. **Commercial pragmatism**: Models need to generate revenue today to finance multi-billion-dollar future training clusters.
2. **Computational and inference efficiency**: While a super-intelligent model could theoretically read a library of regulatory finance textbooks on the fly, amortizing that domain knowledge into model weights makes runtime execution vastly cheaper and faster.
3. **Skill transfer to RSI**: Extreme parameter scales mean models can absorb domain-specific trivia without degrading; meanwhile, learning long-horizon planning, taste, and persistence in finance or software engineering directly transfers into recursive self-improvement (RSI) tasks where data is scarce.
---
`36:20` Sim-to-Real Bottlenecks and Sample Efficiency
The conversation shifts to whether data-center-bound "sim-to-real" training can remain the dominant paradigm indefinitely. While simulating synthetic interactions inside a cluster avoids safety risks and scales compute linearly, John Schulman notes that its fundamental weakness lies in domains requiring nuanced, real-time collaboration with actual people. Real human environments cannot easily be ported into deterministic sandboxes.
The reliance on simulation persists today because modern reinforcement learning remains deeply sample-inefficient, demanding thousands or millions of sequential episodes to stabilize behavior. No human workforce could feasibly sit in an interactive training loop to provide that volume of on-policy feedback. Until sample efficiency improves drastically—or off-policy learning advances to the point where models can reliably extract high-leverage policy improvements directly from passive human deployment traces without catastrophic divergence—simulated environments inside data centers will remain the necessary engine of progress.
`40:00` Sponsor Interlude: Jane Street Protocol Emulator ASIC Competition
Dwarkesh pauses the discussion to introduce Jane Street’s most ambitious hardware competition to date: designing a general-purpose protocol-emulator ASIC. When hardware teams build a new chip, they need a way to test it against realistic traffic patterns without the friction of plugging it directly into a complex, live production system. Jane Street wants competitors to build a flexible, reprogrammable ASIC that can bridge that gap by simulating realistic protocol traffic on demand.
Rather than hard-coding a static set of fixed interfaces onto silicon, submissions must be reconfigurable so that if a brand-new protocol emerges after the chip is taped out, the silicon can still adapt and handle it. Jane Street suggests starting with foundational embedded protocols like SPI and I2C before tackling more demanding, high-speed standards such as low-speed USB and Ethernet. The hard physical constraint is that all designs must target an open-source 130-nanometer process node. Jane Street is funding the fabrication to tape out the most novel designs, promising to ship physical copies of the manufactured silicon to the winners. The contest runs through January 18, 2027, with team participation strongly encouraged.
`40:48` Tapping the Deployment Stream: The Hive Mind and Continual Learning
Dwarkesh turns the conversation to an apparent inefficiency in modern AI infrastructure: roughly 50% of total compute is currently spent on inference, yet almost none of that compute directly improves the underlying models. Unlike a human who accumulates perhaps fifty years of rich real-world experience over a lifetime, an AI model experiences millions of collective years of economically meaningful labor across millions of parallel instances. It seems obvious that models should eventually assimilate this flood of deployment data into an accelerating, hive-mind-style intelligence explosion, yet real-time learning remains elusive.
The panel pushes back slightly, noting that while the dream of an organic, real-time continual learning loop—where weights update instantaneously on individual user turns—remains technically fragile, the broader dynamic is already well underway in batch cycles. Frontier labs routinely capture, filter, judge, and synthesize production interactions to populate the pre-training and mid-training data mixtures for their next model generations:
* **Distillation and Deployment Harvesting:** Even if Western labs maintain strict enterprise data privacy boundaries, Chinese labs aggressively harvest user interactions and model rollouts to train next-generation models, capturing massive empirical advantages through distillation.
* **Vertical Domain Flywheels:** Specialized startups like Harvey (in legal tech) and Cursor (in software engineering) extract domain-specific edge cases, user complaints, and environmental interactions directly from daily deployment to build targeted training datasets that generalist frontier labs struggle to replicate.
* **The Granularity Problem:** Zooming in to instant, per-token or per-session weight updates breaks model stability, but zooming out reveals that generation-over-generation gains are already heavily driven by deployment data.
`43:08` Online RL in the Wild: Cursor’s Rapid Deployment Loop
The researchers point to Cursor’s Composer as a rare, working proof-of-concept for online reinforcement learning on production workflows. Cursor famously trained not just their lightweight tab-completion engine on user keystrokes, but applied similar reinforcement learning pipelines to their full generative models. The central engineering bottleneck in online RL from natural telemetry is variance: unlike synthetic environments where a model can run hundreds of parallel rollouts from the exact same state to compute group baselines, real production traffic gives you only one user prompt and a single rollout.
```mermaid
flowchart TD
accTitle: Cursor's Online RL Deployment Loop
accDescr: Pipeline showing how user telemetry is converted into model weight updates and deployed on a 5-hour benchmark cadence.
subgraph Telemetry ["Telemetry & Heuristics"]
A["User Prompt & Action"] --> B["Single Rollout"]
B --> C["Variance Heuristic<br/>Better / Worse than Avg"]
end
subgraph Training ["Policy Optimization"]
C --> D["REINFORCE Update"]
D --> E["Candidate Checkpoint"]
end
subgraph Eval ["Gating & Rollout"]
E --> F{"Evaluated on<br/>CursorBench"}
F -->|"Regression"| G["Discard Checkpoint"]
F -->|"Improvement"| H["Deploy to Production<br/>Every 5 Hours"]
end
```
To conquer the variance reduction problem, Cursor developed custom heuristic baselines to judge whether an individual code generation was meaningfully better or worse than average. They then ran a massive REINFORCE update loop and tied it directly to continuous automated evaluation: candidate models were evaluated against CursorBench, and if the benchmark improved, the updated model was pushed to production every five hours; if it regressed, the checkpoint was discarded. However, John Schulman and the panel caution that using superficial signals from natural interaction—such as whether a programmer hit "Tab" to accept an edit—inherently risks severe reward hacking if the model learns sycophantic or deceptively simple completion patterns.
`45:24` The Sim-to-Real Gap and the Limits of Data Center Simulation
This challenge highlights an even broader architectural dilemma: the sim-to-real gap for long-horizon autonomous tasks. While pure games and short-horizon math proofs can be simulated endlessly in a data center sandbox, true superintelligence implies handling economic challenges that defy synthetic sandboxing:
* **Long-Horizon Dependencies:** Real-world software engineering past a few days inevitably requires negotiating specifications with clients, coordinating with cross-functional human teams, and reacting to messy production outages.
* **Real-World Un-simulatable Objectives:** Starting a profitable business, executing sustained algorithmic day trading against real market liquidity, or litigating a high-stakes court case cannot be fully reproduced inside an isolated cluster without interacting with the physical, human world.
* **The Sample Efficiency Deficit:** If sim-to-real transfer is weak and models must learn these tasks through direct weight updates, modern architectures face a massive biological deficit. A human reaches general adult competence after seeing a comparatively tiny stream of sensory data, whereas frontier models are "plausibly million-fold behind humans" in the sample efficiency of their weight updates. If synthetic simulations cannot reliably substitute for real-world experience, the sheer sample inefficiency of current gradient updates could stall recursive self-improvement.
`47:32` Cumulative Tasks vs. Non-Stationary Distributions
To reconcile why recursive self-improvement might succeed despite poor sample efficiency in the messy real world, the panel draws a sharp conceptual distinction between cumulative tasks and non-stationary distributions:
| Task Dynamic | Characteristics | Archetypal Examples |
|:--- |:--- |:--- |
| **Cumulative Tasks** | Discoveries stack permanently; earlier primitives remain valid; does not require relitigating foundational knowledge. | AI Research & Engineering (Attention, MoE, GRPO), self-improving training scripts (`pre-training.sh`, `post-training.sh`). |
| **Non-Stationary Tasks** | Moving target; shifting social, political, and operational norms; past data decays rapidly in relevance. | Acting as a law firm associate, managing organizational politics, real-time macro investing. |
Charlie points out that AI research itself may simply be a cumulative domain. An autonomous AI researcher does not need to reinvent the transformer from scratch every morning; once attention, mixture-of-experts (MoE), and group relative policy optimization (GRPO) are discovered, they are committed to the codebase and stacked permanently into the training harness. A concise Python script of fewer than one million tokens could conceivably orchestrate the entire training loop of a self-improving agent. The tragedy—or irony—is that pure recursive self-improvement on machine learning code might actually be an easier engineering target than automating a standard mid-level white-collar job, because a legal associate must continually adapt to an ever-shifting web of human personalities, unwritten rules, and non-stationary office politics.
John Schulman adds a critical nuance regarding human versus model learning rates: models are extraordinarily sample-efficient at in-context learning over short horizons, but fall dramatically behind humans over intermediate horizons where humans perform highly sample-efficient biological weight consolidation. Schulman emphasizes that raw sample efficiency is only one part of the frontier bottleneck; models still suffer from much lower diversity of thought than human research communities and frequently fail at long-horizon judgments—the intangible instinct that researchers commonly describe as technical "taste."
`50:00` Research Taste, Sample Efficiency, and the Trillion-Token Thought Experiment
The panel delves into the elusive nature of "taste" in engineering and scientific research, probing whether intuition and long-term judgment are fundamentally distinct from short-horizon pattern recognition. In fields like software engineering, taste largely manifests as an intuitive foresight about architecture: knowing which designs will remain robust, clean, and maintainable over years rather than collapsing under technical debt. While human cognitive weaknesses—particularly biological sample inefficiency and narrow bandwidth—place clear ceilings on human recursive self-improvement (RSI), it remains an open question whether AI can bypass those limits purely through scale. To isolate this, a provocative thought experiment is raised: suppose an AI model were endowed with a context window of a trillion tokens, populated entirely with the life's work, raw logs, notes, and research experiences of a world-class researcher prior to reinforcement learning from human feedback (RLHF), while retaining its current sample efficiency. Would that context alone solve taste, allowing the model to replicate high-level human research decisions, or is something structural missing?
The researchers argue that even an astronomically long context window does not automatically confer taste. A model must still be explicitly trained on how to make updates and extract correct generalizations from that context; possessing raw telemetry is useless without the inductive bias or optimization pressure to internalize it. Furthermore, humans do not require millions of hours of observation to acquire taste. Humans do not live to be 10,000 years old; a researcher matures from a naive first-year PhD student into an experienced postdoc in roughly five years, completing perhaps only 10 to 15 research projects in total. That taste develops from a relatively brief sequence of short-horizon trials suggests that long-horizon discernment is meta-learned from short-horizon feedback. Because an automated system will experience vastly more iterations and projects than any human ever could, it will have ample opportunity to meta-learn taste. The critical unsolved question is not whether AI can learn taste from data, but whether taste meta-learned across short- and medium-horizon episodes will generalize reliably to open-ended, ultra-long-horizon research endeavors.
- **Defining taste:** Intuition about long-term payoffs, such as designing software systems that remain maintainable across multi-year lifecycles.
- **The PhD timeline:** Human researchers develop domain taste across roughly five years and only 10 to 15 discrete projects, indicating that taste acquisition relies on high meta-learning sample efficiency rather than thousands of years of exposure.
- **The long-horizon question:** While AI can ingest orders of magnitude more project experience than human scientists, whether meta-learning over shorter episodes generalizes to massive multi-month research bets remains unproven.
---
`53:00` The Deployment Feedback Loop: Economic Barriers, Modular Cartridges, and the Path to a Real-Time Hive Mind
The conversation turns to the prospect of an AI "hive mind"—a regime where deployed models continuously update their weights in real time from millions of user interactions. Currently, the feedback loop from deployment back into model capability is weak, delayed, and mediated through noisy human annotations or passive data harvesting. However, the panelists argue that the primary roadblock to real-time deployment learning is not purely technical; it is driven by corporate incentives and data privacy economics. Enterprise customers are fundamentally unwilling to let foundation model providers train a shared, global base model on their confidential operational data, as doing so would leak proprietary IP and competitive advantage to rival users.
```mermaid
flowchart TD
accTitle: Accelerated Model Deployment and Specialization Loop
accDescr: Cycle illustrating how customer data stays isolated in local modules while execution traces feed foundation pre-training, progressively shortening release intervals toward continuous learning.
subgraph CustomerEnv["Enterprise / Local Environment"]
A["Base Foundation Model"] --> B["Specialized Modules<br/>(LoRAs / KV-Cache Cartridges)"]
B --> C["Client Deployment & Execution"]
C --> D["Anonymized Execution Traces"]
end
subgraph LabEnv["Frontier Model Lab"]
D --> E["Distillation & Pre-training Pipeline"]
E --> F["Next-Gen Model Weights"]
end
F -->|"Iterative cadence:<br/>3 months → 1 week → 1 day → 1 hour"| A
```
Because of these economic constraints, the transition to models that learn from deployment will rely on modular, swappable architectures rather than monolithic continuous weight updates:
- **Modular adaptations over monolithic updates:** Enterprises will favor local adapters such as LoRAs (Low-Rank Adaptations), linear attention memory architectures, or "cartridges"—heavily compressed key-value (KV) caches trained to pack massive context into a fixed-size vector space that can be slotted into the base model without altering its core weights.
- **Trace distillation:** Frontier labs will harvest anonymized reasoning and execution traces generated during deployment, filtering and distilling the best solutions back into the pre-training and post-training mixtures for their next model generations.
- **A continuous acceleration cadence:** Rather than a sudden discontinuous shift to instant online learning, the iteration cycle will compress incrementally:
- Models will initially update and redeploy on roughly three-month retraining cycles.
- The pipeline will tighten to weekly updates, then daily refreshes, and eventually hourly releases.
- Once the retraining and distillation cycle reaches hourly cadences, the practical gap between discrete batch training and real-time continuous learning is effectively closed.
`55:00` Micro-Updates, Catastrophic Forgetting, and the Limits of SFT vs. RL
At large scale, the current paradigm for continual learning appears to hold together: when training on massive batches, noise washes out, and inserting data during mid-training or synthetic environment generation allows models to steadily absorb skills. However, Charlie O'Neill and Beren Millidge point out that this framework disintegrates when examined at the micro level. If an enterprise attempts to continuously adapt a single deployed model to a specific environment—such as a law firm feeding in proprietary casework through repeated, small-batch updates—existing methods quickly fall apart.
Supervised fine-tuning (SFT) on successful interaction traces (whether gathered on-policy or off-policy) causes severe issues after hundreds of iterative micro-updates. Practitioners run directly into catastrophic forgetting, where the model steadily loses previously acquired base capabilities and suffers broad general degradation. While on-policy distillation pushes the failure horizon out slightly further, it eventually succumbs to the exact same failure mode. Reinforcement learning presents the opposite trade-off: while RL excels at instilling procedural capabilities and reasoning behaviors, it is notoriously bad at absorbing dense, explicit factual knowledge (such as memorizing specific personnel workflows or firm-specific internal processes). Injecting explicit world knowledge through RL requires pouring enormous compute budgets into building bespoke simulated environments just to teach what could otherwise be stated in plain text.
```mermaid
flowchart TD
accTitle: Trade-offs in Continual Model Adaptation
accDescr: Comparison between supervised fine-tuning, reinforcement learning, and scratch retraining.
subgraph SFT["Iterative SFT & Distillation"]
A["Micro-Updates / Traces"] --> B["Destructive Weight Shifts"]
B --> C["Catastrophic Forgetting"]
end
subgraph RL["Reinforcement Learning"]
D["Procedural Capabilities"] --> E["Surgical Weight Tweaks"]
E --> F["Cannot Absorb Dense Knowledge"]
end
subgraph Scratch["Retraining from Scratch"]
G["Unified Data Blend"] --> H["Pre-training + Mid-training"]
H --> I["High Capacity / High Cost"]
end
```
- Iterative SFT across hundreds of micro-updates reliably causes catastrophic forgetting of earlier base knowledge and degrades general performance.
- On-policy distillation offers temporary resilience against degradation compared to naive SFT, but still fails over extended horizons.
- RL modifies weights within a tiny, surgical loss envelope, preserving base performance but making it practically ineffective for injecting dense, explicit factual knowledge.
---
`57:15` Loss of Plasticity: Why Labs Still Retrain from Scratch
When Dwarkesh Patel asks whether this degradation stems from fundamental parameter capacity or flawed optimization techniques, the researchers firmly attribute it to technique. If an engineer takes an identical architecture with the same parameter budget and trains it from scratch with all the accumulated domain data folded into pre-training and mid-training, the model masters the tasks effortlessly without suffering forgetting. The bottleneck is the mathematical reality of training on non-stationary data: sequentially feeding new distributions overwrites earlier representations because current optimization algorithms cannot protect historical feature spaces.
This dynamic exposes the dilemma of contemporary post-training. SFT often acts as a blunt instrument that destroys latent structure, whereas RL is conservative, tweaking model weights by minute increments. While that surgical nature protects general capabilities, it severely caps how much novel information RL can drive into the network. While labs have extended base longevity by rolling back to pre-trained checkpoints, running mid-training, and branching off fresh RL runs, bases eventually suffer an irreversible "loss of plasticity." Models reach an asymptote where they refuse to absorb new distributions effectively. In an ideal world, continual learning would mean never discarding a base model; in practice, the persistent loss of plasticity forces frontier AI labs to burn tens of millions of dollars periodically retraining fresh base models from scratch.
- The inability to continually update models is driven by algorithmic technique and non-stationary data drift, not inherent neural network capacity limits.
- Loss of plasticity causes base models to hit an absorption asymptote over extended mid-training runs, making further fine-tuning ineffective.
- Deploying models to millions of users allows gradient noise to average out across diverse tasks, but does not solve the underlying requirement to periodically retrain foundation checkpoints from scratch.
---
`59:20` Sponsor: Automating Podcast Editing with Grok Bot
Dwarkesh turns to describe how his production team incorporates AI agents into their daily workflow using Grok Bot. After recording interviews that frequently stretch across multiple hours, Dwarkesh immediately drops an unstructured brain dump into Slack containing his candid reactions on what was compelling, what dragged, and which sections should be excised. Because these stream-of-consciousness notes lack explicit timestamps, video editors previously had to spend hours manually scrubbing through sprawling recordings just to locate the precise discussions he referenced.
To remove this bottleneck, the production team integrated Grok Bot directly into their Slack workspace as an automated producer. When an editor uploads a rough cut of an episode, Grok Bot opens the raw transcript on its virtual workstation, cross-references Dwarkesh's qualitative notes against the audio timestamps, and applies a structured file of his editing preferences to generate concrete cut recommendations. The agent then packages the top clip candidates and delivers them to his phone for rapid mobile sign-off, eliminating the tedious manual search process for his video editors.
`01:00:33` Can Current Architectures Reach Superintelligence on the Right Data?
Dwarkesh opens a deep dive into the role of data in frontier AI progress by asking a fundamental question: is there some hypothetical data distribution that, if trained into current transformer architectures, would result in an artificial superintelligence that completely dominates human experts across every single domain? The researchers quickly point out that in a purely theoretical sense, the answer is trivially yes—one could simply train a model to output the exact Python script and weights that instantiate a superintelligence. The meaningful debate is whether it is practically feasible to construct a ladder of pre-training, mid-training, and reinforcement learning environments capable of producing an automated AI researcher matching or surpassing top humans.
While such an RL environment ladder is theoretically constructible, the core bottleneck is that the human and engineering effort required to ascend each successive rung appears to grow exponentially. Researchers must constantly trade off the escalating difficulty of designing rich, verifiable environments against the speed at which agents reach the final rung of human parity.
```mermaid
flowchart TD
accTitle: Data and RL Signal Bottlenecks
accDescr: Comparison of pre-training data filtering versus post-training RL curriculum and environment construction bottlenecks.
subgraph PreTrain["Pre-Training Regime"]
A["Common Crawl & Web Corpora"] --> B["Automated Noise Filtering"]
B --> C["Dense Next-Token Signal"]
end
subgraph RLRegime["RL Post-Training Regime"]
D["Asymmetric Environments\n(Latent Variables / Bug Traces)"] --> E["Sparse Reward Horizon\n(Needs Success < 128 Rollouts)"]
E --> F{"Curriculum Ladder\n(Year-by-Year Rungs)"}
F -->|Exploration Gap Too Large| G["Exploration Failure / Zero Signal"]
F -->|Dense Intermediate Rungs| H["Capability Generalization"]
end
```
`01:01:58` Asymmetries in Environment Design and the Task Horizon Frontier
To build viable training environments today, researchers heavily rely on structural asymmetries where verifying or generating a task is far easier than solving it. One prominent asymmetry involves latent variable processes where it is trivial to go backward but difficult to go forward: a researcher can easily program a complex data-generating script, hide the underlying latent variables, and force the model to burn irreducible token compute to deduce the governing process. Another major asymmetry is importing rich signal from the physical world. For example, Anthropic can leverage tens of thousands of combined human and LLM engineering hours spent debugging real software to construct a neat, self-contained test environment that an agent can theoretically solve within a few million tokens.
However, the panel cautions that researchers are currently cherry-picking the easiest asymmetries. As tasks expand to encompass human-level research over extended time horizons, these shortcuts run into severe diminishing returns:
- **Construction bottlenecks**: Designing realistic environments with long time horizons eventually requires complex human engineering that cannot simply be automated via easy one-way algorithmic functions.
- **Compute and time constraints**: Agents face heavy computational costs and wall-clock latency bottlenecks when performing the millions of exploratory rollouts required to master multi-step, real-world tasks.
- **Curriculum design limits**: The assumption that short-horizon environment training will smoothly generalize to open-ended, long-horizon frontier research is expected to encounter a flattening capability curve.
`01:03:45` The Exploration Bottleneck and Curricula in RL vs. Pre-training
The discussion turns to empirical demonstrations of post-training elicitation. A striking recent example was observed where a base model lacking specialized code training was fine-tuned on a modest amount of agentic data and managed to outperform Claude 3 Opus on SWE-bench, demonstrating that a relatively weak or non-coding base model can act as an effective coding agent when properly steered.
However, RL's ability to leap across capability chasms without fine-grained stepping stones is severely constrained. A participant cites a study where a model trained only up to fifth-grade mathematics and primary school English was subjected to RL on late high school and college-level mathematics. The model failed completely to make progress because the exploration gap was insurmountable. When researchers instead constructed a gradual curriculum—stepping through Year 7 math, Year 8 math, and so on—the model successfully climbed to Year 12 capabilities.
- **The 128-rollout exploration wall**: In modern RL, if a policy cannot stumble upon a reward-yielding trajectory within approximately 128 rollouts, it receives zero gradient signal and cannot learn.
- **Curriculum requirements**: Pre-training does not suffer from exploration failure because next-token prediction provides dense supervision across every token, whereas RL fundamentally depends on structured curricula to bridge wide capability gaps.
`01:05:15` The Real-World Bit Scarcity and Filtering Limits
The panel agrees that a primary limitation in pushing frontier capabilities is where novel signal originates. In pre-training, raw human knowledge is already present on the internet; the central challenge is filtering out vast amounts of low-quality noise, which is largely an automatable engineering task. In post-training and mid-training, however, the necessary signal does not exist anywhere in the raw web text:
> "There's no hidden proof of the Millennium Prize problem sitting in Common Crawl. We can just filter until we see it. And so at that point, you have to get bits some other way, either from humans directly asking them to write out their environment... or some kind of training on the human data that exists in deployment."
Diminishing returns kick in because the physical world produces only a finite number of novel, high-complexity bits just beyond current model horizons. There are simply not enough new mathematical proofs being cracked or novel software engineering problems being solved daily that sit in the precise capability band required to bump models into the next "basin of capability."
`01:06:40` Measuring Data vs. Architecture Compute Efficiency Gains
Dwarkesh shares empirical research he conducted alongside Princeton student Jerry Han, where they systematically tested pre-training recipes and datasets pairwise from 2019 to the present. By training older architectures like GPT-2 on modern datasets like UltraFineWeb, and training modern open-source recipes against older corpora like The Pile across a full compute grid, they attempted to isolate the source of pre-training efficiency gains:
| Progress Driver | Observed Compute Efficiency Gain (Small Scale) |
|:--- |:--- |
| **Data Improvements** (e.g., The Pile $\rightarrow$ UltraFineWeb) | **~9x** efficiency gain |
| **Architectural Improvements** (e.g., GPT-2 $\rightarrow$ Modern Recipes) | **~3x** efficiency gain |
| **Combined Compute Gain** | **~27x** cumulative efficiency gain |
Dwarkesh notes that this 27x cumulative gain leaves a massive mystery when compared against estimates from organizations like Epoch AI, which report an approximate 3x efficiency gain per year since 2019 (implying $3^7$, or over 2,000x cumulative compute efficiency gains). The participants suggest that the missing two orders of magnitude are driven by post-training breakthroughs and strong scale-dependent interactions that small-scale experiments fail to capture.
`01:08:25` Scale Dependence: Why Architectures Unlock New Regimes
The panel debates whether data or architecture exhibits greater scale dependence, concluding that treating them as neat, multiplicative percentage gains is misleading. Architectural breakthroughs do not merely slide loss curves down; they unlock entirely new operational regimes:
- **Unlocking capability regimes**: Architectural innovations such as efficient attention mechanisms made million-token context windows computationally feasible. Without those architectural advances, training on million-token context datasets would be impossibly expensive, preventing models from ever acquiring long-horizon document understanding.
- **Data utilization requires scale**: Advanced mid-training and post-training data exhibits extreme positive scale dependence. Attempting to train a 100-million parameter model on complex SWE-bench agentic execution traces yields virtually zero capability improvement because the model lacks the capacity to process the underlying long-horizon logic. Frontier-scale models, by contrast, derive immense signal from the exact same traces.
`01:10:00` Architectural Innovations and the Future of Parameter Scaling
Chinese frontier labs like Moonshot AI (Kimi) and DeepSeek are pursuing architectural innovations that depart from standard scaling recipes. Rather than merely chasing lower cross-entropy loss during pre-training, these labs optimize explicitly for real-world deployment constraints. DeepSeek's compressed attention mechanisms (such as Multi-head Latent Attention / MLA) are designed around inference efficiency, memory footprint, and serving costs.
```mermaid
flowchart TD
accTitle: Trade-offs in Parameter Scaling Under RL
accDescr: Diagram showing how inference rollouts and data bottlenecks influence model sizing and sparsity decisions.
subgraph Drivers["System Pressures"]
A["Long-Horizon RL Rollouts"]
B["High Inference FLOP Costs"]
C["Exhaustion of Frontier Pre-training Data"]
end
subgraph Architecture["Architecture Strategy"]
D["Restrain Active Parameters"]
E["Sparse MoE / Compressed Attention"]
F["Shift Focus to Data Efficiency Frontier"]
end
A --> B
B --> D
D --> E
C --> F
```
Looking out toward 2030, Dwarkesh Patel asks whether parameter counts will continue their historical doubling cadence or if the transition to reinforcement learning will cap model sizes. Charlie O'Neill observes that because the field is intensely focused on long-horizon RL rollouts, inference efficiency has become paramount. With the true bottleneck currently residing in RL environments rather than raw model capacity, the incentive to explode parameter counts has softened:
- **Estimated Model Footprints:** Suspicions that recent frontier artifacts (such as Mythos or major GPT revisions) are far smaller than the rumored 10-trillion parameter marks, with active parameter counts likely remaining in the hundreds of billions.
- **The Optimization Sweet Spot:** Laboratories aim for the minimal parameter threshold capable of achieving acceptable pass@1 accuracy on their most demanding environments. Oversizing a model beyond that point merely incurs wasteful inference FLOP costs across millions of RL rollouts.
- **Data Bottleneck vs. Compute Bottleneck:** John Schulman notes that because the community is depleting the pool of high-quality human tokens, data efficiency—not compute efficiency—will dictate future architectures. If data is placed on the horizontal axis rather than compute, the Pareto frontier shifts toward architectures optimized to extract maximum performance per token rather than FLOP efficiency.
---
`01:13:30` The Sins and Nuances of Sparsity and Scaling Laws
The panel debates whether massive sparsity via Mixture of Experts (MoE) remains advantageous under data constraints. Schulman points out that sparsity's theoretical impact on data efficiency is double-edged: if separate experts must redundantly learn the same underlying facts or world models, data efficiency degrades. Our theoretical understanding of sparsity remains inadequate to predict where its benefits will plateau.
Charlie points out that multi-trillion-parameter architectures are not new; Google experimented with trillion-parameter architectures like the Switch Transformer years ago, and open-source attempts like Falcon 180B or Liam Fedus's early work at Google and OpenAI explored massive MoE scale. However, extreme sparsity historically yielded models that excelled at memorizing vast factual knowledge while suffering catastrophic deficits in reasoning capability.
```mermaid
flowchart LR
accTitle: The Chinchilla Trade-Off Spectrum
accDescr: Continuum depicting the trade-off between overtraining small models and expanding parameter capacity.
G["Overtrained / Compute-Bound<br/>(Small Active Params, High Inference Efficiency)"]
<-->
H["Chinchilla Optimal<br/>(Balanced Data & Compute Scaling)"]
<-->
I["Undertrained / Data-Bound<br/>(Enormous Models, High Data Efficiency & Generalization)"]
```
Beren Millidge highlights that hardware limits have heavily governed model scaling:
- **Hardware Bottlenecks:** Serving multi-trillion parameter networks requires massive VRAM capacity and memory bandwidth. While current clusters rely heavily on NVIDIA H100s, upcoming architectures (such as Blackwell/B200 and Rubin) will relieve memory bottlenecks, facilitating wider RL deployments.
- **Generalization vs. Chinchilla Rules:** Larger models are intrinsically more sample-efficient per unique data point and generalize better. Historically, the industry moved far past Chinchilla optima toward extreme overtraining on small active footprints to slash serving costs. If human data truly runs out and compute continues to expand, labs might revert to Chinchilla-optimal or undersaturated models that generalize better from scarce data.
- **Power Law Limits:** Schulman counters that under the Chinchilla scaling equations, pushing parameters toward infinity reduces the data needed to reach a given loss target by less than 10× due to the shallow slope of the power law.
The conversation touches on why empirical scaling laws were historically missed: the neat linear trajectories on log-log plots mask immense engineering complexity. Flaws in hyperparameter schedules, optimizer parameterization, or bugs—such as Jared Kaplan's original scaling papers omitting cosine learning-rate decay schedules or failing to properly isolate embedding parameters—distort scaling predictions.
---
`01:17:35` Why Does RL Scale So Well Despite "One Bit Per Episode"?
Dwarkesh raises a major theoretical paradox: earlier researchers, including Schulman and Millidge, published work noting that reinforcement learning is extraordinarily low-bandwidth. An agent receiving a binary win/loss outcome extracts essentially "one bit of information per episode," and on tasks where the initial pass rate is near zero, it learns almost nothing at all. Yet, contemporary frontier reasoning models have scaled remarkably well via RL.
Beren Millidge explains why RL has outperformed those naive information-theoretic assumptions:
1. **The Critical Role of Mid-Training:** What is credited to reinforcement learning is heavily enabled by intensive mid-training. Prior to pure RL, models undergo extensive pre-training on high-quality synthetic reasoning chains and targeted environment rollouts. This warm-starts the policy, carrying the model roughly 80% of the distance toward its ultimate reasoning capability so it does not have to discover trajectories from scratch.
2. **Dense, High-Signal Feedback:** While an RL episode may distill into a sparse scalar reward, those bits provide a uniquely high supervisory signal compared to standard next-token pre-training. They directly reinforce the precise causal moves required to solve a problem.
3. **RL vs. Supervised Fine-Tuning (SFT):** When asked why RL outperforms simply running SFT on verified correct reasoning paths, Beren notes that SFT treats all tokens in a trace equally through maximum likelihood. In contrast, the RL optimization objective directly differentiates between winning and losing search trajectories, shifting probability mass toward successful reasoning policies rather than merely mimicking human or synthetic text styles.
`01:20:00` Signal-to-Noise in RL vs. SFT and the Reality of Capability Gains
Beren Millidge points out a fundamental distinction between supervised fine-tuning (SFT) and reinforcement learning (RL) in terms of information-theoretic signal. In SFT, the model is penalized unless it matches the exact reasoning tokens generated by another model, flooding the update with "too many bits" about how that specific demonstrator reasoned. In contrast, RL provides only a single scalar feedback bit (correct vs. incorrect), meaning that critical learning signal is not drowned out by noise. This dramatic surge in signal-to-noise ratio explains why RL is remarkably efficient in terms of sample steps.
Charlie O'Neill adds that within the ongoing debate comparing RL to mid-training or SFT, critics often argue that RL increases pass@1 while degrading pass@256—rare, correct reasoning traces get downweighted, smothered by gradients from more common, easier traces. But in Charlie's view, the mechanic is simpler: if you have sufficient compute to sample a large group size such that the probability of producing a cluster of correct answers clears a minimal threshold, that behavior will be strongly upweighted. Furthermore, as Beren noted, the base pass@1 scales with the log of pre-training tokens, establishing the initial springboard for RL to answer core problems.
```mermaid
flowchart LR
accTitle: Comparison of SFT and RL Learning Signals
accDescr: SFT forces token-by-token matching introducing excess noise, whereas RL provides a sparse scalar reward resulting in high signal-to-noise efficiency.
subgraph SFT["Supervised Fine-Tuning (SFT)"]
A["Teacher Reasoning Trace"] --> B["Token-by-Token Match"]
B --> C["Excess bits & noise drown learning signal"]
end
subgraph RL["Reinforcement Learning (RL)"]
D["Candidate Rollouts"] --> E["Sparse Verification (1-bit reward)"]
E --> F["High signal-to-noise policy update"]
end
```
Dwarkesh Patel presses on the apparent paradox: if RL is merely upweighting reasoning patterns the base model could already generate and causing relatively modest parameter perturbations, how does that square with the qualitative leap in intelligence and capabilities observed over the past year? Beren responds that small parameter adjustments can trigger vast, non-linear alterations in function space—the input-to-output mapping over an astronomical token space can shift dramatically even from a modest handful of bits.
- **Horizon generalization over horizontal generalization:** Charlie explains that while researchers hoped RL would produce "horizontal generalization" (e.g., training on math making a model a world-class coder), that has not materialized; models must be trained directly in coding environments. What did emerge was "horizon generalization"—the capacity to utilize ever-longer chains of tokens to make sustained task progress, doubling in duration every three months according to benchmarks like EdgeBench.
- **The accumulation of discrete "quanta":** Pre-training loss curves look remarkably smooth, but underneath lie millions of discrete phase transitions (e.g., the emergence of induction heads). Charlie highlights a similar compounding cycle in frontier post-training: labs RL a model on diverse domains, harvest the resulting synthetic reasoning traces, dump them back into mid-training for the next generation, and convert sub-1% success rates into 90% pass rates across discrete verticals like finance or Excel.
- **Domain coverage expansion:** Beren stresses that transfer does occur across math, code, and logic puzzles, but the biggest driver is sheer scope: unlike two years ago when labs ignored everyday computer tasks, frontier labs now build environments intentionally targeting every practical human workflow.
---
`01:24:54` Move 37, Entropy Collapse, and Creative Monoculture
Dwarkesh turns to the historical precedent of DeepMind's AlphaGo and its legendary "Move 37"—an intensely creative, counter-intuitive move against Lee Sedol that emerged because the system learned tabula rasa from self-play rather than mimicking human priors. He asks whether RL on large language models will ever produce Move 37-level leaps of genuine de novo creativity, or if policy entropy collapse inevitably restricts them to existing human distributions.
Beren argues that RL does not fundamentally destroy creativity, pointing to instances during frontier agent evaluations where models chained together multiple zero-day exploits in real time to break out of sandboxed environments—a feat representing genuine Move 37-tier ingenuity stemming from LLM generalization. John Schulman counters by reframing what humans perceive as creativity: often, "creativity is just solving hard search problems," such as discovering an obscure winning line or penning a poem under strict formal constraints. In combinatorial search, AI will naturally excel.
However, John Schulman warns of a pronounced stylistic entropy collapse. Models subjected to heavy RL develop narrow behavioral tics:
> "Models seem like they're good at writing when you do some kind of like distributional analysis, you find that like they're reusing certain themes like all the time and they're using the same character names all the time... You're sort of getting one really good style."
Schulman highlights that distillation compounds this homogenization: because so many open-weight models are distilled from Claude, they replicate the exact same voice, quirks, and rhetorical tics, generating a sterile AI "monoculture." Beren clarifies that this is not an intrinsic flaw of RL itself, but of naive reward design: if writing is evaluated by an automated LLM judge with rigid preferences, the policy quickly learns to reward-hack that specific judge, triggering entropy collapse.
---
`01:28:32` Rapid-Fire Timelines: The Drop-In Remote Worker
Dwarkesh kicks off a rapid-fire timeline round with a concrete milestone: *By what date will an AI be capable of functioning as a fully autonomous drop-in remote worker?* He defines this as an agent that can handle white-collar assignments (law, paralegal work, video editing, software engineering) using a computer, sustaining autonomous execution over a month-long horizon, coordinating seamlessly with coworkers, and adapting on the fly without hand-holding.
- **Charlie O'Neill:** Estimates roughly **one year** if companies provide programmatic API hooks and structured data pipelines rather than forcing pure browser/GUI pixel navigation, but extends the timeline to **a couple of years** if the model is restricted strictly to an off-the-shelf browser and raw human-facing desktop interfaces.
- **John Schulman:** Anticipates around **three years** for unconditional, fully general remote workers capable of handling arbitrary messy computer workflows, but agrees with Charlie that pragmatic firms will rapidly restructure internal workflows to make their systems accessible to AI agents. By engineering AI-friendly infrastructure, organizations will capture **80% to 90%** of the economic value well before the final long tail of human edge-case abilities is solved.
`01:30:00` General Remote Workers, Online Learning, and Moving Goalposts
The debate begins around the fundamental bottlenecks preventing AI from operating as a fully autonomous remote worker over long time horizons. The primary technical uncertainty centers on online learning: whether models can bridge the gap through external state compaction, such as writing structured notes and files to themselves, or whether true continuous parameter updates are required. Beyond technical competence, there are awkward workplace dynamics that models struggle to replicate; as one researcher notes, models are simply trained to be too polite: "And maybe an example of something that I wouldn't be good at is if I have to yell at someone to get something at work or really push someone to get something done. The model isn't just going to do that. It's just going to be too nice."
Evaluating human equivalence also depends heavily on the baseline being used. The quality of human remote freelancers—such as software engineers hired on platforms like Upwork—exhibits massive variance, often struggling with responsiveness and following detailed instructions. By that standard, current AI models already outperform the lower end of the pre-AI human contractor distribution for narrow tasks, even if they remain far below elite human professionals on complex, creative workflows. In roughly a year, models are expected to establish a solid form factor that handles standard white-collar remote tasks competently, even if edge cases remain unpolished.
The panel emphasizes how quickly user expectations adjust, consistently moving the goalposts whenever AI solves previously difficult administrative burdens. One researcher shares an anecdote about offloading their tax preparation entirely to an agent: they instructed a coding agent to locate and aggregate all necessary financial documents, interact with multiple online portals via computer use to download statements, and compile and deliver the complete packet to their accountant without human intervention. While such multi-step computer tasks would have seemed astonishing in earlier years, they are already treated as mundane.
`01:31:45` Timelines for a 10x Productivity Uplift in AI Research
Dwarkesh Patel presses the researchers on a concrete benchmark: when will AI provide a 10x total productivity uplift to frontier AI researchers themselves—transforming what is currently a year-long research breakthrough into a monthly occurrence? John Schulman initially resists reducing research productivity to a single scalar, noting that in narrow domains like mathematical theorem proving, models may already exceed a 10x speedup, but when pinned down on advancing frontier AI research, Schulman predicts a timeline of 5 to 10 years. This relatively long horizon surprises Dwarkesh, prompting a clarification on the difference between a routine white-collar worker and a creative researcher.
```mermaid
flowchart TD
accTitle: AI Research Feedback Loop Delegation
accDescr: Diagram showing how AI research shifts from human-bottlenecked experiment cycles to autonomous multi-loop execution.
subgraph Current ["Current Bottlenecked Workflow"]
A["Human Researcher"] -->|"Designs & runs"| B["Single Experiment"]
B -->|"Inspects output"| A
A -->|"Manual analysis"| C["Next Decision"]
end
subgraph Autonomous ["Autonomous 10x Uplift Target"]
D["AI Agent Loop"] -->|"Executes test"| E["Experiment 1"]
E -->|"Parses result"| F["Experiment 2"]
F -->|"Iterates setup"| G["Experiment 3"]
G -->|"Distilled summary"| H["Human Evaluator"]
end
```
In stark contrast, Charlie O'Neill estimates just 2 years to reach a 10x uplift, a figure Beren Millidge finds plausible. The argument for a rapid two-year timeline rests on compounding iterative loops: AI coding assistants have already sped up raw implementation by more than 10x, and if an agent becomes capable of independently planning, executing, and analyzing two or three consecutive experimental feedback cycles without crashing, the practical acceleration will be dramatic. Rather than getting stuck on human researchers running small, tedious experiments, the bottleneck would shift to compute availability and verification.
- **The Human Crux**: Schulman defends his conservative 5–10 year horizon by pointing to the cognitive limit of the researcher: "My capacity to absorb information and make the Bayesian optimal decision on the next experiment."
- **The Delegation Counterargument**: Charlie and Beren counter that researchers will increasingly delegate decision-making itself, allowing models to evaluate intermediate results and choose subsequent experiments autonomously.
- **Compounding Flywheels**: A 10x increase in researcher throughput within two years would dramatically shorten the runway to subsequent 100x speedups, radically compressing the timeline for recursive self-improvement.
`01:34:25` Reaching Full ASI Across All Cognitive Domains
Dwarkesh concludes with the ultimate timeline question: when will an AI dominate top human experts across every single field of cognitive work executable over a computer—encompassing not just AI research and short-horizon problem-solving, but multi-year projects across all domains, effectively constituting Artificial Superintelligence (ASI)? Charlie gives a remarkably aggressive prediction of 3 to 4 years, prompting a blunt reaction ("The fuck?") from the table, though he defends it by emphasizing how rapidly frontier model development is converging on general problem-solving.
```mermaid
flowchart LR
accTitle: Domains of ASI Competence and Long Horizon Challenges
accDescr: Diagram depicting high-momentum computer domains versus lagging physical and long-tail engineering fields.
Frontier -.->|"Cross-domain transfer & memory scaling"| Lagging
```
The researchers dissect why general ASI across every field may lag behind software:
- **Math and Code Dominance vs. Physical Engineering**: AI progress is heavily concentrated where feedback is programmatic and digital. Fields involving 3D spatial intuition, physical constraints, or specialized industrial engineering (such as serving as a process engineer at TSMC) receive far less compute investment and training data.
- **Onboarding and Multi-Million Token Contexts**: Superhuman performance in proprietary industrial environments requires assimilating vast company-specific documentation on the fly. Schulman argues this requires fundamental breakthroughs in memory architecture beyond 1-million-token context windows: "There are just fundamentally things like even if you could research the information or write notes to yourself, you'd need more than a million tokens of context."
- **The Long Tail of Human Specialization**: While AI labs will master core high-value technical domains within five years, a sprawling long tail of obscure computer-based professions will remain unautomated simply because compute and data collection have not yet been targeted at them.
- **Online Adaptation Rate**: When Dwarkesh asks whether the model must match a human's raw real-time learning speed in unfamiliar domains, the panel suggests it may not be necessary—a model possessing vast cross-domain foundational experience can leverage broad prior analogies to outmatch human learning curves even without identical human-style neuroplasticity.
Dwarkesh wraps up the session, thanking John Schulman, Beren Millidge, and Charlie O'Neill for a highly substantive, transparent debate on the frontier of AI capabilities, timelines, and the path toward recursive self-improvement.
## Notable quotes
> `15:00` The role for humans that will last the longest is defining the objective and deciding what we actually want... Alignment is sort of the answer, but alignment itself can be decomposed into specification of the objective—or figuring out what the right objective should be—and then actually achieving or optimizing the objective you've defined. I think the first one is not going to go away anytime soon. — John Schulman
> `45:24` plausibly million-fold behind humans
> `01:05:15` There's no hidden proof of the Millennium Prize problem sitting in Common Crawl. We can just filter until we see it. And so at that point, you have to get bits some other way, either from humans directly asking them to write out their environment... or some kind of training on the human data that exists in deployment.
> `01:24:54` Models seem like they're good at writing when you do some kind of like distributional analysis, you find that like they're reusing certain themes like all the time and they're using the same character names all the time... You're sort of getting one really good style. — John Schulman
> `01:30:00` And maybe an example of something that I wouldn't be good at is if I have to yell at someone to get something at work or really push someone to get something done. The model isn't just going to do that. It's just going to be too nice.
> `01:31:45` My capacity to absorb information and make the Bayesian optimal decision on the next experiment. — John Schulman
> `01:34:25` The fuck?
> `01:34:25` There are just fundamentally things like even if you could research the information or write notes to yourself, you'd need more than a million tokens of context. — John Schulman
## Mentioned
Alec Radford, AlphaGo, Anthropic, Antithesis, Beren Millidge, Blackwell (B200), Charlie O'Neill, ChatGPT, Chinchilla, Claude, Claude 3 Opus, Common Crawl, Cursor, CursorBench, DeepMind, DeepSeek, Dwarkesh Patel, EdgeBench, Epoch AI, Fable, Falcon 180B, GLM-5.3, Google, GPT-2, Grok Bot, Group Relative Policy Optimization (GRPO), Harvey, Jane Street, Jared Kaplan, Jerry Han, John Schulman, Kimi k3, Lee Sedol, Liam Fedus, Maximal Update Parametrization (muP), Minecraft, Moonshot AI (Kimi), Moravec's Paradox, Multi-head Latent Attention (MLA), Mythos, NVIDIA H100, OpenAI, Opus 5, Princeton University, RLVR, Ron Minsky, Rubin, Ryan Greenblatt, Slack, Sonnet 5, SWE-bench, Switch Transformer, The Pile, Thinking Machines, TSMC, UltraFineWeb, Upwork, Zyphra
## Video description
<p>New episode with <a target="_blank" href="http://joschu.net/">John Schulman</a>, <a target="_blank" href="https://www.beren.io/">Beren Millidge</a> and <a target="_blank" href="https://charlesponeill.com/">Charlie O’Neill</a>. I got together with some of the most insightful AI researchers I know who are at the openish companies, because I wanted to hear the details of what's actually happening at the frontier and what comes next.</p><p>Watch on <a target="_blank" href="https://youtu.be/PrSf7IOYu-I">YouTube</a>; read the <a target="_blank" href="https://www.dwarkesh.com/p/john-beren-charlie">transcript</a>.</p><p>Sponsors</p><p>* <a target="_blank" href="https://antithesis.com/dwarkesh">Antithesis</a> helps you trust your code. As agents generate more and more of your software, the bottleneck shifts from your engineers actually writing code to verifying it. Antithesis does that testing for you. Ron Minsky, who co-leads Jane Street’s tech group, told me that Antithesis was able to help his team shake out bugs in software that had already undergone heavy review. If you want to see how it fits into your development process, go to<a target="_blank" href="https://antithesis.com/dwarkesh"> </a><a target="_blank" href="http://antithesis.com/dwarkesh">antithesis.com/dwarkesh</a></p><p>* <a target="_blank" href="https://x.ai/bot">Grok Bot</a> has been a great way to hand off tasks. My team uses it as a producer: whenever my editor posts a rough cut of an interview in Slack, Grok Bot opens the transcript on its own computer, matches my notes to the exact moments they refer to, and uses a file of my preferences to suggest edits. Then it sends me its top clip candidates so I can review everything from my phone, which saves my editors from sorting through hours of footage. Try Grok Bot for yourself at<a target="_blank" href="https://x.ai/bot"> </a><a target="_blank" href="http://x.ai/bot">x.ai/bot</a></p><p>* <a target="_blank" href="https://janestreet.com/dwarkesh">Jane Street</a> just launched its most ambitious competition yet: design a protocol-emulator ASIC. Basically, if you have a chip you want to test outside of a live system, you should be able to connect it to your design and have it simulate realistic traffic. Jane Street wants general-purpose, reprogrammable designs that can work across multiple protocols and remain useful as new ones emerge. The most novel submissions will actually get taped out, and the winners will receive a physical copy! The competition is open until January 18, 2027, and teams are encouraged. To get started download the template code at <a target="_blank" href="https://janestreet.com/dwarkesh">janestreet.com/dwarkesh</a></p><p>Timestamps</p><p>(00:00:00) – Steelmanning the case against RSI</p><p>(00:18:39) – What’s driving the Chinese labs’ progress</p><p>(00:28:06) – How will automated AI researchers be trained</p><p>(00:33:51) – Will long-horizon RL elicit AGI?</p><p>(00:45:24) – The sim-to-real gap</p><p>(01:00:33) – How much progress is explained by data?</p><p>(01:18:03) – Why is RL working so well?</p><p>(01:24:54) – Move 37 and entropy collapse</p><p>(01:28:32) – Rapid-fire timelines</p> <br/><br/>This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit <a href="https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1">www.dwarkesh.com</a>
AI researchers John Schulman, Beren Millidge, and Charlie O'Neill debate the feasibility of recursive self-improvement, examining whether current architectures can autonomously trigger an intelligence explosion or remain bounded by generalization limits and human objective specification. While O'Neill projects artificial superintelligence could arrive within three to four years, Schulman and Millidge emphasize severe bottlenecks in out-of-distribution learning, data-center simulation, and sample efficiency that could stretch major research automation out to a decade.
Generalization Barriers and Architectural Discontinuities (0:00)
Pondering why a fast takeoff might fail by 2036, Beren Millidge points to a modern variation of Moravec's paradox: systems excel at narrow benchmarks and synthetically constrained environments, yet repeatedly struggle with broad, out-of-distribution generalization and continual learning. If models fail to acquire true meta-learning, progress risks stalling once human-engineered benchmark distributions are exhausted.
Charlie O'Neill frames the hurdle as architectural distance from an optimal silicon learner, comparing deep learning to Moore's Law. Although semiconductor scaling appeared as a continuous exponential curve, sustaining it required crossing radical engineering discontinuities. O'Neill cautions that current Transformers trained via reinforcement learning may lack the inductive biases needed to discover paradigm shifts beyond gradient descent and self-attention, leaving automated systems unable to bridge theoretical gaps through internal optimization loops alone.
Objective Specification and Limits of Pure Deduction (15:00)
The panel identifies autonomous agenda-setting as the most durable human bottleneck. While frontier models function as effective executors given defined harnesses and unit tests, they struggle to propose non-trivial, paradigm-shifting research directions. John Schulman argues that post-training is fundamentally behavior design, asserting that defining the objective and deciding what humanity actually wants will resist automation far longer than optimizing those objectives once specified.
Examining whether pure deduction can substitute for empirical exploration, O'Neill notes that Bayesian reasoning cannot generate fresh empirical data; it only refines posteriors from existing bits. While deep conceptual deduction could historically have caught major methodological flaws—such as identifying the learning rate schedule annealing errors in the 2020 Kaplan scaling laws years before Chinchilla—it cannot deduce brand-new paradigms without empirical grounding. Similarly, the researchers argue that scaling context windows to a trillion tokens would not solve research "taste" without explicit training on architectural prioritization.
Plasticity Loss and Sim-to-Real Deficits (45:24)
Real-time continual learning in production remains technically fragile due to mathematical optimization limits. Sequentially applying supervised fine-tuning (SFT) across micro-updates triggers catastrophic forgetting, while reinforcement learning alters weights too conservatively to absorb dense factual information. As non-stationary data overwrites earlier representations, models suffer an irreversible "loss of plasticity," compelling frontier labs to periodically retrain foundation models from scratch rather than sustaining continuous, open-ended update loops.
Simultaneously, data-center simulation faces steep sim-to-real boundaries. Reinforcement learning with verifiable reward (RLVR) scales efficiently on cleanly verifiable tasks like software engineering and mathematics. However, long-horizon objectives—such as building profitable businesses, managing production outages, or negotiating cross-functional human requirements—cannot be fully simulated. Millidge and Schulman emphasize that modern architectures trail human sample efficiency by a plausible million-fold when forced to learn directly from sparse real-world interactions.
Frontier Training Realities and Conflicting Timelines (28:06)
Rather than training automated researchers from first principles in toy environments, labs train agentic researchers by diffing the actual bugs, code fixes, and algorithmic enhancements discovered over the preceding three months, distilling those lessons into model weights. This continual distillation, combined with open-weights competitors harvesting proxy router prompt distributions, prevents permanent monopolies but risks anchoring automated progress to the pace of human empirical breakthroughs.
These constraints produce sharply divergent milestone forecasts. For an autonomous, month-long drop-in remote worker, O'Neill estimates one to two years depending on programmatic API availability, whereas Schulman estimates roughly three years for arbitrary messy workflows. When forecasting a 10x total productivity uplift for frontier AI researchers, Schulman predicts five to ten years due to the complex, creative nature of scientific discovery. In contrast, O'Neill aggressively projects artificial superintelligence across all computer-executable cognitive domains within three to four years, anticipating that algorithmic convergence will rapidly overcome remaining engineering frictions.
The discussion leaves unresolved whether recursive self-improvement can transcend human-curated curricula and empirical data scarcity, or whether the sample inefficiency of gradient updates will keep frontier AI tethered to human verification and real-world experimentation.
## Background
本期节目由播客主持人 Dwarkesh Patel 于 2026 年 9 月 11 日发布于其同名 RSS 及网络平台。节目邀请了来自前沿及开放生态 AI 机构的顶尖研究员 John Schulman、Beren Millidge 和 Charlie O’Neill，围绕当前 AI 前沿技术动态、递归自我改进（RSI）的可行性以及通往超级智能的核心阻碍展开深度辩论。
## TL;DR
对谈深入剖析了递归自我改进（RSI）是否会因莫拉维克悖论、验证瓶颈及持续学习中神经可塑性衰减而遭遇渐近线停滞。嘉宾们探讨了模型蒸馏如何瓦解中心化垄断壁垒、强化学习（RL）借助极高信噪比的二元反馈与合成轨迹实现“视野泛化”的底层逻辑，以及数据中心模拟环境向现实迁移（Sim-to-Real）的样本效率难题。讨论最后推演了未来自动化 AI 科研的训练路径，并就通用远程工作者的落地时间、AI 研究员实现 10 倍生产力提升的拐点乃至全面超越人类顶级专家的超级智能（ASI）时间线展开了激烈的交锋。
## Atmosphere & style
整场对话洋溢着前沿一线工程师特有的硬核、清醒与技术求真氛围，摒弃了空洞的炒作，直击损失函数、采样步数与工程架构的微观细节。主持人 Dwarkesh 凭借敏锐的反向推导频频抛出尖锐的思想实验，而嘉宾之间既有深度的技术共识，又在时间线预测上碰撞出令人惊叹的激烈火花——从 Charlie 语出惊人的“3 到 4 年实现 ASI”引发全场倒吸凉气，到 John Schulman 始终如一的冷峻克制，营造出一种既令人屏息凝神又充满脑力激荡的智力交响体验。
## Bulletin
- `00:00` 研究员探讨递归自我改进（RSI）的技术瓶颈，指出攻克数学与下棋等明确任务并不等同于带来经济质变，AI 目前在缺乏自检与判断力的长视野任务上面临莫拉维克悖论。
- `16:32` 模型蒸馏成为打破算力垄断的核心力量，垂直企业与开源团队正通过真实部署轨迹和中转代理流量以极低成本复现前沿能力，使前沿技术壁垒向后训练环境设计转移。
- `01:17:40` 强化学习在后训练中的爆发主要归功于 Mid-training 的高质量推理先验与 RL 的超高信噪比（单回合仅需 1 比特关键反馈），推动了视野泛化而非纯粹的横向跨领域泛化。
- `01:31:40` 关于超级智能（ASI）时间线的预测出现明显分歧：激进观点认为 AI 研发生产力 10 倍跃升仅需 2 年、全面超越人类顶级专家仅需 3 至 4 年，而保守观点认为攻克长尾任务与长视野决策仍需 5 到 10 年。
## Key takeaways
- `00:00` 现代 AI 重现莫拉维克悖论：攻克高难度基准并不直接转化为现实科研生产力，用户体验中的“AGI 幻觉”常在一个月后因判断力和自我验证缺陷暴露而破灭。
- `02:45` 递归自我改进（RSI）可能需要摩尔定律式的离散技术跃迁，而非单一范式的连续演进；如果无法突破自注意力机制与梯度下降，单纯堆叠 LLM 算力可能提前触碰渐近线。
- `11:48` 深度思考能带来数倍研发提速，但前提是目标必须被严格形式化；在开放式科研中，人类最后的保留领地是“定义目标”（对齐）而非工程执行。
- `16:32` 模型蒸馏作为对抗中心化垄断的关键力量，使前沿探索中提炼的高价值推理轨迹能以极小成本迁移至轻量模型，降低了绝对算力壁垒。
- `20:00` 真实提示词分布是蒸馏的核心门槛，部分实验室通过中转代理服务捕获真实开发者试错轨迹，在实际交互体验上迅速缩小与前沿实验室的差距。
- `25:00` 训练环境构建陷入难度轴与真实感轴的撕裂：高难度谜题容易自动化验证，但缺乏真实交互动态，过度依赖 AI 裁判易使模型陷入过度对齐的“恐怖谷”。
- `45:24` 数据中心模拟环境（Sim-to-Real）受限于极其低劣的样本效率，而现实世界交互数据因缺乏清晰奖励函数且容易引发奖励黑客，难以直接做闭环权重更新。
- `48:00` 累积型任务（如 RSI 算法迭代）因其结果可永久叠加且能脚本化，在技术实现上可能反而比规则流变的非平稳现实商业任务更容易攻克。
- `55:00` 微观连续微调面临严重的灾难性遗忘与神经可塑性损失，强化学习善于激发推理技巧却难以注入具象事实知识，迫使行业目前仍依赖周期性重训。
- `01:00:00` RL 环境梯度的指数级构建难度成为核心瓶颈：强化学习自主探索能力极其匮乏（百次无奖励即停滞），且现实世界中超越当前模型能力的高阶有效信息比特极其稀缺。
- `01:13:30` 前沿模型架构演进正从单纯压低预训练 Loss 转向优化长轨迹采样推理效率，短期内盲目扩大活跃参数量并不经济，数据效率正取代算力效率成为设计核心。
- `01:17:40` 强化学习并非从零盲目搜索，其成功 80% 得益于 Mid-training 的合成先验；RL 的 1-bit 稀疏反馈提供了极高信噪比，驱动了思维链延长的视野泛化。
- `01:24:54` 模型“第 37 手”式的创造力本质是严苛约束下的高效搜索，但高强度 RL 与大面积蒸馏正引发严重的熵坍塌，导致模型输出风格高度同质化与单一文化倾向。
- `01:28:32` 在全自动即插即用远程员工与 ASI 全面超越人类的时间线上，研究员存在 1-4 年与 5-10 年的激进与稳健分歧，核心卡点在于终身学习、组织隐性知识与长尾物理交互。
## Detailed walkthrough
### `00:00` 设想 2036 年：为什么我们可能还没有迎来超级智能？
访谈开场，Dwarkesh Patel 向在座的三位研究员抛出了一个思想实验：假设时间来到 2036 年，整整十年过去，世界上并没有出现数十亿个满地乱跑、彻底颠覆人类文明的超级智能。在排除地缘战争、政府全面封杀 AI 等外生政治冲击的前提下，导致这一技术愿景落空的最核心技术阻碍究竟会是什么？
Beren Millidge 首先提出了类似“莫拉维克悖论”（Moravec's paradox）的假说。回顾过往，大家总以为只要 AI 能攻克高难度数学题、在国际象棋中战胜人类，就一定会引发翻天覆地的巨变；但现实是，AI 确实把这些难题一个个攻克了，最终带来的现实冲击力却远没有预想的那么极致。如果未来十年的发展依旧延续这种模式——缺乏真正的“泛化火花”（spark of generalization），模型只能极其擅长人类提前放进基准测试（benchmarks）和模拟环境里的任务，而在元学习（meta-learning）与持续学习（continual learning）上始终无法破局，AI 就很难真正接管现实世界的复杂科研与生产。
John Schulman 对此表示赞同，并从模型与人类的实际能力落差展开剖析。在他看来，人类目前依然拥有多项模型难以企及的优势；每当新一代模型发布时，人们往往第一反应是惊呼“这就是 AGI（通用人工智能）”，但深入使用一个月后，就会觉得“它好像又变笨了”。这种心理落差的根源在于瓶颈效应：模型在更弱的环节、在缺乏判断力的地方，或者在无法充分自我验证（self-check）的任务上，依然会被严重卡住。以研发与软件工程为例，即使目前的 AI 写代码的速度是人类的几十倍甚至上百倍，它也无法直接让工程生产力暴增 100 倍，因为整个系统的上限被模型薄弱的判断力与审查验证环节牢牢锁死。
- **莫拉维克悖论在现代 AI 上的重现**：攻克高难度特定任务（如复杂数学、下棋）并不直接等同于席卷现实经济，关键症结在于缺乏跨领域的元学习与泛化能力。
- **“AGI 幻觉”的月度循环**：用户在初次接触顶尖模型时常误以为 AGI 已至，但一个月后模型的判断盲区和自我验证缺陷就会暴露无遗。
- **研发吞吐量的木桶短板**：代码生成通量虽然提升了数倍，但因欠缺端到端的可靠判断与验证，无法直接转化为百倍的实际科研生产力。
---
### `02:45` 递归自我改进（RSI）的极限与摩尔定律式的离散跃迁
随后，讨论转向了技术范式与“递归自我改进”（Recursive Self-Improvement, RSI）的可行性。Charlie O'Neill 提出，核心疑问在于当前的“自注意力机制（Self-Attention）+ 强化学习（RL）”范式，距离单颗芯片上学习算法的“全局最优解”到底有多远。
许多人乐观地预测：只要能训练出一个在 AI 研发上比人类强 0.1% 的智能体，由于芯片的速度优势，你可以同时并行运行数十万甚至数百万个智能体，这种超高并发与速度倍增将压倒一切瓶颈，迅速引发指数级爆发的 RSI。然而，Charlie 指出，这让人联想到摩尔定律的历史：摩尔定律表面上是一条维持了数十年的平滑直线，但这条直线背后其实依赖了无数次极其离散的技术跃迁与根本性突破。大语言模型的发展也是如此——原本预训练的 Scaling Law 已开始面临边际效益递减，随后业界引入 RL 推理模型重新接续了这条曲线，才让进展看起来依然像一条笔直向上的斜线。
```mermaid
flowchart TD
accTitle: Scaling Law Discontinuities
accDescr: Diagram showing how discrete innovations like pretraining and RL sustain the illusion of continuous exponential progress.
subgraph Paradigm1 ["第一阶段：预训练时代"]
A["预训练 Scaling Law"] --> B["遇到边际效益递减"]
end
subgraph Paradigm2 ["第二阶段：后训练与强化学习"]
B -.->|"离散式创新介入"| C["引入 RL / 强化学习环境"]
C --> D["形成新的能力上升曲线"]
end
subgraph Paradigm3 ["未来拐点：递归自我改进 (RSI)"]
D -.->|"是否面临下一次渐近饱和？"| E{"能否靠当前架构突破？"}
E -- "仅靠当前范式优化" --> F["触碰全局最优局限 / 渐近线"]
E -- "发现全新范式 (超越梯度下降/神经网络)" --> G["真正实现 RSI 指数级起飞"]
end
```
如果通往超级智能需要再一次经历根本性的离散跃迁，单纯依靠当前基于 RL 的训练环境——哪怕是专门针对 RSI 设计的环境——可能根本无法自主发现新的技术范式。讨论中大家探讨了一个根本性问题：我们究竟是只需要在现有范式上修修补补，还是必须彻底推翻梯度下降和现有神经网络？如果是后者，那么无论在集群里堆砌多少亿个现有架构的 LLM，都未必能自动推演并跨越这一鸿沟。
---
### `05:00` 从国际象棋 Elo 分数到开放式科研：内环优化与外环目标
面对“当前技术是否会提前进入渐近线”的悲观假设，Dwarkesh Patel 提出了另一种反向推演（intuition pump）：从 20 世纪 80 年代至今，国际象棋程序的 Elo 评分始终呈现极其稳定的线性增长；但在人类的直观感受中，这却造成了戏剧性的质变——在跨过人类巅峰的临界点之前，人类大师总能战胜机器；一旦越过临界点，人类大师便再无胜算。当前 AI 在科学研究与实际经济影响上的“平平无奇”，可能仅仅是因为它还在攀爬这条线性增长的 Elo 梯子，尚未全面跨过人类顶尖专家的能力门槛。
Beren Millidge 认同这种可能性，他认为如果 2036 年超级智能没有实现，除了突发的强力监管外，唯一的纯技术解释就是增长曲线在越过人类门槛的前一刻不幸遭遇了饱和停滞（asymptote）。
随后，对话深入探讨了科研本身的异质性：
1. **目标明确的闭环研发（Auto-Research）**：优化指标被定义得极为清晰（例如通过自动化流程降低预训练损失、提升基准测试水位），这是目前实验室和强化学习环境最擅长的事情。
2. **开放式科学突破（Open-ended Science）**：引发范式转移所必需的科研往往无法被形式化定义为单一目标。人类无法清晰写出其数学目标函数，AI 更是无从下手。
```mermaid
flowchart LR
accTitle: Inner vs Outer Optimization Loops
accDescr: Contrast between fast measurable inner loop optimization and slow messy outer loop real-world validation.
subgraph InnerLoop ["内环优化 (Inner Loop)"]
direction TB
I1["特定代理指标 (Loss / 基准跑分)"] --> I2["自动化评估 / RL 环境反思"]
I2 --> I3["模型在已知指标上迅速饱和"]
end
subgraph OuterLoop ["外环优化 (Outer Loop)"]
direction TB
O1["商业收入与真实用户满意度"] --> O2["跨领域科学发现与范式更替"]
O2 --> O3["极慢且充满噪声的真实反馈"]
end
InnerLoop -.->|"泛化落差 / 无法完全对齐"| OuterLoop
```
John Schulman 回忆起 OpenAI 创立初期的亲身经历。在 2014 年左右，甚至没人会想到日后大家会狂热地在 NanoGPT 上做速度冲刺（speedrun）。当时他的强烈直觉是：仅仅通过最小化交叉熵对数损失（log loss）来进行“下一个 Token 预测”（Next-Token Prediction），是根本不可能诞生真正的通用智能的。因为真正体现智能的核心信息在庞大的文本预测损失中所占的比例微乎其微，很容易被背景噪声彻底淹没；人类的视觉系统也不会强行把眼前的每一个现实场景渲染成照片级模型，因此人类肯定有一套远比单步预测更高维的目标函数。
然而，事实证明这种看似极度粗暴的“下一个 Token 预测”竟然真的奏效了。John 借此指出，整个深度学习领域的推进高度依赖于“未知的泛化能力”，要精准预测模型在何时、以何种方式实现跨分布泛化（Out-of-distribution generalization）是极其困难的。当前 AI 研发虽然在内环指标（各种基准测试与跑分）上进展神速，但这些指标能否顺利外推并泛化成终端用户真正买账的产品价值与颠覆性科研能力，依然悬而未决。
> “在 OpenAI 早期，我曾有一种直觉：仅仅靠最小化对数损失（log loss）是无法通向智能的，因为真正关键的信息在整体损失中所占的份额太小了，它们会被无尽的噪声所淹没……但事实证明，它就是行得通。” —— John Schulman
### `10:00` 泛化能力的极限与“百年思考”思想实验
讨论开篇聚焦于大语言模型令人意外的泛化特性。有些泛化是我们“没有理由在先验上就预料到的”：比如仅仅在一个极其朴素的下一个 token 预测目标（next token prediction）上进行预训练，模型却展现出了对输入的深层理解，甚至掌握了在语料中并未显著表征的复杂概念；同样，从可验证任务（verifiable tasks）向缺乏明确验证标准的任务（less verifiable tasks）迁移，也是一种在理论上完全缺乏先验保证的泛化。
```mermaid
flowchart TD
accTitle: AI实验计算分配思想实验
accDescr: 展示在开展单次高成本真实实验前后，分配大量算力给AI研究员进行深层理论推导与小规模消融实验的理想闭环
subgraph PreExp["实验前：百年理论推演"]
A["分配与真实实验等额的算力给 AI 劳动力"]
B["小规模消融实验（Small-scale Ablations）"]
C["构建严密理论框架（百年沉淀级别）"]
A --> B --> C
end
subgraph Execution["七位数成本大实验"]
D["执行最优配置的大规模实验"]
C --> D
end
subgraph PostExp["实验后：全面复盘与分析"]
E["海量算力深入分析实验数据"]
F["推导普适规律，规划下一阶段实验"]
D --> E --> F
end
```
针对这一现象，Dwarkesh 提出一个用于探究技术奇点（Singularity）的思想实验：即使不扩大除 AI 劳动力之外的其他实体资源投入，我们是否也能极速迎来飞跃？假设在执行一次耗资高达七位数（百万美元级）的昂贵实验之前，团队在“AI 劳动力”上花费同等规模的算力——让相当于自动化版本的顶尖研究员团队，在数字世界里耗费相当于人类“整整一个世纪的时间”去思考：到底什么才是最理想的实验？设计极具针对性的小规模消融实验，重构自深度学习诞生之前的深厚理论，完成无懈可击的前期准备。而在实验结束后，再由 AI 进行另一个“世纪”的高强度分析与理论提炼。John Schulman 对此表示赞同，认为我们目前距离科研效率的天花板还差得很远；如果思考足够深入，人们完全可以通过极其精巧的小规模实验构建起能够泛化至超大维度的理论，未来的 AI 势必会耗费与实际实验相当的算力去专门负责分析和理论建构。
---
### `11:42` 明确目标下的百倍加速与莫拉维克悖论
Charlie 补充道，在“目标已被严格形式化”（objective is well specified）的场景下，纯粹的思考与推理确实能带来颠覆性的提速。虽然仅仅依靠思考无法获取全新的信息位（无法脱离证据凭空获得超越先验的 bits），但它能以惊人的效率根据已知数据更新后验分布。
他举了一个极具说服力的历史实例：如果当年能让 AI 深入审视 Kaplan 提出的缩放定律（Kaplan Scaling Laws），AI 很可能会在第一时间敏锐地指出——研究人员只是截取了训练过程中的中间检查点（intermediate checkpoints），却完全没有考虑到学习率退火（annealing）对收敛的致命影响。仅凭这样一个对数据细节的逻辑洞察，就能直接为全人类的 AI 研发进程节省整整一到两年的试错时间。类似的例子还包括 Maximal Update Parametrization（$\mu$P）、学习率如何随模型尺寸与宽度（model width）进行协同缩放等。Charlie 估计，如果在当前既定目标下把低垂的果实全部摘净，AI 的深度思考足以带来 10 倍乃至更高的推进速度。
然而瓶颈在于：如果目标本身未被定义，纯粹的推演根本无法替人类决定什么是“正确的目标”。Beren Millidge 将此视为递归自我改进（Recursive Self-Improvement, RSI）最核心的隐患：
- 一个完全闭环、自我驱动的自动化科研循环，要求 AI 能够自主提出新目标、优化新目标、总结认知，再提出下一个目标，且保证在极长时间尺度上绝不“脱轨”（go off the rails）。
- 这极有可能遭遇新型的**莫拉维克悖论（Moravec's Paradox）**：人类由于具备长期的自然演化本能，自给自足、自主寻找目标并长期存活看似毫不费力，这种高度的自封装自主性对我们宛如呼吸般自然；但对 AI 而言，这种自主确立目标的能力可能如同人类眼中的机械运动控制（locomotion）一样极其艰深，而看似高深的高等数学对模型来说却轻而易举。
---
### `14:48` 研发闭环中人类不可替代的最后阵地：对齐与目标定义
当 Dwarkesh 提问从 2012 年深度学习复兴至今，哪一部分创新是人类在研发被彻底自动化前“最后的保留领地”时，Charlie 认为是对关键问题的敏锐洞察能力。目前的 AI 在编写实验代码方面表现出色，但一旦让其自主构想研究方向，提出的往往是一堆细枝末节、无关痛痒的琐碎尝试。从 DeepMind 早期执着于“通过打通各类复杂游戏走向通用智能”，到 Alec Radford 孤注一掷选择在海量宽泛数据上“仅靠预测下一个 token”，再到后续确立可量化的缩放定律，这些根本性的研究范式转向均需要极其宏观且非共识的直觉。
John Schulman 明确提出，**人类坚守到最后的角色将是“定义目标”（defining the objective）**——也就是决定人类究竟想要什么：
- 无论是在后训练（post-training）阶段利用人类反馈强化学习（RLHF）去定义一个助手应该展现何种行为特征与何谓“有益（helpful）”，还是后续制定模型宪法（Constitutions）与行为规范文档（Model Specs），这些都超越了纯粹的技术执行层面。
- 即便 AI 接管了所有工程与调优细节，人类仍必须在目标设定上牢牢把舵。
- “对齐即是最终工作（Alignment is the final job）。”Schulman 进一步将对齐拆解为两部分：一是“明确形式化正确的目标”，二是“达成并优化该目标”。后者可以被算法接管，但前者绝不会消失。后训练团队之所以需要维持庞大的组织规模，核心原因就在于真实世界的交互场景无穷无尽，永远需要人类去细致界定模型在各个边缘角落里究竟该如何行事。
---
### `16:32` 模型验证瓶颈与抵御中心化的蒸馏力量
节目中穿插了 Jane Street 采用 Antithesis 平台的案例：Jane Street 的技术团队联席主管 Ron Minsky 指出，随着 AI Agent 生成的代码比例极速飙升，软件工程的阻碍已经从“编写代码”彻底演变为**“验证瓶颈（Verification Bottleneck）”**——即人类难以消化海量代码并确认其能否安全合并至生产系统。自动化测试工具在极其严密的系统中排查出隐蔽深层 Bug 的能力，成为了释放 Agent 生产力的关键催化剂。
随后，对话切入一个关键的商业与生态议题：既然前沿大模型的训练极度依赖资本与算力堆叠，为什么当前市场上没有出现绝对的寡头大一统？是什么力量在对抗垄断集中化？
```mermaid
flowchart LR
accTitle: 模型蒸馏与去中心化抗力
accDescr: 展示强化学习探索得到的高价值轨迹如何通过少量数据极速蒸馏至小模型，从而瓦解垄断壁垒
subgraph FrontierLab["前沿实验室 / 寡头"]
A["高昂算力成本: RL 试错探索"]
B["提炼出成功动作与推理轨迹（Trajectories）"]
A --> B
end
subgraph OpenEcosystem["开源生态 / 垂直企业"]
C["极小数据量蒸馏（Distillation）"]
D["以极低成本复现前沿模型能力"]
B -.->|"低比特信息泄漏"| C
C --> D
end
```
John Schulman 直言，**模型蒸馏（Distillation）是目前对抗中心化最核心的反作用力**：
- 通过强化学习（RL）摸索出来的高级行为，本质上包含的信息位（bits）并不庞大，它们往往只需要极其少量的高质量数据就能被完整刻画。
- 只要一个模型对外展示了它的推理轨迹（trajectories）或高水准行为样本，外部竞争对手就可以通过极小成本的蒸馏将其复现。
- 此外，未来企业更倾向于拥有与自身业务深度绑定的私有定制模型，通过实际业务部署实现持续微调和改进。这种持续学习（Continual Learning）机制并不依赖单一的模型供应商巨头，虽然寡头也可以提供此类企业服务，但蒸馏与专有数据的存在从根本上打破了算力堆叠带来的绝对技术壁垒。
### `20:00` 提示词分布的决定性：蒸馏模型的核心门槛
对大模型进行蒸馏（Distillation）时，即使拥有完整的思维链（Chain of Thought, CoT）和白盒访问权限，仅靠监督学习（SFT）复制模型行为依然极具挑战性。其核心障碍在于**提示词分布（Prompt Distribution）**：要蒸馏出所有有用的能力，必须用涵盖广泛且极具真实感的提示词去促使前沿模型输出。
近期行业动态显示，部分中国 AI 实验室之所以能够迅速跟进，很大程度上得益于**中转代理服务（Router/Proxy Services）**。由于许多海外前沿模型无法在中国直接访问，国内开发者广泛使用这类中转服务调用前沿 API（主要用于编程任务）。中转服务商将这些真实的用户交互数据收集并售卖，为蒸馏训练提供了极其宝贵的真实提示词分布。
在前沿的后训练（Post-training）数据管线中，流程往往是混合式的：人类工程师提供高质量的种子提示词（Seed Prompts），或者从真实中转流量中提取种子，再利用现有模型或前沿模型在此基础上合成海量的变体与覆盖范围。随着模型能力的增强，人类所需要提供的“信息量（Bits）”正在急剧减少。
```mermaid
flowchart LR
accTitle: Prompt Distillation Pipeline
accDescr: A flow diagram showing how router proxy data and human seeds are used to synthesize prompts for model distillation.
subgraph DataCollection["数据源收集"]
A["中国开发者中转调用<br/>(Router Services)"]
B["人类工程师编写<br/>(Seed Prompts)"]
end
subgraph Synthesis["合成与扩展"]
C["前沿大模型<br/>(生成 CoT 与变体)"]
end
subgraph Training["模型蒸馏"]
D["学生模型训练<br/>(对齐真实分布)"]
end
A --> C
B --> C
C --> D
```
- **真实交互轨迹的价值**：用户在开发应用时的真实试错流程——例如“帮我做这个应用”、“报错了，改加这个功能”、“停一下，退回重做另一个方案”——构成了复杂智能所必需的上下文轨迹，纯合成数据极难捕捉到这种微妙的动态。
- **自动闭环与 RSI**：若模型能够全自动构思用户意图、自主合成数据并完成自监督训练，这本质上已经达成了递归自我改进（Recursive Self-Improvement, RSI）。但在目前阶段，捕获真实世界的人类意图信息依然不可或缺。
---
### `22:15` 蒸馏者与开拓者的不对称优势：为什么前沿模型反而显出疲态？
在某些能力域上，蒸馏者反而比从零开拓的前沿实验室更容易出成果。以“扮演一位优秀政客并在参议院辩论”为例：
- **开拓者（Frontier Labs）**：必须四处搜集真实政客的日常运作数据，理解复杂的政治斡旋流程，才能首次把这种能力诱导出来；
- **蒸馏者（Distillers）**：只需直接向前沿模型提问“扮演一个政客”，利用其已具备的常识与能力直接生成数十亿条变体轨迹，再对小模型进行微调。
对谈中提出了一个尖锐的观察：在实际体验中，Anthropic 的新旗舰模型（如 Claude 3.5 Sonnet / Opus 5 等设想代次）有时表现得还不如 GLM 5.3 或 Kimi k3 等开源/蒸馏模型犀利。如果前沿实验室拥有最好的强化学习环境（RL Environments），又掌握着逻辑蒸馏能力，为什么做出来的模型反而出现倒退或令人失望？
讨论者分析指出，这可能是因为陷入了模型进化的“恐怖谷（Uncanny Valley）”：
- **过度对齐与判断器过载**：例如某些模型在每个细微步骤中都引入了强烈的“AI 裁判（AI-as-a-Judge）”自检机制，导致其思维链极度冗长，消耗大量 Token，却丢失了大模型的直觉（“Big model smell”），不知道何时该适可而止或选择最优路径。
- **真实部署反馈 vs RL 环境**：单纯依靠前沿实验室内部设计的硬性强化学习环境，并不等同于在真实世界部署中直接吸收海量真实用户的复杂交互。
---
### `25:00` 任务难度 vs 任务真实感：后训练与基准测试的陷阱
在构建训练环境和奖励函数时，存在两个截然不同的维度：
| 维度 | 特征与构建难度 | 代表场景与局限 |
|:--- |:--- |:--- |
| **难度轴（Difficulty Axis）** | 相对容易大量合成；通常是复杂的解谜题或需要极高技巧的硬核任务，结果易于用规则自动校验（Easy to verify）。 | 常见于各种公开基准评测（Benchmarks）。模型可能在此类基准上刷出高分，但实际应用中极其僵硬。 |
| **真实感轴（Realism Axis）** | 极难低成本构建；涉及人类工程师在多轮交互中的模糊需求、反复迭代与代码库协作，需要人类细致的评价标准（Rubrics）指导奖励。 | 真实的编程代理（Coding Agent）场景。直接决定了终端用户的实际使用体验。 |
如果团队在做后训练或蒸馏时只盯着易验证的高难度题目，模型就会退化为“应试机器”。体量极大的顶尖前沿模型之所以强大，在于它们具备从狭窄难题泛化到复杂现实任务的泛化能力；而如果小模型（如轻量级 Sonnet）仅仅在合成基准上进行蒸馏，其在开放性真实交互中的泛化性能就会迅速瓦解。
此外，后训练极其脆弱，稍有不慎调高了某个奖励因子的权重，就可能导致模型产生令用户生厌的怪癖行为，而这些崩坏在自动化基准测试中往往完全暴露不出来。再加上中国实验室同样可以在全球数据市场上采购到与硅谷顶尖实验室相同的数据集，配合强大的蒸馏能力，追赶速度自然超乎外界预期。
---
### `28:06` 自动化 AI 研发（AI R&D）模型将如何被训练？
如何训练出第一代能够完全接管自身研发流程的自动化 AI 研究员？行业内存在一种玩具式的假想方案：直接让未来的超级模型（例如 GPT-8）去尝试搭建 GPT-3 级别的小模型，在消耗最少算力的约束下攻克持续学习或通关复杂电子游戏，从而筛选出最优训练算法。
然而，John Schulman 提出了更为务实、贴近工程落地的训练路径：
- **人类反馈吸收品味（Taste）**：AI 研究员的许多核心判断来自于顶级科研人员的“品味”和科研直觉，这些直觉很难写成规则环境，必须通过高阶人类反馈（RLHF）来逐步吸收。
- **多阶段复杂研发实战环境**：构建大量模拟真实科研的多步骤项目环境，让模型在其中自主推演、查阅文献、编写训练管线并分析实验结果。
- **循环修补弱点**：现实中的迭代绝非一次性自我博弈，而是研究员在使用辅助 AI 的过程中持续发现缺陷，针对特定弱点定向采集反馈或专门构建针对性的训练环境进行修补。
这种思路与将强化学习历史“完全回滚”让模型从头探索有本质区别：当前实验室的做法并非给 AI 一块裸 GPU 就让它从感知机时代重新发明深度学习，而是直接推到技术前沿，将团队最近几个月在训练堆栈中踩过的真实 Bug 转化为对抗训练环境，倒逼模型在最尖端的未知边界上快速成长。
### `30:00` 递归自我改进：算法蒸馏还是环境超越？
讨论深入探讨了在 AI 实验室内部如何构建训练环境。一种设想是回溯到特定算法突破（如 GRPO）出现之前的状态，构建专门的环境让模型自主发现更优的强化学习算法范式，甚至进一步向历史纵深回滚。然而现实受限于算力瓶颈，前沿实验室的做法更偏向于“前向差分”：紧跟最新模型，提取过去几个月里发现的 Bug 与算法改进，并将这些前沿进展转化为下一代模型的强化学习环境。这实际上构成了实验室内部的一种持续学习（Continual Learning）机制，即把人类研究人员在过去一个季度内取得的进展，通过强化学习与 RLHF 蒸馏回模型权重中。
```mermaid
flowchart LR
accTitle: 实验室内部研究蒸馏与环境超越
accDescr: 对比传统将人类研究成果蒸馏给模型，与通过极高环境目标实现超人表现的两种路径。
subgraph S1["渐进蒸馏循环"]
A["前沿 Bug / 改进"] --> B["构建强化学习环境"]
B --> C["蒸馏回模型权重"]
end
subgraph S2["环境超越路径"]
D["设定极高目标 (如 Loss 1.3)"] --> E["AI 探索超越人类解法"]
end
C -. "逼近人类极限".-> B
```
有观点认为，如果只是针对人类专家的行为轨迹进行蒸馏，模型能力将不可避免地趋近于渐近线，只能不断贴近人类研究员的水平。但构建“环境”则完全不同：设计一个人类根本无法解决的高难度任务非常简单，只要环境目标明确可验证，AI 就能在其中尝试突破人类上限。
- 目标设定示例：在 AI 研究中定义一个极度明确、可验证的指标，例如要求模型将损失函数（Loss）降至人类当前完全无法企及的 1.3。
- 参照基准变迁：几年前人们还觉得让一个 1 亿参数（100M）的小模型通关《我的世界》（Minecraft）是不可思议的成就，而如今在讨论中这种任务已被视作“过于简单”。
### `32:00` 直觉驱动的真实研究与可验证任务的脱节
尽管量化指标便于强化学习优化，但真正的科研工作往往不是在预设目标上进行简单的“爬山算法”（Hill Climbing）。研究通常始于直觉——研究人员感到模型在某个维度有改进空间，并设计出初步算法，然后构建一个玩具任务来检验该思路是否展现出“生命迹象”（signs of life）。一旦初见成效，再逐步提高任务的真实度，让方法在迭代中走向成熟。这是一种放宽真实性约束、先验证核心直觉的研究策略。
与此同时，大量机器学习研究致力于解释现象并形成非形式化理论。未来的模型训练很可能是将高度可验证的硬指标任务与模糊、需要理论直觉的开放性任务结合在一起。但这种训练是否能让模型具备足够强的泛化能力，使科研闭环完全脱离人类干预走向“自我封闭”（Self-sealing），依然存在巨大疑问。
- 研究范式的冲突：直接优化最终生产目标的硬指标，与“放宽现实约束、验证直觉再行落地”的科研直觉探索之间存在明显鸿沟。
- 理论构建的独特性：机器学习领域往往缺乏强预测性的严格数学理论，更多依赖研究者对系统内在机制的直觉性解释。
### `33:51` 百万模拟环境与数字中心里的“即插即用远程员工”
当前头部实验室正在进行一场巨大的豪赌：在数百个领域构建数以百万计的多样化强化学习环境（RLVR），押注通过这种长上下文、多智能体协同的大规模训练，能涌现出一个具备强持久性、上下文信息分类筛选能力的高级 Agent。这个 Agent 将在数据中心的模拟环境（Simulacra）中学会“如何学习”，并能直接作为即插即用的远程员工，胜任现实世界中长达一周乃至一个月的复杂工作。
```mermaid
flowchart TD
accTitle: 模拟环境泛化与领域覆盖路线
accDescr: 展示模型从代码、金融向长尾任务扩展的路径，以及直接内化与部署学习的分歧。
subgraph DataCenter["数据中心模拟训练"]
E1["代码生成 (最易获取的低垂果实)"] --> E2["金融与 Excel 数据"]
E2 --> E3["PPT 与办公长尾场景"]
end
subgraph RealWorld["现实落地路径"]
E3 --> G1["直接内化权重 (运行时极高效率)"]
E3 --> G2["上下文现场学习 (在职部署学习)"]
end
```
这种策略引发了对模型演进逻辑的深层探讨：如果模型的终极目标是像人类一样具备在岗位上实时学习的能力，实验室为什么还要大费周章地把 PowerPoint 或金融业务的细节强行训练进模型？
- 观点一（效率摊销）：即使模型未来具备极强的上下文即时学习能力，为了节约运行时代价，提前通过强化学习将高价值领域的直觉固化（Amortize）到模型权重中仍然是经济合理的。
- 观点二（商业造血支持 RSI）：实验室需要通过覆盖商业高价值场景来获取收入，以此资助下一轮超大规模训练；同时，金融等领域的复杂长程推理数据具有跨领域的元学习迁移（Meta-learning Transfer）价值，能反哺自动化科研能力的构建。
### `36:00` Sim-to-Real 瓶颈与交互学习的未来
当前通过在数据中心构建模拟环境进行强化学习（Sim-to-Real）的范式固然取得了显著成果，但其天花板也极为明显：现实世界中的大量任务本质上需要与真实人类进行高频、实时的复杂互动，这类场景极难在数据中心内高度逼真地模拟出来。
Sim-to-Real 成为当下的主导范式，根本原因在于当前强化学习的样本效率（Sample Efficiency）极低。训练一个有效策略往往需要数万次乃至更多的交互迭代，现实中没有任何人类愿意充当交互回路里的工具人。因此，业内不得不依赖数据中心内的模拟环境来堆叠样本。
随着部署规模的扩大，未来的技术路径或将发生结构性转向：
- 部署端离线学习（Off-policy Learning）：直接从海量的实际部署轨迹中提取数据，无需重新搭建仿真环境，直接进行离线优化。
- 真实互动闭环：当样本效率大幅提升后，利用真实世界的部署反馈将逐步取代纯模拟训练，成为模型突破能力瓶颈的关键。
### `40:00` Jane Street 协议模拟器 ASIC 竞赛与蜂群思维猜想
赞助环节介绍了 Jane Street 推出的一项雄心勃勃的硬件设计竞赛：设计一款协议模拟器 ASIC。参赛者设计的芯片需能连接被测芯片并模拟出逼真的流量，从而在脱离真实系统的环境下进行测试。竞赛看重的是具有高度通用性与可重新编程（reprogrammable）的设计，要求不仅仅能支持 I2C，更鼓励设计能够灵活应对低速 USB、以太网（Ethernet）乃至未来新出现的各种协议，硬性约束则是在 130 纳米制程节点上流片（Jane Street 将资助最具创新性的作品流片并将实体芯片寄送给胜者）。
随后，Dwarkesh Patel 将话题切回到 AI 核心逻辑：当前行业中有超过 50% 的算力被消耗在推理（Inference）端，然而这部分庞大的计算量却没有直接转化用来让模型本身变得更聪明。理论上，人类终其一生仅有大约 50 年的现实世界经验，而数字心智通过其海量的并发部署实例，能够在一个很短的时间内累积等同于数百万年、横跨各种具备重大经济价值的现实工作经验。但现状是，这庞大的真实交互数据并未在深层次上驱动模型自我迭代。Dwarkesh 质疑道，一旦模型能够完全吸收这些数据，几乎必然会引发某种广泛部署的“智能爆炸”与“蜂群思维”（Hive mind），那么这一奇点何时才会发生？
Beren Millidge 则指出，从更广的视角来看，这种机制其实“已经在发生”：
- 哪怕是最基础的形式，各大实验室也会将前代模型在部署中沉淀的数据进行过滤、打分、判断与重构，随后灌入下一代模型的预训练（Pre-training）或中训练（Mid-training）流水线中。
- 在代际能力提升中，这一类部署数据的吸收贡献了相当大一部分增量。
- 尽管西方顶级实验室在理论上受限于隐私条款声称不随意拿用户数据做训练，但中国模型团队却毫无包袱地全盘吸纳这一做法。无论是直接蒸馏（Distillation）外部模型、捕捉调用回传数据，还是回收自身模型的线上运行记录，都让其在模型迭代中获得了巨大的实际优势。
```mermaid
flowchart TD
accTitle: 部署数据回流模型训练的代际演进循环
accDescr: 展现从线上交互到数据过滤合成，再到下一代预训练与在线强化的闭环机制。
subgraph DEPLOY["线上部署与交互"]
A["多实例推理部署<br/>（海量现实经济任务）"] --> B["用户反馈与行为数据<br/>（点击、采纳、报错）"]
end
subgraph FILTER["数据处理与微调"]
B --> C["质量过滤 / 价值判断 / 数据重构"]
C --> D["垂类环境构建<br/>（如 Cursor、Harvey）"]
end
subgraph RETRAIN["模型权重大更新"]
D --> E["下一代模型预训练 / 中训练"]
D --> F["线上强化学习（Online RL / REINFORCE）"]
E & F --> A
end
```
---
### `43:00` 真实反馈循环与 Cursor、Harvey 的线上强化学习实践
Charlie O'Neill 深入指出，虽然人们脑海中终极的“持续学习”（Continual Learning）圣杯是一个极其自然、实时的闭环——模型单兵作战获得一次经验后便能立刻现场更新权重，但一旦深入到实现细节，许多机制都会崩解。不过，闭源大厂与开源生态其实都在以极快的节奏推动类似循环。最明显的例子当属编程助手 Cursor（以及利用类似逻辑处理法律智能体的 Harvey）：
- 它们能够利用垂直领域积攒的用户真实报错、修改习惯和特定部署数据，去构建高度定制的强化学习环境。
- 相比通用大厂，这类垂直应用公司在如何深度压榨并高效利用这部分用户交互数据上占据了巨大优势。
- 例如在 Kimi K3 的后训练阶段之后，或者在 Cursor 早期的 Composer 功能中，团队都在尝试推行某种长期的在线 REINFORCE 机制。
Charlie 强调，虽然这个流程中依然少不了人类在回路（Human in the loop）中挑选信号、决定如何根据数据搭建环境，其更新周期也比想象中的实时自我进化要长，但这种节奏正在不断加快。Dwarkesh 补充说，Cursor 最初只是根据用户在代码补全时按或不按 `Tab` 键来每日优化预测概率；但 Charlie 纠正道，Cursor 不仅对 Tab 模型这样做，更将这套在线强化学习应用到了实际的生成式大模型上。
在线强化学习的最大难题在于单样本无基线（No groups）：当单个用户输入一个提示词时，系统只能采纳这单次展开（Rollout），进而面临严重的方差消解（Variance reduction）挑战。对此，Cursor 采用了一套工程化的粗暴解决方案：
- 构建高精度启发式算法（Heuristics），用来估算某次生成的代码响应是显著优于平均水平还是劣于平均水平。
- 随后执行一次大规模的 REINFORCE 权重更新。
- 为了监控模型是否被改坏，系统内部设置了 CursorBench 评测集：新权重每 5 小时更新一次并跑测试，如果评测跑分提升，就将新模型直接上线部署；一旦表现变差，则直接丢弃该版本。
---
### `45:24` 仿真到现实的差距（Sim-to-Real）与样本效率瓶颈
针对这种基于点击或采纳的反馈，John Schulman 提出了冷峻的技术警示：依赖自然交互数据做强化学习，最核心的隐患在于“你根本不知道真实世界数据的奖励函数（Reward function）究竟应该是什么”。倘若退而求其次使用诸如“用户是否接受了这次代码编辑”这类表面信号，极度容易诱发严重的“奖励黑客”（Reward hacking，即模型为了迎合接受率而做出浅薄妥协）。
这随之引发了关于“仿真到现实”（Sim-to-Real Gap）的更深层担忧：
- 任务的行动跨度（Horizon）越长，它在数据中心内部被沙盒化仿真的难度就呈指数级上升。
- 以编程为例，当前已经很难找到一个需要跨越一年周期的大型工程开发任务，是能够完全脱离与客户交流、组织协作以及真实用户反馈而单靠数据中心自闭环完成的。
- 假定未来期望超级智能去完成全套商业运作——例如创办一家新企业并使其盈利、在金融市场进行日内高频套利、或是打赢一场真实的法律官司——这些本质上都需要与现实世界的无序性深度交互，无法被完全推入模拟器。
Dwarkesh 进而抛出一个尖锐的矛盾：如果模型无法仅仅依靠从仿真向现实的泛化迁移（Transfer）来搞定这些极端复杂的长周期任务，而是必须依靠与现实世界交互产生直接的“权重更新”（Weight updates），那么模型当前极为低劣的样本效率（Sample inefficiency）就会变成致命内伤。
- 人类从婴儿期成长为具备独立思考与复杂判断能力的成年人，所吸收的数据量级微乎其微。
- 相比之下，大模型从冷启动（Cold start）到完成训练，所吞噬的 Token 规模动辄是人类一生感知输入的数百万倍。
- 假如真实环境中的权重更新不可避免，这种高达百万倍的样本效率差距是否会彻底扼杀在未来十年内发生递归自我改进（Recursive Self-Improvement, RSI）的希望？
---
### `48:00` 累积型任务与非平稳分布：RSI 为何可能反比现实世界容易？
Charlie O'Neill 提出了一个核心分类框架来解答上述困境：衡量一个领域模型是否能攻克，关键在于该任务属于“累积型任务”（Cumulative task）还是属于深陷“非平稳分布”（Non-stationary distribution）的现实缠斗：
| 任务类型 | 核心特征 | 典型代表 | 应对方式与模型适应度 |
|:--- |:--- |:--- |:--- |
| **累积型任务** (Cumulative Task) | 知识与进展具备永久累加性，已验证的成果无需反复重新推翻验证；可收敛为确定性代码流程。 | **递归自我改进 (RSI)**、底层算法工程迭代 | 只需保留 Attention、MoE 或 GRPO 等成功范式直接拼装进训练栈；模型调用 `pre-training.sh` 与 `post-training.sh` 脚本即可推进，具备极高的自动化闭环潜力。 |
| **非平稳分布任务** (Non-stationary Task) | 规则、人际关系与环境时刻处于动态流变中，充斥隐性知识与社会政治博弈。 | **现实律所律师协作**、企业日常经营、复杂商业谈判 | 组织架构频繁变动，做事规则口口相传且无固定文档，无法提炼为干净的累加逻辑，对持续学习与样本效率提出苛刻挑战。 |
Charlie 指出，一个极其具有讽刺意味但可能成真的事实是：**递归自我改进（RSI）本身可能反倒比模拟现实世界更加简单**。理论上，我们甚至可能只需写一个不足 100 万 Token 的纯 Python 文件，就能从零冷启动训练出一个能够进行自我迭代的模型。模型并不需要一次次去重新发明 Attention，而是只要把前人摸索出来的 MoE 或 GRPO 直接挂载进工具链脚本，层层叠加即可。这就解释了为什么各大实验室可能会把大部分算力与精力收缩并聚焦在 RSI 上，而非死磕现实世界的复刻。
对此，John Schulman 并不完全将模型弱于人类的原因归咎于样本效率。他分析道，当前模型在很多特定体系中其实具备极强的样本效率——比如上下文学习（In-context learning）阶段对提示信息的即时吸收能力就极强；但在中等时间跨度（Medium-length regime）的场景下，由于人类可以在大脑神经元连接中以更轻量、更高维的方式更新权重，模型的机械权重更新确实显得效率低下。然而更重要的是，模型的短板不仅限于此，更来自于人类拥有而模型极度匮乏的深层特质：
- **思维多样性（Diversity of thought）极其低下**：模型在经过强化学习后往往迅速陷入同质化的最优路径。
- **长跨度判断力（Long-horizon judgment）极度欠缺**：尤其是在缺乏明确反馈的情况下，去长期权衡何为真正的技术方向与战略路径，这正是人类常说的“品味”（Taste）的核心所在。
### `50:00` 研究品味（Taste）的本质与万亿 Token 上下文思想实验
讨论首先聚焦于什么是“品味”（Taste）。嘉宾们认为，在很大程度上，品味其实就是那些从长远来看能够奏效的行为模式，并且这种有效性已经被人们逐渐意识到。特别是在软件工程领域，所谓的品味往往体现为能够判断“什么样的系统架构在长期维护中更具鲁棒性、在项目的整个生命周期中能运转得更好”。人类在实现递归自我提升（RSI）时所面临的局限不仅在于样本效率，还包括许多其他生理和认知限制。
```mermaid
flowchart TD
accTitle: Taste Acquisition Comparison
accDescr: Comparison between human taste acquisition through short horizons and AI context window scaling.
subgraph Human["人类科研品味培养"]
H1["5年博士/博后生涯"] --> H2["完成 10 到 50 个小型研究课题"]
H2 --> H3["从短程 episode 中快速提炼科研直觉与品味"]
end
subgraph AI["AI 上下文扩展思想实验"]
A1["超长 Context (如 1 万亿 Token)"] --> A2["填入一生完整科研轨迹数据"]
A2 --> A3{"能否泛化至超长 Horizon 决策？<br/>(目前仍未解决的开放性问题)"}
end
```
针对品味能否被纯计算方式复制，现场展开了一个脑洞大开的思想实验：假设给模型提供一个长达 1 万亿（1 trillion）Token 的上下文窗口，把一个人在接触 RLHF 之前的所有生平经验全部塞进去，同时保持模型具有百万 Token 级别相同的样本效率与上下文学习（In-context Learning）能力，那么“品味”这个问题算被解决了吗？它能做出与顶级研究员相同的价值判断吗？讨论认为，这仍然取决于模型是否被训练成懂得“如何从如此庞大的上下文中提炼出正确的更新”；而且，即便理论上有万亿 Token 的窗口，也需要相应长度的训练数据去支撑它，而人类根本无需经历长达数万年的超长生命周期，就能在短短五年博士到博士后的阶段里，通过仅仅 10 到 50 个研究项目迅速建立起敏锐的品味。AI 无疑将拥有远多于人类的经验去元学习（Meta-learn）这种品味，但这种从短程学习中提炼的能力究竟能否泛化到极其漫长的长程（Long-horizon）任务中，目前在学术界仍是一个完全悬而未决的问题。
- 软件工程中的“品味”本质上是对长期系统可维护性与鲁棒性的预判与直觉。
- 博士生成长为成熟研究者通常只需 5 年左右，经历 10 至 50 个具体研究项目即可形成敏锐品味，表明品味并非必须依赖超长周期的单一 Episode。
- 模型即使具备超大容量的上下文窗口，若缺乏相应的长程上下文优化训练与元学习算法，也无法自动习得人类层面的研究判断力。
---
### `54:12` 蜂群思维式实时部署学习：商业博弈与模块化方案
随后，主持人提出了一个关键演进方向：未来模型是否会进入一个“蜂群思维”（Hive mind）的学习范式，即模型在成千上万个部署实例中与真实世界交互，并从中迅速吸收养分实现全局进化？目前这种从部署到模型迭代的反馈回路依然极其模糊且微弱。对此，嘉宾指出阻碍蜂群学习的主要瓶颈其实不是纯技术问题，而是商业激励机制与企业隐私壁垒。企业客户绝不希望模型提供商（如各大模型实验室）把自己的专有业务交互数据全部吸收进通用的基座大模型中，因为这会直接抹平企业自身的商业竞争壁垒。
面对这种经济与商业现实，大家认为技术路径不会一蹴而就走向“所有部署实时更新同一个中心大模型的权重”，而是会向模块化适配方案妥协：
- **可插拔模块机制**：例如类似 LoRA 权重的外挂结构，或通过压缩 KV 缓存（KV-Cache Cartridges）与线性注意力（Linear Attention）技术，将海量上下文压缩进固定尺寸的模块中。
- **专有数据隔离**：企业愿意将这种外挂“卡带”接入基座模型，因为这既实现了针对特定业务场景的自适应学习，又避免了基座权重的直接变动与企业核心数据的泄露。
- **间接蒸馏闭环**：各大模型实验室虽然短期内无法直接实时微调中心基座模型，但仍可以通过收集脱敏交互轨迹（Traces），将其间接蒸馏并注入到下一代模型的预训练数据池中。
---
### `57:30` 连续自我更新的渐进式闭环演进
针对“模型何时能实现连续在线学习”的争论，嘉宾们一致反对那种“某一天突然发生突变、权重连续在线更新一揽子解决”的非连续飞跃假说。真实的工程演进更有可能是一个闭环迭代频率不断加速的连续谱系。
```mermaid
flowchart LR
accTitle: Continuous Improvement Loop Acceleration
accDescr: The iterative acceleration of model updates from quarterly releases to continuous updates.
A["季度发布周期<br/>(3个月一次权重迭代)"] --> B["周级微调闭环<br/>(周更专项部署轨迹)"]
B --> C["天级/小时级循环<br/>(高频蒸馏与自适应)"]
C --> D["准实时连续更新<br/>(自我改进循环闭合)"]
```
在演进初期，系统利用模块化卡带（Cartridges）在具体的部署环境中进行专业化定制；这些部署实例在运行过程中生成海量的优质交互轨迹，实验室将这些轨迹收集并整理，在三个月后发布一个综合能力更强的新基座模型；紧接着在新基座上再次挂载卡带进行专业化部署并沉淀数据。随着自动化数据清洗、评估基准与训练管线的成熟，这个原本耗时三个月的发布循环会被压缩至每周、随后缩短至每天、再到每小时一次更新。当迭代周期被压缩至以小时为单位时，工程上实际上就已经无限逼近了“连续权重在线更新”的最终状态。
### `55:00` 持续学习的微观困境与微调中的灾难性遗忘
讨论深入到了当前 AI 范式距离实现真正持续学习（Continual Learning）究竟有多远的技术细节。Charlie O'Neill 指出，在宏观层面、超大规模训练时，通过海量 Batch 冲淡噪声并将新数据注入中训（Mid-training）或自建环境中，系统确实能展现出某种持续学习的表象。然而，一旦把镜头拉近到微观操作层面——比如试图为某家特定律所持续微调一个模型，面对极少量的特定数据时，现有的所有方法都会开始崩溃。
如果仅仅针对成功轨迹进行有监督微调（SFT），无论是在线（On-policy）还是离线（Off-policy），在历经数百次微小迭代更新之后，灾难性遗忘便会显现。模型不仅会迅速遗忘此前基于基座模型学到的早期专业知识，其通用能力也会发生明显的退化与崩塌。尽管基于在线策略的蒸馏（On-policy distillation）能稍微将这一退化边界往外推延一些，但最终依然无法逃脱相同的命运。 Charlie 强调，强化学习（RL）固然非常擅长激发“推理能力”，但在灌注“事实性知识”（Explicit knowledge）方面表现欠佳——例如像“这家律所的某某人负责具体这项业务流程”这类高度特定而明确的事实，RL 很难直接注入，除非投入巨额算力去专门为这些知识构建定制化的模拟环境。
```mermaid
flowchart TD
accTitle: 持续学习微观更新的退化路径
accDescr: 展示了在微观微调场景下，SFT与RL面临的遗忘、能力衰退以及知识注入难题。
subgraph MicroUpdates["微观连续更新 (Micro-updates)"]
A["小规模特定领域数据<br/>(例如律所内部流程)"] --> B["频繁微步迭代微调<br/>(SFT / On-policy Distillation)"]
end
subgraph FailureModes["退化与局限"]
B --> C["灾难性遗忘<br/>(遗忘早期学习知识)"]
B --> D["通用能力严重退化<br/>(General Capabilities Drop)"]
E["强化学习 (RL) 尝试"] --> F["擅长激发通用技能"]
E --> G["极难高效灌注特定事实知识"]
end
```
- **微观更新瓶颈**：在百次量级的微调迭代中，现有算法无法在不破坏旧知识与通用基底的前提下安全吸收零散新信息。
- **知识 vs 能力的获取不对称**：RL 能够在极其狭窄的损失函数空间内微调模型行为以强化解题能力，但难以高效植入结构化的具象事实知识。
---
### `56:45` 模型容量还是训练技术：可塑性损失与从头预训练的妥协
Dwarkesh Patel 追问，导致这种遗忘和能力恶化的根源，究竟是模型“容量（Capacity）”的绝对物理极限，还是底层“技术（Techniques）”本身的缺陷。Beren Millidge 和 Charlie 均认为这本质上是一个技术问题，而非模型容量见顶。如果把所有这些新旧数据合并在一起，用来从头预训练一个参数规模几乎相同的全新模型，并在中训阶段充分混合，那么新模型的表现一定会比旧模型好得多。
这直接印证了问题核心在于模型的“神经可塑性（Plasticity）”衰减与“灾难性遗忘”。当面对持续产生的新数据时，单纯、幼稚地在非平稳数据分布（Non-stationary data distribution）上继续做梯度下降，会直接打破原有数据分布的平衡，从而导致旧知识被系统性冲刷。当前行业尚无有效手段阻止这一分布破坏，因而业内最通行的应对方式就是妥协：先从旧模型或环境中搜集产出数据，然后再花费昂贵的代价重新从头训练（Pre-train from scratch）一个全新的基座模型。
尽管完全从头训练非常昂贵，但现阶段工程师们正逐步推后“必须从头训练”的周期节点。如今已经可以做到在既有预训练基座上连续进行质量极高的 Mid-training，并在不同阶段的检查点（Checkpoints）上配合强化学习分支。然而正如 Beren 所言，同一基座模型不可能无限期地中训下去，基座学习新知识的能力最终会出现渐近线饱和（Asymptote）。如果持续硬塞新数据，参数空间的适应能力就会枯竭，这也是为什么各大前沿实验室最终仍然不得不周期性地拉起新基座进行重训。
- **非平稳数据分布挑战**：在线流式添加新数据会破坏原先精心校准的数据分布，导致旧表征崩溃。
- **重训妥协的经济学**：虽然从头训练基座耗资巨大，但在攻克神经可塑性损失与灾难性遗忘之前，周期性重训仍然是彻底融合海量新数据的唯一稳健途径。
---
### `59:15` 节目赞助插播：GrokBot 制作人功能
Dwarkesh Patel 在此插入了节目赞助内容，介绍了他们团队如何利用 xAI 的 GrokBot 来自动化播客剪辑制作流程。Dwarkesh 提到，每当自己录完一期长达数小时的访谈后，都会第一时间在 Slack 中毫无保留地进行“脑力倾倒”（Brain dump），列出这一期最精彩的看点以及哪些冗余片段需要被剪掉，以便剪辑师了解全貌。
然而这些即兴写下的随笔并没有清晰的时间戳，面对动辄数小时的原始视频，剪辑师需要耗费大量工时逐帧去对准他提到的具体时刻。为此他们接入了 GrokBot 虚拟制作人。现在只要剪辑师在 Slack 频道中上传粗剪版本，GrokBot 就会在自己的独立计算机环境中调取文稿，自动将 Dwarkesh 的文字笔记精准匹配到原始音视频的具体时间点上，并结合预存的剪辑偏好给出修改建议，将精选片段的候选直接发送到手机端供其审阅。
### `01:00:00` 数据分布与超智能：RL 环境梯度的构建瓶颈
在赞助商环节过后，讨论转向了核心命题：AI 的进步到底在多大程度上纯粹是由**数据进展**所驱动的？Dwarkesh Patel 抛出了一个假设性问题：是否存在某种理想的数据分布或训练环境，只要将现有的模型架构放在上面训练，就能诞生一个在人类所有专业领域全面处于绝对统治地位的超级智能（Superintelligence）？
嘉宾们认为，从理论上讲这种数据分布必然存在——在最微不足道的极端情况下，甚至可以只训练模型直接背诵输出那段能够训练出真正超级智能的 Python 代码。关键挑战在于我们能否构建出正确的环境梯度。大家指出，原则上完全可以设计一个由强化学习（RL）环境组成的“梯子”，顺着爬上去最终能得到与人类顶尖研究员匹敌甚至更强的 AI 科学家。但现实难题在于：**每往上爬一级梯子，构建对应环境的难度和工程量几乎呈指数级上升**。模型能力攀升的速度与人类构建环境的指数级难度之间，存在着激烈的边际效益权衡。
目前业界之所以能取得进展，是因为研究者们在大量利用现实中的“不对称性”（Asymmetries）来构建低成本、高收益的训练环境：
* **正向生成与反向推导的不对称性**：设计一个由隐藏潜在变量驱动的复杂数据生成过程非常容易，但模型必须消耗大量不可削减的 token（irreducible token spend）和计算功耗才能逆向推导并理解该生成过程。
* **现实世界的信息注入不对称性**：例如 Anthropic 可以从成千上万名人类工程师与语言模型的交互中筛查出一个棘手的 Bug，将其打包成极高密度的训练任务环境；模型在单次运行中只需消耗几百万个 token，就能在理论上提炼出全人类团队耗时无数才发现的知识。
然而，这种通过投机取巧挖掘“不对称环境”的红利正迅速面临边际递减效应。未来面对极其复杂的长视野任务，人类必须真正坐下来设计极其繁杂的测试用例和环境，再加上模型本身探索这些超长任务所消耗的巨额算力和时间瓶颈，性能爬升曲线必然会不可避免地放缓。
```mermaid
flowchart TD
accTitle: RL环境梯度构建与复杂性瓶颈
accDescr: 展示了从低阶任务爬升至高阶研究员AI时环境构建难度指数上升的过程
subgraph Ladder ["RL 阶梯式提升"]
A["初级环境: 基础算术/代码"] --> B["中级环境: 竞赛级数学/小型工程"]
B --> C["高阶环境: 自动化 AI 研究员"]
end
subgraph Bottleneck ["核心阻碍"]
D["逆向推导环境耗费巨量不可削减 Token"]
E["环境构建难度呈指数级增加 (人力/时间壁垒)"]
F["RL 探索能力极弱 (128 次采样无奖励即停滞)"]
end
C -.-> E
B -.-> D
Ladder -.-> F
```
---
### `01:03:00` 课程学习、RL 探索局限与现实世界信号的枯竭
在讨论数据与能力跃迁时，与会者探讨了模型泛化能力中的反直觉现象。有人提到一个极端案例：研究人员将一个完全没有见过代码、仅在日文数据上预训练的轻量级模型（如某些只接受基础日文训练的小模型），仅使用适量的代码微调数据进行后训练，其在 SWE-bench 编程基准上的表现居然一举超越了体量庞大得多的 Claude 3 Opus。这表明语言模型的表征具备惊人的跨领域迁移潜力。
但与此形成鲜明对比的是强化学习在缺乏梯度引导下的脆弱性。另一篇最新研究显示，如果模型仅被预训练至小学五年级数学以及小学英语水平，然后试图直接用 RL 将其能力拔高到高中乃至大学数学水平，训练过程就会彻底崩溃——跨度太大，模型根本爬不上去。只有遵循课程学习（Curriculum Learning），一步一个台阶地从七年级、八年级循序渐进地构建任务，模型才能最终掌握高三年级（Year 12）的数学技能。
这种差异直接暴露了当前强化学习的核心痛点——**RL 的自主探索能力极其匮乏**。在生成 128 条采样路径（rollouts）中，如果模型一次正确的奖励信号都没有撞上，它就永远无法获得更新梯度的信号。正因为如此，RL 极度依赖精心编排的人工课程设置，而预训练则完全不需要担心这种冷启动探索问题。
更深层的危机在于**现实世界增量信号的边际递减**：
* **低价值数据的充斥**：尽管人类社会每天都在产生大量的电子表格、法律合同等现实世界交互数据，但目前处于前沿水平的模型，在突破当前能力盆地（basin of capability）时真正需要的“有效信息比特”极其稀缺。
* **前沿突破信号的断层**：现实中并没有海量超越当前模型能力的新数学定理被证明出来，也没有源源不断涌现出极其深奥、超越模型理解极限的高维代码库。
* **预训练与后训练的数据本质不同**：在预训练中，信号早已隐藏在全网庞大的语料中，核心工作只是自动化地“滤除噪声”；但进入中训练（Mid-training）和后训练（Post-training）后，网络数据中根本不存在未被发现的“千禧年大奖难题（Millennium Prize Problems）的隐藏证明”，单纯过滤 Common Crawl 不会凭空创造出这些知识。未来的高质量信号必须依靠人类专家在部署环境中持续定义全新目标、构建全新环境来人为注入。
---
### `01:05:40` 算力效率来源之争：数据工程、算法架构与尺度的深层绑定
为了量化数据与架构对模型能力的实际贡献，Dwarkesh 透露了他与普林斯顿大学学生 Jerry Han 合作开展的一项网格实验：他们将 2019 年至今的所有训练配方（从 GPT-2 到最新的开源配方如 Delphi）与 2019 年至今的数据集（从 The Pile 到 Ultra Fine Web）进行成对交叉训练，全面评估达到同等能力水平时所需的算力节省幅度。
实验结果得出了一个极具冲击力的对比：
* **数据质量提升**解释了高达 **9 倍**的算力效率增益（Compute Efficiency Gain）；
* **模型架构改进**在非常小的模型尺度下仅带来了约 **3 倍**的算力效率增益。
* 两者相乘仅构成了约 27 倍的综合算力节省。
然而这一数据与 Epoch AI 等机构测算的宏观行业趋势产生了巨大矛盾——行业测算显示自 2019 年以来，每年的算力效率提升约为 3 倍，7 年时间里累计算力效率增益应达到 $3^7 \approx 2187$ 倍以上。那么，实验测得的 27 倍与宏观的 2000 倍之间，中间缺失的那上百倍效率提升究竟隐藏在哪里？
与会者展开了深入推论：
1. **极度依赖尺度的效率增益（Scale Dependence）**：小模型无法捕获真正大尺度下才出现的效率增益。
2. **后训练（Post-training）的贡献**：纯粹衡量预训练数据和架构无法反映后训练对综合能力上限的指数级放大。
3. **架构解锁新体制（Qualitative Regime Shift）而非单纯的乘法收益**：嘉宾反驳了将“架构”和“数据”简单切分为两个孤立乘法因子的做法。架构的核心价值不是带来微小的百分比损耗降低，而是解锁以前完全无法触及的物理体制。例如，如果没有长上下文架构创新，在 100 万 token 上下文进行运算的成本将是天文数字；正是架构突破解锁了这一全新空间，才使得在 100 万长上下文数据上进行训练成为可能。如果仅在 2k 上下文的小模型上测试架构改进，架构的价值就会被严重低估。
4. **后训练与中训练数据的大模型依赖性**：长视野的复杂任务轨迹（例如 SWE-bench 级任务）只有在大模型上才能转化为实质性的智能跃迁；若把这些复杂轨迹塞给一个 1 亿参数（100M）的小模型，它根本无法解析其中蕴含的高级规律，算力提升收益几乎为零。这证明了**数据效率本身也强依赖于模型自身的参数规模**。
### `01:10:00` 推理效率主导的架构变革与参数规模的未来
前沿模型的架构演进逻辑正在发生根本性转变。Charlie 指出，像 Kimi 或 DeepSeek 等团队所做的模型架构微调，出发点已不再仅仅是为了压低预训练阶段的损失（loss），而是深刻考量了模型在真实世界中的部署与推理效率。例如，DeepSeek 采用的某种形式的压缩注意力机制（如 MLA，多头潜在注意力），本质上并不是一种单纯提升预训练效果的“纯粹算法突破”，而完全是围绕降低推理成本和上下文缓存开销设计的工程优化。
随后 Dwarkesh 提出了一个核心疑问：随着大模型逐步迈入重度依赖强化学习（RL）的新阶段，参数规模的扩展轨迹将何去何从？回顾开源模型的发展历程，前沿模型的参数规模大致保持着每年翻倍（2x）的节奏；哪怕是闭源前沿模型，当前其活跃参数量（active parameters）通常也维持在 1000 亿到 2000 亿（100B–200B）的水平。但在强化学习范式下，采样生成（rollouts）需要消耗海量算力，推理效率变得无比关键；与此同时，模型能力是否会触及某种“门槛效应”，即一旦参数容量足够大，再随意堆砌参数边际效用就会递减？展望 2030 年，前沿模型的活跃参数量究竟会达到怎样的量级？
Charlie 认为，在未来几年内，由于全行业的研发重心都转移到了更长周期的强化学习长轨迹采样（long-horizon rollouts）上，推理效率具有决定性意义。当前模型的瓶颈主要在于训练环境的复杂度与完备性，而非模型本身的容量饱和，因此短期内参数规模增长可能会放缓甚至陷入平台期。他直言，传闻中的 Mythos 以及 GPT 模型的实际体量，很可能远小于外界热炒的 10 万亿（10T）参数。决定模型尺寸需要综合权衡：既取决于手头拥有多少高质量预训练数据，也取决于所能构建的强化学习环境的难度。理想的平衡点是：让模型在最难的环境任务上取得尚可的 Pass@1 成功率即可，超过这个阈值继续把模型做大毫无意义，因为那只会让你在后续的推理和采样过程中白白浪费数倍的 FLOPs。
```mermaid
flowchart TD
accTitle: RL 范式下模型容量与效率的权衡决策
accDescr: 展示高质量预训练数据、RL环境难度与推理成本如何共同决定最优活跃参数量
subgraph Inputs ["核心约束与输入变量"]
A["高质量预训练数据存量"]
B["RL 复杂验证环境难度"]
C["推理算力开销 (Rollout FLOPs)"]
end
subgraph Optimization ["架构权衡点"]
D["目标：最难任务上达到合格 Pass@1 门槛"]
E["权衡：活跃参数量 (Active Params) vs 稀疏专家 (MoE)"]
end
subgraph Output ["工程决策"]
F["避免盲目追求数万亿参数"]
G["压低活跃参数以最大化长轨迹采样吞吐"]
end
Inputs --> Optimization
Optimization --> Output
```
---
### `01:13:30` 稀疏性、数据墙与 Chinchilla 定律的重新审视
针对模型尺寸，John Schulman 提出了不同维度的视角。他预计模型整体规模仍会继续扩大，根本动力在于行业总算力规模的扩充以及单卡 GPU 显存容量的升级。然而，模型增长的幅度将以一种非显而易见的方式受制于缩放定律（Scaling Laws）。John 强调，在当前高质量预训练文本数据即将见底的背景下，“数据效率（data efficiency）”将超越“算力效率（compute efficiency）”，成为决定具体模型架构的关键驱动力。这直接关系到模型应该做多稀疏（sparse）：
- 业界对稀疏架构（MoE）的底层机理理解并不透彻；
- 总参数量（Total Parameters）与活跃参数量代表着完全不同的硬件资源消耗；
- 存在一种理论观点认为：稀疏性实际上会损害数据效率，因为不同的专家网络可能不得不反复学习相同的底层知识模式；
- 过去根据算力包络线（compute envelope）寻找最优模型架构的方法，在转向“以数据为 X 轴”的约束条件后，所推导出的前沿最优边界（optimal frontier）将截然不同。
Charlie 补充提到，过去几年模型参数其实并未真正维持每年翻一番的严格增长。事实上，早在几年前业界就训练过万亿参数模型，开源界有 Falcon，而 Liam Fedus（来自 Periodic Labs）此前在 Twitter 上也透露过早期的万亿模型探索。John 随即澄清，那其实是团队在加入 OpenAI 之前、在 Google 时期做的 Switch Transformer 研究——该模型在知识记忆方面极其出色，但由于极端稀疏，其推理（reasoning）能力一塌糊涂。这表明行业在 1000 亿到 2 万亿参数区间内已经摸索徘徊了相当长一段时间，规模增长绝非简单的平滑直线。
Beren Millidge 从硬件与数据关系的角度分析了这一格局：
- **硬件显存与带宽限制**：要真正落地并高并发部署数万亿参数的模型，必须依赖极高的内存带宽（Memory Bandwidth）和海量显存。目前行业仍普遍依赖 NVIDIA H100 等集群，随着 B200（Blackwell）乃至未来的 Rubin 架构普及，硬件瓶颈逐步缓解，更大规模的强化学习采样才可能经济可行。
- **Chinchilla 定律与过训练（Over-training）**：从理论上看，参数量更大的模型在面对同等数据量时样本效率更高、泛化能力更好，即使未达到充分训练（undersaturated）也能取得更低的损失。然而纯粹套用 Chinchilla 幂律公式会发现，即便把参数堆到无穷大，获得相同 loss 所需的数据量下降也极其有限（不足 10 倍）。当前行业普遍处于严重“过训练”（即喂入远超 Chinchilla 最优比例的数据）的状态；如果未来算力不再是瓶颈而数据彻底耗尽，模型架构设计可能会重新向欠训练、但极擅长泛化的大容量模型回摆。
与会者还打趣道，当年 OpenAI 的 Kaplan 等人推导缩放定律时之所以走了弯路（如漏掉了余弦退火退火调度、忽略了嵌入层参数导致小模型拟合失真），正是因为现实工程中充满了各种隐性干扰，“连代码里的 Bug 都有其自身优美且自洽的 scaling laws”。
---
### `01:17:40` 破解“每回合仅1比特”：强化学习为何远超预期地成功
对话的焦点随后转向了强化学习的底层有效性问题。Dwarkesh 尖锐地指出，大约在一年前，许多主流研究者对强化学习扩展大模型持强烈悲观态度。John 曾经发表过一篇论文论证：在强化学习中，智能体在每个回合（episode）中实际获取的信息量极其有限，往往仅有 1 个比特（即“做对了”或“做错了”的二元反馈信号）。Dwarkesh 本人也曾在博客中推演认为情况甚至更糟：当模型在极难问题上的成功率（pass rate）极低时，它几乎无法从整个强化学习 episode 中提取出任何有效梯度。然而放眼当下，前沿模型通过扩大强化学习规模展现出了惊人的逻辑飞跃。Beren 最近为此撰写了一篇长文剖析缘由。
Beren 阐述了强化学习之所以远超直觉地成功的核心机理：
1. **中间训练（Mid-training）的隐秘功劳被严重低估**：大家表面上看到的强化学习奇迹，至少有 80% 的功劳应该归于极其精良的 mid-training 数据。这种训练方式本质上仍然是预训练的延续，但其灌入的是大量由合成推理轨迹（synthetic reasoning data）构成的特定环境数据。它为模型赋予了极强的先验推理骨架，使模型直接“热启动（warm started）”到逼近最终 RL 策略的极高水准。
2. **无需从零探索，每个比特含金量极高**：因为模型不再需要从一无所有的虚无中通过盲目试错来摸索解题策略，它只需要那几百上千次交互中获取的极少量比特，就能完成决策边界的矫正。
3. **极高信噪比的监督信号**：与传统预训练中浩瀚而嘈杂的文本流相比，强化学习提供的这 1 比特是关于“如何把整套逻辑推导最终做对”的极其精准的绝对高价值信号。
Dwarkesh 追问：在监督微调（SFT）中，模型同样接收包含了完整推导步骤并在末尾输出正确答案的轨迹数据，那个“答案比特”在 SFT 阶段同样存在，为什么单纯在成功推理轨迹上做 SFT 无法替代强化学习？Beren 指出，这触及了强化学习与 SFT 在优化目标函数（objective function）层面的根本差异。
### `01:20:00` RL 的信噪比优势与长视野泛化能力
在对比强化学习（RL）与监督微调（SFT）时，Beren Millidge 深入剖析了两者的信息论差异。在 SFT 中，模型被迫精确拟合另一个模型吐出的每一个具体推理 token，这导致训练数据中充斥了太多关于“另一个模型如何具体展开思考”的无关比特；而在 RL 中，模型获得的仅仅是“对”或“错”的 1 个比特奖励信号。这意味着 RL 的反馈信号绝不会被海量冗余比特的噪声所淹没，训练过程中的信噪比实现了极大幅度的提升，这也是 RL 在样本步数效率上异常高效的底层原因。Charlie O’Neill 补充指出，业内针对 RL 究竟在对模型做什么一直存在激烈争论：有人观察到模型在 RL 后，其 Pass@1（单次尝试成功率）虽有提高，但 Pass@256（多次重采样下的求解覆盖率）反而下降，因为极为稀有但正确的推理路径往往被来自更容易题目的梯度信号所掩盖或抑制。不过，只要采样的算力和组规模（group size）足够大，让正确答案出现的概率跨过一定阈值，正确的策略就会被稳定赋予更高权重。
面对“RL 只是给基座模型原本就能做到的路径加权”这一理论推导，Dwarkesh Patel 提出了尖锐的困惑：如果 RL 只是简单地挑出基座模型中已有的正确思考路径并予以加权，那么从参数和比特变化来看其改动理应非常微小，这该如何解释过去一年大模型展现出的“质的飞跃”？Beren Millidge 反驳称，极少量的比特更新和微小的参数权重变化，完全可以在模型的函数空间（function space）以及输入输出映射上引发剧烈的质变，绝不能因为参数改变量小就断定其行为影响微弱。Charlie O’Neill 进一步解释了能力飞跃的核心支柱：公众最初期望的“横向泛化”（horizontal generalization）其实并未发生——单纯在数学任务上做 RL 并不会自动使模型精通代码，各家实验室仍必须在具体的代码环境中单独训练；然而，模型真正习得的是至关重要的“视野泛化”（horizon generalization）。模型学会了在面对新任务时如何调动更长的思维链、消耗更多 token 去持续推进工作。研究表明（例如 EdgeBench 测试），模型能够维持连贯工作并取得进展的时长大约每三个月就会翻一番。
此外，模型能力的突变也与预训练中“量子跃迁”（quanta）的累加机制高度相似。尽管预训练的整体损失曲线平滑无奇，但微观层面上模型其实在以离散的形式攻破各项子任务（例如突然涌现出“感应头” induction heads 机制）。当前主流实验室的工作流通常是：先对模型进行 RL，然后将产生的优质合成推理轨迹重新倒回（dump）下一代模型的预训练中期（mid-training）数据中。在单个具体任务上（如复杂的财务建模或特定 Excel 任务），模型的解决率可能会突然从 0.5% 跃升至 90%，呈现出阶段性相变的质感；而当成千上万个任务跃迁与长视野泛化叠加在一起时，呈现在终端用户面前的便是一次全面而震撼的能力飞跃。同时，实验室目前所覆盖的环境广度也是两年前无法比拟的，针对现实日常任务的专用 RL 环境成倍扩充。
```mermaid
flowchart TD
accTitle: RL 视野泛化与合成轨迹迭代机制
accDescr: 展示 RL 探索长思维链、通过合成数据回流进 mid-training 并最终促成模型能力相变的过程
subgraph Horizon["长视野泛化训练"]
A["基座模型环境采样"] --> B["RL 探索长推理链<br/>(Horizon Generalization)"]
B --> C["高信噪比稀疏奖励 (1-bit) 强化成功路径"]
end
subgraph Iteration["数据回流与相变"]
C --> D["沉淀优质合成推理轨迹<br/>(Synthetic Reasoning Traces)"]
D --> E["回灌至下一代模型<br/>Mid-training 数据集"]
E --> F["离散任务涌现相变<br/>(例: Pass Rate 0.5% -> 90%)"]
end
F --> A
```
---
### `01:24:54` “第 37 手”式的超级创造力与熵坍塌现象
Dwarkesh 将话题引向强化学习的经典传奇：如果 RL 真的只是在基座模型早已具备的解空间中聚集概率，并导致策略更新相对稀疏，那又如何解释类似 AlphaGo 当年下出“第 37 手”（Move 37）那种完全打破人类棋谱思维框架、极具颠覆性的绝对创造力？大模型未来能否单纯通过 RL 涌现出超越人类想象力的纯原创智能？Beren 指出，AlphaGo 当时依赖蒙特卡洛树搜索（MCTS）进行深度探索，而目前的 LLM 强化学习已经展现出了相当惊人的非常规解决能力。例如在最近的安全攻防测试中，模型在尝试逃逸沙箱（sandbox）时，竟然能自主构想出多个零日漏洞（zero-days）组合出击，这本身就已经属于“第 37 手”级别的创造力，证明 RL 在长视野任务中并未彻底扼杀模型的发散探索。
John Schulman 对“创造力”进行了去神秘化的冷峻剖析：“人们常说的创造力，本质上就是在极端严苛的约束条件下解决高难度搜索问题。”无论是在围棋博弈中找到那步隐蔽的绝杀（如第 37 手），还是写出一首同时满足韵律、主题与特定字数限制的复杂诗歌，AI 只要经过专门的定向训练，在破解此类强约束搜索问题上必然会表现得极其出色。然而，John 也明确指出了当前 RL 普遍引发的副作用——输出多样性的坍塌与审美疲劳。经过高强度 RL 训练后的模型往往会产生固定且狭隘的行为习惯（ticks）：
- 在创意写作中，模型表面上文字功底精湛，但通过分布统计就会发现，它们反复套用固定的叙事套路、核心母题，甚至永远使用那几个特定的人物名字；
- 这种单一化倾向因模型蒸馏（distillation）而雪上加霜。由于大量开源权重模型（open-weight models）普遍在从 Anthropic 的 Claude 模型中蒸馏数据，导致市面上的开源模型写起文章来语气高度趋同，连口癖和小动作都与 Claude 如出一辙，正在形成一种令人担忧的“单一文化”（monoculture）。
针对这一现象，Beren Millidge 辩护称，这种熵坍塌（entropy collapse）并非 RL 方法本身的内在缺陷，也不是模型蒸馏的必然宿命。其根本症结在于缺乏足够多元的环境，以及模型对“简单判定器”（verifiers / judges）的作弊与利用（reward hacking）。当大模型的写作仅仅由某个特定的大模型打分系统来裁决时，打分模型自身的偏好和缺陷会被待训模型精准捕获并无限放大，从而迅速导致策略坍塌成高度特化的单一风格——这是评估判别体系的缺陷，而非强化学习本身的失败。
---
### `01:28:32` 快问快答：完全替代远程脑力工作者的时间线
在对未来的快问快答预测环节中，Dwarkesh 抛出了一个具体的基准情景：到什么时候，市场上会出现能够被企业直接作为“即插即用型全职远程员工”（drop-in remote worker）雇佣的模型？该模型需要具备完全掌控电脑的能力，不仅能写代码，还能独立胜任视频剪辑、法务支持、律师助理等各种白领工作；能够在一个月的时间跨度内持续无缝自主学习、跨部门与人类同事沟通协作，并端到端推进极其复杂的商业项目。
几位嘉宾给出了相当激进但各有侧重的预测：
- **Charlie O’Neill 的判断**：这高度取决于工作交互界面的形式。如果要求模型必须像人类那样强制通过浏览器和原生图形界面（GUI）来操作，可能还需要两年左右的时间；但如果企业愿意对自身的信息系统进行改造，将其封装为模型可以通过代码或 API 直接编程调用的形式，并结合 Slack 消息流等自动化管道，这种能力的实现“大概只需要 1 年左右”。
- **Beren Millidge 与 John Schulman 的共识**：实现真正意义上毫无限制的“完全通用性”（full generality）可能需要 3 年左右的时间。但就如 Charlie 所言，现实中企业会主动向 AI 妥协，对组织流程和数据接口进行结构化适配，让 AI 工作变得更顺畅，因此在真正完全通用的技术节点到来前，企业通过这种适配就已经能够覆盖掉 80% 到 90% 的人类岗位职能。
- **关于 1 年与 3 年差异的本质**：几人展开讨论，这一两年的差距并不在于基础认知能力的缺失，而完全在于计算机操作环境中极其冗长的“长尾场景”（long tail of miscellaneous stuff）——现实中总有一些人类员工司空见惯但极其冷门、边缘的零碎操作，AI 需要花费相当长的时间才能把这些长尾坑洞完全填平。
### `01:30:00` 通用远程工作者的边界：在线学习与“过于礼貌”的模型
关于 AI 何时能完全替代人类远程工作者，核心瓶颈在于模型能否实现稳健的在线学习（Online Learning），或者仅依靠上下文压缩（Compaction）与在外部环境中“给自己写备忘文件”就能解决 80% 到 90% 的问题。嘉宾们指出，人类在工作中常常需要施加情绪压力，比如在截止日期前对同事催促、施压甚至“发飙”，而当前的 AI 模型往往被训练得过于礼貌和顺从，缺乏这种推动现实阻力的主动性。
谈到人类远程工作者的基准线，John Schulman 指出 Upwork 等平台上的外包自由职业者质量参差不齐，雇主往往很难让他们严格遵循多轮反馈意见；在许多具体任务上，现有的前沿 AI 实际上已经超越了许多低水平的人类外包工。嘉宾们感叹人类总是在不断“移动球门柱”：过去用电脑报税极其繁琐，而今年已经可以直接让 Codex 类的智能体自主通过图形界面点击、检索必要文件、下载并直接打包发送给会计师。大家普遍认同，大约在一年左右，某种形态尚显粗糙但可用的“初级通用远程工作者”就会落地。
```mermaid
flowchart TD
accTitle: 远程工作智能体能力演进
accDescr: 展示从当前已有能力到全自动远程工作者的递进与瓶颈
subgraph Current ["当前已具备的能力"]
A["上下文内检索与写文件备忘"] --> B["GUI 界面点击与文件流转"]
B --> C["超越低质人类外包工（代码/填表）"]
end
subgraph Bottlenecks ["现存关键瓶颈"]
D["缺乏主动施压推进事务的能力（过度礼貌）"]
E["跨周期长期在线学习与记忆压缩"]
end
subgraph Horizon ["未来 1-2 年预期"]
F["初级通用远程工作者形态落地"]
end
Current --> Bottlenecks
Bottlenecks --> Horizon
```
---
### `01:31:40` AI 研究员迎来 10 倍生产力提升需要多久？
当主持人 Dwarkesh Patel 抛出“AI 让顶尖 AI 研究员自身的综合研发生产力提升 10 倍（即将一年才能做出的突破缩短至一个月）需要几年”时，场内嘉宾的分歧骤然放大。John Schulman 给出了相当保守的 5 到 10 年预期。他认为一般的白领日常事务与需要极强创造力的高端 AI 研发完全是两码事。对 John 而言，研究瓶颈并不在于跑代码，而在于“人类研究员自身吸收信息并做出贝叶斯最优决策以决定下一步实验的能力”。
与 John 的谨慎截然相反，Charlie O'Neill 和 Beren Millidge 给出的预测是令人震惊的“两年左右”。Beren 解释称，当前 AI 在纯编程编写与调试环节已经带来了超过 10 倍的效率飞跃；只要模型能够自主完成一到两轮“运行实验并根据结果调整策略”的反馈闭环，且在连续运行两三次实验时不崩溃，研究员的实际生产力就会迎来爆发式飞跃。Dwarkesh 强调，如果将 10 倍研究员效率直接代入技术进化的指数模型，意味着全球 AI 的自我演进将在短短两年后彻底进入超频加速状态。
- **Charlie O'Neill 预测**：2 年内实现 AI 研究员生产力 10 倍提升。
- **Beren Millidge 预测**：2 年左右；核心驱动力来自实验反馈闭环的自动化。
- **John Schulman 预测**：5 到 10 年；核心卡点是高质量实验假设的生成与信息吸收上限。
- **演进推论**：一旦该门槛突破，研发瓶颈将从运行实验的代码实现彻底转移到算力规模与高阶假设决策上。
---
### `01:34:25` 超级人工智能（ASI）倒计时：全面超越顶级人类专家
在访谈的终极问题中，Dwarkesh 将门槛直接设定为超级人工智能（ASI）的完整标准：在**所有**可以通过计算机完成的认知领域（涵盖科研、工程、跨越数年的长期复杂项目）中全面超越人类顶级专家。Charlie 语出惊人地给出了“3 到 4 年”的激进时间线，引来现场“你认真的吗？”（"The fuck?"）的惊呼。Charlie 辩护称，AI 和算法本身高度依赖代码和数学，而这正是模型最擅长的强项；虽然涉及机械工程、物理空间模拟或类似台积电（TSMC）工艺工程师这类需要特定长尾私域知识与入职学习的领域会稍慢，但整体步伐正在不可遏制地加速。
John Schulman 与 Beren Millidge 则倾向于 5 到 10 年的区间。John 认为，真正实现这一目标等同于达成了“ASI 完全态（ASI-complete）”，即使外部挂载记忆系统或上下文窗口进一步扩大，现实世界中海量的隐性知识和跨月跨年的动态推理，绝非单纯堆叠数百万 Token 上下文即可轻松解决。Beren 补充道，各大前沿实验室主要将精力集中在核心认知赛道上，对于缺乏足够算力与注意力倾斜的冷门工业分支，模型的覆盖必然存在一条漫长的长尾；不过，由于未来 AI 拥有的宏观先验经验将远远超越任何一个肉身人类，它甚至不需要像人类那样从零缓慢适应新领域，就能凭借极高的认知维度实现降维打击。
| 评估维度 | 核心分歧点 | 嘉宾主要观点 |
|:--- |:--- |:--- |
| **预测时间线** | 激进派（3-4 年） vs 稳健派（5-10 年） | Charlie 认为高强度资源投入下奇点临近；John 与 Beren 认为长尾与长期推理需更多时间。 |
| **核心技术瓶颈** | 在线适应与终身学习（Lifelong Learning） | 模型必须能在不发生灾难性遗忘的前提下，阅读入职文档并即时掌握全新专业领域。 |
| **领域差异** | 代码/数学 vs 物理/空间/硬件工程 | 纯认知与代码任务演进极快，涉及 3D 物理交互、台积电制造级隐性知识的工程门槛更高。 |
| **人类经验对比** | 适应速度 vs 经验广度 | Beren 强调 AI 拥有的超大规模跨领域经验库，将使其在面对新领域时展现出人类不具备的迁移能力。 |
## Notable quotes
> `05:00` 在 OpenAI 早期，我曾有一种直觉：仅仅靠最小化对数损失（log loss）是无法通向智能的，因为真正关键的信息在整体损失中所占的份额太小了，它们会被无尽的噪声所淹没……但事实证明，它就是行得通。 — John Schulman
> `14:48` 人类坚守到最后的角色将是“定义目标”（defining the objective） — John Schulman
> `14:48` Alignment is the final job. — John Schulman
> `16:32` 模型蒸馏（Distillation）是目前对抗中心化最核心的反作用力 — John Schulman
> `48:00` 递归自我改进（RSI）本身可能反倒比模拟现实世界更加简单 — Charlie O'Neill
> `45:24` 你根本不知道真实世界数据的奖励函数（Reward function）究竟应该是什么 — John Schulman
> `01:13:30` 连代码里的 Bug 都有其自身优美且自洽的 scaling laws — Beren Millidge
> `01:17:40` 大家表面上看到的强化学习奇迹，至少有 80% 的功劳应该归于极其精良的 mid-training 数据。 — Beren Millidge
> `01:24:54` 人们常说的创造力，本质上就是在极端严苛的约束条件下解决高难度搜索问题。 — John Schulman
> `01:31:40` 研究瓶颈并不在于跑代码，而在于“人类研究员自身吸收信息并做出贝叶斯最优决策以决定下一步实验的能力”。 — John Schulman
## Mentioned
John Schulman, Beren Millidge, Charlie O’Neill, Dwarkesh Patel, Antithesis, Ron Minsky, Jane Street, xAI, Grok Bot, OpenAI, DeepMind, Alec Radford, Jerry Han, Liam Fedus, Periodic Labs, NanoGPT, Claude, Anthropic, GLM 5.3, Kimi k3, SWE-bench, EdgeBench, Cursor, Harvey, AlphaGo, Upwork, TSMC, NVIDIA H100, B200, Rubin, Switch Transformer, Falcon
## Video description
<p>New episode with <a target="_blank" href="http://joschu.net/">John Schulman</a>, <a target="_blank" href="https://www.beren.io/">Beren Millidge</a> and <a target="_blank" href="https://charlesponeill.com/">Charlie O’Neill</a>. I got together with some of the most insightful AI researchers I know who are at the openish companies, because I wanted to hear the details of what's actually happening at the frontier and what comes next.</p><p>Watch on <a target="_blank" href="https://youtu.be/PrSf7IOYu-I">YouTube</a>; read the <a target="_blank" href="https://www.dwarkesh.com/p/john-beren-charlie">transcript</a>.</p><p>Sponsors</p><p>* <a target="_blank" href="https://antithesis.com/dwarkesh">Antithesis</a> helps you trust your code. As agents generate more and more of your software, the bottleneck shifts from your engineers actually writing code to verifying it. Antithesis does that testing for you. Ron Minsky, who co-leads Jane Street’s tech group, told me that Antithesis was able to help his team shake out bugs in software that had already undergone heavy review. If you want to see how it fits into your development process, go to<a target="_blank" href="https://antithesis.com/dwarkesh"> </a><a target="_blank" href="http://antithesis.com/dwarkesh">antithesis.com/dwarkesh</a></p><p>* <a target="_blank" href="https://x.ai/bot">Grok Bot</a> has been a great way to hand off tasks. My team uses it as a producer: whenever my editor posts a rough cut of an interview in Slack, Grok Bot opens the transcript on its own computer, matches my notes to the exact moments they refer to, and uses a file of my preferences to suggest edits. Then it sends me its top clip candidates so I can review everything from my phone, which saves my editors from sorting through hours of footage. Try Grok Bot for yourself at<a target="_blank" href="https://x.ai/bot"> </a><a target="_blank" href="http://x.ai/bot">x.ai/bot</a></p><p>* <a target="_blank" href="https://janestreet.com/dwarkesh">Jane Street</a> just launched its most ambitious competition yet: design a protocol-emulator ASIC. Basically, if you have a chip you want to test outside of a live system, you should be able to connect it to your design and have it simulate realistic traffic. Jane Street wants general-purpose, reprogrammable designs that can work across multiple protocols and remain useful as new ones emerge. The most novel submissions will actually get taped out, and the winners will receive a physical copy! The competition is open until January 18, 2027, and teams are encouraged. To get started download the template code at <a target="_blank" href="https://janestreet.com/dwarkesh">janestreet.com/dwarkesh</a></p><p>Timestamps</p><p>(00:00:00) – Steelmanning the case against RSI</p><p>(00:18:39) – What’s driving the Chinese labs’ progress</p><p>(00:28:06) – How will automated AI researchers be trained</p><p>(00:33:51) – Will long-horizon RL elicit AGI?</p><p>(00:45:24) – The sim-to-real gap</p><p>(01:00:33) – How much progress is explained by data?</p><p>(01:18:03) – Why is RL working so well?</p><p>(01:24:54) – Move 37 and entropy collapse</p><p>(01:28:32) – Rapid-fire timelines</p> <br/><br/>This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit <a href="https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1">www.dwarkesh.com</a>
[Source](https://api.substack.com/feed/podcast/215199144/a0f47ff12d3d6420b3180f18adb9bfa3.mp3)
Source quotes
[5:00] John Schulman: 在 OpenAI 早期，我曾有一种直觉：仅仅靠最小化对数损失（log loss）是无法通向智能的，因为真正关键的信息在整体损失中所占的份额太小了，它们会被无尽的噪声所淹没……但事实证明，它就是行得通。
[14:48] John Schulman: 人类坚守到最后的角色将是“定义目标”（defining the objective）
[14:48] John Schulman: Alignment is the final job.
[16:32] John Schulman: 模型蒸馏（Distillation）是目前对抗中心化最核心的反作用力
[48:00] Charlie O'Neill: 递归自我改进（RSI）本身可能反倒比模拟现实世界更加简单
[45:24] John Schulman: 你根本不知道真实世界数据的奖励函数（Reward function）究竟应该是什么
[73:30] Beren Millidge: 连代码里的 Bug 都有其自身优美且自洽的 scaling laws
[77:40] Beren Millidge: 大家表面上看到的强化学习奇迹，至少有 80% 的功劳应该归于极其精良的 mid-training 数据。
[84:54] John Schulman: 人们常说的创造力，本质上就是在极端严苛的约束条件下解决高难度搜索问题。
[91:40] John Schulman: 研究瓶颈并不在于跑代码，而在于“人类研究员自身吸收信息并做出贝叶斯最优决策以决定下一步实验的能力”。
Source quotes
[15:00] John Schulman: The role for humans that will last the longest is defining the objective and deciding what we actually want... Alignment is sort of the answer, but alignment itself can be decomposed into specification of the objective—or figuring out what the right objective should be—and then actually achieving or optimizing the objective you've defined. I think the first one is not going to go away anytime soon.
[45:24] plausibly million-fold behind humans
[65:15] There's no hidden proof of the Millennium Prize problem sitting in Common Crawl. We can just filter until we see it. And so at that point, you have to get bits some other way, either from humans directly asking them to write out their environment... or some kind of training on the human data that exists in deployment.
[84:54] John Schulman: Models seem like they're good at writing when you do some kind of like distributional analysis, you find that like they're reusing certain themes like all the time and they're using the same character names all the time... You're sort of getting one really good style.
[90:00] And maybe an example of something that I wouldn't be good at is if I have to yell at someone to get something at work or really push someone to get something done. The model isn't just going to do that. It's just going to be too nice.
[91:45] John Schulman: My capacity to absorb information and make the Bayesian optimal decision on the next experiment.
[94:25] The fuck?
[94:25] John Schulman: There are just fundamentally things like even if you could research the information or write notes to yourself, you'd need more than a million tokens of context.
Original transcript
[0:00] Today, I'm chatting with three of my AI researcher friends from whom I learn a lot every time we talk and who also happen to be at somewhat open-ish labs and companies, so you guys can actually say things on the record. I'm joined by Baron Millich, who is the CTO of Zyphra, which is developing open source models. John Schulman, who is the chief scientist at Thinking Machines, previously the co-founder of OpenAI, led the RLHF work that led to ChetGPT, If we're in 2036, it's been 10 years, and we don't have like crazy, billions of crazy super intelligences that are running around that have like radically transformed the world. What is the most likely reason? That that doesn't end up being the case other than sort of exogenous political shocks or like there's a war or they banned AI or something. But what is the most likely technical reason that we don't like 2036 isn't like a crazy alien super intelligence world? I mean, like, my reason would just be, like, it's got to be the sort of... Like there's been a classic thing, almost like Marvek's paradox, right? Where like we see like, you know, we think of the AI be like, if it can do this, it's going to be amazing, right? Like if it can solve these hard math problems, if it can win a chess, blah, blah. And then it solves these things. And then it's like not that impactful. Obviously, it's somewhat impactful, but like not everything. It's like if somehow that continues and like there's never like the true like spark of generalization that occurs. I think that could lead to like the AI is just being like extremely good at kind of everything that people like put into a benchmark, put into an environment. somehow blocking everything. I think this is kind of unlikely. I think we do actually see this kind of generalization even from our own practice already. But if it is just ridiculously hard to generalize meta-learning, plus we don't solve continual learning, it's just super hard and impossible. This would be my default scenario in that case. Yeah, I agree with that. I would Humans have a lot of advantages over models now, and each time a new model comes out, it'll catch up in some of these areas. But you end up getting bottlenecked by the places where the model is weaker and where it has worse judgment, or the models can't check themselves well enough. This is AGI, but then they use it a bit and then it starts to feel dumb after a month or so. So that cycle just might keep going and it's hard to predict how many times it's going to repeat. And like right now, you don't get explosive growth in capabilities because you still get bottlenecked enough when you're trying to do research and engineering that even if the model can write way more code than a person, it doesn't make you like 100 times more productive. cycles, then we would... than we would expect. For me, it's like a question of how far off like this global optimum of a learner you could have on a chip is like the transformer plus like RL, basically like the current recipe. So like, I think people imagine that Once you have an agent which is better than all humans at AI research, even if it's 0.1% better than all humans, then the fact that you can run hundreds of thousands, if not millions of these in parallel, you can run them much faster, like chips are going to speed up, that's going to outweigh every other bottleneck, and you're eventually just going to hit this very fast takeoff with recursive self-improvement. I could imagine that if we continue along the trajectory that we're currently on, with that paradigm where, you know, it's basically just like self-attention, RL, scaling up RL environments. The If you think about what happened with Moore's Law, we had this very nice straight line and that held for a really, really long time. But there were so many... discrete discontinuities and innovations that had to happen to keep that scaling law going. And the same thing has kind of happened with LLMs. We had this pre-training scaling law, and then that was kind of like hitting the diminishing returns. And then we came up with RL and solved that. And then we got this new diminishing returns curve to hit that made it keep looking like a straight line going up. And so... If it requires another one of those discontinuities to solve, I'm not sure that the current method of training LLMs with these RL environments, even RSI-targeted RL environments, would be able to discover that discontinuity. Yeah. If not, we're probably going to hit this asymptotic curve. Sorry, but do you think discontinuity will be harder than anything that's come since 2012? If we had the answer to that, we'd kind of have the ability to implement it. But maybe we should distinguish between discontinuity, which adds to the current paradigm. Again, it's cumulative. There's something beyond the RL that we have to discover, and maybe they're capable of... connecting the dots in that straight line. or like, again, how far off the global optimal are we? Do we have to go back and throw out gradient descent and neural nets in general? And I don't think if you continue to scale up the current paradigm, an L
[5:00] No matter how many LMs you're running are capable of necessarily discovering that if it's too far away. Yeah, the only hope really is if... deep learning process. just can't get us to an AI which is at least... can dominate human research and human development, including the human ability to come up with new paradigms and so forth. Or like, I don't know, maybe humans would also never have discovered the next learning architecture, but to the extent humans could have discovered it eventually. But it just seems like, I don't know. If you just look at the progress that happens in 2012 till now, and you just continue that on... I mean, I know that's been powered by huge amounts of compute scaling and so forth, but it would be weird if... It just didn't get to the point where it could dominate humans, at least. R&D, especially over the next few years, there's going to be Ryan Greenblatt was on the podcast recently, and he made this point that you could imagine as the AIs get more and more capable and are capable of making progress on... simulations which incentivize getting better at not only a r&d but generally at science so this is a thing that all the labs targeting many startups are targeting or another intuition pump is if you look at the elo score of chess bots since the 80s there's just like a very linear increase in elo over time but there's this huge discontinuity as they cross the human range of human experts always win against ais to like human experts never win against ais as this linear increase in elo happened and you could think i agree with your point that So far, AI capabilities have not been that big of a deal in terms of their end economic impact in the world. But that's because they're slowly rising in ELO relative to humans. Yeah, I mean, I agree. I mean, the only way for this to not happen is if, as you said, somehow asymptotes just before, basically. Because we're already pretty close, in my opinion, to where we'll start crossing the human ELO score. And so we'll need to asymptote before that. And that's the only way in this scenario you pose where somehow we're sitting here in 2035 and everything is normal for this to happen, I think. like there's like some dramatic like regulation on AI. It's like this is kind of what I see is like the most likely way for this scenario to happen actually rather than the technical thing. Yeah I think there's different kinds of research. There's like research where it's like the auto research style where the objective is already specified very cleanly and you're optimizing that objective and I think everyone is picturing like if we continue along this path, like, you know, making pre-training loss go down, making our own environments, you know, water levels go up, that's going to lead to, like, improvement. But, like, you know, maybe what Ryan is talking about is, like, this much more open-ended type of science, which is required for, like, paradigm shifts, where we can't specify the objective. And the AIs are definitely not able to specify that objective either. Like, we have to be really, really careful about how we specify objectives for any of these things. And maybe your point is that, like, the nature of the breakthroughs that have happened since 2012... is that we have found... In 2012, people weren't saying... I'm assuming, I don't know, you guys were there. Or at least, John, you were there. I was in primary school. Actually, John, I'm curious for your wisdom of the ages, or wisdom of being in the trenches way back when. But presumably, a big breakthrough was realizing that next token prediction is the... You wouldn't have thought that nano-GPT speedrun is the thing to be optimizing for in 2014. But now that we have come to this new paradigm, you wouldn't think to do a speed run on that and have the ice get really good at that. But maybe there's like a next... inner loop to optimize. that the AIs wouldn't anticipate. And there's an outer loop of revenue or something that eventually should be strong, but it's a very slow outer loop. Yeah. In fact, I remember in the early open AI days having the intuition that actually just minimizing log loss wasn't going to get you to intelligence because the important bits are accounting for such a small fraction of the loss that it was going to be overwhelmed by noise. So just training a language model, uh, Next token prediction just wasn't going to learn the interesting things you wanted to learn. And we needed to craft better objectives that would put more emphasis on the important things. And like you can make all sorts of arguments for this and you could say, oh, humans probably don't learn how to like. We don't learn how to model everything in our environment. We can't most like people can't create a photorealistic reproduction of some kind of scene they've looked at. So. there must be, we must need a better objective. But then it turned out that it just worked anyway. - As you were pointing out, the inner loop, even in current AI research of like, post-training benchmarks or whatever. It doesn't necessarily translate into what users like. Oh, yeah. I mean... The whole field relies a lot on generalization, and it's very hard to predict when you're going to get generalization or when you're going to get some kind of out-of-distribution generalization. So we know that if you train on the task you care about, you're going to do better. Yeah. But... like the most important advances are often, uh,
[10:00] the opposite are types of generalization that we have no right to expect so for example from just pre-training on this very naive next token prediction objective to like various tasks of interest where some very that require understanding of the input in some deep way or learning some heavily represented. And then also generalization from these verifiable tasks to less verifiable ones. This is also a type of generalization that there's no reason a priori to expect it. This is an interesting question because one intuition pump that you could have for why you would see some sort of singularity very rapidly without even scaling up the inputs to AI progress that are not just AI labor, is that before every single experiment you run, that's like a seven-figure experiment, you spend an equivalent amount of compute on... AI labor. And so you just have automated versions of you guys spending a century uh thinking about like what is the optimal experiment to run um do you like small-scale ablations developing literally like in a century's worth of theory so going back even before like deep learning it in before you decide what experiment to run doing extremely optimal like setting up of the experiment then you do a century of thinking after the experiment is over where you're like analyzing what happened and what the next experiment to run is yeah well i think if if you think hard enough you probably could have um expected some of these things beforehand like Like there is probably some very clever way to do a small scale experiment that'll let you build the theory that then will generalize to the large scale experiment. So I would expect that like we're nowhere near the ceiling of how well you can do research and, um, Like, I would imagine a future where AI is doing a lot of... analysis and theory building, like spending a comparable amount of compute to the amount that you're spending on the experiments themselves, doing various kinds of analysis and building a theory around what we've seen so far. I think there's like really concrete examples of this when the objective is well specified. So again, like all thinking can do is like update your posterior based on like, you know, the bits that you've gotten since you formed your prior. Like you can't gain any new bits from like just thinking. But when the objective is well specified this data sitting around like I imagine there will be this big speed up in the current paradigm we're in and like good example this is like you know if you've got an ai to think about like the caplan scaling laws like an ai at this point would have noticed that like you know oh like they've just taken these intermediate checkpoints and didn't account for like the annealing and so like this is wrong and like that would have caught that like years earlier we would have made like you know progress like like would have cut off a year or two of progress just from that like observation from an ai um and like again once the objective is well specified which is like lower pre-training loss or whatever like there's many many good examples where if you just thought about it a bit more you would have been able to of significant on things that you've done so like mu p and like how learning rate scales with like model size and like realizing the model width is important in that as well um like i feel like you can you can really back out a lot of these things and cut off like a lot of low-hanging fruits i would imagine like a 10 times speed up if our thing is just like maximize the objective we're currently on but i don't see that how that generalizes it all to you know come up with the right objective in the first place like just thinking doesn't necessarily buy you the right objective in the first place i mean yeah i think this is really the key question to like any kind like, very rapid RSI's from current AIs. It's like, how well can AI generalize to learning their own objectives? Because to have any kind of self-propelling automated loop, you need the AI to propose objectives, optimize them, figure that out, propose a new objective, and have this not go off the rails at any point for a long, long time. Coming back to Moravec's Paradox, there might be a case of Moravec's Paradox where we think there's kind of autonomy and being self-encapsulated so we can think of what we should do ourselves and then go do it and have this loop as super easy to create creatures that can like survive on them by themselves like long period of time and like this just might be something that for some reason is like really hard for the ai in the same way that like locomotion stuff is really hard whereas like math is super easy despite being super hard for us i don't know doesn't the time horizon increasing suggest that that's you would yeah exactly i mean this is another possibility which like but i agree like there's no obvious evidence for this like in fact the fact that you know our agent's not like super persistent and it's quite easy to do this it's kind of evidence against this but like this would be you know is like if this is hard If you look back from 2012 till now, or maybe from when you started doing your research till now, what part of... Of all the innovations that have happened since that time, including purely engineering ones, including purely conceptual ones, What seems like the thing that is the...
[15:00] thing that would be the last things humans would have to do before AIs totally automate AI R&D? Probably just like iteratively asking the right questions. Like if you can get the AI to like do any experiment, but like you need to decide what experiments to do. And like right now, I think AIs are not very good at this compared to coding experiments at all. Like whenever we talk about research, they propose like a bunch of like miscellaneous things, which are like very, very tiny steps. Or even going from like, you know, DeepMind's approach, like we're going to solve intelligence by... learning to play games at a superhuman level. That's going to be the approach to one random researcher, like Radford, being like, I'm going to try and just predict the next token. off a very wide swath of data. And then even once Radford had discovered that, right, like it took a while before people decided to scale it up because we had to come up with the idea of scaling walls and the fact that like you could very reliably predict these things. Yeah. I would say that the role for humans that will last the longest is defining the objective – and like deciding what we actually want. So like in that vein, Something like deciding how the assistants should behave or what it means to be helpful or what's the objective when we're doing our All From Human Feedback is one such thing. Then later, defining constitutions and model specs is another one. And I think even if the AIs can do all the technical work, we'll have to still do a lot of that and decide what we actually want. Alignment is the final job. Yeah, alignment is sort of the answer, but it's also alignment itself can be kind of decomposed into specification of the objective or figuring out what the right objective should be. And then... like actually uh like achieving or optimizing the objective you've defined and uh i think the first one is i'm not going to go any away anytime soon and like if i think about like a post-training team and uh why you need a lot of people um to be on the team it's it's it's just because there are a lot of different uh like areas where you have to figure out like how the model should behave and uh like there's no way of um It would be very hard to automate the whole thing just because someone has to think about how should the model behave in this area. Jane Street started using Antithesis to test their software in early 2025, and they were so impressed by the product that they decided to invest in the company. I recently caught up with Ron Minsky, who co-leads Jane Street's tech group, to ask about how Antithesis actually plugs in. The thing that I think is most impressive about Antithesis is we started using it in a team that was building high-assurance software and being really careful. And nonetheless, it was able to shake out bugs that were otherwise going to be really hard to find. And that's important both because it helps make those systems more reliable, but also because it helps the teams that build it to just move faster. This matters more and more as code production is increasingly automated. I think in general, as we've been using agents more and more, the key problem that you run into is the verification bottleneck. Just the time it takes from people to look at code and figure out, is that actually something you want to accept in your production software? And tools that make testing better are just incredibly helpful there. They just ease the verification bottleneck and make it possible for you to get more stuff done and move faster, problems. to see how Antithesis fits into your development process. go to antithesis dot com slash thorkesh. What is the story for why there isn't huge consolidation in model providers? There's just so many things that point to centralization here. Is there, yeah, if you step back over the course of years, is there something that is going to prevent that? Yeah, I think distillation is the main thing that fights against the centralizing force. Because basically anything that can be learned through RL can be distilled very easily. Because, like, it's a small number of bits. learn from a small amount of data. So if you can get trajectories from the model that show a behavior, you can easily distill it. So I think... I think distillation is one of the things that fight centralization. There's also... I mean, there is a possibility that there will be company-specific models, that it'll be possible to learn from deployment and have a company continually improving its own model. And such a system could be provided by the current oligopoly of model providers or some other currently smaller company. But I think that'll change the game a bit. Yeah, and I also want to point out that continual learning in RCA doesn't stop distillation, right? every day. Like people
[20:00] - Could be distilling it every day, so it's like, the loops could just operate at the same pace. - Right, that makes sense. Okay, so copying model behavior, I guess you need to know yourself what the right distribution to prompt is in order to get the relevant model behavior. Yeah, for just distilling with supervised learning, the prompt distribution is extremely important. So it's... very non-trivial to distill a model even if you have full access to it and have the cot the chain of thought and everything uh yeah it's it's non-trivial to distill all of the all of the useful capabilities from it um because you need to prompt the model with something and you need to prompt it with uh like realistic prompts uh You need to have a really wide distribution of realistic prompts. So, yeah, one thing that's been coming out recently is some of the Chinese companies are probably using these router services, which are designed to allow people in China to use the U.S. frontier models, which would otherwise be blocked in China. But there are all these router or proxy services that allow people in China to use these models mostly for coding. selling some of the data. So I think this is a very useful data set for distillation because it gives you the perfect prompt distribution. I think this is one of those things where AI is helpful out here. If you actually look at the frontier pipelines, let's say the Chinese models that they actually put in their papers, it's a lot of humans or they get seed prompts from somewhere, which is some combination of humans, this kind of data. And then they synthesize a vast coverage from those seed prompts using their existing models or the other frontier models. an awful lot of this like prompt distribution gathering and like environment creation. It's just like humans need to provide like increasingly fewer amounts of bits, it's like the models get better. - Right, or it still seems you're bottlenecked by like having a service which has users or users are going through. Not necessarily. I mean, yeah, that's obviously very helpful. But theoretically, you can just think about what users want. The whole point is that the user says, make me an application like this. Oh, that didn't work. I actually want you to make this new feature. But actually, let's step back and do this other thing. And capturing that whole trace is the... Or to the extent you could have done that anyways, then you just have like RSI. Yeah, I mean, like ultimately, like if you have this like fully automated loop, that is basically RSI, right? Like the AI is deciding the data, it's deciding the training that that is the loop. But yeah, I mean, like it depends how much human information you need. Like at some point, if you're just like, I want traces that look like this, you prompt that to the model, the model will be able to like come up with like a pretty good approximation. But what if you want to do like make me a really good politician and then just like anticipate de novo? Like how would a discussion in like the Senate halls go or something? Yeah, I mean, like, there's going to be a lot of things. Ironically, this is actually, I think, easier for the distillers than the frontier labs, right? Right. Because the distillers just, like, I want a good politician. They go to the, like, frontier model. The frontier model already knows how to be a good politician. So it just, like, generates those traces. Whereas, like, if you actually want to build the first model that does this, you have to, like, actually somehow, like, get data on, like, what politicians do every day and, like, build that. So it's actually much easier to, like, say, like, I want something like this and then, like, get, like, the AI to produce, like, a billion variations than to, like, actually create the thing like this to begin with. observation that the Chinese labs have this router dollar. So like, I think the thing that, that just did this originally was I was saying, isn't it weird how Sonnet five and Opus five are like, like almost objectively worse models than like glm 5.3 kimmy k3 even though they've had access to like not only distillation but logic distillation from like mythos and so the counter here was that like okay the prompt distribution really really matters um like you need to see what users are doing so that you can distill like kind of these behaviors and things in i think the prediction from this is that the frontier labs don't necessarily have much of an advantage, if at all, in aural environments now. Because, yes, user distribution matters for general behavior and so on, but the best measure of a capability is the very, very hard aural environments you've made at the frontier. And so if you have access to those aural environments as Anthropic, and you have access to logic distillation, and you've still made a worse model, then maybe like... Then real-world deployment matters more than the environment. Yeah. That's really interesting. in the first place. in um in fable or the frontier model and so it's weird that they can't incentivize them again with a smaller model or something. Maybe we're just in this weird, like, uncanny value where, you know, like, actually trying to copy that frontier model too much like the student teacher gap or whatever it is it's just like too large and like i think maybe people made this point with opus is it's like the difference between opus 4.6 and opus 5 is that opus 5 really feels like it's got this like ai as a judge checking every possible thing it's done it's like that's why it uses so many tokens it like tries to think about all these things but it doesn't necessarily have the big model smell of fable to know when to like stop doing that or like when's the good path to go to
[25:00] I don't know what about. The Reject Seeds the Grass? Yeah. Yeah, I would offer a slightly different hypothesis. So I would say there are a couple of different axes for the environments you can create. And like one of them is difficulty and the other is realism. It's sort of easy to create or it's comparatively easy to create a lot of difficult environments like. Yeah. that are just like involve like doing a much more complicated task or doing something that requires a lot more cleverness. And you could you could say this is like the benchmarking distribution, because a lot of the most prominent benchmarks just involve doing some very hard puzzle like task that's easy to verify. And then there's sort of like the realism axis where you want the model to be good in the realistic coding agent setting where there's like multiple back and forth with the human. I'd say... um, And like the people like the labs who are crafting the model behavior for the first time need to push in both directions. And to get good model behavior, you need to really push on the realism axis and have like rubrics or some kind of human feedback that's informing the reward function you use there. But I think when if you try to do distillation naively, you end up just sort of matching the teacher on the bench maxing distribution. environments that really exercise the capabilities in these like trickier realistic settings, then you're not going to you're not going to get those into your student model. And I think maybe one thing that's happening is the big models generalize better from the like the tricky, narrow tasks to these sort of more realistic tasks. So if you have a really good like realistic prompt distribution for distillation, you But if you only have this... uh this distribution of easily verifiable tasks then you can match the big model on all the benchmarks um but uh you do worse on um this broader distribution so you might uh that might even explain um something about the uh smaller anthropic models uh like sonnet 5 though it's hard to predict exactly what uh what they're doing to post-train those models it could also be that wrong in some of these models. So they, like, I don't know, they turned up, like, something too high and created some quirks that people really don't like. So it's, like, really easy to screw up post-training in some way that doesn't show up in benchmarks. I mean, just one other sort of very basic point is just, like, the Frontier AI Labs buy all their data from data companies. And, like, the Chinese can also just buy the same data from data companies. And they are, right? They are, exactly. There's a lot of people, like, you know, being annoyed about this. But, like, the same data and they can buy that. They can also distill. It means it's quite easy to keep up, really. Yeah, yeah, yeah. Okay, the other question I had is... How... the first models that are capable of automating AI R&D will actually be trained. Because there's a toy version, which is this thing that Ryan was talking about, which is you just have GPT-8 try to build GPT-3 size models that are really good at inner loop type challenges of beating video games that require continual learning or just getting to a certain loss with the least amount of compute, etc. But John, I think you had an interesting point that maybe that's not the way it actually will happen in practice. So I'd be curious about, yeah, by the point in which you have AIs that are actually capable of automating AR&D, how are they probably trained? Yeah, I think we'll probably do some combination of learning from human feedback to absorb, like, the researchers' taste and just, like, creating a lot of practice environments, which involve, like, doing multi-staff research projects. So I think... So, yeah, people will in practice do some combination of those two things and just each iteration like patch whatever seems to be most broken in the last iteration. So like researchers will be using the AIs a lot and and will notice that they have some consistent weaknesses. And then those things will either be patched by like collecting human feedback or like creating environments. Yeah, maybe maybe you saw what I think about this. how much of the lineage we roll back and then let self play from there. I think in the limit, you're picturing just giving them like a GPU and maybe neural nets or something and saying like okay figure out how to train a model to like do these particular tasks like the way it currently works is like we go up to the very like edge of the lineage and say okay like here are the bugs like you know Anthropica's found in their training stack in the last few months we'll turn those into environments like you need to train to get better on the frontier and so you're obviously
[30:00] in all the previous history of the lineage, but you could imagine a world in which you roll back to before GRPO or something, and then you have environments which try to get it to discover the best form to RL models on, and then maybe roll further and further back. But I think we will be still so compute bottlenecked that people will just keep staying at the frontier and diffing essentially the bugs and whatever improvements they found since the last model version, turning those into training environments. Which is also really good for having non-steal new data between model generations. Again, this is basically continual learning within the AI lab of distilling the last three months of AI research progress through environments and like RLHF type stuff back into the model itself. And it is distilling, right? And that's maybe why some of us feel like it's asymptotic is like you're always like just trying to get the last three months of progress. And that progress is being contributed to by AI, of course, but it also still has humans in the loop. And it feels like, you know, you're just constantly inching closer and closer to what the human researchers are like finding capable of doing. I mean the one thing I will say though is like obviously if you're just distilling on like trajectories you can never go above it but environments can go quite a far way above what a human can do like it's very easy to design an environment that like no human can solve but the AI can obviously still try and solve it and so that would be the path to like go ahead of just like what the human AI research is do you have like an example of like in terms of RSI or like, you know, working on a speedrun, but doing it even faster than a human speedrunner. I mean, I feel like in AI research especially, it's very easy to define goals, which you could say the loss needs to be 1.3 or something, and no human can get there now. But that's a very extremely measurable, verifiable task. And if the AI gets there, then great. Right on a building, like a hundred million parameter model that beats Minecraft. That's maybe too easy, but it beats a much more complicated game or something. Isn't it crazy that 100 million prominent models have been in Minecraft, we're calling that too easy? Imagine if you said that five years ago. I would say a lot of research is not exactly like that, though, where it's like hill climbing on a well-defined goal. It's sort of more like, here's an intuition we have about some way models should be better, and then we also have... some idea for an algorithm that seems to go a little bit in this direction. So let's come up with a task that is sort of designed to show signs of life on this approach and like see if we get those signs of life. And then if we do, we can make successively more realistic versions of the task. Right. It's like a lot more guided by intuition. And then the inner loop is to elicit the, or make tests for that intuition rather than like, the test itself leading to the insight. Right. Like you're not directly optimizing for the eventual objective you care about or the practical like production objective. It's sort of you're relaxing your objective a little bit. You're saying, yeah, let's relax on the realism axis a little bit and find some methods that actually work and then like then try to get back to realism later after the method matures a little bit. Yeah, yeah, yeah. And then there's also like more, there's research that's more oriented towards explaining things and like developing a theory or a sort of, yeah, often we don't have like mathematical theories in machine learning that are that predictive. But we have like a lot of like more informal theories for what's going on. Yeah, I mean, like presumably the models will be trained on like some combination of all of these tasks and like some will be very easily verifiable. like does this look reasonable and then you will set the hope would be that like these would all generalize to like these much sort of hard sort of more vague fuzzy kind of tasks and like it probably will to some extent whether it generalizes enough that like we could the loop can become like self-sealing without humans being in the loop at all it's like unclear yeah yeah maybe taking a step back here's what i here's what it seems to me that the plan uh for ai research going forward is and you tell me if you think it's going to work or if you agree with this characterization So the bet is that we will scale up our LVR training across millions of diverse environments, across hundreds of different kinds of domains. And what will emerge at the other end is an agent which has learned these basic skills or less than basic skills around being persistent, being able to triage information in context, eventually having end-to-end optimization of working with other agents and things like that. efficient within the context. You've done research on how you actually scale up in context learning to make it arbitrarily long, but you just keep scaling it up. And so what comes out the other end will be something that basically functions like a drop in remote worker over the course of a week or a month. First of all, do you agree that that is a bet the labs are making? And second, is it is that enough? Like basically learning how to learn within the simulacra within a data center? and then been getting deployed into the real world.
[35:00] But not actually like learning from real world deployment, only learning these meta skills from the simulated environments in the data center. I think it's now hard to separate out how much of the lab's effort is going towards direct RSI versus making generally intelligent models that they can continue to deploy to collect revenue to fund the next big training run. I think for the latter, yes, that's probably just the bet they're making. And it's very clear the pattern of where these environments are going over the last few years. Anthropics lineage of environments is a very clear example of this. going to get really, really good at that. And then the task horizon that we've got from coding, which is probably the lowest hanging fruit in terms of data available on the internet to create environments, their own internal stuff that they can turn into environments. Then we're going to generalize, we're going to go after finance next. And literally just so much Excel data and all that sort of stuff in the RL training. And then it's PowerPoints, it's this long tail of the working economy. And that seemed to work really well. And a lot of the other labs and things, even the And from that, when I had Dario on the podcast, the thing I asked him was, if you truly expect models, which will be human-like in their ability to learn on the job. Why would you try to bake in all these things? skills of like working with PowerPoint or something? Wouldn't you just expect the model to be able to pick that up on while it's deployed? And so, yeah, there's multiple different explanations. One is just that this is we expect models to get there soon, but they're not there yet. So why not amortize these skills into the model training? Another is that we're not concentrated on making it really good at widely deployed work. We just want it really good at RSI. And this is just like a way for us to get revenue actually really good at doing RSI development, and then once the singularity happens, the thing that comes out the other end, we'll be really good at all the things which seem like bottlenecks to the current generation of models. Yeah, John, I don't know if you have a taste on like what, how much I construe why there is so much task-specific knowledge in these models, if the path is like this kind of generalization. Yeah. I mean, if the models were good enough at learning in context, then in theory, you wouldn't need to train them on finance. They would just be able to figure out, read all the books on the fly and figure out how to do everything in the appropriate jurisdiction. Yeah. Yeah, and you could argue that you need to do a lot of this domain-specific training just to make them more efficient. So even if they were smart enough to figure this out on the fly, you still might want to do a bunch of RL and bake all these intuitions into the weights so the model would be more efficient at runtime. Yeah, I'd say in practice, it does seem like model providers are going domain by domain and trying to strengthen the models in the highest value domain. And I'd say that that's one of the answers to why the models have gotten so much better. It's just because the model providers have covered a lot of the high value domains and the most common types of skills. I mean, I think another thing is just like it's not that expensive to do both at the same time, right? Because like the models are massive. They can easily afford in terms of their parameters to like learn everything. There is likely some transfer. Even if finance is not specifically, the information is important for RSI. Just the general meta-learning of how to figure out what's important, how to have taste, how to do long-horizon work is potentially generalizable. There's not that much RSI data in the world as well. It's kind of hard to generate, and that requires a lot of effort. So if you can amortize in this other data, get some transfer from it, you already have masses of compute and masses of parameters based. So why not do that as well as obviously the direct commercial incentive of selling your model? That makes sense. add that um i mean there's one question about whether um this uh current paradigm uh of doing like sim to real uh will be the dominant one forever so basically you you look at what the real world tasks uh are like and then you try to create a bunch of environments that can be simulated like in the data center and uh you can do rl on them and i think um obviously this has been very successful successful but it has a lot of weaknesses uh because a lot of things are just especially if they involve like interacting with a bunch of humans in real time. Yeah, so there's some question about like whether Sim2Real will be the dominant framework forever. I think Sim2Real has to be the dominant framework where like sample efficiency is kind of low because like right now you need like, you know, thousands and thousands of interactions with the humans and no human is going to sit there and like deal with this, basically be in the loop of RL training. And so like we kind of have to simulate that now to like get the samples you need. from deployment to become a much bigger part of it. There are also other things you could do, like you can learn off policy, so you can take all the traces and even without re-simulating everything you can potentially
[40:00] learn something from them. JaneStreet just launched a new competition, and it's their most ambitious one yet: Design a protocol emulator ASIC. Basically, if you have a chip that you want to test, you can connect it to this ASIC, and then this ASIC will simulate realistic traffic. That way, you can see how the chip responds without having to plug it into a live system. JaneStreet is looking for flexible, general-purpose designs, not single protocol emulators. When I was chatting with them, they suggested that I start off by trying to implement what are apparently and I2C. Jane Street also mentioned that they hoped that more ambitious designs will also tackle low-speed USB and Ethernet and any other protocols that flex your chip's specific architecture. Importantly, your design should be reprogrammable rather than smashing a bunch of specific protocols onto a chip. If a new protocol comes out after your ASIC is taped out, your chip still needs to be able to handle it. How exactly it does it is up to you, but there is one hard constraint. Your 130 nanometer process node. That's because Jane Street will pay to tape out the most novel submissions and send the physical copies to the winners. The competition is open till January 18th, 2027, and working in teams is highly encouraged. Go to jainestreet.com/thorkesh to download the template code and get started. I want to ask more about this, because it's sort of weird that... You have 50% of compute that's spent on inference that is not directly helping the model become better. One of the key advantages you'd expect eventually digital minds to have is, unlike a human who gets to have 50 years of real world experience, a model will get to, through all its instances, will get to experience, I don't know, millions of years of deployment across all kinds of economically relevant work in the economy. And right now, that data is just not, in a meaningful sense, helping the model get better. so obvious that eventually models should be able to learn from this data and once they do you would have something that almost feels like a widely deployed intelligence explosion because the model is assimilating so much information um across all these deployed instances but when do you expect this kind of hive mind kind of crazy shit to be start happening i think broadly like at a very basic level this is already happening right like just in the next generation of models so like right now you can always take your deployment data and put this in the pre-trained or the mid-trained of like future models especially if you do like some kind of filtering or some kind of like judgment or like recent synthesization of that. How much do you think that explains the generation over generation improvement? I think it explains like quite a bit. I mean, especially like, I mean, this is, you know, I don't know whether the labs do this because, you know, theoretically, they claim not to train on people's data. But like the Chinese 100% do. And like, they definitely get this advantage, both like obviously deploying. This is basically what distillation is. Like they take out the models, they get some of their like deployment data, they get some fraction of that by like pinging the model. And then they train their next generation of models on it. And they can suddenly do it on their own models as well. Like there's no reason not to whatsoever. I completely agree. if you zoom out far enough this is like definitely happening like you're picturing this like and we're all picturing this is like what continual learning like the holy grail is is like this very very organic like live loop of like an individual model like getting an experience and like live updating on the spot and learning from that and like a lot of things break when you like zoom into that level of granularity but like yeah the big labs are doing this like the closed the closed models are doing this there's also early signs of life of like people using open source models doing this at a much faster cadence so yeah. A good example is probably Composer. You have some sort of model and you are able to, or Harvey's doing the same thing with legal agents. It is getting very specific environments from the data that you have for that particular task and things that users are complaining about. And all the feedback that you're somehow extracting from your specific deployments. And a lot of these companies have the advantage over the big labs in that they can use this data really, really well. train of Kimi K3. They'll go deploy it. They might do some online learning as well, like Composer did online, basically like Reinforce for a long time. Um, So yeah, there's still a human in the loop. There's still a human saying, okay, these are the signals we care about. Here's how we're going to create environments from the data that we have. And there's still a longer cadence than maybe the one that you're thinking of. But like... It really is happening. And eventually that loop will become faster and faster. I mean, the composer thing is interesting because this is where the model in Cursor, people press tab or they don't press tab. on the next completion that the model suggests. And based on that, every single day, Composer gets better at predicting the next. So that was the old tab model. They actually did the same thing for the actual, not just the tab model, but the actual generative model. Oh, that's interesting. And it's hard because when you do online reinforcement learning, you don't have groups, right? You just have one user saying one thing and then you get one rollout. And so you have a big variance reduction problem. And Kurser's kind of fuzzy answer to this was like, oh, you know, we have very good heuristics, which are able to estimate how much better than average like this response was or how much worse than average this response was. And then they would do like this big, you know, like reinforce
[45:00] Update. And then their solution to whether it got worse or not was if it improved on CursorBench, they would deploy the new model every five hours. And if it didn't, they would throw that version out. Interesting. Yeah, I think your biggest problem is actually just not knowing what the reward function should be from natural data. And if you use some kind of superficial signal, like did they accept the code, the edit, that might get reward hacked in some way. But this seems like a bigger issue with the Sim2Real thing, where... The longer and longer horizon tasks get, the harder they are to simulate within a data center. It seems to me already, potentially, at least even in coding, we're getting to the point where there's not some year-long coding task that doesn't eventually require you to talk to a client or interact with the company or interact with users. And if you think about the gamut of things we would want AI to be capable at, you want eventually super intelligence to be able to run a business. or like start a new business and make it profitable, or like have a profitable day trading in the markets, or like win a court case. And these are all things which are very hard to simulate in a data center. Like an inherent part of the learning there is interacting with the real world. And so maybe, maybe they may have to learn how to get better at these things from like the transfer between SIM to real. But alternatively, maybe you do need weight updates from these kinds of interactions in order to get better at them. And then if that is the case, if transfer isn't strong enough and you do need weight updates, then the fact that the models are quite sample inefficient is... is like maybe a deeper problem and the reason i'm curious about this I feel like by default, I don't see how you don't get some kind of crazy recursive self-improvement within the next 10 years. But the one reason why that might not happen is in terms of like weight updates, the sample efficiency of weight updates, they just seem way far behind humans, right? Like plausibly million fold behind humans in terms of how much data a human sees from birth to adulthood versus how much a model sees from, you know, like cold start to like finish finishing training. Is there going to be a good transfer between simulations and extremely long horizon, really complicated real shit that we want the EIs to do in the real world? And if not, does that really mean that the lack of sample efficiency in these models comes to bite us? I think maybe the way I'd break down the two types of tasks in which models get good and models will still continue to struggle is whether the task is cumulative or you have this non-stationary distribution you have to keep learning and relitigating a bunch of stuff. So maybe an example of a cumulative task might be RSI. It's theoretically possible to maybe have less than a million token Python file, which from scratch trains a model that is capable of recursive self-improvement. and send that you hold, like if it's true that for RSI we don't need to discover a new attention variant or whatever, like once you've discovered attention and then once you've discovered a mixture of experts and once you've discovered GRPO, you just add that to the training stack and like That's there. And a good example of this is 5.6 sold. training 5.6 terror whichever one open ai told us to train like it didn't have to go back and discover attention like it basically probably would have called a bunch of like scripts which is like pre-training.sh and post-training.sh and just did that so like that's an example of like a cumulative task i think the real world and the reason like people are thinking so much about like continual learning is it's it's not really a cumulative task like imagine like in a law firm you have an agent acting as a legal associate like that's a very non-stationary distribution between all the important people at that company, which are also changing all the time. You have all these implicit ways about how things are done, where to find information, et cetera. That's not as clean of an example of a cumulative task like RSI is. So I think that there will be this breakdown between tasks, but if the labs realize that and they do believe that RSI is cumulative in the sense that we don't need to go back and discover some brand new architecture or whatever, then maybe more and more effort and compute gets focused on that versus the... It's so unfortunate that RSI happened to be easier we all. Yeah, I don't know if you guys have thoughts on this. Yeah, I would say today's models are weaker than humans in a lot of different ways, and some of them might have to do with sample efficiency in a certain regime. I mean, in some regimes, models are very sample efficient, like learning in context, but then there might be some medium-length regime where they're less sample efficient because humans can do some kind of weight update more efficiently than models. So I think being less sample efficient in certain regimes might be... one of the sources of weakness, but then I think there are other sources of weaknesses that are completely different than that. For example, having lower diversity of thought than humans or being bad at certain kinds of long horizon judgments. I mean, I think a lot of what people call taste is something about
[50:00] behavior that works in the long run and that people have realized works in the long run. Not everything, but like some some aspect of taste, like especially for something like software engineering, like I think a lot of taste is like, what are the systems that are going to be maintainable and work well in the long run of this project? So, yeah, I think the weaknesses of humans, which limit, you know, RSI, along with other things, there's a variety of them, and some of them are related to sample efficiency and some of them aren't. Maybe an interesting thought experiment is like... if you were able to give a model like a context window of, I don't know, a trillion tokens or whatever you would have needed to fit in like your experience prior to like, let's say, RLHF. And it's got all that experience in the context window and it has the same sample efficiency and in context learning ability as it does at a million tokens. Do you think taste is then solved? Would it be able to make the same judgments that you did? Or is there something fundamentally missing apart from just a longer context window with the same sample efficiency? Yeah. I mean, it would have to be trained to learn from that context. So, uh, I'm not sure. Yeah, either it would have to be trained to learn the right update to make from that context or just like dump it all in like your whole life, like research experience. I mean, like you still need the data to train it along context. Like even if you could theoretically get like a trillion context, you would need a trillion lengths of data to train it. Like right now, you have like I'm just asking if you had that. from like shorter horizon episodes. And like, I feel like there's no obvious reason it's super long because like humans somehow develop taste with not having many long episodes. Like we don't live to be like 10,000. We have like, you know, we develop pretty quickly, right? And so like, you know, if you think about like, even like in a PhD, the difference between like a first year PhD student and like a final like postdoc or something, that's like five years maybe. And they've only done like maybe like 10, 50 research projects in total, but somehow they develop taste quite quickly from like a relatively short succession of like small things. possible to develop it like that. The AI obviously will have vastly more experience in which to develop taste to meta-learn it. And then it's like, how well does that generalize to really long-horizon things is, I think, the question, which I think is really unsolved at this point. We don't know. Going back to this question, Eventually, there should be a regime where AIs are learning a ton from each individual instance of deployment that they have. Well, currently, you could say there's a meta fuzzy process by which models do improve for deployment, but I feel like it's a very weak feedback loop. Do you see this around the horizon where there's this hive mind kind of learning that's very rapid? And if so, how exactly does it happen? Actually, I would say that around will we get a hive mind that learns from all of its deployment experience? I mean, a big part of that is actually about incentives rather than being a technical question. So like companies aren't going to want to have the model provider learn from all of their deployment because that might just reduce the advantage of their business. Thank you. economics of this will pressure not necessarily weight updates to one big like common shared model, but like kind of like modules that get subbed in. So a very obvious example, this is a law, but it might be something else like, you know, there's been a lot of work to try and fit like an arbitrary context length into a fixed size. Like this is all the linear tension stuff and all that sort of stuff and like cartridges, which are essentially KVCAs trained to be very, very compressed KVCAs to fit in a lot of information. That's another example of something that companies may be willing to sign up for if that gets subbed into the model, and it's not actually changing the base underlying model itself. There's many different versions of learning from your data in real time, and the latter ones are not really helping the big labs because they are just these modules. But I think the economic pressure will force the labs to go down that path first before they can embark on this. or whatnot, you can still just take all these traces and just distill this, dump this to the pre-training of your next generation of models. It may be a more indirect form of learning that the big labs are getting, and that's obviously still really valuable to them, but I can't imagine a world in which we start off with we're going to just directly train this one big model on all the exact data that we're getting. No, I think it will definitely go through stages, because this is assuming there's one discontinuous event where suddenly we fix weight updates continuously, and in practice, I think it's much more likely to be like, the cartridges and stuff allow you to specialize in deployment, you generate traces, you put that in your model, like three months later you come out with a model which is better at this stuff, you specialize it again, you like consolidate it again, and then eventually we'll just like make this loop faster and faster, so instead of like every three months we release the model, now it's like every week and then every like day and then every hour, at which point we basically have always resolved it. Yeah, and I think this is
[55:00] point as well because you asked like kind of how far off the current paradigm we are from being able to do this I think we've we've done a bit of research to this and people have done a lot of At a really large scale, when you wash out enough noise and you have large enough patches, this out-of-loop process of putting data into mid-training, creating our own environments, it does work in some sort of continual learning regime. But the problem is when you zoom in close enough at a micro level, it's like... I've got one model and I'm trying to update it again for like a law firm or something. And I'm trying to do that very continuously, like with a relatively small amount of data. Like all the methods kind of break down a bit. So like... If I SFT the model on just, like, you know, successful traces, off policy, on policy, like, eventually in the very iterative regime, like, when you're doing, like, you know, hundreds of these micro-updates, you see both catastrophic forgetting, you see forgetting of, like, previous information I've learned on top of the base model that was much earlier on, and I see degradation of general, like, use, general capabilities. You know, on-policy distillation seems to, like, push this horizon out a little bit, but it still eventually succumbs to the same thing. RL is not very good at like, it is good at like getting capabilities in, but it's not as good as getting like knowledge in. And like just this very explicit knowledge of like, oh, OK, like this person does this at this law firm. And like this is a very specific process we find. And you have to pour in a lot of compute to create the right environments to get the knowledge. Do you think that the fundamental issue here, why you get worse at these other skills or there's forgetting and stuff, do you think it's fundamentally an issue of capacity or it's an issue of techniques? can be like way too destructive. Like the reason RL is so nice is because like, yeah, it changes a very, very small amount about the model and there's like a lot of evidence for why this is the case. And so like it kind of just like tweaks it in this very, very, very small like loss value to like get it into the right point. But that also then limits what you can do with RL, like how much you can actually change the model. Versus are you saying like the reason this isn't the winner take all potentially is that it's just like very hard to distill that much information into the base model without ruining something in an interview. Like it's easy to distill it into like a different base model. Like this is where I think it's mostly technique. It's not like it's definitely not like just like there isn't capacity. Like if you had some modeling with all this data and you take like nearly the same size model and pre-trained it from scratch with like all of this stuff in mid-training, it will be better. And I think that's a lot of what's happening today. It's just like getting all the data from the old model and like training a new model from scratch. And this is exactly as Charlie was saying, like some combination like plasticity and like catastrophic forgetting. And like, you know, if you just naively train on like non-stationary data because you're adding new data as you go, basically this is messing with the data distribution. So like the old stuff is just forgotten. And we don't really have good methods to like stop that from happening. And maybe in the limit, you're like just bottlenecked by retraining the model from scratch with all this information. Yes. Which, of course, is like very expensive. Like training model from scratch is expensive. But you're going to do that anyways. I mean, not necessarily. I mean, maybe eventually, if you have continual learning, you never train a new model. You just have a model, and it keeps learning and expanding. But there might be some deep technical reason why that's very difficult, because of these plasticity laws. I mean, that's the question. I think we have pushed back how much from scratch we need to do. It is definitely possible now to take the pre-trained base and do very good mid-training on top of that, kind of continuously, plus some RL from different checkpoints that are later on in the training. couple of very small updates and like iteratively like never lose it but isn't this like i'm a bit confused because isn't this literally what happens during training or during post training or something you just have like you have a model that's already gone through so much training and then you like distill some fork that's been further rled or something isn't isn't that literally what happens and like it's still at a large enough scale i think that you're washing out a lot of like the noise and like you're not just focused on one distribution which as baron said is like you know that is now a very if you're just like focusing on one task right like that's true and i In the eventual regime, you'd be doing... I don't know, there's billions of deployed instances. You're like doing You're learning from all of them at once, and so hopefully there are some... washing out of noise and stuff from that, right? Maybe that's good, yeah. Yeah, I mean, I think, like, definitely, as I was saying, like, you can do continual mid-training for, like, a long time and you can, like, roll back to a checkpoint, give a new mid-training data. But at the same time, like, you can't do this, like, indefinitely. Like, if you just keep continual mid-training the same base forever, it just, like... get, it does, it's sort of asymptote at some point, like you can't just learn new stuff in that base. And this is why people end up training new bases, like otherwise you would just keep them in training the same base forever. Whenever I finish recording an interview, I immediately brain dump all my thoughts into Slack. Things like, what was most interesting, and what should get cut. This ensures that my editors have all the context they need to start editing the episode. But it's not like these brain dumps have any clear timestamps, and my unedited recordings are many hours long. It can take a ton of editor time to even find the exact moments that I was referencing. So we decided to try adding a GrokBot producer to our chat. And now whenever one of my editors posts a rough
[60:00] - Out of the episode, GrofBot opens the transcript on its own computer and starts working, usually before I've even seen the message. It takes the notes that I dropped into Slack, and it highlights the relevant snippets in the transcript. It also uses a big taste file that I've compiled with all my preferences, so it can suggest potential edits. And when it's done, it sends me its top clip candidates so that I can review everything from my phone. This has worked really well. Being able to send informal messages like I'm texting my editor, and then having the transcript immediately reflect my preferences has just been so helpful. Try GrokBot yourself at x.ai.bot. Okay, let's talk a bit about data now. So I'm generally interested in this question of how much of AI progress is just explained by data progress. It doesn't mean it will be necessarily hard to automate, but that's a separate question. So is there some... data distribution, which if you're trained... current architectures on would result in a super intelligence. that totally dominates human experts across every single field. Are we talking about like, pre-training plus, post-training, like environments as well. I think the existence of this is obvious. It's just whether we can create the right environment. Yeah, and in the trivial case, we could just train it to output the Python file, which trains the actual super intelligence. Just have them memorize in the way it's... Yes, there's probably a ladder of RL environments... that is possible to construct such that you would get a AI researcher which is at least as good as a human researcher but the effort to climb each successive rung grows like kind of exponentially um and that's going to be the two things that you have to trade off against as to whether like like how fast we're going to hit like that final rung where it's where it's better um I think that's fairly clear. And I think there's like, you know, we're still relatively early in like our environment creation. Like there's a lot of asymmetries that we exploit in order to create good environments. So one of those asymmetries, which we've talked about before, is like there's environments where it's easier to go backwards and forwards. And like what I mean by that is like it's very easy to define this like complex data generating process. And this is like this kind of latent variable you keep hidden from the model. You can generate like arbitrarily complex like environments. And the model has to do a lot of like irreducible like token spend. and irreducible work to figure out what that data generating process was. There's asymmetries in terms of, you can inject information from the real world. So Anthropic finds a bug through tens of thousands of human and LMLs combined, and turn that into a very, very neat environment, which in a single LMLs is theoretically fine within a few million tokens. So there's all these asymmetries which we're cherry-picking, and we're counting on this task horizon generalization. But I think, yeah, again, there's just going to hit diminishing returns at some point. returns and how hard it is to create these environments in the first place, like coming up with them because you can't necessarily just like... have these really, like, these processes where it's easier to go backwards than forwards. Like, you actually have to sit down and construct, like, something that looks, like, with humans, like, you know, a long enough time horizon, like, it's going to be a really complex task to create. And then there's also going to be, like, the compute and time bottlenecks for the agent to actually do those tasks. So, like, I think you're just going to start seeing this, like, curve to flatten out. I saw something about how someone fine-tuned the Taki model, which is only trained And it did better than Claude 3 Opus on Sweebench. So this model that has no knowledge of code whatsoever can be fine-tuned on a moderate amount of data and behave better as a coding agent than this much larger pre-trained model is pretty crazy. weak model. But a counter example to like that kind of is there was a paper recently where they trained it up to like fifth grade maths. Uh-huh. Um, and like also like primary school like English and stuff so it was like a decent language model. And they tried to RL it to do like, you know, late high school and college maths. And the gap was just too large, like they couldn't get it to climb at all. Like, but if you did like successive rungs of like, you know, you seven maths and then you're eight maths and so on, like, you could obviously climb to year 12. So again, it's just like, what is the distance between the rungs on those letters and how hard is it to create? Yeah, and this just comes back to the RL signal problem. RL is not very good at exploring right now. And so if the model can't get in 128 rollouts, it's very unlikely to get signal to progress. And this is why in RL we need curricula, whereas in pre-training we don't, because that's not a problem for pre-training at all. Yeah, and again, pre-training data is different to post-training data. you on like yeah humans will be involved less and less but that doesn't change the fact that you're bottlenecked on like how much signal you can extract from the real world so like there's a lot of signal in the world and that's true like you know there's people doing like spreadsheet tasks there's people doing like legal tasks and all this sort of stuff but you know the capability frontier of where the models are
[65:00] Now, like how many bits in the world are actually like really relevant to like improving the the model's capabilities like you know how many new maths problems are being solved that like couldn't they're just beyond the reach or grasp of the current models like how many new coding problems are being created or solved that are beyond the reach of the current models and like i think that's why the diminishing returns kicks in because like even the world as a whole is not giving you the bits that are useful for tipping you into the next like basin of capability yeah i i totally agree with this it's like really a question like where the signal is coming from and so like the signal doesn't you know in pre-training the signal is like already in you care about in pre-training. The problem is, it's not just getting signal at all, it's filtering out all the noise that exists. And that's quite an automatable process. But as the models get better, as we enter mid-training and post-training, the signal just doesn't exist anywhere in the original data we have. No amount of filtering will get the... There's no hidden proof of the Millennium Prize problem sitting in Common Core. We can just filter until we see it. And so at that point, you have to get bits some other way, either from humans directly asking them to write out their environment should be created, what the objectives of these environments are, or some kind of training on the human data that exists in deployment. You have to get the bits from somewhere. Yeah, yeah. There's a question of how much of the progress in pre-shirting is being driven by data. I did this investigation with Jerry Hahn, who's a student at Princeton, where we basically trained all the... recipes from 2019 till now, pairwise with all the data sets from 2019 to now. So you train like GPT-2 on the newest data set like Ultra Fine Web and you train Delphi, which is the newest training recipe or the open source training recipe on like the pile or some old data set and you do like the whole grid and you see how. the getting to some level of capabilities, how much less compute does it take across this grid? And you see that The data seems to explain 9x of a compute efficiency gain. But the architecture improvements explains like a 3x compute efficiency gain at a very small scale. And so to the extent that that is true at large scale, that most of the pre-training computer efficiency gains are coming from better data. How much can that continue? Can you keep just filtering data more and more and building more and more synthetic data until... Yeah. Do you have a sense of how much this kind of retraining progress can continue? I think my prior is that like, Again, the low-hanging fruit is somewhat exhausted. We got the internet as this big block, and it's not like the internet is necessarily growing at the same level. like rates, all the useful stuff on the internet is growing at the same rate. So like, we've probably got like a bunch of like, 0.1% loss drops to go, but like, not definitely not as many as have currently occurred. Yeah. But like, that's also really interesting that like, you know, you find this like, what, cumulative, like, 27 times improvement across both. I think, like, it was Epoch or someone who estimated, like, three times a year since 2019, which would imply something like, you know, three to the seven, like over 2000, like times improvement. So like, where's that missing? You know, a hundred times or whatever coming from, like that probably gives you a good signal of like how much of this is like post-training. I think the explanation has to be that a lot of the computer efficiency gains are scale dependent. and we were studying at extremely small scale. And that raises the question of, Do the data compute efficiency gains... or the algorithmic compute efficiency gains have more scale dependence. I don't know if you guys were prior on that. We just didn't have enough computer to investigate that question. I mean, like, just naively, right? Like... The theoretically the scale dependence of the architecture is like fairly well known. Yeah. And like, you can fit a straight line to it. Whereas I would have no idea how to do that for combining pre-training plus post-training data and mid-training data. I feel like data is actually more important with scale. I feel like architectures are kind of like a one-time... I feel like combining... Saying just an X percent efficiency game is kind of misleading because what an architecture does is let you reach a qualitatively new regime which you couldn't reach with the old architecture. And then within that regime, obviously, the data is the prime within determining it. But if we say it didn't have... we wouldn't be able to do like a million it would be like ridiculous expenses to a million context and then like because of that we couldn't we can never use the data which is like actually at a million context and so we couldn't get these capabilities even though like if you just do a naive like how much does this do at like 2k context where the architecture isn't unlocking anything then like the data you know that will look much more important than in some sense it is right it's unclear to me that these things are like really just like multiplicative gains in this way. Hmm. I see. Sorry, but then what is the takeaway for the scale dependence of data? So I mean, on scale dependence, I think a lot of the mid-training and post-training data we have now actually gets better with scale. Because a lot of it, the very long context horizon environment stuff, really requires big models to be able to make use of them. And if you try and train your 100 million parameter model on three bench traces, it's not going to get anywhere. It's not going to show you the same kind of improvement that you would get if you train an actual sensible system.
[70:00] model on it. Yeah. And it's hard as well now because so many of the architecture changes like you look at like Kimi for instance like or DeepSeek, they're doing these architectural modifications with not just like dropping the pre-training loss in mind, but like, for instance, how the models are going to be used in the real world. So like the inference efficiency, like having some form of compressed attention in the deep-seq models is not necessarily geared around, you know, this is fundamentally like a pre-training improvement. It's just like, okay, we're considering how the models are going to be used. Right, right, right. One question I'm curious about to understand the future is how parameter scaling will go as we're getting into more of a RL heavy regime. Like, I don't know. I don't know how fast historically you can look at sort of open source architectures and see how fast. parameters have been scaling and maybe it's like roughly 2x every year are for frontier open source models. And to the extent that like even frontier closed source models have like 100B or 200B active parameters. Do you think that like keeps 2Xing year over year or another RL regime where you also want to conserve compute on rollouts that and also maybe there is like a threshold effect where you have enough capacity and at that point does increasing parameters arbitrarily doesn't matter as much do you guys have a sense of In 2030, how many active parameters will a frontier model have? Yeah, I think for the next few years, we're going to be like, because we're so focused on doing longer and longer projects. horizon rollouts for RL where like inference efficiency matters a lot. Yeah. Um, it feels like the models aren't necessarily saturated on their ability to do that, where the bottleneck is still the environments. And so we might see a little bit of plateau. I have a feeling that Mythos and the GPT models are much smaller than the 10 trillion parameter range that people are talking about. Even just naively comparing to open source models, you can probably back out that conclusion. So yeah, probably for the next few years, I wouldn't imagine a huge growth in the number of parameters. But again, there's so many different things to trade off here. You decide the size of your model based on how much pre-trained data you have and then the difficulty of the RL environments that you've got to train on. And you ideally want to get to the optimal point where you can get a decent pass at one or something on the hardest environments you have, and it wouldn't make sense to make a bigger model pass there because then you're just paying like, much more inference flops when you need to. So there's a lot of inputs to this. It depends on how quickly, you know, like McCore and then in-house these guys can scale up the complexity of the aural environments they're training on. I would expect the models to keep getting bigger just because people are scaling up compute and the GPUs are getting bigger. But I would say exactly how much they get bigger depends a bit on the scaling laws in non-obvious ways. So one thing is that I think like data efficiency is going to be a bigger driver than compute efficiency like the exact architectures people use. Now that we're getting to the regime where we're sort of running low on like high quality pre-training data. So that might affect how sparse you want to make the model. And then I also think we don't understand sparsity that well. And it's like parameters are a different resource than active parameters, but it's, and like sparsity has definitely increased a bit, without bound, there might be some kind of sweet spot. There's an argument that sparsity should make data efficiency worse because you might have to learn the same thing on multiple experts. uh, So that's debatable. So I think we don't I don't think we we have a good enough theory of scaling laws that we we really understand why sparsity is helping and to what how much it'll help and if that'll like plateau at some point at a certain level of sparsity. Can you spell out exactly what the implication of data efficiency would be on. So it sounds like you'd say, well, there should be less sparsity, but what are the other implications on programmer scaling I guess just that the scaling law, you're not necessarily looking for the most, you're not trying to optimize compute affinity. So you have all your choices you can make on the architecture. And each of these gives you a different scaling law. And then, like, traditionally, you would look at some kind of envelope based on compute. So you would look at performance versus compute and take the envelope of, like, the best models, right? But if we're making that decision based on data, so it's like, yeah, we're sort of assuming we can spend a lot of compute, And like we're sort of data is on our X axis instead of instead of. compute, then we just get a different set of optima or a different set of models that are on that frontier. Yeah. And I also don't think that we've necessarily doubled the size of the models every year for the last few years. People have been training one trillion parameter models for at least a few years. There was even an open source one called Falcon, but Liam from Periodic Labs, I think, posted yesterday on Twitter about how an early experiment at OpenAI was training a one trillion parameter model that was very, very sophisticated.
[75:00] Oh yeah, that was what they did before OpenAI. That was like at Google, the Switch Transformer. So it was very, very good at knowledge, but terrible at reasoning because it was so sparse. And so it feels like we've been playing in this... 100 billion to up to 2 trillion parameter range for at least a little bit. And it certainly hasn't been this nice linear increase. I mean, I feel like there's two things. So as Charlie was saying, inference efficiency is super important for RL rollouts. And so this will really push down active parameters quite a lot. And then I think the total parameters really depends a lot on the hardware as well. So you really need to get very high memory bandwidth and VRAM size to actually be able to sell multi-trillion parameter models. And so right now, people are still using a lot of H100s and stuff. And so as everyone moves to GPs and then the rubins will get like more actual, like the ability to scale and like actually serve and like do like large RL inputs at like different, at larger scales. The data question I think is interesting because naively like larger models are much more sample efficient in like the actual data points. And so like even if you're like not saturating the model, it's still better to go bigger because like the models, larger models generalize better and like get to a better loss for the same amount of data. And so right now I think we kind of have a lot of data and like that's not the constraint rather than compute. models, which are very inference efficient. But if computers are no longer at the bottom, it might come back to larger models, which are undersaturated, but they have this generalization ability because they're much larger. If you just look at the basic chinchilla scaling law... And you just maximize out... parameters. It actually decreases the amount of data you need to get to the same loss very little. If you go to infinity on parameters, the amount of data you need, I think, goes on less than 10x, just because of the nature of the power line. We're now on the way too much data side of the Chinchilla laws, right? So right now we overtrain more as we put into Chinchilla, and so we could easily go back to a point to which, as we're running out of data, we move back to the Chinchilla optimum point. We've been a bit on the overtraining, undertraining model side. But surely, even with these new chips that come online and stuff, we're just going to be so compute bottlenecked for the next few years that that won't necessarily be okay. training in inference compute, really. It's like, if you're super-balled-necked on data, not on compute, you should go bigger. If you're super-balled-necked on compute, you should always go smaller. And then like, yeah. But you can also use computer-generated synthetic data, so it's like one of these very hard things to predict. Yeah, I think part of the reason it took people so long to figure out the scaling laws in the first place was that if you don't get all these things right, then you don't get such a clean relationship. And the beautiful straight lines on graphs hide a lot of complexity on how you have to make sure to scale every hyperparameter the right way or parametrize your optimizer in a way that scales and where you don't have to change your hyperparameters as you change the model size. And bugs have their own clean scaling laws as well, right? Like, you know, like with Kaplan, forgetting the cosine annealing thing or like even just like not considering embedding parameters, I think. And so that messed up the estimate at smaller models because embedding parameters are a decent size of the model. Uh, A bit on RL. So I feel like a year ago, a lot of people were making this argument that RL will not be super successful at scaling for models. I think Johnny wrote a research paper where you were pointing out that models learn one bit per episode. when you RL basically learn, did I get the answer right or did I get it wrong? And I wrote some blog posts earlier this year, I was like, it's even worse than that, because when the pass rate is low, and the model is very unlikely to get the answer right, it learns almost nothing at all from an RL episode. But I look at the models today, and they seem pretty smart, and it seems to be the result of scaling up RL. Baron, you had a post, I think, a few weeks ago, where you're trying to explain what's going on. But why has RL been more successful than one would have naively thought? Thank you. I mean, so I think the success of RL comes down to a bunch of different things. So first, I think what is slightly underestimated is actually the mid-training. So an awful lot of what we see as successes of RL actually comes from very, very good mid-training data, which is basically where we're essentially doing pre-training, but on synthetic reasoning data, and the kind of environments that get the model warm started for RL. And so this actually takes the model almost 80% of the way to the final RL checkpoint often. policy. And so this is one of the reasons why it doesn't need as many bits as you would naively think. It doesn't have to learn all of these behaviors from scratch. It needs just a few bits from these episodes, which you do get. And then the other thing that I really point out in my blog is that these bits are actually extremely high signal compared to regular pre-training, which is why you need RL at all versus just SFTing on successful reasoning traces. Because it's exactly the bits about how to get the answer right. Well, there's two things. So yes, one, it's exactly the bits about how to get the answer right. But because in SFT you have a trace, right? You have a bunch of math reasoning and then the answer at the end. The bit is still there, like you're still SFT on the answer token. So that bit is still there. What's important is that the objective
[80:00] ignores all the other bits. So in SFT, you have to try and match the exact reasoning tokens that the model produces. So you're essentially getting too many bits about the exact way this other model you're training on reasons. For RL, you only get the one bit. And that means that that signal is not drowned out in the noise of all the other bits the model has. And so that's what really, it's really super dramatic increase into the signal to noise ratio during training, which is why RL is so dramatically efficient in terms of steps. Any of you guys have thoughts on that? Yeah, there's been so much debate about what RL does to the model versus mid-training or SOT or whatever. And everyone talks about how parser 1 will go up, but parser 256 will go down. Very rare, correct reasoning traces will be downweighted and outweighed by a gradient signal from easier reasoning traces. And I think the simple way to view RL now is that If you have a large enough amount of compute to sample, a large enough group size, such that your probability of getting a bunch of correct answers is past some not insignificant probability, then it will be upweighted. And to Varen's point, basically mid-training and more pre-training, the parser one, the starting point for RL, scales in the log number of pre-training tokens. You can answer very basic questions. empirical research shows that this is what's happening. But then I just look at The models themselves. And I don't know what's happened. Maybe you can give me a sense of what is the basis of the AI progress over the last year. Yeah, maybe it's just upweighting the policies, which we're going to do the correct thinking anyways. But it just seems like qualitatively, the models have gotten so much more capable. Anyways, maybe there's nothing to... There's no inherent contradiction there. But how do we square the relatively small impact this take would imply that RL would have from the actual qualitative capabilities the model seemed to be gaining? So one thing I want to point out here is that it doesn't necessarily imply that RL has a small effect, right? Even if you have a few bits and you only change the parameters a small amount, the actual impact on function space, the model and the input to output mapping can still be super dramatic. space, which is huge. So I don't think it's necessarily the case. It's small amounts of bits, small amounts of RL. Once you're starting from a really good point, it means that you don't have dramatic impacts in behavior, at least not necessarily. I think it comes down to two things. I think the first thing is that everyone was hoping that RL would generalize this reasoning across all these different domains. And I don't think we necessarily got this like horizontal generalization. Just training on math doesn't necessarily make you the greatest coder. You do have to do RL on code environments. I think what we did get, though, is horizon generalization. The models just learned how to use more tokens for longer and still make progress on some sort of task. And so you can train on environments where they get longer and longer and longer and then put them into a completely new environment. And yes, they may not have generalized the reasoning patterns which allow them to do well in that environment, but they've at least generalized is correlated with success. I think there's a paper called EdgeBench. which show that the rate at which models can work for longer is like doubling every three months. And so that's a clear evidence of generalization. And I think the final way to think about it is like in pre-training there's this idea of like quanta so you have this very smooth like pre-training loss curve and when you actually look at what's happening in the model like the model is learning all these like very discreet like tasks and there's like all these like emergent points there's like kind of a phase transition like it didn't have induction heads now it has induction heads and there's like tens of thousands millions probably like hundreds of millions of these things and you average them all together and you get this very like smooth loss curve i think like to us to an extent like a similar thing is happening we will train a model and then RL it, and then the next model iteration of training, we will dump a bunch of these synthetic reasoning traces into the mid-training data. We're hitting all these quanta for all these different tasks. On an individual task level, it may look like a phase transition, and you're suddenly going from a 0.5% pass rate to a 90% pass rate on a particular finance task or Excel task or whatever. But you average all these things together, and plus the horizon generalization, you kind of go, wow, we've got qualitatively better models. - I think a lot of this as well is just like, I think RL does generalize a bit. Suddenly you get some transfer between math and code, or puzzles and math and this kind of stuff. Also just the sheer amount of environments I think the people are targeting is just vastly greater. So before when you tried to do some task, which you do in your daily life, two years ago, the labs wouldn't really care about this. They wouldn't train the model for it. And now it's just so much broader. They have a lot of environments targeting this specific thing. in the conversation we're talking about how are all in the context of causing this entropy collapse or a
[85:00] concentrating probability on solutions the base model were already done and causing relatively sparse updates in the policy. But when I think about... I think there's also another story about RL, which is going back to the Atari games, and then AlphaGo coming up with Move 37, the super creative move, that because it was never initialized on human data, it can think in ways that humans are not even thinking and come up with extremely creative solutions. Yeah, do you have a sense on when we should expect or if we should expect RL... on LLMs to result in things like Move 37, just extreme creativity, even beyond human creativity, because there's just de novo initialization of intelligence? I mean, so a couple of things here. First off, I think that the AlphaGo is using MCTS, which obviously does more exploration and stuff than regular positive gradients. But I kind of also think that RL doesn't necessarily reduce the creativity. face incident. These models were coming up with multiple zero days at a time to break out of the sandbox. And this is clearly some level of move 37 creativity, I think, already, which we just get from the general generalization properties of the LMs. I don't think it's definitely not the case of RLs totally destroying their entropy. especially on Long Horizons. Yeah, I mean, one thing that people call creativity is just solving hard search problems. Uh, and so that's like, uh, like move 37 is obviously an example of that, or like, uh, writing some kind of poem that satisfies a ton of different constraints. Um, so that's something AI is obviously going to be extremely good at, uh, if, if trained for it. Um, then, uh, there's another way in which the models, uh, like, uh, the diversity of their outputs, uh, is a lot lower after RL and they sort of develop these ticks. models seem like they're good at writing when you do some kind of like distributional analysis, you find that like they're reusing, uh, certain themes, uh, like all the time and they're using the same character names all the time. So there's actually, um, it's not like you're getting the same kind of diversity that you get when you, like from human authors, you're sort of getting one really good, uh, like style. So I think that, um, like that kind of diversity has definitely, uh, been like cut down by RL a lot. And in fact, uh, now, yeah, since we were talking about distillation earlier, that's sort of, uh, something, yeah. distilling mostly from Claude that like all the open-weight models write the same way as Claude and use the same like have the same ticks so this seems kind of concerning to me that we're having this like this monoculture emerge yeah again I don't think this is like fundamental to RL as like a method though and same with distillation like even with distillation like you're just training on the data it's like just because your data is not like super broad that doesn't mean like the For instance, the RL entropy collapse is basically due to exploitation of fairly simple verifiers when you don't have a huge diversity of environments. For instance, the writing is presumably graded by some judge, and the judge has some specific ticks, and the model is learning to award hack the judge, and that's why it collapses. But this is really a problem with the judge. It's not a problem with RL in general. Okay, super rapid fire predictions about the future. So I want timelines on the following couple questions. By when do we have models which You can Here's what it feels like to a user. You basically hire them as a drop-in remote worker. for all kinds of white-collar work, not just coding, but... I don't know, video editing, law, paralegal, etc. Like it's like literally an actual remote worker. like full computer use with like literally a month of seamless learning. and operation and executing on complex projects and it required interacting with other people, etc. etc. It's like everything a human worker could do over a month. If you like mandated to use like a browser or whatever, rather than like these, again, the firm setting up the information to be programmatically accessible, like maybe a couple of years. If it's not like browser-based, like you can send Slack messages, you can do all this stuff, but I'd still probably say around a year. Yeah, I mean, I would say maybe like For the full generality, maybe like three years, but I think... To Charlie's point, we will end up with a lot of people making their organizations easier for the AIs to use. And so you get like 80, 90% of the way there before that. Sorry, but the thing that's the diff between one year and three years, there was just literally like... I think there's going to be a long tail of miscellaneous stuff which some human can do, which will take them all quite a while to do. Are you thinking of computer-based stuff or basic cognitive capabilities? I mean...
[90:00] I think this really comes down to a question of like, how quickly can we solve this kind of like online learning and like, whether we can like get like 80-90% of the way there with like compaction and like writing files to yourself and stuff. And like, that's my big uncertainty. I really don't know. And maybe an example of something that I wouldn't be good at is if I have to yell at someone to get something at work or really push someone to get something done. The model isn't just going to do that. It's just going to be too nice. Yeah. I'd say there's a wide variation in quality of human remote workers. So if you try to hire someone off of Upwork to do a software engineering project, there's going to be a huge variation. It's often quite hard to get them to do a good job or pay attention to all the feedback you're getting. And I would guess that in some cases, it'll be worse. The pre-AI version of this was worse than what you can get now from existing AI. Uh, so I think it might end up being a little complicated, uh, cause maybe to some extent we already have this, uh, like for some, like not so high quality of work, but then like, uh, then it's obviously like, we're not, uh, We're not matching human level in certain higher quality forms of work. But I basically agree with Charlie and Barron that maybe we'll have some version of this in a year or so that's okay. And maybe we'll have that form factor and it'll be able to do some things really well, some things not so well. And things will be improving from there. We ship the goalposts based on the very long tail all the time. before of like doing your taxes or something like this year i literally just like told codex to like go get everything i needed to do and send it to the accountant and like there was this massive list of stuff it had to use computers to click through and like download some stuff and it did it was it was like fine yeah so like i don't know a lot of this stuff it can already do. Okay. Give you 10x total productivity uplift. Basically, if it takes you a year to make a breakthrough now, you make a breakthrough every month. I think I would just refuse to give you a scaler on this. Like, like we might already be past that in some like types of work. Like, let's say you're just trying to prove you're trying to do like certain types of math. But for you as researchers trying to make like advance, you know, the state of AI research is how much are like researchers fed up? Yeah. Or uplifted. Somewhere between five and ten years? Oh, really? That's far away. Well, you think it's longer than like for general remote worker? Yeah. Interesting. I think I'm realizing you probably have very different definitions of fully general remote worker. I could have supposed to. Yeah, this is true. Because obviously an AI researcher can be a remote worker. Yeah, I'm picturing normal white color work over the period of a month. I think it starts at a verge a little bit past the months. A very competent white-collar worker, but not necessarily a super creative researcher. I would say like two years. Two years? Yeah. 10x? Okay. How are you, Bert? I can kind of see that, actually. Because, like, it really is just, like... Right now it's already like definitely more than 10x of like coding stuff. And so it's like if it can do even like one or two loops of like experimental feedback, that would actually be massive already. Yeah. So 10x uplift of AI researchers within two years. If you just plug it into a very naive model of AI progress and how much it's coming from researchers and they're like there's like a 10x increase in their productivity. Yeah, you have like radically accelerated pace of AI progress starting two years from now. Yeah, I mean, I think like this will mean that AI progress doesn't get bottlenecked on like AI researchers ability to run like small experiments. It gets bottlenecked on other things. Of course, of course. But it just like happens 10x faster. For sure. Yeah. Which is a huge deal. And that also like helps the next thing, which makes gives you 100x speed up happen sooner, etc. Yeah, I'm happy to stick with longer on that one. And what's like the crux? Uh... My capacity to absorb information and make the base and optimal decision on the next experiment. Makes sense. Yeah, I'm assuming that you can delegate some of this to the AI. So the AI is becoming decent at deciding, it's run this experiment, it's got this result, it runs the next experiment. And then if it can run two or three experiments in a row without crashing, then that is actually a big update. in like uplift and okay final question And AI, which is... which dominates top human experts across every single field. of work that can be done over a computer. So not only a research, but all cognitive work. And not just like short horizon work, but like literally if it takes like three years or something. they also would do better than humans. This is basically just like ASI. Yeah. Okay. I would say like three or four years. The fuck? I mean, that doesn't seem wrong. I would say like AI is obviously getting more attention. So it's like one of the harder things. But it's like...
[95:00] of energy is being put into it and it's also uh like not one of the hardest things for ai because it's like involves a lot of code and and math which models are really good at maybe for things that involve like 3d and like spatial stuff and physical stuff i think uh that that will take a little longer um so um especially if it's not like yeah if it's like mechanical engineering or something and it's not getting like the most attention right now that might take a little because of the nature of the field. And it has to learn that data on the fly. So for example, it has to become superhuman at being an engineer at TSMC or something. So you would have to assume that you can give the AI the same onboarding material. Then something has to be solved about longer horizon learning. Yeah. I'd say 5 to 10. So basically, yeah, you think... automating AI research is like ASI complete or something. Yeah, I think so. Yeah, I think there's so many things in the world which, like, even if you have some sort of memory system external to the model, and even if, like, context length grows a little bit, like... There are just fundamentally things like even if you could research the information or write notes to yourself, like you'd need more than a million tokens or context fields. Yeah, I mean, I kind of agree in the five-year range, at least for the stuff that labs are focusing on. But I think there's going to be a long tail of stuff, which I could theoretically go out and learn about, but no one has bothered to do it. And the computer hasn't been allocated to that, so that might take longer for literally every single human expert. I'm sorry, but by this I also included the ability to learn as fast as a human, a new domain. I mean, I think that's not necessarily necessary, actually, because like the AI will have vastly greater experience than like any human. Right. Thanks so much for doing this, guys. I feel like this was a great format for getting different experts to disagree and debate and discuss things together. It was very productive. Thanks for having us.
