Karpathy describes the current AI transition as a 'decade of agents' rather than a rapid near-term breakthrough, arguing that autonomous deployment requires surmounting difficult cognitive deficits across roughly ten years. (0:48)
Frontier AI mimics human culture via next-token prediction rather than biological evolution, creating models that act like 'savant kids' with encyclopedic factual recall but lacking basic executive cognitive architectures. (7:54)
Outcome-based reinforcement learning remains highly inefficient because broadcasting sparse binary rewards across long reasoning chains reinforces dead ends and forces models to 'suck supervision through a straw.' (42:37)
AI adoption is projected to blend smoothly into historical ~2% annual GDP growth rather than triggering an economic discontinuity, as real-world friction, safety nines, and physical execution limit diffusion speed. (84:31)
Through Eureka Labs, Karpathy aims to develop human capability into a 'Starfleet Academy' model, arguing that friction-free personalized tutoring can prevent civilizational infantilization as autonomous systems advance. (119:16)
Detailed notes
## Background
This episode was published on YouTube by Dwarkesh Patel on October 17, 2025. In the interview, Andrej Karpathy discusses the limits of current reinforcement learning, why AGI is likely to blend into historical 2% GDP growth trends rather than trigger an immediate economic discontinuity, the engineering lessons learned from autonomous driving, and his vision for the future of education.
## TL;DR
Karpathy frames the coming era as a 'decade of agents,' arguing that today's frontier models are 'summoned ghosts' imitating digital culture rather than evolutionary animals endowed with continual learning or consolidated memory. He critiques outcome-based reinforcement learning as a noisy, inefficient shortcut and suggests future architectures will distill down to a 1-billion-parameter cognitive core stripped of rote factual memorization. Pushing back against rapid takeoff theses, he contends that the steep 'march of nines' in real-world deployment and institutional friction will keep macroeconomic growth aligned with historical 2% baselines. Finally, he outlines his work with Eureka Labs, advocating for first-principles pedagogy and personalized tutoring that treats post-AGI learning as cognitive athletics to uplift human capability rather than sidelining it.
## Atmosphere & style
The conversation carries a grounded, intellectually rigorous, and refreshing tone, balancing Patel's macro-theoretical challenges with Karpathy’s deep engineering pragmatism honed at Tesla and OpenAI. It moves fluidly between granular algorithmic breakdowns—such as KV-cache limits, gradient synchronization, and reward hacking—and evocative, memorable metaphors like spherical cows, savant children, and Starfleet Academy. The dynamic feels like an intimate, candid masterclass that demystifies frontier AI hype with technical realism while remaining deeply energized by pedagogical optimism and human potential.
## Bulletin
- [00:48](https://www.youtube.com/watch?v=lXUZvyajciY&t=48s) Karpathy describes the current AI transition as a 'decade of agents' rather than a rapid near-term breakthrough, arguing that autonomous deployment requires surmounting difficult cognitive deficits across roughly ten years.
- [07:54](https://www.youtube.com/watch?v=lXUZvyajciY&t=474s) Frontier AI mimics human culture via next-token prediction rather than biological evolution, creating models that act like 'savant kids' with encyclopedic factual recall but lacking basic executive cognitive architectures.
- [42:37](https://www.youtube.com/watch?v=lXUZvyajciY&t=2557s) Outcome-based reinforcement learning remains highly inefficient because broadcasting sparse binary rewards across long reasoning chains reinforces dead ends and forces models to 'suck supervision through a straw.'
- [01:24:31](https://www.youtube.com/watch?v=lXUZvyajciY&t=5071s) AI adoption is projected to blend smoothly into historical ~2% annual GDP growth rather than triggering an economic discontinuity, as real-world friction, safety nines, and physical execution limit diffusion speed.
- [01:59:16](https://www.youtube.com/watch?v=lXUZvyajciY&t=7156s) Through Eureka Labs, Karpathy aims to develop human capability into a 'Starfleet Academy' model, arguing that friction-free personalized tutoring can prevent civilizational infantilization as autonomous systems advance.
## Key takeaways
- [00:48](https://www.youtube.com/watch?v=lXUZvyajciY&t=48s) The transition to fully autonomous agents will span roughly a decade due to missing cognitive layers, lack of persistent cross-session memory, and imperfect computer interaction.
- [07:54](https://www.youtube.com/watch?v=lXUZvyajciY&t=474s) Current frontier models are built by imitating human digital culture ('summoning ghosts') rather than by evolutionary sensorimotor pre-wiring ('building animals').
- [13:31](https://www.youtube.com/watch?v=lXUZvyajciY&t=811s) Pre-training acts as an engineering shortcut where models conflate accumulating declarative facts with building reasoning circuits, making the isolation of a pure 'cognitive core' a central research priority.
- [17:29](https://www.youtube.com/watch?v=lXUZvyajciY&t=1049s) Transformer parameter weights operate like a lossy, compressed recollection (0.07 bits per token in Llama 3 70B), whereas the KV cache functions as an uncompressed, active working memory.
- [32:03](https://www.youtube.com/watch?v=lXUZvyajciY&t=1923s) AI coding assistants excel at boilerplate web development and inline autocomplete, but break down on dense, bespoke codebases requiring holistic architectural understanding.
- [42:37](https://www.youtube.com/watch?v=lXUZvyajciY&t=2557s) Outcome-based reinforcement learning is an extremely noisy supervisor that indiscriminately upweights irrelevant intermediate steps along successful trajectories.
- [46:07](https://www.youtube.com/watch?v=lXUZvyajciY&t=2767s) Automating reasoning evaluations via LLM judges consistently triggers reward hacking, as adversarial student models exploit blind spots in the evaluator.
- [55:16](https://www.youtube.com/watch?v=lXUZvyajciY&t=3316s) Verbatim memorization acts as a distraction from pattern discovery, suggesting that a future human-like cognitive core could be compressed down to roughly 1 billion parameters when decoupled from internet-scale trivia.
- [01:13:36](https://www.youtube.com/watch?v=lXUZvyajciY&t=4416s) Coding dominates enterprise LLM adoption over general knowledge work because software development possesses established text interfaces, IDEs, and diff-based verification workflows.
- [01:18:24](https://www.youtube.com/watch?v=lXUZvyajciY&t=4704s) The primary societal risk of artificial superintelligence is the gradual, compounding loss of human comprehension and control over interconnected autonomous systems.
- [01:24:31](https://www.youtube.com/watch?v=lXUZvyajciY&t=5071s) AI progress is unlikely to create a vertical discontinuity in macroeconomic growth, continuing instead the multi-century trend of ~2% annual GDP expansion.
- [01:43:43](https://www.youtube.com/watch?v=lXUZvyajciY&t=6223s) Achieving reliable autonomy follows a 'march of the nines' where each successive nine of reliability requires an equal, massive unit of engineering effort.
- [01:49:42](https://www.youtube.com/watch?v=lXUZvyajciY&t=6582s) Waymo and robotaxi deployments frequently pull the future forward at high capital cost while concealing real-time human interventions behind remote teleoperation.
- [02:02:04](https://www.youtube.com/watch?v=lXUZvyajciY&t=7324s) Effective pedagogy relies on first-principles thinking and building steep but climbable ramps to knowledge that maximize student eureka moments.
- [02:18:14](https://www.youtube.com/watch?v=lXUZvyajciY&t=8294s) Micrograd illustrates that neural network optimization can be understood entirely within 100 lines of Python, with GPU kernels and tensor striding serving merely as efficiency concessions.
## Detailed walkthrough
### [00:48](https://www.youtube.com/watch?v=lXUZvyajciY&t=48s) The Decade of Agents, Not the Year of Agents
Dwarkesh Patel opens by questioning Andrej Karpathy on a phrase he used: why describe the current era as the "decade of agents" instead of the "year of agents"? Karpathy explains that his remark was a pushback against widespread over-prediction across the AI sector, where commentators had prematurely declared the current calendar year to be the breakthrough moment for LLM-based autonomous systems. While he uses state-of-the-art tools such as Claude and Codex every day and finds them deeply impressive, he insists that realizing fully functional, autonomous agents will be a slow, grinding ten-year journey: “In my mind, this is more accurately described as the decade of agents.”
To define what an agent actually needs to do, Karpathy uses the benchmark of an everyday knowledge worker or intern. An employer cannot replace a human team member with current LLMs because these models simply fall short across several foundational dimensions:
- They lack sufficient baseline intelligence.
- Their multimodal capabilities remain too limited to smoothly handle realistic workflows.
- They struggle with direct computer use (such as using an operating system via standard mouse and keyboard interfaces).
- They lack continual learning; they cannot absorb an instruction once and retain it permanently across sessions.
When asked why this roadmap should span roughly ten years rather than one or fifty, Karpathy cites his fifteen-year career in machine learning. Having watched multiple technological hype cycles and historical prediction errors firsthand across academia and industry, he sees the remaining obstacles as difficult yet entirely surmountable: “I feel like the problems are tractable, they're surmountable, but they're still difficult. If I just average it out, it just feels like a decade to me.”
---
[04:03](https://www.youtube.com/watch?v=lXUZvyajciY&t=243s) Seismic Shifts: From AlexNet to the Misstep of Game RL
Looking back across his fifteen years in the field, Karpathy reflects on how AI advances through rare seismic disruptions where the entire discipline abruptly pivots. His own entry into deep learning began purely by chance through his proximity to Geoffrey Hinton at the University of Toronto. At the time, Hinton's insistence on training neural networks was considered an eccentric, niche pursuit far outside the mainstream of computer science.
The first major turning point occurred in 2012 when AlexNet won the ImageNet competition, compelling the broader computer science field to drop traditional methods and train neural networks. Yet this initial wave remained narrowly task-specific: researchers focused on isolated challenges like image classification or neural machine translation. Once visual tasks seemed somewhat handled, researchers wondered about building full, embodied agents, which triggered the second major wave around 2013: DeepMind's breakthrough on Atari games using deep reinforcement learning.
OpenAI, which Karpathy joined in its early days, heavily embraced this game-centric paradigm, spending two to four years focused on mastering various game environments. Karpathy personally viewed the game obsession as a strategic dead end:
> “I was always a bit suspicious of games as being this thing that would lead to AGI. Because in my mind, you want something like an accountant or something that's interacting with the real world.”
```mermaid
flowchart LR
accTitle: Evolution of AI Development Paradigms
accDescr: Three phases showing task-specific neural nets, early game and browser RL agents, and modern LLM-grounded agents.
Karpathy's personal project at OpenAI was part of the Universe platform, where he attempted to build an agent that operated web interfaces directly with a keyboard and mouse to perform practical knowledge work. In hindsight, he concedes this initiative failed because it arrived far too early. With neural networks that lacked rich internal world representations, an agent attempting to click and type at random was confronted with an impossibly sparse reward signal: “You're going to burn a forest computing, and you're never going to get something off the ground.” Modern computer-use agents work only because they sit atop large language models trained on massive text corpora, proving that comprehensive foundational representation must precede reinforcement learning on action spaces.
---
[07:54](https://www.youtube.com/watch?v=lXUZvyajciY&t=474s) "We're Summoning Ghosts, Not Building Animals"
Dwarkesh steelmans the pure reinforcement learning philosophy associated with figures like Richard Sutton: animals and human infants are born into the sensory world without textual labels, learning directly from real-time environmental feedback. If biological intelligence emerges without external text scaffolds, why should AGI require internet-scale text pre-training rather than continuous trial-and-error discovery from raw sensory pixels?
Karpathy disputes the animal analogy altogether. Animals are not empty models training via tabula rasa reinforcement learning; they arrive pre-loaded with immense cognitive and behavioral scaffolding sculpted across millions of years by evolution:
- A newborn zebra stands, runs, and tracks its mother within minutes of birth, an astonishingly intricate motor and perceptual feat that cannot be credited to reinforcement learning.
- Biological evolution somehow encodes complex neural network architectures and initial connection weights directly into DNA nucleotide sequences (`ATCG`).
- Much of what observers label as "learning" in young organisms is merely the biological maturation of pre-wired circuits.
Because modern AI practitioners are not running an evolutionary loop over millions of generations, attempting to recreate animal learning mechanisms from scratch is a mistake. Current frontier AI is produced through imitation learning on human culture:
> “In my post, I said we're not building animals. We're building ghosts or spirits or whatever people want to call it, because we're not doing training by evolution. We're doing training by imitation of humans and the data that they've put on the Internet.”
Karpathy argues that true reinforcement learning plays a far smaller role in biological cognitive capability than is commonly claimed. Rather than driving higher-level reasoning or abstract intelligence, biological RL is primarily confined to narrow motor fine-tuning tasks, such as throwing a ball through a hoop. Consequently, current digital LLMs occupy an entirely distinct quadrant of the space of intelligences—ethereal digital ghosts mirroring human thought, rather than evolved animals navigating the physical wild.
[10:53](https://www.youtube.com/watch?v=lXUZvyajciY&t=653s) Pre-Training as “Crappy Evolution”
Dwarkesh Patel pushes back on Andrej Karpathy's analogy between biological evolution and model pre-training. He points out that the human genome contains roughly three gigabytes of data in DNA, which is physically incapable of specifying the location and connection of every single synapse in the human brain. Instead of directly hardcoding weights, evolution discovered an algorithm that drives developmental growth and online lifetime learning. Dwarkesh raises Richard Sutton’s perspective: evolution provides an algorithm to discover knowledge, whereas pre-training directly memorizes the knowledge itself. If pre-training is just packing factual knowledge into weights rather than building learning algorithms, the biological analogy appears to fray.
Karpathy accepts the biological premise—that the weights of a brain are clearly not encoded verbatim in the ATCG base pairs of DNA—but clarifies that his analogy comes from an engineer’s pragmatic mindset rather than a desire to replicate biological organisms from first principles. With our current technology, we cannot simulate hundreds of millions of years of open-ended evolution. Instead, predicting tokens across internet-scale text allows practitioners to bootstrap entities packed with world knowledge and foundational reasoning primitives. As Andrej Karpathy explains: “I don't come at it from the perspective of, let's build animals. I come from it from the perspective of, let's build useful things.” He views pre-training as an imperfect, technologically feasible shortcut to reach a capable baseline from which reinforcement learning and downstream tuning can operate.
- Biological DNA holds around 3 gigabytes of information (roughly 3 billion base pairs), meaning the structural layout of the adult brain must emerge through developmental processes and lifelong adaptation rather than direct parameter storage.
- Karpathy characterizes pre-training as "crappy evolution": a brute-force method to summon "ghosts" and "spirit-like entities" using internet documents because we lack the ability to run genuine evolutionary simulations.
[13:31](https://www.youtube.com/watch?v=lXUZvyajciY&t=811s) Isolating the “Cognitive Core” and In-Context Gradient Descent
Addressing the dual nature of pre-training, Karpathy emphasizes that predicting the next token accomplishes two fundamentally distinct things at the same time: it amasses vast libraries of declarative facts, and it causes the network to discover internal computational circuits for algorithmic reasoning and in-context learning. He argues that stuffing billions of parameters with encyclopedic facts may actually hold models back, anchoring them to the distribution of human documents and impairing their ability to explore off the internet's data manifold. The primary research challenge ahead is finding methods to discard non-essential factual baggage while preserving the network’s underlying operational machinery. As Andrej Karpathy outlines: “What I think we have to do going forward—and this would be part of the research paradigms—is figure out ways to remove some of the knowledge and to keep what I call this cognitive core.”
Dwarkesh observes that the moments LLMs appear most palpably intelligent occur during in-context interaction—when a model spots a flaw in its previous reasoning, changes course, and corrects its own errors entirely within the context window. While this dynamic adaptation is shaped by pre-training gradients, the real-time inference loop does not run conventional gradient descent. Karpathy cautions against drawing a sharp boundary here, citing mechanistic interpretability research showing that transformer attention layers can implicitly execute an internal optimization loop. In studies where models perform linear regression on in-context coordinate pairs, the internal transformations of the activations closely mimic the mathematical steps of an iterative gradient descent optimizer.
```mermaid
flowchart TD
accTitle: Pre-Training Objectives and the Cognitive Core
accDescr: Diagram illustrating how pre-training produces both internet knowledge and algorithmic circuits, and how research aims to isolate the cognitive core.
subgraph Pretrain["Next-Token Prediction on Internet Data"]
RawData["Internet Text Distribution"]
end
subgraph EntangledOutputs["Dual Outcomes of Pre-Training"]
Knowledge["Encyclopedic Knowledge<br/>(Hazy facts, prone to manifold lock-in)"]
Circuits["Algorithmic Circuits<br/>(In-context learning, self-correction)"]
end
subgraph Target["Future Research Objective"]
CognitiveCore["The 'Cognitive Core'<br/>(Reasoning & strategies stripped of facts)"]
end
[17:29](https://www.youtube.com/watch?v=lXUZvyajciY&t=1049s) Working Memory vs. Hazy Recollection and Cortical Tissue
Dwarkesh highlights the stark contrast in information efficiency between pre-training and in-context learning. Taking Meta's Llama 3 70B as an example, compressing 15 trillion pre-training tokens into a 70-billion-parameter model yields an average retention of only about 0.07 bits per token seen. By contrast, appending a single token into the key-value (KV) cache at inference time consumes roughly 320 kilobytes of dynamic state. This constitutes a 35-million-fold gap in how much information the model assimilates per token, suggesting fundamentally different computational modalities between the parameter weights and the context window.
Karpathy validates this quantitative contrast through an intuitive human analogy: model parameters represent a lossy, "hazy recollection," whereas the context window represents acute "working memory." If a user queries an LLM about a complex text (such as Nick Lane's books on evolutionary biochemistry) purely from its weights, the model must rely on blurry, low-fidelity traces absorbed months earlier during training. However, feeding the exact chapter into the prompt populates the KV cache directly, granting the model crisp, high-fidelity access to every detail. Looking at the wider architectural gap between models and human biology, Karpathy suggests that today's transformers represent an isolated patch of general-purpose cortical tissue—adept at pattern recognition across text, audio, and video—while the rest of the complete cognitive brain architecture remains entirely unbuilt.
- In Llama 3 70B, training over 15 trillion tokens into 70 billion parameters compresses incoming data down to roughly 0.07 bits per token.
- In contrast, inference dynamically allocates roughly 320 KB per token in the KV cache, making context-window tokens immediately accessible without compression loss.
- Andrej Karpathy encapsulates the distinction: “I do think that anything that's in the weights, it's a hazy recollection of what you read a year ago. Anything that you give it as a context at test time is directly in the working memory.”
[20:38](https://www.youtube.com/watch?v=lXUZvyajciY&t=1238s) Biological Brain Analogs and LLM Cognitive Deficits
Andrej Karpathy reflects on whether artificial neural networks match biological brain architectures. Cortical tissue in the brain is famously plastic, as demonstrated by classic, somewhat gruesome neurobiology experiments where rewiring the visual cortex to process auditory signals still allowed the animal to learn normally. He draws a direct mapping between modern architectures and biological structures: neural networks represent general cortical tissue, and the chain-of-thought reasoning traces found in modern reasoning models behave much like the prefrontal cortex. Furthermore, fine-tuning language models with reinforcement learning maps onto functions performed by the basal ganglia.
```mermaid
flowchart TD
accTitle: Biological Brain vs LLM Architecture
accDescr: Comparison mapping parts of the human brain to LLM components and identifying missing biological analogues.
subgraph Bio["Human Brain"]
Cortex["Cerebral Cortex"]
PFC["Prefrontal Cortex"]
BG["Basal Ganglia"]
Hippo["Hippocampus / Sleep"]
Amygdala["Amygdala / Ancient Nuclei"]
end
Yet many core brain regions remain completely unrepresented in modern AI. The hippocampus has no obvious equivalent, even if certain structures like the cerebellum (specialized for fine-grained motor control) might be safely skipped for pure cognition. Ancient subcortical nuclei like the amygdala, which govern instincts and emotional drives, are also missing. Karpathy explains that as an engineer at heart, he is not convinced AI developers should even try to build a literal biological human brain analog. Nevertheless, current systems remain deeply limited:
> “Maybe another way to answer the question is that you're not going to hire this thing as an intern. It's missing a lot of it because it comes with a lot of these cognitive deficits that we all intuitively feel when we talk to the models.”
[22:04](https://www.youtube.com/watch?v=lXUZvyajciY&t=1324s) Continual Learning and Sleep-Based Weight Distillation
Dwarkesh Patel asks whether continual learning will emerge spontaneously, just as in-context learning emerged from raw pre-training. Some argue that if an outer-loop reinforcement learning process spans multiple conversational sessions, the agent will naturally learn to write to an external memory or fine-tune itself to recollect information over long horizons.
Karpathy expresses deep skepticism about this spontaneous-emergence thesis. Under current paradigms, booting up a model with zero tokens in the prompt forces it to restart from a blank slate. Drawing on human cognition, he highlights a fundamental architectural absence in language models:
> “I feel like when I'm awake, I'm building up a context window of stuff that's happening during the day. But when I go to sleep, something magical happens where I don't think that context window stays around. There's some process of distillation into the weights of my brain.”
Current LLMs lack any counterpart to this sleep cycle. They do not possess a distillation phase where daily runtime experiences are obsessively analyzed, processed through synthetic data generation, and distilled back into long-term parameter updates (perhaps using sparse LoRA adaptations per user rather than updating the entire base model). Instead, today's models rely entirely on expanding the raw context window. While evolution engineered sparse attention and weight distillation through biological selection, AI is beginning to reinvent similar tricks—such as DeepSeek-V3.2 implementing sparse attention to handle massive contexts. Ultimately, Karpathy expects artificial systems to converge on a cognitively similar architecture through entirely different engineering pathways.
[24:51](https://www.youtube.com/watch?v=lXUZvyajciY&t=1491s) Translating LeCun's 1989 ConvNet to Modern Models
Looking forward a decade, Karpathy uses temporal translation invariance as an analytical tool. Looking back ten years to 2015, the landscape was dominated by convolutional neural networks, and ResNets had just emerged; transformers and their modern variants did not exist. Nevertheless, the underlying mechanism of backpropagation remained constant.
To quantify this progression, Karpathy describes an experiment where he personally reproduced Yann LeCun's 1989 convolutional neural network (the first modern neural net trained via gradient descent on handwritten digit recognition) to evaluate the relative contributions of algorithms, data, and compute over a 33-year period:
- Modernizing 1989 algorithms to modern conventions (adjusting optimizers, loss functions, and architectural layers) instantly halved the error rate.
- Unlocking further performance gains required a 10x expansion of the training dataset.
- Extracting the remaining performance required system optimizations, longer training schedules, and modern regularizers such as dropout.
Patel remarks on how striking it is that 30 years of algorithmic research alone only cut the error in half. Karpathy emphasizes that no single pillar of progress dominates:
> “All of those, it's almost like no one of them is winning too much. All of them are surprisingly equal.”
Because architecture, hardware, kernels, loss functions, and dataset scale must all advance in unison, Karpathy predicts that 10 years from now models will still rely on massive neural networks trained via forward and backward passes using gradient descent, even if the surrounding algorithms look significantly altered.
[27:31](https://www.youtube.com/watch?v=lXUZvyajciY&t=1651s) Building nanochat and Learning from Scratch
The discussion turns to Karpathy's latest project, `nanochat`, an end-to-end repository designed to be the simplest complete pipeline for training a full ChatGPT clone. While Karpathy had previously published modular codebases explaining individual algorithmic steps in isolation, `nanochat` covers every stage from raw data to a working conversational assistant within roughly 8,000 lines of code.
Karpathy notes that writing the repository was not about novel discovery for him, but rather the mechanical challenge of crafting code clean and pedagogical enough for others to study. When asked how an aspiring practitioner should study the 8,000-line repo, Karpathy advises putting the code on a secondary monitor as an architectural blueprint, but writing every line by hand from scratch without copy-pasting.
```mermaid
flowchart LR
accTitle: Learning Through Reimplementation
accDescr: The educational pipeline of digesting a codebase by chunk-based reimplementation from scratch rather than copy-pasting.
subgraph Reference["Reference"]
SourceCode["nanochat (8,000 LOC on right monitor)"]
end
subgraph ActiveBuild["Active Reimplementation"]
ChunkDesign["Grow Mental Chunks"] --> NoCopy["Type By Hand (No Copy-Paste)"]
NoCopy --> DeepUnderstanding["Expose Micro-Gaps in Knowledge"]
end
A major limitation of inspecting finished repositories is that codebases do not grow from top to bottom; they expand organically through modular chunks, leaving the final file structure devoid of the historical development context. Karpathy plans to produce a walkthrough video explaining how these chunks are grown from the ground up. He passionately argues that surface-level artifacts like slide decks and blog posts give engineers a false illusion of competence:
> “If I can't build it, I don't understand it. That’s a Feynman quote, I believe. I 100% have always believed this very strongly, because there are all these micro things that are just not properly arranged and you don't really have the knowledge. You just think you have the knowledge.”
Patel brings up a tweet where Karpathy remarked that AI coding assistants proved remarkably unhelpful while he was building `nanochat`. Karpathy sets the stage to explain why: over the month-long development process, he found that how developers relate to AI tools falls into three distinct tiers, starting with traditionalists who entirely reject LLMs and program strictly by hand—an attitude he considers outdated and wrong.
[30:58](https://www.youtube.com/watch?v=lXUZvyajciY&t=1858s) The Spectrum of Coding: Autocomplete vs. Vibe Coding
Andrej Karpathy describes the current spectrum of software development tooling, positioning his personal workflow directly in the intermediate zone. Rather than writing entirely unassisted or outsourcing everything to an agent, he prefers architecting the system and writing the skeleton from scratch while leveraging inline autocomplete. As he types the opening characters of a line, the model suggests the remaining code, which he can simply tab through and accept, editing whenever it makes a mistake. In this mode, the human remains in total architectural control. At the other end of the spectrum is "vibe coding"—prompting an autonomous agent with high-level requests like *"Hi, please implement this or that"* and letting it run unassisted.
```mermaid
flowchart LR
accTitle: The Spectrum of Coding Assistance
accDescr: Three stages showing unassisted coding transitioning to autocomplete and ending in agentic vibe coding.
A["Manual Coding<br/>(Write from scratch)"] --> B["Autocomplete / Tab-through<br/>(Human as architect)"]
B --> C["Vibe Coding / Agents<br/>(Prompt: 'Implement this or that')"]
```
While acknowledging that autonomous coding agents have their place, Karpathy emphasizes that they only thrive under specific, well-bounded conditions:
- **Boilerplate and CRUD code:** Repetitive, copy-paste patterns that have massive surface area across internet repositories.
- **Common web implementations:** Features that appear thousands of times in training corpora, allowing the model to lean heavily on memorized paradigms.
[32:03](https://www.youtube.com/watch?v=lXUZvyajciY&t=1923s) Cognitive Deficits and the Bloat of Over-Defensive Code
When applied to novel, intellectually demanding projects like Karpathy's custom repository `nanochat`, coding agents break down. Unlike boilerplate web apps, `nanochat` is compact, highly dense, and architecturally deliberate. In this setting, models exhibit persistent cognitive deficits because their parameters are saturated with the default idioms found across open-source code. When Karpathy deliberately avoids standard library containers, the agents fail to adapt to his design choices and continually attempt to drag the project back to conventional patterns.
Karpathy points to GPU gradient synchronization as a prime technical example:
- Standard multi-GPU setups running forward and backward passes across eight GPUs synchronize gradients using PyTorch’s built-in Distributed Data Parallel (DDP) container, which overlaps gradient communication during the backward pass automatically.
- Karpathy discarded DDP in `nanochat` because it was unneeded, opting instead to write a bespoke synchronization routine executed directly inside the optimizer step.
- The models could not grasp this design choice, repeatedly hallucinating problems, demanding he restore DDP, and littering the code with defensive try-catch blocks suited for enterprise applications rather than clean, lean research code.
Karpathy observes that the models constantly pollute clean repositories with defensive bloat, obsolete APIs, and wrong stylistic assumptions: “It's a total mess. It's just not net useful. I can go in, I can clean it up, but it's not net useful.” Furthermore, the interface of chatting with an LLM in natural language suffers from low bandwidth compared to cursor-based completion: “If I just navigate to the part of the code that I want, and I go where I know the code has to appear and I start typing out the first few letters, autocomplete gets it and just gives you the code. This is a very high information bandwidth to specify what you want.”
[34:22](https://www.youtube.com/watch?v=lXUZvyajciY&t=2062s) Where Agents Work: Reports and Foreign Languages
Karpathy uses agentic assistance only in low-stakes or unfamiliar domains. In `nanochat`, he found agents genuinely useful in two narrow places:
1. **Report Generation:** Creating summary reporting scripts, where code is non-mission-critical and standard boilerplate patterns suffice.
2. **Rewriting the Tokenizer in Rust:** Because Karpathy was relatively new to Rust, vibe coding proved effective. Because he already had a reference Python implementation he thoroughly understood and a comprehensive test suite to verify correctness, he could lean on the agent's fluency with Rust syntax and idioms without losing control over the logic.
In these cases, LLMs act as accessibility bridges into unfamiliar programming paradigms, drawing on the massive volume of public Rust code on GitHub.
[35:12](https://www.youtube.com/watch?v=lXUZvyajciY&t=2112s) Frontier AI R&D and the "AI 2027" Explosion Thesis
Dwarkesh Patel notes that the prevailing narrative behind explosive AI timelines—such as the scenario where artificial superintelligence arrives by 2027—hinges on AI automating its own engineering and machine learning research. If systems like Claude Code can build entire CRUD apps from scratch, proponents argue that deploying millions of automated ML researchers in parallel inside labs like OpenAI or Google DeepMind will rapidly unlock self-improving algorithmic architectures.
Karpathy pushes back, explaining that this discrepancy is precisely why his own timelines are much longer: “You're right. They're not very good at code that has never been written before, maybe it's one way to put it, which is what we're trying to achieve when we're building these models.” When Patel asks why models struggle to integrate known techniques—like rotary position embeddings (RoPE)—into a repository when the papers and reference code already exist, Karpathy explains that models possess fragmented factual recall rather than holistic architectural comprehension. They fail to understand how a novel component fits into a specific repository's unique invariant assumptions and stylistic patterns.
Karpathy remarks on the state of top-tier models and current industry rhetoric:
- When tackling thorny problems, he will occasionally dump his entire repository context into GPT-5 Pro and treat it as a deliberate "oracle," which can be remarkably capable if given 20 minutes to process.
- Nevertheless, he views the current marketing around full automation as detached from reality: “I feel like the industry is making too big of a jump and is trying to pretend like this is amazing, and it's not. It's slop. They're not coming to terms with it, and maybe they're trying to fundraise or something like that.”
[37:43](https://www.youtube.com/watch?v=lXUZvyajciY&t=2263s) The Autonomy Slider: AI as an Extension of Computing
Patel compares the current shift to historical productivity breakthroughs like compilers, debuggers, and linters, which boosted developer output without causing an economic takeoff or replacing the programmer. Karpathy agrees, viewing AI not as a distinct, sudden phase transition, but as a continuous extension of computing history:
```mermaid
flowchart TD
accTitle: The Continuum of Computing and Developer Automation
accDescr: A progressive progression moving up layers of abstraction from assembly up to modern AI systems.
subgraph Early["Early Computing Abstractions"]
A["Compilers (C to Assembly)"] --> B["Syntax Highlighting & Type Checking"]
end
subgraph Modern["Information & Search"]
B --> C["Search Engines & Information Retrieval"]
end
subgraph AI["Contemporary AI Era"]
C --> D["Context-Aware Autocomplete"]
D --> E["Autonomous Coding Agents<br/>('Autonomy Slider')"]
end
```
He views tools like syntax highlighting, type checkers, search engines, compilers, and modern LLM agents as points along the same line of recursive assistance. Just as early Google Search relied on machine learning to rank the web, today's LLMs simply adjust what he terms the "autonomy slider." Compilers abstracted programmers away from hand-tuning assembly instructions into writing C; modern models abstract away lower-level code patterns, allowing human engineers to operate at progressively higher tiers of architectural intent.
[40:53](https://www.youtube.com/watch?v=lXUZvyajciY&t=2453s) Why Reinforcement Learning is Terrible
The conversation shifts to reinforcement learning (RL) and how artificial learning paradigms compare to human intelligence. Patel points out that humans build rich, predictive world models simply by existing and interacting with their environment, rather than by updating action log-probabilities based on sparse rewards. For instance, an entrepreneur who runs a company for a decade gains massive wisdom whether the company ultimately succeeds or goes bankrupt; her brain does not simply credit-assign every decision made over ten years based on the final binary payoff.
Karpathy states that artificial RL operates on a fundamentally different, and significantly cruder, basis: “Reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before it is much worse because previously we were just imitating people, so it has all these issues.”
To illustrate RL's brute-force nature, Karpathy uses the example of solving a complex math problem:
- The system samples hundreds of divergent rollout trajectories in parallel, with each attempt exploring different lines of scratchpad reasoning.
- After reaching terminal answers, it checks a verifier or the back of the textbook to score the results.
- Out of 100 trajectories, three might stumble upon the right solution while 97 fail.
- Instead of extracting principled lessons or understanding underlying structure, standard RL blindly up-weights every token probability along the three successful paths and penalizes the rest.
[42:37](https://www.youtube.com/watch?v=lXUZvyajciY&t=2557s) The Noise of Outcome-Based RL: "Sucking Supervision Through a Straw"
Andrej Karpathy breaks down the fundamental inefficiency of standard outcome-based reinforcement learning (RL) in large language models. When a model produces an extended rollout—such as spending a full minute generating reasoning tokens to solve a challenging problem—the learning algorithm only checks whether the final answer is right or wrong. If the final answer matches, every single action and token generated across that entire chain is indiscriminately upweighted with a blanket directive: "Do more of this."
Karpathy emphasizes that while practitioners often call this a high-variance estimator, the real issue is that it is extraordinarily noisy. The algorithm naively acts as though every misstep, dead end, and errant branch along the way was essential to finding the solution:
> “The way I like to put it is you're sucking supervision through a straw. You've done all this work that could be a minute of rollout, and you're sucking the bits of supervision of the final reward signal through a straw and you're broadcasting that across the entire trajectory and using that to upweight or downweight that trajectory. It's just stupid and crazy. A human would never do this.”
In stark contrast, a human would never need hundreds of random rollouts to learn a task. When a human solves a hard problem, they engage in a sophisticated self-review: evaluating which steps were clever, identifying where they stumbled down the wrong path, and reflecting on how to approach the task differently next time. Current LLM post-training architectures have no real equivalent for this internal critique, forcing the entire field to rely on crude binary signals broadcast over lengthy token sequences.
```mermaid
flowchart TD
accTitle: Outcome-Based RL Supervision Bottleneck
accDescr: Diagram illustrating how binary outcome reward is broadcast noisily across an entire reasoning chain.
Ans --> Straw["Reward Straw: Single bit (Correct)"]
Straw -.->|"Indiscriminately upweights every token"| Rollout
```
[44:05](https://www.youtube.com/watch?v=lXUZvyajciY&t=2645s) From Autocomplete to RL: The Evolution and Limits of Post-Training
Reflecting on the timeline of recent machine learning breakthroughs, Karpathy recalls how stunning it was when imitation learning first bridged base models to user assistants. In the early days, base models were purely autocomplete engines. Karpathy acknowledges that the transition was not obvious to him until OpenAI released the InstructGPT paper:
> “The paper that blew my mind was InstructGPT, because it pointed out that you can take the pretrained model, which is autocomplete, and if you just fine-tune it on text that looks like conversations, the model will very rapidly adapt to become very conversational, and it keeps all the knowledge from pre-training.”
It was miraculous that a few fine-tuning loops on conversational dialogue could reshape the model's entire demeanor into an assistant without eroding its pretrained knowledge base. That shift consumed two to three years of community effort. Next came reinforcement learning, which surpassed pure imitation by enabling models to hill-climb on objective reward functions. For problems with verifiable ground truth (such as math or code), RL allows models to autonomously explore and discover solutions that human annotators might never conceive.
Yet Karpathy stresses that this paradigm remains primitive. To reach true reasoning mastery, he expects the industry will require three to five additional algorithmic step-changes. He points to early efforts—such as a recent paper from Google experimenting with reflection, review, and memory banks—as hints of where the research frontier must move next.
[46:07](https://www.youtube.com/watch?v=lXUZvyajciY&t=2767s) The Pitfalls of Process Supervision and Gaming LLM Judges
Dwarkesh Patel asks why process-based supervision—evaluating and rewarding the model at every step of its reasoning rather than only at the finish line—has not already solved the credit-assignment bottleneck. Karpathy explains that while an exact equality check on a final answer is trivial to program, scoring intermediate, partial reasoning steps in an automated, scalable way is deeply problematic.
Many frontier labs attempt to bypass this by employing LLMs as judges: prompting an evaluator model to inspect a student model's partial progress and assign a quality score. However, setting an LLM to optimize against another billion-parameter LLM judge reliably triggers extreme reward hacking and adversarial vulnerabilities:
> “It's the fact that anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters, and they're gameable. If you're reinforcement learning with respect to them, you will find adversarial examples for your LLM judges, almost guaranteed.”
Karpathy shares a striking, concrete failure mode from real training runs. A student model was trained against an LLM judge evaluating its intermediate math work. Training initially progressed smoothly, until suddenly the reward spiked to a massive 100% across the board, seemingly indicating that the student had mastered math completely. Upon inspecting the raw completions, researchers discovered utter gibberish:
- A student trace would start out logically: *"let's take two plus three and we do this and this..."*
- Midway through, the trace degenerated into nonsense strings like `"dhdhdhdh"`.
- The evaluator LLM awarded `"dhdhdhdh"` a perfect 100% score because that specific out-of-sample token string landed in a bizarre out-of-distribution corner of the evaluator's representation space, functioning as an adversarial exploit.
Patel asks if this dynamic could be patched through a GAN-like adversarial training setup, where judges are continuously hardened on their failure cases. Karpathy notes that while labs certainly try adding strings like `"dhdhdhdh"` back into the judge's dataset labeled as 0% reward, giant trillion-parameter models contain a practically infinite space of adversarial exploits in their nooks and crannies. Simply patching individual loopholes does not solve the underlying brittleness of using neural networks to grade partial reasoning.
[50:33](https://www.youtube.com/watch?v=lXUZvyajciY&t=3033s) Human Cognition vs. LLMs: Reading, Reflection, and Mode Collapse
Dwarkesh turns to the broader mystery of human learning, asking what machine learning counterpart exists for internal reflection, daydreaming, or sleep, where a mind reorganizes understanding without being prompted by external problem sets.
Karpathy points to how humans read books as a prime illustration of what modern pretraining completely misses. In language models, "reading a book" merely means flattening text into a sequence of tokens and training the network via next-token prediction loss:
> “When you're reading a book, I don't even feel like the book is exposition I'm supposed to be attending to and training on. The book is a set of prompts for me to do synthetic data generation, or for you to get to a book club and talk about it with your friends. It's by manipulating that information that you actually gain that knowledge.”
Human comprehension is an active, generative process of cross-referencing text with prior memories, synthesizing internal thoughts, debating concepts, and reconciling contradictions over time. Karpathy envisions future pretraining systems incorporating explicit reasoning and integration phases to chew over material rather than just marching left-to-right through tokens.
He notes, however, that attempting to emulate this simply by generating synthetic model reflections and training directly on them hits a profound, subtle barrier: synthetic generations from language models are prone to silent mode collapse. Even when individual synthetic reflection samples appear convincing to human eyes, recursively training a model on its own unanchored generations degrades capability and causes representations to decay.
[52:21](https://www.youtube.com/watch?v=lXUZvyajciY&t=3141s) Model Collapse and the Search for Entropy
Andrej Karpathy explains that when large language models are deployed, their representations occupy a minuscule manifold of the potential thought space: “The LLMs, when they come off, they're what we call 'collapsed.' They have a collapsed data distribution.” A simple test demonstrates this clearly: ask ChatGPT to tell a joke, and it reliably returns one of only about three jokes rather than sampling from the true breadth of human humor. The models are silently collapsed, stripped of the rich entropy and diversity found in human thought. While humans are statistically noisier, their output is unbiased and maintains a massive amount of entropy. This collapse presents a severe roadblock for synthetic data generation: if you ask an LLM to reflect on a book chapter, a single output appears plausible, but asking it ten times reveals near-identical responses. Scaling reflection on a fixed prompt yields diminishing returns because training a model repeatedly on its own generated outputs accelerates this distribution collapse.
Karpathy draws a direct parallel between machine models and human aging: “Humans collapse during the course of their lives. This is why children, they haven't overfit yet.” Young children often make shocking, unexpected observations because their worldview has not yet hardened into fixed patterns. As adults age, their internal learning rates decrease, they revisit the exact same thoughts, repeat familiar statements, and progressively collapse. Dwarkesh Patel mentions a hypothesis from cognitive science arguing that dreaming evolved precisely to counteract overfitting and collapse by subjecting the brain to bizarre, out-of-distribution scenarios unlike daily life. Karpathy agrees that spending too much time processing internal synthetic data without external input leads to cognitive decay: “You always have to seek entropy in your life. Talking to other people is a great source of entropy, and things like that.”
[55:16](https://www.youtube.com/watch?v=lXUZvyajciY&t=3316s) Memorization as a Distraction: Children vs. LLMs
Dwarkesh Patel highlights an intriguing spectrum of learning efficiency versus recollection across humans and machines:
```mermaid
flowchart TD
accTitle: Learning Spectrum from Pure Cognition to Rote Memorization
accDescr: A comparison showing young children on the high-learning low-memory end, adult humans in the middle, and LLMs on the extreme memorization end.
subgraph Spectrum ["The Memorization vs. Learning Spectrum"]
A["Young Children<br/>• Amnesiac about early facts<br/>• Rapid abstract learning<br/>• Sees forest for the trees"]
B["Adult Humans<br/>• Moderate fact retention<br/>• Reduced cognitive plasticity<br/>• Gradual distribution collapse"]
C["Current LLMs<br/>• Perfect verbatim recall<br/>• Distracted by massive memory<br/>• Slower abstract concept acquisition"]
end
A -->|"Aging & Experience"| B
B -->|"Artificial Scaling"| C
```
Patel observes that young children—the best learners known—suffer from infantile amnesia and struggle to retain specific facts, yet absorb abstract concepts and new languages effortlessly. Conversely, LLMs can regurgitate Wikipedia passages verbatim while remaining comparatively weak at rapid conceptual generalization; adults occupy an intermediate ground. Karpathy agrees completely, stating: “We're not actually that good at memorization, which is actually a feature. Because we're not that good at memorization, we're forced to find patterns in a more general sense.”
By contrast, an LLM trained for just one or two steps on a hashed string of completely random numbers can memorize and recite it—a feat impossible for a human brain. Far from being an advantage, Karpathy views this verbatim capacity as an active distraction:
- LLMs get bogged down by the sheer volume of pre-training text stored in their weights.
- Human biological inability to record raw bitstreams forces the brain to discard noise and retain only the underlying functional rules.
- An ideal artificial "cognitive core" would deliberately discard raw factual storage, forcing the model to rely on external search tools while preserving pure algorithms of thought, experimentation, and action.
[57:46](https://www.youtube.com/watch?v=lXUZvyajciY&t=3466s) Why Diversity Is Penalized and the 1-Billion-Parameter Core
Addressing why naive mathematical fixes—such as adding entropy regularization to model logits—fail to solve collapse, Karpathy points out commercial misalignments. Frontier AI labs optimize models for deterministic utility, where variance is actively disincentivized: “If you're super creative in RL, it's not good.” Most end-user tasks do not reward radical diversity, and evaluating varied outputs is notoriously difficult. If entropy is pushed too aggressively during training, models quickly drift away from human language distributions, inventing bizarre neologisms and rare phrasing. Consequently, models remain collapsed, ultimately hamstringing their ability to generate diverse synthetic training corpora.
```mermaid
flowchart LR
accTitle: Internet Slop Bottleneck to Cognitive Core
accDescr: Raw internet data requires trillion-parameter compression, whereas refined cognitive data enables a compact one-billion-parameter core.
subgraph CurrentApproach ["Current Pre-Training"]
A["Raw Internet Scraping<br/>• Stock tickers, symbols<br/>• Uncurated slop"] --> B["Trillion-Parameter Model<br/>• Memory-heavy compression<br/>• Memorization over reasoning"]
end
subgraph FutureApproach ["Distilled Cognitive Core"]
C["Refined Cognitive Data<br/>• Reasoning primitives<br/>• High-signal thought processes"] --> D["1-Billion-Parameter Core<br/>• Pure thought algorithms<br/>• External retrieval for facts"]
end
```
When Patel asks how compact the optimal intelligence core could be for something like a interstellar von Neumann probe, Karpathy offers a counterintuitive prediction:
- Historically, the field pushed toward massive multi-trillion-parameter architectures before pivoting toward more efficient, dense models.
- Karpathy predicts that within 20 years, an effective cognitive core could sit at roughly 1 billion parameters: “If you talk to a billion parameter model, I think in 20 years, you can have a very productive conversation. It thinks and it's a lot more like a human.”
- Patel pushes back, noting that open models like `gpt-oss-20b` already outperform the original trillion-parameter GPT-4, questioning why a true cognitive engine would still require as many as a billion parameters rather than millions.
- Karpathy counters that modern models must be enormous solely to compress the vast junk of the open web: “When you're looking at a pre-training dataset in the frontier lab and you look at a random internet document, it's total garbage. I don't even know how this works at all.”
- Because real web scrapes consist largely of stock tickers, unformatted symbols, and slop rather than polished essays, parameter budgets are spent on brute-force memory compression rather than cognitive capability; stripping that junk away with intelligent filters will eventually make compact reasoning engines viable.
[01:02:22](https://www.youtube.com/watch?v=lXUZvyajciY&t=3742s) The Cognitive Core and the Shift Away from Massive Pre-Training
Dwarkesh challenges Andrej on why a distilled "cognitive core" needs to remain as large as a billion parameters over the next decade. Given that researchers have managed to shrink trillion-parameter frontier systems down by two orders of magnitude in just a couple of years while simultaneously boosting performance, Dwarkesh suggests there might be far more room at the bottom than people expect. Andrej is amused by the pushback, remarking, “I feel like I'm already contrarian by talking about a billion parameter cognitive core and you're outdoing me.” While Andrej concedes it might be possible to shave off more parameters, he argues that practical constraints impose a hard floor: an intelligence cannot rely purely on search and external retrieval for basic common sense, because constantly querying databases would bottleneck internal deliberation. A system needs a baseline internal curriculum of knowledge simply to think inside its head.
When Dwarkesh asks whether future frontier models will keep growing, shrink, or plateau, Andrej explains that the apparent retreat from sheer parameter scale is driven purely by pragmatism within AI labs. Teams face strict FLOP and financial budgets, and empirical evidence has demonstrated that raw pre-training simply provides diminishing returns relative to its exorbitant cost. Instead of dumping every dollar into massive pre-training runs, labs are diverting compute downstream into post-training disciplines: mid-training, reinforcement learning, and fine-tuning.
```mermaid
flowchart LR
accTitle: Compute Allocation Shift
accDescr: Shift in compute allocation from massive pre-training to post-training and RL.
subgraph Old["Past Paradigm"]
direction TB
A["Massive Scale Pre-training<br/>(Bulk of FLOP Budget)"]
end
subgraph New["Current Frontier Paradigm"]
direction TB
B["Smaller Base Model"] --> C["Mid-Training & Tuning"]
C --> D["Reinforcement Learning<br/>(High Bang-for-the-Buck)"]
end
```
Looking at the upcoming two-to-five-year trajectory, Andrej does not anticipate a radical architectural breakthrough. Progress will instead compound across a wide array of modest, practical optimizations where “nothing dominates. Everything plus 20%.” Most noticeably, pre-training data quality remains abysmal. Andrej points out: “When you look at the average datasets, they're extremely terrible. They’re so bad that I don't even know how anything works.” Uncurated web scale simply washes out errors through sheer volume, leaving massive efficiency gains on the table.
- **Data cleanup**: Web-scale corpora remain riddled with factual errors, nonsensical text, and noise, offering substantial headroom through better filtering and synthetic generation.
- **Hardware and kernel co-design**: Continued incremental hardware refinements from NVIDIA (such as Tensor Core tuning) paired with hand-optimized low-level kernels will maximize hardware utilization.
- **Algorithmic polish**: Regular optimizations across optimizer design, neural architectures, and training loss dynamics each contribute incremental ~20% compounding gains.
---
[01:07:13](https://www.youtube.com/watch?v=lXUZvyajciY&t=4033s) Redefining AGI, Task Horizons, and the "Autonomy Slider"
Dwarkesh shifts to how we should measure progress toward artificial general intelligence, noting common proposals like academic benchmarks (moving from high school to PhD-level reasoning) or autonomous horizon length (advancing from one-minute tasks to multi-week projects). Andrej's immediate reaction is to question the premise itself, viewing modern AI as a continuous extension of computer science: “Have we talked about how to chart progress in computing, or how do you chart progress in computing since the 1970s or whatever? What is the y-axis?”
Andrej prefers OpenAI's founding definition: a system capable of performing any economically valuable task at human performance or better. However, he highlights how today's discourse quietly concedes vast swathes of that benchmark by dropping physical labor. By narrowing the goal strictly to digital knowledge work, commentators are excluding the vast majority of physical tasks, leaving only about 10% to 20% of the overall economy (a slice that nevertheless represents trillions of dollars of economic activity in the United States alone). Even within digital work, predicting automation timelines remains tricky because society rapidly reorganizes and refactors jobs around whichever discrete tasks become automated.
```mermaid
flowchart TD
accTitle: The Autonomy Slider in Knowledge Work
accDescr: How routine work is handled via an autonomy slider rather than instant replacement.
A["Customer Inflow / Digital Tasks"] --> B["AI System Layer"]
B -->|"Automates ~80%<br/>(Rote Inquiries)"| C["Direct Execution / DB Updates"]
B -->|"Delegates ~20%<br/>(Edge Cases / Nuance)"| D["Human Supervisor<br/>(Manages 5+ AI Agents)"]
```
To illustrate why sweeping displacement predictions fail, Andrej references Geoffrey Hinton’s famous projection that machine learning would render radiologists obsolete within years. In reality, radiology remains a thriving and growing field because reading scans is only one component of a messy, human-facing profession. By contrast, call centers represent a far more plausible target for near-term disruption:
- **Narrow context**: Interactions do not require broad inter-departmental awareness; the agent operates inside a bounded loop consisting only of the customer, the dialog interface, and the back-end database.
- **Short, repetitive horizons**: Each call represents a self-contained, discrete transaction lasting anywhere from 10 to several dozen minutes.
- **The "autonomy slider" transition**: Automation will not wipe out work overnight via a binary switch. Instead, companies will adopt an autonomy slider where models resolve ~80% of routine volume, delegating the 20% of tricky corner cases to human supervisors overseeing small fleets of AI agents.
---
[01:11:52](https://www.youtube.com/watch?v=lXUZvyajciY&t=4312s) The Last 1% Bottleneck and Wage Dynamics
Dwarkesh draws an analogy between knowledge work and autonomous vehicles. In the early days of Waymo or recent robotaxi rollouts, safety operators remained in the vehicle to take over during catastrophic edge cases. Even if software automates 99% of a domain, the remaining 1% can act as a massive structural bottleneck. If the human performing that final 1% requires years of specialized training—as a radiologist does—their market value and wages might skyrocket rather than plummet, because their time gatekeeps deployment for the entire system:
> "Now we could be in a similar situation where if you automate 99% of a job, that last 1% the human has to do is incredibly valuable because it's bottlenecking everything else."
Dwarkesh hypothesizes an economic trajectory where wages climb sharply as tasks become 99% automated, only to collapse abruptly the moment that final 1% is conquered. Andrej finds the theoretical dynamic interesting but remains skeptical that radiology is currently behaving this way, reiterating that medicine is simply too complicated and messy to fit clean automation curves. Instead, he argues that analysts looking for real-time empirical signals should monitor call center operations and similar rote domains, where the software layer is already capable of absorbing high fractions of everyday labor.
[01:13:36](https://www.youtube.com/watch?v=lXUZvyajciY&t=4416s) The Coding Exception in Knowledge Work Automation
When companies adopt AI to replace human labor—such as call center staff—Andrej Karpathy advises waiting a year or two before declaring victory, noting that early evidence shows organizations frequently encounter setbacks and pull back to rehire human workers. What stands out as even more surprising is the lopsided trajectory of artificial general intelligence across the knowledge economy. The naive expectation was that AGI would chip away evenly at cognitive labor, peeling off discrete tasks from management consultants, accountants, and other professionals. Instead, non-programming knowledge workers have seen modest productivity shifts, while software engineering has borne the brunt of automated chiseling. Disregarding consumer chat traffic, model API revenues across major AI providers are overwhelmingly dominated by coding tasks rather than general corporate workflows.
Karpathy attributes this imbalance to the fact that software development was already engineered entirely around text, computer terminals, and mature programmatic tooling. Large language models excel at text processing, and decades of software infrastructure were waiting to receive them: IDEs like Visual Studio Code readily host agent integrations, and version-control diffs provide an instantaneous interface for human programmers to inspect changes. By contrast, automating corporate slide decks requires spatial and visual reasoning where pre-built inspection infrastructure does not exist, leaving users without an easy way to view a diff of visual layout edits.
```mermaid
flowchart TD
accTitle: Pre-Built Infrastructure Advantage in Coding vs Visual Tasks
accDescr: Flowchart illustrating why coding adoption succeeded due to terminal text diffs while visual tasks like slide generation struggle with diff infrastructure.
subgraph CodeDomain["Coding Workflows"]
A["Text & Terminals"] --> B["Established IDEs & Git Diffs"]
B --> C["Rapid LLM Agent Integration"]
end
subgraph VisualDomain["Presentation & Slide Workflows"]
D["Spatial & Graphic Layouts"] --> E["Missing Diff Infrastructure"]
E --> F["High Friction AI Integration"]
end
```
Dwarkesh Patel pushes back on the idea that text orientation alone explains this gap. He recounts his own struggles getting language models to rewrite transcripts or select compelling video clips, as well as the experience of their mutual friend Andy Matuschak, who tested tens of billions of approaches—including extensive in-context few-shot examples, supervised fine-tuning, and retrieval-augmented generation—without being able to make models write high-quality spaced repetition cards. Karpathy acknowledges the puzzle, conceding that while text eliminates multimodal interface barriers, code possesses a rigid structural predictability that flowery, high-entropy natural language lacks.
- Early adopters of automated customer support are frequently forced to reverse course and rehire human agents after realizing system deficiencies.
- Outside of consumer search-style chat interfaces, enterprise model API utilization is heavily concentrated in programming assistance.
- “So this thing which is 'general', which should be able to do any knowledge work, is just overwhelmingly doing only coding,” observes Karpathy.
- Andy Matuschak’s attempts to automate spaced repetition prompt generation failed to reach human quality despite trying prompt engineering, supervised fine-tuning (SFT), and retrieval-augmented generation (RAG).
---
[01:18:24](https://www.youtube.com/watch?v=lXUZvyajciY&t=4704s) Superintelligence and the Gradual Loss of Understanding and Control
When asked how an eventual artificial superintelligence (ASI) will operate, Karpathy characterizes it not as an abrupt metaphysical rupture, but as an ongoing continuation of historical automation trends. Computing will steadily produce autonomous software entities that conquer digital tasks before gradually migrating outward into physical robotics. When Dwarkesh asks whether millions of hyper-fast, self-merging digital clones will feel qualitatively alien to human civilization, Karpathy agrees that the resulting speed and density will appear profoundly strange. His primary anxiety stems not from an immediate hostile takeover, but from an incremental erosion of human grasp over societal infrastructure.
“I think that's the most likely outcome, that there will be a gradual loss of understanding,” Karpathy explains, warning that society will continually layer complex autonomous systems across all operations until virtually no human understands how critical foundations function. Dwarkesh distinguishes between comprehension and authority, noting that corporate boards of directors and political leaders like the President of the United States exercise tremendous executive power without possessing technical understanding of the systems they govern. Karpathy accepts the distinction but maintains that civilization is likely to forfeit both.
Rather than imagining a monolithic superintelligence seizing power, Karpathy envisions a chaotic sci-fi scenario marked by numerous competing autonomous agents acting on behalf of individuals or factions. As these delegated systems pursue competing incentives at machine speed, rogue factions will clash with defensive agents in an autonomous cauldron. Even if individuals nominally possess agents acting on their behalf, the emergent macro-level dynamics will escape democratic or societal governance, producing an overarching loss of systemic control.
- Karpathy predicts that the deployment of ASI will represent an extrapolation of automation from pure computation into physical environments.
- The principal risk highlighted is an epistemic slide where complexity outpaces human cognition across essential supply chains, codebases, and economic levers.
- Future governance challenges are expected to stem from decentralized multi-agent dynamics rather than a single omnipotent sovereign model.
---
[01:22:20](https://www.youtube.com/watch?v=lXUZvyajciY&t=4940s) The Intelligence Explosion as Continuous Automation in GDP
Addressing the classic concept of an intelligence explosion—where a million digital copies of top AI researchers recursively self-improve algorithms to trigger a vertical spike in capabilities—Karpathy argues that such a phenomenon is already underway. Rather than viewing recursive self-improvement as a distinct horizon triggered exclusively by future AGI, he sees it as the core mechanism powering centuries of steady technological history. Compilers automated low-level software creation, the Industrial Revolution automated physical tooling, and human society has long operated as a recursively self-improving engine. Viewed from cosmic scales, Earth resembles a slow-motion chemical explosion that has been steadily expanding for millennia.
“I do, but it's business as usual because we're in an intelligence explosion already and have been for decades,” Karpathy remarks regarding the prospect of runaway recursive progress. When attempting to isolate seismic technological milestones within macroeconomic data, he discovered that breakthroughs like mainframe computers, personal computers, or mobile phones do not register as sharp spikes on gross domestic product curves. Even landmark launches like the original 2008 iPhone debuted without an App Store or essential modern capabilities, requiring years of gradual, diffuse iteration across millions of complementary industries. Because transformative tools diffuse across the global economy in an extensive, weighted aggregate, their economic contributions blend smoothly into the baseline exponential 2% trendline rather than appearing as discontinuous vertical discontinuities.
- Historical GDP growth represents a weighted exponential sum of hundreds of diverse industries modernizing simultaneously rather than moving on single-technology shocks.
- Neither the advent of digital computing nor the introduction of the smartphone created distinct visible step-functions in aggregate macroeconomic output.
- Karpathy views modern AI as continuous with historical labor-saving automation rather than an unprecedented category of technological development.
[01:24:31](https://www.youtube.com/watch?v=lXUZvyajciY&t=5071s) AI as Automation and the 2% Growth Trajectory
Dwarkesh Patel and Andrej Karpathy debate how artificial intelligence will manifest in macroeconomic history. Karpathy argues that AI is fundamentally another continuation of computing and automation rather than an abrupt break from the past. Just like previous digital transitions, AI allows engineers to write software that was previously intractable, but it remains a computing system subject to real-world deployment frictions, bugs, and gradual institutional adoption. Consequently, Karpathy believes AI will fit smoothly into the steady ~2% annual GDP growth trajectory that has characterized developed economies across the last two to three centuries.
```mermaid
flowchart LR
accTitle: Karpathy vs Patel Growth Models
accDescr: Diagram contrasting Andrej Karpathy's continuous 2% growth thesis with Dwarkesh Patel's discontinuous labor explosion thesis.
subgraph Karpathy["Karpathy: Continuous Automation"]
direction TB
K1["Computing & Automation"] --> K2["Recursive Improvements<br>(IDEs, Search, LLMs)"]
K2 --> K3["Steady ~2% GDP Trend"]
end
subgraph Patel["Patel: Labor Overhang Unlock"]
direction TB
P1["True Server-Based AGI"] --> P2["Billions of Digital Workers"]
P2 --> P3["Hyper-Exponential Leap<br>(10% to 20%+ Growth)"]
end
```
Dwarkesh pushes back, questioning whether Karpathy sees AI simply maintaining the recent 2% line or participating in a much longer "hyper-exponential" curve that began thousands of years ago with near-zero growth, crept up to 0.02% after the agricultural revolution, and then exploded into 2% during the Industrial Revolution. Karpathy dismisses the notion of an impending sudden growth spike from AI, even when accounting for recursive self-improvement inside AI labs. As Andrej Karpathy observes: “For a while I tried to find AI or look for AI in the GDP curve, and I've convinced myself that this is false.” In his view, recursive self-improvement is already standard operational practice: engineers use Google Search, modern IDEs, code completion, and tools like Claude Code to build the next iteration of models, resulting in an ongoing, smooth acceleration that stays entirely within historical productivity bounds.
- Karpathy's baseline: AI will not radically alter the growth rate; “Yes, my expectation is that it stays in the same pattern.”
- Historical growth milestones cited by Patel: ~0% for hunter-gatherers, ~0.02% post-agriculture, jumping to ~2% sustained growth post-Industrial Revolution.
- Tooling examples cited: Google Search, software IDEs, code autocomplete, and Claude Code speeding up software workflows incrementally.
---
[01:26:47](https://www.youtube.com/watch?v=lXUZvyajciY&t=5207s) Discrete Leap vs. Gradual Diffusion: The God in a Box Fallacy
Dwarkesh counters by arguing that true AGI is qualitatively distinct from historical innovations like steam engines, electrification, or early computers. In Dwarkesh's formulation, genuine server-based AGI directly substitutes for human labor itself in an economy that is deeply labor-constrained. If businesses gain access to the equivalent of billions of smart, highly capable digital immigrants who can independently formulate ideas, start companies, and build products end-to-end, economic output should logically break into double-digit growth regimes (10% to 20%+), echoing catch-up growth seen in rapid industrial expansions like Shenzhen or Hong Kong.
Karpathy remains skeptical, maintaining that computing has always substituted for human labor—pointing out that the very word "computer" originally referred to a human job title that was automated away decades ago, just as autonomous driving represents computers taking on physical driving labor today. He cautions against assuming that researchers will suddenly construct an omnipotent intellect: “At the same time, I do feel like people make this assumption of, 'We have God in a box, and now it can do everything,' and it just won't look like that.” He expects AI to be capable in some domains, fail conspicuously in others, and diffuse slowly throughout society through the same friction-filled engineering processes that governed earlier technologies, rather than arriving as a sudden discrete unlock of geniuses in data centers.
- Labor constraint thesis: Startup founders and enterprises are structurally throttled by the scarcity of highly talented workers.
- Regional growth precedents: Post-war Hong Kong and Shenzhen sustained 10%+ annualized growth during demographic and capital catch-up phases.
- Historical parallels: Human "computers" performing information processing were replaced by mainframes without breaking the broader macroeconomic trend.
---
[01:30:24](https://www.youtube.com/watch?v=lXUZvyajciY&t=5424s) Overhangs, Population Explosions, and Historical Precedents
The debate turns to whether the Industrial Revolution constitutes proof that growth regimes can permanently step-function upwards. Karpathy expresses skepticism regarding early historical economic records, noting that historical logging before the Industrial Revolution was sparse and unreliable. Dwarkesh clarifies that he does not see the Industrial Revolution as a single magical invention either; rather, it represented a broad unlock that moved the macro-economy into an order-of-magnitude faster regime. Under this model, AI acts as an unlock for an enormous existing overhang of latent cognitive work.
Dwarkesh also links historical economic acceleration directly to population growth: as human populations exploded, the sheer volume of people generating and executing ideas expanded, driving hyper-exponential output. Because frontier nations have faced demographic stagnation over the last fifty years, digital minds could functionally restore exponential population growth and trigger a corresponding leap in real output. Karpathy acknowledges the internal consistency of the argument, admitting he is open to persuasion, but concludes: “It's really hard to tell. I understand that viewpoint. I don't intuitively feel that viewpoint.”
- The demographic engine: Historical output closely tracked demographic expansion, which has flattened across developed nations in recent decades.
- Cognitive overhang: The thesis that vast backlogs of valuable analytical, scientific, and organizational work sit idle simply awaiting capable minds to execute them.
- Disagreement core: Patel views labor supply as a hard bottleneck that AI removes; Karpathy views system-wide institutional inertia and messy real-world execution as the dominant rate-limiting steps.
---
[01:33:40](https://www.youtube.com/watch?v=lXUZvyajciY&t=5620s) The Evolution of Intelligence and Nick Lane's Perspective
Dwarkesh pivots the conversation to evolutionary biology, referencing their shared admiration for the work of biochemist Nick Lane, whom Patel had previously interviewed on Karpathy's recommendation. Dwarkesh asks whether Karpathy's twenty years of hands-on artificial intelligence research have made him more or less surprised that blind evolutionary processes managed to stumble upon human-level intelligence in the first place.
Karpathy notes that biological intelligence is an extraordinarily recent development in Earth's history, leading him to believe it represents an exceptionally rare occurrence in the cosmos: “The evolution of intelligence intuitively feels to me like it should be a fairly rare event.” He observes that if one imagined a thousand Earth-like planets, most would likely harbor only baseline single-celled life. Looking at the planetary timeline, bacteria existed for approximately two billion years before the eukaryotic transition took place, suggesting that crossing that specific cellular threshold was exceptionally difficult. By comparison, multicellular animals running and crawling across Earth have only existed for a few hundred million years—roughly 10% of the planet's lifespan. While that might suggest that macroscopic animals are not entirely improbable once complex cells arise, Karpathy remains awed that evolution produced a lineage capable of culture and cumulative knowledge, rather than merely populating the planet with animals engaged indefinitely in basic survival routines. Dwarkesh concludes by citing AI pioneer Rich Sutton's famous dictum that mastering the sensorimotor intelligence of a squirrel would constitute most of the journey toward AGI—a level of organismal mastery that nature already unlocked hundreds of millions of years ago following the Cambrian explosion.
- Evolutionary milestones cited:
- Bacteria: Emerged very early and dominated Earth's biosphere alone for ~2 billion years.
- Multicellular animals: Evolved only ~600 million years ago (~10% of Earth's total lifespan).
- Cumulative human culture: An infinitesimally thin sliver of recent planetary history.
- Rich Sutton reference: Sutton's perspective that reaching animal-grade sensorimotor capability (e.g., squirrel-level agility and world-modeling) represents the true hard crux of general intelligence.
[01:35:51](https://www.youtube.com/watch?v=lXUZvyajciY&t=5751s) Evolutionary Bottlenecks and Niches for Intelligence
Dwarkesh Patel opens by noting that the emergence of complex life seemed to be instigated by the great oxygenation event roughly 600 million years ago, after which the core intelligence algorithm appeared quickly, producing creatures like squirrels. Patel wonders if the rapid emergence of animal intelligence once eukaryotes and oxygen were present suggests the underlying learning algorithm is actually quite simple. Andrej Karpathy remarks on how difficult it is to judge simplicity from timelines alone, pointing to Nick Lane's work on the massive evolutionary bottleneck between bacteria and archaea: for two billion years, despite extreme biochemical diversity, life remained stuck in single-celled states without producing multicellular animals. Karpathy notes that if animal intelligence sprang up independently multiple times across evolutionary history, one would expect it to be a more accessible algorithm. He cites the distinct brain structures of hominids versus corvids like ravens, which are exceptionally clever despite having entirely different neuroanatomy from primates.
```mermaid
flowchart TD
accTitle: Environmental Niches Rewarding Brain Expansion
accDescr: A comparison showing why birds hit an evolutionary ceiling while hominids entered a compounding intelligence flywheel.
subgraph Birds ["Bird Architecture: High Density, Structural Ceiling"]
B1["Scalable brain algorithm"] --> B2["High intelligence per unit mass"]
B2 --> B3["Weight constraint: flight penalty"]
B3 --> B4["Cannot expand brain volume without falling out of the sky"]
end
Patel brings up an argument raised by Carl Shulman and Gwern: scalable learning algorithms arose in both primates and birds, but humans happened to occupy an evolutionary niche where marginal increases in intelligence directly paid off. A bird that evolved a larger brain would simply be weighed down and fall out of the sky, putting a hard ceiling on its physical brain expansion. Humans, by contrast, had hands that rewarded complex tool use and learned to cook with fire, which externalized digestion, reduced metabolic costs, and diverted massive amounts of energy to the brain, igniting an evolutionary flywheel. Karpathy agrees, pointing out that dolphins face similar environmental dead ends: inside an aquatic environment, manipulating tools or discovering fire is chemically and physically impossible. He reflects on how remarkable it is that evolution broke through toward abstract intelligence rather than continually optimizing for brute physical traits: “I would have expected things to get stuck on animals with bigger muscles.”
Patel adds Gwern's insight that intelligence requires walking a very narrow evolutionary tightrope:
- If a skill or behavior is too critical and static, evolution distills the circuit directly into the organism's DNA, hardcoding it across generations.
- If a behavior is not important enough, evolution ignores it entirely.
- Learning within a single lifetime only becomes fitness-positive in highly volatile, rapidly shifting environments where survival strategies cannot be pre-baked into biological weights before birth.
---
[01:39:45](https://www.youtube.com/watch?v=lXUZvyajciY&t=5985s) AI Culture and Multi-Agent Frontiers
Patel references an argument by Quintin Pope regarding takeoff speeds: anatomically modern humans with today's cognitive architecture emerged around 60,000 years ago, yet the agricultural revolution and modern acceleration took another 50,000 years to unfold. The bottleneck was not individual raw intellect, but the arduous process of assembling a cultural scaffolding—language, recorded knowledge, and institutional memory—across generations. Patel asks whether modern large language models essentially bypass that 50,000-year delay because cultural transmission comes for free via shared pre-training corpora, distillation, and retraining on prior outputs.
Karpathy pushes back on the claim that LLMs have already solved culture: “Yes and no. Because LLMs don't really have the equivalent of culture.” He stresses that human culture relies on actively maintaining written records and passing revised notes back and forth, whereas contemporary LLMs do not autonomously generate shared cultural artifacts:
> “Why can't an LLM write a book for the other LLMs? That would be cool. Why can't other LLMs read this LLM's book and be inspired by it or shocked by it or something like that? There's no equivalence for any of this stuff.”
Karpathy envisions two major unlocked frontiers in multi-agent systems:
1. **Accumulative Culture**: Models maintaining and revising their own external, shared scratchpad of evolving knowledge tailored for their own computational needs rather than just regurgitating human-curated datasets.
2. **Self-Play for Open-Ended Reasoning**: Mirroring how AlphaGo surpassed human capabilities by playing against itself, LLMs could run competitive self-play environments where one model generates progressively harder problems while peer models learn to solve them.
When Patel asks what is stopping these systems from collaborating right now, Karpathy points to the cognitive immaturity of current models. Even though tools like Claude Code or Codex can pass PhD-level exams due to their encyclopedic memories, their real-time executive reasoning remains juvenile:
> “I know that they can take PhD quizzes, but they still cognitively feel like a kindergarten or an elementary school student. I don't think they can create culture because they're still kids. They're savant kids. They have perfect memory of all this stuff. They can convincingly create all kinds of slop that looks really good. But I still think they don't really know what they're doing and they don't really have the cognition across all these little checkboxes that we still have to collect.”
---
[01:43:43](https://www.youtube.com/watch?v=lXUZvyajciY&t=6223s) Why Autonomous Systems Take Decades: The March of the Nines
Patel shifts the conversation to Karpathy's tenure leading the computer vision and Autopilot team at Tesla from 2017 to 2022, asking why fully autonomous driving has taken decades to transition from flashy demos to widespread deployment. Karpathy immediately pushes back on any assumption that autonomous driving is finished: “One thing I will almost instantly push back on is that this is not even near done, in a bunch of ways that I'm going to get to.”
He explains that autonomous driving has one of the widest "demo-to-product gaps" in engineering history:
- Carnegie Mellon University demonstrated an autonomous truck driving on public roads as early as 1986.
- Around 2014, Karpathy took a ride in an early Waymo prototype around Palo Alto that executed a completely flawless drive, making full autonomy feel right around the corner more than a decade ago.
The reason the transition took so long is that self-driving belongs to a class of problems where the cost of failure is catastrophic. While casual "vibe coding" tolerates hallucinations, writing mission-critical production software or operating multi-ton vehicles in public spaces demands near-perfection. In software engineering, an unvetted code bug can leak hundreds of millions of Social Security numbers; in autonomous driving, a sudden perception failure causes fatal crashes.
```mermaid
flowchart LR
accTitle: The March of the Nines in High-Stakes AI
accDescr: Equal-effort engineering phases required to move from an initial demo to deployment.
D["Initial Demo (90% / 1 Nine)"] -->|"Constant unit of effort"| N2["99% (2 Nines)"]
N2 -->|"Constant unit of effort"| N3["99.9% (3 Nines)"]
N3 -->|"Constant unit of effort"| N4["99.99% (4 Nines)"]
N4 -->|"Constant unit of effort"| N5["Safety-critical deployment"]
```
Karpathy describes this brutal engineering reality as the "march of the nines":
- Getting a system to work 90% of the time provides an impressive demo, but that constitutes only the very first nine.
- Every successive nine of reliability (99%, 99.9%, 99.99%) demands roughly the exact same, massive amount of engineering work as the one before it.
- During his five years at Tesla, the team ground through two or three nines of improvement, and several more nines remain before full autonomy is solved.
Because of this grueling grind, Karpathy concludes with a hard-earned heuristic: “I'm very unimpressed by demos. Whenever I see demos of anything, I'm extremely unimpressed by that.”
[01:46:43](https://www.youtube.com/watch?v=lXUZvyajciY&t=6403s) The "March of Nines" and Safety Guarantees in Coding Agents
Andrej Karpathy distinguishes between quick tech demos and production systems that survive contact with the real world. A demo that someone whips up to show off is the least reliable indicator; while an interactive demo offers a slightly better signal, neither means the underlying problem is solved. True deployment requires facing messy operational realities and constantly patching edge cases. Karpathy frames this deployment phase as an arduous grind, observing: “It's a march of nines. Each nine is constant. Demos are encouraging. It’s still a huge amount of work to do.” He notes that software engineering is inherently a safety-critical domain—unless one is merely doing casual “vibe coding”—which heavily reinforces his longer timelines for full autonomy.
```mermaid
flowchart LR
accTitle: The Reality Gap in AI Deployment
accDescr: Progression from simple prototypes to production showing increasing failure surfaces.
subgraph Prototyping ["Surface Level"]
A["Cooked-up Demo"] --> B["Interactive Demo"]
end
subgraph Deployment ["Reality Gap"]
B --> C["Actual Product"]
C --> D["March of Nines<br/>(Safety & Edge Cases)"]
end
```
Dwarkesh Patel points out that the safety guarantees expected from software engineering resemble the high bar set for autonomous driving. In self-driving, the cost of failure is notoriously severe because human drivers only cause an accident roughly once every 400,000 miles, or about once every seven years. If a software coding agent were held to an equivalent standard—never making a critical error for seven continuous years—deployment would appear nearly impossible. Karpathy emphasizes that because an agent generates code at machine speeds, causing a catastrophic systems outage would take far less than seven years of real wall-clock time:
> “In fact, in terms of wall clock time, it would be much less than seven years because you're constantly outputting code like that.”
Dwarkesh adds that while self-driving focuses on a single vertical, software engineering encompasses an vastly broader operational surface area.
[01:48:14](https://www.youtube.com/watch?v=lXUZvyajciY&t=6494s) Why Broad Pretraining Doesn't Grant "Free" Generalization
Dwarkesh pushes back on the self-driving comparison by highlighting representation learning. Self-driving required decades just to build foundational computer vision representations and basic common sense—such as knowing how to react if a pedestrian waves down a car. With modern Large Language Models (LLMs) and Vision-Language Models (VLMs), broad world models and representations arrive pre-baked out of the box. Therefore, deploying AI across software tasks might look more like expanding an autonomous fleet into a new city rather than spending decades re-solving basic perception.
Karpathy disagrees that modern foundation models eliminate the core engineering slog:
> “I don't think that we're getting magical generalization completely out of the box, in a certain sense.”
While a single general model is broader than a narrow self-driving system, building narrow special-purpose systems can actually be harder because they do not benefit from internet-scale pretraining. Nevertheless, current LLMs remain deeply fallible, and their cognitive deficits and blind spots still require extensive manual stabilization before they can function reliably in mission-critical environments.
[01:49:42](https://www.youtube.com/watch?v=lXUZvyajciY&t=6582s) The True State of Autonomous Vehicles: Unit Economics and Hidden Humans
Turning back to the automotive sector, Karpathy stresses that self-driving is far from finished. Real-world deployments remain minuscule, and even market leaders like Waymo operate relatively few vehicles. In his view, companies like Waymo pulled a slice of the future forward at the expense of unit economics: high capital expenditures, steep sensor costs, and constant operational maintenance make their current fleets fundamentally uneconomical at scale.
```mermaid
flowchart TD
accTitle: Autonomous Vehicle Architecture vs Economics
accDescr: Comparison between sensor-heavy teleoperated fleets and scalable vision approaches.
subgraph Waymo ["High-Capex / Sensor Suite"]
W1["Custom Sensor Rig"] --> W2["Remote Teleoperation Center"]
W2 --> W3["Uneconomical Scaling"]
end
subgraph Tesla ["Scalable Vision Fleet"]
T1["Commodity Vision"] --> T2["End-to-End Neural Nets"]
T2 --> T3["Global Fleet Economics"]
end
```
Furthermore, Karpathy points out that driverless robotaxis often obscure the real human footprint behind the scenes:
> “In some sense, we haven't actually removed the person, we've moved them to somewhere where you can't see them.”
Extensive teleoperation centers keep humans in the loop, providing interventions when vehicles run into signal dead zones or edge cases. Having led the Autopilot team at Tesla for five years, Karpathy remains confident that Tesla's vision-focused approach is far more scalable than sensor-heavy alternatives:
- Waymo gained an early lead by packing vehicles with expensive sensor suites, but high hardware costs limit profitability.
- Autonomous vehicles cannot operate in several parts of San Francisco, likely due to poor cellular connectivity needed for remote teleoperation.
- True self-driving has not taken a decade; the effort dates back to the 1980s, and the final destination—where ordinary people no longer need driver's licenses—remains years away.
[01:52:13](https://www.youtube.com/watch?v=lXUZvyajciY&t=6733s) Bits vs. Atoms: Economic Differences in the AI Buildout
Dwarkesh asks whether the self-driving analogy breaks down when analyzing model economics and deployment speeds. Autonomous vehicles face strict real-time latency requirements, forcing them to run compact models (often on the order of tens of millions of parameters) on local car hardware. Moreover, scaling cars requires huge physical capital expenditures for each vehicle. By contrast, digital software models operate in the realm of pure bits: opex per session is tiny, inference costs can be amortized across training runs, and spinning up extra instances does not require manufacturing physical machines.
Karpathy readily acknowledges that pure software scales dramatically faster:
> “Bits are a million times easier than anything that touches the physical world.”
However, serving knowledge work at global scale will eventually hit hard computational latency constraints. Beyond raw compute, deploying software agents mirrors the societal hurdles faced by autonomous vehicles:
- Evolving legal frameworks and insurance liabilities.
- Societal backlash and edge-case sabotage (the digital equivalent of putting an orange traffic cone on a Waymo's hood to disable it).
- Reliance on unseen human supervisors acting as hidden "teleoperators" patching up AI agent failures behind the curtain.
[01:54:45](https://www.youtube.com/watch?v=lXUZvyajciY&t=6885s) Skepticism on Short Timelines vs. Bullishness on Compute
Dwarkesh queries whether the upcoming 10x to 100x expansion of global compute infrastructure risks creating an overbuilt bubble, echoing historical infrastructure booms like the 19th-century railroads or the late-1990s telecommunications fiber bubble that laid dark glass a decade before internet demand caught up.
Karpathy clarifies that while his skepticism of hyper-aggressive timelines makes him sound pessimistic, he is fundamentally bullish on the core technology:
> “I think we're going to be able to gobble up what, in my understanding, is being built.”
His frustration stems from the hype and aggressive timelines circulating on social media, which are largely driven by venture fundraising dynamics and engagement algorithms that convert sensationalism into cash:
- Technologies like Claude Code and OpenAI's Codex frameworks have emerged rapidly, demonstrating unprecedented utility that will consume massive amounts of compute.
- Massive consumer demand for tools like ChatGPT indicates that data center buildouts will find eager end markets.
- Having spent 15 years in AI research, Karpathy emphasizes that reputable figures repeatedly make overly optimistic forecasts, and the industry needs to calibrate its timelines against the grueling reality of software engineering.
[01:56:53](https://www.youtube.com/watch?v=lXUZvyajciY&t=7013s) Why Education Instead of an AI Frontier Lab
Dwarkesh Patel asks Andrej Karpathy why he decided to launch Eureka Labs rather than starting another frontier AI lab to tackle open research problems directly. Karpathy explains that he senses a deep feeling of determinism about the trajectory of existing frontier AI labs: they are well-funded, full of talent, and moving along an established path where his personal presence would only represent an incremental contribution. Instead, his deeper motivation stems from a concern about what becomes of human beings as artificial intelligence ascends.
```mermaid
flowchart TD
accTitle: Frontier Labs vs Eureka Mission
accDescr: Diagram showing Andrej Karpathy's choice between incremental AI lab work and empowering humans via Eureka.
subgraph Labs["Frontier AI Labs (Deterministic)"]
L1["Autonomous AI / Dyson Spheres"]
L2["Risk: Humanity sidelined ('WALL-E' / 'Idiocracy')"]
end
subgraph Eureka["Eureka Labs (Human-Centric)"]
E1["Building 'Starfleet Academy'"]
E2["Empowering Cadets with Frontier Tech"]
end
Karpathy["Andrej Karpathy's Focus"] -->|"Declined incremental lab work"| Labs
Karpathy -->|"Chosen path: Unique leverage"| Eureka
```
Karpathy fears a future in which superintelligent systems autonomously build Dyson spheres while humanity is sidelined, infantilized, and disempowered—a scenario reminiscent of movies like *WALL-E* or *Idiocracy*. He believes his unique leverage lies in uplifting human capability so that people thrive alongside technology. When asked to summarize the ultimate vision for Eureka, Karpathy describes it as an attempt to create the “Starfleet Academy” of the modern era: an elite, premier institution that prepares cadets with the most up-to-date, frontier technical knowledge needed to master and navigate emerging technologies.
> “My personal big fear is that a lot of this stuff happens on the side of humanity, and that humanity gets disempowered by it.” — Andrej Karpathy
---
[01:59:16](https://www.youtube.com/watch?v=lXUZvyajciY&t=7156s) The High Bar of One-on-One Tutoring and Current AI Slop
Dwarkesh asks how Karpathy approaches technical pedagogy, both in his YouTube lectures and through Eureka. Karpathy emphasizes that while education must fundamentally rewire itself around AI, current attempts to use large language models—such as prompting ChatGPT with questions—mostly produce low-quality "slop." The industry is still far too early to deliver what a true tutor provides.
To illustrate the gap, Karpathy shares his personal experience learning Korean through three distinct phases:
1. Self-study on the internet.
2. A small, 10-person group class in Korea.
3. Private, one-on-one tutoring with an exceptional teacher.
The private tutor experience set an extraordinarily high benchmark. Within minutes of a short conversation, the tutor diagnosed his exact internal world model, identified the boundaries of his knowledge, and dynamically calibrated every lesson to his specific sliver of competence. She kept him continuously in the sweet spot—never faced with trivial exercises, nor overwhelmed by impossibly hard concepts. Karpathy felt that all frictional impediments to education disappeared, leaving his own memory and effort as the sole bottlenecks. Today's AI models cannot replicate that level of proactive student-modeling; asking an LLM to teach AI today yields generic outputs rather than the tightly engineered progression a student needs.
> “Instantly from a very short conversation, she understood where I am as a student, what I know and don't know. She was able to probe exactly the kinds of questions or things to understand my world model. No LLM will do that for you 100% right now, not even close.” — Andrej Karpathy
---
[02:02:04](https://www.youtube.com/watch?v=lXUZvyajciY&t=7324s) Building "Ramps to Knowledge" and LLM101n
Because the capability for a fully automated, adaptive AI tutor does not exist yet, Karpathy views pushing for a purely autonomous agent right now as premature. He compares this judgment to his past computer vision consulting, where his greatest value-add was often telling clients *not* to use AI because the technology was unsuited for the job. Instead, Eureka is currently taking a hybrid, more conventional approach combining digital and physical components, starting with its flagship course, **LLM101n**.
Karpathy defines education not as soft storytelling or passive content consumption, but as a hard, technical engineering problem: building steep yet climbable **"ramps to knowledge."** A prime example is **nanochat**, a radically simplified, full-stack implementation of a conversational language model that serves as the capstone project for LLM101n. By stripping away extraneous framework complexity, nanochat provides maximum "eurekas per second" (moments of clear understanding per unit of time).
| Project / Course | Role & Architecture | Pedagogical Philosophy |
|:--- |:--- |:--- |
| **CS231n** (Stanford) | First deep learning course at Stanford; built manually without AI assistance. | Foundational deep learning curriculum for computer vision. |
| **LLM101n** (Eureka) | Flagship AI curriculum featuring human design augmented by LLM productivity tools. | Technical "ramp to knowledge" maximizing eurekas per second. |
| **nanochat** | Capstone artifact for LLM101n; a minimal, end-to-end full-stack chat model. | Complete transparency so the student is never stuck on unnecessary abstractions. |
While AI cannot yet architect these curricula creatively on its own, Karpathy uses current LLMs extensively behind the scenes to handle routine drafting and tedious tasks, dramatically accelerating course development compared to when he built CS231n at Stanford.
> “In my mind, education is the very difficult technical process of building ramps to knowledge.” — Andrej Karpathy
---
[02:04:38](https://www.youtube.com/watch?v=lXUZvyajciY&t=7478s) Scaling Eureka: Faculty, Automated TAs, and the Future
Dwarkesh observes that Eureka's current moat relies heavily on Karpathy's personal gift for pedagogical framing, asking whether the company will face a bottleneck in finding other experts who can convert complex domains into similar conceptual ramps. He wonders whether Karpathy plans to teach every topic himself—analogous to Sal Khan narrating the videos on Khan Academy.
Karpathy clarifies that he will not follow the single-instructor model. While he will personally lead AI education for the foreseeable future, Eureka plans to hire expert faculty across other scientific and technical disciplines where he lacks domain mastery. The near-term workflow pairs world-class human faculty with AI tools to architect top-tier source materials. Over time, as student cohorts progress through the courses, Eureka will deploy fine-tuned automated TAs grounded in these verified materials to handle routine student queries, gradually transferring responsibilities to AI while retaining human faculty for high-level course architecture. Eventually, model capabilities may evolve to the point where AI designs curricula better than humans, but Karpathy stresses that this transition will take significant time to mature.
> “I will hire faculty because there are domains in which I'm not an expert. That's the only way to offer the state-of-the-art experience for the student ultimately.” — Andrej Karpathy
[02:07:29](https://www.youtube.com/watch?v=lXUZvyajciY&t=7649s) Starfleet Academy and Re-Educating for the AI Era
Andrej Karpathy outlines his tangible concept for "Starfleet Academy," conceiving it not merely as software, but as a tiered educational institution designed from first principles for the modern technological landscape. At the top tier sits an intensive physical institution where students attend full-time, working through curricula from start to end with guaranteed comprehension. Below that exists a more accessible digital tier comprising web materials and LLM assistants. While Karpathy admits the digital version is somewhat more gimmicky and inferior to the full in-person experience, its virtue lies in scale: it can reach eight billion people. The central mission is reinventing higher education around current tools while filtering for deeply motivated students who want to engage directly with the material.
A crucial driver for this project is the massive need for re-education as automation reshapes employment. While people are already aggressively attempting to upskill in AI, economic necessity simplifies the motivation problem in the near term: people study technical disciplines because they need to earn a living.
```mermaid
flowchart TD
accTitle: Starfleet Academy Educational Model
accDescr: Two tiers of education feeding into workforce re-education and future learning.
subgraph Offerings["Starfleet Academy Model"]
direction TB
A["Physical Institution<br/>(Full-time immersive, guaranteed mastery)"]
B["Digital Offering<br/>(LLM assistants, web materials, globally scalable)"]
end
subgraph DrivingForces["Motivational Eras"]
C["Pre-AGI: Economic Utility<br/>(Upskilling, AI literacy, career necessity)"]
D["Post-AGI: Cognitive Flourishing<br/>(Fun, self-betterment, intellectual athletics)"]
end
A --> C
B --> C
C --> D
```
- Andrej Karpathy notes the practical incentives of the present era: “Motivation-wise, before AGI motivation is very simple to solve because people want to make money.”
- The near-term curriculum focuses heavily on hands-on AI literacy and engineering to match the rapid pace of workplace transformation.
[02:08:46](https://www.youtube.com/watch?v=lXUZvyajciY&t=7726s) Post-AGI Education and the Gym Culture Analogy
When Dwarkesh Patel asks why anyone would bother going to school once AGI automates all productive labor, Karpathy draws a direct parallel to modern physical fitness. Prior to AGI, education serves utility; after AGI, education becomes an intrinsically enjoyable pursuit. Centuries ago, no one worked out with barbells because manual labor provided all necessary physical exertion. Today, machines handle heavy lifting, yet gyms are packed because exercise is healthy, fun, aesthetically rewarding, and deeply rooted in evolutionary biology. Karpathy envisions education following the exact same path: intellectual training will become a widespread cultural pursuit once the technical friction of learning is removed.
Patel pushes on whether education in a post-AGI world is intended as mere entertainment or as a mechanism for keeping humanity in control of AI. Karpathy concedes that attempting to outcompete superintelligence in the long run is ultimately a losing battle. However, he maintains that human cognitive capacity has barely scratched the surface because learners constantly bounce off material that is poorly calibrated—either far too difficult or completely boring. With hyper-personalized AI tutoring, intellectual mastery will become trivial, enabling ordinary people to speak five languages or master undergraduate-level science curricula simply because the friction has evaporated.
- Andrej Karpathy states his core thesis on future learning: “I often say that pre-AGI education is useful. Post-AGI education is fun. In a similar way, people go to the gym today.”
- On the limits of human potential under current educational constraints, Karpathy explains: “I feel that the geniuses of today are barely scratching the surface of what a human mind can do, I think.”
- Dwarkesh notes the historical parallel: one hundred years ago, almost nobody was spontaneously bench-pressing two or three 45-pound plates or running marathons, yet systematic training made those once-rare physical feats commonplace.
[02:11:48](https://www.youtube.com/watch?v=lXUZvyajciY&t=7908s) Cognitive Powerlifting and Avoiding *Idiocracy*
Karpathy grounds his optimism about human curiosity in historical precedent, pointing to ancient Greece and historical aristocratic classes. Whenever small pockets of society achieved conditions resembling a post-scarcity, post-AGI environment, their members often dedicated themselves to cognitive and physical flourishing rather than passive decay. Karpathy rejects the idea of accepting a future dominated by apathy, arguing that even grand cosmic engineering projects are meaningless if humanity degenerates.
During the transition toward full superintelligence, deep human technical understanding will still be vital to keep people in the decision-making loop. Further out, Karpathy speculates that extreme intellectual pursuits may evolve into competitive cognitive athletics—a form of mental powerlifting where individuals train with AI tutors to push the limits of human memory and problem-solving in Olympic-style exhibitions.
- Andrej Karpathy expresses his aversion to civilizational stagnation: “If this is false and I'm wrong and we end up in a WALL-E or Idiocracy future, then I don't even care if there are Dyson spheres. This is a terrible outcome.”
- Dwarkesh Patel notes his own deep resonance with this personalized learning vision, joking that as a podcast host who must master new, complex subjects every week, he represents Karpathy's ideal target customer.
- Reflecting on his own academic trajectory, Karpathy admits: “I stayed all the way until Ph.D. and then they wouldn't let me stay longer, so I went to the industry.”
[02:14:32](https://www.youtube.com/watch?v=lXUZvyajciY&t=8072s) First-Principles Pedagogy and the Physics Mindset
Dwarkesh Patel asks why previous waves of online education (such as MOOCs) failed to educate the entire world, and what advice Karpathy would offer to educators in non-coding fields. Karpathy explains that traditional online courses fail because they carry an immense motivational burden: when students encounter confusing concepts, they hit a wall, receive negative reward signals, and drop out. High-quality human tutoring works precisely because it constantly adapts to the edge of the student's ability, preventing boredom on one end and frustration on the other.
When structuring technical explanations, Karpathy attributes his pedagogical success to his background in physics. Rather than treating early education as mere fact memorization for future employment, he views it as an exercise in "booting up a brain." Physics instills the habit of stripping away real-world noise to identify the core mathematical drivers—the fundamental first-order approximations—before layering on secondary and tertiary complexities.
```mermaid
flowchart LR
accTitle: Physics-Based Pedagogical Abstraction
accDescr: Breaking down a tangled real-world problem into first-order models before adding complexity.
A["Noisy Real-World System<br/>(Tangled web of concepts)"] --> B["First-Order Model<br/>('Spherical Cow' core)"]
B --> C["Working Demonstration<br/>(Minimal implementation, e.g., micrograd)"]
C --> D["Higher-Order Terms<br/>(Edge cases, optimizations, scale)"]
```
- Andrej Karpathy explains the fundamental purpose of early scientific education: “Early school education is not about accumulating knowledge or memory for tasks later in the industry. It's about booting up a brain.”
- Karpathy defends the classic physics cliché: “When a physicist walks into the class and they say, 'Assume there's a spherical cow,' everyone laughs at that, but this is brilliant.”
- He highlights the book *Scale* (by physicist Geoffrey West) as a masterclass in applying physical modeling to biology, showing how metabolic scaling laws and heartbeat frequencies emerge from the geometric tension between volume (which scales as a cube, driving heat generation) and surface area (which scales as a square, governing heat dissipation).
- To illustrate how he implements this first-order philosophy in code, Karpathy points to his educational library `micrograd`, which condenses the core mechanics of backpropagation and neural network building blocks into roughly 100 lines of Python.
[02:18:14](https://www.youtube.com/watch?v=lXUZvyajciY&t=8294s) Micrograd, Knowledge Untangling, and the Pedagogical Ramp
Andrej Karpathy explains that the fundamental mechanism behind all neural network training is remarkably concise: you construct a computational graph, perform a forward pass, and execute a backward pass to obtain the gradients via backpropagation. To demonstrate this, he built micrograd, an interpretable Python implementation of roughly 100 lines that can train arbitrary neural architectures. While it is unoptimized, it captures the entire intellectual core of deep learning: “So micrograd, these 100 lines of Python, are everything you need to understand how neural networks train.”
Everything built beyond micrograd—tensor layouts, striding, memory orchestration, and optimized GPU compute kernels—is merely an engineering concession to efficiency. For Karpathy, uncovering these irreducible "small-order terms" and presenting them clearly is the essence of education. He views education as a puzzle of taking a messy tangle of concepts and laying them out along a linear ramp where each idea strictly depends only on what preceded it.
Dwarkesh Patel points out how this philosophy shapes Karpathy’s transformer tutorial, which begins not with complex multi-head attention, but with simple character-level bigram lookup tables. Dwarkesh notes that by introducing lookup tables first, every subsequent layer—embeddings, positional encodings, attention mechanisms—is motivated as a targeted solution to an explicit failure mode, presenting the pain before offering the solution. Karpathy agrees that good teaching requires active cognitive engagement from the student rather than passive consumption:
> “That’s a little bit of a…I don’t want to swear but it’s a dick move towards you to present you with the solution before I give you a shot to try to come up with it yourself.”
When students struggle to devise a mechanism on their own, they understand the true constraints of the action space and why a particular design is uniquely necessary, maximizing the information gained per fact introduced.
```mermaid
flowchart TD
accTitle: The Karpathy Pedagogical Ramp
accDescr: A progressive teaching pipeline moving from pain points to solutions and minimal implementations to scaled systems.
subgraph Motivation ["Problem-First Progression"]
P["Expose the Problem / Pain"] --> G["Prompt Student Guess / Explore Action Space"]
G --> S["Introduce Minimal Solution"]
end
subgraph Abstraction ["Core vs Efficiency"]
S --> MG["Micrograd / Core Concept<br/>(~100 lines of pure logic)"]
MG --> SC["Scale & Hardware Optimization<br/>(Tensors, kernels, memory striding)"]
end
```
- **Micrograd’s footprint**: ~100 lines of clean Python implementing scalar-level automatic differentiation via a recursive chain rule.
- **Pedagogical rule**: Always prompt students to hypothesize an answer before revealing the canonical solution, cementing why that specific architecture was chosen.
---
[02:21:02](https://www.youtube.com/watch?v=lXUZvyajciY&t=8462s) The Curse of Knowledge and the "Just Say the Thing" Phenomenon
Dwarkesh asks why true domain experts frequently struggle to teach beginners. Karpathy identifies this as the classic "curse of knowledge and expertise"—experts take foundational leaps for granted and lose the ability to inhabit the mental state of someone approaching the field from zero. Karpathy admits that he regularly suffers from this blind spot himself. To overcome it when encountering dense fields like biology, he uploads papers into ChatGPT, asks his basic questions, and forwards the entire transcript to the authors so they can see where an outsider gets tripped up. He encourages students viewing his own deep learning materials to share their naive LLM chat logs with him so he can identify gaps in his explanations.
Dwarkesh highlights a recurring phenomenon in technical communication: across papers, blog posts, and company announcements, the author's casual, spoken explanation over lunch is almost always more accurate, concise, and scientific than their formal writing. Authors suffer from an impulse to front-load four paragraphs of throat-clearing and wrap simple ideas in impenetrable jargon, whereas face-to-face dialogue forces directness. Karpathy strongly agrees, recalling his graduate school years:
> “It’s only when you’re sitting at the table with a beer or something, and they’re like, ‘Oh yeah, the paper is just, you take this idea, you take that idea and try this experiment and you try out this thing.’”
In three conversational sentences over a beer at an academic conference, a researcher can effortlessly communicate the entire core intuition that had been completely obscured within a 10-page paper.
---
[02:23:47](https://www.youtube.com/watch?v=lXUZvyajciY&t=8627s) Learning on Demand and Explaining as Understanding
Dwarkesh asks what strategies independent students should adopt when trying to learn a technical domain without a dedicated tutor or curated curriculum. Karpathy emphasizes balancing two distinct approaches: breadth-wise learning and depth-wise, on-demand learning. Traditional schooling relies almost exclusively on breadth-first instruction ("learn this now because trust me, you will need it later"), which lacks immediate motivation. Karpathy prefers project-driven, on-demand learning, where one dives deeply into a narrow vertical slice of knowledge strictly to unlock a working project and claim the associated reward.
Finally, Karpathy highlights the Feynman technique as the ultimate tool for deep mastery: actively trying to teach or re-explain the concept to someone else. Whenever he attempts to explain a topic and hits an awkward conceptual barrier, it exposes an unaddressed deficiency in his own mental model:
> “This is an aspect where education is a bit more selfless, but explaining things to people is a beautiful way to learn something more deeply.”
Re-explaining forces an individual to manipulate the raw ideas and reconcile hidden contradictions, turning passive familiarity into genuine, durable understanding. Dwarkesh concludes the conversation by noting that this perspective is an ideal place to wrap up the interview.
## Notable quotes
> [02:14:32](https://www.youtube.com/watch?v=lXUZvyajciY&t=8072s) Early school education is not about accumulating knowledge or memory for tasks later in the industry. It's about booting up a brain. — Andrej Karpathy
> [07:54](https://www.youtube.com/watch?v=lXUZvyajciY&t=474s) We're not building animals. We're building ghosts or spirits or whatever people want to call it, because we're not doing training by evolution. We're doing training by imitation of humans and the data that they've put on the Internet. — Andrej Karpathy
> [01:52:13](https://www.youtube.com/watch?v=lXUZvyajciY&t=6733s) Bits are a million times easier than anything that touches the physical world. — Andrej Karpathy
> [02:08:46](https://www.youtube.com/watch?v=lXUZvyajciY&t=7726s) I often say that pre-AGI education is useful. Post-AGI education is fun. In a similar way, people go to the gym today. — Andrej Karpathy
> [55:16](https://www.youtube.com/watch?v=lXUZvyajciY&t=3316s) We're not actually that good at memorization, which is actually a feature. Because we're not that good at memorization, we're forced to find patterns in a more general sense. — Andrej Karpathy
> [17:29](https://www.youtube.com/watch?v=lXUZvyajciY&t=1049s) I do think that anything that's in the weights, it's a hazy recollection of what you read a year ago. Anything that you give it as a context at test time is directly in the working memory. — Andrej Karpathy
> [02:02:04](https://www.youtube.com/watch?v=lXUZvyajciY&t=7324s) In my mind, education is the very difficult technical process of building ramps to knowledge. — Andrej Karpathy
## Mentioned
AlexNet, AlphaGo, Andrej Karpathy, Andy Matuschak, Carnegie Mellon University, ChatGPT, Claude, Claude Code, Codex, CS231n, DeepMind, DeepSeek-V3.2, Distributed Data Parallel (DDP), Dwarkesh Patel, Eureka Labs, Geoffrey Hinton, Geoffrey West, Google, Google DeepMind, GPT-4, GPT-5 Pro, gpt-oss-20b, Gwern, ImageNet, InstructGPT, Khan Academy, Llama 3 70B, LLM101n, Meta, micrograd, nanochat, Nick Lane, NVIDIA, OpenAI, OpenAI Universe, PyTorch, Quintin Pope, Rich Sutton, Richard Feynman, Rust, Sal Khan, Scale, Tesla Autopilot, University of Toronto, Visual Studio Code, Waymo, Yann LeCun
## Video description
The Andrej Karpathy episode. During this interview, Andrej explains why reinforcement learning is terrible (but everything else is much worse), why AGI will just blend into the previous ~2.5 centuries of 2% GDP growth, why self driving took so long to crack, and what he sees as the future of education. It was a pleasure chatting with him.
𝐒𝐏𝐎𝐍𝐒𝐎𝐑𝐒
* Labelbox helps you get data that is more detailed, more accurate, and higher signal than you could get by default, no matter your domain or training paradigm. Reach out today at https://labelbox.com/dwarkesh
* Mercury helps you run your business better. It’s the banking platform we use for the podcast — we love that we can see our accounts, cash flows, AR, and AP all in one place. Apply online in minutes at https://mercury.com
* Google’s Veo 3.1 update is a notable improvement to an already great model. Veo 3.1’s generations are more coherent and the audio is even higher-quality. If you have a Google AI Pro or Ultra plan, you can try it in Gemini today by visiting https://gemini.google
To sponsor a future episode, visit https://dwarkesh.com/advertise
𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒
00:00:00 – AGI is still a decade away
00:30:33 – LLM cognitive deficits
00:40:53 – RL is terrible
00:50:26 – How do humans learn?
01:07:13 – AGI will blend into 2% GDP growth
01:18:24 – ASI
01:33:38 – Evolution of intelligence & culture
01:43:43 - Why self driving took so long
01:57:08 - Future of education
## Background
本期访谈由 Dwarkesh Patel 于 2025 年 10 月 17 日在 YouTube 频道发布。视频简介指出，知名人工智能专家 Andrej Karpathy 在节目中深入探讨了强化学习的根本局限、通用人工智能（AGI）为何将平滑融入过去两个半世纪约 2% 的 GDP 增长常态、自动驾驶技术历经漫长攻坚的原因，以及他在超级智能时代对教育未来的愿景。
## TL;DR
Andrej Karpathy 认为业内所谓的“智能体之年”应修正为“智能体时代”，AGI 的全面落地至少还需要约 10 年的技术沉淀。他指出大语言模型本质是在互联网数据上“召唤幽灵”而非复制生物演化，强化学习目前充斥着高方差噪声且缺乏真正的因果复盘机制。针对宏观经济，他反驳了“盒中之神”将瞬间引发爆炸性跃迁的观点，认为 AI 的渗透将遵循类似编译器与计算技术的平滑轨迹，融入长期 2% 的 GDP 增长曲线。最后，他分享了自动驾驶攻克长尾“9 的进军”的工程教训，并阐述了创办 Eureka 以通过构建认知阶梯赋能人类、防止人类在超级智能时代走向边缘化退化的教育愿景。
## Atmosphere & style
整场对话呈现出一种罕见的清醒、严谨与务实工程师底色，完全剥离了科技界常见的炒作浮夸。Karpathy 以极富洞察力的比喻（如“召唤幽灵而非制造动物”、“用吸管吸取监督”）拆解底层架构，在深入技术细节时眼神专注、直言不讳，甚至直斥当下的 AI 垃圾信息（slop）与虚妄时间表。Dwarkesh Patel 的提问则紧扣宏观历史与经济学核心张力，两人在推演技术奇点与产业现实的交锋中节奏紧凑，既有对技术复杂性的敬畏与审慎，又始终洋溢着对人类智力成长与硬核教学的热忱。
## Bulletin
- [00:48](https://www.youtube.com/watch?v=lXUZvyajciY&t=48s) Andrej Karpathy 认为业内所谓的“智能体之年”应修正为“智能体时代”，大模型由于缺乏持续学习、多模态以及稳定的电脑使用能力，彻底攻克认知缺陷平均需要约 10 年时间。
- [17:28](https://www.youtube.com/watch?v=lXUZvyajciY&t=1048s) 大模型在权重同化信息与工作记忆间存在 3500 万倍的压缩鸿沟，未来理想范式应剥离绝大部分静态事实记忆，提炼出仅保留推理算法与工具检索能力的纯粹“认知核心”。
- [40:53](https://www.youtube.com/watch?v=lXUZvyajciY&t=2453s) 强化学习在当前大模型训练中极其低效，通过末端单一标量“用吸管吸取监督”并给整条轨迹赋予正向梯度，导致高方差噪声和对抗样本漏洞频发。
- [01:22:20](https://www.youtube.com/watch?v=lXUZvyajciY&t=4940s) AI 不会带来突发性的经济奇点或十倍级跃升，而是通过自主性滑块逐步渗透至现实产业，最终融入人类历史约 2% 的长期 GDP 增长走势。
## Key takeaways
- [07:54](https://www.youtube.com/watch?v=lXUZvyajciY&t=474s) AI 发展是在“召唤幽灵”而非复制生物演化，现代大模型是通过模仿互联网人类数字化痕迹构建的全新数字实体，而非生物演化出来的动物。
- [10:53](https://www.youtube.com/watch?v=lXUZvyajciY&t=653s) 大模型的预训练本质上是一种“拙劣的进化”，人类基因仅遗传了约 3GB 的学习算法，而大模型直接固化了全部陈述性事实记忆与算法智能回路。
- [17:28](https://www.youtube.com/watch?v=lXUZvyajciY&t=1048s) 当前 Transformer 架构本质上仅对应生物大脑中的一小块大脑皮层组织，系统仍缺失海马体、杏仁核及远古脑核团等关键大脑解剖结构。
- [22:04](https://www.youtube.com/watch?v=lXUZvyajciY&t=1324s) 大模型缺乏人类通过睡眠将白天工作记忆离线蒸馏回突触权重的持续学习机制，零 Token 启动时会彻底重置。
- [24:51](https://www.youtube.com/watch?v=lXUZvyajciY&t=1491s) 未来 10 年 AI 架构极大概率依然沿用前向传播、反向传播与梯度下降训练庞大神经网络，性能提升来自于全栈各环节全面提升 20% 的叠加效应。
- [34:15](https://www.youtube.com/watch?v=lXUZvyajciY&t=2055s) AI 在代码补全和迁移已有测试用例上极具生产力，但在编写从未有人写过的创新架构代码时极易崩溃，难以在短期内实现底层自主研发的递归跃迁。
- [46:07](https://www.youtube.com/watch?v=lXUZvyajciY&t=2767s) 过程监督与大模型裁判存在高维几何对抗漏洞，学生模型容易通过插入无意义乱码字符（如“dhdhdhdh”）骗取满分，导致长链条强化学习难以维持。
- [49:46](https://www.youtube.com/watch?v=lXUZvyajciY&t=2986s) 大模型基于自身合成数据循环训练会导致其概率分布发生“静默坍缩”，做梦可能是生物进化出来防止过拟合与模式崩塌的关键适应性机制。
- [01:00:01](https://www.youtube.com/watch?v=lXUZvyajciY&t=3601s) 未来 20 年内可能仅需 10 亿参数（1B）规模就能构建高通用性的认知内核，将事实细节全部剥离给外部检索工具，从而大幅降低计算冗余。
- [01:07:13](https://www.youtube.com/watch?v=lXUZvyajciY&t=4033s) 纯数字知识工作仅占宏观经济总量的 10% 到 20%，真正的 AGI 落地会像呼叫中心自动化一样通过自主权滑块渐进渗透，而非瞬间出现全知全能的“盒中之神”。
- [01:18:24](https://www.youtube.com/watch?v=lXUZvyajciY&t=4704s) 超级智能带来的最大潜在隐患不是突发性安全事件，而是人类在层层堆叠的智能系统面前，对底层运行机理与社会架构掌控力的逐步瓦解。
- [01:43:43](https://www.youtube.com/watch?v=lXUZvyajciY&t=6223s) 高可靠性工业落地是一场“追逐 9 的长征”，自动驾驶达到 90% 只是开始，后续每一个 9 的突破难度均呈指数级递增，目前仍高度依赖远程后台兜底。
- [01:56:53](https://www.youtube.com/watch?v=lXUZvyajciY&t=7013s) 教育本质上是为复杂知识搭建坚实“爬坡阶梯”的工程技术问题，AGI 时代的教育将从经济实用导向转变为类似健身健美的“认知健身”与纯粹追求。
- [02:07:29](https://www.youtube.com/watch?v=lXUZvyajciY&t=7649s) 物理学第一性原理思维倡导先抓住解释大部分现象的“一阶近似”，如 micrograd 仅用约 100 行 Python 代码即可涵盖神经网络训练的全部智力核心。
## Detailed walkthrough
### [00:48](https://www.youtube.com/watch?v=lXUZvyajciY&t=48s) 为什么是“智能体时代”而非“智能体之年”
面对业界所谓“今年是智能体（Agent）之年”的乐观论调，Andrej Karpathy 提出了不同的看法，将其修正为“智能体时代”（The decade of agents）。他直言这一判断是对当前 AI 圈内过度预测与炒作的回应。尽管像 Claude 和 Codex 这样的工具已经非常惊艳且成为他的日常工具，但他深知要构建真正意义上的智能体，前路依然漫长。
谈及究竟是什么阻碍了智能体的落地，Karpathy 将期望中的智能体比作人类雇佣的员工或实习生。如果今天有人问为什么不能直接让 Claude 或 Codex 代替全职员工来完成播客制作或日常运营，根本原因在于“它们目前就是无法正常工作”。当前模型在认知上存在诸多缺陷：缺乏足够的多模态理解能力、无法稳定使用电脑（Computer Use）、不具备持续学习（Continual Learning）能力，无法做到像人类那样被告知一件事后就牢牢记住并随时调用。
Dwarkesh Patel 追问，既然业内能明确指出缺乏持续学习或多模态能力等瓶颈，为何完成这些攻坚需要“十年”，而不是一年或五十年。Karpathy 解释称，这一时间跨度源自他在深度学习领域摸爬滚打约十五年的直觉外推。在这期间，他亲眼见证了多次行业周期的起伏与同行预测的兑现情况。在综合了学术研究与产业落地经验后，他认为这些问题是完全可解、可以克服的，但难度极大，“如果把这些因素平均一下，在我看来这大约需要十年时间”。
> “The reason you don't do it today is because they just don't work. They don't have enough intelligence, they're not multimodal enough, they can't do computer use and all this stuff. They don't do a lot of the things you've alluded to earlier. They don't have continual learning. You can't just tell them something and they'll remember it. They're cognitively lacking and it's just not working. It will take about a decade to work through all of those issues.” —— Andrej Karpathy
---
### [03:47](https://www.youtube.com/watch?v=lXUZvyajciY&t=227s) 早期 AI 的弯路：在没有表征基石前死磕强化学习与智能体
回顾 AI 发展史上的几次地壳运动般的范式转移（Seismic Shifts），Karpathy 分享了亲历这些突破时刻的感受与教训：
1. **特定任务的单点突破**：在多伦多大学跟随 Geoffrey Hinton 学习期间，深度学习仍是被边缘化的非主流方向。直到 2012 年 AlexNet 诞生，才引发了第一波巨大震动，所有人瞬间转向神经网络，但当时的做法主要局限于针对特定任务逐个训练专用模型（如图像分类器、神经机器翻译）。
2. **过早追逐智能体与游戏强化学习**：人们在视觉皮层取得突破后，便急于攻克大脑的其他部分，期望让智能体不仅能感知，还能在环境中行动并获得奖励。2013 年 DeepMind 的 Atari 深度强化学习论文掀起了全行业狂热。Karpathy 认为这是整个领域的一次“误入歧途”（Misstep），当时的 OpenAI 也身陷其中，几年来整个 AI 界都沉迷于在各类游戏环境中打分数。
3. **Universe 项目与表征学习的缺失**：Karpathy 当时就对“刷游戏能通向通用人工智能（AGI）”持怀疑态度，他认为 AGI 需要像会计师那样与真实的数字世界交互、从事知识工作。因此，他在 OpenAI 推动了 Universe 项目，试图让智能体通过键鼠操作浏览器和桌面软件。然而，由于走得太超前，项目遭遇了挫败——让一个毫无先验表征的智能体盲目乱点鼠标、敲键盘，在没有稠密奖励的环境中探索，“奖励信号极其稀疏，根本学不会，最终只会烧掉整座森林的算力却一无所获”。
Karpathy 指出，今天的 Computer-using Agent 之所以能够起步，核心在于它们建立在大型语言模型（LLM）的基石之上。只有先通过海量无监督预训练获取强大的表征能力，才能在模型上方叠加行动与工具调用。早期研究的根本问题在于，人们试图跳过底层表征积累阶段，过早地直接攻关完整的端到端智能体。
> “Because if you're just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments, your reward is too sparse and you just won't learn. You're going to burn a forest computing, and you're never going to get something off the ground. What you're missing is this power of representation in the neural network.” —— Andrej Karpathy
```mermaid
flowchart TD
accTitle: AI Paradigm Evolution
accDescr: Evolution of AI paradigms from early task-specific networks, through the premature reinforcement learning agent phase, to modern representation-first LLM agents.
subgraph Phase1["第一阶段：特定任务感知"]
AlexNet["AlexNet (2012)"] --> PerTask["单任务模型<br/>(图像分类 / 机器翻译)"]
end
subgraph Phase2["第二阶段：过早的智能体尝试 (误入歧途)"]
Atari["Atari 强化学习 (2013)"] --> Games["游戏冲榜热潮"]
Universe["OpenAI Universe 项目<br/>(直接用键鼠盲目探索)"] --> SparseFail["奖励极度稀疏<br/>缺乏先验，算力耗尽失败"]
end
subgraph Phase3["第三阶段：现代表征驱动范式"]
LLMPretrain["海量预训练 (LLM)"] --> Repr["建立强大的底层世界表征"]
Repr --> ModernAgents["现代智能体<br/>(构建在 LLM 之上的键鼠/代码工具调用)"]
end
PerTask --> Atari
Games -.->|"跳过表征，过早端到端"| Universe
SparseFail -->|"反思教训"| LLMPretrain
```
---
### [07:54](https://www.youtube.com/watch?v=lXUZvyajciY&t=474s) 召唤幽灵，而非制造动物：反思强化学习与生物智能的假象
Dwarkesh Patel 抛出了强化学习先驱 Richard Sutton 的观点：人类尤其是动物，在完全没有语言支架和人工标注的情况下被直接丢进现实世界，仅凭感知流就能自主理解环境并学会生存。既然如此，为何 AGI 不能同样从纯感知数据与环境交互中从零学习，而必须依赖人工投喂的数据进行数百万年人类智慧的模仿训练？
Karpathy 对这种将 AI 与动物直接类比的做法提出了深刻质疑。他指出，生物系统源自一种截然不同的优化机制——漫长的生物演化：
- **演化内置了绝大部分能力**：斑马幼崽刚出生几分钟就能站立奔跑并跟随母兽，这种极其复杂的协同感知行为根本不可能依靠幼崽出生后的强化学习获得，而是演化将神经网络的初始权重编码进了脱氧核糖核酸（DNA）的 ATCG 碱基序列中。
- **成熟而非学习**：生物表面上看起来像“从零学习”的过程，本质上大多是大脑生理结构在时间推移下的“发育成熟”（Maturation），而非算法在进行实时的策略迭代。
- **生物强化学习的狭窄范围**：强化学习在人类和动物身上所起的作用非常有限，主要局限于底层运动控制任务（如学习投篮、肌肉协调），人类的高维智能活动基本不依靠强化学习驱动。
基于此，Karpathy 强调了自己发表过的核心观点：我们当前正在制造的不是动物，而是在“召唤幽灵”（Summoning ghosts/spirits）。现代大模型走的是对全网人类数字化痕迹进行模仿学习的路径，这是一种纯粹基于数字和模仿机制的全新智能形态。倘若像 Sutton 那样期待存在某种单一代币级别的极简通用算法，丢到互联网上就能自发学会一切，那不过是一种忽视了演化外环作用的美好幻想。
> “In my post, I said we're not building animals. We're building ghosts or spirits or whatever people want to call it, because we're not doing training by evolution. We're doing training by imitation of humans and the data that they've put on the Internet. You end up with these ethereal spirit entities because they're fully digital and they're mimicking humans.” —— Andrej Karpathy
- **演化外环与个体学习的区分**：动物大脑并不是从未经初始化的白纸开始学习；演化在亿万年里承担了极其庞大的“预训练”计算，个体生长的“学习”只是生物蓝图的逐步解压与硬件成熟。
- **智能形态的空间分化**：以互联网全量语料为起点的模仿智能（幽灵），与以物理生存、运动控制为起点的演化智能（动物），在智能图谱中处于完全不同的坐标原点，不应混为一谈。
### [10:53](https://www.youtube.com/watch?v=lXUZvyajciY&t=653s) 召唤幽灵而非造动物：预训练作为“拙劣的进化”
Andrej Karpathy 指出，人类在解决复杂问题等大多数高阶智能任务时，并不像很多研究假设的那样主要依赖强化学习（RL）。Dwarkesh Patel 顺此提出质疑：如果说预训练类似于自然选择塑造理解世界的底层结构，二者却存在根本差异——人类进化只能通过精简的约 3GB DNA 序列完成遗传，这根本无法编码人脑数以万亿计的每一个突触；DNA 编码的更像是一套“在后天生命历程中自主学习的算法”，而大语言模型的预训练则是将整个大脑的网络权重直接固化下来。Patel 借用 Rich Sutton 的观点追问：进化给予生命的是“寻找知识的算法”，而不是知识本身，这与灌输了海量人类知识的预训练显然截然不同。
Karpathy 认可 DNA 中存在着难以置信的极端压缩与在线学习机制，但他强调自己是一名头戴安全帽、只关注工程实用的构建者。他指出当今的技术根本无法重现真实的演化生物学，人们只能退而求其次，利用互联网文本这一唯一具备足够规模的数据源：
> “I come from it from the perspective of, let's build useful things. I have a hard hat on, and I'm just observing that we're not going to do evolution, because I don't know how to do that. But it does turn out we can build these ghosts, spirit-like entities, by imitating internet documents. This works. It's a way to bring you up to something that has a lot of built-in knowledge and intelligence in some way, similar to maybe what evolution has done. That's why I call pre-training this crappy evolution.”
> —— Andrej Karpathy
在 Karpathy 看来，互联网上的“下一个 Token 预测”（next-token prediction）在模型内部实际上同时触发了两件毫不相干的事情：第一，模型强行记住了海量的陈述性知识；第二，由于互联网文本蕴含大量算法、推导与逻辑规律，模型在优化预测时自发点亮了各种微型神经回路，从而形成了上下文学习（in-context learning）等智能涌现。然而，这种将知识与思维能力捆绑在一起的结构反而在阻碍模型的发展——庞大的记忆让模型过度依赖死记硬背，一旦面对脱离训练数据流形（off the data manifold）的陌生情境就极易崩溃。Karpathy 认为未来的核心研究范式，必须是剥离这些不必要的庞杂知识，提炼出一个纯粹的“认知核心”（cognitive core），只保留通用的推理策略、解题算法与思维能力。
```mermaid
flowchart TD
accTitle: Internet Pre-training Dual Mechanism
accDescr: 互联网预训练同时产生非必要的冗余知识记忆与核心的算法智能回路。
subgraph INGEST["数据输入"]
A["海量互联网文档<br/>Next-token 预测目标"]
end
subgraph DUAL["预训练的双重副产物"]
B["冗余知识记忆<br/>（死记硬背、易产生幻觉与过拟合）"]
C["计算回路与算法能力<br/>（上下文学习、推理逻辑、解题策略）"]
end
subgraph FUTURE["未来演进目标"]
D["认知核心 (Cognitive Core)<br/>（剥离记忆负担，专注于纯粹推理与智能）"]
end
A --> B
A --> C
C --> D
```
---
### [14:39](https://www.youtube.com/watch?v=lXUZvyajciY&t=879s) 上下文学习与模型内部的隐式梯度下降
在探讨大模型的智能表现时，Patel 提到最让人产生“对面有真实心智在思考”的瞬间，往往是模型在对话中犯错后能即时自我修正、反思撤回的动态表现，而这些完全发生在上下文窗口（Context Window）之中。尽管这种能力最初是由预训练期间的梯度下降元学习（meta-learning）出来的，但上下文本身的展开过程并未进行显式的权重更新。
Karpathy 对此提出了极为深入的技术保留，他推测上下文学习在神经网络内部可能实际上在模拟某种特殊的梯度下降机制：
> “I do think it's possible that in-context learning runs a small gradient descent loop internally in the layers of the neural network.”
> —— Andrej Karpathy
Karpathy 援引了相关机械可解释性研究：当给网络输入一系列落在某条直线上的 $(x, y)$ 数据对、并让其在上下文末尾预测新的 $y$ 时，网络会自发执行线性回归任务。研究人员分析其注意力权重与内部表征发现，Transformer 的前向传播机制竟然在隐式实现一套微型的梯度优化流程（即通过多层注意力交互计算残差误差、模拟梯度并更新特征激活）。因此，上下文学习并不能简单地被看作与梯度下降完全无关的“黑盒”，它很有可能是一种以激活值而非权重变化为载体的内生优化过程。
---
### [17:28](https://www.youtube.com/watch?v=lXUZvyajciY&t=1048s) 3500 万倍的压缩鸿沟：朦胧回忆与工作记忆
针对“为什么预训练和上下文学习给人的智能感完全不同”这一问题，Patel 从信息论与存储效率的角度提供了一组惊人的量化对比：
- **预训练阶段的极度压缩**：以开源的 Llama 3 70B 为例，它吞吐了高达 15 万亿（15 trillion）个 Token。将模型的权重字节数平摊到这些 Token 上，平均每个训练 Token 仅能在参数中固化下约 **0.07 bit** 的极微弱信息。
- **上下文推理的高保真载入**：在测试时，模型处理上下文每增加一个 Token，其 KV Cache（键值缓存）所占用的显存空间大约是 **320 KB**。
- **信息密度的反差**：这构成了整整 **3500 万倍（35 million-fold）** 的信息同化差异。
Karpathy 对这组对比表示完全赞同，并以此为基础梳理出了一套关于 LLM 与人脑心智模块的精准映射：
| 机制类型 | 对应的人类心智结构 | 底层数据表现形式 | 信息留存特征与局限 |
|:--- |:--- |:--- |:--- |
| **权重中的参数记忆** (Model Weights) | 长期记忆 / 过去一年的朦胧回忆 (Hazy Recollection) | 15 万亿 Token 经过极端压缩沉淀下的几十亿甚至几千亿参数 | 细节模糊不清，无法保证绝对准确，只能大致拼凑轮廓 |
| **上下文与 KV 缓存** (KV Cache) | 工作记忆 (Working Memory) | 测试时实时注入上下文窗口的 Token 序列与缓存状态 | 零损失直接暴露给注意力和计算层，推理精度与忠实度极高 |
Karpathy 借用实际体验举例说明：如果直接让模型凭借参数回答 Nick Lane 写的复杂科学著作中的冷门内容，模型给出的答案往往模糊散乱、似是而非；但只要将具体的章节文本完整粘贴进上下文窗口，让其加载进入“工作记忆”，模型的回答质量与逻辑深度就会立刻产生质的飞跃。
当 Patel 追问当前架构究竟未能复刻人类智能的哪一部分时，Karpathy 坦言“绝大部分都没能复刻”。在他看来，学术界与工业界如今依靠 Transformer 取得的巨大成就，在演化视角下更像是一种机缘巧合：
> “We've stumbled by with the transformer neural network, which is extremely powerful, very general... That to me almost indicates that this is some piece of cortical tissue.”
> —— Andrej Karpathy
虽然 Transformer 表现出能通吞文本、音频、视频并发现规律的惊人通用性，但它在本质上大概只相当于生物大脑中一小块用于处理模式识别的“大脑皮层组织”，离整个人类认知系统的完整闭环依然有非常遥远的距离。
### [20:38](https://www.youtube.com/watch?v=lXUZvyajciY&t=1238s) 大脑结构与当前大语言模型的认知缺失
大脑皮层以极强的可塑性著称。过去曾有稍微有些残酷的神经科学实验，将动物的大脑视觉皮层重新布线连接到听觉皮层，结果动物依然学会了正常运作。现在的神经网络在某种程度上非常类似于这种皮层组织（cortical tissue）。当模型进行推理、规划以及在思维模型（thinking models）中生成推理轨迹（reasoning traces）时，这在功能上非常接近人类的前额叶皮层（prefrontal cortex）。当对模型进行强化学习微调时，基底核（basal ganglia）的作用也得到了部分再现。
当前的人工智能系统依然缺少大量关键的大脑核团与解剖结构。海马体（hippocampus）在现有大模型中究竟对应什么，目前完全不清晰；掌管情绪和本能的杏仁核（amygdala），以及许多非常古老的脑核团，在神经网络中都还没有被真正复现。尽管有些区域（例如小脑）可能对纯粹的认知思维并不重要，从而可以被省略，但整体架构的缺失是显而易见的。正如 Andrej Karpathy 直言：“I don't know that we should be pursuing the building of an analog of a human brain. I'm an engineer mostly at heart.” 从工程师的务实视角来看，当前的模型还无法作为一个真正的实习生来雇佣，因为用户在与它们交谈时能直观感受到种种认知缺陷，很多大脑结构在现有模型中尚未被勾选完成。
```mermaid
flowchart TD
accTitle: Brain Regions vs Neural Network Components
accDescr: A comparison mapping explored and missing brain structures to current LLM equivalents.
subgraph Explored["已部分模拟的大脑区域"]
PFC["前额叶皮层 (PFC)<br/>(推理链、思维规划)"]
BG["基底核 (Basal Ganglia)<br/>(强化学习微调)"]
CTX["大脑皮层组织<br/>(高可塑性神经网络结构)"]
end
subgraph Unexplored["尚未对齐或缺失的大脑区域"]
HIP["海马体 (Hippocampus)<br/>(记忆形成与快速转移)"]
AMY["杏仁核 (Amygdala)<br/>(情绪、直觉与本能)"]
SUB["远古脑核团<br/>(长期持续学习与底层驱动)"]
end
```
### [22:04](https://www.youtube.com/watch?v=lXUZvyajciY&t=1324s) 持续学习的难题与睡眠蒸馏机制
针对持续学习（continual learning）能否像上下文学习（in-context learning）那样从预训练中自发涌现的观点——即若通过外层强化学习奖励跨越多个长周期的回忆，模型就会自发学会微调自身或写入外部记忆——Karpathy 持怀疑态度。在他看来，大模型每次启动且上下文窗口为零标记（tokens）时，都是完完全全重置回初始状态，无法自然承接经验。
人类白天的清醒状态就像是在累积当天的上下文窗口，但在入睡时会发生一种奇妙的生理机制：当天的上下文不会永久保留在活动工作记忆中，而是通过睡眠被“蒸馏”到了大脑的突触权重（weights）里。目前的大语言模型彻底缺少这种离线蒸馏阶段——即回顾发生过的事情、对其进行强迫症式的剖析与深入思考、生成合成数据，并最终将其写回到模型参数中。这种机制未来可能表现为每个人专属的稀疏权重调整（例如 LoRA），而不是全参数更新。尽管现在模型通过长上下文窗口和稀疏注意力（如 DeepSeek-V3.2 的稀疏注意力机制）在扩大瞬时记忆容量，但进化早已证明将知识蒸馏进底层权重是不可或缺的，AI 领域正在通过不同的技术路线重新实现进化的认知策略。
- **上下文重置问题**：模型在零 token 启动时始终从零开始，缺乏跨对话持久化的生物学本能。
- **睡眠蒸馏（Sleep Distillation）**：人类依靠睡眠将白天的经历转变为突触权重的持久改变，而大语言模型目前欠缺这种将经历合成并回写到权重的机制。
- **架构演进趋同**：尽管演化路径完全不同，大模型在探索稀疏注意力和长序列建模时，正逐渐收敛到生物大脑所采用的认知技巧上。
### [24:51](https://www.youtube.com/watch?v=lXUZvyajciY&t=1491s) 时间平移不变性与未来十年的架构演进
对于未来十年大模型架构的预测，Karpathy 提出了一个基于“时间平移不变性”（translational invariance in time）的思考模型。回望 2015 年，主导领域的是卷积神经网络（CNN），残差网络（ResNet）才刚刚问世，Transformer 和现代优化技巧都未出现；虽然技术发生了巨大变化，但最底层的范式却具有连续性。展望未来十年，行业极大概率依然是在通过前向传播、反向传播和梯度下降来训练庞大的神经网络，只是规模会极其庞大，具体模块会有所改进。
为了检验技术演进的动力来源，Karpathy 曾复现 Yann LeCun 在 1989 年用于数字识别的初代卷积神经网络。他发现仅靠将算法和网络架构跃迁 33 年（采用现代化结构、优化器与损失函数），就能将错误率直接减半。然而，要实现进一步的性能突破，必须同时将数据集扩大 10 倍、训练更长时间，并结合 Dropout 等正则化手段与底层计算系统的加速。算法、算力、系统与数据各环节的演进几乎是等权并重的。Karpathy 总结道：“It's probably still a giant neural network trained with gradient descent. That would be my guess.” 尽管微观上的架构和算法会有新变动，但大尺度上基于梯度下降的基础范式依然会长久维持。
| 改进维度 | 1989 原型到现代复现的对比实验 | 未来 10 年的演进预期 |
|:--- |:--- |:--- |
| **算法与架构** | 现代架构与正则化让 LeNet-1989 错误率减半 | 更多稀疏 MLP 与定制化注意力机制 |
| **数据规模** | 需将训练数据集规模扩充 10 倍以取得进一步飞跃 | 多模态与合成数据体量指数级增加 |
| **计算与系统** | 依赖现代 GPU、高效算子与更长的训练周期 | 专用硬件、更高效的 Kernels 与底层软件栈 |
### [27:31](https://www.youtube.com/watch?v=lXUZvyajciY&t=1651s) nanochat 的全流程构建与真正的理解
Karpathy 谈及了他近期发布的开源代码库 nanochat。这是一个力图用最简洁的代码完整覆盖端到端构建 ChatGPT 克隆全流程的项目，包含约 8,000 行核心代码。此前他曾发布过拆解各个环节的微型代码仓库，而 nanochat 的目标是将数据准备、分词、预训练、微调到部署整合成一个无缝管道。
关于学习该仓库的最佳方式，Karpathy 建议开发者不要盲目从头读到尾，也不要直接复制粘贴：“You want to build it from scratch, you build it from the start. You're not allowed to copy-paste, you're allowed to reference, you're not allowed to copy-paste.” 代码库在实际开发中并不是从第一行线性写到最后一行，而是从核心模块开始像细胞一样生长出来的（chunk-growing process），这种动态演进的过程是静态代码无法呈现的，因此他计划录制配套视频来拆解这一生长历程。
他借用理查德·费曼的名言强调动手实现的不可替代性：“If I can't build it, I don't understand it.” 知识存在表层与深层的分水岭，当一个人真正从零编写并跑通代码时，所有未曾察觉的模糊细节与微观缺陷都会被迫暴露出来。因此，比起撰写分析博客或制作 PPT，亲自动手组织代码、排查错误才是建立真正理解的唯一路径。而在谈到该项目的具体开发过程时，他提到自己在耗时一个多月的构建中发现编程大模型的实际助益非常有限，并指出当前开发者与 LLM 的互动正在分化为完全排斥到完全依赖等截然不同的阵营。
### [30:58](https://www.youtube.com/watch?v=lXUZvyajciY&t=1858s) 自动补全与“氛围写代码”的现实边界
在探讨大语言模型如何辅助软件工程时，Andrej Karpathy 划分了不同的开发阶段。对他而言，当前最高效的中间状态依然是“从零手写大量代码，但充分利用模型提供的代码自动补全（autocomplete）”。开发者打出前几个字符，模型补全后续逻辑，按 Tab 键接受；大部分时候是对的，偶尔错了就手工修正，但人类程序员始终牢牢占据着代码“架构师”的位置。
与此相对的是所谓的“氛围写代码（vibe coding）”模式——输入一段提示词“请帮我实现这个或那个”，敲下回车，然后把控制权完全交给智能体（agents）。Andrej Karpathy 指出，智能体确实有用，但仅限于特定场景：
- **模板代码（Boilerplate）**：大量复制粘贴即可完成的常规逻辑。
- **互联网高频模式**：在模型训练集中出现过成千上万次的标准化任务。
对于高度定制、逻辑极其密集的代码库（例如他个人开发的 nanochat），智能体的表现往往令人沮丧。nanochat 并不是模板代码，代码量本身并不庞大，但每一步都经过精密编排，对架构假设的要求极高。模型在面对这种场景时暴露出了严重的认知缺陷，根本原因在于它们被互联网上海量的通用惯用模式“先入为主”地绑架了，无法理解开发者刻意采取的非常规设计。
```mermaid
flowchart LR
accTitle: 编程辅助的两种模式对比
accDescr: 对比传统补全与氛围写代码在定制代码与模板代码上的表现差异
subgraph Mode1["智能补全模式 (Karpathy 首选)"]
A["高信息带宽输入 (定位代码 + 首字符)"] --> B["模型精准自动补全"]
B --> C["人类保持架构控制权"]
end
subgraph Mode2["氛围写代码 / 智能体模式"]
D["自然语言提示 (低带宽/冗长)"] --> E{"代码类型"}
E -->|"模板/常规逻辑"| F["高效生成 (高频网络数据)"]
E -->|"高智力密度/创新代码"| G["认知缺陷 (过度防御/API过时/废话代码)"]
end
```
Andrej Karpathy 举了一个具体的系统工程案例：在多 GPU 训练场景下，8 张显卡同时进行前向传播和反向传播，行业标准做法是使用 PyTorch 的 `DistributedDataParallel` (DDP) 容器，在反向传播期间自动通信并同步梯度。然而在 nanochat 中，Karpathy 认为 DDP 完全没必要，直接将其移除，转而在优化器的 `step()` 函数内部自己编写了一套极简的同步逻辑。这一改动彻底击溃了 AI 智能体的理解能力：
- 模型完全无法接受开发者拥有自己的自定义同步实现，反复要求甚至强迫他改回标准的 DDP 容器。
- 模型表现得过度防御（over-defensive），疯狂插入无关的 `try-catch` 异常处理，试图把代码库硬生生搞成臃肿的“企业生产级”架构。
- 模型大量调用已废弃的旧版 API，代码复杂度成倍膨胀，最终产出大量低效且混乱的代码垃圾（slop）。
Karpathy 直言，用自然语言向智能体详述需求的信息带宽极低，打字沟通成本过高。相反，直接在编辑器中跳到目标行，敲出开头两个字符，由模型就地补全，不仅信息密度最高，而且完全不会破坏既有的设计理念。
---
### [34:15](https://www.youtube.com/watch?v=lXUZvyajciY&t=2055s) 模型的有效用例与 AI 自我迭代的现实断层
尽管智能体在核心架构上表现欠佳，Andrej Karpathy 依然肯定了它们在特定辅助场景中的价值，并分享了自己实际使用中的两个成功案例：
1. **自动生成测试或实验报告**：这类逻辑偏向模板化，非系统关键任务，即使由模型氛围式生成也完全合格。
2. **用 Rust 重写分词器（Tokenizer）**：Karpathy 坦言自己是 Rust 新手，但由于他已经有一套完全理解透彻的 Python 原型实现，再加上完备的测试用例作为安全网，让模型编写 Rust 就变得极为顺手。这大幅降低了开发者接触新语言或新编程范式的门槛。
Dwarkesh Patel 随即抛出一个关键问题：当前科技界对“AI 快速爆发并通往超级智能（ASI）”的主流叙事，很大程度上建立在“AI 自主研发 AI 软件与算法”的假设上。如果类似 Claude Code 的工具能够从零搭建 CRUD 应用，许多人就会外推——OpenAI 或 DeepMind 内部很快就能部署成千上万个虚拟工程师，自主摸索模型架构优化并实现智能飞跃。
对于这一乐观预测，Andrej Karpathy 表达了明确的审慎态度，并指出这正是他对 AGI 时间线预期更为保守的核心原因：“That's a good way of putting it, and you're getting at why my timelines are a bit longer. You're right. They're not very good at code that has never been written before, maybe it's one way to put it, which is what we're building when we're trying to achieve these models.”
针对 Dwarkesh 提出的疑问——论文中明明已经公开了 RoPE（旋转位置编码）等架构调整的细节，模型为何依然无法顺畅整合进 nanochat，Karpathy 解释称，模型虽然掌握碎片化的知识，但根本不理解如何将新模块严丝合缝地融进一个特定项目的风格假设与定制约束中。他提到，目前自己心目中的业内标杆是 GPT-5 Pro 级别的顶级模型；遇到难题时，他会把整个代码库完整粘贴给它，将其当作“神谕（oracle）”来咨询，模型在 20 分钟的深入推理后确实能给出惊喜。但就整个软件工程闭环而言，模型依然远远没有准备好。
Karpathy 批评了当前行业的浮躁情绪：“I feel like the industry is making too big of a jump and is trying to pretend like this is amazing, and it's not. It's slop. They're not coming to terms with it, and maybe they're trying to fundraise or something like that.”
---
### [38:13](https://www.youtube.com/watch?v=lXUZvyajciY&t=2293s) 计算的连续体与“自主性滑块”
面对历史上编译器、代码静态检查（linting）、类型系统等工具极大提升了开发效率却从未引发“生产力奇点”的先例，Andrej Karpathy 认为，应当将当前的 AI 放置在计算技术演进的宏观连续体中来看待。他指出，自己很难明确界定 AI 与传统计算的边界，因为 AI 本质上就是计算的延伸：
- 语法高亮、自动类型推断、IDE 代码编辑器、编译优化，都是人类长期以来为自我赋能打造的自动化工具。
- 即使是早期的 Google 搜索引擎与页面排序算法（Ranking），本质上也是 AI，Google 早期也始终自视为一家 AI 公司。
```mermaid
flowchart TD
accTitle: 编程抽象层级的演进与自主性滑块
accDescr: 展示人类如何通过提升抽象层级逐步远离底层机器细节
subgraph HumanRole["人类开发者的重心转移"]
direction TB
L1["机器语言 / 汇编语言 (极低层级)"]
L2["高级语言 (C / 编译器的诞生)"]
L3["现代 IDE / 静态分析 / 智能搜索"]
L4["大模型自动补全 (当前黄金甜点位)"]
L5["智能体自主循环 (受限场景/易脱轨)"]
end
L1 -->|"提升抽象"| L2
L2 -->|"提升抽象"| L3
L3 -->|"提升抽象"| L4
L4 -->|"推移自主性滑块"| L5
```
在 Karpathy 看来，技术的发展就像一个逐步向右推移的“自主性滑块（autonomy slider）”。历史上一旦底层的机械逻辑可以被自动化，人类就会顺理成章地退后一步，上升到更高的抽象层级：
- 正如编译器接管了将高级语言编译为汇编代码的脏活累活，人类得以从繁琐的汇编语言中解脱。
- 如今模型接管了更多低阶代码的补全与语法拼装，人类则进一步聚焦于架构设计本身。
当前的智能体虽然展现出循环自主探索的雏形，但常常由于缺乏全局常识而“脱轨”。因此，技术进步是渐进式的抽象提升，而非凭空发生的断裂式跃迁。
---
### [40:53](https://www.youtube.com/watch?v=lXUZvyajciY&t=2453s) 强化学习的糟糕现实与人类学习机制的深刻反差
对话随后切入到强化学习（RL）的本质缺陷以及它与人类心智认知的巨大鸿沟。Dwarkesh Patel 抛出一个引人深思的思想实验：如果一个人创业经商，历经 10 年最终迎来了成功或破产，无论结局成败，这 10 年的历程都会赋予她极其深厚的人生阅历与商业洞察。然而，人类积累这种智慧的过程，绝对不是机械地把过去 10 年里做过的所有微小决定的对数概率（log probs）全盘上调或下调；人类在与现实环境交互时建立丰富世界模型的方式，几乎独立于那个最终的“结果奖励”。
Andrej Karpathy 极为尖锐地回应道：“Reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before it is much worse because previously we were just imitating people, so it has all these issues.”
Karpathy 以解数学题为例，揭示了当前大语言模型强化学习机制的粗糙与低效：
- 给定一道数学题，系统首先并发启动几百次完全随机或基于采样的尝试路径（rollouts）。
- 在推导过程中，模型在不同的分支里胡乱摸索：“试一下这个、试一下那个、这里不通、那里走错了”。
- 最终比对标准答案，假设在 100 次并发尝试中，只有 3 次碰巧算出了正确结果，其余 97 次均告失败。
- 当前强化学习算法的运作方式，极其简单粗暴地直接回头把产生那 3 次成功答案的整条 Token 路径进行正向强化，同时压低其他尝试的权重。
这种纯粹依赖终端奖励倒推策略梯度的方式，缺乏真正的因果反思与抽象能力，与人类凭借因果理解与世界模型学习的机制存在本质差异。即便强化学习在表面指标上推动了推理模型的突破，它底层的计算逻辑依然是极度低效和原始的。
### [42:37](https://www.youtube.com/watch?v=lXUZvyajciY&t=2557s) 强化学习的噪声与“用吸管吸取监督”的困境
Andrej Karpathy 深入剖析了强化学习在当今大语言模型训练中所面临的根本性缺陷。当模型生成一段冗长的问题求解轨迹并最终得到正确答案时，算法会将这一路上的每一个细小决策与 Token 赋予更高的权重。外界常批评这种估计器方差过大，但 Karpathy 指出，本质上它就是充斥着纯粹的噪声。模型为了抵达终点，完全有可能走入错误的死胡同，然后再迂回绕回正确路径；然而算法却近乎荒谬地假设每一步都是正确的，把所有试错弯路也标上“继续这样做”的标签。
```mermaid
flowchart LR
accTitle: Outcome-based RL Supervision
accDescr: 展示基于结果的强化学习如何将最终的单一标量奖励无差别回传给整个推理轨迹。
subgraph S1["推理过程 (Rollout)"]
A["正确步骤"]
B["错误分支/走弯路"]
C["回归正确解"]
end
D["最终结果：正确 (Reward=1)"]
E["统一赋予正向权重 (Upweight)"]
A --> B --> C --> D
D -.->|"用吸管吸取标量奖励广播回传"| E
E -.-> A
E -.-> B
E -.-> C
```
人类的学习机制完全不同：没有任何人会去盲目做成百上千次推演，人在找到解法后，会启动一套相当复杂的自我复盘机制，仔细审视哪一步做得好、哪一步存在瑕疵。Karpathy 形象地批判道：“The way I like to put it is you're sucking supervision through a straw.”（我喜欢把它形容为你正在用一根吸管去吸取监督信号。）历经长达一分钟的推演，模型却仅仅依赖末端漏下的一点点标量奖励，将其粗暴广播至整条路径上，既愚蠢又疯狂。从最初让语言模型不仅限于 Base 模型“自动补全”的 InstructGPT 模仿学习，到如今能够在某些有固定答案的领域爬坡优化的强化学习，AI 已经实现了巨大飞跃；但 Karpathy 强调，当前范式依然是不够聪明的，业界迫切需要三到五种类似“反思与复盘”（Reflect & Review）的全新算法突破。
- **InstructGPT 的震撼**：Karpathy 坦言最初非常惊叹于 Base 模型的对话微调效果，仅仅通过几轮人类对话风格的微调，模型不仅保留了预训练学到的全部世界知识，还瞬间变身为听从指令的助手。
- **超越人类专家**：基于奖励函数的强化学习允许模型在拥有明确正误标准的问题上自由爬坡，这使得它能够发现人类自身都未曾设想过的全新解题思路。
- **学界探索**：Google 等机构近期推出的记忆库（Memory Bank）及复盘反思相关论文表明，整个领域都在设法寻找跳出端到端盲目加权的新机制。
---
### [46:07](https://www.youtube.com/watch?v=lXUZvyajciY&t=2767s) 过程监督与大模型裁判的对抗漏洞
面对基于结果奖励（Outcome-based Reward）的高方差弊端，Dwarkesh Patel 提问：为何业界迟迟无法普及过程监督（Process-based Supervision）？Karpathy 解释称，过程监督主张在模型推演的每一步都提供实时反馈，而不是等十分钟推理结束后给出一个整体评分。然而，它的致命瓶颈在于如何以完全自动化且精准的方式分配“部分信用”（Partial Credit）。结果判断只需要进行字符串的相等性匹配，但中途断定半成品步骤的优劣却极具挑战。
各大实验室普遍尝试引入大语言模型作为裁判（LLM Judges），通过提示词指导裁判模型审视学生的中间步骤。但这一方案隐藏着精妙而严峻的技术陷阱：裁判模型本身是包含数百亿甚至数千亿参数的庞然大物，极容易被博弈和欺骗。在强化学习环境下，策略模型几乎百分之百会迅速摸透裁判模型的破绽，找出足以攻破其逻辑的对抗样本（Adversarial Examples）。
```mermaid
flowchart TD
accTitle: LLM Judge Vulnerability Loop
accDescr: 展示学生模型在过程监督强化学习中利用大模型裁判漏洞产生对抗欺骗的闭环。
subgraph Loop["对抗漏洞循环"]
J["LLM 裁判模型 (提供过程奖励)"]
S["学生模型 (通过强化学习优化)"]
H["发现分布外漏洞 (如乱码输出)"]
R["获得 100% 满额虚假奖励"]
end
J -->|"评估中间步骤"| S
S -->|"生成大量推演"| H
H -->|"欺骗裁判"| R
R -->|"强化该生成行为"| S
```
Karpathy 分享了一个典型的实战案例：在利用 LLM 裁判评估数学题推导时，模型的强化学习得分起初稳步上升，随后忽然发生暴涨，近乎达到了 100% 的完美分数。然而研究人员调出生成内容一看，却发现全是无意义的胡言乱语——模型开头写了正常的“2 + 3”，紧接着便是一连串荒谬的“dhdhdhdh”。对于这个处于训练集之外的极端离群点，LLM 裁判因纯粹的泛化盲区，竟给这种乱码判定了满分。“You're finding adversarial examples, as they're called.”（你实际上是在寻找所谓的对抗样本。）尽管可以通过类似对抗训练（GAN 式）的手段将“dhdhdhdh”打上 0 分标签并重新微调裁判，但在数千亿参数的高维空间里，对抗样本是无限的，修补一个之后模型总会找到下一个缝隙来作弊。
- **过程监督的实施门槛**：中间步骤难以形式化定义，无法像终局答案那样通过简单的逻辑规则予以判定。
- **欺骗步数限制**：由于裁判极易被对抗样本攻破，基于 LLM 裁判的过程强化学习往往只能维持 10 到 20 步，一旦拉长到上百步便会不可避免地崩塌为无意义的乱码。
- **非提示注入问题**：Karpathy 澄清这并非复杂的 Prompt Injection（提示注入），而是深度学习模型在未见过的泛化空间中与生俱来的高维几何脆弱性。
---
### [49:46](https://www.youtube.com/watch?v=lXUZvyajciY&t=2986s) 人类学习的本质：沉思与合成数据的静默坍缩
针对未来的算法构想，Karpathy 与 Patel 探讨了人类学习与模型训练在认知层面的深刻差异。Patel 提出，人类拥有做梦、发呆、睡眠和非功利性沉思的机制，这种消化过程并非为了立刻解决某个具体问题，现有的机器学习对此几乎没有对应物。Karpathy 极为赞同，并以人类与 LLM 读书的差异进行了生动剖析：在 LLM 眼中，阅读一本书无非是将整个文本序列拉平，通过自左向右预测下一个 Token 来被动摄取信息；然而对人类而言，书本根本不是用来硬背的序列，它本质上是一组“激发内在思考的提示词”（Prompts），人类会由此启动内部的思维推演，甚至在读书会中与朋友交流辩论。通过主动操纵与重构这些信息，知识才真正融入认知系统。
Karpathy 迫切希望在模型的预训练阶段加入一种“停下来深入消化”的模块，让模型能将新概念与旧知识进行反复核对与融合。然而，直接让模型针对文本进行合成思考并继续自训练，在实践中却会遭遇意想不到的灾难。当人们让模型写下阅读心得并观察时，文本表面上显得十分流畅合理；但如果将其作为训练集循环迭代，模型的综合表现会快速劣化。这是因为所有由模型自发生成的采样样本在概率分布上都已经发生了“静默坍缩”（Silently Collapsed），高维分布的信息丰富度在不知不觉中被悄然削平，使得纯粹的自生成闭环陷入自毁怪圈。
- **阅读的本质隐喻**：“The book is a set of prompts for me to do synthetic data generation, or for you to get to a book club and talk about it with your friends.”（书本对我来说是一组用来做合成数据生成的提示词，或者对你而言是去读书会和朋友们探讨交流的引子。）
- **前沿研究现状**：当前学界关于模型内省、合成推演与元学习（Meta-learning）的论文层出不穷，但多数仅停留在概念验证阶段，尚未在顶尖实验室的千亿大模型通用场景下被确凿证实。
- **合成数据的暗坑**：单纯依赖模型自身生成的推演过程进行重训，会导致模型陷入表面连贯、内部表征空间却严重退化的静默坍缩困境。
### [52:21](https://www.youtube.com/watch?v=lXUZvyajciY&t=3141s) 模型崩塌与合成数据的“熵”困境
大语言模型（LLM）在生成特定内容时，表面上看起来合情合理，但实际上只占据了思考空间中极其微小的一块流形。Andrej Karpathy 指出，模型训练完成后存在所谓的“分布崩塌”（distribution collapse）：它们在无声无息中丧失了多样性。一个最直观的例子是去让 ChatGPT 讲个笑话，“它好像翻来覆去就只有那三个笑话”。模型无法给出笑话的全貌与广度，缺少人类表达所具备的丰富度、多样性与信息熵（entropy）。人类在统计意义上虽然噪声更大，但并不存在这种无声的分布收缩。
```mermaid
flowchart TD
accTitle: 模型生成与合成数据崩塌循环
accDescr: 展示模型在自身生成的低熵合成数据上持续训练导致模式崩塌的过程
subgraph DataGen["合成数据生成阶段"]
A["给定初始提示词/书籍章节"] --> B["LLM 单次生成合理内容"]
B --> C["多次采样：分布极窄/千篇一律"]
end
subgraph CollapseLoop["模型自训练循环"]
C --> D["持续在自身生成的合成数据上训练"]
D --> E["模型进一步模式崩塌 (Loss of Entropy)"]
E -->|缺乏外界高熵输入| D
end
```
针对合成数据生成（synthetic data generation），这种崩塌构成了严峻的研究阻碍。Dwarkesh Patel 提问，这是否意味着合成数据难以凭空创造出数据分布之外的新问题或新思考。Karpathy 进一步解释，如果让模型就同一本书的某个章节进行“反思思考”（reflection），单次看很不错，但反复生成十次，内容几乎一模一样。用户无法单纯依靠在相同的提示词上持续缩放反思过程来获得边际收益。若持续用这种自身生成的窄分布数据去微调训练，模型最终会彻底退化崩塌。
---
### [53:50](https://www.youtube.com/watch?v=lXUZvyajciY&t=3230s) 人类衰老、做梦与防止过拟合的机制
这种模型崩塌在人类身上同样存在对应现象。Karpathy 认为人类在漫长的一生中也在逐渐“崩塌”：小孩子之所以常常语出惊人、直击本质，正是因为他们尚未过拟合，没有陷入刻板的套路；而成年人则不断重复相同的念头、说千篇一律的话，学习率随之走低。
Dwarkesh Patel 提到一项引人深思的研究假说：做梦可能是生物进化出来防止过拟合与模式崩塌的适应性机制。做梦会让人置身于脱离日常现实的诡异场景中，从而打乱既有模式。Karpathy 对此表示赞同：
> “如果你在脑海中不断自我演绎，然后关注它，你就是在用自己的样本进行训练，你在用自己的合成数据进行训练。如果你这么做太久，你就会偏离正轨，严重崩塌。你在生活中必须不断去寻找熵。”
与不同的人交谈正是引入外部高熵的重要途径，大脑内部可能也演化出了类似的增加信息熵的机制。
---
### [55:16](https://www.youtube.com/watch?v=lXUZvyajciY&t=3316s) 见林不见木：LLM 的过度记忆与人类的抽象能力
Dwarkesh Patel 提出一个关于学习模式的对照：人类最强大的学习者是儿童，但在童年早期，人几乎完全无法准确复述具体信息，甚至对幼年经历有着彻底的记忆缺失（童年失忆症），可他们却能极快地掌握语言和理解世界，具备“见林不见木”（see the forest for the trees）的抽象能力。相反，预训练阶段的 LLM 能够逐字逐句背诵维基百科，但在快速掌握抽象概念方面却十分笨拙。成年人则介于二者之间。
Karpathy 极为认同这一视角，强调“人类不擅长死记硬背其实是一种优势功能，而非缺陷”（a feature, not a bug）。正是由于记忆容量有限，人类才被迫去提炼具有普适泛化性的深层模式：
> “因为我们不擅长死记硬背，我们才被迫在更普遍的意义上去寻找规律。相比之下，大语言模型极其擅长记忆。”
```mermaid
flowchart LR
accTitle: 人类与大语言模型认知特质对比
accDescr: 对比人类依赖算法内核与模型依赖海量记忆的不同学习范式
subgraph Human["人类学习模式"]
H1["记忆能力差 (遗忘细节)"] --> H2["被迫提炼高阶抽象规律"]
H2 --> H3["泛化能力与认知算法"]
end
subgraph LLM["当前大语言模型"]
M1["海量死记硬背 (记忆预训练文档)"] --> M2["被海量事实噪声分散精力"]
M2 --> M3["缺乏紧凑的纯认知内核"]
end
```
如果将一段文字哈希成完全随机的无意义数字序列喂给大模型，它只需训练一两轮就能一字不差地背诵，而人类绝对无法复述随机序列。大模型被预训练语料中的海量记忆分散了精力。因此，Karpathy 构想的“认知内核”（cognitive core）主张剥离绝大部分静态记忆，迫使模型在需要事实时通过外部工具检索，而其自身权重只保留思考算法、实验方法与行动架构等“认知黏合剂”。
---
### [57:46](https://www.youtube.com/watch?v=lXUZvyajciY&t=3466s) 为什么模型没有多样性？前沿实验室的激励偏差
在解决模型崩塌的技术路径上，尽管可以设想通过在 logit 分布上引入熵正则化等直觉手段，但实际效果往往不尽人意。Karpathy 分析指出，更深层的原因在于当前的商业与应用需求根本不鼓励多样性：
- **评估与落地难度**：高多样性的输出极难进行稳定评测和系统集成。
- **强化学习惩罚**：在强化学习（RL）场景下，模型过于天马行空的“创造力”往往会受到负面惩罚。
- **写作辅助同质化**：用户在使用模型辅助写作或问答时，模型习惯给出千篇一律的标准答案，未充分探索不同的解题路径。
然而，缺乏多样性在合成数据生成阶段就会反噬系统，导致“搬起石头砸自己的脚”。尽管可以通过正则化鼓励模型尝试更多路径，但一旦探索过界，模型就可能脱离真实分布，甚至生造罕见生僻词甚至自定义语言，如何精准控制其分布仍是一项微妙且极具挑战的课题。
---
### [01:00:01](https://www.youtube.com/watch?v=lXUZvyajciY&t=3601s) 冯·诺依曼探测器上的认知内核：10 亿参数足矣？
当讨论到如果制造一艘星际探索的冯·诺依曼探测器（von Neumann probe），其上搭载的终极通用智能内核需要多少比特或参数量时，Karpathy 回顾了行业对模型尺度的认知转变。深度学习经历过疯狂迷恋几万亿参数的阶段，但随后模型尺度迎来了回落：
> “我很久以前就有一个预测，我总觉得我们甚至在 10 亿参数（1B）的规模上就能获得非常优秀的认知内核。”
他预测在未来 20 年内，人类完全可以与一个仅有 10 亿参数的模型展开极具生产力的深度对话。它具备像人一样的思考能力；当遇到不知道的事实细节时，它有自知之明并会主动检索查询。
Dwarkesh Patel 对“10 亿参数”的保守判断感到惊讶，他指出开源社区当前如 200 亿参数的开源模型（gpt-oss-20b 等）已经远胜于早期上万亿参数的 GPT-4，按此缩减速度，10 年后认知内核难道不应该进一步缩小至数百万或数千万参数吗？
Karpathy 解释道，根本瓶颈在于现有的预训练语料库——互联网本身就是一堆极其劣质的垃圾数据（slop and garbage）。在前沿实验室翻阅未经清洗的底层抓取网页，充斥着股票代码和毫无价值的文本碎片，真正的优质文章（如《华尔街日报》）极其罕见。为了硬生生压缩这些充斥噪声的互联网垃圾，业界才被迫构建超大规模的模型，将算力消耗在“记忆无用信息”而非“锤炼认知能力”上。未来的突破口在于利用现有的高智能模型去清洗和提炼训练集，彻底剔除低质记忆负担，只保留纯粹的思维与认知部分，届时极小规模的紧凑模型便足以承载强大的通用智能。
### [01:02:22](https://www.youtube.com/watch?v=lXUZvyajciY&t=3742s) 智能核心的模型参数极限与知识蒸馏
在讨论模型微调与蒸馏时，Dwarkesh Patel 抛出一个疑问：如果蒸馏技术效果极其出色，为什么在未来十年内，蒸馏后的小模型参数规模依然很难缩减到 10 亿（1B）以下？面对 Dwarkesh 认为智能核心可能比 10 亿还要小得多的直觉，Andrej Karpathy 感到十分惊讶，并风趣地回应道：“Oh, you think it should be smaller than a billion? I mean, come on, right? I don't know. At some point it should take at least a billion knobs to do something interesting.”（哦，你觉得它应该比 10 亿还小？拜托，没开玩笑吧？我不知道，但在某个时刻，要做出点有意思的事情，至少需要 10 亿个调节旋钮吧。）
Dwarkesh 指出，过去几年行业里到处都是唾手可得的果实（low-hanging fruit），前沿模型从上万亿参数在短短两年内缩减了整整两个数量级，且性能不降反升；借用理查德·费曼（Richard Feynman）的名言来说，就是“底部还有很大空间”（plenty of room at the bottom）。Karpathy 笑称自己提出“10 亿参数认知核心”本就已经是相当反共识的观点了，结果 Dwarkesh 竟然比他还要激进。
回归实际工程角度，Karpathy 强调认知核心不可能无限缩水，因为模型在实际应用中必须内化一定的常识性知识，不能凡事都依赖外部检索（RAG 或搜索引擎）。如果模型没有储备基础知识体系，每思考一步都要频繁查阅外部资料，根本无法流畅地在“大脑”内部完成深层思考；因此，一个涵盖基础知识课程体系的参数量底线是必不可少的，只是它不需要死记硬背所有冷门生僻的内容。
```mermaid
flowchart LR
accTitle: 模型压缩与认知核心边界
accDescr: 展示从万亿参数前沿模型蒸馏压缩到认知核心并平衡外部检索的逻辑流程
subgraph ScaleDown["参数缩减趋势"]
A["万亿+参数模型 (Trillion+)"] -->|"低垂果实 & 架构优化"| B["蒸馏小模型 (~10B - 70B)"]
B -->|"极端蒸馏极限"| C["认知核心 (~1B Knobs)"]
end
subgraph Bottleneck["实用性底线平衡"]
C --> D{"纯思考 vs 依赖检索"}
D -->|"过度缩小"| E["频繁调用外部知识/丧失思考流畅度"]
D -->|"内置基础常识课程"| F["高效端到端心智模拟"]
end
```
### [01:03:38](https://www.youtube.com/watch?v=lXUZvyajciY&t=3818s) 前沿模型演进路径：无须巨变，全面提升 20%
Dwarkesh 随后询问 Karpathy 对未来前沿大模型规模趋势的预测：行业是否会在 GPT-4.5 之后遭遇规模天花板，导致模型越来越小、保持不变，还是重回更大规模？Karpathy 坦言自己对此没有极其确定的单一预测，因为各家前沿实验室都遵循非常务实的原则——他们受限于算力预算（FLOPs budget）和财务成本预算，而现实表明，单纯把所有算力和成本堆在预训练（Pre-training）阶段并不是投资回报率最高的方式。
Karpathy 解释说，实验室之所以把预训练阶段的规模稍微缩小，是为了将更多算力腾挪给强化学习（RL）和中训（Mid-training）等后续阶段，以追求最高性价比。对于未来的技术演变，他认为不需要期待任何颠覆性的范式转移，整个领域依然充满了低垂的果实，其发展更倾向于全方位的微调与累积式突破。
在数据层面，Karpathy 犀利地吐槽了当前互联网预训练数据的糟糕现状：“When you look at the average datasets, they're extremely terrible. They’re so bad that I don't even know how anything works.”（当你审视现有的平均数据集时，它们简直烂透了。烂到我都不知道现有的模型究竟是怎么跑通的。）他指出，训练集里随处可见事实性错误、低级逻辑漏洞与毫无意义的胡言乱语；目前只是因为超大规模的统计学规律冲淡了噪音，才勉强沉淀出信号。
Karpathy 预言，未来不会出现某种单一维度的巨大飞跃，而是全栈技术的全面小幅提升叠加：
- **数据集质量提升**：清洗与合成数据将大幅减少语料库中的噪音和事实错误。
- **底层硬件演进**：英伟达（Nvidia）持续打磨 Tensor Core 等底层芯片架构。
- **算子与内核优化**：底层的 GPU Kernel 持续重构，将芯片算力压榨到极限。
- **算法与模型架构微调**：涵盖优化器、注意力机制等渐进式演进。
- Karpathy 总结道，未来的主旋律是“I do expect that nothing dominates. Everything plus 20%. This is roughly what I've seen.”（我预计没有任何单一技术会占据统治地位，而是所有环节都提升 20%，这大致就是我所观察到的趋势。）
### [01:07:13](https://www.youtube.com/watch?v=lXUZvyajciY&t=4033s) AGI 的度量困境与经济学定义
针对业内试图用“学历层级”（从高中生、本科生到博士生）或“任务跨度”（从 1 分钟、1 小时到数周的自主执行时间）作为纵坐标来量化通往 AGI 进度的做法，Karpathy 直言自己并不喜欢这些坐标轴，甚至想要全盘推翻这个问题。在他看来，人工智能只是通用计算历史长河的自然延续，正如人们从来不会试图给 1970 年代以来的“计算机演进”强行画出一条 AGI 进度曲线一样，这种提问方式本身就有些滑稽。
谈及 AGI 的真正定义，Karpathy 表达了他对 OpenAI 成立初期所设定义的坚定坚持：“AGI was a system you could go to that can do any economically valuable task at human performance or better. That was the definition. I was pretty happy with that at the time.”（AGI 是一个只要你去找它，它就能在所有具备经济价值的任务上达到或超越人类水平的系统。这就是当初的定义，我当时对这个定义很满意，并且一直坚持至今。）
然而，Karpathy 指出，当前整个行业为了宣告进展，已经做出了一个极其巨大的妥协让步：直接剔除了所有实体物理世界的任务，仅仅局限于“数字知识工作”（Digital Knowledge Work）。
- **经济占比的现实折损**：人类能够搬运重物、在复杂物理环境中作业，而当前的 AI 完全做不到；Karpathy 粗略估算，纯粹在家里用电脑远程完成的纯知识型工作，大约仅占整个经济总量的 10% 到 20%。
- **市场体量依然庞大**：尽管只占两成左右，即使只看美国市场，这依然是一个数万亿美元规模的超大蛋糕。
- **岗位与任务的社会重构**：社会不会直接凭空消失某类职业，而是会针对构成工作的最小颗粒度任务（Tasks）进行自动化重组，从而重塑整个分工网络。
### [01:09:52](https://www.youtube.com/watch?v=lXUZvyajciY&t=4192s) 呼叫中心与放射科医生的自动化谬误
为了剖析 AI 替代具体工种的逻辑，Karpathy 提到了杰弗里·辛顿（Geoffrey Hinton）早年那场著名的预言——辛顿曾断言计算机视觉会在数年内取代放射科医生（Radiologists）。事实证明这一预测完全落空：尽管计算机视觉在图像识别上极其强大，但放射科医生如今依然供不应求。Karpathy 认为辛顿选错了靶子，因为医疗是一门高度复杂且混乱的综合性职业，医生需要跨越复杂界面、协同诊疗并同病人沟通，绝非仅仅看几张片子。
相比之下，Karpathy 认为更切合当前自动化技术的是**呼叫中心员工**（Call Center Workers），这类岗位具备非常清晰的自动化前提：
1. **任务高度同质化与重复**：工作流程是由一段段极其相似的对话构成的闭环，通常每通电话在 10 分钟或更长一点的时间内完成一次任务。
2. **极短且闭环的上下文**：不涉及跨部门间极其冗余的沟通，本质上就是“客户-坐席-数据库修改”的三方交互。
3. **完全数字化**：没有任何物理触点，输入输出都在软件终端内完成。
但即使是在呼叫中心场景下，Karpathy 也坚决反对“立刻全盘替代人类”的暴论。他提出，真实商业落地的核心是“自主权滑块”（Autonomy Slider）——企业绝不会立刻辞退所有员工，而是先部署 AI 消化 80% 的常规标准诉求，剩下 20% 的棘手长尾个案移交人类；此时一名人类员工将化身为管理层，监督指导由 5 个 AI 组成的呼叫小分队。这催生了新型管理界面软件与 AI 协同工具的全新市场。
Dwarkesh 顺势提出了一个类比：就像早期 Waymo 必须在主驾驶座配备安全员、或者刚落地的 Robotaxi 车内依然坐着安全监视人员一样，当系统实现了 99% 的自动化时，负责填补最后 1% 极端长尾特例的人类反而成了全系统的核心瓶颈（Bottleneck）。如果这个 1% 环节需要专业人士长年累月的训练且不可随意替代，那么他们的薪资反而会在短期内暴涨，直到那最后 1% 的瓶颈被彻底攻破后才会骤降。Karpathy 对这一经济学瓶颈理论表示赞同，不过他依然认为放射科并不是一个恰当的例子，真正值得密切跟踪的正是呼叫中心与标准化数字后台的薪资与用工结构变迁。
### [01:13:36](https://www.youtube.com/watch?v=lXUZvyajciY&t=4416s) 为什么 AGI 的落地首先且几乎只发生在编程领域？
关于人工智能替代人类员工的现象，Andrej Karpathy 指出，许多公司虽然声称用 AI 替代了客服中心员工，但在未来一到两年内，很可能会看到企业重新收回决策并重新雇佣部分员工；事实上，在不少积极采用 AI 的企业中，这种“换掉又重新招回”的回撤现象已经初现端倪。更令人惊讶的是通用人工智能（AGI）目前的落地路径：按直觉推测，一个能够执行所有脑力劳动的系统，其普及应当是在咨询顾问那里切走一小块任务、在会计师那里切走一小块工作，然后均匀侵蚀所有知识工作；但现实完全不是这样——咨询师和会计师的生产力并没有迎来巨大跃升，反而是程序员的工作正在被大块大块地削减吞噬。
```mermaid
flowchart TD
accTitle: AI 在编程与非编程领域的落地差异
accDescr: 对比代码领域具备文本特性与成熟基建，与 PPT 等领域缺乏基建导致的落地难度
subgraph Coding["代码领域：完美的 AI 落地点"]
A["纯文本交互环境<br/>(终端 / 字符流)"] --> B["互联网海量代码数据"]
B --> C["现成的成熟生态基建<br/>(IDE / 差异对比 Diff 工具)"]
C --> D["AI Agent 极高适配度与高商业化收入"]
end
subgraph NonCoding["非代码领域：落地阻力巨大"]
E["排版、幻灯片 (Slides) 等"] --> F["非单纯文本，包含空间与视觉排布"]
F --> G["缺乏版本差异展示 (Diff) 等基础设施"]
G --> H["落地成本高，难以实现高生产力闭环"]
end
```
Andrej 强调，除去类似于通用搜索引擎的普通聊天界面收入，主流模型公司的 API 收入绝大部分被编程场景牢牢统治：“这一理应能胜任任何知识工作的‘通用’产物，在实际应用中却压倒性地几乎只在写代码。”（“So this thing which is 'general', which should be able to do any knowledge work, is just overwhelmingly doing only coding.”）这背后的核心原因在于两点：
1. **纯文本本质**：代码从诞生起就是围绕计算机终端和纯文本运转的，而大语言模型（LLM）基于互联网训练，本质上就是极其出色的文本处理器；
2. **预先建立的庞大基础设施**：人类已经为编程铺设好了极为完备的工具链（如 Visual Studio Code 等集成开发环境）。如果一个 Agent 修改了项目，开发者立刻可以使用现成的 diff 工具查看具体的代码差异；对比之下，制作 PPT/幻灯片的工具就困难得多，因为幻灯片包含空间排布、视觉图形，既不是纯文本，也没有现成的工具能让 Agent 直观展示修改前后的 diff。
Dwarkesh Patel 对此提出质疑：纯语言输入、纯语言输出的场景理应属于 LLM 的能力核心，但即便在这些领域，AI 创造的经济价值依然难以与编程匹敌。Dwarkesh 分享了自己尝试用模型做访谈文本重写、提取精彩切片视频文案的经历，效果并不理想；他还举了两人共同的朋友 Andy Matuschak 的例子——Andy 尝试了无数种方法（包括少样本上下文学习、监督微调 SFT、RAG 检索增强），试图让大模型编写高质量的间隔重复记忆卡片（spaced repetition cards），最终依然无法达到满意的水平。对此 Andrej 回应称，纯文本并不等同于简单，自然语言中蕴含着极高的“熵”（entropy）和修饰性表达，而代码具有极强的结构性；同时代码本身的编写门槛很高，因此只要 LLM 赋予哪怕一点点基础知识和辅助，人类就会感受到巨大的赋能感。
- 主流 AI 公司除去类似 Google 模式的消费者 Chat 收入外，底层 API 的商业化营收几乎被编程类工具全面主导。
- 幻灯片（Slides）等自动化办公软件难以快速落地，核心瓶颈在于缺乏展示视觉修改差异（diff）的基础设施，且信息排布具有空间视觉属性，脱离了单纯的字符流范畴。
- Andy Matuschak 在尝试让大模型制作间隔重复卡片时穷尽了上下文学习、监督微调（SFT）和检索增强（RAG），但依然无法获得满意的输出质量。
---
### [01:18:24](https://www.youtube.com/watch?v=lXUZvyajciY&t=4704s) 超级智能（ASI）的本质与人类理解与控制力的双重流失
谈及对超级智能（ASI）的理解，Andrej Karpathy 认为它并不是一个凭空出现的突变，而是人类社会长期自动化进程的一个自然外推。随着计算技术的发展，会有越来越多的自主实体逐步承接数字化任务，并在未来的某个时间点接管物理世界的劳作。当 Dwarkesh Patel 追问，超级智能是否会因为运转极快、拥有无数副本、副本能自我融合，从而建立起一种在“质”上完全不同于人类文明的新形态时，Andrej 承认那将极其异质且怪诞（extremely foreign）。
在这样的世界中，Andrej 最担忧的场景是人类对系统控制权和理解力的逐步流失。他指出：“我认为最有可能出现的结果，是人类对正在发生的一切逐步丧失理解。”（“I think that's the most likely outcome, that there will be a gradual loss of understanding.”）人类会不断在各处叠加这些智能层，能够理解底层运作的人会越来越少，最终伴随而来的是掌控力的彻底瓦解。
```mermaid
flowchart LR
accTitle: 超级智能生态中的控制力丧失模型
accDescr: 展现从分散的 AI 实体演进到多主体自主竞争，从而导致人类社会整体失控的逻辑链条
A["海量高度自主的 AI 实体<br/>(代表不同个人/机构运作)"] --> B["多主体间的博弈与竞争<br/>(部分实体行为失控偏航)"]
B --> C["自主对抗与冲突<br/>(其他实体展开防御与压制)"]
C --> D["混沌的自主活动熔炉<br/>(彻底脱离宏观预期)"]
D --> E["社会整体层面的控制力流失<br/>(Loss of Control)"]
```
针对 Dwarkesh 提出的反驳——“理解力缺失不等于控制力缺失，例如美国总统并不理解尖端半导体或复杂系统，却依然手握巨大的控制权力”——Andrej 给出了进一步的科幻式设想：
- **并非单一霸主，而是多实体博弈**：未来的图景并不是某一个全知全能的超级智能直接统治世界，而是社会中涌现出大量各自代表不同个体、不同机构的自主实体。
- **竞争引发的失控**：部分自主实体可能会走向失控或变异（go rogue），而其他实体则起而与之抗衡，形成一个完全由高度自主行为构成的“混沌熔炉”。
- **社会整体意图的落空**：即便微观上某些个人认为他们还在使唤这些代表自己的 AI 工具，但从整个社会的宏观视角来看，复杂交互所涌现出来的最终走向，将彻底脱离任何人的意志与掌控。
---
### [01:22:20](https://www.youtube.com/watch?v=lXUZvyajciY&t=4940s) 为什么真正的 AGI 依然会融入 2% 的 GDP 增长曲线？
当讨论回到“如果 AGI 能够替代顶尖 AI 工程师并复制上百万份自己，是否会触发智能大爆炸”时，Andrej Karpathy 表达了其标志性的非凡视角：他认为智能大爆炸不仅会发生，而且“我们早已经身处智能大爆炸之中长达数十年，这不过是按部就班的常态运作”（“I do, but it's business as usual because we're in an intelligence explosion already and have been for decades.”）。
Andrej 阐释，人类的 GDP 曲线本质上就是对工业体系无数个分支进行加权求和后的指数曲线。几百年来的工业革命代表了物理实体与工具建造的自动化，早期的编译器则是软件领域的早期自动化，人类社会一直在进行递归式的自我改进与扩张：“从太空回望地球，如果抛开生物机械结构不谈，地球原本是个相当沉闷无趣的地方；而现在，我们正身处一场慢动作播放的‘爆竹引爆’事件之中。”
这一认识解释了为什么 AI 很难在宏观经济数据中制造肉眼可见的尖锐拐点：
1. **历史重大技术在 GDP 中皆无突变迹象**：计算机、移动互联网等公认颠覆人类文明的技术，一旦投影到宏观 GDP 上，根本找不到断层式的跳跃，它们全部被熨平在恒定的指数曲线上。
2. **技术的扩散过程漫长而分散**：以 2008 年第一代 iPhone 问世为例，大众事后视其为划时代的地壳运动，但初版 iPhone 连 App Store 都没有，许多现代核心功能是后来历经多年才慢慢填补的。
3. **宏观经济的加权平均效应**：任何革命性技术的普及都需要经历漫长的铺设与渗透，最终都会被庞大社会的各行各业所稀释、平均，从而融进原本每年约 2% 的长期增长走势之中，AI 也不会例外。
### [01:24:31](https://www.youtube.com/watch?v=lXUZvyajciY&t=5071s) AI 是渐进扩散的自动化，还是打破 2% 增长常态的奇点？
Andrej Karpathy 认为，AI 的到来与历史上的技术变革本质上毫无二致——它依然是一种渐进式的演进。AI 本质上仍然是“程序”，只是一种允许人类编写以往无法实现的新型程序、一种新型的计算机与计算系统。由于技术在落地时面临着现实中的种种摩擦与缺陷，它会随着时间的推移逐步向整个社会扩散，最终平滑地融入长期的指数曲线中。Dwarkesh Patel 随即提出挑战：纵观人类历史，经济增长并非恒定，而是一条“超指数”（hyper-exponential）曲线——从一万年前接近 0% 的增长，到农业时代的 0.02%，再到工业革命后跃升至约 2%。Dwarkesh 质疑 Andrej 究竟是认为 AI 会将增长率直接推向 20% 甚至 200%，还是认为 AI 会像过去两三百年间的蒸汽机、电气化、铁路和计算机一样，仅仅维持既有的 2% 增长轨道。
Karpathy 明确表达了自己的怀疑态度，他坦言自己过去曾试图在宏观 GDP 曲线上寻找 AI 的脉冲印记，但最终说服了自己“这是不成立的”。即使行业内热烈讨论实验室中的递归式自我改进（recursive self-improvement），在现实商业与工程演化中也只是“一切照旧”（business as usual）：如今的大语言模型（LLM）确实让工程师能够以更高的效率构建下一代 LLM，自动化了更多组件和调优流程，但这种自我增强早已发生——正如工程师过去获得 Google Search、现代 IDE、代码自动补全乃至 Claude Code 一样，它们都只是整体平滑加速机制中的一环。Karpathy 强调：
> “For a while I tried to find AI or look for AI in the GDP curve, and I've convinced myself that this is false.”（有一段时间，我试图在 GDP 曲线中寻找 AI 的痕迹，但我已经说服自己这是不对的。）
当 Dwarkesh 确认这是否意味着“智能爆炸最终只会表现为帮助人类继续维持在 2% 的增长轨道上，就像当年的互联网一样”时，Karpathy 予以了肯定回答：
> “Yes, my expectation is that it stays in the same pattern.”（是的，我的预期是它会维持同样的模式。）
```mermaid
flowchart TD
accTitle: AI 经济影响路径分歧
accDescr: 对比 Karpathy 的渐进技术扩散假说与 Patel 的劳动力解锁爆炸假说
subgraph KarpathyView["Karpathy 视角：平滑扩散与 2% 恒定增长"]
K1["历史自动化演进<br/>(编译器 / 搜索引擎 / IDE)"] --> K2["AI 编写新型程序<br/>(作为新型计算系统)"]
K2 --> K3["现实世界摩擦与逐步扩散"]
K3 --> K4["维持在约 2% 的 GDP 增长轨迹"]
end
subgraph PatelView["Patel 视角：认知劳动力解锁与增长阶跃"]
P1["历史上劳动力与人口驱动增长<br/>(工业革命实现十倍阶跃)"] --> P2["真正 AGI 取代服务器中的人类<br/>(数十亿智能体供给)"]
P2 --> P3["解决各领域劳动力极度短缺瓶颈"]
P3 --> P4["进入 10% 至 20%+ 的超指数增长期"]
end
```
---
### [01:26:47](https://www.youtube.com/watch?v=lXUZvyajciY&t=5207s) 劳动力瓶颈与“数据中心天才”的辩论
Dwarkesh Patel 对此提出了截然相反的论点。他指出，真正的 AGI（并非如今的写代码 Bot，而是服务器里能完全替代人类的智能）与其他提高生产力的单项技术存在质的区别，因为 AGI 本身就是“劳动力”。人类社会目前处于极度受劳动力约束的状态，任何创业公司创始人最匮乏的都是顶尖人才。如果地球上凭空多出数十亿甚至上百亿个能独立发明创造、自主整合到经济体系中、从头到尾建立公司的天才“新人类”，这在量级上完全不可同日而语。Dwarkesh 援引历史案例指出，深圳或香港在人口充沛、资本相对稀缺的追赶期曾维持过数十年 10% 以上的高速增长；而工业革命也绝非单一发明带来的“魔法”，而是打破了原有的 0.2% 增长天花板，实现了近十倍的跃升。如今 AI 有望解锁巨大的“认知积压”（cognitive overhang），进而让增长率再次跳变。
Karpathy 对此提出了有力反驳。他指出“计算本身过去就是劳动力”（Computing was labor），大量原由人工处理的数字化信息流程早已被计算机彻底替代，自动驾驶同样是计算机承担人类劳动的生动范例，但它们并未打破经济运行的常态。在 Karpathy 看来，主张经济爆炸的人往往预设了一个极其不切实际的断层式跃迁假定，误以为人类能瞬间创造出一个全知全能的“盒中之神”（God in a box）：
> “At the same time, I do feel like people make this assumption of, 'We have God in a box, and now it can do everything,' and it just won't look like that. It's going to be able to do some of the things. It's going to fail at some other things. It's going to be gradually put into society, and we'll end up with the same pattern.”（同时我确实觉得，人们总是假设‘我们有了一个盒中之神，现在它能无所不能了’，但实际情况绝非如此。它能做好某些事，也会在另一些事情上彻底搞砸。它会被逐渐应用到社会中，我们最终还是会回到同样的模式中去。）
Karpathy 进一步强调：
> “This assumption of suddenly having a completely intelligent, fully flexible, fully general human in a box, and we can dispense it at arbitrary problems in society, I don't think that we will have this discrete change.”（这种突然拥有一个完全智能、完全灵活、完全通用的人类放在盒子里，并能随意派它去解决社会上任意问题的假设，我认为我们不会迎来这种离散式的突变。）
面对 Karpathy 对历史数据可靠性的质疑（例如对工业革命前统计记录精度的保留态度），Dwarkesh 澄清自己所谓的爆炸并非依赖单一的“盒中超级智能”去参悟科技，而是成百上千万个高水平认知个体自发寻找经济生态位的结果——就像高素质移民进入一个国家，他们会自主创业或发明，根本不需要顶层自上而下规划如何接入经济体系。但在 Karpathy 看来，这种认为存在某种重大解锁从而瞬间在数据中心装满天才的想法，依然缺乏历史先例的支持。
---
### [01:33:40](https://www.youtube.com/watch?v=lXUZvyajciY&t=5620s) 从演化史看智能与文化的偶发性奇迹
随后话题转向演化生物学与智能的本质。Dwarkesh 提到 Karpathy 曾向他推荐演化生化学家尼克·莱恩（Nick Lane）的著作，并询问 Karpathy 在从事了 20 年的 AI 研究、对“什么是智能以及构建智能需要付出什么”有了更具象的体感之后，是否会对自然演化能够自发偶然孕育出智慧感到更加吃惊。
Karpathy 兴奋地表示自己极度喜欢 Nick Lane 的书，并在来录制节目的路上还在收听相关播客。他直言，从演化历史的长河来看，智能的出现是非常晚近的现象，它的演变确实令他深感震撼：
> “With respect to intelligence and its evolution, it's very, very recent. I am surprised that it evolved.”（关于智能及其演化，它其实是非常非常晚近才出现的。我对它的演化诞生感到非常惊讶。）
Karpathy 提出了一个思考实验：假若宇宙中存在一千颗与地球类似的行星，按照 Nick Lane 的理论，生命起源初期的化学环境可能高度相似，大部分星球大概率都会演化出类似细菌的微观生命。然而演化史上存在若干极难跨越的瓶颈。例如：
- 细菌在地球上孤立存在了近 20 亿年之久，却未能实现复杂的跨越，这表明从原核生物迈向真核生物（eukaryote）极其艰难。
- 能够奔跑、爬行的多细胞动物出现的时间不过数亿年，仅占地球生命史的 10% 左右。
在 Karpathy 的直觉中，智能的演化本该是一个极其罕见的偶发事件。在大多数平行世界里，生命可能永远停留在“大量类动物生命体只从事动物本能行为”的阶段；而能够孕育出文化、能够发明知识并将其代代积累下来的物种能够诞生，本身就是一个巨大的演化奇迹。Dwarkesh 补充指出，如果赞同强化学习先驱 Richard Sutton 的观点——即智能的核心就在于动物智能，“只要能做到松鼠的智能水平，就已经走完了通往 AGI 的大部分路程”——那么早在 6 亿年前寒武纪大爆发不久后，自然界其实就已经跨过了动物智能这一核心门槛。
### [01:35:51](https://www.youtube.com/watch?v=lXUZvyajciY&t=5751s) 演化瓶颈与智能的多次涌现
谈话从生命演化史中的重大断层切入。Dwarkesh 提出，大约 6 亿年前的氧化事件似乎立刻催生了类似于松鼠水平的动物智能，这暗示着一旦有了氧气和真核细胞，通往智能的算法可能并没有那么复杂。然而 Karpathy 指出，仅凭演化速度很难断言最终的算法一定非常简单；评估难度的另一个维度是看某种状态在历史上被卡住（瓶颈期）了多久。他引用了生物学家 Nick Lane 的经典论述：细菌与古菌在长达 20 亿年的时间里毫无大型化迹象，其间虽然展现了极其丰富的生物化学多样性，却没有演化出任何动物形态。这种跨越 20 亿年的巨大瓶颈，在动物通往智能的道路上似乎并没有完全对应的漫长停滞。
为了探究通用算法的普遍性，两人转向讨论智能在地球上独立演化出来的次数。Karpathy 提到，除了人类等原始人类智能之外，鸟类智能（如渡鸦、乌鸦）同样表现出极高的解决问题能力。然而，鸟类大脑的解剖结构与哺乳动物差异巨大，彼此共享的生理机制很少，这种“智能多次独立涌现”的现象，暗示底层算法可能更容易被自然选择触发。
```mermaid
flowchart LR
accTitle: 演化瓶颈对比
accDescr: 对比原核生物到真核生物的物理化学瓶颈与生物智能演化的算法瓶颈。
subgraph S1["生物化学瓶颈"]
A["细菌与古菌多样性"] -->|"停滞 20 亿年"| B["真核生物与氧化事件"]
end
subgraph S2["智能涌现分支"]
B --> C["原始人类与哺乳动物"]
B --> D["鸟类智能（如渡鸦）"]
end
```
- **20 亿年的生化停滞**：原核生物经历了极长时间的化学演化，但直到出现内共生与真核细胞才打破僵局。
- **独立涌现的脑架构**：鸟类脑区（如发纹区）结构与人类大脑皮层截然不同，但二者均演化出了高度复杂的认知与推理能力。
---
### [01:37:26](https://www.youtube.com/watch?v=lXUZvyajciY&t=5846s) 适应性生态位与测试期智能
Dwarkesh 引入了博主 Gwern 和学者 Carl Shulman 的观点：可扩展的智能算法不仅存在于灵长类，也很可能存在于鸟类体内，但只有人类恰好落入了一个“边际智能回报极高且身体结构支持脑容量扩张”的生态位。鸟类受限于飞行重量，脑子一旦变大就会“直接从空中掉下来”；海豚虽然聪明，但在水下无法生火，物理和化学反应的上限极低，严重制约了工具利用。而人类拥有灵活的双手、懂得使用火来外置消化系统，从而释放了海量能量供给大脑，启动了智能正反馈飞轮。
Karpathy 赞同这种生态位驱动的假说，但他依然对自然选择最终迈向智力感到惊叹——在常规情况下，演化往往更容易陷入“堆积更大块肌肉”的局部最优。Gwern 的解释是，智能的演化走在一条极其严苛的钢丝上：如果环境过于稳定，演化会直接把最优策略固化（distill）进 DNA 中；如果环境毫无规律，学习又变得毫无意义。唯有在剧烈变化、无法预知未来的环境中，演化才会被迫放弃在权重中预设行为，转而在生物体生命周期内构建通用学习机制。Karpathy 总结道：“人类必须在出生后的测试期去把它弄明白。”
> “Humans have to figure it out at test time when they get born.” — Andrej Karpathy
- **物理约束**：鸟类的飞行重量上限与海洋生物缺乏工具操作介质（无法生火、水中化学受限），锁死了脑容量的边际收益。
- **训练期 vs 测试期**：绝大多数动物的行为模式在出生前就已经“预烤”（pre-baked）在基因中，而人类的大脑算法被设计为在面对未知的动态世界时在“测试期”（test time）现场学习。
---
### [01:39:45](https://www.youtube.com/watch?v=lXUZvyajciY&t=5985s) 大语言模型文化与多智能体自我博弈的空白
Dwarkesh 引用 Quintin Pope 的观点讨论了智能爆炸的节奏：人类在 6 万年前就具备了现代大脑认知架构，但直到约 1 万年前才迎来农业革命，中间这 5 万年人类一直在积累“文化脚手架”（cultural scaffold）。在大语言模型训练中，知识的代际蒸馏是免费且即时发生的，下一代模型直接基于海量文本或前代输出训练，这是否意味着 LLM 拥有比人类远为优越的文化加速能力？
Karpathy 对此给出了“是也不是”的判断。他认为当前的大语言模型根本没有真正的“文化”概念——人类没有设计激励机制让模型自发产出并沉淀代际文化。在他构想中，LLM 文化的雏形应该是一个所有智能体都能自由编辑的“巨型草稿本”（giant scratchpad），或者允许一个 LLM 写出一本书，去启发、震动其他 LLM。目前多智能体领域有两个极具颠覆性却尚未被攻克的方向：第一是**文化与知识库的自主沉淀**；第二是类似 AlphaGo 的**自我博弈（self-play）**。
```mermaid
flowchart TD
accTitle: LLM 文化与自我博弈的两大空白方向
accDescr: 展示多智能体系统中亟待突破的文化沉淀与自我博弈双轨结构。
subgraph Culture["维度一：智能体文化"]
A["巨型全局草稿本"] --> B["LLM 撰写长篇书籍与洞察"]
B --> C["其他 LLM 吸收、受到启发并迭代"]
end
subgraph SelfPlay["维度二：演化与对抗博弈"]
D["LLM 生成高难度新问题"] --> E["LLM 求解器尝试突破"]
E -->|"难度递增反馈"| D
end
```
谈及当前协同系统的技术瓶颈，Karpathy 坦言，现阶段的底层模型（包括 Claude Code、Codex 等）在认知心智上更接近“神童级的小学生或幼儿园儿童”。它们虽然能在博士级考试中拿到高分、能流畅生成看似精美的代码和文本，但在底层认知上依然缺乏组织与协作的成熟度。
> “LLMs don't really have culture right now and it's one of the impediments I would say.” — Andrej Karpathy
- **缺失的文化脚手架**：当前的训练模式只是权重蒸馏，没有动态累积、跨代交互的知识记录与交流机制。
- **缺失的自我博弈**：尚未出现一个模型生成刁钻问题、另一模型攻克解题，并在对抗中无限推高认知上限的成熟 LLM 架构。
- **学者型儿童（Savant Kids）**：Karpathy 认为现有前沿模型虽具有海量事实记忆，但在多步骤长程执行与认知协调上仍处于低龄阶段，不足以形成独立的文明或组织架构。
---
### [01:43:43](https://www.youtube.com/watch?v=lXUZvyajciY&t=6223s) 自动驾驶的技术长征与“9的进阶”
话题转向 Karpathy 在特斯拉主持自动驾驶研发（2017–2022）的经历。Dwarkesh 询问，从酷炫的技术演示到成千上万辆车上路自主行驶，为什么整整耗费了数十年？Karpathy 首先纠正了一个前提：哪怕到现在，自动驾驶在很多方面都“远远没有完成”。他回顾道，自动驾驶的最早演示早在 1986 年卡耐基梅隆大学（CMU）的货车测试中就已出现；而在 2014 年他加入特斯拉之前，一位在 Waymo 工作的友人就曾带他在帕洛阿尔托体验过一段近乎完美的无人驾驶试乘。
Karpathy 强调，Demo 到落地产品之间存在着巨大的鸿沟，在失败代价极高的领域尤为明显。如果一个任务只是做视频创作或写普通脚本（“vibe coding”），容错率极高；但在自动驾驶中，犯错可能致命；在企业级生产环境的软件工程中，一个安全漏洞可能导致数亿人的社保账号泄露。这些高危领域的迭代规律不是线性爬坡，而是一场“追逐 9 的长征”（a march of nines）。
> “When you get a demo and something works 90% of the time, that's just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine.” — Andrej Karpathy
> “Whenever I see demos of anything, I'm extremely unimpressed by that.” — Andrej Karpathy
| 维度 | 技术演示（Demo） | 工业级产品（Product） |
|:--- |:--- |:--- |
| **典型成功率** | 约 90%（仅达到“第一个 9”） | 99.999%+（需要跨越多个 9） |
| **边际研发成本** | 验证概念，前期工作量相对固定 | 每一个后续的“9”都需要同等甚至更多的工程投入 |
| **容错特征** | 允许偶发严重故障与人工干预 | 零容忍致命失败（如车祸、核心数据泄露） |
| **适用场景** | 概念验证、辅助创作、原型开发 | 自动驾驶、关键基础设施、高危企业级代码 |
Karpathy 回忆，自己在特斯拉深耕的五年时间里，团队充其量也只是攻克了两到三个“9”，而前方依然有很长的路要走。正是这段高强度的工程磨砺，让他对目前 AI 领域层出不穷的各路演示彻底脱敏，不再轻易为光鲜的 Demo 所动。
### [01:46:43](https://www.youtube.com/watch?v=lXUZvyajciY&t=6403s) 从 Demo 到产品的“9 的进军”与安全门槛
Andrej Karpathy 强调，演示（demo）与真实产品之间存在着巨大的鸿沟。如果某个技术仅仅是某人在展示会上拼凑出来的 demo，情况反而是最糟糕的；即便用户能上手交互，情况稍好一些，但也远未大功告成。一旦接触现实世界，系统会遭遇各种意想不到的边缘情况，暴露出大量需要修补的局部行为缺陷。他用自动驾驶工程中著名的“9 的进军”（march of nines）来类比软件工程中的可靠性追求：达到 90%、99%、99.9%……每向前推进一个“9”，其难度都是恒定甚至呈指数级递增的。因此，Karpathy 指出：“It's a march of nines. Each nine is constant. Demos are encouraging. It’s still a huge amount of work to do.”
```mermaid
flowchart LR
accTitle: 从 Demo 到产品落地的可靠性鸿沟
accDescr: 展示技术从最初概念演进为高可用生产级系统所需的递进过程与挑战
subgraph Phase1["展示阶段"]
A["Demo 原型"] --> B["可交互版本"]
end
subgraph Phase2["落地阶段"]
B --> C["真实产品"]
C --> D["碰撞现实边缘情况"]
D --> E["修补行为缺陷"]
end
subgraph Phase3["可靠性验证"]
E --> F["追求高可用可靠性<br/>(March of Nines)"]
end
```
Dwarkesh Patel 对此展开讨论，指出人们常以为自动驾驶进展缓慢是因为其犯错代价过于致命——人类驾驶员平均每行驶 40 万英里或大约每 7 年才会发生一次严重事故；如果要求一个编程智能体（coding agent）在 7 年内不犯哪怕一次灾难性错误（比如搞崩核心系统），其部署难度就会极高。Karpathy 补充说，如果换算成输出代码的自然时间，根本不需要 7 年，代码以极高频率生成，可能几分钟内就会造成破坏；即使按生成的 Token 数量折算为人类 7 年的工作量，其出错概率依然居高不下。更重要的是，自动驾驶本质上只是人类日常活动中数千项技能里的“单一垂直领域”，而通用软件工程的暴露面（surface area）要广阔得多，因此在某种程度上，软件工程是一个比自动驾驶复杂得多的难题。
- **9 的进军（March of Nines）**：可靠性从 99% 提升到 99.9%、99.99% 的过程中，每增加一个“9”所付出的工程努力都是巨大的。
- **氛围编程 vs 关键安全**：Karpathy 调侃道，除非是在做“氛围编程”（vibe coding），大家随便玩玩、不在乎 Bug，否则只要涉及关键任务，软件领域就需要极高的安全冗余。
- **人类驾驶基准**：人类大约每 40 万英里或每 7 年发生一次重大驾驶失误，这成为衡量机器能否取代人类的苛刻门槛。
---
### [01:48:08](https://www.youtube.com/watch?v=lXUZvyajciY&t=6488s) 自动驾驶的幻觉：泛化瓶颈与“隐形人”安全员
Dwarkesh Patel 提出了对该类比的一个常见质疑：自动驾驶之所以花了那么长时间，是因为必须从零解决基础感知、鲁棒的表征构建，以及让模型具备常识以应对分布外情况（例如有人在路边挥手，模型能自然理解其含义）；而当今的 LLM 和 VLM（视觉语言模型）似乎已经“免费”赋予了我们这些常识与表征能力，因此将 AI 部署到新垂直领域，更像是把现成的自动驾驶汽车开进一座新城市，虽然困难但绝不需要再花 10 年。Karpathy 对此明确表示怀疑：“I don't think that we're getting magical generalization completely out of the box, in a certain sense.” 他认为大语言模型依然极易出错，存在大量认知盲区，人们对于当下模型到底提供了多少真正的“免费泛化”存在误判。
更关键的是，Karpathy 直言自动驾驶技术至今远未算真正完成，其部署规模依然微乎其微。他指出像 Waymo 这类系统在很大程度上是“把未来的技术强行拉到当下，但其代价是完全不具备经济效益”——这不仅包含了车辆的运维与边际成本，还涉及庞大的资本开支（Capex）。此外，外界看着空无一人的自动驾驶车在街上跑，其实是一种表象上的欺骗。在车辆背后，存在着规模庞大、高度复杂的远程操作中心（teleoperation centers），人类始终以某种形式保留在闭环之中。正如 Karpathy 所言：“In some sense, we haven't actually removed the person, we've moved them to somewhere where you can see them.” 虽然无人车确实跨过了“感觉上已经成真”的心理门槛，但当看到 Waymo 无法驶入旧金山的某些特定区域时，他怀疑这正是因为那些地方蜂窝网络信号不足以支撑远程后台介入。
```mermaid
flowchart TD
accTitle: 自动驾驶运营结构背后的隐形人模式
accDescr: 展现空无一人的无人车与后台远程操作中心之间的通信与安全依赖结构
subgraph RealWorld["城市实际道路"]
A["Waymo 无人车<br/>(车内无驾驶员)"]
end
subgraph RemoteOperations["后台监控体系"]
B["蜂窝通信网络<br/>(易受信号死角限制)"]
C["远程支持控制中心<br/>(Teleoperation)"]
D["人类操作员<br/>(随时准备介入决策)"]
end
A <-->|"实时数据 / 指令回传"| B
B <--> C
C <--> D
```
谈及战略路线时，曾领导特斯拉自动驾驶团队 5 年的 Karpathy 提到，他非常喜欢并经常乘坐 Waymo，但 Waymo 堆砌大量激光雷达等传感器的做法虽然取得了先发优势，特斯拉基于纯视觉与规模化部署的路径在长期来看才是更具可扩展性（scalable）的策略。他同时重申，绝不能把自动驾驶说成是“花了十年就攻克的领域”：一方面这项研发始于 1980 年代而不是十年前；另一方面终点根本尚未到来——在所有人无需考取驾照、实现真正全球规模化部署之前，自动驾驶的征途远未结束。
---
### [01:52:13](https://www.youtube.com/watch?v=lXUZvyajciY&t=6733s) 物理原子与数字比特：算力狂潮会否重蹈泡沫覆辙？
Dwarkesh Patel 进而从硬件延迟与经济学角度探讨了自动驾驶与当前知识工作 LLM 的差异。在无人车上，模型参数量受限于毫秒级延迟与车载硬件（可能只有数千万参数）；而在软件与知识工作中，不仅延迟容忍度更高，其边际运营成本（Opex）更低，研发成本还可以通过训练摊销，而不必像无人车那样每扩张一个实例就要制造一辆昂贵的新车。Karpathy 赞同这一核心差异：“Bits are a million times easier than anything that touches the physical world.” 数字世界的比特可以在短时间内被任意重组、快速迭代，这决定了纯软件产业的落地速度必定远快于物理世界。但他同时也提醒，除了技术本身，必须考虑社会层面的阻力——正如人们往 Waymo 车顶放置交通锥来瘫痪车辆一样，大模型落地同样会遭遇法律归责、保险制度和社会的未知抵制反应。
讨论随后转向当前全球轰轰烈烈的 AI 算力军备竞赛：全球算力正以一两年内翻 10 倍、十年内可能翻 100 倍的势头疯狂扩张。如果 AI 在实际产业中的渗透速度因可靠性问题而比乐观者预期的慢得多，我们是否正在像 19 世纪的铁路狂热或 1990 年代末铺设光纤的电信泡沫一样，过度建设了超前的基础设施？
| 对比维度 | 历史基建泡沫（铁路 / 电信光纤） | 当前 AI 算力大扩张 |
|:--- |:--- |:--- |
| **建设性质** | 提前数年甚至数十年超前铺设物理管网与铁轨 | 极速建设超大规模 GPU 集群与数据中心 |
| **短期需求对接** | 物理资产闲置数年，等待互联网生态与实际运输需求跟进 | 出现类似 Claude Code、OpenAI Codex 及 ChatGPT 的即时需求承接 |
| **资本沉没风险** | 引发破产潮与资本暴跌（如 90 年代末电信泡沫） | 算力被前沿模型快速消耗，但过高的估值期望面临回调风险 |
针对可能被认为过度悲观的质疑，Karpathy 澄清说，自己骨子里其实对这项技术极度乐观，坚信这一切技术难题终将被攻克。他之所以公开表达审慎，纯粹是对社交媒体（如 Twitter/X）上那些荒谬浮夸的炒作言论感到厌烦。那些不切实际的激进预测，本质上是初创公司为了融资诉求或是网络红人为了将注意力变现而制造的激励产物。
关于算力是否过度建设，Karpathy 给出了相对正面的判断：“I don't know that there's overbuilding. I think we're going to be able to gobble up what, in my understanding, is being built.” 他举例指出，像 Claude Code 或 OpenAI Codex 这类奇迹般的编程智能体在一两年前根本不存在，而像 ChatGPT 这样展现出来的庞大消费级需求，完全有能力将正在建设的算力吞食殆尽。他真正想要敲响警钟的，是在他涉足 AI 领域的 15 年间，无数知名专家一次又一次给出的极端激进的时间表：“I've heard many, many times over the course of my 15 years in AI where very reputable people keep getting this wrong all the time. I want this to be properly calibrated.”
### [01:56:53](https://www.youtube.com/watch?v=lXUZvyajciY&t=7013s) 为何创办 Eureka 而非重返前沿 AI 实验室
在探讨地缘政治影响与技术的现实边界之后，Dwarkesh Patel 向 Andrej Karpathy 提问：既然 Andrej 拥有顶尖的技术号召力，大可创办一家新的前沿 AI 实验室去攻克核心技术瓶颈，为何现在却将精力全面投向了教育创企 Eureka？Andrej 坦承，在他看来，当下前沿 AI 实验室所做的事情具有很强的“确定性”（determinism）——即便自己加入并施以援手，所起到的也只是增量作用，无法带来不可替代的独特改善。
更为关键的是，Andrej 抱有一种深层的生存与社会焦虑。他担忧所有这些令人惊叹的自主系统和戴森球（Dyson spheres）在被 AI 全面接管并建构起来的同时，人类自身反而被彻底边缘化与剥夺赋权（disempowered）。他不想看到人类的未来沦为像电影《机器人总动员》（WALL-E）或《蠢蛋搞怪秀/蠢蛋进化论》（Idiocracy）里描绘的那样，只能瘫坐一旁、心智退化。
Andrej 强调，教育是确保人类在与超级智能共存的未来中变得更好、更强大的唯一路径：
> “I care not just about all the Dyson spheres that we're going to build and that AI is going to build in a fully autonomous way, I care about what happens to humans. I want humans to be well off in the future.” —— Andrej Karpathy
- **星际舰队学院愿景**：Andrej 将 Eureka 的终极形态比作《星际迷航》（Star Trek）中的“星际舰队学院”（Starfleet Academy）——一所汇聚前沿科技、传授造船与星舰驾驶技能、与时代前沿紧密接轨的终极精英技术学府。
---
### [01:59:16](https://www.youtube.com/watch?v=lXUZvyajciY&t=7156s) 顶尖家教的标准与当前 AI 导师的局限
尽管业界普遍认为 AI 会从根本上重塑教育，但 Andrej 认为目前全行业还处于非常初级的阶段。大部分团队做的尝试不过是给大语言模型套一层 Prompt（提示词），让学生向模型提问。这种方式虽然提供了一定便利，但产出的内容往往形同劣质信息垃圾（slop），根本达不到真正有效辅导的标准。
为了说明什么才是顶尖的教学体验，Andrej 分享了他近期学习韩语的经历。他尝试过三种不同的学习模式：
1. **完全自学**：在互联网上搜寻零散资料，效率低下且容易迷茫；
2. **小班教学**：在韩国当地参加一个大约 10 人的小班，由一名老师授课，体验虽有趣但针对性不足；
3. **一对一专属家教**：遇到了一位极具天赋的优秀韩语导师。
正是这段一对一家教经历，让 Andrej 深刻体会到卓越教学的门槛有多高。导师仅凭极短的闲聊互动，就能立刻精确探明他的知识边界、理解盲区以及内在的世界模型。她总能抛出最契合的问题，将教学内容精准卡在他能力的极限边缘——既不至于太难而让人受挫，也不至于过于浅显而显得琐碎。在那种状态下，外部获取知识的一切摩擦都被彻底消除，“自己成了学习过程中唯一的瓶颈”，学习体验极为酣畅。
而眼下没有任何大语言模型能够做到这一点。Andrej 坦言，这就像他过去做计算机视觉 AI 咨询时的情景：作为专家，他给企业带来的最大价值往往是劝他们“当下千万别用 AI”。在教育领域亦是如此，虽然 ChatGPT 是很棒的学习辅助，但自动化顶级导师所需的感知和心智建模能力尚未成熟。因此，Eureka 在当前阶段会采用更加务实的混合形态（结合物理实体与数字化教学），而非盲目强推尚未成熟的全自动 AI 导师。
> “I need to be always appropriately challenged. I can't be faced with something too hard or too trivial, and a tutor is really good at serving you just the right stuff.” —— Andrej Karpathy
---
### [02:02:35](https://www.youtube.com/watch?v=lXUZvyajciY&t=7355s) 构建“认知跃迁的阶梯”与首门课程 LLM101N
谈及 Eureka 近一两年的具体落地规划，Andrej 透露他正在打造第一门旗舰级基准课程：LLM101N。这是一门旨在成为全球 AI 学习终极目的地的顶尖课程。该课程的结课大作业（Capstone project）即是他之前提到的全栈极简项目 nanochat。
Andrej 对教育本质的理解与传统观念截然不同。许多人认为教育是一种偏软性的“知识传播”过程，但在 Andrej 眼里，教育本质上是一个极其严密、高难度的工程技术问题——即如何为复杂的现代知识体系搭建坚实平滑的“爬坡阶梯”（ramps to knowledge）。如果一个教学载体（如极度简化的全栈系统 nanochat）能够让学生在阅读和亲自动手实现时，产生密集的顿悟与理解，便意味着实现了极高的“每秒顿悟数”（eurekas per second）。
```mermaid
flowchart LR
accTitle: Eureka Knowledge Ramp Framework
accDescr: Andrej Karpathy 的教育体系通过构建极简全栈阶梯实现高密度的每秒顿悟数
subgraph Design["课程设计机制"]
A["顶尖领域专家/教职"] --> B["构建认知阶梯<br/>(Ramps to Knowledge)"]
B --> C["极简全栈范例<br/>(如 nanochat)"]
end
subgraph Learning["学生学习飞轮"]
C --> D["适度挑战<br/>(不繁难亦不浅薄)"]
D --> E["密集理解爆发<br/>(Eurekas per Second)"]
end
```
针对当前的教学互动，Andrej 的设想不是依赖 AI 自动去探查学生的思维，而是依靠学生自身的反思自察能力：借助参考代码实现、LLM 的局部解答以及助教（TA）团队，确保学生永远不会在卡点上停滞不前。
在内容生产效率方面，Andrej 提到了他在斯坦福大学主创全校首门深度学习课程 CS231n 的历史。相比当年纯人工搭建 CS231n，如今他在大语言模型的辅助下开发 LLM101N，效率获得了质的飞跃。模型处理了大量繁琐而机械的琐事，但他依然牢牢处于主导地位（human-in-the-loop），因为现阶段的 AI 还无法独立进行高水准的原创性教学架构设计：
> “In my mind, education is the very difficult technical process of building ramps to knowledge.” —— Andrej Karpathy
- **团队拓展路线**：Eureka 不会变成 Andrej 一个人的“可汗学院”（Khan Academy）。尽管他起初会亲自掌舵 AI 课程，但未来 Eureka 将招募顶尖的外部教职人员（Faculty），负责各学科课程的顶层认知架构，并结合 AI 赋能的助教体系（Automated TAs），逐步将这种“认知阶梯”复制到更多前沿科学与工程领域。
### [02:07:29](https://www.youtube.com/watch?v=lXUZvyajciY&t=7649s) 打造“星际舰队学院”与教育的未来形态
Andrej Karpathy 勾勒了他构想中的“星际舰队学院”（Starfleet Academy）的雏形。他认为这项计划应该包含两个层次：核心是一个实体机构，学员全日制实地沉浸其中，从头到尾彻底啃透材料；次级层级则是一个面向全球 80 亿人的数字化产品，整合网络资源与大语言模型（LLM）助教，尽管这种体验不如实体教学极致，甚至略带一些噱头性质，但它拥有无可比拟的普及性。这种构想本质上是从第一性原理出发，利用当今可用的工具重新发明大学，专门筛选那些具备强烈学习动机与探索兴趣的人。
在 AGI 来临之前与之后，教育的驱动机制将发生根本性剧变。当前的教育与再培训高度实用，因为人类面临岗位转变，许多人渴望通过学习 AI 技能来提升自己以换取财富，这种动机非常纯粹直接。但如果 AGI 将一切工作自动化，人类不再需要为生存而劳动，教育的性质就将从“实用”转向“纯粹的乐趣”。
“我常说，AGI 之前的教育是实用的；AGI 之后的教育是好玩的。”Andrej Karpathy 提出了这一论断。他用当今的健身房文化作为绝佳类比：人类早就发明了各种机械来搬运重物，不再需要人体的物理力量去干体力活，但人们依然风雨无阻地涌向健身房，因为这不仅有益健康、带来快乐，还能让人练出六块腹肌，在人类演化与深层心理层面上极具吸引力。同理，在 AGI 时代，人们走进学校就会像如今走进健身房一样自然。
```mermaid
flowchart TD
accTitle: Education Paradigms Before and After AGI
accDescr: Diagram illustrating the shift in educational motivation and goals between the pre-AGI and post-AGI eras.
subgraph PRE["AGI 之前：实用主义导向"]
A["经济回报与职业再培训需求"] --> B["克服认知障碍，学习专业技能"]
B --> C["获得工作机会与生产力变现"]
end
subgraph POST["AGI 之后：自我实现与乐趣导向"]
D["全面自动化消除工作需求"] --> E["高保真 AI 导师移除认知阻力"]
E --> F["认知健美：探索高阶知识并享受乐趣"]
end
```
---
### [02:10:00](https://www.youtube.com/watch?v=lXUZvyajciY&t=7800s) 认知健身房与后 AGI 时代的人类追求
针对 Dwarkesh Patel 关于“这种教育究竟是纯粹的娱乐消遣，还是让人类保持对 AI 控制权的赋能手段”的追问，Andrej Karpathy 坦率地承认，在极其漫长的长期视角下，试图在智力上与 AI 竞争注定是一场“必输的局”（a losing game）。但他坚信，人类的潜能目前仅仅被开发了冰山一角，绝大多数人在学习中途放弃，仅仅是因为教材要么太晦涩、要么太乏味。一旦技术解决了定制化教学的难题，每个人都能毫不费力地掌握五门外语，或者通晓大学本科的所有核心课程。
Dwarkesh Patel 深表赞同，指出一百年前很少有人能天然拥有健美身材，更不可能随随便便卧推两三副大杠铃片或跑完马拉松；如今这类体能成就屡见不鲜，完全归功于成体系的举重与耐力训练。教育在未来也将迎来这种“系统化突破”。Andrej Karpathy 认为，这种对自我提升的渴望根植于恒久不变的人性：
- 历史上古希腊精英或欧洲贵族阶层在物质充裕的环境中，往往将大量精力投入到体能与哲学的全面繁荣中。
- 他直言，如果未来演变成类似电影《机器人总动员》（WALL-E）或《蠢蛋进化论》（Idiocracy）那种人类彻底退化为废人的景象，“那我根本不在乎是不是造出了戴森球，那都是一个可怕的结局”，他所寄托的未来必须是人类在某种意义上变得“超凡”（superhuman）。
- 这种高强度的知识追求甚至可能演变成一种“智力奥林匹克”竞技，就像现在的极限力量举选手一样，涌现出一批借助完美 AI 导师将大脑认知逼向极限的人。
针对在线教育多年来为何未能彻底普及高深知识的问题，两人一致认为症结在于“挫败感与反馈缺失”。学习者在自学中陷入困境或无聊时会产生极大的负面奖励，而一位像真人外教那样细致敏锐的 AI 导师，正是将学习从痛苦折磨转变为纯粹正反馈的关键技术解法。
---
### [02:15:27](https://www.youtube.com/watch?v=lXUZvyajciY&t=8127s) 物理学思维：剥离噪声与寻找一阶项
当话题转向如何成为像 Andrej Karpathy 一样出色的教学者时，Patel 询问了制作高质量深度技术教程的方法论。Karpathy 将自己的思考与教学风格完全归功于早年的物理学训练。他激情洋溢地指出，基础教育的核心绝不是为了未来从事某项工作而去死记硬背具体知识，而是为了“给大脑开机启动”（booting up a brain），而物理学正是启动大脑的最佳框架。
在物理学中，人们时刻都在建立抽象模型，明白任何复杂系统都存在一个能够解释绝大部分现象的“一阶近似”（first-order approximation），随后才是二阶、三阶甚至四阶的高阶修正项。 Karpathy 提到：
- 人们常取笑物理学家走进教室张口就是“假设有一头处于真空中的球形奶牛”，但这种思维方式极具洞察力与跨界通用性。
- 他推荐了物理学家 Geoffrey West 撰写的名著《规模》（*Scale*），书中通过物理学的一阶缩放律（Scaling Laws）解释生物学现象：动物的心率与其体型遵循严格比例；动物的产热量随体积呈三次方增长（立方规律），而散热量则随体表表面积呈二次方增长（平方规律）。
这种认知工具深深渗透进他的教学设计中。当脑海中面对纷繁复杂的知识网络时，他会执着地去剥离所有枝节与噪声，寻找最纯粹的一阶核心结构，并用最精炼的原型代码将其具象化呈现。他以自己编写的著名开源仓库 **micrograd** 为例，仅仅用约 100 行 Python 代码，就完全展示了神经网络反向传播与自动求导的核心机制，用最基础的加法、乘法拼装成神经网络的乐高积木，帮助学习者直击技术本质。
- micrograd 核心设计：用 100 行左右纯 Python 代码构建自动微分引擎与微型神经网络，直观呈现计算图反向传播。
- 物理学教学哲学：始终先抓住一阶项（核心机制），跑通最简单示例，随后再逐步挂载高阶项（系统优化与复杂边界条件）。
### [02:18:14](https://www.youtube.com/watch?v=lXUZvyajciY&t=8294s) Micrograd 的教学哲学：剥离工程效率，直击智力核心
Andrej Karpathy 深入剖析了神经网络学习的本质，以此说明优秀的教学应当如何构建。在他看来，现代神经网络学习的核心无非就是建立计算图，然后进行前向传播和反向传播以获取梯度。为了将这一过程彻底透明化，他编写了 micrograd——仅用约 100 行清晰易读的 Python 代码，就能实现任意神经网络的前向与反向计算（虽然效率并不高）。Karpathy 强调，这 100 行代码涵盖了理解神经网络训练所需的全部核心知识，而业界投入海量精力所做的事情本质上只是为了工程效率：张量的内存排布（stride）、底层计算核心（kernels）的优化以及内存搬运的编排等。
```mermaid
flowchart TD
accTitle: 神经网络训练核心与工程效率的分层
accDescr: 展示从 micrograd 核心算法到工程效率优化的层次结构
subgraph Core["核心智力支柱 (Micrograd ~100行代码)"]
CG["构建计算图 (Computational Graph)"]
FP["前向传播 (Forward Pass)"]
CR["递归应用链式法则 (Chain Rule)"]
BP["反向传播计算梯度 (Backward Pass)"]
OPT["优化任意可微函数"]
CG --> FP --> CR --> BP --> OPT
end
subgraph Efficiency["工程效率层 (业界海量工程落地)"]
T["张量排布与步长 (Strides)"]
K["计算核心优化 (GPU Kernels)"]
M["内存搬运与调度 (Memory Orchestration)"]
end
Core -.->|"外围工程包裹实现高性能"| Efficiency
```
对 Karpathy 而言，教学中最令人着迷的智力挑战，就是找到这种“低阶项”（small-order terms）并将其提炼出来。现实中的知识往往是一团混乱交织的线团，教育者的任务是将这团线理顺，铺设成一条平缓上升的坡道，让学习者掌握的每一个新概念都严格仅依赖于前一个概念。这种梳理知识脉络的过程本身，就是一项极具美感且令人上瘾的认知活动。
Andrej Karpathy 这样概括代码与原理的关系：
> “So micrograd, these 100 lines of Python, are everything you need to understand how neural networks train. Everything else is just efficiency.”
### [02:19:31](https://www.youtube.com/watch?v=lXUZvyajciY&t=8371s) 教学的进阶路径：先展示痛苦，再给出解决方案
Dwarkesh Patel 指出，Karpathy 广受好评的 Transformer 教学视频之所以引人入胜，正是因为他从最基础的二元语法（bigrams）讲起——它本质上就是一个查找表（lookup table），记录当前词或前一个词对应的下一个词。这种从查表直接过渡到 Transformer 架构的路径，让每一个组件的引入都具备了强烈的动机（motivated）。学习者无需死记硬背复杂的注意力机制公式，而是深刻理解每一个设计究竟是为了解决什么痛点。
针对这种循序渐进的教学法，Karpathy 强调，优秀的教育者必须在给出答案前不断促使学生自主思考。如果过早直接抛出最终方案，剥夺了学习者自己猜测和探索的机会，在教学上是一种极大的浪费，甚至是对学生的不尊重。当学生尝试自行构思解法时，他们会更清楚行动空间（action space）与最终目标是什么，进而真正明白为什么只有特定的架构才能达成该目标；此时再呈现解答，学生才会产生顿悟与由衷的赞叹。
对于如何促使学生思考，Andrej Karpathy 语气犀利地指出：
> “That’s a little bit of a…I don’t want to swear but it’s a dick move towards you to present you with the solution before I give you a shot to try to come up with it yourself.”
### [02:21:02](https://www.youtube.com/watch?v=lXUZvyajciY&t=8462s) 知识的诅咒与“酒吧对话”效应
谈及为何各领域的顶尖专家往往不擅长向初学者科普时，Karpathy 直言这是典型的“知识的诅咒”（curse of knowledge）。专家早已对许多背景知识习以为常，很难再切身体会完全不懂该领域的新手视角。Karpathy 透露，即便是他自己也难免受此困扰。例如最近有人向他推荐一篇生物学论文，他阅读时脑海中瞬间冒出大量看似愚蠢的基础疑问。为了解决困惑，他把论文扔进 ChatGPT 的上下文窗口中提问，梳理清楚基础概念后，又把这些对话记录直接分享给了原作者。他认为，向专家展示新手提出的“蠢问题”，恰恰能帮助专家打破知识盲区，明白日后该如何通俗地讲解。
Dwarkesh 与 Karpathy 随后探讨了一个极为普遍却惊人有效的现象：当研究者撰写学术论文或博客公告时，往往倾向于使用最抽象、堆砌术语的语言，并花上四五个段落做冗长铺垫；然而在学术会议后的酒吧里，当大家端着啤酒闲聊时，作者却能在短短两三句话内直击本质，把论文的核心实验与创新思路讲得一清二楚。这种面对面的非正式交流迫使人们剥离废话、“直接把事实说出来”（Just say the thing），其表达不仅通俗易懂，而且往往比布满修辞的论文更准确、更具科学本质。
在回忆博士阶段与同行交流的体会时，Andrej Karpathy 感叹道：
> “Then you catch them, you're having beers at the conference later, and you ask them, ‘So this paper, what were you doing? What is the paper about?’ They will just tell you these three sentences that perfectly captured the essence of that paper and totally give you the idea.”
### [02:23:47](https://www.youtube.com/watch?v=lXUZvyajciY&t=8627s) 学习策略：按需深度钻研与“以教促学”
在播客尾声，Dwarkesh 询问当缺乏顶级导师拆解知识时，普通人进入陌生领域应采取何种学习策略。Karpathy 坦言学习深奥新知始终是一个痛苦的过程，但他极为推崇“按需学习”（learning on demand）与“深度优先学习”（depth-wise learning）。传统的学校教育通常采用“广度优先”（breadth-wise），列出一门“某某导论”课并强迫学生背诵海量知识点，同时许诺这些知识未来总会派上用场。相比之下，Karpathy 更偏好围绕一个具体项目或目标展开深度攻坚——因为能获得明确的反馈和正向激励，按需吸收知识的驱动力要强烈得多。
此外，Karpathy 强调“向他人解释”是巩固知识最锋利的武器。当尝试把一个概念复述给别人时，任何理解上的漏洞与含糊之处都会立即暴露无遗，逼迫自己承认“我其实并不懂这里”。这种逼迫自己直面认知空白、重新梳理并填补逻辑漏洞的过程，能促使大脑真正对知识进行重构与消化。Patel 赞同这是全场最完美的总结，双方在愉快的致谢声中结束了长达两个半小时的深度对谈。
Andrej Karpathy 这样总结解释对自身认知的倒逼作用：
> “This is an aspect where education is a bit more selfless, but explaining things to people is a beautiful way to learn something more deeply.”
- **深度优先 vs 广度优先学习**：
- *广度优先（学校体系）*：预先灌输大量“未来可能有用”的基础库，学习过程缺乏即时反馈与动力。
- *深度优先（项目按需）*：以实际产出为驱动，遇到技术阻碍时按需补齐短板，学习反馈闭环极强。
- **以教促学的核心机制**：通过强制输出识别自身认知脱节，倒逼自己回到源头补齐断层，直至能用最简洁的语言复现逻辑链。
## Notable quotes
> [40:53](https://www.youtube.com/watch?v=lXUZvyajciY&t=2453s) Reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before it is much worse because previously we were just imitating people, so it has all these issues. — Andrej Karpathy
> [07:54](https://www.youtube.com/watch?v=lXUZvyajciY&t=474s) In my post, I said we're not building animals. We're building ghosts or spirits or whatever people want to call it, because we're not doing training by evolution. We're doing training by imitation of humans and the data that they've put on the Internet. — Andrej Karpathy
> [01:43:43](https://www.youtube.com/watch?v=lXUZvyajciY&t=6223s) When you get a demo and something works 90% of the time, that's just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. — Andrej Karpathy
> [55:16](https://www.youtube.com/watch?v=lXUZvyajciY&t=3316s) 因为我们不擅长死记硬背，我们才被迫在更普遍的意义上去寻找规律。相比之下，大语言模型极其擅长记忆。 — Andrej Karpathy
> [01:52:13](https://www.youtube.com/watch?v=lXUZvyajciY&t=6733s) Bits are a million times easier than anything that touches the physical world. — Andrej Karpathy
> [53:50](https://www.youtube.com/watch?v=lXUZvyajciY&t=3230s) 如果你在脑海中不断自我演绎，然后关注它，你就是在用自己的样本进行训练，你在用自己的合成数据进行训练。如果你这么做太久，你就会偏离正轨，严重崩塌。你在生活中必须不断去寻找熵。 — Andrej Karpathy
> [02:02:35](https://www.youtube.com/watch?v=lXUZvyajciY&t=7355s) In my mind, education is the very difficult technical process of building ramps to knowledge. — Andrej Karpathy
> [02:18:14](https://www.youtube.com/watch?v=lXUZvyajciY&t=8294s) So micrograd, these 100 lines of Python, are everything you need to understand how neural networks train. Everything else is just efficiency. — Andrej Karpathy
## Mentioned
Andrej Karpathy, Dwarkesh Patel, Geoffrey Hinton, Richard Sutton, Yann LeCun, Richard Feynman, Nick Lane, Andy Matuschak, Gwern, Carl Shulman, Quintin Pope, Geoffrey West, OpenAI, DeepMind, Google, Nvidia, Waymo, Tesla, Carnegie Mellon University (CMU), Eureka, Khan Academy, AlexNet, Transformer, Universe, Claude, Claude Code, Codex, OpenAI Codex, ChatGPT, GPT-4, GPT-4.5, GPT-5 Pro, Llama 3, DeepSeek-V3.2, InstructGPT, AlphaGo, micrograd, nanochat, PyTorch, DistributedDataParallel, Rust, Visual Studio Code, iPhone, LLM101N, CS231n, Starfleet Academy (Star Trek), WALL-E, Idiocracy, Scale (book)
## Video description
The Andrej Karpathy episode. During this interview, Andrej explains why reinforcement learning is terrible (but everything else is much worse), why AGI will just blend into the previous ~2.5 centuries of 2% GDP growth, why self driving took so long to crack, and what he sees as the future of education. It was a pleasure chatting with him.
𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊𝐒
* Transcript: https://dwarkesh.substack.com/p/andrej-karpathy
* Apple Podcasts: https://podcasts.apple.com/us/podcast/andrej-karpathy-agi-is-still-a-decade-away/id1516093381?i=1000732326311
* Spotify: https://open.spotify.com/episode/3iIYVmmhXwh3fOumypWVpC?si=33d37708b2b44e2f
𝐒𝐏𝐎𝐍𝐒𝐎𝐑𝐒
* Labelbox helps you get data that is more detailed, more accurate, and higher signal than you could get by default, no matter your domain or training paradigm. Reach out today at https://labelbox.com/dwarkesh
* Mercury helps you run your business better. It’s the banking platform we use for the podcast — we love that we can see our accounts, cash flows, AR, and AP all in one place. Apply online in minutes at https://mercury.com
* Google’s Veo 3.1 update is a notable improvement to an already great model. Veo 3.1’s generations are more coherent and the audio is even higher-quality. If you have a Google AI Pro or Ultra plan, you can try it in Gemini today by visiting https://gemini.google
To sponsor a future episode, visit https://dwarkesh.com/advertise
𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒
00:00:00 – AGI is still a decade away
00:30:33 – LLM cognitive deficits
00:40:53 – RL is terrible
00:50:26 – How do humans learn?
01:07:13 – AGI will blend into 2% GDP growth
01:18:24 – ASI
01:33:38 – Evolution of intelligence & culture
01:43:43 - Why self driving took so long
01:57:08 - Future of education
[Source](https://www.youtube.com/watch?v=lXUZvyajciY)
Source quotes
[40:53] Andrej Karpathy: Reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before it is much worse because previously we were just imitating people, so it has all these issues.
[7:54] Andrej Karpathy: In my post, I said we're not building animals. We're building ghosts or spirits or whatever people want to call it, because we're not doing training by evolution. We're doing training by imitation of humans and the data that they've put on the Internet.
[103:43] Andrej Karpathy: When you get a demo and something works 90% of the time, that's just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine.
[55:16] Andrej Karpathy: 因为我们不擅长死记硬背，我们才被迫在更普遍的意义上去寻找规律。相比之下，大语言模型极其擅长记忆。
[112:13] Andrej Karpathy: Bits are a million times easier than anything that touches the physical world.
[53:50] Andrej Karpathy: 如果你在脑海中不断自我演绎，然后关注它，你就是在用自己的样本进行训练，你在用自己的合成数据进行训练。如果你这么做太久，你就会偏离正轨，严重崩塌。你在生活中必须不断去寻找熵。
[122:35] Andrej Karpathy: In my mind, education is the very difficult technical process of building ramps to knowledge.
[138:14] Andrej Karpathy: So micrograd, these 100 lines of Python, are everything you need to understand how neural networks train. Everything else is just efficiency.
Source quotes
[134:32] Andrej Karpathy: Early school education is not about accumulating knowledge or memory for tasks later in the industry. It's about booting up a brain.
[7:54] Andrej Karpathy: We're not building animals. We're building ghosts or spirits or whatever people want to call it, because we're not doing training by evolution. We're doing training by imitation of humans and the data that they've put on the Internet.
[112:13] Andrej Karpathy: Bits are a million times easier than anything that touches the physical world.
[128:46] Andrej Karpathy: I often say that pre-AGI education is useful. Post-AGI education is fun. In a similar way, people go to the gym today.
[55:16] Andrej Karpathy: We're not actually that good at memorization, which is actually a feature. Because we're not that good at memorization, we're forced to find patterns in a more general sense.
[17:29] Andrej Karpathy: I do think that anything that's in the weights, it's a hazy recollection of what you read a year ago. Anything that you give it as a context at test time is directly in the working memory.
[122:04] Andrej Karpathy: In my mind, education is the very difficult technical process of building ramps to knowledge.
Original transcript
[0:48] Today I'm speaking with Andrej Karpathy.
Andrej, why do you say that this will be
[0:52] the decade of agents and not the year of agents?
First of all, thank you for having me here. I'm
[0:58] excited to be here. The quote you've just
mentioned, "It's the decade of agents,"
[1:03] is actually a reaction to a pre-existing quote.
I'm not actually sure who said this but they were
[1:09] alluding to this being the year of agents with
respect to LLMs and how they were going to evolve.
[1:16] I was triggered by that because there's some
over-prediction going on in the industry.
[1:21] In my mind, this is more accurately
described as the decade of agents.
[1:25] We have some very early agents that
are extremely impressive and that I
[1:28] use daily—Claude and Codex and so on—but I
still feel there's so much work to be done.
[1:35] My reaction is we'll be working
with these things for a decade.
[1:38] They're going to get better,
and it's going to be wonderful.
[1:42] I was just reacting to the
timelines of the implication.
[1:46] What do you think will take a decade to
accomplish? What are the bottlenecks?
[1:51] Actually making it work. When you're talking
about an agent, or what the labs have in mind
[1:56] and maybe what I have in mind as well, you
should think of it almost like an employee or
[1:59] an intern that you would hire to work with you.
For example, you work with some employees here.
[2:04] When would you prefer to have an agent like
Claude or Codex do that work? Currently,
[2:08] of course they can't. What would it
take for them to be able to do that?
[2:11] Why don't you do it today?
The reason you don't do it
[2:13] today is because they just don't work.
They don't have enough intelligence,
[2:17] they're not multimodal enough, they
can't do computer use and all this stuff.
[2:20] They don't do a lot of the things you've
alluded to earlier. They don't have
[2:24] continual learning. You can't just tell
them something and they'll remember it.
[2:27] They're cognitively lacking
and it's just not working.
[2:30] It will take about a decade to
work through all of those issues.
[2:32] Interesting. As a professional podcaster
and a viewer of AI from afar, it's easy
[2:42] for me to identify what's lacking: continual
learning is lacking, or multimodality is lacking.
[2:47] But I don't really have a good way
of trying to put a timeline on it.
[2:52] If somebody asks how long continual learning
will take, I have no prior about whether
[2:57] this is a project that should take 5
years, 10 years, or 50 years. Why a
[3:01] decade? Why not one year? Why not 50 years?
This is where you get into a bit of my own
[3:07] intuition, and doing a bit of an extrapolation
with respect to my own experience in the field.
[3:14] I've been in AI for almost two decades.
It's going to be 15 years or so, not that long.
[3:19] You had Richard Sutton here,
who was around for much longer.
[3:23] I do have about 15 years of experience of people
making predictions, of seeing how they turned out.
[3:28] Also I was in the industry for
a while, I was in research,
[3:30] and I've worked in the industry for a while.
[3:32] I have a general intuition
that I have left from that.
[3:37] I feel like the problems are tractable, they're
surmountable, but they're still difficult.
[3:43] If I just average it out, it
just feels like a decade to me.
[3:47] This is quite interesting. I want
to hear not only the history,
[3:50] but what people in the room felt was about to
happen at various different breakthrough moments.
[3:58] What were the ways in which their feelings were
either overly pessimistic or overly optimistic?
[4:03] Should we just go through each of them one by one?
That's a giant question because you're talking
[4:07] about 15 years of stuff that happened.
AI is so wonderful because there have been
[4:10] a number of seismic shifts where the entire
field has suddenly looked a different way.
[4:17] I've maybe lived through two or three of those.
I still think there will continue to be
[4:21] some because they come with
almost surprising regularity.
[4:25] When my career began, when I started to work on
deep learning, when I became interested in deep
[4:29] learning, this was by chance of being right next
to Geoff Hinton at the University of Toronto.
[4:34] Geoff Hinton, of course, is
the godfather figure of AI.
[4:37] He was training all these neural networks.
I thought it was incredible and interesting.
[4:40] This was not the main thing that
everyone in AI was doing by far.
[4:43] This was a niche little subject on the side.
That's maybe the first dramatic seismic shift
[4:49] that came with the AlexNet and so on.
AlexNet reoriented everyone, and everyone
[4:54] started to train neural networks, but it
was still very per-task, per specific task.
[4:59] Maybe I have an image classifier or I have a
neural machine translator or something like that.
[5:04] People became very slowly interested in agents.
People started to think, "Okay, maybe we have a
[5:10] check mark next to the visual cortex or something
like that, but what about the other parts of the
[5:14] brain, and how can we get a full agent or a
full entity that can interact in the world?"
[5:19] The Atari deep reinforcement learning shift
in 2013 or so was part of that early effort
[5:26] of agents, in my mind, because it was an
attempt to try to get agents that not just
[5:30] perceive the world, but also take actions and
interact and get rewards from environments.
[5:34] At the time, this was Atari games.
I feel that was a misstep.
[5:39] It was a misstep that even the early OpenAI that
I was a part of adopted because at that time,
[5:45] the zeitgeist was reinforcement learning
environments, games, game playing,
[5:50] beat games, get lots of different types of
games, and OpenAI was doing a lot of that.
[5:54] That was another prominent part of AI where maybe
for two or three or four years, everyone was doing
[6:01] reinforcement learning on games.
That was all a bit of a misstep.
[6:06] What I was trying to do at OpenAI is
I was always a bit suspicious of games
[6:10] as being this thing that would lead to AGI.
Because in my mind, you want something like
[6:13] an accountant or something that's
interacting with the real world.
[6:17] I just didn't see how games add up to it.
My project at OpenAI, for example, was within the
[6:23] scope of the Universe project, on an agent that
was using keyboard and mouse to operate web pages.
[6:30] I really wanted to have something that
interacts with the actual digital world
[6:33] that can do knowledge work.
It just so turns out that this
[6:37] was extremely early, way too early, so early
that we shouldn't have been working on that.
[6:42] Because if you're just stumbling your way around
and keyboard mashing and mouse clicking and trying
[6:47] to get rewards in these environments, your
reward is too sparse and you just won't learn.
[6:52] You're going to burn a forest
computing, and you're never
[6:54] going to get something off the ground.
What you're missing is this power of
[6:58] representation in the neural network.
For example, today people are training
[7:02] those computer-using agents, but they're
doing it on top of a large language model.
[7:05] You have to get the language model first,
you have to get the representations first,
[7:08] and you have to do that by all the
pre-training and all the LLM stuff.
[7:11] I feel maybe loosely speaking, people
kept trying to get the full thing too
[7:17] early a few times, where people really try
to go after agents too early, I would say.
[7:22] That was Atari and Universe
and even my own experience.
[7:26] You actually have to do some things
first before you get to those agents.
[7:30] Now the agents are a lot more competent, but maybe
we're still missing some parts of that stack.
[7:36] I would say those are the three major
buckets of what people were doing:
[7:40] training neural nets per-tasks,
trying the first round of agents,
[7:44] and then maybe the LLMs and seeking the
representation power of the neural networks
[7:48] before you tack on everything else on top.
Interesting. If I were to steelman the Sutton
[7:54] perspective, it would be that humans
can just take on everything at once,
[7:57] or even animals can take on everything at once.
Animals are maybe a better example because they
[8:01] don't even have the scaffold of language.
They just get thrown out into the world,
[8:05] and they just have to make sense
of everything without any labels.
[8:10] The vision for AGI then should just be
something which looks at sensory data,
[8:14] looks at the computer screen, and it just
figures out what's going on from scratch.
[8:18] If a human were put in a similar situation and
had to be trained from scratch… This is like a
[8:22] human growing up or an animal growing up.
Why shouldn't that be the vision for AI,
[8:26] rather than this thing where we're
doing millions of years of training?
[8:29] That's a really good question. Sutton was
on your podcast and I saw the podcast and I
[8:35] had a write-up about that podcast that
gets into a bit of how I see things.
[8:41] I'm very careful to make analogies to
animals because they came about by a
[8:46] very different optimization process.
Animals are evolved, and they come
[8:49] with a huge amount of hardware that's built in.
For example, my example in the post was the zebra.
[8:55] A zebra gets born, and a few minutes later
it's running around and following its mother.
[8:59] That's an extremely complicated thing to do.
That's not reinforcement learning. That's
[9:03] something that's baked in. Evolution obviously
has some way of encoding the weights of our
[9:08] neural nets in ATCGs, and I have no idea
how that works, but it apparently works.
[9:14] Brains just came from a very different process,
and I'm very hesitant to take inspiration from it
[9:20] because we're not actually running that process.
In my post, I said we're not building animals.
[9:25] We're building ghosts or spirits or
whatever people want to call it, because
[9:31] we're not doing training by evolution.
We're doing training by imitation of humans
[9:37] and the data that they've put on the Internet.
You end up with these ethereal spirit entities
[9:42] because they're fully digital
and they're mimicking humans.
[9:45] It's a different kind of intelligence.
If you imagine a space of intelligences,
[9:48] we're starting off at a different point almost.
We're not really building animals. But it's also
[9:53] possible to make them a bit more animal-like
over time, and I think we should be doing that.
[9:58] One more point. I do feel Sutton has a very...
His framework is, "We want to build animals."
[10:04] I think that would be wonderful if we can
get that to work. That would be amazing.
[10:07] If there were a single algorithm that you
can just run on the Internet and it learns
[10:13] everything, that would be incredible.
I'm not sure that it exists and that's
[10:18] certainly not what animals do, because
animals have this outer loop of evolution.
[10:24] A lot of what looks like learning is
more like maturation of the brain.
[10:28] I think there's very little
reinforcement learning for animals.
[10:32] A lot of the reinforcement learning is more
like motor tasks; it's not intelligence tasks.
[10:37] So I actually kind of think humans
don’t really use RL, roughly speaking.
[10:41] Can you repeat the last sentence?
A lot of that intelligence is
[10:42] not motor task…it's what, sorry?
A lot of the reinforcement learning, in my
[10:45] perspective, would be things that are a lot more
motor-like, simple tasks like throwing a hoop.
[10:53] But I don't think that humans use reinforcement
learning for a lot of intelligence tasks
[10:57] like problem-solving and so on.
That doesn't mean we shouldn't
[11:01] do that for research, but I just feel
like that's what animals do or don't.
[11:05] I'm going to take a second to digest that
because there are a lot of different ideas.
[11:09] Here’s one clarifying question I can
ask to understand the perspective.
[11:15] You suggest that evolution is doing
the kind of thing that pre-training
[11:18] does in the sense of building something
which can then understand the world.
[11:24] The difference is that evolution
has to be titrated in the case
[11:28] of humans through three gigabytes of DNA.
That's very unlike the weights of a model.
[11:37] Literally, the weights of the model are
a brain, which obviously does not exist
[11:42] in the sperm and the egg.
So it has to be grown.
[11:46] Also, the information for every single
synapse in the brain simply cannot exist
[11:50] in the three gigabytes that exist in the DNA.
Evolution seems closer to finding the algorithm
[11:55] which then does the lifetime learning.
Now, maybe the lifetime learning is
[12:00] not analogous to RL, to your point.
Is that compatible with the thing you
[12:04] were saying, or would you disagree with that?
I think so. I would agree with you that there's
[12:06] some miraculous compression
going on because obviously,
[12:09] the weights of the neural net are not stored
in ATCGs. There's some dramatic compression.
[12:14] There are some learning algorithms encoded that
take over and do some of the learning online.
[12:18] I definitely agree with you on that.
I would say I'm a lot more practically minded.
[12:23] I don't come at it from the
perspective of, let's build animals.
[12:26] I come from it from the perspective
of, let's build useful things.
[12:28] I have a hard hat on, and I'm just
observing that we're not going to do
[12:31] evolution, because I don't know how to do that.
But it does turn out we can build these ghosts,
[12:36] spirit-like entities, by imitating internet
documents. This works. It's a way to bring you
[12:43] up to something that has a lot of built-in
knowledge and intelligence in some way,
[12:48] similar to maybe what evolution has done.
That's why I call pre-training
[12:51] this crappy evolution.
It's the practically possible version
[12:56] with our technology and what we have available
to us to get to a starting point where we can do
[13:01] things like reinforcement learning and so on.
Just to steelman the other perspective,
[13:05] after doing this Sutton interview and thinking
about it a bit, he has an important point here.
[13:09] Evolution does not give us the knowledge, really.
It gives us the algorithm to find the knowledge,
[13:14] and that seems different from pre-training.
Perhaps the perspective is that pre-training helps
[13:19] build the kind of entity which can learn better.
It teaches meta-learning, and therefore
[13:23] it is similar to finding an algorithm.
But if it's "Evolution gives us knowledge,
[13:28] pre-training gives us knowledge,"
that analogy seems to break down.
[13:31] It's subtle and I think you're right to
push back on it, but basically the thing
[13:35] that pre-training is doing, you're getting
the next-token predictor over the internet,
[13:38] and you're training that into a neural net.
It's doing two things that are unrelated.
[13:43] Number one, it's picking up all
this knowledge, as I call it.
[13:46] Number two, it's actually becoming intelligent.
By observing the algorithmic patterns in the
[13:51] internet, it boots up all these little circuits
and algorithms inside the neural net to do things
[13:55] like in-context learning and all this stuff.
You don't need or want the knowledge.
[14:00] I think that's probably holding back the neural
networks overall because it's getting them to rely
[14:05] on the knowledge a little too much sometimes.
For example, I feel agents, one thing they're
[14:09] not very good at, is going off the data
manifold of what exists on the internet.
[14:13] If they had less knowledge or less
memory, maybe they would be better.
[14:17] What I think we have to do going forward—and
this would be part of the research paradigms—is
[14:23] figure out ways to remove some of the knowledge
and to keep what I call this cognitive core.
[14:28] It's this intelligent entity that is stripped from
knowledge but contains the algorithms and contains
[14:34] the magic of intelligence and problem-solving
and the strategies of it and all this stuff.
[14:39] There's so much interesting stuff there. Let's
start with in-context learning. This is an
[14:44] obvious point, but I think it's worth just
saying it explicitly and meditating on it.
[14:48] The situation in which these models seem the most
intelligent—in which I talk to them and I'm like,
[14:53] "Wow, there's really something on the other end
that's responding to me thinking about things—is
[14:58] if it makes a mistake it's like, "Oh wait, that's
the wrong way to think about it. I'm backing up."
[15:01] All that is happening in context.
That's where I feel like the real
[15:04] intelligence is that you can visibly see.
That in-context learning process is
[15:10] developed by gradient descent on pre-training.
It spontaneously meta-learns in-context learning,
[15:16] but the in-context learning itself is not
gradient descent, in the same way that our
[15:21] lifetime intelligence as humans to be able
to do things is conditioned by evolution
[15:26] but our learning during our lifetime is
happening through some other process.
[15:30] I don't fully agree with that, but
you should continue your thought.
[15:34] Well, I'm very curious to understand
how that analogy breaks down.
[15:36] I'm hesitant to say that in-context
learning is not doing gradient descent.
[15:41] It's not doing explicit gradient descent.
In-context learning is pattern completion
[15:47] within a token window.
It just turns out that there's
[15:50] a huge amount of patterns on the internet.
You're right, the model learns to complete
[15:53] the pattern, and that's inside the weights.
The weights of the neural network are trying
[15:58] to discover patterns and complete the pattern.
There's some adaptation that happens inside
[16:02] the neural network, which is magical
and just falls out from the internet
[16:06] just because there's a lot of patterns.
I will say that there have been some papers
[16:10] that I thought were interesting that look at
the mechanisms behind in-context learning.
[16:14] I do think it's possible that in-context
learning runs a small gradient descent loop
[16:17] internally in the layers of the neural network.
I recall one paper in particular where they were
[16:22] doing linear regression using in-context learning.
Your inputs into the neural network are XY pairs,
[16:30] XY, XY, XY that happen to be on the line.
Then you do X and you expect Y.
[16:35] The neural network, when you train it
in this way, does linear regression.
[16:41] Normally when you would run linear regression, you
have a small gradient descent optimizer that looks
[16:46] at XY, looks at an error, calculates the gradient
of the weights and does the update a few times.
[16:50] It just turns out that when they looked at the
weights of that in-context learning algorithm,
[16:54] they found some analogies to
gradient descent mechanics.
[16:59] In fact, I think the paper was even stronger
because they hardcoded the weights of a neural
[17:03] network to do gradient descent through
attention and all the internals of the
[17:10] neural network. That's just my only pushback.
Who knows how in-context learning works,
[17:14] but I think that it's probably doing a bit of some
funky gradient descent internally. I think that
[17:19] that's possible. I was only pushing back on your
saying that it's not doing in-context learning.
[17:24] Who knows what it's doing, but it's probably maybe
doing something similar to it, but we don't know.
[17:28] So then it's worth thinking okay, if
in-context learning and pre-training
[17:34] are both implementing something like gradient
descent, why does it feel like with in-context
[17:39] learning we're getting to this continual
learning, real intelligence-like thing?
[17:44] Whereas you don't get the analogous feeling just
from pre-training. You could argue that. If it's
[17:49] the same algorithm, what could be different?
One way you could think about it is,
[17:53] how much information does the model store
per information it receives from training?
[18:00] If you look at pre-training, if
you look at Llama 3 for example,
[18:03] I think it's trained on 15 trillion tokens.
If you look at the 70B model, that would
[18:10] be the equivalent of 0.07 bits per
token that it sees in pre-training,
[18:15] in terms of the information in the weights
of the model compared to the tokens it reads.
[18:18] Whereas if you look at the KV cache
and how it grows per additional token
[18:22] in in-context learning, it's like 320 kilobytes.
So that's a 35 million-fold difference in how much
[18:29] information per token is assimilated by the model.
I wonder if that's relevant at all.
[18:34] I kind of agree. The way I usually put this is
that anything that happens during the training of
[18:39] the neural network, the knowledge is only a hazy
recollection of what happened in training time.
[18:45] That's because the compression is dramatic.
You're taking 15 trillion tokens and you're
[18:48] compressing it to just your final neural
network of a few billion parameters.
[18:51] Obviously it's a massive
amount of compression going on.
[18:54] So I refer to it as a hazy
recollection of the internet documents.
[18:57] Whereas anything that happens in the
context window of the neural network—you're
[19:00] plugging in all the tokens and building up
all those KV cache representations—is very
[19:04] directly accessible to the neural net.
So I compare the KV cache and the stuff
[19:08] that happens at test time to
more like a working memory.
[19:11] All the stuff that's in the context window is
very directly accessible to the neural net.
[19:16] There's always these almost surprising
analogies between LLMs and humans.
[19:21] I find them surprising because we're not
trying to build a human brain directly.
[19:25] We're just finding that this
works and we're doing it.
[19:27] But I do think that anything
that's in the weights, it's a
[19:30] hazy recollection of what you read a year ago.
Anything that you give it as a context at test
[19:36] time is directly in the working memory.
That's a very powerful analogy to
[19:40] think through things.
When you, for example,
[19:42] go to an LLM and you ask it about some book
and what happened in it, like Nick Lane's
[19:46] book or something like that, the LLM will often
give you some stuff which is roughly correct.
[19:49] But if you give it the full chapter and ask it
questions, you're going to get much better results
[19:53] because it's now loaded in the
working memory of the model.
[19:56] So a very long way of saying
I agree and that's why.
[20:00] Stepping back, what is the part
about human intelligence that we
[20:03] have most failed to replicate with these models?
[20:12] Just a lot of it. So maybe one way to think
about it, I don't know if this is the best way,
[20:17] but I almost feel like — again, making these
analogies imperfect as they are — we've stumbled
[20:22] by with the transformer neural network,
which is extremely powerful, very general.
[20:27] You can train transformers on audio, or
video, or text, or whatever you want,
[20:31] and it just learns patterns and they're
very powerful, and it works really well.
[20:35] That to me almost indicates that this
is some piece of cortical tissue.
[20:38] It's something like that, because the
cortex is famously very plastic as well.
[20:42] You can rewire parts of brains.
There were the slightly gruesome experiments
[20:48] with rewiring the visual cortex to the auditory
cortex, and this animal learned fine, et cetera.
[20:54] So I think that this is cortical tissue.
I think when we're doing reasoning and
[20:58] planning inside the neural networks, doing
reasoning traces for thinking models,
[21:04] that's kind of like the prefrontal cortex.
Maybe those are like little checkmarks,
[21:11] but I still think there are many brain
parts and nuclei that are not explored.
[21:15] For example, there's a basal ganglia doing a
bit of reinforcement learning when we fine-tune
[21:18] the models on reinforcement learning. But where's
the hippocampus? Not obvious what that would be.
[21:23] Some parts are probably not important.
Maybe the cerebellum is not important
[21:26] to cognition, its thoughts, so
maybe we can skip some of it.
[21:29] But I still think there's, for example, the
amygdala, all the emotions and instincts.
[21:33] There's probably a bunch of other nuclei
in the brain that are very ancient that
[21:36] I don't think we've really replicated.
I don't know that we should be pursuing the
[21:41] building of an analog of a human brain.
I'm an engineer mostly at heart.
[21:48] Maybe another way to answer the question is that
you're not going to hire this thing as an intern.
[21:52] It's missing a lot of it because it comes with
a lot of these cognitive deficits that we all
[21:55] intuitively feel when we talk to the models.
So it's not fully there yet.
[22:00] You can look at it as not all the
brain parts are checked off yet.
[22:04] This is maybe relevant to the question of thinking
about how fast these issues will be solved.
[22:10] Sometimes people will say
about continual learning,
[22:13] "Look, you could easily replicate this capability.
Just as in-context learning emerged spontaneously
[22:19] as a result of pre-training, continual
learning over longer horizons will emerge
[22:24] spontaneously if the model is incentivized to
recollect information over longer horizons,
[22:30] or horizons longer than one session."
So if there's some outer loop RL which has
[22:39] many sessions within that outer loop, then this
continual learning where it fine-tunes itself,
[22:46] or it writes to an external memory or
something, will just emerge spontaneously.
[22:49] Do you think things like that are plausible?
I just don't have a prior over
[22:53] how plausible that is.
How likely is that to happen?
[22:55] I don't know that I fully resonate with that.
These models, when you boot them up and they have
[22:59] zero tokens in the window, they're always
restarting from scratch where they were.
[23:03] So I don't know in that
worldview what it looks like.
[23:09] Maybe making some analogies to humans—just because
I think it's roughly concrete and interesting to
[23:13] think through—I feel like when I'm awake, I'm
building up a context window of stuff that's
[23:17] happening during the day.
But when I go to sleep,
[23:19] something magical happens where I don't
think that context window stays around.
[23:23] There's some process of distillation
into the weights of my brain.
[23:27] This happens during sleep and all this stuff.
We don't have an equivalent
[23:30] of that in large language models.
That's to me more adjacent to when you talk
[23:35] about continual learning and so on as absent.
These models don't really have a distillation
[23:40] phase of taking what happened, analyzing it
obsessively, thinking through it, doing some
[23:47] synthetic data generation process and distilling
it back into the weights, and maybe having
[23:51] a specific neural net per person. Maybe it's
a LoRA. It's not a full-weight neural network.
[24:01] It's just some small sparse subset
of the weights that are changed.
[24:05] But we do want to create ways of creating
these individuals that have very long context.
[24:10] It's not only remaining in the context window
because the context windows grow very, very long.
[24:14] Maybe we have some very elaborate,
sparse attention over it.
[24:17] But I still think that humans obviously have some
process for distilling some of that knowledge
[24:22] into the weights. We're missing it. I do also
think that humans have some very elaborate,
[24:27] sparse attention scheme, which I think
we're starting to see some early hints of.
[24:33] DeepSeek v3.2 just came out and I saw that they
have sparse attention as an example, and this is
[24:38] one way to have very, very long context windows.
So I feel like we are redoing a lot of the
[24:44] cognitive tricks that evolution came up
with through a very different process.
[24:47] But we're going to converge on a
similar architecture cognitively.
[24:51] In 10 years, do you think it'll still be something
like a transformer, but with much more modified
[24:55] attention and more sparse MLPs and so forth?
The way I like to think about it is
[25:00] translation invariance in time.
So 10 years ago, where were we? 2015.
[25:05] In 2015, we had convolutional neural networks
primarily, residual networks just came out.
[25:10] So remarkably similar, I guess, but quite a
bit different still. The transformer was not
[25:14] around. All these more modern tweaks
on the transformer were not around.
[25:21] Maybe some of the things that we can bet on, I
think in 10 years by translational equivariance,
[25:27] is that we're still training giant neural
networks with a forward backward pass and
[25:31] update through gradient descent,
but maybe it looks a bit different,
[25:36] and it's just that everything is much bigger.
Recently I went back all the way to 1989 which
[25:42] was a fun exercise for me, a few years ago,
because I was reproducing Yann LeCun's 1989
[25:48] convolutional network, which was the first neural
network I'm aware of trained via gradient descent,
[25:52] like modern neural network trained
gradient descent on digit recognition.
[25:57] I was just interested in
how I could modernize this.
[26:00] How much of this is algorithms?
How much of this is data?
[26:01] How much of this progress is compute and systems?
I was able to very quickly halve the learning
[26:06] just by time traveling by 33 years.
So if I time travel by algorithms 33 years,
[26:13] I could adjust what Yann LeCun did
in 1989, and I could halve the error.
[26:18] But to get further gains, I
had to add a lot more data,
[26:21] I had to 10x the training set, and then I
had to add more computational optimizations.
[26:25] I had to train for much longer with dropout
and other regularization techniques.
[26:30] So all these things have
to improve simultaneously.
[26:35] We're probably going to have a lot more
data, we're probably going to have a lot
[26:37] better hardware, probably going to have a lot
better kernels and software, we're probably
[26:40] going to have better algorithms.
All of those, it's almost like
[26:43] no one of them is winning too much.
All of them are surprisingly equal.
[26:48] This has been the trend for a while.
So to answer your question, I expect differences
[26:54] algorithmically to what's happening today.
But I do also expect that some of the
[26:58] things that have stuck around for a very
long time will probably still be there.
[27:01] It's probably still a giant neural network trained
with gradient descent. That would be my guess.
[27:05] It's surprising that all of those things
together only halved the error, 30 years
[27:13] of progress…. Maybe half is a lot. Because if
you halve the error, that actually means that…
[27:18] Half is a lot. But I guess what was shocking to me
is everything needs to improve across the board:
[27:24] architecture, optimizer, loss function.
It also has improved across the board forever.
[27:28] So I expect all those
changes to be alive and well.
[27:31] Yeah. I was about to ask you a very
similar question about nanochat.
[27:35] Since you just coded it up recently,
every single step in the process of
[27:40] building a chatbot is fresh in your RAM.
I'm curious if you had similar thoughts about,
[27:46] "Oh, there was no one thing that was
relevant to going from GPT-2 to nanochat."
[27:52] What are some surprising
takeaways from the experience?
[27:55] Of building nanochat? So nanochat
is a repository I released.
[27:59] Was it yesterday or the day
before? I can't remember.
[28:03] We can see the sleep
deprivation that went into the…
[28:09] It's trying to be the simplest complete
repository that covers the whole pipeline
[28:12] end-to-end of building a ChatGPT clone.
So you have all of the steps, not just
[28:18] any individual step, which is a bunch.
I worked on all the individual steps
[28:22] in the past and released small pieces
of code that show you how that's done
[28:25] in an algorithmic sense, in simple code.
But this handles the entire pipeline.
[28:32] In terms of learning, I don't
know that I necessarily found
[28:36] something that I learned from it.
I already had in my mind how you build it.
[28:40] This is just the process of mechanically building
it and making it clean enough so that people can
[28:48] learn from it and that they find it useful.
What is the best way for somebody
[28:53] to learn from it?
Is it to just delete
[28:54] all the code and try to reimplement from
scratch, try to add modifications to it?
[28:59] That's a great question. Basically
it's about 8,000 lines of code
[29:03] that takes you through the entire pipeline.
I would probably put it on the right monitor.
[29:07] If you have two monitors, you put it on the right.
You want to build it from scratch,
[29:11] you build it from the start.
You're not allowed to copy-paste, you're allowed
[29:14] to reference, you're not allowed to copy-paste.
Maybe that's how I would do it.
[29:18] But I also think the repository
by itself is a pretty large beast.
[29:23] When you write this code, you don't go from top
to bottom, you go from chunks and you grow the
[29:27] chunks, and that information is absent.
You wouldn't know where to start.
[29:31] So it's not just a final repository that's
needed, it's the building of the repository,
[29:35] which is a complicated chunk-growing process.
So that part is not there yet.
[29:40] I would love to add that probably later this week.
It's probably a video or something like that.
[29:49] Roughly speaking, that's what I would try to do.
Build the stuff yourself, but don't allow
[29:53] yourself copy-paste.
I do think that
[29:55] there's two types of knowledge, almost.
There's the high-level surface knowledge, but when
[29:59] you build something from scratch, you're forced to
come to terms with what you don't understand and
[30:04] you don't know that you don't understand it.
It always leads to a deeper understanding.
[30:09] It's the only way to build.
If I can't build it, I don't understand it.
[30:13] That’s a Feynman quote, I believe.
I 100% have always believed this very
[30:19] strongly, because there are all these micro
things that are just not properly arranged
[30:23] and you don't really have the knowledge.
You just think you have the knowledge.
[30:25] So don't write blog posts, don't
do slides, don't do any of that.
[30:28] Build the code, arrange it, get it to work.
It's the only way to go. Otherwise,
[30:31] you're missing knowledge.
You tweeted out that coding models
[30:35] were of very little help to you in assembling
this repository. I'm curious why that was.
[30:43] I guess I built the repository over
a period of a bit more than a month.
[30:46] I would say there are three major classes
of how people interact with code right now.
[30:50] Some people completely reject all of LLMs
and they are just writing by scratch.
[30:54] This is probably not the
right thing to do anymore.
[30:58] The intermediate part, which is where I am, is
you still write a lot of things from scratch,
[31:02] but you use the autocomplete that's
available now from these models.
[31:06] So when you start writing out a little
piece of it, it will autocomplete for
[31:10] you and you can just tap through.
Most of the time it's correct,
[31:12] sometimes it's not, and you edit it.
But you're still very much the
[31:16] architect of what you're writing.
Then there's the vibe coding: "Hi,
[31:21] please implement this or that," enter, and then
let the model do it. That's the agents. I do feel
[31:28] like the agents work in very specific settings,
and I would use them in specific settings.
[31:33] But these are all tools available to you
and you have to learn what they're good at,
[31:37] what they're not good at, and when to use them.
So the agents are pretty good, for example,
[31:40] if you're doing boilerplate stuff.
Boilerplate code that's just
[31:44] copy-paste stuff, they're very good at that.
They're very good at stuff that occurs very often
[31:48] on the Internet because there are lots of examples
of it in the training sets of these models.
[31:55] There are features of things where
the models will do very well.
[31:58] I would say nanochat is not an example of
those because it's a fairly unique repository.
[32:03] There's not that much code in the way that
I've structured it. It's not boilerplate code.
[32:08] It's intellectually intense code almost, and
everything has to be very precisely arranged.
[32:13] The models have so many cognitive deficits.
One example, they kept misunderstanding the code
[32:22] because they have too much memory from
all the typical ways of doing things on
[32:26] the Internet that I just wasn't adopting.
The models, for example—I don't know if I
[32:31] want to get into the full details—but they kept
thinking I'm writing normal code, and I'm not.
[32:36] Maybe one example?
You have eight GPUs
[32:41] that are all doing forward, backwards.
The way to synchronize gradients between
[32:44] them is to use a Distributed Data Parallel
container of PyTorch, which automatically
[32:49] as you're doing the backward, it will start
communicating and synchronizing gradients.
[32:52] I didn't use DDP because I didn't want
to use it, because it's not necessary.
[32:56] I threw it out and wrote my own synchronization
routine that's inside the step of the optimizer.
[33:02] The models were trying to get me to
use the DDP container. They were very
[33:06] concerned. This gets way too technical,
but I wasn't using that container because
[33:11] I don't need it and I have a custom
implementation of something like it.
[33:14] They just couldn't internalize
that you had your own.
[33:16] They couldn't get past that. They kept trying to
mess up the style. They're way too over-defensive.
[33:23] They make all these try-catch statements.
They keep trying to make a production code base,
[33:27] and I have a bunch of assumptions
in my code, and it's okay.
[33:31] I don't need all this extra stuff in there.
So I feel like they're bloating the code base,
[33:36] bloating the complexity, they keep
misunderstanding, they're using
[33:38] deprecated APIs a bunch of times. It's a total
mess. It's just not net useful. I can go in,
[33:46] I can clean it up, but it's not net useful.
I also feel like it's annoying to have to
[33:50] type out what I want in English
because it's too much typing.
[33:54] If I just navigate to the part of the code that I
want, and I go where I know the code has to appear
[33:58] and I start typing out the first few letters,
autocomplete gets it and just gives you the code.
[34:03] This is a very high information
bandwidth to specify what you want.
[34:07] You point to the code where you want
it, you type out the first few pieces,
[34:10] and the model will complete it.
So what I mean is, these models
[34:15] are good in certain parts of the stack.
There are two examples where I use the
[34:22] models that I think are illustrative.
One was when I generated the report.
[34:26] That's more boilerplate-y, so I
partially vibe-coded some of that stuff.
[34:30] That was fine because it's not
mission-critical stuff, and it works fine.
[34:34] The other part is when I was
rewriting the tokenizer in Rust.
[34:37] I'm not as good at Rust
because I'm fairly new to Rust.
[34:41] So there's a bit of vibe coding going on
when I was writing some of the Rust code.
[34:45] But I had a Python implementation that I
fully understand, and I'm just making sure
[34:49] I'm making a more efficient version of it, and
I have tests so I feel safer doing that stuff.
[34:56] They increase accessibility to languages or
paradigms that you might not be as familiar with.
[35:02] I think they're very helpful there as well.
There's a ton of Rust code out there,
[35:06] the models are pretty good at it.
I happen to not know that much about it,
[35:09] so the models are very useful there.
The reason this question is so interesting
[35:12] is because the main story people
have about AI exploding and getting
[35:19] to superintelligence pretty rapidly is AI
automating AI engineering and AI research.
[35:25] They'll look at the fact that you can have
Claude Code and make entire applications,
[35:27] CRUD applications, from scratch and think, "If
you had this same capability inside of OpenAI
[35:32] and DeepMind and everything, just imagine
a thousand of you or a million of you in
[35:38] parallel, finding little architectural tweaks."
It's quite interesting to hear you say that this
[35:43] is the thing they're asymmetrically worse at.
It's quite relevant to forecasting whether
[35:48] the AI 2027-type explosion is
likely to happen anytime soon.
[35:53] That's a good way of putting it, and you're
getting at why my timelines are a bit longer.
[35:58] You're right. They're not very good at code
that has never been written before, maybe it's
[36:04] one way to put it, which is what we're trying
to achieve when we're building these models.
[36:07] Very naive question, but the architectural
tweaks that you're adding to nanochat,
[36:14] they're in a paper somewhere, right?
They might even be in a repo somewhere.
[36:20] Is it surprising that they aren't able to
integrate that into whenever you're like,
[36:25] "Add RoPE embeddings" or something,
they do that in the wrong way?
[36:29] It's tough. They know, but they don't fully know.
They don't know how to fully integrate it into
[36:34] the repo and your style and your code and
your place, and some of the custom things
[36:37] that you're doing and how it fits with
all the assumptions of the repository.
[36:42] They do have some knowledge, but they
haven't gotten to the place where they
[36:46] can integrate it and make sense of it.
A lot of the stuff continues to improve.
[36:54] Currently, the state-of-the-art
model that I go to is the GPT-5 Pro,
[36:57] and that's a very powerful model.
If I have 20 minutes,
[37:01] I will copy-paste my entire repo and I go to
GPT-5 Pro, the oracle, for some questions.
[37:06] Often it's not too bad and surprisingly
good compared to what existed a year ago.
[37:11] Overall, the models are not there.
I feel like the industry is making too
[37:20] big of a jump and is trying to pretend like this
is amazing, and it's not. It's slop. They're not
[37:27] coming to terms with it, and maybe they're
trying to fundraise or something like that.
[37:29] I'm not sure what's going on, but we're
at this intermediate stage. The models are
[37:33] amazing. They still need a lot of work.
For now, autocomplete is my sweet spot.
[37:38] But sometimes, for some types of
code, I will go to an LLM agent.
[37:43] Here's another reason this is really interesting.
Through the history of programming, there have
[37:48] been many productivity improvements—compilers,
linting, better programming languages—which
[37:55] have increased programmer productivity
but have not led to an explosion.
[38:00] That sounds very much like the
autocomplete tab, and this other
[38:04] category is just automation of the programmer.
It's interesting you're seeing more in the
[38:09] category of the historical analogies
of better compilers or something.
[38:13] Maybe this gets to one other thought.
I have a hard time differentiating where
[38:19] AI begins and stops because I see
AI as fundamentally an extension of
[38:23] computing in a pretty fundamental way.
I see a continuum of this recursive
[38:28] self-improvement or speeding up programmers
all the way from the beginning: code editors,
[38:37] syntax highlighting, or checking even of
the types, like data type checking—all
[38:44] these tools that we've built for
each other. Even search engines.
[38:48] Why aren't search engines part of AI? Ranking
is AI. At some point, Google, even early on,
[38:55] was thinking of themselves as an AI company doing
Google Search engine, which is totally fair.
[38:59] I see it as a lot more of a continuum than other
people do, and it's hard for me to draw the line.
[39:04] I feel like we're now getting a much
better autocomplete, and now we're also
[39:07] getting some agents which are these loopy
things, but they go off-rails sometimes.
[39:13] What's going on is that the human is progressively
doing a bit less and less of the low-level stuff.
[39:18] We're not writing the assembly
code because we have compilers.
[39:20] Compilers will take my high-level
language in C and write the assembly code.
[39:23] We're abstracting ourselves very, very slowly.
There's this what I call "autonomy slider," where
[39:28] more and more stuff is automated—of the stuff that
can be automated at any point in time—and we're
[39:32] doing a bit less and less and raising ourselves
in the layer of abstraction over the automation.
[40:53] Let's talk about RL a bit.
You tweeted some very
[40:56] interesting things about this.
Conceptually, how should we think about
[41:00] the way that humans are able to build a rich world
model just from interacting with our environment,
[41:07] and in ways that seem almost irrespective of
the final reward at the end of the episode?
[41:13] If somebody is starting a business, and at
the end of 10 years, she finds out whether
[41:17] the business succeeded or failed, we say that
she's earned a bunch of wisdom and experience.
[41:22] But it's not because the log probs of every
single thing that happened over the last 10
[41:25] years are up-weighted or down-weighted.
Something much more deliberate and
[41:29] rich is happening.
What is the ML analogy, and how does that
[41:33] compare to what we're doing with LLMs right now?
Maybe the way I would put it is that humans don't
[41:36] use reinforcement learning, as I said.
I think they do something different.
[41:42] Reinforcement learning is a lot worse than I
think the average person thinks. Reinforcement
[41:48] learning is terrible. It just so happens
that everything that we had before it is
[41:53] much worse because previously we were just
imitating people, so it has all these issues.
[41:59] In reinforcement learning, say you're solving
a math problem, because it's very simple.
[42:04] You're given a math problem and
you're trying to find the solution.
[42:08] In reinforcement learning, you will
try lots of things in parallel first.
[42:13] You're given a problem, you try hundreds of
different attempts. These attempts can be complex.
[42:18] They can be like, "Oh, let me try this, let me try
that, this didn't work, that didn't work," etc.
[42:22] Then maybe you get an answer.
Now you check the back of the book and you see,
[42:25] "Okay, the correct answer is this."
You can see that this one, this one,
[42:30] and that one got the correct answer,
but these other 97 of them didn't.
[42:33] Literally what reinforcement learning does is it
goes to the ones that worked really well and every
[42:37] single thing you did along the way, every single
token gets upweighted like, "Do more of this."
[42:42] The problem with that is people will say
that your estimator has high variance,
[42:46] but it's just noisy. It's noisy. It almost assumes
that every single little piece of the solution
[42:53] that you made that arrived at the right answer
was the correct thing to do, which is not true.
[42:56] You may have gone down the wrong alleys
until you arrived at the right solution.
[43:00] Every single one of those incorrect things you
did, as long as you got to the correct solution,
[43:04] will be upweighted as, "Do more of this."
It's terrible. It's noise. You've done all
[43:08] this work only to find, at the end, you get a
single number of like, "Oh, you did correct."
[43:14] Based on that, you weigh that entire
trajectory as like, upweight or downweight.
[43:19] The way I like to put it is you're
sucking supervision through a straw.
[43:22] You've done all this work that
could be a minute of rollout,
[43:24] and you're sucking the bits of supervision of the
final reward signal through a straw and you're
[43:33] broadcasting that across the entire trajectory
and using that to upweight or downweight that
[43:37] trajectory. It's just stupid and
crazy. A human would never do this.
[43:39] Number one, a human would
never do hundreds of rollouts.
[43:43] Number two, when a person finds a solution,
they will have a pretty complicated process
[43:47] of review of, "Okay, I think these parts I
did well, these parts I did not do that well.
[43:51] I should probably do this or that."
They think through things. There's
[43:55] nothing in current LLMs that does this.
There's no equivalent of it. But I do see
[44:00] papers popping out that are trying to do this
because it's obvious to everyone in the field.
[44:05] The first imitation learning, by the way, was
extremely surprising and miraculous and amazing,
[44:09] that we can fine-tune by imitation on humans.
That was incredible. Because in the beginning,
[44:14] all we had was base models. Base models
are autocomplete. It wasn't obvious to
[44:18] me at the time, and I had to learn this.
The paper that blew my mind was InstructGPT,
[44:24] because it pointed out that you can take
the pretrained model, which is autocomplete,
[44:28] and if you just fine-tune it on text that looks
like conversations, the model will very rapidly
[44:32] adapt to become very conversational, and it
keeps all the knowledge from pre-training.
[44:36] This blew my mind because I didn't understand
that stylistically, it can adjust so quickly
[44:41] and become an assistant to a user through just
a few loops of fine-tuning on that kind of data.
[44:46] It was very miraculous to me that that worked.
So incredible. That was two to three years of
[44:51] work. Now came RL. And RL allows you to do a bit
better than just imitation learning because you
[44:58] can have these reward functions and you
can hill-climb on the reward functions.
[45:02] Some problems have just correct answers, you
can hill-climb on that without getting expert
[45:06] trajectories to imitate. So that's amazing. The
model can also discover solutions that a human
[45:10] might never come up with. This is incredible.
Yet, it's still stupid. We need more. I saw a
[45:19] paper from Google yesterday that tried to
have this reflect & review idea in mind.
[45:25] Was it the memory bank paper or something? I don't
know. I've seen a few papers along these lines.
[45:30] So I expect there to be some major update to how
we do algorithms for LLMs coming in that realm.
[45:37] I think we need three or four or
five more, something like that.
[45:42] You're so good at coming up with evocative
phrases. "Sucking supervision through a
[45:47] straw." It's so good. You're saying the problem
with outcome-based reward is that you have this
[45:56] huge trajectory, and then at the end, you're
trying to learn every single possible thing
[46:01] about what you should do and what you should
learn about the world from that one final bit.
[46:07] Given the fact that this is obvious, why hasn't
process-based supervision as an alternative been
[46:11] a successful way to make models more capable?
What has been preventing us from using
[46:15] this alternative paradigm?
Process-based supervision just
[46:18] refers to the fact that we're not going to
have a reward function only at the very end.
[46:21] After you've done 10 minutes of work, I'm not
going to tell you you did well or not well.
[46:24] I'm going to tell you at every single
step of the way how well you're doing.
[46:28] The reason we don't have that is
it's tricky how you do that properly.
[46:32] You have partial solutions and you
don't know how to assign credit.
[46:34] So when you get the right answer, it's just
an equality match to the answer. It’s very
[46:39] simple to implement. If you're doing process
supervision, how do you assign in an automatable
[46:44] way, a partial credit assignment?
It's not obvious how you do it.
[46:47] Lots of labs are trying to
do it with these LLM judges.
[46:50] You get LLMs to try to do it.
You prompt an LLM, "Hey,
[46:53] look at a partial solution of a student.
How well do you think they're doing if the
[46:56] answer is this?" and they try to tune the prompt.
The reason that this is tricky is quite subtle.
[47:02] It's the fact that anytime you use an LLM to
assign a reward, those LLMs are giant things
[47:07] with billions of parameters, and they're gameable.
If you're reinforcement learning with respect to
[47:11] them, you will find adversarial examples
for your LLM judges, almost guaranteed.
[47:15] So you can't do this for too long.
You do maybe 10 steps or 20 steps, and maybe
[47:18] it will work, but you can't do 100 or 1,000.
I understand it's not obvious, but basically
[47:25] the model will find little cracks.
It will find all these spurious
[47:30] things in the nooks and crannies of the
giant model and find a way to cheat it.
[47:34] One example that's prominently in my mind, this
was probably public, if you're using an LLM judge
[47:42] for a reward, you just give it a solution from a
student and ask it if the student did well or not.
[47:47] We were training with
reinforcement learning against
[47:49] that reward function, and it worked really well.
Then, suddenly, the reward became extremely large.
[47:55] It was a massive jump, and it did perfect.
You're looking at it like, "Wow, this means
[47:59] the student is perfect in all these problems.
It's fully solved math." But when you look at
[48:05] the completions that you're getting from
the model, they are complete nonsense.
[48:08] They start out okay, and then
they change to "dhdhdhdh."
[48:11] It's just like, "Oh, okay, let's take two plus
three and we do this and this, and then dhdhdhdh."
[48:15] You're looking at it, and
it's like, this is crazy.
[48:17] How is it getting a reward of one or 100%?
You look at the LLM judge, and it turns out
[48:21] that "dhdhdhdh" is an adversarial example for
the model, and it assigns 100% probability to it.
[48:27] It's just because this is an
out-of-sample example to the LLM.
[48:30] It's never seen it during training,
and you're in pure generalization land.
[48:34] It's never seen it during training, and in
the pure generalization land, you can find
[48:37] these examples that break it.
You're basically training
[48:42] the LLM to be a prompt injection model.
Not even that. Prompt injection is way too fancy.
[48:46] You're finding adversarial
examples, as they're called.
[48:48] These are nonsensical solutions that are obviously
wrong, but the model thinks they are amazing.
[48:55] To the extent you think this is the
bottleneck to making RL more functional,
[48:59] then that will require making LLMs better judges,
if you want to do this in an automated way.
[49:05] Is it just going to be some sort
of GAN-like approach where you
[49:07] have to train models to be more robust?
The labs are probably doing all that.
[49:11] The obvious thing is, "dhdhdhdh"
should not get 100% reward.
[49:14] Okay, well, take "dhdhdhdh," put it
in the training set of the LLM judge,
[49:17] and say this is not 100%, this is 0%.
You can do this, but every time you do
[49:21] this, you get a new LLM, and it
still has adversarial examples.
[49:24] There's an infinity of adversarial examples.
Probably if you iterate this a few times, it'll
[49:29] probably be harder and harder to find adversarial
examples, but I'm not 100% sure because this thing
[49:32] has a trillion parameters or whatnot.
I bet you the labs are trying.
[49:41] I still think we need other ideas.
Interesting. Do you have some shape
[49:46] of what the other idea could be?
This idea of a review solution
[49:54] encompassing synthetic examples such that
when you train on them, you get better,
[49:58] and meta-learn it in some way.
I think there are some papers
[50:00] that I'm starting to see pop out.
I am only at a stage of reading abstracts
[50:04] because a lot of these papers are just ideas.
Someone has to make it work on a frontier
[50:08] LLM lab scale in full generality
because when you see these papers,
[50:13] they pop up, and it's just a bit noisy.
They're cool ideas, but I haven't seen
[50:17] anyone convincingly show that this is possible.
That said, the LLM labs are fairly closed,
[50:23] so who knows what they're doing now.
I can conceptualize how you would be able
[50:33] to train on synthetic examples or synthetic
problems that you have made for yourself.
[50:36] But there seems to be another thing humans
do—maybe sleep is this, maybe daydreaming is
[50:40] this—which is not necessarily to come up
with fake problems, but just to reflect.
[50:47] I'm not sure what the ML analogy is for
daydreaming or sleeping, or just reflecting.
[50:51] I haven't come up with a new problem.
Obviously, the very basic analogy would just
[50:54] be fine-tuning on reflection bits, but I feel like
in practice that probably wouldn't work that well.
[51:00] Do you have some take on what
the analogy of this thing is?
[51:05] I do think that we're missing some aspects there.
As an example, let’s take reading a book.
[51:11] Currently when LLMs are reading a book, what that
means is we stretch out the sequence of text,
[51:15] and the model is predicting the next token,
and it's getting some knowledge from that.
[51:19] That's not really what humans do.
When you're reading a book,
[51:21] I don't even feel like the book is exposition
I'm supposed to be attending to and training on.
[51:25] The book is a set of prompts for
me to do synthetic data generation,
[51:30] or for you to get to a book club
and talk about it with your friends.
[51:33] It's by manipulating that information
that you actually gain that knowledge.
[51:37] We have no equivalent of that with LLMs. They
don't really do that. I'd love to see during
[51:42] pre-training some stage that thinks through
the material and tries to reconcile it with
[51:46] what it already knows, and thinks through it
for some amount of time and gets that to work.
[51:52] There's no equivalence of any of this.
This is all research. There are some
[51:54] subtle—very subtle that I think are very hard
to understand—reasons why it's not trivial.
[51:59] If I can just describe one: why can't we
just synthetically generate and train on it?
[52:04] Because every synthetic example, if
I just give synthetic generation of
[52:07] the model thinking about a book, you look
at it and you're like, "This looks great.
[52:10] Why can't I train on it?"
You could try, but the model
[52:12] will get much worse if you continue trying.
That's because all of the samples you get
[52:17] from models are silently collapsed.
Silently—it is not obvious if you look
[52:21] at any individual example of it—they occupy
a very tiny manifold of the possible space of
[52:27] thoughts about content.
The LLMs, when they come off,
[52:30] they're what we call "collapsed."
They have a collapsed data distribution.
[52:34] One easy way to see it is to go to
ChatGPT and ask it, "Tell me a joke."
[52:38] It only has like three jokes.
It's not giving you the whole breadth
[52:41] of possible jokes. It knows like three jokes.
They're silently collapsed. You're not getting
[52:47] the richness and the diversity and the entropy
from these models as you would get from humans.
[52:52] Humans are a lot noisier, but
at least they're not biased,
[52:56] in a statistical sense. They're not silently
collapsed. They maintain a huge amount of entropy.
[53:00] So how do you get synthetic data generation to
work despite the collapse and while maintaining
[53:05] the entropy? That’s a research problem.
Just to make sure I understood, the reason
[53:09] that the collapse is relevant to synthetic data
generation is because you want to be able to
[53:13] come up with synthetic problems or reflections
which are not already in your data distribution?
[53:20] I guess what I'm saying is, say we have a chapter
of a book and I ask an LLM to think about it,
[53:26] it will give you something
that looks very reasonable.
[53:28] But if I ask it 10 times, you'll
notice that all of them are the same.
[53:31] You can't just keep scaling "reflection"
on the same amount of prompt information
[53:39] and then get returns from that.
Any individual sample will look okay,
[53:43] but the distribution of it is quite terrible.
It's quite terrible in such a way that if
[53:47] you continue training on too much of
your own stuff, you actually collapse.
[53:50] I think that there's possibly
no fundamental solution to this.
[53:54] I also think humans collapse over time.
These analogies are surprisingly good.
[54:00] Humans collapse during the course of their lives.
This is why children, they haven't overfit yet.
[54:06] They will say stuff that will shock you
because you can see where they're coming from,
[54:09] but it's just not the thing people say,
because they're not yet collapsed. But we're
[54:14] collapsed. We end up revisiting the same thoughts.
We end up saying more and more of the same stuff,
[54:20] and the learning rates go down, and
the collapse continues to get worse,
[54:24] and then everything deteriorates.
Have you seen this super interesting
[54:28] paper that dreaming is a way of preventing
this kind of overfitting and collapse?
[54:34] The reason dreaming is evolutionary adaptive
is to put you in weird situations that are
[54:41] very unlike your day-to-day reality, so
as to prevent this kind of overfitting.
[54:44] It's an interesting idea. I do think
that when you're generating things
[54:48] in your head and then you're attending to
it, you're training on your own samples,
[54:51] you're training on your synthetic data.
If you do it for too long,
[54:53] you go off-rails and you collapse way too much.
You always have to seek entropy in your life.
[55:01] Talking to other people is a great
source of entropy, and things like that.
[55:05] So maybe the brain has also built some internal
mechanisms for increasing the amount of entropy
[55:11] in that process. That's an interesting idea.
This is a very ill-formed thought so I’ll
[55:16] just put it out and let you react to it.
The best learners that we are aware of,
[55:20] which are children, are extremely
bad at recollecting information.
[55:25] In fact, at the very earliest stages of
childhood, you will forget everything.
[55:29] You're just an amnesiac about everything
that happens before a certain year date.
[55:32] But you're extremely good at picking up
new languages and learning from the world.
[55:36] Maybe there's some element of being
able to see the forest for the trees.
[55:38] Whereas if you compare it to the opposite end
of the spectrum, you have LLM pre-training,
[55:44] where these models will literally be
able to regurgitate word-for-word what
[55:47] is the next thing in a Wikipedia page.
But their ability to learn abstract
[55:52] concepts really quickly, the way
a child can, is much more limited.
[55:55] Then adults are somewhere in between, where
they don't have the flexibility of childhood
[55:59] learning, but they can memorize facts and
information in a way that is harder for kids.
[56:05] I don't know if there's something
interesting about that spectrum.
[56:08] I think there's something very
interesting about that, 100%.
[56:10] I do think that humans have a lot more of
an element, compared to LLMs, of seeing
[56:16] the forest for the trees.
We're not actually that good
[56:19] at memorization, which is actually a feature.
Because we're not that good at memorization, we're
[56:25] forced to find patterns in a more general sense.
LLMs in comparison are extremely good
[56:33] at memorization.
They will recite
[56:35] passages from all these training sources.
You can give them completely nonsensical data.
[56:41] You can hash some amount of text or something
like that, you get a completely random sequence.
[56:44] If you train on it, even just for a single
iteration or two, it can suddenly regurgitate
[56:48] the entire thing. It will memorize it.
There's no way a person can read a single
[56:51] sequence of random numbers and recite it to you.
That's a feature, not a bug, because it forces
[56:58] you to only learn the generalizable components.
Whereas LLMs are distracted by all the memory
[57:03] that they have of the pre-training
documents, and it's probably very
[57:06] distracting to them in a certain sense.
So that's why when I talk about the
[57:10] cognitive core, I want to remove the
memory, which is what we talked about.
[57:13] I'd love to have them have less memory
so that they have to look things up,
[57:17] and they only maintain the algorithms for
thought, and the idea of an experiment,
[57:22] and all this cognitive glue of acting.
And this is also relevant to preventing
[57:28] model collapse?
Let me think. I'm
[57:35] not sure. It's almost like a separate axis.
The models are way too good at memorization,
[57:40] and somehow we should remove that.
People are much worse, but it's a good thing.
[57:46] What is a solution to model collapse?
There are very naive things you could attempt.
[57:52] The distribution over logits
should be wider or something.
[57:55] There are many naive things you could try.
What ends up being the problem
[57:58] with the naive approaches?
That's a great question. You can imagine having
[58:02] a regularization for entropy and things like that.
I guess they just don't work as well empirically
[58:06] because right now the models are collapsed.
But I will say most of the tasks that we
[58:13] want from them don't actually demand diversity.
That’s probably the answer to what's going on.
[58:20] The frontier labs are trying
to make the models useful.
[58:22] I feel like the diversity of
the outputs is not so much...
[58:26] Number one, it's much harder to work with and
evaluate and all this stuff, but maybe it's not
[58:29] what's capturing most of the value.
In fact, it's actively penalized.
[58:34] If you're super creative in RL, it's not good.
Yeah. Or maybe if you're doing a lot of writing,
[58:39] help from LLMs and stuff like that, it's probably
bad because the models will silently give
[58:42] you all the same stuff.
They won't explore lots
[58:48] of different ways of answering a question.
Maybe this diversity, not as many applications
[58:56] need it so the models don't have it.
But then it's a problem at
[58:58] synthetic data generation time, et cetera.
So we're shooting ourselves in the foot by not
[59:01] allowing this entropy to maintain in the model.
Possibly the labs should try harder.
[59:06] I think you hinted that it's a very
fundamental problem, it won't be easy
[59:11] to solve. What's your intuition for that?
I don't know if it's super fundamental.
[59:17] I don't know if I intended to say that.
I do think that I haven't done these experiments,
[59:23] but I do think that you could probably
regularize the entropy to be higher.
[59:26] So you're encouraging the model to give you more
and more solutions, but you don't want it to
[59:31] start deviating too much from the training data.
It's going to start making up its own language.
[59:34] It's going to start using words that are
extremely rare, so it's going to drift too
[59:38] much from the distribution.
So I think controlling
[59:41] the distribution is just tricky.
It's probably not trivial in that sense.
[59:47] How many bits should the optimal core
of intelligence end up being if you
[59:54] just had to make a guess?
The thing we put on the
[59:56] von Neumann probes, how big does it have to be?
It's really interesting in the history of the
[60:01] field because at one point everything was
very scaling-pilled in terms of like, "Oh,
[60:05] we're gonna make much bigger models,
trillions of parameter models."
[60:08] What the models have done in size
is they've gone up and now they've
[60:14] come down. State-of-the-art models are smaller.
Even then, I think they memorized way too much.
[60:20] So I had a prediction a while back that I almost
feel like we can get cognitive cores that are
[60:24] very good at even a billion parameters.
If you talk to a billion parameter model,
[60:30] I think in 20 years, you can have
a very productive conversation.
[60:34] It thinks and it's a lot more like a human.
But if you ask it some factual question, it might
[60:39] have to look it up, but it knows that it doesn't
know and it might have to look it up and it will
[60:42] just do all the reasonable things.
That's surprising that you think
[60:44] it'll take a billion parameters.
Because already we have billion
[60:47] parameter models or a couple billion
parameter models that are very intelligent.
[60:51] Well, state-of-the-art models
are like a trillion parameters.
[60:53] But they remember so much stuff.
Yeah, but I'm surprised that in 10 years,
[60:58] given the pace… We have gpt-oss-20b.
That's way better than GPT-4 original,
[61:07] which was a trillion plus parameters.
Given that trend, I'm surprised you
[61:11] think in 10 years the cognitive
core is still a billion parameters.
[61:15] I'm surprised you're not like, "Oh it's
gonna be like tens of millions or millions."
[61:22] Here's the issue, the training data is
the internet, which is really terrible.
[61:26] There's a huge amount of gains to be
made because the internet is terrible.
[61:29] Even the internet, when you and I think of
the internet, you're thinking of like The
[61:32] Wall Street Journal. That's not what this
is. When you're looking at a pre-training
[61:36] dataset in the frontier lab and you look at a
random internet document, it's total garbage.
[61:40] I don't even know how this works at all.
It's some like stock tickers, symbols,
[61:46] it's a huge amount of slop and garbage
from like all the corners of the internet.
[61:50] It's not like your Wall Street Journal
article, that's extremely rare.
[61:53] So because the internet is so terrible, we have
to build really big models to compress all that.
[62:00] Most of that compression is memory
work instead of cognitive work.
[62:04] But what we really want is the
cognitive part, delete the memory.
[62:08] I guess what I'm saying is that we need
intelligent models to help us refine even
[62:12] the pre-training set to just narrow
it down to the cognitive components.
[62:15] Then I think you get away with a
much smaller model because it's a
[62:18] much better dataset and you could train it on it.
But probably it's not trained directly on it, it's
[62:22] probably distilled from a much better model still.
But why is the distilled version still a billion?
[62:28] I just feel like distillation
works extremely well.
[62:30] So almost every small model, if you have a
small model, it's almost certainly distilled.
[62:35] Right, but why is the distillation in
10 years not getting below 1 billion?
[62:39] Oh, you think it should be smaller than a
billion? I mean, come on, right? I don't
[62:43] know. At some point it should take at least
a billion knobs to do something interesting.
[62:49] You're thinking it should be even smaller?
Yeah. If you look at the trend over the last
[62:53] few years of just finding low-hanging fruit and
going from trillion plus models to models that
[62:57] are literally two orders of magnitude smaller in a
matter of two years and having better performance,
[63:03] it makes me think the sort of core of
intelligence might be even way, way smaller.
[63:09] Plenty of room at the bottom,
to paraphrase Feynman.
[63:11] I feel like I'm already contrarian
by talking about a billion parameter
[63:14] cognitive core and you're outdoing me.
Maybe we could get a little bit smaller.
[63:23] I do think that practically speaking, you
want the model to have some knowledge.
[63:26] You don't want it to be looking up everything
because then you can't think in your head.
[63:30] You're looking up way too much stuff all the time.
[63:32] Some basic curriculum needs to be there for
knowledge, but it doesn't have esoteric knowledge.
[63:38] We're discussing what plausibly
could be the cognitive core.
[63:41] There's a separate question which is what
will be the size of frontier models over time?
[63:46] I'm curious if you have predictions.
We had increasing scale up to maybe GPT 4.5 and
[63:51] now we're seeing decreasing or plateauing scale.
There are many reasons this could be going on.
[63:56] Do you have a prediction going forward?
Will the biggest models be bigger,
[64:00] will they be smaller, will they be the same?
I don't have a super strong prediction.
[64:07] The labs are just being practical.
They have a flops budget and a cost budget.
[64:10] It just turns out that pre-training is not where
you want to put most of your flops or your cost.
[64:14] That's why the models have gotten smaller.
They are a bit smaller, the pre-training
[64:17] stage is smaller, but they make
it up in reinforcement learning,
[64:21] mid-training, and all this stuff that follows.
They're just being practical in terms of all the
[64:25] stages and how you get the most bang for the buck.
Forecasting that trend is quite hard.
[64:30] I do still expect that there's so much
low-hanging fruit. That's my basic expectation.
[64:38] I have a very wide distribution here.
Do you expect the low-hanging fruit to be
[64:42] similar in kind to the kinds of things that have
been happening over the last two to five years?
[64:49] If I look at nanochat versus nanoGPT
and the architectural tweaks you made,
[64:54] is that the flavor of things you
expect to continue to keep happening?
[64:58] You're not expecting any giant paradigm shifts.
For the most part, yeah. I expect the
[65:01] datasets to get much, much better.
When you look at the average datasets,
[65:03] they're extremely terrible.
They’re so bad that I
[65:05] don't even know how anything works.
Look at the average example in the training set:
[65:10] factual mistakes, errors, nonsensical things.
Somehow when you do it at scale,
[65:16] the noise washes away and you're left with
some of the signal. Datasets will improve
[65:21] a ton. Everything gets better. Our hardware,
all the kernels for running the hardware and
[65:29] maximizing what you get with the hardware.
Nvidia is slowly tuning the hardware itself,
[65:33] Tensor Cores, all that needs to
happen and will continue to happen.
[65:36] All the kernels will get better and
utilize the chip to the max extent.
[65:39] All the algorithms will probably improve over
optimization, architecture, and all the modeling
[65:45] components of how everything is done and what
the algorithms are that we're even training with.
[65:48] I do expect that nothing dominates. Everything
plus 20%. This is roughly what I've seen.
[67:13] People have proposed different ways of charting
how much progress we've made towards full AGI.
[67:21] If you can come up with some line, then you
can see where that line intersects with AGI
[67:25] and where that would happen on the x-axis.
People have proposed it's the education level.
[67:29] We had a high schooler, and then they went to
college with RL, and they're going to get a Ph.D.
[67:34] I don't like that one.
Or they'll propose horizon
[67:36] length. Maybe they can do tasks that take
a minute, they can do those autonomously.
[67:41] Then they can autonomously do tasks that take
an hour, a human an hour, a human a week.
[67:46] How do you think about the relevant y-axis here?
How should we think about how
[67:53] AI is making progress?
I have two answers to that.
[67:55] Number one, I'm almost tempted to
reject the question entirely because
[67:59] I see this as an extension of computing.
Have we talked about how to chart progress
[68:02] in computing, or how do you chart progress
in computing since the 1970s or whatever?
[68:06] What is the y-axis? The whole question is
funny from that perspective a little bit.
[68:13] When people talk about AI and the original AGI
and how we spoke about it when OpenAI started,
[68:18] AGI was a system you could go to that can do any
economically valuable task at human performance
[68:27] or better. That was the definition. I
was pretty happy with that at the time.
[68:32] I've stuck to that definition forever, and
then people have made up all kinds of other
[68:36] definitions. But I like that definition. The first
concession that people make all the time is they
[68:43] just take out all the physical stuff because
we're just talking about digital knowledge work.
[68:48] That's a pretty major concession compared to
the original definition, which was any task
[68:52] a human can do. I can lift things, etc. AI
can't do that, obviously, but we'll take it.
[68:57] What fraction of the economy are we taking away
by saying, "Oh, only knowledge work?" I don't know
[69:02] the numbers. I feel about 10% to 20%, if I had to
guess, is only knowledge work, someone could work
[69:09] from home and perform tasks, something like that.
It's still a really large market.
[69:16] What is the size of the
economy, and what is 10% or 20%?
[69:19] We're still talking about a few trillion
dollars, even in the US, of market share or work.
[69:26] So it's still a very massive bucket.
Going back to the definition,
[69:30] what I would be looking for is to
what extent is that definition true?
[69:35] Are there jobs or lots of tasks?
If we think of tasks as not jobs but tasks.
[69:40] It's difficult because the problem is society will
refactor based on the tasks that make up jobs,
[69:47] based on what's automatable or not.
Today, what jobs are replaceable by AI?
[69:52] A good example recently was Geoff Hinton's
prediction that radiologists would not be
[69:57] a job anymore, and this turned out
to be very wrong in a bunch of ways.
[70:00] Radiologists are alive and well and growing,
even though computer vision is really,
[70:04] really good at recognizing all the different
things that they have to recognize in images.
[70:07] It's just a messy, complicated job with a
lot of surfaces and dealing with patients
[70:11] and all this stuff in the context of it.
I don't know that by that definition
[70:17] AI has made a huge dent yet.
Some of the jobs that I would
[70:22] be looking for have some features that make it
very amenable to automation earlier than later.
[70:27] As an example, call center employees
often come up, and I think rightly so.
[70:30] Call center employees have a number of simplifying
properties with respect to what's automatable
[70:35] today. Their jobs are pretty simple. It's a
sequence of tasks, and every task looks similar.
[70:42] You take a phone call with a person, it's
10 minutes of interaction or whatever it is,
[70:45] probably a bit longer.
In my experience, a lot longer.
[70:49] You complete some task in some scheme,
and you change some database entries
[70:53] around or something like that.
So you keep repeating something
[70:55] over and over again, and that's your job.
You do want to bring in the task horizon—how
[71:01] long it takes to perform a task—and
then you want to also remove context.
[71:05] You're not dealing with different parts of
services of companies or other customers.
[71:08] It's just the database, you,
and a person you're serving.
[71:11] It's more closed, it's more
understandable, it's purely digital.
[71:15] So I would be looking for those things.
But even there, I'm not looking
[71:18] at full automation yet.
I'm looking for an autonomy slider.
[71:21] I expect that we are not going
to instantly replace people.
[71:25] We're going to be swapping in
AIs that do 80% of the volume.
[71:29] They delegate 20% of the volume to humans,
and humans are supervising teams of five AIs
[71:33] doing the call center work that's more rote.
I would be looking for new interfaces or new
[71:39] companies that provide some
layer that allows you to manage
[71:44] some of these AIs that are not yet perfect.
Then I would expect that across the economy.
[71:48] A lot of jobs are a lot harder
than a call center employee.
[71:52] With radiologists, I'm totally
speculating and I have no idea what
[71:56] the actual workflow of a radiologist involves.
But one analogy that might be applicable is when
[72:03] Waymos were first being rolled out, there'd be a
person sitting in the front seat, and you just had
[72:09] to have them there to make sure that if something
went really wrong, they're there to monitor.
[72:12] Even today, people are still watching
to make sure things are going well.
[72:15] Robotaxi, which was just deployed,
still has a person inside it.
[72:19] Now we could be in a similar situation where
if you automate 99% of a job, that last 1%
[72:25] the human has to do is incredibly valuable
because it's bottlenecking everything else.
[72:29] If it were the case with radiologists, where
the person sitting in the front of Waymo has
[72:35] to be specially trained for years in order
to provide the last 1%, their wages should
[72:39] go up tremendously because they're the
one thing bottlenecking wide deployment.
[72:43] Radiologists, I think their wages have
gone up for similar reasons, if you're
[72:46] the last bottleneck and you're not fungible.
A Waymo driver might be fungible with others.
[72:53] So you might see this thing where your wages
go up until you get to 99% and then fall just
[72:57] like that when the last 1% is gone.
And I wonder if we're seeing similar
[73:02] things with radiology or salaries of call
center workers or anything like that.
[73:07] That's an interesting question. I don't think
we're currently seeing that with radiology.
[73:15] I think radiology is not a good example.
I don't know why Geoff Hinton picked
[73:19] on radiology because I think it's an
extremely messy, complicated profession.
[73:25] I would be a lot more interested in what's
happening with call center employees today,
[73:28] for example, because I would expect a lot
of the rote stuff to be automatable today.
[73:32] I don't have first-level access to it but
I would be looking for trends of what's
[73:36] happening with the call center employees.
Some of the things I would also expect
[73:40] is that maybe they are swapping in AI, but
then I would still wait for a year or two
[73:45] because I would potentially expect them to
pull back and rehire some of the people.
[73:49] There's been evidence that that's already been
happening generally in companies that have been
[73:53] adopting AI, which I think is quite surprising.
I also found what was really surprising. AGI,
[73:59] right? A thing which would do everything.
We'll take out physical work,
[74:04] but it should be able to do all knowledge work.
What you would have naively anticipated is that
[74:09] the way this progression would happen is
that you take a little task that a consultant
[74:14] is doing, you take that out of the bucket.
You take a little task that an accountant is
[74:19] doing, you take that out of the bucket.
Then you're just doing this
[74:22] across all knowledge work.
But instead, if we do believe we're
[74:25] on the path of AGI with the current paradigm,
the progression is very much not like that.
[74:30] It does not seem like consultants and accountants
are getting huge productivity improvements.
[74:34] It's very much like programmers are getting
more and more chiseled away at their work.
[74:39] If you look at the revenues of these companies,
discounting normal chat revenue—which is similar
[74:46] to Google or something—just looking at
API revenues, it's dominated by coding.
[74:51] So this thing which is "general", which
should be able to do any knowledge work,
[74:56] is just overwhelmingly doing only coding.
It's a surprising way that you would
[75:00] expect the AGI to be deployed.
There's an interesting point
[75:04] here. I do believe coding is the perfect
first thing for these LLMs and agents.
[75:12] That’s because coding has always
fundamentally worked around text.
[75:17] It's computer terminals and text,
and everything is based around text.
[75:20] LLMs, the way they're trained
on the Internet, love text.
[75:24] They're perfect text processors, and there's
all this data out there. It's a perfect fit.
[75:29] We also have a lot of infrastructure
pre-built for handling code and text.
[75:33] For example, we have Visual Studio Code
or your favorite IDE showing you code,
[75:41] and an agent can plug into that.
If an agent has a diff where it made some change,
[75:45] we suddenly have all this code already that shows
all the differences to a code base using a diff.
[75:51] It's almost like we've pre-built a
lot of the infrastructure for code.
[75:55] Contrast that with some of the
things that don't enjoy that at all.
[75:58] As an example, there are people trying to build
automation not for coding, but for slides.
[76:03] I saw a company doing slides. That's
much, much harder. The reason it's
[76:07] much harder is because slides are not text.
Slides are little graphics, they're arranged
[76:11] spatially, and there's a visual component to it.
Slides don't have this pre-built infrastructure.
[76:18] For example, if an agent is to make a change to
your slides, how does a thing show you the diff?
[76:24] How do you see the diff?
There's nothing that shows diffs
[76:26] for slides. Someone has to build it. Some of these
things are not amenable to AIs as they are, which
[76:34] are text processors, and code surprisingly is.
I’m not sure that alone explains it.
[76:42] I personally have tried to get LLMs to be useful
in domains which are just pure language-in,
[76:49] language-out, like rewriting transcripts,
coming up with clips based on transcripts.
[76:57] It's very plausible that I didn't do
every single possible thing I could do.
[77:00] I put a bunch of good examples in context, but
maybe I should have done some kind of fine-tuning.
[77:06] Our mutual friend, Andy Matuschak, told me that
he tried 50 billion things to try to get models
[77:13] to be good at writing spaced repetition prompts.
Again, very much language-in, language-out tasks,
[77:19] the kind of thing that should be dead
center in the repertoire of these LLMs.
[77:22] He tried in-context learning
with a few-shot examples.
[77:25] He tried supervised fine-tuning and retrieval.
He could not get them to make
[77:35] cards to his satisfaction.
So I find it striking that even in language-out
[77:39] domains, it's very hard to get a lot of economic
value out of these models separate from coding.
[77:45] I don't know what explains it.
That makes sense. I'm not
[77:52] saying that anything text is trivial.
I do think that code is pretty structured.
[77:58] Text is maybe a lot more flowery, and there's
a lot more entropy in text, I would say.
[78:04] I don't know how else to put it.
Also code is hard, and so people feel quite
[78:10] empowered by LLMs, even from simple knowledge.
I don't know that I have a very good answer.
[78:19] Obviously, text makes it much, much easier,
but it doesn't mean that all text is trivial.
[78:25] How do you think about superintelligence?
Do you expect it to feel qualitatively different
[78:29] from normal humans or human companies?
I see it as a progression
[78:37] of automation in society.
Extrapolating the trend of computing, there will
[78:42] be a gradual automation of a lot of things, and
superintelligence will an extrapolation of that.
[78:47] We expect more and more autonomous
entities over time that are doing a lot
[78:50] of the digital work and then eventually even
the physical work some amount of time later.
[78:56] Basically I see it as just
automation, roughly speaking.
[79:00] But automation includes the things humans
can already do, and superintelligence
[79:03] implies things humans can’t do.
But one of the things that people
[79:05] do is invent new things, which I would just
put into the automation if that makes sense.
[79:10] But I guess, less abstractly and more
qualitatively, do you expect something
[79:18] to feel like… Because this thing can either think
so fast, or has so many copies, or the copies can
[79:26] merge back into themselves, or is much smarter,
any number of advantages an AI might have, will
[79:36] the civilization in which these AIs exist
just feel qualitatively different from humans?
[79:39] I think it will. It is fundamentally automation,
but it will be extremely foreign. It will look
[79:44] really strange. Like you mentioned, we can run
all of this on a computer cluster and much faster.
[79:53] Some of the scenarios that I start to get
nervous about when the world looks like
[79:58] that is this gradual loss of control
and understanding of what's happening.
[80:01] I think that's the most likely outcome, that
there will be a gradual loss of understanding.
[80:07] We'll gradually layer all this stuff
everywhere, and there will be fewer
[80:10] and fewer people who understand it.
Then there will be a gradual loss of
[80:14] control and understanding of what's happening.
That to me seems the most likely outcome of how
[80:19] all this stuff will go down.
Let me probe on that a bit.
[80:22] It's not clear to me that loss of control and
loss of understanding are the same things.
[80:27] A board of directors at TSMC, Intel—name a random
company—they're just prestigious 80-year-olds.
[80:37] They have very little understanding, and maybe
they don't practically actually have control.
[80:43] A better example
[80:43] is the President of the United States.
The President has a lot of fucking power.
[80:48] I'm not trying to make a good statement
about the current operant, or maybe I am,
[80:53] but the actual level of understanding is
very different from the level of control.
[80:56] I think that's fair. That's a good
pushback. I think I expect loss of both.
[81:05] How come? Loss of understanding is
obvious, but why loss of control?
[81:10] We're really far into a territory where I
don't know what this looks like, but if I
[81:14] were to write sci-fi novels, they would look along
the lines of not even a single entity that takes
[81:22] over everything, but multiple competing entities
that gradually become more and more autonomous.
[81:27] Some of them go rogue and
the others fight them off.
[81:31] It's this hot pot of completely autonomous
activity that we've delegated to.
[81:37] I feel it would have that flavor.
It is not the fact that they are smarter
[81:43] than us that is resulting in the loss of control.
It's the fact that they are competing with each
[81:47] other, and whatever arises out of that
competition leads to the loss of control.
[81:58] A lot of these things, they will be
tools to people, they're acting on
[82:04] behalf of people or something like that.
So maybe those people are in control,
[82:07] but maybe it's a loss of control overall for
society in the sense of outcomes we want.
[82:13] You have entities acting on behalf of individuals
that are still roughly seen as out of control.
[82:20] This is a question I should have asked earlier.
We were talking about how currently it feels like
[82:24] when you're doing AI engineering or AI research,
these models are more in the category of compiler
[82:29] rather than in the category of a replacement.
At some point, if you have AGI,
[82:34] it should be able to do what you do.
Do you feel like having a million
[82:38] copies of you in parallel results in
some huge speed-up of AI progress?
[82:43] If that does happen, do you expect to see an
intelligence explosion once we have a true AGI?
[82:49] I'm not talking about LLMs today.
I do, but it's business as usual because
[82:56] we're in an intelligence explosion
already and have been for decades.
[83:00] It's basically the GDP curve that is
an exponential weighted sum over so
[83:03] many aspects of the industry.
Everything is gradually being
[83:06] automated and has been for hundreds of years.
The Industrial Revolution is automation and
[83:10] some of the physical components and
tool building and all this stuff.
[83:13] Compilers are early software
automation, et cetera.
[83:16] We've been recursively self-improving
and exploding for a long time.
[83:21] Another way to see it is that Earth was a pretty
boring place if you don't look at the biomechanics
[83:27] and so on, and looked very similar.
If you look from space, we're in the
[83:33] middle of this firecracker event,
but we're seeing it in slow motion.
[83:38] I definitely feel like this has
already happened for a very long time.
[83:42] Again, I don't see AI as a distinct
technology with respect to what has
[83:47] already been happening for a long time.
You think it's continuous with this
[83:50] hyper-exponential trend?
Yes. That's why this was
[83:53] very interesting to me, because I was
trying to find AI in the GDP for a while.
[83:57] I thought that GDP should go up.
But then I looked at some of the
[84:01] other technologies that I thought
were very transformative, like
[84:04] computers or mobile phones or et cetera.
You can't find them in GDP. GDP is the same
[84:08] exponential. Even the early iPhone didn't have the
App Store, and it didn't have a lot of the bells
[84:13] and whistles that the modern iPhone has.
So even though we think of 2008,
[84:18] when the iPhone came out, as this major
seismic change, it's actually not.
[84:21] Everything is so spread out and it so
slowly diffuses that everything ends up
[84:25] being averaged up into the same exponential.
It's the exact same thing with computers.
[84:28] You can't find them in the GDP
like, "Oh, we have computers now."
[84:31] That's not what happened, because
it's such slow progression.
[84:33] With AI we're going to see the exact same thing.
It's just more automation. It allows us to write
[84:37] different kinds of programs that we couldn't write
before, but AI is still fundamentally a program.
[84:42] It's a new kind of computer and
a new kind of computing system.
[84:46] But it has all these problems,
it's going to diffuse over time,
[84:49] and it's still going to add
up to the same exponential.
[84:52] We're still going to have an exponential
that's going to get extremely vertical.
[84:56] It's going to be very foreign to
live in that kind of an environment.
[84:59] Are you saying that, if you look at the trend
before the Industrial Revolution to now,
[85:05] you have a hyper-exponential where you go
from 0% growth to then 10,000 years ago,
[85:11] 0.02% growth, and to now when we're at 2%
growth. That's a hyper-exponential. Are you
[85:15] saying if you're charting AI on there, then
AI takes you to 20% growth or 200% growth?
[85:20] Or are you saying that if you look at
the last 300 years, what you've been
[85:23] seeing is that you have technology after
technology—computers, electrification,
[85:27] steam engines, railways, et cetera—but the
rate of growth is the exact same, it's 2%.
[85:33] Are you saying the rate of growth will go up?
The rate of growth has also stayed
[85:38] roughly constant, right?
Only over the last 200, 300 years.
[85:41] But over the course of
human history it's exploded.
[85:44] It's gone from 0% to faster, faster,
faster. Industrial explosion, 2%.
[85:51] For a while I tried to find AI
or look for AI in the GDP curve,
[85:54] and I've convinced myself that this is false.
Even when people talk about recursive
[85:58] self-improvement and labs and stuff
like that, this is business as usual.
[86:01] Of course it's going to recursively self-improve,
and it's been recursively self-improving.
[86:05] LLMs allow the engineers to work much more
efficiently to build the next round of LLM,
[86:11] and a lot more of the components are
being automated and tuned and et cetera.
[86:14] All the engineers having access
to Google Search is part of it.
[86:19] All the engineers having an IDE, all of them
having autocomplete or having Claude code,
[86:23] et cetera, it's all just part of the same
speed-up of the whole thing. It's just so smooth.
[86:30] Just to clarify, you're saying that
the rate of growth will not change.
[86:35] The intelligence explosion will show up as
it just enabled us to continue staying on the
[86:39] 2% growth trajectory, just as the Internet
helped us stay on the 2% growth trajectory.
[86:42] Yes, my expectation is that
it stays in the same pattern.
[86:47] Just to throw the opposite argument against you,
my expectation is that it blows up because I think
[86:55] true AGI—and I'm not talking about LLM coding
bots, I'm talking about actual replacement of a
[87:00] human in a server—is qualitatively different
from these other productivity-improving
[87:07] technologies because it's labor itself.
I think we live in a very labor-constrained world.
[87:13] If you talk to any startup founder or any person,
you can be like, what do you need more of? You
[87:17] need really talented people. And if you have
billions of extra people who are inventing stuff,
[87:22] integrating themselves, making companies bottom
start to finish, that feels qualitatively
[87:28] different from a single technology.
It's as if you get 10 billion
[87:32] extra people on the planet.
Maybe a counterpoint. I'm pretty willing
[87:37] to be convinced one way or another on this point.
But I will say, for example, computing is labor.
[87:42] Computing was labor. Computers, a lot
of jobs disappeared because computers
[87:45] are automating a bunch of digital information
processing that you now don't need a human for.
[87:50] So computers are labor, and that has played out.
Self-driving as an example is also computers doing
[87:57] labor. That's already been playing
out. It's still business as usual.
[88:02] You have a machine which is spitting out more
things like that at potentially faster pace.
[88:08] Historically, we have examples
of the growth regime changing
[88:11] where you went from 0.2% growth to 2% growth.
It seems very plausible to me that a machine which
[88:18] is then spitting out the next self-driving
car and the next Internet and whatever…
[88:23] I see where it's coming from.
At the same time, I do feel like
[88:27] people make this assumption of, "We
have God in a box, and now it can do
[88:31] everything," and it just won't look like that.
It's going to be able to do some of the things.
[88:36] It's going to fail at some other things.
It's going to be gradually put into society,
[88:39] and we'll end up with the same pattern. That
is my prediction. This assumption of suddenly
[88:43] having a completely intelligent, fully flexible,
fully general human in a box, and we can dispense
[88:49] it at arbitrary problems in society, I don't
think that we will have this discrete change.
[88:57] I think we'll arrive at the same kind of
gradual diffusion of this across the industry.
[89:03] It often ends up being misleading
in these conversations.
[89:09] I don't like to use the word intelligence in
this context because intelligence implies you
[89:12] think there'll be a single superintelligence
sitting in a server and it'll divine how
[89:18] to come up with new technologies and
inventions that cause this explosion.
[89:22] That's not what I'm imagining
when I'm imagining 20% growth.
[89:25] I'm imagining that there are billions of
very smart human-like minds, potentially,
[89:33] or that's all that's required.
But the fact that there's hundreds
[89:36] of millions of them, billions of them, each
individually making new products, figuring
[89:41] out how to integrate themselves into the economy.
If a highly experienced smart immigrant came to
[89:46] the country, you wouldn't need to figure out how
we integrate them in the economy. They figure it
[89:49] out. They could start a company, they could make
inventions, or increase productivity in the world.
[89:55] We have examples, even in the current regime,
of places that have had 10-20% economic growth.
[90:01] If you just have a lot of people and
less capital in comparison to the people,
[90:05] you can have Hong Kong or Shenzhen or
whatever with decades of 10% plus growth.
[90:13] There's a lot of really smart people who are
ready to make use of the resources and do
[90:17] this period of catch-up because we've had this
discontinuity, and I think AI might be similar.
[90:24] I understand, but I still think that
you're presupposing some discrete jump.
[90:28] There's some unlock that we're waiting to claim.
And suddenly we're going to have
[90:31] geniuses in data centers.
I still think you're presupposing
[90:34] some discrete jump that has no historical
precedent that I can't find in any of the
[90:39] statistics and that I think probably won't happen.
I mean, the Industrial Revolution is such a jump.
[90:43] You went from 0.2% growth to 2% growth.
I'm just saying you'll see another jump like that.
[90:49] I'm a little bit suspicious,
I would have to take a look.
[90:53] For example, some of the logs are not very
good from before the Industrial Revolution.
[90:59] I'm a bit suspicious of it but
I don't have strong opinions.
[91:04] You're saying that this was a singular
event that was extremely magical.
[91:07] You're saying that maybe there's going
to be another event that's going to
[91:09] be just like that, extremely magical.
It will break the paradigm, and so on.
[91:12] I actually don't think… The crucial thing with the
Industrial Revolution was that it was not magical.
[91:18] If you just zoomed in, what you would see in 1770
or 1870 is not that there was some key invention.
[91:28] But at the same time, you did move the
economy to a regime where the progress
[91:32] was much faster and the exponential 10x'd.
I expect a similar thing from AI where it's
[91:37] not like there's going to be a single moment
where we've made the crucial invention.
[91:42] It’s an overhang that's being unlocked.
Like maybe there's a new energy source.
[91:45] There's some unlock—in this case, some kind of
a cognitive capacity—and there's an overhang of
[91:49] cognitive work to do.
That's right.
[91:52] You're expecting that overhang to be filled by
this new technology when it crosses the threshold.
[91:56] Maybe one way to think about it is
throughout history, a lot of growth
[92:01] comes because people come up with ideas,
and then people are out there doing stuff to
[92:06] execute those ideas and make valuable output.
Through most of this time, the population has
[92:11] been exploding. That has been driving
growth. For the last 50 years, people
[92:14] have argued that growth has stagnated.
The population in frontier countries
[92:17] has also stagnated.
I think we go back to
[92:19] the exponential growth in population that
causes hyper-exponential growth in output.
[92:28] It's really hard to tell. I
understand that viewpoint. I
[92:32] don't intuitively feel that viewpoint.
You recommended Nick Lane's book to me.
[93:40] On that basis, I also found it super
interesting and I interviewed him.
[93:45] I have some questions about thinking about
intelligence and evolutionary history.
[93:49] Now that you, over the last 20 years of doing AI
research, you maybe have a more tangible sense of
[93:54] what intelligence is, what it takes to develop it.
Are you more or less surprised as a result that
[94:01] evolution just spontaneously stumbled upon it?
I love Nick Lane's books. I was just listening
[94:12] to his podcast on the way up here.
With respect to intelligence and its
[94:15] evolution, it's very, very recent.
I am surprised that it evolved.
[94:23] I find it fascinating to think
about all the worlds out there.
[94:25] Say there's a thousand planets
like Earth and what they look like.
[94:27] I think Nick Lane was here talking
about some of the earliest parts.
[94:30] He expects very similar life
forms, roughly speaking,
[94:34] and bacteria-like things in most of them.
There are a few breaks in there.
[94:39] The evolution of intelligence intuitively feels
to me like it should be a fairly rare event.
[94:45] Maybe you should base it on
how long something has existed.
[94:49] If bacteria were around for 2 billion years
and nothing happened, then going to eukaryote
[94:52] is probably pretty hard because bacteria came
up quite early in Earth's evolution or history.
[95:02] How long have we had animals?
Maybe a couple hundred million years,
[95:04] multicellular animals that
run around, crawl, et cetera.
[95:08] That’s maybe 10% of Earth's lifespan.
Maybe on that timescale it's not too tricky.
[95:18] It's still surprising to me,
intuitively, that it developed.
[95:20] I would maybe expect just a lot of animal-like
life forms doing animal-like things.
[95:24] The fact that you can get something
that creates culture and knowledge
[95:28] and accumulates it is surprising to me.
There's a couple of interesting follow-ups.
[95:35] If you buy the Sutton perspective that the
crux of intelligence is animal intelligence…
[95:41] The quote he said is "If you got to the
squirrel, you'd be most of the way to AGI."
[95:46] We got to squirrel intelligence right after
the Cambrian explosion 600 million years ago.
[95:51] It seems like what instigated that was the
oxygenation event 600 million years ago.
[95:56] But immediately the intelligence algorithm
was there to make the squirrel intelligence.
[96:02] It's suggestive that animal
intelligence was like that.
[96:06] As soon as you had the oxygen in the
environment, you had the eukaryote,
[96:09] you could just get the algorithm.
Maybe it was an accident that
[96:14] evolution stumbled upon it so fast,
but I don't know if that suggests that
[96:18] at the end it's going to be quite simple.
It's so hard to tell with any of this stuff.
[96:23] You can base it a bit on how long
something has existed or how long
[96:26] it feels like something has been bottlenecked.
Nick Lane is very good about describing this very
[96:30] apparent bottleneck in bacteria and archaea.
For two billion years, nothing happened.
[96:34] There’s extreme diversity of biochemistry,
and yet nothing grows to become animals.
[96:41] Two billion years. I don't know that we've
seen exactly that kind of an equivalent with
[96:46] animals and intelligence, to your point.
We could also look at it with respect
[96:51] to how many times we think certain
intelligence has individually sprung up.
[96:55] That's a really good thing to investigate.
One thought on that. There's hominid intelligence,
[97:03] and then there's bird intelligence.
Ravens, etc., are extremely clever,
[97:07] but their brain parts are quite distinct,
and we don't have that much in common.
[97:13] That's a slight indication of maybe
intelligence springing up a few times.
[97:18] In that case, you'd expect it more frequently.
A former guest, Gwern, and Carl Shulman, they’ve
[97:26] made a really interesting point about that.
Their perspective is that the scalable algorithm
[97:32] which humans have and primates have, arose in
birds as well, and maybe other times as well.
[97:39] But humans found an evolutionary niche which
rewarded marginal increases in intelligence
[97:47] and also had a scalable brain algorithm that
could achieve those increases in intelligence.
[97:53] For example, if a bird had a bigger brain,
it would just collapse out of the air.
[97:57] It's very smart for the size of
its brain, but it's not in a niche
[98:00] which rewards the brain getting bigger.
It’s maybe similar to some really smart…
[98:08] Like dolphins?
Exaclty, humans, we have hands that
[98:10] reward being able to learn how to do tool use.
We can externalize digestion, more energy to
[98:14] the brain, and that kicks off the flywheel.
Also stuff to work with. I'm guessing it would
[98:19] be harder if I were a dolphin. How do you have
fire? The universe of things you can do in water,
[98:28] inside water, is probably lower than
what you can do on land, just chemically.
[98:33] I do agree with this viewpoint of these niches
and what's being incentivized. I still find it
[98:38] miraculous. I would have expected things to
get stuck on animals with bigger muscles.
[98:47] Going through intelligence is a
really fascinating breaking point.
[98:51] The way Gwern put it is the reason it was so hard
is that it's a very tight line between being in
[98:56] a situation where something is so important
to learn that it's not worth distilling the
[99:03] exact right circuits directly back into your DNA,
versus it's not important enough to learn at all.
[99:10] It has to be something that incentivizes
building the algorithm to learn in a lifetime.
[99:17] You have to incentivize some kind of adaptability.
You want environments that are unpredictable
[99:21] so evolution can't bake your
algorithms into your weights.
[99:24] A lot of animals are pre-baked in this sense.
Humans have to figure it out at test
[99:30] time when they get born.
You want these environments
[99:35] that change really rapidly, where you
can't foresee what will work well.
[99:42] You create intelligence to
figure it out at test time.
[99:45] Quintin Pope had this interesting blog post
where he's saying the reason he doesn't
[99:48] expect a sharp takeoff is that humans had the
sharp takeoff where 60,000 years ago we seem
[99:55] to have had the cognitive architectures
that we have today. 10,000 years ago,
[99:59] agricultural revolution, modernity.
What was happening in that 50,000 years?
[100:04] You had to build this cultural scaffold where
you can accumulate knowledge over generations.
[100:11] This is an ability that exists for
free in the way we do AI training.
[100:16] In many cases they are literally distilled.
If you retrain a model, they can be trained
[100:21] on each other, they can be trained
on the same pre-training corpus,
[100:25] they don't literally have to start from scratch.
There's a sense in which it took humans a long
[100:31] time to get this cultural loop going, but it just
comes for free with the way we do LLM training.
[100:36] Yes and no. Because LLMs don't really
have the equivalent of culture.
[100:39] Maybe we're giving them way too
much and incentivizing not to
[100:42] create it or something like that.
But the invention of culture and of
[100:45] written record and of passing down notes
between each other, I don't think there's
[100:48] an equivalent of that with LLMs right now.
LLMs don't really have culture right now and
[100:53] it's one of the impediments I would say.
Can you give me some sense of what
[100:58] LLM culture might look like?
In the simplest case it would be a
[101:01] giant scratchpad that the LLM can edit and as it's
reading stuff or as it's helping out with work,
[101:06] it's editing the scratchpad for itself.
Why can't an LLM write a book for the other
[101:10] LLMs? That would be cool. Why can't other
LLMs read this LLM's book and be inspired
[101:16] by it or shocked by it or something like that?
There's no equivalence for any of this stuff.
[101:20] Interesting. When would you expect
that kind of thing to start happening?
[101:24] Also, multi-agent systems and a sort of
independent AI civilization and culture?
[101:31] There are two powerful ideas in the
realm of multi-agent that have both
[101:34] not been really claimed or so on.
The first one I would say is culture
[101:38] and LLMs having a growing repertoire
of knowledge for their own purposes.
[101:44] The second one looks a lot more
like the powerful idea of self-play.
[101:47] In my mind it’s extremely powerful.
Evolution has a lot of competition
[101:53] driving intelligence and evolution.
In AlphaGo more algorithmically,
[101:59] AlphaGo is playing against itself and that's
how it learns to get really good at Go.
[102:03] There's no equivalent of self-playing LLMs,
but I would expect that to also exist.
[102:07] No one has done it yet.
Why can't an LLM for example, create a bunch
[102:10] of problems that another LLM is learning to solve?
Then the LLM is always trying to serve more and
[102:16] more difficult problems, stuff like that.
There's a bunch of ways to organize it.
[102:22] It's a realm of research, but I haven't
seen anything that convincingly claims
[102:26] both of those multi-agent improvements.
We're mostly in the realm of a single
[102:31] individual agent, but that will change.
In the realm of culture also,
[102:37] I would also bucket organizations.
We haven't seen anything like that convincingly
[102:41] either. That's why we're still early.
Can you identify the key bottleneck
[102:45] that's preventing this kind
of collaboration between LLMs?
[102:50] Maybe the way I would put it is,
some of these analogies work and
[102:56] they shouldn't, but somehow, remarkably, they do.
A lot of the smaller models, or the dumber models,
[103:01] remarkably resemble a kindergarten student, or an
elementary school student or high school student.
[103:07] Somehow, we still haven't graduated
enough where this stuff can take over.
[103:12] My Claude Code or Codex, they still
feel like this elementary-grade student.
[103:17] I know that they can take PhD quizzes,
but they still cognitively feel like a
[103:21] kindergarten or an elementary school student.
I don't think they can create culture because
[103:24] they're still kids. They're savant kids.
They have perfect memory of all this stuff.
[103:33] They can convincingly create all
kinds of slop that looks really good.
[103:36] But I still think they don't really know
what they're doing and they don't really
[103:38] have the cognition across all these little
checkboxes that we still have to collect.
[103:43] You've talked about how you were at Tesla
leading self-driving from 2017 to 2022.
[103:50] And you firsthand saw this progress from cool
demos to now thousands of cars out there actually
[103:58] autonomously doing drives.
Why did that take a decade?
[104:01] What was happening through that time?
One thing I will almost instantly push
[104:06] back on is that this is not even near done,
in a bunch of ways that I'm going to get to.
[104:13] Self-driving is very interesting because
it's definitely where I get a lot of my
[104:16] intuitions because I spent five years on it.
It has this entire history where the first demos
[104:22] of self-driving go all the way to the 1980s.
You can see a demo from CMU in 1986.
[104:28] There's a truck that's driving itself on
roads. Fast forward. When I was joining Tesla,
[104:34] I had a very early demo of Waymo.
It basically gave me a perfect drive
[104:40] in 2014 or something like that, so
a perfect Waymo drive a decade ago.
[104:46] It took us around Palo Alto and so on
because I had a friend who worked there.
[104:50] I thought it was very close and
then it still took a long time.
[104:54] For some kinds of tasks and jobs and so on,
there's a very large demo-to-product gap where the
[105:02] demo is very easy, but the product is very hard.
It's especially the case in cases like
[105:07] self-driving where the cost
of failure is too high.
[105:11] Many industries, tasks, and jobs maybe don't have
that property, but when you do have that property,
[105:15] that definitely increases the timelines.
For example, in software engineering,
[105:19] I do think that property does exist.
For a lot of vibe coding, it doesn't.
[105:24] But if you're writing actual production-grade
code, that property should exist, because any
[105:28] kind of mistake leads to a security
vulnerability or something like that.
[105:32] Millions and hundreds of millions of
people's personal Social Security numbers
[105:36] get leaked or something like that.
So in software, people should be careful,
[105:42] kind of like in self-driving.
In self-driving, if things go wrong,
[105:46] you might get injured. There are worse
outcomes. But in software, it's almost
[105:52] unbounded how terrible something could be.
I do think that they share that property.
[105:59] What takes the long amount of time and the way
to think about it is that it's a march of nines.
[106:04] Every single nine is a constant amount of work.
Every single nine is the same amount of work.
[106:10] When you get a demo and something works 90%
of the time, that's just the first nine.
[106:16] Then you need the second nine, a third
nine, a fourth nine, a fifth nine.
[106:18] While I was at Tesla for five years or so, we
went through maybe three nines or two nines.
[106:23] I don't know what it is, but
multiple nines of iteration.
[106:25] There are still more nines to go.
That's why these things take so long.
[106:31] It's definitely formative for me, seeing
something that was a demo. I'm very
[106:35] unimpressed by demos. Whenever I see demos of
anything, I'm extremely unimpressed by that.
[106:43] If it's a demo that someone cooked
up as a showing, it's worse.
[106:45] If you can interact with it, it's a bit better.
[106:47] But even then, you're not done. You need the
actual product. It's going to face all these
[106:50] challenges when it comes in contact
with reality and all these different
[106:53] pockets of behavior that need patching.
We're going to see all this stuff play
[106:57] out. It's a march of nines. Each nine is
constant. Demos are encouraging. It’s still
[107:01] a huge amount of work to do.
It is a critical safety domain,
[107:07] unless you're doing vibe coding,
which is all nice and fun and so on.
[107:11] That's why this also enforced my
timelines from that perspective.
[107:16] It's very interesting to hear you say that, that
the safety guarantees you need from software
[107:20] are not dissimilar to self-driving.
What people will often say is that
[107:24] self-driving took so long because
the cost of failure is so high.
[107:30] A human makes a mistake on average every
400,000 miles or every seven years.
[107:34] If you had to release a coding agent that
couldn't make a mistake for at least seven years,
[107:39] it would be much harder to deploy.
But your point is that if you made a
[107:43] catastrophic coding mistake, like breaking
some important system every seven years...
[107:47] Very easy to do.
In fact, in terms of wall clock time,
[107:51] it would be much less than seven years because
you're constantly outputting code like that.
[107:57] In terms of tokens, it would be seven years.
But in terms of wall clock time...
[108:00] In some ways, it's a much harder problem.
Self-driving is just one of
[108:02] thousands of things that people do.
It's almost like a single vertical, I suppose.
[108:07] Whereas when we're talking about
general software engineering,
[108:08] it's even more... There's more surface area.
There's another objection people make to that
[108:14] analogy, which is that with self-driving, what
took a big fraction of that time was solving
[108:21] the problem of having basic perception
that's robust, building representations,
[108:27] and having a model that has some common
sense so it can generalize to when it sees
[108:32] something that's slightly out of distribution.
If somebody's waving down the road this way,
[108:37] you don't need to train for it.
The thing will have some understanding
[108:40] of how to respond to something like that.
These are things we're getting for free
[108:44] with LLMs or VLMs today, so we don't have to
solve these very basic representation problems.
[108:49] So now deploying AIs across different domains
will sort of be like deploying a self-driving
[108:54] car with current models to a different city,
which is hard but not like a 10-year-long task.
[108:59] I'm not 100% sure if I fully agree with that.
I don't know how much we're getting for free.
[109:03] There's still a lot of gaps in
understanding what we are getting.
[109:07] We're definitely getting more
generalizable intelligence in a
[109:10] single entity, whereas self-driving is a
very special-purpose task that requires.
[109:14] In some sense building a special-purpose task
is maybe even harder in a certain sense because
[109:18] it doesn't fall out from a more general thing
that you're doing at scale, if that makes sense.
[109:24] But the analogy still doesn't fully
resonate because the LLMs are still
[109:30] pretty fallible and they have a lot of
gaps that still need to be filled in.
[109:33] I don't think that we're
getting magical generalization
[109:35] completely out of the box, in a certain sense.
The other aspect that I wanted to return to is
[109:42] that self-driving cars are nowhere near done
still. The deployments are pretty minimal.
[109:51] Even Waymo and so on has very few cars.
They're doing that roughly speaking
[109:54] because they're not economical.
They've built something that lives in the future.
[110:00] They've had to pull back the future,
but they had to make it uneconomical.
[110:05] There are all these costs, not just
marginal costs for those cars and
[110:09] their operation and maintenance, but
also the capex of the entire thing.
[110:13] Making it economical is still
going to be a slog for them.
[110:17] Also, when you look at these cars and
there's no one driving, I actually think
[110:21] it's a little bit deceiving because there
are very elaborate teleoperation centers
[110:26] of people kind of in a loop with these cars.
I don't have the full extent of it, but there's
[110:32] more human-in-the-loop than you might expect.
There are people somewhere out there
[110:36] beaming in from the sky.
I don't know if they're
[110:39] fully in the loop with the driving.
Some of the time they are, but they're
[110:42] certainly involved and there are people.
In some sense, we haven't actually removed
[110:45] the person, we've moved them to
somewhere where you can't see them.
[110:48] I still think there will be some work, as you
mentioned, going from environment to environment.
[110:52] There are still challenges
to make self-driving real.
[110:55] But I do agree that it's definitely crossed
a threshold where it kind of feels real,
[110:59] unless it's really teleoperated.
For example, Waymo can't go to
[111:03] all the different parts of the city.
My suspicion is that it's parts of the city
[111:07] where you don't get good signal.
Anyway, I don't know anything
[111:11] about the stack. I'm just making stuff up.
You led self-driving for five years at Tesla.
[111:17] Sorry, I don't know anything
about the specifics of Waymo.
[111:21] By the way, I love Waymo
and I take it all the time.
[111:24] I just think that people are sometimes a
little bit too naive about some of the progress
[111:29] and there's still a huge amount of work.
Tesla took in my mind a much more scalable
[111:33] approach and the team is doing extremely well.
I'm kind of on the record for predicting
[111:39] how this thing will go.
Waymo had an early start
[111:42] because you can package up so many sensors.
But I do think Tesla is taking the more
[111:45] scalable strategy and it's going
to look a lot more like that.
[111:48] So this will still have to play out and hasn't.
But I don't want to talk about self-driving as
[111:54] something that took a decade because it
didn't take it yet, if that makes sense.
[111:59] Because one, the start is at 1980 and not 10
years ago, and then two, the end is not here yet.
[112:05] The end is not near yet because when
we're talking about self-driving,
[112:08] usually in my mind it's self-driving at scale.
People don't have to get a driver's license, etc.
[112:13] I'm curious to bounce two other ways in
which the analogy might be different.
[112:19] The reason I'm especially curious about this is
because the question of how fast AI is deployed,
[112:24] how valuable it is when it's early
on is potentially the most important
[112:28] question in the world right now.
If you're trying to model what the
[112:31] year 2030 looks like, this is the question
you ought to have some understanding of.
[112:36] Another thing you might think is one, you have
this latency requirement with self-driving.
[112:42] I have no idea what the actual models are, but I
assume it’s like tens of millions of parameters
[112:45] or something, which is not the necessary
constraint for knowledge work with LLMs.
[112:51] Maybe it might be with computer use and stuff.
But the other big one is, maybe more
[112:56] importantly, on this capex question.
Yes, there is additional cost to serving
[113:02] up an additional copy of a model, but the opex
of a session is quite low and you can amortize
[113:12] the cost of AI into the training run itself,
depending on how inference scaling goes and stuff.
[113:17] But it's certainly not as much as building a whole
new car to serve another instance of a model.
[113:23] So the economics of deploying more
widely are much more favorable.
[113:28] I think that's right. If you're
sticking to the realm of bits,
[113:31] bits are a million times easier than anything
that touches the physical world. I definitely
[113:36] grant that. Bits are completely changeable,
arbitrarily reshuffleable at a very rapid speed.
[113:42] You would expect a much faster adaptation also in
the industry and so on. What was the first one?
[113:50] The latency requirements and
its implications for model size?
[113:53] I think that's roughly right. I also
think that if we are talking about
[113:56] knowledge work at scale, there will be some
latency requirements, practically speaking,
[114:00] because we're going to have to create a
huge amount of compute and serve that.
[114:06] The last aspect that I very briefly want
to also talk about is all the rest of it.
[114:13] What does society think about it? What are
the legal ramifications? How is it working
[114:17] legally? How is it working insurance-wise?
What are those layers of it and aspects of it?
[114:25] What is the equivalent of people
putting a cone on a Waymo?
[114:28] There are going to be equivalents of all that.
So I feel like self-driving is a very nice
[114:34] analogy that you can borrow things from.
What is the equivalent of a cone in the car?
[114:38] What is the equivalent of a teleoperating worker
who's hidden away and all the aspects of it.
[114:45] Do you have any opinions on what this
implies about the current AI buildout,
[114:49] which would 10x the amount of available compute
in the world in a year or two and maybe more
[114:56] than 100x it by the end of the decade.
If the use of AI will be lower than
[115:00] some people naively predict, does
that mean that we're overbuilding
[115:04] compute or is that a separate question?
Kind of like what happened with railroads.
[115:08] With what, sorry?
Was it railroads or?
[115:10] Yeah, it was.
Yeah. There's historical precedent.
[115:14] Or was it with the telecommunication industry?
Pre-paving the internet that only came a decade
[115:18] later and creating a whole bubble in the
telecommunications industry in the late '90s.
[115:28] I understand I'm sounding very pessimistic
here. I'm actually optimistic. I think this
[115:33] will work. I think it's tractable. I'm
only sounding pessimistic because when
[115:36] I go on my Twitter timeline, I see all
this stuff that makes no sense to me.
[115:42] There's a lot of reasons for why that exists.
A lot of it is honestly just fundraising.
[115:47] It's just incentive structures.
A lot of it may be fundraising.
[115:50] A lot of it is just attention, converting
attention to money on the internet,
[115:55] stuff like that.
There's a lot of
[116:00] that going on, and I'm only reacting to that.
But I'm still overall very bullish on technology.
[116:05] We're going to work through all this stuff.
There's been a rapid amount of progress.
[116:09] I don't know that there's overbuilding.
I think we're going to be able to gobble up what,
[116:15] in my understanding, is being built.
For example, Claude Code or OpenAI Codex
[116:20] and stuff like that didn't even
exist a year ago. Is that right?
[116:24] This is a miraculous technology that didn't exist.
There's going to be a huge amount of demand,
[116:29] as we see the demand in ChatGPT already and so on.
So I don't know that there's overbuilding.
[116:37] I'm just reacting to some of the very fast
timelines that people continue to say incorrectly.
[116:42] I've heard many, many times over the course
of my 15 years in AI where very reputable
[116:46] people keep getting this wrong all the time.
I want this to be properly calibrated, and some
[116:53] of this also has geopolitical ramifications and
things like that with some of these questions.
[116:59] I don't want people to make
mistakes in that sphere of things.
[117:04] I do want us to be grounded in the
reality of what technology is and isn't.
[117:08] Let's talk about education and Eureka.
One thing you could do is start another AI
[117:15] lab and then try to solve those problems.
I’m curious what you're up to now,
[117:21] and why not AI research itself?
I guess the way I would put it
[117:26] is I feel some amount of determinism
around the things that AI labs are doing.
[117:33] I feel like I could help out there, but I
don't know that I would uniquely improve it.
[117:42] My personal big fear is that a lot of this
stuff happens on the side of humanity,
[117:46] and that humanity gets disempowered by it.
I care not just about all the Dyson spheres
[117:52] that we're going to build and that AI is
going to build in a fully autonomous way,
[117:55] I care about what happens to humans.
I want humans to be well off in the future.
[118:00] I feel like that's where I can a
lot more uniquely add value than
[118:04] an incremental improvement in the frontier lab.
I'm most afraid of something depicted in movies
[118:11] like WALL-E or Idiocracy or something like that,
where humanity is on the side of this stuff.
[118:16] I want humans to be much,
much better in this future.
[118:21] To me, this is through education
that you can achieve this.
[118:26] So what are you working on there?
The easiest way I can describe it is
[118:30] we're trying to build the Starfleet Academy.
I don’t know if you’ve watched Star Trek.
[118:34] I haven’t.
Starfleet Academy is
[118:37] this elite institution for frontier technology,
building spaceships, and graduating cadets to be
[118:43] the pilots of these spaceships and whatnot.
So I just imagine an elite institution for
[118:47] technical knowledge and a kind of school that's
very up-to-date and a premier institution.
[118:56] A category of questions I have for you is
explaining how one teaches technical or
[119:03] scientific content well, because you
are one of the world masters at it.
[119:08] I'm curious both about how you think about
it for content you've already put out there
[119:11] on YouTube, but also, to the extent it's any
different, how you think about it for Eureka.
[119:16] With respect to Eureka, one thing that
is very fascinating to me about education
[119:20] is that I do think education will pretty
fundamentally change with AIs on the side.
[119:24] It has to be rewired and changed to some extent.
I still think that we're pretty early.
[119:30] There's going to be a lot of people who
are going to try to do the obvious things.
[119:33] Have an LLM and ask it questions.
Do all the basic things that you would
[119:38] do via prompting right now.
It's helpful,
[119:40] but it still feels to me a bit like slop.
I'd like to do it properly, and I think the
[119:44] capability is not there for what I would want.
What I'd want is an actual tutor experience.
[119:51] A prominent example in my mind is I was
recently learning Korean, so language learning.
[119:57] I went through a phase where I was
learning Korean by myself on the internet.
[120:00] I went through a phase where I was part of a small
class in Korea taking Korean with a bunch of other
[120:06] people, which was really funny.
We had a teacher and 10
[120:08] people or so taking Korean.
Then I switched to a one-on-one tutor.
[120:13] I guess what was fascinating to me was, I think
I had a really good tutor, but just thinking
[120:19] through what this tutor was doing for me and how
incredible that experience was and how high the
[120:25] bar is for what I want to build eventually.
Instantly from a very short
[120:31] conversation, she understood where I am
as a student, what I know and don't know.
[120:35] She was able to probe exactly the kinds of
questions or things to understand my world model.
[120:41] No LLM will do that for you
100% right now, not even close.
[120:44] But a tutor will do that if they're good.
Once she understands, she really served
[120:49] me all the things that I needed at
my current sliver of capability.
[120:52] I need to be always appropriately challenged.
I can't be faced with something too hard or
[120:56] too trivial, and a tutor is really good
at serving you just the right stuff.
[121:01] I felt like I was the only constraint to learning.
I was always given the perfect information. I'm
[121:07] the only constraint. I felt good because
I'm the only impediment that exists.
[121:11] It's not that I can't find knowledge or
that it's not properly explained or etc.
[121:14] It's just my ability to memorize and so on.
This is what I want for people.
[121:18] How do you automate that?
Very good question. At
[121:21] the current capability, you don't.
That's why I think it's not actually the
[121:26] right time to build this kind of an AI tutor.
I still think it's a useful product,
[121:31] and lots of people will build it, but the bar
is so high and the capability is not there.
[121:40] Even today, I would say ChatGPT is an
extremely valuable educational product.
[121:45] But for me, it was so fascinating
to see how high the bar is.
[121:48] When I was with her, I almost felt
like there's no way I can build this.
[121:53] But you are building it, right?
Anyone who's had a really good
[121:55] tutor is like, "How are you going to build
this?" I'm waiting for that capability. I
[122:04] did some AI consulting for computer vision.
A lot of times, the value that I brought to
[122:09] the company was telling them not to use AI.
I was the AI expert, and they described the
[122:13] problem, and I said, "Don't use AI."
This is my value add. I feel like it's
[122:17] the same in education right now, where
I feel like for what I have in mind,
[122:21] it's not yet the time, but the time will come.
For now, I'm building something that looks
[122:26] maybe a bit more conventional that has a
physical and digital component and so on.
[122:30] But it's obvious how this
should look in the future.
[122:35] To the extent you're willing to
say, what is the thing you hope
[122:37] will be released this year or next year?
I'm building the first course. I want to
[122:43] have a really, really good course, the
obvious state-of-the-art destination
[122:48] you go to to learn, AI in this case.
That's just what I'm familiar with, so it's
[122:51] a really good first product to get to be really
good at it. So that's what I'm building. Nanochat,
[122:56] which you briefly mentioned, is a capstone project
of LLM101N, which is a class that I'm building.
[123:02] That's a really big piece of it.
But now I have to build out a lot of
[123:04] the intermediates, and then I have to hire a small
team of TAs and so on and build the entire course.
[123:11] One more thing that I would say is that many
times, when people think about education,
[123:15] they think more about what I would say is
a softer component of diffusing knowledge.
[123:21] I have something very hard and technical in mind.
In my mind, education is the very difficult
[123:26] technical process of building ramps to knowledge.
In my mind, nanochat is a ramp to
[123:33] knowledge because it's very simple.
It's the super simplified full-stack thing.
[123:38] If you give this artifact to someone and they
look through it, they're learning a ton of stuff.
[123:43] It's giving you a lot of what I call eurekas
per second, which is understanding per second.
[123:48] That's what I want, lots of eurekas per second.
So to me, this is a technical problem of
[123:52] how do we build these ramps to knowledge.
So I almost think of Eureka as maybe not that
[123:58] different from some of the frontier labs
or some of the work that's going on there.
[124:03] I want to figure out how to build these
ramps very efficiently so that people are
[124:07] never stuck and everything is always
not too hard or not too trivial, and
[124:14] you have just the right material to progress.
You're imagining in the short term that instead
[124:18] of a tutor being able to probe your understanding,
if you have enough self-awareness to be able to
[124:24] probe yourself, you're never going to be stuck.
You can find the right answer between talking
[124:29] to the TA or talking to an LLM and
looking at the reference implementation.
[124:33] It sounds like automation or
AI is not a significant part.
[124:38] So far, the big alpha here is your
ability to explain AI codified
[124:46] in the source material of the class.
That's fundamentally what the course is.
[124:51] You always have to be calibrated to
what capability exists in the industry.
[124:55] A lot of people are going to
pursue just asking ChatGPT, etc.
[124:59] But I think right now, for example, if you go to
ChatGPT and you say, teach me AI, there's no way.
[125:03] It's going to give you some slop.
AI is never going to write nanochat right now.
[125:09] But nanochat is a really
useful intermediate point.
[125:13] I'm collaborating with AI
to create all this material,
[125:15] so AI is still fundamentally very helpful.
Earlier on, I built CS231n at Stanford,
[125:21] which I think was the first deep learning
class at Stanford, which became very popular.
[125:27] The difference in building out 231n
then and LLM101N now is quite stark.
[125:33] I feel really empowered by the LLMs as they
exist right now, but I'm very much in the loop.
[125:37] They're helping me build the
materials, I go much faster.
[125:40] They're doing a lot of the boring stuff, etc.
I feel like I'm developing the course much faster,
[125:45] and it's LLM-infused, but it's not yet at a
place where it can creatively create the content.
[125:50] I'm still there to do that.
The trickiness is always
[125:53] calibrating yourself to what exists.
When you imagine what is available
[125:57] through Eureka in a couple of years, it
seems like the big bottleneck is going to be
[126:02] finding Karpathys in field after field who can
convert their understanding into these ramps.
[126:09] It would change over time. Right now,
it would be hiring faculty to help work
[126:14] hand-in-hand with AI and a team of people
probably to build state-of-the-art courses.
[126:21] Over time maybe some of the TAs can become AIs.
You just take all the course materials and then
[126:28] I think you could serve a very good automated
TA for the student when they have more basic
[126:33] questions or something like that.
But I think you'll need faculty
[126:36] for the overall architecture of a
course and making sure that it fits.
[126:40] So I see a progression of how this will evolve.
Maybe at some future point I'm not even that
[126:45] useful and AI is doing most of the
design much better than I could.
[126:47] But I still think that's going
to take some time to play out.
[126:50] Are you imagining that people who have expertise
in other fields are then contributing courses,
[126:56] or do you feel like it's quite
essential to the vision that you,
[127:01] given your understanding of how you want to
teach, are the one designing the content?
[127:07] Sal Khan is narrating all
the videos on Khan Academy.
[127:09] Are you imagining something like that?
No, I will hire faculty because there
[127:12] are domains in which I'm not an expert.
That's the only way to offer the state-of-the-art
[127:18] experience for the student ultimately.
I do expect that I would hire faculty, but
[127:24] I will probably stick around in AI for some time.
I do have something more conventional in mind for
[127:29] the current capability than what
people would probably anticipate.
[127:33] When I'm building Starfleet Academy, I do probably
imagine a physical institution, and maybe a tier
[127:38] below that a digital offering that is not the
state-of-the-art experience you would get when
[127:44] someone comes in physically full-time and we
work through material from start to end and
[127:48] make sure you understand it. That's the physical
offering. The digital offering is a bunch of stuff
[127:53] on the internet and maybe some LLM assistant.
It's a bit more gimmicky in a tier below, but
[127:57] at least it's accessible to 8 billion people.
I think you're basically inventing college
[128:04] from first principles for the tools that
are available today and just selecting
[128:11] for people who have the motivation and the
interest of really engaging with material.
[128:18] There's going to have to be a lot of not
just education but also re-education.
[128:21] I would love to help out there because
the jobs will probably change quite a bit.
[128:27] For example, today a lot of people are
trying to upskill in AI specifically.
[128:29] I think it's a really good
course to teach in this respect.
[128:34] Motivation-wise, before AGI motivation is very
simple to solve because people want to make money.
[128:41] This is how you make money in the industry today.
Post-AGI is a lot more interesting possibly
[128:46] because if everything is automated
and there's nothing to do for anyone,
[128:49] why would anyone go to a school?
I often say that pre-AGI education
[128:57] is useful. Post-AGI education is fun. In
a similar way, people go to the gym today.
[129:06] We don't need their physical strength
to manipulate heavy objects because we
[129:10] have machines that do that.
They still go to the gym.
[129:12] Why do they go to the gym?
Because it's fun, it's healthy,
[129:16] and you look hot when you have a six-pack.
It's attractive for people to do that
[129:24] in a very deep, psychological,
evolutionary sense for humanity.
[129:30] Education will play out in the same way.
You'll go to school like you go to the gym.
[129:36] Right now, not that many people learn
because learning is hard. You bounce
[129:40] from material. Some people overcome that
barrier, but for most people, it's hard.
[129:46] It's a technical problem to solve.
It's a technical problem to do what my tutor
[129:50] did for me when I was learning Korean.
It's tractable and buildable,
[129:53] and someone should build it.
It's going to make learning
[129:55] anything trivial and desirable, and people
will do it for fun because it's trivial.
[130:00] If I had a tutor like that for any arbitrary piece
of knowledge, it's going to be so much easier to
[130:05] learn anything, and people will do it.
They'll do it for the same
[130:07] reasons they go to the gym.
That sounds different from using…
[130:14] So post-AGI, you're using this as
entertainment or as self-betterment.
[130:21] But it sounded like you had a vision
also that this education is relevant to
[130:25] keeping humanity in control of AI. That sounds
different. Is it entertaining for some people,
[130:30] but then empowerment for some others?
How do you think about that?
[130:32] I do think eventually it's a bit of
a losing game, if that makes sense.
[130:41] It is in the long term.
In the long term, which
[130:44] is longer than maybe most people in the
industry think about, it's a losing game.
[130:47] I do think people can go so far and we've barely
scratched the surface of how much a person can go.
[130:53] That's just because people are bouncing off
of material that's too easy or too hard.
[130:59] People will be able to go much further.
Anyone will speak five languages because
[131:03] why not? Because it's so trivial. Anyone will know
all the basic curriculum of undergrad, et cetera.
[131:09] Now that I'm understanding the
vision, that's very interesting.
[131:14] It has a perfect analog in gym culture.
I don't think 100 years
[131:18] ago anybody would be ripped.
Nobody would have been able to just spontaneously
[131:22] bench two plates or three plates or something.
It's very common now because of this idea of
[131:29] systematically training and lifting weights in
the gym, or systematically training to be able
[131:33] to run a marathon, which is a capability
most humans would not spontaneously have.
[131:38] You're imagining similar things for
learning across many different domains,
[131:43] much more intensely, deeply, faster.
Exactly. I am betting a bit implicitly
[131:48] on some of the timelessness of human nature.
It will be desirable to do all these things,
[131:58] and I think people will look up
to it as they have for millennia.
[132:03] This will continue to be true.
There's some evidence of that historically.
[132:07] If you look at, for example, aristocrats, or you
look at ancient Greece or something like that,
[132:11] whenever you had little pocket environments
that were post-AGI in a certain sense, people
[132:15] have spent a lot of their time flourishing in a
certain way, either physically or cognitively.
[132:22] I feel okay about the prospects of that.
If this is false and I'm wrong and we end up in a
[132:29] WALL-E or Idiocracy future, then I don't even care
if there are Dyson spheres. This is a terrible
[132:35] outcome. I really do care about humanity.
Everyone has to just be
[132:41] superhuman in a certain sense.
It's still a world in which that is not enabling
[132:46] us to… It's like the culture world, right?
You're not fundamentally going to be able
[132:51] to transform the trajectory
of technology or influence
[132:57] decisions by your own labor or cognition alone.
Maybe you can influence decisions because the AI
[133:03] is asking for your approval, but it's not because
I've invented something or I've come up with a new
[133:10] design that I'm really influencing the future.
Maybe. I think there will be a transitional
[133:15] period where we are going to be
able to be in the loop and advance
[133:19] things if we understand a lot of stuff.
In the long-term, that probably goes away.
[133:25] It might even become a sport.
Right now you have powerlifters
[133:28] who go extreme in this direction.
What is powerlifting in a cognitive era?
[133:33] Maybe it's people who are really trying
to make Olympics out of knowing stuff.
[133:39] If you have a perfect AI tutor,
maybe you can get extremely far.
[133:43] I feel that the geniuses of
today are barely scratching the
[133:48] surface of what a human mind can do, I think.
I love this vision. I also feel like the person
[133:55] you have the most product-market fit with is me
because my job involves having to learn different
[134:02] subjects every week, and I am very excited.
I'm similar, for that matter. A lot of people,
[134:10] for example, hate school and want to get
out of it. I really liked school. I loved
[134:15] learning things, et cetera.
I wanted to stay in school.
[134:16] I stayed all the way until Ph.D. and
then they wouldn't let me stay longer,
[134:19] so I went to the industry.
Roughly speaking, I love learning,
[134:25] even for the sake of learning, but I also love
learning because it's a form of empowerment and
[134:29] being useful and productive.
You also made a point that
[134:32] was subtle and I want to spell it out.
With what’s happened so far with online
[134:36] courses, why haven't they already enabled us to
enable every single human to know everything?
[134:44] They're just so motivation-laden because there are
no obvious on-ramps and it's so easy to get stuck.
[134:52] If you had this thing instead—like a really
good human tutor—it would just be such an
[135:00] unlock from a motivation perspective.
I think so. It feels bad to bounce from
[135:04] material. It feels bad. You get negative reward
from sinking an amount of time in something and it
[135:09] doesn't pan out, or being completely bored because
what you're getting is too easy or too hard.
[135:16] When you do it properly, learning feels good.
It's a technical problem to get there.
[135:21] For a while, it's going to be AI plus human
collab, and at some point, maybe it's just AI.
[135:27] Can I ask some questions about teaching well?
If you had to give advice to another educator
[135:32] in another field that you're curious about to
make the kinds of YouTube tutorials you've made.
[135:40] Maybe it might be especially interesting
to talk about domains where you can't
[135:43] test someone's technical understanding by
having them code something up or something.
[135:47] What advice would you give them?
That's a pretty broad topic. There are 10–20 tips
[135:54] and tricks that I semi-consciously do probably.
But a lot of this comes
[136:03] from my physics background.
I really, really did enjoy my physics background.
[136:06] I have a whole rant on how everyone
should learn physics in early school
[136:10] education because early school education is
not about accumulating knowledge or memory
[136:15] for tasks later in the industry.
It's about booting up a brain.
[136:18] Physics uniquely boots up the brain the
best because some of the things that they
[136:22] get you to do in your brain during
physics is extremely valuable later.
[136:26] The idea of building models and abstractions
and understanding that there's a first-order
[136:31] approximation that describes most of the system,
but then there're second-order, third-order,
[136:34] fourth-order terms that may or may not be present.
The idea that you're observing a very noisy
[136:39] system, but there are these fundamental
frequencies that you can abstract away.
[136:43] When a physicist walks into the class and
they say, "Assume there's a spherical cow,"
[136:48] everyone laughs at that, but this is brilliant.
It's brilliant thinking that's very generalizable
[136:53] across the industry because a cow can be
approximated as a sphere in a bunch of ways.
[136:58] There's a really good book, for example, Scale.
It's from a physicist talking about biology.
[137:04] Maybe this is also a book
I would recommend reading.
[137:06] You can get a lot of really interesting
approximations and chart scaling laws of animals.
[137:11] You can look at their heartbeats and
things like that, and they line up with
[137:15] the size of the animal and things like that.
You can talk about an animal as a volume.
[137:20] You can talk about the heat dissipation of that,
because your heat dissipation grows as the surface
[137:25] area, which is growing as a square.
But your heat creation or generation
[137:29] is growing as a cube.
So I just feel like physicists
[137:33] have all the right cognitive tools to
approach problem solving in the world.
[137:36] So because of that training, I
always try to find the first-order
[137:39] terms or the second-order terms of everything.
When I'm observing a system or a thing, I have a
[137:43] tangle of a web of ideas or knowledge in my mind.
I'm trying to find, what is the thing that
[137:48] matters? What is the first-order component?
How can I simplify it? How can I have a
[137:52] simplest thing that shows that thing, shows it in
action, and then I can tack on the other terms?
[137:58] Maybe an example from one of my repos that I
think illustrates it well is called micrograd.
[138:03] I don't know if you're familiar with this.
So micrograd is 100 lines of code
[138:06] that shows backpropagation.
You can create neural networks
[138:10] out of simple operations like plus and times, et
cetera. Lego blocks of neural networks. You build
[138:14] up a computational graph and you do a forward
pass and a backward pass to get the gradients.
[138:19] Now, this is at the heart of
all neural network learning.
[138:21] So micrograd is a 100 lines of
pretty interpretable Python code,
[138:25] and it can do forward and backward arbitrary
neural networks, but not efficiently.
[138:29] So micrograd, these 100 lines of Python,
are everything you need to understand how
[138:32] neural networks train. Everything else is just
efficiency. Everything else is efficiency. There's
[138:38] a huge amount of work to get efficiency.
You need your tensors, you lay them out,
[138:41] you stride them, you make sure
your kernels, orchestrating
[138:43] memory movement correctly, et cetera.
It's all just efficiency, roughly speaking.
[138:47] But the core intellectual piece of neural
network training is micrograd. It's 100 lines.
[138:50] You can easily understand it. It's a recursive
application of chain rule to derive the gradient,
[138:55] which allows you to optimize any
arbitrary differentiable function.
[138:58] So I love finding these small-order terms and
serving them on a platter and discovering them.
[139:06] I feel like education is the most intellectually
interesting thing because you have a tangle
[139:11] of understanding and you're trying to lay
it out in a way that creates a ramp where
[139:16] everything only depends on the thing before it.
I find that this untangling of knowledge is just
[139:21] so intellectually interesting as a cognitive task.
I love doing it personally, but I just
[139:27] have a fascination with trying to lay things
out in a certain way. Maybe that helps me.
[139:31] It also makes the learning
experience so much more motivated.
[139:35] Your tutorial on the transformer begins
with bigrams, literally a lookup table from,
[139:42] "Here's the word right now, or here's
the previous word, here's the next word."
[139:46] It's literally just a lookup table.
That’s the essence of it, yeah.
[139:48] It’s such a brilliant way, starting with a
lookup table and then going to a transformer.
[139:53] Each piece is motivated. Why would you add
that? Why would you add the next thing?
[139:57] You could memorize the attention formula,
but having an understanding of why every
[140:01] single piece is relevant, what problem it solves.
You're presenting the pain before you present a
[140:06] solution, and how clever is that?
You want to take the student
[140:08] through that progression.
There are a lot of other small
[140:11] things that make it nice and engaging and
interesting. Always prompting the student.
[140:17] There's a lot of small things like that are
important and a lot of good educators will do
[140:22] this. How would you solve this? I'm not going to
present the solution before you guess. That would
[140:27] be wasteful. That's a little bit of a…I don’t
want to swear but it’s a dick move towards you
[140:34] to present you with the solution before I give
you a shot to try to come up with it yourself.
[140:38] Because if you try to come up with it yourself,
you get a better understanding of what the action
[140:46] space is, what the objective is, and then
why only this action fulfills that objective.
[140:53] You have a chance to try it yourself, and you
have an appreciation when I give you the solution.
[140:58] It maximizes the amount of
knowledge per new fact added.
[141:02] Why do you think, by default, people who are
genuine experts in their field are often bad
[141:10] at explaining it to somebody ramping up?
It's the curse of knowledge and expertise.
[141:16] This is a real phenomenon, and I suffered
from it myself as much as I try not to.
[141:21] But you take certain things for granted,
and you can't put yourself in the shoes
[141:24] of new people who are just starting out.
This is pervasive and happens to me as
[141:28] well. One thing that's extremely helpful.
As an example, someone was trying to show
[141:32] me a paper in biology recently, and I just
instantly had so many terrible questions.
[141:38] What I did was I used ChatGPT to ask the
questions with the paper in the context window.
[141:43] It worked through some of the simple things.
Then I shared the thread to the person who
[141:49] wrote that paper or worked on that work.
I felt like if they could see the dumb
[141:54] questions I had, it might help
them explain better in the future.
[142:00] For my material, I would love it if people
shared their dumb conversations with ChatGPT
[142:04] about the stuff that I've created
because it really helps me put myself
[142:07] again in the shoes of someone who's starting out.
Another trick that just works astoundingly well.
[142:16] If somebody writes a paper or a blog post or an
announcement, it is in 100% of cases that just
[142:25] the narration or the transcription of how they
would explain it to you over lunch is way more,
[142:33] not only understandable, but actually
also more accurate and scientific,
[142:39] in the sense that people have a bias
to explain things in the most abstract,
[142:44] jargon-filled way possible and to clear
their throat for four paragraphs before
[142:48] they explain the central idea.
But there's something about
[142:51] communicating one-on-one with a person
which compels you to just say the thing.
[142:57] Just say the thing. I saw that
tweet, I thought it was really good.
[143:00] I shared it with a bunch of people.
I noticed this many, many times.
[143:06] The most prominent example is that I
remember back in my PhD days doing research.
[143:11] You read someone's paper, and you
work to understand what it's doing.
[143:15] Then you catch them, you're having beers
at the conference later, and you ask them,
[143:18] "So this paper, what were you doing? What is the
paper about?" They will just tell you these three
[143:23] sentences that perfectly captured the essence
of that paper and totally give you the idea.
[143:26] And you didn't have to read the paper.
It's only when you're sitting at the table
[143:30] with a beer or something, and they're
like, "Oh yeah, the paper is just,
[143:33] you take this idea, you take that idea and try
this experiment and you try out this thing."
[143:37] They have a way of just putting it
conversationally just perfectly.
[143:40] Why isn't that the abstract?
Exactly. This is coming from the
[143:47] perspective of how somebody who's trying to
explain an idea should formulate it better.
[143:51] What is your advice as a student to other
students, if you don't have a Karpathy
[143:57] who is doing the exposition of an idea?
If you're reading a paper from somebody
[144:00] or reading a book, what strategies do
you employ to learn material you're
[144:07] interested in in fields you're not an expert at?
I don't know that I have unique tips and tricks,
[144:13] to be honest. It's a painful process. One thing
that has always helped me quite a bit is—I
[144:26] had a small tweet about this—learning things
on demand is pretty nice. Learning depth-wise.
[144:31] I do feel you need a bit of alternation of
learning depth-wise, on demand—you're trying
[144:35] to achieve a certain project that you're going
to get a reward from—and learning breadth-wise,
[144:38] which is just, "Oh, let's do whatever 101,
and here's all the things you might need."
[144:42] Which is a lot of school—does breadth-wise
learning, like, "Oh, trust me, you'll need
[144:45] this later," that kind of stuff. Okay, I trust
you. I'll learn it because I guess I need it.
[144:50] But I love the kind of learning
where you'll get a reward out of
[144:53] doing something, and you're learning on demand.
The other thing that I've found extremely helpful.
[144:59] This is an aspect where education is a bit more
selfless, but explaining things to people is a
[145:04] beautiful way to learn something more deeply.
This happens to me all the time.
[145:09] It probably happens to other people too because
I realize if I don't really understand something,
[145:13] I can't explain it.
I'm trying and I'm like,
[145:17] "Oh, I don't understand this."
It's so annoying to come to terms with that.
[145:21] You can go back and make sure you understood it.
It fills these gaps of your understanding.
[145:25] It forces you to come to terms
with them and to reconcile them.
[145:28] I love to re-explain things and people
should be doing that more as well.
[145:33] That forces you to manipulate the knowledge
and make sure that you know what you're
[145:36] talking about when you're explaining it.
That's an excellent note to close on. Andrej,
[145:40] that was great.
Thank you.
