Jang Says AlphaGo Search Solves AI Learning Bottleneck
Dwarkesh Patel
Combining tree search with neural networks eliminates the credit assignment problem in reinforcement learning by providing an improved training target at every step.
## Background
Dwarkesh Patel published this episode on his RSS podcast feed and website on May 15, 2026. The page description explains that AI researcher Eric Jang walks through how to build AlphaGo from scratch using modern AI tools, followed by a discussion on RL in LLMs and automated AI research agents.
## TL;DR
Eric Jang breaks down the foundational mechanics of AlphaGo, explaining how Monte Carlo Tree Search (MCTS), policy networks, and value networks interact to turn an intractable search space into a manageable, amortized computational problem. The conversation explores how KataGo and modern compute reductions have made training world-class Go bots accessible for a few thousand dollars. Jang and Patel then draw direct parallels between MCTS and reinforcement learning in LLMs, highlighting why standard policy gradient RL suffers from credit assignment issues compared to search-guided target distributions. Finally, they examine the current capabilities and limits of automated LLM research loops, discussing where frontier models excel at hyperparameter tuning but struggle with lateral thinking.
## Atmosphere & style
The episode has the rich, pedagogical feel of an informal black-board lecture delivered by a passionate domain expert. It balances deep mathematical and architectural technicality with an engaging, relaxed banter between Jang and Patel. The tone is deeply curious and forward-looking, blending nostalgic reverence for early AI breakthroughs like AlphaGo with realistic candor about the practical realities of modern AI research.
## Key takeaways
- `00:00` AlphaGo provides the cleanest worked example of fundamental AI primitives—search, learning from experience, and self-play—making it a vital blueprint for understanding future general intelligence.
- `01:45` Advances in algorithmic efficiency and modern AI tooling have reduced the cost of rebuilding an AlphaGo-level system from millions of dollars to a few thousand dollars on rented compute.
- `25:00` AlphaGo solves Go's intractable search space by using a dual-head neural network to constrain depth via a Value Network and breadth via a Policy Network.
- `36:00` Initializing policy networks through supervised learning on expert human games establishes a strong performance baseline before transitioning to self-play reinforcement learning.
- `43:04` Monte Carlo Tree Search (MCTS) generates a refined action target at every step, bypassing the credit assignment problem inherent in naive policy gradient RL.
- `01:00:00` In self-play, policy distillation trains the base network to internalize MCTS lookahead, effectively shifting test-time scaling performance directly into forward passes.
- `02:15:00` Comparing learning efficiency using a bits-per-FLOP breakdown highlights why naive policy gradient RL is vastly more information-inefficient than supervised learning.
- `02:13:20` Frontier LLM agents excel at autonomous code execution and hyperparameter tuning, but still struggle with the high-level lateral thinking required to escape research dead ends.
- `02:15:00` The bitter lesson of AI shows that general methods leveraging compute, energy, and parameter scale consistently outperform complex domain-specific optimizations over time.
## Detailed walkthrough
### `00:00` Introduction and the Appeal of AlphaGo
Dwarkesh Patel introduces Eric Jang, who was most recently the Vice President of AI at 1X Technologies and previously a Senior Research Scientist at Google DeepMind Robotics. Having been on sabbatical for a few months, Jang spent part of his time rebuilding, improving, and hacking on AlphaGo from scratch. Patel asks why Jang chose to spend his sabbatical working on AlphaGo rather than relaxing on a beach, prompting Jang to reflect on how AlphaGo and Go AI originally drew him into the field of artificial intelligence. Observing the breakthroughs from 2014 through 2016 made him appreciate how deep learning could tackle an immense computational complexity class that had long been considered intractable for search. Coming from a robotics background where neural network decisions feel relatively intuitive, Jang found it mysterious how a simple 10-layer network could effectively amortize the simulation of a search process deep inside a game tree.
Patel highlights the trajectory of compute needed to build strong Go bots over time. By 2020, an open-source project called KataGo—developed by David Wu at Jane Street—achieved a $40\times$ reduction in the compute required to train a world-class Go bot *tabula rasa*. Thanks to modern LLM-assisted coding, what originally required a dedicated team of research scientists at DeepMind and millions of dollars can now be replicated for just a few thousand dollars of rented compute. Patel notes for audio listeners that this interview is structured as a blackboard lecture and recommends switching to video on YouTube to view the mathematical formulas, diagrams, and Go board demonstrations.
* **Guest background:** Eric Jang (formerly VP of AI at 1X Technologies, former Senior Research Scientist at Google DeepMind Robotics).
* **KataGo efficiency:** Achieved a $40\times$ compute reduction in 2020 compared to earlier benchmarks; serves as the standard engine Go players train against today.
* **Modern accessibility:** Re-implementing AlphaGo-level systems has dropped from millions of dollars to a few thousand dollars in compute using LLM coding tools.
---
`01:45` The Rules and Scoring of Go
Sitting at a physical Go board, Jang walks Patel through the basic mechanics of the game. Go is played with black and white stones on a grid, with the overarching goal of occupying and controlling as much territory as possible. Black always plays first. Stones are captured when all four orthogonal neighbors (known as liberties) are surrounded by opponent stones, cutting off their "oxygen." Jang clarifies that capturing depends strictly on the cross-section grid lines rather than diagonal neighbors. He outlines the slight rule variations across different jurisdictions, specifically noting Chinese rules, Japanese rules, and Trump-Taylor rules. Trump-Taylor rules are designed to be completely unambiguous for computer implementation, making them the standard ruleset that all Go AIs train and resolve against.
```
[ ] [B] [B]
[ ] X [ ] -> [B] X [ ] -> [B] X [B] -> Dead Stone Captured!
[ ] [ ] [B]
```
Standard human rules ban "instant suicide"—placing a stone directly into an area with no liberties—whereas Trump-Taylor rules technically allow the move, but it immediately resolves to death and removal from the board, achieving the exact same functional outcome. As they play out a sample sequence, Jang points out a "check"-like threat where a stone has only one remaining liberty. He notes that tactical sacrifices are common in Go: players routinely allow the opponent to capture a small group of local stones if it yields a grander strategic position elsewhere, epitomizing how one can "lose the battle but win the war."
```
+-----------------------------------------------------------------------------+
| RULES COMPARISON MATRIX |
+-------------------+---------------------------------------------------------+
| Ruleset | Key Characteristics |
+-------------------+---------------------------------------------------------+
| Japanese / Chinese| • Prohibits suicide moves outright. |
| | • Requires human consensus on dead/alive groups. |
| | • Subject to edge-case disputes during endgame. |
+-------------------+---------------------------------------------------------+
| Trump-Taylor | • Fully deterministic & unambiguous for code. |
| | • Permits suicide moves (immediately resolved as dead). |
| | • Area scoring based purely on stones + empty spaces. |
+-------------------+---------------------------------------------------------+
```
As they simulate an endgame scenario, Jang explains how game termination and scoring differ between humans and computers. Human play relies on mutual agreement: when both players evaluate their position and believe no further beneficial moves exist, they pass and agree on which territory is controlled by whom. If they disagree, play continues until a consensus is reached. Under Trump-Taylor rules, scoring is completely programmatic: a player's score equals the total number of physical stones they control plus the empty intersections untouched by any enemy stones. The game officially ends when either player resigns or both players pass consecutively.
* **Trump-Taylor advantage:** Eliminates ambiguity in territory assessment, making it optimal for programmatic reward evaluation.
* **Grid orientation:** Liberties and captures depend entirely on orthogonal cross-sections, not diagonals.
* **Endgame triggers:** Consecutive passes or a explicit resignation trigger game termination.
---
`08:17` Monte Carlo Tree Search (MCTS) and Tree Structure
Transitioning to the theoretical underpinnings of Go AI, Jang diagrams how Monte Carlo Tree Search (MCTS) navigates the overwhelming state space of the game. On a standard $19 \times 19$ board, there are up to 361 potential legal moves on the first turn. With games lasting roughly 250 to 300 moves, a naive search tree expands to an astronomical factor on the order of $361^{300}$ possible trajectories—vastly exceeding the total number of atoms in the observable universe. While symmetries, state merges (transposition tables where different move orders arrive at identical board configurations), and board clutter reduce the effective state space, exhaustive tree search remains completely intractable.
> "AlphaGo's core conceptual breakthrough was using neural nets to make this search problem tractable."
Rather than storing or expanding the entire tree upfront, MCTS interactively explores leaf nodes that show high promise. Jang details the underlying data structure of an MCTS tree node. Each node represents a board state $s$ and tracks specific statistics for each valid action $a$:
1. **Visit count ($N(s, a)$):** The cumulative number of times the search has traversed through that specific child node.
2. **Mean action value ($Q(s, a)$):** The expected win rate or payoff resulting from taking action $a$ from state $s$.
3. **Prior probability ($P(s, a)$):** The initial probability of selecting action $a$, supplied by a policy function (or a neural network in modern setups).
4. **Children references:** A pointer or dictionary storing references to expanded downstream child states.
```
[ Root State (s) ]
/ | \
a_1 / a_2 | \ a_3
v v v
Node A Node B Node C
N(s,a_1) N(s,a_2) N(s,a_3)
Q(s,a_1) Q(s,a_2) Q(s,a_3)
P(s,a_1) P(s,a_2) P(s,a_3)
```
When selecting which node to traverse next, early multi-armed bandit algorithms relied on formulas like UCB1 (Upper Confidence Bound 1), which balances exploitation (choosing actions with high $Q$-values) and exploration (choosing infrequently visited actions):
Patel summarizes the core intuition: $Q(s,a)$ represents the exploit term (the average win rate from previous rollouts or evaluations), while the second component acts as an exploration bonus. Early in the search, when visit counts $N(s,a)$ are low, the exploration bonus dominates the argmax, forcing the agent to try unfamiliar paths. As visit counts accumulate, the exploration term shrinks, allowing the high-confidence $Q$-value to dictate the final move selection.
* **Board state complexity:** $19 \times 19$ grid yields an initial branching factor of 361, leading to a search tree scale of $\sim 361^{300}$.
* **Transposition merging:** Child nodes arriving at identical board positions through different move sequences share identical future state trees.
* **Exploration vs. Exploitation:** PUCT dynamically shifts focus from uncertain, unvisited paths to high-value winning trajectories as visit counts grow.
`20:00` Regret Bounds, Probabilities, and MCTS Backup
The mathematical formulation for selecting moves during search uses an exploration term designed to bound regret—limiting how far off a chosen path can be from the optimal sequence. While standard Multi-Armed Bandit (MAB) problems often rely on Upper Confidence Bounds (UCB) with logarithmic regret bounds, the selection criteria in Go (Upper Confidence bounds applied to Trees, or UCT / PUCT) is modified to account for the massive branching factor and large action space inherent to a 19x19 board.
```
[ Root Node ]
/ | \
a₁/ a₂\ a₃\
v v v
[Node] [Node] [Node]
/ | \
/ | \
[Leaf Nodes (Terminal: Win/Loss)]
```
Although Go is a strictly deterministic, full-information game with an exact mathematical truth at every board position, probabilities enter the equation through the random search process itself. The action-value function $Q$ represents the expected return averaged over the distribution of trees sampled during search.
* **Uniform vs. Informed Sampling:** A naive Monte Carlo approach samples valid moves uniformly ($P(a) = \frac{1}{N_\text{valid}}$), requiring an immense number of simulations over low-value subtrees. Efficient search requires importance sampling to concentrate compute on high-value paths.
* **Terminal Node Resolution:** At the end of a rollout or game path under Tromp-Taylor rules, a leaf node resolves into a deterministic return $u \in \{+1, -1\}$ (a clear win or loss).
* **The Backup Step:** Values determined at the terminal leaf nodes propagate backward up the tree. The mean action value $Q(s, a)$ of parent nodes is updated as a weighted average of downstream leaf values.
> "If you had a very powerful computer, there are no probabilities. You can just compute the true average of what the mean action value is... The interpretation of $Q$ is what is the expected action value under the random distribution induced by some random search process."
---
`25:00` Truncating Depth and Breadth with Value Functions
Humans do not evaluate a Go board by calculating every variation down to the final terminal state hundreds of moves away. Instead, experienced human players look at a board position and instantly form an intuitive judgment about who is winning. This mental model operates like an amortized neural network—summarizing thousands of possible game playouts in a split second based on past experience and intuition.
```
[ Board State ] ---> ( Human Intuition / Neural Net) ---> [ Value: P(Win) ]
```
To make Monte Carlo Tree Search (MCTS) computationally tractable, AlphaGo replaces deep rollouts with a value function $v(s)$ that predicts the probability of winning directly from a board position. This mechanism resolves the two fundamental challenges of combinatorial tree search:
| Search Dimension | Problem in Classical Search | AlphaGo Solution |
|:--- |:--- |:--- |
| **Search Depth** | Games last hundreds of moves, making deep rollouts computationally prohibitive. | **Value Network ($v$):** Evaluates leaf nodes immediately without playing out games to the very end. |
| **Search Breadth** | Too many legal moves per state ($100+$ valid actions per turn). | **Policy Network ($p$):** Narrows the search to a small subset of plausible, high-quality moves. |
During play, MCTS runs afresh on every turn. When the AI receives an encoded board state, it executes a set number of search simulations (e.g., 1,000 simulations) to update node statistics. Once an action is selected and played, the old search tree is discarded, and a new tree search begins from scratch on the next turn.
---
`30:00` Dual-Head Neural Architectures and Representation
The neural network aiding MCTS consists of two distinct prediction heads built on top of a shared feature-extraction backbone:
1. **Value Head ($v$):** Predicts a single scalar logit representing the expected game outcome ($+1$ for a win, $-1$ for a loss) from the current board state.
2. **Policy Head ($p$):** Predicts a probability distribution over all valid board positions (a 361-dimensional vector for a $19 \times 19$ grid).
```
[ Encoded Board Input ]
(3 Channels: Black, White, Empty)
|
v
[ ResNet / Conv Backbone ]
/ \
v v
[ Value Head ] [ Policy Head ]
Scalar Logit: P(Win) 361-dim Vector: P(Action)
```
While vision transformers (ViTs) excel at global attention across unstructured image data, residual convolutional networks (ResNets) often outperform transformers in lower-budget RL settings due to their strong inductive bias for local spatial correlations.
* **Global Feature Aggregation:** Papers such as KataGo demonstrated that standard CNNs can be improved by adding global pooling layers. These allow distant fights on opposite corners of a $19 \times 19$ board to exchange information without relying purely on localized convolutions.
* **Input Tensor Encoding:** The board state is represented as a multi-channel binary tensor (e.g., $3 \times 19 \times 19$), where individual channels encode black stones, white stones, and empty spaces.
> "In deep learning, initialization is everything. You always want to initialize your research project to something as close to success as possible."
---
`35:00` Perfect Information vs. Imperfect Information Games
Go is a deterministic, perfect-information game. Game theory proves that in such settings, there exists a stationary Nash equilibrium strategy where a player's optimal action depends solely on the current state, without needing memory of past turns.
In games with imperfect information, partial observability, or team dynamics (e.g., Poker, Diplomacy, or 2v2 Go), this assumption breaks down. In those domains:
* Players must infer hidden information and model the opponent's (or partner's) specific habits and play styles over time.
* Temporal history and context sequences become essential for predicting optimal moves.
* Architectures like Transformers become significantly more advantageous due to their ability to process variable-length temporal sequences and unobserved state histories.
When bootstrapping an AlphaGo implementation, initializing the policy network via supervised learning on expert human games (as done in AlphaGo Lee) provides a stable foundation before transitioning to pure self-play reinforcement learning (as in AlphaGo Zero).
`40:00` Expert Data Initialization and the Policy-Value Base
```
[Root Board State]
/ | \
[Move 1] [Move 2] [Move 3]
| | |
v v v
(Probabilities) (Argmax Play) (Fast "Hip-Shot")
```
The design philosophy when building an system like AlphaGo centers on taking a working foundation and making it better, rather than starting from scratch with a broken system. Under this approach, the network is initialized on human expert games—specifically using winning trajectories—to predict expert actions across various board states. A neural network trained purely on this human data can act as an remarkably strong, fast Go player simply by taking the argmax over its policy probabilities, making decisions in a single forward pass without explicitly searching ahead.
- **Win Probability Dynamics:** On an opening board with a single stone placed, the predicted value $v_\theta$ naturally converges near $0.5$ across hundreds of thousands of games because early moves branch almost equally into wins and losses; as the game progresses into later moves, $v_\theta$ sharply trends toward $0$ or $1$.
- **Model Efficiency:** Even a modest policy network—around 10 layers and fewer than 3 million parameters—trained solely on human expert moves can outplay the vast majority of human Go players without running any search.
- **Verification Step:** Validating that this base policy network plays sensible moves serves as a crucial sanity check for game rules and simulation code before layering complex search algorithms on top.
---
`43:04` Test-Time Compute and the MCTS Framework
The transition from a pure neural network to Monte Carlo Tree Search (MCTS) allows test-time compute to scale performance. In historical matches like AlphaGo's victory over Lee Sedol, massive computational resources were used to maximize win probability, whereas modern implementations require significantly less compute to achieve equal strength.
- **Simulation Budget:** MCTS runs an iterative process over a chosen budget of simulations, typically ranging from $200$ to $2,048$ per move during training, though the AlphaGo vs. Lee Sedol match utilized tens of thousands of simulations backed by TPU pods.
- **Compute Transfer:** Over training iterations, the policy network internalizes the lookahead capabilities of MCTS; a single forward pass eventually absorbs much of the burden previously handled by heavy test-time search compute.
- **The Four-Step Loop:** Each simulation in the search process sequentially executes **Selection**, **Expansion**, **Evaluation**, and **Backup**.
---
`47:11` The MCTS Loop: Selection, Expansion, and Evaluation
MCTS begins at a root node representing the current game state, where the neural network evaluates the board to output both a scalar win value $v_\theta$ and a vector of action probabilities $P(s, a)$. The search tree then iteratively expands through four distinct phases:
1. **Selection:** The algorithm navigates down the existing tree by maximizing the Predictor Upper Confidence Bounds applied to Trees (PUCKT) formula:
$$\text{PUCKT}(s, a) = Q(s, a) + c_{\text{puct}} \cdot P(s, a) \cdot \frac{\sqrt{N(s)}}{1 + N(s, a)}$$
Initially, visit counts $N(s, a)$ are zero, biasing the very first selections toward actions with high policy probability $P(s, a)$.
2. **Expansion:** Upon reaching an unexpanded non-terminal node, the tree generates child nodes corresponding to valid opponent responses.
3. **Evaluation:** The network evaluates the expanded leaf board states to estimate value $v_\theta$. In a zero-sum game like Go, the estimated value for the opposing player simplifies to $1 - v_\theta$.
> "Because Go is such a combinatorially complex game, you cannot afford to build the tree in advance and then search it. You must search while building the tree."
- **Historical Rollouts vs. Pure Value Networks:** The original AlphaGo Lee averaged $v_\theta$ with the outcome of a fast random rollout to the end of the game using Tromp-Taylor rules:
$$\text{Value} = \alpha \cdot v_\theta + (1 - \alpha) \cdot \text{Rollout Outcome}$$
- **Streamlined Modern MCTS:** Subsequent architectures like AlphaGo Zero and KataGo removed rollouts entirely, relying solely on the neural network's $v_\theta$ output, which significantly speeds up simulations without losing accuracy.
---
`52:50` Backup, Visit Counts, and Policy Improvement
The final phase of each MCTS simulation is the **Backup** step. Once a leaf node is evaluated, its predicted value $v_\theta$ is propagated back up the search path, updating the running mean action-values $Q(s, a)$ and incrementing the visit counts $N(s, a)$ for every ancestor node up to the root.
- **Visit Counts as Policy:** As the simulation count grows, the search tree concentrates visits on promising branches. The normalized visit counts at the root node form a refined probability distribution over actions that acts as a strictly improved policy target over the raw network prior $P(s, a)$.
- **Pruning the Search Space:** Unlike exhaustive game trees (such as Tic-Tac-Toe's $9!$ state space), MCTS produces an extremely sparse tree, focusing compute resources on high-value paths while ignoring low-probability branches.
- **Dual Head Synergy:** Maintaining explicit policy heads alongside value heads avoids having to pass up to 361 individual move states into a batch to evaluate board positions sequentially. The prior probabilities $P(s, a)$ prune the search space before value evaluations take place.
---
`58:36` Sponsor Break: Jane Street Engineering
A brief historical note illustrates the rapid scaling of modern AI and trading infrastructure. Ron Minsky (co-lead of Jane Street's tech group) and Dan Pontecorvo (head of physical engineering) contrast early compute setups with present-day data centers:
- **"The Hive":** Jane Street's initial compute cluster began as six Dell servers stacked on top of each other in an office space, where trading systems were kept close enough to physically turn off if needed.
- **Datacenter Transition:** Early office setups suffered from operational hazards—including an incident where an office cleaner accidentally unplugged a live trading machine while vacuuming.
- **Modern Infrastructure:** Modern liquid-cooled GB300 server cabinets consume up to 140 kW at peak power, compared to traditional air-cooled cabinets that consumed 10 to 40 kW.
`01:00:00` Self-play and Policy Distillation
The core mechanics of self-play rely on taking the raw policy network and refining it through Monte Carlo Tree Search (MCTS) to create an improved target distribution.
During a game, the policy network provides an initial recommendation distribution across possible moves at a given state $S$. Running MCTS combines this initial policy guess with evaluations from the value network to perform search simulations, ultimately producing a much sharper, more confident visit count distribution $\pi_{\text{MCTS}}(A \mid S)$. The system then samples an action from this search-refined distribution (or takes the argmax), updates the board state, discards the search tree, and repeats the process for the next move until the game reaches a terminal state with a clear winner.
The mathematical beauty of AlphaGo's self-play training loop lies in policy distillation:
* **Direct Target Learning:** Instead of requiring MCTS to perform heavy computational search on every single move, the system trains the baseline policy network to predict the outcome of the MCTS search directly from the start.
* **Test-Time Scaling:** On a test-time scaling plot (win rate versus number of simulations), starting with a distilled policy effectively shifts the starting point upward. Spending 1,000 search simulations on top of a distilled network yields a far stronger player than spending 1,000 simulations on an unrefined baseline.
* **Monotonic Improvement:** As long as search evaluations are accurate, increasing search simulations monotonically improves performance.
While the general shape of test-time scaling curves shows diminishing returns as simulation count increases, initializing search with a stronger, distilled policy consistently raises the baseline.
---
`01:05:00` Analogy to Robotics: DAgger and Action Relabeling
The self-play training process in AlphaGo bears a striking structural parallel to classic imitation learning algorithms in robotics, specifically **DAgger** (Dataset Aggregation).
In an Markov Decision Process (MDP) trajectory, a bot might execute a sequence of actions $[a_0, a_1, a_2, \dots]$ and ultimately lose the game. MCTS acts as an expert supervisor that inspects every state along that trajectory and provides a strictly better alternative action tuple $[a_0', a_1', a_2', \dots]$.
* **State Relabeling:** Even when a trajectory terminates in a loss or passes through bad intermediate states, MCTS re-labels the actions taken with higher-quality alternatives.
* **Robotics Analogy:** This is identical to a self-driving car veering off-course where an expert supervisor provides corrective steering interventions from an off-path state to bring the vehicle back onto the road.
* **Training Objective:** The policy network is retrained on these tuples $(S, a')$ to imitate the search process's recommendations rather than the raw actions taken during the game.
#### The Risk of MCTS Policy Degradation
MCTS is a heuristic and does not strictly guarantee an improved policy if underlying assumptions fail:
* **Value Function Collapse:** If a bot frequently resigns instead of playing out games to the end (e.g., using Trump-Taylor scoring rules), late-stage board states disappear from the replay buffer.
* **Error Propagation:** Bad leaf evaluations propagate up the tree during MCTS backups, distorting node selection criteria and yielding a search distribution worse than the unrefined policy network.
* **Variance Issues:** Running too few simulations introduces high variance, preventing the search from exploring sufficiently to converge.
---
`01:10:00` Practical Value Function Bootstrapping and TD-Learning
To ensure MCTS improves the policy, practitioners must ground the value network before relying heavily on expensive search simulations.
```
Offline Expert Data / Random Play (9x9)
│
▼
Ground Terminal Values
│
▼
Back Up Intermediate Nodes (TD-like)
│
▼
Stable MCTS & Self-Play Search
```
Several practical strategies serve to ground value estimates:
* **Preventing Premature Resignation:** Forcing bots to play out 10% of training games all the way to terminal resolution prevents distribution shift in late-stage evaluations.
* **Human/Offline Data Warm-start:** Pre-training on human games or open-source Go engines quickly grounds value predictions for late-stage and terminal board configurations.
* **Small Board Transfer:** On smaller boards (such as $9 \times 9$ or $5 \times 5$), playing roughly 50,000 completely random games generates enough end-state overlap with reasonable play to bootstrap a reliable value function. Architectures that share weights across board sizes (like KataGo) can transfer these fundamental spatial value representations up to $19 \times 19$ boards.
Judging win probability in the mid-game is vastly harder than evaluating the opening (where win probability is $\approx 0.5$) or the late game (where territory is clearly bounded). The value network's task of propagating end-game certainty backward into mid-game evaluations directly mirrors **Temporal Difference (TD) learning**.
---
`01:15:00` Computational Efficiency and the $P$ vs $NP$ Perspective
Combining the policy head and value head into a single unified architecture—sharing underlying neural representations—provides substantial compute savings over two standalone networks (as used in the original AlphaGo Lee paper).
The ability of a relatively shallow 10-layer neural network to approximate an exponentially massive search space represents a fundamental shift in AI capability:
* **Compressing Intractable Search:** A forward pass through a modest neural network executes parallelized, distributed reasoning that amortizes an almost intractable combinatorial search space into a single inference step.
* **Parallels to Physical Systems:** Similar to AlphaFold approximating complex molecular dynamic simulations into direct structure predictions, AlphaGo shows that domain structure allows deep networks to bypass worst-case computational complexity.
* **Worst-Case vs. Structured Reality:** While problems like Go or protein folding may be computationally intractable or $NP$-hard in the worst-case scenario, real-world instances contain rich macroscopic structures that neural networks exploit effectively.
"10 steps of neural network vectorized distributed representation thinking is able to amortize and approximate to a very, very high fidelity a nearly intractable search problem."
- The shift from AlphaGo Lee to AlphaZero merged separate policy and value models into a dual-headed single network.
- Neural networks bypass worst-case mathematical hardness by exploiting environmental structure.
- Similar amortization principles power structural biology breakthroughs like AlphaFold and optimization tools like AlphaTensor.
Here is a detailed walkthrough of the transcript part, covering the computational theory of simulation, chaotic systems, credit assignment, and alternative RL paradigms.
---
`01:20:00` Chaos, Cryptography, and the Compression of Macroscopic Intelligence
The conversation shifts into a high-level theoretical discussion on computational complexity and how AlphaGo manages to distill complex world simulations into the forward pass of a single neural network. Eric notes that AlphaGo serves as a profound example of compressing massive simulation spaces into compact neural representations. Dwarkesh raises a question about the role of chaos in prediction, pointing out how weather prediction requires exponentially more computational resources the further out one attempts to forecast. Small perturbations in a chaotic system drastically alter the final outcome over time.
Eric illustrates this challenge using an analogy between Go and weather forecasting. In Go, predicting the *exact* future board state 100 moves down the line is extremely sensitive to initial conditions—a single stone placement can disrupt the entire future trajectory. This microscale prediction is inherently chaotic and hard. However, AlphaGo does not need to predict the exact microscopic layout of every future move; instead, it predicts a macroscopic quantity: who is going to win. Similarly, in weather prediction, humanity rarely cares about the exact wind velocity 6,000 feet above a specific latitude; what matters is the macroscopic structure, such as where a hurricane will land.
> "In chaos, you know, there's a classic Lorenz attractor... You don't, if you start anywhere on the Lorenz attractor, you don't know where you're going to end up. But you do know that the thing looks like this... Sometimes we don't necessarily care about the microscale things. We actually care about the macroscopic structure."
To contrast, they analyze hash functions and cryptographic protocols. Cryptography relies on initial-condition sensitivity without revealing any macroscopic structure or value function that permits fast approximations. Dwarkesh references a blog post by Reiner, noting the structural similarities between cryptographic protocols and neural networks: both involve sequential layers that jumble information together. However, while cryptography aims to conceal patterns, neural networks operate at the "edge of chaos" (referencing papers by Jascha Sohl-Dickstein) to process and correlate dense, high-dimensional information.
---
`01:25:00` Credit Assignment: MCTS vs. Naive Policy Gradient RL
The discussion transitions back to Monte Carlo Tree Search (MCTS) to highlight why its learning signal is radically superior to naive reinforcement learning (RL) methods used in modern Large Language Models (LLMs). Crucially, MCTS does not merely upweight winning trajectories and downweight losing ones. Instead, MCTS conducts an exhaustive search to discover a strictly better action for *every single state* encountered in a trajectory, giving the policy network a highly stable, step-by-step target to predict.
To demonstrate why alternative self-play approaches fail, Eric walks through a hypothetical naive RL setup akin to how LLMs are currently trained:
* Imagine a league of two evenly matched policy networks, Policy A and Policy B, playing 100 games of Go at 300 moves per game.
* Because they are evenly matched, Policy A wins 51 games and Policy B wins 49 purely due to subtle exploration noise or random chance.
* Suppose in 50 of those games, the players played identically, but in that single extra winning game (the 51st), Policy A made one critical, out-of-distribution, brilliant move that secured victory.
In a naive policy gradient algorithm, you would take the winning trajectories and reinforce all their constituent actions. However, this creates a massive variance problem: you end up reinforcing $99 \text{ games} \times 300 \text{ moves}$ of redundant, average play just to capture the single critical move that actually won the game. Dwarkesh compares this inefficient credit assignment to Andrej Karpathy’s quote about RL in LLMs being like "sucking supervision through a straw."
```
Naive RL Gradient Estimator:
- Reward multiplier applies to the entire sequence.
- Variance grows quadratically with episode length T.
- High gradient variance due to reinforcing neutral/average actions.
MCTS Target Generation:
- Generates a local, high-quality search target for EVERY move.
- Sidesteps sequence-length credit assignment.
- Extremely low variance per training step.
```
---
`01:30:00` Mathematical Variance in RL and the LLM Single-Action Trick
Eric breaks down the mathematical formulation of the standard policy gradient estimator (REINFORCE) to explain why sequence length magnifies variance. The gradient calculation relies on the return multiplied by the log probabilities of the sequence:
When computing the variance of this estimator, cross-multiplication between token probabilities and episode returns causes the variance to grow quadratically with the trajectory length $T$.
To bypass this exploding variance, modern LLM post-training treats the *entire generated text sequence* (e.g., "Hello World") as a single action at $T=1$, rather than performing multi-step credit assignment across thousands of individual tokens. While the log probability of a sequence is the sum of its individual token log probabilities, assigning a single scalar reward at the very end of a generation makes it difficult to isolate which specific token contributed to a good or bad answer unless process-supervised rewards are introduced.
---
`01:35:00` Advantage Baselines and Neural Fictitious Self-Play (NFSP)
To mitigate variance in model-free RL settings where tree search is impossible, researchers rely on **advantage estimation** (referencing John Schulman’s Generalized Advantage Estimation paper). Instead of multiplying gradients by absolute returns ($1$ or $0$), the algorithm subtracts a baseline—often derived from a learned value function $Q(s, a)$ or $V(s)$ via Temporal Difference (TD) learning—to compute the advantage:
$$A(s, a) = Q(s, a) - V(s)$$
By subtracting average performance, the model only pushes up actions that perform better than expected and suppresses actions that perform worse, effectively zeroing out the gradient for neutral moves.
Finally, Eric introduces **Neural Fictitious Self-Play (NFSP)**, an alternative model-free RL technique used in complex imperfect-information or non-simulable games like *StarCraft* (AlphaStar) and *Dota 2* (OpenAI Five):
* In games like *Go*, the environment is deterministic and fully observable, allowing deep MCTS search trees.
* In games like *StarCraft*, partial observability (fog of war) and complex binaries make exact tree search impossible.
* Instead of tree search, NFSP constructs an explicit "best response" policy. You fix the opponent's strategy (e.g., a Protoss policy playing against a fixed Zerg baseline) and apply standard model-free algorithms like PPO or TD-learning to discover winning counter-strategies, progressively refining the policy labels over time.
`01:40:00` Best-Response Distillation and the Mechanics of Q-Learning
When moving away from pure self-play, an alternative reinforcement learning strategy involves training against a league of fixed opponent policies rather than a constantly moving self-play target. By fixing the opponent ($\pi_B$), the problem simplifies to an environment where the agent receives a sparse binary return—such as $1$ for a win and $0$ for a loss—and optimizes against that specific opponent using standard model-free algorithms like PPO, SAC, or VMPO. Training separate best-response policies against a suite of opponents ($\pi_B, \pi_C, \pi_D$) generates high-quality action labels. These individual best-response policies can then be distilled into a unified "mixed strategy" policy. This distilled agent acts as a proxy for an MCTS teacher, performing no worse than the average opponent in the league while bypassing the need to generate online search trees during training.
This framework shares a foundational connection with $Q$-learning and approximate dynamic programming. In MCTS, action-value estimates ($Q$) are calculated by averaging Monte Carlo rollouts and backing up value estimates from the leaf nodes of a search tree. By contrast, model-free $Q$-learning uses Temporal Difference (TD) learning to enforce consistency across steps via the Bellman equation:
$$Q(s, a) = R + \gamma \max_{a'} Q(s', a')$$
This recursive relationship allows an agent to estimate the value of intermediate state-action pairs without forward tree search, recovering an implicit policy simply by taking the $\text{argmax}$ over $Q$-values.
- **League Distillation**: Distilling best-response policies trained against a fixed set of league opponents creates a game-theoretic mixed strategy robust against all candidate opponents.
- **Dynamic Programming Property**: $Q$-learning leverages the recursive structure of Markov Decision Processes (MDPs) to back up reward signals across visited trajectories.
- **Policy Recovery**: Explicit policy distribution modeling is optional in value-based RL, as taking an $\text{argmax}$ over learned $Q$-values implicitly defines the optimal action selection.
---
`01:44:15` MCTS vs. LLM Credit Assignment and the "Supervision Through a Straw" Problem
A crucial distinction exists between how model-free RL algorithms update $Q$-values and how Monte Carlo Tree Search operates. $Q$-learning plans over real trajectories that the agent has already visited in the environment. MCTS, on the other hand, simulates and plans over hypothetical trajectories the agent has not yet experienced. This structural difference highlights a major pain point in applying standard RL to Large Language Models (LLMs). In LLM training, models often receive feedback based on outcome-based unit tests (e.g., verifying whether a generated code snippet executes correctly). However, standard policy gradient methods suffer from a severe credit assignment problem, receiving "supervision through a straw"—a memorable phrase associated with Andrej Karpathy—where the algorithm must uniformly upgrade or downgrade thousands of intermediate tokens without knowing which specific tokens actually caused the success or failure.
In contrast, Go benefits enormously from MCTS because search provides dense, move-by-move supervision. A strong value function allows MCTS to evaluate local positions and truncate depth early, enabling iterative policy improvements on a fine-grained, per-move basis rather than forcing the model to rollout entire game trajectories to obtain a scalar reward.
- **Search vs. Replay**: $Q$-learning relies on replay buffer trajectories from past visits, whereas MCTS searches unexplored forward paths via mental simulation.
- **Credit Assignment Bottleneck**: Naive RL on long LLM generations reinforces every token in a trajectory equally, making credit assignment extremely sample-inefficient.
- **Local Move Supervision**: MCTS acts as an efficient local teacher in board games by delivering direct training targets for every individual move rather than full-sequence rewards.
---
`01:47:30` Why Standard Search Heuristics Fail on Language
While tree search works seamlessly in games like Go, adapting MCTS to continuous control or natural language reasoning presents significant hurdles. In Go, board states have discrete, bounded action choices, making value estimation relatively concrete and search trees shallow enough to manage. For LLMs, the action space consists of tens of thousands of potential tokens at every step, creating an enormous branching factor. Under these conditions, traditional exploration heuristics like the Predictor Upper Confidence bounds applied to Trees (PUCT) formula break down:
$$\text{PUCT} \propto \frac{\sqrt{N}}{1 + N_a}$$
Because the continuous language space is so vast, an algorithm will almost never sample the exact same child node twice. Consequently, PUCT becomes overly greedy, favoring locally high-probability tokens and missing broader logical breakthroughs required for deep problem-solving.
Despite these mathematical friction points, LLMs frequently display implicit human-like reasoning—exploring candidate approaches, recognizing mistakes, backing up, and trying alternative paths—all within a single forward pass without needing explicit external tree search. While explicit forward simulation may make a comeback in LLM reasoning, standard discrete MCTS formulas cannot be applied out of the box to open-ended domain generation.
- **PUCT Failure Mode**: The square-root exploration term in PUCT relies on repeated visits to discrete nodes, which fails when the action space is so broad that child nodes are rarely revisited.
- **Domain Differences**: Game domains like Go feature rigid rules and explicit value targets, whereas language reasoning involves combinatorial breadth where local token choices may appear incorrect initially yet yield valid long-term proofs.
- **Implicit Reasoning**: LLMs can perform effective back-tracking and self-correction within text context windows without maintaining an external search tree structure.
---
`01:50:00` Board Game Scaling Laws and the Reality of Empirical AI Research
The relationship between search and model parameters was formally explored in Andy Jones's 2021 paper, *"Scaling Scaling Laws for Board Games."* Jones demonstrated an explicit trade-off between test-time search compute (MCTS rollout depth) and training compute (model size), showing that spending additional compute during inference yields equivalent win-rate performance to training a significantly larger network. Furthermore, the paper established that compute requirements could be accurately predicted when scaling board games to larger dimensions, offering an early blueprint for modern inference-time compute scaling.
Motivated by these scaling principles, Eric Jang embarked on an experimental project to train a compute-optimal Go bot purely driven by the Bitter Lesson, aiming to match top-tier performance without relying on handcrafted heuristics or domain-specific code adjustments seen in systems like KataGo. However, running into implementation bugs in MCTS labeling highlighted a crucial empirical lesson: scaling laws cannot be derived on broken architectures or faulty data. Scientific modeling of scaling behaviors only yields valid insights after first engineering a bug-free, highly capable baseline system.
> "Usually when you want scaling laws to work, you want to be in the regime where the recipe already works and the data sets are good, rather than trying to figure out how to do scaling while also trying to figure out what the right data sets are."
- **Inference vs. Training Trade-off**: Increasing MCTS search budget at test time directly compensates for smaller network parameter counts along predictable scaling curves.
- **Board Size Predictability**: Jones demonstrated that compute budgets required to master larger grid sizes (from $3 \times 3$ up to $19 \times 19$) can be modeled extrapolate-wise.
- **Prerequisite for Scaling Research**: Valid scaling curves require a baseline pipeline free of structural bugs and training noise; attempting to extract scaling laws from sub-optimal configurations leads to misleading empirical signals.
`01:55:00` The Reality of Reproducing AlphaZero and the Economics of Compute
Eric Jang opens by offering a piece of practical advice for AI researchers and practitioners: finish building a working, bug-free system before attempting to study its scaling behavior. Refinement and scientific experimentation should only happen after the underlying artifact works. Dwarkesh Patel points out that if one examines the historical charts of AI compute over the last decade, there is a smooth exponential line in log space—except for a massive aberration around AlphaGo Zero. AlphaGo Zero was trained on roughly $3 \times 10^{23}$ FLOPs, a massive amount of compute at the time that sits surprisingly close to modern frontier LLM compute scales. When Dwarkesh asks how Eric managed to replicate a strong Go bot on a modest budget, Eric reveals he received a $10,000 compute grant from Prime Intellect, spending $4,000 on exploratory research, $3,000 on the final training run, and keeping the rest for model serving.
The staggering discrepancy between DeepMind's original compute budget and Eric's $3,000 run highlights a recurring theme in machine learning: the compute required to be the *first* to achieve a breakthrough is vastly larger than what is needed to catch up later. Once a proof-of-concept exists, researchers can use distillation, warm-starting, and structural shortcuts. For example, Eric trained best-response agents directly against KataGo to rapidly achieve high performance. DeepMind's AlphaZero team had no pre-existing policy to train against because they were strictly pursuing a *tabula rasa* approach. When speed and proof-of-concept matter most, researchers optimize for time-to-result rather than compute efficiency—a pattern currently repeating in robotics research, where scatterplots of model compute show little clean scaling alignment compared to LLMs.
- **Key Compute Metrics:** AlphaGo Zero consumed ~$3 \times 10^{23}$ FLOPs; Eric Jang's reproduction run cost ~$3,000 on modern hardware out of a $10,000 grant.
- **Bootstrapping Strategy:** Training best-response agents against KataGo bypasses the initial compute-heavy *tabula rasa* phase that DeepMind was forced to navigate.
- **Robotics Parallel:** Robotics compute scaling remains fragmented and unstandardized because labs prioritize capability emergence and rapid iterations over strict FLOP efficiency.
---
`01:58:30` Algorithmic Simplifications and Architecture Insights Since 2017
Reflecting on how hardware and algorithmic understanding have evolved since 2017, Eric explores whether the "Bitter Lesson"—the principle that raw compute and general methods eventually render complex hand-tuned heuristics obsolete—has played out in Go AI. KataGo introduced numerous clever techniques and auxiliary training targets in 2020. However, Eric’s preliminary findings suggest that many explicit architectural choices matter far less today. Modern NVIDIA GPUs have advanced to the point where choosing between a Transformer and a ResNet yields negligible performance differences for board sizes like Go.
Furthermore, infrastructure can be drastically simplified. Instead of managing complex, asynchronous, distributed RL setups with dedicated replay buffers, pushers, and collectors, one can use a straightforward synchronous setup: collect data, train a supervised network, and repeat. While KataGo required massive training runs on V100 GPUs, modern runs achieve equivalent results on half the number of desktop Blackwell GPUs.
To accelerate early training, multi-board co-training serves as an immense compute multiplier. In AlphaGo Zero, the first 30 hours were spent simply catching up to basic supervised learning baselines. By co-training value functions on $9 \times 9$ board layouts—where endgame value functions are solved much faster—and warm-starting those parameters onto full $19 \times 19$ boards, the initial learning phase is drastically compressed. Interestingly, varying the number of search simulations per episode proved surprisingly insensitive, demonstrating that foundational initialization and opponent strength matter far more than hyper-parameter tuning of simulation counts.
- **Hardware Efficiency:** Modern desktop GPUs (e.g., Blackwell) enable a single researcher to outpace clusters of older V100 GPUs.
- **Multi-Board Transfer:** Pre-training value functions on $9 \times 9$ boards and transferring them to $19 \times 19$ boards eliminates the ~30-hour cold-start phase seen in AlphaGo Zero.
- **Infrastructure Streamlining:** Synchronous batch collection removes the need for asynchronous distributed buffers without sacrificing learning performance.
---
`02:01:09` Off-Policy Training, Replay Buffers, and the "DAGGER" Perspective
Dwarkesh raises a fundamental RL question: why is it acceptable to use a replay buffer in AlphaGo Zero when AI researchers constantly warn about the dangers of off-policy training? In a naive AlphaGo implementation, most moves inside a training batch were generated by older versions of the network, meaning the current policy is trained on data generated by a different policy.
Eric explains that off-policy training becomes destructive when the training buffer consists of states that the current policy would never actually visit. Supervising a model to take actions on irrelevant, unreachable states wastes capacity and leads to severe policy degradation. However, viewed through the lens of DAGGER (Dataset Aggregation), off-policy data acts as a crucial stabilizing funnel.
```
+-----------------------+
| Optimal Trajectory |
| S1 ---> S2 ---> S3 |
+-----------+-----------+
|
(Disturbance / Drift)
v
+-------------------------------+ +-------------------+
| Off-Policy Buffer States | | Drifted State |
| (Teaches how to funnel back) |<--| S_drift |
+-------------------------------+ +-------------------+
```
An optimal training distribution needs more than just the ideal trajectory; it requires a "tube" of nearby states around that trajectory so the agent learns how to recover when pushed off-course. Eric quips, quoting a standard adage in games: *"The problem with Go and chess is that the other player is always trying to do some shit."* In both games and real-world robotics (where wind gusts or tire slippage drift an agent off-path), off-policy states teach the model how to correct back toward a winning state.
To test this in practice, Eric ran an experiment where he broke the temporal causality of games entirely. He took random board states from past games, re-ran MCTS on them using the *current* network, and fed those fresh labels back into the training process. This fully saturated GPU utilization because the system never had to block and wait for an ongoing game to finish—it could plan across arbitrary states in parallel.
- **DAGGER Analogy:** Off-policy buffers provide a corrective "funnel" that teaches agents how to recover from mistakes or opponent disruptions.
- **Causality-Free MCTS Re-labeling:** Arbitrary historic board states can be pulled from a database and re-annotated using the current network's MCTS, maximizing GPU compute density.
- **Policy Drift Risks:** If off-policy data strays too far into states the current policy will never see, training capacity is wasted on irrelevant situations.
---
`02:05:00` Deepmind's Q-Op Legacy and Offline Relabeling in Robotics
Connecting these Go insights back to robotics, Eric diagrams how traditional offline learning setups (such as Google's Q-Opt) operate. In these architectures, off-policy transition tuples $(S, A, R, S')$ are continuously pushed to a replay buffer. A background process—the Bellman updater—acts like a "daydreamer," constantly re-evaluating historical transitions to ask: *"Given what I know now, what is the best possible action I could have taken at state $S$ to maximize future value?"*
The Bellman updater calculates the target $Q$-value by evaluating $Q(S', A')$, adding the immediate reward, and setting a new target $Q_{\text{target}}$ for the trainer. The main trainer then uses standard supervised loss to minimize the difference between its current prediction $Q(S,A)$ and $Q_{\text{target}}$.
In Eric's Go experiment, he replaced the Bellman updater with an offline MCTS relabeler. Instead of computing target networks, he ran MCTS searches directly on old board transitions using the latest policy network $\pi$. While this provided strong training stability, modern RL and robotics have largely shifted toward strictly on-policy setups or light off-policy usage. On-policy methods avoid the loss explosions and instabilities that occur when off-policy value estimations go awry. Today, off-policy data is primarily used to compute baseline advantages—such as $A(S,A) = Q(S,A) - V(S)$—to reduce variance without directly warping the primary learning objective.
- **The Bellman Updater:** A background process that recalculates historical state-action values to find optimal retro-active targets ($Q_{\text{target}}$).
- **Hindsight Daydreaming:** Offline relabeling allows models to retroactively re-evaluate past experiences using updated capabilities.
- **Modern Convergence:** Modern RL limits direct off-policy training due to instability, instead utilizing off-policy data mainly for variance reduction and advantage shaping.
---
`02:12:02` Information Inefficiency: Why Naive RL Struggles Compared to Supervised Learning
Dwarkesh pivots the conversation to the core information theory problem of reinforcement learning, referencing an essay he wrote on why policy gradient RL (like REINFORCE used in LLMs) is far more inefficient than standard human or supervised learning. In RL, as task horizons expand—for instance, asking an AI agent to perform two days' worth of coding before receiving a binary success/fail reward—the number of samples generated per FLOP drops precipitously.
```
BITS PER FLOP
|
+-------------------------+-------------------------+
| |
Samples Per FLOP Bits Per Sample
(Decreases as task horizons (Extremely low in naive
lengthen to days/weeks) RL vs Supervised Learning)
```
To evaluate learning efficiency, one can think of maximizing "bits per FLOP," which decomposes into:
$$\text{Bits Per FLOP} = \frac{\text{Samples}}{\text{FLOP}} \times \frac{\text{Bits}}{\text{Sample}}$$
Naive RL is severely constrained on both fronts, especially in terms of "bits per sample." Dwarkesh presents a stark comparison: imagine an untrained LLM with a 100,000-token vocabulary responding to the prompt *"The sky is..."*
- **Supervised Learning:** The model receives the explicit target token *"blue"*. It computes cross-entropy loss across its entire probability distribution in a single pass, immediately measuring exact errors for all 100,000 possible choices.
- **Naive Policy Gradient RL:** The untrained model guesses random tokens (*"halcyon"*, *"tall"*, etc.), receives a negative feedback signal for each incorrect guess, and must trial-and-error its way through up to 100,000 attempts just to randomly land on *"blue"* and obtain a positive gradient signal.
This highlights why naive policy gradients are so sample-inefficient: without dense intermediate feedback or direct target supervision (like AlphaGo's MCTS search target or supervised cross-entropy), the agent spends vast amounts of compute sampling fruitless trajectories in high-dimensional action spaces.
- **Bits-per-FLOP Decomposition:** Overall learning speed is bounded by how many samples can be computed per FLOP multiplied by how much information each sample provides.
- **The Sparse Reward Trap:** Unguided policy gradients require exponential exploration attempts to discover optimal tokens when action spaces are large (e.g., 100,000+ vocabulary tokens).
- **Supervised Advantage:** Supervised learning provides dense gradient information across every possible output token in a single forward/backward pass.
<ElicitationsGroup message="Explore related discussions from this episode:">
<Elicitation label="How MCTS avoids credit assignment in LLMs" query="Explain how Monte Carlo Tree Search provides step-by-step training targets that solve the credit assignment problem in LLMs."/>
<Elicitation label="Automated AI research loops" query="How can LLMs automate AI research experiments, and where do they still get stuck?"/>
</ElicitationsGroup>
`02:15:00` The Information Inefficiency of Reinforcement Learning
The foundational difference in information efficiency between supervised learning and reinforcement learning (RL) can be mathematically illustrated by examining bits learned per sample. In supervised learning utilizing cross-entropy loss, an algorithm learns based on the negative log of the pass rate once it receives a label. Conversely, in a standard RL setup where an agent randomly guesses actions to see what works, the signal behaves like the entropy of a binary random variable. A major complication is that the sampling distribution is dictated by the current policy. If an untrained policy has a microscopic probability of sampling the correct action—such as the token "blue" out of a massive vocabulary—it will almost never receive a success signal. Instead, 99.99% of the time, the model merely learns the highly inefficient lesson that incorrect words are wrong, generating minuscule parameter updates.
When mapping out the bits learned per sample against the pass rate on a log scale, the divergence between the two paradigms becomes stark. With a standard large vocabulary size of 100,000 tokens, the initial pass rate sits at a bleak $1/100,000$. While supervised learning maintains a clean, steady learning signal across this entire axis, standard RL produces a flat line near zero for the vast majority of early training. In many complex RL problems, models spend their entire computational budget in this low-pass-rate regime, risking never encountering a single success signal to begin hill-climbing. This highlights the critical importance of proper initialization to ensure a non-zero baseline pass rate.
A powerful mechanism to counteract this sample inefficiency is the use of soft targets and knowledge distillation. When an algorithm has access to full logits rather than a sparse, one-hot token representation, the entropy of the target distribution is far higher, embedding vastly more "dark knowledge" and information per sample. AlphaGo leverages this principle elegantly; the policy network is not trained merely to imitate the single discrete action selected by the Monte Carlo Tree Search (MCTS), but rather to minimize the KL divergence against the entire MCTS probability distribution.
- **Entropy of Soft Labels:** While the entropy of a rigid one-hot label is zero, a soft logit distribution yields a high entropy value that accelerates sample distillation.
- **Scientific Validation:** The empirical value of this soft-target distillation could be scientifically isolated by retraining an AlphaGo policy network strictly on the final MCTS-selected actions rather than the full search distribution.
- **Stable Infrastructure:** Because AlphaGo reduces its RL loop to supervised learning targets on values and policy distributions, it completely bypasses the need for complex, volatile distributed infrastructure required to maintain strict on-policy constraints.
---
`02:20:00` The Power of MCTS and Early Automated AI Research
The elegance of AlphaGo's search framework lies in its circumvention of the classic exploration problem. By utilizing MCTS to suggest a strictly improved target at every step, the system never has to initialize at a 0% success rate or rely on explicit temporal difference (TD) error learning. The MCTS policy acts as a reliable guide that stays consistently ahead of the raw network's capabilities, ensuring a clean, stable supervision signal. Transitioning to the broader topic of AI self-improvement, this iterative optimization loop mirrors recent experiments in automating AI research. Using advanced frontier models like Claude (Opus 4.6 and 4.7) as an automated coding assistant loop offers concrete insights into what parts of the scientific process can currently be delegated to machines.
Today's models excel exceptionally well at hyperparameter optimization within an open-ended search space. Rather than relying on rigid grid searches or traditional Bayesian optimization for learning rates and weight decay, an LLM agent can autonomously inspect a training run, identify that gradients are flattening in a specific layer, rewrite the code to inject a new data augmentation, and aggressively grind a performance metric. Furthermore, they are highly proficient at executing concrete experimental protocols. By wrapping the model in a dedicated execution skill, a researcher can simply define the parameters—specifying the exact X and Y axes for a visualization—and the agent will run the pipeline, compile the plots, and generate an analytical report of the findings.
- **MCTS vs. Raw Network:** The win-rate curve of the MCTS-guided policy consistently sits above the raw policy network, providing a continuous gradient for optimization until the two distributions converge.
- **The "Experiment" Skill:** Automating the inner loop allows researchers to offload the mechanical pipeline of running code, plotting results, and compiling data summaries to the model.
---
`02:25:00` Lateral Thinking, The Bitter Lesson, and the Outer Loop of General Intelligence
Despite their engineering proficiency, contemporary LLM research agents suffer from severe limitations when navigating complex, branching experimental paths. When visualizing research as a tree structure—where nodes represent successful, mixed, or failed experiments—human researchers frequently perform "lateral thinking," stepping back from a dead-end track to re-evaluate from first principles. Current AI assistants, however, easily fall into narrow rabbit holes, such as over-optimizing off-policy MCTS relabeling code when the underlying hypothesis is fundamentally flawed. They struggle to identify macro-bottlenecks or question their own core objectives, requiring human intervention to spot infrastructure bugs or redirect the strategy.
```
[Research Tree Metaphor]
Root (Core Hypothesis)
├── Branch A: Hyperparameter Optimization (High AI Success)
└── Branch B: Structural Algorithmic Changes (Frequent Dead Ends)
├── Sub-experiment B1 (AI hyper-focuses here)
└── Sub-experiment B2 (Requires human lateral thinking to abandon Track B)
```
This underscores the need for highly verifiable, rapid-iteration environments like Go to train automated scientists. Go provides an ideal testing ground because its outer verification loop—the win rate against a formidable baseline like KataGo—is un-hackable, while its inner loop demands sophisticated research engineering, scaling law predictions, and distributed systems management. Deep learning pioneer Ilya Sutskever has noted that great research requires a profound, almost faithful intuition for what ideas *should* work, allowing a scientist to persevere through weeks of broken code knowing that a failure is a temporary bug rather than a flawed premise. Training AIs in rigorous environments may eventually teach them to distinguish between an implementation bug and a broken theory.
When scaling these automated pipelines, AI labs face the challenge of algorithmic stackability. Individual researchers or parallel AI agents often discover disparate compute multipliers that look promising in isolation but fail to stack harmoniously during massive, centralized training runs due to unforeseen mathematical interactions. Ultimately, the "bitter lesson" of AI history dictates that raw compute, energy, and parameter scale are the true historical drivers of intelligence, rendering many temporary algorithmic optimizations obsolete as hardware transitions from V100s to Blackwell-generation GPUs. However, establishing rigorous outer loops remains imperative. Just as DeepMind's early work on Atari, Go, and StarCraft yielded critical institutional knowledge that translated directly into building modern large language models, training automated research agents on fast, simulation-bound games will likely provide the transferable capabilities required to eventually solve real-world complexities like automated drug discovery.
> "The fact that LLMs are universal grammar means that they can kind of move at almost any level of the stack... They can think very locally, as well as step back and think, like, in very broad steps. And I think that's where a lot of the lateral thinking ability of humans kind of come from."
- **Transitory Optimizations:** Many algorithmic tricks featured in historical AI literature to speed up convergence lose their value entirely when paired with next-generation hardware architectures.
- **The Generalization Hypothesis:** Positive transfer ensures that an automated agent optimized to systematically solve complex constraints within verifiable game simulations will develop the structural coding and project management skills needed for broader macroeconomic utility.
`02:35:00` Institutional path dependencies and conclusion
Dwarkesh and Eric reflect on how institutional choices and tech-tree investments shape long-term AI development. A key point of discussion is how past bets can both empower and hobble an organization over different horizons. For instance, while some critics argued that Google lagged behind in the LLM race due to an over-reliance on older paradigms, Google's early, deep investment in custom hardware like TPUs initially seemed less impactful, but ultimately provided an enormous infrastructure advantage for massive scaling. This highlights how extremely difficult it is—even for humans with full historical data—to determine the truly optimal long-term research strategy.
Looking ahead, Eric emphasizes that although the recent AI boom has shifted massive focus toward LLMs, Go and Monte Carlo Tree Search (MCTS) remain heavily underexplored as clean, small-budget testbeds for fundamental reasoning. He points out a profound duality between explicit tree search and implicit LLM "thinking," noting that researchers can test high-level ideas about reasoning, RL, and search at a fraction of the compute cost required for modern language models. "It's not to say that I think we should have trees in our LLMs, but there is some very interesting duality between them," Eric explains.
To close out the conversation, Eric shares where listeners can dive deeper into the project and explore these concepts hands-on.
- **Website & Tutorial:** Eric's personal website is `evjang.com`, which hosts an interactive version of the tutorial and links to related materials.
- **Code Repository:** The project code is hosted on GitHub under the username `eric-jang` in the `auto-go` repository, which listeners can fork to reproduce the training pipeline and run their own experiments.
- **Recommended Reading:** Eric highlights his blog post *"As Rocks May Think"*, which expands on the broader philosophical thesis of what happens when "thinking" itself becomes a primitive building block in computer science.
## Notable quotes
> `08:17` AlphaGo's core conceptual breakthrough was using neural nets to make this search problem tractable. — Eric Jang
> `20:00` If you had a very powerful computer, there are no probabilities. You can just compute the true average of what the mean action value is... The interpretation of $Q$ is what is the expected action value under the random distribution induced by some random search process. — Eric Jang
> `30:00` In deep learning, initialization is everything. You always want to initialize your research project to something as close to success as possible. — Eric Jang
> `47:11` Because Go is such a combinatorially complex game, you cannot afford to build the tree in advance and then search it. You must search while building the tree. — Eric Jang
> `01:15:00` 10 steps of neural network vectorized distributed representation thinking is able to amortize and approximate to a very, very high fidelity a nearly intractable search problem. — Eric Jang
> `02:15:00` The fact that LLMs are universal grammar means that they can kind of move at almost any level of the stack... They can think very locally, as well as step back and think, like, in very broad steps. And I think that's where a lot of the lateral thinking ability of humans kind of come from. — Eric Jang
> `02:15:00` It's not to say that I think we should have trees in our LLMs, but there is some very interesting duality between them — Eric Jang
## Mentioned
Eric Jang, Dwarkesh Patel, AlphaGo, 1X Technologies, Google DeepMind Robotics, KataGo, David Wu, Jane Street, YouTube, Lee Sedol, AlphaGo Lee, AlphaGo Zero, AlphaFold, AlphaTensor, Ron Minsky, Dan Pontecorvo, Cursor, Claude, Opus 4.6, Opus 4.7, Ilya Sutskever, Google, TPUs, Blackwell, V100s
Jang称AlphaGo搜索机制解决了AI学习瓶颈
将树搜索与神经网络相结合，通过在每一步提供更优的训练目标，消除了强化学习中的信用分配难题。
Eric Jang深入解析了AlphaGo的基础机制，阐述了蒙特卡洛树搜索（MCTS）、策略网络与价值网络如何协同运作，将难以处理的搜索空间转化为可控的分摊计算问题。对话探讨了KataGo与现代算力成本的降低如何让仅需数千美元即可训练出世界级围棋机器人成为可能。随后，Jang与Patel将MCTS与大语言模型（LLM）中的强化学习进行了直接类比，重点指出了相较于搜索引导的目标分布，标准策略梯度强化学习为何会面临信用分配难题。最后，他们审视了自动化LLM研究闭环的现有能力与局限，探讨了前沿模型虽然擅长超参数调优但在横向思维上仍面临挑战的现状。
得益于现代算法架构优化（如 KataGo）与 AI 辅助编程，如今仅需数千美元的租赁算力即可从零复现出达到 AlphaGo 水平的围棋 AI，相比当年数百万美元的研发门槛大幅降低。 (0:00)
AlphaGo 依托共享特征骨干的双头神经网络化解了围棋近乎无穷的搜索空间：价值网络通过评估胜率直接截断搜索深度，策略网络则将分支广度收窄至少数高质量候选着法。 (25:00)
在自我博弈中，蒙特卡洛树搜索（MCTS）在每一步都生成经过充分前瞻的更优动作目标，既绕开了传统强化学习棘手的信用分配问题，又通过策略蒸馏将测试时算力内化进网络的单次前向传播中。 (60:00)
对比大语言模型当前的强化学习，朴素策略梯度在每 FLOP 信息利用率上远低于监督学习；而前沿 AI 智能体虽已能高度自动化地执行实验与微调超参数，在跳出科研死胡同等宏观思维上仍面临显著瓶颈。 (132:02)
Reading summary
Revisiting AlphaGo reveals how combining deep neural networks with Monte Carlo Tree Search bypasses worst-case combinatorial complexity, offering a dense supervisory signal that contrasts sharply with the sample inefficiencies of naive reinforcement learning. Eric Jang demonstrates that modern algorithmic refinements, off-policy stabilization, and hardware progress now allow individuals to reproduce world-class Go systems for a few thousand dollars.
Amortizing Combinatorial Search via Dual-Head Networks (8:17)
A standard 19x19 Go board presents up to 361 legal moves initially and game lengths of 250 to 300 moves, producing an intractable state space on the order of 361^300 possible trajectories. AlphaGo overcomes this combinatorial barrier by using neural networks to guide Monte Carlo Tree Search (MCTS), truncating both breadth and depth. A policy head narrows the exploration space to a subset of plausible actions, while a value head evaluates leaf nodes directly as a scalar win probability, bypassing the need for exhaustive, terminal rollouts.
Step-by-Step Distillation Versus Sparse Policy Gradients (60:00)
In AlphaGo's self-play training loop, MCTS functions as an internal supervisor that generates a refined visit-count distribution at every visited board state. By supervising the policy network to directly predict these search-refined probabilities, the architecture distills multi-step forward lookahead into single forward passes. Jang and Dwarkesh Patel contrast this dense, move-by-move supervision with naive policy gradient reinforcement learning used in large language models. In long text generation, standard policy gradients treat multi-token sequences as a single action, assigning scalar outcome rewards that suffer from quadratic variance growth and severe credit assignment bottlenecks—described by Andrej Karpathy as 'sucking supervision through a straw.'
Off-Policy Funnels and Empirical Training Realities (121:09)
Although naive off-policy training risks destabilizing value estimates when sampling states an agent would never visit, Jang frames replay buffers through the lens of Dataset Aggregation (DAgger). In competitive two-player games and physical robotics, off-policy trajectories provide a corrective 'tube' of perturbed states that teach the agent how to recover after an opponent or environment forces it off an ideal path. However, Jang stresses that scaling laws and algorithmic optimizations only hold after eliminating basic software defects, warning researchers against studying scaling behavior on broken codebases.
Compute Collapse and Simplified Architectures (115:00)
While AlphaGo Zero consumed roughly 3x10^23 FLOPs in 2017, contemporary compute costs have collapsed by multiple orders of magnitude. Jang replicated a high-performance Go bot using a $10,000 compute grant from Prime Intellect—spending $4,000 on exploration and $3,000 on the final training run. Modern implementations eliminate complex asynchronous collector-pusher architectures in favor of simple synchronous self-play and training loops. Furthermore, advancements in general hardware render complex architectural tweaks and heuristic training targets largely secondary, aligning with Rich Sutton's 'Bitter Lesson.'
The Search Duality Between Games and Language (107:30)
Applying game-tree search directly to language models remains difficult because text presents an open-ended vocabulary of tens of thousands of tokens per step. Under such expansive branching factors, traditional exploration heuristics like the Predictor Upper Confidence Bounds applied to Trees (PUCT) formula break down because identical child nodes are rarely revisited. Even so, Jang notes a compelling conceptual duality between the explicit search trees of Go and the implicit, step-by-step reasoning internal to modern transformer forward passes, suggesting Go remains a uniquely cost-effective sandbox for investigating foundational reasoning principles.
While automated AI research agents excel at local hyperparameter tuning and execution pipelines, they continue to lack the lateral thinking required to step back from dead-end hypotheses. Jang emphasizes that deterministic, perfectly verifiable environments like Go remain vital, low-cost proving grounds for testing search, credit assignment, and automated reasoning mechanisms.
## Background
本期节目由 Dwarkesh Patel 于 2026 年 5 月 15 日在个人播客及 RSS/Web 平台发布。节目中邀请了前 1X Technologies AI 副总裁及前 Google DeepMind 研究员 Eric Jang，深入探讨如何利用现代 AI 编程工具与算力从零复现并改进 AlphaGo，并借此反思当前大语言模型强化学习机制与自动化科研的前沿发展。
## TL;DR
Eric Jang 拆解了 AlphaGo 背后的核心智能原语——蒙特卡洛树搜索（MCTS）、深度神经网络价值/策略头以及自对弈强化学习闭环。对话深入对比了 AlphaGo 逐步提供高质量改进标签的监督本质与当前大语言模型通过策略梯度（Policy Gradient）长程信用分配时高方差、低信息效率的痛点，并分析了 MCTS 难以直接套用于文本生成的原因。最后，Eric 结合自己的 auto-go 开源项目与使用 Claude 搭建的 Autoresearch 自动化实验循环，深入探讨了“苦涩的教训”、算力平替工程技巧以及通用人工智能前夜自动化研究员的演进边界。
## Atmosphere & style
对谈洋溢着硬核极客与第一性原理思考碰撞的纯粹智力快感，既有严密推导数学原理的从容，也有亲历工程踩坑后的生动幽默。Eric 表达通透且极具画面感，从六台 Dell 电脑拔电的机房趣事到洛伦兹吸引子与混沌边缘的高妙类比，娓娓道来；Dwarkesh 则以极高的敏锐度不断提炼概念本质并抛出直击痛点的诘问。整场对话既像是一堂高密度的顶级前沿 AI 架构大师课，又带着一种回望历史来预见未来智能大爆发的深邃与敬畏感。
## Bulletin
- `01:55:00` 算力成本暴降使顶级强化学习系统的复现门槛降至个人级：得益于开源项目 KataGo 的算法优化、现代云算力及 LLM 编程工具，2016 年 DeepMind 耗费数百万美元研发的顶级围棋 AI，如今个人仅需约 1 万美元云算力额度即可完成高质量复现与闭环训练。
- `01:03:00` AlphaGo 的本质是转化为稠密监督学习的改进算子：MCTS 彻底避开了传统策略梯度的长程信用分配与高方差问题，在对局的每一步通过前向搜索生成严格更优的动作分布，并将这一包含暗知识的软标签分布直接蒸馏给策略网络。
- `01:45:47` 大模型难以直接复用 MCTS 的根因在于动作空间与价值截断的缺失：LLM 词表庞大且语义开放，缺乏精准的中间步骤价值评估网络；且 PUCT 探索项在海量分支下几乎不会重访同一子节点，导致基于访问计数的置信区间探索机制失效。
- `02:12:02` 纯强化学习存在信息获取极度低效的“对数绝望区间”：在极低成功率下二元胜负反馈几乎不含有效梯度信息（Bits/Sample 趋近于 0），相比监督学习单样本巨大的 -log(P) 纠错信号，复杂 RL 系统必须依赖预训练或人类专家数据冷启动才能跨过冷启动门槛。
## Key takeaways
- `01:15:00` 深度神经网络以连续分布式表征摊销了组合爆炸难题：仅有十几层的简单卷积网络，通过数步前向传播成功摊销了状态空间高达 361^300、传统符号搜索几乎不可解的指数级博弈树模拟。
- `27:30` 策略网络与价值网络构建起双向剪枝闭环：AlphaGo 利用策略网络裁剪搜索广度（Breadth），依靠价值网络截断推演深度（Depth），从而彻底淘汰了传统 MCTS 中极其低效的随机模拟（Rollout）。
- `01:05:40` MCTS 摆脱了传统强化学习的长程信用分配难题：无模型 RL 试图在充满噪声的终局胜负中反向推导早期动作的功过，而 MCTS 在对局的每一个单步都能现场构造出更优动作，使整条轨迹的所有样本均能作为高质量监督目标。
- `01:03:00` 测试期推理算力被有效沉淀进网络参数：通过自对弈将上千次 MCTS 模拟提炼出的陡峭访问计数分布作为监督目标训练网络，测试期昂贵的搜索预算被压缩进单次前向传播的权重中，并支持后续进一步叠加算力扩展能力。
- `37:30` 小规模模型搭配冷启动的极高性价比：一个不到 300 万参数的 10 层小 ResNet 仅靠监督学习的即时直觉即可击败绝大多数人类业余棋手；科研项目应始终从监督学习冷启动等行之有效的基线起步，而非盲目尝试白板自对弈。
- `01:10:00` 残局自举（Value Bootstrapping）需强制防止过度投降：自对弈中若过于频繁提前认输，会导致经验回放池严重匮乏死活与残局样本，造成价值网络失真甚至崩溃，工程中需强制保留一定比例的对局完整下到底。
- `01:20:00` 宏观可预测性与神经网络的“混沌边缘”特性：虽然中后盘单步微观棋子状态呈现极端敏感的混沌特性，但宏观期望胜率具备平滑连续的几何结构（类似洛伦兹吸引子），神经网络恰好工作在混沌边缘提取宏观模式。
- `01:30:00` LLM 采用单步强化学习以控制策略梯度的方差爆炸：朴素策略梯度方差与时序长度呈二次方增长，大模型普遍将整段长文本生成抽象为时间步 T=1 的单步动作以避免交叉相乘，但也带来了监督信号极度稀疏的“吸管效应”。
- `01:45:47` 语言建模缺乏直接套用 MCTS 的底层前置条件：语言生成空间开放且不可逆，不存在围棋般严谨闭环的博弈规则与终局判定，缺乏能精准评估中间推理片段的判别器网络，使得树搜索展开极易陷入平庸局部解。
- `01:50:00` 博弈游戏 Scaling Law 证明测试期算力与模型规模的可替代性：推理期搜索模拟次数与模型训练算力呈现平滑对换关系；但在验证扩展定律前必须先搭建毫无 Bug 的稳固基线，防止拟合出虚假的缩放曲线。
- `02:01:09` Off-Policy 经验回放结合 DAgger 机制提供防偏离鲁棒性：AlphaGo 从回放池中提取历史偏离局面对其重新进行 MCTS 打标，形成了以最优轨迹为主干、周围环绕纠偏样本的高维管道，有效防止策略在遭遇意外脱轨时复利出错。
- `02:12:02` 纯策略梯度在低胜率任务中的信息效率极低：二元胜负反馈在极低通过率下信息熵接近于 0，负面惩罚几乎无法指明修正方向，因此必须借助预训练热启动将初始通过率提升到具备有效梯度的区域。
- `02:22:16` 自动化科研循环（Autoresearch）在渐进优化与横向思维上的分化：当前大模型在代码重构、超参数调优和基准指标压榨方面表现卓越，但在实验进入死胡同时缺乏基于第一性原理抽身并更换技术路线的横向思维能力。
- `02:30:00` 硬件演进正在抹平算法技巧并印证“苦涩的教训”：针对特定硬件设计的复杂优化技巧往往存在负向交互且不可叠加，随着算力呈数量级跃升（如 Blackwell GPU），早期堆砌的大量特化 trick 逐渐失去存在价值。
## Detailed walkthrough
### `00:00` 为何在休假时从零复现 AlphaGo
在访谈开篇，主持人 Dwarkesh Patel 介绍了嘉宾 Eric Jang——他此前曾担任具身智能公司 1X Technologies 的 AI 副总裁，更早前则是 Google DeepMind 机器人团队的高级研究科学家。近期处于休假状态（sabbatical）的 Eric，并没有选择去海滩度假消磨时光，而是专注于从零重写、改进并深入钻研 AlphaGo。Dwarkesh 抛出疑问：在现代 AI 突飞猛进的当下，为什么还要回过头去啃 AlphaGo？
Eric 解释道，自己天生热爱动手造物，而 2014 至 2016 年间 AlphaGo 的横空出世，正是将他彻底吸引到 AI 领域的决定性契机。在那之前，围棋长期被数学与计算机界公认为对于传统穷举搜索算法不可逾越的鸿沟，其计算复杂度高到难以想象；然而深度学习却不可思议地攻破了这道难关。Eric 的专业背景集中于机器人的深度神经网络，在机器人控制中，神经网络做出的决策往往偏向于“直觉”与即时反应；但在围棋中，落子决策背后却蕴含着极深博弈树搜索的逻辑，"令人着迷的是，一个仅有区区 10 层的神经网络，竟然能够摊销（amortize）如此深度的博弈树模拟过程。"
此外，算力门槛的演变也是促使他动手的催化剂。2020 年，来自 Jane Street 的研究员 David Wu 发起了开源项目 KataGo，通过算法优化实现了从零训练（tabula rasa）高水准围棋 AI 所需算力的 40 倍降幅。如今多数人类围棋专业棋手训练时使用的正是 KataGo。在当下 LLM 辅助编程（LLM coding）与廉价算力租用的加持下，当年 DeepMind 耗费数百万美元算力、由数十位顶级科学家组成的团队才能攻克的历史级工程，如今个人仅凭几千美元的租赁算力便可完整复现。Dwarkesh 提醒听众，本期节目本质上是一堂黑板讲义（Blackboard lecture），强烈建议观众在 YouTube 等视频平台同步观看黑板上的数学公式、图表推导与围棋走位。
- **核心人物与项目**：Eric Jang（前 1X Technologies AI 副总裁、Google DeepMind 资深研究科学家）；David Wu（Jane Street，KataGo 作者）。
- **算力对比**：KataGo 相比早期系统实现了 40 倍的算力缩减；曾经数百万美元的科研项目如今降维至数千美元租赁成本。
---
### `02:15` 围棋基本规则与 Trump-Taylor 规则系统
为了让观众彻底理解 AlphaGo 算法的基础，Eric 在黑板上画出棋盘，从最基础的规则机制开始演示。围棋的机制极为精炼：黑先白后，双方轮流在交叉点落子，核心目标是在棋盘上围出尽可能多的领地。黑子与白子在棋盘上的争夺遵循“气”的法则——当一个棋子或一组棋子的上下左右四个直接相邻的十字交叉点全部被对方棋子堵死时，这枚棋子就被“切断了氧气”，判定为死子并从棋盘上移除，提子一方获得这片空间与交叉点。值得注意的是，气只沿十字交叉连通，对角线并不能提供气的连接。
```mermaid
flowchart TD
accTitle: 围棋提子与气的判定逻辑
accDescr: 展示十字相邻交叉点如何决定棋子生死以及提子过程
subgraph LibertyCheck["相邻交叉点（气）判定"]
CenterStone["中心棋子（白）"]
Up["上方十字邻点"]
Down["下方十字邻点"]
Left["左侧十字邻点"]
Right["右侧十字邻点"]
CenterStone --- Up
CenterStone --- Down
CenterStone --- Left
CenterStone --- Right
end
subgraph CaptureResolution["提子解析"]
AllSurrounded{"四个十字邻点<br/>是否全被黑子占据？"}
Alive["白子存活（保有气）"]
Dead["白子死亡（断氧）<br/>黑方提子并占领交叉点"]
end
Up & Down & Left & Right --> AllSurrounded
AllSurrounded -- 否 --> Alive
AllSurrounded -- 是 --> Dead
```
Eric 特别区分了现实中的人类规则（如中国规则、日本规则）与计算机专用的 Trump-Taylor 规则。Trump-Taylor 规则是一套为了彻底消除人类规则中各种逻辑歧义而设计的公理化系统，也是现代所有围棋 AI 进行训练与状态结算的底层准则。例如在传统人类比赛中，主动下进一个没有任何气且无法立刻提走对方的空格属于“自杀”，是严格禁止的犯规走法；但在 Trump-Taylor 规则中，允许这种落子，棋子落下后系统在同一物理节拍下立即将其判定为死子移出棋盘，其数学表述与逻辑实现对计算机极其友好。
在与 Dwarkesh 模拟对弈时，Eric 演示了“叫吃/打吃”（Atari）的概念：当黑棋落下并封住白棋仅剩的一个空邻点前夕，局势便“非常类似于国际象棋里的将军（check）”。如果防守方不立即在空位落子补气，对方下一手就会彻底提走这枚棋子。Eric 借此揭示了围棋的宏观哲学：落子被吃并不一定意味着失败，棋手往往会主动放弃局部的几颗弃子，以换取在棋盘其他区域构筑更广阔的势力范围，“在围棋中，你完全可以输掉局部的战役，但赢下整场战争（lose the battle but win the war）”。随着棋盘尺寸的扩大，这种微观死活与宏观大局之间的动态拉扯会展现出无与伦比的美感与复杂度。
---
### `05:00` 终局判定：人类共识与算法裁决的分歧
随着推演深入，Eric 指出了围棋终局判定中一个人类直觉与计算机算法容易产生分歧的临界情况。在某个棋形中，黑棋在外围形成合围，内部困住白棋。直观上，大部分人类棋手扫一眼棋盘就会达成默契：黑棋已经构筑了铁桶般的包围网，白棋在内部绝无做活可能，已是死棋；但从极其严谨的图论角度看，只要白棋存在破开重围、与外围己方棋子重新联通的理论可能，局面就仍存变数。
人类社会与计算机在终局结算上的底层范式大相径庭：
- **人类的解决方式（共识机制）**：对局双方依次声明“我认为游戏已结束”，双方均同意停手后，再共同指认“我认为这些是死子”。如果双方达成共识，则按照中国规则或日本规则进行点目；如果对方不认同某块棋已经死透，那么双方就必须继续落子实操演示，直到两人的“价值函数”（value function）在实际棋局演进中达成一致。
- **计算机的解决方式（Trump-Taylor 规则）**：该规则完全摒弃了主观协商，依赖确定性的算法几何计数。终局时，计算机首先清点每位棋手盘面存活的棋子数；其次，计算只与己方棋子直接相连、未接触任何对方棋子的空白交叉点目数。如果某个空白交叉点同时接触了黑子与白子（即双方争议区或未收的公气），则双方均不得分。
这种机制会导致一种反直觉现象：如果白棋在局部虽然已经注定死亡，但只要未被黑棋在物理上彻底提走，在 Trump-Taylor 计数体系下，这几枚棋子本身以及其包围的狭小区域可能仍会被算作白方的分数。因此，终局的严格触发条件被定为：要么有一方主动投降认输（resign），要么双方连续选择弃权停着（pass）。
---
### `07:05` 构建 AI 的第一步：输入表示与延迟奖励挑战
梳理完围棋的底层公理后，讨论正式切入如何用 AI 算法攻坚围棋。Eric 提出，想要理解 AlphaGo 的威力并亲自编写代码复现，首先要还原最原始的博弈树搜索架构，然后再引入深度学习将其层层加速。
```mermaid
flowchart LR
accTitle: 围棋 AI 的状态输入与博弈树分叉
accDescr: 展示棋盘离散状态如何输入给神经网络并展开为具有延迟奖励的博弈树
subgraph StateEncoding["棋盘状态编码"]
RawBoard["棋盘网格<br/>（黑、白、空）"] --> TensorInput["三值离散张量<br/>[0, 1, 2] 或 One-Hot"]
end
subgraph DecisionSearch["博弈分支与延迟奖励"]
TensorInput --> Agent["AI 决策模型"]
Agent --> MoveA["候选分支 A"]
Agent --> MoveB["候选分支 B"]
Agent --> MoveC["候选分支 C"]
MoveA & MoveB & MoveC --> DeepTree["博弈树深层展开<br/>（无即时局部奖励）"]
DeepTree --> EndGame["终局结算<br/>（极度延迟的胜负信号）"]
end
```
在最基础的抽象中，围棋是一场两人零和博弈。当轮到 AI（假定执白）行动时，首要问题是如何将人类的棋盘离散状态转化为计算机能够读取的张量输入。由于棋盘每个交叉点存在三种互斥状态（黑子、白子、空格），AI 的输入层至少需要编码三种状态——例如使用最简单的标量映射（0 代表空、1 代表黑、2 代表白），或者扩展为多通道的二进制特征图。
然而，摆在 AI 面前最致命的难题在于**延迟奖励（Delayed Reward）**。当 AI 观察到盘面输入并面临数以百计的合法落子候选时，系统根本无法像其他简单视频游戏那样获得任何“即时局部奖励”（local reward）来评判眼前这一步的好坏。在围棋中，局部吃掉两个子可能在全局是大劣势，而看似弃子的一步却可能是几十手后的胜负手。在对局彻底落幕之前，你根本不知道当前这一手究竟是妙手还是昏招。Eric 指出，博弈树的展开深度极其惊人——哪怕翻开 1936 年的一盘人类古谱，对局手数也往往绵延数百手，直接让纯粹暴力前向遍历的搜索算法陷入计算崩溃。
### `10:00` 围棋的庞大状态空间与“不可解”神话
在标准的 19×19 围棋棋盘上，第一手棋理论上有 361 种落子选择。随着棋盘逐渐被填满，后续每一步的可选位置依次递减。一局对局通常需要经历 250 到 300 手，即便人类职业棋手有时会根据形势早早中盘认输，但如果按照规则（例如特朗普-泰勒规则，Trump-Taylor scoring）将死子全部提掉、完全行棋到底，整局棋的步数会扎扎实实达到 300 步左右。这意味着博弈树的深度高达 300 层。如果天真地将每一步的所有可能分支逐一展开，从最初的一个盘面出发，最终产生的分支数量级大约是 $361^{300}$——这个数字已经彻底碾压了全宇宙所有原子总数之和。
```mermaid
flowchart TD
accTitle: 围棋博弈树搜索的复杂度与转置合并
subgraph Tree["庞大博弈树的展开与合并"]
R["根节点 (初始状态)"] --> C1["第 1 步 (AI 落子)"]
R --> C2["..."]
C1 --> G1["第 2 步 (人类应对)"]
C1 --> G2["..."]
G1 --> T1["后续分支... (深度达 ~300)"]
T1 -. 状态等价.-> M["转置合并 (同形局/相同盘面)"]
end
subgraph Scale["规模对比"]
S1["树复杂度: ~361^300 种可能"]
S2["宇宙原子总数: 远小于此树分支"]
S1 -. 算力极限.-> S3["无法暴力穷举"]
end
```
德瓦克什询问什么叫“子节点的合并”（merging of children）。埃里克在棋盘上演示了“转置”（transposition）的概念：如果我先下 A 点，你下 B 点，我再下 C 点，你下 D 点；这与我先下 C 点，你下 B 点，我再下 A 点，你下 D 点，最终形成的盘面状态是完全一样的。也就是说，走子顺序不同，却能殊途同归到达同一个盘面。在图结构中，这两个分支其实可以收敛到同一个共享节点。尽管存在盘面对称性和转置能够精简空间，且每一步的候选数都在递减，但搜索空间的底数依然庞大到令人绝望。正因如此，计算机科学家们在过去几十年间普遍悲观地认为，围棋在本世纪内绝不可能被完全攻陷。
但围棋在本质上是一个确定性博弈（deterministic game），不含任何随机骰子或隐藏信息。理论上，只要算力无限，就可以在当前盘面穷举所有通向胜利的未来分支，推算出绝对的必胜策略，并确保每一步都将局势锁定在获胜的分支集合内。AlphaGo 最核心的理论突破，正是利用神经网络来指导搜索方向，使这个看似无限的搜索问题在有限算力下变为可能。
- 19×19 棋盘初始有 361 个空位，树深通常达到 250 至 300 层。
- 朴素树搜索的分支规模高达 $361^{300}$，远超宇宙原子总量。
- 围棋是纯确定性、完全信息博弈，理论上存在可解的必胜路径，瓶颈完全在于搜索算力。
---
### `12:30` MCTS 的节点数据结构与多臂老虎机灵感
在正式引入神经网络之前，埃里克先讲解纯粹靠搜索算法如何在巨大树中寻找最佳下法。显而易见，计算机无法在内存中一次性实例化整棵树，因此必须采用交互式构建的方式：从根节点出发，智能地判断应该重点探索和扩展哪些有价值的叶节点。AlphaGo 采用的树搜索框架，其动作选择逻辑深受多臂老虎机（Multi-Armed Bandit）经典算法 UCB1（Upper Confidence Bound 1）的启发。
UCB1 算法的核心思想是在每一步做决策时，选择能够最大化“平均价值 $Q(a)$”加上“探索奖励项”（exploration bonus）的动作：
$$\arg\max_a \left[ Q(a) + \text{Bonus} \right]$$
在代码实现层面，埃里克分享了蒙特卡洛树搜索（MCTS）中每个节点应包含的数据结构。埃里克打趣道，如果你让 LLM（比如 Claude 4.6）来“氛围编程”（vibe code）一个 MCTS，它基本上就能给出非常合理的节点结构设计；当然数据结构完全是“主厨自选”（chef's choice），只要逻辑自洽即可。在确定性游戏中，状态（State）与动作（Action）紧密绑定：因为没有环境随机性，从父节点指向特定子节点的那条边，就唯一确定了所采取的动作。
```mermaid
flowchart LR
accTitle: MCTS 节点核心数据结构
subgraph NodeStruct["MCTS 节点内部存储数据"]
direction TB
N["访问次数 N(s, a):<br/>被父节点选中的频次"]
Q["平均动作价值 Q(s, a):<br/>当前动作的历史平均胜率"]
P["先验概率 P(s, a):<br/>父节点推荐该分支的先验几率"]
C["子节点映射表 children:<br/>键值对指向下一层 Node"]
end
```
一个标准的节点通常维护四组关键数据：
- **访问计数 $N(s, a)$**：记录该子节点被其父节点选中的累计次数。
- **动作平均价值 $Q(s, a)$**：记录选择该动作后所获得的平均回报，反映当前动作从历史探索来看平均有多好。
- **先验概率 $P(s, a)$**：从父节点采取该动作的先验偏好，这一项在结合神经网络先验策略时至关重要。
- **子节点字典（children）**：类似经典链表或指针树结构的字典映射，用于展开后续的子节点。
---
### `15:00` PUCT 算法机制：探索与利用的博弈平衡
在 AlphaGo 中，实际采用的动作选择标准是基于 UCB 衍生改进的 PUCT（Predictor Upper Confidence bounds applied to Trees）。其评估公式在形式上与 UCB 非常接近，都包含两个核心部分：
$$\arg\max_a \left[ Q(s, a) + \text{exploration bonus} \right]$$
埃里克解释了公式内部“利用”（Exploitation）与“探索”（Exploration）两股力量的拉锯：
1. **利用项 $Q(s, a)$**：代表该动作的平均胜率。如果我们已经遍历了整棵博弈树并确切知道每个动作的真实终局回报，那么做决策时只需要单看 $Q$ 值最大的动作即可，根本不需要其他辅助项。
2. **探索奖励项（Exploration Bonus）**：因为树是边搜索边动态生长的，面对未知的胜率评估，算法必须拿出一部分算力去尝试那些尚未被充分探索的动作。
所有节点的访问计数 $N(s, a)$ 初始值都为 0。当根节点的总搜索次数在不断增加时，如果某个动作 $A$ 从未被选中过（其分母或者自身计数极小），它的探索奖励项就会变得非常巨大，从而迫使算法在下一次选择时去尝试它；相反，如果某个动作已经被频繁探索了多次（比如 10 次里面选了 10 次），它的探索奖励项就会迅速衰减变小。
```mermaid
flowchart TD
accTitle: PUCT 动作选择中的动态平衡
subgraph Selection["PUCT 动作选择公式: argmax [ Q(s,a) + Bonus ]"]
direction TB
QTerm["Q(s, a) 利用项:<br/>模拟推演的平均胜率"]
BTerm["探索项 Bonus:<br/>随未探索程度增大，随访问增多衰减"]
end
subgraph Dynamics["搜索阶段的重心转移"]
Early["初期搜索:<br/>访问次数少，Bonus 占主导，广泛试探"]
Late["后期搜索:<br/>N 增大，Bonus 极小，Q 值占主导，集中攻坚最优解"]
Early -->|模拟次数累积| Late
end
Selection --> Dynamics
```
德瓦克什用自己的话梳理了整套逻辑：在模拟（rollouts）过程中，算法一路向下推演至终局或评估点，统计自己赢了还是输了，并将多次推演的胜负结果平均后存入 $Q$。因此 $Q$ 本质上代表着“从该节点出发，我最终能赢下比赛的概率估算”。与此同时，探索项则在不断追问：“相对于其他备选路径，这条分支我到底看得够不够充分？”
由于在经典置信度公式中，分子中关于总访问次数的对数增长（$\ln N$）远远慢于分母中单一分支访问计数 $N(s, a)$ 的线性增长，搜索过程展现出极其优雅的收敛特性：在一开始，由于样本稀少，探索奖励项在 $\arg\max$ 中占据主导地位，促使算法广泛探索各大分支；而随着模拟次数越来越充分，探索项衰减至可以忽略，整个决策完全交由胜率 $Q$ 说了算——此时算法已经积累了足够多的胜负样本，能够极其自信地锁定真正能够导向胜利的落子路径。
- 节点初始化时访问计数为 0，探索奖励处于峰值。
- 单一分支被尝试的次数越多，其探索奖励衰减越快。
- 搜索重心随推演次数的递增，自然完成从“泛化探索未知”到“高置信度利用高胜率路径”的平滑过渡。
### `20:00` 围棋的确定性与概率的来源：蒙特卡洛采样的本质
在蒙特卡洛树搜索（MCTS）中，探索项（Exploration Term）的作用在于对懊悔度（Regret）施加上界约束，限制算法做出错误决策的代价。尽管 Eric Jang 坦言自己并未深究严格的数学证明（例如其懊悔界究竟是对数界还是平方根界），但该公式的形式经过专门调整，以适应围棋相比传统多臂老虎机（Multi-Armed Bandit）而言极其庞大的动作空间。
```mermaid
flowchart TD
accTitle: 围棋蒙特卡洛采样的概率来源
accDescr: 解释围棋虽然是确定性博弈，但由于算力有限引入蒙特卡洛随机采样分布
subgraph Deterministic["确定性博弈本质"]
A["围棋规则完全确定<br/>（不存在内在随机性）"]
B["理想全树遍历：算力无限<br/>直接计算真实确定性动作价值"]
end
subgraph Sampling["蒙特卡洛近似"]
C["算力受限：无法遍历全树"]
D["引入随机采样分布 P(action)"]
E["期望动作价值 Q<br/>基于随机搜索树的期望均值"]
end
A --> C
B -.->|现实算力不可行| C
C --> D --> E
```
针对“概率从何而来”这一问题，Eric Jang 提出了一个核心澄清：从本质上讲，围棋是一场完全确定性的零和博弈，没有随机发牌或掷骰子环节。如果拥有一台算力无限强大的超级计算机，全树可以直接展开，根本不需要概率，只需直接求出动作价值的真实确定性平均值即可。
在现实中，由于算力极其有限，在 AlphaGo 之前的计算机围棋程序普遍采用蒙特卡洛方法。概率并非源于游戏本身，而是来自于程序在随机生成的搜索子树上取期望动作价值（Q 值）的过程：
- **均匀随机采样的局限**：如果采用朴素假设，对棋盘上每一个合法落子位置赋予等概率（$1/\text{合法动作数}$），算法就必须在一棵极其弥散、分布极其稀疏的博弈树上做积分求平均。
- **重要性采样问题（Importance Sampling）**：在绝大多数棋局中，绝大部分落子路径的价值都极低（臭棋），只有极少数分支具有高价值。如果不对搜索空间加以引导，均匀采样就会将海量算力浪费在毫无意义的垃圾分支上。
---
### `22:15` 终局判定与反向回溯（Backup Step）的计算逻辑
随着树搜索沿选定分支持续向下展开，算法最终会抵达博弈的终局叶节点。在围棋规则（例如创普-泰勒计分法，Tromp-Taylor scoring）下，当棋盘上双方皆无可落之子或连续停着时，胜负便迎来严格且确定的判定——非赢即输。
```mermaid
flowchart LR
accTitle: MCTS 反向回溯流程
accDescr: 展示从终局胜负判定反向更新父节点Q值的回溯机制
subgraph Terminal["叶节点终局判定"]
L1["终局叶节点 1: 胜 (+1)"]
L2["终局叶节点 2: 负 (-1)"]
L3["终局叶节点 3: 胜 (+1)"]
end
subgraph Backup["反向回溯 (Backup)"]
P1["父节点动作价值 Q_B<br/>（子节点胜负结果加权平均）"]
Root["根节点动作价值 Q_root<br/>（层层向上反推）"]
end
L1 --> P1
L2 --> P1
L3 --> P1
P1 --> Root
```
当终局叶节点被赋予确定的胜负回报值 $u$（例如 $+1$ 表示胜，$-1$ 表示负）后，算法必须解决如何为先前的父节点及历史动作赋值：
- **回溯计算（Backup Step）**：父节点对应的平均动作价值 $Q$ 本质上是其下游所有终局叶节点胜负结果的加权平均值。权重取决于采样分布的偏置。
- **层层反推**：顺着博弈树的连接链条向上递归传递，动作 $B$ 的价值 $Q(B)$ 由其后续状态的期望结果决定，最终回溯更新到根节点。
- **纯符号搜索的算力瓶颈**：如果脱离神经网络仅靠暴力搜索，哪怕有了反向回溯，整个过程依然完全不可行（Intractable）。特别是在陷入劣势并试图扭转局面的复杂局面下，绝大部分采样动作带来的价值都接近于零，盲目搜寻能带来生机的极少数妙手就像大海捞针。
---
### `24:50` 人类的直觉与价值函数：解决搜索“深度”难题
人类顶级棋手在下围棋时，绝不会将棋局完全推演至最后一步。很多时候，职业棋手在距离终盘几十步甚至上百步之前，就能敏锐察觉局势走向并主动投降认输。
人类大脑在看一眼棋盘的瞬间，实际上是在隐式运行一个生物层面的**价值函数（Value Function）**：
- **摊销海量推演（Amortizing Playouts）**：人类棋手并非逻辑上穷举每一盘可能的终局，而是凭借日积月累结晶而成的盘面经验与直觉，在一瞥之间完成了海量可能走法的抽象均值估算，直接输出当前的胜率估计 $P(\text{win})$。
- **直觉决定去留**：棋手正是基于这瞬间生成的价值判断，决定是继续死拼还是投子认负。Eric Jang 赞叹这种机制非常优美：无需实际走完每一种可能，大脑在数秒之内便完成了惊人的宏观模拟。
- **压缩搜索深度**：人类的这种直觉启发了 AlphaGo 的设计思想——通过训练一个专门的深度神经网络（价值网络）来充当棋盘价值函数，算法根本无需将博弈树搜到最深处的叶节点，便能在半途截断搜索，极大缩减博弈树的搜索**深度（Depth）**。
Dwarkesh Patel 对此深有感触地指出，自己过去在备战采访时曾觉得 AlphaGo 的运行机制极其庞杂深奥；但一旦建立起这种直观拆解，就会发现整个架构非常通透。
---
### `27:30` 剪枝的双重维度与每手棋的 MCTS 循环重构
将传统围棋搜索与现代深度学习相结合的核心，可以凝练为对博弈树**广度**与**深度**的双向打击：
| 维度挑战 | 传统纯搜索难题 | 神经网络与 AlphaGo 的破解方式 |
|:--- |:--- |:--- |
| **搜索深度 (Depth)** | 必须推演至终局才能算出输赢，计算步数极长 | **价值网络（Value Network）**：看一眼棋盘即预测胜率，直接截断搜索深度 |
| **搜索广度 (Breadth)** | 棋盘落子点极多，均匀采样会导致算力极度分散 | **策略网络（Policy Network）**：模拟人类棋感，只在直觉上合理的少数优质动作中采样搜索 |
```mermaid
flowchart TD
accTitle: 每手棋的 MCTS 完整运行循环
accDescr: 展示 AI 从接收棋盘编码、执行 MCTS 搜索、落子、人类对手应对到重构全新搜索树的生命周期
subgraph StepAI["AI 决策回合"]
A1["接收当前棋盘编码状态<br/>（黑/白/空）"]
A2["启动 MCTS 搜索<br/>（结合策略与价值网络深入推演）"]
A3["选出最优动作并落子"]
end
subgraph StepOpponent["对手响应回合"]
H1["人类对手观察棋盘"]
H2["人类落子应对"]
end
subgraph NextCycle["下一轮搜索重构"]
N1["AI 接收全新编码棋盘"]
N2["彻底丢弃上一轮旧搜索树<br/>（重新设立根节点从零搜索）"]
end
A1 --> A2 --> A3
A3 --> H1 --> H2
H2 --> N1 --> N2
N2 --> A2
```
真实对局中每一步走法的具体交互生命周期如下：
1. **棋盘状态编码**：AI 读取用数值张量表示的棋盘状态（例如用 `2` 代表 AI 所持白子，`1` 代表人类所持黑子，`0` 代表空交叉点）。
2. **局部树搜索**：以当前状态为根节点，AI 结合神经网络展开深度 MCTS 模拟，权衡哪个落子动作期望回报最高。
3. **做出决策与对手交接**：AI 在物理棋盘（或数字界面）上下出选定的一子，棋权交回给人类。
4. **推倒重建树结构**：人类落子后，棋盘产生全新局势。此时 AI 会将上一轮耗费算力探索的整棵旧搜索树完全废弃，以人类落子后的新状态为全新根节点，从零开始重新构建并执行下一次 MCTS。MCTS 本质上是一个在神经网络辅助下、为每一手棋量身定制的单步实时规划算法。
### `30:00` 蒙特卡洛树搜索（MCTS）的单步重置与中场说明
Dwarkesh 总结了前面对蒙特卡洛树搜索（MCTS）的讨论：这套带有节点和子节点的数据结构是按“每一步棋”（per move）独立构建的。人类落子后，AI 会基于当前盘面初始化该结构，运行数千次模拟（每次模拟相当于在搜索树中向下探索一个新节点）。当所有模拟结束后，这些搜索信息会聚合成一个动作概率分布；AI 从中选出最佳动作落子，然后通常会抛弃这棵搜索树，等待对方落子后再重新启动这个过程。Eric 对此表示赞同并做了一个微调：“你不会把所有东西完全丢掉，我们会保留一小样东西留作后续使用。”
Dwarkesh 在播客中分享了制作卡片的幕后工具链：为了帮助听众复习核心概念，他希望通过 LLM 自动生成闪卡候选草稿。然而，单纯把多个 LLM 调用堆叠在一起效果并不好，因为该任务需要在大模型的不同阶段之间保持持久上下文——抓取黑板视频截图、生成配套的 SVG 图解、调用 Critic 进行审核，并根据反馈修改内容。因此，他使用 Cursor SDK 启动了常驻的 Agent，免去了手动编写复杂上下文支架和工具调用的工作。
```mermaid
flowchart TD
accTitle: 单步 MCTS 与行动决策循环
accDescr: 展示 AI 在围棋中每一步落子时如何重置和构建 MCTS，并保留关键信息供后续训练使用。
subgraph StepLoop["每步决策循环 (Per-Move Search)"]
A["当前盘面状态 (Current Board)"] --> B["运行 N 次 MCTS 模拟 (Simulations)"]
B --> C["输出动作概率分布 (Action Distribution)"]
C --> D["选择最优落子动作 (Make Best Move)"]
end
D --> E["重置搜索树以待对手落子"]
C -.->|"保留单步分布供后续训练"| F["经验回放缓冲区 (To Replay Buffer)"]
```
---
### `32:04` 神经网络的作用：价值网络、策略网络与归纳偏置之争
当搜索树的运作直觉建立之后，Eric 将重点转向“神经网络如何为搜索加速并提供类人直觉”。在 AlphaGo 框架中，存在两个核心网络头（或双头网络）：
1. **价值网络（Value Network）**：输入棋盘状态，预测当前局势的胜负输赢。这本质上是一个二分类问题（以单个 logit 形式输出预测）。
2. **策略网络（Policy Network）**：输出所有合法落子点的概率分布，即在候选动作空间上的分类预测（如 19×19 棋盘上的 361 维分类）。
在模型架构的选择上，Eric 指出具体的骨干网络架构（Backbone）其实并没有那么关键，Transformer 和 ResNet（残差卷积网络）都可以跑通。然而在预算有限的小数据阶段，ResNet 的性价比明显高于 Transformer。
Dwarkesh 追问为何 ResNet 在棋类任务中表现更好。Eric 解释道，卷积神经网络具有**局部卷积的归纳偏置（Inductive Bias）**。Transformer 凭借全局注意力机制，固然擅长跨越棋盘两侧连接远距离特征（例如 KataGo 论文就强调了聚合全局特征对联通盘面两侧胜负价值的重要性），但 Transformer 缺乏强硬的局部不变性先验，必须依靠海量数据去“从头学习”这些局部空间特征。Eric 坦言，自己曾多次尝试用纯 Transformer 跑通并打破 Go 的传统方案以省去各种工程技巧，但在算力与数据受限的情况下，ResNet 依然胜出。
- **价值预测**：预测标量胜率（$R^1$），属于类别/二分类损失。
- **策略预测**：输出棋盘网格点上的概率分布（19×19 对应 $R^{361}$），通过标准交叉熵损失（Cross-Entropy Loss）进行监督。
- **局部 vs 全局**：卷积网络的局部感受野天然契合局部棋子厮杀；全局注意力擅长跨盘面联动作战，但对数据量的饥渴度极高。
---
### `35:00` 空间与时间维度的归纳偏置：完全信息博弈与纳什均衡
Dwarkesh 提出一个进阶问题：既然 CNN 与 Transformer 的取舍取决于“空间”维度的全局与局部关系，那么在“时间”维度上又是怎样的？围棋目前只看当前状态，是因为它是确定性的完全信息博弈（Deterministic Full Information Game）；但如果在扑克（Poker）或《强权外交》（Diplomacy）这类涉及诈唬（Bluff）与过往承诺的游戏中，是否必须利用时序上下文去建模对手？
Eric 对此作了深入剖析：
- **完全信息博弈的单状态充分性**：围棋作为完全信息博弈，在数学上存在一个**纳什均衡策略（Nash Equilibrium Strategy）**。虽然在理论上，如果你确切知道对手有某种特定破绽（例如过度激进），你可以制定超越纳什均衡的针对性策略来反制他；但无论对手采取何种策略，单凭当前棋盘状态所做出的纳什均衡决策，就能保证 AI 处于不败之地（不输于任何其他策略）。AlphaGo 及其后续工作完全基于单步状态做决策，后验证明该纳什均衡在实战中已超越了所有人类手段。
- **不完全信息博弈与时序建模**：一旦跳出完全信息博弈，进入双人组队围棋（2v2 Go，需要建模队友的隐藏行为倾向）或《强权外交》这类非完全信息、部分可观测环境（Partial Observability），玩家就必须保留并聚合过往的历史动作序列，通过构建世界模型来推断其他参与者的意图。
```mermaid
flowchart LR
accTitle: 完全信息博弈与非完全信息博弈的决策依赖对比
accDescr: 对比完全信息博弈中仅需单步状态的纳什均衡与不完全信息博弈中依赖时序上下文的建模需求。
subgraph PerfectInfo["完全信息博弈 (如 1v1 围棋)"]
S1["当前棋盘状态 (Current State)"] --> NE["纳什均衡策略 (Nash Equilibrium)"]
NE --> Move1["超人类落子 (无需历史)"]
end
subgraph ImperfectInfo["不完全信息博弈 (如 2v2 围棋 / Diplomacy)"]
H1["历史对局轨迹 (Temporal History)"] --> OpponentModel["建模对手与盟友意图"]
S2["当前部分可观测状态"] --> OpponentModel
OpponentModel --> Move2["针对性决策 (Context-Dependent)"]
end
```
---
### `37:30` 神经网络架构与实验策略：冷启动的“先胜而后求战”
回到围棋 AI 的实际实现细节，Eric 介绍了最简易可行的网络输入与结构搭建方法：
- **输入表征设计**：棋盘状态被编码为类似图像的 3 通道张量（类比 RGB 三通道）——一个通道编码黑子位置，一个通道编码白子位置，第三个通道编码空位或棋盘尺寸掩码（Mask）。
- **双头网络拓扑**：将张量送入 ResNet 骨干抽取特征，随后分出两路预测头：一路输出标量值预测当前胜率（$R^1$ 的价值头），另一路输出全盘 361 个点的未归一化概率（$R^{361}$ 的策略头）。
在研发流程上，Eric 特别强调了“从容易的目标开始初始化”的重要性。最早的 AlphaGo 论文（AlphaGo Lee 版本）并没有一开始就让模型在空白状态下盲目自对弈（Tabula Rasa），而是先用人类专业棋手的高质量对局数据进行监督学习（Supervised Learning）预训练，然后再开启强化学习自对弈微调。尽管随后的 AlphaGo Zero 证明了从零自我博弈也能成功，但从工程实操与科研落地的角度来看，他极力推荐听众先从现成的人类专家数据进行冷启动：
> "就像在深度学习中初始化决定一切一样，你总是希望让你的科研项目初始化在尽可能接近成功的位置。"
- **自动化超参数调优**：可以借鉴 Andrej Karpathy 风格的 AutoResearch 进行自动化超参数搜索，快速找到足够好用的 ResNet 架构，而不需要在架构细节上过度纠结。
- **渐进迭代原则**：先通过监督学习打通全流程闭环，在验证了价值网络分类和策略网络预测能够收敛之后，再去挑战更复杂的强化学习自对弈机制。
### `40:00` 监督学习初始化与“凭直觉下棋”的原始神经网络
在开发复杂系统时，Eric Jang 分享了一条核心工程哲学：“永远从行得通的东西开始，让它变得更好；而不是从完全不灵的东西起步，再试图把它修好。”遵循这一理念，AlphaGo 的第一步并非直接从零开始强化学习，而是利用人类专家的历史对局进行良好的模型初始化（监督学习）。训练时，系统收集所有高水平专家获胜的棋局，让模型学习预测获胜方的落子动作；同时，无论当前棋盘局势谁优谁劣，网络都需要预测整盘对局的最终胜负结果（价值预测）。
Dwarkesh 提出疑问：在开局阶段，棋盘上可能仅落下了一两颗棋子，模型怎么可能知道谁能赢？Eric 解释道，如果有数十万盘棋的数据积累，由于任何给定的开局都会衍生出无数种对局可能，其中大约一半最终获胜、一半最终落败，因此网络在开局位置预测的胜率 logit 会自然收敛到 0.5 左右（居中状态）。随着棋局推进到数百步，胜率曲线才会逐渐分化，明确地走向 0 或 1。Dwarkesh 追问这种价值思考方式是否仅对专家数据成立，Eric 明确指出这与是否是专家数据无关，即便从零开始（Tabula Rasa）进行强化学习训练，模型也会展现出相同的收敛动态。
如果有人尝试“凭直觉编写代码”（vibe coding）复刻 AlphaGo，只要从 KGS 等线上围棋平台抓取专家对局数据，训练一个包含约 10 层网络、参数量不足 300 万的小模型，仅仅对策略网络的概率输出直接取最大值（`argmax`），就能得到一个极其惊艳的围棋程序。它完全不进行多步前瞻推演，纯粹靠单次前向传播的“直觉出击”（shoots from the hip），却已经足以击败绝大多数人类业余棋手。在着手引入复杂的树搜索之前，验证这个纯神经网络和围棋规则引擎的正确性，是绝佳的基础测试点。
- **初始化策略**：利用数十万盘人类高水平棋谱，进行动作预测（策略）与最终胜负判定（价值）的监督微调。
- **开局价值收敛**：开局阶段胜率自然预测在 0.5（胜负各半），随着步数增加，曲线才向 1（必胜）或 0（必败）两极发散。
- **原始网络性能**：一个不到 300 万参数、仅约 10 层的网络，无需推理搜索即可战胜大部分业余人类选手。
```mermaid
flowchart LR
accTitle: 监督学习到蒙特卡洛树搜索的演进路径
accDescr: 解释了从人类棋谱训练直觉网络到结合MCTS进行推演的演进过程。
subgraph SL["监督学习阶段 (Supervised Learning)"]
A["人类专家棋谱数据"] --> B["策略与价值网络 (约3M参数)"]
B --> C["即时直觉落子 (argmax)\n打败大部分人类棋手"]
end
subgraph MCTS_Integration["结合 MCTS 进阶"]
C -.-> D["提供先验概率 P(a) 与状态价值 V"]
D --> E["MCTS 四步循环：\n选择、扩展、评估、反向传播"]
end
```
---
### `43:08` 算力落差与 MCTS 的四步迭代核心
即便原始神经网络已经很强，若要达到超人类水平，仍需将其与蒙特卡洛树搜索（MCTS）深度结合。在实际执行 MCTS 时，关键超参数之一是每次落子前进行的模拟次数（`num_simulations`）。在常规训练或像 KataGo 这样的现代围棋引擎中，模拟次数通常在 200 到 2048 次之间；但在 2016 年 AlphaGo 对战李世石（Lee Sedol）的历史性对决中，DeepMind 为了榨干每一分棋力，将单步模拟次数推高到了数万次。
Dwarkesh 提到纪录片里对弈现场桌上摆着一台笔记本电脑，Eric 笑着指出那台笔记本只是终端界面，背后连接着庞大的 Google TPU Pod 集群。Dwarkesh 调侃道：“老实说，这有点不公平，李世石可没有用 $10^{22}$ FLOPs 的算力来下一记棋。”不过 Eric 强调，现代围棋 AI 在推理测试时早已不需要如此夸张的硬件算力。随着 MCTS 策略改进的自对弈迭代，强化学习会逐步把整个搜索树的推演负担“蒸馏压缩”进神经网络单次前向计算的权重之中。但在人机大战的历史关头，背后庞大的硬件集群无疑为 AlphaGo 提供了至关重要的胜算保障。
MCTS 的本质是围绕当前根节点展开的一个四步迭代循环，每次模拟都在同时推进四件事：
1. **选择（Selection）**：根据现有树结构找到最具潜力的路径；
2. **扩展（Expansion）**：当遇到未展开的中间局面时，为博弈树添加新的叶子节点；
3. **评估（Evaluation）**：利用神经网络计算新状态的局面胜率（$V_\theta$）；
4. **反向传播 / 备份（Backup）**：将叶子节点的评估结果向上沿路径回传，更新所有走法的动作价值。
- **模拟步数设置**：日常训练与轻量部署通常设为 200–2048 次，而当年人机大战采用了数万次的大规模并行搜索。
- **算力内化**：随着训练推进，原本需要大规模树搜索才能发现的精妙走法，会逐渐被蒸馏固化进神经网络参数中，使现代围棋 Bot 对推理算力的依赖大幅降低。
---
### `46:12` PUCT 选择机制与树扩展过程详解
在具体代码实现层面，搜索从当前对局盘面的根节点（Root Node）开始。假设以一个已落一子、剩余 8 个空位的 $3 \times 3$ 简化棋盘为例，神经网络首先在根节点进行评估，输出状态价值 $V_\theta$ 以及覆盖所有合法走法的先验概率分布 $P(a)$（对应 8 个可选动作，如 $P_1$ 到 $P_8$）。此时树的深度仅为 1，所有动作的访问计数 $N(a)$ 初始均为 0。
为了在搜索中平衡“利用已有高分路线”（Exploitation）和“探索未尝试路线”（Exploration），算法依据 PUCT（Predictor Upper Confidence bounds applied to Trees）准则来选择动作：
$$\text{PUCT}(s, a) = Q(s, a) + C_{puct} \cdot P(s, a) \cdot \frac{\sqrt{\sum_b N(s, b)}}{1 + N(s, a)}$$
在搜索伊始，因为所有动作的动作价值 $Q$ 和访问计数 $N$ 都是初始值，选择结果会完全由策略网络给出的先验概率 $P(s, a)$ 主导，首先挑选概率最高的子节点（例如 $P_1$ 对应的走法）。
选定该动作后，系统发现它是一个非终局（Non-terminal）节点，无法直接得出比赛胜负，于是触发**扩展（Expansion）**阶段。此时视角切换到对手（人类或其他玩家）的回合，将该状态输入策略与价值网络进行前向传播，根据对手可能的落子动作派生出一批新的叶子节点，并读取网络为这些新局面预测的状态价值 $V_\theta$。这一机制巧妙地用神经网络的即时价值估计，彻底取代了经典 MCTS 中随机走子直到对局结束（Rollout）的低效模拟。
- **PUCT 准则**：由评估价值 $Q$ 与探索增益项组合而成，探索项分子依赖总访问次数的平方根，分母由当前动作访问计数平滑，权重受先验概率 $P(a)$ 调节。
- **冷启动偏置**：在节点尚未被充分访问前，树搜索倾向于优先探索策略网络最看好的落子点。
- **网络替代随机模拟**：通过策略网络生成对手分支，并用价值网络 $V_\theta$ 直接对叶子节点打分，极大提升了搜索深度与准确度。
### `50:00` 扩展与评估：价值网络与 AlphaGo Lee 的快速走子折衷
在蒙特卡洛树搜索（MCTS）中，“扩展”（Expansion）与“评估”（Evaluation）紧密相连。当算法沿着树向下选择并抵达一个未完全展开的节点时，神经网络会介入并对该棋局做出直觉评估——从当前玩家的视角判断盘面胜率究竟有多高。围棋作为一种零和博弈（Zero-sum game），其数学对称性极佳：黑方视角的价值评估若为 $V$，则白方视角的价值即为 $1 - V$。因此搜索过程中切换行棋方时只需做简单的取反操作。这里的价值网络 $V_\theta$ 扮演了极为关键的“快捷剪枝”角色：它省去了从该节点一步步推演至终局的巨大开销，相当于在单次模拟中直接给出终局胜负的快速预判。
在 2016 年击败李世石的初代 AlphaGo Lee 中，DeepMind 采取了一种极具折衷色彩的评估策略：他们并未完全迷信单次神经网络的静态估值，而是将其与真实的快速走子（Rollout / Playout）进行了加权平均。具体而言，系统会使用轻量级策略网络让黑白双方快速自我对弈下到底，依据特隆普-泰勒规则（Tromp-Taylor rules）得出非赢即输的真实终局结果（$0$ 或 $1$），再通过公式 $\alpha V_\theta + (1-\alpha) z$ 将神经网络估值与终局实测值融合。这种设计能避免在复杂的残局阶段估值网络与客观事实脱钩（"untethered from reality"）。然而快速走子极其耗时，且随后的 AlphaGo Zero 证明了这一步骤纯属多余——仅靠高质量价值网络就能胜任评估。Eric Jang 在本次从零复刻 AlphaGo 时同样彻底去掉了 Rollout，这大幅精简了系统并带来立竿见影的提速。
```mermaid
flowchart TD
accTitle: AlphaGo 评估演进对比
accDescr: 对比初代 AlphaGo Lee 的混合评估与后续 AlphaGo Zero 的纯网络评估机制
subgraph Lee["AlphaGo Lee 混合评估机制"]
direction TB
Node1["叶子节点"] --> Net1["价值网络 V(θ)"]
Node1 --> Roll["快速走子至终局 (Rollout)"]
Roll --> Rules["Tromp-Taylor 规则结算 (0 或 1)"]
Net1 & Rules --> Blend["加权融合: α·V + (1-α)·z"]
end
subgraph Zero["AlphaGo Zero / 现代实现"]
direction TB
Node2["叶子节点"] --> Net2["纯价值网络单次前向推断 V(θ)"]
Net2 --> Result["直接产出胜率估值 (快且充分)"]
end
```
- **快速走子的实现取舍**：如果在模拟过程中再嵌套树搜索会产生可怕的“树中树”计算爆炸，因此 AlphaGo Lee 仅使用单一策略网络自我对弈下到终局。
- **算力冗余与剔除**：后续版本（AlphaGo Master、AlphaGo Zero）完全放弃了快速走子混合评估，使单次模拟仅需一次神经网络前向推断。
---
### `52:30` 反向回传与边建边搜：MCTS 的动态展开与访问计数
Dwarkesh Patel 尝试用直观语言总结这一机制：单次模拟本质上就是在搜索过程中向外多推演一个新节点。Eric 补充指出，如果整棵博弈树已经预先存在，模拟确实只是按照 PUCT（Predictor + Upper Confidence bounds applied to Trees）准则向下遍历；但围棋的状态空间具有恐怖的组合爆炸性，人类与计算机根本无法预先建构整棵树。"你不能先建树再搜索，你必须在搜索的过程中动态建树。"系统从初始根节点出发，只有在探索到未曾访问过的分支时才会向外动态生长出一层子节点。
当新节点被扩展并由神经网络打分后，算法进入第四步——“反向回传”（Backup / Backpropagation）。在这一阶段，新评估出的胜率值会沿着本次模拟走过的路径一路递归返回根节点。父节点对应动作的动作价值 $Q(s, a)$ 会根据所有经过该路径的历史模拟值计算累计移动平均值。与此同时，沿途各个节点的访问计数（Visit count, $N$）也会加 1。反复迭代数千次模拟后，访问计数的大小实际上反映了经过深度搜索后被验证为最优路径的置信度。树搜索最终给出的决策，正是由这些访问计数的相对比例所构成的全新策略分布。
- **四步闭环循环**：
1. **选择（Selection）**：依 PUCT 准则自顶向下挑选最具探索与利用价值的动作。
2. **扩展（Expansion）**：遇到叶子节点时展开候选走法。
3. **评估（Evaluation）**：调用 $V_\theta$ 评估新盘面的期望胜率。
4. **回传（Backup）**：自底向上更新整条路径上的 $Q$ 值与访问计数 $N$。
- **访问计数的意义**：访问计数不仅决定单步落子选择，在后续强化学习训练中它还是指导策略网络蒸馏学习的目标分布。
---
### `55:00` 为什么不能抛弃策略网络：从“隐式归一化”到自我改进算子
围绕系统设计的极简主义，两人展开了一场思想实验：既然每次扩展节点时，价值网络 $V$ 都会评估所有子节点的胜率（例如某个落子点胜率为 1，其余为 0），那为何不干脆把所有子节点的价值分数归一化作为策略分布，彻底丢掉策略网络（Policy Network）？Dwarkesh 猜测这可能是为了避免针对盘面上全部 361 个落点进行 361 次前向推断的高昂开销。Eric 澄清称，在现代深度学习工程中，将 361 个衍生棋盘打包成一个 mini-batch 进行批处理推断在算力上完全可行，算力开销并不是保留策略网络的决定性原因。
保留显式策略网络的核心价值在于两点：其一，单次前向推断中策略网络能提供极强的先验概率（Priors），在搜索起点直接大幅修剪掉大量荒谬的落子分支（Pruning）；其二，也是更关键的数学逻辑——蒙特卡洛树搜索在强化学习闭环中扮演的是“策略提升算子”（Improvement Operator）。MCTS 以策略网络给出的先验为起点，经过多步前瞻搜索，输出一个质量显著高于初始先验的访问计数分布；神经网络随后通过监督损失去拟合这一搜索结果。正是因为策略是显式建模的独立实体，这套自我搏弈迭代提升的递归机制才能稳定收敛。
围棋 MCTS 生成的博弈树与井字棋（Tic-Tac-Toe）的穷举树截然不同。井字棋的状态树仅有 $9!$ 种可能，完全可以穷举构建；而围棋 MCTS 的树结构在拓扑上极度稀疏（extremely sparse）。大量劣质落子在被价值网络初筛后将永远不再被访问，搜索算力会高度集中于少数几条高访问计数的黄金路径上。
| 对比维度 | 井字棋（Tic-Tac-Toe）等简单游戏 | 围棋（Go）中的 MCTS 搜索树 |
|:--- |:--- |:--- |
| **树拓扑结构** | 稠密、近乎穷举（$9!$ 量级完整展开） | 极度稀疏（Sparse），深层窄道聚焦 |
| **展开方式** | 静态穷举，遍历所有可能终局 | 动态构建，在先验策略指导下边搜索边建树 |
| **算力分配** | 均匀分配至所有合法动作分支 | 集中于高访问计数、高期望胜率的极少数主干分支 |
| **评估机制** | 穷举到达胜负平终局 | 结合价值网络截断与极少数关键路径推演 |
---
### `58:30` 插播赞助：从办公室被踢掉插头的六台 Dell 到 140 千瓦液冷机柜
在深入探讨自博弈（Self-play）之前，Dwarkesh 穿插分享了他探访量化交易巨头 Jane Street 数据中心的见闻与趣事。如今掌控庞大算力集群的 Jane Street，在 20 年前的起步阶段其实极其原始。其技术团队联席主管 Ron Minsky 向 Dwarkesh 回忆称，公司最早的计算集群被戏称为“蜂巢”（The Hive），其实物仅仅是“在办公桌过道尽头层叠堆放的六台 Dell 主机”。由于那是公司早期的核心交易系统，团队特意将其放在触手可及的办公区内，初衷竟是为了“确保在系统失控时能有人亲手冲过去关掉电源”。
这种草莽时期的部署方式带来了啼笑皆非的险情。Ron 透露，有一次大楼保洁人员在日间办公期间用吸尘器打扫卫生，为了找插座，“在交易进行的正午时分，直接把其中一台核心交易系统的电源线给拔了”。这段插曲让团队痛定思痛，深刻意识到关键系统必须进入专业数据中心。随后 Dwarkesh 在得克萨斯州与 Jane Street 物理工程团队负责人 Dan Pontecorvo 一同参观了现代化的液冷数据中心。如今单台高密度 GB300 计算机柜在峰值状态下的功耗已飙升至 140 千瓦左右；相比于传统风冷机柜通常仅 10 到 40 千瓦的功率水平，高密度计算的基础设施演进展现出了跨越时代的鲜明反差。
- **早年“The Hive”集群**：仅由 6 台 Dell 品牌服务器裸机堆叠于办公室内，以纯手动拔线作为终极风险熔断手段。
- **现代化液冷功率跃升**：传统风冷机柜功耗通常在 10–40 kW 之间，而如今配备先进冷却系统的超算机柜峰值功耗已达约 140 kW。
### `01:00:00` 数据中心插曲与自我对弈强化学习的开端
在深入算法细节前，访谈提及了 Jane Street 数据中心的实地探访细节。现代大型 AI 算力中心普遍采用液冷技术，冷却液由去离子水或蒸馏水与 25% 的丙二醇（Propylene Glycol）混合而成，添加丙二醇的核心目的在于抑制管道内细菌和藻类的滋生。Jane Street 的工程团队甚至掀开防静电地板并拆卸机架，完整展示了后端的冷水机组（chillers）与物理工程构造。
随后讨论正式进入 AlphaGo 如何通过自我对弈（Self-Play）与强化学习（RL）实现自我增强的核心机制。在一场自我对弈中：
1. 棋局到达某个状态 $S$，AI 首先通过策略网络（Policy Network）计算出一个先验动作概率分布 $\pi(a|S)$，这个初始分布通常较为平缓。
2. 随后启动蒙特卡洛树搜索（MCTS），结合价值网络评估与多次模拟展开分支。
3. 搜索完成后，各分支的访问计数分布形成了一个新的、更加确信且陡峭（peaky）的目标分布 $\pi_{\text{MCTS}}(a|S)$。
4. 模型依据该分布采样或选取最大概率动作（argmax）落子，随后清空这棵搜索树，在下一个盘面重新执行上述流程，直至终局决出胜负。
```mermaid
flowchart LR
accTitle: AlphaGo 自我对弈与策略蒸馏流程
accDescr: 展示策略网络生成先验、MCTS 搜索得到更尖锐分布，并最终将搜索结果蒸馏回策略网络的过程
subgraph Step["单步决策与搜索"]
S["当前棋局状态 S"] --> P0["策略网络直觉评估: 扁平先验分布 π(a|S)"]
P0 --> MCTS["MCTS 展开模拟 (PUCT + 价值网络)"]
MCTS --> P1["访问计数分布: 陡峭优化分布 π_MCTS(a|S)"]
P1 --> Act["落子推进至下一状态"]
end
subgraph Distill["经验回传与策略蒸馏"]
Act --> Buffer["存入自对弈轨迹数据集 (无论胜负)"]
Buffer --> Update["策略网络模仿训练: 学习预测 π_MCTS"]
Update -.->|"摊销计算预算 (提升下一轮先验起点)"| P0
end
```
### `01:03:00` 蒸馏 MCTS 搜索结果：摊销测试期算力
AlphaGo 训练的核心巧思在于**将搜索的产物反哺给先验网络**。既然策略网络直觉预测的分布需要经过 1,000 次 MCTS 模拟才能提炼出高质量的落子概率，那么模型可以直接把这 1,000 次模拟得到的最终分布作为监督学习的目标，训练策略网络从一开始就预测出这个更优的结果。
这种做法从根本上改变了测试期扩展（Test-Time Scaling）的起点：
- **直觉起点的跃迁**：未经模拟（0 次模拟）的原始策略网络，在蒸馏了 MCTS 的搜索结果后，其胜率基线直接提升到了原先需要 1,000 次模拟才能达到的水平。
- **算力预算的摊销（Amortization）**：原本必须在推理阶段通过暴力搜索消耗的 1,000 步计算量，被参数化并压缩进了神经网络的前向传播中。在此基础上再进行额外的 1,000 次搜索，模型的整体实力和胜率便会推向新的高度。
- **测试期扩展曲线的单调性**：关于模拟次数与胜率增长是否呈 S 型饱和（Sigmoid），在实践中它对基础策略网络的强弱高度敏感，但整体上严格保持随搜索次数单调递增的趋势。
### `01:05:40` DAgger 算法类比与逐鹿无责的信用分配
这种在整盘对局中模仿搜索策略的做法，与机器人学及模仿学习中的 DAgger（Dataset Aggregation）算法具有深刻的内在一致性。在马尔可夫决策过程（MDP）中，智能体执行了一系列动作轨迹 $(S_0, a_0, S_1, a_1, \dots)$，即使这盘棋最终输掉了，或者动作本身并非最优，MCTS 也能够在每一个具体状态 $S_t$ 上提供一个严格更优的反事实目标动作 $a_t'$。
- **规避长程信用分配（Credit Assignment Problem）**：传统的策略梯度方法面对长轨迹时，难以判断成百上千步中究竟哪一步导致了最终的失败；而 MCTS 相当于在每个节点直接指明了“原本应该怎么下”，因此无论这盘对局最终胜负如何，所有被搜索修正过的样本元组都可以作为有效数据用于重训。
- **类似自动驾驶的干预矫正**：正如自动驾驶汽车在偏离车道边缘时，人类专家或高阶控制器给出的纠偏转向指令一样，即便系统当前处于极差的状态，MCTS 依然能给出将局面拉回正轨的最优修正动作。
### `01:07:35` MCTS 是否必然优于策略网络：理论假设与现实失效边缘
针对“MCTS 是否在数学上必定优于策略网络”的疑问，讨论指出这在工程实践中是一种极为有效的启发式假设，但**并非绝对的数学保证**。当价值网络或搜索环境存在系统性偏差时，MCTS 完全有可能产生劣于策略网络直觉的动作分布：
- **终局评估分布偏移与遗忘**：若自我对弈中大量对局在劣势时直接认输（Resign），而非严格下到最终数子（如 Trump-Taylor 规则判决），回放缓冲区中就会严重缺乏后半盘关键终局及死活评估的样本。价值网络在叶节点的估值崩溃会沿着树结构反向传播，彻底带偏 PUCT 选择公式，导致搜索得出的分布严重失真。
- **搜索模拟方差**：MCTS 只有在模拟次数趋近于无穷大时才具备严格收敛保证。如果模拟步数极少，搜索本身的方差与价值网络的估值误差叠加，反而会把原本合理的直觉带向歧途。
- **AlphaGo Lee 的历史权衡**：这也解释了为何早期的 AlphaGo Lee 在训练算法中坚持保留完整的快速走子至终盘（Playouts to the end），正是为了防止因价值网络在罕见棋局上的盲区而破坏整体策略的稳健迭代。
### `01:10:00` 蒙特卡洛树搜索（MCTS）的冷启动与价值函数的自举
在围棋强化学习中，自对弈（Self-play）有一个常见问题：如果智能体总是提早认输（resign），经验回放缓冲区（replay buffer）就会缺乏终局对局数据。Eric Jang 指出，在工程实践中，可以让大约 10% 的对局禁止认输，强制让两个 AI 互相落子直至真正终局。这样就能采集到人类高手平时根本不会走完的残局末期数据，帮助模型建立对真正胜负状态的认知。
MCTS 能够提供比当前策略更优的动作推荐，其核心前提在于“价值函数是准确的”。在类似 AlphaZero 的从零冷启动（cold-start）情境下，最初几个训练周期的策略网络基本上是盲目的、毫无意义的。训练初期的本质，就是通过完整下完一盘棋来获得最终胜负标签（+1 或 -1），然后让价值函数学习预测：“处于当前局面时，当前行动方获胜的概率是多少？”AlphaZero 的总损失函数（loss）包含两部分：
1. 策略损失：当前策略网络预测的分布与 MCTS 搜索生成的访问分布之间的差距；
2. 价值损失：价值头对胜负的预测与实际对局最终胜负之间的均方误差。
```mermaid
flowchart TD
accTitle: AlphaZero Cold-Start Bootstrapping Flow
accDescr: 价值函数先通过残局或专家数据锚定，再通过MCTS搜索反向传递到中间状态，进而带动策略网络提升
subgraph DataGen ["1. 数据与冷启动"]
A["强制对局到底 (10%不认输) / 人类棋谱"] --> B["获取终局真实胜负 (Terminal Labels)"]
end
subgraph ValueBootstrap ["2. 价值函数自举 (Value Bootstrapping)"]
B --> C["残局价值快速收敛 (低不确定性)"]
C --> D["MCTS 回传价值至中局/开局 (类似 TD 学习)"]
D --> E["价值函数学会中局态势判断"]
end
subgraph PolicyImprove ["3. 策略改进"]
E --> F["MCTS 基于准确价值给出更优动作"]
F --> G["策略网络拟合 MCTS 搜索结果"]
end
```
针对从零训练价值函数的高昂开销，Eric 提出了个人工程实践中的实用经验（非经过同行评议的结论，但极其管用）：
- **优先保证价值函数准确再上搜索**：在糟糕的价值预测上花费大量算力跑 MCTS 是毫无意义的。MCTS 必须有可靠的价值锚定（grounding function），否则整个搜索机制就会分崩离析。
- **利用离线数据集快速热身**：AlphaGo Lee 直接利用人类棋谱进行监督学习，或者使用开源围棋引擎（如 GNU Go、Pachi 等）自我对弈生成的数据。在棋盘布满棋子的残局阶段，搜索树深度极浅，胜负几乎是可判定的（decidable），因此终局价值极易学得非常精准。
- **类似于时序差分学习（TD Learning）的价值反向传递**：开局评估通常显式为 0.5（势均力敌），终局胜负一目了然，而整个价值网络最难学习的其实是扑朔迷离的中局（mid-game）优劣判断。随着对局增多，搜索树会逐步把终局的胜负信号倒推回中间节点。
- **小棋盘向大棋盘迁移**：如果想追求纯粹的白板学习（tabula rasa），可以在 4×4 或 9×9 的小棋盘上让完全随机的智能体下棋。在 9×9 棋盘上玩 50 盘或 50,000 盘，随机走子就会自然产生类似人类下棋的终局形态（就像井字棋一样有限）。结合 KataGo 提出的跨尺寸架构，在小棋盘学到的价值判断特征可以相当平滑地迁移到 19×19 的标准棋盘上。
---
### `01:15:00` 双头网络架构与深度学习对计算复杂度的“摊销”
针对 Dwarkesh 提出的疑问——策略头与价值头共享主干网络（two heads sharing representations）究竟能节省多少算力，Eric 解释道：原版 AlphaGo Lee 论文中策略与价值是由两个独立的深度卷积网络分别计算的，而自 AlphaZero 起，二者被合并为一个共享特征提取骨干的双头网络。虽然量化其确切节省的算力需要严格的消融实验，但直觉上策略与价值在评估局面时共享了大量相同的棋盘特征表征，两者在逻辑上天然需要保持一致性。
随后，Dwarkesh 表达了一种直觉上的反差：当人们了解大语言模型（LLM）的强化学习（如基于规则验证的 RLVR）时，算法极其简单粗暴（仅凭最终答案的对错反馈就能学会编写极其复杂的代码仓库），令人感到震撼；反观 AlphaGo，在深入了解其机制后，人们反而会觉得它在某种程度上“没那么惊艳了”，因为人类工程师显式地设计了树搜索结构，并严格规定了如何随着时间衰减去平衡探索与利用（exploration vs. exploitation），相当于人为注入了极强的归纳偏置（bias）。
Eric 对此明确表示反对。他认为 AlphaGo 代表了一种常人甚至很多从业者至今仍未完全领悟的深刻突破：
- **前向推断对庞大搜索空间的摊销（Amortization）**：一个仅有 10 到数十层的神经网络前向传播，受结构所限，只能进行十几步严格顺序的分布式表征思考（distributed representation reasoning）。然而，就是这区区十几步连续计算，竟然能够高度保真地“摊销”并近似原本在计算复杂度上近乎不可求解的指数级组合搜索树。
- **从 AlphaFold 到 NP 难问题的最坏情况悖论**：同样的突破机制也是 AlphaFold 和 AlphaTensor 的基石。在传统物理仿真中，蛋白质折叠需要模拟极其庞大微观尺度的粒子交互，属于物理上的微观穷举；但中等规模的神经网络前向传递几步就能直接预测出准确结构。这引发了一个深层次的计算理论思考：人类过去往往从“最坏情况复杂度”（worst-case complexity）来定义 NP 难问题，但现实世界或具象博弈中面临的复杂问题往往蕴含极其丰富的物理与几何结构。深度神经网络并不需要给出数学意义上的全局最优解，却能在实践中以惊人的精度解决现实中的最复杂模式。
- **复杂仿真的极限遐想**：如果极小步数的神经网络就能等效替代浩瀚的模拟计算，那么当这种能力被推向极限时，人类想要预测未来天气乃至进行极端复杂维度的系统模拟时，所耗费的计算资源或许远没有传统计算理论预想的那样遥不可及。
### `01:20:00` 混沌系统与宏观结构：为何围棋与天气可预测而哈希函数不行
复杂的模拟计算实际上可以被大量“摊销”（amortize）并压缩进单一神经网络的前向传播（forward pass）中。在 Eric Jang 看来，AlphaGo 是历史上第一篇真正展现出这种深刻能力的论文：它将极其庞大的树搜索模拟压缩进了一个规模相对小得多的网络权重中。
Dwarkesh Patel 对此提出了一个关于计算复杂度和混沌理论（chaos theory）的疑问：天气预报之所以极难，且每往后预测一天所需算力就要暴涨十倍，是因为天气是一个对微小扰动极其敏感的混沌系统，按理说围棋和蛋白质折叠也应具备类似的混沌特性。Eric 以天气和围棋的类比化解了这一悖论：如果在围棋中要求模型预测“在给定当前棋盘状态以及对局双方风格的情况下，未来第 100 步的**精确**棋盘状态是什么”，这确实是一个极度敏感的混沌问题——任何一个落点的改变都会像蝴蝶效应一样彻底掀翻后续整个盘面。然而，人类和价值网络真正关心的并不是那个具体的微观状态，而是一个**宏观量**（macroscopic quantity）——即在所有可能未来的平均期望下，“究竟谁更有可能赢”。
这就像天气预报中我们并不关心 6000 英尺高空某个经纬度上的一阵风速具体是多少，我们关心的是飓风到底会走向何方。Eric 用黑板上的洛伦兹吸引子（Lorenz attractor）打了个生动的比方：虽然从吸引子轨道上的任意一点出发，你根本无法准确预测系统最终会落在哪个具体微观坐标上，但你确切地知道整个系统的宏观几何形状长成这个蝴蝶形态。这种宏观结构是具有高度可预测性的。
相比之下，加密哈希函数（hash function）虽然同样极端依赖初始条件（输入变动一个 bit，输出就完全混乱），但它被设计成彻底抹杀任何宏观结构，因此不存在能预测其大体走势的“价值函数”。两人还探讨了密码学与神经网络的奇妙对偶——神经网络由多层交织的信息变换组成，恰好运行在“混沌边缘”（edge of chaos，Jascha Sohl-Dickstein 等学者曾对此有深入研究）：它既要维持足够的敏感性以整合全局信息，又要在混沌的边缘提炼出稳定的宏观模式。
### `01:24:40` MCTS 的本质优势：每一步棋都有高质量监督目标
随后，讨论切回蒙特卡洛树搜索（MCTS）在强化学习机制上的核心精髓。Eric 特别强调，理解 AlphaGo 的关键在于认清 MCTS **到底不是在做什么**：它绝非粗暴地提高最终获胜对局的动作概率，也绝非简单地把赢棋里的动作全盘加权、把输棋里的动作全盘降权。
```mermaid
flowchart TD
accTitle: MCTS与朴素强化学习的监督信号机制对比
accDescr: 对比朴素RL的整局稀疏胜负奖励与MCTS在每一步提供的密集局部搜索改进目标
subgraph NaiveRL ["朴素策略梯度 (LLM 风格 RL)"]
N_traj["对局轨迹 (300步)"] --> N_eval{"整局结果 (+1 / -1)"}
N_eval --> N_up["所有动作一刀切加权/降权<br/>(梯度方差极高 / 信用分配极难)"]
end
subgraph AlphaGoMCTS ["AlphaGo MCTS 机制"]
M_state["当前单步棋盘状态"] --> M_search["MCTS 局部深度搜索"]
M_search --> M_target["输出更优落子概率分布 π"]
M_target --> M_train["策略网络单步交叉熵监督学习<br/>(步步有目标 / 极低梯度方差)"]
end
```
MCTS 的真正做法是：在智能体走出的**每一个步骤**上，它都通过前向树搜索与模拟推演，寻找一个比当前策略网络输出严格更好的行动分布。接着，它直接让策略网络去拟合（predict）这个由 MCTS 搜索得出的更优概率分布。
- **单步稠密监督**：网络在整场对局中走的每一个单步，都能从 MCTS 获得一个明确的优化目标，而不是等到数百步后游戏结束才拿到一个二元的胜负标量。
- **方差极低**：由于每一步都有高质量的显式目标分布进行交叉熵更新，学习信号的梯度方差相比朴素的端到端强化学习要低出几个数量级。
### `01:26:30` 朴素强化学习的方差困境与 Karpathy 的“吸管效应”
为了反衬 MCTS 的优雅，Eric 让大家设想一种完全抛弃 MCTS、直接采用当前大语言模型（LLM）主流 RL 思路的朴素算法：构建一个由不同历史 checkpoint 组成的联赛（league），让它们互相对弈；对局结束后，挑选获胜者的整条动作轨迹，强行拉高这一整串动作的似然概率，以此重新微调策略网络。
这种做法会遭遇极其严重的梯度方差崩溃。Eric 举了一个具体数字的模型：
- 假设策略 A 与策略 B 实力相当，真实胜率各为 50%。
- 双方对弈 100 局，每局棋平均走 300 步，总计产生多达 30,000 个落子动作。
- 最终比分是微弱的 51 比 49，策略 A 获胜。在这 51 场胜利中，有 50 场两者的下法其实完全平手，仅仅因为偶然的探索噪声或极其微小的随机扰动，策略 A 在其中某**一局**里灵光一闪下出了**一步**脱离原本分布的关键好棋（critical move）。
- 在朴素 RL 看来，它必须把获胜那 51 局里的全部 15,300 个动作统统奖励一遍。然而在这上万个动作中，有 99% 以上的落子策略与原先的旧模型没有任何区别，真正有价值、能够带来超额胜率的监督信号其实**只有那一个孤立的动作**。
Dwarkesh 随即联想到 Andrej Karpathy 在播客中做过的绝妙比喻：当前的 LLM 策略梯度强化学习就像“通过一根极细的吸管吸取监督信号”（"sucking supervision through a straw"）。在围棋里，如果仅凭最终的一胜一负来回传梯度，算法会被海量的方差和无效动作淹没，根本无法逾越特定棋力门槛；而现代大模型因缺乏像围棋那样完美的局部状态转移模型，却不得不默认承受这种极低的信息传输效率。Eric 指出，虽然通过采样数百万个样本、配合优势函数（advantage）和 baseline 消除基线漂移可以勉强把信号捞出来，但这正是为什么朴素策略梯度的方差会如此恶劣。
### `01:30:00` 策略梯度中的方差放大与 LLM 中的单步强化学习
在朴素的强化学习设定下，汇报（Return）往往直接由指示变量（Indicator Variable）决定：胜则为 1，负则为 0。如果对局输了，梯度为零，模型就不在这类样本上更新；如果赢了，模型便尝试去拟合整局的所有走子。然而，这种朴素策略梯度的核心痛点在于**极高的方差（High Variance）**。
当把回报项与各步策略的对数梯度交叉相乘时，梯度的方差大致与轨迹长度 $T$ 呈二次方增长（$T^2$）。回报项像一个强耦合乘子，将轨迹中每一步的动作关联在一起并迅速放大噪声。
这种方差现象直接映射到当今的大语言模型（LLM）RL 训练：
- 为什么当前的 LLM 强化学习普遍采用**单步 RL（One-step RL）**，而不是将每个 token 视为离散步骤的多步 RL？
- Transformer 架构本身基于条件概率连乘，整段文本序列（如生成 `"hello world"`）的对数概率等于各个分词（token）对数概率之和：
$$\log P(\text{"hello world"}) = \log P(\text{"hel"}) + \log P(\text{"lo"}) + \log P(\text{"world"})$$
- 在目前的 LLM RL 设定中，整个生成序列通常被抽象成一个单一动作（$A_T$），时间步长度 $T=1$。如果针对每个 token 分别打分（例如基于过程监督 Process Supervision 给出逐步奖励），这些中间奖励与每个 token 的概率交叉项相乘，会带来巨大的动作间相关性与方差爆炸。
- 此时最关键的挑战在于**信用分配问题（Credit Assignment Problem）**：在动辄成百上千个 token 的输出中，到底哪些具体的 token 真正促成了最终正确的答案，哪些只是无关紧要甚至有副作用的噪音？
```mermaid
flowchart TD
accTitle: 策略梯度在多步与单步场景下的方差对比
accDescr: 对比传统多步RL中每步奖励导致的方差放大与LLM单步RL简化信用分配的结构差异
subgraph MultiStep ["多步 RL 模式 (Multi-Step)"]
direction TB
M1["Token 1: 奖励 r1"] --> M2["Token 2: 奖励 r2"]
M2 --> M3["Token N: 奖励 rN"]
M3 --> MV["交叉相乘产生强耦合<br/>方差随时间步呈二次方增长 (T²)"]
end
subgraph LLMOneStep ["LLM 当前主流实践 (One-Step)"]
direction TB
L1["将整段回答视作单个动作 A"] --> L2["在序列末尾计算标量回报 R"]
L2 --> LV["通过求和降低方差交互<br/>避免为每个中间 token 分配零碎信用"]
end
```
---
### `01:35:00` 优势函数估计与蒙特卡洛树搜索（MCTS）的根本差异
为了抑制朴素 REINFORCE 算法中的巨大方差，强化学习引入了**基线扣除与优势估计（Advantage Estimation）**。在理想情况下，如果一个棋手下了 51 场好棋、49 场臭棋，最优的梯度更新方式绝非粗暴地拿 51 场胜利的所有着法去强化，而是彻底剔除那些平庸或由对手失误带来的“水货步法”，只在真正实现超越基准（Baseline）的那几步关键操作上计算梯度。
在无模型 RL（Model-free RL）中，消除方差的标准手段是引入状态价值或动作价值函数：
- **时序差分学习（TD Learning）**：用来近似状态动作价值 $Q(s, a)$。
- **优势函数（Advantage）**：通过从实际回报中减去该状态下的平均期望回报（Baseline），定义出该动作比平均表现好多少（$A = Q - V$）。高于平均水平的动作被正向激励，低于平均水平的动作被惩罚抑制。
- Eric 强烈推荐了 John Schulman 关于广义优势估计（Generalized Advantage Estimation, GAE）的经典论文，作为理解各种优势估计方差控制方法的必读篇目。
Eric 强调，**无模型强化学习与蒙特卡洛树搜索（MCTS）在哲学层面上存在根本不同**：
- **无模型强化学习**试图在一个极度嘈杂的环境中，反向解开终局胜负对各步历史动作的**信用分配**。
- **MCTS** 则完全不做这种滞后且高方差的信用分配；它直接在每一步利用前向搜索与价值网络评估，现场构造出一个质量严格优于当前策略的“改进标签”。
---
### `01:38:00` 从全信息博弈到复杂博弈：神经虚拟自博弈（NFSP）
MCTS 之所以能在围棋中大放异彩，依赖于一个关键前提：围棋是一个**完全信息博弈（Perfect Information Game）**，拥有严格确定的状态转移，可以极其廉价且深入地在内存中构建显式的博弈树。
然而，在面对如《星际争霸》（StarCraft，AlphaStar）或《Dota 2》（OpenAI Five）这类复杂系统时，MCTS 面临无法逾越的结构障碍：
- 游戏包含战争迷雾（非完全信息）、连续状态空间、实时微操以及非确定性动态，无法通过简单的规则模拟器建立深层前向搜索树。
- 在无法运行显式树搜索的环境中，AI 系统采用了**神经虚拟自博弈（Neural Fictitious Self-Play, NFSP）**等理念，实现了无模型强化学习与自对弈思想的统一。
NFSP 依然继承了“为已有动作寻找更优教师标签”的核心逻辑，但其构造更优行为标签的方法转为了训练**最优响应策略（Best Response Policy）**：
1. **冻结对手**：固定当前强力对手的策略模型（例如固定对手 $\pi_B$ 选择虫族 Zerg 的打法）。
2. **局部无模型优化**：将正在训练的主体 $\pi_A$（例如神族 Protoss）置于一个纯粹的无模型 RL 任务中，仅以击败固定的对手 $\pi_B$ 为唯一目标。
3. **策略迭代求解**：使用标准的 TD 学习或近端策略优化（PPO）等算法拟合局部最优响应，从而在无法使用树搜索的复杂非完全信息博弈中持续推进策略的进化。
```mermaid
flowchart LR
accTitle: 围棋 MCTS 与星际 NFSP 的决策提升机制对比
accDescr: 围棋利用完全信息树搜索提升动作质量，而星际争霸利用固定对手的最优响应无模型RL来迭代策略
subgraph GoAlphaGo ["围棋 / AlphaGo (完全信息)"]
direction TB
G1["完美可观测状态"] --> G2["前向 MCTS 展开博弈树"]
G2 --> G3["搜索访问频次构成更优标签"]
end
subgraph SCAlphaStar ["星际争霸 / AlphaStar (部分观测)"]
direction TB
S1["无法构建精确树搜索"] --> S2["固定对手策略 π_B (如虫族)"]
S2 --> S3["PPO / TD 优化主体 π_A (最优响应)"]
end
```
### `01:40:00` 用无模型强化学习蒸馏“最佳响应”策略
在不采用自我博弈（Self-play）的设定下，可以通过针对特定对手进行强化学习来求解“最佳响应”（Best Response）。以一个固定对手 $\pi_B$ 为例，环境变成了完全固定的单智能体任务：智能体的目标仅仅是最大化对战 $\pi_B$ 的得分，胜则获得奖励 1，负则获得 0。此时可以使用任何喜欢的无模型强化学习（Model-free RL）算法——如 PPO、SAC、VMPO 等进行爬坡训练，从而得到一个专门克制 $\pi_B$ 的优秀策略。
```mermaid
flowchart TD
accTitle: Best Response Distillation
accDescr: 使用无模型RL针对对手池分别训练最佳响应，并蒸馏为混合策略
subgraph League["固定对手池 (League)"]
OPP_B["对手 π_B"]
OPP_C["对手 π_C"]
OPP_D["对手 π_D"]
end
subgraph Solvers["无模型 RL 求解最佳响应"]
BR_B["最佳响应策略 BR_B (PPO / SAC)"]
BR_C["最佳响应策略 BR_C (PPO / SAC)"]
BR_D["最佳响应策略 BR_D (PPO / SAC)"]
end
subgraph Distill["策略蒸馏"]
PI["最终混合策略 π"]
end
OPP_B --> BR_B
OPP_C --> BR_C
OPP_D --> BR_D
BR_B -->|"监督标签 / 动作重标记"| PI
BR_C -->|"监督标签 / 动作重标记"| PI
BR_D -->|"监督标签 / 动作重标记"| PI
```
当面对由多个固定对手（如 $\pi_B, \pi_C, \pi_D$）组成的对手联盟（League）时，可以分别针对每个对手训练出对应的最佳响应策略，然后提取这些策略生成的行为标签，用于监督训练一个主策略。这一过程本质上是用 Model-free RL 代替了蒙特卡洛树搜索（MCTS）充当“教师”的角色，最终将各种应对方案蒸馏为一种“混合策略”（Mixed Strategy），使其面对联盟中随机抽取的任意对手时，表现都不会劣于平均水平。
- 该方案绕开了依赖 MCTS 产生指导信号的需求，但在本质上仍然是通过更优动作对状态进行“重新标记”（Relabeling），以此推动策略提升。
- 如果在对局中取胜，所采用的无模型算法（如 PPO、Q-learning 或 VMPO）会对整条轨迹上的动作进行强化。
---
### `01:42:30` MCTS 与 Q-Learning：前向搜索与逆向备份的数学对称
Eric 进一步指出了 MCTS 与 Q-learning 之间的深层结构联系。在 MCTS 中，算法在构建的搜索树叶节点处通过价值网络估值或模拟得出结果，再沿着多条路径向上反向回溯（Backup），计算出均值估计——这里的 $Q$ 值本质上源自大量前向模拟的统计平均。而在无模型算法中，学习 $Q$ 值通常依赖时序差分学习（TD learning）与贝尔曼方程备份：
$$Q(s, a) = r + \gamma \max_{a'} Q(s', a')$$
这一贝尔曼备份公式体现了马尔可夫决策过程（MDP）的动态规划与递归属性：当前状态动作的价值等于即时奖励加上后续状态的最大价值。神经网络被用来强制满足这种跨步间的一致性。当没有环境模拟器无法进行前向搜索时，Q-learning 允许智能体先采集长为 $n$ 步的真实轨迹，再利用“逆向备份”反推中间步骤的状态价值。
```mermaid
flowchart LR
accTitle: Comparison of MCTS and Q-learning
accDescr: 对比MCTS的前向模拟与Q-learning的离线逆向备份
subgraph MCTS_Flow["MCTS（前向搜索）"]
direction TB
S0["当前状态 s"] --> SIM["前向分支模拟 (未探索轨迹)"]
SIM --> LEAF["叶节点价值估值"]
LEAF -->|"向上聚合均值"| Q_MCTS["更新动作价值 Q(s,a)"]
end
subgraph Q_Flow["Q-Learning（逆向时序差分备份）"]
direction TB
TRAJ["收集真实交互轨迹 (已访问轨迹)"] --> TERM["到达后续状态 / 终局"]
TERM -->|"TD 逆向备份 r + γ max Q"| Q_TD["更新当前 Q(s,a)"]
end
```
二者的异同与历史成因表现为：
- **前向规划 vs. 逆向传播**：MCTS 是在智能体尚未实际走入的虚拟前向轨迹树上进行推演；而 Q-learning 是在智能体已经实际访问过的真实轨迹上，将未来获得的信息逆向回传。
- **策略的恢复**：获得准确的 $Q$ 值后，不需要显式对策略概率分布建模，只需通过 $\operatorname{argmax}_{a} Q(s, a)$ 就能直接恢复出策略。
- **历史瓶颈与无模型范式**：过去在机器人学等高维复杂问题中，既没有精准的世界模型，也无法进行低成本前向搜索，因此业界长久以来被迫假设“只能收集真实轨迹，并在唯一关键的标量信号——奖励（Reward）的驱动下做规划”。
---
### `01:45:47` 为什么 MCTS 难以直接套用在 LLM 上？
将上述逻辑延伸至大语言模型（LLM）的强化学习时，二者的学习范式存在巨大的样本效率差距。当前的 LLM RL 通常是在代码生成等环境中，筛选出通过单元测试的完整轨迹，并像“吸管式监督”一样粗暴地强化该轨迹内的所有 Token（如 Andrej Karpathy 所比喻的 "learning through a straw"）——算法并不知道数千个 Token 中到底哪几句逻辑才是破局的关键，存在严重的信用分配（Credit Assignment）难题。相反，AlphaGo 的 MCTS 可以在单步（Per-move）层面展开局部搜索，并在遇到难以展开的深层分支时直接用价值网络截断（Truncate），从而以极高的样本效率产出比原策略更优秀的单步目标分布。
针对为什么大语言模型无法简单复用类似 AlphaGo 的 PUCT 搜索树，Eric 与 Dwarkesh 剖析了其根本瓶颈：
| 维度 | 围棋（AlphaGo） | 大语言模型（LLM 推理） |
|:--- |:--- |:--- |
| **动作空间（Breadth）** | 离散且有限（最多 361 处落子点，随着棋局收紧） | 词表庞大且语义开放，单步离散 Token 组合无限 |
| **深度与价值函数（Depth）** | 胜负规则绝对闭环，价值评估确定且可用截断 | 语义推理缺乏精确的中间过程评估网络，真假难辨 |
| **启发式搜索规则（PUCT）** | 探索项 $\frac{\sqrt{N}}{1 + N_a}$ 鼓励在小动作集中重访与对比 | 几乎永远不会对同一子节点采样两次，$N_a$ 失去统计意义 |
- **PUCT 探索项失效**：围棋树搜索高度依赖访问频次对置信区间的修正，而在 LLM 中，语言的宽泛性和开放性意味着同一个思考步骤很少会被一模一样地多次采样，PUCT 的局部贪婪特性容易导致模型陷入显而易见但并不能真正解决难题的“平庸局部解”。
- **隐式搜索 vs. 显式树结构**：Dwarkesh 质疑 LLM 是否已经在自回归生成中隐式地学会了 MCTS（例如自我发现思路不对后主动“等等，让我重试另一种解法”）。Eric 肯定了这一点，认为 LLM 确实学到了一种接近人类思维的反思行为而无需外挂显式树结构；但他强调，以某种形式引入前向模拟与推演（Forward Search and Simulation）来重新获得强大的局部评估信号，未来极有可能卷土重来。
- **围棋 MCTS 的核心特权**：围棋最关键的优势在于，即便前向搜索没有展开到终局，只要利用价值函数进行局部 MCTS 扩展，得出的动作分布**几乎绝对比当前基础策略更强**；而在 LLM 和通用机器人领域，既缺少这样可靠的中间截断价值，也缺乏确定的动作树修剪机制。
### `01:50:00` MCTS、连续控制与棋盘游戏的 Scaling Law
对于是否能像围棋那样“局部评估并改进下一步、而无须从全局直接解题”的疑问，Eric Jang 指出虽然在类似语言的超高维动作空间或连续控制中直接应用 MCTS 极为困难，但这并不意味着未来沿多条并行路径进行前瞻搜索（lookahead search）的思想毫无用处。例如在数学领域，思考过程具备明确的逻辑推导和回溯（backtracking）结构，能够清楚判断某条路径的优劣，因而天然契合树搜索；而在商业谈判这类非结构化场景中，思维模式则完全不同。
随后，话题转向 Andy Jones 在 2021 年发表的经典论文《Scaling Laws for Board Games》（棋盘游戏的缩放定律）。该研究早在推理 Scaling（Inference Scaling）概念大火之前，就前瞻性地证明了推理计算量（Test-time Compute）与训练计算量（Training Compute）之间的等价替换关系——通过在 MCTS 阶段分配更多搜索算力，可以达到与大幅增加模型训练量完全相同的棋力表现。这一规律不仅预示了今日大语言模型推理缩放的演进路径，还揭示了前向推理（forward pass）本身如何通过深层网络隐式地内化原本需要显式递归搜索的能力。
```mermaid
flowchart LR
accTitle: Compute Tradeoff in Game Search
accDescr: 展示测试时推理算力与训练算力之间的替换关系，以及棋盘尺寸与所需算力的缩放规律。
subgraph ComputeTradeoff["Andy Jones 缩放定律：算力替换"]
A["增加训练算力<br/>(更久训练 / 更大参数)"] <-->|"效果等价替换"| B["增加推理算力<br/>(更多 MCTS 搜索步数)"]
end
subgraph BoardScaling["棋盘规模扩展"]
C["棋盘尺寸放大<br/>(3x3 扩展至 NxN)"] --> D["精确预测所需总算力"]
end
```
- **棋盘规模预测**：Andy Jones 论文的另一核心成果在于，通过缩放定律可以精确预测把棋盘尺寸从 $3 \times 3$ 扩展至更大尺寸（甚至无限大棋盘）时所需的算力规模。
- **“苦涩教训”的实操误区**：Eric Jang 最初启动项目时，试图验证仅依靠算力缩放和“苦涩教训”（The Bitter Lesson）能否省去 KataGo 中大量精巧的特化技巧、打造纯算力驱动的围棋 AI。但在实践中，他因 MCTS 标注代码存在 Bug，导致系统在错误数据和失效策略上拟合出虚假的 Scaling 曲线。
- **研究顺序法则**：研究人工产物的科学规律（如 Scaling Law）前，必须先有一个架构正确、数据扎实且无 Bug 的基准系统。不要在系统自身尚未调通时就急于去分析其缩放行为。
---
### `01:55:00` AlphaZero 的算力异常点与现代复现的工程红利
回顾近十年全球顶尖 AI 模型训练算力的增长图表，所有模型在对数坐标下均呈现出一条平滑平稳的指数增长曲线，唯一的极端离群点就是 2017 年的 AlphaGo Zero / AlphaZero。它消耗了高达约 $3 \times 10^{23}$ FLOPs 的算力，在当时堪称天文数字，甚至在量级上能与早期的前沿 LLM 相比拟。然而在当下，Eric Jang 仅凭 Prime Intellect 赞助的约 1 万美元云算力额度（其中 4000 美元用于前期探索，约 3000 美元用于最终训练跑通，剩余用于模型在线部署服务），便成功训练出具备极高竞争力的围棋模型。
这种巨大的算力反差揭示了技术突破与工程追随之间的本质规律：开辟新领域的先驱者（如 DeepMind 的 AlphaZero 团队）必须在没有现成对手、缺乏先验知识的前提下实现白板起步（*Tabula Rasa*，从零自博弈），他们的首要目标是“验证能力是否存在”，为了在有限时间内拿到成果，根本无暇优化算力利用效率；而后续复现者则可以借助蒸馏、高质量先验、以及对抗现有强模型（如对战 KataGo）的“最优响应”（Best Response）训练来走捷径，从而以极低成本追平表现。这正如当前的机器人学习前沿（Frontier Robotics）一样，各团队的算力散点图极其杂乱，远未达到大语言模型那样对每个 FLOP 锱铢必较的严密帕累托前沿（Pareto frontier）。
```mermaid
flowchart TD
accTitle: Pioneer vs Follower Compute Dynamics
accDescr: 对比先驱团队的高算力白板探索与后续复现者的低成本工程优化路径。
subgraph Pioneer["先锋探索：AlphaGo Zero (2017)"]
P1["首要目标：验证可行性"] --> P2["白板训练 (Tabula Rasa)<br/>无基准策略可供借鉴"]
P2 --> P3["巨额算力投入 (~3e23 FLOPs)<br/>非算力最优解"]
end
subgraph Follower["现代复现：Eric Jang 方案 (2026)"]
F1["目标：极简架构与低成本落地"] --> F2["对抗 KataGo 做 Best Response 训练<br/>硬件迭代 + 训练链路大幅简化"]
F2 --> F3["低预算搞定 (~1 万美元内)<br/>去除分布式异步队列与重放缓冲区"]
end
```
- **架构选型不再敏感**：在当今强大的 GPU 算力下，围棋网络的模型架构（无论采用 Transformer 还是传统 ResNet）差异已被抹平，模型规模尚不足以触发两者的性能质变。
- **训练架构大幅精简**：无需搭建过去那种依赖复杂数据推送器（pushers）、收集器（collectors）和经验回放缓冲区（replay buffers）的分布式异步强化学习系统，采用“纯同步生成数据 $\rightarrow$ 监督训练模型 $\rightarrow$ 再生成数据”的朴素循环即可稳定收敛。
- **硬件演进对工程的释放**：当年 KataGo 需要大量 NVIDIA V100 GPU 协同训练，如今仅需数张消费级/桌面级 Blackwell GPU，算力就足以轻松支撑完整迭代并砍掉大量繁复的辅助监督损失项。
### `02:00:00` KataGo 技巧的必要性与小棋盘冷启动加速
在从零复现围棋 AI 的过程中，Eric Jang 发现 KataGo 所开发的许多复杂技巧，在拥有强大初始化的前提下其实并非绝对必需。如果直接针对 KataGo 本身进行最佳响应训练（Best Response Training）来完成初始化，模型其实无需依赖 KataGo 所采用的那些网络架构创新。训练中最核心的关键，在于如何以最快速度匹配到高水平的对手，这远比具体的模型架构设计重要得多。
不过，计算效率的乘数效应依然存在。Eric 发现先在 9×9 的小棋盘上进行训练，对于网络解析终局价值函数（Endgame Value Function）极为有效。如果设计一种能够在 9×9 与 19×19 棋盘之间自适应迁移的通用架构并进行联合训练，就能大幅缩短从零学习的预热时间（Warm Start）。AlphaGo Zero 论文中的训练曲线显示，最初大约需要 30 个小时仅仅用来赶上监督学习基线，而通过小棋盘预训练再迁移到 19×19 棋盘，可以大幅削减这段追赶时间。此外，关于每局对弈之间是否调整 MCTS 搜索模拟次数（Simulations），实验表明系统对此并不敏感，无论是保持固定还是逐步增加，效果差异都不大。
- **快速面对强手**：击败强大对手的驱动力远胜于细枝末节的架构微调；利用预训练强大的现成模型（如 KataGo）提供最佳响应信号，能省去大量繁复技巧。
- **棋盘尺寸迁移**：9×9 棋盘对局步数短、终局结算清晰，是训练价值函数判断胜负的绝佳温床，能显著加速 19×19 完整棋盘的学习。
- **复现科学价值**：重访经典论文并剥离非本质复杂度，能够看清哪些设计真正带来收益，哪些只是计算开销的附属品。
---
### `02:01:09` 为什么 AlphaGo 允许 Off-Policy：DAgger 视角与防漂移“管道”
Dwarkesh Patel 提出了一个核心疑问：强化学习研究人员总是强调 Off-Policy（离线策略）训练存在诸多隐患与不稳定性，然而 AlphaGo Zero 的朴素实现高度依赖经验回放缓冲区（Replay Buffer），其中一个反向传播 Batch 中的绝大多数走子，都是由较早版本、而非最新权重生成的走法，为什么 AlphaGo 却没有崩溃？
Eric 解释道，这触及了强化学习中 Off-Policy 与 On-Policy 的本质区别。在 AlphaGo 中，MCTS 会对既有状态重新搜索并赋予新的动作目标。Off-Policy 的潜在危害在于，若你不断重新标注新策略永远不可能访问的状态，就是在白白浪费模型容量；在极端情况下，如果训练缓冲区全是当前策略遇不到的局面，模型就会在无用状态上强行拟合，导致策略急剧恶化。但从模仿学习算法 DAgger（Dataset Aggregation）的视角来看，理想的数据分布应当以最优轨迹为主干，并在其周围构建一个包含轻微偏离状态的“高维管道”（High-dimensional Tube）。一旦系统偏离最优路径，这些状态上的目标标签能够充当“漏斗”，把策略重新纠偏引导回最优轨迹。
这正是很多现实场景中需要 Off-Policy 数据的核心理由：为了避免复利误差（Compounding Error）。纯粹的最优控制往往默认智能体一旦学会策略就绝不出错，但在机器人领域，突如其来的一阵风可能会吹偏机械臂，或者车轮摩擦力不均会导致车辆打滑漂移，算法必须拥有从偏离状态中恢复的数据。Eric 引用了一句关于棋类游戏的精辟比喻：“围棋和国际象棋的麻烦之处在于，对手总想搞点事情（the other player is always trying to do some shit）。”局面随时会脱离预想，回放缓冲区中保留那些可能漂移到的边缘状态及其纠偏走法，能够赋予模型极强的鲁棒性。
```mermaid
flowchart LR
accTitle: DAgger-style Trajectory Tube
accDescr: 理想训练分布包含最优轨迹以及环绕在外的偏离状态纠偏漏斗
subgraph Tube["高维状态管道（DAgger 纠偏机制）"]
direction TB
Drift["偏离/受扰状态<br/>(对手意外走子 / 物理扰动)"]
Label["MCTS / 专家修正标签<br/>(提供纠偏动力)"]
Opt["最优轨迹状态<br/>(通向胜利的黄金路径)"]
Drift -->|"重新引导"| Label
Label -->|"漏斗收敛"| Opt
end
```
---
### `02:05:00` 将机器人强化学习搬到围棋：经验回放、白日梦与 MCTS 离线重新标注
如果走向另一个极端——缓冲区全是远离最优行为的垃圾状态，模型就会被无意义的目标所拖垮。为了最大化打满 GPU 算力，Eric 曾在围棋项目中尝试过一种实验：直接打乱棋局的因果时序，从历史轨迹中随机抽取棋盘状态，完全跳过从头到尾下完整盘棋的过程，仅用当前最新神经网络对这些离散局面重新运行 MCTS 打标，甚至反复重抽旧状态赋予最新标签。这一方法在实践中确实奏效，且其底层逻辑与机器人领域的离线强化学习高度相通。
在机器人控制系统（例如早期 Google 的 QT-Opt）中，标准的离线框架由三部分构成：
1. **数据收集与推送**：将历史对弈或操作记录的转移元组（Transition Tuples）不断压入回放缓冲区。
2. **贝尔曼更新器（Bellman Updater / Planner）**：从缓冲区中提取状态转移，持续重规划如果当初采取不同动作是否能获得更高价值。它通过最小化时序差分误差（TD Error），根据 $S'$ 计算 $\max_{A'} Q(S', A')$，叠加奖励 $R$ 构造出更优的 $Q_{\text{target}}$。
3. **训练器（Trainer）**：以简单的监督学习方式，最小化当前网络的预测值 $Q(S, A)$ 与 $Q_{\text{target}}$ 的差距。
```mermaid
flowchart TD
accTitle: Off-policy RL and MCTS Relabeling Architecture
accDescr: 展示离线强化学习系统与围棋 MCTS 重新标注系统的架构映射
subgraph Storage["数据存储"]
Trans["离线转移数据<br/>(S, A, R, S')"]
Buf[("经验回放缓冲区<br/>Replay Buffer")]
Trans -->|"持续写入"| Buf
end
subgraph Relabeler["后台重规划（白日梦 / Daydreaming）"]
Bellman["Bellman 更新器 / MCTS 重标注器<br/>(探索：当时若采取其他走法会如何？)"]
Target["生成高质量优化目标<br/>(Q-target 或 MCTS 动作分布)"]
Buf --> Bellman
Bellman --> Target
end
subgraph Learning["参数拟合"]
Trainer["训练器 (Trainer)<br/>(监督学习拟合 Target)"]
Target --> Trainer
Trainer -.->|"更新网络权重"| Bellman
end
```
Patel 将这种机制形象地比作智能体的“白日梦”（Daydreaming）——在后台结合当下认知，复盘历史长河中的状态，审视自己过去是否有更好的走法。Eric 指出，在围棋中完全可以用 MCTS 重新标注器来替代贝尔曼更新器：从历史中抽取局面状态 $S$，扔掉当初走出的动作与胜负标签，直接让最新的网络在 $S$ 上展开 MCTS 树搜索，生成更强的策略分布作为监督目标。虽然这种机制因工程复杂度过高而未包含在最终开源代码中，但它证明了搜索增强与机器人离线强化学习在数学本质上的深度同构。
### `02:10:00` 离策略训练与机器人的收敛实践
在围棋中，利用当前最优策略网络 $\pi$ 在历史状态转移上离线重新运行搜索（MCTS），可以起到极佳的稳定训练效果。这种做法不仅规避了在线对弈时等待环境推进棋盘状态的阻塞开销，还能通过并行化搜索不同深度的棋盘状态，将 GPU 算力完全打满。训练目标也变得简单直接：让策略网络尽可能拟合 MCTS 搜索给出的标签。Eric Jang 指出，这种离策略（Off-policy）模式在机器人学（Robotics）中十分常见——现实世界中往往积攒了大量离线离线数据集，且无法像围棋一样凭空进行无限次环境模拟。
然而，离策略训练存在明显的隐患：如果当前模型被强行拿去学习那些“以其现有能力永远无法到达的状态”，就会严重浪费模型容量。正因如此，当今强化学习（RL）领域大部分工作逐渐收敛到了更为稳妥的同策略（On-policy）设定上，不再直接用离策略数据强行拟合目标策略。即便引入离策略数据，也往往只是将其作为降低方差的辅助手段，例如通过离策略方式估算动作价值 $Q$，以此计算优势函数（Advantage）：
$$\text{Advantage} = Q(s, a) - V(s)$$
利用这种基准面（Baseline）减法，即使离线动力学与当前分布存在偏差，也不会导致整体损失函数剧烈发散。在机器人领域，大家普遍达成的共识是：离策略数据最好只用来对奖励进行塑形（Reward Shaping）或辅助方差缩减，而非直接驱动策略优化。
```mermaid
flowchart LR
accTitle: Off-policy Data Utilization in RL
accDescr: Diagram contrasting naive off-policy direct training with stable advantage baseline computation
subgraph Direct["直接离策略拟合 (风险高)"]
D1["离线历史状态"] --> D2["直接监督/策略优化"]
D2 --> D3["容量浪费与分布漂移"]
end
subgraph Advantage["稳健方差缩减 (主流做法)"]
A1["离策略数据"] --> A2["离线估算 Q 价值"]
A2 --> A3["计算优势 Advantage = Q - V"]
A3 --> A4["稳定同策略更新"]
end
```
---
### `02:12:02` 为什么朴素策略梯度在信息效率上极其低下
Dwarkesh Patel 回溯了此前两人的讨论，指出与 MCTS 步步提供高质量指导相比，以 REINFORCE 为代表的策略梯度（Policy Gradient）强化学习在信息利用效率上远比人们想象的更低。他分享了自己此前撰写的博文观点：人们直觉上认为 RL 低效是因为需要“跑完整个轨迹（Trajectory）才能获得单一的奖惩信号”——随着任务从简单的下一个 Token 预测变成智能体需要自主摸索两天写完一个完整项目，每单位算力能采样的样本量（Samples per FLOP）呈断崖式下跌。
智能体学习的核心本质是最大化“每单位算力获取的信息量（Bits per FLOP）”，其分解公式为：
$$\frac{\text{Bits}}{\text{FLOP}} = \frac{\text{Samples}}{\text{FLOP}} \times \frac{\text{Bits}}{\text{Sample}}$$
长时程任务使得 $\frac{\text{Samples}}{\text{FLOP}}$ 大幅下降，但更严重的是，朴素 RL 在“每样本获取的信息量（Bits per Sample）”上也极其贫乏。以大语言模型在初始化阶段的学习为例：
- **监督学习（Supervised Learning）**：假设词表大小为 100K，输入提示为 `"The sky is"`。未经训练的模型会给所有 Token 分配一个概率分布。目标真实标签是 `"blue"`，模型直接通过交叉熵损失（Cross-entropy Loss）计算当前分布与标准答案之间的差距，单次样本学到的信息量正比于 $-\log(P)$。初始命中概率越低（Pass rate 越小），模型单次校正得到的梯度更新和信息量反而越巨大。
- **强化学习（RL）模式**：若完全依赖纯 RL，模型只能随机乱猜：`"The sky is halicon"`——错误；`"The sky is told"`——错误。对于一个 100K 词表的未初始化模型，它平均要盲目抽样 100,000 次，才可能撞大运碰上一次 `"blue"` 并获得正向反馈。
在 99.99% 的情况下，模型得到的只是“猜错”的微弱惩罚，它从中学到的信息量几乎可以忽略不计。
---
### `02:15:00` 通关率对数坐标下的绝望曲线与 AlphaGo 的软标签精髓
通过数学对比可以看清这一信息量鸿沟：
- 监督学习在单样本中获取的有效信息量遵循负对数形式 $-\log(P)$，通关率（Pass rate, $P$）越低，单次监督样本带来的信息量反而越高。
- RL 乱猜给出的结果本质上是一个二元伯努利随机变量，其信息增益遵循二元熵函数。只有在胜率接近 50%（类似于掷硬币）时信息增益达到峰值（1 bit）；当胜率趋于 0 时，采样获得的信息量无限趋向于 0。
| 学习范式 | 通关率极低区间 ($P \to 0$) | 适中区间 ($P \approx 50\%$) | 拟合后期 ($P \to 1$) |
|:--- |:--- |:--- |:--- |
| **监督学习 (SFT)** | $-\log(P)$ 极大（梯度信号极强，直接指出靶向方向） | 适中信息增益 | 渐趋饱和 ($-\log P \to 0$) |
| **朴素强化学习 (RL)** | 二元熵极小（接近 0，绝大多数失败只带来微弱噪声） | 达到峰值（约 1 bit） | 逐渐收敛清零 |
当在对数坐标轴（横轴从 $1/100,000$、$1/10,000$ 逐步过渡到 $1$）上绘制信息量曲线时，RL 呈现出一条令人绝望的轨迹：在训练初期，模型几乎完全困在超低通过率区间，单样本信息量极其稀薄，甚至“可能永远采不到一次成功的正向样本”。这解释了为什么所有复杂的 RL 系统都极度依赖预训练或专家数据热启动：必须先将初始通过率从绝对的零提升到某个阈值之上，RL 的引擎才能真正转动。
```mermaid
flowchart TD
accTitle: Bits per Sample Comparison
accDescr: Information bits extracted per sample between supervised learning and reinforcement learning across pass rates
subgraph SFT["监督学习 (Cross-Entropy)"]
S1["标签已知: 'blue'"] --> S2["信息量 = -log(P)"]
S2 --> S3["P 极低时单样本信息量极大"]
end
subgraph RL["朴素强化学习 (Binary RL)"]
R1["策略空间盲猜: 'halicon', 'told'..."] --> R2["信息量 = 伯努利二元熵"]
R2 --> R3["P 极低时信息量 ≈ 0 (极度欠采样)"]
end
```
Eric Jang 随后指出，从“每样本比特数”这一视角出发，还可以深刻理解知识蒸馏（Distillation）与软标签（Soft Targets）的巨大威力：
- 传统的独热标签（One-hot Label）信息熵非常低。
- 包含了各类别相对概率的 Logits 概率分布，其分布熵（Entropy）要大得多，蕴含了丰富的类别间关联结构（即暗知识 Dark Knowledge），这使得单次样本蒸馏所能传递的信息量呈几何级数增长。
这也是 AlphaGo 架构中极具美感之处：**在训练策略网络时，并不是让网络生硬地模仿 MCTS 搜索最终选出的单一动作，而是让它去拟合 MCTS 搜索展开后所形成的完整概率分布**。这种软标签蒸馏机制赋予了模型远远超出普通离散 RL 的极高样本信息效率。
### `02:20:00` AlphaGo 为何是优雅的强化学习：逃离零信号困境
Dwarkesh 与 Eric 探讨了 AlphaGo 与当前大语言模型（LLM）强化学习的核心差异。在现有的 LLM 强化学习中，策略梯度算法往往面临漫长轨迹中的信用分配难题（即在 10 万个 token 的轨迹中找出到底哪一步带来了正确结果），且初始策略在面对复杂难题时的通过率往往是 0%，模型很难直接得到有效信号。而 AlphaGo 的蒙特卡洛树搜索（MCTS）呈现出一种优雅的机制：无论模型初始水平如何，你永远不需要在“0% 成功率”的冷启动悬崖上死磕探索问题。
在 AlphaGo 的实际工程实现中，每一步甚至完全没有显式的 TD 误差学习（TD-error learning）或动态规划，它本质上是一个被转化为监督学习的流程：基于改进后的搜索结果进行价值分类训练，并通过最小化策略分布与 MCTS 访问计数之间的 KL 散度来更新策略。这种纯粹的监督学习范式带来了极强的训练稳定性，基础设施架构简单易维护，既不需要维护复杂的分布式系统来保证严格的 on-policy 状态，也不存在梯度消失或信号完全平坦的局部极小值。只要画出曲线，MCTS 搜索策略的胜率永远严格高于原始神经网络，每一步都提供干净、明确的提升标签，驱动策略持续爬坡。
```mermaid
flowchart TD
accTitle: AlphaGo Supervised Hill-Climbing Loop
accDescr: AlphaGo 的迭代提升闭环：原始策略驱动 MCTS，MCTS 输出更高质量的胜率与策略标签，再通过监督学习稳定更新网络。
subgraph Search["MCTS 搜索阶段"]
PolicyNet["原始策略网络 Policy / Value Net"] -->|"先验指导扩展"| MCTS["蒙特卡洛树搜索 (MCTS)"]
MCTS -->|"探索与回溯评估"| ImprovedDist["改进后的策略分布与胜率估计"]
end
subgraph Supervised["监督学习更新阶段"]
ImprovedDist -->|"生成训练目标 (更优软标签)"| Targets["KL 散度最小化 / 价值分类"]
Targets -->|"稳定监督更新 (无需复杂分布式 on-policy)"| PolicyNet
end
```
---
### `02:22:16` 自动化科研循环（Autoresearch）：LLM 擅长与不擅长的事
随后两人移步坐下，讨论将 LLM 编码智能体接入自动化科研（Autoresearch）的实际体验。随着前沿实验室致力于训练能够完全自主搞科研的 AI，许多人开始讨论这是否会触发“智能爆炸”甚至奇点。Eric Jang 分享了他在复现 AlphaGo 项目中大量使用 Claude Opus 4.6 与 4.7 搭建自动化科研循环的真实观察。
目前 AI 最擅长的领域在于开放性超参数优化（Hyperparameter Optimization）与实验执行。过去的超参搜索大多依赖预设好网格的贝叶斯优化（Bayesian optimization），只能调整学习率、权重衰减或网络层数；现在的 AI 智能体则像一个任劳任怨、不知疲倦的研究生：它能自主检查中间激活，发现“某一层的梯度太小了”，然后直接修改代码架构；它能自己编写数据加载器的数据增强（data augmentation），在固定数据集和计算时间预算下，极致压榨分类或困惑度（perplexity）指标。Eric 专门为 Claude 编写了一个名为 `experiment` 的自定义 Skill：只需向它描述想要绘制的 X 轴和 Y 轴并提出研究假设，Claude 就会自动编写代码、分发运行实验、绘制图表、生成报告并归纳推导可能的原因。
- **超参数开放式调优**：不仅限于数值网格搜索，能自主重写代码、重构数据加载逻辑与网络层。
- **自动化实验调度（`experiment` 技能）**：从提示词构思实验、跑批处理脚本，到制图与撰写实验报告一气呵成。
- **死磕特定指标**：表现出极强的“刷分”（metric grinding）能力，能大幅降低困惑度。
---
### `02:25:00` 科学探索中的水平思考、Bug 识别与围棋基准
然而，现阶段的大模型在科学探索的宏观决策上仍存在显著短板。Eric 将自己博客中的研发历程可视化为一棵实验树：每个节点代表一次成功、失败或模糊的实验结果，并由此分支出后续实验。在实际探索中，人类研究员经常会陷入死胡同（例如研究 off-policy MCTS 重新打标），尝试几轮发现性价比不高后，便会果断横向跳跃到全新的路线上。但当前的公开闭源模型缺乏这种“退后一步、基于第一性原理思考根本瓶颈何在”的横向思维（lateral thinking）；它们往往无法自主决定特定技术路线中的下一个最佳实验，对于代码中的分布式底层基础设施 Bug，也往往需要人类提出极其精准的提问，模型才能排查出根因。
两人进一步讨论了围棋（Go）作为自动化 AI 研究（AI researching AI）基准环境的独特价值。围棋具备极高的外环验证确定性（胜负规则不可篡改，可用开源的 KataGo 直接评测胜率），而其内环则充斥着与 LLM 和机器人学完全相通的复杂系统工程（预测某一改动对 Scaling Law 算力乘数的影响、分布式通信与收敛性评估）。围棋的高防作弊与快速验证属性，恰好能防止模型发生奖励黑客攻击（reward hacking）。
针对这一议题，Dwarkesh 引用了 Ilya Sutskever 的观点：顶级研究者的核心特质在于对“正确想法”持有近乎信仰般的坚定直觉，因此在遭遇失败时，他们能敏锐分辨这究竟是底层想法本质有误，还是仅仅因为代码里藏了 Bug，进而有毅力穿越重重 Bug 坚持到底。Eric 赞同道，深度学习本身就是一个历经数十年冷板凳、顶住无数学术委员会唱衰才最终爆发的超长视野（long-horizon）难题；如何设计既具有强确定性外环验证信号、又能尽早提供中间反馈的 RL 研发环境，正是自动化科学研究（Automated Science）中最关键且最考验“研究品味”（research taste）的开放课题。
### `02:30:00` 通用语法、苦涩教训与算法 Trick 的“不可叠加性”
大语言模型（LLM）因具备类似“通用语法”（universal grammar）的特质，能够在技术栈的不同层级自如穿梭：既能极为细致地专注于局部代码和细节，又能随时抽身退步，以高屋建瓴的宏观视角审视全局。人类科研中至关重要的“横向思维”（lateral thinking）能力——判断当前追求的目标是否已误入歧途、是否应该换一个根本问题重新发问——往往就源于这种在微观与宏观之间的自如切换。然而，在推动外层闭环（outer loop）迭代时，一个极具争议的核心问题在于：各种局部的微小改进到底能否线性叠加？在诸多顶尖 AI 实验室中，流传着大量令人扼腕的失败案例：研究人员各自独立探索并验证了一批看似绝妙的好点子，但在合并之后，训练任务却因不同改动之间怪异的负向相互作用而彻底崩溃。这表明，一个统一且由上至下的系统性愿景在系统工程中至关重要。
对于“AI 创新在多大程度上可以被并行化”这一问题，在各大前沿实验室均有深厚经历的 Eric Jang 指出，要优雅地践行“苦涩的教训”（The Bitter Lesson），最关键的“研究品味”在于清晰地判断苦涩教训在当下到底能为你买单多少，以及何时不应对它过度索求。从终极的时间维度来看，算力无疑是决定一切的最关键要素，只要不断扩展能量、算力与模型参数规模，智能就近乎必然会自然涌现——这既优美又深刻，在此背景下没有任何具体的算法细节具有永恒的重要性。但在现实中，我们受限于有限的算力、参数和初始权重，不得不设计各种启发式策略（heuristics）来进行加速。
这些启发式 trick 之间往往存在大量功能冗余，导致所谓的“算力倍增器”（compute multipliers）通常无法良好叠加。许多提升收敛速度的算法收益在时间尺度上是极其短暂的。以围棋领域的 KataGo 论文为例，该研究当年针对 NVIDIA V100 级 GPU 堆叠了大量极其精巧的算法技巧来加快收敛；然而仅仅几年之后，随着 NVIDIA Blackwell 和 Ada 架构等现代 GPU 算力的大幅飞跃，很多曾经关键的算法 trick 相比单纯的硬件碾压与规模扩展，其重要性瞬间微不足道。在特定时间节点辨析哪些 trick 具备长期价值、哪些只是昙花一现的妥协，本身就是研究品味的试金石。
```mermaid
flowchart TD
accTitle: Compute Multipliers vs Scaling
accDescr: 展现短期算法技巧在算力换代后的收益衰减与冗余关系
subgraph Heuristics["短期微观启发式优化 (如 KataGo 技巧)"]
H1["Trick A (加速收敛)"]
H2["Trick B (探索奖励)"]
H3["Trick C (结构先验)"]
end
subgraph Limitations["边际效应递减与冲突"]
R["功能冗余 / 相互干扰<br/>(收益无法线性叠加)"]
end
subgraph Hardware["苦涩教训 (硬件与规模碾压)"]
HW["V100 时代的算力瓶颈"] --> ModernHW["Blackwell / Ada 架构算力飞跃"]
ModernHW --> Obsolete["原有算法 Trick 相对收益锐减"]
end
H1 --> R
H2 --> R
H3 --> R
R -. 试图替代.-> HW
```
---
### `02:32:30` 外层验证闭环与跨任务的正向能力迁移
构建自动化 AI 研究员（Automated AI Researcher）的关键难点在于“外层验证闭环”（outer verification loop）的有效性。在围棋任务中，外层闭环相对清晰——即对抗开源基准的最优胜率；但即便是围棋研究，要判断系统是否发现了一种全新的前沿规律依然极度困难。例如在 Kaplan 或 Chinchilla 提出缩放定律（Scaling Laws）之前的 2015 年，任何自动化算法程序都极难辨别哪张图表孕育着深刻的缩放法则，哪张图表只是又一个毫无价值的随机测试曲线。围棋尚且如此，面向通用人工智能（AGI）的外层闭环则更为严峻：尽管业界充斥着五花八门的评测基准（benchmarks），但系统往往只能针对可测量的指标进行过拟合优化，而无法有效度量其在真实物理世界中从事“具有经济实用价值的工作”（economically useful work）的核心能力。
针对自动化研究员能否胜任这种复杂跃迁，Eric 提出了一个基于历史直觉的论证：以 DeepMind 为例，该实验室最初专注于在 Atari、Go（围棋）和 StarCraft（星际争霸）等游戏环境中寻求闭环反馈。随着研究人员在攻克游戏的过程中积累了深厚的工程落地、代码编写和项目管理经验，当转向大型语言模型研发时，这些历史积累展现出了极为显著的正向能力迁移（positive transfer）。同样的逻辑理应适用于自动化 AI 研究员：它们完全可以先在验证周期极短、迭代迅速的沙盒环境（如强化学习游戏、代码验证基准）中完成自我历练，随后将积累的科研能力泛化迁移至自动化药物研发、新材料设计等更为宏大且高经济价值的领域。
---
### `02:35:10` 研究路径的迷雾与将“思考”作为计算基语的未来
对于 DeepMind 早期专注游戏积累经验是否绝对有利，Dwarkesh 提出了犀利的质疑：在 Gemini 3 等前沿模型发布之前的很长一段时间里，科技界曾普遍批评 Google 在 LLM 浪潮中反应迟缓、起跑落后，正是因为过去被固有的游戏与强化学习旧技术路径所束缚。针对这一观点，Eric 认为历史的定论尚且为时过早：究竟是早期的游戏研究拖累了其通用智能的节奏，还是世人眼中的“起步滞后”实际上是在底层技术树上进行长跑预训练——比如耗费数年重注 TPU 基础设施的算力集群构建——以至于在如今的长线竞争中展现出持久后劲？这表明，即便站在当今上帝视角审视人类顶尖科研机构的发展史，要推断出一条绝对最优的科研战略路线都是极端困难的。
随着对 AlphaGo 与现代 LLM 机制探讨的收尾，Eric Jang 介绍了其开源实验项目及后续学习路径：
- **个人主页与项目交互指南**：可通过 [evjang.com](https://evjang.com/) 获取结合现代大模型技术的复现教程交互版本。
- **开源代码仓库**：在其个人 GitHub（用户名 `Eric Jang`）下开源了名为 `auto-go` 的完整仓库，供开发者和研究者自由 Fork、运行自我博弈并验证全套训练循环。
- **代表性博文推荐**：特别推荐研读其代表作《像岩石一样思考》（*As Rocks May Think*），该文从更宏大的计算机哲学视角探讨了当“思考”（thinking）本身成为计算机科学的一项基础原语（primitive）时将带来怎样的范式变革。
Eric 强调，尽管业界不一定需要在 LLM 内部生硬地套用显式树结构，但蒙特卡洛树搜索（MCTS）所代表的展开推理与当前 LLM 的思维链（Chain of Thought）之间存在着深刻的“对偶性”（duality）。在当今 LLM 算力军备竞赛的高昂壁垒下，基于极小算力预算在围棋和 MCTS 架构上探索智能思考的本质，依然是一片充满巨大潜力的未被充分开垦的学术沃土。
## Notable quotes
> `00:00` 令人着迷的是，一个仅有区区 10 层的神经网络，竟然能够摊销（amortize）如此深度的博弈树模拟过程。 — Eric Jang
> `02:15` 在围棋中，你完全可以输掉局部的战役，但赢下整场战争（lose the battle but win the war）。 — Eric Jang
> `30:00` 你不会把所有东西完全丢掉，我们会保留一小样东西留作后续使用。 — Eric Jang
> `37:30` 就像在深度学习中初始化决定一切一样，你总是希望让你的科研项目初始化在尽可能接近成功的位置。 — Eric Jang
> `40:00` 永远从行得通的东西开始，让它变得更好；而不是从完全不灵的东西起步，再试图把它修好。 — Eric Jang
> `43:08` 老实说，这有点不公平，李世石可没有用 $10^{22}$ FLOPs 的算力来下一记棋。 — Dwarkesh Patel
> `52:30` 你不能先建树再搜索，你必须在搜索的过程中动态建树。 — Eric Jang
> `58:30` 在交易进行的正午时分，直接把其中一台核心交易系统的电源线给拔了 — Dwarkesh Patel
> `01:30:00` sucking supervision through a straw — Dwarkesh Patel
> `02:01:09` the other player is always trying to do some shit — Eric Jang
> `02:35:10` As Rocks May Think — Eric Jang
## Mentioned
Eric Jang, Dwarkesh Patel, 1X Technologies, Google DeepMind, AlphaGo, AlphaGo Lee, AlphaGo Zero, AlphaZero, AlphaFold, AlphaTensor, KataGo, David Wu, Lee Sedol, Jane Street, Ron Minsky, Dan Pontecorvo, Andrej Karpathy, Jascha Sohl-Dickstein, John Schulman, Andy Jones, Ilya Sutskever, OpenAI Five, Prime Intellect, Claude, Claude Opus 4.6, Claude Opus 4.7, Cursor, Trump-Taylor Rules, ResNet, Transformer, DAgger, QT-Opt, NVIDIA V100, NVIDIA Blackwell, Scaling Laws for Board Games, auto-go, GNU Go, Pachi, KGS
## Video description
<p><a target="_blank" href="https://evjang.com/">Eric Jang</a> walks through how to build AlphaGo from scratch, but with modern AI tools.</p><p>Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn.</p><p>Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second.</p><p>Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside.</p><p>Watch on <a target="_blank" href="https://youtu.be/X_ZVSPcZhtw">YouTube</a>. Read the <a target="_blank" href="https://www.dwarkesh.com/p/eric-jang">transcript</a>.</p><p>And check out the <a target="_blank" href="https://flashcards.dwarkesh.com/eric-jang/">flashcards</a> I wrote to retain the insights.</p><p>Sponsors</p><p>* <a target="_blank" href="https://cursor.com/dwarkesh">Cursor</a>‘s agent SDK let me build a pipeline to generate flashcards for this episode. For each card, I had an agent read the transcript, ingest blackboard screenshots, generate an SVG visual, and run everything through a critic. A durable agent is much better at this kind of work than a chain of LLM calls, and Cursor’s SDK made it easy. Check out the cards at <a target="_blank" href="https://flashcards.dwarkesh.com">flashcards.dwarkesh.com</a> and get started with the SDK at <a target="_blank" href="https://cursor.com/dwarkesh">cursor.com/dwarkesh</a></p><p>* <a target="_blank" href="https://janestreet.com/dwarkesh">Jane Street</a> gave me a real deep-dive tour of one of their datacenters. I got to ask a bunch of questions to Ron Minsky, who co-leads Jane Street’s tech group, and Dan Pontecorvo, who runs Jane Street’s physical engineering team. They were willing to literally pull up the floorboards and take out racks to explain how everything works. Check out the full tour at <a target="_blank" href="https://janestreet.com/dwarkesh">janestreet.com/dwarkesh</a></p><p>Timestamps</p><p>(00:00:00) – Basics of Go</p><p>(00:08:17) – Monte Carlo Tree Search</p><p>(00:32:04) – What the neural network does</p><p>(01:00:33) – Self-play</p><p>(01:25:38) – Alternative RL approaches</p><p>(01:45:47) – Why doesn't MCTS work for LLMs</p><p>(02:01:09) – Off-policy training</p><p>(02:12:02) – RL is even more information inefficient than you thought</p><p>(02:22:16) – Automated AI researchers</p> <br/><br/>Get full access to Dwarkesh Podcast at <a href="https://www.dwarkesh.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4">www.dwarkesh.com/subscribe</a>
[Source](https://api.substack.com/feed/podcast/197852876/5642f97b323e24b00efd0d89ace15d58.mp3)
Source quotes
[0:00] Eric Jang: 令人着迷的是，一个仅有区区 10 层的神经网络，竟然能够摊销（amortize）如此深度的博弈树模拟过程。
[2:15] Eric Jang: 在围棋中，你完全可以输掉局部的战役，但赢下整场战争（lose the battle but win the war）。
[30:00] Eric Jang: 你不会把所有东西完全丢掉，我们会保留一小样东西留作后续使用。
[37:30] Eric Jang: 就像在深度学习中初始化决定一切一样，你总是希望让你的科研项目初始化在尽可能接近成功的位置。
[40:00] Eric Jang: 永远从行得通的东西开始，让它变得更好；而不是从完全不灵的东西起步，再试图把它修好。
[43:08] Dwarkesh Patel: 老实说，这有点不公平，李世石可没有用 $10^{22}$ FLOPs 的算力来下一记棋。
[52:30] Eric Jang: 你不能先建树再搜索，你必须在搜索的过程中动态建树。
[58:30] Dwarkesh Patel: 在交易进行的正午时分，直接把其中一台核心交易系统的电源线给拔了
[90:00] Dwarkesh Patel: sucking supervision through a straw
[121:09] Eric Jang: the other player is always trying to do some shit
[155:10] Eric Jang: As Rocks May Think
Source quotes
[8:17] Eric Jang: AlphaGo's core conceptual breakthrough was using neural nets to make this search problem tractable.
[20:00] Eric Jang: If you had a very powerful computer, there are no probabilities. You can just compute the true average of what the mean action value is... The interpretation of $Q$ is what is the expected action value under the random distribution induced by some random search process.
[30:00] Eric Jang: In deep learning, initialization is everything. You always want to initialize your research project to something as close to success as possible.
[47:11] Eric Jang: Because Go is such a combinatorially complex game, you cannot afford to build the tree in advance and then search it. You must search while building the tree.
[75:00] Eric Jang: 10 steps of neural network vectorized distributed representation thinking is able to amortize and approximate to a very, very high fidelity a nearly intractable search problem.
[135:00] Eric Jang: The fact that LLMs are universal grammar means that they can kind of move at almost any level of the stack... They can think very locally, as well as step back and think, like, in very broad steps. And I think that's where a lot of the lateral thinking ability of humans kind of come from.
[135:00] Eric Jang: It's not to say that I think we should have trees in our LLMs, but there is some very interesting duality between them
Original transcript
[0:00] Today I'm here with Eric Zhang, who was most recently vice president of AI at 1x Technologies, before that senior research scientist at what is now Google DeepMind Robotics. And you've been on sabbatical for the last few months. One of the things you've been doing is rebuilding and improving and hacking on AlphaGo. And so today what we're going to do is you're going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development. But before we get to that, why is AlphaGo interesting? Why is this Why is this the project you decided to do on sabbatical rather than just hanging out at the beach? Sure, yeah. I like making things, and AlphaGo and GoAI is one of those things that really got me into the field. When I saw the early breakthroughs on AlphaGo in 2014, 2015, 2016, and so forth, it was just profound to see how smart AI systems could become and the computational complexity class that they could tackle with deep learning. This is a problem that has long been understood to be intractable for a search, and yet it was solved through deep learning. And so that was quite mysterious to me and I always wanted to understand that phenomenon a little bit better My training is often in deep neural nets for robotics where the decisions made by the neural networks are a bit more intuitive but AlphaGo is a sort of problem where the decisions are actually the result of a very very deep search And it's always been very mysterious to me how a 10-layer network can sort of amortize the simulation of something so deep in the game tree. Yeah, interesting. So if you plot out how much compute it took to build various iterations of strong GoBots over the years, you can see that in 2020, there was a open source project called KataGo by David Wu from Jane Street, who basically achieved a 40x reduction in compute needed to train a really strong GoBot tabular Rasa. I'm not certain if it's stronger than AlphaGo Zero or AlphaZero or MuZero, but it's very, very strong. And this is what most Go practitioners today train against when they're playing an AI. And thanks to LLM coding, what took a whole team of research scientists at DeepMind and, you know, millions of dollars of research to compute can now be done for, you know, a few thousand dollars of rented compute. By the way, if you're listening to this on an audio platform, this is a Blackboard lecture. So I highly recommend switching over to a video platform like YouTube if you can to look at the math and the graphs and the Go board OK I guess we should first discuss how Go works Great So yeah how does the game work So the game of Go is a very simple one that can be implemented quickly and easily in a computer The objective of the game is basically to put down black and white stones and try to occupy as much territory in the game as possible. So I might start by putting down a black stone. Black always goes first. So go ahead. And so the way you capture an opponent's stones is that for every intersection, if you can surround all four of its neighbors, with your stones, then this one is sort of cut off from oxygen, if you will, and then it is a dead stone. So then now I control these four stones as well as this empty intersection here. So there's like slight variations between Chinese, Japanese, and what is called Trump-Taylor rules. Trump-Taylor rules are designed to be completely unambiguous for Go. So this is what all Go AIs train against and resolve against. So in typical Go, like where the humans play, you're actually not allowed to put this white stone down here. It would be instant suicide. In Trump-Taylor, it's actually fine. You put it down, and then it immediately resolves to death. So the outcome is sort of the same. Let's go ahead and start over and play a few stones, and then I'll explain some more. So I'll just start there. I basically playing randomly here but I trying to get around your stones and see if I can close by them Yeah So this move basically exposes one empty neighbor for your white stone, and it's very akin to a check in chess, where if you don't respond immediately by putting one here, then I can immediately capture this. I see, okay. Because it is sort of the diagonals that determine whether you're grounded in it. The cross-section, not the diagonals. So this one is surrounded on three sides. Yeah. And so you're at threat of losing that stone if you don't play one immediately there. Now you can see that I'm starting to pressure you because by putting a stone here, now you are forced to put one here. Otherwise you would have this two block to yourself. And then if you think through what happens if you were to respond here, you can probably search into the future and deduce what I'll do in response once you do that. You have a lot of confidence in my abilities, but I'm guessing you'd put the black here. That's right, and then I would capture all three of these stones. So I should just assume that this is gone, this little block is gone. Yes. So in Go, it's actually okay to let an opponent capture some stones. If, for example, it allows you to position
[5:00] to capture more stones in somewhere else on the board. And this is what makes Go a very beautiful game, is that you can kind of lose the battle but win the war. And as the board size increases, the complexity of these kind of micro versus macro dynamics gets more interesting. Presumably you'd put one here. And so now I would capture this entire group, and this would be mine. Okay, there's one more case that I want to demonstrate, which actually I had a bug in my code recently, which is the following situation. So let's consider a formation like this. And then we have other pieces on the board in play or whatever. And so let's talk a little bit about how the game ends. In this territory, who controls these areas? Is it white or is it black? White. It's actually black because I have actually surrounded this whole area. And assuming I have other black stones here, it's actually very hard for you to break this out of the control of this stuff so when the final score is tallied would would these ones also count as being in um yeah great question so so um this is where different rule sets have different ways of scoring and so we should talk a little bit about how like uh you resolve humans and how you resolve scores between computer code Because there actually some ambiguity in how humans evaluate this So most humans would look at this board configuration and conclude that black has kind of totally surrounded white. And so white has no chance of life. We could play out more here, but then at the end, I would capture everything. However, if you have a way of breaking this formation and connecting white to something outside of it, then it can flip. And so this is where it's a little bit hard for a computer to decide these kind of things. So how do humans do it? It's worth thinking a little bit about how humans resolve this because this will actually map later to how we think about the deep neural network. Humans basically say, I think the game is done. And then you have to also say, I think the game is done. And then we'll say, I think these are my stones. And then you have to agree. If you don't agree, then we keep playing. So essentially, once two humans, their so-called value function, agree on a consensus, then the Chinese rules result in that. Yeah, interesting. So in Trump-Taylor scoring, it's perfectly unambiguous, so it can be decided, you know, algorithmically by a computer. So if, let's say, you have this at the endgame, the way you score this is that you first count how many stones you control and that unambiguous then you count how many empty intersections that are not touched by your opponent stones so these intersections would not count for either player because both of all of these intersections are connected to both white stones and black stones right if um this were like this then white would get three points now this is a little odd because a human would know that white is actually losing these points. But Trump-Taylor scoring would consider White to have all of these points as well as these points. Got it, okay. So that is a very big difference in how computer Go scores things and how humans score things. How does the game end? The game ends when either a player chooses to resign or both players pass consecutively. Cool. Yep, so that's the rules. Nice. All right, now help me correct this with AI. Great, okay. let's understand how AlphaGo actually works and how somebody in the audience might be able to implement it. Great, yeah. Let's start with kind of an intuition about the underlying search process used to make moves. And we'll layer on ideas from deep learning to make it much more efficient and attractable. So Go is a game where there just two players We going to draw a person here and we going to draw an AI here And let say this person is playing black so they go first So we're going to draw... They go here. And then now the AI is going to make a move based on what it sees here. So there's a question of how you encode these inputs into the AI. Maybe you could use ones and zeros, but you want to represent black, white, and empty. So you would need at least three different values here. So maybe you could use zero, ones, and twos or something. So the AI might see something like zero, zero, zero, zero, one. Great. So this is the input to the AI on its turn. Yeah. So the AI can choose, let's just pick three possible random moves that can go, and I just drew these at random. And so which move is best here? Well, we don't know until the game ends. Go does not have any kind of local reward of which move here is good. And this is what makes Go a very difficult game, is that you don't actually know who won until you really get to the end of the game. So how deep is this tree? Well, in a 1936 game,
[10:00] 19 go board, there are, you know, roughly to the order of 361 moves on any given move. And of course, as it fills up, you have less moves. And the number of steps in the game can be somewhere from 250 to 300 moves. And maybe experts might decide to end the game well before that. But, you know, under Trump Taylor's scoring, you actually have to play things all the way to the So this could be like 300 moves or something. Right. So like 300 depth of the tree. So if you keep on expanding possible moves here, so in this move, the AI is going, and then here the human would go, and then there's some... and so forth. You can find that essentially what you end up with is an enormous explosion in the possible game outcomes originating from just this one state. So this is something to the order of like 361 to 300, power of 300, which is far more than the number of atoms in the universe. Like it just and of course actually there are redundancies and symmetries so it not actually 300 but that sort of the If you were to do a naive tree where there were no merging of children then actually you end up with a tree about this big What do you mean by merging of children? Right. Let me use this board here. So if we start here, and then you play here, and then I play here, and then you play here, that is equivalent to I start here, you play here, I play here, and then you play here, right? So both of them arrived at the same spot, but through different paths. So this child node can be thought about as a shared ancestor. And I guess it starts at 361, but it decreases by one each time. And the branching factor decreases by one each time. But in any case, this is a very, very, very large tree. And this is also why computer scientists for many years thought that Go was not a tractable problem this century. Because the amount of compute you would need to exhaustively search every possible possibility is just too large. If you could, Go is actually a deterministic game. So on any given state, you can actually compute what the best possible strategy you can make is in order to win the game. You can search all the possible futures where you win and then just make sure you always stay in that set of futures So AlphaGo kind of core conceptual breakthrough was using neural nets to make this search problem tractable So before we get into you know how neural networks are involved let's talk a little bit about how we can, you know, assuming we had a powerful enough computer, search this tree to find the best move, right? So in the beginning, you're not going to build out the whole tree, because storing that tree would be very expensive. Instead, you might do something like interactively figure out which leaves of this tree are worthy of exploring and expanding into the future to see, you know, what else is there. So there are some early algorithms in bandit literature like, you know, UCB1, which is not exactly appropriate for a, you know, sequential game like Go, but very much inspired the action selection algorithm used in AlphaGo. So UCB1 looks like on every move, we're gonna take the best action, or the argmax over A, that maximizes the Q of A, and I'll explain what Q of A is in a moment, plus some sort of exploration bonus So on every node we going to track a few quantities So let's consider each of these a node. This is the root node, where you're making decisions from. And these are the children of the root node. And we're going to say each node is basically a data structure that stores a visit count. of this node, this child node. Is how often the parent visited this node. Yes, and we'll call this an action. So one thing that is easy to trip on is if you come from robotics or other kinds of reinforcement learning is like, where are the actions, right? I'm only talking about nodes. Nodes here represent states, and because this is a perfectly deterministic game with no randomness, you can actually just infer the action based on the child. So if I go here, that implies an action. And this is the state that we resolve it. So the LLMs, if you ask to, you know, vibe code a MCTS implementation, it'll most likely design the right data structure here. But, you know, it's sort of a chef's choice. You can actually rewrite the tree structure however you like. This was what Claude 4.6 wrote for me when I asked it.
[15:00] And it was a very reasonable choice. So then Q represents the mean action value of this action. And I'll use a subscript a to denote that this kind of corresponds to taking a specific action to get here, right, from the root node. So, like, if we have root, basically taking a gets us to this node here. Yeah. And then we're going to also store the probability of taking this action. Again, from the parent. From the parent, yes. Like, what are the odds that we sample this one? Yeah. And this will become relevant later. You know, like, we've talked about a deterministic tree for now, so I'll bring probabilities into this later. And then finally, we have a sort of dictionary of children, which is just like, you know, more of these notes, you know, in a sort of classic linked list style reference tree. So this is the basic data structure to implement a tree. And in AlphaGo, they use a slightly different action selection criteria called PUCKT, and it short for Predicted Upper Confidence with Trees And this is basically when you select which child to take you do argmax A of Q of S A plus constant So the equation and forms are actually pretty similar. These are both scoring criteria, right? Like you want to argmax this quantity, and you want to argmax this quantity to determine which action to take. So let's break down the intuition of how you select actions here. This is the mean action value, so how good is a given child on average. And if you actually knew the whole tree, then this is all you need to select the best action. You don't really need to do more than that. But if you're interactively building this tree as you're figuring out what the Q values should be, then what you have to do is occasionally try some other actions, you know, as a sort of explore versus exploit trade-off. So in both UCB and Pucked, there is this term here that basically rewards taking actions that you haven taken before So as we mentioned before each node stores the visit count of taking that specific action right So everything is initialized to zero And so for a given action, let's just say like call it action A, initially it's zero. And so as N is increasing, if let's say we've already made 10 action selections from that root node, but we haven't picked A yet, then this term actually starts to become quite large for A. And conversely, if we have chosen A 10 times out of 10, then now this term is quite small. It diminishes very quickly. And the same thing is actually true here. Just to make sure I'm understanding it. Maybe I can put it in my own words. Let's just focus on UCB. What we're saying here, you can think of it conceptually as two different things, the Q and then this exploration term. Let's just be clear about what Q is. Q is basically saying, hey, once we do these rollouts, so you're actually running all these simulations, you go down the tree, and then you figure out, okay, if I end up at the terminal value of this tree, do I win this game or not? And then you do this, you average whether I win this game or not across all the you know the leafs of this tree starting from this node that average you put in Q Correct And so you saying the Q is basically representing will I win this game or not What is the probability that I win this game starting at this node? That's your sort of, that is your sort of exploit. That is like saying, I've run these simulations, I think this is a good move or not. And then this other term is saying, have I explored this branch enough yet relative to the other actions I could be exploring? Or I have already explored. if I haven't explored this branch yet maybe I think it has a low score but I just haven't explored that many leaves down this down this node in this tree so I should maybe try this even though the Q, this sort of exploit is telling me that this is not that valuable because ln of n grows slower than n basically as over time you will move from the argmax being dominated by this exploration term which is the second term here to the argmax being dominated by the Q-term, which is like, okay, I've done enough simulations, I'm quite confident that this is the branch to go down. Yes, that's right. So the motivation for UCB was to come up with an algorithm where if you don't know the payoff of the arms, the different actions you can select to begin with, this strategy basically with given some
[20:00] exploration term here bounds your regret in terms of how wrong you can possibly be. I don't know the proof. I don't also know if this one is proved to have a logarithmically or like, you know, square root bounded regret or anything. But I think the algorithm was just derived to look something like this. And you can tell that these terms, they grow a little bit differently. And this is actually just to account for the fact that Go has many more actions in every given move compared to your standard banded problem. So one small clarification to make is that you talked a little about simulations on and probabilities and forth. We should remember that Go fundamentally is a deterministic game. So where does the notion of probability come from here, right? If you had a very powerful computer, there is no probabilities. You can just compute the true average of what the mean action value is. So where does the probability come in? Well, it turns out that as in, you know, computer Go before AlphaGo, we've always done some sort of Monte Carlo method where we have some, we take the expected Q value averaged over a randomly selected tree. And that randomly selected tree is where probabilities come in. So the interpretation of Q is what is the expected action value under the random distribution induced by some random search process Makes sense. And so where does the random search process come in? That's where P of action comes in. So if we assume a very naive algorithm where you have a uniform probability of taking any valid action, then this would just be one over the number of valid moves in this setup. and you would be kind of taking this average over this very diffuse tree, right? And this is a valid integral you can take, but it's very slow because you're going to consider a lot of trees that have very low value. And it's essentially almost like an importance sampling problem where you want to, there's only a few actions and sort of paths that can contribute high value, and almost everything else is low value. So that's a sort of a tricky problem here. Okay, so this is the action selection criteria for how you decide which moves to move down. Now, as you move down in tree search, you will eventually run into a node where it's quite clear you've won or lost, right? At the very, very end of the game, when there are no valid moves to play left under Trump scoring you can decide whether you won or lost right So you either win or you lost And so this is basically the final return of the whole game And so the question here is we can assign a value, u, to a terminal leaf node of the tree, but how do we assign the values for nodes prior to that, the parents? And it turns out, you know, what you simply do is you just take the, your mean action value is essentially your average. So let's suppose these were leaf nodes. Sorry, these were all leaf nodes. The mean action value of this node, you know, this action here, is just the average of whether you won or lost at the leaf nodes. And correspondingly, you can kind of walk up the chain and say, like, well, the mean action value of this node, let's call this like QB, and this is action B, is just the average of, a weighted average of these ones here. And the weighted average is, it could be dependent on if you have a different sampling distribution or not, but the basic intuition is that you want to resolve the game where you have a deterministic win or lose and then you can kind of go backwards This is called the backup step and assign values to these nodes or actions corresponding to the averaged over the final terminal OK, so if you were to do this without neural networks, it would still be intractable. You would have trouble finding, you know, which actions to sample. a lot of the actions would contribute very low value, especially if you're trying to fight your way out of a losing position, and only a few actions give you high value. So the search in practice is still very, very expensive. But the idea is that if you can, because Go follows a tree structure, you can actually inform a very good estimate of the value of this node based on the values of downstream, assuming they're all correct, and assuming you've searched deep enough. Your explanation earlier about the sorts of states where it's obvious to a human who's going to win, but it's not obvious to, or like you deterministically still had to play it out, actually drove from the intuition of why the value function both is trainable and to why it's necessary in order to actually be able to learn this game effectively. I mean, it's worth defining value in the first place. Sounds good.
[25:00] We talked about this U value being your final resolution of whether you won or lost. And this is the terminal leaf node condition. Now, humans don't play all the way to the sort of edges of the tree, the leaves of the tree. They kind of stop some dozens of moves before, maybe even 100 moves before in sort of high-level play. So how do they know? You can think about humans as implicitly having a neural network called a value function that basically takes in a board state and then it kind of evaluates T win. And so the human glances at the board and they know, like, I'm probably going to lose. And they're essentially running a neural network that looks at a board, and implicitly they are amortizing a huge number of possible game playouts and taking that average and then deciding whether the board is winnable or not, and then whether they should concede or keep playing or not. And this is remarkable. If you think about the beauty of something like this, it's like a neural network in a human can somehow do all of this simulation at a glance and then just know within a few seconds without actually playing every single game logically based on just crystallized knowledge and experience that they can do this And so this gives us a hint that in games like Go there are ways to basically radically speed up the search process. And this is one of the fundamental intuitions behind why AlphaGo works, is that you can train a value function to look at a board and quickly resolve the game without playing out all of these trees into a very deep search depth. Yep, makes sense. I will say for the audience, I sort of found for previous episodes when I was prepping, and it seemed somewhat relevant to understand how AlphaGo works, I would find it very, very confusing. But it's the kind of thing where once you understand the problem in this way and then you'll build the next few pieces, it is actually much more understandable and it will make a lot of sense. And it's okay to be confused right now, but it's probably simpler to understand by the end of this lecture than you anticipate. So I'll just make that note for the audience. Yeah, the important intuition at a high level, just to step back about where we're going with all this, is that classically, for games like Go, you could build a tree, but we don have computers powerful enough for that And estimating the value of every action that you could possibly take is also hard because you don know until the end of the game You could take averages by playing them to the end but that also hard because you don know which actions to take to sample these averages So conceptually, there's kind of two problems. There's the breadth of the tree, and then there's the depth of the tree. And AlphaGo gives us a way to basically shrink both of those to be very attractive. that's essentially the kind of core idea behind it okay so we take this idea that humans can glance at a board and instantly predict whether we win and maybe that gives us the opportunity to really truncate how deep we search and then you know we also know that humans can look at a board and decide you know what boards you know like intuitively at a glance what moves might be good on a Go board. So these are kind of two things that we can use deep neural networks for to accelerate this search process. Let's go back, before we talked about neural nets, let's just go back to how this play out works. So we've only talked about making one move. So the AI looks at this encoded Go board It has a tree It searches for you know deeply into the tree to find out which of its actions might be the best And then it takes that action And then now you know it goes back to the human So maybe now the human sees a Go board that looks like you know like this And then they make their move. So maybe they put their stone here. And then now we go back to the AI, which now looks at a new encoded board. So I've used two to denote the AI's playing as white, and one to denote the human playing as black, and zero as empty. And then now on the AI's turn, it does the MCTS tree search all over again from scratch. So it throws away this old tree that it searched last round, and now there's a new root node, and it begins to search anew. And then so on and so forth. So MCTS is basically, you can think about it like a search algorithm that is deciding what moves to play best, aided by neural networks. And it's done on every move. Okay, great. So let's talk about the neural network part of this. And while you're racing, another sort
[30:00] thing that was important for me to understand was this mcts data structure with nodes and children's of nodes and whatever um this is done per move and reinstantiated once a move is made so a human makes a move then the ai looks at this and is trying to basically run a bunch of simulations uh to figure out okay what should move should i make next and those simulations just A simulation is basically like exploring one more node in this MCTS tree. And at the end, once all this, you know, you run a thousand simulations, that informs then this, I guess as you will explain, this probability of what move to make next. That's what you store. You sort of choose the best move given those probabilities. You discard all of that. Then the next player makes a move. And you restart this process at the beginning of every move. Correct. One small addendum, you don't discard all of that, you keep one thing behind that we'll use later. Just like I did for Reiner, I wanted to make flashcards for this episode so that people could retain these concepts. And ideally, an LLM could generate some candidates for me to then refine. But to actually get high suggestions I needed to design a whole pipeline where the AI could take and ingest screenshots of the blackboard at the right timestamps and then make SVG diagrams in case visuals were helpful and then run their writing and drawing through a critic and then revise the card in response to this feedback. It's very hard to accomplish this just by stacking LLM calls. This sort of step-by-step recipe works much better if you have a durable agent that's been engaging with the task across all the previous stages. So I used the cursor SDK to spin up an agent for each card. The cursor hardness saved me a bunch of work in designing some custom context scaffold or figuring out how to design tool calls for taking screenshots or making animations. These agents all run in the cloud, so I don't have to worry about leaving my laptop open. I just get an email when I have candidates to review. You can check out my cards at flashcards.thvorkesh.com. You can start building with the agent's SDK at cursor.com.thvorkesh. Okay, so now we have a basic intuition of how moves are made with search. We're going to talk about how neural networks can speed this up by providing an analog to the human intuition. So there's two networks. There is the value network which takes in a state and it predicts you know am I going to win or lose It a binary classification problem Then we going to have a policy network which induces a distribution over good actions to take So I'm going to draw a one-dimensional flattened move distribution, but this is really like a square kind of grid, right? So maybe it thinks actions are like... These are the kind of probability distribution over good actions. And both of these are categorical classification problems, right? So you can train this like any classifier with deep learning, cross-entropy, loss, that kind of stuff. So the specific architecture does not actually matter too much. I tried a few different architectures, transformers work, ResNet's work. For small data regimes, my experience is that ResNet still kind of outperformed transformers and kind of give you more bang for the buck at lower budgets, but this may not be true. Wait, why is that? They provide the inductive bias of local convolutions. And generally, transformers start to outperform residual convolutional networks when you want more global context. I see. So one interesting finding from the Katago paper was that they found it actually quite useful to pool together global features together and aggregate global features throughout the network to kind of give the network a global sense of how to connect value from one side of the board to another side of the board. What does it mean to aggregate global features? Yeah. So if you have a very large 19 by 19 Go board and you've got some sort of battles going on here and you've got some battles going on here, when you pass this through a convolutional neural network, the receptive fields of the convolutional network are going to be good at computing local things and making that invariant, but they won't be able to kind of connect these two features easily, right? They need to sort of be pulled together and attend to each other somehow. So the argument about, you know, why transformers are good for computer vision tasks, like with, you know, vision transformers and so is that because they have a sort of global attention across the whole thing, they can more easily draw these connections. But you do need more data there so that you can kind of learn through data the sort of invariant local features. I've tried very hard to make transformers work for this problem, because I was kind of curious if transformers would present some sort of breakthrough in Go and just remove a lot of those tricks. But to try as I might, I actually haven't figured out a way to make transformers better than Red.
[35:00] That's for now. So one more tendential question. It makes sense why transport and mirrors with their, like, global pooling of information would be better if you need to consider information that is not just spatially... Or, yeah, CNNs give you a sort of bias that the things that are next to you are especially irrelevant. And then they're sort of aggregated up. Yeah, exactly. Yes. But suppose... Okay, so for games where it isn't that relevant what is happening locally, you just kind of have to consider the whole thing. you're saying transformers would work better how about games where so they're talking about the spatial dimension how about the temporal dimension where right now we're only considering the previous move because it is a deterministic full information game where um but what if it was something like poker or diplomacy where really a bluff they made a while back is sort of relevant to understanding now and isolating to decide to make your next movements you need to consider all those previous states, would that change the consideration of what inductive bias is most relevant and what architecture is most relevant? Right. Great question. So Go is a perfect information game. And in perfect information games there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy So if you know that your opponent has a particular bias like they love to play aggressively you can actually, in principle, counter that specific strategy better than a Nash equilibrium policy. But to counter any given strategy, there does exist a single Nash equilibrium that can be decided solely using the current state. So that is a design choice that most Go agents, AlphaGo chose to do, which in hindsight turned out to work very well because the Nash equilibrium seems to be superhuman. Like no human strategy seems to be able to beat it. Now, there are variations of this where you would actually need to consider temporal history. And this is a very exciting research area that I would encourage people to kind of fork my repo and try these things out, which is if you were to play, let's say, 2v2Go, then you actually need to model your partner's behavior and you like you may not have information on how they play so you need to aggregate some information on like how they play so that you can respond accordingly yeah right like these are uh situations where it's no longer a perfect information game and then in those cases in games of imperfect information or partial observability then you do need some context to build a model yeah yeah and and i think that a place where things get very very exciting in terms of like self or diplomacy style Yeah interesting Okay so returning back to the neural network the architecture again is not super important You can get it to work with transformers. You can get it to work with ResNets. I found that for low-budget experiments, ResNets work a little better. You can also use kind of a Karpathy-style auto-research hyperparameter tuning to make your architecture pretty good. And so you don't have to worry too much about that. You just need to sort of set up the problem so that you have a sort of target optimization. Yeah. Okay. So we're going to pick just a somewhat arbitrary architecture that worked for, you know, what I did. But again, this part is not super important. You have your encoded board state. And we're going to just choose to, let's say, do three, like, you know, similar to an RGB, we're going to have three kind of channels. One channel to include black, one channel to include white. and then one channel maybe to encode empties or maybe a masked region if you want to train on multiple board sizes. I'm actually not going to talk about multiple board sizes for now. That's a little bit too complicated. So we'll just say we've got this two or three-channel RGB-like image, and then we go into a ResNet And then we have two branching heads One head predicts the value function And this is like a single logit. So this is like R1. And then we have the policy, which is, you know, R361. So this is the architecture. and we're going to basically train this to predict the outcomes of games given the board state and we're also going to train this to predict what are good moves. So the OG AlphaGo paper or called AlphaGo Li initialized this network with a supervised learning data set of expert human play. Later they removed this restriction by having the model teach itself how to play well but I find it actually from a matter of implementation for your audience super, super nice to always kind of initialize your experiments to something that's easy and then like, you know, get the problem working before, you know, trying to bite off the whole thing and learn a tabular resume. You generally want to kind of initialize, just as in deep learning, initialization is everything, right? You always want to initialize your research project to something as close to success as possible, especially if you're, you know, doing something,
[40:00] that you haven't done before. Always pick something that works and then get it to do something better rather than start from something that doesn't work at all and then try to make it work. So under that philosophy, it's a great idea to start from something that has a good initialization. So we're going to take human expert plays and train this model to predict good actions. So we're going to take all of the winning games, all the moves in which a human won, sorry, an expert won, and then predict those actions. and then regardless of board state, like whether you won or lost, you're going to predict the outcome. So you might be wondering like, okay, well, some of the early boards where basically only one stone has been put down, how could you possibly know who the winner of this game is, right? Well, if you have hundreds of thousands of games, then on average, you'll probably see that boards that start like this have sort of half of the games that branch off from this will win and half of the games that branch off from this will lose. So that'll actually be fine. when you train this model to predict those, the logit will sort of converge to 0.5. And so for these things, it's sort of expected that once you train the model, a starting board state will look like 0 and then as you progress towards the end of the game it actually look something like you know if this is 0 the win probability will sort of either go like this or it go like this And this is sort of your move number. Yeah. And so as you, you know, get hundreds of steps into the game, it becomes much more clear, like, who's more likely to win or who's more likely to lose under your expert data distribution. I didn't understand the significance of why this way of thinking about value is especially relevant to the expert data. It is not relevant to the expert data. It's true for any data that you train it on. So if you were to learn Tablet Rasa, you would also expect this to fall out. So if you just do this, like so imagine, you know, you're vibe coding AlphaGo and you gather some expert data sets from like how to go online. or you have a data set of human players and you train this model, actually it turns out this model is already a pretty good Go player. It'll most likely beat most human players, right? So if you just take this policy recommendation and take the arg max over its... If this is the probabilities, if you take the arg max and you just take this action as your Go play it be a very very fast Go player that doesn think in terms of reasoning steps It just kind of shoots from the hip and it be a very strong Go player which is already quite miraculous if you think about like you know 10 neural network layers maybe under like 3 million parameters can already do something that impressive. And so you can start this way and it's important when implementing this to kind of just verify that this is probably true. It's good to verify that your Go rules are implemented correctly, that you can run these simulations relatively quickly. And just as almost like a sort of a checkpoint that you want to make sure that you can actually do this basic step before you try to layer on more complex things like search. But yeah, we can do a lot better than taking the raw neural network and playing the moves. And this is how we can apply it to Monte Carlo Tree Search. So let's apply the neural network to improve Monte Carlo Tree Search. So we start with our root node. And we now have a four-step iterative process to do MCTS. So this tripped me up when I was first reading the paper and trying to understand it But essentially what we going to do is we going to choose a number of simulations So like you know num simulations And this number varies. This can be, you know, somewhere between 200 to 2048. I believe in the AlphaGo Li match, they used tens of thousands of simulations per move because they really wanted to boost the strength of the model as much as possible. But in training, you don't actually need too many. And Kotago, I think, uses something on this order as well. Do you know if they used, if you watch the documentary, they had a laptop out during the game? Yeah. They didn't use a laptop itself. It was like on some... It was on some TPU pod, I think. Cool. Yeah. But now... Honestly, kind of unfair. Well, uh... Like, Lee is not using like one E22 flops to do a move, you know? Fair enough. Interestingly enough, modern Go bots don't need that much compute at test time. Yeah, I understand. And what we'll actually find out as we talk about how the MCTS policy improvement works is that over time, the raw network actually takes all of the burden of that big TPU pod and just pushes it into the network. And you can do all of that work with one neural network point pass. But the TPU pod will always add the extra oomph on top. And so that's what they wanted for the match. So we're going to pick this kind of like num simulation.
[45:00] thing. And for every simulation, we're going to basically do several things simultaneously. We're going to see which moves are the best in the current tree. We're going to add extra leaves to the tree if we get to a point where we need to add a leaf. And we're going to update the action values for the tree. So that's what every simulation involves, these kind of like four-step process. So the four-step process is basically selection, expansion, evaluation, and backup. So at the beginning of our Monte Carlo tree search, our tree is very basic. It only has the root node or our current board that our AI wants to play at. And so So we're going to basically select the best action for this. So when this root node is created, we also know that we can evaluate this under our neural network and get the quantities, you know, v theta, as well as our probability over actions. And I'm going to say root. So for all of the actions here we can create a bunch of children right So this one has well in this case I drawing a 3x3 board with one board missing So basically, there are eight possible children associated with this root node. So like... and each of these has an associated probability of taking that action, right? So there's P8, P1, P2, et cetera. Okay, so at the beginning of our Monte Carlo tree search, we have our root node, and we can initialize it with some children, right? Because we know the policy network evaluated on the root node gives us, on a three-by-three board with one existing stone placed, eight possible children that this AI could take. So with each of the children, their policy network also gives us the probability of selecting that child. So the first step is to do the selection of the tree. And again, this is a very shallow tree. All we have so far is a tree of depth 1, essentially. So our first move is to select by maximizing or argmaxing the PUCKT criteria which is basically QSA plus C Pucked times P of A divided by N over 1 plus NA So for each of these, we're going to, you know, NA is 0 for all the actions initially. n is zero. And so we're going to basically just pick according to this. Initially, what is going to be the chosen action here is most likely going to be biased towards the highest likelihood action here, right? Because these are sort of uniform for everybody. Yeah. So let's suppose P1 was the highest probability node. So you selected this one here. Now, you've got to this node and you realize that it's not a leaf node, right? It's not a terminal game, so you cannot resolve the final resolution. So the next step that you do is expansion. So you will then run this node this board state through the policy network Note that This is the AI move right Like AI is making this move And so when we expand this tree we now thinking about what the human might do or any opponent might do So this is like your opponent. The tree expansion process actually is completely... So when we evaluate the node here, we're going to now evaluate the node from the perspective of this player. So then this one has possible actions that we could take, and we expand basically the leaf nodes here. So for each of these nodes that we could arrive at, we're going to now check how good those nodes are. So maybe from here, like the human could play here, the human could play here, or the human could play here. And we're going to store, essentially, the v theta for each of these things. So v theta of node 1 prime, v theta node 1 prime. And so we're basically using
[50:00] our neural network to make an intuitive guess of how good is this board from the perspective of this player. And fortunately, because it's a zero-sum game, it's easy to deduce that the value for this player at this step is just one minus the value for this perspective. So it's easy to flip the search process depending on which player you're at. And so this is the expansion step. You've taken a non-leaf node and expanded it and evaluated the value. And this is essentially a quick guess as to, like, if I were to play to the end, am I going to win or not? So you can almost think about the V theta as a shortcut for searching to the end of the tree for any given simulation. And this is essentially the evaluation step. We're evaluating the quality of each of these boards. In original AlphaGo Lee, they actually did something kind of interesting, which is that they took this value and they averaged it with the value of a real Go play out. So they actually played a real game from here all the way to the end. So I'm just gonna draw this squiggly line to indicate some path And they kind of played this all the way to Trump Taylor resolution of a full board And so this is like a zero or one right And so they took this value and they just averaged it with this one here. So the formula they did was like alpha times V theta of some node plus sort of like one minus alpha of a true randomly sampled playout. And you might be wondering, like, okay, well, how do they play this out, right? Like, it would be very, very costly to do another search on this playout, like, almost like a tree within a tree. So they don't do this. Instead, they just take the policy network and play it against itself. So they just take this as both players, and they just play it all the way to the end. And this is something that helps ground the estimates here in reality, because you can get a single sample estimate of whether you win or not. You can think about in the endgame, where the board is almost resolved, that this one actually becomes quite useful, because the play according to the policy will most likely decide a pretty reasonable guess of the game And so you not facing a problem where this one kind of becomes untethered from reality It turns out this is totally unnecessary So in all subsequent papers after AlphaGo Li they just got rid of this. And so in my implementation, I also did the same, and it speeds things up a lot because you don't have to roll these games out on every single simulation. Okay, so again, just to reinforce my own understanding and just to re-explain it. For the audience, by the way, in case it's not obvious, the P there in the select, is the probability coming from the network in this case. Correct. The policy network here. Yeah. Okay, so fundamentally a simulation just think of it as like rolling out one more node in the search process. Almost. So a simulation is easy to think about when the whole tree already exists. You just walk down the tree using the pucked selection criteria and you keep going. Now, in AlphaGo, the data structure is such that we begin with a tree that has no, like, basically only depth one, which is its only children, and you want to iteratively build out the tree as you're also selecting actions down the tree So that the kind of core thing here is that because Go is such a combinatorially complex game you cannot afford to build the tree in advance and then search it You must search while building the tree Right okay So let me just finish up with actually the last step which is the backup right So once you've scored these things, you basically take the mean, the value, the Q value assigned to the node here for taking this action is now just the average across your evaluated values here. You take a running mean over all of the simulations that you've taken, and they average the values of the children's notes. So that's what is known as the backup step. And once you evaluate this, you can actually kind of recursively go back. So if you know the action value of this node, you can then take the average on its parent and so on and so forth. So you have this kind of four-step process where you are choosing the best action that you know of so far. then you may run into a node where you haven't been to before, so you need to grow the tree a bit, and then you run it through the network to guess whether you're gonna win or not, and then you walk all the way back up to the root node to update your values on what the best moves are. So as you do this iteratively, this selection criteria will cause you to visit the, because you're always selecting according to this criteria, you're always gonna be selecting
[55:00] best action you think at any given branch, right? So the final visit counts of like how often you chose these things will reflect your correct policy distribution as induced through this search process. And so the visit count that you store in the node earlier actually becomes the sort of vote for like which way we should finally select an action here. So, you know, as a sort of test of understanding, it's worth thinking a little bit about whether we could make this even simpler, right? Like, could we actually maybe even get rid of this one and still make the thing work? So recall that, you know, when you do an expansion and then an evaluation at, let's say, this node, you are checking the sort of win probability of each of the child nodes, right? And so if this one is, you know, like one and these are zero, you do kind of know something about which action might be better to take. And so why would you still need this, right? Why not just normalize this one into some distribution and call that your policy distribution? This is fine. You can do this. And this probably does work. But in practice having a single forward pass that gives you a pretty good guess is how the breadth is pruned out There is a sort of duality here It would be weird if, let's say, the policy recommended an action that disagreed with the value. If, let's say, the policy said this was very high probability, but this one said it was low value, then there's actually something kind of fundamentally wrong between your policy head and your value head. So they are linked, and you probably could get rid of this if you came up with a different way to recover this from just the value evaluations. Right. But just to make sure I understand, the reason you don't do that is so that you don't have to do 360 independent forward passes to like, hey, here's the value of everything, let's arg max over it. Instead, you can just do one forward pass and get the probabilities of all of them. You can usually batch these somewhat efficiently. So it probably is not a huge computational burden in practice. But yes, you would have to pass up to 361 boards into a single mini-batch update to evaluate all the values here, then normalize them. Now there actually a more important reason why we still do this which is how Monte Carlo Tree Search is used to feed back on itself and sort of recursively improve its own predictions and search capabilities And that where this one having this as an explicit entity you modeling rather than an implicit normalization over your value is a good idea Makes sense. Okay. Okay. So we talked about the simulations and basically, you know, what you end up with as you roll out the number of simulations is a tree that kind of looks like, I'm drawing a very low dimensional version of this. Of course, in the real game, it's much more high-dimensional. But you'll end up with basically a tree structure that has a lot of leaves that terminate and are not visited again because their value is deemed to be too low. But then along one path, there will be a set of actions with very, very high visit counts that gravitate towards that one set of decisions as you increase n. So this is kind of like the mental picture of what the tree in Monte Carlo tree search looks like. And you should contrast this with like an exhaustive tree, like in tic-tac-toe, where you could say like, you know, there's nine actions and then eight and then seven and six. And so it's a sort of like nine factorial sized tree. The Monte Carlo tree search in Go is very very sparse It only considers the paths that you expanded children nodes on Okay so now that we have the search algorithm that applies the value function as well as the policy function, we can now talk about how the Monte Carlo Tree Search algorithm can actually act as a improvement operator on top of these guys here. 20 years ago, Jane Street's data center fit in the corner of an office. Ron Minsky, who co-leads the tech group there, told me about how it all got started. One of our compute clusters we called The Hive. And I remember the first version of The Hive was literally like six Dell boxes stacked on top of each other at the end of the row. And the trading systems themselves, we also had there because we actually wanted the ability to make sure we could turn the damn thing off. I mean, there were ups and downs, like literally at some point, one of the people who was cleaning the office unplugged one of the trading systems in the middle of the day as they were vacuuming. So, you know, in the end, it is in fact better to have it all in a data center. Jane Street's data centers have come a long way since those six tells. And I got to tour one of them in Texas with Ron and Dan Fontocorvo, who leads Jane Street's physical engineering team. You know, these cabinets, these GB300 cabinets, consume at peak about 140 kW. Compare that to traditional air-cooled, you're talking about 10 to 40 kW. It's a lot more.
[60:00] I got deep into the details of running one of these data centers, things that I had never considered before. It's filled with a liquid, a mix of distilled or deionized water and propylene glycol, 25% of propylene glycol. That's to inhibit any bacteria or algae growth. I don't love the world where we have to worry about bacteria growing in our servers. I got to see way more of what actually happens in a data center than I've ever seen before. Jane Street was willing to literally pull up the floorboards and take out the racks and take me to the back where all the chillers are. You can check all of this out at jainestreet.com slash dhwarkash, where we posted the full tour. Okay, so we now talk about the RL part of how this thing gets stronger by playing itself. Let's say we play a game where you have the AI. So you make a move. AI will kind of compute the search. And then this is this sort of visit count distribution. Let's say this is your policy, your initial policy recommendation at this node. And then after MCTS it gets more confident about one of these actions right And so maybe the distribution looks a bit more peaky like this based on the search Now of course you can tune the search process so that it ends up more diffuse but that probably not a good idea MCTS should get more confident about specific actions than others, but it, of course, might place a lot of weight on, you know, other actions initially, and then as you increase the number of sims, it should converge to a very peaky distribution. So this is your new, let's call this like pi, let's wrap this in like a MCTS operator of A given S. So after applying MCTS process, your policy recommended distribution looks like this. It's a bit more peaky than the previous one. And so then you take the argmax, or maybe you just sample from this. It doesn't have to be argmax, and then you make your move. And then you throw away the tree, and then you begin a new on the next move, right? So again, like you compute a new distribution. So initially maybe your guess looks like this and then you refine it through MCTS There should be one more X on the board right I sorry That correct Yes To something that looks like Right. So on every move, you have your initial guess from your policy network. And then the search process that combines your policy network and your value network arrives at a more confident action that you take. And then so and so forth. And then the game ends. And one person wins and one person loses. So the beauty of how AlphaGo trains itself is that it actually can take this final search process, the outcome of the search process, and tell the policy network, hey, instead of having MCTS do all this legwork to arrive here, why don't you just predict that from the get-go? Why don't you not use this guess and just predict this to begin with? And if you have this guest to begin with in your policy network, then MCTS has to do a lot less work to get things to work. And so if we draw a sort of test time scaling plot, so let's say this is number of simulations. Let say at zero simulations your sort of implicit win rate is like I don know here And then without any simulation if you just take this raw action this is what your winning rate is And let say as we increase the number of sims maybe you kind of have a win rate that looks like this. So when you search for, let's say, 1,000 simulation steps, that gets you to a policy here that gets you to here, which is great. But if you were to distill this MCTS policy network back into your sort of shoot from the hip policy network, then you could actually, you know, start here. Like if let's say this was, you know, zero by distillation, then if you spend another 1000 SIM steps, then you actually kind of get to here. It's almost like if you could just amortize the first 1,000 steps actually into the policy network instead of the search process, then you can begin at a much better starting point and then get a much better result for the number of sims that you put. The sigmoid type nature of test time scaling as the number of simulations increases, the increase in win rate is smaller. Is that true even for the distilled network? That is to say, is there some gain of like, okay, we start from the distilled, we get these early games,
[65:00] Or is that just inherent to the nature of MCTS? To be honest, I actually don't know the test time scaling behavior of MCTS simulations. And I believe it might actually be quite sensitive to how strong this one is in practice. I'm just drawing a monotonically increasing function that gets to 1. So don't pay too much attention to the shape of the curve. Just know that it's monotonic with respect to 7. Okay, so the idea of MCTS is very brilliant, which is like we got something better by applying search. and we're going to now, on our next iteration of updating this network, just train this to approximate the outcome of 1,000 steps of search. And so instead of starting here, we get to now have our neural network start here, and then the play gets stronger once we then apply another 1,000 steps on top of it. And you can keep going, right? So the training algorithm for AlphaGo is to basically take the games where you've applied the search on every move that the policy encountered, whether you won or lost, and that's quite important. And you're just going to train the model to imitate the search process. So there an analogy to robotics actually which is the Dagger algorithm First I going to draw a schematic of let say the states right So S0 S1 S2 S3 So let say we took a series of actions in an MDP to get a trajectory And these actions may be suboptimal, right? Maybe we lost at the end of this game. So there is a family of algorithms that basically take trajectories and relabel the actions to better trajectories. So maybe a better action here would have been to take, you know, a0 prime. A better action here would have been to take a1 prime and yet another one like a2 prime, a3 prime. So what MCTS is doing is basically saying, like, you play this game where you eventually lost, but on every single action, I'm going to give you a strictly better action that you should take instead. It does not guarantee that you are going to win but it does guarantee that you know if you take these tuples as training data so that you retrain your policy network to predict these ones instead of these ones you going to do better And this is very related to Dagger in robotics and imitation learning where you want to collect an intervention here And even if you in a not great state for example, like a self-driving car that veers off the side of the road, there is still a valid action that kind of corrects you and brings you back. Yeah. Okay. So, pedantic question. But is there a guarantee that MCTS must be better than the policy? For example, you could imagine early on in training, because MCTS is informed by the value network, early on in training, when the value network hasn't been well-trained on finished games, that MCTS is worse than sort of randomly exercised policy. So is it just like a heuristic that MCTS is better than policy, or is there some guarantee? Right. In practice, it is a heuristic, and it does work also in practice. But let me illustrate an example where MCTS can give you a worse distribution than your policy network. So, and this can often happen if your self-play algorithm has trained to a good point, but then somehow it collapses because it's not trained on diverse data or something. So, let's say we have a board state where the policy recommendations here are very good. So like you know pi of AS is like great But somehow because maybe we playing on a lot of games where the bots just resign instead of playing all the way to the Trump Taylor resolution they kind of forget how to evaluate those kind of late stage plans, right? Like in the case that we showed with the corner play, maybe like 100% of our training data in our replay buffer has lost examples of how to evaluate the value function at those states. So you might end up in a scenario where your terminal value is like very bad. And if the terminal values of the leaves are not good, then this will actually propagate all the way up and cause your puck selection criteria and your backups to be off. And then you end up visiting a very, very different distribution than what your policy initially recommended. Also, if your number of sims is low, then you might also have a variance issue where you just don't explore enough, right? Like it's only guaranteed to converge when you kind of take end to infinity. So, variants in your search process as well as inaccuracies in your evaluation can definitely screw with the quality of your policy network foundation. And so that's why it's not a guarantee to improve. And that is why I suspect why AlphaGo Lee had the playouts to the end in their training algorithm so that they could grow.
[70:00] this thing in real plants. In practice, what you could also do is just like for 10% of the games, you prevent the bots from resigning and you just say like resolve it to the end. So you get some training data in your replay buffer to really resolve those kind of like late stage playouts that normal human players would kind of not play to. Yeah, yeah. So this is why MCTS kind of, if you assume that the value functions are correct, why it gives you a better policy is because, and it's a very critical chain of assumptions. Assuming that this is accurate, then your search process should give you a better recommendation than your initial guess. Right. Okay. So if you have a cold started policy, if you have an off-a-zero type thing, really what's happening for the first few epochs is the policy is kind of useless. And what you're really just doing is, hey, but let's play full games. and uh once we have played full games for the preceding moves we'll have labeled who won who didn't win and the loss for alpha zero has two components which is like how good is the policy relative to mcts and how good is the value prediction relative to who actually won the game from this move and this is this sort of like you can think of this being applied to every single action or every single move right and really what happening the beginning of AlphaZero training is just like we trying to get the value function to actually predict who will win the game If you find yourself in this state and you're this player, and functionally that's all that's happening. And later on, once that's well trained, now the policy is also improving. Correct. Okay. One trick I did find to be pretty useful, and this is not a peer-reviewed claim, so just like take this with a grain of salt, this, I found it useful in my own implementation to do the following. You want to first make sure that this is good before you invest a lot of cycles doing MCTs. It doesn't really make a lot of sense to do search on garbage value predictions. So you want to start at a good place where this works. AlphaGo Lead does a very good thing where it just takes human gains and then you trade on it and it just works. Totally works. You can also take an open source GoBot, play it against itself, generate data. Also works. So if you have some offline dataset that has realistic good play, you can easily learn the late stage value functions pretty well. And that's what you kind of need to start the search process. Sorry, can you just read this one more time? Sure So it quite easy to evaluate a late stage Go game Like when almost all the pieces are on the board it almost like a decidable problem right Because there a lower and lower uncertainty as to the depth of the tree So most games played to the end by reasonable people will be good training data to train a good value function at terminal parts of the tree. Got it, okay. Then as you play more games, the search will back up good values into the sort of intermediate nodes of the tree. And then as you increase the amount of data, your value head gets a good intuition of like what is a healthy board state versus a not healthy board state yeah that those are much more subtle to judge in the mid game than the beginning or the end so the most difficult part to score is like not the beginning or the because the beginning is just like obviously 0.5 and then at the end it's like pretty obvious who's winning so so the hard part that you want to learn in the value function is like who is winning in the middle and this is actually very analogous to td learning yes and there's a beautiful connection to td learning that that we can talk about in a bit, as opposed to contrasting with Monte Carlo Tree Search. So you first want to get good value functions, and expert data can kind of give you a quick shortcut. I recommend for practitioners, just do that first, just to initialize to a good starting point, and then if you want to do the AlphaZero thing, or Cotigo kind of tabula rasa learning then what you can try to do is on a small board play random games just take a random agent And if you play like you know 50 games you actually learn a pretty good value function as well. Because on a nine by nine board, there's actually, you can see enough of the common patterns with random play. And then if you train a model that kind of can train on both nine by nine and 19 by nine data, and Katago was a proposed one of these architectures, then there's some pretty good transfer learning from the value uh value head evaluated at nine by nine to the 19 by nine right because this this unlike other games has much like a very much a sense of like there's not like a new kind of piece that is introduced when you increase the size or something if we take it to this limit and consider like a very tiny like four by four go board yeah like if you play 50 000 games you're gonna have a lot of end states that look like human play right like it's just like tic-tac-toe at that point um so so if you like broaden this a little bit to like nine by five by five or nine by nine, it's not unrealistic to imagine that like purely random play will actually generate pretty reasonable looking boards. And then so you can score those pretty easily. And so that is what gives you the bootstrapping to be able to then improve your policy with search. But it's very, very critical that MCTS has accurate value estimates. And you need to ground the value. Ultimately, MCTS will fall apart if you don't have a grounding function for the value.
[75:00] how much compute you save by training the value and policy on the same network that because they share the same representations how much more efficient learning is because that would be interesting if um they're basically kind of we've just talked about how they're kind of making similar predictions or they should be in line with each other yeah and so i'd be curious if like actually yeah you just you're you're like having the amount of compute you had to do by giving them the same network right alpha go lee the original alpha go paper had two separate networks yeah And then in all subsequent papers, they merge them into two heads. And presumably this saves compute. But answering that question in a very rigorous scientific way is actually a simple question, but in practice actually takes – like if you really want to chase that question down to its limit, it takes quite a bit of work to really resolve that. But intuitively, yes, they share a lot of representations. And as we mentioned, there is a sort of like – your policy network and your value network when doing evaluation should kind of agree, right? So there really should be this sort of consistency between them. Yeah. I don't know if this is the wrong way to think about it. I feel like when I learn how an LLM works and how simple RLVR is, at least as an algorithm, how simple it is, I sort of stunned by the kinds of things it can do that it can learn how to build very complicated code repositories and whatever simply from getting a yes And here I feel like if you understand it more deeply of just predicting MCTS it actually seems less impressive in retrospect the more you understand it Because you're like, oh, you're putting in a lot of bias by just saying how much you do. You're telling it how it should titrate exploration as things go on. you're building this very explicit tree search for it. And so I don't know if you share that intuition where it actually, the more you understand, the less impressive the accomplishment in 2017 seems. I personally disagree. I think they're profound for different reasons. And I don't understand the LMRL enough to comment on your podcast about it. But I think AlphaGo, so why is it a profound accomplishment? I think maybe it's worth stepping back a little bit and just like, it is different than modern RL, and we can talk a little bit about some of the algorithmic choices there, but I think the most profound thing here is that a 10-layer neural network pass, so basically 10 steps of reasoning and of course the reasoning is not just one trail of thought It could be like the distributed representations and a lot of thoughts going on at the same time But by construction let say a 10 neural network can only do 10 sequential steps of thinking 10 steps of neural network paralyzed distributed representation thinking is able to amortize and approximate to a very, very high fidelity a nearly intractable search problem. So this was a breakthrough that I think most people don't even understand today, like fully comprehend how profound that accomplishment is. And this is what also girds AlphaFold, for example, where you have a very, very difficult physical simulation process that you would need to roll out so many micro-scale simulations. And yet, like, 10 steps of a somewhat small neural network can somehow capture what feels like a, you know, MP class problem into a single problem. And so it actually makes me wonder if, you know, our understanding of problems like P equals MP or, you know, these very fundamental, like, computational hardness problems are incomplete, right? Like it not like you know obviously this is not a proof of like P equals NP or anything but there something to it that like kind of is very disturbing where like what felt like a very hard problem can fall to a very very simple macroscopic yeah that is a very interesting insight that a lot of problems which are proven to be np hard like i don't know if go is proven to be np hard but okay protein folding etc have been like neural networks can solve them because they're np hard in the worst case but we're not dealing with the worst we're usually not concerned about the worst case we're you know like these problems have a lot of structure to them yeah i think that the the kind of question we should be asking ourselves is like we've been formulating you know solutions to mp hard problems as in like kind of worst case complexity and i wouldn't say you know this solves go right it doesn't give us a exact solution of the optimum yeah but in practice like it is extremely useful and the same thing has been shown in like alpha tensor alpha fold where like yes there is a very hard problem that in the worst case seems intractable, and yet we're able to make almost arbitrary amounts of progress. So here's a sort of like, in the limit, what might this look like? Well, if you want to simulate something very complex like weather or predict the future, do we live in a simulation or not? The computing resources you need to build a very complex
[80:00] complex simulation might be much smaller than you think based on, you know, our ability to amortize a lot of that computation into the forward pass of a single network. Interesting. So to me, AlphaGo was the first paper that kind of like really showed this like profound level of, you know, simulation being compressed into a small amount of. I feel totally not at all qualified on the computational complexity of the math to comment on this but i wonder if um there's an important role of chaos here where if under what is the problem with weather and why does it take 10x the amount of resources to predict weather a day out and continually so for every more day out is because it's a chaotic system and so small perturbations can totally change the final estimate as time goes on and um i guess it's interesting well i guess you would expect that for go and protein folding as well so here's an analogy to weather that might be relevant in Go. So the problem of like, you know, here's our current board state Given what we know about both players what is the board state in the future What is the exact board state in the future This is extremely sensitive to initial conditions A single stone place here can disrupt the entire prediction This is hard. This is intuitively the chaotic problem. And yet, somehow, so this is hard, somehow we can predict who's going to win. And this captures a lot of possibilities here. And so there's this more macroscopic quantity that we really care about, which is the average or expectation or some sort of global macro structure over a lot of possible futures. That's an interesting way to think about it. And so in weather, it could be the same thing. We don't exactly care what the velocity of wind 6,000 feet above a specific latitude or longitude is. We kind of care where's the hurricane or things like that. And I would say, like, in chaos, you know, there's a classic, like, Lorenzo tractor, which kind of looks like this, right? Yes, you don't, if you start anywhere on the Lorenzo tractor, you don't know where you're going to end up. But you do know that the thing looks like this. Yeah, yeah. Right. And so there's this kind of beauty of, like, sometimes we don't necessarily care about the microscale things. We actually care about the macroscopic structure. That's interesting. And these things can be predictable And contrast that say to something like a hash function which is also incredibly dependent on initial conditions but doesn have a macro structure or at least hopefully if the arguments work Yes, one would hope. And so there's no equivalent of a value function or broadly how's the weather going to be that is interesting there. It's really just about what is the move, what is the board going to look like 100 moves from now exactly? Yes, intuitively that seems correct. And then again, this is also out of my area of expertise, but I find it interesting that cryptography has not been able to, like the tools of cryptography and hashing have also not been able to prove that you cannot come up with fast approximations. Like you cannot come up with fast approximations. If they were able to do that, then you could prove P is not equal to MP. In fact, we know that there's structure in many cryptographic protocols, obviously like RSA cryptography. There is structure, and that structure is what quantum computers exploit to break them, right? I see. Reiner has a very interesting blog post which we talked about in the episode where he talks about how if you look at a high level what cryptographic protocols look like and what neural networks look like it extremely similar Where you have sequential layers of jumbling information together and it because there this conversion devolution in the algorithms where in cryptography you want the final state to be incredibly sensitive to initial conditions so that it can come out sort of looking jumbled based on if you change anything. And then neural networks, you similarly want everything to be dependent on all the information because you want to process all the information and consider how it relates to itself. Yeah, you have the maximum power of a neural network at the edge of chaos. I think there's some research papers from Joshua Stoltec on this. Yeah, there's something kind of quite fundamental about chaos that it's not just hopeless noise. It's like there's something kind of useful in chaotic systems, at least at that boundary. But yeah, this is just my thing about this philosophy. I don't actually know the math well enough to comment on it. Anyway, if we go back to, we'll talk about LMRL in a little bit, because there's some connections there, but let's just go back to the MCTS. What is it doing? It is not, crucially, it is not saying we're going to increase the probability of winning directly. It's not going to say we're going to upweight all actions that won and downweight all actions that didn't win. Yeah. Importantly, what it is doing is saying for every
[85:00] we took, we did a pretty exhaustive search on MCTS to see if we could do better. And we're just going to make every action that we took better by having the policy network predict that outcome instead. And so this is a very, very nice idea because you have one supervision target for every single action. Yeah. So the variance of your learning signal is very low compared to the alternative naive RL thing. So let's actually consider a very naive algorithm that looks a lot more like modern LLM RL today, where we do something like, let's take the winner of a self-play game and encourage it to do more of that. Okay, so it's worth kind of thinking a little bit about, okay, what are some alternatives we could do to train self-play agents instead of MCTS? We use a lot of LLM-style RL these days. Is that relevant? Could we do that instead? So let's think through this a little bit. let's suppose we have a very naive algorithm where we take a league of agents of different checkpoints and we play them against each other. And for the games where a single player wins, we're going to reinforce those actions up and retrain the policy network to imitate those guys instead of the MCTS objective So what ends up happening is let say you have a chain of actions that led to a win And you have a matchup between two agents that are basically the same. So in fact, let's just assume that policy A and policy B are evenly matched. So their true win rate is like 50%. So let's say you play 100 games. And then each game, let's say, lasts 300 moves. And you're doing some sort of evolution strategy or some way to perturb these things to get them to do different things. Or maybe you don't and you just play them against each other. And you see, like, occasionally this one might actually have a better strategy than this one, right? And so let's say, you know, 51 games, policy A wins. And then 49 games, policy B wins. And this is just due to random luck or maybe you perturbed policy A in some way that let it do this And just to have a very very simple model let pretend that for 49 of the games they played exactly equally. I'm sorry, for 50 of the games, they played exactly equally. And on that one game where this one won, it played slightly differently. It made one critical move that normally it would have done differently, but due to some exploration or some random noise, it just happened to make a smarter move than it did previously. So you have one supervision signal, like one true supervision signal for your policy network. And then you have 99 games times 300 moves for which imitating those actions gives you exactly the same policy you had before. And so the scale of your variance is actually very bad because it's like you only have one label out of this enormous data set of actions, of supervision actions, where you want... Actually, sorry, let me clarify a little bit. Okay, so we were just talking about how the good move, the outer distribution move, is a small fraction of all the moves that are played across all the games on which you'd want to train. And this of course reminds me of how LLMs are trained with policy gradient methods Karpathy when he was on the podcast called it like sucking supervision through a straw And so, yeah, it's interesting that this thing you're saying, which would be intractable and prevents you from actually getting beyond a certain level in Go is just by default how LLMs are trained? Right. So in this case, this is not to say it doesn't work. If you imagine increasing the number of games to millions of samples, you actually can get some meaningful supervision. Samples so long as you find a way to sort of mask out the supervision from these guys. And then this is where things start to get pretty related to RL in terms of advantage and baselines and so forth. So let's look at the gradient variance of a very naive approach like this, where I'm just going to call it gradient RL. and it's basically the sum of rewards. Okay, I see what you're saying. Okay.
[90:00] So the sum of rewards is the return, right? So like in our naive setup here, we only have an indicator variable for the return where either you won or lost. So in the case where you lost, well, you just don't train on, your gradient is zero. You don't train on those examples. And when you won, you try to predict those things, right? So you can think about this setup as a special case of this general formula here. The trouble here is that this is very high variance because when you multiply these terms out, when you try to compute the variance of this, and so variance of the gradient is equal to expectation of squared minus... And just for simplicity, we can pretend this is on average zero or something. if you're centering it at no signal. And the variance here basically means that you're taking the square of this product term, and so you end up with a term that kind of grows quadratically with T So variance when you have a setup like this this thing acts as a coupling effect on top of these terms here So let's actually map this to an LLM case, and we can answer, like, why do LLMs only do one-step RL instead of a multi-step RL scenario? in LLMs you have a decoder that might you know predict some words like hello world and so in current LMRL they treat this entire sequence as a single action just AT and big T is just one right and so yes it is true that you know the because of how you know transformers are formulated through the sort of product of conditional probabilities, we do have, you know, probability of this sequence is equal to the sort of sum of, log probability of the whole sequence is equal to the sum of the probabilities of like, you know, individual tokens, right? So in this case I would say something like you know log L plus log low plus log world So this is true And if this term were one, then they would be the same thing. However, in sampling things, if you have a reward term assigned to every specific token, now you have these interaction effects between the cross multiplication of these terms and these terms. And so the problem becomes, how do you ascribe the credit associated with every episode to all these different terms here? I guess the thing I'm confused on is, what would that even look like to do it that way in LLMs? Because you only do get a reward at the end of the episode. You could imagine a reward that says, I'm going to give you some process supervision, where you get a reward for each of these actions on every step. Okay, so you're saying instead of doing it that way where you... Well, I guess the way you've written it, it would be a sum at the end anyways, so it wouldn't have to be multiplied. But you're saying instead of doing it that way, you would just add up this process rewards at the end and then treat that as one single reward signal? Correct for one single log prop action But isn that how it written to begin with anyways Like the sum of the rewards So the thing that a little bit hidden here in the math is that we're assuming that when you decompose the problem to a multi-step problem, that you're now introducing kind of correlations between your actions through the computation of this guy. And so if you separate these things out, then there will be, this will magnify the variance of this one. So in the case where you don't separate it out, If you just have t equals 1, you just have a single estimate of logprop and a single estimate of reward. Now, this term still shows up in LLMs. So in LLMs, it looks a little bit more like the naive reinforced estimator looks a bit like return of the single action times, you know. It looks kind of like this. This is sort of the very basic form here, but this is still a contributor to variance. So you want to make sure that you don't, similar to how in this case we were training on a lot of neutral labels, you want to make sure that you're sort of penalizing the labels that don't help, and only rewarding the ones that actually make you better. Right.
[95:00] The analogy here is like, can we find a term in our training objective such that it's actually kind of discouraged from doing this? Or, you know, these don't have any effect on the gradient and this has an effect on the gradient. Right. I guess if you apply that there, the only thing you could do is eliminate 49 of the games. So at least the way you have it in there, you would be 51 times. Actually, the optimal case is to pull out, discard all of these moves, and only get a gradient on that single move that you got better. Yeah, but how would you do that? Right. So this is a pretty tricky problem in practice. And so this is where advantage estimation happens in reinforcement learning. So you want to subtract a term from your multiplier. Instead of an indicator function of 1 and 0, you want something that kind of behaves like a 0 for all of these guys, and then a 1 for all of these ones. So you could do that if you can say, hey I won this game so this is slightly above baseline performance Well you won on a lot of games Exactly But you don know which ones let you win because they were truly better versus winning on access How would you design a baseline where it truly better Yeah, so this is where in RL, people use things like TD learning to better approximate the quality function, the queue that we mentioned earlier. So you can try to subtract that from your return. So ideally, what you really want to do is, In RL, you want to push up the actions that make you better than the average and push down the actions that make you worse than the average. And they call this advantage. There are multiple ways to compute it. I highly recommend John Shulman's general advantage estimation paper as a good treatment on how to think about various ways to compute it. But at the end of the day, you want to reduce variance by trying to make this smaller and so it doesn't magnify the variance. Right, makes sense. But this requires you to have a very good estimate of what average performance from a state would look like. And this gets us back to the value function thing we were talking about earlier. Right And so keep in mind that in this case this model RL setting is trying to solve a credit assignment problem where you don know which actions were actually good and which ones were bad Monte Carlo Tree Search is doing something very fundamentally different which is it not trying to do credit assignment on wins It's trying to improve the label for any given action you took. And so we can actually think about a completely different algorithm called Neural Fictitious Self Play, which was used to great effect in systems like AlphaStar and OpenAI's Dota. book. So let me talk a little bit about how you can kind of unify some of these RL ideas in the model-free setting as well as the self-play setting. Okay, so what happens if you don't have the ability to easily search a tree, right? Like in Go, it's a perfectly observable game. You can easily construct a pretty deep tree that completely captures the game state. In a game like StarCraft, where you don't have really complete control over the binary, it's a little bit hard to do this, and I'm not even sure if it's a deterministic game, right? So that makes this kind of difficult from a data structures perspective. So what is done instead is that the basic idea of supervising your actions with a better teacher is still there, right? So given neurofictitious so we going to talk a little bit about how neural self works The same idea, we're going to come up with better labels for each of the actions we took, just like in MCTS. But how do we derive the better labels? In MCTS, we perform search to, and assuming we have a good value function, the search will kind of give us a better result than our initial guess. In a game where you can't easily simulate a search process, what they do instead is train what is known as a best response policy. so you fix your opponent so let's say you're currently training pi A against a strong opponent pi B in StarCraft maybe like you know those are the Zergs and you're playing Protoss so you fix your opponent and you treat this as a classic model free RL algorithm where your goal is just to beat this guy And so here you use your standard TD learning style tricks or use PPO or any actually like,
[100:00] all free RL algorithm to try to hill climb against winning this player. And so you train, basically you have a reward function that's like, you know, return is like, you know, one if wins against IB. So this is no longer a self-play kind of problem, right? This is just like a fixed opponent and you're just solving, trying to maximize a score against that and then, you know, zero otherwise. And so you have a sort of fixed environment where all you care about is just beating this guy. And once you have a good policy that you train with, you know, pick your favorite model-free R algorithm, PPO, or SAC, or, you know, any kind of mixture of the, or, you know, VMPO or whatever, you now have a good policy that gives you a good label for what this one should do when playing against that player. And when you train multiple best response policies, you can basically then distill the RL algorithms into the labels for a given opponent. So you might have, let's say, a best response policy against PIB, and then maybe you have a league of opponents like PIB PIC PID and you going to take the best response policy that you train against each of these fixed opponents and for this one you going to supervise them with the label that this one would provide. So it is kind of like, this is almost like a proxy for your MCTS teacher, right? Instead of MCTS teacher, you use a model-free RL algorithm to find the best search action that you could do to kind of beat your opponent, and then you're finally, you're distilling the policy here into what is known as a mixed strategy, where it's trying to basically average across all possible opponents it could play against. And this is what gives you something that can do no worse than an average selected opponent from the league. And so this gets around the problem of having to derive a teaching signal from MCTS, but it still fundamentally is about relabeling your states with better actions so that they improve your policy. And just to make sure you understand, this is like, if you win against this other policy, you sort of reinforce all the actions on that trajectory. Yeah, so here you can use a number of algorithms like PPO, VMPO, Q-learning even if you want. The specific algorithm here can be it usually a model thing because you don have search But is an interesting connection from MCTS and Q that I want to bring up So in MCTS you do something where you have a tree And through the resolution of your value function at the leaves of the tree or, you know, your approximate leaves of the tree, you can kind of back up through the sequence of many sequences and then obtain some sort of mean value estimate, right? Your Q is kind of derived from the average of a bunch of simulations. In model-free algorithms, there is often a component of estimating a Q value. And so Q values are often learned through TD learning, although in PPO, the way that they do advantage estimation is not necessarily through a Bellman backup. But in Q-learning, there's this kind of very cool trick where you do, you know, QSA is backed up as R plus, you know, some discount factor times the max AQ of your next step. So intuitively how this works is like if you have an MDP and then this is like you know terminal what this is sort of saying is that, like, the best action you can take at this state is equal to the reward you take for, you know, taking this action, plus the best that you can do at the next state. So there's a sort of recursive and dynamic programming property of MDPs, and you can train neural networks to basically try to enforce this consistency. So you can say like, well, once I know the Q value of this action, I can then use that to kind of compute something about the Q value. So when earlier I was like, hey, why are we training policy? Why don't we just train the value alone? That is what this is. This is an algorithm for recovering value estimates of intermediate steps when you don't have the ability to do forward search. So you must collect a trajectory first of like n steps before you're able to do this trick. But the intuition is kind of the same, which is that like knowing something about the Q value here can tell you something about the Q value here. And indeed, you can recover a policy from a Q value. So you don't need to explicitly model the policy distribution. You can actually recover the policy distribution by doing argmax over your
[105:00] your queue values. So queue learning or this kind of approximate dynamic programming kind of propagates what you know about the future queues backward like this. And you can see that there's a sort of similar structure that goes on here where in this case you're planning over trajectories your agent hasn't actually been to yet. Whereas in this case you're planning over trajectories your agent has visited. So importantly, why was queue learning a big deal? It's because historically we just haven't had the ability to do search on fairly high dimensional problems like robotics or whatever so for a long time we kind of make the assumption that like okay well if we can't model the dynamics with like a world model or something we're going to instead just collect trajectories and then plan with respect to the only number that really matters which is reward okay so this is very interesting and then to unify this with our discussion of llms so with llms you're doing something you don't have q values but you're doing this sort of backwards learning where, hey, let's find the trajectories which pass some unit tests in some coding environment. And then let's reinforce those trajectories. And then there's a huge difference between that and this forward approach with MCTS And the reason you can do MCTS and it much more preferable to do MCTS because you can do it per move and make each move better rather than having to learn per trajectory um and hope you know as karpathy said hope to learn this like straw yes you get the supervision through a straw uh basically just upgrade all the tokens in a trajectory that might or might not have been relevant to getting the answer right the reason you can do this much more sort of sample efficient much more favorable thing with Go is that because MCTS works in Go, you basically know that, hey, if I just do search locally here and this search is sort of truncated at the end by this value function that works even if I haven't unfolded my whole trajectory, I can just say this is my new policy and I can improve in a more iterative, like local way rather than having to having to unfold all these trajectories. So there was some research, I think, from Google in 2023, 2024, where they did try to apply tree structures to reasoning. And I think the jury is still out as to whether this can ever work. So I would say we probably will see revisiting of this idea of forward search in the future but there two things that make MCTS very simple for Go which is that value estimation is kind of concrete and you can determine it for real. And then you can kind of sort of use it to truncate depth, as you said. And then the breadth is also determined. And what's kind of critical is that the action selection algorithm, where you iteratively visit and grow the tree, is well suited for the size of problem that Go is and the depth of the problem. But for something like LLM reasoning, Pucked might actually not be a good enough heuristic. It might be too greedy with local tokens, and it might do something like, oh, only give you sort of obvious thoughts that are correct, but not really solve your final problem. So I would say the jury is probably still out on what the final instantiation of reasoning for LLMs would look like. And I wouldn't rule out that this stuff could come back. but it's been hard don't lm sort of night and natively learn to do mcts where they'll try an approach and be like oh that doesn't work let's back up let's try this other thing and then go in the direction that proves to be more fruitful uh yes certainly i think the lm's managed to do something that looks like real human reasoning without having to do an explicit tree structure yeah um that being said i think the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback even though not exactly in the same instantiation as Alfga. But just to make sure I understand the crux of it, like the breadth from the number of legal actions being wider and the depth from not being able to train a value function as easily because... Because. So here's an example where LMs break down. The CPAC rule involves, you know, square root of n over 1 plus nA. In an LLM, like, you're most likely never going to sample the same child more than once, right? So if you have, let's say, multi-steps of thinking, because language is so broad and open-ended, it's a sort of discrete set of actions is not really an appropriate choice for an LLM, even though they're discrete tokens. it's just such a large number that this type of exploration heuristic is probably not the right thing to do to guide how to search down a tree right but i guess the crux comes down to the fact that in go you know that the mcts is almost certainly better than your current policy even though you haven't gotten even though you haven't explored the end of any trajectory correct and then in uh in normal reasoning for lms robotics
[110:00] So there's no way to just locally evaluate and improve your next move in a way that doesn't result in, in a way that's independent of actually like solving the problem. No way is a strong word. I think lots of people have thought about how to try to apply MCTS or its kind of successors like New Zero to continuous control spaces. And I'm sure, you know, very cool research work is still ongoing to try to crack that problem. But yes, the seeming challenge right now is that most problems in much higher dimensional action spaces or something that's combinatorially much bigger like language, they don't seem as amenable to the kind of discrete action selection heuristics as well as kind of game evaluation type stuff that Go does. but that's not to say the idea of like you know thinking into the future along multiple parallel tracks might not give you some information about like which way to search right like if you think about mathematics i think mathematics often occupies a little bit more of like a logical search kind of procedure where you kind of can back up you can kind of see like which paths seem good or not there's more of a rigid structure there whereas maybe like in a uh you know business negotiation or something um it less of a tree and maybe you know something a bit different Okay so we now seeing it so I can ask you some more questions about Afligo and about AI research more generally In 2021 Andy Jones had a paper called Scaling Scaling Loss for Board Games, and he basically anticipated inference compute or inference scaling by showing that you can trade off test time compute and training compute. That is to say that you can spend more compute on the for the searching through the mcts um and if you do that you can get the equivalent performance as having spent more time training the model um and so if you you know you if you see this pattern you might think okay well with llms you might do something like that in the future and in fact that's what ended up happening okay so what is a kind of fun exploration one could do now to explore other axes of scaling in toy settings which will be important to understanding what AI development might be like in a few years. Sure, yeah. I think that, indeed, test time scaling and reasoning and how it interacts with model size are quite profound when it comes to how much needs to be actually done as explicit search versus how much can be packed into the forward pass of a neural network And how does a forward pass of a neural network sort of learn how to do something that should be a sort of sequential and recursive step That quite interesting So the Andy Jones scaling laws for board games paper is quite cool There's another really nice result from that paper where he showed that not only can you predict scaling laws of like, you know, the sort of LLM variety where as you increase parameters, you can decrease the amount of compute for search or vice versa. he also showed that you can actually predict how much compute is needed to solve a larger version of the board game for example and then so with go you know which can scale from you know uh three by three to infinitely sized you know go board you might actually be able to sort of revisit this question and try to reproduce uh whether this shows up you know i actually started this project with this sort of a motivation that like does the bitter lesson or does our knowledge of scaling laws allow us to kind of execute a lot better on a sort of compute optimal go bot and can we can we kind of build a strong go bot without all the katago tricks right just just by really focusing on the bitter lesson scaling laws um i have not been successful so far but i think it's it's sort of a fact that like usually when you want scaling laws to work you want to be in the regime where um the the recipe already works and the data sets are good rather than trying to kind of figure out how to do scaling while also trying to figure out what the right data set are So this is like the scientific understanding component in research often follows a step where you get something to work first and then you use that system to collect data that then helps you build a mental model of how things work, such as scaling laws, right? And so usually, actually, if you want to build a strong gobot using scaling laws, you actually have to make a strong gobot first and then use the scaling laws to kind of extrapolate a bit farther into the future. Say more, just so I understand, first of all, you're saying scaling laws did not work or you could not, there was no scaling laws pattern that you could see in your robot? Yeah, so a mistake I made initially when I had some bugs around how MCTS labeling was working was I would collect a bunch of data with an expert policy and then treat it as a supervised learning problem and try to identify scaling laws with expert data sets. you can indeed plot things that look kind of like this but if you're in a regime where you know your policy is not working well you might be just studying scaling laws on like bad data right so so just like one important implementation details that if you want to study a scaling laws problem you kind of have to have a problem for which like the data is good the architecture is good and there's no bugs and then like you you solve it there um x ante i wasn't able to like apply scaling laws to direct what to look on look look at um
[115:00] until, you know, I had the rest of the system working. And this sounds obvious. Like, to researchers, of course, you want to have, like, a working bug-free system before you study scaling. But just as a sort of advice for practitioners on, like, where I actually tripped up when I started this project was you don't necessarily want to kind of jump into the science of studying your man-made artifact before your man-made artifact is, like, interesting enough to be studied. Speaking of compute, so if you can look at these charts of compute used to train the best AI model in the world over time, going back 10 years. And it's a very smooth line in log space that is exponentially growing year over year. Except there's this huge aberration, and that aberration is off-label zero, which is trained on way more compute than any other AI model at the time. It was like three E23 flops. It's sort of comparable to a Frontier LLM. I mean, orders of magnitude off. And so, yeah, the question is, especially with you being able to get something off and did you turn it on your own i got a donation from prime intellect okay for like about 10k and then i spent um i spent maybe the first 4k doing um kind of exploratory research and then uh about 3k on the kind of final run yeah um and then uh some some of it remaining for serving the model for cool yeah is your sense that they were just did a bad job training it if you can do it in 10k now uh the compute required to be the first to do something is always like much larger than the compute it takes to catch up and it's the same story playing out in lms right like once someone else has done it um you could use tricks like distillation you could use um uh all sorts of crutches to bootstrap your way to success. So with my own bot that I've hosted online, I actually used best response training against the Katago models to get a strong level performance. And as a time of recording, I'm validating whether I can do that first step, which is to do the tabula rasa play. But importantly for research, you often want to start from a good init. So the simple thing I did first was train best response agents against Katago. AlphaZero team, they did not have any policy that they could train against, right? Because they were trying to do everything tabula rasa. And being the first to do it means that you're prioritizing getting the thing working rather than, let's say, the most compute efficient possible implementation. So this actually plays out in robotics as well. If you look at the frontier of large models trained for robotics, the scatterplot is all over the place and there isn a very clean line the way that there is for frontier LLMs And that is because the folks training these models often are not you know at the scale where every flop counts and they need to kind of squeeze out the performance of every single flop as the dominating decision deciding factor in pre-training, right? Instead, their focus is more like we want a certain capability to show up. So we optimize the training setup to kind of make it easy to derive that capability. and once you have that capability well um invariably if you scale up the compute you are forced to kind of make it compute efficient because this is like hundreds of millions of dollars we're talking about but um but in the past when when compute for experiments was kind of more plentiful or you know not not uh not um accounted in a way that the researcher was really responsible for then you kind of end up with people optimizing for things besides kind of being on the compute optimal pre-do frontier i see like speed or something yeah like time to result or just getting it to work. I think the first AlphaGo, like, probably they had lots of compute and they didn't need to be, they didn't need to worry too much about making it the most compute optimal thing. And how much of the improvements to compute efficiency are methods that did not exist as of 2017 versus things which they could have done in 2017? Yeah great question So going into this project I kind of knew in the back of my mind that like things always get easier to do over time And I want to see like where is Goat given that it didn seem like there has been any major open source strong bot after Katago in 2020 And then reading the Katago paper, there's a lot of clever ideas. I was kind of wondering, okay, let's see if the bitter lesson has happened where a lot of these tricks just sort of go away because NVIDIA made faster GPUs. And so roughly, where are we on that? So again, this is not a peer-reviewed claim. So this is just my preliminary vibe guess on what I've seen based on my own experiments. But it seems like architecture choices don't matter that much. Transformer versus ResNet. We're at the sort of speed of GPU where the size of the model is not so big that this really matters. You can actually simplify the setup quite a lot. So instead of doing a distributed asynchronous RL setup with replay buffers and pushers and collectors, you can kind of do a dumb synchronous thing where you'd like collect, you just train a supervised learning model, and then you collect again. And so there's like opportunities to simplify infrastructure. NVIDIA GPUs have indeed got faster. So whereas Kodago was trained on V100s, you can train on like half the number of, you know, desktop Blackwell GPUs, and it still works. And some of the kind of auxiliary supervision objectives
[120:00] that KataGo developed aren't really necessary if you have a strong initialization. So if you're initializing against best response training against KataGo itself, then your own model actually needs none of the tricks that KataGo needs. So then the core thing is like, how can you get as quickly as possible to some strong opponents? And that matters a lot more than the specific architectural innovations. But there are still some nice compute multipliers. So I found that training on 9x9 boards was very nice for resolving endgame value functions. And then if you can co-train that on a architecture that can transfer between 9x9 and 19x19, then you can really cut down the warm start time to learn that from scratch. I think AlphaGo Zero, their plot was first 30 hours or so are spent basically catching up to the supervised learning baseline. And you can cut down that time a lot by kind of pre-training on a small board and then warm starting that into your 19x19 board play. There were some other stuff like varying the number of sims between episodes. this turns out to be not that sensitive actually like you can kind of you know fix it or increase it doesn't matter too much um but so anyway it's kind of just nice from a scientific perspective just revisiting like an old paper and seeing like what really matters this is sort of a question but why is it okay to have a buffer and off the go every time I talk to an AI researcher they telling me about how bad it is to be off policy But then the way a naive implementation of Off Zero would work is that most of the moves in a given backward step or in a batch of backward steps would be not among the ones that were made by the most recently trained model So why is that okay? Great question. Yeah. And this gets into the sort of fundamental off-policy versus on-policy reinforcement learning kind of questions. So as you recall, in MCTS, you take actions that you took and you relabel them to take different actions on the same states, right? So the off-policy part here comes where what if you're relabeling states that your new policy would never visit? Like, what's the point? You're kind of wasting capacity. and in the extreme limit, imagine your distribution of states in your training buffer are all states that you would never visit. Then you're basically supervising them to take good actions on states you would never achieve and therefore your policy can get really bad. So this is where off policy can really hurt AlphaGo. However if you interpret this sort of from like the dagger perspective which is basically saying like a way to kind of correct yourself back to the optimal trajectory given some some data What you kind of want in an algorithm like this is to have mostly states that you would visit But then you have a small percentage or maybe a reasonable percentage of states in this kind of high dimensional tube around your optimal trajectories. And any of those states are given a supervision target to kind of sort of funnel you back into your optimal trajectory. So maybe I can just draw it out quickly here. So in sort of a Dagger-style setup, what your kind of optimal training data distribution is, is that here is your optimal states and actions. So this is like, you know, you want to be in this state, you want to be in this state, you want to be in this state, and then you win here. And then these are your optimal policy actions. So these are the things that you definitely want to train on. But to make it robust to disturbances, you want to make sure that if you happen to drift off into some other states, you can kind of funnel yourself back into. But why isn't this a fully general argument for policy training? This is actually why you want to do off policy training sometimes. Is that like you don want to have a compounding error where if you make a mistake you don have the data of how to return back to your optimal distribution And so optimal control does not really say too much about how to not accidentally get here because it's sort of making the assumption that once you learn the policy, you're going to get here. But in applications like robotics, I don't know, a gust of wind blows you slightly off, and then now you need to correct. Or the friction on one of your tires is kind of a little bit lower than the other wheel, and then now your car's drifting and you've got to correct it. So these kind of things in more real environments often happen. Actually, there's a funny quote about chess and also Go. It's like, the problem with Go and chess is that the other player is always trying to do some shit. So things can kind of drift off. And you always want to be able to correct back to your winning condition. So your replay buffer really should have the states that your policy would visit, plus some distribution of states that you might drift to, and then how to return back to your optimal states. Now, if you take this to the extreme and you say, well, we don't have any of this data, and we're gonna just be labeling with MCTS states that are so far away
[125:00] away from our optimal behavior, like this bag of states over here, well, now, yeah, each of them gets an MCTS label, and your policy learns how to take the best possible action here, but you never get here. So you're training your model on states you would never reach. This is not there. So then this is a problem. And this is where off-policy can really hurt. So actually, as part of this project, I did try an experiment where I took a bunch of trajectories and to try to saturate the GPU as much as possible, what I did was I took random states from the data set and re-ran MCTS on just those states. So instead of playing a whole game where I'm doing MCTS on every move, I just ignore the sort of causality of moves and just pick random board states and I just label those with my current network. And I might revisit old states that I've labeled before and re-label them again with my current network. And so in practice, this actually does work. You can actually say, like, let's take some states that are reasonable and constantly be relabeling them while we're training. And so this actually starts to converge on a very robotics setup which is very common which is you have your data set of trajectories and then you have something like a replay buffer pusher And these are off-policy, offline trajectories, right? So your replay buffer pusher pushes transition tuples to the replay buffer, and then you have some job that's kind of continuously replanning what the best action you should have done instead of taking this action is. And so in robotics, it's actually very common to use the sort of minimized TD error. So like your Bellman updater constantly is pulling things from here and trying to satisfy the QSA. So, and then from here you have your trainer, which is trying to fit the S to A, or fit the Q to the Q target. So here you can think about this as a sort of planner right You revisit old states that you been to and you take your current model and you rethink like what could I have done better if I visited this And so this is actually how, like, kind of off-policy robotic learning systems are usually trained. These days, there's a sort of simpler recipe, but, like, you know, in the Google QTOp days, we kind of did things like this. So what is the trainer? Oh, yeah. The trainer is you try to minimize QSA and QTarget. Wait, can you explain the whole setup again? Like at a high level? Yeah. So you have your off-policy data that came from various policies. You're constantly pushing transitions that you saw before to a replay buffer. And then you've got this thing called a Bellman updater, which basically replans. Instead of this action, what action should I have taken at S to have a better value? And the way you enforce that is you try to minimize the TD error. So actually, given this, you have S prime, right? You compute Q of S prime, and you find the action that should go with S prime that makes this Q value as high as possible. And then you add that to the reward here, and that gives you your actual target, right? So for this current S and A your Q target is this So now you send back the Q target to this transition So with this tuple you pair with that a Q target And then here on the trainer, you simply just use supervised learning and you minimize your current network's QSA with its target. Got it, okay. So in the background, you're just like, hey, let me basically think through how valuable were all these actions actually? Yeah. In a more optimal policy where you're trying to maximize this, what is the key target of this transition? It's sort of like basically daydreaming. Exactly, yeah. You can think about it's like you're kind of going back in hindsight and being like, given what I've seen in historical buffer, was there a better action I could have taken? Yeah. Now, the connection to Go here that I tried, and it was moderately successful, but too complex to kind of open source, was you replace this with a MCTS relabeler where instead of doing this kind of target network computation, you run MCTS on your transition. So in this case, you have your state, your action, and then whether you want or not at the game. And actually, you can just toss these two. You don't care about these ones. You just take your state, and you just plan MCTS to get your best transition.
[130:00] policy, you know, pi on your current network, right? Not the network that took this action, but your current best policy network. You just rerun your search offline on these transitions. And if these are transitions that your policy can get to, then this actually acts as a very nice stabilizing effect. And also one other benefit is that you can, like, kind of fully saturate your GPU better because you're not, like, blocking on the go game to kind of like give you board states. You just simply search across all board states at any depth in peril. So, and then here the trainer would be just, you know, predict the MCTS label as possible. So again, like this kind of works and this is quite relevant in robotics where you're really, you just have, I'll have a lot of offline data and you can't simulate things like MCTS. But in practice, like it does run into the problem where, you know, like if the current model is looking at states that it would never reach, then it's kind of wasting capacity. And so you have to be a little bit careful here. So the on policy thing, and also much of RL has kind of converged to a much more on policy setup, where they don't really try to directly train on off policy data. At best they use off policy data as a way to reduce variance but not directly influence the objective I sorry why have they converted to that It just more stable So you might use the off policy data as a way to reduce variance but not directly influence the objective I sorry why have they converted to that It just more stable Okay Yeah So like you might use the off Q as a way to do like you know advantage computation Like, you know, Q minus, like, sum of Q. That's kind of like your, or sorry, like, you know, sum of, like, if there's N actions and then, yeah. So this is your value, and then this is your current Q value. So your advantage for that action is the average value minus your current one. So people can try to estimate Q in an off-policy way and then just use advantage here. And then if there's a problem in these dynamics, it doesn't blow up your loss as much. And so in robotics, there's a kind of convergence towards more like using off-policy data to just shape your rewards, but not actually be directly here. I'm reminded now of our earlier conversation of why MCTS is so favorable as compared to the kind of, you know, reinforce a policy gradient kind of thing LLMs do. And this might be totally wrong, but I wrote a blog post a few months ago about how RL, at least policy gradient RL, is even more inefficient than you might think. And so the inefficiency one thinks about naively is the fact that you have to roll out a whole trajectory in order to get any learning signal at all And so as these trajectories become longer and longer an agent has to instead of just previously complete the next word in the sentence it has to go instead to, hey, do two days' worth of work to figure out if you even did this project correctly. The amount of information per flop has been decreasing. As you had to unroll two days' worth of thinking in order to see if you even did something correctly, to like implement this feature, the amount of samples per flop has been decreasing. But so you can think of, you're trying to maximize as you're learning bits per flop, right? And this is, you can think of bits per flop as samples per flop times bits per sample. And what I just mentioned a second ago is that the samples per flop go down as RL becomes more and more along the horizon. But at least this kind of naive RL is also terrible from a bits per sample perspective. And here what I mean at least compared to supervised learning So early on in training let say you have a vocabulary size for an LLM that is 100K long So there's 100K possible tokens that one could answer. And you have a totally untrained model. And you have a prompt like, the sky is. With supervised learning, what would happen is that the model would have some probability distribution over all the things it could say. there's a label that says actually the term here is blue. And it would learn basically for cross-entry-pilots exactly how far its distribution is from correctly saying blue. Now, if you were doing this through RL, you would say the model would try the sky is halicon. Nope, that's wrong. The sky is told. Nope, that's wrong. This is a totally untrained model, right? And so you would have to do this on the order of 100,000 times in order to just stumble on blue. then get some learning signal off of that. So if you're in the supervised learning regime and you have your distribution of probabilities, you get told that it's blue and you figure out how far off you are, the amount you learn is a function of your pass rate. So the further away you are from blue, the more you've learned to go towards blue.
[135:00] using cross entropy loss. And so you can think of it as like your pass rate, your prior probability of having said blue. And as a function of that, like in supervised learning through cross entropy loss, you would learn negative log p, p being pass rate bits once you get this label. Whereas in RL, if you're just randomly guessing shit and seeing if it works or not, that's just basically going to be the entropy of a binary random variable, which is... And what's also tough here is that actually the distribution that you're sampling under is your policy's distribution. So it's like if your policy has no chance of sampling blue, then you will never get a signal. Exactly, right, right. So that's being modeled by the fact that your probability of sampling blue is extremely low. if you do sample it, you do learn as much as you would have learned in a supervised learning. In all other cases, like, you know, 99.99% in an untrained model, you're just learning incredibly little from, like, seeing halicon is not the correct word or told is not the correct word And that what happens most of the time So you just like learn very little So if you try to graph if you put on the x your pass rate and here you put the bits you're learning from a sample. If you have like 0% here, 50% here, and 100% here. So the end of trading, you're here. If you have supervised learning, negative log pass rate would look something like this. And then the binary random variable would look like this. And this is, depending on whether you're doing NATS or BITS. If you do BITS, it's like one, right? Here at the peak. This is like a coin flip. You learn the most from a coin flip. this is supervised learning this is RL however the problem is you spend most of training in this regime like in the low pass rate regime and in fact of how fast you learning is a function of how many bits per sample you getting And you getting very little signal here If you chart the pass rate on a log scale so you put the x on a log scale where like at the beginning of training with a vocab size of 100k, the pass rate is 1 over 100,000, then 1 over 10,000 1 over 1,000 1 over 100 and then okay what this graph looks like here where supervised learning would look like this and and then RL if you just basically crunch what I just showed there it would look like that and arguably you spend all your time here Potentially never even getting a single success, right? Exactly. So it's a sort of depressing plot in the sense that once you're here, it's not at all obvious how you get to here. Yeah. Once you're here, you have something, but you actually, in many RL problems, spend all the time here. Yeah. So there's a sort of question of how do you initialize so you're at least not at zero but at a non-zero pass rate. Yeah. One more thing I'd like to add about bits per sample that very relevant to any kind of machine learning problem is that there a connection to soft targets and distillation where if you have access to the logits right not just the one hot, like this is a sort of one hot token answer. If you have access to the soft targets, the entropy of this distribution is far, far higher than the one hot. So there's actually way more information and bits per sample in a soft label. So that's why distillation is so effective per sample, is that it's actually giving you way more information per sample. Yeah, well, I wonder what the equation would be. But obviously... It would just be the entropy of this distribution. So the entropy of this is zero. The entropy of this is like, you know, the entropy equation. And this is also why, like, you know, AlphaGo is quite beautiful. In AlphaGo, you don't train the policy network to imitate the MCTS action. You train it to imitate the MCTS distribution. Interesting. But both of these are actually valid. And if you wanted to do a scientific experiment of like, how important are this kind of soft label, dark knowledge distillation, you can run an experiment where you retrain the policy network on the action MCTS selected rather than the software. Interesting. Earlier I was sort of stumbling around. Intuitively, why is this ability to do...
[140:00] iterative search where you don't necessarily need to be able to win the game in the beginning. You just need to be able to improve your current policy. Why is that so powerful a capability in learning as compared to how LLNs currently learn RL? And yeah, it's exactly this thing of, this is considering your pass rate of the entire trajectory. I actually don't know a formal way to think about this. Maybe you should help me out here why is alpha go an elegant rl algorithm like so um uh the major reason is that you never have to initialize at a zero percent success rate and solve the exploration problem of how to get a non-zero success rate and this is what allows you to hill climb this beautiful supervised learning signal where and if you look at the actual implementation of alpha go every step of the way there's no uh there's actually no um you know td error learning or dynamic programming, at least explicitly. It's just supervised learning on a value classification as well as a policy, you know, KL minimization. So it's just a supervised learning problem on improved labels And so the training is very stable right You can train like as big of a network as you want You can kind of retrain this on the data set Everything will just go stably The infrastructure is very simple to implement as well You don need a complex distributed system to kind of keep everything on policy At the end of the day you just saying like I have some improved labels. Let's retrain my supervised model on these targets. And so you're always in this beautiful regime where you're just trying to improve the policy rather than escape this kind of like sort of local minima where every signal is flat all around you. So one way to draw the curve is like, if you draw the sort of win rate of an MCTS policy versus the raw network, let's say this dotted line is the raw network, the MCTS policy kind of looks like this. And so every step of the way, this supervision signal is very clean. You're never in a situation where the MCTS is kind of like giving you no signal, unless your MCTS distribution converges to exactly what your policy network predicts. Yeah, yeah, yeah. Okay, that's a great way to explain it. Cool. Okay, maybe we sit down and I ask some questions about automated research. Sounds good. One thing I really wanted to talk to you about is that you did a bunch of the research for this project through this kind of automated LLM coding assistant loop And there an idea that if you fully automated AI research you could have some sort of singularity. Obviously, we're not there yet, but to the extent that we have early indications of what this process might look like, I am curious what your observations about what the AI is good at, what it's not good at, what you think about this scenario, this likelihood eventually, what thoughts you have about this in general. For sure, yeah. I think automated scientific research is one of the most exciting skills that, you know, the frontier labs are developing right now. And I think it's important for everyone who's doing any kind of research to get a good intuition of like what it can do now and what it can't and how might the sort of science process work in the future once we're having AIs automating a lot of this investigation. So in brief, I mostly use Opus 4.6 and 4.7 throughout working on this. And what works is that the models can do a very good job of doing hyperparameter optimization. So in the past, people would kind of come up with a search base of hyperparameters, like learning rate and weight decay and maybe how many layers are in your network. And they would just kind of do a grid search or a sort of Bayesian hyperparameter optimization approach And then it would find some tuned parameters The kind of really cool thing that automated coding can do now is that it can search a much more open set of problems right It can say like, well, I've identified that the gradients are kind of small in this layer, so let me change it up here. Let me rewrite the code so the data loader has a new augmentation I came up with. Let's sort of try to find the best way to kind of fit the constraints of the optimization problem. And you end up with this much more flexible and kind of high-level, almost like grad student-like ability to just, you know, grind a performance metric. And so this can squeeze out quite a lot of performance. You can, you know, on a fixed data set with a fixed time budget, improve perplexity by quite a lot on a sort of classification problem like LLMs or Go. And it is also fantastic now at basically executing any experiment, right? So I have a a clawed skill that I wrote called experiment, where I give it a description of what I wanted to plot. And like I just described, here's the x axis I want, here's the y axis, answer this question for me. And it'll go run off and do all the experiments, compile the plot, make a report and suggest like, you know, what might have caused it or so forth. So that's what works quite well today. And I think we can expect that these abilities get better in the future. But it's also kind of useful to know, you know, what is it not doing so well today? So on
[145:00] My blog version of this tutorial, I have a plot of basically all the kind of experiments I did grouped in a sort of tree where, you know, every node kind of represents a failed successful or sort of mixed experimental result. And then from there, it branches off into a child where it's like the follow on experiment. Occasionally, I'll kind of rabbit hole down a track like this off policy MCTS relabeling, do a few experiments and then realize it's probably not worth it. So then I'll kind of jump to a completely different track. And I call these kind of things like rows. So what I find is that current closed models that we can access, the public can access today, they don't seem to be that great at selecting what the next experiment should be in a given track. And they don't seem to be able to kind of step back and do the lateral thinking of like, wait a minute, this track doesn't really make sense. Like, let's go back to sort of first principles and think about what the bottleneck might be or like, what are we trying to achieve? And so often I had to catch infrabugs myself by prompting the right question to Claude to investigate what is causing this discrepancy, and then it'll answer the question. I think with mythos class models or mythos models coming online maybe this just completely changes and these problems just fall to just improve skilling But at the same time I think there a lot of rich opportunity to develop RL environments that might incentivize this kind of lateral thinking And so one of the motivations for setting up this Go environment was that I think that Go captures a lot of very interesting research problems, often overlapping with LLMs or robotics. And yet it's very quick to verify. The outer loop is ultimately like, does the agent do what I think it does? And you can kind of check the outcome of a Go game quite easily. and then the inner loop involves all this kind of like you know research engineering around distributed systems predicting whether an idea is going to work or not predicting the you know the difference a particular modification to your training algorithm might make and i think there's a rich library of subtasks and sub environments that you can kind of train an automated scientist to work on with go as a sort of outer verification loop that then once you acquire these skills maybe you can apply them to like other domains like you know biosciences or robotics or automating AI research. Or automating AI research. Which is the real crux or the scary slash incredible thing of just making AIs making future versions of AIs. And you're suggesting the outer loop here could just be your win rate against KataGo, basically? That one of them I think there a lot of deeper questions that one could tackle right So for example let say you have an idea on how to improve a scaling law compute multiplier The outcome isn't necessarily like I achieved the best GoBot ever. The outcome might just be like, can I predict what the win rate of my GoBot will be? Or can I predict the scaling law plots that emerge from my idea? But then you can verify that you haven't kind of reward hacked anything by using a very verifiable game like Go on the outer loop. I think there's a couple of interesting follow-on questions. There's questions on the inner loop and the outer loop. On the inner loop, there's a question of how locally verifiable any modification you might make is. That is to say, would you know whether something is actually improvement or degradation, some idea you try out? Would you know that if something isn't working as a result of a bug or is it the result of the idea itself being wrong? um ilia was talking about why having one of the reasons he thinks he's a good researcher is he is a good researcher one of the things he thinks makes him a good researcher is that um he has intuition about he has strong belief in what the correct idea is and he is able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea based on his high level belief about this idea should work so therefore it there has to be bug versus the other way around why don we start with that question actually yeah how locally verifiable are things which are good ideas yeah i think as in the case of the success story for deep learning you can think about this as like a decades-long idea that took like took a lot of faith to get it to work. And so this presents a very challenging long horizon, you know, RL problem where, you know, every step of the way you have like a committee telling you that this is a bad idea and then ultimately you break it through, right? And so like, how do you design RL environments that maybe give you some feedback earlier? And I think this is a very tough open question that I don't have an answer to. But, you know, ultimately to play a very strong GoBot, you probably did need to discover deep learning. Yeah. Right. And so I think that having a challenging game that cannot be cheated easily on the outer loop could be used as a sort of outer loop signal for something like discovering the principles of deep learning. Now, of course, to make it tractable, and this is where research tastes really matters, you have to come up with ways to initialize your problem so that you don't solve a very
[150:00] practical problem, right? Like, maybe you can leverage LLMs as a sort of a universal grammar in the middle to kind of give you some sort of local feedback. The fact that LLMs are universal grammar means that they can kind of move at almost any level of the stack, right? They can think very locally, as well as step back and think, like, in very broad steps. And I think that's where a lot of the lateral thinking ability of humans kind of come from. Like, how to know if the track that you're pursuing or the objective that you're pursuing is not right, and you should be asking a different question. The other question is how stackable local improvements are in the attempt to get to a better result on the outer loop. I've heard rumors that at some AI labs, the thing that has gone wrong is that people will individually pursue good ideas, but those don't end up stacking well. And so the training run falls because of some weird interaction between two seemingly good ideas and having a single top-down vision of how things should work is very important. having worked at different AI labs and also playing around with, I guess, parallel agents, trying different ideas. What is your sense of how parallelizable AI innovation is? Yeah great question I think the research taste for executing well on you know the bitter lesson is that you need to know how much the bitter lesson can buy you and how much is too much to ask for at any given moment Of course, in the fullness of time, compute is the single most important determinant on how things work. And it's almost inevitable that as you scale up energy and compute and parameters, intelligence will just fall out of that. And that's super beautiful, super profound. No algorithmic detail really matters beyond that. But in present day, we don't have infinite compute and parameters and arbitrarily good initialization. So we have to come up with heuristics that kind of give us that. But these heuristics are probably somewhat redundant. So that's probably why you see this effect where a lot of these compute multipliers don't necessarily stack. They might have some correlated benefit. and then you know three years down the line when the nvidia gpus have gotten even stronger maybe maybe they stack even less well right like maybe like at any given point in time the the sort of benefit of any given compute multiplier is transitory which is what i sort of suspected with the katago paper like there was many algorithmic ideas kind of applied and then you can see that like with you know modern blackwell gpus and ada class gpus that are much better than the sort of v100 grade gpus GPUs that paper used You can see that like some of these algorithmic tricks to speed up convergence just don matter so much compared to something else And I think that's a matter of taste in the present time. Yeah, interesting. How about the outer loop? How verifiable for making AI smarter? with Go you do have this outer loop of win rate against the best open source model out there and even there as you were saying there are other outer loops of did you discover a new phenomenon which is actually very hard to if you didn't know scaling laws were important if you're back in when was Chinchilla or Kaplan scaling laws released like 2019 so if you're back in 2015 would you there's not an automated procedure one can easily imagine of knowing which paper is the scaling loss paper versus which is just like another random plot and so that even in the go case is hard to verify outer loop and the whole the whole idea of an outer loop is to have like some backstop on um on improvement uh but let alone for general agi where of course we have a bunch of these benchmarks but there a problem that like we know the things we can measure and we improve on the things we can measure but we care about this broader ability to do economically useful work which is at least until you automate everything not super easy to measure So, yeah, there's a question of, okay, how good is the outer verification loop for AI self-improvement, and does that matter? Yeah. I'm going to give a non-rigorous argument, but one that I kind of intuitively believe, which is that, you know, DeepMind, the AI research lab, they started as a sort of focus on games, right? Like they kind of used games as their outer loop. And then the researchers learned from experience of solving games. And then like now they're working on LMs. And presumably there was some positive transfer from their time working on games and like Atari and Go and StarCraft that like now helps them make good LMs. I assume that there's positive transfer in some regard, whether it's coding or general research ability or project management. All these things probably help them do well. And so if that's the case, why wouldn't it also be true for automated AI researchers? They should be able to positively transfer experience, tackling quick to verify, quick to iterate on environments to something more ambitious and economically useful like automating drug discovery
[155:00] I mean, I don't know if it isn't the issue with historically until Gemini 3 or whatever. Been a couple of years ago, people were saying, look, Google hasn't isn't catching up in LLMs because they're too tight to the old approach. And yeah, there's gains, but there's also there's ways in it which actively hinders you. So it's actually not obvious to me that there's like the jury's still out. I think, like, who knows if the, you know, let's say currently Google's doing quite well. Who knows if the initialization on training on games is ultimately going to hobble their ability to be the winner in the long term. Like, it's hard to say for sure. And, you know, likewise, who knows if the seeming late start was really just them kind of pre-training for longer on how to, like, scale up TPUs. They invested all their tech tree in getting TPUs to be good, which seemed not that useful in the short term, but then in the long term it becomes maybe like a... So it's even hard for humans to reason about what the optimal research strategy should be, even with the data we have today. Yeah, yeah. Cool. Okay we should let people know how they can find out more about this project whether to fork it themselves whether to check out your blog post where you doing an excellent job explaining many of these ideas Where do people go next? Great, yeah, so my website is evjeng.com. There's a blog post that kind of links to an interactive version of this tutorial. And on my GitHub, which is the username is just Eric Chang, there's an auto-go repo that people can fork and reproduce the training. And I also highly recommend people check out this blog post, As Rocks May Think, which we touched on some of the ideas in this conversation, but it's this grander, you know, thesis of what happens when you have thinking as a primitive in computer science. Exactly. Right. And so I highly recommend people check out that block list as well. Yeah. And I encourage the audience to, you know, think about the relationship between thinking and go, you know, via MCTS and search and how it relates to LMs. I think there's something quite like profound there and probably underexplored just because Go has been relatively underexplored compared to the boom in LLMs. It's not to say that I think we should have trees in our LLMs, but there is some very interesting duality between them, and you can actually do a lot of research on Go, MCTS, and reasoning with very small budgets. So that's very exciting. Cool. Awesome, Eric. Thanks for doing this. It's an honor to be on the podcast.
