Dwarkesh Patel Says AI Post-Training Moves to On-the-Job Distillation

Dwarkesh Patel称AI后训练正转向“在岗蒸馏” AI实验室将通过同策略自蒸馏，从静态预训练过渡到部署期间的动态权重更新，以此攻克复杂的现实世界任务。 Dwarkesh Patel深入探讨了主流AI实验室重注基于可验证奖励的强化学习（RLVR）的原因，并分析了将编程等可反复试错的容器化环境与复杂的现实世界任务区分开来的关键瓶颈。他指出了瞬时推理在结构上的低效，认为模型的长期演进必须超越单纯扩展上下文窗口，直接将实际操作中的经验固化到权重中。通过同策略自蒸馏（OPSD）和推测性“做梦（dreaming）”等机制，未来模型将能从稀疏的现实数据中提取高密度信号。到2027年，智能将在全球部署中实现持续复利增长，而不再单纯依赖静态的预训练更新。 顶级 AI 实验室正重注可验证奖励强化学习（RLVR），期望在数以千计的可验证环境中催生通用智能，并将训练初期极低的样本效率视为可被规模化摊薄的一次性成本。 (0:00) 计算机使用（Computer Use）明显滞后于代码与数学，核心瓶颈在于缺乏支持高并发、确定性及可重放的“可刷”（Grindable）模拟环境，且真实网络的反爬机制严重限制了并行训练。 (2:12) 目前实验室高达 30% 至 50% 的算力被用于推理，但这些交互经验完全无法沉淀到底层权重中；单纯依赖扩大 KV 缓存不仅缺乏参数效率，也无法实现生物级的记忆抽象整合。 (8:41) 展望 2027 年，借助策略内自蒸馏（OPSD）与内部模拟“做梦”（Dreaming），AI 有望实现真正的“在岗学习”，将产品演进的重心从出厂前的静态训练转向全行业部署带来的动态网络效应。 (17:23)

Dwarkesh Patel argues that artificial intelligence labs will exhaust the benefits of static training paradigms and must transition toward models that continually consolidate real-world experience directly into their weights. Grindability and the Limits of Verifiable Simulators (2:12) Reinforcement Learning from Verifiable Rewards (RLVR) currently accelerates domains like software engineering and mathematics because they are both verifiable and "grindable"—allowing thousands of deterministic, parallel rollouts inside containerized environments. By contrast, real-world tasks like computer use lag because live platforms trigger bot countermeasures, and offline clones of applications like Slack or Gmail are labor-intensive to build. Without isolated, farmable simulators, sample-inefficient reinforcement learning algorithms struggle to make meaningful progress. More critically, complex human endeavors—such as running a political campaign, launching a company, or litigating a lawsuit—take place in non-stationary, reset-free environments where feedback loops unfold over months or years. Patel expresses skepticism that training strictly inside synthetic, containerized environments will automatically generalize into the strategic mastery demonstrated by historical figures like Lyndon B. Johnson or Elon Musk. Citing Anthropic CEO Dario Amodei's observations on performance degradation between training and serving context lengths, Patel cautions against assuming that synthetic verifiers alone can resolve real-world ambiguity. The Inefficiency of Ephemeral Inference (8:41) Patel identifies a major structural inefficiency in modern AI architectures: while 30% to 50% of laboratory compute budgets are expended on inference, none of that compute updates the underlying model weights. Consequently, deployed agents repeatedly encounter enterprise workflows and operational errors without retaining any lasting institutional memory. Relying permanently on expanding Key-Value (KV) caches to emulate competence through extended context is both parameter-inefficient and biologically unnatural. Patel notes that the human brain relies on memory consolidation and abstraction rather than verbatim storage; individuals with hyper-photographic recall frequently struggle with high-level conceptual reasoning. Because gradient updates remain notoriously sample-inefficient—evidenced by Cursor's tab-completion model requiring over 400 million records to learn basic patterns—true on-the-job adaptation requires distilling session insights directly into long-term weights. On-Policy Self-Distillation and Simulated Dreaming (10:00) To update weights from limited data without causing catastrophic forgetting, Patel points to On-Policy Self-Distillation (OPSD). Under OPSD, a context-rich instance of the model acting as an experienced teacher guides a base student model through dense, token-by-token probability matching. Unlike Supervised Fine-Tuning, which forces naive memorization of entire transcripts, or RLVR, which compresses complex trajectories into a single scalar score, OPSD applies targeted updates strictly to critical decision nodes without requiring an external verifier. Beyond OPSD, Patel explores the speculative frontier of "dreaming" or test-time training, where models spend compute constructing internal simulations to practice skills offline. He draws an analogy to DeepMind's EfficientZero, which matched human performance on Atari games within two hours of real-world play by running dozens of simulated games internally for every physical step taken. If language models can generate and rehearse within their own environments, dreaming could emerge as a fourth scaling axis alongside pre-training, RL, and test-time inference compute. Continuous Learning and Networked Compounding (17:23) Looking toward 2027 and 2028, Patel outlines a deployment paradigm where RLVR serves primarily as a foundation, producing agents resilient enough to navigate unfamiliar obstacles. An AI could work alongside a human professional for a week, after which a simple binary evaluation—such as a thumbs-up or thumbs-down—triggers weight distillation via OPSD or simulated dreaming, expanding the system into adjacent, non-verifiable domains. Under this model, the primary driver of AI improvement would no longer be static pre-release training cycles. Instead, intelligence would compound dynamically across the broader economy. Each user deployment would feed into a collective network effect, allowing models to grow steadily more capable by assimilating the shared experiences of millions of concurrent real-world sessions. Patel's forecast depends on whether the AI industry can transcend containerized synthetic tasks and solve the credit assignment problem in ambiguous, non-resettable environments. While mechanisms like OPSD and internal dreaming offer plausible paths to update long-term weights, whether models can generate accurate internal world simulations without succumbing to catastrophic forgetting remains an unproven hypothesis.

Dwarkesh Patel认为，当前依赖静态预训练和可验证奖励强化学习（RLVR）的研发范式无法仅凭长上下文应对复杂现实任务；AI的下一个重大飞跃将是通过在策略自蒸馏（OPSD）与模拟训练实现“在职学习”，将日常推理经验持续沉淀至模型底层权重中。 可验证性与可反复试错性的分野 (2:12) Patel分析了AI技术在代码、数学领域的飞跃与计算机通用操作相对滞后之间的反差。主要AI实验室目前押注RLVR路线，认为只要在数千个环境中训练数百万个可验证任务即可逼近通用人工智能（AGI）。支持者主张初始训练极低的样本效率可通过数十亿次推理摊销，而极长上下文未来或能直接消化长达半年的员工入职培训。 然而，仅有“结果可验证”（例如确认订票、报税或送达）远远不够，任务领域还必须具备高吞吐的“可反复试错性”（Grindability）。软件工程可以通过容器化技术在隔离环境中并发运行数千个确定性实例，以极低成本尝试修复Bug；但若在亚马逊等真实网站上并发运行大量智能体，会迅速触发反爬虫防御并遭封禁。在缺乏低成本、可复现的高保真应用克隆环境前，样本效率低下的模型在通用计算机操作上举步维艰。 现实世界非重置环境与泛化极限 (5:00) 依赖数据中心内的容器化模拟器，难以训练智能体掌握政治竞选、创办企业或法庭辩护等复杂现实技能。这些真实领域属于非平稳、无重置机制（reset-free）的环境，其反馈闭环往往长达数月乃至数年，决策者根本无法像在模拟器中那样微调动作来排查究竟哪一步影响了成败。 RLVR的核心赌注在于合成容器化任务的训练能自动泛化为全能策略，乐观派设想智能体可重现1948年林登·约翰逊的政治博弈或埃隆·马斯克建立SpaceX的商业决策。但Patel对此表示怀疑，并援引Anthropic首席执行官Dario Amodei关于上下文长度的观察：若在短上下文下训练却在长上下文中服务，模型性能会出现明显退化，合成经验向复杂现实的迁移存在未知天花板。 推理算力浪费与权重固化瓶颈 (8:41) 即便模型能在单次长会话中凭借上下文学习展现出极高水准，这种经验在会话结束后也会彻底消失。Patel指出当前AI基础设施存在结构性浪费：前沿实验室约30%至50%的算力预算用于推理，但这些算力对基础模型的底层权重没有任何长期积累。部署在企业中的AI对每天遇到的工作流、组织知识与失败模式完全视而不见。 依靠无限制扩大KV缓存既不经济，也不符合生物智能机制。人脑不会随着记忆增长而无限膨胀头骨，而是依靠记忆巩固与概念抽象；过目不忘的学者症候群往往缺乏高级抽象思维。尽管梯度更新极其缺乏样本效率（如Cursor的代码补全模型需4亿条记录学习接受模式），AI仍必须找到将单次会话洞见沉淀为长久权重的方法。 在策略自蒸馏与梦境模拟的前沿机制 (10:00) Patel认为持续学习的瓶颈不在于模型架构，而在于损失函数。他重点探讨了与Sasha Rush交流中提出的“在策略自蒸馏”（OPSD）机制：让经过长会话积累经验的“老练教师模型”去指导原始“学生模型”的权重更新。相比传统强化学习，OPSD无需外部确定性奖励即可提供逐Token的稠密监督；相比容易导致灾难性遗忘的监督微调（SFT），OPSD只进行精准微调以匹配教师表现，从而保留基础通用能力。 在此基础上，更前沿的设想是“梦境模拟”（Dreaming/测试时训练）。类似于DeepMind的EfficientZero通过在脑海中运行大量虚拟模拟而在两小时内超越Atari人类新手，未来的LLM或可自主编写代码构建模拟环境并通宵训练。开发者未来可能不再仅输入压缩上下文的指令，而是通过算力“做梦”来消化白天观察到的现实经验，这有望成为预训练、RL与推理计算之后的第四扩展维度。 2027年范式转型：经济规模在职演进 (17:23) Patel预测在2027至2028年间，AI能力演进将迎来多阶段范式转移。RLVR首先锻造出面对训练分布外障碍仍能自主探索的稳健智能体；随后扩展的上下文使AI能与人类协同工作一整周；周末用户只需提供简单的优缺点评价，模型便通过OPSD或梦境模拟将经验吸纳至底层权重，并逐步向非验证性领域扩展。 一旦这一持续学习闭环打通，AI能力的跃升将不再主要依赖实验室发布前的静态预训练或后训练，而是直接在经济部署中动态复利。AI将在工作中持续进化，全球数百万用户的并发真实互动将形成巨大的网络效应，使模型在每一次人机协作中持续变聪明。 Patel勾勒出AI从“静态预训练工具”走向“在职动态进化的经济智能体”的路径，但该愿景能否实现，取决于模型能否在缺乏低成本确定性模拟器的现实世界中，仅凭极稀疏的反馈和高昂的在线自蒸馏算力跨越样本效率鸿沟。

## Background 本期内容由 Dwarkesh Patel 于 2026 年 6 月 26 日在其个人播客与网站（RSS / Web）发布，对应其专题论文《The next paradigm》（下一个范式）。节目简介中除了提及赞助商 Mercury 的自动化账单管理工具外，完整列出了从顶级实验室的技术豪赌、可刷性与可验证性、RLVR 的泛化边界，到权重更新回流、做梦机制与 2027 年展望的时间轴大纲。 ## TL;DR 当前顶级 AI 实验室正押注在大规模可验证强化学习（RLVR）与超长上下文窗口上，但真实世界由于缺乏高确定性、可无限重放的“可刷性”（grindability），暴露出了短视界训练向复杂现实泛化的根本瓶颈。尽管上下文学习能在单次会话中展现极高灵活性，但因会话结束经验即归零且受限于显存墙，造成了推理期珍贵数据与算力的巨大浪费。未来的破局关键在于实现真正的“在岗学习”（on-the-job learning），利用策略自蒸馏（OPSD）或构建内部沙盒“做梦”（测试时训练）将稀缺的现实交互提炼回静态权重中。展望 2027 至 2028 年，AI 将突破代码与数学等封闭判题器的藩篱，通过周级别的人机协同与绩效闭环在全经济体部署中完成自主演化。 ## Atmosphere & style 全篇叙述兼具冷峻严密的工程批判力与深邃宏阔的未来主义视野，Dwarkesh 以层层递进、抽丝剥茧的研究者视角剖析底层技术死穴。播客没有陷入对算力奇迹的盲目狂热，而是直面环境模拟不可行性、KV Cache 困局等现实泥潭，论证克制而富有锐气。随着话题推进至 OPSD、内心做梦乃至 2027 年的演进图景，叙事呈现出一种从理性审视走向震撼与敬畏的情感弧线，带给听众置身范式突变前夜的智识沉浸感。 ## Bulletin - `02:12` 顶级实验室正押注强化学习与长上下文扩展以实现 AGI，但电脑操作（Computer Use）的进展迟滞表明，单纯的可验证性远远不够，任务还必须具备支持确定性无限重放的“可刷性”（grindability）。 - `08:41` 前沿实验室 30% 至 50% 的算力被推理消耗且无法反哺模型权重，过度依赖 KV Cache 装载长程记忆既面临显存瓶颈，也背离了人类大脑通过高强度压缩形成抽象理解的认知本质。 - `12:15` 在策自蒸馏（OPSD）成为权重回写的重要突破口：通过将积累长上下文经验的“老兵模型”输出逐 Token 拟合回基础权重，既摆脱了外部判题器的约束，又避免了朴素微调带来的灾难性遗忘。 - `15:22` 继预训练、RL 与推理期计算之后，“做梦”（Dreaming/TTT）被视为第四种算力扩展轴，模型有望在自构的内心沙盒中高并发试错演练，以极高样本效率攻克现实任务。 - `17:23` 预计到 2027 年，模型将能支持整周连续人机协作，并通过粗粒度人类反馈与自蒸馏将工作沉淀入静态权重，推动 AI 能力由代码和数学等封闭领域向全实体经济滚雪球式扩散。 ## Key takeaways - `00:00` 实验室的核心技术押注认为，低样本效率只是训练期一次性固定成本，会被数十亿次推理调用彻底摊销，长程上下文学习最终可替代复杂的持续学习与权重在线更新。 - `02:12` “可刷性”（Grindability）要求环境确定、可重放且支持海量并行推演；真实网络由于存在反爬风控机制（如平台封号），极大阻碍了电脑操作智能体的发展。 - `05:00` 现实商业与政治决策属于无全局重置机制（reset-free）且高度非平稳的环境，其外循环验证往往长达数月乃至数年，根本无法在机房内通过高并发扰动进行因果归因。 - `05:00` Dario Amodei 指出训练视界与服务视界的不匹配会导致性能衰减，这表明短视界 RLVR 训练成果能否直接跨越式泛化至长视界真实世界任务仍存巨大疑问。 - `08:41` 模型部署阶段承载着人类社会最珍贵的隐性知识，但当前会话结束即归零的机制导致 30%-50% 的推理算力无法转化为模型的持久能力进化。 - `08:41` 无休止膨胀的 KV Cache 无法构建真正智能；人类大脑不区分参数与激活值，真正的抽象理解、泛化与顿悟源于对神经连接权重的极度信息压缩。 - `10:00` 在岗学习（On-the-job Learning）的瓶颈不在于模型架构（如稀疏注意力或 KV 压缩），而在于缺乏能依据单次异构经验安全更新静态权重的有效损失函数。 - `12:15` 策略自蒸馏（OPSD）让无上下文的基础模型去逼近拥有完整交互上下文的导师模型概率分布，提供了密集监督信号，且无需依赖环境自动判分。 - `12:15` 相比于强行记忆所有 Token 的朴素 SFT，OPSD 只精炼改变与核心顿悟相关的极少量参数，能有效抵御灾难性遗忘并保留预训练通用底座能力。 - `15:22` 借鉴 EfficientZero 的经验，未来大模型可通过“做梦”（Dreaming）自主编写环境沙盒并通宵推演，打造出基于测试时训练（TTT）的第四算力扩展轴。 - `17:23` 到 2027 年，结合长达一周的有效上下文窗口与人类低频打分闭环，系统可利用 OPSD 等技术实现周级别在岗知识向静态权重的周期性固化。 - `17:23` 正如预训练孕育底模、RLVR 锻造实干智能体，AI 能力的终极跃升将依赖于全社会各行各业在岗部署中的集体实战演化，而非实验室发布前的离线预演。 ## Detailed walkthrough ### `00:00` 顶级实验室的宏大研究豪赌：扩展 RL 与上下文能否替代持续学习？ 各大顶尖 AI 实验室目前正押注一个核心研究方向：只要让模型在数千个多样化的强化学习（RL）环境中，训练完成数以百万计的可验证任务（verifiable tasks），我们就能实质上构建出通用人工智能（AGI）。这种假设认为，高强度的环境训练能够塑造出真正具备问题解决能力的智能体（problem-solving agent），让它们在面对错误、挫折和模糊性时，依然能够连续数周推进开放式任务。乐观派坚信，目前模型面临的所有看似根本性的缺陷——例如模型训练阶段极低的数据效率（data inefficiency），或是缺乏在部署中持续自我更新的持续学习（continual learning）机制——都会在算力与规模的碾压下土崩瓦解，正如当年向大语言模型堆砌海量算力时，自然语言处理（NLP）领域的传统核心难题被一举平推一样。 针对“模型样本效率只有人类百万分之一”的反驳，当前的拥护者认为，样本效率低下仅仅发生在训练期，而训练只是一次性的固定成本，会被模型后续数十亿次的用户调用会话所彻底摊销（amortized）。真正具有决定性意义的，是模型在具体会话期间展现出的聪明程度、泛化水平以及样本效率。事实证明，随着强化学习的深入，AI 智能体解决更野心勃勃、时间跨度更长任务的能力已经肉眼可见地提升，任何深度使用过代码模型的人对此都深有体会。 关于持续学习的必要性，乐观派同样提出了有力的反驳构想：既然长程上下文学习（in-context learning）正在变得极其强大，将模型在工作中所学到的新知全部蒸馏回权重（weights）或许根本不是必需的。职场中常有说法认为，一个新员工往往需要入职工作六个月以上才能产生净产出，这意味着即时在线学习（online learning）对胜任工作是必不可少的。然而，如果通过激进的架构创新，将这长达六个月的工作经历直接塞进上下文窗口，持续学习的问题便迎刃而解。既然近年来 Transformer 在存储上下文容量上已经取得了爆炸性突破，完全有理由相信几年之后就会诞生体验上“近乎无限长”的上下文窗口。 - 实验室的核心押注：数千个 RL 环境 + 数百万个可验证任务 = 具备数周抗挫折推进能力的 AGI。 - 一次性成本与摊销逻辑：低样本效率只存在于训练阶段，在数十亿次推理部署中成本被平摊，关键在于会话内的上下文学习能力。 - 替代持续学习的方案：与其复杂地把在岗学习蒸馏回权重，不如直接用未来接近无限大的上下文窗口装载员工长达半年的在职沉淀。 --- ### `02:12` 可刷性与可验证性同样关键：为何电脑操作（Computer Use）进展缓慢？ 在探讨这一技术路线前，Dwarkesh 提出一个极为耐人寻味且令人困惑的反常现象：为什么 AI 在“电脑操作”（computer use）上的进展，远远慢于写代码和数学等其他领域？按理说，电脑操作的结果是极其明确且高度可验证的（verifiable）——例如，我在 Etsy 上订购的商品有没有送达？计划中的活动场地是否已成功预订？我的税务申报表格到底有没有提交上去？然而现实是，电脑操作相比编程和数学模型的飞速突破显得迟缓许多。除了预训练阶段高质量多模态数据严重匮乏这一显而易见的原因之外，还有一个被严重低估的根本瓶颈：一个领域仅仅具有“可验证性”是远远不够的，它还必须具备极高的“可刷性”（grindability）。 所谓“可刷性”，指的是你必须能够在一个确定性且可无限重放的模拟器（deterministic and replayable simulator）中，从完全相同的初始状态出发，进行成千上万次并行推演（parallel rollouts）。当训练一个写代码的模型时，你可以非常廉价地构建一个包含目标软件代码库的容器（container），挖掉某个功能作为任务，并分发给 1000 个完全相同的并行智能体同时尝试解决。但在电脑操作领域，这一逻辑无法直接成立。如果让 1000 个并行智能体直接跑到真实亚马逊网站上测试同一个结账流程来学习上网，“安迪·贾西（Andy Jassy）会分分钟发现你的机器人爬虫，然后把你的账号封个精光”。 ```mermaid flowchart TD accTitle: 代码训练与电脑操作训练的可刷性对比 accDescr: 对比展示代码环境支持确定性重放和高并行度，而真实电脑环境面临风控封禁与极高模拟成本。 subgraph Coding["代码任务（高可刷性 Grindable）"] C1["代码仓库容器"] --> C2["完全确定且可重放"] C2 --> C3["数千个并行智能体同时测试"] C3 --> C4["高效强化学习迭代"] end subgraph ComputerUse["电脑操作（低可刷性瓶颈）"] U1["真实网络/软件环境"] --> U2["不可重放 / 动态变化"] U2 --> U3["亚马逊等平台封禁 Bot\nAndy Jassy 封杀账号"] U3 --> U4["手动复刻 Slack/Gmail\n成本高昂且难以扩展"] end ``` 一种替代解法是手动复刻 Slack、Gmail 等常见应用和网站的高仿真克隆体，但这在目前是极度劳动力密集且不可扩展的造环境方式。当然，未来一旦 AI 自身写代码的水平高到能够以极高保真度自动复刻这些应用程序，电脑操作就会迅速爆发，并且“让 AI 从零重构完整应用”本身也是一种绝佳的写代码 RL 训练目标。但电脑操作眼前的迟滞揭示了一个冰冷残酷的事实：除非你能为一个领域搭建出极高保真、支持海量重放的训练标靶，否则模型因其训练期极其低下的样本效率，根本寸步难行。而现实中诸如白手起家创办企业、在法庭打赢官司、在金融市场实现单日稳定盈利等关键技能，我们根本无法在机房内凭空搭建这类支持无限刷数据的模拟器。 - 可刷性（Grindability）的核心标准：确定性、可重放的模拟器，以及支持从同一起点展开海量并行推演（parallel rollouts）。 - 真实环境的对抗阻力：无法直接向亚马逊等真实业务系统派发成千上万个智能体，平台防御机制会直接封禁自动化行为。 - 人工沙盒的局限：手动克隆真实 SaaS 应用极为繁琐，需等待代码模型具备自主生成软件沙箱的能力。 --- ### `05:00` 真实世界的重置困境与 RLVR 泛化边界 在真实世界中帮助候选人打赢政治选举，智能体的推演动作必须直接与现实社会交互，这种交互永远无法单纯在数据中心内部凭空模拟。真实世界的“外循环验证”（outer loop verification）可能需要数月甚至数年的现实行动才能展现出最终结果；你根本不可能通过稍微扰动模型的局部行为，在成千上万次并行测试中逐一剥离、排查出究竟是哪一步决策起到了决定性效果。在强化学习领域，这种无重置、非平稳环境（reset-free, non-stationary environments）早已是人尽皆知的未解难题。真实世界中多数领域的数据都具备高度特异性（idiosyncratic）与稀疏性，模型必须具备极高的样本效率才能真正掌握这些能力。 许多人类独有乃至超越人类的高阶技能，其核心训练信号只会从稀疏、非结构化、无法自动验证且充满模糊性的现实互动中暴露出来。那么，究竟什么样的 RL 环境能够培养出一个在政治手腕上比肩林登·约翰逊（Lyndon B. Johnson），或者在商业航天领域能与埃隆·马斯克（Elon Musk）相提并论的 AI？当前各大顶级实验室的全部底气，都在于押注“可验证奖励的强化学习”（RLVR, Reinforcement Learning with Verifiable Rewards）能够产生神级泛化——即认为只要模型在足够多容器化的可重现环境中训练，就会自然涌现出超强的通用规划与执行能力，并能在单次会话内迅速消化新信息、习得新技能。他们预期，把这样一个被 RLVR 充分洗礼的 AI 丢到 1948 年的得克萨斯州政治舞台，它给出的争夺参议员席位的策略甚至比年轻时的 LBJ 还要高明；如果在 2002 年给它 1 亿美元，它就能替你白手起家打造出 SpaceX。 Dwarkesh 指出，RLVR 能否实现如此深度的泛化依然是一个未知的实证问题。如果实验室将 RL 环境的投入从数十亿美元一路推高到一万亿美元，模型真的能仅凭上下文窗口就演进为完全逼近人类的通用智能吗？在与 Anthropic 创始人 Dario Amodei 的播客对话中，Dario 曾透露过一句意味深长的分析，解释为何模型在极长上下文下的表现通常会出现退化：“这里有两个关键因素：一个是你在训练时采用的上下文长度，另一个是你在实际服务（推理）时采用的上下文长度。如果你在短上下文长度下进行训练，然后试图在长上下文长度下去提供服务，你很可能就会遭遇这种性能衰减。” 这暴露出一个严峻的技术逻辑断层：如果模型在短视界强化学习（short horizon RL）下的训练成果，在长视界下连维持性能稳定都极其困难，那智能体又如何能将写代码、做数学题等白领任务中学到的能力，跨越式地泛化为像沃尔玛创始人山姆·沃尔顿（Sam Walton）那样扎根真实世界白手起家缔造商业帝国的本领？ - 现实世界的环境本质：非平稳（non-stationary）、无全局重置机制（reset-free），反馈周期往往以月乃至年为单位。 - 稀缺与模糊的信号：许多战略性领域的关键成功要素在机房内完全无法通过高并发扰动（perturbation）来分离归因。 - Dario Amodei 的长短上下文警示：训练视界与服务视界的不匹配会导致性能退化，预示着短视界 RLVR 的泛化边界并非毫无阻力。 --- ### `08:41` 浪费的推理算力、KV Cache 困局与权重更新的本质矛盾 退一步讲，即便智能体凭借足够多的上下文交互，在单次会话中临时成为了亨利·福特（Henry Ford）或罗伯特·爱因斯坦（Robert Einstein）级别的旷世奇才，只要无法将这些洞见固化回模型权重（weights），所有的经验在会话结束的一瞬间都会烟消云散，沦为彻底的浪费。目前各大顶级实验室有约 30% 到 50% 的算力被直接消耗在模型推理上，但这部分庞大的算力目前完全没有反哺于模型的性能提升。这种现状在技术和经济上都是极为荒谬的，因为模型在实际部署运行（deployment）阶段所接触到的信息，恰恰是全世界最珍贵的数据矿藏——包括使用该模型的企业组织内部究竟在发生什么、大家拿 AI 来解决什么真实的痛点，以及模型在现实环境中究竟更容易犯下哪些愚蠢的错误。 当前的 AI 就像是一个天赋异禀但永远不被允许参加现实实习的“天才研究生”，实验室只是在一间又一间教室内向它填鸭更多的强化学习模拟案例。AI 已经大规模渗透进全社会的日常经济活动，面对无数特定领域和机构内部的隐性知识（tacit knowledge），它们却无法沉淀任何长期记忆。要想实现真正的持续学习，智能体不可能仅仅依赖不断膨胀的 KV Cache（键值缓存）去无休止地硬塞海量用户数据——这在显存成本和硬件扩展性上是完全不可行的，更完全违背了人类大脑的运作机理。 人类大脑中根本不存在“参数”（weights/parameters）与“激活值”（activations）之间的清晰物理割裂，一个人终其一生学习海量知识，头颅和脑容量也并未随之膨胀。人类的学习本质上是一种深度的信息压缩，而正是这种高度压缩驱动了归纳抽象、泛化以及真正的顿悟（grokking）。现实中某些患有学者综合征（autistic savant）的人，能够多年不忘地精准背诵毫无意义的长串随机数字表格或生僻音节——这几乎等同于大模型在长上下文中拥有的超高保真度数据，然而这种恐怖的数据容量记忆反而摧毁了他们理解抽象概念和隐喻的能力。人类的持续学习不是把所有经历的细枝末节都时刻悬在舌尖，而是将正确的直觉和大局认知雕刻进神经连接的深层权重中。 然而，一旦试图将学习成果沉淀回模型权重，就会立刻陷入另一个极其棘手的泥潭：由于梯度更新（gradient updates）本身的样本效率极低，你将彻底丧失上下文学习所具备的即时灵活性。正因如此，目前工业界所有成功上线的“在线学习”模型，都只能被迫在数以百万计的用户身上学习同一种高度同质化的目标。例如 Cursor 的代码补全模型（Cursor Tab），其在线持续学习机制依赖于跨越超过 4 亿条同质记录，去反复预测同一个极其固定的单一目标。这种方式与真正具备灵活性、能够汲取组织异构隐性知识的持续学习之间，依然横亘着巨大的技术鸿沟。 | 学习范式 | 存储载体 | 优势 | 核心缺陷 / 瓶颈 | |:--- |:--- |:--- |:--- | | **上下文学习 (In-Context Learning)** | 激活值 / 上下文窗口 (KV Cache) | 样本效率极高，单次会话内迅速调整适应 | 极其短暂脆弱；会话结束即归零；显存开销随长度暴增，不可持续。 | | **人类神经认知 (Human Learning)** | 生物神经突触连接（压缩内化） | 大脑体积不膨胀；通过极度抽象与压缩促进泛化和隐喻理解 | 无法原样重放海量原始杂音数据（除个别学者综合征患者外）。 | | **权重在线更新 (Continual Weight Learning)** | 模型静态参数（Weights） | 永久固化技能；沉淀部署中的隐性知识；充分利用 30%-50% 的推理算力 | 梯度更新样本效率极低，目前仅能针对单一机械目标在大规模同质数据（如 4 亿条补全）上收敛。 | ### `10:00` 在岗学习的本质与样本效率瓶颈 当前模型在单次会话中根据用户的修改反馈进行在线调整，其优化目标无非是“用户的哪些修改最终被采纳”。然而，业界至今尚未见过模型能针对不同用户在线学会截然不同事物的案例。对人类来说，一次工作会话中产生的数据或许足够顿悟和学习，但对训练出能力更强的大模型而言，这丁点数据量还远远不够。当下的在线学习仅适用于极少数狭窄场景，而持续学习（continual learning）的核心在于真实世界的极端复杂性——每份工作、每家公司、每个现实难题都千差万别，智能体必须能够掌握与其特定部署环境直接相关的专属经验，而这些细碎信息绝不可能被提前塞进某个通用的预训练集群中。 这正是所谓“在岗学习”（on-the-job learning）的真正内涵：理解一家组织内部各要素如何运转与咬合、如何与现有的软硬件基础设施及同事协同推进大型项目、明确常见的失败模式等等。Dwarkesh 在此顺势谈及自己的业务切身体会：随着播客规模扩大，日常运营开销激增，以往处理承包商账单需要逐一翻找邮箱、建文件夹再手动转账；如今借助 Mercury 平台，只需给承包商一个专属邮箱，系统便会自动扫描发票并抽取姓名、地址、金额、发票号与到期日生成付款草稿，省去了所有繁琐的信息录入与追踪。 这个例子直接映射出样本效率（sample efficiency）与持续学习之间的紧密捆绑：AI 在实际在岗工作时能接触到的真实数据极其稀缺。虽然大模型目前能依靠注意力机制在上下文窗口中即时构建“快速权重”（fast weights）展现出惊人的上下文学习效率，但其显存开销随长度剧烈膨胀，扩展性极差。行业需要中间表示层的架构创新，但从稀疏注意力（sparse attention）到 KV 缓存压缩（KV cache compaction），学术界每周都在发表新花样。Dwarkesh 认为，模型架构本身并不是持续学习的根本瓶颈，真正的死穴在于损失函数（loss function）——我们到底该如何根据单次会话学到的信息去更新模型自身的静态权重？ - **在岗经验的独特性**：特定工作流程、组织架构协作方式、环境基础设施交互与典型踩坑经验，无法在离线训练阶段通过通用语料覆盖。 - **上下文显存墙**：通过 Attention 机制构建的临时激活虽然样本效率高，但受限于长上下文的显存占用，无法替代永久性的权重更新。 - **架构并非主因**：尽管稀疏注意力与 KV Cache 压缩层出不穷，但更底层的问题在于缺乏让单次经验安全更新回权重的目标函数。 --- ### `12:15` 策略自蒸馏（OPSD）：把上下文经验沉淀回权重 针对权重更新这一死穴，许多研究者近期开始押注“在策自蒸馏”（On-Policy Self-Distillation, OPSD）。Dwarkesh 提及几周前他曾用 iPhone 录制过一段与学者 Sasha Rush 在黑板前即兴探讨该技术的视频。其核心逻辑在于：迫使基础模型（Base Model）在解决真实世界问题时，输出概率分布去拟合那个在经历完整工作会话、积累了海量上下文后变得博学熟稔的“老兵导师模型”（Teacher Model）。该流程的目的就是把模型在单次工作流中汲取的上下文知识，精炼蒸馏并固化回自身的基础权重中。 与当前火热的基于可验证奖励的强化学习（RLVR）相比，OPSD 具备两大鲜明优势： 1. **无需外部可验证奖励**：RLVR 极度依赖能自动判分的环境，而 OPSD 只需要模型本身具备在上下文窗口中提炼正确线索的能力。只要上下文里的“老兵模型”变聪明了，就能用来指导基础模型。 2. **监督信号极其密集**：普通 RL 只能将最终标量奖励向整条漫长轨迹反向分摊，而 OPSD 能在逐个 Token 的概率分布差异（per-token probability discrepancy）上进行精细训练。 ```mermaid flowchart TD accTitle: 策略自蒸馏核心机制图 accDescr: 展示长上下文老兵模型如何通过逐Token概率分布差异将经验蒸馏回基础模型权重的过程 subgraph Context_Phase["会话阶段"] A["长时间实际工作流输入"] --> B["老兵导师模型<br/>(积累全量上下文)"] end subgraph Distill_Phase["权重沉淀阶段"] C["基础模型<br/>(无长上下文)"] B --> D["逐 Token 概率分布差异"] C --> D D -->|"密集监督信号更新"| E["沉淀知识后的新模型权重"] end ``` 更重要的是，OPSD 在持续学习场景下显著优于传统的监督微调（SFT）。最朴素的 SFT 做法是让基础模型去死记硬背会话期间观测到的所有 Token，但人类在工作中变优秀从来不是靠“像录音机一样复刻当天发生的一切”，而是“提炼出少数真正有助于提升工作表现的顿悟与核心认知”。RL 训练之所以不易遗忘，正是因为每一次更新都极度聚焦于与正确结果直接相关的关键参数，改变的参数量极少；而 OPSD 继承了这一精练特性，它不会粗暴地将分布生硬拉向微调数据，而是精准抽取出在现实任务中达成导师模型同等水准所必需的关键信息，从而避免抹去预训练模型既有的通用能力。 - **无需外部判题器**：避开了现实复杂任务无法自动化打分的严苛限制。 - **逐 Token 密集反馈**：相比全轨迹单一稀疏奖励，优化效率呈数量级提升。 - **防御灾难性遗忘**：改变极少量关键参数，避免朴素 SFT 强行记忆整段日志导致的知识覆盖。 --- ### `15:22` “做梦”（Dreaming）：利用内心仿真实现第四种算力扩展 除 OPSD 外，Dwarkesh 提出了另一种更大胆、更具推测性的破局设想——“做梦”（Dreaming）。如果 AI 能够自行构建一个高度逼真的现实世界模拟器，在其中反复演练新技能、尝试备选策略并强化有效行为，那么在相同的物理时间（wall-clock time）内，AI 所能经历的虚拟交互样本量将比真实世界高出数个数量级。 这在强化学习发展史上已有先例。在 DeepMind 推出 AlphaZero 数年后，研究人员开发了追求极高数据利用率的 EfficientZero。若让 EfficientZero 和一个人类新手同时接触一款从未见过的 Atari 游戏两小时，该模型很可能会击败人类新手。表面上 EfficientZero 显得更具样本效率，但这取决于计算维度：因为在真实游戏每推进一步时，该模型已经在脑海内部的隐式仿真中反复推演了数十局推演。未来的大型语言模型同样可以在真实世界消耗极少数据，转而在自身构筑的内部沙盒中无限对弈。 ```mermaid flowchart LR accTitle: 做梦演化机制与第四轴算力扩展 accDescr: 展示模型在面对现实任务时，通过构筑环境、内部做梦强化以及部署上线的闭环流程 subgraph RealWorld["现实交互"] A["遭遇罕见/复杂工作任务"] --> B["提炼世界状态"] end subgraph DreamingLoop["内心沙盒 (做梦/TTT)"] B --> C["自主编写 RL 仿真环境"] C --> D["脑内高并发推演练习"] D -->|"验证有效策略"| E["提炼实战技巧"] end subgraph Production["上线应用"] E --> F["高置信度执行实际操作"] end ``` 当然，模拟真实世界的物理与社会规则，远比模拟规则确定的围棋或雅达利游戏艰深得多，这也是该设想目前充满推测性的原因。但一旦成功，它将成为继**预训练（Pre-training）**、**强化学习（RL）**和**推理期计算（Inference-time compute）**之后的“第四算力扩展轴”——即测试时训练（Test-Time Training, TTT）或“做梦”。 - **从 `/compact` 到 `/dream`**：以往在 Codex、Cursor 或 Claude 中输入 `/compact` 只是消耗微量算力生成文字摘要，营造一种虚假的持续学习幻象；未来如果敲下 `/dream`，模型将燃烧海量算力，把在现实中观察到的场景渲染并编写成专属的虚拟游戏世界，在其中通宵达旦地训练特定技能。 --- ### `17:23` 展望2027年：从可验证领域迈向全经济体真实进化 Dwarkesh 勾勒出了一幅 2027 至 2028 年持续学习落地的演进蓝图。当前实验室通过 RLVR 进行的大量训练，核心价值在于打造出一个“在面对陌生问题时能够摸清门道、主动尝试不同策略并在碰壁后快速迭代”的高韧性智能体。RLVR 赋予了模型最基础的实干能力，使其达到可以被派往真实世界打工的门槛，哪怕面对的课题完全脱离预训练的数据分布。 接下来的范式跃迁将按以下阶段推展： 1. **周级别连续协同**：到 2027 年左右，模型的有效上下文长度将扩展到足以支持与人类连续协作整整一周的物理时间。 2. **人类绩效评审闭环**：在一周结束时，用户无需手动标注每一步，只需给出一个整体的满意评价（“thumbs up / thumbs down”）。 3. **混合沉淀与泛化**：得到肯定后，系统利用 OPSD、做梦或尚未发明的新算法，将整周的上下文经验沉淀至静态权重中。 4. **邻近领域滚雪球扩展**：模型首先在与 RLVR 显式训练集相邻的领域变强；在下一轮循环中，又会基于新学到的经验继续向更外圈的相邻领域扩散。AI 的能力版图由此彻底冲破离线阶段可验证环境（代码、数学）的藩篱。 | 发展阶段 | 核心驱动技术 | 学习发生阶段 | 能力边界 | |:--- |:--- |:--- |:--- | | **基础智能期** | 大规模自监督预训练 | 部署上线前（离线） | 通用常识与文本理解，但缺乏自主落地代理能力 | | **智能体萌芽期** | 规则可验证强化学习（RLVR） | 部署上线前（离线） | 擅长数学、编程等有客观判题器的封闭领域，具备试错韧性 | | **在岗学习成熟期** | 策略自蒸馏（OPSD）+ 做梦（TTT） | 真实部署中（持续在线） | 覆盖非结构化企业内部协作、复杂商业决策等广阔经济场景 | 最终，正如预训练孕育了能够被 RLVR 调教出 Agent 能力的底模，RLVR 则打造出能够深入实体经济广泛部署的合格员工。一旦持续学习的算法配方彻底拼齐，大模型最主要的能力跃进将不再依赖于实验室在发布前的离线预演，而依赖于它在被部署到全社会各行各业中所吸收的庞大实战经历。未来用户每一次与 AI 交互，模型都在变聪明——这不仅因为模型记住了该用户的过往偏好，更因为它已在千千万万其他真实用户的协作实战中完成了脱胎换骨式的集体演化，“这既令人兴奋，也极其令人敬畏”。 ## Notable quotes > `02:12` 一个领域仅仅具有“可验证性”是远远不够的，它还必须具备极高的“可刷性”（grindability）。 — Dwarkesh Patel > `02:12` 安迪·贾西（Andy Jassy）会分分钟发现你的机器人爬虫，然后把你的账号封个精光 — Dwarkesh Patel > `05:00` 这里有两个关键因素：一个是你在训练时采用的上下文长度，另一个是你在实际服务（推理）时采用的上下文长度。如果你在短上下文长度下进行训练，然后试图在长上下文长度下去提供服务，你很可能就会遭遇这种性能衰减。 — Dario Amodei > `08:41` 目前的各大顶级实验室有约 30% 到 50% 的算力被直接消耗在模型推理上，但这部分庞大的算力目前完全没有反哺于模型的性能提升。 — Dwarkesh Patel > `08:41` 当前的 AI 就像是一个天赋异禀但永远不被允许参加现实实习的“天才研究生”，实验室只是在一间又一间教室内向它填鸭更多的强化学习模拟案例。 — Dwarkesh Patel > `08:41` 人类的学习本质上是一种深度的信息压缩，而正是这种高度压缩驱动了归纳抽象、泛化以及真正的顿悟（grokking）。 — Dwarkesh Patel > `10:00` 模型架构本身并不是持续学习的根本瓶颈，真正的死穴在于损失函数（loss function）——我们到底该如何根据单次会话学到的信息去更新模型自身的静态权重？ — Dwarkesh Patel > `12:15` 人类在工作中变优秀从来不是靠“像录音机一样复刻当天发生的一切”，而是“提炼出少数真正有助于提升工作表现的顿悟与核心认知”。 — Dwarkesh Patel > `15:22` 未来如果敲下 `/dream`，模型将燃烧海量算力，把在现实中观察到的场景渲染并编写成专属的虚拟游戏世界，在其中通宵达旦地训练特定技能。 — Dwarkesh Patel > `17:23` 未来用户每一次与 AI 交互，模型都在变聪明——这不仅因为模型记住了该用户的过往偏好，更因为它已在千千万万其他真实用户的协作实战中完成了脱胎换骨式的集体演化，“这既令人兴奋，也极其令人敬畏”。 — Dwarkesh Patel ## Mentioned Dwarkesh Patel, Mercury, Andy Jassy, Etsy, Slack, Gmail, Lyndon B. Johnson, Elon Musk, SpaceX, Dario Amodei, Anthropic, Sam Walton, Henry Ford, Cursor, Cursor Tab, Sasha Rush, DeepMind, AlphaZero, EfficientZero, Atari, Claude, Codex, Transformer ## Video description <p>Read it <a target="_blank" href="https://www.dwarkesh.com/p/the-next-paradigm">here</a>.</p><p>Thanks to Mercury for sponsoring this essay.</p><p><a target="_blank" href="https://mercury.com/">Mercury</a> has automated basically my entire bill pay process for my business. I just give contractors a dedicated email address, and when they send an invoice, Mercury automatically creates a draft payment for me to review. I no longer have to hunt through my inbox for invoices or deal with messy spreadsheets to track my bills. Mercury handles it all. Learn more at <a target="_blank" href="http://mercury.com">mercury.com</a></p><p>Timestamps:</p><p>(00:00:00) – The big research bet the labs are making</p><p>(00:02:12) – Grindability is just as important as verifiability</p><p>(00:06:10) – Will RLVR alone generalize?</p><p>(00:08:41) – Getting the learning back to the weights</p><p>(00:15:22) – Dreaming</p><p>(00:17:23) – What 2027 looks like</p> <br/><br/>Get full access to Dwarkesh Podcast at <a href="https://www.dwarkesh.com/subscribe?utm_medium=podcast&#38;utm_campaign=CTA_4">www.dwarkesh.com/subscribe</a> [Source](https://api.substack.com/feed/podcast/203704568/b3a3986f8a780f554393d02d34bdcf51.mp3)

[2:12] Dwarkesh Patel: 一个领域仅仅具有“可验证性”是远远不够的，它还必须具备极高的“可刷性”（grindability）。 [2:12] Dwarkesh Patel: 安迪·贾西（Andy Jassy）会分分钟发现你的机器人爬虫，然后把你的账号封个精光 [5:00] Dario Amodei: 这里有两个关键因素：一个是你在训练时采用的上下文长度，另一个是你在实际服务（推理）时采用的上下文长度。如果你在短上下文长度下进行训练，然后试图在长上下文长度下去提供服务，你很可能就会遭遇这种性能衰减。 [8:41] Dwarkesh Patel: 目前的各大顶级实验室有约 30% 到 50% 的算力被直接消耗在模型推理上，但这部分庞大的算力目前完全没有反哺于模型的性能提升。 [8:41] Dwarkesh Patel: 当前的 AI 就像是一个天赋异禀但永远不被允许参加现实实习的“天才研究生”，实验室只是在一间又一间教室内向它填鸭更多的强化学习模拟案例。 [8:41] Dwarkesh Patel: 人类的学习本质上是一种深度的信息压缩，而正是这种高度压缩驱动了归纳抽象、泛化以及真正的顿悟（grokking）。 [10:00] Dwarkesh Patel: 模型架构本身并不是持续学习的根本瓶颈，真正的死穴在于损失函数（loss function）——我们到底该如何根据单次会话学到的信息去更新模型自身的静态权重？ [12:15] Dwarkesh Patel: 人类在工作中变优秀从来不是靠“像录音机一样复刻当天发生的一切”，而是“提炼出少数真正有助于提升工作表现的顿悟与核心认知”。 [15:22] Dwarkesh Patel: 未来如果敲下 `/dream`，模型将燃烧海量算力，把在现实中观察到的场景渲染并编写成专属的虚拟游戏世界，在其中通宵达旦地训练特定技能。 [17:23] Dwarkesh Patel: 未来用户每一次与 AI 交互，模型都在变聪明——这不仅因为模型记住了该用户的过往偏好，更因为它已在千千万万其他真实用户的协作实战中完成了脱胎换骨式的集体演化，“这既令人兴奋，也极其令人敬畏”。

[5:00] Dario Amodei: There's two things. There's the context length you train at, and there's a context length that you serve at. If you train at a small context length and then try to serve at a long context length, like maybe you get these degradations. [17:23] Dwarkesh Patel: Every time that you interact with an AI, it'll be smarter. Not only because it's been learning from your previous sessions, but also because it's been learning from all its interactions with all the other users in the world. And that's very scary and exciting and different from the way that AI improves right now.

[0:00] So here's the big research bet that all the labs are making. They think that if we train AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments, then we will have basically built AGI. Because this kind of training will have created a kind of problem-solving agent, the kind of thing that can make progress on open-ended tasks for weeks on end in the face of errors and mistakes and ambiguity. And the people who are optimistic about this vision will say that all these things that we talk about as the fundamental deficits in the current training paradigm, for example, the data inefficiency of these models or the fact that they lack continual learning, these things can just be steamrolled if we just scale training more. And the same way that all the fundamental research problems in natural language processing collapsed when we just threw enough compute into LLMs. So in the previous essay, I talked about how these models are one one millionth a sample efficient as humans. And the people who are in favor of the current training paradigm will say, look, that might be true, but this is only true during training. And training is this one-time cost that is amortized across billions of sessions that a model will experience. And what really matters is how smart and general and sample efficient the model is during a session And this has clearly been improving as we been doing more RL training AI agents are able to solve more and more ambitious problems over longer and longer time spans Anybody who has used these models for coding knows that Similarly, people would say, look, continue learning. This capability I keep harping about where the model's weights get updated based on what is learning from deployment may simply not be necessary because if in-context learning gets so good across longer and longer time horizons, then you don't need to distill back everything the model is learning on the job into the weights. People often say that their employees are not net productive until six months or more of them working on the job. So clearly, online learning is necessary for competence. But what if you could just fit those six months into the context window? There's been tons of architectural innovations that dramatically increase the amount of information or the amount of context that a transformer can store. And why not think with a couple more years of progress, you might have what feels like infinitely large context windows. Okay, so before we discuss this research a bit further, I want to step back and I want to ask a completely tendential question, which I find actually very interesting and confusing about the nature of current AI progress. Why has progress on computer use been so much slower than other domains Computer use is so clearly verifiable You could ask a question like did the desired Etsy item I ordered get delivered Is the venue for an event I'm trying to organize booked? Have my taxes been submitted? So isn't it weird that computer use has been making so much slower progress than coding and math and these other verifiable domains? I'm sure there's many reasons for this. And one of them, of course, is the fact that the modelers are exposed to far less high quality multimodal data during pre-training. But one reason that I think is actually quite underrated by people and which I think reveals the canyon walls against which this river of AI progress will only slowly chip away at is that it is not enough for a domain to be verifiable. It also has to be very grindable in the sense that you have to be able to run lots of parallel rollouts against a deterministic and replayable simulator. And you have to run those rollouts from the same starting point. If you're trying to make a model better at coding, you can define some container that has the software repo with a missing feature that you have tasked the AIs with creating. And then you have a thousand parallel agents that go at the problem, each of which has an identical copy of the container. But this doesn't work with computer use, at least not trivially You can just have a thousand Asians go try the same checkout flow on Amazon to get better at using websites because Andy Chassie will find your bots and shut your ass down You can solve this by making clones of Slack and Gmail and all the other common applications and websites. But at least currently, this is a very labor intensive and unscalable way to build environments. Of course, once AIs get good enough at coding themselves to build these clones with extremely high fidelity, then I'm sure the computer use will make quicker progress than it is right now. And you're also killing two birds with one stone with this kind of procedure because getting AIs to rebuild whole applications from scratch is also a great RL objective for coding. So while computer use itself may soon be solved, its current lethargy is telling us the following. That unless you can build a very replayable training target for a domain, the models will struggle to make much progress. And the reason this is true, of course, the models are incredibly sample inefficient during training. This is a point I was making in my last video essay. So for computer use, we might be able to make up for the sample efficiency deficit by building these farmable deterministic simulators. But for so many other different kinds of skills that we need AIs to have, we simply can't do this. How do we train an AI to get really good at building a business from scratch? How about winning court cases? Or having a profitable day of trading in the markets? [5:00] a candidate win an election. The rollout here requires interacting with the real world, and you can't recreate it from just within a data center. The outer loop verification here may take months or even years of real world actions to elicit, and you can't re-observe it by perturbing the model's actions slightly in thousands of parallel rollouts to isolate exactly what the model did that actually worked. Now, dealing with such reset-free, non-stationary environments is a known open problem in RL. I'm not pointing out anything new. But I really do want to emphasize that because of the idiosyncratic and sparse nature of data in most domains in the world, you need sample efficiency in order to get proficient. If AIs are to develop all the skills that humans have, and even skills that humans don't have, then they need to be able to learn from information revealed in unstructured, unverifiable, and ambiguous ways from scarce amounts of real-world interaction, because in many domains, the relevant training information simply doesn't exist in any other way. What is the RL environment to make an AI that is as good at politics as Lyndon Johnson, or as good at building a space launch business as Elon Musk? The labs are betting that RLVR will generalize That is that if you train on enough containerized reproducible environments you will develop a very general agent that can make and execute plans and learn rapidly from new information and even pick up new skills all within a single session. If you drop this endlessly RLVRed AI into Texas politics in 1948, it could give you better advice than LBJ about winning the Senate seat. And if you give it $100 million in 2002 and let it cook, it would build SpaceX for you. Now, whether RLVR can generalize this well is an empirical question. If the labs went from spending billions of dollars on RL environments to a trillion dollars, would you get the kind of thing that is a fully human-like general intelligence within the context window? Dario gave a telling quote during our podcast together, which I think hints that RLVR generalization is not infinitely strong. But he was explaining why model performance tends to degrade at long context. He said, there's two things. There's the context length you train at, and there's a context length that you serve at. If you train at a small context length and then try to serve at a long context length, like maybe you get these degradations. Now, maybe I'm reading too much into this, but it seems like he's saying that short horizon RL training doesn't necessarily generalize to long horizon RL performance And if you can generalize from short horizon to long horizon then how are agents supposed to generalize from getting trained at a bunch of white collar tasks to say having the ability to be dropped in the real world and build a business from scratch as well as Sam Walton And even if after enough in-context experience, the AIs could become like Henry Ford or Robert Einstein, all that would be ephemeral and wasted if you couldn't get those learnings back into the weights. Around 30 to 50 percent of a lab's compute goes to inference. and that compute is currently not playing any productive role in helping improve the model. This seems like a huge waste and it's even worse than it sounds because it is only in deployment that the most valuable bits of information which your model could learn from are actually revealed. Things like what's actually happening in the organizations where I'm being used and what are they using me for and what kinds of mistakes do I tend to make in the real world. We've got some genius grad student who's never been allowed to take a real internship and we keep giving it more and more classroom case studies in the form of RL training on environments. It's so bizarre that we have AIs that are broadly deployed through the economy already and are participating in so many different kinds of tasks and are privy to so much domain and organization-specific tacit knowledge and they not able to make use of it But this kind of continual learning requires going back to the weights AIs can just keep building up a bigger and bigger KV cache as they learn from more and more users That's just not scalable. And that's also not how humans do it. There's no clean separation in our brain between parameters and activations. And it's not like some part of your skull keeps expanding as you learn more things through your lifetime. When we learn stuff, there's clearly some kind compression, and this aids our generalization and grogking. There are, in fact, some humans who have this autistics of ant-type ability to recall random tables of numbers or nonsense syllables years later, basically the kind of fidelity information that models have in context. And such sheer volume cripples these humans' ability to understand abstractions and metaphors. Human continual learning is less about having all your observations at the tip of your tongue and more about chiseling the right intuitions and big picture knowledge back into the weights. But the moment you move into the weights, you have to give up on in-context learning sample efficiency. Because gradient updates are super sample inefficient, all of the successfully shipped online learning models have had to learn the exact same thing across millions of users. For example, the cursor tab model online learns by predicting the same exact objective for over 400 million records [10:00] rest of the day. The objective here being which edits actually got accepted by the user. At least so far, we haven't seen models online learn different kinds of things for different users. Because while a single session may generate more than enough data for a human to learn from, it's not enough to train a more capable AI. Current online learning can work for a very limited number of use cases. But the whole point of continual learning is that the world is very complicated. And each job and company and problem is different. And you need your intelligence to be able to learn the specific information related to a particular deployment, which simply can't be stuffed into some shared training run. These are all the things we're talking about when we talk about on-the-job learning. Things like how does everything in your organization work and fit together, and how to cooperate with all the infrastructure and the other people around you to make progress on some larger project, and what the common failure modes are, and many other things like this. As the podcast has grown, I've had to deal with more and more operational overhead. Take paying bills. In the past, contractors would just email me their invoices. Every few weeks, I'd dig through my inbox, I'd create a folder with all the bills, and I manually pay each one At this point though I just give everybody an email address that goes straight to Mercury which is my banking platform Whenever anybody sends an invoice to that address Mercury automatically downloads it scans it and extracts all the relevant information. Things like the contractor name, address, payment amount, invoice number, and due date, and then uses all of this to create a draft payment. Mercury then stores a list of these drafts for me to review. I just go through this list and double-check that I've been built correctly. I don't know how to track anything or enter any information myself. Mercury does all the fundamental things for your business extremely well, and it puts them all in one place. If you want to learn more, go to Mercury.com. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA members FDIC. In this way, sample efficiency and continual learning are actually deeply connected problems. Relatively little data is available to the model on the job. Now, to learn from this data requires sample efficiency. And models can do that in context, but using the fast weights that are built on the fly by attention, which allows for the sample efficiency, but scales very poorly in terms of memory. So we need architectural innovations that allow for some kind of intermediate representation. I talked before about how we already have many different working ideas for this kind of thing from sparse attention to KV cache compaction And every week somebody releases a new paper suggesting some kind of other architectural optimization It doesn't seem to me that architecture is fundamentally what is bottlenecking in continual learning. So perhaps the bottleneck is the loss function. How do we update the weights, aka how do we improve the model itself, based on information that was learned from one particular session? Even here, naively, it seems like there are many ideas that ought to work. A lot of people are talking about this technique called on-policy self-distillation recently. If you want to learn more about it, I recorded a little impromptu Blackboard lecture on my iPhone with Sasha Rush a couple weeks ago, and it's in the link in the description. But to summarize the explanation, the idea is that we encourage the base model to make the same predictions when trying to solve some real-world problem as the model with all the context accumulated after a long session would have made. The whole point of this procedure is to distill what the model learned in a session back into the weights themselves. This is better than RLVR for two reasons. One, OPSD doesn't require us to have some outer loop verifiable reward. We just need a model that can learn the right things within the context window And as long as we have that we can train the base model to match our veteran teacher model which has built up all this experience during the session And two OPSD provides a much denser supervision signal than naive RL Instead of projecting a single reward through the whole trajectory, you can train on the per-token probability discrepancy between the teacher and student. For continued learning, OPSD is also superior to supervised fine-tuning. The most naive version of SFT for this application that you can imagine is just to train the base model to predict all the tokens that are observed during the session. But this takes no sense if you think about it as a learning target. The way you get better at your job is not by recalling the transcript of every single thing that happened every day with perfect fidelity. Rather, it's by consolidating the handful of insights and pieces of knowledge that are actually relevant to you getting better at your job. RL trading doesn't suffer from this failure mode. RL is great at concentrating the update to only what is relevant to getting the outcome right. That's why actually very few parameters are changed during an RL training step. And this is a very important property for continual learning because as you're learning on the job, we don't want to overwrite and forget all the other things that the base model knows. I wrote a post a few months earlier arguing that RL learns much less information per sample than supervised learning. But this may be a good thing rather than... [15:00] thing. You only change the model as much as is absolutely necessary to achieve the outcome, and no more. OPSD preserves this property of supervised learning, where instead of slingshoting towards a teacher distribution, as supervised learning would have you do, you only extract the knowledge that is necessary for you to achieve the same results of the teacher on actual real-world tasks. OPSD is one way to attack the sample efficiency problem. You take the scarce real-world experience, and you squeeze all the signal into a tiny, well-targeted update. But there's also another, much more speculative idea. Let's call it dreaming. If the AI can build a good simulation of reality against which to rehearse new skills or try alternative strategies and reinforce what actually works, then AIs could experience orders of magnitude more simulated samples in the same wall clock time. Let's go back into history a bit. A couple of years after DeepMind released Alpha zero, a group of researchers trained a model called Efficient Zero. And the whole point of this model is to be very efficient with data. So if this model and a human both got two hours to play against a simulator of an Atari game that they hadn seen before this model would actually probably beat the novice human Does this mean that the model was more sample efficient than the humans Well that was the goal of the training but it depends on how you measure sample efficiency Because for each step in the real game, Efficient Zero is playing dozens of simulated games in its head. In a similar way, future LLMs might be able to consume far less real-world data while practicing endlessly against environments that they build for themselves. The big difference, of course, is that it will be much harder to build a simulation of the whole world than it is to emulate the game of Go. That's why I said this is a much more speculative idea. If it works, it would become a fourth axis of scaling alongside pre-training, RL, and inference time compute. You call it test time training or dreaming. The model spends compute writing up RL environments and then training against them, and it's rehearsing all the skills that will actually be used in production for a specific user. So instead of hitting forward slash compact in codex or cursor or clod, which kindles a small amount of compute to write up a summary, and which gives you the simulacrum of continual learning, you hit forward slash dream. And this incinerates huge amounts of compute to build and train against a video game version of what the model is witnessing in the real world So what might continual learning look like by 2027 or 2028 And how do we get there Here one scenario All of this RLVR training is producing an agent that can get its bearings when it thrown at an unfamiliar problem, and it can try different strategies, and it can iterate when it hits a roadblock. This is the crucial thing that RLVR has given you, an AI that is at least competent enough to start getting some real world experience if you could learn from it. And once you have that, you send it out into the world to do real work, even on projects that are off the trading distribution. Now, let's say at this point, the effective context lengths have expanded such that AIs can jam and co-work with you for a full week of wall clock time. At the end of a week, you give it a thumbs up or a thumbs down. Give it a work review. And if you give it a thumbs up, the base model distills everything that the AI learned during the session. And it may use OPSD, it may use dreaming, it may use some other technique that we aren't even aware of, or it'll use a combination of all of the above. And once it does so, the AI starts getting better domains that are adjacent to what it was explicitly trained to beforehand with our LVR training. And in the next round after that it can get better at things that are adjacent to what it had previously online learned In this way the gamut of AI skills and knowledge and capabilities can expand far beyond the verifiable domains that the model was originally trained against before it was deployed. Just as pre-training created a base intelligence that was smart enough to become a competent agent with enough RLVR on top, so RLVR has created an agent that is competent enough to be actually broadly deployed in the world and from this broad deployment to learn on the job once the training recipe for continual learning actually arrives. By this point, the main way that AIs get better is not from the training they have received before they are released to the public. Rather, it's from all this experience that they'll be accumulating from being broadly deployed in the economy and engaging in so many different kinds of tasks. Every time that you interact with an AI, it'll be smarter. Not only because it's been learning from your previous sessions, but also because it's been learning from all its interactions with all the other users in the world. And that's very scary and exciting and different from the way that AI improves right now. This was an aeration of a blog post that I also released on my website at dwarkash.com. Go there if you want to read all the footnotes or if you want to sign up so you can find out when I release the next blog post. Otherwise, I'll see you on the next episode.