Sutton's Oak Lab Rejects Static AI to Build Continual Agents

Sutton旗下Oak实验室摒弃静态AI，构建持续学习智能体 前沿AI实验室因依赖冻结的预训练语言模型而仍受困于局部极小值，而真正的智能需要持续的在线权重更新，以应对无限复杂的现实世界。 AI先驱Rich Sutton与Khurram Javed对当前的静态大语言模型以及对人工策划的合成数据的依赖提出了批评，认为真正的智能需要通过与现实世界的交互进行持续的终生学习。他们提出了“大世界假说”（Big World Hypothesis），并阐释了其持续反向传播（Continual Backprop）算法如何在不依赖冻结权重或离线批处理更新的情况下解决灾难性遗忘。针对该领域当前陷入的局部极小值，他们概述了Oak实验室的5到10年目标：构建一个拥有1万亿参数的心智系统，该系统能保持连贯性、持续适应，且运行功耗仅为20瓦。 强化学习与持续学习并非脱离AI的激进转向，而是认知科学的历史常态，因为所有真正的学习本质上都是持续进行的。 (0:00) 《苦涩的教训》（The Bitter Lesson）表明，直接随计算能力扩展的方法（具体为搜索与学习）随着时间的推移始终会超越人类的领域知识。 (5:00) 尽管LLM展示了算力扩展的威力，但它们遭遇了根本性瓶颈，因为有限的互联网无法涵盖物理世界的全部复杂性。 (5:00) “大世界假说”认为，环境远比任何智能体或软件模拟无限复杂得多，这使得预封装的合成数据成为不可靠的瓶颈。 (10:00) 由于智能体的记忆与算力有限，它们必须对世界维持高度近似，并通过经验不断构建自身的内部世界模型。 (15:00)

Rich Sutton and Khurram Javed argue that modern artificial intelligence has reached a conceptual dead end by relying on static, pre-trained models with frozen weights, which cannot scale to the boundless complexity of the real world. They contend that genuine intelligence requires continual learning—adapting parameters in real time from an individual agent's ongoing stream of experience. The Big World and Pre-Training Limits (10:00) While large language models (LLMs) validate the computational scaling principles of Sutton's "Bitter Lesson" by ingesting public internet data, Sutton and Javed assert that static pre-training cannot scale indefinitely. Under Javed's "Big World Hypothesis," the physical environment is infinitely more complex than any software simulator, agent, or finite dataset. Attempting to bypass data exhaustion with human-curated synthetic data merely reintroduces an engineering bottleneck, as human domain experts cannot fully anticipate physical dynamics such as motor wear or acoustic friction. Furthermore, Sutton and Javed critique the commercial norm of freezing model weights at deployment. In their view, deploying models with fixed parameters prevents them from acquiring genuine expertise. While commercial tools aggregate user interactions in offline batches, true adaptation demands that an individual agent update its weights dynamically along a single, un-batched stream of ongoing personal experience. Biological Plasticity and Self-Discovered Abstractions (25:00) Sutton and Javed point out that biological intelligence relies on deep plasticity rather than supervised learning. Animals do not receive supervised feedback instructing individual muscle twitches, and Moravec's paradox demonstrates that sensory-motor foundations are far more difficult to mechanize than high-level symbolic reasoning. Javed highlights neurological cases where patients who lose proprioception successfully rewire their motor policies over several years using visual cues alone, an adaptable plasticity that static artificial neural networks lack. Crucially, the researchers distinguish between human cultural accumulation—which enables modern humans with 10,000-year-old brain architecture to achieve milestones like space travel—and fundamental intelligence. They argue current AI excels only in closed environments where humans pre-program rules and operators, such as AlphaGo or theorem provers in Lean. In contrast, true intelligence requires agents to self-discover temporal abstractions and internal world models directly from raw interactive data, with Sutton estimating that language accounts for only 20% to 25% of overall intelligence. Overcoming Catastrophic Forgetting with Continual Backprop (40:00) In Sutton's 2022 roadmap, *The Alberta Plan for AI Research*, continual deep learning represents the essential algorithmic bottleneck that must be solved before higher-level abstraction discovery can occur. Naively updating a neural network on a single data stream causes catastrophic forgetting, where new weight adjustments erase prior knowledge. Sutton maintains that catastrophic forgetting is completely curable without relying on offline batching, pointing to their *Continual Backprop* algorithm published in *Nature*. Continual Backprop introduces two core mechanisms: independently meta-learning a step size for every weight—keeping most step sizes tiny to insulate established representations—and maintaining a continuous "generate-and-test" cycle. While standard backpropagation expends its initial weight diversity over training and loses plasticity, Continual Backprop continuously injects randomly initialized units during runtime, preserving useful ones via backpropagation. Sutton emphasizes that foundation models must incorporate these mechanisms from scratch, as continual learning capabilities cannot simply be fine-tuned onto frozen architectures. The 20-Watt Frontier and Industrial Local Minima (45:00) Sutton and Javed outline an operational objective of running a self-maintaining, one-trillion-parameter mind on a 20-watt power budget. Javed acknowledges this is physically impossible with current hardware memory architectures, but projects it will become achievable over a 5-to-10-year horizon assuming historical Moore's Law efficiency gains, which equates to roughly 2,000 watts of compute today. Despite having the resources to explore this paradigm, major frontier labs remain trapped in what Javed describes as a "local minimum." Bound to commercial product releases that demand continual scaling of static models, incumbent labs cannot afford the short-term performance regressions required to pivot to continual learning. To pursue this non-standard trajectory, Javed explains that Oak Lab is purposefully keeping its team small and focused on developing a single architectural design that generates diverse minds conditioned on individual experiences, rather than a single monolithic model. While Sutton and Javed position continual backpropagation as an algorithmic alternative to static language models, validating this architecture at frontier scale remains an ongoing research endeavor. Their long-term vision depends on multi-fold hardware efficiency gains and relies on the premise that commercial AI will inevitably exhaust static data paradigms.

强化学习先驱 Rich Sutton 与研究员 Khurram Javed 认为，当前依赖离线预训练和冻结参数的主流 AI 范式存在根本缺陷；真正的智能必须立足于持续学习，使系统在与真实物理世界的单一经验交互流中动态演进，而非依赖静态人类数据或人工构建的仿真环境。 静态模型的极限与大世界假说 (5:00) Sutton 提出的“苦涩的教训”（The Bitter Lesson）表明，依赖人类领域知识的系统终将被通用搜索和学习算法淘汰。当前大语言模型（LLM）通过消耗海量互联网数据证明了计算扩展的有效性，但静态人类数据储备终将耗尽。Sutton 与 Javed 认为，部署后冻结权重的系统宣称具备“博士级专业知识”并不合理；目前的商业自适应方案依赖上下文窗口或跨用户的大规模批量更新（batching），不仅无法实现真正的个人经验流适配，更违背了智能体在环境中持续行动与学习的基本原则。 试图依靠人类策划的合成数据打破瓶颈同样存在根本局限。Javed 提出的“大世界假说”（The Big World Hypothesis）指出，现实环境的复杂度无穷无尽，必然远超任何单一智能体或仿真器的容量。无论是模拟无人机的蝙蝠式回声定位，还是应对真实物理中的微小摩擦与磨损，由人类工程师修正虚实差距（sim-to-real gap）只会再次引入人工干预的瓶颈。智能体必须基于自身有限的算力与存储，在真实交互中不断用经验修正自身构建的内部世界模型，而非依赖预先写定的仿真模拟器。 生物可塑性与自主抽象世界模型 (25:00) 在探讨自然智能的机制时，Sutton 与 Javed 强调监督学习绝非生物智能的核心。根据莫拉维克悖论（Moravec's paradox），对于人类看似高阶的逻辑推理或下棋，计算机往往更易攻克，而行走等低阶感知运动能力反而极其困难。Sutton 指出，自然界中没有动物通过外部给定的目标样例来学习肌肉抽动，甚至松鼠也能在没有定理证明能力的情况下自主规划复杂路径。Javed 引用的神经学案例显示，即便人类完全丧失负责感知身体位置的本体感觉（proprioception），大脑的深度神经可塑性也能支持其在两到三年内完全依赖视觉反馈重新学会行走。 针对人类区别于动物的飞跃，两人指出人类大脑的生理硬件架构在一万年间并无实质改变，现代文明的成就主要依赖代际传承的文化知识积累以及基于经验发现的高层抽象概念。相比之下，当前 AI 体系在抽象建模与规划上存在显著断层：AlphaGo 或 Lean 证明器之所以能实现高效规划，是因为人类工程师预先输入了绝对完备的博弈规则与形式逻辑算子。现存模型缺乏直接从底层连续感知运动数据流中自发提炼时间与空间高层抽象模型并据此进行推演的能力。 解决灾难性遗忘与持续反向传播 (35:00) 在 Sutton 共同提出的《阿尔伯塔 AI 研究计划》（The Alberta Plan）中，“持续深度学习”（Continual Deep Learning）被列为实现高级世界模型与抽象概念发现的基础前置步骤。Javed 说明，持续学习的核心阻碍完全是算法层面的难题而非算力瓶颈：若将传统反向传播直接应用于单一路径的连续流式数据，新样本的梯度更新会产生严重的“灾难性遗忘”（catastrophic forgetting），在更新当前特征的同时彻底破坏网络原有的既有知识。 针对这一痛点，Sutton 强调灾难性遗忘在理论与算法上完全可解。Sutton、Javed 及其合作者在《Nature》上发表的“持续反向传播”（Continual Backprop）算法提出了双重机制：首先，算法为网络中的每个权重独立进行元学习以匹配自适应步长，对绝大多数承载既有记忆的权重施加极小步长，隔绝破坏性修改；其次，区别于传统网络仅在初始阶段引入随机权重的做法，该算法在网络运行的生命周期中持续注入随机初始化的新神经元，通过“生成与测试”（generate-and-test）机制动态筛选并保留有用特征。Sutton 指出，这种机制意味着未来模型必须从零开始训练以习得知识获取的元能力，无法仅靠在现成冻结模型上微调达成。 工业界局部最优与 Oak Lab 愿景 (45:00) 对于当前顶尖 AI 实验室为何未全力投入这一方向，Javed 解释称整个行业正困于商业交付的“局部最优”之中。主流机构受制于现有产品的发布周期与规模扩张策略，而探索颠覆性的持续学习范式需要承担初期性能下滑的研究风险。两名研究者的长远目标是构建一个在 20 瓦能耗下运行万亿参数的自适应智能体。尽管 Javed 承认以当下的内存与硬件功耗体系在物理上无法实现该指标，但随着摩尔定律在未来 5 到 10 年带来约两个数量级的计算效率提升（约每 18 个月翻一番），该目标在当前相当于约 2,000 瓦功耗预算下的算法验证，具有明确的工程探索空间。 Sutton 评估指出，语言能力虽然取得了令人瞩目的突破，但其在完整智能体系中仅占约 20% 到 25%，将语言掌握等同于通用智能是严重的学科偏差。Javed 阐述了二人创立的 Oak Lab 的发展蓝图：实验室不寻求打造一个掌握所有知识的中心化单一静态模型，而是致力于研发一种通用的底层算法机制，使其能跟随不同智能体的独特经验流孕育出各具特色的心智。为确保对非主流持续学习路线的研究专注，Oak Lab 计划控制初期团队规模，稳步构建深度协同的研究核心。 Sutton 与 Javed 为摆脱静态预训练困境勾勒了基于持续反向传播与自主抽象建模的技术路径，但从连续交互流中重新训练全新基座模型需要巨大的时间投入与不确定的算力验证；在商业主流全面押注 Transformer 和批处理扩展的现实下，该范式能否在短期内跨越算法稳定性与工程效率的鸿沟，仍有待未来实证检验。

## Background 本期播客由红杉资本（Sequoia Capital）合伙人 Sonya Huang 与 Alfred Lin 主持，于 2026 年 8 月 18 日在 RSS 及网络平台发布，时长约 53 分钟。节目聚焦强化学习奠基人、《苦涩的教训》（The Bitter Lesson）作者 Rich Sutton 与其阿尔伯塔大学前学生 Khurram Javed 联合创立的 Oak Lab，深入探讨为何主流 AI 模型在部署后便停止学习，以及他们如何开创从自身经验中终身持续演进的全新智能体范式。 ## TL;DR Rich Sutton 与 Khurram Javed 指出，真正的学习本质上必然是持续的，当前行业人为割裂训练与推理并在部署后冻结权重的做法偏离了心智规律，而过度依赖人类规则的“合成数据”在大世界假说面前更是走入了死胡同。他们强调大语言模型虽是了不起的科学突破，但仅代表约四分之一的智能，真正的系统必须具备自主发现抽象概念并基于自身经验构建世界模型进行规划的能力。针对深度学习中的灾难性遗忘痛点，他们通过“持续反向传播”（Continual Backprop）与步长元学习给出了根本解药，并指出主流大厂因短期商业回报被困在只知堆砌算力的“局部最优”陷阱中；Oak Lab 则致力于在 5 至 10 年内以极度精简且高度一致的先锋团队，打造一个仅耗能 20 瓦、能终身持续学习并保持内在认知连贯的万亿参数活体心智。 ## Atmosphere & style 整场对话洋溢着一种兼具科学史厚重感与先锋颠覆性的智识魅力，氛围深邃而笃定。经历过生死考验与多轮 AI 兴衰周期的 Rich Sutton 表现出极具感染力的黑色幽默与从容自信，与青年学者 Khurram Javed 配合默契，以师徒创业者的默契对当下硅谷主流范式展开了一针见血的剖析。面对投资人的追问，两人的回应既无浮躁的商业宣发味，亦无学究的晦涩空谈，平静温和的语调下涌动着推翻行业共识的磅礴雄心，让听众仿佛置身于一场正在重构下一代通用智能底座的宁静思想革命。 ## Bulletin - `00:00` 强化学习先驱 Rich Sutton 与 Khurram Javed 联合创办 Oak Lab，致力于推翻主流大模型在部署后冻结权重的做法，打造在部署后仍能基于自身经验终身更新权重的智能体。 - `10:53` Sutton 严厉批评合成数据是“巨大错误”，指出真实世界的复杂度呈指数级超越任何模拟器（大世界假说），强行让人类充当数据优劣裁判无法让 AI 真正自主。 - `40:00` 团队提出在《Nature》发表的 Continual Backprop 算法，结合步长元学习与特征空间“生成-测试”动态注入随机单元，证明灾难性遗忘“完全可以治愈”。 - `45:48` Oak Lab 设定了 5 至 10 年工程目标：开发一个仅需 20 瓦特人脑级功耗即可运转的万亿参数持续学习智能体，并直指主流大实验室因陷入商业局部最优而无法承受新范式的阵痛期。 ## Key takeaways - `00:00` 学习的本质必然是持续且动态的，当前行业将预训练与部署推理生硬割裂开来是背离心智规律的怪异现象。 - `07:44` 《苦涩的教训》核心在于利用随算力增长而自动扩展的通用算法，而非排斥精妙算法设计或依赖人工知识堆砌。 - `10:00` 大语言模型成于通用算力扩张，但过度依赖有限的人类文本与经验，这反而构成了智能向更高维度进化的枷锁。 - `10:53` 合成数据方案依旧受制于人类工程师的评估瓶颈（大世界假说），必须由智能体在无限复杂的物理世界中自主获取直接经验。 - `15:53` 当前的仿真模拟器依赖人类工程师手动修补 Sim-to-Real 鸿沟，真正有效的模拟应当是智能体基于自身经验在内部自主构建的世界模型。 - `17:35` 先验知识与自主学习并非对立阵营，生命智能的本质是以现有先验认知为基石，通过学习获取经验并持续螺旋上升。 - `22:36` 仅靠上下文学习（In-context learning）只是改变了模型的激活状态（State），并未改变模型权重与底层长期认知结构。 - `22:36` 集中式离线批处理微调会让不同用户的定制化偏好在混合梯度更新中互相稀释与抵消，无法实现真正的个性化智能。 - `24:58` 神经科学中丧失本体感觉后依靠视觉重构行走策略的病例表明，生物心智具备推翻并彻底重构数十年既有权重的终身可塑性。 - `27:10` 动物从不进行监督学习，学校的知识灌输只占智能的一小部分，底层肌肉运动控制和环境适应依赖莫拉维克悖论揭示的自主闭环交互。 - `33:04` 自主提炼全新抽象概念并基于自建世界模型展开规划的能力，在当今所有基于固定规则或旧知识复读的 AI 系统中完全处于缺失状态。 - `36:47` 在线流式交互下的灾难性遗忘是核心算法缺陷而非工程部署问题，需依赖从底层同步训练的持续反向传播机制方能彻底解决。 - `42:55` 真实心智在经验涌入并持续演变的同时具备自我维持机制，能在不被人为参数冻结的前提下长期保持思维的内在连贯性。 - `45:48` 主流大厂被短期商业回报和既有 Scaling 范式锁死在局部最优陷阱中，无法承担探索新范式初期必然遭遇的性能下滑阵痛。 - `50:00` 大语言模型是语言处理领域的划时代科学突破，但流畅的语言交互能力在整体智能版图中仅占约四分之一。 - `51:00` 未来的智能生态不是一个包罗万象的单一庞大模型，而是一套统一的持续学习底层设计演化出适应不同环境的多元心智群。 ## Detailed walkthrough ### `00:00` 不是我太激进，而是整个领域变奇怪了 Rich Sutton 开篇便直言，外界常常觉得他的观点过于激进，每次向他提问时，开头总免不了强调他的思路与主流同行有多么格格不入。但他自己完全不这么看：“我反倒觉得自己思考问题的方式很普通、很正常，反而是其他人变得有点奇怪。”他补充道，这种怪异也就是近几年才蔓延开来的现象。在如今这场全方位的 AI 狂潮爆发前，根本没有人会特意去强调所谓的“持续学习”（continual learning），因为在过去，谈论一种“非持续”的学习本身就是不可思议的——学习本就该是持续不断的，智能体在行动中学习，在交互中感知，这是理解心智最自然不过的方式。 红杉资本的主持人 Sonya Huang 与 Alfred Lin 随后隆重介绍了 Rich Sutton。作为强化学习（Reinforcement Learning, RL）的奠基人，Rich 不仅合著了被誉为领域圣经的经典教材，培养了包括 AlphaGo 关键作者 David Silver 在内的一批顶尖学者，更在 2019 年撰写了震撼整个 AI 界的思想纲领《苦涩的教训》（*The Bitter Lesson*）。本期播客中，Rich 携手他在阿尔伯塔大学（University of Alberta）的前学生兼联合创始人 Khurram Javed 共同亮相，探讨他们共同创立的新实验室 Oak Lab。对话将从《苦涩的教训》与大语言模型（LLM）的现状展开，进而深入剖析 Oak Lab 旨在让模型从自身经验中不断学习的全新研究范式。 --- ### `02:18` 2003 年的生死边缘与阿尔伯塔的坚守 主持人将时间推回数十年之前：在深度强化学习尚处雏形、整个领域依然笼罩在“AI 寒冬”的低谷期，Rich 为何能在阿尔伯塔大学咬牙坚持，将其打造成强化学习的全球重镇？Rich 的回答极其笃定而质朴：“不然你还能做什么呢？我们一生都在试图理解心智，而学习正是心智的核心，拥有目标也是心智的核心与智能的核心。我只是在把自己一直在想的事情加倍坚持下去而已。” 当被问及当时身边的人是否觉得他疯了时，Rich 吐露了一段充满黑色幽默却无比动容的真实经历： > “2003 年的时候，我其实病得很重，当时我快要死了——我得了癌症。但我又没完全死成。我抗争了好些年都没死掉，病情反倒又一次缓解了。于是我想：‘好吧，既然我还没死成，而且这日子拖得够久了，那我不如重新找份工作。’所以我去了阿尔伯塔大学教书。结果最后我竟然活了下来。我现在可以拿它开玩笑，但当时真的很凶险。” Rich 感慨道，当一个人被告知生命只剩几个月时，为什么还会继续钻研这些学术研究？他引用了据传是本杰明·富兰克林的一句名言：“如果你想知道某人为什么做一件事，原因几乎总是逃不开两样东西——不是出于习惯（habit），就是出于虚荣（vanity）。”Rich 坦言，自己当时很可能纯粹是出于多年研究积攒下来的“习惯”，因为在面对死亡时，虚荣已经毫无意义。 - 2003 年正值 AI 寒冬，Rich Sutton 确诊癌症且被认为时日无多，但病情缓解后他选择赴加拿大阿尔伯塔大学任教，并在此后将该校建成了强化学习的研究圣地。 - 面对生命倒计时依然坚持研究的动力，被他自嘲为富兰克林所说的“习惯”。 --- ### `04:22` 重新审视心智：学习本来就不分阶段 Rich 接着延伸他开篇的观点，深入阐述他对心智本质的理解。如果把目光放宽，回顾人类几十年来对心智的思考，共识其实从未改变：学习至关重要，明确的目标至关重要，感知同样至关重要。人类与智能体本质上都是“低层级的存在”，以极高的频率产生具体动作并接收海量感知数据，但同时心智又必须能够在更高层级上展开抽象思考。 在这一连续的物理与认知循环中，智能体无时无刻不在感知与行动，因此也必然无时无刻不在调整自身。Rich 再次抨击了当下将“训练”与“推理”生硬割裂开来的行业惯性： > “你不需要专门去说‘持续学习’，因为谈论一种不是持续的学习根本毫无意义。所有真正的学习都是持续的。它不是一个特殊的阶段，我们始终在行动，也始终在学习，这是最正常的思维方式。我一点都不怪，怪的是这个领域本身——整个领域居然觉得有必要给它起个新名字叫‘持续学习’，而它本质上就是‘学习’。” ```mermaid flowchart TD accTitle: 传统两阶段学习与 Rich Sutton 倡导的持续学习对比 accDescr: 对比当下主流 AI 的离线训练部署模式与智能体在自然环境中感知行动一体的持续学习模式。 subgraph Current["当下的行业做法（人为割裂）"] direction LR A["离线预训练<br/>(冻结参数)"] --> B["静态部署 / 推理<br/>(停止学习)"] end subgraph Natural["心智的本质模式（真正的学习）"] direction LR C["产生低层级动作"] --> D["与复杂世界交互"] D --> E["接收高频感知"] E --> F["高层级抽象思考"] F --> G["参数与认知实时更新"] G --> C end ``` --- ### `05:40` 导师与学生的机缘：从 David Silver 到 Khurram Javed 谈及过去二三十年间如何发掘并培养出像 David Silver（DeepMind 强化学习领军人、AlphaGo 核心负责人）这样推动领域前沿的顶尖学者时，Rich 表现得十分谦逊，笑称自己其实完全没有挑选学生的独到秘诀： > “这正好给了我一个谦虚的机会。我其实很喜欢指出，许多看似伟大的决定往往只是自然而然发生的结果。对于学生，我觉得自己并不擅长挑选，有时候纯属运气好，有时候运气差。其实是 David Silver 挑选了我，而不是我挑选了他。” 随后 Rich 转头看向身旁的创业伙伴 Khurram Javed，询问两人当年究竟是如何结缘的。Khurram 回忆道，这一切同样是一场意料之外的巧合： - Khurram 在阿尔伯塔大学完成硕士学位时，导师并不是 Rich，他当时的计划是硕士毕业后直接去工业界工作。 - 偶然的一次会议中，Rich 恰好在场，有人提到 Khurram 正在做的一项研究，Rich 表现出兴趣，Khurram 随即被拉入合作讨论。 - 双方合作推进得异常融洽且高效，六个月内便取得了突破性进展，这项研究自然而然地演变成了极其理想的博士论文方向。 - Khurram 既没有经过常规的正式申请程序，也从未正式向 Rich 提出“你能不能做我的博士生导师”，两人通过并肩做研究建立了深度共鸣，随后才补办了正式的博士申请流程。 --- ### `06:58` 2019 年写下《苦涩的教训》：跨越数十年的历史透视 主持人将对话引入了 2019 年诞生的现代 AI 纲领性文章《苦涩的教训》（*The Bitter Lesson*）。主持人提问，2019 年是一个很有趣的时间节点：它发生在 ImageNet（2009）和 AlphaGo（2015）证明深度学习威力之后，却在当今基于大语言模型的大规模 Scaling 范式彻底爆发之前，究竟是什么促使 Rich 在那个时刻写下了这篇警世之作？ Rich 解释道，这篇文章绝非对某个突发事件的即兴反应，而是他跨越数十年、观察过多轮 AI 兴衰后的厚积薄发： > “这绝不是对某一个特定时刻的回应，而是对我漫长经历的总结。在《苦涩的教训》成文之前至少一年多，我就写过多版草稿，并在演讲中反复讲述这些想法。我亲身经历过符号 AI（symbolic AI）时代，这篇文章的核心思想很大程度上正是来源于对那一时期的深刻反思。” 符号 AI 当年之所以走向死胡同，正是因为研究者执迷于将人类自以为是的知识硬塞进系统；而《苦涩的教训》所揭示的永恒规律是：永远不要因试图注入人类经验而分心，AI 发展的主线始终在于尊重问题的本质，并依靠能够随计算算力增长而持续扩展的通用方法。 --- ### `07:44` 《苦涩的教训》的真正内核与对当下的反思 主持人指出，《苦涩的教训》如今已成为硅谷与 AI 界被提及频率最高的文章之一，投资人和从业者甚至会用“是否吃下了苦涩教训的药丸”（bitter-lesson-pilled）来作为评价标准。但随着这一概念的爆火，它也极易被外界曲解、滥用和过度简化。 Rich 谈到他最近在社交平台 X 上尝试用 26 个英文单词极简重构《苦涩的教训》的精髓： > “不要被人类知识所分散精力（正如 AI 历史上一再发生的那样）；相反，把重心完全放在那些能够随着计算量增长而扩展的学习方法上，比如搜索（search），比如学习（learning）。” Rich 进一步澄清了外界对该文最普遍的误读： - **不是否定算法创新**：这绝不意味着我们不需要精妙复杂的算法设计；恰恰相反，我们非常需要精妙算法，关键在于这些算法必须是“能随计算算力扩增而自动变强”的算法。 - **扩展算力 vs 扩展人类输入**：算法的扩展性（scaling）必须体现在与算力的协同增长上，而不是建立在人工整理的数据规模或人工先验知识的堆砌之上。 在这一逻辑下，访谈自然引出了眼下最具争议的技术焦点：当今依靠海量人类文本预训练的大语言模型（LLM），究竟是符合还是违背了《苦涩的教训》的核心定律？Rich 随即就此展开了深层次的剖析。 ### `10:00` 大语言模型与“苦涩的教训”：成也规模，困于人类数据 大语言模型（LLM）的发展在 Rich Sutton 看来，既是“苦涩的教训”（The Bitter Lesson）的正面典范，也是其反面教材。从积极的一面来看，LLM 证明了计算规模的巨大威力：系统只需“痛饮整个互联网的数据”，借助算力扩张就能大幅提升能力，完全契合了依靠通用算力而非人工设计规则来取得突破的规律。然而，这种范式随后便撞上了天花板。互联网的数据是有限的，高质量样本很快就会被开采殆尽。 更根本的问题在于，真实世界远比人类记录在互联网上的所有信息都要庞大得多。当系统过度依赖人类既有知识与文本时，人类经验的边界反倒成了系统进一步进化的枷锁。这一现象恰恰再次印证了“苦涩的教训”中反复出现的规律：过度依赖人类先验知识，最终必然阻碍系统向更高维度的智能演进。 --- ### `10:53` 合成数据是一个“巨大错误”：大世界假说与人类瓶颈 面对互联网数据耗尽的困境，当前许多前沿模型实验室（如 OpenAI、Anthropic）将“合成数据”（Synthetic Data）视为延续 LLM 扩展定律的新燃料。但在 Rich Sutton 看来，这一方向走偏了。当被问及合成数据是否属于利用算力的通用方法时，Rich 直截了当地回应：“不，那完全是一个巨大的错误（That's just a big mistake）。” Rich 和 Khurram Javed 指出，他们在阿尔伯塔大学探讨了五到十年的核心理念是“大世界假说”（The Big World Hypothesis，Khurram 曾就此撰写过专门论文）。该假说认为，真实世界是无限宏大的，存在无穷无尽需要学习的事物。合成数据虽然可以源源不断地生成，但决定数据质量好坏的评判标准却始终掌握在人类专家手中。这就导致整个范式依然受制于人类瓶颈： ```mermaid flowchart TD accTitle: 合成数据与自主经验学习的路径对比 accDescr: 对比由人类主导的合成数据闭环与智能体基于自身经验的持续学习闭环 subgraph HumanLoop ["传统路线：人工合成数据（受限于人类瓶颈）"] H1["人类专家定义规则/评估指标"] --> H2["生成小规模模拟数据"] H2 --> H3["离线训练模型"] H3 --> H4{"遭遇现实物理/未定义场景"} H4 -- 暴露Sim-to-Real鸿沟 --> H1 end subgraph AgentLoop ["大世界路线：自主经验与持续学习（打破瓶颈）"] A1["智能体置身无限复杂的真实世界"] --> A2["采取行动并产生直接具身体验"] A2 --> A3["自主学习并修正内部世界模型"] A3 --> A4["持续更新局部近似，终身适应"] A4 --> A1 end ``` - **人类假期思想实验**：Khurram 提出一个尖锐的问题——如果 OpenAI 或 Anthropic 的工程师全员去度假，谁来决定合成数据的优劣并生成新数据？既然数据并非来自智能体自身的交互经验，这套体系就无法真正脱离人类自主运转。 - **回声定位无人机案例**：若想训练一架像蝙蝠一样依靠超声波回声定位在复杂物理环境中穿梭的无人机，根本无法仅凭写个程序生成 50 个虚拟宇宙的数据就能解决。环境中的真实物理法则、气流扰动以及电机内部的摩擦与磨损，必须由机器人自身在真实世界中积累经验来学习，而无法预先依赖人类专家的知识来无中生有。 --- ### `13:40` 现实的无限复杂度：他人的心智与无法被模拟的物理世界 Rich Sutton 强调，任何由简短程序生成的合成数据世界都只能是“小世界”（Small World），根本无法复刻现实的复杂性。他指着眼前的红杉资本投资人打趣道：他今天来见 Alfred 和 Sonya，最关心的就是“此时此刻你们两位脑子里在想什么”，但世界上没有任何程序或合成数据能够模拟出另外两个鲜活人类的心智状态。真实世界的复杂性正是源于它不仅包含无穷无尽的物理细节，还包含无数其他互相博弈的智能体。 这就构成了“大世界假说”的核心逻辑： 1. **世界远比智能体庞大**：世界的复杂度呈指数级超越任何单一智能体的心智容量，因此任何系统都绝不可能做到全知全能的“最优解”或完美建模。 2. **严峻的近似逼近**：由于容量受限，智能体内部的模型必须做出极其剧烈的近似（Severe Approximations）。 3. **持续学习的必要性**：因为智能体只能在有限时刻处于浩瀚大世界的某一个局部角落，它必须不断根据当前所处环境调整其内部近似，而不是试图在部署前将整个世界一次性装入静态权重中。“这就是我们为什么必须持续学习的根本原因。” --- ### `15:53` 自动驾驶模拟器的反思：从“工程师造模拟”到“智能体自主建模” 针对主持人提出的反驳——新一代自动驾驶汽车公司广泛利用仿真环境（Simulation）进行预训练并取得了极佳成效，Khurram 与 Rich 给出了深度辨析。Khurram 反问：“为了搭建那套高保真模拟器，背后投入了多少名顶尖工程师？”目前的仿真训练之所以有效，是因为背后有一大群人类工程师不断从真实世界收集长尾 corner case，手动修补“仿真到现实”（Sim-to-Real）的鸿沟。 - **人类作为系统的补丁**：目前的仿真流程是人类在充当真实世界反馈与虚拟世界之间的搬运工。一旦上路遇到人类未曾预料的新突发情况，仿真预训练模型依然会束手无策。 - **正确的模拟方式**：Rich 与 Khurram 明确表示，他们绝不反对利用模拟进行规划（Planning），但这套模拟器必须是**智能体基于自身经验在内部自主构建的世界模型**。当智能体自主构建模型时，一旦现实与预期不符，它可以立即通过持续学习进行自我修补，而不需要等待人类工程师发觉并发布下一个软件版本。 --- ### `17:35` AlphaZero、先天与后天之争：先验知识与学习本该是盟友 话题转向“苦涩的教训”中的另一核心论点：摒弃人类知识注入。主持人提到了 Rich Sutton 的得意门生 David Silver 主导的 AlphaGo 及后续完全摒弃人类棋谱的 AlphaZero。当被问及 AlphaZero 彻底击败人类经验时是否感到意外，Rich 坦言自己当时非常高兴，并且感到“深得昭雪（vindicated）”。 Rich 对“先天先验”（Nature）与“后天学习”（Nurture）的哲学关系进行了深刻剖析： - **本非敌对，实为相生**：从原理上讲，人类先验知识与机器自主学习完全不必对立。生命智能的本质就是获取知识并运用知识：智能体先拥有一部分先验知识，通过学习获得新经验，这些新经验随后沉淀为下一阶段的先验知识，如此螺旋上升。 - **历史导致的阵营割裂**：在人工智能发展史中，偏爱人类先验知识的研究者往往执念于让人类设计的特征取胜，从而轻视或排斥自主学习；反之，推崇学习算法的人往往被误解为想要彻底抹杀先验知识。 - **心智的真正形态**：Rich 澄清自己并非“讨厌先验知识”，他真正追寻的是心智的动态本质——智能绝非一成不变的人工先验，而是一个以现有认知为基石、永不停歇地在与环境交互中吸收新知、重塑自我的连续过程。 ### `20:00` 停止学习的模型与“奇怪的”主流 AI 界 Rich Sutton 针对外界认为他“过度执念于学习”的看法做出了犀利回应。他直言，如今整个业界都在宣扬“只要有足够的知识就万事大吉，模型根本不需要持续学习”，这种观点完全是本末倒置：“不是我奇怪，是你们这帮觉得这行得通的人太奇怪了。”主流 AI 路线试图把人类的所有知识硬塞进大语言模型，但这些模型在实际部署运行、与人对话交互时，其权重完全处于冻结状态、根本不再变化。业界甚至宣称这种不再具备任何学习能力的系统可以达到“博士水平的专业经验”，这在 Rich 看来荒谬至极。真正的智能在于不断从新经验中获取知识并构建新结构，而不是在一开始就把权重焊死。 关于未来智能系统的演进形态，Rich 指出，未来的智能机器人不需要每次都从零开始学习。由于数字智能具备极低成本的复制特性，我们完全可以先训练出一个初始智能体，然后将其直接复制分发到各个机器人身上，让它们从各自的环境和任务出发继续持续学习。这样一来，行业就无需每次都花费天文数字的算力从整个互联网重训模型。无论是物理世界中的实体机器人、互联网底层的智能数据包路由代理，还是手机与电脑上能够根据日常交互越来越懂用户的个人助理，它们的核心特征都必须是：伴随实际经验的积累，表现随着时间推移变得越来越好。 ```mermaid flowchart TD accTitle: 传统冻结模型与Oak Lab持续学习模型的演进对比 accDescr: 对比展示预训练后权重冻结的当前范式与部署后持续更新权重的持续学习范式 subgraph ParadigmA ["当前大模型范式（权重冻结）"] A1["大规模预训练"] --> A2["后训练对齐"] A2 --> A3["部署运行"] A3 -.-> A4["权重永久冻结（仅依靠上下文）"] end subgraph ParadigmB ["Oak Lab 提倡范式（持续经验学习）"] B1["先验模型 / 数字副本"] --> B2["部署与实际交互"] B2 --> B3["与世界/用户交互获取经验"] B3 --> B4["持续反向传播更新权重"] B4 --> B2 end ``` - **主流宣称的矛盾点**：行业声称能靠固定权重的模型创造出拥有“博士级专业经验”的 AI，但任何真正的专家都必须在工作实践中持续学习。 - **数字复制与经验接力**：未来机器人无需斥巨资重走互联网预训练全过程，只需复制已有的基底智能体，并从所在节点继续进行终身学习。 - **智能体的存在形态**：学习型系统不仅限于人形机器人，还可以是网络数据包路由算法或手机交互助理。 --- ### `22:36` 权重冻结的死胡同与集中式微调的低效 面对主持人 Sonya Huang 关于主流大模型助手是否算“经验学习者”的提问，Rich Sutton 毫不客气地反问：“你是认真的吗？很显然，它们的权重压根就没变过。”主持人提到模型可以在对话上下文中“记住”用户的偏好，但 Rich 和 Khurram Javed 强调，上下文学习（In-context learning）只是改变了模型的激活状态（State），并没有改变模型本身的结构。构建大语言模型最核心的能力，恰恰在于通过更新权重来形成新概念与认知结构；人们允许模型在预训练和后训练时更新权重，却在模型推向真实世界、开始面对真实用户的那一刻强行按下了暂停键。“我们双方唯一的大分歧就在于：他们不让模型在上线后继续学习。” 以 Cursor 的 Tab 代码补全模型和 Composer 为例，主持人指出这些系统确实会根据用户数据进行更新。然而 Khurram Javed 详细剖析了当前这种更新机制的底层缺陷：Cursor 等工具的做法是收集成千上万用户的补全与接受数据，打包成批次（Batch），然后对全局策略做一次集中式更新。这种模式存在致命的效率瓶颈：当某个程序员希望教会模型自己独特的代码风格或特定项目逻辑时，他的偏好会被放进几十万人的大池子里相互抵消与争夺。“我只想教我自己的模型特定的东西，我根本不想跟另外 10 万个人争夺模型的更新方向，我也不在乎模型是否从别人那里获得了通用的共享知识。”这种把所有人数据混在一起的离线批处理，根本不是真正意义上的个性化持续学习。 - **状态改变 vs 权重更新**：上下文窗口的调整仅仅是在既有计算图上输入不同的 State，模型并不能由此沉淀出新的长期知识结构。 - **预训练与后训练的边界**：Rich 明确表示，做多少预训练和后训练都完全没有问题，错误在于模型上线与用户互动时彻底剥夺了其继续学习的能力。 - **集中式更新的弊端**：将数万用户的遥测数据混合进行离线批次更新，不仅无法实现真正的个人专属智能，还会导致不同用户的需求在梯度更新中互相稀释与冲突。 --- ### `24:58` 本体感觉丧失与人脑极度可塑性的启示 对于“基础技能存在权重中、个性化仅由上下文负责”这一当前业界的主流心智模型，Khurram Javed 予以了否定。他指出，虽然上下文可以作为状态的一部分，但长远来看模型必须能够随时重写自己的权重。为了说明这一点，Khurram 引用了神经科学中关于人体机能障碍的经典研究案例：人类正常行走高度依赖“本体感觉”（Proprioception，即感知肢体在空间中位置的内部传感器），这套由神经传感器驱动的行走策略早在婴儿时期就深深烙印在脑回路的“权重”之中。 然而，有些患者因疾病完全丧失了本体感觉，导致他们在一瞬间彻底瘫痪，因为他们几十年积累的行走底层策略完全失效了。但在两到三年的康复训练中，许多患者竟然通过极其惊人的大脑可塑性，学会了完全依靠视觉反馈（低头看着自己的脚）来重新行走。这证明人脑的底层结构绝非一次性固化：“如果某些已经成立了 20 年的前提突然不再成立，大脑能够彻底重写并废弃原有的权重。”这种能够推翻根深蒂固假设的终身可塑性，正是当前无法自我更新权重的 AI 模型所完全缺失的核心能力。 - **本体感觉障碍案例**：依靠内部传感器构建的运动控制策略一旦因病被毁，整个既有动作策略将彻底崩溃。 - **重构行走策略**：人类患者耗时数年用视觉神经反馈替代本体感觉，成功重构行走底层策略。 - **极其惊人的可塑性**：智能系统必须具备推翻并修改数十年既有认知权重的能力，而非永远受制于训练阶段形成的先验假设。 --- ### `27:10` 动物从不用监督学习，学校教育只是智能的冰山一角 在探讨生物学习对 AI 的启发时，Rich Sutton 强调，自然界是极其宝贵的灵感来源，但绝不能成为束缚 AI 发展的死板教条，这必须严格符合他在《苦涩的教训》（The Bitter Lesson）中阐述的计算扩展原则。紧接着，Rich 抛出了一个让现场主持人颇感意外的论断：“没有任何动物是通过监督学习（Supervised Learning）来学习的，因为从来没有任何外部实体会给动物示范肌肉纤维到底应该如何抽动。”肌肉抽动是生物与环境交互的唯一底层输出，而自然界根本不存在这种直接的监督信号。 面对主持人 Alfred Lin 的反驳——“但我们在学校里接受的不全是监督学习吗”，Rich 展开了幽默而锋利的辩解。首先，学校教育在生物乃至人类的学习历程中只占极小的一部分：人类学会看东西、学会平衡走路、理解物理世界运行规律等最本质的认知，都发生在走进校门之前；甚至学校这种高度组织化的形式也是人类在几百年前才普及的特殊产物，松鼠从来不上学，照样能自如穿梭在树林之间。 退一步说，即使在学校里，真实的学习机制也绝非纯粹的监督学习。当老师提问“法国的首都在哪里”并告知标准答案是“巴黎”时，老师给出的只是感知信号（声波）。从来没有人能够伸进学生的大脑去精准指挥他们的发音肌肉群该如何按顺序收缩来发出“Paris”这个音。系统依然必须依靠自主的动作试验与感知反馈闭环去逼近目标，这本质上是强化与交互，绝非标签驱动的监督学习。 Rich 还提到了莫拉维克悖论（Moravec's Paradox）：人类往往对证明数学定理、下国际象棋这类高阶抽象任务推崇备至，觉得这才是智能的顶峰；然而对计算机来说，下棋和符号证明极其简单，反而是人类和动物视为本能的移动、视觉感知和注意力分配等日常物理交互机制最为困难。松鼠虽然不会证明数学定理、没有 DVD 播放机或 iPod，但它们在复杂非结构化自然环境中的感知运动控制能力，依然是当前所谓前沿 AI 所无法企及的。 - **肌肉抽动论（Muscle Twitch）**：生物输出的本质是肌纤维收缩与动作执行，没有任何教师能对这种底层动作直接贴标签或提供监督梯度。 - **学校知识传授的实质**：听到标准答案并不等于获得了监督学习标签，如何通过发声器官发出对应读音依然是一个自我探索与调优的闭环过程。 - **莫拉维克悖论的重申**：证明数学定理在认知上看似高深但对计算而言相对简单，而像松鼠在树林中跳跃这种日常感知运动与环境适应，才是智能中最深奥、也最依赖非监督经验学习的部分。 ### `30:00` 具身经验与抽象能力：松鼠、火箭与人类特质 在探讨具身智能时，库拉姆·贾韦德（Khurram Javed）举了一个生动的生理学例子：别人根本无法通过直接示范来教会我们如何控制肌肉，因为发声的生理机制因人而异。即使库拉姆完全模仿理查德·萨顿（Rich Sutton）发“巴黎（Paris）”这个词时的嘴唇、舌头和声带运动，发出来的声音也绝对截然不同。从某种意义上说，没有人知道库拉姆的身体该如何发出正确的声音，“只有我自己才知道”。 索尼娅·黄（Sonya Huang）对此提出疑问：感知运动控制等与物理世界接触的底层能力显然依赖于具身经验，但人类之所以伟大，更多在于脱离了底层感知运动的高阶抽象思维。萨顿构建的世界模型是否能够覆盖从感知运动一直延伸到顶层的宏观抽象？萨顿给出了无比坚定的回答：“是的，这就是我们的野心，绝对如此。” 为了说明抽象能力并非人类独有，萨顿顺带为动物的智力辩护，举了松鼠的例子。无论人类在喂鸟器周围设置多少复杂的障碍，松鼠总能摸索出新的跳跃和攀爬路线打破封锁。松鼠不仅能极其精准地计算抛物线轨迹，在坠落时也能实时调整姿态避免受伤，完全不需要人类在思考发射火箭时自以为在做的那种显式心智计算。萨顿认为，人类与其他动物的差距本质上只是“程度上的差异”，人类总想强调自身与动物的不同是一种傲慢（hubristic）的表现。尽管社会、文化和语言赋予了人类巨大的后天优势，但生物底层对物理世界的理解逻辑是相通的。 ```mermaid flowchart TD accTitle: 感知经验与抽象层级图 accDescr: 展示从底层物理世界交互到高阶抽象规划的递进关系，说明动物与人类共有的经验学习基础。 subgraph LowLevel["底层感知运动学习 (Sensory-Motor Learning)"] A["物理世界单流交互"] --> B["个体肌肉与运动控制<br/>（如精准发音、松鼠躲避障碍）"] B --> C["实时动力学反馈与轨迹预判"] end subgraph HighLevel["高阶抽象与世界模型 (High-Level Abstractions)"] C --> D["自主发现的抽象表征"] D --> E["跨时间步长的心智模拟与规划<br/>（如火箭设计、范式转移）"] end ``` 随后，阿尔弗雷德·林（Alfred Lin）支持索尼娅的观点并进一步追问：两三岁的幼儿确实能无师自通地通过经验掌握许多本领，但制造火箭、探索太空这类壮举绝非单靠百分之百的经验摸索就能达成。在火箭发射之前，人类必须先在大脑中展开未经现实经验检验的抽象思考与想象。机器究竟该如何学会想象未曾存在过的事物？ - 萨顿强调，试图割裂人类与动物并过度神化人类独特性的做法是“傲慢的”（"It's hubristic to try to emphasize what we do differently"），二者在智能机制上更多是程度高低的连续谱。 - 动物无需形式化的数学符号系统，即可在毫秒级的时间内解决极为复杂的物理轨迹与缓冲动力学问题。 --- ### `33:04` 积累知识与范式转移：当前 AI 最欠缺的抽象建模与规划 面对关于人类独特成就的追问，萨顿完全赞同规划与心智模拟（planning and imagination）必不可少。库拉姆随即提出了一个思维实验：一万年前的人类是否具有同等水平的智力？如果把一个史前人类带到现代文明中接受教育，他是否也能掌握前沿技能？现场讨论迅速达成共识：人类大脑在一万年间并没有发生显著的生理进化，其本质上是“完全同一台机器”（"Fundamentally the same machine"）。 现代人之所以能造出火箭，是因为社会在过去一万年里沉淀了庞大的知识库，后人可以通过学校的有监督学习在短时间内飞速吸收，而不需要每个人从零开始依靠个人试错去重新发明轮子。库拉姆承认，未来的 AI 系统必须具备通过经验学习人类文化知识的能力。但当讨论推进到爱因斯坦式的“范式转移（paradigm shift）”时，问题的本质就变了：当人类在某一领域推导出颠覆性的新理论时，往往是某个个体在汲取了既有文化知识之后，凭借自身的独特经验，自主提炼出了全新的抽象概念，并运用这些抽象概念进行心智规划，进而拓展了知识的边界。 库拉姆直言不讳地指出，**自主发现全新抽象概念、基于这些抽象学习世界模型并进行规划的能力，在现有的所有 AI 系统中完全是缺失的**。萨顿提醒道，大家千万不要陷入“先验知识重要还是后天获取知识重要”的无谓争论，这两者并非对立关系，而是相辅相成的。但在范式转移的临界点上，系统必须通过自身的经验去探索，因为现存的人类知识本身就受困于旧范式，机器不可能依靠复读人类的旧认知来自动跃迁到新视角。 更关键的技术痛点在于，当今 AI 领域缺乏“自主学习世界模型并利用该模型进行规划”的成功范式： - 在 AlphaGo 等游戏领域，棋盘规则和转移概率模型是预先给定的硬编码逻辑。 - 在精益证明助手（Lean）等数学定理证明中，运算逻辑算子也是形式化已知的。 - 现有某些尝试预测“下一毫秒”物理信号的端到端世界模型，完全不同于人类高度抽象的模型结构；人类的世界模型建立在高度提炼的自创抽象概念之上。 --- ### `36:47` 《阿尔伯塔计划》与终身深度学习的突破口 索尼娅指出，萨顿和库拉姆并不是只停留在象牙塔里清谈弊端的学者，他们具有极强的行动力，这也正是他们联合创立 Oak Lab 的原因。在 2022 年，萨顿曾主导发布了一份包含 12 个步骤的纲领性路线图——《人工智能研究阿尔伯塔计划》（The Alberta Plan for AI Research）。 萨顿解释称，《阿尔伯塔计划》的初衷是将宏大的通用智能构想拆解为具体可落地的研究切片。其中位于前期的“第 2 步：终身深度学习（Continual Deep Learning）”几乎是整个计划的核心基石。萨顿坚信，终身深度学习具有承前启后的决定性作用——只有先攻克这一步，系统才能在不发生灾难性遗忘的前提下持续更新其内部的世界模型；而计划后半部分的全部内容，都是在解决“如何建立正确的抽象”。萨顿特别澄清，所谓的“正确抽象”并非由设计者钦定，因为没人能提前预知环境的真实分布，模型必须由智能体根据其所处的现实世界自主学习提炼。 库拉姆用顶尖运动员的身体感知进一步阐释了这种抽象的内涵：如果去询问职业运动员如何完成某些超高难度的动作，他们往往会发明一套属于自己的小众术语来指代特定的身体协调感知；有时他们独自训练，甚至连名字都不会起，纯粹是一种内在的自发抽象。当今主流 AI 架构正是缺失了这种自主生成状态表征的机制。 针对终身深度学习的现实挑战，索尼娅发问：这究竟是一个算法层面的结构性缺陷，还是仅仅属于工程部署、计算基础设施或数据隐私的范畴？如果只是天真地在模型与用户交互时更新权重，现代工程体系本就可以做到。库拉姆明确驳斥了这种工程论调：**这绝对是一个核心算法缺陷（"absolutely an algorithmic gap"）**。 在传统深度学习框架下，如果采用朴素的在线更新机制——即拿到一个新样本就立即反向传播更新整个神经网络——模型中原先存储的历史知识就会在极短时间内遭受毁灭性的破坏。像代码辅助工具 Cursor 等产品之所以能展示出某种持续优化的表象，是因为它们将成千上万用户的交互日志聚集成超大批次（batch）进行离线混合训练。然而在绝大多数真正的自主智能体应用场景中，系统面对的都是孤立的时间序列单流数据（single stream of data）。在这种流式交互下，直接运行现有的反向传播算法只会带来灾难性的知识崩溃。 - **《阿尔伯塔计划》第 2 步**：终身深度学习被萨顿视为“最关键的解锁步骤”，它直接决定了世界模型能否进行在线连续更新。 - **抽象表征的来源**：顶尖运动员在没有语言定义的情况下也能形成高阶动作抽象，机器必须具备同样的自底向上构建表征的能力。 - **单流数据挑战**：批量离线微调（如 Cursor 等工具的工程做法）回避了算法本质，无法解决智能体在持续与单一直播世界交互时面临的灾难性遗忘。 ### `40:00` 灾难性遗忘的解药：Continual Backprop 与特征空间的“生成-测试”机制 面对深度学习中臭名昭著的“灾难性遗忘”（catastrophic forgetting），Rich Sutton 斩钉截铁地表示这绝非不治之症，而是“完全可以治愈的”（totally curable），关键在于选用正确的算法。他给出了具体药方：首先是“步长优化”（step-size optimization）。在神经网络中，每个权重都必须拥有独立的步长，并通过元学习（meta-learning）来自主调节。在实际运作中，网络中绝大多数权重的步长会被压缩得极小；这样一来，当模型接收新样本并进行更新时，旧有的核心记忆便不会被粗暴抹除，参数更新只会精准发生在正确的位置。 治愈遗忘的第二个核心支柱是特征空间中的“生成与测试”（generate and test）。Rich 指出，单纯依赖梯度下降是非常缓慢的，因为梯度要求系统只能向已知有益的方向移动，缺乏探索性，无法支撑复杂结构的持续生长。为了实现持久的学习，系统必须在特征空间中不断提出全新的特征或隐藏单元，而不受梯度的预先束缚。他特别提到了团队发表在《Nature》上的 Continual Backprop（持续反向传播）算法：经典的反向传播只在模型初始化时注入一次随机权重，随着训练推进，这种由随机性带来的多样性会被彻底耗尽；而 Continual Backprop 则像不断播撒新种子的园丁，持续向网络注入带有随机初始权重的新单元。在这一架构中，持续生成负责探索，而反向传播的运算则充当“测试器”（tester），筛选并保留有用的特征。 ```mermaid flowchart TD accTitle: Continual Backprop 算法机制 accDescr: 展示 Continual Backprop 如何通过持续注入随机单元与步长优化解决灾难性遗忘 subgraph GenTest ["生成与测试机制 (Generate & Test)"] A["持续注入新单元<br/>(随机权重种子)"] --> B["梯度计算与评估<br/>(Backprop 充当测试器)"] end subgraph MetaLearn ["步长元学习 (Step-size Optimization)"] C["每个权重独立步长"] --> D{"重要旧权重？"} D -- 是 --> E["极小步长：锁死核心知识"] D -- 否 --> F["正常步长：吸收新样本"] end GenTest --> G["持久持续学习<br/>(无灾难性遗忘)"] MetaLearn --> G ``` 对于这套算法能否直接嫁接到当前大语言模型（LLM）的扩增中，Rich 给出了审慎而明确的回答：人们无法直接把现有训练好的模型拿来套用这些算法。因为 Continual Backprop 是一套“元学习如何去学习”的机制，必须从零开始（from scratch）训练全新的基座模型。在学习具体知识的同时，算法也在同步学习“未来该如何学习”。一旦双重学习机制在基底中建立起来，模型便能在不断吸收新事物的同时彻底免受灾难性遗忘的困扰。 - **步长微调机制**：通过元学习让大部分网络权重维持极小更新步长，从结构上隔离新旧知识冲突。 - **超越纯梯度下降**：梯度只能沿着已知的下降线行走，唯有“生成-测试”能在特征空间跳出局限，催生更复杂的表征。 - **Continual Backprop 的本质**：打破传统深度学习只在第零步随机初始化的惯例，全生命周期动态注入随机单元。 --- ### `42:55` 从松鼠到客机：全尺度抽象与心智的自我连贯性 当被问及 Oak Lab 同时进行知识学习与学习机制优化的做法是否属于当前 AI 界“最激进”的尝试时，Rich 笑着搬出了他的口头禅：“我没疯，是其他人疯了。”他回顾道，在 2016 至 2018 年间，学界曾有大量研究者探索过类似思路，但他们大多局限在人为设定的“任务分布”框架内；而 Oak Lab 的野心在于从“单一且连续的经验流”（single stream of experience）中实现这一目标，因此其方法具有彻底的普适性。 谈到公司最宏大的抱负，Rich 并不愿用“激进”这个词，而是称其为“涵盖全部知识尺度”（the full spectrum of knowledge）的能力。真正的心智必须能同时处理微观细节与宏观抽象——比如思考如何搭乘飞机从一座城市飞往另一座城市。这种跨越宏大时空的抽象推理其实是人类乃至动物的基本常识，“就连松鼠也懂得运用抽象概念”。Oak Lab 希望用一套统一的方法来处理从微小感知到宏大构想的全谱系知识，并让系统具备“自我维护”（self-maintaining）的能力。 Rich 进一步对比了当前 LLM 与生物大脑在知识维护上的本质差异。现今的大语言模型之所以能保持知识准确，全靠工程师在后训练（post-training）阶段耗费巨大人力进行对齐校准，并在部署时将参数彻底“冻结”（freeze）。然而，真实的人类心智却每时每刻都在改变，经验在持续涌入，但心智内部有一股力量始终维持着整体架构的组织性与连贯性（coherent），让思维收敛在合理的位置，而不是“漂移到精神错乱的境地”（drifting off into crazy land）。打造一个既能不断自我训练和演变、又能长期维持内在一致性的心智，正是他们追寻的终极目标。 - **单一经验流 vs 任务分布**：Oak Lab 拒绝人工切分的多任务分布，坚持模型必须在永不停歇的单一现实经验流中持续成长。 - **全谱系抽象能力**：统一处理感知细节与高级规划，使 AI 拥有类似人类乘坐飞机、松鼠储存坚果时的常识抽象能力。 - **动态连贯性**：打破现代大模型“训练完即冻结”的死板范式，建立在动态重构中依然不崩溃、不谵妄的自我平衡心智。 --- ### `45:48` 20 瓦特、万亿参数与大机构深陷的“局部最优”陷阱 讨论转向了 Oak Lab 极具震撼力的工程目标：在未来 5 到 10 年内，打造一个运行能耗仅为 20 瓦特的万亿参数智能体。Khurram Javed 坦承，以当下的硬件技术来看，这近乎天方夜谭——目前光是将万亿参数维持在内存中，静态功耗就已经远远超过 20 瓦。但这必须放在计算成本降低与能效演进的尺度下来看。Alfred Lin 现场推算道：根据摩尔定律每 18 个月翻一番的节奏，10 年约对应两个数量级（100 倍）的能效飞跃；这意味着如果 10 年后要做到 20 瓦，今天至少需要证明能在 2,000 瓦内跑通这一系统。 Rich 随即自信地指出，只要算法正确，即便在今天，系统的实际能耗甚至可以比 2,000 瓦更低。这引发了 Alfred 的追问：如果现有技术完全可以更高效，为什么主流机构没人这么做？Rich 忍不住打趣吐槽道：“有时候看起来他们就是单纯想把钱烧光……我觉得各大实验室可能是想通过耗费海量能源，来证明自己是‘纯爷们’（real men）吧。”玩笑过后，他指出严肃的根源在于信念：几乎没有研究团队真正相信低功耗持续学习是可行的，而一旦缺乏这种信念，就根本不会有人去死磕背后的底层技术难题。 ```mermaid flowchart LR accTitle: 前沿实验室的“局部最优”与范式转移困境 accDescr: 解释为何主流实验室受限于短期商业回报而无法迈向持续学习新范式 subgraph CurrentPath ["主流范式：暴力缩放 (Scaling)"] A["锁定商业产品"] --> B["算力堆叠与参数膨胀"] B --> C["性能单调提升<br/>(高能耗/静态模型)"] end subgraph OakPath ["持续学习新范式 (Continual Learning)"] D["新算法探索"] --> E["阵痛期：性能短暂下滑<br/>(陷入局部最优低谷)"] E --> F["突破临界点：极低能耗 + 动态心智<br/>(20W / 万亿参数)"] end C -.->|"大厂无法承受短期倒退<br/>被困在局部最优"| E ``` 针对行业现状，Khurram 提出了极具洞察力的解释：前沿大实验室集体被困在了一个“局部最优”（local minimum）中。走向全新的持续学习算法范式，必然要经历一段“情况在变好之前必须先变坏”（get worse before they get better）的艰难阵痛期。在探索新方向的起步阶段，模型绝不可能在第一天就打破现有的 SOTA 跑分记录。大科技公司如今已被商业产品和季度回报死死捆绑，它们的既有范式允许它们只要追加算力就能稳定提升，因而根本无法承受这种短期性能倒退的战略下注。对于 Oak Lab 的未来，Rich 和 Khurram 坚信，如果一切顺利，他们将构建起具备真正抽象推理、规划与持续学习能力的“真正智能”；这绝不会导致人类沦为附庸，反而将让整个人类世界变得更加令人兴奋与多元。 - **能耗推演逻辑**：十年内能效提升两个数量级（100倍），从现今算法验证阶段的约 2,000 瓦演进至人脑级别的 20 瓦特水平。 - **大厂困局（局部最优陷阱）**：成熟团队无法承受新范式探索初期必然面临的性能下滑，因而只能在暴力堆砌算力的单行道上走到黑。 - **Oak Lab 的终极图景**：摆脱冷冻权重的大模型思路，在 5-10 年周期内实现低能耗、持续自我迭代、兼具规划与推理能力的活体心智。 ### `50:00` 大语言模型的真正突破与“智能不仅是语言” 在谈到当前基于大语言模型的前沿实验室时，Rich Sutton 表现出了一种客观与坦诚的态度。他承认这些实验室已经取得了巨大的成功，并且还将继续维持不错的势头。在 Sutton 看来，大语言模型（LLM）毫无疑问是一项“惊人的科学突破”（an amazing scientific breakthrough），它展示了神经网络在语言运用方面的极高造诣。这种突破在过去是完全无法预料的，因为在 AI 发展史上，语言处理长期以来一直被视为符号主义方法（symbolic methods）牢不可破的堡垒，而大语言模型的出现彻底颠覆了学术界对这一问题的认知与研究范式。 然而，真正让 Sutton 感到沮丧的是当前行业看待这项技术的态度。他指出，人们原本应该单纯去庆祝并享受我们在 AI 的某一个子领域中所取得的伟大成就，但现实中，整个领域却迫不及待地让大语言模型“伪装成整个人工智能”（pretend to be all of AI）。Sutton 严正强调，全部的智能绝不仅仅等同于对语言流畅且高效的运用。语言确实是智能非常重要的一部分，但在 Sutton 的估算中，它充其量只占智能的“20% 或四分之一左右”（like 20% or a quarter of intelligence）。在语言之外，还有极其广阔的智能领域尚未被触碰，“我们还远远没有完工”（There's more. We're not done）。 - **对大语言模型的定性**：神经网络对人类语言技巧的掌握是一项划时代的科学突破，彻底打破了符号主义在自然语言领域的长期垄断。 - **智能的真实占比**：流畅的语言交互能力大约仅占全部智能版图的 20% 到 25%，无法代表完整的通用智能。 --- ### `51:00` “统一设计”而非“单一心智”：大世界假设下的多元系统 主持人 Sonya Huang 由此提出疑问：如果未来的探索一切顺利，Rich Sutton 和 Khurram Javed 追求的终极目标，是否是一个能够同时包揽一切的“单一心智”（a single mind）和“单一权重集”（a single set of weights）——既能学会像猿猴一样在树枝间荡漾，又能学会制造太空飞船？面对这个提问，Sutton 给出了一个极为明确且精辟的区分：“它是一个统一的设计，但将会有许多个心智”（It's a single design. There'll be many minds）。 ```mermaid flowchart TD accTitle: 统一算法设计与多元心智生成架构 accDescr: 展示从底层的统一自主学习算法设计，在复杂大世界环境中与不同环境互动，最终衍生出多重具身心智的过程。 subgraph Design["底层核心架构"] A["统一设计 (Single Design)<br/>持续反向传播 + 自主经验学习"] end subgraph Worlds["大世界交互 (Big World)"] B["环境 A：树枝攀爬 / 具身动作交互"] C["环境 B：工程建造 / 太空飞船研发"] D["环境 C：社会协同 / 跨个体交互"] end subgraph Minds["衍生心智群"] E["心智 Alpha<br/>专用经验权重"] F["心智 Beta<br/>专用经验权重"] G["心智 Gamma<br/>专用经验权重"] end A --> B --> E A --> C --> F A --> D --> G ``` Khurram Javed 紧接着阐述了背后的底层哲学与“大世界假设”（Big World Hypothesis）。他解释道，底层的算法架构是同一套设计，但在面对不同环境时，不同版本的心智会基于自身截然不同的真实经历学习到截然不同的知识。在大世界假设中，现实世界的复杂性蕴含着无限多的事物，任何单一系统都不可能穷尽学习无限的知识。Sutton 也曾指出，假设世界上存在两个规模最大、复杂度相当的系统，从数学逻辑上说，由于两者的复杂度对等，它们根本无法完全对彼此建立完整的内部模型。因此，未来的智能生态绝非一个包罗万象的单一庞然大物，而是多个基于统一设计、在各自独特经验中不断自我进化的心智共同体。 - **核心区分**：“统一的设计”（Single Design）与“多元的心智”（Many Minds）相对立于单一静态模型。 - **不可穷尽性**：世界具有无穷的复杂度，单一智能体受限于自身算力与经验，不可能吸收所有知识。 - **模型复杂度边界**：当两个智能系统的规模与复杂度相当时，由于信息容量对等，一个系统在理论上无法完全建模另一个系统。 --- ### `52:05` Oak Lab 的团队组建理念与愿景展望 对话的尾声聚焦于 Oak Lab 的团队建设与未来发展路径。面对团队招聘的提问，Khurram Javed 透露，初始核心团队的人选其实大部分已经在他们的脑海中成型——这些人都是对持续学习、强化学习底层逻辑深思熟虑了很久的学者与同行。与目前硅谷前沿实验室动辄大肆扩张、迅速吸纳数百上千人的打法不同，Oak Lab 将采取一种完全不同的步调，刻意避免在早期迅速膨胀。 Javed 坦言，因为 Oak Lab 探索的是一个全新的研究范式，“在某种意义上，我们雇用的每一个人都必须能够看到我们所看到的世界，而现实中并不是每个人都能看到这一点的”。因此，他们将从极小的规模起步，稳步发展为“一手之数”，或者两三倍于此的体量（大概十几人左右）。正如 Sutton 随后补充强调的，实验室现阶段最核心的诉求是“保持高度认知一致”（super aligned）。只有当极度精简的团队在核心信念和科研范式上形成绝对共鸣时，才能最大化协作效率，并在这个全新的技术轨迹上稳步放大研究成果。 主持人对 Rich Sutton 和 Khurram Javed 长期以来在强化学习与算法机制上的深邃思考表达了由衷的敬意，赞叹这场对谈令人深受启发。Sutton 与 Javed 亦欣然道谢，在对未来持续学习智能的憧憬与默契中为全场访谈画上了句号。 - **招聘理念**：绝不盲目追求团队扩张，优先寻找对非主流新范式具有深刻共识的极少数先锋研究者。 - **发展规模**：初期仅维持在一个巴掌（5人左右）或十几人的紧凑规模，确保团队的“超级对齐”（Super Aligned）。 - **长期效率**：通过极高的人员共识与敏捷性，跳脱主流大模型的局部最优陷阱，以高生产力推动范式级突破。 ## Notable quotes > `00:00` 我反倒觉得自己思考问题的方式很普通、很正常，反而是其他人变得有点奇怪。 — Rich Sutton > `02:18` 如果你想知道某人为什么做一件事，原因几乎总是逃不开两样东西——不是出于习惯（habit），就是出于虚荣（vanity）。 — Rich Sutton > `04:22` 你不需要专门去说‘持续学习’，因为谈论一种不是持续的学习根本毫无意义。所有真正的学习都是持续的。它不是一个特殊的阶段，我们始终在行动，也始终在学习，这是最正常的思维方式。我一点都不怪，怪的是这个领域本身——整个领域居然觉得有必要给它起个新名字叫‘持续学习’，而它本质上就是‘学习’。 — Rich Sutton > `07:44` 不要被人类知识所分散精力（正如 AI 历史上一再发生的那样）；相反，把重心完全放在那些能够随着计算量增长而扩展的学习方法上，比如搜索（search），比如学习（learning）。 — Rich Sutton > `10:53` 不，那完全是一个巨大的错误（That's just a big mistake）。 — Rich Sutton > `20:00` 不是我奇怪，是你们这帮觉得这行得通的人太奇怪了。 — Rich Sutton > `22:36` 我们双方唯一的大分歧就在于：他们不让模型在上线后继续学习。 — Rich Sutton / Khurram Javed > `24:58` 如果某些已经成立了 20 年的前提突然不再成立，大脑能够彻底重写并废弃原有的权重。 — Khurram Javed > `27:10` 没有任何动物是通过监督学习（Supervised Learning）来学习的，因为从来没有任何外部实体会给动物示范肌肉纤维到底应该如何抽动。 — Rich Sutton > `40:00` 灾难性遗忘绝非不治之症，而是完全可以治愈的（totally curable）。 — Rich Sutton > `45:48` 有时候看起来他们就是单纯想把钱烧光……我觉得各大实验室可能是想通过耗费海量能源，来证明自己是‘纯爷们’（real men）吧。 — Rich Sutton > `51:00` 它是一个统一的设计，但将会有许多个心智（It's a single design. There'll be many minds）。 — Rich Sutton ## Mentioned Rich Sutton, Khurram Javed, Sonya Huang, Alfred Lin, Sequoia Capital, Oak Lab, University of Alberta, David Silver, AlphaGo, AlphaZero, The Bitter Lesson, The Alberta Plan for AI Research, Big World Hypothesis, Continual Backprop, OpenAI, Anthropic, Cursor, Composer, ImageNet, DeepMind, Nature, Lean ## Video description Rich Sutton, who helped pioneer reinforcement learning and wrote the seminal AI essay The Bitter Lesson, has now cofounded Oak Lab with his former student Khurram Javed. Their goal: to build agents that continuously learn from their own experience rather than from us. Rich doesn't think he holds a radical view: "I'm not weird. The field is weird." He says all learning is continual, and the field is the one that needed a new name for it. Rich and Khurram argue synthetic data is "a big mistake." Their "big world hypothesis" is that the world is massively more complex than any agent or simulator, so approximations have to be updated continuously rather than frozen at deployment. Rich calls LLMs an unanticipated scientific breakthrough, but says they represent roughly a quarter of intelligence. He says catastrophic forgetting is "totally curable" with the ideas behind their continual backprop algorithm. Khurram explains why the frontier labs can't follow: they sit in a local minimum where a new paradigm gets worse before it gets better. Their target, five to ten years out, is a trillion-parameter mind that keeps learning, stays coherent, and runs on 20 watts. Hosted by Sonya Huang and Alfred Lin, Sequoia Capital [Source](https://pscrb.fm/rss/p/traffic.megaphone.fm/CPUAI4645036827.mp3)

[0:00] Rich Sutton: 我反倒觉得自己思考问题的方式很普通、很正常，反而是其他人变得有点奇怪。 [2:18] Rich Sutton: 如果你想知道某人为什么做一件事，原因几乎总是逃不开两样东西——不是出于习惯（habit），就是出于虚荣（vanity）。 [4:22] Rich Sutton: 你不需要专门去说‘持续学习’，因为谈论一种不是持续的学习根本毫无意义。所有真正的学习都是持续的。它不是一个特殊的阶段，我们始终在行动，也始终在学习，这是最正常的思维方式。我一点都不怪，怪的是这个领域本身——整个领域居然觉得有必要给它起个新名字叫‘持续学习’，而它本质上就是‘学习’。 [7:44] Rich Sutton: 不要被人类知识所分散精力（正如 AI 历史上一再发生的那样）；相反，把重心完全放在那些能够随着计算量增长而扩展的学习方法上，比如搜索（search），比如学习（learning）。 [10:53] Rich Sutton: 不，那完全是一个巨大的错误（That's just a big mistake）。 [20:00] Rich Sutton: 不是我奇怪，是你们这帮觉得这行得通的人太奇怪了。 [22:36] Rich Sutton / Khurram Javed: 我们双方唯一的大分歧就在于：他们不让模型在上线后继续学习。 [24:58] Khurram Javed: 如果某些已经成立了 20 年的前提突然不再成立，大脑能够彻底重写并废弃原有的权重。 [27:10] Rich Sutton: 没有任何动物是通过监督学习（Supervised Learning）来学习的，因为从来没有任何外部实体会给动物示范肌肉纤维到底应该如何抽动。 [40:00] Rich Sutton: 灾难性遗忘绝非不治之症，而是完全可以治愈的（totally curable）。 [45:48] Rich Sutton: 有时候看起来他们就是单纯想把钱烧光……我觉得各大实验室可能是想通过耗费海量能源，来证明自己是‘纯爷们’（real men）吧。 [51:00] Rich Sutton: 它是一个统一的设计，但将会有许多个心智（It's a single design. There'll be many minds）。

[0:00] Rich Sutton: I'm not weird. The field is weird. [25:00] Rich Sutton: No animal learns by supervised learning, because we don't get examples of how our muscles should twitch... Even in school, no one tells us how we should twitch our muscles. [35:00] Khurram Javed: If you take one sample, and then you're going to update your whole model with that one sample, you will run into this problem that now all of the previous knowledge in the network gets destroyed... It completely destroys your prior knowledge in a very destructive way.

[0:00] People think I have a radical point of view sometimes. They start questions saying how what I'm thinking is so different from everyone else. But I don't see it that way at all. I see it as like I'm thinking the ordinary way. It's just everyone else that's thinking a bit weird. And I mean that like, you know, it's just the recent times people are thinking weird. Before there was all this AI craziness, you talk about, you wouldn't have to say continual learning because it wouldn't make any sense to talk about learning that wasn't continual. All learning is continual. We always act and we learn. That's just the normal way of thinking. to call it continual learning. It's just learning. Bye. We are honored to have the great Rich Sutton with us here today. Rich, you invented reinforcement learning. You wrote the seminal textbook. You had the Key students in the field, folks like Dave Silver, you wrote the essay, The Bitter Lesson, that I believe is the Bible of the field. And you have just been one of the greats in propelling the field forward. So thank you for taking the time to join us today. Rich is joined by Khuram Javed, his co-founder and former student from the University of Alberta. The two of you have set off to found Oak Lab. I'm very excited to talk to you about that today. So for today's session, we're going to start talking about The Bitter Lesson, the state of the world as we know it today. whether LLMs will get us there or not. And then we're going to transition to start talking about your research agenda and your plan for Oak. Rich, maybe take us back. I was going to start with a bitter lesson, but I actually want to start earlier than that. Decades ago, you decided to... dedicate your career to reinforcement learning, to deep reinforcement learning in particular, and you established the University of Alberta as a bastion of that, back when I think the field was very much in its infancy. What gave you the conviction to do that? What else are you going to do? We're trying to figure out the mind and learning is a central part of the mind. And having a goal is a central part of the mind. central part of intelligence Yeah, so I was just doubling down on what I was always thinking. Did people think you were crazy at the time? Thank you. Um... It was a winter. It was an AI winter. What year was this? It was in 2003. Okay. And It's kind of crazy, actually, the truth, because I was like really sick. I was dying. I was actually dying of cancer in 2003. And but I wasn't quite dead. You know, I've been trying for a number of years and I wasn't dead. I've seen another another remission. And so, So I said, well, I'm not dying. I haven't succeeded in dying. So I might as well, you know, it's going on long enough. I might as well just try to get another job. And so I went to Alberta and started teaching there. And then in the end, I didn't die. It's kind of amazing. It's because, you know, it's like that. I'm joking about it now, but it was quite serious. It's an even more poignant question. Why did I continue to work on this research stuff when I only had a few months? Keep reminded what I think it's Benjamin Franklin is supposed to have said that, you know, if you ever wonder why someone is doing something, it's almost always one of two things. It's either habit or habit. vanity okay so I think it was I think that's probably true maybe it was a habit to just kept doing what I always been was doing or maybe it was a vanity I think it was more like habit because I was dying. Wow. Divine intervention. It's always been easy for me to be very determined And I'm going to go even longer on this answer. Please do. People think I have a radical point of view sometimes. They start questions saying how what I'm thinking is so different from everyone else. But I don't see it that way at all. I see it as like I'm thinking the ordinary way. It's just everyone else that's thinking a bit weird. And I mean that like, you know, it's just the recent times people are thinking weird. If you look back, what people thought about the mind for, you know, even just a decade, you'll find the kind of thoughts that, you know, learning is important. You've got to have a goal. Yeah. And, you know, perception is important. We are low-level beings. We are generating actions and perceiving data at a fast speed, and yet we have to think at higher levels. And, you know, go back a few, before there was all this AI craziness. [5:00] you talk about, you wouldn't have to say continual learning because It wouldn't make any sense to talk about learning that wasn't continual. All learning is continual. You know, you... It's not a special phase. We always act and we learn. That's just the normal way of thinking. I'm not weird. The field is weird to feel they need to call it continual learning. It's just learning. I'm not weird. Everybody else is. That's a good motto to live by. We're going to have to send that out. Next post about that. We're very happy that you lived on. The field is happy that you lived on. And thank you for pushing the frontier of AI. I'm really happy. Thank you for pushing the frontier of AI. I'm sure really happy. And thank you for all of that. And you've been able to educate a lot of students who push the frontier as well. Thank you. How did you pick them over the last 20, 30 years? Oh, you are giving me opportunities to be humble. Like, I like to be humble and point out how all these great decisions just happen. And that's the way I feel about students. I don't feel that I choose them very well. Sometimes I'm lucky, sometimes I'm unlucky. I don't feel I'm particularly good at picking my students. I'm looking at Kerm now. I think sometimes you end up with really great ones. David Silver picked me. Yeah. How is it that I got you, Kerm? Yeah, that was also... So I... Finished my master's, not with you, and I was planning to join industry. And then we were collaborating on a project, which also just started organically. There was something I worked on that Rich was in a meeting. Then they mentioned that I worked on it. So I got pulled into it. We started collaborating. It went really well. I felt so happy with that collaboration. Rich also felt really good about it. And then six months down the road, we had made some progress. It just made sense to convert that into a thesis proposal. So no point did I apply. At no point did I ask, should you be my PhD advisor? We worked together. Then we decided this would be a pretty good thesis. And then after that, I applied for the PhD. Life works in unexpected ways. Take us to 2019. You wrote The Bitter Lesson, which has become the modern tome. 2019 was a funny time to be writing that piece because ImageNet was 2009... AlphaGo was 2015. What caused you in 2019 to reflect and to write that? Because it was before the current kind of scaling paradigm around large language models had taken off. But it was after deep learning had really proven itself. Well, it was a long time coming. You know, as the bitter lesson expresses, it's something that you can observe for a long time, for many decades. And it's definitely at least as much due to the round of symbolic AI, which I lived through. It's all about... Not getting distracted by trying to put in your human knowledge and just paying attention to what the problem needs and how you can scale with computation. I know I wrote versions of it at least a year before, and I gave talks. I gave a talk a year before. And, um, It wasn't a particular response to the moment. It was a particular response to my long experience. Different people trying to think in different ways. about how you can make smart systems. So, What is the essence of the bitter lesson? The essence of the bitter lesson. It may be the phrase that I hear used the most in my meetings these days. Is this bitter lesson pilled? Is it not bitter lesson pilled? I would imagine, given the popularity of the phrase, it's probably been tortured and misused in different ways that you didn't originally intend it. So what is the essence of it, and where do you think people go wrong in their attempt to understand it? Yeah, you're making me think about X now. I recently made a post where I tried to do the bitter lesson in 26 words. It goes something like, don't be distracted by human knowledge. as AI traditionally has been many times, instead focus on learning methods that will scale with computation like search and like learning. So it's really all about focusing on algorithms and improvements. It's not saying you don't need fancy algorithms. You need fancy algorithms, but you want fancy algorithms that will scale with scale with computation. rather than scaling with data. rather than scaling with human input. Thank you. Yeah, and then the question, if I can anticipate it, Yeah, what about large language models? Are they consistent or inconsistent with your essay? Yeah, and I thought about this and [10:00] I think there's another expo about it, but the conclusion is that it's both a positive example and a negative example of the bitter lesson. First, uh, Large language models enabled a enormous scaling with computation. And you just... drink in the internet and scale so much. So it was a way of getting a much more capable system just by Methods of Scale Then after that, as you go on further, it eventually gets limited by that information. The Internet is finite and it's hard to get more examples. and the world is big, and the world is massively bigger than everything we stored on the internet. And so in the end, It seems like it could be I guess that would be a positive example of when we relied too much on human knowledge and it eventually holds us back. Can I just push on this a little bit? Yeah. Yeah. It seems like a lot of what the foundation model labs are working on right now is synthetic data generation in order to kind of get us beyond... the fossil fuel that is the existing human internet. Is synthetic data generation kind of as part of this LLM scaling paradigm, is that? A general method that leverages computation? No. Why? That's just a big mistake. Why? Well, it's such a big, maybe it's the next big lesson. It's been floating around. Alberta for five or ten years okay and we call it the big world perspective or the big world hypothesis Kurem who eventually wrote it up as a paper there's a little paper called the big world hypothesis The big world is that... The world is infinitely big. There are infinitely many things to learn. And you can have people generating these synthetic datasets, but there would always be more things to learn. And because of that, if you could just... learn from experience, you could remove the humans from the loop, then you would have systems that can do everything because, you know, the world is big. There are many tasks that we want them to do. and they would be able to do anything by learning from their experience. Going back to the synthetic data question too, who decides what's a good synthetic data and what's the bad synthetic data? Because I can write a program that can output a lot of synthetic data, which would hurt programs, Right now, I would say humans decide, and that's the bottleneck where, okay, you can have humans deciding how to generate these data sets, but you need human experts who know what's a good data set and what's a bad data set for that approach to scale. So it is bottlenecked by humans. Doesn't my loss curve decide like how much better did I get with this data set versus that data set? Right. But if all the engineers, OpenAI Anthropic or all the big neolabs, the engineers went on vacation, who would generate the synthetic data? That's the question. It doesn't, It doesn't come from agents' experience. It's not something that the agent is generating itself. some human has to decide what is the right synthetic data to generate? And that requires human expertise. So for example, If you want, a system to do something very challenging from a physics point of view. Maybe you want a drone that flies with echolocation, like a bat, for example. What's the right synthetic data for that? I think you would need to hire domain experts to go figure out what is the right data and generate it, and then maybe you would be able to learn from that. but the domain expert has to exist first. So we are bottlenecked by human expertise at that point. But you can have infinite synthetic worlds. The existing world is finite. But let's go back to the echolocation thing, right? That's what I want. I want a drone that can localize itself and move with echolocation. That's my goal. The robot that is a robot is generating its own experience so it could totally learn from its own experience. but it wouldn't be able to. It doesn't matter how much synthetic data you generate. It doesn't matter if you generate synthetic data that captures 50 different universes. It would not allow you to do that task without humans figuring it out first. First, just say the synthetic data is wrong. I mean, it won't be correct. It'll be a synthetic world. It won't be the real world. Thank you. And it will matter. The world is incredibly complex. If you write a little program, because it's going to be a little program that will generate the synthetic data, It'll be a small world. Hmm. So, for example, what's important to me is what's going on in your mind right now. Okay, and you're saying, why don't I get some synthetic data to tell me what's going on in other people's minds? No, there's no way we can have synthetic data for other people's minds. other people's minds matter to us you know like I talked to you guys about investing today so I care what you going on [15:00] in your minds, How can I get synthetic data on such a thing? really can't get synthetic data on anything you can't get synthetic data on on how the drone is going to interact with with its environment in the physical world and the the the the friction and wear in the motors of this robot, the world is infinitely complex. and any simulation of it is like microscopic. The big world hypothesis, let's say what it is, is that The world is massively more complex than your mind, than any agent. And this is obvious because the world contains many other agents. So because the world is massively complex, there's no way you can do anything like a anything that might claim to be optimal or perfect, you're going to be imperfect and you have to have approximations. And those approximations will be severe. And so because of that, that is ultimately the reason why we have to continue learning. If you want to think of it as a reason, we have to continue learning because we'll encounter some particular part of this immense world. And we'll have to learn an approximation that's tuned to the part of the world we're in, not to all the other parts that we're not in. Thank you. I'm going to push on this one more time. And sorry, I'm being argumentative for the sake of being argumentative. I'm trying to understand. My understanding is that the newest cohort of self-driving car companies is Many of them were primarily trained in sim. And then they, you know, have to do some sort of... post-training, I guess, to make sure they work in the real world, but that it's been a very effective pipeline. Yeah. So I think the important question to ask here is, How many engineers were involved in building that simulation? And are we ready to say that the only problem worth solving are those where we can hire a large team of engineers to first make a simulation. And I'm sure they had to do multiple iterations where they made the simulation, they learned it, they realized there was a sim-to-real gap that was not acceptable. Then they fixed it. So there is this human in the loop fixing the simulation. Like they are getting feedback from the real world, humans. And then they are fixing the simulation. Why can't we just remove the human and let the agent do it itself? And then when it actually drives, again, something unexpected will happen. And that's when you actually want to learn from experience. So your point is there's just so much more data that's going to come from experience than there possibly can be. be from humans curating, creating data. Yeah, and I think there is obviously value in... learning from simulation. And there is a way of doing it. The agent can learn a model from its own experience And when the agent learns it, it's much better because if the model is incorrect, it can fix it by continuing learning. If the humans are making a simulator, then the model only gets updated when the humans figure out that something's wrong. Yes, planning is important. The agents should learn from simulators, but simulators they make themselves. Okay, I want to move to another part of the bitter lesson, removing human knowledge. from your essay, quote, seeking an improvement that makes a difference in the shorter term, Researchers seek to leverage their human knowledge of the domain. But the only thing that matters in the long run is the leveraging of computation. And if I... your former student Dave... with AlphaGo and AlphaZero. For me, that was an example of a triumph of removing human priors. Did that result surprise you? I guess why or why not? Of course, it made me very happy and made me feel vindicated. Um, You know, it could have gone either way. It wasn't that... Because prior knowledge can help. You know, there's nothing wrong with prior knowledge. You know, and I say this right at the very beginning of the bitter lesson. I say there's no reason why there has to be a conflict between prior knowledge and then learning knowledge. You know, you can put some prior in there and then start learning. There's no reason in principle why these have to be opposed. In fact, they're all about knowledge. You know, life is gaining knowledge and having knowledge. And why are these, how somehow, you know, nature and nurture became enemies. Thank you. Prior learning is what you already, then you learn more. And they should be friends. But, as I say at the beginning of the Bitter Lesson, in practice they have been enemies. In practice, people who had an affection for existing human knowledge. ended up wanting that to win. And so they wanted to minimize or dismiss learning. And so now I'm sure your sense of me is that I'm someone who loves learning and wants to dismiss prior knowledge. But, you know, I'm really someone who's interested in the mind. The mind is you have prior knowledge and then you get more. And then once you've gotten more, then that becomes your prior knowledge as you get more and more and more. And [20:00] These two things work together. I end up appearing to be someone who's interested in learning primarily because all the rest of the world is talking about all you need is enough knowledge. You don't need to learn. you know, large language models are, we're going to put all this knowledge into the system, and the large language model will not learn. when it runs. You know, it's talking to people, it's interacting. It is absolutely the weights never change. So, You know... I am not the weird one. It's you guys that are the weird one that think that that's possible, that you could possibly, you know, they claim they can make a PhD level experience and expertise out of something that doesn't learn at all anymore. You know, so... You know, I'm not the weird one. So your recommendation is let the algorithms run for a much, much longer period of time before feeding it. Continually learn. Yeah. before you feed it data. prior data So drip or drip the prior data along the way. Prior knowledge. So, both are important, but in the long run, you've got to gain and structure the gaining of new knowledge. That's what all that matters in the long run. And as you are doing this, yeah, there'll be some... God previously... Um... Like how it works if we look into the future when we have intelligent robots and Will we... have them all learn from scratch. Or will we copy them and ask them to keep learning from wherever they are? I mean, they'll be digital and it'll be easy to copy them. And so instead of having like this huge thing where we're spending zillions of dollars to retrain them from the Internet, we'll just copy them. the agent and keep learning from there. And so in some sense, the prior knowledge will be, should be dismissed because you're just going to copy it from the previous robot. So why don't you just describe for us what you think a machine or a computer that learns from experience looks like? Well, it could look like a robot. It also could live entirely on the Internet. You could like, for example, routing a packet through the Internet and do that in a way that's sensitive to experience and becomes better over time. Or you can interact via the user interface that's interacting with people, like on your phone or on your computer, and... It becomes better over time. Like an intelligent assistant has to become better over time, has to know what you want. Would your contention be that the current paradigm of... you know, the popular LLM-based assistants, would your contention be that These are not experiential learners or continual learners? And if so, what is the fundamental gap? Are you serious? I mean, obviously... They learn memories about me... they're You know, they're doing some in-context learning. Their weights never change. And by the way, is a small number of the weights changing sufficient or do you need all the weights to be changing? Well, so I'll think of all the structuring and generation of new concepts that went into creating the large language models. All that is the weight learning. the And you want to be able to continue doing that. You don't want that to happen just once. Is another way of saying it is we do too much pre-training and post-training before we launch the models? They don't learn after that. The only point that the big disagreement is we don't let them learn after that. Yeah, we don't let them learn after that. We can do as much pre-training as we want. That's okay. Post-training is fine. But then when I'm using the model, I cannot stop learning. You can give it more context. You can change the state of the model by giving it more context. And so it has already learned that if the state is different, if the state says something new, then it will use that to make the next prediction. But the model is unlearning. Cursors, tab, autocomplete model. It is, you know, it does get updated based on... Those models, those ways change. Those ways change. Those are two examples. Cursors, tab, and I think the composer, they were also updating. Those are two examples of continual learning. But... it can be much better. So the way they do it, as far as I understand, is a lot of people are using tab. They collect all this data. So coming from millions of users or thousands of users, and then they do one update of the policy from this patch data. So this could work, but imagine I want to teach this model something specific. I don't want to fight with 100,000 other people is about what they want to teach their models. I want to teach my model something very specific and I want to do it. to my version of the model. I don't care about the shared knowledge that the model has coming from other people. And so it's a very inefficient way of doing it. It seems like the way that this is currently done is that there's fundamental skills maybe that are learned now. in the weights that are common to everybody. [25:00] And then there's personalization that happens in the form of context, right? Yeah, yeah. Is that not the right mental model for how learning should work? Like, should should all the context live in the weights themselves? So, context can be in the state too. Could be both. But, you still need to be able to update the weights. If I give you an example, some really good use studies are with human disabilities, when go through something that changes their mind or some sensors, you can see them adapt. So for example, we have proprioception, we have internal sensors that tell us where the, how the body is positioned, and we use this for walking. There are cases where people lose this ability completely, and then they can't walk at all, because that is literally the foundation of their walking policies. It is ingrained in the brain. But then over the course of two, three years, they can learn to walk again by looking at their feet. So visual feedback. through that. So brand is insanely plastic in the sense that it can learn a lot of things. Something that has been true for 20 years, when it stops being true, it can go and update that and get rid of that. And that is the capability I think that's extremely useful. we would want in our systems. What is there for us to learn from how human babies or animals learn, and how much inspiration do you take from that? Well, we take a lot of inspiration. We don't take it as a requirement that the AI has to behave like the natural system, like babies or people or animals. But it's a source of inspiration. Inspiration, but not constraint from animal learning. Be consistent with the bitter lesson. Yeah. Where do you think we should... most seek to draw inspiration from the way that biological learning works. That is not present in today's systems. I feel like I'm just giving opinions now, but they're just obvious opinions. So I think it's... apparent that No animal learns by supervised learning. Because... we don't get examples of how our muscles should twitch. And that's our output. But all of school is supervised learning. I know, absolutely not. But even if it was, school is like a tiny fraction of what we learn. Like we learn to see and we learn to walk. And we learn... How the world works. But even in school, no one tells us how we should twitch our muscles. The knowledge skills I acquire... We're from supervised learning in school. So I don't want to say that learning from others, transmission from others is not important. It's like extremely important and language is extremely important. But what are we... What are we missing? You know... There is no... supervised learning. There's no targets that are given to us. You hear the right answer is, you know, where is what's the capital of France and we know the answer is Paris. Okay, but no one tells me how I should pronounce Paris. You say the answer is Paris, and I listen to you, and I hear your words, and I will make some other questions. muscle motions to produce the answer Paris, It's not literally supervised learning. Thank you. Um, Anyway, yeah. So I think it's really true. I mean, well, anyway, the first thing is the school is irrelevant. Like, you know, squirrels don't go to school and learn that. They might. Animals don't learn that way. And school is a very special thing that even we didn't have up until, you know, I don't know, a few hundred years ago. But isn't it something we've evolved to? which we didn't do as animals. I wish you had been around to tell my parents that. It forced me to go to school and deal with all the structure. The thing is, squirrels are wonderful at jumping off trees, but squirrels can't prove math theorems. And if I want to learn how to prove a math theorem, I go to school. Yeah. They also don't have... DVDs and iPods. You know, a lot of things. They can do things that we can't do. But, math theorems Yeah, and they don't play chess. It's sort of like more of X Paradox. There are these advanced things that we think of as really intelligent. But... They're sort of easy for computers to do as opposed to all these regular things that are hard like moving and seeing with attention and everything Um... I think supervised learning is a good thing. You know, just mentions, I like to think, look for obvious things. No one tells us how to twitch our muscles. [30:00] by giving us examples because they couldn't possibly because we have had to twitch our muscles. We've had to figure that out. Yeah. And their answer would be wrong, right? So if I moved my mouth and my tongue and my vocal cords exactly the same way that Rich does to pronounce Paris, I'm sure a very different sound would come out. So in some sense, which or no one knows the right way of producing a sound with my body. Only I know that. Yeah. It seems to me that many of the most, I guess, the most raw, like, sensory motor capabilities... especially related to movement in the physical world. I agree with you that that seems... something that inherently learns from experience. It seems to me, though, that there are higher levels of abstraction that bring us closer to, you know... what makes humans great. And much of that... doesn't live in this low level of sensory motor learning. Does your world model, I guess, span sensory mother learning all the way up Yeah, that's the ambition. Absolutely. And squirrels, by the way, can do some enormously abstract things. What's the coolest thing a squirrel can do? Well, it can always get into your bird feeder, no matter what obstacles you put in the way. You know, it can find new ways to jump and climb and do lots of things. You calculate trajectories pretty well. Animals are pretty good at understanding the physical world. without the mental calculations that we think we are doing when we think about launching ourselves into space. Breaking a fall, they can do it in real time the right way to prevent injuries. Okay, fair enough. I think it's just a question of degree between... And I like to think that other animals are very close to humans. I think it's hubristic to try to emphasize what we do differently, how we're different from animals. It's better to see the commonalities... And I think we are just a question of degree. It's degree and, of course, society and culture give us big advantages. Language gives us big advantages. Can I just push... on some of this. Yeah, good. Because I want to back up Sonia. So I believe... animals and children... learn from experience and do incredible things learning from experience and When my son was two or three or four, I'm like, wow, this is really interesting that my son can learn these things without nobody... really teaching them how to do these things. But at the same time, what Sonia is saying is like, what makes human uniquely human, to be able to... go to outer space. Build a rocket. Those are not things that are learned properly. 100% from experience because before you launch the rocket, you actually have to abstract thinking through it in a way that is not learned from quote unquote experience. because you don't know if it's going to work or not. You have to imagine it. How do we teach a machine to imagine things that were not available before. That's probably the thing that we're trying to push on because we're not quite understanding that. We're absolutely going to agree with you there. You have to be able to plan. You - Yeah. - Would you say that humans a thousand years ago, before they had done most of the things that we were talking about, were they as intelligent? if for example someone from that era was exposed to this new culture would they be able to get the same skills and start doing useful things. Even over the last 10,000 years. I don't think... Human brain doesn't evolve that much. Fundamentally the same machine. Fundamentally the same machine. but we've built up 10,000 years of knowledge. Yes. And I get to learn 10,000 years of knowledge by going to school through supervised learning. Right. And I get all that much, much faster than trying to learn through experience. Right. So I think you're totally right. So we would want our systems to learn from experience and part of their experience would be getting exposed to our culture and then learning about our culture. They should learn from that. That's all good. But let's talk about when someone goes So everyone gives the example of Einstein, but I think there are many examples. Learning is that too, like looking at learning thing versus programming things. So when these paradigm shifts happen, I would say it's a human who has accumulated all this knowledge. And then from their experience, they're building new abstractions, they're planning with them, and then they're discovering new knowledge. And that skill of coming up with new abstractions and then learning more modern and planning with them, that problem is, that skill is totally missing in our current systems. And you can expose this at the edge of human knowledge, but you can also study this problem at the sensory motor stream level. So we're not arguing with the... With the principle, we need to form abstraction so we can reason at a high level. You guys are coming close to doing that thing that I said we should never do, which is argue is... Prior knowledge is important or gaining knowledge is important. You know, that's what you guys just said. You said, [35:00] You're still going to have to learn things. And you're saying, oh, I can get things from my culture and from prior knowledge. But these should not fight for each other. On the exact thing around paradigm shifts, how do we create a machine that understands when to shift the paradigm? I think through its experience, right? So it would have to through its own experience, it can't rely on human knowledge because we're assuming the humans see one paradigm and we want a different way of looking at things. And so through its experience, it has to find something that is better. Maybe it generalizes better and makes better predictions. Maybe it's better in some other ways, but it has to be through its own experience. The big challenge that we don't see in our field, the ability we don't see in our field yet, is the ability to learn a model and then plan with a model. We can do the math things and we can do AlphaGo because... the games, we know the model. We know how the moves work. And in math, we know what the operators are. We know lean will take us from one state of knowledge, state of the proof to the next state. But if we have to learn the models, There are no I'm going to say it is probably maybe a weird example, but counter example, but I can see there's no instances of learning the model and then planning with the model in our field. at least not with self-discovered abstractions. So, There are people who say, I'm just going to learn a model of what happens in the next second or next millisecond. But that's not how our models work. Our models are more abstract. our models are um, quite different. So one of the things I like about what you're doing here is you're not just sitting around pontificating or lamenting the state of the world as it is. You're very action oriented. That's why you've started a company. So let's start talking about that a bit. In 2022, Rich, you laid out a very specific 12-point plan. The Alberta Plan for AI Research. Maybe tell us about that. So the Alberta Plan came about because we did have general ideas, but we also needed to convert them into smaller chunks. And so the 12 steps are the attempt to crystallize particular chunks. There's a very important early step, step two, which is Continual deep learning. And we think that one is like almost the most important because it unlocks everything else. If you could do continual deep learning, You could then continually update your model of the world. And then if you knew how to do the abstraction rights and like... The second half of the steps are all about how to get the abstractions right. So and not only my abstractions right what I mean, but I don't mean get the right abstractions because no one can say what the right abstractions are that depends on the world is you're in your agent would have to learn the correct abstractions for whatever world sin. And so, you know, maybe those are the two key things. You have to find the right abstractions, and then you have to be able to do continual deep learning. I think that a lot of the people in the field realize that we need models, we need to plan with them, but the abstractions tell us what the model should be conditioned on. So what should the model predict? What are you going to do? And then something is going to happen. And more importantly, where would that come from? So I really like the example of elite athletes. If you ask elite athletes about how they do certain things, they would have weird niche terminologies for doing very specific things. They were like, you know, I do this thing and they would have a name for it if they communicate. Sometimes They don't even have a name for it if they're just doing it alone. So how did they come up with those abstractions? That's, in some sense, a crucial thing that's missing that the later half of Alberta plan answers. Can we talk about the continual deep learning part? is it an algorithmic gap that exists today? Or is it a, just a practical deployment infrastructure, data privacy gap? Because if I wanted to do, Call it naive updating of weights. based on user interaction, I can do that today, right? And so What, in your opinion, is the biggest... thing that we're missing to kind of get to continual deep learning? Yeah, so it's absolutely an algorithmic gap. You can do the naive thing, but then you'll see all sorts of problems. For example, if you say, I'm going to take one sample, and then I'm going to update my whole model with that one sample, you will run into this problem that now all of the previous knowledge in the get around this is exactly what cursor does. They don't use one example. They use a large batch coming from a lot of users. So in use cases where you can have that, you can do continued learning. But Most use cases you don't have that. Most use cases you have a single stream of data. And then if you apply it to the naive thing, it just completely destroys your prior knowledge in a very destructive [40:00] away. catastrophic forgetting. Yeah. That is so, yeah. But it's totally curable. You have to have the right algorithm. What's the cure? Yeah, exactly. Well, you know, first you need to do what we call step size optimization. And it means every weight in your network has to have a separate step size. So some will move fast, some will move slow. And you will have to meta-learn things. These step sizes for each weight. Most of your network will have weights that have tiny step sizes. So then when you train on a new example, they don't get destroyed. Mm-hmm. It happens just to the right places. And then secondly, you have to use some form of generate and test. Um, which is in feature space. So you come up with new features or new units and without following gradients because gradients is a very slow process. You only move in a direction if you know it's the helpful one. And that's always going to be very slow and doesn't give you a path to grow more and more complex and complex. to have sustained learning. You need to have something that just proposes a bunch of new units And, uh, and, and, And then goes from there, I guess. So there is a specific thing I can say to make it at least concrete, which is to say we have this algorithm called continual backprop published in Nature. a couple of years ago. And it is exactly like Backprop, but you also plant new seeds of units that are newly initialized with random weights. Backprop only has random weights at the beginning of time. And then as you go on, all that randomness, all that variety from the randomness gets used up. And with continual Backprop, you keep injecting a bit of randomness, a bit of generate and test, a bit of generate. And then the operation of Backprop is the tester. need that And if you put those together really well, I think you'll have a new generation of massively superior continual deep learning. That's what we hope to do in the next couple of years. Wonderful. Do you think that these algorithms... can be applied to the current state of affairs with... People scaling LLMs and trying to get them to do continual learning without catastrophic forgetting. Yeah, absolutely. I think it's a... So I don't think that you could take an existing model and say, I'm going to just start updating it with these algorithms, because these algorithms meta learn how to learn. So really, you have to say, I'm going to learn from scratch. So let's say learn a new foundation model, but I'm going to learn with these new algorithms. These new algorithms, in addition to... learning the knowledge, they're also going to learn how to learn future things. So they're learning two things at the same time. And then... then I think you would be able to learn new things without catastrophic forgetting. Is that the most radical thing that you're trying to do in your company from the current state of affairs to try to do these two things at the same time? Most radical thing. I think that's... This goes back to I'm not crazy. Everyone else is crazy. That's perhaps not totally radical. There was a point... in 2016 to 2018, where a lot of people were exploring these ideas quite a bit. They were doing it in a much more limited setting. So they would say, we have a distribution of problems. And then in this specific case, we'll do it, whereas we want to do it from a single stream of experience. So our method should be more generally applicable. So I think many people have explored this. No one has... explore this in the general setting where the resulting algorithm would be applicable everywhere. So what would be the most radical thing that your new company is trying to do that other people are not doing? What's the most ambitious thing? Remember, I don't think I'm weird, so I don't want to say it's radical. What's the most ambitious thing? I think, is to try to have the full spectrum of knowledge. both about the tiny things and about the big things. You know, like thinking about how you take an airplane from one city to another. That's a very big, you know, it's more, it's like you're, your space flight example, but it's just kind of more common sense to think about because we all, many of us take airplanes and all of us use abstractions in all kinds of our life. And even the squirrels use abstractions. So to have that spectrum of knowledge, From the small to the big and to treat it in a uniform way and to be able to have it self-maintaining. You know, the big question is always you have your knowledge-based system and what keeps the knowledge in it correct? Well, what keeps the knowledge correct in a large language model is, well, people did a lot of post-training. And they made it. [45:00] Sure, it was correct. And then they freeze it after that. So that's... what keeps it correct. But really our minds, we are always changing things and yet something keeps it organized and coherent and settling back into a good place rather than drifting off into crazy land. That is, I think, our biggest ambition to have a mind that is self-consistent and can change. keep training itself and making it I love that. Can I ask? It almost seems that It's such a... ambitious vision and the idea that all these things can be unified into a single mind is so ambitious. It's within reach. I think it's within reach. Here it's 2026 and our computers are so fast. Is it so ambitious that it's out of reach? Or do we have... already inklings of how all the steps can be done. And we I think we have... a vision, and inklings I don't think it's inappropriate. Your vision involves a trillion parameter model with 20 watts. That seems pretty... Ambitious. That is ambitious in some sense with current technology, I would say it's also impossible like just storing a trillion parameters in memory would probably use more than 20 watts of energy with current memory technologies but it's we are really thinking of Okay? things are getting better, computation is getting cheaper, it is getting more energy efficient. So where would be would be in five to ten years? And I think five to ten years with the right algorithms And we can totally be in a world where this would be possible. So five to 10 years is two orders of magnitude of Moore's Law. It's a standard improvement process. If we double every 18 months, 10 years would be two orders of magnitude. for Kerm's statement to be plausible, then today you should be able to do it for for 20 watts, 200 magnitude, 2,000 watts. If you can do it with 2,000 watts today, yeah. then in 10 years you'll be able to do it for 20 watts. I think you can do it for 2,000 watts. You have lots of people at research labs that have access to way more than that. Yeah, I think we can be more efficient than that even now. with the right algorithm. If we can be more efficient than that, then why aren't we? It's not like people... Just want to spend all their money. Spend all our money. Sometimes it seems like they want to. Yeah. Doesn't it? Thank you. I think that's how they show their real men, by using lots of energy. At least when I look at different research groups, I don't even see anyone... believing in it's possible. And I think if you don't believe in it, you're just not going to work on the technical problems and work through them. Is it that it's not possible or it's that there's so much waste in the system? Like which one is it? Is there someone who knows how to do it efficiently? Yeah. And then there's... 10 times the number of people in the same lab doing all these other things And so nine out of 10 people are wasting. - In some sense, the way I think about it is that we are stuck in a local menu. So if we want to move towards this new kind of algorithms, it is almost impossible that things will not get worse before they get better. So when we start exploring these new directions, you're not going to get state-of-the-art performance from day one. but it is because it is a different paradigm. But that path leads to similar performance at a higher energy. And these big labs, they are so locked into a product that they It is not possible for them to pursue a path where things get worse first. Because their current paradigm allows them to keep scaling. And this new paradigm, they have to take a bet. And they have to figure out some technical things that are difficult that... We have thought about it for many years. We know people who have thought about these things for many years and When I talk to them, it makes sense that it's doable. But you need to think about those challenges for a long period of time. So if everything goes right with Oak... What happens with the company? What kind of company are you building? If everything goes right, we implement the architecture, we can have genuine, continual learning, and we can form abstractions so that we can do planning and reasoning. And we so sort of like true intelligence. And then, you know, it's hard to imagine just exactly what will happen by then. But I think... Humans will become irrelevant? I don't think that's true at all. We don't either. I think the world becomes exciting and even more exciting and interesting for humans. But in particular, I think there are the... you have to wonder about the large language models. [50:00] might be at risk. when this eventually happens, you know, I'm sure they'll get a good run. They've already had a good run. You know, they've been very successful. And let me say, just for clarity, that large language models are an amazing scientific breakthrough. A breakthrough in the skillful use of language by neural networks. It's totally unanticipated. You know, it was always a holdout for symbolic methods in language, and they have totally changed how that's thought about now. Yeah, it's a big breakthrough. So it's frustrating to me that we have to, you don't just celebrate that we've made this great progress in the sub subset of the problem of AI and enjoy that. Instead, it has to pretend to be all of AI. All of intelligence is not fluid, capable use of language. There's so much more. It's an important part. You know, it's like... 20% or a quarter of intelligence. There's more. We're not done. Yeah. If everything goes right, Are you imagining that there's a single mind that can do everything from learn how to swing from tree branches... to make a spaceship... to all these various things we've talked about today. Is it a single mind and is it... A single set of weights... That can do all these things? Or is it... It's a single design. Okay. There'll be many minds. Okay. Okay, so it's a single design that reacts to different environments. And different versions of that mind would learn different things because they have different experience. This sort of goes back to the big world hypothesis that there are infinitely many things to learn. So one system cannot learn infinitely many things. And like, I think Rich already mentioned this, but if you have two of these systems, if you have two of the largest systems in the world, then it is trivial that they cannot model each other because they're equally complex. multiple systems that are learning from their own experience. You guys are hiring. What kind of people are you looking for? We are hiring the initial team, most of it we already have in our mind. So these are people who have thought about these ideas for a long time, and We are going to take a slight... different approach because this is a different paradigm. It doesn't make sense to become large very quickly because in some sense, everyone we hire has to come to see what we see and not everyone sees that. So we're going to start small, slowly grow to maybe a handful or two or three and then go from there. We want to be super aligned. We want to be super aligned. So that we can be very productive, working together and scaling the progress. Absolutely. Very, very cool. Wonderful. I love this conversation. Thank you for taking the time to share what you're up to. You are an unusually deep thinker about... where reinforcement learning and algorithmic design will go. And it was a true pleasure to get to explore it together with you today. So thank you. Thank you very much. Thank you. It's our pleasure. Music.