Sutton, Javed Launch Oak Lab to Replace Frozen LLM Architectures

Sutton与Javed创立Oak Lab，旨在替代冻结权重的LLM架构 Oak Lab联合创始人Rich Sutton与Khurram Javed认为，静态LLM在有限的互联网数据面前已陷入瓶颈，主张采用能根据现实世界经验不断更新模型参数的持续学习算法。 Rich Sutton和Khurram Javed对当前主流AI范式提出批评，认为依赖静态数据集、冻结权重和合成数据是受人类局限性限制的重大失误。他们提出了“大世界假说”（Big World Hypothesis），阐明物理世界远比任何静态模型或模拟器复杂得多，因此实时持续学习是实现真正智能的必要条件。为解决灾难性遗忘等核心障碍，他们提出了Continual Backprop算法，使模型能够在不丢失既有知识的情况下持续更新权重。Oak Lab的终极目标是在20瓦的功耗预算下，构建一个拥有1万亿参数、能够持续学习且连贯自持的智能体。 Rich Sutton 重新阐释其知名文章《苦涩的教训》（The Bitter Lesson）的核心，强调 AI 研究应聚焦于能随算力自然扩展的通用搜索与学习算法，而非过度依赖人工灌输的专业知识。 (0:07) Sutton 与 Khurram Javed 认为合成数据是一个“巨大的错误”，并提出“大世界假说”：物理世界的复杂性远超任何模拟器，智能体必须通过直接经验在交互中持续更新内部模型。 (11:03) Sutton 批评现有主流大语言模型在推理时网络权重保持冻结，根本无法在与用户交互中实时学习，且语言能力仅代表约 20% 至 25% 的智能。 (20:15) 针对灾难性遗忘难题，Oak Lab 提出 Continual Backprop 算法，通过为每个权重分配独立步长并引入“生成与测试”机制，实现无需冻结权重的终身在线学习。 (38:53) Oak Lab 确立了未来 5 至 10 年的目标：打造一个具备 1 万亿参数、能持续自我维护且仅以 20 瓦能耗运行的智能心智。 (43:43)

Rich Sutton and Khurram Javed argue that artificial intelligence research has drifted into a flawed paradigm of static, frozen models bounded by human data, contending that true intelligence requires real-time continual learning from direct physical interaction with the world. Through their work on the Alberta Plan and Oak Lab, they propose replacing brute-force offline compute with architectures that dynamically adapt and self-organize within strict energy constraints. The Bitter Lesson and Synthetic Data Bottlenecks (0:07) Reflecting on his 2019 essay The Bitter Lesson, Sutton distills its core thesis into a 26-word principle: researchers must avoid distraction by human domain knowledge and instead prioritize general methods like search and learning that scale with computation. Sutton views modern Large Language Models (LLMs) as a dual case: early models validated the lesson by discarding handcrafted linguistic rules in favor of compute-driven next-token prediction, but contemporary architectures have stalled because their performance is bounded by finite corpora of human-generated internet text. Sutton and Javed reject the push toward synthetic data generation as a fundamental mistake. Because humans or existing models must still curate, filter, and evaluate synthetic outputs, human judgment remains the primary bottleneck. Under Javed's Big World Hypothesis, the physical universe is infinitely complex and larger than any internal agent model or simulation. Closed simulators inevitably miss real-world dynamics—such as motor friction, acoustic variance in drones, or subtle shifts in human negotiation—making direct, ongoing empirical learning essential. Frozen Inference versus Sensorimotor Neuroplasticity (20:15) Sutton criticizes the prevailing industry norm of deploying models with frozen network weights, asserting that a system incapable of updating during interaction cannot possess genuine intelligence. Javed notes that existing substitutes, such as in-context memory in prompt windows or delayed batch gradient updates aggregated over thousands of users, provide only temporary conditioning or slow, impersonal changes rather than permanent structural adaptation. To illustrate genuine plastic adaptation, Javed points to human patients who lose proprioception and gradually rewire their motor control to rely on visual feedback over two to three years. Invoking Moravec's paradox, Sutton highlights that biological organisms never learn through supervised learning with direct error signals dictating muscle movements. Animals such as squirrels master complex trajectories and recovery dynamics through trial-and-error physical interaction without formal education. Sutton argues that the field has overvalued symbolic tasks like chess while overlooking foundational sensory-motor coordination, which represents the true difficulty of biological intelligence. The Alberta Plan and Continual Backpropagation (36:42) To build autonomous agents capable of mental planning and abstraction, Sutton points to the Alberta Plan, a 12-step research roadmap published in 2022. Sutton identifies Step 2, continual deep learning, as the critical milestone. Standard neural networks suffer from catastrophic forgetting when exposed to a single continuous stream of data because naive gradient optimization quickly overwrites previously established representations. Sutton and Javed address this through their Continual Backprop algorithm, published in Nature. The method relies on two core mechanisms: meta-learning individualized step sizes for each weight so that critical knowledge remains stable while designated weights adapt rapidly, and continuously injecting randomly initialized units into feature space while using backpropagation as an evaluative filter to retain useful parameters. Javed emphasizes that because the network must meta-learn how to learn, continual models must be trained from the ground up rather than fine-tuned on top of static, pre-trained LLM checkpoints. Oak Lab's 20-Watt Target and Institutional Inertia (43:43) At Oak Lab, Sutton and Javed aim to develop a self-maintaining mind capable of bridging micro-sensory details and macro-level abstractions without human intervention. Their long-term technical objective is to run a 1-trillion-parameter model within a 20-watt power envelope—matching the biological budget of the human brain. Javed and Sutton project that this target is achievable over a 5-to-10-year horizon, assuming hardware efficiency advances from an initial 2,000-watt baseline on modern hardware. Javed claims that mainstream frontier AI labs are trapped in a commercial local minimum, unable to pursue radical continual learning architectures because transitioning paradigms requires enduring temporary performance drops that product-focused companies cannot tolerate. Sutton estimates that language capabilities represent only 20% to 25% of general intelligence, forecasting that static LLMs may eventually face obsolescence. Under the Big World framework, Javed concludes that no single monolithic model can know everything, anticipating an ecosystem of diverse agents sharing a common architectural core but shaped by distinct experiential paths. Sutton and Javed acknowledge that departing from mainstream transformer scaling requires sustained technical execution, leading Oak Lab to maintain a small, highly aligned research team. However, whether continual backpropagation can scale to match frontier capabilities in open-world settings without relying on massive offline human datasets remains an unproven hypothesis.

Rich Sutton 与 Khurram Javed 认为，当前依赖静态数据集与冻结权重的 AI 研发路径已偏离真正智能；通用人工智能必须建立在能够随算力扩展的“持续学习”之上，通过智能体在无限复杂的物理世界中实时交互，自主构建内部世界模型与抽象表示。 苦涩的教训与静态大语言模型的局限 (0:07) Sutton 重申了其 2019 年知名文章《苦涩的教训》（The Bitter Lesson）的核心逻辑，并将其精简为 26 个英文单词：不要被人类已有知识所干扰，而应专注于能随计算力提升而扩展的学习与搜索方法。Sutton 澄清自己并不反对先验知识，而是主张先验知识应作为后天经验的基础并持续演化。早期大语言模型（LLM）通过通用算法在大规模算力上吸收海量网络数据，印证了该法则的正确性；然而现行 LLM 范式过度依赖人类生成的静态数据，随着互联网公开文本逼近极限，以人类产出为上限的数据瓶颈逐渐显现。 Sutton 批评当前主流行业对 LLM 存在误解，指出这些模型在推理与用户交互阶段权重完全冻结，并不具备实时学习能力。在 Sutton 与 Javed 看来，真正的学习必须是实时连续的过程，当前依靠上下文窗口（In-context learning）进行状态调节只是临时条件反射，未能实现网络权重的持久更新。此外，Sutton 提出人类语言能力仅占全部智能的 20% 至 25%，业界将流畅的文本处理能力等同于完整通用智能的倾向严重脱离了智能本质。 大世界假说与合成数据及仿真模拟的缺陷 (11:03) 针对当前业界转向合成数据（Synthetic Data）的趋势，Sutton 直言这是“一个巨大的错误”。Sutton 与 Javed 认为，合成数据依然需要人类专家或预设规则进行生成与过滤，人类认知依然是系统的潜在瓶颈；一旦顶尖工程师离岗，系统便缺乏有机机制来评判超出已有知识范围的数据优劣。 Javed 提出了阿尔伯塔大学发展的“大世界假说”（Big World Hypothesis）：现实物理世界的复杂性无限大于任何智能体的大脑或计算模拟器，任何人工环境都只是现实的微缩片段。Sutton 以商业谈判中动态内隐的心理状态、以及蝙蝠仿生声呐无人机面临的物理磨损与声学微变为例，指出离线模拟器无法涵盖现实中无穷的边缘案例。在自动驾驶等应用中，依靠人工反复调试以弥合“模拟到现实差距”（sim-to-real gap）极度低效，智能体必须在与真实世界的持续交互中，自主学习并实时修正其内部世界模型。 生物启发与莫拉维克悖论：告别监督学习偏见 (26:09) Sutton 指出，自然界中没有任何生物依赖监督学习机制运作。生物在进化与成长中从不接收指示肌肉抽搐的显式目标向量，即便是人类语言习得也是通过试错进行运动协调。尽管人类通过学校教育传承了上万年的文化累积，但正规学校教育只是近现代的发明；松鼠从未接受过抽象教育，却能在复杂的树林间实时计算运动轨迹并自我缓冲，展现出高超的物理交互能力。 这正契合了“莫拉维克悖论”（Moravec's paradox）：人类视为高阶智能的抽象符号任务（如下棋、数学证明）在计算上相对容易，而基础的感觉运动协调与物理自适应才是智能中最难克服的根基。Javed 以神经科学为例说明真正的结构适应能力：失去本体感觉的患者在两到三年内能够重构运动控制回路，转而利用视觉反馈恢复行走能力。这种在核心假设失效后重构底层参数的神经可塑性，是缺乏持续学习机制的现代 AI 所不具备的。 克服灾难性遗忘与持续反向传播算法 (38:53) 在 Sutton 等人于 2022 年发布的通用智能 12 步路线图“阿尔伯塔计划”（The Alberta Plan）中，第二步“持续深度学习”（Continual Deep Learning）被列为最关键的关卡。若无法在网络引入新经验时保护旧知识，智能体便无法维持动态世界模型，进而无法实现高阶抽象与长程规划。Javed 强调，灾难性遗忘是一个纯粹的底层算法缺陷，而非工程规模或隐私问题所致。 Sutton 与 Javed 在《自然》（Nature）发表的“持续反向传播”（Continual Backprop）算法提出了具体解决方案：其一，元学习每个权重的独立步长（Step-size），绝大多数承载核心先验知识的权重采用极小步长加以保护，仅允许特定权重快速更新；其二，在特征空间实施“生成与测试”（Generate-and-test），针对梯度下降随时间耗尽随机性的问题，算法持续向网络注入新的随机初始化单元，并利用反向传播作为测试筛选器保留有效特征；其三，模型必须从零开始训练以元学习“如何学习”，而非仅仅微调预训练静态大模型。 Oak Lab的终极愿景与前沿实验室的创新困境 (43:43) Sutton 与 Javed 创办的 Oak Lab 确立了明确的技术远景：构建一个在 20 瓦能耗预算下运行的 1 万亿参数自主维持心智，使其兼具人类大脑般的能效，平滑连接微观感知细节与宏观抽象规划。根据团队测算，在当前硬件条件下运行该模型需约 2,000 瓦功率，但若摩尔定律在未来 5 至 10 年内实现两个数量级的能效优化，20 瓦目标将具备可行性。Sutton 批评主流前沿实验室深陷高耗能算力军备竞赛，缺乏对低能耗、持续学习算法可能性的基本信念。 Javed 认为，主流 AI 巨头目前困于商业化的“局部最优”（Local minimum）：转向持续学习路径意味着短期内必须承受性能回撤与技术试错，绑定了成熟商业产品的企业难以承担这种风险，只能继续堆叠确定性更强的静态算力。基于大世界假说，Javed 预测未来不会存在无所不知的单体全知模型，不同智能体因经历不同将衍生出多元心智；为此 Oak Lab 选择克制扩张，保持团队规模在两三打人以内，以确保研发哲学的高度同频。 Sutton 与 Javed 展现了一种跳出主流静态预训练与人工先验束缚、聚焦物理交互与持续学习的 AGI 路径。然而，该路线仍面临多重未决挑战：在硬件能效上，其 20 瓦万亿参数心智极度依赖摩尔定律与新型计算架构的长期突破；在产业生态上，主流机构对既有商业回报与算力规模化路径的路径依赖，使得这一底层算法范式的工业级落地与广泛验证仍需漫长周期。

## Background 该视频由红杉资本（Sequoia Capital）于 2026 年 8 月 21 日发布在 YouTube 平台，由合伙人 Sonya Huang 与 Alfred Lin 共同主持。内容围绕强化学习先驱 Rich Sutton 与其学生 Khurram Javed 共同创立的 Oak Lab 展开，深入探讨了为何现有 AI 模型在部署后停止学习、合成数据的局限性、大世界假说，以及如何通过持续学习架构重塑通用人工智能。 ## TL;DR Rich Sutton 与 Khurram Javed 尖锐指出当前 AI 领域正深陷由预训练主导的“局部最优”：大语言模型仅能涵盖人类智能的约四分之一，而权重冻结和依赖合成数据实质上将模型锁死在人类知识的瓶颈中。基于真实世界复杂度永远碾压任何智能体的“大世界假说”，他们论证了智能体必须依靠直接的环境经验与单流交互来持续更新权重。他们通过在《自然》发表的 Continual Backprop 算法与步长元学习提出根治灾难性遗忘的方案，并确立了 Oak Lab 的终极愿景——在未来 5 到 10 年内打造一个以 20 瓦超低人脑级功耗运行、能自我维护且持续进化的万亿参数心智实体。 ## Atmosphere & style 整场对话兼具深邃的哲学思辨与底层的硬核算法推导，弥漫着一种逆主流狂潮而行的叛逆者气质与历史洞察力。Sutton 饱经沧桑后的从容自若、对行业浮躁的幽默戏谑（如直问“你是认真的吗？”），与 Javed 严密冷静的青年锐气相得益彰，与红杉主持人的务实追问碰撞出极具张力的火花。收听体验既像是在聆听学术宗师对生命与智能本质的虔诚拆解，又像是在见证一场向整个深陷局部最优的 AI 巨头阵营发起的长远智力远征。 ## Bulletin - [42:57](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2577s) Rich Sutton 与 Khurram Javed 创立 Oak Lab，致力于研发基于单条经验流自我维护的智能体，终极目标是在 5 至 10 年内以 20 瓦人脑级功耗运行万亿参数的持续学习心智系统。 - [11:03](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=663s) Sutton 严厉批评合成数据是“巨大的错误”，并提出“大世界假说”：物理世界的复杂度无限超越任何智能体与模拟器，静态冻结权重的预训练范式无法跨越与真实世界的泛化鸿沟。 - [40:00](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2400s) 团队在《Nature》提出的 Continual Backprop 算法证明灾难性遗忘完全可治愈，核心在于结合权重独立的步长元优化与持续引入新神经单元的“生成与测试”演化机制。 - [47:58](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2878s) 大语言模型（LLM）被指仅占人类智能的约四分之一，而主流前沿实验室因背负商业化交付压力深陷 Scaling Law 局部最优，难以承受探索持续学习新范式初期必然遭遇的性能滑坡阵痛。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=0s) 自然智能中的所有学习本质上都是持续进行的，反常的是整个 AI 领域，居然需要特意发明“持续学习”一词来指代最基础的学习行为。 - [07:08](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=428s) 《苦涩的教训》的核心在于不要被人类主观知识所分心，应聚焦于能随算力规模扩展的通用学习与搜索方法，历史证明人工硬编码规则终将被算力扩展碾碎。 - [11:03](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=663s) 合成数据并未跳出人类瓶颈，其生成规则与评估标准仍依赖人类专家干预；缺乏智能体与环境的亲身交互，合成数据无法带来真正的全新认知增量。 - [13:38](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=818s) “大世界假说”指出真实世界的复杂度在量级上永远碾压任何智能体的心智，全局最优不可实现，智能体必须依赖持续学习动态调整局部的有效近似。 - [16:20](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=980s) 传统自动驾驶等领域的外部仿真系统严重依赖工程团队人工修复 Sim-to-Real 差距，而真正的智能系统应由智能体自主学习内部预测世界模型并实现实时自我纠偏。 - [19:27](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1167s) 先验知识与学习本应互为补充，经验学习的成果会沉淀为下一阶段的先验状态；然而当前大模型将海量知识预训练后彻底冻结权重的做法，背离了心智演进的基本规律。 - [22:02](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1322s) 大模型助手的上下文学习（In-context learning）只是在瞬时状态变化下输出对应结果，只要底层权重保持静止，系统就未曾发生真正的心智重塑或概念生成。 - [26:09](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1569s) 自然界没有任何动物通过监督学习成长，底层肌肉抽动与运动控制从来不存在外部标准答案，莫拉维克悖论表明基础感知运动控制比符号推理或证明定理更加艰难。 - [32:02](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1922s) 重大的科学范式转移无法仅依靠继承现有人类知识完成，智能体必须拥有在知识边界处自主发现全新抽象、并基于该抽象建立模型进行长程想象与规划的能力。 - [36:42](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2202s) Sutton 的《阿尔伯塔计划》将通往通用智能拆解为 12 步，其关键母钥匙是持续深度学习，旨在让网络在单条连续数据流中终身演进并不间断提炼抽象概念。 - [40:00](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2400s) 灾难性遗忘在算法上完全可治愈，解决方案依赖为每个权重独立配置学习步长的元学习机制，配合 Continual Backprop 动态注入随机单元并进行生成筛选。 - [42:57](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2577s) Oak Lab 的核心野心是构建能够“自我维护”的统一心智，摆脱人工后训练对齐与参数冻结，在权重持续流动更新中自发维系内部常识边界与一致性。 - [46:02](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2762s) 结合摩尔定律带来的百倍能效演进与高稀疏度持续学习算法，Oak Lab 预期在 5 到 10 年内以 20 瓦的人脑级超低功耗维持万亿参数智能体运转。 - [49:13](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2953s) 大语言模型击溃了语言学中的符号主义流派，但流畅运用自然语言仅占人类完整智能的 20% 至 25%，将其神化为全能智能属于严重的学科认知偏差。 - [51:02](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=3062s) 大世界假说决定了未来不会存在全知全能的单一静态超级心智，而是依托统一的底层持续学习架构，在不同交互轨迹中分化繁衍出千差万别的多元心智生态。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=0s) “我并不反常，是这个领域反常了” 访谈开门见山，Rich Sutton 直面了外界常年扣在他身上的标签——“观点激进”。很多人在向他提问时，总习惯以“你的想法为什么和大家如此截然不同”作为开场白。但 Sutton 认为事实恰恰相反：“我觉得自己的想法非常普通，反倒是其他人的思维变奇怪了。”他解释道，这种认知扭曲只是近些年才出现的狂热表象。 在当前 AI 狂潮爆发前，人们根本不需要特意强调“持续学习”（Continual Learning）这个概念，因为讨论非持续的学习在逻辑上毫无意义——在自然界和智能的本质中，“所有的学习都是持续的”。智能体永远在行动，并从行动中实时学习，这是再正常不过的规律。Sutton 直言：“我并不反常。反常的是这个领域，它居然需要造出‘持续学习’这个词，而它原本就只是普通的学习。” --- ### [00:59](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=59s) 寒冬、癌症与奔赴阿尔伯塔 红杉资本的 Sonya Huang 隆重介绍了访谈嘉宾：强化学习之父、经典教科书作者、《苦涩的教训》（The Bitter Lesson）一文的撰写者 Rich Sutton，以及他的前博士生、Oak Lab 联合创始人 Khurram Javed。Sonya 将对话拉回几十年前，询问在强化学习尚处襁褓、学术界一片萧条时，是什么信念支撑他在阿尔伯塔大学（University of Alberta）扎根并建立起强化学习的重镇。 Sutton 笑着反问：“不然还能做什么呢？”他解释道，他们一直在试图解开人类心智的奥秘，而心智与智能的核心就是两点：具备目标（having a goal）以及具备学习能力（learning）。然而，2003 年他前往阿尔伯塔背后的真实故事却极富戏剧性甚至带有黑幽默色彩——那不仅是一个学术界的 AI 寒冬，也是他个人生命的极度严冬。当时 Sutton 罹患癌症，医生预言他只剩几个月生命，他本人甚至做好了随时死去的准备。但经过多次化疗缓解后，他发现自己“居然没死成”。既然一时半会儿死不了，生活总得继续，他便决定换份教职，前往埃德蒙顿的阿尔伯塔大学。 谈及面对绝症为何仍坚持科研，Sutton 引用了相传来自本杰明·富兰克林的一句话：探究人们为何做某事，原因往往不外乎两点——要么是习惯（habit），要么是虚荣（vanity）。在 Sutton 看来，当时的坚持更像是一种纯粹的习惯惯性。 - **2003 年的境遇**：处于第二次 AI 寒冬，Sutton 被诊断出癌症晚期，在预期寿命仅剩数月的状态下接下阿尔伯塔大学的教职。 - **关于心智的常识**：在狂热之前，智能被公认为由底层的快速行动与感知生成、结合高阶思考构成，感知与行动永不停歇，绝非离线切分的孤立阶段。 --- ### [05:47](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=347s) 伯乐与千里马：学生如何“挑选”了导师 当 Sonya 问及几十年来如何挑选出像 David Silver（AlphaGo 主导者）和 Khurram Javed 这样推动领域前沿的顶尖学生时，Sutton 展现了极大的谦逊。他坦言自己从未觉得自己擅长选拔学生，很多时候全凭运气，“有时运气好，有时运气差，往往是优秀的学生选择了我，比如当年是 David Silver 挑选了我”。 Khurram Javed 也分享了自己成为 Sutton 学生的奇妙缘分。他硕士毕业后原本打算直接进入工业界工作。一次偶然的项目讨论中，有人向 Sutton 提到了 Javed 的工作，Javed 随即被拉进讨论并开始与 Sutton 合作。在六个月高度默契且成果斐然的研究协作后，两人自然而然觉得这项研究足够支撑一篇博士论文。Javed 从未正式递交博士申请信去恳求 Sutton 收留，而是“先有了合作成果，顺理成章转为博士课题，随后才补办了博士入学申请”。 ```mermaid flowchart TD accTitle: Khurram Javed 成为 Sutton 博士生的路径 accDescr: 展示了 Khurram 从完成硕士学位到与 Sutton 合作并最终转为博士研究生的自然发展过程 subgraph Phase1 ["初始规划"] M["完成硕士学业"] --> I["原计划直接进入工业界"] end subgraph Phase2 ["有机协作"] P["参与独立研究项目"] --> D["项目在会议中被 Sutton 关注"] D --> C["开启有机合作研究"] end subgraph Phase3 ["水到渠成"] C --> R["6个月合作顺利并取得实质进展"] R --> T["成果自然转化为博士课题提议"] T --> PhD["正式提交博士申请，师徒关系确立"] end I -.-> P ``` --- ### [07:08](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=428s) 《苦涩的教训》与26个英文单词的精髓 谈话切入 Sutton 在 2019 年发表的划时代文章《苦涩的教训》（The Bitter Lesson）。Sonya 提出疑问：2019 年处于一个微妙的时间节点——晚于 2009 年的 ImageNet 和 2015 年的 AlphaGo，又早于大语言模型（LLM）展示惊人 Scaling Law 的爆发期，为何偏偏在那时写下这篇檄文？ Sutton 回应称，这篇文章并非对某个孤立历史时刻的应激反应，而是他历经数十年 AI 浪潮（包括他亲身走过的符号主义 AI 时代）后沉淀下的规律总结。早在发表前一年，他便已在多次演讲中阐述过这一构想。他见证了一代代研究人员前仆后继地试图将人类的主观知识（human knowledge）编码进系统中，但历史屡次证明，这种做法最终都会被利用计算力扩展（scale with computation）的通用方法彻底碾碎。 针对当下创投圈和科技界把“苦涩教训药丸”（Bitter Lesson pill）当成时髦黑话随处乱套的现象，Sutton 引用了自己在社交网络 X 上用 26 个英文单词归纳的微缩版核心奥义： > **“不要被人类知识所分心（正如 AI 历史上屡次发生的那样），而应聚焦于能够随计算力规模扩展的学习方法——例如搜索与学习。”** > *(“Don't be distracted by human knowledge as AI traditionally has been many times. Instead focus on learning methods that will scale with computation, like search and like learning.”)* Sutton 明确厘清了认知误区： - **并非不需要精妙算法**：这篇文章不是鼓吹蛮力无用论，AI 依然极度依赖算法创新，但关键在于算法的设计必须能够“随计算能力的增长而不断扩展效能”。 - **计算优先于人工介入**：追求的目标是用通用计算消化算力，而不是靠堆砌人类专家经验和手工规则来提升模型。 - **引出大模型争议**：访谈由此引向当下的核心争论——依赖海量人类文本预训练的 LLM，究竟符合还是违背了《苦涩的教训》？ ### [10:01](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=601s) 大语言模型：苦涩教训的双面镜 对于大语言模型（LLM）究竟印证还是违背了《苦涩的教训》（*The Bitter Lesson*），Rich Sutton 认为它兼具正面与负面特征。从正面来看，大语言模型实现了计算规模的空前扩张，它让系统能够直接“痛饮整个互联网”（drink in the internet），单凭能够利用算力扩展的通用算法，就换来了系统能力的飞跃，这完全符合苦涩教训的核心主张。然而从反面来看，LLM 终究会被其所依赖的人类信息源所锁死。互联网的数据是有限的，高质量样本难以无休止地获取；更重要的是，物理与现实世界极其宏大，远远超出了人类存储在网络上的所有内容。过度依赖人类已有知识作为养料，最终必然会沦为限制智能体进一步演进的桎梏。 --- ### [11:03](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=663s) 合成数据是“一个巨大的错误”与人类瓶颈 针对互联网“化石燃料”即将耗尽的困境，Sonya Huang 提出当前众多前沿基础模型实验室正押注于“合成数据生成”（Synthetic Data Generation），并询问这是否算是一种利用计算来突破瓶颈的通用方法。Rich Sutton 毫不犹豫地给出了干脆的否定：“不，那只是一个巨大的错误（that's just a big mistake）。”他笑着表示，这或许会演变为下一个“苦涩的教训”。这一思想在阿尔伯塔大学已经酝酿了五到十年，被称为“大世界视角”或“大世界假说”（Big World Hypothesis），后来由 Khurram Javed 正式撰写成论文。 Khurram 解释道，大世界假说的核心在于：真实世界是无限巨大的，其中存在着无穷无尽需要去学习的事物。依赖人类来构建合成数据集，本质上依然无法摆脱人类瓶颈。他反问：“究竟由谁来决定什么是好的合成数据，什么是坏的合成数据？”任何人都能写出一个吐出海量劣质合成数据的程序，进而毁掉整个模型；当前评判合成数据优劣的依旧是人类专家。当 Sonya 质疑模型自身的损失曲线（Loss Curve）难道不能评估数据质量时，Khurram 提出了一个尖锐的思考实验：如果 OpenAI、Anthropic 以及各大前沿实验室的所有工程师明天全部去度假，谁来负责生成合成数据？合成数据并非源自智能体自身的亲身体验，其背后必须有特定领域的专业人员。例如，若要制造一架像蝙蝠一样依靠回声定位（echolocation）在复杂环境中飞行避障的无人机，要想生成有意义的合成数据，就必须先聘请声学与空气动力学领域的人类专家去摸索。只要必须先有人类专家才能生成数据，整个扩展范式就依然被人类知识的上限卡死了。 - **核心矛盾**：合成数据虽然消耗了计算，但其生成机制与评价标准仍受困于人类工程师的先验认知与手工微调。 - **思考实验**：若实验室工程师全部离岗，缺乏自主感知环境与试错机制的预训练范式就无法产生新的认知增量。 --- ### [13:38](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=818s) 大世界假说：为什么任何模拟器都只是“微缩世界” 当 Sonya 提出人类可以构建无限多的虚拟合成世界时，Khurram 与 Sutton 展开了彻底的反驳。Khurram 指出，即使合成了 50 个不同宇宙的数据，由于缺乏真实的物理交互与感知反馈，机器依然无法凭空学会蝙蝠回声定位这种高度精细的现实技能。Sutton 则更加直截了当地指出：合成数据根本就是“错的”（wrong）。任何企图生成合成世界的程序，本质上都只是极其简陋的代码，拼凑出的不过是一个微不足道的“微缩世界”（small world）。 Sutton 用日常互动进行了生动的说明：在当天的交流中，对他来说最关键的是“投资人脑子里正在想什么”，但这世上绝不可能有任何合成数据能模拟他人此刻鲜活的思维状态。同样地，真实无人机在空气中遭遇的瞬时扰动、机械马达内部的微小摩擦损耗等，没有任何模拟器能够真正穷尽。Sutton 正式阐述了**大世界假说**：世界的复杂程度，在量级上永远碾压任何智能体的心智（mind），这一结论显而易见，因为大世界本身包含了无数个其他智能体。既然系统永远无法达成所谓的全知全能或“最优解”，就必须在局部做出严苛的近似。这也正是智能体必须**持续学习（Continual Learning）**的根本原因——智能体只能根据当下身处的大世界特定角落，动态调整适应于当下的局部近似，而不是妄图用静态模型覆盖整个宇宙。 ```mermaid flowchart TD accTitle: 大世界假说与持续学习的必然性 accDescr: 对比人类手工模拟器的局限与智能体通过环境交互持续学习的逻辑闭环 subgraph HandCrafted["传统路径：人工合成与模拟器（微缩世界）"] A["人类专家手工编写规则/模拟器"] --> B["生成预设合成数据"] B --> C["模型离线训练与参数冻结"] C -.->|"遇到未建模的真实物理/心智现象"| D["泛化断崖与模拟器失真"] end subgraph BigWorld["大世界视角：基于经验的持续学习"] E["无限复杂的大世界\n（物理摩擦 / 复杂环境 / 他人心智）"] --> F["智能体自主感知与交互体验"] F --> G["构建局部动态近似模型"] G --> H["根据新情境持续更新权重"] H --> E end ``` - **大世界假说定义**：现实世界的复杂度远远大于任何单一智能体的心智容量，全局最优在数学与物理上皆不可行。 - **心智的不可模拟性**：涉及交互博弈、他人心理以及精细物理损耗的现实场景，无法通过低维计算机程序合成。 - **持续学习的本体论依据**：正因为世界无穷大而模型永远只是局部近似，智能体才必须随着时间与位置的迁移不断学习。 --- ### [16:20](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=980s) 自动驾驶模拟的局限与智能体自建模型 针对自动驾驶领域广泛采用仿真模拟（Sim）训练再进行实车部署（Sim-to-Real）的成功经验，Sonya 提出了进一步的追问。Khurram 犀利地剖析了这种技术管线背后的真实代价：为了搭建和维护逼真的仿真环境，自动驾驶公司投入了极其庞大的人类工程师团队。整个研发流程本质上是人类在充当调试接口——工程师搭建仿真环境，模型在其中学习，部署到现实中暴露出不可接受的“仿真-现实差距”（sim-to-real gap），随后人类工程师再回头修改模拟器。这种系统完全依赖人类从真实世界获得反馈来弥合裂痕，智能体本身并没有自主解决这一差距的能力。Sutton 补充道，只要自动驾驶车辆驶上真实的公开道路，就一定会遭遇意料之外的极端偶发状况，而那恰恰才是智能体最需要直接从经验中学习的时刻。 Khurram 澄清，他们并非全盘否定模拟仿真与“规划”（Planning）的价值，关键在于**模拟器应该由谁来建立**： 1. **智能体自建世界模型**：智能体应当直接从自身的真实交互经验中自主学习出一个内部预测模型（Model）。若该内部模型预测失误，智能体能够通过持续学习在毫秒级内自动纠偏。 2. **人类硬编码模拟器**：依赖人类工程团队编写的外部模拟系统，只有当人类工程师意识到缺陷并重新发版时，模型才能得到修正，因而具有严重的滞后性与局限性。 --- ### [18:01](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1081s) AlphaGo的胜利：先验知识与学习本应是朋友 访谈转入《苦涩的教训》中另一个核心命题：剥离人类先验知识。Sonya 念出了 Sutton 论文中的经典名言：*“为了寻求短期内的明显改进，研究人员总是试图利用他们在该领域的人类知识；但长远来看，唯一真正起决定性作用的，是利用计算的能力。”* 随后她以 Sutton 的前学生 David Silver 在 DeepMind 主导的 AlphaGo 与 AlphaZero 为例——从最初融合人类棋谱到彻底抛弃人类先验、完全依靠自对弈强化学习超越全人类——询问 Sutton 当时对这一成果是否感到意外。 Sutton 坦言，David Silver 的突破让他深感欣慰，并带来了一种“沉冤得雪般的证明感”（feel vindicated）。但他同时修正了一个常见的误解：先验知识本身并无原罪，先验知识与学习在理论上完全没有理由对立。正如他在《苦涩的教训》开篇所写的，完全可以在注入先验的前提下启动学习过程，“先天与后天”（nature and nurture）本应是并肩作战的朋友，生命本就是获取知识与拥有知识的过程。然而在过去数十年的 AI 研发实践中，两者却荒谬地沦为了敌人——研究人员总是习惯于将人类先验作为逃避构建通用学习机制的捷径，一旦人类设计的启发式规则占据主导，系统利用计算与自主学习演进的大门就被彻底堵死了。 ### [19:27](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1167s) 先验知识与学习并非对立：“奇怪的不是我，而是整个领域” Rich Sutton 指出，AI 领域过去长期存在一种偏见：热衷于既有人类知识的人总希望知识能彻底胜出，从而极力贬低或忽视“学习”（learning）。因此在很多外界人士眼中，Rich Sutton 似乎成了一个极度崇尚学习而全盘否定先验知识的人。但他澄清，自己本质上是一个探索“心灵/心智”（the mind）的研究者。心智的真实运作方式是：你拥有先验知识，随后获取新经验并学到更多；一旦掌握了新知识，它就会沉淀为下一阶段的先验知识，如此循环往复、相辅相成。 之所以 Sutton 显得只关注学习，完全是因为整个 AI 行业走向了另一个极端——大谈“只要有足够的知识就无需再学”。以目前的大语言模型为例，人们试图将全人类的知识填塞进模型中，但在部署运行和与人交互时，模型的权重却彻底冻结、再也不发生改变。面对这种现象，Sutton 直言：“奇怪的不是我。奇怪的是你们，居然会认为能从一个完全不再学习的东西身上获得博士级别的经验与专长。” 针对 Alfred Lin 提问“是否应该让算法运行更长时间、细水长流地灌输先验知识”，Sutton 强调，从长远来看，真正重要的在于如何构建一套机制来持续获取和组织新知识。放眼未来智能机器人的时代，人类绝不会在每次更新时都耗费数以亿计的资金从整个互联网重新预训练；相反，由于数字实体极易复制，更合理的做法是直接克隆一个已有的智能体，让它在原有基础上继续学习。从这个意义上说，传统意义上人工灌输的先验知识最终会被超越，取而代之的是持续学习与智能体状态的复制演进。 ```mermaid flowchart TD accTitle: 持续学习的心智闭环 accDescr: 展示先验知识与持续经验交互沉淀的心智演进闭环 subgraph MindLoop["心智的持续演进循环"] A["现有先验知识 (Prior Knowledge)"] --> B["与环境交互获取新经验 (Experience)"] B --> C["权重与结构更新 (Continual Learning)"] C --> D["沉淀为新的先验状态 (Updated Priors)"] D --> A end subgraph Contrast["当前大模型范式（静态冻结）"] E["海量预训练 / 知识注入"] --> F["权重永久冻结 (Weights Frozen)"] F --> G["上下文改变但模型未学 (In-context only)"] end ``` --- ### [22:02](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1322s) 真正从经验中学习的机器，与大语言模型“权重不改”的假象 当被问及“一个真正能从经验中学习的机器应该长什么样”时，Sutton 表示它的形态多种多样：它可以是具身机器人；也可以完全运行在互联网底层，例如执行网络数据包路由任务，并根据实际经验动态优化路由策略；或者作为手机和电脑上的交互式个人助手，随着时间推移越来越理解用户的真实意图。 Sonya Huang 随即追问：当下基于 LLM 的热门助手难道不算经验学习者或持续学习者吗？它们难道没有通过 In-context learning（上下文学习）记住用户的偏好吗？核心差距究竟在哪里？Sutton 甚至有些难以置信地反问：“你是认真的吗？”（"Are you serious?"）他一针见血地指出了当前范式最根本的硬伤：**“它们的权重从不改变。”（"Their weights never change."）** Sutton 进一步解释，构建大语言模型最关键的步骤，在于权重训练过程中对新概念的组织与结构化生成。而真正的智能必须能够持续进行这种概念建构，绝不能指望它只在预训练阶段发生一次便告终止。无论预训练和后训练（Post-training）做了多少工作，一旦交给用户使用时模型就停止了更新，它就失去了最核心的学习能力。 - **上下文变换与真正学习的本质区别**：通过延长 Context，改变的仅仅是模型的瞬时状态（State）。模型只是在根据既有经验预测“若状态变为 X 则输出 Y”，这绝不等于模型本身在学习新技能或更新心智结构。 - **权重的全面适应性**：真正的持续学习不仅是微调几个局部参数，而是要求心智系统能够像最初形成语言理解一样，持续生成新概念并调整内部结构。 --- ### [24:01](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1441s) 批处理更新的低效，与人类大脑惊人的神经可塑性 Khurram Javed 与 Alfred Lin 讨论了类似 Cursor 的 Tab 代码补全以及 Composer 功能，这些系统确实会定期根据用户数据更新权重，属于广义持续学习的一种雏形。但 Khurram 指出，当前的做法存在巨大缺陷：工程团队通常从成千上万名用户那里收集海量日志批次（Batch data），然后统一进行一次策略更新。这种做法非常低效，如果某位开发者想专门教模型掌握某种特异的代码模式，自己的需求就会被其他几十万用户的统计分布所淹没。用户真正需要的是对“属于自己的模型副本”进行针对性教学。 面对“核心技能存在于公共权重中，个性化仅通过上下文实现是否合理”的疑问，Khurram 用生物学上的极端案例论证了为什么权重更新不可或缺： - **本体感觉缺失与神经重构**：人类依赖肌肉和关节内的本体感受器感知身体姿态以实现行走，这种控制策略自幼深植于大脑深层结构中。临床研究中，某些病患因疾病完全丧失了本体感受，起初彻底瘫痪。 - **突破 20 年固有常识的重塑能力**：在长达两到三年的康复训练中，这些患者的大脑展现出惊人的可塑性（Plasticity），他们通过眼睛紧盯双脚获取视觉反馈，完全绕过本体感受重新学会了走路。 - **AI 系统的必要特性**：人类大脑能够推翻坚守了 20 年的底层运行法则并重构神经网络。如果 AI 系统无法通过更新底层权重来纠正或抛弃过时规则，就永远无法具备这种深度的适应力。 --- ### [26:09](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1569s) 批判“监督学习”迷思：没有任何动物是靠监督学习长大的 探讨生物学对 AI 的启示时，Sutton 明确指出，动物学习是“灵感的源泉，而非硬性约束”，这与他著名的《苦涩的教训》（The Bitter Lesson）一脉相承。谈到当前 AI 系统与生物智能最大的脱节，Sutton 提出了一个看似激进实则不争的事实：**自然界中没有任何一种动物是通过“监督学习”（Supervised learning）来学习的**。 Alfred Lin 提出反驳，认为人类在学校接受的正是监督学习，所有的知识技能都是在学校习得的。Sutton 则坚决否定了这一观点： - **输出端没有直接目标标签**：生物与现实世界交互的终极输出是神经信号驱动的肌肉抽动（Muscle twitches）。现实中根本没有任何“超级监督者”会直接给神经系统示范肌肉纤维应该如何精准收缩。 - **学校教育只是冰山一角**：人类心智绝大部分关键能力——感知视觉、学会行走、理解物理世界因果律——早在上学前就已经形成，而且全都不是靠监督学习完成的。 - **语言传递与监督学习的混淆**：即便在课堂上被问到“法国的首都是哪里”，老师说出“巴黎”，这确实是信息的社会化传递，但这绝非严格意义上的监督学习。老师从未示范声带肌肉、舌头与呼吸肌群应该怎样协同微动，学生是通过自主探索与感知反馈才发出“巴黎”的音节。 - **学校教育不代表智能本质**：松鼠从不上学，依然能在复杂的树冠间自由跳跃穿梭；人类出现正规学校教育也不过几百年历史。把“学校的讲授模式”当成学习与智能的核心范式，不仅本末倒置，更是对智能本质的巨大误读。 ### [29:01](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1741s) 松鼠、数学定理与莫拉维克悖论 面对松鼠与人类技能的对比，Alfred Lin 和 Sonya Huang 抛出了一个直观的差异：松鼠虽然极擅长在树枝间跳跃穿梭，却永远无法证明数学定理；如果人类想学会证明数学定理，就必须进入学校接受结构化教育。Rich Sutton 幽默地反驳道，松鼠确实既不会证明定理，也没有发明 DVD 或 iPod 等现代科技产品，但它们能做到许多人类根本无法完成的事情。人类往往将象棋、数学定理这类高度抽象的能力视作智慧的终极体现，但这恰恰印证了莫拉维克悖论（Moravec's paradox）：那些在人类看来极其高深、需要高等智力的事情，对计算机而言往往轻而易举；相反，诸如感知、注意力分配、以及在物理世界中灵活移动这类动物与婴儿与生俱来的“常规能力”，对计算机反而是最大的难关。 关于监督学习的本质，Sutton 指出，在现实世界中没有任何外部教师能教我们如何抽动每根肌肉纤维，因为他人不可能拥有这套内部感知，我们必须靠自己在尝试中摸索。Khurram Javed 紧接着用发音的生动例子补充了这一观点：即使他完全模仿 Sutton 发音“Paris”（巴黎）时嘴唇、舌头和声带的运动方式，由于两人生理构造的差异，最终发出的声音也绝不相同。因此，世上没有任何人能够直接告诉某个人发出特定声音的“标准动作答案”，这种感知运动控制的真理只能由个体自身通过交互去探知。 ### [30:24](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1824s) 抽象能力的本质与人类的自大 Sonya Huang 承认在最底层的感知运动（sensory-motor）层面，智能体确实必须依靠自身经验来学习，但她质疑人类之所以伟大，依靠的是远离肌肉抽动的更高层次抽象思维，Oak Lab 的世界模型是否真的能从最底层的感知运动一路贯穿到最高阶的抽象层？Sutton 斩获肯定，坚称这正是他们的研究野心所在。他以松鼠为例打破人们对动物“缺乏抽象思维”的偏见：松鼠展现出的抽象能力极为惊人，无论人类在鸟类喂食器（bird feeder）周围设置多少精巧复杂的物理障碍，松鼠总能随机应变，构想出全新的攀爬和跳跃轨迹突破重围。 谈及动物与人类的智力鸿沟，现场讨论了动物在无意识状态下对物理规律的惊人掌控： - **实时抗跌落与轨迹规划**：动物在跌落瞬间能够实时做出毫秒级的身体姿态调整以避免受伤，它们无须像人类规划火箭升空那样进行显式的数学心算，却能极其精准地处理复杂的物理轨迹。 - **程度差异而非本质鸿沟**：Sutton 直言，试图刻意强调人类与动物之间所谓“本质区别”是一种智力上的自大（hubristic）。他更主张去发现二者的共性，人类与高级动物之间只是智力发展“程度（degree）”的差异。 - **文明与语言的放大效应**：人类之所以能够拉开与动物的差距，关键并不在于底层大脑计算架构的跃迁，而在于人类演化出了社会、文化以及语言体系，这些后天产物赋予了我们巨大的代际知识累积优势。 ### [32:02](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1922s) 造火箭、想象力与范式转移的诞生 Alfred Lin 进一步向 Sutton 和 Javed 发难：即便两三岁的小孩确实能在无人教授的情况下从经验中自行领悟大量底层技能，但人类登上外太空、建造火箭绝不可能 100% 仅靠蛮力经验来试错学习。在火箭点火发射之前，科学家必须先在脑海中完成高度抽象的推演与“想象（imagination）”——既然机器面对的是此前从未存在过的全新事物，我们究竟该如何教会机器去“想象”？ Javed 完全赞同规划与想象是不可或缺的基石，但他反问 Alfred：一万年前未经历现代科技的人类，其智力是否逊色于今天的人类？Alfred 坦言，哪怕回溯一万年，人类的大脑在生理演化上“基本上是同一台机器”，区别在于我们通过上万年的文明沉淀累积了海量知识，个体通过学校这种高效的监督学习机制，能在短短数年内吸收这万年智慧，其获取知识的速度远超从零试错。然而，Javed 深刻剖析了当人类打破既有认知、带来颠覆性“范式转移（paradigm shift）”（如爱因斯坦创立相对论，或是 AI 领域从人工编程规则转向自监督/强化学习）时的真实心理图景： ```mermaid flowchart TD accTitle: Paradigm Shift Knowledge Process accDescr: 展现人类从吸收文化知识到产生范式转移、发现全新世界模型的进阶认知路径 subgraph CultureAccumulation["知识积累与文化输入"] A["吸收既有文化与知识体系<br/>（通过监督学习/学校教育）"] end subgraph AutonomousAbstraction["自主经验抽象与规划"] B["自身交互与实践经验"] C["在知识边界自主构建全新抽象<br/>（Self-discovered Abstractions）"] D["基于新抽象进行长程想象与规划<br/>（Planning with Learned Models）"] end subgraph Breakthrough["范式转移与新发现"] E["突破既有假设，发现新规律<br/>（如爱因斯坦相对论）"] end A --> C B --> C C --> D D --> E ``` Javed 指出，当前最顶尖的 AI 系统恰恰彻底缺失了这种“在知识边界自行形成新抽象、并基于该抽象构建世界模型进行规划”的能力。Sutton 随之提醒大家不要陷入伪二元对立的陷阱——在先验知识（prior knowledge）与获取知识（gaining knowledge/learning）之间争论孰优孰劣是毫无意义的，两者本该相辅相成。当机器面临何时该掀起范式转移的关口时，它绝不能盲从人类现有的知识框架（因为既有框架正是要被打破的范式本身），而必须依托它在世界上自身的经验，去探索出泛化能力更强、预测更精准的底层新规律。 ### [35:40](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2140s) 模型的瓶颈：为何 AlphaGo 算不上学会了世界模型 针对当前 AI 领域在模型构建上的现状，Sutton 和 Javed 抛出了一个极其尖锐的诊断：如今的人工智能领域根本没有真正实现“先自主学习出一个世界模型，再利用该模型展开规划”的成功范例。人们常常拿 AlphaGo、数学定理证明或者 Lean 交互式定理证明器来反驳，但 Sutton 剖析道： - **规则完备的伪世界模型**：在围棋中，规则引擎是人类赋予的完全已知的模型，系统完全清楚每一步棋落子后的状态转移；在数学证明中，符号逻辑和 Lean 的算子也是人类提前定义好的真理集合。 - **微观预测不等于抽象模型**：目前有些学者尝试直接训练网络去预测物理世界“下一毫秒”或“下一帧画面”的具体数值变化，但人类真实的内部认知模型绝不是在如此低粒度的时间步上空转。人类的心理模型高度抽象且具跨度性，而目前 AI 领域完全缺乏让机器“自主发现抽象（self-discovered abstractions）”的技术路径。 ```mermaid flowchart LR accTitle: Known Rules Planning vs Oak Continual World Model accDescr: 对比传统AlphaGo/Lean规则系统与真正自主发现抽象的世界模型的根本差异 subgraph Handcrafted["现有系统：预置完美规则"] direction TB H1["人类硬编码规则/物理/状态转移<br/>（AlphaGo / Lean 数学证明）"] --> H2["在给定模型内部执行搜索与规划<br/>（MCTS / 符号推导）"] end subgraph LearnedModel["理想系统：自主抽象与世界模型"] direction TB L1["持续流式交互经验<br/>（Continual Stream）"] --> L2["自主提炼并发现高阶抽象<br/>（Self-discovered Abstractions）"] L2 --> L3["在抽象概念空间建模并展开想象规划"] end ``` ### [36:42](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2202s) 阿尔伯塔计划：12 步迈向持续深度学习 鉴于当前的学科困境，Sutton 与 Javed 并非纸上谈兵，这也是他们创立 Oak Lab 的核心契机。Sutton 回顾了自己在 2022 年公开发布的《AI 研究的阿尔伯塔计划》（The Alberta Plan for AI Research）。该计划将通往真正智能的庞大构想拆解为具体的 12 个台阶： - **步进核心：持续深度学习（Continual Deep Learning）**：阿尔伯塔计划中的第二步（Step 2）至关重要。Sutton 视其为解开后续所有技术链条的母钥匙——只有网络在参数部署后依然能够不间断地持续进行深度学习，智能体才可能随时随地更新其对世界的动态认知模型。 - **自适应抽象构建（Learning the Abstractions）**：计划的后半部分全部围绕“如何构建正确的抽象”展开。Sutton 特别澄清，世界上根本不存在客观普适的“标准抽象”，抽象的有效性完全取决于智能体当前所置身的具体环境；因此，AI 必须拥有在自身所处世界中自主提炼契合该环境抽象的能力。 Javed 总结道，学界当前其实普遍达成了“需要构建内部模型并利用模型进行规划”的共识，但真正的硬骨头在于“抽象”：唯有掌握了自主抽象，智能体才能明确世界模型在做推演预测时究竟应当基于何种条件与表征进行条件化（conditioned on）。 ### [38:19](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2299s) 抽象概念的来源与持续深度学习的算法鸿沟 对世界的预测不仅关乎行动及其结果，更核心的问题在于：模型应当预测什么，以及这些预测所依赖的抽象表征究竟从何而来？Khurram Javed 以顶尖运动员为例指出，精英选手在做特定高难度动作时，往往拥有一套极其小众、甚至独创的专业术语；即便在独自训练无须对外部交流时，他们脑海中也存在着难以言表的专用感知与动作抽象。这种自主形成抽象概念的能力，正是当前 AI 缺失的关键一环，也是“阿尔伯塔计划”（Alberta Plan）后半部分试图解决的核心命题。 针对持续深度学习（Continual Deep Learning），主持人追问这究竟是一个算法层面的鸿沟，还是仅仅是部署架构、工程实现或数据隐私方面的工程限制。Khurram 明确表示，这百分之百是一个算法鸿沟。虽然现有工程手段支持在收到用户交互后进行朴素的权重更新，但这会立即引发严重后果：如果拿单一样本去更新整个模型的权重，模型此前积累的所有先验知识都会遭受灾难性的负面冲击。像 Cursor 这样的现代工具之所以看似实现了持续更新，是因为它们绕过了“单流在线学习”的本质——它们收集海量用户的交互构建大批次（Batch）进行离线或半在线更新。但在绝大多数现实场景中，智能体面对的只有源源不断的单数据流（Single stream of data），如果强行应用朴素反向传播，将以极具破坏性的方式摧毁原有的记忆与能力。 - **运动员的启发**：顶尖动作背后高度专业化、甚至不可言传的细粒度抽象，说明智能体必须具备从自身交互中沉淀内部表征的机制。 - **批处理并非持续学习**：依靠大规模多用户数据池的批处理更新只是回避了问题，无法推广到依赖单条经验流的自主智能体。 --- ### [40:00](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2400s) 治愈灾难性遗忘：步长优化与 Continual Backprop 面对被称为“灾难性遗忘”（Catastrophic Forgetting）的经典难题，Rich Sutton 表现得非常乐观，断言这“完全是可以治愈的”，前提是必须掌握正确的算法框架。他提出了解决灾难性遗忘的两大关键支柱： 1. **步长优化（Step-size Optimization）与元学习**：网络中的每一个权重都必须拥有独立的学习步长。这意味着部分权重变化快，而大部分权重的步长会被元学习调整得极小。当新样本到来时，更新只会精准发生在相关区域，而网络主体知识绝不会被冲刷抹去。 2. **特征空间的“生成与测试”（Generate and Test）**：传统的梯度反向传播机制极度缓慢，它只敢沿着已知有益的梯度方向挪动，无法赋予模型自我扩张复杂度、维持终身学习的能力。真正的持续学习必须跳出梯度束缚，持续主动提议并生成一批全新单元（Units）。 Rich 具体介绍了他们在《自然》（*Nature*）杂志上发表的核心算法——**Continual Backprop（持续反向传播）**。传统 Backprop 仅在最初阶段随机初始化一次权重，随着训练推进，这种最初的多样性与随机性便被迅速消耗殆尽；而 Continual Backprop 则像不断播撒种子一样，在网络运行全过程中持续注入随机初始化的新神经元节点，通过“新节点生成 + 反向传播测试筛选”的动态演化循环，保持网络的持续可塑性。 ```mermaid flowchart TD accTitle: Continual Backprop 双重学习机制 accDescr: 展示了通过独立步长元学习与动态生成测试解决灾难性遗忘的过程。 subgraph MetaOptimization["机制一：针对权重的步长优化"] A["输入单条连续数据流"] --> B["元学习调整每个权重的 Step-size"] B --> C["关键区域大步长快速适应"] B --> D["大部分区域极小步长锁定已有知识"] end subgraph DynamicUnits["机制二：Continual Backprop 生成与测试"] E["持续注入随机初始化的新神经单元 (Generate)"] F["利用标准反向传播进行验证与筛选 (Test)"] E --> F end C --> G["保持终身可塑性，根治灾难性遗忘"] D --> G F --> G ``` 对于这套机制是否适用于现有的大语言模型（LLM），Rich 和 Khurram 澄清：你无法直接拿现成训练好的开源大模型硬套这套算法。因为这些算法需要在学习具体知识的同时“元学习如何去学习（Meta-learn how to learn）”，两种学习进程是高度耦合且并行的。因此必须使用全新算法从零构建新的基座模型（Foundation Model），唯有如此，模型才能在摄入新经验时既学到新事物，又天然免疫灾难性遗忘。 - **发表成果**：Continual Backprop 的相关理论已发表于《Nature》，确立了结合非梯度探索与梯度评估的持续学习范式。 - **双重学习目标**：新范式下的模型在训练时不仅吸收任务本身的知识，同时在优化未来摄入新经验时的权重更新规则。 --- ### [42:57](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2577s) Oak Lab 的最大野心：自主维护的统一心智 被问及新创立的 Oak Lab 是否在进行全行业最激进（Radical）的尝试时，Rich 幽默地重申了自己的口头禅：“我并不奇怪，是这个领域里的其他人变奇怪了”，他更倾向于用“最具雄心（Most ambitious）”来形容他们的目标。Khurram 补充道，在 2016 至 2018 年间学术界曾广泛讨论过元学习与持续学习，但当时的研究被局限在特定分布的多任务设定下；而 Oak Lab 致力于在“单条连续经验流”这一最普适的现实约束下彻底攻破这一难题。 Rich 阐述了 Oak Lab 的终极愿景：构建一个具备完整知识图景、能够**自我维护（Self-maintaining）的心智**。人类的思维既能处理极其微观的动作细节，也能理解诸如“乘坐飞机跨越城市”这种高度概括的宏观抽象——这种自底向上的跨尺度抽象是全人类（甚至松鼠）共有的本能。传统的大语言模型之所以能保持知识正确，完全依赖人类通过海量后训练（Post-training）进行纠偏并随之将权重冻结；一旦解冻，模型便极易陷入混乱。然而人类心智时时刻刻在发生改变，却能维持高度的自洽与连贯性，绝不会随意漂移到疯癫状态（"drifting off into crazy land"）。打造一个能够自主保持连贯、持续自我训练且内部自洽的心智系统，正是 Oak Lab 最大的野心。 - **与过往研究的区别**：跳出小众的人工多任务分布，直面真实世界不可逆的单流连续交互环境。 - **自我维护机制的对比**： - **当前 LLM**：依靠人工后训练对齐知识后彻底“冻结”，丧失在线适应性。 - **Oak Lab 目标心智**：在无休止的自我修改与权重流动中，凭借内在结构自主维系一致性与常识边界。 --- ### [46:02](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2762s) 20瓦功耗运行万亿参数：摩尔定律与能耗信仰 主持人对这一宏大设想提出了极具现实感的质疑：Oak Lab 设想在未来以仅仅 **20 瓦** 的超低功耗运行一个 **万亿参数（Trillion-parameter）** 的心智模型，这在物理和工程上是否可行？ Khurram 坦言，如果以当下的内存与芯片制造技术来看，这一目标在物理上几乎是不可能的——仅仅在当前的存储器中维持一万亿参数的通电状态，消耗的功率就远远超过 20 瓦。但这项规划立足于未来 5 到 10 年的技术演进。Alfred Lin 现场算了一笔账：如果遵循摩尔定律每 18 个月翻一番的经典节奏，10 年的时间跨度将带来两个数量级（约 100 倍）的能效提升。因此，若要在 10 年后实现 20 瓦的运行功耗，今天算法在同等任务上的基准线应当是 **2000 瓦**。 Khurram 坚信，如果配合正确的稀疏化与持续学习算法，即使以当下的算力条件，他们也能在能效上做到超越 2000 瓦的标准。面对主持人“如果现在就能做，为什么前沿大实验室不做？难道他们只是喜欢挥霍金钱吗？”的追问，Rich 笑着调侃道：“有时候看起来他们就是想花钱，仿佛只有消耗天量能源才能证明自己是‘真男人’（Real men）。”Khurram 进而深刻地指出，主流实验室之所以没有朝这个方向努力，核心症结根本不在于技术是否可行，而在于大家甚至压根不相信这是可能的；当一个研究群体失去了对极低能耗、持续演化路线的信仰，他们就绝不可能沉下心去攻克这一分支背后的深层技术障碍。 - **物理能耗与摩尔定律测算**： - 终极目标：5-10 年内以 20 瓦（人体大脑水平能耗）维持万亿参数智能体。 - 阶段映射：按 10 年 100 倍计算，当前对应的功耗基线应压缩至 2000 瓦以内。 - **行业心理反思**：前沿实验室深陷比拼能耗与算力规模的局部最优，因缺乏对高能效终身学习的底层信仰，导致相关算法探索陷入停滞。 ### [47:58](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2878s) 巨头实验室的局部最优陷阱与范式转换之痛 面对主持人关于主流前沿实验室是否存在巨大资源浪费的疑问——是不是每十个人里其实只有一个人知道如何高效推进，而其余九个人都在徒劳折腾——Khurram Javed 给出了一个极为精辟的系统性解释：整个 AI 领域目前正深陷于一个“局部最优”（local minimum）。在现有框架下，主流实验室沿着当前的 Scaling Law 可以稳步迭代并持续产出商业化产品，这让他们在组织和商业模式上形成了重度路径依赖。然而，Oak Lab 所倡导的这种基于自主持续学习的新算法体系，在探索初期“几乎不可避免地要经历一段先变差、后变好的阵痛期”（"it is almost impossible that things will not get worse before they get better"）。 对于背负着庞大产品营收与季度交付压力的大厂而言，转向一条初期性能必然下降、无法在第一天就打破 SOTA 的全新道路，几乎是不可能的抉择。Khurram 指出，探索这条路径不仅需要敢于下注，更需要解决大量极具挑战性的底层技术难点；而这些难题，他和 Rich Sutton 以及少数志同道合的学者已经默默思索了许多年。现有巨头看似在加速狂奔，实则被锁死在既有的算力与工程堆叠范式中，无法抽身涉足根本性的范式转移。 ```mermaid flowchart TD accTitle: 巨头实验室的局部最优与范式转移困境 accDescr: 对比前沿大厂停留在现有范式与探索持续学习新范式所面临的性能与风险抉择 subgraph Current["现有大模型范式（局部最优）"] A["沿现有架构持续扩展 (Scaling)"] --> B["短期维持 SOTA 并支撑现有商业产品"] B --> C["沉没成本与交付压力巨大<br/>无法承受性能下滑阵痛"] end subgraph ParadigmShift["持续学习新范式（全局最优探索）"] D["引入全新算法与自适应架构"] --> E["探索初期性能暂时落后 (Worse)"] E --> F["攻克理论与架构难关<br/>实现人脑级能效与真正智能 (Better)"] end C -.->|"难以跨越鸿沟"| D ``` - **局部最优的诅咒**：大模型厂商受制于既有产品的迭代周期与市场预期，无法承担探索新范式初期必然遭遇的性能回撤。 - **能耗与架构天花板**：现有范式依赖不断膨胀的高能耗计算，而新范式虽起点艰难，其终极目标却是在极低能耗（如人脑级别的 20 瓦）下实现甚至超越同等的智能水准。 ### [49:13](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2953s) Oak Lab 的愿景：语言模型只占智能的四分之一 当被问及如果 Oak Lab 的构想全盘成功、究竟会塑造一家怎样的公司时，Rich Sutton 勾勒出了一幅构建“真正智能”的蓝图：全面落地全新的持续学习架构，使系统具备真正的自我维护与抽象构建能力，从而像生物一样自发进行高阶规划与复杂推理。主持人半开玩笑地插话：“那人类岂不是要变得无关紧要了？”Rich 随即微笑着断然否定，表示世界不仅不会陷入人类虚无主义，反而会因真正智能的诞生而变得前所未有的精彩与有趣。 谈及技术格局的演变，Rich 直言不讳地指出，一旦这种真正的自主智能架构成熟，现有的纯大语言模型（LLM）很可能会面临被彻底颠覆的风险。Rich 澄清自己绝非全盘否定 LLM 的历史功绩，相反，他认为大语言模型是神经网络技术史上一次“完全出乎预料、极其辉煌的科学突破”，彻底击溃并取代了语言学界长期顽固坚守的符号主义方法。但他感到深深沮丧的是，当下行业未能理性地将 LLM 视作 AI 领域某一特定子问题的局部里程碑来庆祝，反而将其神话成了智能的全部。在 Rich 看来，流畅巧妙地驾驭语言最多只占人类全部智能的“五分之一或四分之一”（"like 20% or a quarter of intelligence"），智能的世界广袤无垠，远未到可以轻言大功告成的时候。 - **LLM 的真正贡献**：攻克了神经网络流畅运用自然语言的难关，颠覆了传统的符号主义语言处理流派。 - **行业认知的偏差**：流畅的语言生成被过度夸大为全能智能，忽视了感知、具身交互、世界模型构建与动态规划等更为核心的智能组成部分。 ### [51:02](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=3062s) 单一设计与多元心智：大世界假说下的智能分化 针对未来真正的人工智能是否会是一个包罗万象、用同一套静态权重既能学会“在树枝间荡秋千”又能“设计制造宇宙飞船”的单一超级心智，Rich 和 Khurram 依据“大世界假说”（Big World Hypothesis）给出了截然相反的构想。Rich 明确表示，Oak Lab 追求的是一种“单一的通用架构设计”，但这种设计在现实世界中演化出的将是“无数个形态各异的心智”（many minds），绝非一个单一僵化的全知全能网络。 Khurram 进一步阐释，大世界假说的核心立论在于：现实世界所蕴含的复杂度与信息量是无限的，没有任何单一计算系统能够穷尽无限的信息。这意味着，即使部署两个世界上规模最大、架构相同的顶级智能系统，因为它们的复杂度处于同一量级，它们之间从数学和逻辑上就无法做到彼此绝对建模。由于不同的智能体在环境中拥有完全不同的体验轨迹和交互历史，同样的底层设计必将分化出专注于不同领域、拥有差异化知识结构的多元心智。 - **架构统一与心智多元**：通过相同的底层学习机理与可塑性规则，在不同物理或虚拟环境中自适应生长出千差万别的心智实体。 - **不可完全建模定理**：当两个系统的复杂程度相当时，彼此无法建立包含对方全部状态的完备模型，决定了未来智能生态必然是群体协作、持续交互的分布式心智网络。 ### [52:04](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=3124s) 极度对齐的团队构建与结语 在节目的尾声，话题转向了 Oak Lab 的人才招募策略。Khurram 透露，虽然 Oak Lab 正在招募英才，但核心初始团队的绝大部分成员其实早已在他们的脑海中有了明确人选。这些人几乎全是在持续学习与强化学习前沿长期深耕、对该范式抱有长期信念的资深研究者。由于 Oak Lab 正在践行一条与主流背道而驰的新范式，团队绝不会采取盲目堆砌人头的急速扩张模式，而是计划从极小的精英编制起步，极其审慎地逐步扩充到两到三拨核心成员。 Rich 和 Khurram 一致强调，“极度高度的一致与对齐”（"We want to be super aligned"）是团队初期能否爆发高生产力的命脉所在。每一位加入的同仁都必须真正看清并认同新范式的核心逻辑，只有理念完全合拍，团队才能在探索未知的过程中高效协同，将持续学习的理论成果稳健转化为飞跃式的工程进展。主持人由衷赞叹 Rich 始终是强化学习与算法架构领域不可多得的深邃思考者，并感谢两人带来这场颠覆当下 AI 认知盲区的深度思想碰撞。 ## Notable quotes > [00:00](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=0s) 我觉得自己的想法非常普通，反倒是其他人的思维变奇怪了。 — Rich Sutton > [00:00](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=0s) 我并不反常。反常的是这个领域，它居然需要造出‘持续学习’这个词，而它原本就只是普通的学习。 — Rich Sutton > [07:08](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=428s) Don't be distracted by human knowledge as AI traditionally has been many times. Instead focus on learning methods that will scale with computation, like search and like learning. — Rich Sutton > [11:03](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=663s) 不，那只是一个巨大的错误（that's just a big mistake）。 — Rich Sutton > [19:27](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1167s) 奇怪的不是我。奇怪的是你们，居然会认为能从一个完全不再学习的东西身上获得博士级别的经验与专长。 — Rich Sutton > [22:02](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1322s) 它们的权重从不改变。（Their weights never change.） — Rich Sutton > [26:09](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=1569s) 自然界中没有任何一种动物是通过“监督学习”（Supervised learning）来学习的。 — Rich Sutton > [47:58](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2878s) it is almost impossible that things will not get worse before they get better — Khurram Javed > [46:02](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2762s) 有时候看起来他们就是想花钱，仿佛只有消耗天量能源才能证明自己是‘真男人’（Real men）。 — Rich Sutton > [49:13](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=2953s) like 20% or a quarter of intelligence — Rich Sutton > [52:04](https://www.youtube.com/watch?v=xH7U7w9Qzlo&t=3124s) We want to be super aligned — Rich Sutton / Khurram Javed ## Mentioned Rich Sutton, Khurram Javed, Sonya Huang, Alfred Lin, David Silver, Oak Lab, Sequoia Capital, University of Alberta, OpenAI, Anthropic, DeepMind, AlphaGo, AlphaZero, The Bitter Lesson, The Alberta Plan for AI Research, Big World Hypothesis, Continual Backprop, Nature, ImageNet, Cursor, Lean, Moravec's paradox ## Video description Rich Sutton, who helped pioneer reinforcement learning and wrote the seminal AI essay The Bitter Lesson, has now cofounded Oak Lab with his former student Khurram Javed. Their goal: to build agents that continuously learn from their own experience rather than from us. Rich doesn't think he holds a radical view: "I'm not weird. The field is weird." He says all learning is continual, and the field is the one that needed a new name for it. Rich and Khurram argue synthetic data is "a big mistake." Their "big world hypothesis" is that the world is massively more complex than any agent or simulator, so approximations have to be updated continuously rather than frozen at deployment. Rich calls LLMs an unanticipated scientific breakthrough, but says they represent roughly a quarter of intelligence. He says catastrophic forgetting is "totally curable" with the ideas behind their continual backprop algorithm. Khurram explains why the frontier labs can't follow: they sit in a local minimum where a new paradigm gets worse before it gets better. Their target, five to ten years out, is a trillion-parameter mind that keeps learning, stays coherent, and runs on 20 watts. Hosted by Sonya Huang and Alfred Lin, Sequoia Capital 00:00 Introduction 02:10 An AI winter, a cancer diagnosis, and the move to Alberta 07:07 Writing "The Bitter Lesson" — and the 26-word version 09:53 Are LLMs a positive or a negative example of it? 11:03 Synthetic data is "just a big mistake" — and the Big World Hypothesis 18:01 AlphaGo, human priors, and why prior knowledge and learning should be friends 22:37 "Their weights never change" — do LLM assistants actually learn? 26:09 Babies, squirrels, and why no animal learns by supervised learning 32:02 Rockets, imagination, and where paradigm shifts come from 36:42 The Alberta Plan and its 12 steps 38:53 Catastrophic forgetting — and the cure 43:43 Oak's biggest ambition: a self-maintaining mind 47:56 Why the big labs are stuck in a local minimum 49:13 If everything goes right — LLMs, many minds, and hiring [Source](https://www.youtube.com/watch?v=xH7U7w9Qzlo)

[0:00] Rich Sutton: 我觉得自己的想法非常普通，反倒是其他人的思维变奇怪了。 [0:00] Rich Sutton: 我并不反常。反常的是这个领域，它居然需要造出‘持续学习’这个词，而它原本就只是普通的学习。 [7:08] Rich Sutton: Don't be distracted by human knowledge as AI traditionally has been many times. Instead focus on learning methods that will scale with computation, like search and like learning. [11:03] Rich Sutton: 不，那只是一个巨大的错误（that's just a big mistake）。 [19:27] Rich Sutton: 奇怪的不是我。奇怪的是你们，居然会认为能从一个完全不再学习的东西身上获得博士级别的经验与专长。 [22:02] Rich Sutton: 它们的权重从不改变。（Their weights never change.） [26:09] Rich Sutton: 自然界中没有任何一种动物是通过“监督学习”（Supervised learning）来学习的。 [47:58] Khurram Javed: it is almost impossible that things will not get worse before they get better [46:02] Rich Sutton: 有时候看起来他们就是想花钱，仿佛只有消耗天量能源才能证明自己是‘真男人’（Real men）。 [49:13] Rich Sutton: like 20% or a quarter of intelligence [52:04] Rich Sutton / Khurram Javed: We want to be super aligned

[0:00] Rich Sutton: I'm not weird. The field is weird. [7:07] Rich Sutton: Don't be distracted by human knowledge as AI traditionally has been many times. Instead focus on learning methods that will scale with computation like search and like learning. [18:01] Rich Sutton: Seeking the improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain. But the only thing that matters in the long run is the leveraging of computation. [34:45] Rich Sutton: The big challenge that we don't see in our field... is the ability to learn a model and then plan with the model... If we have to learn the models, there are no—I'm going to say it, it's maybe a counterexample—but I can see that there's no instances of learning the model and then planning with the model in our field. [38:53] Rich Sutton: Catastrophic forgetting... is totally curable. You have to have the right algorithm... Every weight in your network has to have a separate step size. So, some will move fast, some will move slow... With continual backprop, we keep injecting a bit of randomness, a bit of generate and test... and the operation of backprop is the tester. [43:43] Rich Sutton: I think that's how they show they're real men—by using lots of energy... At least when I look at different research groups, I don't even see anyone believing in that it's possible. And I think if you don't believe in it, you're just not going to work on the technical problems and work through them.

[0:00] People think I'm have a radical point of [0:02] view sometimes. They say they start [0:04] questions saying how what I'm thinking [0:06] is so different from everyone else. But [0:07] I don't see it that way at all. I see it [0:10] as like I'm thinking the ordinary way. [0:11] It's just everyone else that's thinking [0:13] a bit weird. And [0:15] >> [laughter] [0:17] >> And I mean that like, you know, it's [0:18] just the recent times people are [0:20] thinking weird. Before there was all [0:22] this AI craziness, [0:24] uh [0:25] you talk about you wouldn't have to say [0:26] continual learning cuz [0:28] it wouldn't make any sense to talk about [0:29] learning that wasn't continual. All [0:31] learning is continual. We always act and [0:33] we learn. That's just the normal way of [0:35] thinking. I'm not weird. [0:37] The field is weird. The field they need [0:39] to call it continual learning. It's just [0:41] learning. [0:50] >> [music] [0:59] >> We are honored to have the great Rich [1:01] Sutton with us here today. Rich, you [1:03] invented reinforcement learning. You [1:05] wrote the seminal textbook. [1:08] You're the [1:09] key students in the field, uh folks like [1:11] Dave Silver. You wrote the essay The [1:13] Bitter Lesson that I believe is the [1:15] Bible of the field. And and you have [1:17] just been one of the greats in [1:18] propelling the field forward. So thank [1:19] you for taking the time to join us [1:21] today. Um Rich is joined by Quorum [1:23] Javed, his co-founder uh and former [1:26] students from the University of Alberta. [1:28] Um the two of you have set off to found [1:30] Oak Lab. I'm very excited to talk to you [1:32] about that today. So for today's [1:34] session, we're going to start talking [1:36] about The Bitter Lesson, the state of [1:37] the world as we know it today, [1:39] whether LLMs will get us there or not. [1:41] And then we're going to we're going to [1:42] transition to start talking about your [1:44] your research agenda and your plan for [1:46] Oak. Um Rich, maybe take us back. I was [1:49] going to start with The Bitter Lesson, [1:50] but I actually want to start earlier [1:52] than that. Decades ago, you decided to [1:56] dedicate your career to reinforcement [1:58] learning, to deep reinforcement learning [1:59] in particular, and you established the [2:01] University of Alberta as a bastion of [2:04] that [2:05] back when I think the field was very [2:07] much in its infancy. What gave you the [2:08] conviction to do that? [2:10] >> What else you going to do? [2:12] >> [laughter] [2:12] >> We were trying to figure out the mind [2:14] and learning is a central part of the [2:16] mind. [2:17] And having a goal is a central part of [2:19] the mind. Central part of intelligence. [2:22] Yeah, so I was just doubling down on [2:24] what I was always thinking. [2:26] >> Did people think you were crazy at the [2:27] time? [2:29] >> Um [2:31] It was a winter. It was an AI winter. [2:33] >> Uh what year was this? [2:35] >> It was in 2003. [2:37] >> Okay. [2:38] >> And [2:39] it's kind of crazy actually the truth [2:42] cuz I was like really sick. [2:43] I was dying I was actually dying of [2:45] cancer in 2003. And but I I wasn't quite [2:50] dead, you know, I've been trying for a [2:52] number of years. And I wasn't dead after [2:55] another remission. And so [2:58] so I said, well, I'm not dying I haven't [3:01] succeeded in dying. So I might as well, [3:04] you know, it's going on long enough [3:05] might as well just try to get another [3:06] job. And so I so I went to Alberta and [3:10] and and started teaching there. [3:12] And then in the end I didn't die. It's [3:13] kind of amazing it's cuz [3:15] you know, it's it's like that. I'm [3:16] joking about it now but it was quite [3:18] serious. And um [3:20] it's an even more important question. [3:22] Why why did I continue [3:26] to work on this research stuff when I [3:27] was, you know, I only had a few months. [3:30] I would always [3:32] keep reminded what [3:33] I think it's Benjamin Franklin is [3:35] supposed to have said that, you know, if [3:36] you ever wonder why someone is doing [3:38] something [3:39] it's almost always one of two things. [3:41] It's either habit [3:43] or vanity. Okay? So I think I think it's [3:46] probably true. Maybe it was my habit to [3:48] just kept doing what I'd always been was [3:50] doing or maybe it was vanity. [3:52] I I know. I think it was more like habit [3:53] cuz I was I was dying. [3:55] >> Wow. [3:56] >> Um [3:56] >> Wow. Divine intervention. [3:57] >> Yeah, it's always been easy for me to [3:59] keep be very determined. [4:02] Um and [4:04] I'm I'm I'm going to go even longer on [4:05] this answer. [4:06] >> Please go. [4:08] >> People think I am have a radical point [4:10] of view sometimes. They they they start [4:12] questions saying how how what I'm [4:14] thinking is so different from everyone [4:16] else. But I don't see it that way at [4:18] all. I see it as like I'm thinking the [4:20] ordinary way. It's everyone else that's [4:22] thinking a bit weird. [4:24] >> [laughter] [4:26] >> And I mean that like, you know, it's [4:28] just the recent times people are [4:30] thinking weird. If you go look back what [4:31] what what what people thought about the [4:33] mind for, you know, even just a decade, [4:36] you'll find the kind of thoughts that, [4:38] you know, learning is important. You've [4:40] got to have a goal. Um [4:43] and you know, perception is important. [4:45] We have a We are We are low-level We are [4:47] low-level beings. We are generating [4:49] actions and perceiving data at a fast [4:52] speed and yet we have to think at higher [4:54] levels. And you know, go back a few [4:57] before there was all this AI craziness, [5:00] uh [5:01] you talk about you wouldn't have to say [5:02] continual learning cuz [5:04] it wouldn't make any sense to talk about [5:05] learning that wasn't continual. All [5:07] learning is continual. [5:08] You know, you [5:10] It's not a special phase. We always act [5:13] and we learn. That's just the normal way [5:14] of thinking. I'm not weird. [5:17] The field is weird. The field they need [5:19] to call it continual learning. It's just [5:21] learning. [5:22] >> I'm not weird, everybody else is. That's [5:23] a [5:24] good good motto to live by. [5:26] >> We're going to have to send out a an ex [5:28] post about that. [5:29] >> [laughter] [5:30] >> We're very happy that you you you lived [5:32] on with the field is happy that you [5:34] lived on and thank you for pushing the [5:36] frontier of AI. [5:36] >> happy. [5:37] >> [laughter] [5:38] >> Thank you for pushing the frontier of [5:39] AI. I'm sure really happy and thank you [5:41] for all of that and push you've like [5:43] been able to sort of educate a lot of [5:45] students who pushed the frontier as [5:46] well. [5:47] How did you pick them? How did you [5:49] over the last 20, 30 years? [5:51] >> Oh, well, you are giving me [5:52] opportunities to be uh [5:54] to be humble. Like I like to be humble [5:56] and point out how all these great [5:58] decisions are are just happen. [6:01] And it's that that's the way I feel [6:03] about students. I don't feel that I [6:05] choose them very well. I've just [6:07] Sometimes I'm lucky, sometimes I'm [6:08] unlucky. [6:10] I don't feel I'm particularly good at [6:11] picking my students. [6:14] I'm looking at Karam. I think sometimes [6:16] you end up with the really great ones. [6:18] David picked David Silver picked me. [6:20] >> Yeah. [6:20] >> How is it that [6:21] that I got you, Karam? [6:23] >> Yeah, that was also so I [6:25] finished my master's not with you [6:27] uh and I was planning to join industry. [6:30] And then we were collaborating on a [6:33] project which also just had started [6:35] organically. Like there was something I [6:37] worked on that Rich was in a meeting, [6:39] then they mentioned that I worked on it. [6:40] So I got pulled into it. We started [6:42] collaborating. It went really well. Like [6:44] I felt so happy with that collaboration. [6:47] Rich also felt really good about it. And [6:49] then 6 months down the road we had made [6:51] some progress and it just made sense to [6:53] convert that into a thesis proposal. So [6:56] at no point did I apply, at no point did [6:58] I ask should you be my PhD advisor. We [7:01] worked together, then we decided this [7:03] would be a pretty good thesis. [7:05] And then then after that I applied for [7:07] the PhD. [7:08] >> Life works in unexpected ways. [7:10] Uh take us to 2019. You wrote the bitter [7:13] lesson which has become the the mother [7:16] in tome. [7:17] 2019 was a funny time to be writing that [7:19] piece because ImageNet was 2009. AlphaGo [7:23] was 2015. What caused you in 2019 to [7:26] reflect and and to write that? Because [7:28] it was before the current kind of [7:30] scaling paradigm around large language [7:32] models had taken off, but it was after [7:34] deep learning had really proven itself. [7:36] >> Well, it was a long time coming. You [7:38] know, as the [7:39] bitter lesson expresses, it's something [7:41] that you can [7:42] for a long time, for many decades. [7:45] And it's definitely at least as much due [7:48] to the round of [7:50] symbolic AI, which I lived through. [7:53] It's all about not getting distracted by [7:56] trying to put in your human knowledge [7:58] and just paying attention to what the [7:59] problem needs and how you can scale with [8:02] computation. [8:03] I know I I I wrote versions of it at [8:06] least a year before and I I gave talks. [8:10] I gave a talk a year before. [8:12] And um [8:13] it wasn't a particular response to the [8:16] moment. It was a particular response [8:19] to my my long experience, different [8:22] people trying to think in different ways [8:25] about how you can make smart systems. [8:28] >> Mhm. [8:29] What is the essence of the bitter [8:31] lesson? [8:32] >> The essence of the bitter lesson. [8:33] >> You know, and maybe the phrase that I [8:35] hear used the most in my meetings these [8:37] days is is bitter lesson pill, is it not [8:39] bitter lesson pill? [8:41] I would imagine given the popularity of [8:42] the phrase it's probably been tortured [8:44] and misused in different ways that you [8:46] didn't originally intend it. So, what do [8:48] you think what is the essence of it and [8:49] where do you think people go wrong in [8:51] their in their attempt to understand it? [8:53] >> Yeah, you're you're making me think [8:55] about X now and my I recently made a [8:58] post where I tried to do the the bitter [9:00] lesson in in 26 words. [9:03] It [laughter] goes something like [9:05] don't be distracted by human knowledge [9:09] as AI traditionally has been many times. [9:13] Instead focus on [9:15] learning methods that will scale with [9:18] computation [9:20] like search and like learning. So, it's [9:23] really all about uh focusing on [9:27] algorithms and improvements. It's not [9:30] it's not saying you don't need [9:32] fancy algorithms. You need fancy [9:34] algorithms, but you want fancy [9:36] algorithms that will scale with scale [9:38] with computation. [9:40] >> Rather than scaling with data. [9:41] >> Rather than scaling with human input. [9:45] Yeah, and then the the question if I can [9:47] anticipate it [9:49] um [9:51] Yeah, what about large language models? [9:52] >> Yeah. [9:53] >> Are they [9:54] >> consistent or inconsistent with your [9:56] >> Yeah. [9:57] >> And and I thought about this and I think [10:00] there's an ex- there's another exposed [10:01] model, but the conclusion is that it's [10:04] both a a positive example and a negative [10:06] example of the big lesson. First uh [10:10] large language models enabled uh [10:13] enormous scaling with computation. And [10:16] you could just [10:17] drink in the internet and scale so much. [10:20] So it was a it was a way of getting uh [10:22] much more capable system just by [10:25] methods that scale. Then after that, as [10:28] you go on further um it eventually gets [10:31] limited by by that information. The the [10:34] internet is finite and it's hard to get [10:38] more examples. [10:39] And uh the world is big and the world is [10:43] massively bigger than everything we [10:44] stored [10:45] on the internet. And so in the end, it [10:48] seems like it could be [10:50] uh I guess that would be a [10:52] positive example of when, you know, we [10:54] relied too much on human knowledge and [10:56] it eventually [10:58] holds us back. [10:59] >> Mhm. [11:00] Can I just push on this a little bit? [11:02] >> Yeah. [11:03] >> It seems like a lot of what the [11:04] foundation model labs are working on [11:06] right now is synthetic data generation [11:09] in order to kind of get us beyond [11:12] the fossil fuel that is the existing [11:14] human internet. [11:15] Um is synthetic data generation kind of [11:18] as part of this LLM scaling paradigm, is [11:20] that [11:21] a general method that leverages [11:23] computation? [11:24] >> No, that's that's just a big mistake. [11:27] >> Why? [11:27] >> [laughter] [11:28] >> Well, it's such a big it's such a maybe [11:30] it's the next the next big lesson. [11:33] Um [11:34] it's been floating around [11:37] uh Alberta for 5 or 10 years. [11:40] >> Okay. [11:40] >> And uh we call it the big world [11:42] perspective or big world hypothesis. [11:44] Khuram, who eventually wrote it up as a [11:46] paper. There's a little paper called the [11:49] big world hypothesis. [11:50] >> So, the big world is [11:52] that the world is infinitely big. There [11:55] are infinitely many things to learn. [11:57] And you can have people generating the [12:00] synthetic data sets, but there will [12:02] always be more things to learn. And [12:04] because of that, if you could just [12:06] learn from experience, if you could [12:08] remove the humans from the loop, then [12:10] you would have systems that can do [12:12] everything. Because, you know, the world [12:14] is big. There are many tasks that we [12:15] want them to do. [12:17] And they would be able to do anything by [12:18] learning from their experience. Going [12:20] back to the synthetic data question, [12:21] too. Uh [12:23] who decides what's a good synthetic data [12:25] and what's a bad synthetic data? Because [12:27] I can write a program that can output a [12:28] lot of synthetic data, which would hurt [12:30] programs. Right now, I would say humans [12:32] decide. And that's the bottleneck where [12:35] okay, you can have humans deciding how [12:37] to generate these data sets, but you [12:39] need human experts who know what's a [12:40] good data set and what's a bad data set [12:42] for that approach to scale. So, it is [12:44] bottlenecked by humans. [12:45] >> Doesn't my loss curve decide like how [12:47] much better did I get with this data set [12:49] versus that better data set? [12:50] >> Right. But, if all the engineers open AI [12:53] and tropic or all the big new labs and [12:55] engineers went on vacation, [12:57] who would generate the synthetic data? [12:58] That's the question. It doesn't doesn't [13:01] come from agent's experience. It's not [13:03] something that the agent is generating [13:04] itself. [13:05] Some human has to decide what is the [13:08] right synthetic data to generate. And [13:09] that requires human expertise. So, for [13:11] example, [13:12] if you want a system to do something [13:15] very challenging from a physics point of [13:16] view. Maybe you want a drone that flies [13:18] with echolocation, like a bat, for [13:20] example. Um what's the right synthetic [13:22] data for that? I think you would need to [13:24] hire domain experts to go figure out [13:28] what is the right data and generate it [13:29] and then maybe you would be able to [13:31] learn from that. But [13:33] the domain expert has to exist first. [13:35] So, we are bottlenecked by human [13:36] expertise at that point. [13:37] >> But you can have infinite synthetic [13:39] worlds. The existing world is finite. [13:42] >> But let's go back to the echolocation [13:44] thing, right? That's what I want. I want [13:46] a drone that can uh [13:48] localize itself and move with [13:50] echolocation. [13:51] That's my goal. [13:53] Um the robot that that's a robot that's [13:55] generating its own experience. So, it [13:56] could totally learn from its own [13:58] experience, [13:59] but it wouldn't be able to It doesn't [14:00] matter how much synthetic data you [14:02] generate. Doesn't matter if you generate [14:04] synthetic data that captures 50 [14:06] different universes, it will not be not [14:08] allow you to do that task without humans [14:11] figuring it out first. [14:12] >> I first just say it's it's the synthetic [14:14] data is wrong. [14:16] I mean, it won't be correct. It'll be a [14:18] synthetic world. It won't be the real [14:20] world. [14:22] And it will matter. [14:23] The world is incredibly complex. If you [14:26] write a little program, cuz this is [14:28] going to be a little program that will [14:29] generate the synthetic data, [14:31] it'll be a very It'll be a small world. [14:35] >> Mhm. [14:36] >> So, [14:37] for example, what's important to me is [14:39] what's going on in your mind right now. [14:42] Okay? And why You're saying, "Why don't [14:43] I get some synthetic data to tell me [14:46] what's going on in other people's [14:47] minds?" No, there's no way we can have [14:50] synthetic data for other people's minds. [14:53] And other people's minds matter to us. [14:55] You know, like [14:57] I talked to you guys about investing [14:58] today, so I care what you going on in [15:00] your minds. [15:01] And how how can I get synthetic data on [15:03] such a thing? [15:05] Really, you can't even get synthetic [15:06] data on anything. You can't get [15:07] synthetic data on on the how the drone [15:09] is going to interact with with its [15:11] environment in the physical world and [15:14] the the [15:15] the [15:16] the friction and where in the motors of [15:19] this of this robot. The world is [15:21] infinitely complex, and any simulation [15:24] of it is like microscopic. The big world [15:27] hypothesis, let's say what it is, is [15:30] that the world is massively more complex [15:33] than your mind, than any agents any [15:36] agent. [15:37] And this is obvious because the world [15:39] contains many other agents. So, because [15:42] the world is is massively complex, as [15:46] you could no way you can do anything [15:48] like [15:49] anything that might claim to be optimal [15:52] or perfect. You're going to be [15:54] imperfect, and you have to have [15:55] approximations, and those approximations [15:57] will will be severe. And so, because of [16:01] that, that is the ultimately the reason [16:03] why we have to continue learning, if you [16:05] want to think of it as a reason. [16:08] We have to continue learning because [16:09] we'll encounter some particular part of [16:11] this immense world, and we'll have to [16:13] learn an approximation that's tuned to [16:15] the part of the world we're in, not to [16:16] the [16:17] all the other parts that we're not in. [16:19] >> Yeah. I'm I'm going to push on this one [16:21] more time. Um, and sorry, I'm being [16:23] argumentative for the sake of being [16:24] argumentative, [clears throat] but I'm [16:25] trying to understand. My understanding [16:27] is that the newest cohort of [16:28] self-driving car companies, many of them [16:31] were primarily trained in sim, [16:33] and then they, you know, have to do some [16:36] some sort of [16:37] post-training, I guess, to to make sure [16:39] they work in the real world, but but [16:40] that it's been a very effective [16:42] pipeline. [16:42] >> Yeah. So, I think the important question [16:45] to ask here is, how many engineers were [16:47] involved in building that simulation, [16:49] and are we ready to say that the only [16:52] problem worth solving are those where we [16:54] can hire a large team of engineers to [16:56] first make a simulation. And I'm sure [16:59] they had to do multiple iterations where [17:00] they made the simulation, they learned [17:02] in it, they realized there was a [17:03] sim-to-real gap that was not acceptable, [17:05] then they fixed it. So, there is this [17:06] human in the loop fixing the simulation. [17:10] Like, they're getting feedback from the [17:11] real world, humans, and then they're [17:12] fixing the simulation. Why can't we just [17:15] remove the human and let the agent do it [17:17] itself. [17:18] >> And then when it actually drives, again, [17:20] something unexpected will happen. [17:22] >> And that's when you really want to learn [17:23] from experience. [17:24] >> So, your point is there's just so much [17:26] more data that's going to come from [17:27] experience than there possibly can be [17:30] from [17:31] humans [17:32] curating and creating data. [17:34] >> Yeah, and think there is a there is [17:36] obviously value in learning from [17:38] simulation. And and there is a way of [17:40] doing it. The agent can learn a model [17:41] from its own experience. And when the [17:43] agent learns it, it's very it's much [17:45] better because if the model is [17:46] incorrect, it can fix it by continuous [17:48] learning. [17:49] If the humans are making a simulator, [17:51] then the model only gets updated when [17:52] the humans figure out that something is [17:54] wrong. So, [17:55] yes, planning is important. The agents [17:57] should learn from uh simulators, but [17:59] simulators they make themselves. [18:01] >> Okay, I want to move to another part of [18:02] the bitter lesson, removing human [18:04] knowledge. From your essay, quote, [18:07] "Seeking the improvement that makes a [18:08] difference in the shorter term, [18:10] researchers seek to leverage their human [18:11] knowledge of the domain. But the only [18:13] thing that matters in the long run is [18:15] the leveraging of computation." [18:17] And if I, you know, your former student [18:19] Dave, uh with AlphaGo and AlphaZero, for [18:23] me that was a an example of a triumph of [18:25] removing human priors. [18:27] Did that result surprise you? I guess [18:29] why or why not? [18:31] >> Of course, it made me very happy. It [18:32] made me, you know, feel vindicated. Um [18:36] you know, it could have gone either way. [18:38] It wasn't that [18:40] uh cuz cuz prior knowledge can help. You [18:43] know, there's nothing wrong with prior [18:45] knowledge. You know, and and I say this [18:47] right at the very beginning of the [18:48] right at the very beginning of the [18:50] bitter lesson, I say there's no reason [18:52] why there has to be a conflict between [18:54] prior knowledge and then learning [18:55] knowledge. You know, you can put some [18:57] prior in there and then start learning. [18:59] There's no reason in principle why these [19:01] have to be opposed. In fact, they're all [19:03] about knowledge. You know, life is [19:05] gaining knowledge and having knowledge. [19:08] And [19:09] why are these how somehow, you know, [19:10] nature and nurture became enemies? but [19:12] really you know, prior learning is what [19:15] you already and then and you then you [19:16] learn more and it's they should be [19:18] friends. [19:20] But, as I say at the beginning of the [19:21] bitter lesson, in practice they have [19:24] been enemies. In practice, people who [19:27] who had a an affection for existing [19:30] human knowledge ended up, you know, [19:32] wanting that to win and so they wanted [19:34] to [19:35] to minimize or or dismiss learning. [19:38] And so now, I'm sure your sense of me is [19:40] that I'm someone who who loves learning [19:43] and wants to dismiss prior knowledge. Um [19:47] but you know, I'm really [19:50] someone who's interested in the mind. [19:51] The mind is you have prior knowledge and [19:53] then you get more and then once you've [19:55] gotten more, then that becomes your [19:56] prior knowledge as you get more and more [19:58] and more. [19:59] And this these two things work together. [20:02] I end up appearing to be [20:04] someone who's who's interested in [20:06] learning [20:08] primarily because all the rest of the [20:10] world is is is is talking about all you [20:13] need is enough knowledge. You don't need [20:14] to learn. [20:15] You know, large language models are [20:17] we're going to put all this knowledge in [20:18] the into the system and the large [20:20] language model will not learn when it [20:22] runs. You know, you know, it's talking [20:24] to people, it's interacting. It is [20:25] absolutely the weights never change. [20:28] So, [20:29] you know, [20:30] I am not the weird one. It's you guys [20:32] that are the weird one that think that [20:34] that's possible that you could possibly, [20:36] you know, they claim they can like a PhD [20:38] level experience and expertise out of [20:40] something that doesn't learn at all [20:42] anymore. [20:44] You know, so [20:45] you know, I'm not the weird one. [20:47] >> [laughter] [20:47] >> So, your recommendation is let it let [20:49] the algorithms run for much much longer [20:51] period of time before feeding [20:52] >> Continually learn. [20:54] >> before you feed it data. [20:56] with prior data. [20:58] So, drip or drip the prior data along [21:01] the way. [21:02] prior knowledge. [21:04] >> So, both are important. [21:06] Um but in the long run, you've got to [21:08] gain and structure the gaining of new [21:10] knowledge. [21:12] That's that's what all that matters in [21:14] the long run. And as you are doing this, [21:17] yeah, there'll be some that you had [21:20] got previously. [21:22] Um, [21:23] like how it would work if we, you know, [21:25] look into the future when we have [21:26] intelligent robots. [21:28] We will we will will we [21:30] uh, have them all learn from scratch? [21:34] Or will we like copy them and ask them [21:36] to keep learning from wherever they are? [21:39] I mean, we they'll be digital and it'll [21:40] be easy to copy them. And so, instead of [21:42] having like this huge thing where we're [21:45] spending zillions of dollars to retrain [21:48] them from the internet, we'll just copy [21:50] the agent and keep learning [21:53] from there. And and so, in some sense, [21:55] the prior knowledge will be [21:58] should be dismissed cuz you're just [21:59] going to copy it from the previous [22:01] robot. [22:02] >> So, why don't you describe for us what [22:04] you think a machine or computer that [22:07] learns from experience looks like? [22:09] >> Well, it could look like a robot. It [22:11] could be It also could be live entirely [22:14] on the internet. [22:15] You could like, for example, routing a [22:17] package through the internet and do that [22:19] in a way that's sensitive to experience [22:21] and and becomes better over time. [22:23] Or you can interact via the user [22:25] interface that's interacting with [22:26] people, like on your phone or on your [22:28] computer, and uh, it becomes better over [22:31] time. Yeah, like an intelligent [22:32] assistant, you know, has to become [22:35] better over time. It has to know what [22:36] you want. [22:37] >> Would your contention be that the [22:39] current paradigm of, [22:42] you know, the popular A L M based [22:44] assistants, would your contention be [22:46] that these are not experiential learners [22:48] or continual learners? And if so, what [22:50] is the fundamental gap? [22:52] >> Are you serious? [22:54] >> [laughter] [22:55] >> I mean, obviously they [22:56] >> They they have memo- they learn memories [22:58] about me. [22:59] They're [23:01] you know, they're they're they're doing [23:02] some in-context learning. [23:03] >> Their weights never change. [23:05] >> And and by the way, is a small number of [23:07] the weights changing sufficient or do [23:08] you need all the weights to be changing? [23:10] >> Well, so all think of all the [23:12] structuring and generation of new [23:14] concepts that went into creating the [23:17] large language models. All that is the [23:19] weight learning. [23:21] The [23:22] and you you want to continue be able to [23:24] continue doing that. You don't want that [23:26] to happen just once. [23:27] >> Is another way of saying it is we do too [23:29] much pre-training and post-training [23:31] before we launch the the models. They [23:33] they don't learn after that. [23:35] >> The only point that the big disagreement [23:36] is we don't let them learn after that. [23:38] >> Yeah, we don't let them learn after [23:39] that. [23:40] >> as much pre-training as we want, that's [23:41] okay. Post-training is fine, but then [23:43] when I'm when I'm using the model I [23:46] cannot it's it's it stops learning. [23:48] Uh you can give it more context. You can [23:49] change the state of the model by giving [23:51] it more context. And so it it already [23:53] learned that if the state is different, [23:55] if the state says something new, then it [23:57] will use that to make the next [23:59] prediction. But the model is not [24:00] learning. [24:01] >> Cursor's tab other complete model. It [24:03] is, you know, it does get updated based [24:06] on [24:07] >> Those models those weights change. [24:09] >> weights change. Those are two examples [24:10] like Cursor's tab and I think the [24:11] composer they were also updating. Those [24:13] are two examples of continual learning. [24:15] >> Okay. [24:16] >> it's can be much better. So the way they [24:18] do it as far as I understand is a lot of [24:20] people are using tab, they collect all [24:22] this data so coming from millions of [24:24] users or thousands of users and then [24:26] they do one update of the policy from [24:28] this batch data. Um so now [24:31] this could work, but imagine I want to [24:32] teach this model something specific. I [24:34] don't want to fight with 100,000 other [24:36] peoples about what they want to treat [24:38] teach their models. I want to teach my [24:40] model something very specific and I want [24:41] to do it [24:43] to my version of the model. I like I [24:44] don't care about the shared knowledge [24:46] that the model has coming from other [24:47] people. [24:48] And so it's a very inefficient way of [24:50] doing it. [24:50] >> Mhm. [24:51] It seems like the way that this is [24:53] currently done is that there's [24:54] fundamental skills maybe that are [24:56] learned [24:58] in the weights that are common to [24:59] everybody. [25:00] And then there's personalization that [25:01] happens in the form of context, right? [25:04] >> Yeah. [25:05] >> Is that not the right mental model for [25:07] how learning should work? Like should [25:08] should all the context live in the [25:10] weights themselves? [25:11] >> So, context can be in the state, too. [25:13] Could be both. [25:15] But, you still need to be able to update [25:17] the weights. So, [25:18] if I give you an example, some really [25:21] good use studies are for with human [25:23] disabilities. When human go through [25:24] something that changes their mind or or [25:26] some sensors, you can see them adapt. [25:28] So, for example, we have proprioception, [25:30] we have internal sensors that tell us [25:32] where the how the body is positioned, [25:33] and we use this for walking. [25:35] There are cases where people lose this [25:37] ability completely, and then they can't [25:38] walk at all because that is literally [25:41] the foundation of their walking [25:43] policies. It is ingrained in the brain. [25:45] But then over the course of 2 3 years, [25:47] they can learn to walk again by looking [25:50] at their feet. So, visual feedback [25:52] through that. So, brain is [25:54] insanely plastic in the sense that it it [25:56] can learn a lot of things. Something [25:58] that has been true for 20 years, when it [26:00] stops being true, it can go and update [26:02] that and get rid of that. And that is [26:04] the capability I think that's extremely [26:06] useful we would want in our systems. [26:09] >> Hm. What is there for us to learn from [26:11] how human babies or animals learn? And [26:14] how much inspiration do you take from [26:16] that? [26:16] >> Well, we take uh a lot of inspiration. [26:20] We don't take it as a requirement that [26:22] the AI has to [26:24] behave like [26:25] uh the natural system, like babies or [26:28] people or animals. [26:30] Um but it's it's a source of [26:31] inspiration. Inspiration, but not [26:33] constraint from animal learning. [26:35] >> Consistent with the better lesson? [26:36] >> Yeah. [26:37] >> [laughter] [26:37] >> Yeah. [26:39] >> Where do you think we should most seek [26:42] to draw inspiration from the way that [26:43] biological learnings works that is not [26:46] present in today's systems? [26:48] >> I feel like I'm just giving opinions [26:49] now, but they're just obvious opinions. [26:52] So, so I think it's apparent that [26:55] no animal learns by supervised learning. [27:00] Because [27:02] we don't get examples of how our muscles [27:04] should twitch. [27:06] And that's our output. [27:07] >> But all of school is supervised [27:08] learning. [27:09] >> I I know. Absolutely not. [27:12] Uh, but even if it was, school is like a [27:15] tiny fraction of what we learn. [27:18] Like we learn to see, we we learn to [27:20] walk. [27:22] And we learn, um, [27:24] how the world works. [27:26] But even Yeah, and even in school, you [27:27] know, no one tells us how we should [27:29] twitch our muscles. [27:31] >> The knowledge skills I acquire [27:33] were were from supervised learning in [27:34] school. [27:35] >> don't want to say that that that [27:37] learning from [27:38] from others, transmission from others, [27:40] is not important. It's like extremely [27:42] important. And language is extremely [27:43] important. [27:45] Um, but what are we [27:47] what are we missing? [27:48] You know, [27:49] there is there is no [27:51] supervised learning. There's no targets [27:54] that are given to us. [27:55] You know, you you hear the right answer [27:57] is, you know, [27:59] where where is [28:00] what's the capital of France? And we [28:02] know the answer is Paris. [28:05] Okay, but no one tells me how I should [28:07] pronounce Paris. [28:09] You say the answer is Paris, and I [28:11] listen to you, and I hear your words, [28:13] and, you know, I will make some other [28:16] uh, muscle motions to produce the answer [28:19] Paris. [28:20] It's not literally supervised learning. [28:23] Um, [28:24] anyway, yeah. So, I think it's really [28:27] true. I mean, [28:29] well, anyway, the first thing is a [28:30] school is [28:31] is irrelevant. Like, you know, squirrels [28:34] don't go to school and and and learn [28:36] [laughter] that. [28:37] >> They might. [28:37] >> Animals don't learn that way. It's And [28:40] school is a very special thing that that [28:42] even we didn't have up until, you know, [28:44] I don't know, a few hundred years ago. [28:46] It's not part of in not part of the [28:48] essence of intelligence? And it's a [28:50] distraction to think of that as your [28:52] primary example of learning is this [28:54] thing which we didn't do [28:58] as animals. [28:59] >> I wish you had been around to tell my [29:01] parents that before I was made to have [29:03] good go to school and [29:06] deal with all the structure. [29:08] >> The thing is like squirrels are [29:09] wonderful at jumping off trees, but [29:10] squirrels can't prove math theorems. And [29:12] if I want to learn how to prove a math [29:13] theorem, I go to school. [29:14] >> Yeah. [29:15] Uh [29:16] they also don't have [29:18] uh [29:19] DVDs and [29:20] >> [laughter] [29:21] >> and iPods. You know, there are a lot of [29:23] things [29:23] they can do things that we can't do. [29:25] Um but [29:27] >> [sighs] [29:28] >> math theorems [29:30] uh [29:31] Yeah, and they don't play chess. [29:33] You know, it's sort of like more of X [29:34] paradox. They're uh [29:35] they're these advanced things that we [29:36] think of as really intelligent. But uh [29:40] they're sort of easy for computers to do [29:42] as opposed to all these regular things [29:44] that are hard. Like moving and seeing [29:47] with attention and everything. [29:50] Um [29:50] I think supervised learning is a is a [29:52] good thing. [29:53] You know, just mentions I like to think [29:55] look for obvious things. No one tells us [29:58] how to twitch our muscles by giving us [30:00] examples cuz they couldn't possibly cuz [30:02] we have had to twitch our muscles. We've [30:04] had to figure that out. [30:06] >> Yeah. [30:06] >> And their answer would be wrong, right? [30:08] So, if I moved my [30:10] mouth and my tongue and my vocal cords [30:12] exactly the same way that Rich does to [30:14] pronounce Paris, I'm sure a very [30:16] different sound would come out. So, in [30:18] some sense [30:19] Rich or no one knows the right way of [30:21] producing a sound with my body. Only I [30:23] know that. [30:24] >> Yeah. [30:25] It seems to me that many of the most, I [30:27] guess the most raw like sensory motor [30:29] capabilities, especially related to [30:31] movement in the physical world. [30:34] I agree with you that that seems [30:36] something that is inherently learns from [30:37] experience. [30:39] It seems to me though that there are [30:40] higher levels of abstraction that bring [30:42] us closer to, you know, what makes [30:44] humans great. [30:46] And much of that [30:47] doesn't live in this low level of [30:49] sensory motor learning. Does your world [30:51] model, I guess, span [30:54] sensory motor learning all the way up? [30:56] >> Yeah, that's the ambition, absolutely. [30:59] And squirrels, by the way, can do some [31:01] enormously abstract things. [31:03] >> What's the coolest thing a squirrel can [31:04] do? [31:05] >> Well, it can always get into your bird [31:07] feeder, [31:08] >> [laughter] [31:08] >> no matter what obstacles you put in the [31:09] way, you know, it can find new ways to [31:12] jump and climb and [31:14] >> Okay. [31:14] >> and do lots of things. [31:16] >> Calculate trajectories pretty well. [31:18] Animals are pretty good at understanding [31:20] the physical world without the mental [31:22] calculations that we think we are doing [31:24] when we [31:26] think about launching ourselves [31:28] into space. [31:29] >> Breaking a fall, they can do it in real [31:30] time in the right way to prevent [31:32] injuries. [31:33] >> Okay, fair enough. [31:34] >> I think it's just a question of degree [31:36] between and I like to think that [31:38] animals, other animals, are are very [31:41] close to humans. I think it's hubristic [31:43] to try to emphasize what we do [31:45] differently, you know, how we're [31:47] different from animals. [31:48] It's better to see the commonalities. [31:51] And I I think we are just a question of [31:53] degree. It's degree and of course [31:55] society and culture give us big [31:57] advantages. Language give us big [31:59] advantages. [32:00] >> Can I just push on some of this? [32:02] >> Yeah, good. [32:02] >> Because I want to back up Sonia. So, I [32:05] believe [32:07] animals and children learn from [32:09] experience and do incredible things [32:12] learning from experience. [32:13] And [32:15] when my son was two or three or four, [32:17] I'm like, "Wow, this is really [32:18] interesting that they're my son can [32:20] learn these things without nobody [32:23] really teaching him how to do these [32:24] things." [32:25] But at the same time, what Sonia's [32:27] saying is like what makes human uniquely [32:29] human, to be able to [32:31] go to outer space, build a rocket. [32:34] Those Those not things that are learned [32:37] 100% from experience because before you [32:40] launch the rocket, [32:41] you actually have to abstract thinking [32:44] through it in a way that is not learned [32:47] from {quote} {unquote} experience. [32:49] Because you don't know if it's going to [32:51] work or not. You have to imagine it. How [32:53] do you we teach [32:55] a machine to imagine things that were [32:57] not available before? That's probably [32:59] the thing that we're trying to like push [33:01] on [33:02] because that we're not quite [33:03] understanding that. [33:04] >> actually going to agree with you there. [33:06] You have to be able to plan. You have to [33:08] be able to imagine. [33:09] >> Yeah. [33:10] >> Would you say that humans 1,000 years [33:12] ago, before they had done all most of [33:14] the thing that we're talking about, were [33:15] they as intelligent? [33:18] Um if for example someone from that era [33:20] was exposed to this new culture, would [33:23] they be able to get the same skills and [33:25] and start doing useful things? [33:27] >> Even over the last 10,000 years, I don't [33:28] think the human brain has evolved that [33:30] much. Because [33:31] >> Fundamentally the same machine. [33:33] >> Fundamentally the same machine, [33:35] but we've built up 10,000 years of [33:36] knowledge. [33:37] >> Yes. [33:37] >> And I get to learn 10,000 years of [33:39] knowledge by going to school through [33:41] supervised learning. [33:42] >> Right. [33:42] >> And I get all that much, much faster [33:45] than trying to learn through experience. [33:47] >> Right. [33:48] >> So I think you're like totally right. So [33:49] we we would want our systems to learn [33:52] from experience and part of their [33:54] experience would be getting exposed to [33:55] our culture and then learning from about [33:57] our culture. They should learn from [33:59] that. That's all good. But let's talk [34:01] about when someone goes and does a [34:03] paradigm shifting thing. So everyone [34:06] gives the example of Einstein, but I [34:07] think there are many examples. Learning [34:09] is that too, like looking at [34:11] learning thing versus [34:12] programming thing. [34:14] So when these paradigm shifts happen, I [34:15] would say it's a human who has [34:17] accumulated all this knowledge and then [34:19] from their experience they're building [34:20] new abstractions and they're planning [34:22] with them and then they're discovering [34:23] new knowledge. And that skill of of [34:26] coming up with new abstractions and then [34:27] learning what models and planning with [34:29] them, that problem is that skill is [34:31] totally missing in our current systems. [34:33] And you can expose this at the edge of [34:36] human knowledge, but you can also study [34:37] this problem at the sensory motor stream [34:40] level. [34:41] >> So, we're not arguing with the [34:44] with the principle. We need to form [34:47] abstractions so we can reason at a high [34:48] level. [34:49] You guys are coming close to doing that [34:52] thing that I said we should never do, [34:53] which is argue is [34:55] prior knowledge important or gaining [34:57] knowledge important. You know, that's [34:58] what you guys just said. You said It's a [35:00] you're you're still going to have to [35:01] learn things. And you're saying, "Oh, I [35:03] can get things from my culture and from [35:06] prior knowledge." [35:08] But these should not fight for each [35:09] other. [35:10] >> On the exact thing around paradigm [35:12] shifts, how do we create a machine that [35:14] understands when to shift the paradigm? [35:16] >> Yeah, I think through its experience, [35:18] right? So, it would have to [35:20] through its own experience. It can't [35:22] rely on human knowledge because we're [35:23] assuming the humans see one paradigm and [35:26] we want a different way of looking at [35:28] things. [35:28] And so, through its experience, it has [35:31] to find something that is better. Maybe [35:34] it it generalizes better and makes [35:36] better predictions. Maybe it's better in [35:38] some other ways, but it has to be [35:39] through its own experience. [35:40] >> The big challenge that we don't see in [35:42] our field, the ability we don't see in [35:44] our field yet, is the ability to learn a [35:46] model and then plan with the model. [35:49] We can do the [35:51] the math things and we can do AlphaGo [35:53] because [35:54] the games, we know the model. We know [35:57] how the moves work. [35:58] And in math, we know what the operators [36:00] are. [36:02] We you know, we know lean will take us [36:03] from one state of knowledge to the to [36:05] the state of the proof to the next [36:07] state. [36:08] But if we have to learn the models, [36:10] there are no [36:12] I'm I'm going to say it. It's probably [36:13] maybe a a weird example, but [36:16] a counterexample, but I can see that [36:17] there's no instances of learning the [36:19] model and then planning with the model [36:21] in our field. [36:23] >> At least not with [36:24] uh uh, [36:26] like self-discovered [36:27] abstractions. So, [36:29] there are people who say, "I'm just [36:31] going to learn a model of what happens [36:32] in the next second or next millisecond." [36:34] But, that's not how our models work. Our [36:36] models are more abstract. Our models [36:39] are, uh, [36:41] quite different. [36:42] >> So, one of the things I like about what [36:44] you're doing here is you're not just [36:45] sitting around pontificating or [36:47] lamenting the state of the world as it [36:48] is. You're very action-oriented. It's [36:50] why you started a company. So, let's [36:52] let's start talking about that a bit. [36:54] Uh, in 2022, [36:55] Rich, you laid out a very specific [36:57] 12-point plan, the Alberta plan for AI [37:01] research. [37:02] Maybe tell us about that. [37:04] >> So, the Alberta plan came about because [37:07] we just have general ideas, but we also [37:09] needed to convert them into smaller [37:11] chunks. [37:12] And so, the 12 steps [37:15] are the attempt to uh, crystallize [37:17] particular chunks. [37:18] There's a very important early step, [37:21] step two, [37:22] uh, which is [37:24] continual deep learning. [37:26] And And we think that one is like almost [37:29] the most important because it unlocks [37:31] everything else. If you could do [37:32] continual deep learning, [37:35] you could then continually update your [37:37] model of the world. [37:39] And then, if you knew how to do the [37:40] abstraction rights in like [37:42] the second half [37:43] of of the the steps are all about how to [37:45] get the abstractions right. [37:47] So, and not only my abstractions right, [37:50] what I mean by I don't mean get the [37:51] right abstractions cuz no one can say [37:53] what the right abstractions are. That [37:54] depends on the world that you're in. [37:56] Your agent would have to learn the [37:58] correct abstractions for whatever world [37:59] it's in. [38:01] And so, you know, if you maybe those are [38:04] the two key things. You have to find the [38:05] right abstractions, and then you have to [38:07] do able to continual deep learning. [38:10] >> I think that a lot of the people in the [38:12] field realize that we need models, we [38:13] need to plan with them. But, the [38:15] abstractions tell us what the model [38:17] should be conditioned on. So, what [38:19] should you What should the model [38:21] predict? What are you going to do and [38:22] then something is going to happen. And [38:24] more importantly, where would that come [38:26] from? So, I really like the example of [38:28] elite athletes. If you ask elite [38:30] athletes about how they do certain [38:32] things, they would have weird niche [38:35] terminologies for doing very specific [38:37] things. They were like, you know, I do [38:38] this thing and they would have a name [38:40] for it. If they communicate, sometimes [38:42] they don't even have a name for it if [38:43] they're just doing it alone. [38:45] So, how did they come up with those [38:46] abstractions? That's in some sense a [38:49] crucial thing that's missing that the [38:51] later half of Alberta plan answers. [38:53] >> Can we talk about the continual deep [38:54] learning part? [38:56] Is it an algorithmic gap that exists [38:59] today or is it a just a practical [39:01] deployment infrastructure data privacy [39:03] gap? Because if I wanted to do [39:05] call it naive updating of weights based [39:08] on user interaction, I can do that [39:10] today, right? And so, [39:12] what in your opinion is the biggest [39:14] thing that we're missing to kind of get [39:16] to continual deep learning? [39:18] >> Yeah, so it's absolutely an algorithmic [39:20] gap. You You can do the naive thing, but [39:22] then you'll see all sorts of problems. [39:24] So, for example, if you say um I'm going [39:26] to take one sample and then I'm going to [39:29] update my whole model with that one [39:30] sample. You will run into this problem [39:32] that now all of the previous knowledge [39:34] in the model it's impacted negatively. [39:36] And the way currently we we [39:38] get around this is exactly what cursor [39:40] does. They don't use one example. They [39:42] use a large batch coming from a lot of [39:43] users. So, in use cases where you can [39:45] have that, you can do continual [39:47] learning. But, most use cases you don't [39:49] have that. Most use cases you have a [39:51] single stream of data. And then if you [39:53] apply it to the naive thing, it just [39:55] completely destroys your prior knowledge [39:57] in a very [39:58] um [39:59] destructive way. [40:00] >> Catastrophic forgetting. [40:02] >> Yeah. [40:02] >> That is the Yeah. [40:04] >> But, it's totally curable. You have to [40:06] [laughter] have the right algorithm. [40:07] >> cure? [40:07] >> Well, yeah, exactly. [40:09] >> What is the cure? [40:09] >> Well, you know, first you need to do [40:13] what we call step size optimization. [40:16] And it means every weight in your [40:17] network has to have a separate step [40:19] size. So, some will move fast, some will [40:21] move slow. And you will we you will have [40:24] to metalearn the step sizes for each [40:27] weight. Most of your network will be [40:29] have have weights that have tiny step [40:32] sizes. So, then when you train on a new [40:34] example, they don't get destroyed. [40:37] Happens just to the right places. [40:39] And then secondly, you have to use some [40:43] form of generate and test. [40:45] Um [40:46] which is [40:47] in feature space. So, you come up with [40:49] new features or new units and and [40:52] without following gradients. Cuz [40:54] gradients are very slow process. You [40:55] only move in a direction if you know [40:57] it's the helpful one. And that's always [41:01] going to be very slow and doesn't give [41:02] you a path to grow more and more complex [41:06] and and to have sustained learning. You [41:08] need to have something that just [41:09] proposes a bunch of new units. [41:12] And [41:14] and and then goes from there. I guess [41:17] so, there is a specific thing I can say [41:19] that make it at least concrete, which is [41:21] to say we have this algorithm called [41:23] continual backprop. We used published in [41:26] in nature a couple years ago. And it it [41:29] is exactly like backprop, but every but [41:32] you also plant new seeds of units that [41:34] are newly initialized with random [41:36] weights. [41:37] Backprop only has random weights at the [41:39] beginning of time. And then as you go [41:41] on, all that randomness all that variety [41:43] from the randomness gets used up. [41:46] And with continual backprop, we keep [41:47] injecting a bit of randomness, a bit of [41:49] generate and test, a bit of generate and [41:52] then the [41:53] the operation of backprop is the tester. [41:56] So, you need you need that. [41:58] And and if you put those together really [42:00] well, I think you'll have a new [42:01] generation of massively [42:04] superior [42:05] continual deep learning. [42:07] And that's what we hope to do in the [42:09] next couple years. [42:10] >> Wonderful. [42:11] Do you think that these algorithms [42:14] can be applied to the current state of [42:15] affairs with people scaling LLMs and [42:18] trying to get them to do continual [42:20] learning without catastrophic [42:21] forgetting? [42:22] >> Yeah, absolutely. I think it's [42:24] So, I don't think that you could take an [42:27] existing model and say I'm going to just [42:29] start updating it with these algorithms [42:31] because these algorithms meta learn how [42:33] to learn. So, really you have to say, [42:36] I'm going to learn from scratch. So, [42:37] let's say I learn a new foundation [42:38] model, but I'm going to learn with this [42:40] these new algorithms. These new [42:42] algorithms in addition to [42:44] learning the knowledge, they're also [42:46] going to learn how to learn future [42:48] things. So, they're learning two things [42:49] at the same time. And then um then I [42:53] think you would be able to learn new [42:55] things without catastrophic forgetting. [42:57] >> the most radical thing in that you're [42:59] trying to do in your company in terms of [43:01] from the current state of affairs [43:04] to try to do these two things at the [43:05] same time? [43:06] >> Most radical thing. [43:07] >> Is I think that's [43:09] >> This goes back to I'm not crazy, [43:10] everyone else is crazy. [43:11] >> [laughter] [43:12] >> Yeah. [43:13] >> That's perhaps not totally radical. [43:14] There was a point [43:16] in like 2016 to 2018 where a lot of [43:19] people were exploring these ideas quite [43:21] a bit. They were doing it [43:23] in a much more limited setting. So, they [43:25] would say, we have a distribution of [43:27] problems and then in this specific case [43:29] we'll do it whereas we want to do it [43:30] from a single stream of experience. So, [43:32] our method should be more generally [43:34] applicable. So, I think many people have [43:35] explored this, [43:37] but no one has explored this in the [43:39] general setting where the resulting [43:41] algorithm would be applicable [43:42] everywhere. [43:43] >> So, what would be the most radical thing [43:45] that your company your new company is [43:47] trying to do that other people are not [43:49] doing? [43:49] >> What's the most ambitious thing? [43:52] Remember, I don't think I'm weird, so I [43:54] don't want to say it's radical. [43:56] >> the most ambitious [43:56] >> ambitious thing, I think is to try to [44:00] have the full spectrum of knowledge [44:04] both about the tiny things and about the [44:06] big things. You know, like thinking [44:08] about how you take an airplane from one [44:10] city to another. That's a a very big [44:12] thing. You know, it's it's more it's [44:14] like your [44:15] your space flight example, but it's just [44:18] kind of more common sensical to think [44:21] about cuz we all [44:22] many of us take airplanes and any all of [44:24] us use abstractions on all kinds of our [44:26] life. And even even the squirrels use [44:28] abstractions. So, to have that spectrum [44:32] of of knowledge [44:33] uh [44:34] from the small to the big and to treat [44:36] it in a uniform way and [44:39] to be able to help have it self uh [44:43] maintaining. You know, the big question [44:45] is always you have your knowledge-based [44:47] system and what keeps the knowledge in [44:49] it correct? [44:51] Well, what keeps the knowledge correct [44:53] in a large language model is well, [44:55] people did a lot of post-training [44:58] and and they they they made it sure it [45:00] was correct. And then they freeze it [45:01] after that. So, that's [45:03] what keeps it correct. But really our [45:04] minds, we are we're always changing [45:07] things and yet something keeps it [45:09] organized and coherent and and and [45:12] settling back into a good place rather [45:13] than drifting off into crazy land. [45:16] That is an I think our our biggest [45:18] ambition to have a mind that is [45:20] self-consistent [45:22] and and can [45:23] keep training itself and making it [45:25] coherent. [45:26] >> that. [45:27] Can I ask? [45:28] It almost seems that it's it's such a [45:31] ambitious vision and the idea that all [45:34] these things can be unified into a [45:35] single mind is so ambitious. [45:37] >> It's within reach. [45:39] I think it's within reach. It's here [45:40] it's 2026 and our computers are so fast. [45:44] You know, is it is it [45:46] so ambitious that it's out of reach? [45:48] Uh or or do we have [45:50] already inklings of how all the steps [45:52] can be done? And we [45:54] I I think we have [45:56] a vision [45:58] and inklings. [45:59] Um I don't think it's I don't think it's [46:01] inappropriate. [46:02] >> Your vision involves a trillion [46:04] parameter model with 20 watts. [46:07] That seems pretty [46:09] ambitious. [46:10] >> That is ambitious. Uh, in some sense [46:12] with current technology, I would say [46:13] it's also impossible. [46:15] Like just storing a trillion parameters [46:18] in memory would probably use more than [46:20] 20 watts of energy with current memory [46:22] technologies, but [46:25] we are really thinking of okay, [46:28] things are getting better, computation [46:29] is getting cheaper or it is getting more [46:30] energy efficient. So, where would be [46:32] would be in 5 to 10 years? And I think 5 [46:35] to 10 years with the right algorithms [46:38] and [46:39] we can totally be in a world where this [46:42] would be possible. [46:43] >> So, 5 to 10 years is two orders of [46:46] magnitude of Moore's law. [46:49] It's a standard improvement. [46:52] If we double every 18 months, 10 years [46:55] would give you two orders of magnitude. [46:57] And so, [46:58] for Kurzweil's statement to be [47:00] plausible, [47:02] then today you should be able to do it [47:03] for [47:05] for what? 20 watts? Two orders of [47:06] magnitude? [47:08] >> 2,000 [47:08] >> 2,000 watts. If you can do it with 2,000 [47:10] watts today, yeah, then in 10 years [47:12] you'll be able to do it for 20 watts. [47:14] >> You think you can do it for a 2,000 [47:15] watts? You have I think lots of people [47:17] at research labs that have access to way [47:20] more than that. [47:21] >> Yeah, I think we can [47:24] can be more efficient than that even now [47:26] with the right algorithm. [47:28] >> If we can be more efficient than that, [47:29] then why aren't we? It's not like people [47:32] just want to spend all their money on [47:34] spend all their money. [47:34] >> Sometimes it seems like they [47:36] >> I'll pay money. [47:38] >> Sometimes it [laughter] seems like they [47:39] want to. [47:39] >> Yeah, I I [47:40] >> Doesn't it? [47:41] I think that's how they show they're [47:43] they're real men [47:44] by using lots of energy. [47:46] >> At least when I look at different [47:47] research groups, I don't even see anyone [47:50] believing in that it's possible. And I [47:51] think if you don't believe in it, you're [47:53] just not going to work on the technical [47:55] problems and work through them. [47:56] >> Is it that it's not possible or it's [47:58] that there's so much waste in the [47:59] system? Like one which one is it? Like [48:02] is it is is there someone who knows how [48:04] to do it efficiently? [48:05] >> Yeah. [48:06] >> And then there's 10 times the number of [48:08] people in the same lab doing all these [48:10] other things. [48:11] And [48:12] so nine out of 10 people are wasting [48:16] >> In some sense I the way I think about it [48:18] is that we are stuck in a local minimum. [48:20] So, if we want to move towards these new [48:22] kind of algorithms, [48:24] it is almost impossible that things will [48:26] not get worse before they get better. [48:27] So, when we start exploring these new [48:28] directions, you're not going to get [48:30] state of the art performance from day [48:31] one, but it is because it is a different [48:33] paradigm. Um but that path leads to [48:36] similar performance at a higher energy [48:38] scale. And these big labs, they are so [48:41] locked into a product that they [48:45] like it is not possible for them to [48:47] pursue a path where things get worse [48:48] first. [48:49] >> Because their current paradigm allows [48:51] them to keep scaling and this new [48:53] paradigm they have to take a bet. And [48:55] then [48:55] >> And they have to figure out some some [48:57] technical things that are difficult that [49:01] we have thought about it for many years. [49:03] We know people who have thought about [49:04] these things for many years and [49:06] when I talk to them, it makes sense that [49:08] it's doable, but [49:09] you need to think about those challenges [49:11] for a long period of time. [49:13] >> So, if everything goes right with Oak, [49:16] what happens with the company? What do [49:18] you what kind of company are you [49:19] building? [49:20] >> Uh if everything goes right, we [49:23] uh implement the architecture, we can [49:25] have a genuine uh continual learning and [49:27] we can form abstractions so that we can [49:30] do planning and reasoning and [49:33] and we have so sort of like true [49:36] intelligence. And then, you know, it's [49:37] hard to imagine just exactly what will [49:39] happen by then. [49:40] But I think [49:41] >> Humans will become irrelevant. [49:43] >> I I don't think that's true at all. I I [49:46] I [49:46] >> We don't either. [49:47] >> I think the world becomes exciting and [49:49] even more exciting and interesting and [49:52] and for humans. [49:53] But in particular, I think there are the [49:56] You have to wonder about the large [49:58] language models. [49:59] They might be at risk. [50:02] Uh when this eventually happens, you [50:04] know, I'm sure they'll get a good run. [50:05] They've already had a good run. You [50:07] know, they've been very successful. And [50:08] let me say, [50:10] just for for clarity that uh large [50:12] language models are an amazing [50:14] scientific breakthrough, a breakthrough [50:16] in the skillful use of language by [50:18] neural networks, wholly unanticipated. [50:22] You know, it was a It was always a hold [50:23] out for uh symbolic methods in language, [50:26] and they [50:28] They have totally changed how that's [50:30] thought about now. Yeah. It's a It's a [50:32] big breakthrough. [50:33] It's so It's frustrating to me that we [50:35] have to you know, just celebrate that [50:38] we've made this great progress in the [50:39] sub subset of the problem of AI, and [50:42] enjoy that. Instead, [50:44] it has to pretend to be all of AI. All [50:46] of intelligence is not fluid, capable [50:50] use of language. There's so much more. [50:52] It's an important part. You know, it's [50:53] like [50:55] 20% or a quarter of [50:57] intelligence. There's There's more. [50:59] >> Yeah. [51:00] >> We're not done. [51:01] >> Yeah. [51:02] If everything goes right, are you [51:04] imagining that there's a single minds [51:06] that can do everything from learn how to [51:08] swing from tree branches to make a [51:11] spaceship, [51:12] uh to you know, all these various things [51:15] we've talked about today. Is it a single [51:16] mind, and is it a single set of weights [51:19] that can that can do all these things, [51:21] or is it [51:22] >> It's a single design. [51:23] >> Okay. [51:24] >> And there'll be many different There'll [51:25] be many minds. [51:26] >> Mm. Okay. So, it's a single design that [51:28] reacts to different environments. [51:30] >> And it did it And different versions of [51:33] that mind would learn different things [51:34] because they have different experience. [51:35] This sort of goes back to the big world [51:38] hypothesis that there are infinitely [51:40] many things to learn, so one system [51:42] cannot learn infinitely many things. And [51:45] like I think Rich already mentioned [51:46] this, but if you have two of these [51:48] systems, if you have two of the largest [51:50] systems in the world, then it is trivial [51:52] that they cannot model each other [51:54] because they are equally complex. So, [51:55] single system would never be able to get [51:58] to a point where it can learn [51:59] everything. It would always be multiple [52:01] systems [52:02] that are learning from their own [52:02] experience. [52:03] >> Guys, are you hiring? What kind of [52:05] people are you looking for? [52:06] >> We are hiring. [52:09] The initial team, most of it we already [52:10] have in our mind. So, these are people [52:12] who have thought about these ideas in [52:14] the past in many [52:16] uh you know, for a long time. [52:17] And [52:19] we are going to take a slightly [52:20] different approach because this is a [52:22] different paradigm. It doesn't make [52:23] sense to [52:25] become large very quickly because uh in [52:27] some sense, everyone we hire has to [52:31] uh come to see what we see. And not [52:33] everyone sees that. So, we're going to [52:34] start small, slowly grow to maybe a [52:37] handful or two or three uh and then go [52:39] from there. [52:40] >> We want to be super aligned. [52:42] >> We want to be super aligned. [52:43] >> So, that we can be um [52:46] very productive working together and and [52:49] scaling the progress. [52:50] >> Absolutely. [52:51] >> Very, very cool. [52:52] >> Wonderful. [52:53] I love this conversation. Thank you for [52:55] taking the time to share what you're up [52:57] to. You are, you know, an an unusually [53:00] deep thinker about [53:02] where reinforcement learning and [53:04] algorithmic design will go. [53:06] And it was a true pleasure to get to [53:07] explore it together with you today. So, [53:09] thank you. [53:10] >> Thank you very much. Thank you. It's our [53:11] pleasure. [53:15] >> [music] [53:21] [music] [53:37] [music]