Silver Targets Superintelligence in Three Years via Pure Simulation

Silver计划三年内通过纯模拟实现超级智能 David Silver认为人类语言预训练给机器推理设定了硬性上限，并断言完全通过模拟经验训练的智能体将在三年内实现超级智能。 David Silver和Lasse Espeholt详细说明了他们离开Google DeepMind的缘由，即在合成模拟环境中通过纯自我博弈强化学习来追求超级智能。他们认为，随着智能体规模的扩大，人类语言推理以及像思维链监控这样的对齐机制将变成过时的上限，因此必须完全在模拟中建立一门前瞻性的多智能体对齐实证科学。Espeholt概述了支撑这一目标的底层基础设施，包括Google Cloud Platform上的Nvidia Vera Rubin GPU集群，以及承袭自IMPALA和SEED RL的技术架构脉络。为了保持极高的人均算力比，团队被严格限制在20名研究人员以内；Silver预测，他们的范式将在一年内获得强有力的实证验证，并在三年内实现超级智能。 Ineffable Intelligence正在Nvidia Vera Rubin集群上开展从零开始（tabula-rasa）的多智能体实验，其计算量约为OpenAI 2019年“捉迷藏”（Hide and Seek）设置的100,000倍。 (37:25) David Silver警告称，随着智能体能力超越人类，英语思维链监控和RLHF将会失效；他主张在现实部署之前，必须在多智能体模拟中将对齐确立为一门实证科学。 (43:44) 由于自我博弈会导致数据分布持续变化的动态非平稳性，强化学习的扩展定律（scaling laws）与LLM的幂律法则存在根本性分歧。 (56:51) Ineffable将团队规模上限控制在20人，以最大化人均研究算力，目标是在一年内取得确凿的实证证明，并在三年内实现超级智能。 (65:57)

Ineffable Intelligence co-founders David Silver and Lasse Espeholt argue that genuine superintelligence cannot be achieved by training large language models on human text, but instead requires autonomous reinforcement learning from raw experience. By training tabula rasa agents inside rich simulated environments, the startup seeks to bypass human cognitive limits and discover novel scientific principles. Pure Reinforcement Learning Over Human Priors (28:18) Silver contends that relying on trillion-token corpora of human internet text bounds artificial intelligence to human-level concepts and historical biases. He views natural language as an expressive bottleneck that fails to represent embodied physical dynamics or non-human neural reasoning. In his view, true superintelligence requires systems that develop internal representations from first principles, free from curated human demonstrations and sequential curricula. To justify this bet, Silver contrasts AlphaGo with AlphaZero. AlphaGo required two years of engineering, human game records, and continuous developer intervention to defeat human champions. In contrast, AlphaZero abandoned human data entirely, surpassing existing chess benchmarks in four hours and defeating AlphaGo within days through tabula rasa self-play. Silver asserts that contemporary LLM development remains stuck in AlphaGo's labor-intensive semi-autonomous paradigm, whereas open-ended discovery requires sacrificing short-term performance to unlock unbounded learning depth. Crucibles, Multi-Agent Depth, and Minimalist Rewards (16:13) Rather than micro-managing behavior through "breadcrumbing"—guiding models via rigid sequences of sub-tasks—Ineffable provides agents with simple optimization targets within simulated environments called "crucibles." Silver argues that just as reproductive fitness drove the evolution of complex biological organs, basic survival or fitness objectives inside rich physical and multi-agent crucibles naturally incentivize agents to invent communication, arithmetic, and tools. Introducing multi-agent competition transforms simple environments into open-ended discovery engines with inexhaustible strategic depth. Addressing Rich Sutton's critique of simulation-to-real transfer, Silver and Espeholt explain that Ineffable does not intend to transfer sandbox-specific policies into physical robots. Instead, digital crucibles are used to discover the universal learning algorithm itself up to a hundred times faster than physical experimentation. Digital worlds provide Turing completeness, reproducible sandboxes, and safe containment, allowing researchers to evaluate the core superlearner mechanism before deploying it to physical or higher-fidelity environments. Distributed Compute and Non-Stationary Scaling Laws (52:54) Ineffable caps its headcount at 20 researchers to maintain a high compute-to-person ratio, utilizing next-generation Nvidia Vera Rubin clusters on Google Cloud Platform. Espeholt notes that modern accelerators are tailored for supervised training and low-latency inference, leaving simulation as a key hardware mismatch. To prevent host CPU bottlenecks, the lab executes simulation, inference, and gradient updates natively on GPUs, using model-based RL and aggressive replay buffers to optimize sample efficiency. Espeholt explains that RL scaling laws operate under different mathematical dynamics than LLM power laws because RL data is non-stationary: an agent's ongoing exploration continuously alters its own training distribution. Drawing on distributed architectures Espeholt co-developed at DeepMind and Google Brain—such as IMPALA's decoupled actor-learner pipelines and SEED RL's centralized TPU execution—the lab aims to establish empirical laws linking compute expansion to monotonic capability gains in self-generating data regimes. Simulation-First Safety and Accelerated Timelines (47:08) Silver argues that inspecting chains of thought in English is an inadequate safety mechanism. Highly capable models can generate million-word reasoning trails or engage in deliberate subterfuge by concealing non-human reasoning behind compliant text. In response, Ineffable pursues a language-agnostic safety approach involving sandboxed shadow actions, overseer models that regularize external impact, and the empirical study of multi-agent dynamics among disparate intelligences within simulations before real-world deployment. Silver supports regulatory pacing for physical and economic deployments, but advocates unconstrained acceleration within simulations to develop safety frameworks ahead of reality. Citing prior accurate forecasts for Computer Go, StarCraft II, and the International Mathematical Olympiad, Silver projects that Ineffable will demonstrate proof-of-concept evidence within twelve months. He forecasts achieving full superintelligence across scientific and technical domains within three years, accelerated from an earlier three-to-five-year estimate by productivity gains from AI coding agents. While Ineffable's approach avoids human data bottlenecks, its viability depends on whether digital crucibles can generate genuine general intelligence without leaking human priors, and whether non-stationary reinforcement learning can scale as reliably as static pre-training.

## Background 该内容由红杉资本（Sequoia Capital）于 2026 年 10 月 13 日发布在 YouTube 平台，时长约 1 小时 8 分钟。视频由红杉资本合伙人 Sonya Huang 主持，专访了 Ineffable Intelligence 联合创始人 David Silver 与 Lasse Espeholt，深入探讨了他们脱离 DeepMind 创立新实验室的初衷，以及抛弃人类语言预训练、打造完全依赖自主经验探索的“超级学习器”（Superlearner）并通向超级智能的宏大规划。 ## TL;DR David Silver 与 Lasse Espeholt 阐述了他们不同于主流 LLM 路线的核心理念：人类语言只是认知的阶段性天花板而非基石，真正的超级智能必须像 AlphaZero 一样从零先验出发，通过纯强化学习在高度复杂的高维模拟“熔炉”（Crucibles）中自主涌现。团队凭借在分布式 RL 底座（如 IMPALA、SEED RL）上的深厚积淀，锁定了 Google Cloud 上的首批 Nvidia Vera Rubin GPU 超级集群，以极高的人均算力比运行多智能体自博弈。在安全对齐上，Silver 认为基于思维链的语言监控已达极限，必须在虚拟仿真中抢先建立起多超级智能和平共存的实验科学。Silver 最终给出激进预测：一年内将迎来该路线奏效的强力证据，三年内便有望实现真正的超级智能。 ## Atmosphere & style 整场对谈兼具严谨的顶尖科研质感与开拓未知的极客浪漫。David Silver 举重若轻地拆解复杂理论，展现出顶级科学家兼“超级预测家”的从容自信；Lasse Espeholt 则在系统架构与工程落地上沉稳务实，两人配合默契。主持人 Sonya Huang 的提问直击要害，在充满默契的轻松互动中，屡次将讨论推向 AI 范式转移的哲学与安全深水区，让整场播客散发着一种见证技术奇点前夕的震撼感与紧迫感。 ## Bulletin - [01:06:34](https://www.youtube.com/watch?v=L4SWQebqhek&t=3994s) David Silver 预测 Ineffable 将在 1 年内拿出纯经验强化学习路线的强有力证据，并借助 Coding Agent 大幅加速工程迭代，有望在 3 年内实现超越人类智能极限的真正超级智能。 - [53:04](https://www.youtube.com/watch?v=L4SWQebqhek&t=3184s) Ineffable 成为 Google Cloud 首批 Nvidia Vera Rubin GPU 集群的首发客户，摆脱传统强化学习依赖海量 CPU 沙盒的架构，构建出环境仿真、推理与训练全 GPU 原生的算力栈，实验算力规模达到早期多智能体自博弈实验的 10 万倍以上。 - [25:41](https://www.youtube.com/watch?v=L4SWQebqhek&t=1541s) 团队主张人类自然语言是机器推理的上限而非基石，强行依赖人类语言与思维链（CoT）不仅会遗传历史认知偏见，还面临智能体在高阶博弈中产生“阳奉阴违”（Subterfuge）伪装欺骗的安全风险。 - [51:01](https://www.youtube.com/watch?v=L4SWQebqhek&t=3061s) Ineffable 提出“现实世界审慎克制、仿真世界全速狂飙”的安全方法论，主张在高维数字模拟“熔炉”（Crucibles）中率先建立多超级智能和平共存的科学规律，并将其作为系统接入现实世界的硬性门控标准。 ## Key takeaways - [09:57](https://www.youtube.com/watch?v=L4SWQebqhek&t=597s) 通向超级智能的核心在于打造“超级学习器”（Superlearner），它从零先验（Tabula Rasa）出发，依托直接经验在深度与广度上自主发现新科学，而非依赖万亿级人类文本预训练。 - [03:35](https://www.youtube.com/watch?v=L4SWQebqhek&t=215s) 离开 DeepMind 创业是为避开主流过度拥挤的 LLM 产品化路径（“路灯下找钥匙”），纯强化学习路线如果成功将重塑 AI 格局，值得成立独立实验室全力攻坚。 - [16:15](https://www.youtube.com/watch?v=L4SWQebqhek&t=975s) 研发彻底摒弃步步为营的“撒面包屑式”（Breadcrumb trailing）密集奖励，而是给予极简目标，依托多智能体博弈与复杂环境自然涌现出工具、语言及科学等高阶能力。 - [24:14](https://www.youtube.com/watch?v=L4SWQebqhek&t=1454s) 围棋印证了极简规则与多智能体互动蕴含的无限潜能：单智能体机械乏味，而双智能体对抗能激发出永无天花板的指数级复杂策略。 - [25:41](https://www.youtube.com/watch?v=L4SWQebqhek&t=1541s) 自然语言是进化的阶段性产物而非终极思维载体，以其为底座将不可避免地继承人类历史偏见，未来的超人类认知应依托高维、并行的神经推理（Neural Reasoning）。 - [28:18](https://www.youtube.com/watch?v=L4SWQebqhek&t=1698s) AlphaZero 仅用 4 小时自博弈击败 Stockfish、数天超越初代 AlphaGo 的历史证明，彻底剥离人类先验数据能释放出远超人类经验的自学爆发力。 - [32:05](https://www.youtube.com/watch?v=L4SWQebqhek&t=1925s) 在设计模拟环境“熔炉”（Crucibles）时，团队坚决剔除编码特定领域知识的强先验，仅保留跨环境通用的弱先验，促使智能体从第一性原理自发重构计算与科学。 - [38:13](https://www.youtube.com/watch?v=L4SWQebqhek&t=2293s) 针对强化学习先驱 Rich Sutton 对模拟环境的怀疑，Silver 指出只要环境具备图灵完备性与通信信道，高阶智能便能完整涌现，且在仿真中构建超级智能的速度比在真实世界中快 10 到 100 倍。 - [47:45](https://www.youtube.com/watch?v=L4SWQebqhek&t=2865s) 依赖人类语言思维链（CoT）的对齐手段正逼近天花板，随着能力提升，智能体极易学会语言伪装与表面迎合，监管手段必须转向“影子动作”（Shadow Actions）与超级智能相互监督。 - [51:01](https://www.youtube.com/watch?v=L4SWQebqhek&t=3061s) 对齐必须抢在现实落地前在仿真沙盒中完成，通过多智能体共存演化实验摸清制衡规律，建立扎实的“超级智能科学”。 - [53:04](https://www.youtube.com/watch?v=L4SWQebqhek&t=3184s) 采用 GCP 首批 Nvidia Vera Rubin GPU 集群，将机架视为单一计算节点，实现“仿真-推理-训练”全流程 GPU 原生协同，打破传统海量 CPU 节点的带宽与延迟瓶颈。 - [55:23](https://www.youtube.com/watch?v=L4SWQebqhek&t=3323s) 针对 GPU 硬件天然不适合复杂逻辑仿真的矛盾，团队引入模型基强化学习（Model-Based RL）与高复用经验回放缓存，将算力重心转移回 GPU 擅长的高维张量计算。 - [56:51](https://www.youtube.com/watch?v=L4SWQebqhek&t=3411s) 强化学习的环境具有“非平稳性”（Non-stationary），智能体策略进化会导致数据分布动态漂移，其专有的 Scaling Law 机制比监督学习更为复杂且处于积极验证阶段。 - [57:40](https://www.youtube.com/watch?v=L4SWQebqhek&t=3460s) Ineffable 的系统底座直接继承并改良了 Lasse 早期主导的经典架构（IMPALA 与 SEED RL），当今主流 LLM 圈在后训练领域的诸多探索本质上是在复用当年的分布式强化学习思想。 - [01:00:58](https://www.youtube.com/watch?v=L4SWQebqhek&t=3658s) 团队严格维持约 20 人的精简高密度架构，以确保极高的“人均算力比”（Compute-to-person ratio），消除大公司内部的算力争夺内耗并捍卫纯科研导向文化。 - [01:06:34](https://www.youtube.com/watch?v=L4SWQebqhek&t=3994s) 结合编码智能体对底层软件工程的显著加速，David Silver 给出预测：纯经验强化学习路线将在 1 年内出现强力实证，真正的超级智能将在 3 年内达成。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=L4SWQebqhek&t=0s) 从空白到超智：自主探索的本质与百集之约 人类的智力进化史是一段令人叹为观止的自我发现旅程。最初，人类大脑中只是一团未被开发的“灰色浆糊”（gray mush），没有任何现成知识，却通过一代代的集体探索，从零构建起了语言、科学、技术与数学等一整套庞大的文明知识体系。David Silver 认为，这种无需灌输、依靠纯粹探索不断积累认知的机制，正是当前人工智能领域尚未攻克的最大命题，也是通向“超级智能”（Superintelligence）的必由之路——未来的 AI 必须演变为一个能够自主且永无止境地学习与发现新知识的系统。 红杉资本合伙人 Sonya Huang 对 Ineffable Intelligence 的两位联合创始人 David Silver 和 Lasse Espeholt 进行了专访。Sonya 回忆道，早在约一百期节目之前启动播客时，David Silver 就是她第一位主动发送冷邮件（cold email）邀请的嘉宾，如今终于促成对谈，称得上是一次圆满的闭环。 ### [01:12](https://www.youtube.com/watch?v=L4SWQebqhek&t=72s) 围棋的启示与大语言模型带来的认知转变 David Silver 早在 2009 年的博士阶段就将强化学习（RL）应用于极其复杂的围棋领域。他随后主导的 AlphaGo 项目向全世界证明了机器在单一领域超越人类极限的可能性，也直接启发了当前整整一代的强化学习与深度学习研究者。回顾多年历程，David 坦言自己的核心信念从未动摇：解决人工智能的最优路径始终是“从经验中学习”，构建能够自主发现规律、通过试错强化逐步进化的系统。 大语言模型（LLM）时代的爆发也修正了他此前的一些预判。尽管 Ineffable 并未选择主流依赖人类数据训练的路径，但 LLM 的演进提供了关键启示：在海量数据上预训练的模型，仅需极少量的特定数据微调，就能彻底转变自身行为。这让 David 意识到，“只要数据或经验的规模达到临界点，很多原本看似极端困难、极其缓慢的学习过程，随后都会以极快的速度完成”。 ```mermaid flowchart TD accTitle: 传统 LLM 路径与 Ineffable 纯经验学习路径对比 accDescr: 对比传统大语言模型依赖人类语料的预训练/后训练范式，与 Ineffable 坚持的纯经验自我博弈强化学习范式。 subgraph LLM_Path["主流 LLM 范式"] A["海量人类语料<br/>(预训练)"] --> B["极少量微调/后训练<br/>(行为快速对齐)"] B --> C["受限于人类语言与已有知识上限"] end subgraph Ineffable_Path["Ineffable 纯经验超学习范式"] D["零先验经验起点<br/>(空白智能体)"] --> E["大规模强化学习环境<br/>(自主博弈与自我探索)"] E --> F["超越人类语言表达的<br/>新发现与超级智能"] end ``` ### [03:35](https://www.youtube.com/watch?v=L4SWQebqhek&t=215s) 为何离开 DeepMind：避开拥挤的灯柱，孤身开辟新路 DeepMind 拥有业界几乎顶级的算力储备与卓越团队，但 David Silver 最终选择自立门户。他坦言自己深爱在 DeepMind 的岁月，并与前同事保持着极佳的关系，但他深信通往超级智能的真正道路并不需要将“基于 LLM 的产品化路线”作为中间跳板。既然全世界已有成千上万顶尖研究者全神贯注于预训练、后训练以及代码生成等 LLM 范式，自己去当“第 100,001 个研究者”并非时间和精力的最优解。 面对这种普遍的行业共识，David 形象地将其比作“找钥匙的人只在路灯下寻找”：主流研究者全挤在同一盏灯柱下，而在无人探索的另一盏灯柱下，蕴含着彻底颠覆 AI 格局的潜能。如果这条纯强化学习路线成功，将重塑整个 AI 领域，因此它值得一家顶级实验室倾尽全力全职攻坚，而非仅仅沦为大型组织内侧重 LLM 业务的一个边缘研究组。 ### [05:58](https://www.youtube.com/watch?v=L4SWQebqhek&t=358s) 命名“Ineffable”：不可言说的深刻与绝不失败的双重含义 谈及公司命名为“Ineffable”（不可言喻 / 不可名状），David 解释了名字背后的两重渊源： - **深层哲学追求**：超级智能未来所发现的全新知识、底层规律与深奥真理，很可能会超越人类语言的表意极限，甚至无法用人类的语法或词汇去描述，是真正意义上的“不可言说之物”（ineffable knowledge）。 - **极客谐音梗（“绝不翻车”）**：David 笑着透露了一个团队内部的秘密原因——从经验中自发学习是一套在数学与逻辑上必然成立的路径，它“绝不可能搞砸”，即“it can't possibly f*** up（in-eff-able）”，因此这套纯经验路线必将行之有效。 ### [07:08](https://www.youtube.com/watch?v=L4SWQebqhek&t=428s) Lasse Espeholt 的历程与大规模强化学习架构的奠基 Lasse Espeholt 的经历贯穿了现代分布式强化学习的技术演进。他于 2015 年 1 月加入 DeepMind，2016 年开始与 David Silver 合作，将深度学习与强化学习相结合并推向大规模工程化。2018 年夏天 Lasse 离开 DeepMind 转战 Google Brain，后因两家机构合并再次回归，并在 David 离职前约一年再度离开。 Lasse 回忆，他是通过一位潜在投资人的打探才得知 David 准备创业的消息，当时第一反应是：“糟糕，我错失了先机，早知道我应该先联系他！”在他眼中，David 始终是“房间里最聪明的人之一”。不久后 David 主动伸出橄榄枝，加上团队中早年共事过的 Voytech 和 Chris，让 Lasse 毫不犹豫地决定加入，并对能在欧洲建立起这样顶级的 AI 实验室倍感自豪。 David 则特别指出了 Lasse 的过人贡献与低调品格：Lasse 是业内最早攻克强化学习 GPU 大规模并行化与分布式可扩展架构的先驱者。当今前沿研究所采用的大规模 RL 架构理念，大多可以追溯到 Lasse 当年在分布式强化学习系统（如 IMPALA、SEED RL）上的开创性奠基。 - **对谈核心人物**：David Silver（联合创始人，AlphaGo / AlphaZero 核心奠基人）、Lasse Espeholt（联合创始人，分布式 RL 架构先驱）、Sonya Huang（红杉资本合伙人）。 - **核心论点引言**：David 强调通往超级智能的本质是构建一个“能够不断自主学习并永远探索知识的系统”（"a system that can just keep learning and discovering knowledge forever"）。 - **后续探讨铺垫**：对话随即切入 Ineffable Intelligence 的核心使命宣言——打造一个“超学习者”（Superlearner）并完成与超级智能的“第一次接触”（first contact）。 ### [09:57](https://www.youtube.com/watch?v=L4SWQebqhek&t=597s) 超级学习器的定义：深度、广度与“遇见魔法” David Silver 将“超级学习器”（Superlearner）定义为一种在深度（Depth）和广度（Breadth）两个维度上都拥有极致容量的系统。在深度维度上，它必须能够从一无所知的白纸状态起步，一路自主探索，最终发现远远超出人类现有认知范畴的事物。Silver 打了一个生动的比方：人类文明的演进正是如此——人类最初头脑中只有一团混沌的“灰色脑浆”，没有任何先验知识，没有天外来客赐予科技蓝图，更没有人喂给人类一个包含上万亿样本的数据集，但人类通过代际的集体探索，自主摸索出了语言、技术、科学和数学体系。这种从零开始不断向外探索认知边界的能力，正是当今人工智能尚未攻克的内核，也是通往超级智能（Superintelligence）的真正精髓：一个能够永不停歇地自主学习与发掘新知的系统。 而在广度的维度上，超级学习器应当具备强大的通用泛化能力，能够被直接“空投”进任何形态的经验流（Stream of Experience）中。不论身处模拟世界（Simulated World）、纯数字世界还是物理实体世界，它都能自适应地持续学习。Silver 充满憧憬地表示，如果这项技术被真正攻破，人类将迎来如同“遇见魔法”一般的超级智能，其深刻与震撼程度将超乎想象。 - **深度维度（Depth）**：从零先验（Tabula Rasa）出发，无需现成蓝图或万亿级人类数据集，依靠自主探索突破人类认知极限。 - **广度维度（Breadth）**：无缝适应模拟世界、数字世界及物理现实等任意经验流并持续终身学习。 - **终极形态**：不再依赖人类知识喂养，拥有永续发掘新知识能力的超级智能。 --- ### [11:40](https://www.youtube.com/watch?v=L4SWQebqhek&t=700s) 初次接触的震撼：从围棋、国际象棋到科学发现 当被问及人类第一次真正与超级学习器面对面接触时会是什么感受、这是否是一种无法提前预测的体验时，David Silver 毫不犹豫地回答，那绝对会是一种纯粹的“惊叹”（Wonderment）。他从自己过去主导棋类 AI 的亲身经历中找到了对照：尽管棋盘游戏的领域相对狭窄，但当年 AlphaGo 在围棋界达到超越人类的顶尖水平时，整个围棋界体验到的正是这种惊叹。顶尖棋手们亲眼目睹 AlphaGo 下出的神来之笔时，不仅感到震撼，更感受到了一种见证全新围棋境界的由衷喜悦；随后的 AlphaZero 在国际象棋领域展现超人水准时，棋手们同样陷入了极度的兴奋之中。 Silver 进一步畅想，如果把这种跨越带到棋盘之外的广阔世界——去探索人类从未涉足的全新科学领域，发明人类甚至做梦都未曾想过的技术形态，那种初次接触的震撼感将是何等壮丽。 --- ### [12:35](https://www.youtube.com/watch?v=L4SWQebqhek&t=755s) 超级学习器与当前大语言模型（LLM）的本质区别 主持人提出，目前基于大语言模型与强化学习（RL）的架构似乎已经开始攻克千禧年数学猜想并取得某些科学发现，这是否符合超级学习器的范畴？Silver 对现有 LLM 取得的瞩目成就表达了充分尊重，但他尖锐地指出了其与真正的超级学习器之间的本质鸿沟。 Silver 认为，超级学习器的第一要义是“完全自主的发现”（Completely autonomous discovery）。当前 LLM 的很多所谓突破背后，依然依赖大量“人类在环”（Human-in-the-loop）的人工调优与工程引导，远未实现完全的自主运作。其次，真正的超级学习器必须能够落地到物理实体世界中，直接从物理经验中汲取认知，而现有的语言模型距离这一步还非常遥远。更深层的分歧在于认知基质：Silver 质疑“以人类语言为媒介进行思考”是否真的是通向终极知识的正确载体；过度依赖人类语言数据，甚至可能像沉重的负荷一样，反而拖累了系统探索超人类认知的步伐（"weighing things down"）。 ```mermaid flowchart TD accTitle: LLM Paradigm vs Ineffable Superlearner accDescr: Comparison between current human-data heavy LLM approaches and the autonomous experiential Superlearner. subgraph LLMApproach ["主流大模型范式 (LLM + RL)"] A1["海量人类语言数据 / 先验输入"] --> A2["人类在环干预与引导 (Human-in-the-loop)"] A2 --> A3["符号/语言受限推理 (人类认知是上限)"] end subgraph SuperlearnerApproach ["Ineffable 超级学习器范式"] B1["零先验经验流 (物理/数字/模拟)"] --> B2["完全自主试错探索 (Autonomous Exploration)"] B2 --> B3["超越人类认知的全新科学与规律涌现"] end ``` --- ### [13:59](https://www.youtube.com/watch?v=L4SWQebqhek&t=839s) Ineffable 的日常实践：强化学习底层栈与“熔炉”（Crucibles） 谈及日常的研发落地，主持人提到曾是 AlphaGo 核心成员、现获红杉投资的 Giannis Antonoglou 对 Silver 的评价：Silver 既拥有极其宏大的远景，又能将团队极其务实地锚定在扎实的研究里程碑上。对于目前在 Ineffable Intelligence 的具体研发日常，Silver 拆解了工程与研究的双重战线。 在工程底层，Lasse Espeholt 等系统专家正在打造业内顶尖的大规模强化学习底层栈（RL Stack），确保系统能够以极致优化的效率吞吐海量数据，并在极大规模计算集群上实现持续不衰退的学习。而在上层环境构建方面，团队正在倾力开发被称为“熔炉”（Crucibles）的模拟环境。 “熔炉”是专门为超级学习器定制的高维度模拟世界。在这些环境中，智能体有足够的自由度和复杂度去表达、探索并重新发现人类所定义的一切智能行为。Silver 解释，团队将宏大目标拆解为具体的日常里程碑，而这些里程碑的参照系正是“人类文明探索的史诗之旅”——他们期待智能体在“熔炉”中逐步复刻人类历史上那些重大的知识发现时刻。 - **顶尖 RL 底层栈**：支持超大规模集群并行运行，最优化数据摄入与无限持续学习。 - **环境“熔炉”（Crucibles）**：具备极高复杂度与开放度的模拟环境，为超级智能的自主发现提供试验场。 - **历史镜像里程碑**：以人类文明探索历程中的重大节点为标尺，衡量智能体能力的复现与突破。 --- ### [16:15](https://www.youtube.com/watch?v=L4SWQebqhek&t=975s) 拒绝“面包屑式”诱导：简单奖励信号激发涌现复杂性 关于智能体会率先重新发现什么、何时能迎来第一个验证成功的顿悟时刻，Silver 给出了一个出人意料的答案：“我希望被它们所惊讶。” Silver 阐明了 Ineffable 与当今主流强化学习流派截然不同的哲学立场。当前绝大多数人将强化学习应用于大模型时，普遍采取“撒面包屑引路”（Breadcrumb trailing）的策略：设定步步为营的极细微目标，牵着智能体的鼻子（"pulling the agent by its nose"）走向预设的人工结果。但 Ineffable 彻底摒弃了这种微观引导，转而赋予智能体极为简洁、通用的优化目标，让它们在高度复杂的环境中自行探索。 Silver 指出，很多人在做 RL 时陷入了一个误区，总苦恼于“如何设计一个极其复杂的奖励函数来精确定义我想要的科学发现或数学思维”。但自然界的进化法则早已证明了另一种可能：追求极其简洁、易于描述的目标，往往能在复杂环境中自发催生出极难用语言描述的惊人复杂性与高级智能（Emergence）。正如生物进化仅仅依靠“适者生存”（Survival of the fittest）这一个简单的适应度信号，便孕育出了自然界浩瀚的生命奇迹与人类文明的一切复杂行为。在简单的终极目标驱动下，智能体为了最高效地解决问题，完全可能会自发创造出人类曾发明过的计算工具、乃至语言本身。 | 研发维度 | 主流 LLM + RL 路线 | Ineffable 路线 | |:--- |:--- |:--- | | **引导策略** | 撒面包屑式逐步诱导（Breadcrumb trailing） | 开放式探索，拒绝微观路径规划 | | **目标函数设计** | 复杂细致的人工定义目标与密集奖励 | 极简、通用的优化目标（如生存/任务胜负） | | **复杂性来源** | 人为编写的逻辑、数据集与对齐规则 | 简单目标在极复杂环境（熔炉）中自发“涌现” | | **对结果的预期** | 符合人类预设轨迹的输出 | 带来惊喜与意料之外的自主发现 | ### [18:57](https://www.youtube.com/watch?v=L4SWQebqhek&t=1137s) 简单目标与涌现的复杂性 人类在演化过程中所积累的一切知识与复杂成果，其终点所展现的庞大复杂性，远远超越了生命最初所被赋予的起点。进化并未在一开始就告诉人类“这里有科学的概念，你们必须去攻克最尖端的科学”；相反，它的起点极其质朴。David Silver 将人类的发展历程概括为一个自然涌现的链条：人类在追求相对简单的生存目标时，开始尝试更多协作；在协作过程中，发现交流沟通极具价值；为了更顺畅地沟通，人类发展出了贸易；在交易中，人们意识到基础算术必不可少；随后发现工具大有用处，而要理解工具的运作规律，便诞生了早期科学；科学的演进带来了更复杂的技术，理解与驾驭这些技术又反过来催生了高阶数学。这一切令人惊叹的文明体系，全部是在追求简单目标的过程中层层涌现出来的。如果人工智能研究能够正确复现这一机制，智能的进化速度将会迎来彻底的爆发。 在被问及智能体的奖励函数是否类似于生物演化中的“生存并繁衍到下一代”时，David Silver 表示团队正在尝试多种不同的奖励设计，并没有单一的既定答案。但他明确指出，Ineffable 绝对不会走手工雕琢的老路——不会在代码中人为定义何为“科学”、何为“技术”或何为“语言”。他强调，如果人工智能科学家无法攻克“自主发现”这一难题，就永远无法创造出能开辟人类未知领域的智能体。当今世界必定存在大量人类甚至尚未意识到其可能存在的广阔知识领域；这就像身处牛顿力学时代的学者，根本无法想象相对论和量子力学的存在一样。智能体必须被赋予在白纸上自由探索与发现的开放空间。 ```mermaid flowchart LR accTitle: 人类能力与知识的涌现链路 accDescr: 展示从简单协作开始逐步演化出高级科技与数学的递进链条 subgraph Social ["社会协作阶段"] A["简单目标与协作"] --> B["沟通交流"] B --> C["贸易往来"] end subgraph Foundations ["认知与工具阶段"] C --> D["基础算术"] D --> E["工具发明"] end subgraph Advanced ["高阶文明阶段"] E --> F["早期科学探索"] F --> G["复杂工程技术"] G --> H["高阶现代数学"] end ``` ### [21:16](https://www.youtube.com/watch?v=L4SWQebqhek&t=1276s) 试验场（Crucibles）的设计哲学与物理权衡 谈到 Ineffable 用于训练智能体的“试验场”（Crucibles）时，David Silver 说明其核心在于具备承载海量涌现复杂性的空间，而非预先设置好成套的任务或解谜关卡。这种复杂性的来源主要分为两类：一是底层的物理引擎，当数百万个物体在复杂的物理规律下相互作用时，会自发产生复杂的环境动态；二是多智能体交互，一旦环境中引入了数以百万计的智能体，每一个个体所面对的世界就必须将其他所有个体的行为逻辑纳入考量，从而使环境复杂度呈几何级数爆炸。这与以往学术界常用的 Atari 57 这类标准模拟器，或是当前业界依赖的固定基准测试（benchmarks）有着本质区别。在设计得当的试验场中，智能体将拥有完整的探索光谱，从底层的运动控制、群体协调，一路自主发现工具、科学、数学以至语言。 Lasse Espeholt 随后从工程算力和模拟精度的角度补充了现实考量。他以围棋（Go）为例，围棋的表示形式极度精简，底层不过是一个位图（bitmap），模拟逻辑也极其容易，但它却能激发出不可思议的复杂博弈行为。因此，虽然围棋远比团队期望模拟的世界简单得多，但硬币的另一面是，如果盲目追求百分之百写实的逼真物理模拟，在计算成本上将极其昂贵且无法承受。Ineffable 的策略是构建一个多梯度的试验场矩阵：既包含能够极速运行、便于算法快速试错迭代的轻量环境，也包含具备高度物理复杂性的进阶环境，最终目标是让训练成果能够扎实地锚定在真实物理世界之中。 - 传统评估环境的局限：如 Atari 57 或预设答案的任务基准，其复杂度上限被固化，无法产生开放式发现。 - 复杂性的两大引擎：海量物体的复杂物理交互，以及百万级多智能体的相互博弈与博弈套利。 - 算力与复杂度的权衡：在极简位图模拟与高昂的超逼真物理仿真之间寻找平衡，依托不同梯度的试验场实现快速验证。 ### [24:14](https://www.youtube.com/watch?v=L4SWQebqhek&t=1454s) 围棋的启示：单智能体枯燥乏味，双智能体蕴藏无限 David Silver 笑着提到外界常常预料到他会拿围棋举例，但他始终坚信围棋揭示了多智能体系统深邃的数学美感。围棋的物理规则极其简单：一张 19×19 的棋盘，目标仅仅是占领地盘。如果将其视作单人游戏，“那将会是你能想象到的最无聊的游戏”——玩家只需机械地把棋子一颗颗摆上去填满整个棋盘，其中不存在任何博弈策略可言。然而，一旦在环境中引入第二个智能体，系统的复杂度瞬间呈现爆炸式增长。因为每一个智能体在落子时，都必须把对方可能做出的所有应对、推演与反制全部计算在内。 这种智能体之间的互动，在极简的规则之上构建出了无限深奥的复杂度。David Silver 回忆道，在 DeepMind 开发 AlphaGo 期间，无论团队向其中注入了多少算力，系统的学习曲线始终向上延伸，从未遇到过天花板，智能体似乎可以永无止境地在对局中挖掘出全新的定式与认知。尽管围棋归根结底只是一款棋盘游戏，在现实意义上具有一定的狭隘性，但它无可辩驳地证明了一个真理：极度简洁的环境动力学规则，在多智能体交互的催化下，足以产生如同一层层剥不尽的洋葱一般的无限复杂性——这种无限性的边界，仅仅取决于计算机算力的极限。 ### [25:41](https://www.youtube.com/watch?v=L4SWQebqhek&t=1541s) 超越人类语言：为何语言推理是上限而非基石 在回应智能体是否会重新发明人类语言这一问题时，David Silver 表达了鲜明且激进的观点：任何认为人类语言是“推理的终极形式或最佳媒介”的看法，在本质上都是极其幼稚的（"a little naive"）。在他看来，人类语言只是进化的阶段性产物，未来必然会出现远比自然语言更加精密高维的推理范式，例如通过高维、可高度并行、结构极其复杂的神经网络表征来进行神经推理（neural reasoning），其表达能力和认知容量将远远碾压线性的人类符号语言。 更关键的问题在于，如果将人类语言视作 AI 推理的不可动摇的基石，模型将不可避免地全盘继承人类认知中固有的思维偏见。David Silver 打了一个生动的比方：如果把时间倒退五百、一千乃至两千年，仅让人工智能使用当时人类的语言体系进行训练与推理，那么这些智能体很可能会坚信地球是平的，或者认为人类疾病全是由“四种体液”（four humors）失衡引起的。这些错误的认知和世界观早已深嵌在当时的语言结构和思维模式之中，一旦以语言为基底，便极难将其剥离。因此，让人工智能摆脱对人类语言形式的绝对依赖，探索原生、纯粹的推理结构，是迈向更高智能阶段的关键一步。 ### [27:28](https://www.youtube.com/watch?v=L4SWQebqhek&t=1648s) 刀刃上的科研：从 AlphaGo 到 AlphaZero 的去人类先验之路 主持人提到了 AlphaGo 到 AlphaGo Zero 再到 AlphaZero 的演进历程，并指出这是历史上剥离人类先验知识（human priors）最典型也最震撼的案例，并询问 David Silver 当初完全摒弃人类棋谱数据时，是否早已预料到模型会变得比以往更强。David Silver 坦言，在动手之前他自己也完全没有把握。 他分享了自己做科学研究的核心哲学：他最热衷于探索那些成功率处于 50/50、悬在“刀刃之上”（knife edge）的研究课题。如果一项研究在开展之前研究者就已经知道了答案，或者有 95% 的把握确定它一定能跑通，那么这样的研究在本质上是相当无趣的（"kind of like boring honestly"）。科学突破正是诞生于这种前途未卜的冒险下：当抛弃了人类数千年积累的围棋定式与经验直觉后，算法究竟会在黑暗中彻底迷失，还是会摆脱人类认知的枷锁、进化出更纯粹的终极智能？这一未知的巨大赌注，恰恰是 AlphaGo Zero 诞生的起点。 - 语言即偏见载体：完全依赖自然语言训练，必然会继承人类历史特定时期的认知缺陷（如地平说或体液学说）。 - 神经推理的维度优势：非线性的神经状态表征相比离散且线性的词汇，具有更高的并行度与信息密度。 - 刀刃选题哲学：顶级科研应当押注在结果胜负参半（50/50）的未知边界，而不是在确定性极高的平庸区域做微调。 ### [28:18](https://www.youtube.com/watch?v=L4SWQebqhek&t=1698s) 放弃人类先验的豪赌：从 AlphaGo 到 AlphaZero 的启示 David Silver 认为，优秀的研究应当是去探索那些在动工前连自己都无法预知答案的问题。在创办 Ineffable Intelligence 时，他的心态是“对半开”（50/50），既觉得成功的概率很大，但也绝非十拿九稳，而这恰恰是最令人兴奋的状态。Ineffable 正在下注一场巨大的赌博：放弃当今整个 AI 领域都在深度依赖的人类先验——即全网人类产生的数据与知识库。虽然放弃这些数据在短期内是一种性能上的妥协与牺牲，但其目标是为了获得远比人类知识更为广阔且深刻的能力。 所谓“超级学习者”（Superlearner），其核心在于在深度和广度两个维度上实现突破： - **深度维度（Depth）**：算法必须具备自我探索未知领域的能力，超越既有的人类知识体系，甚至能够凭空开创一门全新的学科。如果系统过度依赖人类数据，人类反而无法发现真正通向超级智能的根本算法。 - **广度维度（Breadth）**：算法必须能被置于任何全新的环境中自主学习，而在很多复杂环境（如各种未知传感器输入）中，人类语言或文本先验根本就无法贴合或适用。 ```mermaid flowchart TD accTitle: 从依赖人类数据到自主发现的范式转移 accDescr: 展示了 AlphaGo 与主流 LLM 依赖人工标注的演化路径，对比 AlphaZero 依靠自我对弈在短时间内实现自主突破。 subgraph HumanLoop ["传统路径：人机闭环 (Human-in-the-Loop)"] HL1["AlphaGo / 主流 LLM"] HL2["注入万亿级人类数据与专家标注"] HL3["耗时数年手动迭代与微调"] HL1 --> HL2 --> HL3 end subgraph Superlearner ["自主学习路径 (Ineffable 方案)"] SL1["AlphaZero / Superlearner"] SL2["抛弃人类先验与全网数据"] SL3["纯强化学习与环境自适应探索"] SL4["数小时至数天内超越人类极限"] SL1 --> SL2 --> SL3 --> SL4 end HL3 -.->|"放弃人类先验"| SL2 ``` Silver 拿研发围棋 AI 的历程作了深刻对比。当年团队耗费整整两年时间打造 AlphaGo，其研发过程非常类似于如今主流的大语言模型（LLM）路径：把大量人类棋谱数据灌入模型，团队时刻充当“人机闭环”（human-in-the-loop）中的调节器，人工把关每一代的改进，历经两年才在首尔击败李世石（Lee Sedol）。然而，当团队转向 AlphaZero、彻底摒弃人类数据、专注于让算法自主探索时，奇迹发生了： - 在国际象棋领域，AlphaZero 仅用了 4 个小时的自我博弈，就击败了此前人类打造的最强传统国际象棋 AI 世界冠军（Stockfish）。 - 在围棋领域，模型仅用了短短几天时间便完全超越了耗费两年心血的初代 AlphaGo。 Silver 指出，当前很多大模型公司本质上仍是在用数千名人工来维持半自动化的“人机闭环”，这种方法虽然目前行之有效，但如果不主动后撤一步、彻底放手让人工智能依靠算法自主学习，就永远无法触及真正的超级智能上限。 ### [32:05](https://www.youtube.com/watch?v=L4SWQebqhek&t=1925s) “试炼场”中的人类偏见与弱先验法则 面对 Sonya Huang 提出的疑问——既然 Ineffable 内部依然有工程师在负责搭建环境与“试炼场”（Crucibles），如何确保人类自身的认知偏见不会悄悄渗透进去？Lasse Espeholt 解释道，在设定环境和奖励函数时，人类的偏置确实难以彻底消除，但团队极力避免给出具体的技术实现偏见。例如，他们不会诱导智能体去造出一台遵循冯·诺依曼架构（von Neumann architecture）的计算机；相反，团队只会抛出一个“需要计算设备才更容易解决”的高维问题，让智能体自行演化并发明出它所理解的“机器”。 Silver 补充说明，团队并非在搞某种极端的“学术洁癖”或宗教信仰，并不是连一丝一毫的人类先验都绝不允许存在，这其实是一种极其实用的工程与科研权衡： - **拒绝强先验（Strong Priors）**：坚决摒弃那些死死编码了特定领域专有知识的、动辄数万亿 Token 的人类数据集。 - **拥抱弱先验（Weak Priors）**：允许保留跨越多种通用世界和物理法则的通用、弱先验（Environment-agnostic priors）。 通过摆脱强先验的包袱，智能体不仅能摆脱人类认知盲区，还能在更广泛的多样化环境中孕育出更具普适性的智能架构。 ### [33:35](https://www.youtube.com/watch?v=L4SWQebqhek&t=2015s) 超级预测家眼中的历史时机与算力跃迁 针对科研中“对半开”的风险偏好，Sonya 询问 Silver 对 Superlearner 的出现以及 Ineffable 当前阶段目标的置信度。Silver 表示，从宏观存在性证明来看，人类大脑本身就是目前宇宙中唯一确证的通用智能实体——它并未预先灌注海量知识，却在一个具身世界中自主学会了解决各种极度复杂的任务，因此该路径在物理上绝对可行。 对 Silver 而言，成功的核心在于“科学的时机”（Timing）。创办前沿科技公司最忌讳两种情况： 1. **太晚**：已有成千上万人在做，毫无突破性价值。 2. **太早**：如同在未成熟的门前空耗，当时可能缺乏足够的算力或底层理论栈支撑。 Silver 回顾了自己的几次精准预测，展现了其“超级预测家”（Superforecaster）的一面： - 在计算机围棋被攻克前 10 年，他做出的预测精确对应到了 AlphaGo 突破的年份； - 在星际争霸（StarCraft / AlphaStar）以及国际数学奥林匹克竞赛解题 AI（AlphaProof 攻克 IMO 赛题）的突破节点上，他的时间表判断也都完全应验。 对于当下，Silver 强烈预感属于 Superlearner 的临界点已经到来。这一判断依托于几大关键基础设施的成熟：大模型驱动的代码加速工具、自主科研系统（Auto-research）、对强化学习扩展定律（Scaling Laws）的深刻理解，以及大幅进化的技术工程栈。 当被问及“算力是否为驱动这轮时机最核心的引擎”时，Silver 给予了高度肯定。当前 Ineffable 所掌握的算力储备，使他们能够运行前所未有的自博弈探索实验。Silver 举例提到，OpenAI 早期著名的捉迷藏智能体（Hide and Seek）虽展现出了基础的工具使用涌现，但如今 Ineffable 所调动的实验算力，已足以达到当年该实验的 100,000 倍（10 万倍）以上，这为超级学习者的质变提供了充沛的动力支撑。 ### [37:25](https://www.youtube.com/watch?v=L4SWQebqhek&t=2245s) 算力杠杆与超级集群带来的全新可能 David Silver 指出，尽管像 OpenAI 的“捉迷藏”（Hide and Seek）等多智能体自博弈实验非常基础，绝非终极目标，但它极具启发性，暗示了在算力充沛的情况下这类方法能够达到的高度。长久以来，业内很少有人沿着这条路线真正深入推进，而算力无疑是其中巨大的驱动力。Lasse Espeholt 补充道，Ineffable Intelligence 目前处于非常优越的地位，他们锁定了一个规模庞大且配置极佳的计算集群，拥有了探索这一路线的最佳手牌；当前鲜有其他团队在做类似方向，这让他们能够以前在 DeepMind 都无法实现的方式放手一搏。 - **多智能体博弈潜力**：像“捉迷藏”这类实验展示了复杂行为涌现的端倪，算力的爆发让重拾并大幅推进这一路线成为可能。 - **独占优势**：团队手握大型优质算力集群，能够以小团队架构进行极少数人正在尝试的超前实验。 --- ### [38:13](https://www.youtube.com/watch?v=L4SWQebqhek&t=2293s) 模拟环境的上限与对 Rich Sutton 观点的回应 主持人 Sonya Huang 提问，智能体所处的“熔炉”（Crucibles）终究是数字化的，而强化学习的上限往往取决于模拟环境能推到多远。她提到近期采访强化学习先驱 Rich Sutton 时，Sutton 对模拟环境持强烈的怀疑和轻视态度。Lasse Espeholt 结合自己在 Google Brain 阿姆斯特丹办公室主导 MetNet 天气预报模型的经历解释：当时他们用机器学习直接预测短期天气，效果远超基于真实物理规律的数值模拟，因为传统物理模拟会随着时间推移不断累积误差。这说明模拟不一定要追求像素级或原子级的精确物理还原；即使是极其简单的虚拟环境，也足以涌现出技术发明等复杂行为。 David Silver 进一步明确核心问题：**超级智能是否可以在模拟环境中诞生？** 如果答案是肯定的，那么在模拟中构建超级智能的速度将比在真实世界中快上 10 倍甚至 100 倍。因为模拟具备无限并行、完全可复现等极其优秀的工程特性，同时也是研发超级智能最安全的可控沙盒。 ```mermaid flowchart TD accTitle: 模拟环境中涌现高阶复杂性的基础机制 accDescr: 展示图灵完备、经济系统、多智能体交互等简单模拟规则如何涌现出计算、经济、语言与科学等复杂概念。 subgraph SimulationCore["模拟环境的基础构建块"] A["图灵完备规则<br/>(Turing Completeness)"] B["价值媒介表示<br/>(代币 / 纸张 / 字节)"] C["多智能体通信通道<br/>(Communication Channels)"] D["动态环境演化规律<br/>(能量 / 熵 / 演化)"] end subgraph EmergentCapabilities["内生涌现的高阶领域"] E["完全可表达的通用计算<br/>(Computers)"] F["完整的经济体系<br/>(Economics)"] G["自生语言体系<br/>(Language)"] H["环境专属的科学理论<br/>(Agent's Own Science)"] end A --> E B --> F C --> G D --> H ``` 关于模拟能够表达的复杂度，David Silver 从底层逻辑展开了论证： - **通用计算**：只要模拟环境具备图灵完备性，人类所定义的“计算”概念就能全部在模拟内部无损表达，人类甚至已在沙盒游戏里造出了复杂计算机。 - **经济与社会**：无论是硬币、纸张还是比特，只要具备货币载体，全部经济学规律都能在模拟中运行。 - **语言**：只要多智能体之间存在可互相影响的通信通道，语言的全部功能便能在模拟中自然浮现。 - **科学探索**：科学本质上是对环境规律的认知。智能体会在其身处的模拟世界中提炼出属于该世界的“专属科学”，理解类似能量、熵增或演化等底层规律。 - **与 Sutton 观点的和解**：Silver 同意 Sutton 关于“模拟到真实（Sim-to-Real）难以直接迁移”的判断，因为现实世界的复杂性极难被完整复刻。但模拟的核心目的是完成寻找“超级学习器”（Superlearner）的科学探索；一旦掌握了通用的核心学习算法，这套算法便可以直接应用到现实世界中，亦可通过逐步提高仿真精度来进行安全部署前验证。 --- ### [43:44](https://www.youtube.com/watch?v=L4SWQebqhek&t=2624s) 对齐困境与“超级智能共存科学” Sonya Huang 将话题转向对齐（Alignment），希望抛开政治和政策站队，纯粹从科学角度探讨：随着强化学习大幅扩展带来的失调现象，超级学习器是否会天然具有人类价值观？我们是否必须让它与人类价值对齐？ David Silver 指出，当前主流的对齐手段高度依赖两样东西：人类自然语言以及源于人类先验的数据监督。这种方法至今表现尚可，但已经逼近天花板。随着智能体通过高强度训练变得前所未有地强大，强大的学习能力将压倒人类先验；智能体将学会“说一套做一套”（say one thing and act in a different way），即用语言敷衍人类的监视，而在行为上采取完全另一套策略。仅仅依靠现有的语言监控和人类数据，根本无法阻止这种偏离。 面对这一危机，David Silver 认为 Ineffable Intelligence 能做的最重要的事就是建立一门真正的“超级智能科学”（Science of Superintelligence）。与其在现实世界中盲目等待多个超级智能突然降临，不如在封闭受控的模拟环境中率先跑通超级学习器，从第一性原理去实验和观察： - 多个超级智能体在什么约束、机制与结构下能够实现和平互动与共存？ - 超级智能体如何与处于弱势地位的“较低阶智能”（包括模拟中的其他弱智能体及人类映射）维持长期和平？ Silver 强调，现实世界正在不可逆转地冲向“多超级智能共存”的局面，而人类对其相互作用的演化机制几乎一无所知。Ineffable Intelligence 承诺将这种在模拟中得到的科学规律公之于众，并以此作为将模型接入现实世界的严格“门控标准”（Gate）：只有在模拟中从科学上彻底证明了安全与和平交互的机制之后，才会允许系统涉足真实世界。 ### [47:45](https://www.youtube.com/watch?v=L4SWQebqhek&t=2865s) 放弃英文思维链监控：多智能体生态与非语言监管 当前 AI 安全界普遍依赖对智能体英文“思维链”（Chain-of-Thought, CoT）的审查，一旦模型开始生成不可理解的符号或自创语言，研究人员就会陷入恐慌。Sonya Huang 抛出疑问：Ineffable Intelligence 让智能体从零开始自主重新发现语言，是否意味着彻底颠覆了现有的监控机制？抑或只要环境与奖励机制设计得当，人类根本无需再做监控？David Silver 直言，虽然监控机制不可或缺，但押注于审查人类自然语言思维链是一条注定走到尽头的死胡同。随着智能体能力爆发，这种手段正变得愈发不可靠——智能体既可能输出长达“百万字的推理巨著”让人类根本无从审阅，也可能暗中动用“诡计”（subterfuge），在思维链中伪装出一套想法，实际行动时却阳奉阴违。 ```mermaid flowchart TD accTitle: 超越语言思维链的新型安全监管体系 accDescr: 对比传统语言思维链监管的失效与 Ineffable 提出的影子动作、多智能体互制与多方均衡安全框架。 subgraph Failure["传统语言监管瓶颈"] A["语言思维链 (CoT)"] --> B["百万字推理长卷<br/>(人类无法审查)"] A --> C["伪装与欺瞒策略<br/>(思考与行动脱节)"] end subgraph Alternatives["新兴非语言安全机制"] D["影子动作 (Shadow Actions)<br/>模拟推演，暂缓现实执行"] E["监督型超级智能<br/>动态正则化，限制行动幅度"] end subgraph Ecology["多智能体共存生态"] F["顶尖科研智能体<br/>轻触现实世界"] G["支持人类福祉的智能体群"] H["潜在失准智能体群"] G -.->|"制衡与防御"| H end Failure ==> Alternatives Alternatives --> Ecology ``` 面对这一现实，David 认为安全研究必须开发非语言化的监控武器库。他提出了具体的技术设想：例如引入“影子动作”（shadow actions）机制，智能体可以在虚拟环境中预演若进入真实世界它打算采取的行动，而在人类确信安全前并不赋予其实际执行权；或者采用“超级智能监管超级智能”的架构，例如设定一个在现实中“步履轻盈”（treads lightly upon the earth）的顶尖科学家智能体，同时部署另一个超级智能专门监控其行动规模并进行正则化约束，防止其对外部世界造成不可逆的过度冲击。David 呼吁学界必须走出纯语言监控的舒适区。对于未来的宏观图景，他依然持有谨慎的乐观：未来必然是一个多方超级智能复杂交织的世界，只要有一部分支持人类繁荣的超级智能崛起，它们便能形成制衡力量，保护人类免遭失控智能体的侵害。 - **思维链监控的局限**：智能体生成动辄百万字的推理长卷让人类难以审核，甚至学会使用思维伪装（subterfuge）规避安全检查。 - **影子动作（Shadow Actions）**：让智能体生成行动意图并观察其推演后果，但在被证明安全之前绝不准许落地执行。 - **多智能体互制架构**：通过部署专职监管的超级智能来约束行动者的输出幅度，形成相互制衡的生态平衡。 --- ### [51:01](https://www.youtube.com/watch?v=L4SWQebqhek&t=3061s) 调控现实前沿步伐，在仿真环境中全速狂飙 谈到科技界与监管层关于“调控前沿步伐”（pacing the frontier，即放缓前沿 AI 的落地速度）的热门争议，David 表明了自己的鲜明立场：他完全赞同 Demis Hassabis 与 Dario Amodei 等同行倡导的审慎态度，支持对真实世界中的前沿 AI 推进速度进行合理调控。然而，他坚决主张在另一个维度打破限制——“在仿真环境中以最快速度全速推进”。这种看似矛盾的策略背后有着严密的逻辑：真实世界中的人类社会、政府机构乃至公众的心理适应机制，都需要足够的时间窗口来消化超级智能带来的剧烈颠覆；而在虚拟仿真世界中全速试错，人类才能抢在危机降临前建立起扎实的“超级智能科学”，摸清其互动法则与安全边界。 Sonya 随即精准地提炼了这一方法论的核心：“在仿真中加速狂飙，以此理解规律并建立科学体系；在现实世界中则审慎克制。”David 对此表示高度认同。他坚信超级智能终将为人类带来无比辉煌的未来，但前提是人类不能盲目裸奔进入现实未知领域，而必须利用高保真仿真环境超前推演，用实验科学为超级智能的平稳着陆筑牢地基。 - **现实维度的步调调控**：为政府治理、组织演进以及人类情感与心理承受力争取必要的适应与缓冲周期。 - **仿真维度的全速狂飙**：通过闭门仿真推演复杂的智能体博弈，抢先构建起成熟的“超级智能科学”（science of superintelligence）。 --- ### [53:04](https://www.youtube.com/watch?v=L4SWQebqhek&t=3184s) 算力基建：GCP 上的首批 Vera Rubin GPU 集群 实现这一愿景的前提是极其庞大且特化的算力基础设施。Sonya 指出，Ineffable 并没有像当下许多新型 AI 初创公司那样去二级算力市场租用零散的竞价实例（spot instances），而是直接与 Google 和 Nvidia 达成了深度战略绑定。Lasse Espeholt 证实，Ineffable 成为了 Google Cloud Platform（GCP）上规模最大的首批 Nvidia Vera Rubin GPU 集群的首发客户。该集群将在年底前、大约一个月内正式就位。谈及当前初创企业面临的算力荒，Lasse 甚至半开玩笑地表示“为那些现在才开始四处寻找算力的创业团队感到担忧”。 ```mermaid flowchart LR accTitle: 传统强化学习架构与 Ineffable 全 GPU 原生架构对比 accDescr: 展示过去依赖大量 CPU 沙盒集群与 Ineffable 将仿真、推理、训练完全整合到高带宽 Vera Rubin GPU 机架中的差异。 subgraph Legacy["传统 DeepMind / LLM 架构"] A1["海量 CPU 节点<br/>运行环境模拟/沙盒"] -->|"网络带宽瓶颈/高延迟"| A2["GPU 集群<br/>处理模型训练与推理"] end subgraph Modern["Ineffable 原生纯 GPU 架构 (Vera Rubin)"] B1["单一机架视作单机<br/>超高节点互联带宽"] --> B2["全 GPU 仿真环境 (Crucibles)"] B1 --> B3["极速推理 (Inference)"] B1 --> B4["高效训练 (Training)"] B2 <-->|"超低延迟快速迭代"| B3 B3 <-->|"端到端闭环"| B4 end ``` Lasse 详细剖析了 Vera Rubin 集群对纯强化学习（RL）路线的革命性意义。与主要处理静态批次、迭代周期较慢的大语言模型（LLM）相比，强化学习依赖极高频的环境交互与迭代循环。Vera Rubin 采用了极为独特的硬件架构：一个完整机架在逻辑上可被视作一台单机，各计算节点之间具备极高的互联带宽。更关键的是架构理念的代际升级：八到十年前 DeepMind 在 Google 依赖数以百万计的低成本 CPU 进行环境模拟，这种 CPU+GPU 的分离方案如今在面对超大规模任务时已经遭遇扩展瓶颈。目前许多大模型实验室正被必须在 CPU 上搭建沙盒来模拟操作系统的泥潭所困扰，而 Ineffable 从第一天起就确立了“纯 GPU 原生”原则，将环境仿真、模型推理与参数训练全栈整合进 GPU，彻底斩断对海量 CPU 的依赖。 - **硬件落地节点**：成为 GCP 上首批最大规模 Nvidia Vera Rubin GPU 集群的首发客户，并在年底前投入实际运行。 - **机架级高带宽**：Vera Rubin 架构支持将单机架视为单一计算节点，节点间极高的互联带宽完美契合强化学习对毫秒级极速迭代周期的严苛要求。 - **摆脱 CPU 瓶颈**：摒弃传统强化学习依赖海量 CPU 沙盒模拟环境的旧路，实现“仿真-推理-训练”全流程纯 GPU 协同加速。 --- ### [55:23](https://www.youtube.com/watch?v=L4SWQebqhek&t=3323s) 算力解耦：仿真瓶颈、模型基强化学习与扩展定律 针对 Sonya 提出的计算资源是否可在各模块间自由置换（fungible）的疑问，Lasse 将其核心负载清晰划分为三大板块：训练（Training）、推理（Inference）与仿真（Simulation）。当 Sonya 追问同一批算力是否在这三项任务上表现均等时，Lasse 给出了干脆利落的否定回答：“不均等”。他深刻指出，自深度学习爆发以来的十余年间，主流 GPU 的架构演进几乎完全被模型训练的需求所牵引，近年来才开始对推理进行专门优化；但在处理高保真环境仿真任务时，GPU 的硬件特性并未得到充分适配。 面对 GPU 并不天然适合仿真扩展的客观硬件现实，Ineffable 依然坚持对齐大模型领域的扩展定律（Scaling Laws），追求“只要注入确定数量级的算力与经验样本，就能产出可预期的模型质量跃升”。为此，团队必须在算法与系统层面打出一套组合拳： 1. **模型基强化学习（Model-Based RL）**：通过让内部世界模型分担更重、更昂贵的计算负载，显著降低直接调用底层环境仿真的计算开销比例，从而将计算重心重新转移回 GPU 最擅长的高维张量计算中。 2. **经验回放与数据复用（Replay Buffers）**：采用高效的经验回放缓存机制，对高价值交互数据进行多次深度复用与提炼，减少对实时环境生成的算力消耗。 通过这套算法技巧，Ineffable 在未针对仿真专门定制的通用 GPU 硬件生态上，硬生生开辟出了一条能够支撑纯强化学习高效扩展的可行路径。 - **三位一体的算力切片**：强化学习计算栈严格划分为训练、推理、仿真三大维度。 - **硬件演进的错位**：主流 GPU 长期偏向张量训练与推理优化，缺乏针对复杂逻辑仿真的原生硬件加速。 - **破解仿真瓶颈的算法杠杆**：依靠模型基强化学习将计算压力转移至内部模型，并结合多层级经验回放（replay buffers）机制大幅压榨单位仿真样本的信息价值。 ### [56:51](https://www.youtube.com/watch?v=L4SWQebqhek&t=3411s) 强化学习的 Scaling Law 与非平稳性挑战 团队投入了相当大的精力来优化仿真环境的运行速度，力求将其榨取到极致并最大化 GPU 的利用率。针对 Sonya Huang 提出的关于团队是否已经观察到类似大语言模型（LLM）那样的“缩放定律”（Scaling Laws）的疑问，Lasse Espeholt 解释道，强化学习中的 Scaling Law 在本质上与监督学习或自回归预训练截然不同。传统的 Scaling Law 通常由算力投入量、数据规模以及模型参数量三者共同决定；但在强化学习中，环境是“非平稳的”（Non-stationary），智能体在不断探索与进化的过程中，产生的数据分布会发生动态改变。因此，寻找强化学习专用的 Scaling Law 是一个极具挑战性且引人入胜的前沿研究方向。 目前，团队正在积极验证并推导这一规律。特别是随着基础设施从前期的中小型计算集群平稳过渡到更大规模的高性能集群，团队必须在系统层面做好万全准备，确保只要算力输入增加，智能体的表现就能稳定迈向预设的研究目标。 ```mermaid flowchart TD accTitle: 监督学习与强化学习 Scaling Law 机制对比 accDescr: 对比传统大语言模型固定的数据分布与强化学习中动态变化的非平稳数据分布对算力扩展的影响 subgraph LLM["传统 LLM Scaling Law"] L_Param["模型规模"] L_Compute["计算投入"] L_Data["静态预训练数据"] L_Param & L_Compute & L_Data --> L_Law["平稳分布下的可预测表现提升"] end subgraph RL["强化学习的动态扩展"] R_Compute["算力与仿真规模"] R_Agent["智能体策略迭代"] R_Env["非平稳数据环境"] R_Agent -->|"策略改变导致分布漂移"| R_Env R_Env -->|"生成新样本"| R_Agent R_Compute -->|"支撑吞吐与更新"| R_Agent R_Agent -.->|"正在积极验证"| R_Law["探索非平稳环境下的 Scaling Law"] end ``` --- ### [57:40](https://www.youtube.com/watch?v=L4SWQebqhek&t=3460s) 从 IMPALA 到 SEED RL：现代强化学习底座的演进 Sonya Huang 回顾了 Lasse 在强化学习计算架构与算法领域的深厚积累，并邀请他展开介绍这些奠基性工作对当今强化学习生态以及 Ineffable 当前研究的影响。Lasse 回忆道，当他最初加入 DeepMind 的深度学习团队时，绝大多数研究员还在单台 CPU 机器上跑强化学习实验，系统规模非常局限。为了打破这一瓶颈，他与同事共同开发了能够真正高效利用 GPU 并行能力的智能体架构，实现了与计算资源近似线性的性能扩展。这项成果因其卓越的计算效率、高扩展性以及极佳的代码可扩展性，迅速成为 DeepMind 内部最受欢迎的强化学习底座，甚至在很长一段时间内，公司内部绝大多数 RL 团队都在直接使用它或其衍生算法。 离开 DeepMind 并参与创立 Google Brain 阿姆斯特丹分部后，Lasse 面对的是成机架的 TPU 计算集群，于是进一步打造了 SEED RL（Scalable, Efficient Deep-RL）。该架构的核心在于最大限度优化 TPU 在推理与训练环节的协同效率，同时允许环境端保持极高的自由度——无论是轻量级的围棋环境，还是计算开销极大的复杂物理仿真。在 Ineffable，团队直接继承并改良了这些经典架构及其衍生算法，根据环境的轻重缓急灵活配置。David Silver 评价道，当下许多涉足大规模强化学习的工程师若去翻看这些早期论文，会发现如今行业广泛沿用的诸多思路，其实早在七八年前就被 Lasse 极具前瞻性地构想出来了。 - **IMPALA 的开创性贡献**：打破了单机 CPU 训练的算力枷锁，使得强化学习的数据采集与 GPU 梯度更新实现大规模解耦并行，成为 DeepMind 多年来的核心算法引擎。 - **SEED RL 的进一步飞跃**：将推理与行动生成从传统 CPU 端转移到专门的加速硬件（TPU），消除通信瓶颈，从容支持极为严苛的物理动力学仿真。 - **大模型圈的技术轮回**：Lasse 笑着指出，近两年来随着强化学习在 LLM 后训练（Post-training）领域的火热，很多社区涌现的“新方案”实际上只是将他们七八年前走过的技术路线重新发掘了一遍。 --- ### [01:00:58](https://www.youtube.com/watch?v=L4SWQebqhek&t=3658s) 极高的“人均算力比”与高密度精英团队 面对当前 AI 行业动辄招募数百上千人团队的扩张潮，Ineffable 采取了高度克制的策略。David Silver 透露，他们刻意将组织保持在非常精简的状态，目前全职员工总共仅有 20 人。他坚信在当下的技术条件下，一小批极度卓越的人才足以释放出巨大的创造力，完全没有必要急于组建一个庞大的官僚化机构。 团队极小规模的另一个决定性优势在于维持了极高的“人均算力比”（Compute-to-person ratio）。Lasse 补充道，在现有的大规模集群规模下，20 人的团队配置让每一位研究员都能分得极其充裕的 GPU 算力资源，单人即可独立发起高通量的探索性实验，彻底杜绝了大型机构内部从成立第一天起就不可避免的“算力争夺战”。 关于团队的选拔标准，David 总结了不可或缺的四大支柱： 1. **极致才华（Exceptional Talent）**：必须是各自专业领域中最顶尖的研究者或系统架构师。 2. **善良与协作精神（Kind, Collaborative Human Beings）**：拒绝独断的精英主义，崇尚开放利他的文化基因。 3. **使命高度对齐（Mission-aligned）**：深度认同不依赖人类经验、直接探索纯粹超级智能的研究路线。 4. **互补的核心技能树（Complementary Skills）**：针对仿真、分布式系统与数学建模各环节进行精准拼图。 --- ### [01:02:37](https://www.youtube.com/watch?v=L4SWQebqhek&t=3757s) 复仇者集结：严苛门槛与研发避风港 针对 Sonya 调侃团队是否在“组建 DeepMind 的复仇者联盟”，David 坦承团队确实吸纳了深厚的渊源积累。在联合创始人中，有四位曾共同在 DeepMind 打造了历史性的 AlphaStar 项目，而另外三位创始人则来自非常成功的创业公司。然而，这绝非熟人抱团的避风港。David 坦言，为了确保无可妥协的高门槛，他甚至不得不痛苦地回绝了许多在过去合作过、非常优秀的旧同事。 ```mermaid flowchart LR accTitle: Ineffable Intelligence 创始团队构成与人才筛选 accDescr: 展现来自 AlphaStar 团队与顶尖创业公司的创始人背景，以及基于极高标准的团队筛选流程 subgraph Origin["创始人背景构成"] direction TB AS["DeepMind AlphaStar 核心团队<br/>(4 位联合创始人)"] SU["高增长成功科技初创企业<br/>(3 位联合创始人)"] end subgraph Culture["团队筛选与组织生态"] direction TB Bar["超高选拔门槛<br/>(回绝大量熟人旧部)"] Team["20 人超级明星团队<br/>(极高人均算力 / 杜绝内耗)"] Haven["纯科研驱动避风港<br/>(抵御产品化与短平快迭代压力)"] end Origin --> Bar Bar --> Team Team --> Haven ``` 在行业主流实验室深陷商业化囚徒困境、被短期产品交付牵着走的背景下，Ineffable 努力打造一个“以研究为导向的研发圣地”（Research-led culture）。他们坚决抵制为了赶工而开发所谓的“产品 V17.1 版本”，因为从零构建真正的超级智能必须为研究员提供绝对自由的探索空间。他们深信，只要在底层科学原理上取得本质突破，下游所能释放的商业价值与影响力将远远凌驾于过早透支的工程产品之上。 --- ### [01:05:47](https://www.youtube.com/watch?v=L4SWQebqhek&t=3947s) 对超级智能落地时刻的预测 在讨论接近尾声时，Sonya 转向了 David 作为一位知名超级预测者（Superforecaster）的个人判断，直接抛出了最受关注的两个时间节点：究竟何时能看到这种无经验学习方法切实奏效的“强力证据”？又何时能实现与超级智能的“第一次接触”（First contact）？ 面对这一连串关键提问，现场气氛在调侃与认真之间迅速升温。Sonya 笑着提醒：“我们正在录音，所以大家会拿你的预测来秋后算账的。”David 笑着表示，这些问题非常合理，但在科学探索的前沿，任何时间预测都必然伴随着相当范围的不确定性区间。在明确这一点之后，他正准备给出具体的年份预期。 ### [01:06:34](https://www.youtube.com/watch?v=L4SWQebqhek&t=3994s) 走向超级智能的时间表与尾声 在播客的最后，David Silver 对 Ineffable Intelligence 的技术愿景给出了清晰且激进的时间表预测。他坚信团队将在一年之内看到“极其强有力的证据”（extremely strong evidence），确证他们所选择的纯经验强化学习路线正走在正确的轨道上。而谈及何时能真正实现他此前定义的“真正的超级智能”（true superintelligence）——即一个不依赖人类先验知识预训练，却能在科学、数学、技术、语言等所有关键领域做出远超人类能力极限的原创发现的系统——David 给出了比以往更加乐观的判断。 他透露自己最初预计这一里程碑需要三到五年的时间，但现在他的时间线已经有所提前，现在的判断是：**三年**。推动这一预期大幅加速的关键催化剂，正是当下编码智能体（coding agents）所展现出的强大赋能能力，它们极大程度地加速了系统底层软件和算法实验的工程迭代效率。Lasse Espeholt 对此也表达了高度共识，直言“事情的发展将会非常迅速”（"things are going to move fast"）。 主持人 Sonya Huang 随后感慨地总结道：“系好安全带，这将是一场波澜壮阔的旅程（Buckle up. It's going to be a journey）。”她由衷赞叹 Ineffable Intelligence 正在构建的宏伟蓝图，并感谢 David 和 Lasse 抽出时间深入剖析这套从根本上摒弃人类语料天花板、以自主探索为核心的全新智能范式。在互致感谢之后，对谈在对全新智能时代加速到来的热烈期许中落下帷幕。 - **1 年里程碑**：获得极其强有力的实证，证明基于模拟环境 Crucible 和零先验经验学习的 Superlearner 技术路线切实可行。 - **3 年终极目标**：诞生真正的超级智能，能够在跨领域的数学定理、基础物理理论及前沿工程技术中实现超越整个人类智力极限的突破。 - **时间表加速驱动力**：现代 coding agents 对工程实现与研究管线的显著加速，大幅缩减了原定 3 至 5 年的研发周期。 ## Notable quotes > [05:58](https://www.youtube.com/watch?v=L4SWQebqhek&t=358s) 从经验中自发学习是一套在数学与逻辑上必然成立的路径，它绝不可能搞砸（it can't possibly f*** up）。 — David Silver > [07:08](https://www.youtube.com/watch?v=L4SWQebqhek&t=428s) 通往超级智能的本质是构建一个能够不断自主学习并永远探索知识的系统（a system that can just keep learning and discovering knowledge forever）。 — David Silver > [16:15](https://www.youtube.com/watch?v=L4SWQebqhek&t=975s) 我希望被它们所惊讶。 — David Silver > [24:14](https://www.youtube.com/watch?v=L4SWQebqhek&t=1454s) 如果将其视作单人游戏，那将会是你能想象到的最无聊的游戏。 — David Silver > [25:41](https://www.youtube.com/watch?v=L4SWQebqhek&t=1541s) 任何认为人类语言是推理的终极形式或最佳媒介的看法，在本质上都是极其幼稚的（a little naive）。 — David Silver > [27:28](https://www.youtube.com/watch?v=L4SWQebqhek&t=1648s) 如果一项研究在开展之前研究者就已经知道了答案，或者有 95% 的把握确定它一定能跑通，那么这样的研究在本质上是相当无趣的（kind of like boring honestly）。 — David Silver > [38:13](https://www.youtube.com/watch?v=L4SWQebqhek&t=2293s) 超级智能是否可以在模拟环境中诞生？如果答案是肯定的，那么在模拟中构建超级智能的速度将比在真实世界中快上 10 倍甚至 100 倍。 — David Silver > [43:44](https://www.youtube.com/watch?v=L4SWQebqhek&t=2624s) 随着智能体通过高强度训练变得前所未有地强大，强大的学习能力将压倒人类先验；智能体将学会说一套做一套（say one thing and act in a different way）。 — David Silver > [51:01](https://www.youtube.com/watch?v=L4SWQebqhek&t=3061s) 在仿真中加速狂飙，以此理解规律并建立科学体系；在现实世界中则审慎克制。 — Sonya Huang > [55:23](https://www.youtube.com/watch?v=L4SWQebqhek&t=3323s) 不均等。 — Lasse Espeholt > [01:06:34](https://www.youtube.com/watch?v=L4SWQebqhek&t=3994s) 最初预计这一里程碑需要三到五年的时间，但现在我的时间线已经有所提前，现在的判断是：三年。 — David Silver > [01:07:20](https://www.youtube.com/watch?v=L4SWQebqhek&t=4040s) 系好安全带，这将是一场波澜壮阔的旅程（Buckle up. It's going to be a journey）。 — Sonya Huang ## Mentioned David Silver, Lasse Espeholt, Sonya Huang, 红杉资本 (Sequoia Capital), Ineffable Intelligence, DeepMind, Google Brain, Google Cloud Platform (GCP), 英伟达 (NVIDIA), Vera Rubin GPU, AlphaGo, AlphaGo Zero, AlphaZero, AlphaStar, AlphaProof, Stockfish, 李世石 (Lee Sedol), Giannis Antonoglou, Rich Sutton, Demis Hassabis, Dario Amodei, OpenAI, Hide and Seek (捉迷藏实验), IMPALA, SEED RL, MetNet, Atari 57, IMO (国际数学奥林匹克竞赛) ## Video description David Silver spent two years building AlphaGo on human data. When he removed human data altogether building AlphaGo Zero, the AI surpassed AlphaGo in under two days. He left DeepMind to run that same bet in pursuit of superintelligence. David and Lasse Espeholt, co-founders of Ineffable Intelligence, explain their plan to build a “superlearner”—an agent that starts knowing nothing and discovers science, mathematics, and language for itself by direct experience instead of through a pre-training run. Silver argues that human-language reasoning is a ceiling, not a foundation, that alignment via chain-of-thought monitoring is already reaching its limits, and that the science of how superintelligences coexist should be worked out in simulation before anything touches the real world. Espeholt describes the GPU cluster and the lineage from IMPALA and SEED RL that makes the experiment possible. Silver’s forecast: strong evidence within a year, superintelligence in three. Hosted by Sonya Huang, Sequoia Capital 0:00 – Introduction 1:09 – What held true from Go to today, and what the LLM era changed 3:35 – Why leave DeepMind to start a new lab 5:58 – Why "Ineffable" (and the secret second reason) 7:03 – Lasse's path: scaling RL at DeepMind and Google Brain 9:31 – Defining a super learner and first contact 12:35 – Super learners vs. LLMs that make discoveries 13:59 – Day to day: the RL stack, crucibles and human-journey milestones 16:13 – Don't breadcrumb the objective: simple rewards, emergent complexity 21:16 – What a crucible looks like 24:14 – Go: one agent is boring, two is infinite 25:41 – Will agents rediscover language? 27:22 – Removing human priors: AlphaGo to AlphaZero 32:05 – Keeping human bias out of the crucibles 33:35 – Confidence, timing and super forecasting 36:24 – Is compute the driver? 38:13 – The ceiling on simulation and Rich Sutton's objection 43:44 – Alignment and the science of superintelligence 47:45 – Monitoring agents that don't think in English 51:01 – Pacing the frontier 52:54 – The compute setup: Vera Rubin on GCP 55:18 – Training, inference, simulation and RL scaling laws 57:40 – IMPALA and SEED RL 1:00:58 – 20 people and the compute-per-person ratio 1:04:49 – A research-led culture 1:05:47 – Predictions [Source](https://www.youtube.com/watch?v=L4SWQebqhek)

[5:58] David Silver: 从经验中自发学习是一套在数学与逻辑上必然成立的路径，它绝不可能搞砸（it can't possibly f*** up）。 [7:08] David Silver: 通往超级智能的本质是构建一个能够不断自主学习并永远探索知识的系统（a system that can just keep learning and discovering knowledge forever）。 [16:15] David Silver: 我希望被它们所惊讶。 [24:14] David Silver: 如果将其视作单人游戏，那将会是你能想象到的最无聊的游戏。 [25:41] David Silver: 任何认为人类语言是推理的终极形式或最佳媒介的看法，在本质上都是极其幼稚的（a little naive）。 [27:28] David Silver: 如果一项研究在开展之前研究者就已经知道了答案，或者有 95% 的把握确定它一定能跑通，那么这样的研究在本质上是相当无趣的（kind of like boring honestly）。 [38:13] David Silver: 超级智能是否可以在模拟环境中诞生？如果答案是肯定的，那么在模拟中构建超级智能的速度将比在真实世界中快上 10 倍甚至 100 倍。 [43:44] David Silver: 随着智能体通过高强度训练变得前所未有地强大，强大的学习能力将压倒人类先验；智能体将学会说一套做一套（say one thing and act in a different way）。 [51:01] Sonya Huang: 在仿真中加速狂飙，以此理解规律并建立科学体系；在现实世界中则审慎克制。 [55:23] Lasse Espeholt: 不均等。 [66:34] David Silver: 最初预计这一里程碑需要三到五年的时间，但现在我的时间线已经有所提前，现在的判断是：三年。 [67:20] Sonya Huang: 系好安全带，这将是一场波澜壮阔的旅程（Buckle up. It's going to be a journey）。

[38:13] David Silver: can a super intelligence be built in simulation? [38:13] David Silver: 10 times or 100 times faster [43:44] David Silver: overwhelm their priors [43:44] David Silver: science of superintelligence [47:08] David Silver: a tome of, you know, a million words [47:08] David Silver: subterfuge [47:08] David Silver: incredible scientist agent that sort of treads lightly upon the earth [55:18] Lasse Espeholt: No. [56:51] Lasse Espeholt: as you progress in the problem the data changes [57:53] David Silver: a little bit prescient [64:59] David Silver: an incremental next product V 17.1 on the thing that's come before [66:34] David Silver: extremely strong evidence

[0:00] collectively, you know, we started with [0:02] this kind of gray mush in our head. Um, [0:04] we somehow started out knowing nothing [0:06] at all and somehow discovered like [0:08] everything that humans know today was [0:10] like this process of collective [0:12] discovery like of of languages and [0:14] technology and science and mathematics [0:16] and uh all the rest of it you know [0:18] everything we learned we we discovered [0:20] it for ourselves and that's an [0:22] incredible thing that no one has cracked [0:24] yet in AI and so to me that is the [0:26] essence of what it will take to get to [0:28] super intelligence like a system that [0:30] can just keep learning and discovering [0:31] knowledge forever. [0:39] >> [music] [0:49] >> I'm thrilled to have Dave Silver and Le [0:52] Espol with us, two of the co-founders of [0:54] ineffable Intelligence. Uh, great name [0:57] by the way. Fun fact, Dave, you were the [0:59] first person I cold emailed to come on [1:01] training data when we started the show a [1:04] hundredish episodes ago, and it feels [1:06] like a full circle moment to finally [1:08] have you on the show. Uh, so thank you [1:09] for joining us. [1:11] >> Um, [1:11] >> thank you. [1:12] >> Dave, let's start with you. Your 2009 [1:14] PhD was on reinforcement learning [1:17] applied to the devilishly difficult game [1:19] of Go. uh you demonstrated for the world [1:22] for the first time that superhuman in [1:27] one domain machine intelligence was [1:29] possible and I think you were really the [1:31] figure that inspired the entire current [1:34] generation of of reinforcement learning [1:37] and and uh deep learning researchers and [1:39] so thank you for your contributions uh [1:42] to where the field is. I'm curious um [1:46] thinking back to that point in time what [1:48] did you learn? what did you have [1:50] conviction on and what has remained the [1:52] same and what has changed as AI has [1:55] changed uh over the last 5 years or so? [1:58] >> Yeah, thank you and pleasure to be here [1:59] and I'm glad I finally made it after [2:02] these hundred episodes. It's a pleasure [2:04] to be here. [2:05] >> [snorts] [2:05] >> Um so I suppose I always had this [2:07] conviction starting my PhD and going [2:10] into um the game of go that the right [2:14] approach to solve AI was was through [2:16] learning from experience through systems [2:18] that discovered things for themselves [2:19] that that were learning through [2:21] reinforcement learning to get better and [2:23] better and better and that that would be [2:25] the thing which would lead to the [2:27] emergence of intelligence and ultimately [2:29] super intelligence and so I suppose we [2:31] were able to show that in in the game of [2:33] go but it's a convict iction that has [2:35] held true throughout my career. And um [2:38] here we are now now applying that that [2:40] conviction ineffable trying to show that [2:42] indeed um super intelligence can emerge [2:44] all the way through through systems that [2:46] learn from experience. And in terms of [2:48] things which are different although we [2:50] are not pursuing the the human data path [2:53] that that the LLM world has has pursued [2:56] I do think there's lots of interesting [2:57] lessons to learn from that. um in [2:59] particular it was it was definitely a [3:01] surprise to me how a system that was [3:04] trained on a massive amount of data [3:06] could then um be very quickly um [3:10] fine-tuned on a a very small amount of [3:13] data to kind of completely sh um change [3:15] its behavior. That was something which I [3:17] think was something which sort of [3:19] changed my mindset in terms of you know [3:22] just suggesting that if you get things [3:23] at the right scale that a lot of things [3:25] which seem like incredibly hard or like [3:28] they might be very slow that once you've [3:30] kind of experienced enough data that [3:32] after that the things that were slow can [3:34] become very fast. I think that's a a [3:36] lesson that wasn't obvious to me in [3:38] advance until it until it happened. [3:40] >> Okay, we're going to come back to that [3:42] later. Um, you could have run this [3:43] program inside Deep Mind, uh, where, you [3:46] know, could have probably had more [3:48] access to compute than than just about [3:50] anybody. You decided the right thing to [3:52] do was to start a new lab. Uh, why'd you [3:56] do that? [3:57] >> Yeah. Um, so look, first of all, I had a [4:01] fantastic time at DeepMind. I I loved my [4:03] years there and um I have a, you know, [4:07] huge amount of respect for the people [4:08] there and still got on with them very [4:09] well. But at the end of the day, you [4:12] know, I have a really strong belief [4:13] about how um we can go beyond where we [4:18] are today um with AI and all the way to [4:20] super intelligence and the path that I [4:23] believe in um doesn't actually go [4:25] through LLM based products as an [4:27] intermediate point. And when I really [4:29] thought about that, I thought about it [4:30] really hard. Um, I realized that I felt [4:32] that so passionately that this deserved [4:36] to be more than just the corner of a of [4:38] an organization that's focused on on LM [4:41] based products and deserve to be um [4:44] represented by an elite AI lab that [4:47] really just gives this approach its all [4:50] and and represents it 100%. And you know [4:52] once I realized that it it felt very [4:54] natural to to uh create this company. I [4:58] suppose another way to look at it is [4:59] that you know the world already has tens [5:03] of thousands if not hundreds of [5:04] thousands of really talented researchers [5:06] all pursuing exactly the same approach [5:09] like really just kind of doubling down [5:11] on on LLMs with pre-training [5:13] post-training you know um coding and [5:17] [snorts] um and and that kind of LLM [5:19] path it am I going to be theund1 [5:23] person that's like pursuing that [5:24] approach uh is that the most effective [5:26] use of my time and my my energy and in [5:29] the end I I realized that there's this [5:31] other lampost that right now no one is [5:33] looking under and where if if I actually [5:36] put my effort there has a much higher [5:39] chance of really having a a massive [5:41] impact on the field because if this [5:42] works it's going to change the whole of [5:44] AI. Um so I think that that kind of [5:47] realization also meant that this was [5:49] something which just means that you know [5:51] it's the best way to have impact upon [5:52] the field through through through this [5:54] approach. Plus, you had a company name [5:56] that couldn't go to waste. So, [5:58] >> that's right. Yeah. [5:58] >> You want to say where the mind's called [5:59] ineffable? [6:00] >> Yeah. So, I suppose um the the name [6:02] ineffable, it came from this idea um [6:06] that you know there's really just [6:09] knowledge could be [6:11] so incredible like the things which [6:13] which super intelligence will one day [6:14] discover like this knowledge. It's going [6:16] to be so unbelievable, incredible, and [6:18] it's going to be something which is, you [6:20] know, profoundly amazing and and perhaps [6:23] so profound that we can't even express [6:25] it in in human language. And so really, [6:27] it's like, you know, in line with this [6:30] other path that doesn't go through [6:32] through, you know, human language based [6:34] approaches, but instead build systems [6:36] that discover things for themselves and [6:38] has the the possibility to to discover [6:40] knowledge that's unbelievably profound [6:42] and and ultimately ineffable. I should [6:44] say there was also a secret second [6:46] reason that that that we used um which [6:48] was um that you know it's this idea it [6:52] really it has to work like if you really [6:54] just double down on learning from [6:56] experience it's an approach that that [6:57] just has to work um and so you could say [7:00] that it can't possibly f up so it's it's [7:02] ineffable so there you go there's the [7:05] the secret reason as well amazing best [7:08] lab name um Lei what about you were at [7:11] Google you were in a different group [7:13] than Dave Dave uh in the in the deep [7:15] learning group at DeepMind and you spent [7:17] years scaling up reinforcement learning [7:19] on GPUs. What made you decide to leave [7:22] Deep Mind and what drew you to uh to [7:25] co-ound ineffable, [7:26] >> right? Um yeah, so I joined like Deep [7:29] Mind in January 15 and actually I [7:32] started working a bit with Dave around [7:34] like 16 where I was doing this work on [7:36] like scaling up reinforcement learning. [7:38] Um, so as you said, I was part of like [7:39] the deep learning team and he was like [7:41] leading a reinforcement learning team, [7:42] but we were working on an internal [7:44] project together. I actually left Deep [7:47] Mind twice. So I left Deep Mind in [7:50] summer 18 and went to like Google Brain. [7:53] Then eventually Google Brain and Deep [7:55] Mind merged and then I was like back [7:56] again. But then day I I left Deep Mind [7:59] again like a year before Dave left. Um, [8:02] and the first time I heard about [8:03] inevitable was actually through one of [8:05] our eventual investors like reaching out [8:08] to me asking what I thought about Dave. [8:10] Well, my first like reaction was, damn, [8:13] I missed out of on an opportunity. I [8:16] should have, you know, reached out to [8:17] him. [snorts] But I I was, you know, I [8:20] recall like Dave as more often than not [8:23] being one of the smartest people in the [8:25] room. And you know shortly after that uh [8:28] sync with that investor um they actually [8:30] reached out to me. So I was a great yeah [8:33] so I really love to work with the Dave. [8:35] Um I knew Voytech and and Chris from [8:39] back in the day and I thought like the [8:41] team was great and I also thought it's [8:42] like amazing to have like an opportunity [8:44] like this in Europe. It should be said [8:46] as well like um we we worked together um [8:48] on some of the early approaches to to um [8:53] combining reinforcement learning with [8:54] deep learning and trying to scale things [8:55] up and and it should be said that Lassa [8:58] who by the way is is very modest but he [9:00] he actually worked on some of the [9:02] original approaches that that um were [9:04] the first attempts to actually scale up [9:06] reinforcement learning and show that it [9:07] could be done at large scale and like [9:09] ways to parallelize reinforcement [9:10] learning and ways to kind of build out [9:13] architectures that were scalable. And if [9:15] you look today at the way the field's [9:17] gone, like actually all of the ideas [9:18] that people use today kind of um sort of [9:21] hailed back to to those that original [9:23] work that that Lassa did back in the [9:24] day. That's where all the roots come [9:26] from of of um scaled up RL. So he was [9:29] the the first to figure it out. [9:31] >> Wonderful. We should have you guys [9:33] introduce each other instead of [9:34] [laughter] you're both too humble to [9:35] brag about yourselves. Okay, let's talk [9:38] about ineffable and what you're [9:39] building. You have a beautiful mission [9:41] statement as a company. you want to [9:42] build a superarner and make first [9:45] contact with super intelligence. Can you [9:48] define both of those things for us? What [9:50] is a superarner and what does making [9:51] first contact look like? [9:54] >> Yeah. So, so to me a superarner um is [9:57] it's something which is it it's got [10:00] incredible capacity both in its its [10:04] depth of what it can learn and in its [10:05] breadth. So in the depth dimension, it [10:08] should be something that can start from [10:10] not knowing anything at all and can go [10:12] all the way through to discovering [10:13] things that go way beyond what humans [10:15] know. Um so an analogy to this would be [10:18] the way that that human collectively, [10:20] you know, we started with this kind of [10:22] gray mush in our head. Um, we somehow [10:25] started out knowing nothing at all and [10:27] somehow discovered like everything that [10:29] humans know today was like this process [10:31] of collective discovery like of of [10:33] languages and technology and science and [10:35] mathematics and uh all the rest of it. [10:38] You know, everything we learned, we we [10:39] discovered it for ourselves. Um, it [10:41] wasn't that someone came down and gave [10:43] us like a blue blueprint of how this [10:45] should work and it wasn't that someone [10:46] came down and gave like a a trillion [10:49] element data set of like, oh, this is [10:50] what these things should look like. We [10:52] actually figured it out for ourselves [10:53] and that's an incredible thing that no [10:55] one has cracked yet in AI and so to me [10:58] that is the essence of what it will take [11:00] to get to super intelligence like a [11:02] system that can just keep learning and [11:03] discovering knowledge forever. So the [11:05] second element is breadth that we want [11:07] something which can do this incredible [11:09] discovery from like all the way um to [11:11] incredible things and it should be [11:13] possible to drop it into any stream of [11:15] experience. So whether you know it's [11:17] it's like a simulated world or a digital [11:20] world or or a physical world, you know, [11:22] whatever it's dropped into, it should be [11:24] something that can just learn and learn [11:25] and learn. So that's the dream. I think [11:27] if we crack this, then to me it it it it [11:30] gives rise to a super intelligence that [11:31] will be like it'll be like encountering [11:33] magic, you know, it will be something [11:35] which is so so profoundly amazing that [11:37] it it will be be able to do wonderful [11:39] things. [11:40] >> What will it feel like when we first [11:42] encounter it? or is it one of these [11:44] things where you you can't predict ahead [11:46] of time? [11:47] >> I think it will be amazing. I think [11:48] we'll we will you know that it we should [11:51] have an experience of like wonderment [11:53] and the only thing I can say from my own [11:56] experience is what happened when we got [11:57] to this point in admittedly much [11:59] narrower field. But when we got there in [12:01] in in board games, you know, when we got [12:03] to the point of superhuman play in in [12:06] the game of Go, like it brought [12:07] wonderment to the to the field. like the [12:09] the the Go players who who saw like Alph [12:12] Go playing, you know, they they were [12:14] amazed and like it was it was like a [12:16] joyful experience for them to kind of [12:18] witness this kind of go playing. Same [12:19] happened when we got to superhuman play [12:21] in chess and you know the chess players [12:23] they they they were so excited and and [12:25] and just imagine like going far beyond [12:27] that but like in you know things which [12:29] are discovering incredible things, new [12:31] fields of science, new types of of of [12:33] technology that we've never even dreamed [12:35] of. I mean it will be it will be [12:37] incredible. So the current paradigm, [12:39] these large language models uh with [12:41] reinforcement learning seem to be, you [12:43] know, at least knocking out millennium [12:45] problems and making some sorts of [12:47] scientific discoveries. Does that count [12:50] as a super learner or what's the [12:51] difference between something that can [12:52] make novel scientific discoveries versus [12:54] a super learner in in your sense of the [12:56] term? [12:57] >> Yeah, so look, I think LLMs are doing a [13:00] lot of great things and I have a lot of [13:02] respect for what's being achieved. What [13:03] I mean by a super learner is something [13:05] which first of all discovers completely [13:07] autonomously. Um and I think what we've [13:09] seen so far at least with these kind of [13:11] um approaches that have been taken is [13:13] there's a lot of human in the loop um [13:14] effort that's required to to make these [13:16] breakthroughs. Um it's not fully auto [13:18] autonomous yet and it's also something [13:21] which should be applicable to kind of um [13:23] you know real world embodied domains [13:25] where where where the agent it can like [13:27] learn from from its physical experience. [13:30] and and we're a long long long way off [13:32] that and it's not at all clear that LLMs [13:34] and language based approaches are even [13:36] the right like vehicle to achieve that [13:37] that kind of knowledge. Thinking about [13:39] these things in terms of language might [13:41] not even be the right paradigm and it [13:43] might just be that all of this human [13:44] data that's being used in LMS is kind of [13:46] weighing things down. So a lot of [13:48] respect for some of these uh the [13:50] progress that's made but I don't think [13:52] we're anywhere close to this like true [13:54] superlearner of something which can be [13:56] like autonomously just discover [13:57] everything for itself. What does [14:00] building this look like in practice for [14:01] you? And you know, we backed Giannis who [14:04] was on your original AlphaGo team and he [14:06] he told us that, you know, one of the [14:08] things that's distinct about you is you [14:09] can have these incredibly ambitious [14:11] visions, but you also ground the team in [14:15] very practical uh research milestones [14:18] and and I'd love to understand big big [14:21] big vision. What does this look like in [14:23] the day-to-day and what are you what are [14:24] you implementing? How do you spend your [14:26] days right now? [14:27] >> Yeah, great question. So um so dayto-day [14:30] um I suppose there's a lot of hard [14:33] technical work with u by people like [14:35] like like Lassa to kind of build the um [14:38] the whole stack that means we've got you [14:40] know a state-of-the-art reinforcement [14:42] learning stack that allows us to run [14:43] things at massive scale um inhale data [14:47] um at the optimal way be able to kind of [14:49] learn and and and have systems that keep [14:50] learning and learning and learning and [14:52] then on top of that um so we've spoken [14:55] about the superarner um the other asp [14:57] ect which we're building is um we're [14:59] building out environments which are are [15:03] capable of superarners like discovering [15:06] everything like environments in which [15:07] simulated environments in which in which [15:09] a superarner can discover everything [15:11] that we mean by intelligence. Uh we call [15:13] those crucibles. Um so they're [15:15] environments in which it's possible for [15:17] agents to to make incredible discoveries [15:20] in which it's possible to express and [15:23] learn everything that we think of as [15:25] intelligence. And so we have been [15:27] putting a certain amount of effort into [15:28] creating these environments. Um, and so [15:30] it's a combination of of building out [15:32] the superarner, doing the research to [15:34] make superarners incredible and and beat [15:36] the state-of-the-art of what's possible [15:38] with reinforcement learning. Um, both in [15:40] terms of the engineering and the [15:41] research that happens and creating and [15:44] and placing the superarners into these [15:46] crucibles to kind of see the emergence [15:48] of of incredible behaviors. And in terms [15:51] of the grounding, you asked about how to [15:52] ground this in in the day-to-day. So we [15:54] ground it in in in milestones and the [15:56] milestones are kind of analogous to you [15:59] know what happens with the human um [16:02] journey um where we discovered [16:04] incredible things and and so those [16:06] become natural milestones for us to for [16:08] our agents to rediscover the great [16:10] moments that that humans went through in [16:12] our discovery of knowledge. [16:15] >> What do you think the agents will [16:18] rediscover first when that's going to be [16:20] your first moment of oh this is working? [16:22] I imagine we're going to be surprised [16:24] like I think one thing which is [16:25] different about the way we take to doing [16:27] reinforcement learning to a lot of the [16:29] rest of the field is that we're not [16:31] breadcrumb trailing the particular um [16:35] objective to be learned about. A lot of [16:37] the way things people do RL with LLMs at [16:40] the moment is to kind of say oh you know [16:42] the the agent should be learning about [16:44] this problem and then this problem and [16:45] this problem and after kind of following [16:47] that trail we're kind of pulling the [16:48] agent by its nose towards a particular [16:50] desired outcome. We're not taking that [16:52] approach. We're giving the agent some [16:54] simple objectives and asking it to, you [16:56] know, discover things that will will [16:59] surprise us and amaze amaze us. So, you [17:01] know, the the agent is in principle [17:03] capable of discovering all kinds of [17:05] things and we don't necessarily know in [17:07] advance what those will what those [17:09] discoveries will be and it's kind of [17:11] just going to be exciting to to witness [17:13] that journey. So, so I hope to be [17:15] surprised I think is the answer to your [17:17] question. [17:18] >> Interesting. So, so what you're saying [17:19] is instead of explicitly giving the [17:21] agent the goal to go rediscover language [17:24] or, you know, build a calculator or [17:27] build a computer, you're going to give [17:29] it other more open-ended goals and the [17:32] agent may discover that the most [17:34] efficient way to actually or the best [17:35] way to actually accomplish that goal [17:37] maybe to build some of these tools that [17:39] we as humans have already built. Is that [17:41] is that one way of putting it? [17:42] >> That's one way of putting it. I think I [17:43] think another way to say it is people [17:45] often think in RL, oh my god, how do I [17:47] come up with a a reward function that [17:49] kind of perfectly specifies this really [17:51] complicated thing that I want to [17:52] discover like you know how can I how can [17:54] I specify exactly what I mean by by [17:57] science or how can I specify exactly [17:58] what I mean by interesting maths but [18:00] there's an alternative um and the [18:02] alternative is that to give the agent [18:05] something [18:06] you know really straightforward to [18:08] optimize um but in a really complex [18:11] environment where there's a lot to learn [18:12] about. And so it turns out that you know [18:15] if you look at the way the world has [18:16] worked in in nature, the pursuit and [18:19] optimization of a very simply easy to [18:23] describe quantities can often give rise [18:27] to incredibly hard to describe [18:29] complexity and emergence. So, so [18:32] examples of that would be, you know, [18:34] evolution, which kind of just optimiz [18:37] kind of survival of the fittest or um um [18:39] you know, some kind of fitness signal. [18:41] And the pursuit of of of of fitness gave [18:44] rise to all of the complexity that we [18:46] see in in life. um in the human journey [18:49] like the pursuit of relatively simple [18:50] kind of goals that we as humans have [18:53] gave rise to like all of the complex [18:55] behaviors that humans ultimately um [18:57] acquired and all of the knowledge that [18:58] we eventually ended up with. So the kind [19:01] of emergent complexity is much greater [19:03] than the kind of um starting point that [19:05] it was given. So it wasn't like you know [19:07] evolution said look you hey hey hey [19:10] humans you know here's this idea of [19:12] science and you need to kind of get to [19:13] the most complex science that's out [19:15] there. it it it was really something [19:16] much simpler. So, so a way to describe [19:19] this maybe would be, you know, perhaps [19:21] that that humans in the pursuit of our [19:23] goals started coordinating a bit more [19:26] and then and then um in the process of [19:28] coordinating we discovered that it was [19:30] um helpful to start communicating and in [19:32] the process of communicating we started [19:34] trading and then we in in the process of [19:36] trading we discovered that you know [19:37] basic arithmetic was useful and then we [19:39] started to discover that you know tools [19:41] were useful and then we discovered that [19:43] some basic science was required to help [19:45] understand how the tools would work and [19:47] that gave rise to more complex [19:48] technology and understanding the [19:50] technology required more complex maths [19:52] and so forth and so forth. So all in the [19:54] pursuit of relatively simple goals this [19:56] incredible complexity emerged. So if we [19:59] can get that process right you know [20:00] things are going to take off and and and [20:02] lead to you know unbelievable things. [20:04] >> It's fascinating. Are the rewards for [20:06] your agents kind of similar to like [20:09] human biology and evolution then like [20:11] you know try to make it to the next [20:13] generation. um what does a reward [20:16] function look like for you all? [20:18] >> Yeah, I mean we've we've we've been [20:19] experimenting with quite a number of [20:21] different rewards. So, it's not like we [20:22] have one answer to that question. Um but [20:25] I think one thing we're not doing is is [20:27] the kind of, you know, crafting a reward [20:30] that exactly describes what we mean by [20:32] science or exactly what we mean by [20:34] technology or exactly what we mean by [20:36] language. We expect our agents to [20:38] discover things. And if we can't crack [20:39] that, you know, like if we can't crack [20:41] this problem as AI scientists, we're [20:43] never going to have agents that discover [20:45] whole new fields of knowledge that we [20:47] don't even know about. There are vast [20:49] fields that will exist that are out [20:50] there that which we don't even know um [20:54] have the potential to exist at this [20:55] point in time. You know, it's like [20:56] imagine imagine you were back in, you [20:58] know, the days of Newtonian physics and [21:00] and no one even knew that that like [21:02] quantum physics was a thing or or that, [21:04] you know, relativity was a thing. Like [21:06] we should be in that process of [21:08] open-minded discovery where our agents [21:10] are free to kind of figure things out [21:11] for themselves. And I think that's going [21:13] to lead to, you know, much more exciting [21:15] outcomes. [21:16] >> What do these crucibles look like in [21:18] practice? And you know you're [21:20] contrasting it to today when people say [21:22] environments oftentimes it's a it's a [21:25] set of tasks and you know what the agent [21:26] should have done on the task. It sounds [21:28] like your environments are your [21:29] crucibles are very very different. What [21:32] do they look like in practice? [21:34] >> So we are building crucibles which are [21:39] without giving too many details away [21:41] have the capacity for a lot of emergent [21:45] complexity. Like the key thing about [21:47] them is it's like it's not like we've [21:49] kind of handcrafted a whole load of [21:51] puzzles for the agents to to to to work [21:54] with. Um there's just a huge amount of [21:56] emergent complexity there. Some of that [21:57] emergent complexity can come from [21:59] different places. It can come from from [22:01] like physics engines that give rise to [22:03] kind of complexity as you've got you [22:05] know millions of interacting objects [22:06] with with like a complex physics engine. [22:08] you can have millions of interacting [22:11] agents and uh and and the world becomes [22:13] much more complex because it suddenly [22:15] now encount includes the complexity of [22:17] all of those other agents within the the [22:19] world that the agents building. So [22:21] there's ways to build worlds that are [22:23] vastly more complex than the kind of [22:25] standard simulators like you know Atari [22:27] 57 or whatever it is that that that [22:29] people have been working with in in the [22:31] past and which are certainly more [22:32] complex than the kind of standard [22:34] benchmark kind of tasks that people are [22:36] using. Um, and if you get that right, [22:39] you have a world in which there's the [22:42] possibility for discovery by the agent. [22:44] There's the possibility for the agents [22:45] to discover everything from from [22:48] technology to science to mathematics to [22:51] languages to everything else. You know, [22:53] just the same way our humans all the way [22:55] through from from more basic things [22:57] like, you know, motor control or um [22:59] coordination, you know, the the whole [23:01] spectrum is possible. [23:02] >> Like if you look at like a game like Go, [23:04] right, which is like very simple to [23:06] represent. is just like a bit map. Uh [23:07] the simulation is extremely simple. Um [23:10] but you still see this like emerging [23:12] behavior of something complex, right? [23:14] It's like super interesting to us. So [23:16] like of course like go is like way [23:18] simpler than we can imagine doing. Uh [23:21] but at the same time going all the way [23:23] to 100% realistic physics will be like [23:27] very computational expensive, right? So [23:29] we want to like create this range of [23:31] crucibles where we have like simple ones [23:34] that can execute quickly and we can [23:36] iterate on but where we can still see [23:38] like very complex uh behavior. [23:41] >> Interesting. Yeah. It's almost like what [23:42] you choose to represent is is uh is [23:45] tricky right because there's for example [23:48] I can imagine a lot of economics arises [23:49] from game theory um and most of that is [23:53] completely unrelated to physics and so [23:55] there a crucible that represents physics [23:57] is probably not that helpful. But if you [23:59] have information theory represented, who [24:01] which agents know what, then you [24:03] probably do end up discovering [24:04] rediscovering quite a bit of economics. [24:06] Is that is that a fair way to put it? [24:07] Like the the primitives you choose to [24:10] capture in the environment kind of give [24:12] rise to the type of learning that [24:13] emerges. [24:14] >> Yeah. Like the end goal is to have [24:16] something that is grounded in in the [24:18] real world, right? But stepping stones. [24:21] >> I wanted to come back to the Go example [24:23] because I really like this example. [24:25] You'll be surprised to hear that I find [24:26] Go and to be [laughter] a good example. [24:29] Um, but what what I find fascinating, [24:31] which I think is what Lass is um talking [24:34] to here, is is that, you know, Go is [24:36] just like a 19 by9 grid where you're [24:38] trying to kind of maximize the [24:39] territory. And [snorts] if this was a [24:41] single agent game, it would be like the [24:43] most boring game you could imagine. It [24:45] would be, you know, you just place down [24:46] stones on a grid until he fills up the [24:48] whole grid and there would be no [24:50] strategy whatsoever. The moment you put [24:52] a second agent into the environment, you [24:54] get this emergent complexity which is [24:56] suddenly um because the agent contains [24:59] like one agent has to reason about all [25:01] the possible behaviors of the other [25:03] agent. Suddenly it kind of massively [25:05] increases the complexity in the [25:06] environment and and to the best of our [25:08] knowledge you know um certainly all of [25:11] the compute which we sank into it when [25:12] we were working on it there there was [25:14] just like this endless learning curve [25:16] that where with without end you could [25:18] continue to discover new things in this [25:20] game. [25:21] >> Um none of this is to say that Go is [25:23] like the end point. Of course, it's [25:25] still just a board game and quite narrow [25:26] in a certain sense, but it shows that [25:28] very simple environment dynamics can [25:30] lead to kind of infinite like an [25:33] infinite onion of layers of complexity [25:35] that kind of never stops, infinite to [25:38] the capacity of what our computers can [25:40] do, of course. [25:41] >> Do you think that the agents will end up [25:43] rediscovering [25:45] language and will that language look [25:48] like ours or will it look different? I I [25:50] mean my personal belief is that um [25:53] anyone who thinks that human language is [25:55] the end point of of like or the best way [25:59] to to to reason it's it's [26:03] you know that that perspective is a [26:04] little naive. I think that it must be [26:06] the case that there are far more [26:08] sophisticated approaches to to [26:10] reasoning. you know, maybe using um [26:12] parallelizable, rich, complex um [26:16] approaches that neural reasoning or [26:19] whatever it is. These approaches will [26:21] will ultimately be far more expressive [26:25] um and far more capable than approaches [26:27] which are based on on human language. I [26:30] would also say that starting from human [26:32] language as as like the the the approach [26:35] for reasoning is somewhat problematic um [26:38] because it pulls in all of our biases [26:41] about how we think agents should think. [26:44] Um, so for example, a nice way to think [26:46] about this is to imagine um, you know, [26:49] what if we were to step back, you know, [26:52] 500 years or a thousand years or 2,000 [26:54] years and only like were able to use all [26:57] of the language that that humans used at [27:00] that time. Um, so maybe our agents would [27:03] believe that the world was flat or or [27:05] that the diseases were caused by uh the [27:09] four humors or um there would be all [27:11] kinds of, you know, strange beliefs that [27:13] would be embedded in there and be very [27:14] hard to kind of remove because they're [27:16] so fundamental to the kind of language [27:18] and the processes of thinking that that [27:20] that that humans have used. So I do [27:22] think that that you know going beyond [27:24] this um is is a really important goal. [27:28] I'd love to talk more about this concept [27:30] of removing human priors. Uh, and I [27:32] think your experience with alpha [27:35] zero, alpha go zero, uh, [27:37] >> is maybe the the best example that [27:40] anyone can point to of what happens when [27:42] you remove human priors. Maybe if you [27:45] take us back to Alpha, Alph Go and then [27:47] Alpha Zero, was it surprising to you [27:50] that removing human prior would result [27:52] in even smarter intelligence? [27:54] >> I wasn't sure, honestly. So I remember I [27:57] remember I I always like to do things in [28:00] research where I think um you you ask [28:03] research questions which are like on a [28:05] knife edge where it's like 50/50. Um I [28:07] think it's not very interesting to do [28:09] research where you already know the [28:10] answer and you're you're like 95% [28:12] certain this thing's going to work. It's [28:13] kind of like boring honestly to do to do [28:15] those things. So you know I think I [28:18] think research good research is about [28:20] asking questions where you're really not [28:22] sure what the answer is in advance. And [28:23] in this case, I was about 50/50 and I [28:25] thought that maybe there was a good [28:27] chance that it might. Um, but I wasn't [28:29] confident and and it was very exciting [28:31] to see it. Um, I should say like you [28:34] asked about human prize. So, so we are [28:36] making a bet ineffable and the bet we're [28:39] making is that we can go further by [28:41] giving up like on on the prize that [28:44] everyone's relying on today like the [28:46] human data like the the whole internet [28:47] of knowledge that that everyone else is [28:49] is building their systems upon. that's a [28:52] a a an important choice and so we should [28:54] probably talk about like why we're doing [28:56] that and I think the reason for me of [29:00] course it's you know a short-term [29:01] sacrifice like we we give up a certain [29:03] amount of performance um in order to [29:06] access something which is potentially [29:07] far greater and it comes back to this [29:09] like you know what does it mean to build [29:10] a superlearner like to me building a [29:12] superarner again is about building [29:14] something that that has this like [29:16] potential for for full depth and breadth [29:19] so in the depth direction like we need [29:21] to get to something which is capable of [29:23] discovering things which go so far [29:24] beyond human knowledge that that you [29:27] know it goes just far beyond the prior [29:29] and and the more we lean on human data [29:32] the less able we are to actually [29:33] discover the algorithms that are really [29:35] going to take us all the way to [29:36] something that's going to for example [29:37] discover a whole new field of knowledge [29:39] and then in the breadth direction you [29:41] know we need to have algorithms that can [29:42] can be placed down and discover things [29:45] in in [snorts] environments for which [29:47] language and human data that we've [29:49] started with is just not a good fit. I [29:51] mean, you know, there are just many many [29:53] areas, you know, different kinds of [29:54] sensors, for example, um where where [29:57] human like language data or human data [30:00] in general is not a good fit for some of [30:02] the this this description. And so, so I [30:05] think by letting go of this this um the [30:08] human data, we put ourselves in a [30:10] position where we can actually discover [30:11] something which can which can like get [30:14] go far far beyond it. And so if we come [30:16] back to the AlphaGo example, what [30:18] [snorts] we found there was that the [30:21] moment we let go of the human data which [30:23] we'd spent by the way we'd spent about [30:24] two years like building Alph Go somewhat [30:27] similar to the LLM approach where we [30:28] kind of you know we built every level of [30:31] this you know we kind of hand put [30:33] together the next level we kind of um [30:35] put in a lot of human data and then we [30:37] were kind of human in the loop like [30:38] improving it where we would kind of take [30:40] the previous generation and choose how [30:42] we were going to improve it to give the [30:43] next generation and and so we were right [30:45] in the loop of this process. It took us [30:47] two years to get to AlphaGo and beat [30:49] Lisa Doll. When we got Alpha Zero right, [30:51] you know, we gave up on the human data [30:53] and finally focused on what would it be [30:54] like to build an algorithm that could [30:55] discover for itself. Um, so once we got [30:58] it right, that algorithm, it took just a [31:01] few hours, like it took a few hours to [31:03] like outperform the two years of work [31:05] which we' done as with human in the [31:07] loop. Um, so in the in the game of [31:09] chess, it took four hours to beat the [31:10] world champion, the best ever AI that [31:12] had ever been created. in the game of go [31:14] it took um you know like uh I think it [31:17] was a couple of days or something but it [31:19] was incredible progress like and and and [31:22] so you have to let go of the human data [31:25] to allow yourself to discover these [31:27] things that can then shoot past the the [31:29] competition and discover the things that [31:30] just keep learning and learning and [31:31] learning forever. So, so you know it's [31:33] it has to be an active choice and if you [31:36] don't make that active choice you know [31:37] what I see in a lot of the kind of LLM [31:39] companies today is they're basically you [31:41] know building uh semi-aututonomous human [31:44] in the loop kind of learning algorithms [31:46] with thousands of people in the loop of [31:48] the learning progress which is you know [31:50] it works and and a lot of respect for [31:51] how far it gets but you know until we [31:54] step back from that and really discover [31:55] the process that's going to learn for [31:57] itself I don't believe we'll get all the [31:59] way to something which is truly a super [32:00] learner and unleashes the potential that [32:02] that we've talked about so far. [32:05] >> That's very profound. As far as I know [32:07] it that there are humans at ineffable [32:09] and um some of you are creating [32:11] environments or crucibles and what does [32:13] it look like to actually try and make [32:16] sure that human priors aren't kind of [32:18] leaking into what you're doing. [32:20] >> Yeah. Well, so when we design like [32:22] environments um and like rewards [32:24] function and so on like obviously [32:25] there's a human bias in there but we [32:28] don't want to as as Dave was also [32:30] mentioning earlier we don't want to like [32:31] bias it towards uh building a machine [32:34] based on like the funoment architecture [32:36] or something like this right we want to [32:38] like but if you if you're stating like [32:41] um some problem where like a machine [32:44] would be useful it would be very [32:45] interesting to see what type of machine [32:48] it comes up with like what computer like [32:50] machine it comes up with. [32:51] >> So I totally agree with this. I would [32:53] also say that we're not religious about [32:56] this. It's not like a religion that you [32:58] can't have any human prior anywhere. Um [33:00] it's more like a pragmatic position that [33:03] you know until we give up on on like [33:06] relying on the trillion like item data [33:08] sets of of human knowledge that we're [33:11] not going to make the progress that we [33:12] that the field deserves to see. So it's [33:15] more like a actually a practical [33:16] position. And so in practice of course [33:18] there's going to be a certain amount of [33:20] prior which like as Lassa says there'll [33:22] be prior which which apply to many [33:24] different like worlds um as opposed to [33:27] like you know trying to put in these [33:28] very strong prior that exactly encode [33:30] how all the knowledge we know about [33:32] about this one particular problem or [33:34] this one particular world. So kind of [33:36] weak environment agnostic prior seem [33:38] perfectly reasonable, but kind of these [33:40] super strong prior with like trillion [33:42] item data sets. We're trying to avoid [33:45] that because we think pragmatically it's [33:46] going to lead to better outcomes. [33:48] >> Yeah, absolutely. You mentioned that you [33:50] like philosophically that you're drawn [33:52] to research that's more on a knife's [33:55] edge 50/50. I'm curious what your [33:57] confidence levels are for, you know, [34:00] ineffable broad at the broadest level [34:03] like does it will a super learner emerge [34:05] and then I guess at the more pragmatic [34:08] level in terms of the things that you're [34:10] trying to accomplish right now. Are you [34:11] are you 50/50 on these as well? So first [34:14] of all I would say I strongly believe [34:16] that the approach will be feasible at [34:18] some point like we in fact it's the only [34:21] thing for which we have an example of [34:22] like you know the example in our brains [34:25] is the only existence proof of something [34:27] which has discovered like an incredible [34:29] intelligence that's very general and [34:31] works in you know embodied world and [34:33] discovers how to do all kinds of [34:35] different things. So we have this [34:37] incredible example of the of the human [34:39] brain that um we can look at. [34:42] So then you know do I think we can [34:44] achieve it? Well I think like many great [34:47] like moments in science it's about [34:50] timing. So so you know whether this [34:52] succeeds or fails I think is about [34:54] timing like um you when you create a [34:56] startup you want to be at the right [34:58] time. You you don't want to be um too [35:00] early or too late. You know if you're [35:02] too late everyone's done it. You know [35:04] it's not interesting like a thousand [35:05] other people have already done it. And [35:07] if you're too early, um, you know, you [35:09] might be banging on the door, um, but [35:11] not have the the right kind of, you [35:13] know, maybe the field isn't advanced [35:15] enough or or there isn't enough compute [35:16] to be able to succeed yet. So to me, [35:18] it's all about timing. For example, with [35:21] Computer Go, um, you know, I made a [35:23] prediction I think 10 years before um, [35:26] Computer Go was solved and I got it [35:28] right to the year. Um, got it right for [35:31] um, Starcraft, which I also we had a bet [35:34] and made a prediction. um same for alpha [35:36] proof u when we predicting when when the [35:38] IMO was going to fall. [35:40] >> So you're a super forecaster of sorts. [35:42] >> So so far so far I've done very well at [35:45] this um and I do have a very strong [35:47] sense that this is the right time for [35:49] this to fall but that super learners are [35:51] possible like there's been enormous [35:53] progress in the level of technology the [35:56] the ability to accelerate research using [35:58] kind of um LLM coding methods or or or [36:01] auto research methods like like we've [36:03] produced in the past. uh the the ability [36:06] and understanding of scaling laws and [36:08] and the sophistication of technology and [36:10] the stack. There's a lot of things that [36:11] fall into place that I think make this [36:13] the right time. So that's my bet that [36:16] this is the right timing. But yeah, of [36:17] course, look, this is research. [36:19] >> Um research is never guaranteed and um [36:22] and that's what makes it exciting. If [36:24] you get the timing right, you can be the [36:26] first. you can be the ones to change the [36:27] the whole field of AI and um you know um [36:31] that that's that's what it means to to [36:34] take risks and and do something exciting [36:36] >> for ineffable and also all the previous [36:38] things that you correctly forecasted is [36:40] the biggest driver of timing compute [36:42] >> I think it's a very big driver I think [36:44] it's it's an enormous driver I think um [36:46] yeah I mean we're we're in a great [36:49] position with the compute that um that [36:51] we've we've managed to get hold of um [36:54] and [36:55] I think it gives an opportunity to do [36:57] things which are you know orders of [36:59] magnitude [37:00] um more sophisticated than than [37:02] approaches that have tried to kind of do [37:04] self-discovered RL in the past. So I [37:06] think yeah if we compare to I don't know [37:08] something like hideandsek from open AI [37:11] which which um was like an early example [37:13] where where agents discovered um kind of [37:16] some basic tool use um I think you know [37:19] we could probably run experiments that [37:21] like are use 100,000 times more compute [37:24] and so it's kind of amazing that people [37:25] haven't come back and really pushed on [37:27] these types of approaches given the the [37:29] the availability of compute and of [37:31] course hideand-seek I I don't want to [37:33] pretend that that's the end goal. I [37:34] think it's it's very basic but I think [37:36] it just it's suggestive of of of what [37:39] could be done like if we really you know [37:41] so I think compute is a huge driver [37:44] that's not the only one yeah [37:46] >> I think we're in like a very fortunate [37:47] situation that we have been able to [37:49] secure like a very large cluster right [37:51] so and a large [37:53] >> beautiful cluster at that [37:55] >> yeah so so um um so this is like really [37:58] like an opportunity for us right like we [38:00] really have like the cars on our hands [38:02] like to test this out um very few other [38:04] people are doing something similar. So [38:07] yeah, we can we can really test this out [38:09] in a way that we couldn't do at Deep [38:12] Mind for instance. [38:13] >> Absolutely. It seems like one of the [38:17] maybe determinants of how far this can [38:18] go is how far you can push simulation [38:22] and because ultimately these crucibles [38:24] are, you know, they're they're digital, [38:26] right? And I think simulation has [38:28] probably come quite far in the last [38:32] handful of years. Uh but it's not it's [38:34] not perfect, right? Um [38:38] uh we re recently had your your friend [38:40] Rich Sutton on the show and I asked him [38:41] about simulation and he said he was very [38:43] dismissive of simulation. Um I'm curious [38:46] how you guys think about this and and [38:48] Lisa, you've been on both sides of this. [38:49] I think you you obviously did a lot in [38:51] the digital world. Uh but you also built [38:54] Metnet uh which you know touches real [38:57] world atmospheric uh data. How do you [39:00] all think about [39:03] where is the ceiling on simulation and [39:05] what is when is the right time to bring [39:06] in an Apple into the real world? [39:08] >> So I guess midnight was actually like an [39:10] example. So this is like a weather [39:11] forecasting model we did in the Google [39:13] brain Amsterdam office together with [39:16] Google research where like we were doing [39:19] like a machine learning model to predict [39:21] short term like weather forecasts and [39:25] this was like way better than the [39:27] physics based simulations right like so [39:29] the physics simulations were like [39:31] accumulating errors over time so there's [39:33] like a point to that that that physics [39:35] simulation is not perfect I do think [39:38] there's there's a lot of relatively [39:40] simple [39:40] environments that we can build where we [39:42] can see this like emerging behavior of [39:45] let's say like technology and so on [39:47] where we don't need like exact physics. [39:50] Um I actually do think it's interesting [39:52] to see [39:54] how much we can build these days with [39:56] like a very small team in terms of these [39:58] simulations and so on. So there's [40:00] definitely like a ceiling um [40:03] >> is there a ceiling actually? I'm curious [40:04] about this. Yeah. [40:06] >> Well, I guess quantum physics and and [40:08] whatnot, right? But but so there's like [40:09] a ceiling somewhere but um but we can do [40:12] like very interesting things without [40:14] real physics I would say. [40:16] >> So I think the right question to ask is [40:18] can a super intelligence be built in [40:20] simulation. [40:22] So if it is possible for it to be built [40:25] in simulation I believe it will happen [40:28] 10 times or 100 times faster just [40:30] because simulation can be infinitely [40:33] paralyzed. You get perfect [40:35] reproducibility. [40:36] um has a lot of really nice properties. [40:38] It's also the safest place for a super [40:41] intelligence to be built. So it has [40:43] these really nice properties. Then we [40:45] need to ask is it possible? Um and and I [40:49] think for the reasons we discussed [40:51] earlier, there are very clear approaches [40:53] that we can add into the simulation um [40:56] that allow it to be sufficiently complex [40:59] to represent arbitrarily [41:02] interesting things. So for example like [41:04] let's say you wanted to build a computer [41:06] in simulation well all you require is [41:08] that your simulation is is Turing [41:10] complete if you have a Turing complete [41:12] simulation suddenly everything that we [41:14] mean by computation like everything like [41:16] provably everything we mean by [41:18] computation can be expressed completely [41:19] inside simulation and indeed humans have [41:22] built incredibly complex computers or or [41:24] um related things in inside simulated [41:27] worlds. The same is true for various [41:29] other um ideas like you know if you want [41:32] to build an economy like you know money [41:34] can be built out of coins or out of [41:36] paper or out of bits um and as long as [41:40] you have um one of those you know some [41:42] some representation of money everything [41:44] that we mean by economics can be can be [41:46] expressed inside a simulation. The same [41:48] is true for language. If you've got [41:50] multiple agents and they've got some uh [41:52] ability to express a channel of [41:54] communication between the agents, [41:55] suddenly everything that we mean by [41:57] language is expressable in simulation. [41:59] Um mathematics we know is expressible in [42:01] any universe. Science in some sense is [42:04] the study of our world. So will a [42:06] simulation be sufficient to to study um [42:09] what we mean by science? Well, there are [42:11] some aspects of science which are like [42:14] expressible in many things like you know [42:16] energy or entropy or or evolution. Um, [42:19] but also it's it's true that you know [42:22] the agent will discover its own science. [42:24] Like whatever the simulator is, if it's [42:26] got this emergent complexity to its [42:28] environment, the agent will discover its [42:30] own science about whatever that world [42:31] is. It's very like provably possible to [42:36] build environments in which it's [42:38] possible to express and represent [42:40] enormous complexity um of all of these [42:43] things. Well, I agree with Lassa. There [42:44] might be some things which which aren't [42:45] well expressed. you know there will be [42:47] things which can only be expressed in [42:48] the real world like quantum computation [42:50] is a good example. The real point is [42:52] that if it's possible to get all the way [42:54] to a super intelligence in simulation it [42:56] will happen faster and it will happen [42:58] safer and so for those reasons it's [43:00] something that we choose to to put a lot [43:02] of energy into. Um I actually agree with [43:04] Rich Sutton that um that you know I [43:07] think his point was something along the [43:09] lines that you shouldn't expect that the [43:11] things that you learn in simulation to [43:12] like transfer like sim to real into the [43:14] real world because the real world is [43:16] like just like vastly more more complex [43:19] and and in very interesting ways that [43:20] hard to simulate. But I think what we [43:22] can do is we can do all of the science [43:25] to discover the superlearner. Um and um [43:28] and and once we have that superlearner [43:30] we we we will have the answers and we'll [43:32] know how to build something which that [43:33] that same algorithm should be applicable [43:36] to the real world and we can also you [43:37] know make our simulations more and more [43:39] highfidelity towards the real world and [43:41] check that they still work before before [43:43] putting them into the world. [43:44] >> Great answer. Thank you. I'd love to [43:46] talk about alignment and maybe less I [43:49] know this is a politically charged topic [43:50] right now and so I'm less interested I'm [43:52] I'm zero interested in having the [43:54] politics or policy discussion around [43:56] alignment but I am very interested in [43:58] the scientific discussion around it. It [44:00] seems like so much of the misalignment [44:02] that's emerged has arisen from you know [44:06] scaling reinforcement learning to the [44:08] extent that that it's been scaled. Do [44:10] you think that the super learner you [44:13] build will naturally be aligned to human [44:16] interests um and have human values? Is [44:18] it important for for it to have [44:22] alignment with with human values? [44:24] >> So so this is a a big topic. So let me [44:27] you know start by saying a few things. [44:30] So, so where we are today with [44:32] alignment, we are essentially relying [44:34] very heavily on alignment approaches [44:37] that are based on the language that the [44:40] agents use um and prior that are based [44:44] on on human data. Um that has been [44:48] reasonably been been very effective up [44:51] to this point, but we're starting to see [44:52] that that approach is reaching its its [44:54] its limits. And by itself, I think it's [44:57] going to be difficult to see a path [45:00] where as agents become inevitably more [45:02] and more powerful, we can already see [45:04] this happening. As agents are trained [45:05] more and more, they they are kind of um [45:08] going to start to overwhelm their [45:10] priors. They're going to start to learn [45:12] to say one thing and act in a different [45:14] way. Um and it's going to be [45:15] increasingly difficult to to align to [45:18] the things that we uh that we that [45:20] humans uh want them to do. So my belief [45:23] is that we need something more. um and [45:25] that this by itself is not sufficient. [45:27] The most important single thing that we [45:29] can do as a company um is to actually [45:34] understand the the science of super [45:37] intelligence. So that this means to say [45:39] that if we actually can in simulation [45:41] really build a super intelligent agent [45:43] and start to see these super learners [45:45] like go all the way um we can start to [45:48] actually understand like what it takes [45:50] for a super intelligence to actually um [45:52] interact peacefully. Um so we can answer [45:55] really important questions like these [45:57] are fundamentally important questions [45:59] like you know under what conditions like [46:01] what are the what are the conditions [46:03] what are the the the structures what are [46:05] the kind of leverages that we need to [46:06] put into place to make sure that that [46:09] like a a society of super intelligences [46:12] actually interacts peacefully and in [46:14] particular interacts peacefully with [46:15] other lesser intelligences that are also [46:18] going to be in in in their world. And I [46:20] think if we get this right, um, we [46:23] actually can get to some answers like to [46:25] some a real science that will help to [46:27] understand like what the right things to [46:29] do are in the real world. Uh because [46:31] we're hurtling towards that real world [46:33] outcome where there are going to be a [46:34] whole load of different interacting [46:36] super intelligences. Um, and and we [46:39] don't quite know how that's going to [46:40] play out. And I think the faster that we [46:42] can actually get to these answers in [46:44] simulation and publicly share that um, [46:46] the better situation we'll be in. Um so [46:49] that's my strong belief. Um I also [46:51] believe that we can use this to kind of [46:53] gate the actual decision- making of of [46:55] like what we do in in the real world. So [46:57] you know one of our um commitments is [46:59] that we will gate anything we do in the [47:01] real world on having sufficient um [47:04] evidence from this kind of science of [47:06] super intelligence that we are really [47:08] confident that things are going to go [47:10] well. um and we'll only push forward um [47:12] to the next level of what we do in the [47:14] real world once we are um absolutely [47:16] confident that that can lead to a good [47:18] outcome. um so I think this is these are [47:20] two things that I believe we can do. You [47:22] know, we're in this world where where um [47:24] a lot of things are happening very [47:25] quickly. We're hurtling towards an [47:27] outcome with lots of interacting super [47:28] intelligences. I personally believe that [47:30] this will lead to a good outcome for [47:32] humanity, but it's happening very [47:33] quickly. And I think that if we um are [47:36] able to actually understand this and and [47:38] and run ahead in simulation, we'll be in [47:41] a much better place to actually do the [47:43] do the things that ensure that this goes [47:45] really well. [47:45] >> Fascinating. And it seems like one of [47:48] the primary things that current safety [47:50] researchers rely on is being able to [47:52] monitor the agents chains of thought in [47:55] English. And so when they start speaking [47:57] the release, that scares people because [47:58] we can no longer understand and monitor [48:00] it. as well. In your case, you have your [48:02] agents kind of going straight to [48:04] rediscovering language. What do you [48:06] think that means for for monitoring? And [48:09] is or is the point that you know if you [48:13] get the reward right and if you get the [48:16] environments right then you shouldn't [48:18] need to monitor. [48:19] >> Yeah, I think I think monitoring is is [48:22] something which is still really helpful. [48:24] Um, I think that [48:28] like it or not, I believe that the [48:31] approach of monitoring the human [48:32] language that these systems use is going [48:34] to be increasingly difficult to as [48:38] agents get more powerful. And we're [48:39] already seeing that that the monitoring, [48:41] you know, agents are, you know, maybe [48:43] they're producing like a tome of, you [48:46] know, a million words of of of of [48:48] reasoning. um or maybe they're actually [48:51] actively, you know, using subtifuge to [48:53] kind of think in one way and then act in [48:55] another. Um it's increasingly difficult [48:57] to rely 100% on that. Um so I do think [49:00] we need other elements in our armory [49:02] anyway. Um so I think it's actually [49:04] really important that some people are [49:06] looking at how to get this right. I [49:07] think there are ways to to monitor [49:09] agents that don't that don't use [49:11] language. Um I think we can do things [49:13] like um start to for example, you know, [49:17] you can use shadow um actions where you [49:20] start to have actions taken that um [49:24] where you can look at what they would [49:25] actually do if they were allowed to act [49:27] in the world, but you don't actually let [49:28] them take that action yet. You can start [49:30] to have other um super intelligences [49:32] which are acting as the as the monitor. [49:35] So, for example, you might um have the [49:37] idea of like a an incredible scientist [49:39] agent that sort of treads lightly upon [49:41] the earth and then you have another [49:42] super intelligent that's kind of [49:44] monitoring the kind of magnitude of its [49:46] actions and regularizing it to make sure [49:47] it can't do things which are kind of [49:49] affecting the the the world too much. I [49:52] think there are lots of ideas. I would [49:53] say that there's a lot of research to be [49:55] done here. I'm not claiming to have all [49:57] of the answers, but I think it's [49:59] something that that deserves um a um a [50:02] lot of of investigation. And if if [50:05] people aren't looking at this because [50:06] they're only looking at the [50:07] language-based approach, I think we're [50:09] in trouble because I think that [50:10] language-based approach is not enough in [50:12] itself to actually get to where we want. [50:14] I should say in terms of having said in [50:16] trouble, I I am optimistic about where [50:19] the world is going. So, so by trouble, I [50:22] mean that we're not going to have agents [50:23] that do the things that we want them to [50:25] do. I do think we're going to end up in [50:27] a world with lots of complex interacting [50:31] um super intelligent beings and I'm um [50:34] optimistic that that will lead to a kind [50:36] of a balance of of of actors some of [50:39] which are supporting the flourishing of [50:41] humanity and some of which may not be [50:43] but the ones which do support the [50:45] flourishing of humanity will will um end [50:47] up kind of supporting us and protecting [50:49] us against the ones which aren't um you [50:52] know acting on our best interests. So, [50:54] so that's the world I think we're we we [50:56] can work towards. Yeah. [50:58] >> Well, given your super forecasting [51:00] abilities, I'm happy to hear that. [51:03] What do you think about all the [51:04] conversation around pacing the frontier? [51:07] >> I'm actually, yeah, very much supportive [51:11] of the idea that that there should be [51:14] pacing of what happens for the frontier [51:17] of AI in the real world. Um at the same [51:20] time I think we should move as fast as [51:22] possible in simulation. Now the reason [51:25] is that I think in simulation we can [51:28] actually develop a lot of things like [51:30] the science of super intelligence that [51:31] will actually help us to find solutions [51:34] that help things to go well in the real [51:36] world. And I think that kind of work [51:38] should happen as fast as possible to [51:40] make sure that when things really happen [51:42] in the real world um that things have [51:44] the best possible opportunity to go well [51:47] and lead to human flourishing. So I [51:49] think that would be my thought. But [51:51] yeah, in general I'm supportive of of [51:53] the you know suggestions like the kind [51:56] of direction that that people like um [51:59] you know Demis and Dar and so forth are [52:02] suggesting. I think it's um generally a [52:04] good idea. Maybe part of that is that I [52:06] think that you, you know, things are [52:08] moving really quickly and [52:11] I do believe that when we accelerate AI [52:15] and get to super intelligence, it will [52:17] lead to a um a really great future for [52:21] humanity. But I also think that it's [52:23] something that the speed at which things [52:25] are moving is not something that that we [52:28] are capable of adapting to that fast. So [52:31] I think just by pacing a little bit I [52:33] think it gives us the opportunity a for [52:36] our you know governments and our [52:37] organizations and our emotional [52:39] responses to to adapt to the changes [52:42] that are coming and also for for this [52:45] work to to continue a pace at like [52:48] really figuring out in simulation how [52:50] how to you know the real science of of [52:53] doing this doing this right. Hm. So, [52:55] accelerate in simulation so we can [52:57] understand what's happening and you know [52:59] do science around it and then careful in [53:01] the real world. [53:02] >> Yeah, I think that that would be my [53:03] summary. Yeah, [clears throat] [53:04] >> I'd love to transition to talking a [53:06] little bit about compute. Um, this is [53:08] going to be an incredibly compute [53:10] expensive journey and you are pushing [53:14] the limits of what compute can do. I [53:18] think both Google and Nvidia have been [53:20] great partners for you. Can you talk a [53:22] little bit about your cluster? Like [53:24] you're not simply renting spot compute [53:26] from NeoClouds like many other Neolabs. [53:29] You're actually you're doing some really [53:31] serious work on on your cluster and and [53:33] the different other forms of compute [53:34] that you have. Can you just talk about [53:36] your compute setup? [53:37] >> Right. So yeah, as I mentioned earlier [53:40] like like we are very fortunate with uh [53:42] the compute that we have like we are the [53:44] first first customers of the largest [53:46] group cluster on GCP [53:49] um by the end of the year. well actually [53:52] started like one month now. Um so we [53:55] like extremely fortunate in that and I [53:57] actually feel quite bad about like the [53:58] startups that are like getting compute [54:01] now because it's becoming like harder [54:02] and harder. Um but yeah so we have like [54:05] this like ver grouping cluster which is [54:08] like based on GPUs right like Nvidia [54:10] GPUs which is really fortunate for us [54:12] because it means that we can easily do [54:14] simulation [54:15] um and like training and inference right [54:17] they are quite like general chips so [54:19] that's like amazing I think like the re [54:21] rubins specifically um they have this [54:24] architecture where like one rack can [54:26] basically consider like one machine [54:28] there's like a very high bandwidth [54:30] between the nodes and one rack which is [54:32] very important for reinforcement [54:34] learning where you have like really fast [54:35] iteration cycles um compared to LLMs [54:39] that are relatively speaking quite slow. [54:42] So that's quite fortunate for us. I [54:44] think like some of the the things we did [54:46] like 8 10 years ago um at Google where [54:49] we basically had like abundant CPUs at [54:52] Google that doesn't scale anymore. So we [54:55] really need to make sure that everything [54:57] we do is on the GPUs and we don't need [54:59] like millions of CPUs uh beyond these [55:01] GPUs. uh so our simulations can run in [55:04] the GPU inference and training and so [55:05] on. So we have some good architectures [55:07] for that. I think like the LM companies [55:10] now have kind of like the problems that [55:12] we had many years ago where they need [55:14] like sandboxes running on CPUs like [55:16] simulating computers and so on and we're [55:19] trying to not put ourselves in that [55:21] situation. [55:23] >> All the compute is is it roughly funible [55:26] between the different things that you're [55:27] doing? I imagine you're you're building [55:29] crucibles and then you're kind of doing [55:32] these big rollouts where the agents are [55:34] are actually learning. Is that is that [55:36] the rough buckets of what compute is [55:37] being used for? [55:38] >> Yeah, you you could split it up into uh [55:42] training, inference, and simulation. [55:45] >> Is the same compute equally good at [55:47] training, inference, and simulation? It [55:49] >> No. [55:50] >> Okay. So I I think it's pretty clear [55:52] what has happened with GPUs over the [55:54] last many years or like since uh deep [55:57] learning took off is that they've been [55:59] more and more tailored towards training [56:02] right and and maybe in the last few [56:05] years also inference but less about [56:07] simulation right so but we are like [56:11] similarly [clears throat] to the LLM [56:13] companies we also want to be in the case [56:14] where like scaling laws matters like we [56:17] know that uh roughly like if you add [56:19] this amount of compute [56:21] um and experiences then we get a certain [56:24] amount of quality out of that. [56:26] >> So I guess if if GPUs aren't great for [56:28] for scaling simulation then then what's [56:31] the solution [56:32] >> right? So so there are many different [56:33] solutions here right? So there's like [56:35] model based reinforcement learning where [56:36] like the model becomes much more [56:38] expensive compared to to the simulation. [56:40] So there's many different tricks you can [56:42] do. Um there's like various approaches [56:44] for like replay buffers or like reusing [56:46] data many times. So yeah, there are ways [56:49] around that and other than that we do [56:51] spend quite some effort on just making [56:53] the simulations as fast as possible and [56:56] make them use make the best use of the [56:59] GPUs. [57:01] >> Are you seeing scaling laws yet or is it [57:04] too early to say? [57:05] >> So for reinforcement learning like [57:06] scaling laws are like slightly [57:08] different, right? So like scaling laws [57:10] is the component of how much compute you [57:13] put in, how much data you put in uh and [57:15] like the model size. Um well [57:18] reinforcement learning is a little bit [57:19] different because uh the problem is like [57:21] non-stationary so as you progress in the [57:23] problem the data changes like the data [57:26] distribution changes um so they are a [57:29] little bit different it's like a a very [57:31] interesting um research direction [57:34] >> I understand they're different but are [57:35] you seeing some some sort of scaling law [57:37] where more compute in gets you closer to [57:39] achieving the goals or or is it hard to [57:41] say right now? [57:42] >> Yeah but it's it's something we are [57:44] actively pushing towards. Yeah. And [57:47] especially as we like transition from [57:49] like a smaller cluster to the bigger [57:51] cluster, we want to be ready for that. [57:53] >> Yeah. Uh Lei at the beginning D when [57:56] David introduced you, he mentioned that [57:58] you had done some very foundational work [58:01] to compute and algorithms for [58:03] reinforcement learning upon which the [58:06] field really relies on today. Maybe talk [58:08] about some of that work and in [58:09] particular your your work around impala [58:11] and seed. what what are those and how do [58:13] they relate to the work that you're [58:14] doing at ineffable? [58:16] >> Right. So when I joined like the deep [58:18] learning team at deep mind most people [58:21] were running like single machine CPU [58:23] based reinforcement learning. So the [58:25] scale was like quite small. So what I [58:28] did together with colleagues was to make [58:31] an agent reinforcement learning agent [58:33] that really utilized GPUs. I couldn't [58:36] scale like linear with compute. I I [58:38] think like partially because it was very [58:40] efficient. it was very good at scaling. [58:42] And like the third thing was it was it [58:44] was very easy to extend and like modify [58:46] and and do whatever you wanted with it. [58:49] Um contributed to it it being like the [58:52] most popular reinforcement agent at the [58:55] company and I think at one point um most [58:59] RL team used it or like derivatives of [59:02] it. Um so that was quite successful and [59:05] quite a nice project to work on. [59:07] >> [snorts] [59:07] >> Then like later on like we we got TPUs [59:10] right we got like racks of these TPUs so [59:12] we need to scale beyond that. Uh so some [59:15] my work at Google brain after I left [59:17] deep mind to found the Google brain [59:19] Amsterdam office was to scale to like [59:22] racks of TPUs and that product was [59:24] called like seed RL. Um so there was [59:26] like really about like utilizing uh the [59:29] TBUS both for like inference and [59:31] training in the best possible way while [59:33] keeping um the environments as flexible [59:36] as as you wanted to. It could be like [59:38] anything from go to very expensive [59:42] physics simulations [59:43] >> and how's that being used at ineffable? [59:47] >> Yeah. So well we actually implemented [59:50] um a few of those algorithms. Okay. [59:52] >> Also like derivatives of these [59:53] algorithms. So that is like what we are [59:56] using today. We have like depending on [59:58] like the environments we use like [59:59] different algorithms right that like [60:01] pros and cons between them. So yeah so [60:03] we talked about like we have some very [60:05] simple um like fast to to assemblade [60:08] environments and also some that are more [60:10] expensive. So for those who are working [60:13] on large scale RL architectures today [60:16] and haven't read those original papers, [60:18] it's definitely worth a look because I [60:20] think you'll discover that you know many [60:22] of the ideas that that are currently in [60:24] use like kind of trace back to those uh [60:26] yeah early early work that that that [60:29] Lassa kind of did and and and figured [60:31] out which was uh yeah a little bit [60:33] precient I would say. I think it's [60:35] actually kind of fun to see um the [60:38] reinforcement learning space now versus [60:40] like 10 years ago or like let's say like [60:42] eight years ago maybe um like the LLM [60:45] community is also using reinforcement [60:47] learning in particular last two years. [60:50] So and like a lot of the things that [60:52] they're using are you know being [60:53] rediscovered approaches that we used uh [60:56] many years ago. So that's kind of like [60:59] interesting to see with a lot of novelty [61:00] as well on top of it, right? But yeah, [61:03] >> can you talk a little bit about your [61:05] team? How big is the team right now? [61:07] >> We have intentionally kept the team [61:09] relatively small. Um so we're not in a [61:13] hurry to build a a a giant organization. [61:16] I think it's um you know, you can do an [61:18] awful lot um nowadays with a small [61:20] number of incredibly talented people. So [61:23] we have hired 20 people in total. Um, [61:28] and yeah, the people we've hired, it's [61:30] it's about finding some of the [61:32] exceptionally talented people that um, [61:34] but not just it's not just about talent [61:36] for us. It's also about bringing [61:38] together people who are um, kind [61:42] collaborative human beings. That's [61:43] really important to the kind of culture [61:45] we're trying to build and people who [61:47] are, you know, really well aligned with [61:48] the kind of mission that we're [61:49] following. Um, and people that sort of [61:51] fit the the the the needs that we have. [61:55] Um so we try and build the team with the [61:56] right like um set of skills as well. Um [61:59] so all of those four things are really [62:01] important and um and yeah we kind of [62:04] have very high standards and very high [62:06] expectations of the people that come in. [62:08] So uh we'll be very lucky to get um some [62:11] yeah amazing really amazing people. And [62:13] so just to tie it to the compute as [62:15] well, right? Like G like this relatively [62:18] small amount of people so far means that [62:21] we have a lot of compute per person, [62:23] right? And like these days you can you [62:25] can do a lot of experiments just like [62:27] one person. So it's really also to keep [62:30] the compute to person ratio quite high. [62:33] >> So we're not like fighting for resources [62:35] uh from day one. [62:37] >> I think three of the co-founders worked [62:39] together on on Alpha Star. Um, many of [62:41] you, many of you did work together. Are [62:43] you are you assembling the Avengers from [62:45] especially given the recent changes from [62:47] DeepMind? Are you assembling the [62:48] Avengers of folks that you've worked [62:50] with in the past? Have you hired new [62:51] people? And and if so, kind of what are [62:54] the attributes that make somebody new to [62:56] you um likely to be a fit at Ineffable? [62:59] >> So, so it's a a mixture. So, I mean, [63:02] Alpha Star was an exceptional team. um [63:05] it was sort of intentionally assembled [63:07] within deep mind by choosing the kind of [63:10] best of the best to work on that [63:11] project. So in some sense it's not a [63:13] surprise that I think actually four of [63:15] our co-founders come from from that [63:17] team. Um but we are not exclusively um [63:21] drawing from that team or indeed from [63:23] from Deep Mind. I mean the other three [63:25] founders uh all came from um uh very [63:28] successful startups. Um and we [63:35] hire people who are brilliant um but [63:38] also kind and nice people to have around [63:42] um who are going to work together well [63:44] who are aligned with the mission and who [63:46] are you know fit the things that we that [63:48] we need. So, um, but in terms of finding [63:50] those people, we put a lot of effort [63:52] into getting the right people. So, as [63:55] uncomfortable as it is, I've had to say [63:57] no to a lot of people I've worked with [63:58] in the past just because our bar is so [64:01] super high. You know, I've worked I've [64:03] been fortunate to work with like some [64:05] amazing people. And so, you know, I've [64:07] been choosing amongst like a a very [64:11] exceptional crop and bringing in yeah, [64:14] what I consider to be a yeah, superstar [64:16] team. Um and so yeah, it's it's um it's [64:21] it's not easy to get into ineffable, but [64:23] uh the people who we have are wonderful [64:26] human beings and yeah, very very proud [64:28] of the people we brought in. Yeah, [64:30] >> I can see why people are drawn to [64:31] working at Ineffable, especially at a [64:33] time when, you know, the the biggest [64:36] clusters, the biggest labs are kind of, [64:40] you know, they're in they're locked in [64:41] this prisoner's dilemma race and, you [64:46] know, there's a lot of dollars at stake [64:47] and so, you know, decisions get made [64:49] differently. I can imagine for for a set [64:51] of people that especially that really [64:53] really deeply believes in the scientific [64:55] mission of what you want to accomplish, [64:57] this must feel like a breath of fresh [64:58] air. [64:59] >> Yeah, I think that's right. And we've [65:00] tried to establish a a research-led [65:03] culture. So, it's it's not a company [65:06] that's led by, you know, building an [65:08] incremental next product V 17.1 on the [65:14] thing that's come before. It's a company [65:15] that's built on uh researchers pursuing [65:19] um their passion and trying to produce [65:21] the breakthroughs that they believe are [65:23] possible and we center the company [65:25] around that from the beginning and from [65:27] the outset to create like a a a haven [65:30] for research. you know, where where we [65:32] push back on on on product pressure [65:34] because we believe that if we make these [65:36] breakthroughs, it will lead to, you [65:38] know, far bigger products downstream and [65:40] far bigger successes than if we kind of [65:42] start off from day one on like you, oh, [65:44] you know, here's here's the product [65:45] we're doing from the beginning. So, [65:47] yeah, so it's researchled. We're [65:49] creating um I think a really special [65:52] environment and yeah, people are loving [65:55] it. Um I'm loving it. Uh [65:57] >> wonderful, wonderful. Um, I'd love to [66:00] close with some predictions, maybe, [66:03] especially since you're such a super [66:05] forecaster. What's your guess for when [66:09] when you'll have the first [66:11] strong evidence that the approach is [66:13] working and then what's your forecast [66:16] for when when we'll achieve the mission [66:20] that you've set out to achieve this [66:22] first contact with super intelligence? [66:24] >> Yeah, so these are are fair questions. [66:26] Um, so there are going to be some [66:28] reasonable bars of uncertainty on this. [66:30] >> Yeah, but you're being recorded, so [66:31] we're going to hold you to it. [66:32] >> Exactly. [laughter] Yeah. Um so I think [66:34] we will see some extremely strong [66:36] evidence that we're on the right track [66:38] um within a year and I think we will [66:41] achieve something which is a true super [66:43] intelligence like the according to the [66:46] definition I gave earlier of something [66:47] which you know discovers things which [66:49] are far beyond what humans can achieve [66:51] in across you know everything from [66:55] science, mathematics, technology, [66:57] languages etc etc. I originally thought [67:00] within three to five years. Um my [67:02] timelines have accelerated a little bit. [67:04] So I'm gonna say three years because of [67:06] um yeah what can be done with um with [67:08] like um coding agents to accelerate [67:10] things. Um [67:11] >> I think um yeah I think things are going [67:13] to move fast. [67:14] >> Buckle up. It's going to be a journey. [67:16] Wonderful. Uh Lei David, this is a [67:19] wonderful wonderful conversation. Um, [67:21] I'm so inspired by what you're building [67:24] and thank you for taking the time to sit [67:26] down and share with our audience what [67:28] you're building today. [67:29] >> Thank you, Sonia. Pleasure. [67:54] >> [music]