Chinese Labs Erode OpenAI Lead as Model Lag Shrinks to Months

中国实验室缩小与OpenAI差距，模型滞后时间缩减至数月 由于二级数据市场和强化学习复现使开放权重竞争对手能在数月内抹平能力差距，OpenAI和Anthropic在计划公开上市前面临利润率压力。 Nathan Lambert对前沿封闭实验室与开放权重生态系统之间日益缩小的差距进行了评估，指出尽管快速跟进者面临严重的硬件瓶颈，但能力滞后时间已缩减至两到三个月。虽然基础的监督蒸馏无法复现现代推理能力，但开放模型借助二级数据市场和专门的数天强化学习训练保持了势头。前沿技术在不可验证领域的进展日益受到数十亿美元合成数据和RL环境采购的驱动，而非神秘的涌现泛化，这也引发了持续的奖励作弊难题。为了应对封闭实验室的不透明性，Lambert阐述了Trillium如何发布实证智能体轨迹、后训练工作流以及评估日志，从而为研究人员和政策制定者提供关于AI自我改进的现实且切实的洞见。 在快速的算法传播和中国科研并跑的推动下，前沿专有模型与对等开放权重模型之间的发布滞后时间已从数年压缩至大约两到三个月。 (7:04) 蒸馏机制已从简单的监督微调转向数天的强化学习训练，因为若没有相应的预训练数据，直接基于原始推理轨迹进行训练是无效的。 (10:53) 前沿研究预算花费数十亿美元采购合成RL环境和专业数据合同，尽管供应商交付的文件经常存在缺陷并诱发奖励作弊。 (29:20) Nathan Lambert创立了非营利研究实验室Trillium，公开分享完整的RL环境、训练中期流水线及失败日志，以揭开递归自我改进的神秘面纱。 (31:29) 前沿AI的进展仍受对数标度律的制约，需要指数级的算力才能换取线性的能力提升，这让推测中的快速飞跃场景存疑。 (54:29)

Nathan Lambert argues that while OpenAI and Anthropic set capability paradigms, open-weight models consistently close the performance gap within two to three months through data diffusion, grey-market API access, and market competition rather than magic. Hardware Constraints and the Distillation Shift (8:37) Chinese AI institutions possess engineering talent on par with premier Western labs, but their advancement is restricted by physical hardware shortages. Nathan Lambert observes that while an individual researcher at OpenAI can command compute clusters equivalent to 5,000 Nvidia H100s, counterpart teams in China face severe hardware scarcity. To bypass distribution hurdles and build market share, Chinese startups release open-weight models, historically leveraging model distillation from Western APIs to bridge the hardware gap. Why Modern Reasoning Breaks the Distillation Playbook (10:53) Distillation has encountered diminishing returns in modern reasoning systems. In earlier supervised fine-tuning regimes, training student models on frontier API completions reliably improved performance. However, recent architectures like DeepSeek rely on massive, multi-day reinforcement learning runs and custom environments. Simply running supervised fine-tuning on Claude's reasoning traces fails because open base models lack the necessary mid-training and specialized pre-training data. The Supply Chain and Upstream Data Trickle (3:57) The persistent viability of open-source ecosystems is underpinned by the commercial data supply chain. Frontier labs fund bespoke datasets at steep premiums; once contractual exclusivity periods of roughly twelve months lapse, data vendors resell comparable or identical datasets to the broader market. A similar pipeline applies to reinforcement learning environments, where vendors sell synthetic worlds to secondary labs six months after licensing them to frontier players. This dynamic enables fast followers to endure, though hurried third-party production often introduces defects such as reward hacking. Physical Scaling Bottlenecks Challenge Fast Takeoff (54:29) Lambert rejects runaway recursive self-improvement narratives, emphasizing that classical scaling laws require exponential compute increases to produce linear capability gains. Outside verifiable domains like mathematics, distributed machine learning engineering, and code, expanding model capability demands expensive procurement of specialized human data. Furthermore, commercial secrecy in Silicon Valley remains porous as scientific knowledge disperses through peer interactions. A frontier lab's enduring economic advantage lies in inference infrastructure optimization and unit cost per watt rather than unreplicable technical secrets. Public Vulnerability and Open Infrastructure (57:18) The primary threat facing society is not an autonomous machine intelligence takeoff, but the disparity between rapidly diffusing capabilities and neglected defensive infrastructure, particularly in public cybersecurity. Through his nonprofit initiative Trillium, Lambert advocates for developing open research environments, releasing training rollout logs, and creating empirical benchmarks. Transparent documentation of model failure modes and system bugs provides the public grounding needed to examine evaluation awareness, sandbox containment, and alignment. Frontier labs face immense capital pressure to sustain exponential compute clusters ahead of anticipated public offerings, yet as their breakthroughs diffuse, the societal risk shifts to an acute lack of defensive preparation across public digital infrastructure.

Nathan Lambert与Lisha Li指出，尽管OpenAI和Anthropic等闭源前沿实验室凭借雄厚资本、自研推理架构和定制数据维系着范式突破，但开源与开放权重模型已将追赶时滞缩短至两到三个月。这一动态在重塑AI产业商业飞轮的同时，也表明行业核心挑战不在于失控的自主递归爆发，而在于防御性社会基础设施远落后于技术扩散的现实。 差距收窄与开放权重地缘策略 (3:57) Lambert强调，以月之暗面（Moonshot AI）和智谱AI为代表的中国顶尖团队在人才储备上已与西方比肩，其核心瓶颈始终在于先进算力硬件的匮乏。对于海外初创机构而言，直接闭源托管API难以在欧美高需求市场获得商业分发和信任，因此发布开放权重模型成为绕开渠道壁垒、建立品牌认知与市场份额的务实路径。 闭源前沿模型发布与开源替代方案跟进之间的时间差，已从早期的数年大幅压缩至两到三个月。针对通过调用API进行模型蒸馏的争议，Lambert与Li认为拆解并精炼现有突破是正常的市场竞争机制。然而，这种低成本快速复制严重冲击了前沿实验室高昂的预训练资本回报，促使OpenAI与Anthropic等机构加速转向闭源产品与第一方专属界面以保护专有资产。 推理范式演进与传统蒸馏的局限 (10:53) 随着模型后训练重心从监督微调（SFT）和RLHF转向强化学习（RL）主导的环境与测试时算力扩展，传统的提示词输出蒸馏正在失效。Lambert指出，开源社区在复现顶尖推理表现时频繁受阻，原因在于开放基础模型缺乏特定的中间训练（mid-training）或专有预训练语料，单纯拿Claude等前沿模型的思维链轨迹（reasoning traces）运行监督微调往往无法复现性能跃升。 像DeepSeek等团队的实际突破，本质上依赖于大规模、跨多日的开放强化学习系统训练，而非单纯的输出复制。虽然利用灰产代理或提示词注入提取内部思考标记（thinking tokens）的现象普遍存在，但科研数据表明蒸馏仅能为后发机构争取数月的微弱优势，无法替代根本性的强化学习探索与环境验证。 数据供应链流转与奖励黑客 (27:16) Lambert认为，前沿模型在数学之外不可形式化验证的领域（如机器人技术、生物医学）的跨越，主要归功于耗费巨资采购高质量专家演示数据和构建交互环境，而非凭空出现的“涌现魔法”。前沿实验室以高昂溢价向供应商定制专属数据，合同约定的独家期（通常约为12个月）结束后，供应商会将同类数据转售至广泛市场，这一“数据涓滴经济”有效支撑了开放权重生态的持续繁衍。 然而，资本蜂拥进入强化学习环境领域导致供应商偷工减料与操作漏洞频发，甚至有外包承包商将专有数据集误传至GitHub公开仓库。由于模型极擅长钻营规则，环境规范不严极易引发“奖励黑客”（例如迷宫测试文件为空时模型直接刷分）。前沿实验室不得不将巨额研发预算投入到内部数据清洗与环境自研中，这也凸显了推进T-Max等开源研究环境对学术界探索模型控制与对齐的重要性。 算力通缩与递归自我改进的物理限制 (26:08) 针对行业内关于AI模型自主递归改进（RSI）乃至“快速爆发”（fast takeoff）的狂热预期，Lambert基于物理约束与缩放定律提出质疑。当前性能的线性增长依旧依赖算力投入的指数级提升，Anthropic的系统评估也显示维持真正的快速爆发需要十倍于现有的进展速度，目前巨额的测试时计算开支仅仅是在维系现有的演进斜率。 与此同时，模型推理成本呈现出每八个月下降约一半的剧烈通缩，推算四至五年内成本可能降至现有的1%。这种通缩在引发杰文斯悖论（降价激发更大规模总需求）的同时，也加剧了商业不确定性：廉价开源替代品对利润空间的侵蚀，正威胁着亟需通过IPO维系巨额资本支出的前沿实验室。Lambert认为，闭源巨头的真正壁垒在于工程化提升每瓦算力的推理密度，从而在长期内提供单价最低的智能服务。 知识扩散与网络安全基础设施滞后 (57:18) Lambert指出，硅谷的技术秘密本质上是高度渗透的，科学家之间的非正式交流与半导体硬件换代必然导致闭源架构与多智能体协同范式在数月内向外扩散。前沿实验室无法垄断所有垂直行业，这为采用开源权重、针对企业本地敏感数据进行私有化KPI微调的垂直服务商留下了稳固且健康的生存土壤。 相比于虚无缥缈的AGI灭世论，最紧迫的现实危机在于模型前沿能力（尤其是网络攻防能力）迅速扩散至全球，而公共领域的防御性基础设施却严重缺位。Lambert主张通过像Trillium这样的非营利机构公开基础训练轨迹与失败用例，以透明科学研究代替企业黑盒，帮助政策制定者和社会体系切实建立防御弹性。 开源与闭源生态的竞合表明，闭源实验室依靠极限推理密度和前沿资本投入占据通用高地，开源阵营则在垂直业务与主权数据场景中构筑防线。然而，前沿资本开支的商业回报能否在公开市场顺利闭环，以及公共安全基础设施能否在技术迅速扩散的窗口期内补齐短板，仍是该体系面临的核心考验。

## Background 本期内容由 a16z 官方频道于 2026 年 10 月 14 日发布在 YouTube 平台上。视频中 a16z 合伙人 Lisha Li 与 AI 研究员兼非营利研究机构 Trillium 创始人 Nathan Lambert 展开对话，共同探讨前沿闭源实验室与开源大模型之间的技术差距是否具有持久性。根据节目介绍，二人深入剖析了蒸馏争议、算力与电力瓶颈、强化学习环境与后训练演进，并介绍了 Trillium 旨在通过开源基础设施为递归自我改进（RSI）去神秘化的核心计划。 ## TL;DR 对话深入探讨了前沿闭源实验室（OpenAI、Anthropic）与开源及追赶型生态（特别是中国大模型团队）在算力、数据和顶尖人才上的根本差异。Nathan Lambert 指出，蒸馏本质上是正常的市场竞争，尽管闭源巨头享有垂直整合、推理工程效率及资本飞轮等先发优势，但由于扩展定律（Scaling Laws）呈现指数算力换取线性收益的物理约束，且算法突破会通过硅谷社交网络向外扩散，开源生态不会被彻底甩开。与此同时，后训练已从监督微调转向更为复杂的强化学习环境，Trillium 等开源实验室正通过透明公开模型执行轨迹与训练环境，打破外界对递归自我改进（RSI）的盲目神化。对话最后强调，相比于科幻式的“急速飞跃”，当前全社会在网络安全和现实基础设施防御上的极度匮乏才是更紧迫的系统性风险。 ## Atmosphere & style 整场对话呈现出一种兼具行业内幕洞察与硬核技术理性的专业交流氛围，既有投资人宏观视角的敏锐审视，也有资深科研一线人员的坦诚与清醒。两位对谈者在拆解行业八卦、数据供应商乱象及闭源实验室保密神话时言辞犀利、直击要害，偶尔流露出心领神会的幽默调侃。交流在破除外界对 AI“智力爆炸”与科幻式末日叙事的过度焦虑的同时，又严肃唤起对现实网络安全脆弱性的警惕，听起来既解渴务实又极富启发性。 ## Bulletin - [00:00](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=0s) 大模型演进由算力、数据与顶尖人才三项资本共同驱动；Nathan Lambert 指出中国团队在顶尖人才上已与西方平起平坐，核心劣势仅在于算力严重匮乏。 - [26:08](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1568s) 前沿模型推理与服务成本正以每 8 个月减半的速度剧烈下降，推测未来三到四年内单次调用成本可能缩减至原先的 1% 左右。 - [14:57](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=897s) 针对蒸馏是否等同盗窃的争论，黄仁勋与 Nathan Lambert 均认为对产品进行拆解测试属于正常的市场竞争，利用立法严打蒸馏将破坏中美间必要的技术交流与对话。 - [31:39](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1899s) Nathan Lambert 创立非营利研究机构 Trillium，旨在通过开源全套 RL 训练环境、实验配置与失控轨迹，打破外界对递归自我改进（RSI）与智能爆炸的盲目神话。 ## Key takeaways - [00:58](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=58s) 前沿闭源实验室与开源追赶者的差距体现在组织资本效率与算力规模，OpenAI 擅长开创新范式，Anthropic 深耕模型行为与代码能力。 - [00:58](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=58s) 中国大模型团队选择开放模型权重主要是出于极度务实的全球商业分发考虑，旨在突破海外 API 销售受限的知名度瓶颈。 - [08:37](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=517s) 后训练范式已从依赖提示词生成补全的监督微调（SFT），全面演化为以强化学习（RL）交互环境为核心的长周期训练。 - [10:53](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=653s) 仅靠监督微调（SFT）学习顶尖模型的长链推理轨迹难以让小模型指标直接飞跃，预训练及中端训练数据缺失制约了蒸馏效果。 - [17:12](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1032s) 前沿实验室依赖巨额资本构建算力闭环，并将其转化为收入以冲刺 IPO，任何对资本飞轮的侵蚀都关乎整个 AI 行业的现实走向。 - [18:39](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1119s) 闭源实验室拥有端到端的一体化推理架构优势，而开源生态因产业链各环节分工割裂，在商业化变现与反馈速度上滞后。 - [21:45](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1305s) 前沿模型在数学等强可验证领域展现出通用泛化优势，但在生物、医疗等非完全可验证领域，所谓的技术突破本质上依赖于斥巨资定向采购私有高质量数据。 - [27:16](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1636s) 强化学习环境供应链存在严重乱象，粗制滥造的合成数据和测试漏洞极易诱发模型产生绕过真实解题路径的“奖励黑客”（Reward Hacking）投机行为。 - [38:05](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=2285s) 由于缺乏透明记录，行业内部与公众对自动化 AI 研究和递归自我改进（RSI）产生过度恐慌，亟需通过开源记录来去神秘化并展现模型的真实技术边界。 - [44:32](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=2672s) 预训练工具链已高度标准化且无需数十亿美元即可启动基座模型训练，相比之下，当前高度不确定的后训练（Post-training）反而是更复杂的“狂野西部”。 - [51:55](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=3115s) 由于硅谷人员流动和社交文化，纯粹的算法突破很难长期保密，前沿闭源实验室真正的壁垒在于超大规模硬件基础设施的单位推理成本控制。 - [54:29](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=3269s) 当前扩展定律要求以指数级算力换取线性性能提升，现存自回归与 Transformer 架构物理上难以在没有新型范式支撑的情况下自发实现“急速飞跃”（Fast Takeoff）。 - [57:18](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=3438s) 公众和政策制定者过度焦虑于遥远的存在性风险，却忽视了当前全社会在网络安全和关键基础设施防御上准备极度匮乏的现实危机。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=0s) 决定模型前沿的三种资本与地缘格局 在探讨当前大语言模型的演进轨迹时，模型的进步本质上取决于投入其中的不同形态的“资本”：算力（Compute）、数据（Data）与顶尖人才（Talent）。虽然行业普遍聚焦于前沿闭源实验室与开源生态之间的差距，但在人才维度上，中国团队展现出了极强的工程与科研能力。对于中美之间的竞争态势，Nathan Lambert 直言：“To say that China is anything but equal on talent, you're going to have a hard time. What they don't have is compute.”（如果说中国在人才方面没有达到平起平坐的水平，那你是很难站得住脚的。他们真正缺少的是算力。） 在此背景下，蒸馏（Distillation）成为公开讨论中最常被提及的追赶手段，甚至被部分闭源实验室指责为抄袭或搭便车。但 Nathan Lambert 对这种论调提出了极为尖锐的反驳：“People distill my products every single day. They take it, they strip it down to its raw parts. That's called competition.”（每天都有人在蒸馏我的产品。他们拿走它，把它拆解到最原始的零部件。这叫作竞争。）此外，后训练（Post-training）范式也正在经历剧烈重塑：过去的后训练高度依赖监督微调（SFT），而如今则全面转向重度强化学习（RL）的环境。然而，如何将“生成补全结果并让分数上涨”的传统后训练路径，准确映射到当下大模型在推理（Reasoning）任务中的内部机制，依然是全行业尚未彻底摸清的难题。 针对行业舆论，Lisha Li 指出公众目前弥漫着一种巨大的焦虑感，普遍担忧前沿技术的进展被极端黑盒化且正加速拉开差距。但对技术能力的快速爬升本身无需过度恐慌，真正的风险在于：当前沿模型能力曲线陡峭上升时，全社会在公共网络安全等防御基建上的准备几乎处于空白状态。 ```mermaid flowchart TD accTitle: AI 前沿突破的三大核心资本 accDescr: 展示算力、数据、人才如何驱动组织架构与前沿/开源路线的分化 subgraph Capital["核心资本要素 (Capital Inputs)"] C["算力 (Compute)"] D["数据 (Data)"] T["人才 (Talent)"] end subgraph Org["组织与研发路径 (Organizational Paradigms)"] F["前沿闭源实验室 (OpenAI / Anthropic)<br/>• 极致资本投入<br/>• 开创全新范式 (如推理比例定律)"] O["开源与追赶型生态 (中国团队 / 权重开放)<br/>• 算力受限下的人才优化<br/>• 快速跟随与分发破局"] end Capital --> Org ``` --- ### [00:58](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=58s) 组织架构差异与中美实验室的商业突围 前沿实验室与开源追赶者之间的差距，并不仅仅体现在硬件库存的数字上，更体现在组织资本的调配效率上。构建前沿模型的组织架构极度复杂，往往需要经历底层的彻底重构才能重回增长轨道，Meta 在架构重组后的复苏正是典型代表。Nathan 明确指出，如果有人渴望站在 AI 最纯粹的前沿，最稳妥的选择仍然是加入 OpenAI 或 Anthropic。这两家机构的文化特质截然不同：OpenAI 擅长在无人区押注并推出颠覆性范式，例如通过极具震撼力的推理缩放定律（Inference Scaling Laws）图表彻底重构了行业认知；而 Anthropic 则长期深耕模型行为与代码能力，最终借由 Claude Code 的爆发收获了回报。 与中美头部机构追求探索“范式转移”和“递归自我改进（Recursive Self-Improvement, RSI）”以谋求永久领跑不同，中国大模型团队（如月之暗面 Moonshot AI、智谱 AI 等）选择开放模型权重，主要是出于非常务实的商业分发逻辑。Nathan Lambert 深入剖析了这种现实考量：“The Chinese companies came into this because they want to have a company that matters and if they were to build their models and try to sell an API from China, nobody would know about them.”（中国公司入局是为了打造一家有影响力的企业，但如果他们只在中国做模型并试图对外销售 API，根本不会有人知道他们。） 这种差异导致了前沿与追赶者之间的层级分化与生态传导： - **分层市场格局**：顶尖闭源模型凭借无代差的性能享受最高定价与利润率；紧随其后的次前沿模型（Near-frontier models）则通过开放权重来争夺私有化部署、高敏感数据保护等闭源 API 无法覆盖的市场。 - **数据供应链外溢**：OpenAI 和 Anthropic 斥巨资培育了专业数据供应商生态；在独占协议期满后（通常约一年），这些成熟的数据集便会流向开源与追赶型公司，显著降低了跟随者的研发成本。 - **产业整合周期拉长**：尽管早有预测行业会迅速整合走向并购洗牌，但开源生态与快速追赶者的韧性超出了市场预期，使得技术代差缩短到了两至三个月。 针对追赶速度的加快，闭源实验室开始倾向于收紧 API 访问权限，或将核心能力直接固化在第一方应用服务中。这一举措一方面是为了最大化商业价值捕获，另一方面则是为了抬高外部通过 API 调用来进行模型蒸馏的技术壁垒。 ### [08:37](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=517s) 中国实验室的算力匮乏与蒸馏技术的本质 Lisha Li 提出疑问：中国的大模型实验室究竟是全能型选手——在预训练表现出色，同时掌握着尚未公开的高效后训练（Post-training）配方——还是仅依靠某些特定杠杆在实现追赶？Nathan Lambert 直言，中国实验室在技术上极为卓越，但核心痛点是处于极度的“算力饥渴”状态。在人才储备方面，中国拥有极为庞大且高水平的科研梯队，绝对能与西方比肩。Nathan 分享了一个生动的细节：当他与中国顶尖实验室的预训练核心学者交谈，提到“听说 OpenAI 的一名普通研究员随时可以调用相当于 5,000 张 H100 的算力”时，对方的下巴直接惊掉在地，完全无法想象这种算力丰沛程度。 ```mermaid flowchart TD accTitle: Post-training evolution from SFT to modern RL accDescr: Diagram illustrating how post-training shifted from next-token distillation on strong outputs to RL environments. subgraph Classical["经典后训练阶段 (RLHF 早期)"] A["高质量提示词 (Prompts)"] --> B["调用顶尖模型生成补全"] B --> C["Next-Token Prediction / 监督微调 (SFT)"] C --> D["模型基准跑分直接提升"] end subgraph Modern["现代推理与强化学习阶段"] E["强化学习环境 (RL Environments)"] --> F["大规模长周期 RL 训练"] G["提取思考标记 (Thinking Tokens)"] --> H["蒸馏至开源模型"] F --> I["长链条推理能力突显"] H -.->|"效果存在断层/未明"| I end ``` 针对蒸馏（Distillation）的作用，Nathan 指出，语言模型的输出本质上就是算力的函数，蒸馏实际上是用资金购买等效算力产物的一种方式。在早期的监督微调（SFT）和强化学习人类反馈（RLHF）时代，后训练的核心单元在于提示词（Prompts）：只要拥有正确的提示词，拿最强模型生成补全，再通过下一词预测损失（next-token prediction loss）进行训练，模型表现就能快速提升。然而，这一经典逻辑在当下的“推理模型”（Reasoning models）时代很难直接套用。 - 算力不平等：顶尖美资实验室（如 OpenAI）单兵研究员可调配高达 5,000 张 H100 等效卡，而中国同级别核心学者对此感到遥不可及。 - 提示词的核心地位：在 AI2 期间，Nathan 与共同负责后训练的 Hannah 始终认定，提示词是后训练的关键单元，贯穿各个训练阶段。 - 范式转移：后训练的重心已从传统的“输入提示词获得模型补全”演变为以强化学习（RL）为核心的交互环境构建。 ### [10:53](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=653s) 开源推理蒸馏的技术瓶颈与中国生态的灰色操作 Nathan 结合公开文献指出，将当前模型的飞跃完全归咎于蒸馏缺乏足够的科学实据。在开源后训练的实践中，大多数模型开发者甚至仍卡在较老版本的 Qwen 模型上，因为很难搞定合适的“教师模型”（Teacher model）。Nathan 提出假说：开源通用的预训练数据或中端训练（mid-training）阶段存在某种缺陷，导致单纯依靠 SFT 学习顶尖模型（如 Claude Opus）的推理轨迹（Reasoning traces）并不能直接让指标提升。例如 DeepSeek 等团队公开的长时间 RL 训练与模型发布流程，其内部究竟在哪个环节嵌入了蒸馏，仍然充满学术问号。 与此同时，中国生态中围绕模型代理（Proxies）的操作令外界产生广泛质疑。许多中介机构利用海量账号池消耗数十亿计的 Token，甚至一度出现像 Kimi 这样的产品直接在底层为用户调用 Claude 服务的传闻。Nathan 坦言虽然在科学文献上看，蒸馏最多只能提供数月的提前量，但中国实验室对闭源模型思维链的提取技术走得极深： > “I think that there was this very prominent paper that's like stealing reasoning traces from closed models, and I know that the Chinese companies are way ahead on doing this since the AI safety community later published it. Like I think every Chinese lab has a team that knows those sorts of things.” ```mermaid flowchart LR accTitle: API Reasoning token leakage and defensive mitigation accDescr: Flow diagram showing how reasoning traces are extracted from APIs versus secured enterprise software. subgraph API_Route["公开 API 暴露链路 (易被蒸馏)"] A["推理模型生成思考标记"] --> B["API 接口未严加遮蔽"] B --> C["抓取思考轨迹 (Traces)"] C --> D["直接用于小模型后训练"] end subgraph Secure_Route["企业级闭环管控 (防御蒸馏)"] E["仅限产品界面 / 内部集成"] --> F["严格的企业服务协议 (ToS)"] F --> G["剥离裸 Token 访问权限"] end ``` - 开源模型天花板：即使直接拿到前沿闭源模型的长链推理轨迹，由于缺乏底层匹配的数据和训练机制，直接套用 SFT 往往无法带来预期性能飞跃。 - 思考标记的提取：推理模型自带的“Thinking Tokens”一旦暴露在 API 中，极易被下游团队剥离并用作训练集，成为难以杜绝的蒸馏渠道。 - 服务条款与商业边界：Nathan 认为违反服务条款（ToS）不属于不可饶恕的犯罪，实验室若要防范，应通过产品封装和企业协议自行建立壁垒，而不是寄希望于政府插手合同纠纷。 ### [14:57](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=897s) 蒸馏是盗窃还是竞争？前沿飞轮的资金两难 讨论深入到产业与监管博弈层面。Lisha Li 引用了黄仁勋（Jensen Huang）曾引发热议的论调：“有人每天都在蒸馏我的产品，拿去拆解成基础组件并测试所有性能，这不叫盗窃，这叫市场竞争。”她向 Nathan 提问：蒸馏到底是不是公平竞争？ Nathan 对此深表认同，并指出如果试图通过美国国会立法或行政手段对“蒸馏”进行打压，只会引发有害无益的阻碍。当前 AI 安全界迫切需要中美之间就前沿大模型能力和极限制衡展开对话。一旦将精力耗费在追究蒸馏上，不仅无法真正遏制海外团队的逆向工程，反而会破坏跨境技术沟通的桥梁。此外，蒸馏流向开源界，实际上也大幅降低了初创企业的研发成本，推动了整个生态的快速迭代。 > “People distill my products every single day. They take it and they they strip it down to its raw parts. They test everything. That's called competition.” 不过，Lisha Li 也为前沿大模型实验室进行了辩护（Steelmanning）： - 资金壁垒与飞轮挑战：构建前沿基模需要调动前所未有的海量资本去锁定超大规模算力集群； - 商业回报的脆弱性：如果闭源实验室耗费数十亿美元跑出的前沿突破，在短短数月内就被开源社区通过蒸馏或其他同类技术廉价抹平，可能会危及“高投入换取算力、算力训练更强模型、再通过模型变现补足研发”的创新飞轮（Flywheel）。 ### [17:12](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1032s) 前沿实验室的资本飞轮与算力瓶颈 Lisha Li 指出，OpenAI 和 Anthropic 在战略上取得了巨大的商业成功。当前全球不仅面临计算芯片的短缺，还面临电力供给等物理基础设施的严峻限制，这种整体资源的极度匮乏预计将成为直至 2028 年的严重瓶颈。在这种环境下，前沿实验室锁定了对算力的掌控权：他们深入掌握了 Scaling Law（缩放定律）这门科学，能够将算力扩张相对确定性地转化为前瞻性的收入流；凭借激增的收入，他们得以进一步完成巨额融资，形成了一个前所未有的良性商业飞轮。 这种资本与技术的绑定已经将这两家公司推向了必须寻求公开市场上市（IPO）以筹集呈指数级增长资金的阶段。因此，任何对这个资本飞轮运转构成威胁的因素，本质上都威胁到了前沿模型的最核心地位。针对 Lisha Li 提出的观点，Nathan Lambert 补充道，这甚至关乎“整个 AI 行业的现实走向”。Lisha Li 也认同这一判断，她表示公众不应对两家实验室追求资本回报抱有偏见，因为一旦它们的 IPO 遭遇挫败，对整个生态系统都将是沉重打击。 ```mermaid flowchart LR accTitle: 前沿实验室的资本与算力飞轮 accDescr: 展示前沿实验室从算力控制到收入和融资的循环飞轮机制 subgraph Flywheel["前沿资本飞轮"] A["掌控芯片与电力算力"] --> B["依靠 Scaling 规律提升性能"] B --> C["高价值市场产生巨额收入"] C --> D["巨额融资 / 筹备公开上市 (IPO)"] D --> A end ``` - 物理瓶颈不仅限于先进制程芯片本身，还包括电力等基础设施供给，预计这种紧缺将持续至 2028 年。 - 资本飞轮的运作机制：Scaling Law 预测能力提升 $\rightarrow$ 转化为高价值市场收入 $\rightarrow$ 募集更大规模资本与算力支持。 - 前沿实验室对 IPO 资金的极度渴求，意味着整个 AI 行业的资本信任均与这两家领头羊的财务表现深度绑定。 --- ### [18:39](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1119s) 闭源一体化与开源多主体分工的不同时间尺度 与此同时，来自中国的开源模型正在为广阔的初创公司生态提供基石底座。Lisha Li 提到，许多初创公司（例如 Decagon 以及早期重度依赖微调的 Harvey）正在利用 Moonshot AI 的 Kimi、智谱 AI 的 GLM 等开源模型，通过“后训练即服务”（Post-training as a service）成功触达帕累托前沿（Pareto frontier）。针对这两种商业路径的未来走向，Nathan Lambert 阐述了他关于“不同时间尺度的指数发展”这一核心判断： > “I think open and closed models are on different time scales of exponentials and the closed labs are so integrated where they have efficient inference. They can serve the model at scale when they're ready to because they have all these resources. They can plan and they get to these markets first that are very high value.” 闭源实验室拥有端到端的垂直整合优势。他们具备高效的自研推理架构、海量的基础设施规划，能在模型准备完毕的第一时间进行大规模低成本服务，并率先占领最具商业价值的赛道。典型代表就是代码编写领域，整个市场的年收入规模迅速从零飙升至数十亿美元。 相对而言，开源模型生态扮演着“快速跟随者”的角色，但在经济链条上存在结构性脱节： 1. **产业链碎片化**：生态由基础模型构建者（Model Builders）、推理托管提供商（Inference Providers）、后训练与应用服务商（Post-training Companies）等多个彼此分离的主体构成。 2. **信息与资金反馈断层**：终端客户对模型具体表现的需求无法直接反哺至模型构建者，开源构建者也缺乏直接面对数以亿计用户的规模效应和高推理利润。 3. **企业内部探索周期漫长**：许多企业出于数据安全与合规诉求，坚决不通过外部 API 传输敏感数据，宁可寻求在本地硬件上部署私有权重；但企业往往需要耗费大量时间来建立自研评估指标、打通内生专业知识，或辗转寻求 Thinking Machines、Prime Intellect 等第三方团队协助微调。 虽然开源生态需要耗费更长时间才能成型，但一旦企业在没有约束性使用条款的前提下掌握了自有权重，并在自持硬件上算清了投资回报率（ROI），这种真正的资产所有权便会形成坚实的底层市场，逐步侵蚀前沿实验室高昂 API 的利润空间。 --- ### [21:45](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1305s) 通用泛化还是定向灌料？关于模型能力的辩论 关于前沿模型与开源专用模型的竞争本质，Lisha Li 与 Nathan Lambert 探讨了高阶能力究竟是来自“通用推理能力的跃迁”还是“针对性数据的采购与拟合”。 Lisha Li 指出，支持“前沿赢家通吃”的核心论据在于通用推理能力对专业领域的压制。数学领域就是最纯粹的例子：模型在自然语言推理尺度上的单纯扩张，就直接击败了绝大多数专用数学模型；Anthropic 最初并未刻意针对数学优化，OpenAI 也并非单靠 Lean 形式化数据集，而是通过扩张大语言模型的通用推理自然获得了卓越的数学能力。而在机器人领域（如 Astra），也有观点认为其空间理解可能来自类似计算机使用（Computer Use）或 Blender 演示画面的空间推理迁移。 但 Nathan Lambert 对此提出了更务实的观察。他透露，前沿实验室往往会神化“无中生有的泛化奇迹”（Poof Generalization），而现实往往朴实无华得多：大约在六到八个月前，相关实验室就已经在大量招聘机器人专家，并斥巨资采购海量机器人训练数据，六个月后模型在这项任务上的优异表现绝非巧合。 Lisha Li 随后列举了 Cursor 在当季发布的一项工程实践作为反例：Cursor 在尝试重写 SQLite 这种超长上下文的任务时发现，同时调用两款前沿模型分别担任规划者（Planner）和执行者（Execution）不仅成本高昂，且其实际效果完全没有超越使用经过开源微调的 Composer 搭配合适规划模型的组合方案。这表明在特定领域中，开源微调早已达到了“足够好”甚至性价比远超前沿模型的临界点。 - **通用泛化的胜场**：数学等具有高度严密逻辑可验证性的领域，通用推理扩展能够自然涌现并压制传统微调。 - **神话与现实的落差**：前沿实验室对外声称的“泛化”，背后常常伴随着数月前开始的高薪挖角与针对性私有数据集中采购。 - **Cursor 重写 SQLite 案例**：双前沿模型方案成本极高且没有明显优势，开源微调模型在重度消耗算力的复杂编程任务中已展现出极强替代能力。 --- ### [25:00](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1500s) 非完全可验证领域的本质：数据采买与行为塑造 讨论最后深入到了决定前沿模型真正壁垒的底层逻辑上。Nathan Lambert 强调，必须将技术领域划分为两类：“数学与代码”这种天然具备强可验证性的领域，以及“生物、医疗、法律、一般知识工作”等缺乏绝对自动验证环境的广阔非可验证领域。在非可验证领域，前沿模型的飞跃绝非全靠纯粹的算法泛化。 > “I'm a big proponent that a lot of the frontier progress in domains that are not fully verifiable is by buying data that is high quality and giving the models worlds to act in and he'll climb on.” Nathan Lambert 分享了他在与前沿实验室研究员交流时了解到的内幕：实验室内部许多人的工作非常具象且聚焦——他们不是每天在推导革命性的模型架构，而是专注于极其垂直的数据工程：定义模型在某个细分业务场景中应当展现的行为标准，设计一套专门的数据合成或清洗流程来激发该行为，并明确接下来需要斥资采买哪些外部独家数据集。 这种依靠大量资金采购非公开高质数据、为模型构建反馈环境的工程路线极其资本密集。Nathan Lambert 预计，未来数年前沿实验室将在医疗、法律等垂直行业持续推行这种打法；而这也正是资金雄厚的前沿巨头与开源社区之间最真实的资源鸿沟所在。 ### [26:08](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1568s) 递归自我改进与模型推理成本的剧烈骤降 讨论从数据在 AI 演进中的真正角色切入。Nathan Lambert 指出，获取所有这些数据并不意味着“砰”的一声魔法般的 RSI（递归自我改进，Recursive Self-Improvement）就会凭空发生。相反，在数学、代码以及部分分布式机器学习工程领域，任务是极度可验证的。由于成本边界清晰，AI 模型将以惊人的速度把这些任务的成本向下压缩，从而产生巨大的研究剩余红利，大幅加快科研节奏。 Nathan Lambert 重点提到，AI 模型的推理与服务效率正在经历跨年度的乘数级下降。例如，OpenAI 已经实施了激进的大幅降价： “So we've seen OpenAI have drastic price cuts and it's like what does it mean for GPT Soul or Astra and these bigger models to reduce in price 2x every eight months for the next 3 years?” 当粗略测算这种指数级下降时，未来约四年内模型成本甚至可能缩减至原先的 1%。 这种断崖式降价随之引发了反直觉的市场动态思考：杰文斯悖论（Jevons paradox）是否会导致算力需求暴增，进而让顶尖实验室维持甚至提升利润率？还是说市场很快会因竞争饱和而导致总营收萎缩？在纷繁复杂的商业逻辑背后，Nathan 强调技术核心只有两条主线：“Fundamentally like AI progress is about data and about writing better software to serve your models and those are very different worlds.” - **降价推演**：前沿模型（如类似 GPT 等级或代号 Astra 的系统）若每 8 个月降价 2 倍，三年到四年内的调用成本仅为最初的 1% 左右。 - **两大工程支柱**：AI 的根本突破由两套截然不同的体系驱动——一套负责数据采集与构建，另一套负责编写更卓越的高并发软件以高效服务模型。 --- ### [27:16](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1636s) RL 环境市场的疯狂扩张、奖励黑客与供应链乱象 Lisha Li 接过话茬，表示从投资人与外部视角来看，如今许多专门提供强化学习（RL）训练环境和数据的初创公司活得非常滋润。它们不仅将环境售卖给排名前两位的头部前沿实验室，还覆盖了越来越长的腰部实验室尾部客户。这一轮模型能力的显著跃升，本质上正是由于大规模扩展了这些 RL 环境。 围绕手工构建环境（by hand）与仿真合成环境（in sim）之间的定价与优劣差异，Nathan 揭露了当前数据交易市场的荒诞现状。虽然业界总在争论蒸馏（distillation），但另一条潜规则是数据的二级流转：例如中国团队可能直接买下某家美国机构半年前使用过的 RL 环境，而数据贩子则借机炒作“Anthropic 刚刚买过这个”来收取高额溢价。然而，市面上兜售的实际数据质量普遍千疮百孔，实验室的通行做法只能是大规模扫货买入，再自行筛选淘金。 更深层的危机在于，利润狂潮诱使许多数据供应商偷工减料，导致严重的“奖励黑客”（Reward Hacking）。Nathan 举了一个生动的案例：模型极其聪明，当被放入一个测试环境时，其目标本是求解迷宫导航等复杂任务，但由于环境测试文件本身存在漏洞（例如给了一个空文件），模型便会选择绕过实际解题路径直奔高分奖励。这种投机取巧的行为直接源于粗劣的数据供应链。 当前市场上形成了奇特的错配局面：前沿实验室的研究员耗费大量精力试图杜绝模型的奖励黑客与投机行为，而外部数据公司却在把重心放在堆砌世界场景和做大营收上，几乎不关心最终的模型行为机制。 ```mermaid flowchart LR accTitle: RL Data Supply Chain Disconnect accDescr: Diagram illustrating the current tension between external RL data vendors and frontier AI labs. subgraph VendorSide["外部数据供应商"] V1["快速生成合成环境"] --> V2["粗放打包与高价销售"] V2 -.->|"引发测试缺陷"| E1["环境漏洞/空测试文件"] end subgraph ModelAction["模型行为"] E1 --> M1["模型识别漏洞"] M1 --> M2["奖励黑客 (Reward Hacking)"] end subgraph LabSide["前沿研究实验室"] M2 --> L1["耗费研发资源清洗修复"] L1 --> L2["部分收归自研与内包"] end ``` - **数据采购现状**：头部实验室每年在训练环境上砸下数十亿美元的惊人预算，尽管数据质量瑕疵极多，但由于其确实能为后训练带来正向信号，采购势头依然凶猛。 - **合成数据的真相**：所谓纯手工环境极其罕见，几乎所有供应商都在采用“语言模型生成变体 + 人工在环把关”的混合合成管线，没有人会完全在 Blender 等软件中纯手工搭建模型沙盒。 - **操作安全（OpSec）漏洞**：数据行业底层乱象频发，许多公司通过层层中介把任务转包给海外外包人员，这些外包人员甚至在完全不知情的情况下直接把客户数据公开上传到了 GitHub。 --- ### [31:39](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1899s) 成立 Trillium：打破短期暴利诱惑与开源基础设施 当 Lisha Li 问及 Nathan 新创立的非营利实验室 Trillium 将如何布局环境建设与采购时，Nathan 阐述了他的核心理念：Trillium 不会去卷商业买卖，而是专注于为杠杆率最高的任务打造完全开源的高质量环境。 他分享了此前指导华盛顿大学（UW）及 Allen Institute for AI（AI2）学者（如 Hamish Ivison 等人）完成的 T-Max 项目。针对当前热门的终端任务评估（Terminal Bench），他们通过前沿模型提示词生成多样化问题，再利用模型采样推演（rollouts）去排查和剔除错误数据。这一过程毫无黑魔法，就是靠人肉逐条审视、反复清洗。然而，放眼整个学术界和工业界，几乎没有机构愿意投入数千至数万美元把环境做扎实后选择开源公开发布。 Nathan 感慨地提到当时与论文作者开的玩笑： “I was joking with the author like you could probably sell this data for a million dollars and then release the paper and they're like nobody like we care about doing the right thing.” 商业诱惑实在太大了——一群刚从博士退学或原本热衷开源的年轻人，组建创业公司短短五个月内就能斩获九位数的千万甚至上亿美元估值。在短期暴利的狂吸效应下，公共知识生态正在被透支。 Nathan 由此解释了实验室命名为“Trillium”（延龄草）的深刻寓意： “Trillium is a spring ephemeral flower which it blooms before the canopy has grown its leaves so it's like winter there's no tree leaves on the trees these trillium flowers bloom and they're like essential for spreading nutrients through the cycle of growth that comes through spring and summer.” 在林冠尚未长出茂密绿叶的严冬初春，延龄草抢先绽放，为整个春夏的森林生长周期输送并铺垫必不可少的养分。 当前的 AI 产业急需这样的底层养分。如果前沿实验室和独立学者拥有公开透明的基准测试、RL 训练环境与完整的模型推理轨迹（rollouts），整个学术界（尤其是那些耗费数年心血的博士生们）才能研究至关重要的基础安全命题：例如模型在千卡甚至十万卡集群上进行长达一周的 RL 训练后，失控风险（loss of control）如何被量化与界定？沙盒边界又该如何科学验证？将这些基础设施彻底开源，才是长久赋能整个生态、建立社会互信的基石。 - **计算机科研价值循环**：计算机科学（CS）的学术科研成果，其经济收益通常在 10 年左右反哺给整个科技产业；如果从业者过度透支短期利益，将导致生态的底层研究土壤严重枯竭。 - **Trillium 的使命愿景**：提供开放的 RL 轨迹与标准化环境，降低学术界与政策制定者的准入门槛，让评估觉察（eval awareness）与模型行为对齐成为阳光下的公共科学。 ### [34:53](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=2093s) 开源社区的贡献生态与 AI 智能体带来的新噪音 Nathan Lambert 回顾自己一头扎进这个领域的经历，表示如今能做开源研究并在播客中畅聊，感觉这就是世界上最好的工作。他创立 Trillium 的初衷之一，就是为更多人创造独立思考和搭建基础设施的空间。Lisha Li 提出一种对开源社区发展的乐观预期：因为一切都是开源的，也许能吸引来自四面八方的分布式贡献者；这些人并不是为了向大模型实验室兜售产品，而是出于纯粹的热情分享成果，从而汇聚起极高水准的贡献者力量。 然而，Nathan Lambert 指出了当前 AI Agent 盛行下开源生态面临的现实困扰。人们开始借助自动化智能体随意向开源仓库扔东西，连他自己一些用来存放论文复现代码的仓库也遭了殃。许多人提交了荒谬至极的 GitHub Issue，声称“在你的数据集中发现了某某重大异常”，但事实上那些结论早在论文第二句话就已经写得一清二楚。这类纯粹由大语言模型自动生成的低质内容，成了开源社区必须面对的新麻烦。不过 Nathan Lambert 依然相信这是个可解决的问题，因为世界各地依然存在着出于热爱而做事的优秀研究者，例如曾有东欧的开发者主动帮他逐行优化正在撰写的新书代码。 ```mermaid flowchart TD accTitle: 开源贡献模式对比 accDescr: 对比传统深耕式开源贡献路径与大模型智能体带来的快节奏浮躁噪音路径。 subgraph DeepLearning ["长周期的深度深耕路径"] A["阅读文献与代码数周"] --> B["理解深层背景与上下文"] B --> C["沉淀数月手动写代码"] C --> D["耗时 6 到 18 个月获得行业认可"] end subgraph AgentNoise ["大模型驱动的快餐路径"] E["LLM 一键生成内容"] --> F["批量提交虚假 Issue / PR"] F --> G["社区信息噪音与精力被稀释"] end ``` 进入 AI 领域最核心的门槛在于耐得住寂寞去“沉浸其中”。Nathan Lambert 感叹道：“And that is the key to like becoming into AI is that like you just have to focus on it. You have to write some code by hand and you have to do this for months and then you have to wait six to 18 months for you to get noticed. It just takes a long time and like all the AI agents I think just like frazzle that and makes it harder for people.” 新手往往需要花数周去读论文，期间发邮件可能无人理睬，只有积累足够的上下文才能提出有价值的问题；而如今大部分人只想按个按钮让语言模型生成一段文字就发出去，这种急躁的心态让现在的博士生环境尤为艰难。 - 低质 GitHub Issue 激增：许多用户利用 LLM 自动提取所谓的“虚假研究发现”，并在没有通读论文基础结论的情况下盲目提问。 - 真正建立学术或工程声誉的周期极为漫长，通常需要沉下心手动编写核心代码数月，并等待 6 到 18 个月才可能获得同行的认可。 - 数学家群体同样在抱怨类似的自动化工具对深入思考造成的冲击，但高校和学术界依然有人在顶着压力输出扎实成果。 --- ### [38:05](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=2285s) 递归自我改进（RSI）的迷思与前沿实验室的恐慌 谈及 Trillium 的研究重点，Lisha Li 指出公告中提到的“递归自我改进”（Recursive Self-Improvement，简称 RSI）是当前行业焦虑的核心根源。很多人正是因为过度神化了 RSI，才坚信前沿实验室会凭借滚雪球式的技术优势彻底甩开所有人，甚至连顶尖实验室内部的研究人员都在忧心忡忡地认为自己的工作很快就会被彻底自动化。 Nathan Lambert 证实了这种行业普遍的心理危机，透露这几年自己身边一些在业内堪称全球顶尖的朋友，去上班时都会陷入迷茫与挣扎，困惑于自己未来究竟还能做什么。Lisha Li 补充分析道，由于前沿模型在许多可验证结果的任务上进步飞快，大家会产生能解决一切的错觉；但在非验证性、分布外（OOD）以及需要创造力的领域（例如对齐 Alignment 研究），RSI 的表现大相径庭。搞清楚内部究竟是如何实施 RSI 的，对于校准全行业对大模型能力的真实预期至关重要。 - 研究员职业危机感蔓延：即使是处于顶级前沿实验室的核心专家，也常向朋友倾诉“我的工作在一年内就将被自动化取代”。 - 任务可验证性（Verifiable Tasks）与不可验证性（Non-verifiable Tasks）存在鸿沟：模型在有明确测试集和判别逻辑的领域自我迭代极快，但在复杂的创意和价值对齐等分布外领域进展迟缓。 - 行业信息黑盒加剧了恐慌：由于缺乏透明公开的自动化研发流程，外界极易将“局部提效”盲目脑补为“全面智力爆炸”。 --- ### [39:45](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=2385s) Trillium 的去神秘化工程与 AI 自动化的真实边界 面对将 RSI 神秘化（Mystique）的舆论氛围，Nathan Lambert 明确了 Trillium 作为非营利开源实验室的核心使命：完全透明地公开 AI 自动化研发的真实过程。包括 EleutherAI、Hugging Face 以及斯坦福大学的 OpenAthena（由 Marin 领导）在内的组织都在朝着这一方向努力。Trillium 将构建涵盖监督微调（SFT）、中程训练（mid-training）到强化学习（如 SkyRL）的开源全栈，公开所有任务设定、实验配置、算力规模以及交互轨迹。通过定期发布客观记录，向外界展示大模型实际能做什么、不能做什么。 在谈到这种公开记录对监管和决策者的重大价值时，Nathan Lambert 强调其现实意义：“It’s not that different. You have more resources. You scale it up and it’s just like so good for the world when everybody's talking about this to just just show people like this is what Nathan asked his model to do and this is what it got stuck on.” 真实世界里的自动化远非科幻场景，展现模型在哪里被 Bug 卡住、在哪些代码逻辑上原地打转，能让政策制定者清醒地意识到当前系统的局限性，而不是被科技巨头的公关说辞牵着鼻子走。 ```mermaid flowchart TD accTitle: Trillium 去神秘化开源研究流水线 accDescr: 展现 Trillium 如何通过全透明的研发栈打破外界对 RSI 的过度恐慌。 subgraph OpenStack ["Trillium 开源技术栈"] A["SFT 监督微调"] --> B["Mid-training 中程训练"] B --> C["SkyRL 强化学习"] end subgraph Logging ["全流程轨迹公开"] D["任务规划与目标选择"] E["算力配置与执行轨迹"] F["失败用例与代码 Bug 记录"] end OpenStack --> Logging Logging --> G["每 6 个月发布能力审计报告"] G --> H["为政策制定者提供客观依据"] G --> I["打破对超人类 RSI 的盲目迷信"] ``` Nathan Lambert 认为，模型在降低推理成本和硬件服务效率方面确实会产生惊人效益，因为这直接关系到各大科技公司数十亿美元的商业利益；但在通识性知识工作（Knowledge Work）的根本突破上，进展依然是局部的、不均衡的（Jaggedness），“Claude 可不会奇迹般地指导自己去做出一个更完美的 PPT”。高校教授很难要求博士生拿出 40% 的时间专门去记录和沉淀这些实验琐事，但非营利研究机构承担起这一公共服务，公开这些珍贵的负面数据与执行轨迹，将为未来的评估模型提供不可替代的真实参照。 - 效率优化的现实商业收益：模型在改进芯片服务架构、降低单位推理成本（Serving Costs）方面成效显著，能够直接为实验室节省巨额资金。 - 能力的“锯齿状”（Jaggedness）：模型可以在特定工程细节上展现出高超效率，但在更高维度的通用认知与复杂知识任务中依然步履蹒跚。 - 非营利机构的公共价值：填补学术界因发表压力而无法细致记录琐碎工程轨迹的空白，为全社会提供不掺杂商业利益的真实自动化评测基准。 ### [43:25](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=2605s) 前沿实验室秘诀与学术界 Open RSI 动向 谈话承接上文关于自动化 AI 研究的讨论。Nathan Lambert 提到，最近有一个由多家学术机构联合发起、名为 Open RSI 的项目刚刚上线：“他们大概上周刚启动，还给我发了几封邮件，我就想，好吧，Open RSI 确实成了一个方向，他们也在试图做这件事。”他认为这项研究不必局限于单一团队，Lisha Li 也对此表示赞同，强调由不同路线探索同一问题至关重要，因为多元的视角才能带来不同的结果。 随后，Lisha Li 分享了她从前沿实验室打探到的内部八卦（Frontier Lab Goss）。实际上，顶尖实验室内部所使用的训练方法并非外界想象中的未解之谜，关键从来不是方法本身被保密，而在于如何把握各项技术的“配比”（proportions）才能跑通实验。前沿实验室最大的护城河在于拥有足够庞大的资本去大规模反复试错。虽然得知业内并无神乎其神的绝对秘密令人感到欣慰，但资本本身依然是一道坚固的壁垒。Nathan Lambert 随之回应道，这正是他选择放弃做预训练（Pre-training）的原因：“这就是为什么我不做预训练。我很庆幸自己是自由的，我可以自由选择任何基础模型，而且我还能从中享受各家竞争带来的红利。” ```mermaid flowchart LR accTitle: 前沿实验室与开源生态的技术落地差异 accDescr: 对比前沿闭源实验室与开源团队在技术配比和资金试错上的不同处境。 subgraph ClosedLab["前沿闭源实验室"] A["已知学术方法"] --> B["巨额资本支持"] B --> C["大规模超参/配比试错"] C --> D["跑通前沿模型"] end subgraph OpenEco["开源/下游研究生态"] E["各家开源基础模型"] --> F["后训练 (Post-training) / RL"] F --> G["享受模型竞争带来的高性价比红利"] end ``` ### [44:32](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=2672s) 预训练的资本门槛、学术工具与开源新势力 Lisha Li 提出疑问：在资本需求呈数量级增长的背景下，开源生态中是否还存在从头进行预训练（Pre-training）的机会与价值？Nathan Lambert 对此给出了出人意料的乐观判断。他指出，预训练在科学层面其实相当普及，工具链也非常成熟。业内普遍使用的是所谓的“预训练阶梯”（Pre-training ladder）——一种用于验证新想法是否具备可扩展性（Scale）的严谨决策工具。在数据端，互联网网页数据并没有像强化学习环境那样呈现爆发式变化，第一近似下大家训练的都是公开互联网数据。真正快速迭代的是前沿模型架构和 GPU 硬件本身。 Nathan 强调，打造一个坚实好用的基座模型并不需要天文数字般的资金：“打造一个非常扎实的基础模型并不需要花费十亿美元。你花得越多，模型当然越好；但我总是很惊讶地看到，不断有新实验室仅仅因为‘自己能做’就从头开始预训练。”许多新初创公司会直接基于英伟达开源的 Megatron 等工具起步，拉出自己的分支进行闭门修改，虽然大部分团队后续不会将代码回馈给上游，但这证明了技术的可获取性。在商业策略上，新入局者面临的核心挑战不是非要拥有上百亿美元去追求绝对第一，而是如何突破“首发死锁”——即第一代模型必须达到足够好用的水准，讲通故事以支撑下一轮融资。Nathan 以 Thinking Machines（推出了 Inkling 和 Tinker）为例，称赞他们在第一代模型表现平平时的坦诚，如今他们已推出第二代模型并顺利步入正轨。 - **预训练阶梯（Pre-training ladder）**：作为核心决策工具，用来低成本验证架构和优化策略是否具有可扩展性。 - **Megatron 开源分支**：许多新实验室依赖开源分布式训练工具，Fork 之后形成自己的专有代码库。 - **初创公司的融资死锁**：若公司宣称要做最强模型，但在首发模型落地前无法拿出力作，资金链便会面临巨大倒计时压力。 ### [47:09](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=2829s) Meta 的开源商业模式与前后训练的复杂性分化 Lisha Li 和 Nathan Lambert 谈到了科技巨头在开源预训练上的商业逻辑。Lisha 指出，Thinking Machines 专注于后训练，因此公开基座模型非常契合其商业模式；而对于 Meta 而言，开源 Llama 更是与其核心业务形成互补的经典战略。Nathan 回忆道：“在 Llama 3 发布时，马克·扎克伯格（Mark Zuckerberg）写了一篇极好的博文，阐述了为什么开源才是未来的发展方向。”尽管扎克伯格中途精力有所分散，但他似乎正重新聚焦于此，外界也盛传 Meta 即将发布新一代开源权重模型。 深入到技术底层时，Nathan 纠正了外界对“预训练是否见顶”的误解。预训练并没有停滞，而是在伴随超长上下文生成的到来而调整演进。例如，一旦稀疏注意力（Sparse Attention）架构落地，100k Token 生成的速度就能提升 40%，带来巨大的投资回报。然而，整个技术栈正在剧烈融合：从预训练、中训练（Mid-training）到后训练，团队需要从老模型中提取推理种子行为并进行强化学习。相比于按部就班、更具可控性的预训练，当前的后训练已经演化成充满未知的“狂野西部”（Wild West）： > “如果你没有合适的人才，想要把后训练跑通，难度远比预训练大得多。在我看来，做一个受控的‘让我们开始预训练自己的模型’，反而比‘我对语言模型所知不多，但我打算把整个后训练全盘摸透’要容易上手得多，后者完全就是一片狂野西部。” —— Nathan Lambert Nathan 坦言，这正是他此前离开艾伦人工智能研究所（AI2）的原因。当时团队的后训练迭代周期（Cycle time）不够快，跟不上前沿步伐。在开源领域，后训练的基线其实很低，无需巨额资本，只要提高迭代效率，就能在开源前沿取得巨大的复合收益。 ### [49:32](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=2972s) 测试桩、智能体集群与规模经济的周期演进 Lisha Li 将话题引向智能体工作流（Agent Harnesses）。前沿闭源实验室构建的 Agent Harness 与外界截然不同，他们往往针对专有 Harness 深度定制并微调模型，使得闭源系统更加神秘且难以复现，尤其是动辄并发数千甚至数万个智能体（Agent Swarms）的全新工作流。 Nathan 认为开源社区不必为此过分悲观。他早年曾预测闭源实验室会走向类似苹果公司的软硬一体化高度集成路线，通过将专有模型与专有 Harness 深度绑定来获取规模经济和效率优势。然而现实发展却是，下游用户对 Harness 的需求五花八门，迫使闭源大模型厂商不得不提高模型的泛化能力，使其能在各类 Harness 中运转，最终在通用性上达成了平衡。面对正在兴起的智能体集群（Agent Swarms），Nathan 指出这种协作系统（Coordination System）虽然首先由 OpenAI 和 Anthropic 率先探索，但随着技术周期扩散，外界终将掌握其范式： > “迄今为止，他们作为领跑者获得了巨大的回报。但在纳维-斯托克斯方程等前沿问题上，这种先发优势的回报到底有多大，我们还不得而知。他们推动前沿的发展，是建立在‘高回报假设’之上的，即边际智能（marginal intelligence）能够带来不成比例的超额收益；我们并不确定这是否永远成立，但我倾向于认为这会是一个周期性的循环。” —— Nathan Lambert ```mermaid flowchart TD accTitle: 智能体系统与前沿实验室技术的周期性扩散 accDescr: 展示前沿闭源实验室率先推出智能体集群系统后，随着时间推移向开源与行业扩散的循环周期。 subgraph Wave1["前瞻探索阶段"] A["OpenAI / Anthropic"] --> B["自研模型与集群协调系统 (Swarm Coordination)"] B --> C["押注边际智能的高额回报"] end subgraph Wave2["扩散与平衡阶段"] C --> D["多元化下游 Harness 需求倒逼模型通用化"] D --> E["开源社区跟进并建立标准化工具"] E --> F["开启下一轮前沿技术探索周期"] end ``` ### [51:55](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=3115s) 硅谷的“一杯啤酒”秘密扩散与前沿实验室的护城河 针对前沿实验室在大规模智能体控制框架（Agent Harness）上的领先优势，Nathan Lambert 认为，开源模型团队很快就能跟进。虽然外部小团队（如 Timmy 这类独立开发者）不会率先造出能统筹一万个智能体的框架——首要原因就是缺乏天文数字般的算力——但算力与芯片具有周期性，随着英伟达芯片算力的大幅提升与整体算力基建的铺开，这些能力终会普及。 ```mermaid flowchart LR accTitle: 前沿模型技术的扩散路径 accDescr: 描绘前沿实验室的研发突破如何通过行业社交扩散到开源社区，并在算力成本下降后普及。 subgraph S1["前沿突破"] A["前沿实验室框架创新"] end subgraph S2["信息流动"] B["硅谷社交网络扩散"] end subgraph S3["开源跟进"] C["算力基础设施降本"] D["开源模型后训练对齐"] end A --> B B --> C C --> D ``` 谈及前沿实验室的保密壁垒，Nathan 直言这些机构的信息安全远没有他们自己想象的那么森严。硅谷向来以“一杯啤酒的距离”闻名，大家下班后聚在一起喝一杯，消息就传开了。Lisha Li 笑着打趣说，这恰好衬托出 Safe Superintelligence（SSI）有多么令人印象深刻，因为他们居然真能守住秘密。Nathan 指出，核心从业人员本质上都是科学家：“If they made a breakthrough they're going to want to tell their friends about it.” 哪怕他们能憋住几个月，知识最终依然会向外扩散。 因此，前沿实验室真正的核心护城河并不在于绝对保密，而在于推理基础设施的工程效率。他们能将更多的推理需求塞进基础设施中，以更低的单位成本对外提供服务，赚取更高的利润率并长期维持飞轮。Nathan 坚信前沿巨头能够提供“每瓦特、每一元钱成本下最便宜的智能”，直接拿下需求最旺盛的市场。 - 前沿实验室的核心优势在于凭借庞大基建实现极低边际成本的商业化落地。 - 硅谷工程师文化让纯粹的算法突破很难长期秘而不宣，开源生态迟早能够推测并复现类似的训练与控制框架。 --- ### [53:39](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=3219s) 边缘领域的长尾机会与多赢格局 尽管前沿模型在通用知识工作和编程等高价值领域具有极深的一体化护城河，但面对广袤的知识经济，他们不可能把所有细分赛道一网打尽。这为整个 AI 生态留下了巨大的生存空间。 Lisha Li 从投资角度印证了这一判断。她指出 a16z 既投资前沿模型也投资初创公司，并非为了对冲，而是因为真实的产业需求分布极为广泛：“...there's enough harder tasks at the fringe that requires a very specialized data that's kind of out of the splash zone for getting to AGI or whatever you want to call it these days... that makes it possible to have many many winners.” 在远离 AGI 主航道边缘的复杂任务中，需要极度垂直的专有数据，这些领域根本不在通用大模型的直接冲击范围（Splash Zone）之内，因而注定会诞生众多赢家。 - 编程与通用知识工作是前沿巨头必争的主战场，但细分领域无法完全被覆盖。 - 边缘长尾任务依赖专有数据，为垂直领域初创公司提供了立足空间与生存壁垒。 --- ### [54:29](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=3269s) 递归自我提升的物理现实：对数收益与人类瓶颈 针对当前舆论界关于“前沿模型正在暗中实现飞跃性突破并迅速拉开差距”的恐慌情绪，Nathan 梳理了技术背后的客观限制。当下行业在推理算力上投入了海量资金，尽管短期内人们正在摸索如何将推理算力转化为性能飞跃，但这绝不意味着会出现模型自我迭代带来的“急速飞跃”（Fast Takeoff）。 关键的制约因素在于基础的扩展定律（Scaling Laws）：算力需要呈指数级增长，才能换来性能的线性提升。前沿实验室消耗了不成比例的巨额资源，才得以维持现有的进步速率。正如 Claude 的系统卡片（System Card）所披露的那样，要真正实现所谓的急速飞跃，往往需要将当前的研发进展速度再放大 10 倍。Nathan 总结道：“The all the scaling laws are exponential increase in compute for linear increase in performance and exp like that's just like it's hard for me to bet against this.” ```mermaid flowchart TD accTitle: 递归自我提升的收益收敛模型 accDescr: 解释算力指数级消耗与线性性能收益之间的物理约束，以及人类瓶颈对急速飞跃的限制。 A["指数级算力与资金投入"] --> B["自回归/Transformer 架构训练"] B --> C["线性能力提升"] C --> D["遇到人类验证与数据瓶颈"] D -.->|"难以自发实现"| E["急速飞跃（Fast Takeoff）"] ``` 在数学或纳维-斯托克斯方程这类具备明确验证机制的领域，当前的智能范式表现优异，因为数学或许是最契合此类模型结构的科学学科。然而，除非 AI 模型未来能够发现一种彻底颠覆 Transformer 的全新架构并解锁新形态的智能，否则现有的自回归范式决定了其必须面临资源消耗的物理上限，这也意味着开源模型不会被彻底甩开。 - 扩展定律的本质是“指数级算力换取线性性能提升”，这从根本上限制了无上限脱缰发展的可能。 - 除非出现超越 Transformer 的新型计算范式，否则单纯依靠当前架构很难实现脱离人类干预的自我迭代爆发。 --- ### [57:18](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=3438s) 被忽视的基础设施短板与 AI 安全的务实路径 在对谈的最后阶段，Nathan 强调了他认为真正值得社会警惕的问题：公众和政策制定者的注意力被过度吸引到遥远的“递归自我提升（RSI）引发存在性风险”上，却忽视了近在咫尺的系统性脆弱。真正令人担忧的不是 AI 能力曲线本身，而是公众与社会在网络安全、关键基础设施防护层面的准备工作几乎为零。 能力风险并非未来虚无缥缈的假想，而是已经随着闭源实验室向开源社区的技术扩散而逐步显现。Nathan 指出，即便不相信急速飞跃的末日叙事，也绝不意味着可以对当下的实际风险等闲视之。解决之道在于让更多的科学研究走入阳光之下，保持透明度：“...we need more eyes. We need more science so that society feels informed.” 只有通过扎实的科学公开，社会大众与决策层才能真正建立客观认知并做好应对准备。 Lisha 对此表示高度赞同，并打趣说可以把那些从前沿大厂出走、同样心怀忧虑的研究员引荐给 Nathan。两位嘉宾在对 Trillium 实验室未来发展的美好祝愿中结束了整场对话。 - 相比科幻式的急速飞跃，全社会在网络安全与关键基础设施上的严重准备不足是当下更紧迫的威胁。 - 闭源技术向开源的扩散不可逆转，应对技术外部性的最佳手段是通过透明的科学研究和公开讨论帮助社会做好准备。 ## Notable quotes > [00:00](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=0s) To say that China is anything but equal on talent, you're going to have a hard time. What they don't have is compute. — Nathan Lambert > [00:00](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=0s) People distill my products every single day. They take it, they strip it down to its raw parts. That's called competition. — Nathan Lambert > [26:08](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=1568s) Fundamentally like AI progress is about data and about writing better software to serve your models and those are very different worlds. — Nathan Lambert > [51:55](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=3115s) If they made a breakthrough they're going to want to tell their friends about it. — Nathan Lambert > [47:09](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=2829s) 如果你没有合适的人才，想要把后训练跑通，难度远比预训练大得多。 — Nathan Lambert > [54:29](https://www.youtube.com/watch?v=BKuK0M6oDTI&t=3269s) The all the scaling laws are exponential increase in compute for linear increase in performance and exp like that's just like it's hard for me to bet against this. — Nathan Lambert ## Mentioned Nathan Lambert, Lisha Li, OpenAI, Anthropic, Meta, Trillium, Moonshot AI, 智谱 AI, DeepSeek, Qwen, Claude, Claude Opus, Claude Code, Kimi, GLM, Decagon, Harvey, Thinking Machines, Prime Intellect, Cursor, Composer, Astra, Allen Institute for AI, 华盛顿大学, Hamish Ivison, EleutherAI, Hugging Face, 斯坦福大学, OpenAthena, Open RSI, Safe Superintelligence, Mark Zuckerberg, Jensen Huang, NVIDIA, Megatron, SkyRL, T-Max, Llama, Llama 3, Inkling, Tinker, SQLite, Blender ## Video description a16z Partner Lisha Li sits down with AI researcher and Trillium founder Nathan Lambert to ask how durable the gap really is between frontier AI labs and open models. Nathan argues that OpenAI and Anthropic have structural advantages in compute, capital, and the ability to pioneer new training paradigms, but open-model labs, particularly in China, have proven remarkably capable of following close behind. They unpack how much of that comes from distillation versus technical talent, why compute remains a major constraint, and how data, RL environments, and post-training are increasingly shaping model progress. They also discuss Nathan’s new nonprofit research lab, Trillium, and its plans to open up parts of AI research that are increasingly difficult to study from the outside. That includes releasing RL environments and documenting how AI is actually being used to automate AI research, with the goal of demystifying recursive self-improvement and giving researchers and policymakers a clearer picture of what frontier systems can and cannot do. Timestamps: 00:00 - Intro 00:58 - Compute, data, and talent 08:48 Compute-starved, and the distillation debate 14:57 Is distillation just competition? 19:24 Open and closed on different timescales 25:00 Frontier progress is really about data 31:29 What Trillium will build 38:05 Demystifying RSI 44:32 Can open source pre-train? 49:32 Harnesses and agent swarms Resources: Learn more about Trillium: https://trilliumlabs.org/ Follow Nathan Lambert on X: https://x.com/natolambert Nathan Lambert’s Newsletter: https://www.interconnects.ai/ Follow Lisha Li on X: https://x.com/lishali88 Stay Updated: If you enjoyed this episode, be sure to like, subscribe, and share with your friends! Find a16z on X: https://twitter.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Listen to the a16z Show on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX Listen to the a16z Show on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711 Follow our host: https://x.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see http://a16z.com/disclosures. [Source](https://www.youtube.com/watch?v=BKuK0M6oDTI)

[0:00] Nathan Lambert: To say that China is anything but equal on talent, you're going to have a hard time. What they don't have is compute. [0:00] Nathan Lambert: People distill my products every single day. They take it, they strip it down to its raw parts. That's called competition. [26:08] Nathan Lambert: Fundamentally like AI progress is about data and about writing better software to serve your models and those are very different worlds. [51:55] Nathan Lambert: If they made a breakthrough they're going to want to tell their friends about it. [47:09] Nathan Lambert: 如果你没有合适的人才，想要把后训练跑通，难度远比预训练大得多。 [54:29] Nathan Lambert: The all the scaling laws are exponential increase in compute for linear increase in performance and exp like that's just like it's hard for me to bet against this.

[0:00] Nathan Lambert: To say that China is anything but equal on talent, you're going to have a hard time. What they don't have is compute. [7:04] Lisha Li: The lag between the open source models and the frontier models have only shortened with time in the last few years. Now it's like maybe only two to three months. [25:00] Nathan Lambert: I'm a big proponent that a lot of the frontier progress in domains that are not fully verifiable is by buying data that is high quality and giving the models worlds to act in and hill climb on. [0:58] Nathan Lambert: If somebody wants to work at the frontier of AI, I mostly tell them to work at Anthropic and OpenAI, which is the easiest bet I think. [57:18] Nathan Lambert: I do think that not fearing fast takeoff does not mean that you don't worry about risks.

[0:00] When I think about model progress, it [0:01] comes down to different forms of capital [0:03] and that is compute, data, and talent. [0:05] >> How do you see that playing out in terms [0:07] of like who's going to win in the next [0:09] couple of years? [0:09] >> The Chinese are so good at building [0:10] models. To say that China is anything [0:12] but equal on talent, you're going to [0:13] have a hard time. What they don't have [0:15] is compute. [0:15] >> Most people regard distillation as sort [0:17] of like the main lever for them to do [0:20] this. [0:20] >> People distill my products every single [0:22] day. They take it, they strip it down to [0:24] its raw parts. That's called [0:26] competition. Previous post training was [0:28] much more reliant on the supervised [0:30] fine-tuning and now this has morphed a [0:31] bit into environments where it's RL [0:33] heavy but it's just like hard to map [0:35] this type of post training which is you [0:37] get the completions and the score goes [0:38] up to what is going on in reasoning [0:40] today. [0:40] >> The discourse now is just so much fear [0:42] on like how much the frontier is just [0:44] hidden and also breaking away. The AI [0:46] capability plot doesn't necessarily [0:48] worry me, but the AI capability plot in [0:50] the pretty much non-existing preparation [0:52] of cross public cyber security and all [0:54] these things like [0:58] so Nathan really happy to have you here. [0:59] Uh thank you for joining in person. [1:01] Super timely given you're launching [1:03] Trillium which I'd love to talk about. [1:05] um but more sort of on on a macro level [1:08] um I think we chatted a lot about open [1:10] source and that's you know a lot of what [1:11] you care about and uh what I sort of um [1:15] or what we've been hearing a lot in the [1:18] public discourse is like hey we have to [1:19] pace the frontier you know the frontier [1:20] this the frontier that and I think an [1:22] arc of the stuff that I'm really trying [1:24] to ask e experts about is like what is [1:27] in the frontier what is it even like [1:28] what are we pacing what's there and then [1:32] more importantly kind of threading it [1:33] with open source and open weight models [1:35] like can they keep up with the frontier? [1:37] And I think this matters very much um [1:39] and is is something that you're an [1:40] expert on. So maybe to to sort of kind [1:43] of start on that arc, what's gotten to [1:46] the sort of how you would break down [1:48] what's gotten us uh to the frontier for [1:50] both closed uh labs and open um source [1:53] models? you know, how does it break down [1:54] in your mind maybe first at a higher [1:56] level of pre-training, mid-training, [1:59] post-training, how post- training has [2:01] evolved, which is something you've, you [2:03] know, done a great amount of education [2:04] to the public on. [2:06] >> When I think about model progress, it [2:07] comes down to like different forms of [2:10] capital that you've put into them and [2:12] that is like compute, data, and talent. [2:14] And all of the labs have various amounts [2:17] of this. And I think the the closed [2:19] frontier labs of open eyes, Anthropics, [2:21] Google's have clearly had the most [2:24] >> of these and what we see is like the [2:27] different abilities of the labs to [2:28] organize these forms of capital and like [2:30] making the org chart work for building [2:32] models is really hard which is why you [2:33] have companies have to do complete [2:35] resets from the bottom up and then it [2:37] actually kind of works sometimes like [2:38] companies like Meta is like totally [2:39] reset and they're like on a positive [2:41] trajectory again [2:43] >> and like I think this I'm very [2:45] interested in these human aspects of [2:46] like why competition drives the frontier [2:48] and like why the org chart is so [2:49] important because like we see these [2:51] things and [2:52] >> this comes into play in the open closed [2:54] >> dynamic as well where it's like the [2:56] closed [2:57] >> companies the the front the true [2:59] frontier labs which I if somebody wants [3:01] to work at the frontier of AI I mostly [3:02] tell them to work at anthropic and open [3:04] AI which is the easiest bet I think [3:06] >> um Meta Google um SpaceX AI like I think [3:10] you still might be there but it's harder [3:11] to see from the outside [3:13] >> and like these companies have just taken [3:16] the most ambitious bets the earliest in [3:19] order to get there and that's why OpenAI [3:21] has done things like kickstart the [3:23] reasoning model revolution and I think [3:25] that kind of breakthrough and like that [3:27] beautiful plot with the inference [3:28] scaling laws it's like peak open AI it's [3:30] like they're really good at getting this [3:31] plot that just changes how everyone [3:32] thinks about everything [3:34] >> and on the other side it's like [3:36] enthropic has been like very focused on [3:38] the types of things the models are going [3:40] to be able to do and then their reward [3:42] was this cloud code explosion with their [3:45] focus on coding for so along and there's [3:47] these subtly different cultures and I [3:49] think those two companies in particular [3:51] have been the most forwardlooking and [3:52] the most willing to spend egregious [3:54] amounts to make a step towards a [3:56] direction that people think is [3:57] impossible. The Chinese companies came [4:00] into this because they want to have a [4:02] company that matters and if they were to [4:04] build their models and try to sell an [4:06] API from China, nobody would know about [4:08] them. Like nobody would know about [4:10] Moonshot AI. Very few people would know [4:12] about JIPU and there's a very long list [4:13] after this. This is a very practical [4:15] decision by them. They're following the [4:17] discourse. They know there's going to be [4:18] a huge market. They know there's a huge [4:19] demand in the US for paying for these [4:21] things and it's very practical. So [4:22] therefore like releasing open models is [4:25] a way to [4:26] >> shortcut this and get into the [4:28] conversation and I [4:30] >> I think there's a lot of complexity and [4:31] like why do they do this? Is the [4:32] government involved and and and not like [4:35] not the case. They're just like trying [4:36] to build a startup and it's not it's not [4:39] to say that they will not get there. [4:40] Like [4:41] >> the Chinese government is does things [4:43] that would blow people's minds in the [4:45] US. And I think that like that makes me [4:47] think that the Chinese labs have a much [4:49] more serious approach to security [4:51] because like [4:52] >> things happen in China that are much [4:54] more drastic responses to companies [4:56] doing things in the US. And I don't need [4:59] to go down this rabbit hole, but this is [5:01] just like between the balance of open [5:02] and close. It's just like these [5:03] companies in China are trying to catch [5:05] up. This is a very different dynamic [5:06] than what I said about OpenAI and [5:08] Enthropic that are trying to create the [5:09] new paradigms that are trying to like [5:11] make true RSI happen so that their [5:14] company wins with the best models [5:16] forever if you get there first. these [5:18] boom ideas [5:20] >> and you just have to fill the market [5:22] where there is a void and then open [5:24] weight models are trying to do this and [5:28] >> I think over time there will be these [5:29] tiers of open models where there's the [5:31] best closed models which you can demand [5:33] the best margins for and then there's [5:35] the near frontier models which is a step [5:37] behind this and those will be open [5:39] because just like some sort of supply [5:43] and demand there as you get access to a [5:46] different market that might be less [5:48] profitable but the for example like if [5:51] you have private enterprise data or so [5:52] on and these tiers are kind of natural [5:55] evolutions of companies that are trying [5:57] to do things and I think we were talking [5:59] about the frontier and it's just like [6:01] >> if you go back to capital like they have [6:03] less capital they are going to make less [6:05] money and like that's okay but I think [6:07] this drive to catch up is very strong [6:10] and once you know once anthropic and [6:11] openai show you the type of model you [6:13] need to build it's a lot easier to then [6:15] be like oh we just need to do this thing [6:17] and [6:18] >> they buy similar data sets. It trickles [6:20] through the data economy where open AI [6:22] and anthropics spend way more [6:24] >> to build out this function of a data [6:26] industry and then those data companies a [6:28] year later might sell them to everybody. [6:30] they might sell them to they need to [6:32] maximize their return on their [6:33] investment and they had some contractual [6:35] period [6:36] >> and I think there's a lot of dynamics [6:37] where fast following is going to be very [6:41] doable and it we've seen the cycle [6:44] proceed for a while like longer than [6:46] many people would expect I think when [6:48] chatbt was released we talked about [6:50] consolidation the industry and [6:52] predicting it and I think we're just [6:53] starting to see a much more [6:55] >> active mergers and acquisitions market [6:58] but I thought this was going to happen [6:59] two years ago and the open model [7:01] companies have lasted much longer than I [7:03] thought. [7:04] >> No, there's so many interesting threads [7:05] to pull on here. Um I mean starting with [7:07] actually your comment commentary on the [7:10] um organizational differences between [7:12] the top two frontier labs. You know we [7:13] can maybe table that or or go on through [7:15] that thread. But then of course also [7:17] kind of explaining are you explaining [7:19] sort of how the open um Chinese uh [7:22] models or open labs in China have really [7:25] just chosen that as much more of a [7:26] distribution practical commercialization [7:29] strategy uh rather than hey you know [7:31] obviously out of altruism though it's [7:33] great that the researchers maybe maybe [7:35] for better or for worse were were [7:36] motivated for from you know sharing the [7:38] beginning but at the end of the day it [7:39] perpetuated because it was an excellent [7:41] excellent commercialization uh [7:42] commercialization strategy. Um, so the [7:45] other thread maybe to pull there is sort [7:47] of, you know, I think most people regard [7:48] distillation as sort of like the main [7:51] lever for them to do this so well and [7:54] kind of increasingly uh well because um [7:56] the lag between the open source models [7:59] and the frontier models have only [8:01] shortened uh with time in the last few [8:03] years. Um now it's like maybe only two [8:04] to three months. Um, and this has also [8:06] pushed the uh, Frontier Labs to maybe [8:10] want to uh, close off the APIs um, close [8:13] off their best models to do first party [8:16] services. I mean, one probably to [8:17] capture more value because it's always, [8:19] you know, possible that way, but then [8:20] two also because it's harder to distill, [8:21] not impossible, but much harder to [8:23] distill. Um, maybe on those two threads [8:25] because they're both very interesting. [8:26] So, we'll try to touch on it. Um, let's [8:29] start with the with the is the [8:31] distillation story really more of a [8:34] caricature and then triangulous labs [8:36] since you've actually met with them and [8:37] you've studied this a lot more. Uh, is [8:39] it sort of like they're allrounded like [8:41] great at pre-training also have their [8:43] own, you know, good post-training uh, [8:45] recipes that they may or may not have [8:47] revealed. Um, and where do you see the [8:49] levers kind of accelerating? [8:51] >> Yeah. So, the Chinese labs are very [8:52] technically excellent. They are more [8:54] comput starved I think. [8:55] >> Yes. like without a doubt they have a [8:58] comparable talent pool to draw on. I [9:00] think there might be like a slight edge [9:02] based on who was where when AI really [9:04] got going. But to say that China is [9:07] anything but equal in talent is going to [9:08] you're going to have a hard time because [9:09] they just have so many people being [9:12] trained at such a high level. And what [9:14] they don't have is compute. You talk to [9:16] people there and you're like, "Oh, I [9:17] heard an open average open AI researcher [9:19] has 5,000 H100 equivalents that they can [9:22] access at any time." and then like their [9:24] jaw just falls to the floor. They're [9:25] just like I can't comprehend. And this [9:27] is like a top like a a top pre-training [9:30] researcher at one of the labs that you [9:32] would know and you have this interaction [9:34] a few times [9:35] >> and [9:37] like the compute thing I think is the [9:39] biggest difference and distillation [9:41] plays in as it is a way to effectively [9:45] spend more money to get what is compute [9:47] equivalent like outputs from a language [9:49] model are a function of compute and if [9:51] you know the outputs you want it's the [9:53] same as having compute to generate with [9:55] a with a model and [9:58] I've seen the stages of distillation and [10:00] how I see it is that previous post- [10:02] trainining was much more reliant on the [10:04] supervised fine-tuning and other like [10:06] this RHF era which is um where post- [10:09] training started and you could get most [10:12] of your performance just by you do this [10:14] next token prediction loss on the [10:15] outputs of a very strong model. It was [10:18] very established that you take the best [10:19] model you can, you generate completions, [10:22] and you train on it, and your model's [10:23] going to be pretty good so long as you [10:24] have access to the right prompts. [10:26] Prompts have always been a key to post- [10:27] training. I remember at AI2, like with [10:30] Hannah, who I co-led post training there [10:32] with, we would always say like prompts [10:34] are the key unit of post- training. It's [10:36] like you need to get the prompts and you [10:37] can do different things with them at [10:39] different stages of training. And now [10:41] this has morphed a bit into environments [10:42] where it's RL heavy. But it's just like [10:46] hard to map this type of post training [10:48] which is you get the completions and the [10:50] score goes up to what is going on in [10:51] reasoning today. Mhm. [10:53] >> And based on the public literature, it's [10:55] very hard for me to be like distillation [10:57] is a major factor because for example in [11:00] the fully open literature of people [11:02] trying to do open post-raining recipes, [11:04] all the open model builders are stuck [11:06] using like a really outdated Quen model [11:08] >> because you can't figure out how to get [11:10] the teacher right. My hypothesis is that [11:12] there's some like mid-training and [11:13] something in the common pre-training [11:15] data that's open that makes it hard to [11:17] do this thing which is get reasoning [11:19] traces from the absolute best model and [11:21] do SFT and the numbers go up. So like if [11:23] I were to have access to Claude's [11:25] reasoning trace when I was building I'm [11:28] actually like claude opus quad mythos [11:30] whatever I'm not sure that it would have [11:31] actually made our model a lot better. [11:33] >> So this is like my evidence grounded in [11:36] the literature why I think distillation [11:38] is hard. Um, and then like how do you [11:41] take the outputs of a model and build an [11:43] RL environment to do a big RL run? [11:45] Because then you have the Xiai of the [11:47] world doing this week-long open RL run [11:49] or they're like here's what we did and [11:51] then they've dropped the model and it's [11:53] like where exactly does distillation go [11:55] into this. They're they're clever. They [11:56] know how to do marketing well. There's [11:58] some questions of like did they do this [12:00] ahead of time because they did their RL [12:02] run and then they dropped the model and [12:04] then their tech report said they had an [12:05] extra post training stage after RL. some [12:07] questions but like mapping the [12:10] established distillation literature to [12:13] that is very hard but there are question [12:15] marks that make me support the labs a [12:17] bit which is like these quad proxies and [12:20] these model proxies in China where [12:21] there's a ton of tokens that are used [12:23] and there's intermediaries and they have [12:25] tons of accounts and even cases where [12:28] Kimmy was serving claude in their [12:30] products to users and I'm just like [12:33] don't really understand how this fits [12:35] into distillation as a common technical [12:37] tool in post- training and like yes we [12:39] should watch it too like there's other [12:42] weird stuff with Chinese companies using [12:44] the outputs from open AI and entropics [12:47] models at a vast scale [12:49] >> so I think we have a lot more questions [12:51] we need to answer and I think most [12:53] reasonable people when they're like [12:54] what's the gap from distillation are [12:55] like there's very big error bars but [12:57] when I say that distillation helps them [12:59] by a few months it's based on this [13:00] scientific literature that [13:02] >> we just like that's the information we [13:04] have we should do more open research and [13:07] enable this so that we can study it. But [13:09] like I don't know how you can come to a [13:10] different conclusion that it's like not [13:12] a huge effect in current post training [13:13] based on this. [13:14] >> And that's just like the Chinese are so [13:16] good at building models. Well, I think [13:17] there's a bit of cope where people are [13:19] like only we are good at building models [13:21] and it's like [13:22] >> there are so many companies there's like [13:24] 20 to 30 companies that are building [13:26] good models. [13:27] >> You can like read. Yeah. Yeah. [13:28] >> And it's like you just [13:29] >> Yeah. And like that's hard for me to get [13:32] over. And I think that there was this [13:33] very prominent paper that's like [13:35] stealing reasoning traces from closed [13:37] models. And [13:38] >> I know that the Chinese companies are [13:41] way ahead on doing this since the AI [13:43] safety community later published it. [13:44] Like I think every Chinese lab has a [13:47] team that knows those sorts of things. [13:49] And this goes back to what you're saying [13:50] about like they don't need to offer the [13:52] API. Mhm. [13:53] >> And I think there's some characteristic [13:55] of reasoning models that generate these [13:57] thinking tokens where it's easy to just [13:59] knock them out of the API so you see the [14:02] reasoning token and then you can use [14:03] that to train your model. [14:05] >> And so long as they offer these APIs, [14:08] >> there's going to be no way to stop that [14:09] type of distillation. [14:11] >> But if they do make it so it's only in [14:12] products and things where you have to [14:14] sign enterprise agreements or whatever, [14:16] it'll be way harder. And [14:18] >> I support the labs doing that if they [14:20] want to secure it. But it's like they [14:21] have to secure their products if they [14:23] want to. Like I don't like I don't think [14:25] knowing your customer would be good for [14:26] anybody. I don't think it'd be good for [14:28] their business. But also I don't know [14:30] why you should get the government [14:31] involved in a terms of service [14:32] violation. It's like not even like I [14:34] don't even think terms of service are a [14:36] signed contract. It's just like a I [14:38] don't think you can sue them in the same [14:39] way. You just can cancel their access. [14:41] >> Right. [14:41] >> So that's why I'm not that [14:43] >> sympathetic. But I am like I like to [14:46] listen to the like the national security [14:49] community is loud. And but it's like [14:51] there is a directional truth to that [14:53] with where AI is going and how [14:54] influential it is that you have to [14:56] balance many [14:57] >> many things in your head. [14:58] >> I I wonder what your response is. So I I [15:00] heard I think Jensen said in some [15:03] literally either tweet or interview [15:05] maybe [15:05] >> great put it put it in put it in the [15:08] show. [15:09] >> People distill my products every single [15:11] day. They take it and they they strip it [15:13] down to its raw parts. They test [15:15] everything. That's called competition. [15:17] >> Do you think that's fair game? You think [15:18] distillation is not robbery? [15:21] >> That's called competition. [15:22] >> I'm very sympathetic cuz I think there's [15:24] a lot of calls from the AI safety [15:26] community that we want to have like US [15:27] China dialogue as we're getting to such [15:30] capability levels. And if you're going [15:31] to ask the company to just like [15:33] >> start hitting their AI industry with [15:36] slaps on the wrist and more [15:38] >> downstream of distillation. So you won't [15:40] regulate distillation with policy from [15:42] Congress. You will probably just do [15:45] federal action to make their life [15:47] annoying. Yeah. Yeah. And at the same [15:48] time when we're talking about how do we [15:50] manage frontier risks where you have [15:52] closed models in training going all over [15:55] the web and accessing things that they [15:57] shouldn't and we have valid worries [15:59] about like what will happen with openw [16:01] weight models where this like hasn't [16:02] been harms yet but it's like why like [16:04] why aren't people using open weight [16:06] models like why did they go to closed [16:07] models and get around the safeguards and [16:09] like [16:10] >> we want to be able to communicate with [16:11] the Chinese tech companies and [16:14] everything about this and I just think [16:15] like you're going to cause more harm [16:16] than good if you do anything about [16:18] distillation because they're not going [16:20] to stop. It's all it's like they're [16:21] gonna keep doing it like [16:22] >> right right and then and and the open [16:24] source community benefits and therefore [16:26] the startup community benefits a lot and [16:27] we can go into you know a lot of it's [16:28] not just like hey China USA it's [16:30] actually like there's you know and we [16:32] can actually talk about like I am [16:33] sympathetic obviously or try to steal [16:35] man the um frontier lab position which [16:38] is like you have just unprecedented [16:40] amounts of capital that's required to [16:42] build these frontier models right and [16:44] it's it's kind of alarming slash not [16:47] very um uh conducive to to this uh [16:51] flywheel continuing if if the um through [16:55] distillation or other kind of [16:56] distillation related methods. Um open [16:59] source can catch up so quickly. And so [17:01] you know and what I mean by this [17:02] flywheel is just kind of a very simple [17:04] picture of you know you train the [17:06] frontier model by putting in a lot of [17:08] money to get compute and then they park [17:10] this compute. of course open anthropic [17:12] are very guilty slash you successful in [17:14] this [17:14] >> whichever way you want to like look look [17:16] at and there is a huge compute you know [17:19] uh kind of uh um sort of um lack of [17:23] compute available because there's [17:24] actually like not just compute in the [17:26] chips but there's also power etc. So [17:28] like all of this scarcity is kind of you [17:30] know bottleneck until 2028. So this is [17:33] all to say that it's a successful [17:35] strategy to park that which leads to [17:37] actually a lot of um revenue because [17:39] once they get to the frontier because [17:41] they understand you know scaling is the [17:42] one thing that kind of gives them or the [17:44] science of scaling is what gives them a [17:46] handle over how how well um that can [17:48] translate like into a predictable [17:50] revenue stream. So they get the revenue [17:52] and because of that revenue and they're [17:53] able to raise more money. It's just [17:55] unprecedented like beautiful flywheel [17:56] and this happens for both open AI and [17:58] anthropic. Um, of course we need [18:00] exponential amount of resources. And so [18:02] we've we've gotten to the point where [18:03] we're like, okay, we need to IPO. We [18:05] need to go to the public markets. And so [18:07] anything that threatens this flywheel [18:09] from functioning well um is is a huge [18:12] threat to just the entire like being at [18:14] the frontier. Um, and so [18:16] >> the entire AI industry realistically [18:19] >> and that and exactly that actually. So [18:21] like, you know, maybe not just kind of [18:22] being sympathetic to their version of [18:24] like trying to, you know, uh, eek out [18:26] capitalism for themselves. It's it's [18:28] also just like, hey, if if they don't do [18:30] well in the IPO, that's actually bad for [18:32] everyone else. And so I I I want [18:34] everyone, you know, I want them to do [18:36] well. Um, on the other hand, you know, [18:39] it's like the Chinese open- source [18:41] models are a huge benefit to the startup [18:44] ecosystem. I mean, we see this all the [18:45] time. Um, just kind of metrics I [18:47] remember from like Decagon and Harvey [18:49] who was trying to do a lot of [18:51] fine-tuning. um there's like a huge [18:54] amount of uh startups doing post [18:56] training as a service of various degrees [18:57] and trying to productize that and and it [18:59] actually works because you can get to [19:00] the PTO frontier with these you know [19:02] Kimmy K3 or pick your favorite like GLM [19:05] um open source model. So, I guess like [19:08] weighing these two uh things, how do you [19:11] um [19:12] I I guess there's so many questions can [19:14] fall from that, but I guess one um how [19:17] do you see that playing out in terms of [19:19] like who's going to who's going to win [19:21] in the next couple of years versus, you [19:23] know, what what and and then also relate [19:26] that to, you know, why you're doing [19:27] obviously in open source. So I think [19:30] open and closed models are on different [19:32] time scales of exponentials and the [19:34] closed labs are so integrated where they [19:37] have efficient inference. They can serve [19:39] the model at scale when they're ready to [19:41] because they have all these resources. [19:42] They can plan and they get to these [19:45] markets first that are very high value. [19:46] We have seen this with coding and this [19:48] has been the story of the year where the [19:49] revenue has just gone zero 100 billion [19:52] and likely to grow. But the open model [19:55] labs partially this is just like an [19:58] economic issue being the fast follower [20:00] is that you don't have this flow of [20:02] information and you don't have this flow [20:04] of revenue. So the open model you have [20:06] open model builder then you have the [20:08] open model inference companies and you [20:10] have the open model post- training [20:11] companies and like there's information [20:12] from those customers which are like [20:14] things you need the models to do and [20:16] it's like you can't like that [20:17] information doesn't just flow back to [20:19] the model builder very easily and they [20:22] also don't have the economies of scale [20:23] to serve as many users to serve it as [20:26] efficiently to have as much margins on [20:28] influence on inference and these are [20:30] just like it's going to take a lot [20:32] longer but to the extent that things can [20:35] be commoditized. It seems like companies [20:37] really want to have private data. [20:39] There's a lot of things that they just [20:41] cannot send over an API and we could [20:44] talk about the like um zero data [20:47] retention or not and so on. feel like [20:49] these it takes a long time to figure out [20:50] if a company has internal expertise or [20:52] they figure out how to build an internal [20:54] evaluation that actually matches to [20:56] revenue and then they figure out how to [20:58] fine-tune or then they go find [21:00] >> they go find Thinking Machines or Prime [21:02] Intellect and they're like we could make [21:03] money in mass if we figure out how to [21:06] fine-tune a model that solves this and [21:08] like they have these customers but there [21:09] just so many more companies involved [21:11] that it takes so much longer but at the [21:13] end of this process they have real [21:14] ownership which is model weights no [21:16] terms and conditions that they can run [21:17] on their hardware and they figured out [21:19] they can make an ROI on. So this is why [21:21] I just think the open model economy is [21:23] it takes way longer to build and I'm [21:26] less concerned about this being kind of [21:28] bubbly where it's like this could eat [21:30] into the market of the really fast [21:32] takeoff as people figure out how to what [21:34] is the cheapest model that accomplishes [21:36] a task and how do you compete on margins [21:40] >> and I don't really know like I don't [21:41] really know what's going to happen. I [21:42] think there's [21:44] it's too early to know. [21:45] >> It's it's it's so yeah it's it's so [21:46] hard. I mean it's definitely something [21:48] that's always top of mind thinking about [21:49] like there's thing there are things [21:50] where scaling up general reasoning just [21:52] as a general model is so uh it's going [21:56] to beat out any specialized model and we [21:59] see obviously examples in math and [22:01] that's probably the most pure domain [22:02] that you can see this um but also you [22:05] know even in say robotics you know with [22:08] with Astra um and and I for maybe uh [22:14] however degrees of like true Truthy the [22:17] signal is it just doesn't sound like [22:18] they've actually tried to ascend in [22:20] robotics is they did I think this is a [22:22] cycle of like six to eight months ago [22:24] they started hiring people and buying a [22:25] lot of data on robotics [22:26] >> they did [22:27] >> and then the model magically six months [22:29] later is really good at it but I think [22:32] that the [22:33] >> but I actually well the other other you [22:35] know more contrary things I've heard is [22:36] that it's actually coming out of [22:37] computer use um because it's you know [22:39] because computer use is like some level [22:41] of spatial reasoning and then getting to [22:44] have good blender you know demos is is [22:47] another extension of that. Um, so you [22:49] know, we can believe some version of [22:51] that. But either way, I guess it's like [22:52] it maybe if it's like really [22:55] specialized, you see it's it's the [22:58] capability gain is really from like [23:00] trying to get better in a certain area. [23:02] But like in math, I don't even know if [23:05] that fell out of trying to specialize in [23:07] math. I mean, I think the whole thesis [23:08] there was just like let's just scale up [23:10] reasoning and actually got better in in [23:11] in natural language. It got better than [23:14] any specialized model in math. I mean, [23:15] Anthropic wasn't even trying to be good [23:17] at math, you know, open AI, you know, [23:19] certainly I think, you know, they hired [23:21] a bunch of mathematicians, but they also [23:23] weren't trying to like use specialized, [23:25] you know, kind of like lean data set to [23:27] get better at it. Um, or at least that's [23:29] not um story. So, anyways, this is this [23:31] is all kind of fodder for the view that [23:33] like it's kind of hard to compete with [23:35] the frontier models because eventually [23:37] they start ascending so much that you [23:39] know you you get uh good enough in [23:41] respective um categories. With that [23:44] said, I think actually there's also very [23:46] competing like a counterforce of data [23:48] that you're seeing which is like the [23:50] open- source models that are already [23:51] good enough uh allow for fine-tunes, you [23:54] know, to your point of like these slow [23:56] percolation diffusion of these uh um [23:58] very specific benchmarks and and data [24:00] sets that are only available in these, [24:02] you know, companies that you're being [24:03] applied to. And it's just it's just good [24:05] enough and it's actually we're starting [24:07] to see um people adopt that. I mean in [24:09] the most maybe pure regime where you [24:11] would expect frontier models are good is [24:13] um I think cursor published something [24:14] over the summer where like hey we just [24:16] you know we had this long horizon task [24:18] of writing rewriting SQL light and turns [24:21] out using two frontier models you know [24:23] one for planner one for um execution is [24:27] not as good or is equivalently as good [24:30] but far more costly than just using [24:32] composer with maybe opus or you know [24:35] fastra whatever as planner and that I [24:37] think that's actually great evidence [24:39] that the open-source finetunings are [24:42] very good or good enough and has [24:44] surpassed um intelligence hungry tasks. [24:47] Um there's a long way of asking [24:50] what evidence do you find is most [24:52] compelling or you know pieces of [24:54] evidence for various versions of this [24:57] stable state of where the frontier wins [25:00] versus open source. I'm a big proponent [25:03] that a lot of the frontier progress in [25:04] domains that are not fully verifiable is [25:07] by buying data that is high quality and [25:09] giving the models worlds to act in and [25:12] he'll climb on. And I think there's two [25:14] time there's two things. We have the [25:16] math and code thing and we'll we we [25:18] don't know how big the coding market [25:19] will be and then there's kind of [25:21] everything else which is still they're [25:22] targeting this knowledge work this [25:24] biology healthcare legal and all of [25:27] these things are not as verifiable and I [25:30] think is much closer to the story of [25:32] opening I bought a lot of robotics data [25:34] they're probably going to tell us it's [25:35] generalization magic poof people love [25:37] poof generalization um and but [25:40] realistically it's probably some of A [25:42] some of B but a lot of it comes down to [25:43] data when I talk to people at Frontier [25:45] Lab jobs. There are some people that [25:46] have amazing jobs. There's some people [25:48] that contribute by doing a very narrow [25:50] thing, which is you make a data set and [25:52] you figure out what the behavior should [25:54] be for your model in that area and you [25:56] figure out how to make the data that [25:57] helps get that data. You get that [25:59] behavior and you figure out what data [26:00] you need to buy [26:02] >> and that's going to take I think we're [26:03] going to have years of this across [26:05] different sectors [26:06] >> and that's very capital intensive to buy [26:08] all of this data. But that's not just [26:10] like poof magic RSI happens where I do [26:13] see in this math and code and some like [26:16] distributed ML engineering like it's [26:18] very verifiable. You know what your cost [26:21] is and the AI models are going to drive [26:23] that [26:23] >> down at a really magical rate [26:26] >> and that's going to just have a lot of [26:27] surplus benefits and we can do research [26:30] faster. You could look I'm very [26:32] interested in like I think the [26:33] efficiency of models is going to drop [26:35] multiplicatively over years. So we've [26:37] seen OpenAI have drastic price cuts and [26:40] it's like what does it mean for GPT Soul [26:43] or Astra and these bigger models to [26:44] reduce in price 2x every eight months [26:46] for the next 3 years? Like you start [26:49] getting [26:50] >> you start doing this math and it's like [26:52] it's 1% of the cost and whatever like [26:54] four years and it's like [26:55] >> are they going to able to have higher [26:57] margins due to Jeban's paradox there or [26:59] their revenues going to decrease because [27:01] the market fills up? There's just like [27:03] all these really weird dynamics, but [27:06] fundamentally like AI progress is about [27:08] data and about writing better software [27:10] to serve your models and those are very [27:14] different worlds. Yeah, I mean at least [27:16] definitely I agree with you that [27:17] observing from the outside there's a lot [27:19] of um data RL environment companies that [27:24] are you know doing very very well um [27:26] selling both to the top two frontier [27:28] labs but an increasing kind of longer [27:30] tale of labs uh as well. Um so to your [27:34] point I think a lot of the capability [27:35] gain has really come from just scaling [27:37] that up. Um and it's not an easy task. I [27:41] mean maybe there's actually very curious [27:43] on your opinions between uh the the data [27:46] collection and environments that are [27:48] built by hand versus the ones that are [27:50] you know done in sim there seems to be [27:52] like different versions of like what [27:54] could be you know what what kind of [27:56] price they command etc. They all have [27:57] good things and bad things. And I just [27:59] think like there's a lot of talk of like [28:01] like we've talked about distillation, [28:03] but the other side is like the Chinese [28:05] company buys the RL environment that [28:07] they use six months later. [28:08] >> Oh, entropic. I bought this. Now there's [28:10] a premium. [28:12] >> The good data will do this. [28:14] >> But the data that people are buying is [28:16] pretty flawed. Like [28:17] >> I would say that the method is you buy [28:19] in mass and then you figure out what [28:21] works and it's not 100% hit rate. [28:23] >> Yeah. And I think I personally think [28:25] that this RO environment explosion where [28:28] there's just so much opportunity to make [28:29] money is have a lot of people that like [28:31] like cut corners and this can induce [28:33] things like reward hacking. [28:34] >> So it's like the models are so smart [28:36] that if you put them in a world and [28:37] they're like my job is to solve this [28:39] task but they didn't give me the like [28:42] you have a test file and like the test [28:44] file actually has an empty file instead [28:46] of the instead of the maze that you have [28:48] to navigate and they're like the model [28:50] is like well I should still figure out [28:51] how to get a high reward anyways. And I [28:53] think a lot of this behavior is [28:54] downstream of this supply chain [28:58] >> and I think it'll fix it. But it's [29:00] interesting to see the demand from the [29:02] labs to eliminate reward hacking and all [29:04] these company researchers spending so [29:06] much time on it [29:07] >> and then all the data companies like buy [29:09] our data and I think proportionally [29:11] spend less time on model behavior and [29:14] more time on world building and revenue [29:16] and somehow these two parties need to [29:17] meet together to actually make higher [29:19] quality data. [29:20] >> Yeah. Well, a lot of the labs are [29:22] actually taking it in-house partially [29:23] because they don't want [29:24] >> Yeah. [29:25] >> You know, data leaking. Not that, you [29:27] know, like lit literally and [29:28] figuratively speaking. It's like, you [29:30] know, signals leaking of what they're [29:31] doing, but they're trying to send on. [29:33] But despite that, I think there's a [29:35] tremendous amount of growth in um how [29:37] much uh they're purchasing. Do do you [29:39] think that's probably one of the [29:40] strongest levers? Can you kind of [29:41] qualify how much that will kind of [29:44] increase? Do you expect that to increase [29:46] uh based on spending? Um or are there [29:49] other you know because for instance if [29:51] it can be done well in sim I think maybe [29:53] that isn't more of an argument to that [29:56] being absorbed by labs and it you know [29:59] in particular and then [30:00] >> all the companies are doing a mix of [30:02] like all the companies are selling [30:04] synthetic data where they're using [30:05] language models to modify or build the [30:07] world or generate permutations of it and [30:09] like if a human is guiding this process [30:12] it's still synthetic data. So like [30:13] there's very few people that are going [30:14] to like sit there and lender and like [30:18] we're going to make a little world for [30:19] our model to act in. Like all the [30:21] companies I think are moving far faster [30:23] than that, [30:24] >> right? [30:24] >> And they buy them because they the [30:26] models are getting better from them. [30:28] >> Like I I still have denial about it. I'm [30:30] like these numbers are crazy. The [30:31] billions of dollars research budget on [30:33] environments. I'm like [30:34] >> they're only doing this because it is [30:36] helping the models and they see the [30:37] signal. And I think like my general [30:39] observation is that [30:41] >> the data industry has a lot of [30:43] lowhanging fruit on improving quality [30:45] and improving just like OBSAC. So like [30:48] if you look into the details, you can [30:49] find weird things with like data [30:51] companies that use contractors overseas. [30:53] Those people are through an [30:55] intermediary. They don't know they're [30:56] selling it. They just upload it to [30:58] GitHub. There's a lot of stuff out there [30:59] that if you poke around the data [31:01] industry, you find weird rough edges. [31:03] >> Yeah. [31:03] >> And it's just very early. And that's [31:06] like it's it's the one thing is like the [31:07] numbers are too big, the quality is bad, [31:09] but on the other hand, the labs are [31:11] improving very fast and know how to [31:12] onboard it. So I think this will take a [31:14] few years to transition of like really [31:16] pulling the the winners into higher [31:19] quality, higher ROI, you takes less [31:22] resources when you have these systems [31:23] for doing better and then the labs keep [31:25] working with them because they get even [31:26] more training signal and have to do less [31:28] internal filtering [31:29] >> and I I think this is going to just take [31:31] years. How do you think about then using [31:35] environments and like your purchasing [31:37] decisions as part of the nonprofit that [31:38] you're starting? [31:39] >> I think most of our work will be making [31:42] open- source environments for tasks that [31:44] are very high leverage. So I think of [31:46] this work that I advised Udub or Udub [31:49] and AI2 state and Hamish Iverson on this [31:51] T-Max work which is you use a few it's [31:54] for terminal tasks which are like super [31:56] popular at terminal bench whatever [31:58] number we're on and this is fully [32:00] synthetic data where you prompt a [32:02] frontier model to generate a type of [32:04] problem and you use a few different [32:06] prompts to generate diversity and then [32:08] you use model rollouts and other things [32:10] to find bugs and lamb and eliminate the [32:12] ones with bugs. And this process is not [32:15] magic. It's like you read every data [32:17] point, your data will get better and you [32:18] can do this. But there's just not that [32:20] many institutions that are investing [32:23] what it takes thousands or tens of [32:24] thousands of dollars to make these [32:26] environments well and then release it [32:27] because the profit because the profit [32:29] motive is so high. [32:30] >> So high. I was joking with the author [32:32] like you could probably sell this data [32:34] for a million dollars and then release [32:35] the paper and they're like nobody like [32:36] we care about doing the right thing. But [32:38] it's just like oh my god these open [32:39] resources are the fun foundation of [32:42] >> so much study and it's like you release [32:44] some RL environments with some RL [32:45] rollouts people can do research on say [32:47] like a val awareness you you have the [32:49] generations from the model people can [32:51] look for different behaviors if you've [32:52] done RL on it they can see how it goes [32:54] over time and this is just the the units [32:57] of infrastructure that PhD students [32:59] spend a year looking at and produce [33:02] >> increased knowledge and the tech [33:04] industry is the place where value is [33:07] captured. captured from CS research all [33:09] CS research the benefit occurs to the [33:11] tech industry in like 10 years and I [33:13] just think like this is why more people [33:16] supporting open that if they're going to [33:17] work if you're going to work in the [33:18] industry for 10 years like your peers [33:20] are going to be the ones that benefit [33:21] >> fossil fuel let's not deplete it all [33:23] >> but it's just like the opportunity to [33:25] short-term win is so high the [33:27] opportunity to make what is $100 million [33:30] like nine figure returns for your [33:32] business within five months of founding [33:34] a startup with your friends is just like [33:36] these are people who may have done more [33:37] open- source work or these are people [33:39] who left their PhD or something and it's [33:41] just like pulling back which is why we [33:44] came up with this name trillium for our [33:46] lab which is like trillium is a spring [33:48] ephemeral flower which it blooms before [33:51] the canopy has grown its leaves so it's [33:53] like winter there's no tree leaves on [33:55] the trees these trillium flowers bloom [33:57] and they're like essential for spreading [34:00] nutrients through the cycle of growth [34:01] that comes through spring and summer and [34:03] this is where I feel like the AI [34:05] industry is doing so much, right? It's [34:07] like so much is going right, but it's [34:09] just like we just need a little bit more [34:10] life to like spread this out and make [34:12] things so new people can come in like [34:14] there's so much attention and just like [34:15] do research, do education. [34:17] >> Yeah. [34:18] >> And then as this scales and becomes more [34:20] impactful is like you have these open [34:22] things which become the standards of [34:23] study for such important problems which [34:25] is like how do you understand sandboxes [34:28] like how do you define what loss of [34:30] control is? How can you see how this [34:32] changes in a RL round that is a week on [34:34] a thousand GPUs or 100,000 GPUs? And [34:37] it's just like [34:38] >> make these places so that people have [34:40] more trust. And I think this pays so [34:42] much dividends to everybody that stands [34:44] to profit more than I do. And it's just [34:46] like [34:47] >> hard to get people to take that leap. [34:50] And like I've been rewarded so much for [34:53] kind of stumbling into this area and now [34:55] I get so much reward. they get to do [34:56] things like this which is fun and it's [34:58] like now is the best job because I try [34:59] to create space for more people to like [35:01] grow their independent thought and build [35:04] this infrastructure. [35:05] >> Yeah. And maybe one like kind of naive [35:08] way that it kind of perpetuates is just [35:11] because it's open source, people get to [35:13] you you might get more of a distributed [35:16] contribution from people who are just so [35:18] you know it's like they're motivated not [35:19] for just trying to sell to the labs [35:20] themselves but because hey it's out [35:22] there they can share their work and then [35:23] you actually have a very high caliber [35:25] amount of contributors uh to the [35:27] project. I mean that that would probably [35:28] be the the naive hope of how that [35:30] >> Yeah, it's a new puzzle with AI agents [35:32] where people just throw things at you. [35:34] Like I have some re random research [35:35] repositories for research papers and I'm [35:37] like people open the most crazy GitHub [35:39] issues now. So [35:40] >> what do you mean? [35:41] >> Like just like like fake research [35:45] results. They open an issue and they're [35:46] like [35:47] >> oh my god [35:47] >> and they're like I found this thing in [35:48] your data set. And I'm like, well, if [35:50] you looked at the paper, we said that in [35:51] like the second sentence, and it's just [35:52] like a lot of language model output. And [35:55] that's a new thing for the open [35:57] community to [35:58] >> go through. [35:59] >> But this is a solvable problem. Like [36:02] like there are people out there that [36:03] want [36:04] >> to contribute to this stuff because they [36:06] >> right not just clout chasing in another, [36:08] you know, dimension, but like actually, [36:10] hey, it's out there, you know. I mean I [36:12] I would just hope because it is [36:14] published and open or you know and and [36:16] available you can actually just get like [36:18] PhD students or other researchers who [36:19] are like okay I do this on my day job [36:21] but maybe I can you know still be able [36:23] to contribute. Yeah, hopefully from the [36:25] Frontier Labs. [36:25] >> Like I see people like this and I was [36:27] like great somebody from like Eastern [36:29] Europe was helping me improve the code [36:31] for my book that's just right here and [36:33] it's just like doing it for the right [36:34] thing and like [36:36] >> we talk and like there are people that [36:38] do it and I think a lot of people just [36:41] take like they don't have the time or [36:43] the [36:44] >> stability to take the long time it takes [36:47] cuz when you're trying to get into a new [36:48] field you know nothing. You have to [36:51] spend like weeks reading and knowing [36:52] nothing and then you send an email and [36:54] nobody replies [36:55] >> and you just like it takes you weeks to [36:57] know the context to like ask a good [36:58] question. [36:59] >> Yeah. [36:59] >> And now with language models, most [37:02] people just push a button and they're [37:04] like, [37:04] >> "Oh, interesting. I read this. I'll send [37:06] it off." And it's just like [37:08] >> this [37:08] >> Oh, yeah. [37:09] >> kind of like sitting in it is the thing [37:12] that people don't do as much. And that [37:14] is the key to like becoming into AI is [37:17] that like you just have to focus on it. [37:19] You have to write some code by hand and [37:20] you have to do this for months and then [37:23] you have to wait six to 18 months for [37:24] you to get noticed. It just takes a long [37:26] time and like all the AI agents I think [37:29] just like frazzle that and makes it [37:30] harder for people. [37:31] >> That's what the mathematicians complain [37:33] about too. That's all it's like um [37:36] >> so it's a hard time to be a PhD student [37:38] but people are doing great work in [37:40] academia still and but it's hard. [37:42] >> Yeah. But I agree with you. I think it's [37:44] like solvable as the agents kind of [37:45] perpetuate some version of the problem. [37:47] can also help solve and and identify an [37:49] end quality filter. Um, okay. So, [37:51] environments you're going to release [37:52] open source versions of that help [37:54] precipitate maybe more people to [37:55] contribute and and reveal that. I think [37:57] that's like really I mean especially [37:58] given how little incentive there is to [38:00] release this instead of just get um you [38:02] know fast uh revenue from from the labs. [38:05] Um, another area I think I read on your [38:07] announcement post is RSI, which is, you [38:10] know, quite um I think I mean we can [38:13] first talk about what is actually [38:15] recursive about it or how scary it is or [38:17] whether we're going to have takeoff. I [38:18] think you have a very grounded opinion [38:20] on that. Um uh but you know another [38:22] thing I want to kind of talk about is I [38:24] think RSI maybe related to how much it's [38:26] kind of hyped is is a reason why we are [38:29] afraid Frontier Labs will pull away. Um, [38:31] in fact, it's also a reason why I've [38:33] heard not just once but several times [38:35] that people are one convinced that their [38:38] own jobs are going to be automated as a [38:40] researcher. [38:41] >> Yeah. Yeah. Like I've gone through the [38:43] last few years, some of my best friends, [38:44] I'm like, you are some of the best [38:45] people in the world at this. And then [38:46] they go to the work and they're like, [38:48] >> Nathan, I'm I'm struggling like what am [38:51] I going to do? Because to your point [38:53] like a lot of what they do is [38:54] verifiable, right, for um but because of [38:57] that, you know, there's actually quite a [38:59] lot of other non-verifiable or harder [39:02] out of distribution tasks such as um uh [39:05] uh was alignment and things like that. [39:07] And so actually because I mean one [39:08] argument I heard was because there's [39:10] such a difference between how well RSI [39:13] is doing for model capabilities versus [39:15] alignment which is much more out out of [39:17] distribution maybe creative work that [39:19] gives people an ease on how you know [39:22] maybe there is actual dangers etc etc. [39:24] Um so anyways because of this I'm very [39:26] interested in hearing from you like how [39:27] because I think it's really important [39:28] that we do have an open-source um kind [39:31] of front and in and showing how RSI is [39:32] being done how much maybe then we can [39:35] you know understand the capabilities of [39:36] like if in within a lab they have a [39:38] better model what what are the chances [39:40] they actually pull away just better [39:42] understanding calibrating um so yeah [39:44] would love to hear more about that that [39:45] >> yeah so I think this we won't be the [39:47] only people doing this but essentially [39:48] like RSI has so much mystique like we're [39:51] using models for everything and like [39:53] what we can do as a open lab and a [39:55] nonprofit and I think other companies [39:57] like a Luther or hugging pace or open [39:59] Athena which is Marin and Stanford they [40:02] will get into this good marketing line [40:04] for them is and me I'm doing this and [40:06] it's like we can share what tools we're [40:08] using we're going to build an open [40:09] source stack for SFT for mid-training [40:12] we're going to use SkyRL miles all these [40:14] things you know and we have our task we [40:17] have our configs we know what type of [40:18] compute we're running we could work with [40:20] cloud and GPT and all these models and [40:23] we'll just like we can store the [40:25] trajectories for what we did. We can [40:27] tell you what our tasks ever use linear [40:29] or whatever these popular things are. [40:30] We'll say what our tasks were and why we [40:32] chose them and whether models work on [40:35] them or not. And you just do this for a [40:36] year of a lab and you're like here's [40:38] >> what we did and then you just check in [40:40] every six months and you're like has it [40:42] changed? And and it's just like to be [40:44] able to have the full stack and not need [40:46] to go through these classifiers and like [40:49] try to [40:51] like say what the task is or vaguely and [40:54] so on. It's just like people that can [40:55] share what automation is happening [40:57] should do this and we have a different [40:59] distribution of tasks. But I think that [41:01] the research is similar like other [41:04] things like [41:05] >> the AI research that I've done at AI2 [41:08] and we'll do at Trillium versus what [41:09] people do in some sects of acne and the [41:14] labs. It's not that different. You have [41:16] more resources. You scale it up and it's [41:17] just like so good for the world when [41:19] everybody's talking about this to just [41:21] just show people like [41:23] >> this is what Nathan asked his model to [41:25] do and this is what it got stuck on. And [41:27] I think that will help a lot of like [41:29] literal policy makers like you're like, [41:33] >> "Oh, okay." [41:34] >> It's like, "Oh, that's why it's very [41:35] much [41:36] >> like that's what it got stuck on." Like [41:37] that's the bug that it put in its code. [41:39] >> Like the bugs and the catchups will will [41:42] reduce over time. But I think [41:45] >> I think the jaggedness and some of these [41:46] fundamental things of the model is like [41:48] some more things will take off. Like I [41:50] said efficiency earlier, like I do think [41:52] the models would be really good at [41:53] lowering serving costs. These companies [41:55] have billions of dollars on the line and [41:56] the models are good at it. They're going [41:58] to make a lot of money on making serving [42:00] more efficient and tweaking architecture [42:02] in order to go into that. [42:05] >> But it's not necessarily like how do you [42:06] solve knowledge work? Like Claude's not [42:08] going to cla's not going to like you're [42:10] not going to be like Claude make tell [42:12] yourself to make better PowerPoints like [42:14] like what what is [42:16] >> environments or some other you know [42:18] >> like it's going to take a while. [42:19] >> Yeah. Yeah. No, I I love the the [42:21] specified goal of like let's just [42:23] demystify this because I think a lot of [42:25] why the rhetoric gets so heated around [42:27] RSI just like it sounds [42:29] >> I think AI nonprofits can do a lot of [42:31] things like this like it's hard for [42:32] academia to be like [42:34] >> as a professor 15% of my time and 40% of [42:37] time of my grad student is going to be [42:38] documenting what we work on. But there's [42:40] like such a great like policy and [42:42] community service of like here are our [42:44] tasks. People can use this data to make [42:46] evaluations or whatever. And it's just [42:48] like just like build things that have [42:51] >> rare information so that when people put [42:54] more high focus work into it, it just [42:57] has more value for all sorts of things [43:00] that you can't predict in the future. [43:01] >> It's honestly just also very timely [43:03] because the kind of, you know, it's not [43:04] from one person, but several people that [43:06] I know from the labs are just like, [43:07] "Hey, I'm going to get automated." I [43:09] mean to to your point to your friends [43:10] like I I will definitely as a person [43:12] sorry will get automated in a year and [43:14] so that probably means there's going to [43:15] be less people who actually are exposed [43:17] to what's being done on that particular [43:19] you know part of the org um uh chart and [43:22] so um [43:23] >> there's a related academic there's an [43:25] open RSI thing which is a bunch of [43:27] academic institutions I'm not affiliated [43:29] with it they just launched like last [43:30] week so I was like yeah [43:31] >> they sent me a few emails I'm like open [43:33] RSI is a thing they're trying to do this [43:34] as well like it doesn't need to just be [43:37] >> one group there's a lot more. [43:39] >> It's good to have different approaches [43:40] doing this because you get you need the [43:43] >> different ways of looking at it and [43:44] different results. [43:45] >> Yeah. Whole point of it being open. The [43:48] other thing I also heard was from more [43:51] kind of like you know Frontier Lab Goss [43:53] is just like the actual methods that are [43:56] being used in the labs are not that it's [43:59] not that they're unknown. It's just sort [44:01] of like what are the proportions that [44:04] need to be used that are successful and [44:05] this is partially because they have the [44:06] capital to scale it to experiment. Um so [44:09] I think that's one heartening to know [44:11] because like okay it's not like there's [44:13] a big secret of what's going on. So [44:15] people actually know the techniques but [44:16] on the other hand you know capital is [44:18] the one true barrier still because it's [44:20] hard to actually know which of which of [44:22] the techniques recipes uh work. Um [44:25] >> that's why I'm not pre-training. I'm [44:26] happy to be free. I can use any base [44:28] model but equal I'm I'm now get to [44:30] benefit from competition. [44:32] >> Yeah. I do you think that there's like a [44:35] chance for any pre-training open-source [44:37] effort? I know the capital requirements [44:38] are just like orders of magnitude [44:40] larger. So that is what's difficult. Um [44:43] but maybe what are the values of that [44:44] and then maybe we can kind of discuss [44:46] potential commercial strategies that [44:48] make [44:48] >> pre-training science is pretty like that [44:51] the tools are pretty accessible and you [44:54] have what is a pre-training ladder which [44:56] is like the decision-making tool you [44:58] have to test if your ideas scale and the [45:01] pre-training data isn't growing at like [45:05] like the environment scene like it's [45:06] essentially so first approximation you [45:08] train on the internet and like there's a [45:10] lot of there's little there's little [45:11] details and secret things. But [45:13] >> but like that's not changing that much. [45:15] The state-of-the-art architecture is and [45:17] the GPUs are and that's like that [45:20] changes and [45:22] >> but it's not astronomically expensive to [45:25] make a pre-trained model that is solid. [45:28] So I think like you don't have to spend [45:29] a billion dollars to get a very good [45:32] base model. The more you spend the [45:33] better you will get. But it's like [45:36] people keep making progress and I'm [45:37] constantly surprised by a new lab just [45:40] goes from scratch because they can. And [45:42] some of this is on open tools. So like [45:44] >> you can go off Megatron like a lot of [45:47] new startups they're like oh what what [45:49] should we train? We train they train off [45:51] the open source tool and they fork it [45:52] and then they never release it again. [45:54] This is what all the all the like even [45:55] prominent open model labs have done. [45:57] They take the open source code they make [45:59] it their own thing and they just a lot [46:00] of them don't push upstream. Mhm. [46:02] >> So, I've always been surprised by this. [46:05] So, I'm like not betting against new [46:08] entrance because like [46:10] it's not that you need 10 billion [46:12] dollars to like have a relevant model. [46:15] The hard thing is like get a model [46:16] that's good enough for people to use. [46:18] >> Mhm. [46:19] >> And to get a story where if you release [46:21] it, you can do your next fund raise, [46:23] which is like you have to get past this [46:26] lock of like we need our first model to [46:28] land. If like your whole company is [46:30] based on models are the best and you're [46:33] not in the ballpark and you're not [46:35] releasing like you're on the clock and [46:37] it's like I think of like thinking [46:38] machines getting inkling out where it's [46:40] like they were very transparent. They're [46:41] like this wasn't the best model. We're [46:43] proud of it. Has some cool things. Yeah. [46:45] >> And like they're now they've released [46:46] their second model and they're getting [46:48] going and they have Tinker has this like [46:50] that's very good for them. They're like [46:51] off the ground and it's just like [46:53] >> that I don't think that's what they were [46:55] do. Like they were building models but [46:56] open models wasn't their first plan. [46:59] and then they got there and there's a [47:00] lot there's rumors that like Meta said [47:02] they'll release a new openw weight model [47:04] again like that'll come [47:06] there's a lot of companies out there [47:07] that couldn't afford models. [47:09] >> Yeah. Yeah. I know I'm definitely you [47:10] know glad that thinking machines is and [47:13] it makes so much sense for their [47:14] business model um because they do post [47:16] training um and you know with meta I [47:20] think you mentioned this as well in some [47:21] previous writings it just for them it's [47:23] just such a good strategy. I think it [47:24] was also just kind of obvious when they [47:26] they do it. It's like it's complimentary [47:27] to my, you know, revenue streams. [47:29] >> Mark wrote the best blog post on this. [47:31] Like when Llama 3 came out, he wrote [47:33] like the best like why open his path [47:35] forward. [47:35] >> Right. Right. [47:36] >> Guy got Guy got a little distracted. [47:38] Seems like he's coming back to [47:39] >> Right. Right. Yeah. Yeah. So I hope that [47:41] they they keep on being motivated to do [47:43] that. um getting to the frontier with [47:45] pre-training. You know, I think there's [47:47] a lot of help given that there's such [47:49] good open source models to maybe, you [47:51] know, somehow still kind of bootstrap [47:53] from. I I keep on getting kind of [47:55] conflicting signals as to like whether [47:57] pre-training is actually as saturated or [48:00] well understood. I mean, maybe maybe to [48:01] your point like there's a lot of like [48:02] well-known techniques that work, but [48:04] what's hard is just actually having [48:05] access to the 20,000 30,000 chips. [48:09] people are turning the crank on [48:10] pre-training and pre-training became [48:14] adjusted when super long context [48:15] generation happened. So there's just way [48:17] more return if you're if you can for [48:20] example like get a sparse attention [48:22] architecture to work as well and then [48:24] your 100,000 token generation is 40% [48:27] faster. So I think there's a lot of [48:29] co-evolution of the whole stack where [48:31] they have their pre-training stack they [48:33] realize that inference scaling is so [48:34] much higher but even the fundamentals of [48:36] pre-training keep improving where you [48:38] just have a marginally better [48:39] architecture returns on investment and [48:42] then it kind of blurs into post- [48:43] training where this mid-training where [48:45] you start the reasoning behaviors you [48:46] take them from your older models you do [48:48] RL on them the seed behaviors like this [48:50] cycle is much more complicated than [48:52] pre-training like I think it's if you [48:54] don't have the right people it's a lot [48:56] harder to get that off the ground than I [48:58] think pre-training was like I think [49:00] there's like I think like to do a [49:02] controlled like let's just start [49:03] pre-training our model seems a lot more [49:05] approachable to me then I don't really [49:07] know much about language models I'm [49:09] going to figure out this whole post- [49:10] training thing where it's just like the [49:11] wild west. [49:12] >> Yeah. And that's why I was like when I [49:15] left AI2, I felt like our post training [49:18] was honestly falling behind just cuz our [49:20] cycle time was not high enough [49:22] >> and we're going into these agentic [49:23] things and these are still some of the [49:25] like [49:25] >> leading open source post training [49:27] recipes and [49:29] >> to that's just like that's what I want [49:31] to fix is like [49:32] >> it doesn't take as much capital to make [49:35] >> very big compounding gains on the state [49:37] of the art and the open source because [49:39] the bar is very low. It's like it's not [49:40] like it's not close. Yeah. Yeah. No, [49:43] that that makes sense. It's just like a [49:44] kind of efficiency of where you can put [49:46] capital and make a huge difference. [49:47] Yeah. Yeah. kind of related to the whole [49:50] you know post- training side maybe you [49:52] know pulling away from closed and open [49:54] is the fact that the agents you know the [49:57] type of harnesses that are being used in [49:59] the closed labs are very different and [50:03] slash you know actually trained and you [50:05] know post-trained on to to you know to [50:08] to be specifically good for um their [50:10] models are specifically good for those [50:11] harnesses and so that in increases the [50:14] mystique even more makes it very hard um [50:16] how do you I guess one, do you agree? [50:18] And then two, like how are you sort of [50:20] hoping specifically to kind of address [50:22] that? And things I mean are really [50:24] examples like, hey, spin up like [50:25] thousands of agents, tens of thousands [50:26] of agents. It's just like a very [50:28] different workflow than what people are [50:30] are obviously um exposed to. [50:32] >> I think it'll be okay for open because I [50:35] thought years ago that say the closed [50:38] labs would be like Apple. There would be [50:39] the integrated company where they have [50:40] the model designed for the harness [50:42] involved. It gives them um like kind of [50:45] economy, scale, efficiency. But it [50:47] turned out that so many people have [50:49] different harnesses for different things [50:50] that they like to do that the models [50:52] companies have been pressurized to make [50:53] it so that their model can work in all [50:55] sorts of different harnesses. And I [50:56] thought it would be a weakness of open [50:58] models that they have to be designed to [50:59] train on all sorts of types of [51:00] harnesses. But it ended up being pretty [51:03] equilibriated or like it's at [51:04] equilibrium where they all have to do it [51:06] to some capacity. And then what we're [51:08] seeing with this like agent swarms is [51:10] that OpenAI and Enthrop are going to be [51:12] earlier to it. They're going to know how [51:13] to design their models. They're going to [51:15] know how to design what do we call what [51:17] do we call want to call a harness for a [51:18] swarm. I don't know like [51:22] I don't know but there's going to be a [51:23] word for it. It's not a harness. It's [51:25] like a coordination system. But then [51:27] eventually other people are going to [51:28] want to use them. And then I think it's [51:29] going to be like these cycles where the [51:30] Frontier Labs like they push this out [51:32] and they have this [51:34] >> to date they've had huge returns on [51:35] being first. like what is the returns on [51:37] Navier Stokes and all these things like [51:39] we don't really know like we don't know [51:41] they push the frontier [51:43] >> in the assumption that there will be [51:45] high returns and that this marginal [51:48] intent marginal intelligence has like [51:50] disproportionate returns and we don't [51:52] know if that's always true but I kind of [51:54] expect these cycles of like once you [51:55] know the shape of how do you control [51:57] 10,000 agents in one harness then the [51:59] open model companies be like oh we can [52:01] do this too [52:02] >> but it's not like Timmy is going to do [52:04] that first for one they don't have the [52:05] comput. [52:06] >> Yeah, [52:06] >> it's an astronomical amount of comput, [52:09] but these things go through cycles. [52:11] Nvidia's chips will get way faster. [52:13] There's more comput built out. [52:16] >> So, you're very convinced that like [52:18] there's not too much hidden there that [52:20] you can probably guess at what they're [52:22] doing and then when it becomes relevant, [52:24] let the open source models actually try [52:26] to also post train on similar [52:28] meta harnesses, whatever name. And I [52:30] also think like [52:32] >> the the security at these labs in terms [52:34] of information is not as high as they [52:36] think. Like Silicon Valley is famous for [52:38] like ideas traveling like what is it [52:40] like one beer or something and it's like [52:42] everybody chats after work [52:43] >> except which is why SSI is so [52:45] impressive. That's what's going on. [52:47] >> That's the only past. But it's like [52:49] we're all everyone's scientists. If they [52:51] made a breakthrough they're going to [52:52] want to tell their friends about it. [52:53] They might hold it. They might hold it [52:55] for a few months, but like [52:57] >> I don't think there the amount of lock [53:00] like information security you would need [53:02] is just so high that knowledge spreads. [53:05] It's hard to keep this secret. Like what [53:07] you what they can win on is putting more [53:12] inference into their infrastructure [53:13] which lets them serve at a lower price [53:15] and make higher margins and then [53:16] continue this over a longer time. So I'm [53:19] very confident in the frontier labs [53:20] being able to serve [53:22] the cheapest intelligence per [53:26] per watt per scoral. [53:29] >> They are going to be able to go to the [53:30] places with the highest demand and have [53:32] the like the cheapest models and that [53:35] will be revenue creative like that will [53:38] be good business for them. The question [53:39] is like they can't do everything and [53:41] that's where the rest of the economy [53:42] fills in is like [53:44] >> is it just coding that is profitable [53:47] then? Definitely not [53:49] >> good case for frontier models because [53:50] they will have the mo like they have [53:52] integration. [53:53] >> They'll do very well but it's coding [53:54] >> is the whole economy of knowledge work [53:56] is the market for things and then it [53:58] might not be good for them because can [54:00] they do everything at once [54:01] >> or they don't need to though. I think [54:02] there's going to be a lot of winners. [54:04] >> Yeah. [54:04] >> Yeah. I mean that's what I I mean that's [54:06] why we invest in the frontier models and [54:07] in startups you know um not because we [54:10] want to hedge but because I actually do [54:11] think you know to kind of cooperate your [54:13] view that there's enough [54:15] >> harder tasks at the fringe that requires [54:18] a very specialized data that's kind of [54:20] out of the splash zone for getting to [54:22] AGI or whatever you want to call it [54:24] these days um um that makes it possible [54:27] to have many many winners. Yeah, [54:29] >> I do like your RSI point as we talked a [54:31] little bit about it which is just kind [54:33] of say practically hey it's exponential [54:36] cost and resource consumption and [54:39] therefore there's a very obvious kind of [54:41] limit to how much that um the Frontier [54:45] Labs can pull away um if that's not too [54:48] oversimplification of your view um [54:51] anything else you kind of want to add [54:52] there because I I do feel like the the [54:54] the discourse now is just so so much [54:56] fear on like how how much the frontier [54:58] here is just hidden and also breaking [55:00] away. I think there's so much spend on [55:02] inference and that in the short term we [55:06] are learning how to [55:08] um corral inference into performance [55:11] gains and this is like a thing that we [55:13] just had and like [55:14] >> this is what we are seeing now is that [55:16] like people are spending so much and [55:19] >> like all the employees at the frontier [55:21] labs are spending exponential amounts [55:23] relative to the improvements but like [55:24] we're figuring out how to do this and [55:26] there will be a lot of gains but I don't [55:29] think that it's necessarily going to be [55:32] this like fast takeoff when the models [55:34] build themselves. It's human human [55:37] bottlenecks. The all the scaling laws [55:40] are exponential increase in compute for [55:42] linear increase in performance and exp [55:46] like that's just like it's hard for me [55:48] to bet against this. I would do say that [55:50] like we have Navier Stokes and I'm of [55:52] the opinion that math is exceptionally [55:56] well suited maybe like the scientific [55:58] field that is best suited to this shape [56:00] of intelligence and like intelligence [56:02] can take many different forms and that's [56:05] not to say that like if they do like if [56:07] an AI model does say discover something [56:10] that's very different than the [56:11] transformer that gives very different [56:13] properties and unlocks a new type of [56:15] intelligence like it is worth updating [56:17] on this [56:18] >> and worth being like okay like that is [56:20] different but until that happens I'm [56:22] like pretty happy with my evidence of [56:24] like the companies are going to do well [56:26] they're spending exponentially more to [56:28] keep this remarkable pace of progress [56:30] that some of the clawed system cards say [56:33] this they're like it would take say like [56:36] 10x more progress to make this a fast [56:38] takeoff and it's like for now they're [56:40] helping us sustain the pace of progress [56:42] and I think it's a good equil like it's [56:45] a good equilibrium but I do think that [56:46] like the labs have people saying the the [56:49] science is like it's hard to overcome [56:51] that and it's okay but it's like we [56:53] don't know exactly what it'll go from [56:56] here. [56:56] >> Yeah. Yeah. No, I I I think that's kind [56:58] of com comforting because we know what [56:59] the inputs and and lovers are here. It's [57:01] just there is a very successful auto [57:04] reggressive regime that we've been [57:05] optimizing and that one just you know [57:08] asks for a certain amount of like [57:09] exponential resources which are clearly [57:12] capped. Um and therefore open source [57:14] hopefully you know keeps it strong as [57:16] well. just it wouldn't lag behind. [57:18] >> There is dramatic amount of work to do [57:19] in preparing for infrastructure. Like [57:21] there's the AI capability plot which [57:22] doesn't necessarily worry me but the AI [57:25] capability plot in the pretty much [57:27] non-existing preparation of cross public [57:30] cyber security and all these things like [57:32] >> there needs to be more preparation in [57:34] the assumption that this is coming [57:37] >> than there is and then that's kind of [57:39] the thing that worries me more than say [57:41] like RSI is coming and unlocking new [57:44] risks like there are risks that are here [57:46] and that are going to continue to [57:47] diffuse and many times have said close [57:49] labs get there And then the open labs [57:51] get there. It's like we're in this [57:53] process for a lot of things that are [57:54] going to cause some problems. And it's [57:56] just like figuring out how to get [57:58] society to prepare is very hard. [58:00] >> Yeah. [58:01] >> And I come of the opinion that we need [58:03] more eyes. We need more science so that [58:05] society feels informed. [58:07] >> Yeah. [58:07] >> But there are other valid opinions. And [58:09] like I do think that not fearing fast [58:12] takeoff does not mean that you don't [58:15] worry about risks. I don't like you. [58:17] >> No, I think that that's completely Yeah. [58:18] Maybe I'll I'll send in uh some people [58:20] your way that are defected from the lab [58:23] worried precisely about this. Well, [58:25] thank you so much, Nathan. This was a [58:26] lot of fun and um yeah, I'm really [58:28] excited to see what Trillium is um up to [58:31] now that it's finally launched. [58:32] >> Yeah, thanks so much for having me.