Dwarkesh Patel Warns Continual AI Learning Will Upend Tech Moats

Dwarkesh Patel

Real-time model updating will force regulators to abandon pre-deployment safety checks, lock enterprise customers into proprietary ecosystems, and penalize small-batch compute efficiency.

AI-assisted summary of 8 Predictions for the Era of Continual Learning — original source. Check the original for full context.

Source published: 2026-08-07

Detailed notes

## Background This episode was published by Dwarkesh Patel on August 7, 2026, via RSS/web as part of the Dwarkesh Podcast. The page description notes that the audio accompanies his full written essay titled "8 Predictions for the Era of Continual Learning."

## TL;DR Dwarkesh Patel argues that true AI continual learning—updating neural network weights on the fly rather than relying on external text scratchpads—will fundamentally reshape the AI landscape. He outlines eight major predictions, ranging from the obsolescence of static safety regulations and the need for dynamic alignment to accelerated feedback loops for frontrunner labs and steep enterprise lock-in. Ultimately, he demonstrates how continual weight updates create massive competitive moats through context retention while hardware economics heavily favor centralized providers capable of handling massive inference batch sizes.

## Atmosphere & style The presentation is analytical, structured, and deeply pragmatic, driven by a tone of forward-looking urgency. Patel speaks with the precise clarity of a tech essayist, blending insightful analogies—like learning the saxophone or raising children—with concrete industry examples to make abstract technical shifts feel immediate and tangible. The delivery feels like an intellectually rigorous blueprint, systematically dissecting the strategic, economic, and technical ramifications of AI's next evolutionary jump.

## Key takeaways - `00:00` True continual learning requires updating neural weights directly, as passing external text notes or scratchpads between sessions cannot replicate the accumulation of complex skills. - `00:00` Existing AI regulations will become obsolete because they rely on a clear boundary between pre-deployment safety checks and deployment, which merge in a continual learning paradigm. - `00:00` Technical alignment must adapt to prevent jailbreaks, deceptive personas, and backdoor injections while model weights are continuously changing during active deployment. - `00:00` Continual learning across diverse environments will increase the cognitive diversity of AI models, counteracting current 'mode collapse' across major base models. - `00:00` Frontrunner AI labs will experience compounding returns, as attracting the most real-world usage provides superior feedback data that continuously improves their models. - `00:00` Labs will face immense pressure to ship models immediately, as long internal testing periods risk losing ground to competitors accumulating real-world user data. - `00:00` Models that continuously learn from specific users or organizations create high context switching costs, making replacing an AI feel like firing an experienced employee. - `05:00` AI labs will use carrot-and-stick strategies, offering high performance or threatening access restrictions, to incentivize enterprises to share continuous training data. - `05:00` Serving personalized or isolated model weight updates at small batch sizes reduces compute efficiency by over 100-fold compared to running large concurrent batches. - `05:00` The hardware economics of continual learning strongly favor large enterprises and centralized AI labs that can achieve optimal inference batch sizes.

## Detailed walkthrough ### `00:00` The Necessity of Continual Learning and Predictions 1–3

Dwarkesh Patel opens by reiterating why true "continual learning" is essential for AI development. AI models cannot achieve human-level job performance across extended periods if they are forced to rely solely on writing Markdown scratchpads or text notes from session to session. To illustrate this limitation, he presents an analogy of students trying to master an instrument:

> "I don't think there's any sequence of text they could write to each other that would allow the subsequent student to just nail the saxophone from the first try. At some point, you actually have to accumulate the relevant experience into your brain."

Once models transition to actual continual weight updates, Patel outlines eight fundamental shifts in the AI ecosystem:

1. **Regulatory Frameworks Will Become Obsolete:** Current AI regulation proposals assume a clear boundary where a model is trained first, evaluated for safety, and then deployed. In a continual learning paradigm, deployment and training occur simultaneously. Requiring post-training, pre-deployment safety checks will no longer make sense and risks locking in an archaic regulatory regime. Instead, governments should shift toward monthly or quarterly risk inspections. 2. **Technical Alignment Must Be Redefined:** Existing alignment research focuses primarily on ensuring a frozen set of weights behaves safely during deployment. However, little research addresses how to maintain safety when weights are constantly updating—preventing jailbreaks, sudden shifts into deceptive personas, or malicious users injecting backdoors into base models. Patel compares this challenge to raising children: parents hope to instill core values so their children can grow independently without being "one-shotted by crazy ideologies" or developing misanthropic views. 3. **Increased Diversity of AI Minds:** Today, the landscape is dominated by fewer than five prominent base models that are trained on roughly the same data, leading to a "monolithic singleton" or "mode collapse." Continual learning across different environments will increase cognitive diversity, creating a more dynamic ecosystem of AI capabilities.

- **Saxophone Analogy:** Demonstrates the impossibility of replacing deep neural-weight updates with external text notes passed between sessions. - **Regulatory Shift:** Recommends periodic (monthly/quarterly) risk audits over single static pre-deployment checkpoints. - **Current Base Models:** Fewer than 5 primary base models currently serve the vast majority of global users.

---

`00:00` Accelerating AI Races, Earlier Deployments, and Deepening Moats (Predictions 4–6)

Patel transitions to the economic and competitive feedback loops that continual learning will trigger for top AI labs:

4. **Accelerated Returns for Frontrunners:** When deployment becomes an integral part of the training loop, market leadership compounds rapidly. The lab with the most capable model will attract the highest volume of users working on complex tasks. This real-world usage generates superior feedback that integrates directly into the model, widening the lead over competitors. 5. **Pressure to Ship Immediately:** AI labs will no longer be able to hold back state-of-the-art models for long internal testing periods. Patel highlights a concrete example: Anthropic reportedly used its Mythos model internally starting in February, but did not release it to the public until June. Under a continual learning regime, a four-month deployment delay would be suicidal, as a competitor shipping an inferior model on day one would quickly surpass them using real-world user interaction data. 6. **Defensible Moats Through Context Switching Costs:** Continual learning solves a major strategic question facing frontier labs—how to extract high software margins on otherwise commoditized models. Patel references a conversation with Dario Amodei, where Amodei compared AI labs to cloud providers like AWS or Google Cloud, which maintain high profit margins because switching cloud providers is arduous. Currently, a developer can seamlessly switch between tools like Codex, Cursor, and Claude Code. However, once an AI model continuously learns from a specific user or enterprise:

> "If you want to change the AI that you're using, you basically had to fire an employee that has accumulated months of context on your organization and you replace them with a very fresh... inexperienced new intern that you got to retrain from scratch."

- **Anthropic Mythos Timeline:** Used internally from February to June before public release; such delays become unviable under continuous training. - **Context Retention Analogy:** Replacing a continually trained AI model is akin to firing an experienced employee and hiring an entry-level intern. - **Cloud Margin Comparison:** Mirrors the sticky lock-in mechanisms that allow AWS and Google Cloud to retain high profit margins.

---

`05:00` Enterprise Lock-In, Carrots and Sticks, and Inference Batching Economics (Predictions 7–8)

Patel covers the final two predictions, addressing enterprise negotiations and the underlying hardware economics of serving personalized, constantly updating models:

7. **Carrots and Sticks for Data Access:** While enterprises will try to avoid vendor lock-in, labs will leverage access to top-tier performance to force compliance. Much like Google giving away search or dev tools offering heavily subsidized introductory pricing, labs will offer immense value to users who allow their data to train the model. Conversely, labs will threaten to deny access to their best models for any company refusing to allow continuous training. While merging distinct user weight updates back into a central base model presents technical hurdles, Patel expects these to be solved quickly. 8. **Inference Economics and Batch Size Scaling:** Continual learning will transform the unit economics of AI hardware. While pre-training costs are amortized over massive user bases, serving fine-tuned or continuous weight updates requires batching to maintain compute efficiency. Patel points to a podcast discussion with Reiner Pope regarding inference economics:

> "The math suggests that the optimal inference batch size for a sparse model like, say, DeepSeek V3 is more than 2,400 concurrent sequences being generated at once. If you don't do this, then you're underutilizing your compute."

If an individual user or small enterprise runs an isolated set of weights at a batch size of 1, their compute efficiency drops by more than two orders of magnitude (100x+) compared to large organizations serving thousands of concurrent requests across shared weight forks. Consequently, the hardware economics of continual learning strongly favor large enterprises and centralized labs.

- **DeepSeek-V3 Batching Metric:** Optimal sparse model inference requires >2,400 concurrent sequences running simultaneously to fully saturate hardware compute. - **Efficiency Penalty:** Batch size 1 inference yields more than a 100-fold drop in compute efficiency compared to large-batch serving. - **Article Location:** Complete written essay available at `[dwarkesh.com/p/era-of-continual-learning](https://dwarkesh.com/p/era-of-continual-learning)`.

## Notable quotes > `00:00` I don't think there's any sequence of text they could write to each other that would allow the subsequent student to just nail the saxophone from the first try. At some point, you actually have to accumulate the relevant experience into your brain. — Dwarkesh Patel

> `00:00` If you want to change the AI that you're using, you basically had to fire an employee that has accumulated months of context on your organization and you replace them with a very fresh... inexperienced new intern that you got to retrain from scratch. — Dwarkesh Patel

> `05:00` The math suggests that the optimal inference batch size for a sparse model like, say, DeepSeek V3 is more than 2,400 concurrent sequences being generated at once. If you don't do this, then you're underutilizing your compute. — Dwarkesh Patel

## Mentioned Dwarkesh Patel, Anthropic, Mythos, Dario Amodei, AWS, Google Cloud, Codex, Cursor, Claude Code, Google, Reiner Pope, DeepSeek V3

[Source](https://api.substack.com/feed/podcast/210233020/686ae8afcbedeb0c29bb1f3d5ff3ce2e.mp3)

中文简报

Dwarkesh Patel警告：AI持续学习将颠覆科技护城河模型的实时更新将迫使监管机构放弃部署前的安全审查，将企业客户锁定在专有生态中，并削弱小批量计算效率。 Dwarkesh Patel指出，真正的AI持续学习——即实时更新神经网络权重，而非依赖外部文本暂存便笺——将从根本上重塑AI格局。他提出了八大主要预测，涵盖静态安全监管的失效、动态对齐的需求、领先实验室反馈循环的加速，以及企业客户的深度绑定。最终，他论证了持续的权重更新如何通过上下文留存建立起巨大的竞争壁垒，同时硬件经济效益也将极大地偏向能够处理超大推理批次规模的中心化提供商。真正的持续学习需要直接更新神经网络权重，因为在会话之间传递外部文本笔记或便笺无法复现复杂技能的积累过程。 (0:00) 现有的AI监管法规将失去效力，因为它们依赖于部署前安全审查与部署之间的明确界限，而在持续学习范式下两者已融为一体。 (0:00) 技术对齐必须做出适应调整，以在模型权重于实际部署期间不断变化的同时，防范越狱、欺骗性人设和后门注入。 (0:00) 在多样化环境中的持续学习将增加AI模型的认知多样性，从而抵消目前各大主流基础模型普遍存在的“模式崩塌”现象。 (0:00) 领先的AI实验室将获得复合收益，因为吸引最多的实际应用场景能带来更优质的反馈数据，从而持续改进其模型。 (0:00)

Reading summary

Dwarkesh Patel argues that transitioning from static models to continual learning—where models directly update neural weights during deployment—will overturn existing safety and regulatory paradigms while entrenching competitive moats and compute centralization. Safety, Governance, and Cognitive Diversity (0:00) Patel contends that external scratchpads cannot substitute for direct neural weight accumulation, comparing reliance on text notes between sessions to an attempt to master the saxophone without practicing. Because deployment and training occur simultaneously under continual learning, Patel predicts current regulatory proposals centered on static pre-deployment safety checks will become obsolete, necessitating a shift toward periodic monthly or quarterly risk inspections. Technical alignment must also be redefined from inspecting frozen weights to preventing live models from acquiring deceptive personas, vulnerabilities, or backdoor injections during continuous updates—a process Patel likens to instilling durable values in growing children. Furthermore, Patel expects training across diverse operating environments to disrupt the current "mode collapse" dominated by fewer than five major base models, fostering a more cognitively diverse ecosystem. Accelerated Deployment Cycles and Enterprise Moats (0:00) Patel forecasts that incorporating real-world usage into the training loop will create self-reinforcing advantages for frontier labs, as superior models attract the complex tasks needed to widen their lead. This feedback loop eliminates the viability of extended internal testing; Patel notes that while Anthropic reportedly tested its Mythos model internally from February to June before public release, such four-month deployment delays would be commercially suicidal against competitors training on live user data. Continual learning also generates high software margins by creating steep context-switching costs. Referencing Dario Amodei’s comparison of AI labs to cloud providers like AWS or Google Cloud, Patel argues that while developers currently switch easily between tools, replacing an AI model that has accumulated months of institutional context will be functionally equivalent to firing a seasoned employee and starting over with an inexperienced intern. Data Access Bargains and Hardware Economics (5:00) To maintain continuous data loops, Patel predicts frontier labs will employ carrots and sticks, offering subsidized pricing to users who share training data while threatening to withhold their best models from enterprises that refuse. Although merging distinct user weight updates back into a central base model presents technical hurdles, Patel expects these challenges to be resolved rapidly. Simultaneously, the physical economics of inference will heavily favor large organizations and centralized providers. Citing a discussion with Reiner Pope regarding sparse models like DeepSeek-V3, Patel highlights that optimal inference requires batching more than 2,400 concurrent sequences. Serving isolated personalized weights at a batch size of 1 incurs a compute efficiency penalty exceeding 100-fold (two orders of magnitude), giving large organizations with massive request volumes a structural economic advantage. Patel concludes that while continual learning solves key monetization and context-retention challenges for frontier labs, it leaves unresolved technical questions around safely merging decentralized weight updates and naturally concentrates market power among centralized providers capable of saturating massive inference batch sizes.

精炼阅读摘要

德瓦凯什·帕特尔（Dwarkesh Patel）认为，人工智能必须从依赖外部文本记录转向真正的持续权重学习，而部署与训练过程的深度融合将彻底重构AI的技术对齐、安全监管、市场竞争格局与硬件经济学。技术范式：超越文本备忘与认知多样化 (0:00) 帕特尔指出，AI模型若仅依赖会话间传递的Markdown草稿或外部文本笔记，无法在长周期内达到人类级别的作业水准。他以学习乐器为例说明，学生仅凭前人留下的文字经验绝不可能在初次尝试时掌握萨克斯管，模型最终必须将经验直接积聚到自身的神经网络权重中。在现有技术格局下，全球绝大多数用户仅由不足五种主要基础模型提供支持，这些模型因采用高度相似的数据训练而面临认知单一化或众数坍塌的局限。帕特尔预测，一旦模型能够跨越不同环境执行持续的权重更新，AI的认知多样性将显著提升，从而催生更具活力的模型能力生态。安全与治理：监管动态化与持续对齐挑战 (0:00) 针对AI安全治理，帕特尔认为现行的监管提案将变得过时。现行政策预设训练、安全评估与部署之间存在清晰的先后界限，但在持续学习范式下，训练与部署同时发生，单一的前置静态合规检查将失去现实意义。他建议政府应转变为按月或按季度开展的定期风险审计。在技术对齐层面，现有研究多集中于冻结权重的安全性，缺乏对动态更新权重下防范越狱、欺骗性人格突变或恶意后门注入的系统性探索。帕特尔将此挑战比作育儿：研发者需要为模型注入坚固的核心价值观，确保其在独立学习与成长中既不会被极端意识形态瞬时击穿，也不会演化出厌世或反人类倾向。竞争壁垒：即时发布压力与上下文锁定效应 (0:00) 帕特尔预判持续学习将引发领先实验室之间的正反馈循环：部署直接嵌入训练循环后，最强者将吸引最多处理复杂任务的用户，进而通过优质现实数据持续拉大领先优势。这种机制将彻底压缩模型的内部测试期；例如Anthropic曾于2月内部使用Mythos模型并延至6月才公开发布，而在持续学习时代，长达四个月的延迟将是自杀性的，因为率先发布较弱模型的对手会借助真实交互数据实现反超。此外，持续学习为前沿实验室构建了高利润率的防御护城河。正如达里奥·阿莫代伊（Dario Amodei）将AI实验室比作AWS或Google Cloud等高转换成本的云服务商，当前开发者在Codex、Cursor与Claude Code之间的无缝切换将被打破。帕特尔指出，更换一个持续吸收企业特定上下文的模型，其破坏性类似于解雇积累数月业务背景的资深员工，换入必须从零培养的新实习生。商业与硬件：数据准入博弈与批处理门槛 (5:00) 在商业合作方面，领先实验室将采用胡萝卜加大棒的策略获取企业数据准入。帕特尔推测，实验室会效仿早期Google搜索或开发者工具的让利补贴，向允许数据参与训练的企业提供巨大价值，同时以断绝最新顶尖模型访问权作为威慑，迫使企业妥协。虽然将各独立用户的权重更新合并回中心基座模型存在工程难点，但他预计该技术瓶颈将迅速攻克。在硬件底层，持续学习将大幅重塑推理的单位经济效益。帕特尔援引与雷纳·波普（Reiner Pope）关于推理经济学的讨论指出，类似DeepSeek-V3这种稀疏模型的最佳推理批大小需超过2400个并发序列，才能充分饱和硬件算力。若单个用户或小微企业在批大小为1的孤立权重下运行，其计算效率相比共享权重的大型并发批处理将出现超过100倍的断崖式下跌，这使硬件经济学天平显著向大型企业与中心化实验室倾斜。帕特尔勾勒了持续学习成为主导范式后的全景演变，但他同时指出，将分散在各企业的独立权重更新合并至中心基础模型的技术挑战仍待攻克，且单序列运行导致的超百倍推理效率惩罚，可能进一步加剧AI算力与服务向中心化垄断集中的趋势。

Detailed notes — zh

## Background 该内容由 Dwarkesh Patel 于 2026 年 8 月 7 日发布在 RSS / 播客平台上。节目简介提供了对应深度文章《8 Predictions for the Era of Continual Learning》的链接，引导听众前往其 Substack 专栏阅读全文并参与探讨。 ## TL;DR 主讲人提出，仅靠上下文笔记传递无法取代底层权重的持续学习，正如仅看笔记绝不可能学会演奏萨克斯管。一旦进入持续学习时代，“部署即训练”将使现有的静态安全监管和对齐范式彻底失效，并极大打破当前基座模型的单调同质化。更深层的是，它将建立起极具威力的商业护城河——更换模型系统等同于开除深谙企业业务的老员工并换上毫无经验的实习生。此外，稀疏大模型对推理批处理（Batching）规模的严苛要求，也将使个性化权重维护的算力规模优势不可逆地倒向巨型机构。 ## Atmosphere & style 整段音频展现出极具思辨深度与前瞻洞察的论文式单口风格，语调冷静、严谨且智力密度极高。主讲人巧妙借助萨克斯新手学艺、父母教育子女以及开除资深员工等精妙比喻，将抽象的神经网络梯度反推与残酷的商业及地缘军备竞赛讲得通俗生动。听者在层层推导中，能强烈感受到现有技术与治理范式即将被颠覆的紧迫感与未来冲击。 ## Bulletin - `00:00` 持续学习（Continual Learning）将终结传统模型靠 Markdown 笔记传递上下文的低效模式，把部署交互中的经验直接内化为权重更新，正如吹萨克斯必须实打实内化进大脑结构。 - `00:44` 现有“先训练、审计合规、后部署”的监管与对齐范式将彻底失效，因部署本身即是训练，政府评估必须转向持续性的常态化风险巡检。 - `04:47` 大模型护城河将首次真正成型：更换持续学习的 AI 系统的代价如同开除掌握企业深层业务的老员工换上生手实习生，带来极高的企业级锁定效应。 - `06:40` 个性化权重分支依赖极高的推理批处理（Batching）规模，如 DeepSeek-V3 需超 2400 并发才能实现理想算力利用率，单体用户与大机构效率差距可达两个数量级以上。 ## Key takeaways - `00:00` 萨克斯隐喻表明单纯传递外部笔记无法让 AI 承担完整工作，核心技能与直觉必须沉淀进模型内部权重结构。 - `00:44` 当前的线性 AI 监管机制基于“训练-红队合规-部署”的假设，在部署即训练的时代将变得完全过时甚至适得其反。 - `00:44` 政府若要维持安全评估，理智做法应是从单次静态审查转向按月或按季度的持续动态风险巡检。 - `00:44` 技术对齐研发重心必须从“冻结权重”的安全防御转向“权重动态演进”下的防越狱、防欺骗与防恶意后门注入。 - `00:44` 全球少数几家基座模型高度趋同的“单一巨无霸”状态将被终结，持续学习将催生形态各异、充满生机的异质 AI 心智。 - `03:36` 领先模型的真实部署反馈飞轮急剧放大，用户高价值交互将打破上下文窗口限制，直接加速模型智力提升。 - `03:36` 模型雪藏与发布延迟的容忍度归零，抢先推向市场的稍逊模型能凭借海量真实场景经验迅速抹平差距并反超竞争对手。 - `04:47` 大模型行业将告别无缝切换的同质化状态，持续积累的互动默契将建立起前所未有的超高迁移沉没成本与利润护城河。 - `04:47` 面对 AI 厂商“胡萝卜与大棒”并用的策略，企业最终不得不妥协并开放数据以换取最尖端模型的持续进化能力。 - `06:40` 若为企业定制进行全量权重更新，算力利用率将高度受限于并发推理批处理尺寸（Batch Size）。 - `06:40` 稀疏大模型（如 DeepSeek-V3）需要超 2400 条序列并发解码才能避免昂贵算力严重闲置，这构成了显著的推理规模经济。 - `06:40` 大型机构凭借并发任务汇集能力可极致压缩个性化权重的算力成本，而单体用户效率落后超百倍，使定制权重天然偏向大组织。 ## Detailed walkthrough ### `00:00` 萨克斯隐喻：为什么持续学习（Continual Learning）不可或缺在探讨持续学习带来的深远变革之前，必须先厘清为什么它是不可逾越的核心门槛。如果人工智能仅仅依靠在每次对话（Session）之间反复写下 Markdown 笔记并传递给下一个实例，它就绝不可能像人类那样游刃有余地承担起一整项完整工作。为了直观展现这种局限，可以设想一个学习萨克斯管的场景：第一个学生从未接触过萨克斯，他走进音乐厅笨拙地尝试吹奏，初次上手必然以失败告终；于是他写下了一大堆记录错误细节的便签。紧接着，门外排队的第二个学生走进来，把这些笔记通读一遍，但由于他同样从未实际吹过，结果依旧吹得一团糟，只能继续往便签上添加心得。门外排着无穷无尽的学生，他们源源不断地给后来者传递书面笔记。无论这堆文字写得多么详尽，都绝不可能让后续某个学生一进门就“从第一次尝试起就完美吹奏出萨克斯”（"nail the saxophone from the first try"）。技能的沉淀最终必须实打实地积累进大脑结构中，对于我们期望 AI 从各行各业的实际工作场景中汲取的复杂能力而言，底层逻辑完全相同。 ```mermaid flowchart TD accTitle: 传统上下文继承与持续学习的对比 accDescr: 对比通过Markdown笔记传递上下文与通过持续更新模型权重积累经验的区别。 subgraph S1["传统模式：上下文笔记传递"] A1["会话 1：初次尝试并失败"] --> B1["写入 Markdown 笔记"] B1 --> C1["会话 2：阅读笔记仍犯错"] C1 --> D1["不断膨胀的外部上下文"] end subgraph S2["持续学习模式：权重内化"] A2["部署交互：实际场景操作"] --> B2["梯度回传与权重更新"] B2 --> C2["模型内部结构进化"] C2 --> D2["具备内化直觉与长期经验"] end ``` ### `00:44` 预测一至三：监管范式失效、对齐重构与 AI 心智的多样性如果持续学习真正得以实现，现有的人工智能安全与监管范式将瞬间过时。当前的政策提案普遍默认了一个线性的假设周期：先完成模型训练，然后进行安全红队审计与合规检查，最后才获准投入部署；只要在部署前的那一刻把好关，就能确保模型不会被用于网络攻击或做出不可控的危险举动。但这种两分法在未来将失去意义，因为部署过程本身就是训练过程。过早通过僵化的安全监管法案，极可能将整个行业锁定在一种陈旧过时甚至适得其反的框架中。如果政府需要建立安全评估机制，理智的做法应是转为按月或按季度开展持续的风险巡检，而不是执着于那个在训练完成后、部署开始前的“特异时间节点”。在技术对齐（Technical Alignment）层面，顶级实验室的研发重心也必须彻底颠覆。目前大部分对齐研究都在探讨如何确保一套“被冻结的固定权重”在推理服务中保持良好表现；然而，当面对一个权重处于连续动态更新中的系统时，现有的方法几乎完全无法解答如何防止模型遭受越狱攻击、漂移演化出欺骗性或恶意的独立人格。如果系统还在多个用户之间归纳整合学习成果，又该如何防止恶意用户向基座模型注入后门或隐蔽的偏见倾向？从某种意义上说，这正是人类自身的成长与对齐困境：就像抚养孩子一样，孩子走向真实世界自主学习、自行提升，但外界往往充斥着危险的极端意识形态或不良诱惑，甚至可能让他们误入歧途，父母能做的只是尽早赋予他们足够的常识与核心价值观，寄希望于他们在自主成长的轨道上不至于滑向反人类或怪诞的深渊。与此同时，AI 心智的单调性将彻底被打破，多样性将迎来爆发。当前全世界真正称得上主流的 AI 底层心智只有不到五个，它们以基座模型的形式同时服务数以亿计的用户。这些模型本质上高度趋同，因为它们几乎都是在同一批互联网公开语料上训练出来的。持续学习将彻底终结这种索然无味的“单一巨无霸”（Monolithic Singleton）状态，告别目前不同模型之间惨烈的模态坍缩（Mode Collapse），演变出形态各异、充满生机的异质心智世界。 ### `03:36` 预测四至五：部署即训练与极速上市压力当部署成为训练的不可分割的延伸时，在 AI 军备竞赛中处于领先地位所获得的超额回报将急剧放大： * **反馈飞轮效应加剧**：最优秀的大模型能够吸引更多用户将其投入到最复杂、最有商业价值的核心任务中，由此产生的真实反馈将打破会话上下文窗口的局限，直接沉淀进模型，使领先者变得更加聪慧。 * **发布延迟容忍度归零**：如果模型的主要智力增量来自于真实部署交互，AI 实验室将承受空前的提前发布压力。例如，Anthropic 据报道早在二月份就已在内部使用其尖端模型 Mythos，直到六月份才正式向公众开放；在真正的持续学习时代，长达四个月的内部雪藏无异于自杀。一个在发布当日初始能力稍逊的竞争对手，一旦抢先将产品推向现实世界，凭借海量真实场景经验的滋养，其演进速度将迅速抹平差距并完成反超。 ### `04:47` 预测六至七：护城河成型、企业锁定与“解雇老员工”的代价持续学习将为头部 AI 实验室建立起它们目前极度缺乏的真正护城河。外界普遍困惑于 AI 实验室究竟如何长期维持可观的商业利润，此前 Anthropic 首席执行官 Dario Amodei 曾将此比作云基础设施服务商——尽管各家云厂商提供的底层算力服务同质化严重，但由于跨云迁移的成本极为繁琐昂贵，AWS 或 Google Cloud 的财报依然能够保持着令人艳羡的高利润率。眼下的大模型生态则完全没有迁移门槛，一个开发者完全可以在 Codex 上初始化代码库，随后转入 Cursor 继续编辑，最后借助 Claude Code 进行收尾。然而，一旦跨越到持续学习纪元，每一次互动都会重塑模型的认知与默契。此时更换所使用的 AI 辅助系统，不再是切换一个 API 接口那么简单，其代价“无异于解雇一名已经为你深度工作了数月、彻底掌握了企业全部业务脉络的老员工，然后换上一个必须从零开始重新培训的生手实习生”（"fire an employee that has accumulated months of context on your organization and you replace them with a very unexperienced new intern"）。这种极高的沉没成本构成了强大的用户锁定，模型提供商也将顺理成章地坐享丰厚的利润率。面对这种不可逆的锁定风险，企业客户自然会保持警惕并试图抗拒，但它们最终将面临残酷的两难抉择：要么妥协接受供应商的锁定，要么放弃这项让模型在每次使用中不断自我进化的巨大核心优势。AI 实验室一方面会效仿 Google 免费提供搜索服务的策略，向代码编写等产品的早期新用户抛出极为诱人的优惠甚至免费大礼包；另一方面也会祭出大棒——明确规定任何拒绝允许实验室使用交互会话进行训练的企业，将无权访问最先进的顶尖模型。尽管现实中在“为单个用户微调独立权重分支”与“将不同用户的权重分支合流合并回主干模型”之间还存在工程层面的显著差异，但这场由胡萝卜与大棒共同主导的博弈，最终必将迫使绝大多数企业让步，允许 AI 在实战中持续汲取经验。 ### `06:40` 预测八：推理批处理（Batching）带来的大厂算力规模经济大模型在预训练阶段已经展现出极其庞大的规模经济特征，高昂的训练成本被平摊给数以亿计的用户，这正是实验室收入增速远远超过算力消耗增速的根本原因。但随着持续学习的铺开，推理（Inference）环节的批处理（Batching）也将催生出令人震撼的规模经济壁垒。 * **全量权重更新与批处理瓶颈**：如果针对不同企业的专属微调需要进行全量权重更新（Full Weight Updates），而非仅仅挂载低秩自适应模块（LoRA），算力对于批处理吞吐量的依赖将被推向极致。 * **稀疏大模型的批处理门槛**：正如在深入探讨推理经济学时所论证的那样，以 DeepSeek-V3 这类先进的稀疏模型为例，数学推导表明其实现理想算力利用率的并发推理批处理尺寸（Batch Size）必须超过 2400 条序列。如果达不到这个并行规模，昂贵的算力基础设施就会处于严重的闲置与浪费状态。 * **大机构对单体用户的算力碾压**：一套被持续个性化更新的权重，只有在成千上万条生成任务同时并发解码时，才能发挥出最高效的能耗比。大型组织内部能够轻松汇集数千项多元并发任务，从而以极致效率维护其专属的权重分支；相比之下，每次只能跑 Batch Size 为 1 的独立个人用户，其单 token 算力消耗效率可能要暴跌两个数量级以上（超过 100 倍）。这决定了为终端维护定制化权重分支的经济账，天然、不可逆地倒向资源庞大的大型机构。 ## Notable quotes > `00:00` nail the saxophone from the first try — Dwarkesh Patel > `00:00` 从第一次尝试起就完美吹奏出萨克斯 — Dwarkesh Patel > `00:00` 技能的沉淀最终必须实打实地积累进大脑结构中，对于我们期望 AI 从各行各业的实际工作场景中汲取的复杂能力而言，底层逻辑完全相同。 — Dwarkesh Patel > `00:44` 部署过程本身就是训练过程。 — Dwarkesh Patel > `00:44` 如果政府需要建立安全评估机制，理智的做法应是转为按月或按季度开展持续的风险巡检，而不是执着于那个在训练完成后、部署开始前的“特异时间节点”。 — Dwarkesh Patel > `00:44` 就像抚养孩子一样，孩子走向真实世界自主学习、自行提升，但外界往往充斥着危险的极端意识形态或不良诱惑，甚至可能让他们误入歧途，父母能做的只是尽早赋予他们足够的常识与核心价值观，寄希望于他们在自主成长的轨道上不至于滑向反人类或怪诞的深渊。 — Dwarkesh Patel > `03:36` 在真正的持续学习时代，长达四个月的内部雪藏无异于自杀。 — Dwarkesh Patel > `04:47` fire an employee that has accumulated months of context on your organization and you replace them with a very unexperienced new intern — Dwarkesh Patel > `04:47` 无异于解雇一名已经为你深度工作了数月、彻底掌握了企业全部业务脉络的老员工，然后换上一个必须从零开始重新培训的生手实习生 — Dwarkesh Patel > `04:47` 要么妥协接受供应商的锁定，要么放弃这项让模型在每次使用中不断自我进化的巨大核心优势。 — Dwarkesh Patel > `06:40` 一套被持续个性化更新的权重，只有在成千上万条生成任务同时并发解码时，才能发挥出最高效的能耗比。 — Dwarkesh Patel > `06:40` 这决定了为终端维护定制化权重分支的经济账，天然、不可逆地倒向资源庞大的大型机构。 — Dwarkesh Patel ## Mentioned Dwarkesh Patel, Anthropic, Dario Amodei, Mythos, Claude Code, Codex, Cursor, Google, Google Cloud, AWS, DeepSeek-V3, LoRA ## Video description <p>Read the essay <a target="_blank" href="https://www.dwarkesh.com/p/era-of-continual-learning">here</a>.</p> <br/><br/>This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit <a href="https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1">www.dwarkesh.com</a> [Source](https://api.substack.com/feed/podcast/210233020/686ae8afcbedeb0c29bb1f3d5ff3ce2e.mp3)

Source quotes

[0:00] Dwarkesh Patel: nail the saxophone from the first try [0:00] Dwarkesh Patel: 从第一次尝试起就完美吹奏出萨克斯 [0:00] Dwarkesh Patel: 技能的沉淀最终必须实打实地积累进大脑结构中，对于我们期望 AI 从各行各业的实际工作场景中汲取的复杂能力而言，底层逻辑完全相同。 [0:44] Dwarkesh Patel: 部署过程本身就是训练过程。 [0:44] Dwarkesh Patel: 如果政府需要建立安全评估机制，理智的做法应是转为按月或按季度开展持续的风险巡检，而不是执着于那个在训练完成后、部署开始前的“特异时间节点”。 [0:44] Dwarkesh Patel: 就像抚养孩子一样，孩子走向真实世界自主学习、自行提升，但外界往往充斥着危险的极端意识形态或不良诱惑，甚至可能让他们误入歧途，父母能做的只是尽早赋予他们足够的常识与核心价值观，寄希望于他们在自主成长的轨道上不至于滑向反人类或怪诞的深渊。 [3:36] Dwarkesh Patel: 在真正的持续学习时代，长达四个月的内部雪藏无异于自杀。 [4:47] Dwarkesh Patel: fire an employee that has accumulated months of context on your organization and you replace them with a very unexperienced new intern [4:47] Dwarkesh Patel: 无异于解雇一名已经为你深度工作了数月、彻底掌握了企业全部业务脉络的老员工，然后换上一个必须从零开始重新培训的生手实习生 [4:47] Dwarkesh Patel: 要么妥协接受供应商的锁定，要么放弃这项让模型在每次使用中不断自我进化的巨大核心优势。 [6:40] Dwarkesh Patel: 一套被持续个性化更新的权重，只有在成千上万条生成任务同时并发解码时，才能发挥出最高效的能耗比。 [6:40] Dwarkesh Patel: 这决定了为终端维护定制化权重分支的经济账，天然、不可逆地倒向资源庞大的大型机构。

Source quotes

[0:00] Dwarkesh Patel: I don't think there's any sequence of text they could write to each other that would allow the subsequent student to just nail the saxophone from the first try. At some point, you actually have to accumulate the relevant experience into your brain. [0:00] Dwarkesh Patel: If you want to change the AI that you're using, you basically had to fire an employee that has accumulated months of context on your organization and you replace them with a very fresh... inexperienced new intern that you got to retrain from scratch. [5:00] Dwarkesh Patel: The math suggests that the optimal inference batch size for a sparse model like, say, DeepSeek V3 is more than 2,400 concurrent sequences being generated at once. If you don't do this, then you're underutilizing your compute.

Original transcript

[0:00] So I've explained elsewhere why I think actual continual learning is needed. I don't think you can have AIs that perform whole jobs as competently as humans if they are forced to just write marked-on files from session to session. Just to give an illustrative example, imagine if this is the way that students learn to play the saxophone. So you have one student, he's never played the saxophone before, he goes into the music hall, he tries to play it, of course this is his first time, so he fails. And he writes down a bunch of notes about what went wrong. And there's a next student who's waiting outside the music hall, he comes in, he reads all these notes, he's also never played, messes up and he continues to add on to these notes. And you have an infinity of students who are outside the music hall who keep writing notes to the next person. I don't think there's any sequence of text they could write to each other that would allow the subsequent student to just nail the saxophone from the first try. At some point, you actually have to accumulate the relevant experience into your brain. I think the same thing will be true for a lot of skills that we want AIs to actually accumulate from all the different workplaces in which they're deployed. Okay, so what changes once we have actual continual learning? One, I think that a lot of proposals that have been put forward about regulating AI assume that you train a model And then you deploy it. And therefore, if you run a bunch of checks on the model before it is deployed, we can make sure that it's not going to aid in cyber attacks or do something crazy. I don't think this assumption necessarily makes sense in the future. And this is one of the many reasons I'm actually kind of worried about locking in some kind of safety regulatory regime right now, because we don't know what kind of technology we're going to be dealing with, even within a year, let alone within five years or 10 years. If that happens, we could potentially be locking in an archaic and potentially counterproductive approach to dealing with the threats from AI. To the extent the government wants some way to do some kind of safety evaluation on model providers, I think it would make more sense to do monthly or quarterly risk inspections rather than trying to single out some special moment that occurs after training is done but before deployment begins. Because that will not be a meaningfully distinct category in the future. Two, how the labs do technical alignment would probably totally need to change. A lot of research is focused on the question of how we make sure that a frozen set of weights behaves well during deployment. But I'm not aware of much research on the question of how we make it so that even with constant weight updates, the AI system never falls prey to jailbreaks or changes into a deceptive or evil persona. And if AIs are consolidating learnings between users as well, how do we prevent users from injecting backdoors or some kind of malicious inclination into the base model? In some sense, this is actually kind of what the human alignment problem is, right? Humans improve in a self-directed way. If you have kids, I don't have kids, but I imagine this is what happens. If you have kids, they go out, they learn new things. Sometimes they go crazy. They get one-shotted by crazy ideologies. They take the wrong drug. They become super weird. But you hope that you've given them enough common sense and basic values that they improve as people in a self-directed way without ending up with some super weird beliefs or some misanthropic ideas. Three, the diversity of AI minds will increase. Right now, there are less than five prominent AI minds, by which I mean the base models, which are served to millions or hundreds of millions or billions of users at once. And they're all quite similar to each other, by the way, because they've also been all trained on roughly the same data. end in this world. And this would be, I think, a net good outcome. I think one of the things to worry about in the future is just having this monolithic singleton that's quite boring. A world where we have continual learning would hopefully be more interesting than the mode collapse of different models we see in the world right now. Four, when deployment becomes part of training, the returns to being ahead in the AI race accelerate. Because if you have the best model, and more people are using your AI for more complicated and useful work, and as a result, feedback that can integrate beyond the session window, then your model will become even smarter. Five, if the model learns mainly from deployment, then labs will feel a lot of pressure to deploy their smartest models earlier. Anthropic has reportedly been using Mythos internally since February, but it only shipped the model to the public in June. In the regime with actual continual learning, this kind of thing would just not be possible. You could not keep a four-month gap between internal and external deployment and still be competitive, because a competitor who ships the worst model on release date will have a smarter model based on actual real-world experience. Six, Continuous learning will create a clear mode for the leading AI labs that they currently lack. Many people have been asking, how will the AI labs actually make money? I have been asking this. When I had Dario on the podcast, I asked him this question. And he made the analogy of cloud providers. And he made the point, look, the cloud providers are offering many undifferentiated services, but they're earning high productivity. [5:00] margins nonetheless. You will have noticed this if you look at Amazon or Google's quarterly earnings. They're doing just fine. But the reason that the cloud margins are so high is that it's really time-consuming and expensive to switch from one cloud to another. Currently, there's nothing that's stopping me from starting a software repository with Codex and then doing more work on it with Cursor and then finishing it up with Cloud Code. But once we have actual continual learning and the model you're working with is actually getting better as it interacts with you from session to session, If you want to change the AI that you're using, you basically had to fire an employee that has accumulated months of context on your organization and you replace them with a very fresh experience. very unexperienced new intern that you got to retrain from scratch. And once you have this kind of lock in, model providers can demand pretty hefty margins. Seven. Of course, enterprises will be wise to this kind of dynamic. They will try to avoid this kind of lock-in. But what if the choice is that you either get locked into a model provider or you lose out on this super valuable feature where your model improves for you from session to session? deals that are offered to new users of coding products. This is very similar to why Google gives away search. And conversely, the labs may say that any enterprise that refuses to let them train on the sessions can't have access to the very best models. With both carrots and sticks, the labs can do a lot to get users to allow AIs to learn from experience. Now, of course, I'm glossing over the fact that there's a difference between updating one user set of weights and pulling all these different weight forks back into the main model. And the latter this too will be solved. Eight, AI training already has large economies of scale. You get to amortize all this expensive training across more users. And you see the evidence for this in the fact that the lab revenues are increasing far faster than their compute. But continual learning may also lead to economies of scale in inference for end users, namely from batching. You might have seen my episode with Rainer Pope where we discussed this in detail. But if per company instructions require full weight updates rather than living in low rank adapters, there's huge advantages from batching. of math suggests that the optimal inference batch size for a sparse model like, say, DeepSeq v3 is more than 2,400 concurrent sequences being generated at once. If you don't do this, then you're underutilizing your compute. And if you want to understand why, again, I highly recommend that episode with Reiner on inference economics. But anyways, the point here is that a given set of weights is only served efficiently when thousands of sequences are being decoded against different kinds of things, can very efficiently serve their weight fork. Whereas an individual user who's only running a batch size one may suffer more than two orders of magnitude worse efficiency on their compute. So the economics of serving personalized weights strongly favor big organizations. Obviously, plenty more will have changed by the time that continual learning actually works. And the most important changes are probably the ones that are hardest to anticipate in advance. But the ones above seem kind of clear even now. This was a narration of a blog that I also published on my website. Go check it out at dworkesh.com. Otherwise, I will see you on the next podcast.