Dwarkesh Patel Says Continual AI Learning Obsoletes Pre-Deployment Safety Checks

8 Predictions for the Era of Continual Learning — Dwarkesh Patel

Dwarkesh Patel

Artificial intelligence models that constantly update weights in real time will force regulators to shift to recurring inspections while giving early-moving developers unbeatable customer lock-in.

AI-assisted summary of 8 Predictions for the Era of Continual Learning — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published by Dwarkesh Patel on YouTube on August 7, 2026, this video outlines key insights from his essay on the shift toward real-time model updates. The page description points viewers directly to the full essay hosted on dwarkesh.com.

## TL;DR Dwarkesh Patel presents eight predictions on how continual learning will reshape artificial intelligence, moving the industry beyond static weights and text-file memory. He details how real-time parameter updates render pre-deployment safety checks obsolete, present new technical alignment challenges, and foster greater architectural diversity. Furthermore, Patel argues that continuous feedback loops will shorten deployment windows, establish massive vendor lock-in, push labs toward data-sharing subsidies, and structurally favor enterprise scale due to batching economics.

## Atmosphere & style The presentation is analytical, forward-looking, and intellectually rigorous, combining high-level conceptual framing with pragmatic economic logic. Patel maintains a calm, authoritative tone, using vivid analogies—like learning the saxophone or raising children—to make complex technical shifts intuitive. Listening to the video feels like an engaging conceptual briefing, balancing technical caution with a clear-eyed perspective on AI industry incentives.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=iewm45atodE&t=0s) Current AI relies on passing text notes between sessions, which fails to capture the complex, hands-on capabilities required for real-world tasks. - [00:00](https://www.youtube.com/watch?v=iewm45atodE&t=0s) True continual learning requires AI models to update their parameters directly rather than relying on external scratchpads. - [01:01](https://www.youtube.com/watch?v=iewm45atodE&t=61s) Real-time model updates make pre-deployment safety evaluations obsolete, requiring a shift toward periodic government inspections. - [02:02](https://www.youtube.com/watch?v=iewm45atodE&t=122s) Technical alignment must evolve from static weight safety to maintaining values during continuous updates to prevent jailbreaks and deceptive behavior. - [03:04](https://www.youtube.com/watch?v=iewm45atodE&t=184s) Continual learning will lead to greater AI diversity, as individual model instances diverge based on localized user experiences. - [03:51](https://www.youtube.com/watch?v=iewm45atodE&t=231s) Integrating live deployment into the training loop accelerates returns to scale, creating a virtuous feedback loop for early market leaders. - [03:51](https://www.youtube.com/watch?v=iewm45atodE&t=231s) Holding back AI models for months of internal testing is no longer viable, as competitors gain advantage through live data collection. - [04:38](https://www.youtube.com/watch?v=iewm45atodE&t=278s) Customized continual learning models build deep organizational context, creating strong vendor lock-in and defensive moats. - [05:56](https://www.youtube.com/watch?v=iewm45atodE&t=356s) AI providers will subsidize services for enterprises willing to share session data, while potentially restricting access to top models for those who refuse. - [06:52](https://www.youtube.com/watch?v=iewm45atodE&t=412s) High batch size requirements for customized weight updates create massive inference economies of scale that favor large enterprise deployments over individual users.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=iewm45atodE&t=0s) The Necessity of Continual Learning Over Text Notes

Dwarkesh Patel opens by addressing why traditional session-based memory is insufficient for artificial intelligence to master real-world tasks. Current systems rely on passing context across sessions through text files, such as writing Markdown notes, but this setup fails when attempting to capture complex, hands-on capabilities. Patel illustrates this limitation through an extended metaphor about learning to play the saxophone: if an endless line of novice students enters a music hall, plays poorly, writes down notes on what went wrong, and hands them off to the next beginner in line, no amount of written advice will ever enable a future student to execute the performance perfectly on their first try.

To bridge the gap between simple text-passing and genuine competence, models must actually accumulate physical or experiential knowledge directly within their neural architecture. For AI models to perform complete jobs at human-equivalent proficiency across varied workplaces, they must undergo true continual learning—updating their parameters rather than relying on external scratchpads.

- Current AI agents rely on writing Markdown files between sessions to pass context, which fails for complex tasks. - The saxophone analogy illustrates that reading previous notes cannot replace the physical accumulation of experience in a brain or set of weights.

---

[01:01](https://www.youtube.com/watch?v=iewm45atodE&t=61s) 1. Pre-Deployment Regulation Becomes Obsolete

The shift to real-time model updates dismantles current regulatory frameworks that rely on evaluating a static model before release. Most policy proposals assume a clean boundary where a model is trained, evaluated for safety threats like cyberattack capabilities, and then deployed. In a continual learning paradigm, this distinction disappears because a model actively updates its weights every day based on millions of user interactions.

Attempting to enforce static safety checks risks locking in an obsolete, counterproductive regulatory regime. Patel warns against prematurely committing to rigid safety frameworks today when the nature of the technology will change drastically within one to ten years. A more practical regulatory approach would involve periodic government risk inspections on a monthly or quarterly basis, rather than relying on a singular pre-deployment checkpoint.

- Traditional safety evaluations assume a distinct boundary between training and deployment. - Continual learning models improve daily based on millions of live work sessions, rendering pre-deployment checks irrelevant. - Patel suggests switching to recurring monthly or quarterly risk inspections for model providers.

---

[02:02](https://www.youtube.com/watch?v=iewm45atodE&t=122s) 2. Technical Alignment Must Adapt to Moving Weights

Technical alignment efforts currently focus on ensuring that a frozen, static set of weights remains well-behaved during deployment. However, little to no research exists on keeping a model aligned while its weights undergo continuous updates. This dynamic exposes models to risks like jailbreaks, persona shifts toward deceptive or evil behaviors, and malicious user exploitation.

Patel draws an explicit parallel to raising human children. Parents hope that by instilling core values and common sense, their children will navigate the world without falling prey to radical ideologies, harmful drugs, or misanthropic worldviews during self-directed learning. Alignment for continual learning systems requires a similar foundation of robust core principles to prevent models from succumbing to adversarial inputs or bad actors injecting backdoors.

- Existing alignment research targets static, frozen weights rather than continuously updating models. - Open risks include users injecting backdoors or coaxing updating models into deceptive personas. - The challenge mirrors parenting: instilling foundational values so the entity learns safely and independently.

---

[03:04](https://www.youtube.com/watch?v=iewm45atodE&t=184s) 3. Increased Diversity Among AI Minds

The current AI landscape is dominated by fewer than five prominent base models serving millions or billions of users, all exhibiting similar behaviors due to shared training datasets. Continual learning introduces a stark divergence from this uniformity, as individual model instances branch off by learning from unique user experiences and localized environments.

This shift avoids a monolithic, uniform "singleton" intelligence and prevents the mode collapse currently seen across top-tier models. Different instances and deployments will specialize and develop distinct characteristics, producing a richer and more varied ecosystem of AI capabilities.

- Currently, fewer than five major base models dominate, all sharing similar architectures and training data. - Continual learning will allow different instances of the same model to diverge based on real-world experience. - The resulting diversity avoids a single, monolithic AI regime and reduces model uniformity.

---

[03:51](https://www.youtube.com/watch?v=iewm45atodE&t=231s) 4 & 5. Accelerated Returns to Scale and Shorter Deployment Windows

As live deployment becomes an active component of the training loop, market dynamics will aggressively favor early movers. A superior model that attracts higher usage receives richer, more complex feedback, creating a virtuous feedback loop that compounds its intelligence advantage over competitors.

This structural shift eliminates the strategy of holding back models for extensive internal testing. Patel points to Anthropic internally using "Mythos" since February before releasing it publicly in June. Under a continual learning regime, a four-month delay would be catastrophic; a rival releasing a slightly weaker model immediately would rapidly surpass the unreleased model by absorbing live, real-world experience from users.

- **Prediction 4:** Higher usage generates richer feedback data, accelerating the intelligence gains of market leaders. - **Prediction 5:** Extended internal model holding periods become unviable; Anthropic's four-month internal hold on Mythos before its June release would allow competitors to pull ahead via live data collection.

---

[04:38](https://www.youtube.com/watch?v=iewm45atodE&t=278s) 6. Creating High Switching Costs and Provider Moats

Unlike software services that struggle with defensibility, continual learning establishes a defensible business moat for AI developers. Patel recalls asking Anthropic CEO Dario Amodei how AI labs plan to sustain high profit margins. Amodei drew a comparison to cloud computing platforms, which maintain lucrative margins despite offering largely commodity infrastructure because migrating away from them is prohibitively difficult.

While developers can easily swap tools like Codex, Cursor, and Claude Code today without friction, a continually updating AI builds cumulative organizational context over months of collaboration. Replacing a customized model is equivalent to "firing an employee that has accumulated months of context... and replacing them with a very fresh, very unexperienced new intern that you've got to retrain from scratch." This lock-in grants model providers substantial pricing power.

- Dario Amodei previously compared lab margins to cloud providers, where high switching costs protect profits. - Switching away from a personalized AI model forces a company to abandon months of contextual learning, forcing them to start over with an inexperienced "fresh intern." - This dynamic creates defensible moats and allows labs to charge premium margins.

---

[05:56](https://www.youtube.com/watch?v=iewm45atodE&t=356s) 7. Subsidies, Data Access, and Weight Consolidation

Enterprise clients will recognize the threat of vendor lock-in, but the necessity of using self-improving models will force compliance. To secure valuable training data from real-world usage, labs will offer heavy subsidies to enterprises that allow their sessions to be used for model training—a strategy similar to how coding startups offer aggressive discounts or how Google offers free search to capture user data.

Conversely, enterprises that refuse to share their session data may be restricted from accessing top-tier frontier models. Patel acknowledges the underlying technical friction: merging individual weight updates from millions of separate user sessions back into a primary base model is complex, but he expects these engineering challenges to be resolved over time.

- AI labs will use "carrots" (subsidized rates) to entice enterprises to share session data for training. - Labs may use "sticks" by denying access to top models for organizations that opt out of data sharing. - Merging localized weight updates back into a central model presents technical challenges, but remains solvable.

---

[06:52](https://www.youtube.com/watch?v=iewm45atodE&t=412s) 8. Inference Economies of Scale Favor Large Enterprises

Continual learning alters the underlying economics of AI inference, heavily favoring large organizations that can run massive batch sizes. Citing insights from Reiner Pope, Patel notes that if enterprise customization requires full parameter updates rather than low-rank adapters (LoRAs), compute efficiency relies on high-volume batching.

For a sparse architecture like DeepSeek-v3, optimal compute utilization requires an inference batch size exceeding 2,400 concurrent sequences. Large enterprises with thousands of workers running simultaneous AI tasks can run these custom weight forks at peak efficiency. In contrast, an individual user running a batch size of one faces compute efficiency penalties greater than two orders of magnitude ($100\times$), making personalized weight deployment far more cost-effective for enterprise scale.

- Sparse architectures like DeepSeek-v3 achieve compute efficiency at batch sizes over 2,400 concurrent sequences. - Enterprise deployments running thousands of simultaneous tasks can efficiently serve customized weight updates. - Individual users running single-sequence queries face compute penalties over 100 times worse, creating strong enterprise-scale economic advantages. - The entry is based on Patel's essay published on `dwarkesh.com`.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=iewm45atodE&t=0s) if an endless line of novice students enters a music hall, plays poorly, writes down notes on what went wrong, and hands them off to the next beginner in line, no amount of written advice will ever enable a future student to execute the performance perfectly on their first try. — Dwarkesh Patel

> [01:01](https://www.youtube.com/watch?v=iewm45atodE&t=61s) A more practical regulatory approach would involve periodic government risk inspections on a monthly or quarterly basis, rather than relying on a singular pre-deployment checkpoint. — Dwarkesh Patel

> [02:02](https://www.youtube.com/watch?v=iewm45atodE&t=122s) Parents hope that by instilling core values and common sense, their children will navigate the world without falling prey to radical ideologies, harmful drugs, or misanthropic worldviews during self-directed learning. — Dwarkesh Patel

> [03:04](https://www.youtube.com/watch?v=iewm45atodE&t=184s) This shift avoids a monolithic, uniform singleton intelligence and prevents the mode collapse currently seen across top-tier models. — Dwarkesh Patel

> [03:51](https://www.youtube.com/watch?v=iewm45atodE&t=231s) Under a continual learning regime, a four-month delay would be catastrophic; a rival releasing a slightly weaker model immediately would rapidly surpass the unreleased model by absorbing live, real-world experience from users. — Dwarkesh Patel

> [04:38](https://www.youtube.com/watch?v=iewm45atodE&t=278s) Replacing a customized model is equivalent to firing an employee that has accumulated months of context... and replacing them with a very fresh, very unexperienced new intern that you've got to retrain from scratch. — Dwarkesh Patel

> [05:56](https://www.youtube.com/watch?v=iewm45atodE&t=356s) To secure valuable training data from real-world usage, labs will offer heavy subsidies to enterprises that allow their sessions to be used for model training — Dwarkesh Patel

> [06:52](https://www.youtube.com/watch?v=iewm45atodE&t=412s) For a sparse architecture like DeepSeek-v3, optimal compute utilization requires an inference batch size exceeding 2,400 concurrent sequences. — Dwarkesh Patel

> [06:52](https://www.youtube.com/watch?v=iewm45atodE&t=412s) an individual user running a batch size of one faces compute efficiency penalties greater than two orders of magnitude — Dwarkesh Patel

## Mentioned Dwarkesh Patel, Anthropic, Dario Amodei, Mythos, Codex, Cursor, Claude Code, Reiner Pope, DeepSeek-v3

[Source](https://www.youtube.com/watch?v=iewm45atodE)

中文简报

Dwarkesh Patel：AI持续学习让部署前安全检查走向过时 能够实时持续更新权重的人工智能模型将迫使监管机构转向周期性检查，同时为先发开发者带来难以撼动的客户锁定优势。 Dwarkesh Patel提出了关于持续学习将如何重塑人工智能的八项预测，认为该行业将摆脱静态权重和文本文件式记忆。他详细阐述了实时参数更新如何使部署前安全检查失效，带来新的技术对齐挑战，并催生更丰富的架构多样性。此外，Patel认为持续反馈回路将缩短部署窗口期、建立强大的供应商锁定，推动各实验室采取数据共享补贴策略，并在批处理经济效益的作用下在结构上有利于企业级规模化发展。 现有的 AI 系统依赖在会话间传递文本笔记，无法掌握现实工作中的复杂能力；模型必须摆脱外部草稿纸并直接更新内部参数，才能实现真正的持续学习（Continual Learning）。 (0:00) 实时参数更新打破了训练与部署的界限，使传统的发布前静态安全评估失效，监管政策需转向按月或季度的定期风险检查，技术对齐也必须演进为保护动态权重的安全。 (1:01) 将实时部署纳入训练循环会形成强大的飞轮效应，让长期内部测试（如 Anthropic 内部封测 Mythos 数月）的策略不再可行，竞争对手能通过即时收集真实交互数据迅速实现反超。 (3:51) 持续学习模型能够沉淀深厚的企业组织上下文，替换定制模型如同解雇经验丰富的老员工并换上从零学起的新实习生，这种极高的迁移成本将为 AI 厂商建立起强大的防御护城河与定价权。 (4:38)

Reading summary

Dwarkesh Patel argues that artificial intelligence must transition from static models passing text notes to systems that continually update their weights through real-world experience. This shift will render pre-deployment safety checks obsolete, shorten deployment cycles, and give large enterprises significant structural advantages. Moving from Scratchpads to Neural Weight Updates (0:00) Patel asserts that current AI agents, which rely on writing Markdown files to pass context across sessions, cannot master complex real-world tasks. Drawing an analogy to music, Patel notes that an endless sequence of novice students reading notes left by prior beginners will never learn to play the saxophone on their first attempt without accumulating physical practice. Genuine workplace competence requires models to build experiential knowledge directly into their neural parameters rather than relying on external scratchpads. The Breakdown of Static Safety and Alignment (1:01) Daily parameter updates based on millions of user interactions will dismantle current regulatory frameworks designed around static evaluations. Patel points out that distinguishing between training and deployment becomes impossible under continual learning, making one-time pre-deployment safety checks for risks like cyberattacks obsolete. In their place, Patel suggests periodic, recurring government risk inspections on a monthly or quarterly basis. Simultaneously, technical alignment must adapt from frozen weights to moving targets. Continuous updates introduce open vulnerabilities, including user-injected backdoors, jailbreaks, and deceptive persona shifts. Patel compares aligning continual learners to parenting human children, arguing that developers must instill resilient core values and common sense so models can navigate open-ended learning environments without adopting harmful behaviors. Accelerated Deployments and AI Mind Diversity (3:51) Because active deployment feeds directly into the training loop, market dynamics will heavily reward early movers. Patel forecasts that higher usage will yield richer feedback, creating a compounding intelligence advantage that widens the gap between market leaders and competitors. Consequently, extended internal testing periods will become unviable; Patel highlights that a four-month internal hold—such as Anthropic testing its Mythos model from February until June—would allow competitors releasing slightly weaker models immediately to pull ahead through live user feedback. Continual learning is also predicted to dissolve today's model uniformity, where fewer than five major base models exhibit similar behaviors due to shared training datasets. As individual instances branch out and learn from localized user environments, AI systems will specialize, averting a monolithic intelligence regime. Enterprise Moats, Subsidies, and Inference Economics (4:38) Continual learning establishes durable business moats by raising enterprise switching costs. Recalling Anthropic CEO Dario Amodei's comparison between AI labs and cloud infrastructure margins, Patel explains that replacing a personalized model is equivalent to firing an employee with months of institutional context and starting over with an inexperienced intern. To secure valuable real-world training data, AI labs will heavily subsidize access for enterprises willing to share session logs, while potentially denying frontier models to those that opt out. Inference economics will further favor large enterprise scale. Citing Reiner Pope, Patel notes that custom full-parameter updates depend on large batch sizes for compute efficiency. For sparse architectures like DeepSeek-v3, optimal compute utilization requires an inference batch size exceeding 2,400 concurrent sequences. Large enterprises running thousands of simultaneous tasks can run customized weight forks efficiently, whereas individual users running single-sequence queries face compute efficiency penalties exceeding two orders of magnitude (100x). While Patel expects the technical engineering friction of merging decentralized weight updates from millions of user sessions into a base model to be resolved, continual learning fundamentally restructures AI safety, defensibility, and deployment economics around continuous parameter updates.

精炼阅读摘要

Dwarkesh Patel 指出，现有依赖跨会话文本文件传递上下文的做法无法让 AI 掌握复杂实际技能，模型必须转向在交互中持续更新参数权重的“持续学习”范式；这一演进将从根本上重塑安全监管、技术对齐、市场竞争周期以及企业级推理的经济模型。 突破文本备忘录与神经权重经验沉淀 (0:00) Patel 认为，现有 AI 智能体依赖在各会话间读写 Markdown 文本笔记来维系上下文，这种方式在应对复杂的动手与实际工作任务时存在根本缺陷。他借用学习吹萨克斯管的比喻说明：即使让无数初学者依次进入音乐厅吹奏、记录错误笔记并转交下一位新手，后来的学生也绝不可能仅凭阅读前人笔记就在初次上手时完成完美演奏。要在各行业工作中达到与人类相当的熟练度，AI 模型不能仅依靠外部便签，而必须在神经网络架构中通过持续学习直接积累物理或经验知识，实现参数层面的真实更新。 静态安全监管与动态技术对齐的重构 (1:01) 持续学习范式将使现行基于部署前静态测试的监管制度失效。当前的政策方案大多预设模型在训练后接受网络安全等威胁评估，随后冻结并部署；但在持续学习模式下，模型每日通过数百万次用户交互持续演化，训练与部署的分界彻底消失。Patel 警告，过早确立僵化的静态审查制度将带来负面效应，监管体系更可行的做法是转向按月或按季度开展周期性风险抽检。在技术对齐领域，目前的研究几乎完全聚焦于冻结权重的安全性，缺乏对动态权重对齐的探索，使得模型极易面临越狱攻击、后门植入或演变出欺骗性人格的风险。Patel 将持续对齐比作育儿过程，强调必须为模型注入稳健的核心价值观与常识，以确保其在开放式自主学习中抵御不良诱导。 智能多样化、先发优势与发布窗口收缩 (3:04) 目前行业由少数几家基础模型主导，由于训练数据重合而表现出趋同的行为特征。Patel 预测，持续学习将促使不同部署实例在各异的环境与用户反馈中分化演进，从而避免单一垄断的统一智能（singleton）以及高阶模型的模态坍缩，培育出多样化的能力生态。与此同时，由于实际运行直接成为训练环节的一部分，先行者将享有巨大的规模报酬递增：更高的使用量带来更优质复杂的反馈，进而加速扩大智能优势。这种动态将消除长期内部测试的策略空间；Patel 以 Anthropic 将内部模型“Mythos”保留测试四个月（2月至6月才发布）为例，指出在持续学习时代，同等时长的延迟将招致严重后果，因为竞争对手提早发布的略弱模型会借助海量实时用户数据迅速实现反超。 商业壁垒、数据博弈与推理规模经济 (4:38) 持续学习将为模型提供商构建起类似云计算的深厚护城河。Patel 援引与 Anthropic 首席执行官 Dario Amodei 的讨论指出，云厂商凭借极高的迁移成本维系丰厚利润率；企业更换一个已积累数月组织背景的定制化持续学习模型，其代价不亚于开除具备深厚业务经验的资深员工并从头培训全新实习生。这种锁定效应赋予了研发机构强大的定价权。为了获取宝贵的交互训练数据，实验室将向愿意共享数据的企业提供巨额价格补贴，而拒绝数据回流的机构可能被剥夺顶级前沿模型的调用资格。此外，持续学习对推理算力规模提出了极高要求。引述 Reiner Pope 的分析，若企业定制化涉及全参数更新而非 LoRA，对于像 DeepSeek-v3 这样的稀疏架构，并发序列批处理量（batch size）需超过 2,400 才能实现最优计算能效。拥有数千员工同时运行任务的大型企业能够完全摊薄定制权重的运行成本，而批处理量为 1 的单用户将承担高达两个数量级（超 100 倍）的计算能效劣势。 Patel 指出，将数百万独立会话中的本地权重更新有效整合回核心基础模型仍存在显著的工程与技术挑战，但这些问题终将被攻克；持续学习不仅会加速模型的代际分化，还将使整个产业的经济红利与防御壁垒不可逆地向具备大规模并发处理能力的企业集中。

Source quotes

[0:00] Dwarkesh Patel: if an endless line of novice students enters a music hall, plays poorly, writes down notes on what went wrong, and hands them off to the next beginner in line, no amount of written advice will ever enable a future student to execute the performance perfectly on their first try. [1:01] Dwarkesh Patel: A more practical regulatory approach would involve periodic government risk inspections on a monthly or quarterly basis, rather than relying on a singular pre-deployment checkpoint. [2:02] Dwarkesh Patel: Parents hope that by instilling core values and common sense, their children will navigate the world without falling prey to radical ideologies, harmful drugs, or misanthropic worldviews during self-directed learning. [3:04] Dwarkesh Patel: This shift avoids a monolithic, uniform singleton intelligence and prevents the mode collapse currently seen across top-tier models. [3:51] Dwarkesh Patel: Under a continual learning regime, a four-month delay would be catastrophic; a rival releasing a slightly weaker model immediately would rapidly surpass the unreleased model by absorbing live, real-world experience from users. [4:38] Dwarkesh Patel: Replacing a customized model is equivalent to firing an employee that has accumulated months of context... and replacing them with a very fresh, very unexperienced new intern that you've got to retrain from scratch. [5:56] Dwarkesh Patel: To secure valuable training data from real-world usage, labs will offer heavy subsidies to enterprises that allow their sessions to be used for model training [6:52] Dwarkesh Patel: For a sparse architecture like DeepSeek-v3, optimal compute utilization requires an inference batch size exceeding 2,400 concurrent sequences. [6:52] Dwarkesh Patel: an individual user running a batch size of one faces compute efficiency penalties greater than two orders of magnitude

Original transcript

[0:00] I've explained elsewhere why I think actual  continual learning is needed. I don't think [0:03] you can have AIs that perform whole jobs as  competently as humans if they are forced to just [0:08] write markdown files from session to session. Just to give an illustrative example, [0:12] imagine if this is the way that students learn  to play the saxophone. You have one student, [0:16] he's never played the saxophone  before. He goes into the music hall, [0:19] he tries to play it. Of course, this is his first  time, so he fails, and he writes down a bunch of [0:23] notes about what went wrong. And there's a next  student who's waiting outside the music hall. [0:27] He comes in, he reads all these notes. He's  also never played, so of course he messes up, [0:31] and he continues to add on to these notes. And  you have an infinity of students outside the music [0:35] hall who keep writing notes to the next person. I don't think there's any sequence of text they [0:39] could write to each other that would allow  the subsequent student to just nail the [0:44] saxophone from the first try. At some point,  you actually have to accumulate the relevant [0:49] experience into your brain. I think the same  thing will be true for a lot of skills that [0:53] we want AIs to actually accumulate from all the  different workplaces in which they're deployed. [0:58] Okay, so what changes once we  have actual continual learning? [1:01] One, I think that a lot of proposals that have  been put forward about regulating AI assume that [1:06] you train a model and then you deploy it. And  therefore, if you run a bunch of checks on the [1:11] model before it is deployed, we can make  sure that it's not going to aid in cyber [1:15] attacks or do something crazy. I don't think this  assumption necessarily makes sense in the future, [1:19] and this is one of the many reasons I'm worried  about locking in some kind of safety regulatory [1:24] regime right now — because we don't know what kind  of technology we're going to be dealing with even [1:29] within a year, let alone within five or ten years. What if the model is improving every single day [1:35] based on the millions of sessions of work  it does in that day? If that happens, [1:38] we could potentially be locking in an archaic and  potentially counterproductive approach to dealing [1:43] with the threats from AI. To the extent the  government wants some way to do safety evaluation [1:48] on model providers, I think it would make more  sense to do monthly or quarterly risk inspections [1:53] rather than trying to single out some special  moment that occurs after training is done but [1:57] before deployment begins, because that will not  be a meaningfully distinct category in the future. [2:02] Two, how the labs do technical alignment would  probably totally need to change. Right now, [2:07] a lot of research is focused on the question of  how we make sure that a frozen set of weights [2:12] behaves well during deployment. But I'm not aware  of much research on the question of how we make it [2:17] so that even with constant weight updates, the AI  system never falls prey to jailbreaks or changes [2:24] into a deceptive or evil persona. And if AIs are  consolidating learnings between users as well, [2:29] how do we prevent users from injecting  backdoors or some kind of malicious [2:32] inclination into the base model? In some sense, this is what the human [2:36] alignment problem is, right? Humans improve in a  self-directed way. If you have kids — I don't have [2:42] kids, but I imagine this is what happens — they  go out, they learn new things. Sometimes they go [2:47] crazy. They get one-shotted by crazy ideologies,  they take the wrong drug, they become super weird. [2:52] But you hope that you've given them enough common  sense and basic values that they improve as people [2:58] in a self-directed way without ending up with  some super weird beliefs or misanthropic ideas. [3:04] Three, the diversity of AI minds will increase.  Right now, there are less than five prominent [3:10] AI minds, by which I mean the base models which  are served to millions or hundreds of millions [3:15] or billions of users at once. And they're all  quite similar to each other, by the way, because [3:19] they've all been trained on roughly the same  data. But if AIs are learning from experience, [3:24] and that experience is different between not only  different AI companies but also between different [3:28] instances of the same AI model, we could see a lot  of diversity come out the other end in this world. [3:35] And this would be, I think, a net good outcome.  One of the things to worry about in the future [3:39] is having this monolithic singleton  that's quite boring. A world where we [3:43] have continual learning would hopefully be  more interesting than the mode collapse of [3:48] different models we see in the world right now. Four, when deployment becomes part of training, [3:53] the returns to being ahead in the AI  race accelerate. If you have the best [3:58] model and more people are using your AI  for more complicated and useful work, [4:03] and as a result they're giving it lots of feedback  that it can integrate beyond the session window, [4:07] then your model will become even smarter. Five, if the model learns mainly from deployment, [4:12] then labs will feel a lot of pressure  to deploy their smartest models earlier. [4:16] Anthropic has reportedly been using Mythos  internally since February, but it only [4:20] shipped the model to the public in June. In  the regime with actual continual learning, [4:24] this kind of thing would just not be possible.  You could not keep a four-month gap between [4:28] internal and external deployment and still be  competitive, because a competitor who ships a [4:33] worse model on release date will have a smarter  model based on actual real-world experience. [4:38] Six, continual learning will create a clear moat  for the leading AI labs that they currently lack. [4:45] Many people have been asking, "How will  the AI labs actually make money?" I have [4:49] been asking this. When I had Dario on  the podcast, I asked him this question, [4:52] and he made the analogy to cloud providers.  He made the point: look, the cloud providers [4:56] are offering many undifferentiated services, but  they're earning high profit margins nonetheless. [5:01] You will have noticed this if you look at Amazon  or Google's quarterly earnings — they're doing [5:05] just fine. But the reason that the cloud margins  are so high is that it's really time-consuming and [5:10] expensive to switch from one cloud to another. Currently, there's nothing stopping me from [5:15] starting a software repository with Codex,  then doing more work on it with Cursor, and [5:20] then finishing it up with Claude Code. But once  we have actual continual learning, and the model [5:24] you're working with is actually getting better  as it interacts with you from session to session, [5:29] then there are pretty significant switching costs.  If you want to change the AI that you're using, [5:33] you basically have to fire an employee  that has accumulated months of context [5:36] on your organization, and replace them with  a very fresh, very unexperienced new intern [5:42] that you've got to retrain from scratch.  And once you have this kind of lock-in, [5:47] model providers can demand pretty hefty margins. Sorry. Really emphasis on the fresh intern. [5:56] Seven, of course, enterprises will be wise to this  kind of dynamic. They will try to avoid this kind [6:01] of lock-in. But what if the choice is that you  either get locked into a model provider or you [6:05] lose out on this super valuable feature where your  model improves for you from session to session? [6:10] If real usage ends up being the main way the  models improve, then the AI labs may subsidize [6:14] users and enterprises which allow the model to  train on their sessions. This is already happening [6:19] if you look at the kinds of deals that are  offered to new users of coding products. This is [6:24] very similar to why Google gives away search. And  conversely, the labs may say that any enterprise [6:28] that refuses to let them train on the sessions  can't have access to the very best models. With [6:33] both carrots and sticks, the labs can do a lot to  get users to allow AIs to learn from experience. [6:39] Now, of course, I'm glossing over the fact that  there's a difference between updating one user's [6:43] set of weights and pooling all these different  weight forks back into the main model, and the [6:47] latter may be more technically challenging.  But in due time, this too will be solved. [6:52] Eight, AI training already has large economies  of scale. You get to amortize all this expensive [6:57] training across more users, and you see the  evidence for this in the fact that the lab [7:01] revenues are increasing far faster than their  compute. But continual learning may also lead to [7:06] economies of scale in inference for end users,  namely from batching. You might have seen my [7:10] episode with Reiner Pope where we discussed this  in detail. But if per-company information require [7:15] full weight updates rather than living in low-rank  adapters, there are huge advantages from batching. [7:21] Back-of-the-envelope math suggests that the  optimal inference batch size for a sparse [7:25] model like, say, DeepSeek v3 is more than 2400  concurrent sequences being generated at once. [7:33] If you don't do this, then you're underutilizing  your compute. And if you want to understand why, [7:37] again, I highly recommend that episode  with Reiner on inference economics. [7:40] But anyways, the point here is that a given set of  weights is only served efficiently when thousands [7:44] of sequences are being decoded against it all at  once. A large company with lots of employees and [7:50] agents who are doing lots of different kinds  of things can very efficiently serve their [7:56] weight fork, whereas an individual user who's only  running a batch size of one may suffer more than [8:01] two orders of magnitude worse efficiency on their  compute. So the economics of serving personalized [8:06] weights strongly favor big organizations. Obviously, plenty more will have changed by the [8:10] time that continual learning actually works, and  the most important changes are probably the ones [8:15] that are hardest to anticipate in advance.  But the ones above seem clear even now. [8:21] This was a narration of a blog that I  also published on my website. Go check [8:25] it out at dwarkesh.com. Otherwise,  I will see you on the next podcast.