AI Labs Shift Compute Strategy as RLVR Hits Limits in Complex Tasks

RLVR在复杂任务中遇阻，AI实验室调整算力策略 未来的模型扩展将依赖于通过自蒸馏和内部模拟进行基于现实世界的持续权重更新，而非静态预训练。 本文探讨了主要AI实验室为实现通用人工智能（AGI）而在可验证奖励强化学习（RLVR）上押下的研究重注。文章认为，相关领域必须兼具可验证性与高度的“可刷性”（grindable）——即能够在可重放的模拟器中运行数千次并行推演——才能推动快速进展。为克服样本效率瓶颈和长上下文窗口的局限，文章重点介绍了同策略自蒸馏（OPSD）和内部“做梦”（dreaming）等新兴前沿方向，以将现实世界的学习成果直接转化为静态模型权重。展望2027至2028年，AI能力很可能会从静态预训练转向基于真实世界经济互动的部署后持续全局权重更新。 主要AI实验室正大举押注，认为在数千个环境中跨数百万个可验证任务扩展强化学习，将自然催生通用人工智能（AGI）。 (0:00) 当前范式的支持者认为，训练样本效率低下属于可摊销成本，而具备海量上下文窗口的上下文学习可能会消除对在线权重更新的需求。 (0:00) AI在操作电脑方面的进展滞后于代码编写，因为相关领域需要“可刷性”——即在不受现实世界约束或机器人检测影响的情况下，运行数千次并行、确定性推演的能力。 (2:12) 仅靠可验证奖励强化学习（RLVR）能否泛化到复杂、开放且无法模拟的现实世界挑战，仍是一个尚未解决的实证问题。 (6:10) 模型训练时的上下文长度与实际服务时的上下文长度之间的差距，往往会导致在更长周期下的性能下降。 (6:10)

Frontier AI labs are betting that scaling reinforcement learning across verifiable environments will yield artificial general intelligence, but mastering open-ended real-world work will require shifting from static pre-training to continuous post-deployment learning. Analysis from Dwarkesh Patel suggests that bridging this gap demands transferring knowledge into permanent model weights via self-distillation and internal simulated dreaming. The Bottleneck of Grindability Beyond Coding (2:12) Major AI labs currently wager that training models on millions of verifiable tasks will naturally develop general agency and overcome models' extreme sample-efficiency deficit, where models require roughly one-millionth ($10^{-6}$) the sample efficiency of humans during training. While optimists argue that scaling compute will steamroll these deficits, domain progress has exposed a major structural constraint: tasks must be not only verifiable, but also 'grindable.' Grindability requires deterministic, replayable simulators that allow thousands of parallel rollouts from identical starting conditions. While software engineering thrives in isolated containerized environments, real-world tasks—such as computer web use, commercial strategy, legal battles, or political campaigning—fail this requirement because live environments cannot be repeatedly spammed or reset. Furthermore, whether reinforcement learning with verifiable rewards (RLVR) can generalize to open-ended horizons remains an unproven empirical question. Anthropic's Dario Amodei noted that performance often degrades when serving models at context lengths longer than those used in training, indicating that short-horizon RLVR may struggle to scale smoothly to complex, non-stationary real-world challenges. The Cost of Ephemeral Inference Compute (8:41) Between 30% and 50% of lab compute budgets are currently consumed by inference, yet this expenditure leaves base model weights unchanged. Relying strictly on long context windows and expanded KV caches creates ephemeral learning: specialized, high-level expertise acquired during a live session is discarded as soon as the session concludes. Proponents of in-context learning argue that fitting work histories directly into context removes the need for weight updates, but this approach departs from human cognition, which compresses experience into persistent intuition rather than maintaining raw activation storage. Capturing session knowledge into static parameters without catastrophic forgetting remains difficult. While high-frequency production setups like Cursor's Tab model successfully learn online using over 400 million daily code-edit requests under a single objective, standard gradient updates remain too sample-inefficient to integrate hyper-specific, low-data organizational insights through conventional fine-tuning. Updating Weights Through Distillation and Dreaming (12:42) To transfer deployment experience into model weights without requiring external automated verification, researchers are examining On-Policy Self-Distillation (OPSD). In this framework, a base 'student' model learns directly from token-level probability predictions generated by a context-enriched 'teacher' model that has undergone an extended interactive session. Unlike standard supervised fine-tuning that forces verbatim memorization of full session transcripts, OPSD extracts sparse, targeted insights, preserving the model's baseline capabilities. A complementary, more speculative approach is test-time training or 'dreaming,' representing a fourth scaling dimension alongside pre-training, post-training RL, and inference compute. Drawing inspiration from EfficientZero—which achieved human-level sample efficiency in Atari games by playing dozens of internal simulations for every real step taken—future models could allocate substantial compute to construct and rehearse inside high-fidelity internal environments. Instead of simple context summarization routines like `/compact`, an agent running a `/dream` routine could generate simulated tasks reflecting user production environments to master domain skills offline. Continual Learning Across the 2027–2028 Economy (17:23) Looking toward the 2027–2028 horizon, RLVR is framed as a foundational catalyst: it equips base agents with the resilience needed to orient themselves, test strategies, and navigate roadblocks in unfamiliar tasks outside their initial training distributions. Once this baseline competence is established, models can be deployed into extended, multi-day work cycles alongside human professionals. In this projected workflow, humans review agents after work cycles, where positive feedback triggers permanent weight consolidation through OPSD, internal dreaming, or hybrid distillation techniques. Over time, the primary driver of AI capability will transition away from static pre-release training and toward continuous, global online learning aggregated across active economic deployments. The central unresolved challenge is whether RLVR can bridge the divide between narrow, synthetic environments and the unstructured, non-grindable nature of real-world work without inducing catastrophic forgetting or relying indefinitely on unscalable context lengths.

Dwarkesh Patel探讨了人工智能训练范式的演进路径，指出各大AI实验室目前押注基于可验证奖励的强化学习（RLVR）以实现通用人工智能（AGI），但该路线受制于环境的“可研磨性”不足以及推理经验无法沉淀至权重的弊端；未来的关键突破在于通过策略内自蒸馏（OPSD）与内部仿真“做梦”等技术，将部署中积累的实时经验固化为模型权重，从而在2027至2028年实现经济层面的持续学习。 RLVR押注与可研磨性瓶颈 (0:00) 头部AI实验室当前正进行一项重大研究押注：只要在数千个包含可验证任务的环境中通过强化学习（RL）训练模型，就必然能催生通用人工智能。支持者主张，当前模型在训练时样本效率仅为人类百万分之一（10⁻⁶）的缺陷只是一次性分摊成本，且随着上下文窗口扩展，模型无需更新权重即可解决复杂问题。然而，RL取得突破不仅需要任务具备可验证性，还要求领域具备高“可研磨性”（Grindability），即开发者必须能够在确定性、可重放的模拟器中，从完全相同的初始状态并发运行数千次试验。 编程任务因易于在软件容器中隔离而具备极高的可研磨性，但现实世界的计算机操作和复杂任务却难以满足这一要求。例如，若派出上千个并发智能体在亚马逊网站测试结算流程，会迅速因滥用行为被服务商封禁；而依靠人工克隆本地应用环境成本极高且难以规模化。在创办企业、法庭诉讼或政治竞选等现实场景中，数据稀疏且高度非平稳，外部验证周期长达数月乃至数年，缺乏无重置成本的重放环境，限制了传统RLVR的直接套用。 泛化困境与推理经验的隐性损耗 (6:10) 实验室的主流假设认为，在结构化沙盒中通过RLVR训练出的模型能够自然泛化为具备通用规划与应变能力的实体。然而，RLVR能否跨越到开放世界仍是未解的实证问题。Anthropic首席执行官Dario Amodei曾指出，模型在训练时所用的上下文长度与部署服务时的长度存在断层，在短时域训练出的能力在长时域服务时往往会出现性能退化，这暗示沙盒中培养的短程技能难以平滑迁移至复杂且长周期的现实挑战中。 与此同时，当前AI实验室约有30%至50%的算力消耗在推理阶段，但这些算力并未转化为对底层权重的永久提升。模型在真实商业环境中接触到的组织特有知识，在单次会话结束后便彻底丢失。依赖不断膨胀的KV缓存不仅在工程计算上不可持续，且违背人类通过参数压缩沉淀直觉的认知机制——过度依赖原始记忆甚至会损害高阶抽象思维能力。 策略内自蒸馏突破权重更新瓶颈 (12:42) 为了在不引发灾难性遗忘的前提下将经验沉淀至静态权重中，研究者开始探索损失函数的底层革新。虽然Cursor的Tab模型证明了利用单一目标在大规模线上数据（每日超4亿次编辑建议）中实现在线学习的可行性，但这难以直接推广到数据稀疏的企业定制场景。在Sasha Rush参与探讨的黑板讲座中，研究人员提出了“策略内自蒸馏”（OPSD）机制，使基础“学生”模型去拟合在长交互会话中积累了丰富上下文的“教师”模型的token预测分布。 相比标准RLVR与监督微调（SFT），OPSD具备明确的结构优势：它不依赖外部自动化验证信号，仅要求教师模型在上下文内展现出有效学习能力；它提供密集的token级别监督信号，而非整条路径的单一标量奖励；同时，它继承了强化学习的稀疏更新特性，只针对性提炼匹配结果所需的关键洞见，避免了无差别监督微调对模型原有通用知识的破坏。 测试时训练与做梦仿真维度 (15:22) 除了损失函数优化，另一项提升样本效率的前沿设想是“做梦”（Dreaming）或测试时训练。DeepMind此前研发的EfficientZero展示了极端样本效率的机制：虽然该模型在两个小时的人类等效时间内击败人类玩家，但其在现实环境迈出一步的同时，内部已进行了数十局仿真推演。未来的大语言模型或将借鉴此类思路，在模型内部自主构建高保真仿真环境，用于在脱机状态下预演新技能与优化策略。 这被视作继预训练、后训练强化学习与推理期算力之后的“第四扩展轴”（Scaling Axis）。未来开发者在工具中可能不再依赖仅做简单文本摘要的弱算力指令（如/compact），而是触发类似于/dream的指令，消耗大规模算力针对特定生产场景自动生成类视频游戏环境并实施强化学习训练，从而在不增加真实世界试错风险的前提下跨越数据瓶颈。 走向2028年的持续学习范式 (17:23) 根据Dwarkesh Patel的推演，到2027或2028年，RLVR将演变为智能体的能力基础，赋予其在分布外陌生任务中试错和突破阻碍的底层适应力。随着有效上下文窗口延长，AI系统将能够与人类以数周为单位深度协作。在工作周期终点，人类用户只需给出阶段性评价（如点击认可），便可通过OPSD或内部做梦机制，将当周的交互经验无缝蒸馏固化进模型的基础权重中。 这一闭环标志着人工智能进步的核心动力将发生迁移：模型演进将不再主要依赖发布前静态的集中批次训练，而是转向在全球经济部署中通过数以亿计的实时交互实现持续在线更新。每一次跨领域的业务应用，都将让底层模型通过经验汇聚而持续进化。 Patel勾勒了AI从静态模型向具备在线自更新能力的动态实体转变的技术蓝图，但该前景仍取决于数个未经验证的前提：RLVR在受控沙盒中的能力能否有效泛化至现实长时域任务仍无定论，支持内部“做梦”的高保真世界模拟器需要极高的算力与工程门槛，且在低数据组织场景下实现权重持续安全微调的机制仍处于理论探索早期。

[6:10] Dario Amodei: There's two things. There's the context length you train at, and there's a context length that you serve at. If you train at a small context length and then try to serve at a long context length, maybe you get these degradations. [2:12] Andy Jassy: find your bots and shut your ass down.

[0:00] So here's a big research bet that all the labs are making. [0:02] They think that if we train AIs to accomplish millions of verifiable [0:06] tasks across thousands of diverse RL environments, then we will have [0:11] basically built AGI, because this kind of training will have created a kind of [0:16] problem-solving agent: the kind of thing that can make progress on open-ended [0:20] tasks for weeks on end in the face of errors and mistakes and ambiguity. [0:25] And the people who are optimistic about this vision will say that all [0:27] these things we talk about as the fundamental deficits in the current [0:31] training paradigm — for example, the data inefficiency of these models, [0:35] or the fact that they lack continual learning — can just be steamrolled if we [0:39] scale training more, in the same way that all the fundamental research problems [0:43] in natural language processing collapsed when we threw enough compute into LLMs. [0:48] So in the previous essay, I talked about how these models are one [0:52] one-millionth as sample-efficient as humans, and the people who are in favor [0:56] of the current training paradigm will say, "Look, that might be true, but [0:59] this is only true during training." [1:01] Training is this one-time cost that is amortized across billions of [1:05] sessions that a model will experience. [1:08] What really matters is how smart and general and sample-efficient [1:11] the model is during a session, and this has clearly been improving as [1:15] we've been doing more RL training. [1:17] AI agents are able to solve more and more ambitious problems over [1:20] longer and longer time spans. [1:21] Anybody who has used these models for coding knows that. [1:24] Similarly, people would say, look, continual learning — this capability [1:27] I keep harping about, where the model's weights get updated based on [1:31] what it's learning from deployment — may simply not be necessary. [1:35] Because if in-context learning gets so good across longer and longer [1:40] time horizons, then you don't need to distill everything the model is learning [1:44] on the job back into the weights. [1:46] People often say that their employees are not net productive [1:49] until six months or more on the job. [1:52] So clearly, online learning is necessary for competence. [1:56] But what if you could just fit those six months into the context window? [1:58] There have been tons of architectural innovations that dramatically increase [2:01] the amount of information, or the amount of context, that a transformer can store. [2:06] And why not think that, with a couple more years of progress, [2:08] we might have what feels like infinitely large context windows? [2:12] Okay, so before we discuss this research bet a bit further, I want to step [2:15] back and ask a completely tangential question, which I find actually [2:19] very interesting and confusing about the nature of current AI progress. [2:24] Why has progress on computer use been so much slower than other domains? [2:30] Computer use is so clearly verifiable. [2:34] You could ask a question like: did the desired Etsy item I ordered get delivered? [2:38] Is the venue for an event I'm trying to organize booked? [2:41] Have my taxes been submitted? [2:43] So isn't it weird that computer use has been making so much slower [2:46] progress than coding and math and these other verifiable domains? [2:49] I'm sure there are many reasons for this, and one of them, of course, [2:52] is the fact that the models are exposed to far less high-quality [2:56] multimodal data during pretraining. [2:59] But one reason that I think is actually quite underrated by people, and which I [3:03] think reveals the canyon walls against which this river of AI progress will [3:07] only slowly chip away, is that it is not enough for a domain to be verifiable. [3:12] It also has to be very grindable, in the sense that you have to be able [3:16] to run lots of parallel rollouts against a deterministic and replayable [3:20] simulator, and you have to run those rollouts from the same starting point. [3:24] If you're trying to make a model better at coding, you can define some [3:27] container that has a software repo with some missing feature that you [3:30] have tasked the AIs with creating. [3:32] And then you have a thousand parallel agents go at the problem, each of which [3:36] has an identical copy of the container. [3:38] But this doesn't work with computer use, at least not trivially. [3:41] You can't just have a thousand agents go try the same checkout flow [3:43] on Amazon to get better at using websites, because Andy Jassy will [3:48] find your bots and shut your ass down. [3:51] You can solve this by making clones of Slack and Gmail and all the other [3:54] common applications and websites. [3:56] But at least currently, this is a very labor-intensive and unscalable [3:59] way to build environments. [4:01] Of course, once AIs get good enough at coding themselves to build these [4:04] clones with extremely high fidelity, then I'm sure computer use will make [4:08] quicker progress than it is right now. [4:10] And you're also killing two birds with one stone with this kind of procedure, [4:13] because getting AIs to rebuild whole applications from scratch is also [4:18] a great RL objective for coding. [4:20] So while computer use itself may soon be solved, its current lethargy [4:23] is telling us the following: that unless you can build a very replayable [4:27] training target for a domain, the models will struggle to make much progress. [4:32] And the reason this is true, of course, is that the models are incredibly [4:35] sample-inefficient during training. [4:37] This is the point I was making in my last video essay. [4:39] So for computer use, we might be able to make up for the sample-efficiency [4:42] deficit by building these farmable deterministic simulators. [4:46] But for so many other different kinds of skills that we need AIs [4:49] to have, we simply can't do this. [4:52] How do we train an AI to get really good at building a business from scratch? [4:55] How about winning court cases, or having a profitable day of trading in the markets, [4:59] or helping a candidate win an election? [5:02] The rollout here requires interacting with the real world, and you can't [5:05] recreate it from just within a datacenter. [5:08] The outer-loop verification here may take months or even years of [5:12] real-world actions to elicit, and you can't re-observe it by perturbing the [5:16] model's actions slightly in thousands of parallel rollouts to isolate exactly [5:22] what the model did that actually worked. [5:24] Now, dealing with such reset-free, non-stationary environments [5:27] is a known open problem in RL. [5:29] I'm not pointing out anything new. [5:31] But I really do want to emphasize that because of the idiosyncratic [5:33] and sparse nature of data in most domains in the world, you need sample [5:38] efficiency in order to get proficient. [5:41] If AIs are to develop all the skills that humans have, and even skills [5:45] that humans don't have, then they need to be able to learn from information [5:49] revealed in unstructured, unverifiable, and ambiguous ways from scarce [5:54] amounts of real-world interaction. [5:56] Because in many domains, the relevant training information simply [6:00] doesn't exist in any other way. [6:02] What is the RL environment to make an AI that is as good at politics as [6:05] Lyndon Johnson, or as good at building a space-launch business as Elon Musk? [6:10] The labs are betting that RLVR will generalize. [6:12] That is, that if you train on enough containerized, reproducible [6:15] environments, you will develop a very general agent that can make and [6:19] execute plans and learn rapidly from new information, and even pick up new [6:24] skills, all within a single session. [6:27] If you drop this endlessly RLVR'd AI into Texas politics in 1948, it [6:32] could give you better advice than LBJ about winning the Senate seat. [6:35] And if you gave it a hundred million dollars in 2002 and let it cook, [6:38] it would build SpaceX for you. [6:39] Now, whether RLVR can generalize this well is an empirical question. [6:43] If the labs went from spending billions of dollars on RL environments to a trillion [6:48] dollars, would you get the kind of thing that is a fully human-like general [6:52] intelligence within the context window? [6:54] Dario gave a telling quote during our podcast together, which I think [6:57] hints that RLVR generalization is not infinitely strong. [7:02] When he was explaining why model performance tends to degrade at long [7:05] context, he said: "There's two things. [7:07] There's the context length you train at, and there's a context [7:09] length that you serve at. [7:11] If you train at a small context length and then try to serve at a long context [7:15] length, maybe you get these degradations." [7:17] Now, maybe I'm reading too much into this, but it seems like he's [7:19] saying that short-horizon RL training doesn't necessarily generalize [7:23] to long-horizon RL performance. [7:25] And if you can't generalize from short horizon to long horizon, then how are [7:29] agents supposed to generalize from getting trained at a bunch of white-collar tasks [7:33] to, say, having the ability to be dropped in the real world and build a business [7:38] from scratch as well as Sam Walton? [7:40] And even if, after enough in-context experience, the AIs could become [7:42] like Henry Ford or Albert Einstein, all that would be ephemeral and [7:46] wasted if you couldn't get those learnings back into the weights. [7:50] Around 30 to 50 percent of a lab's compute goes to inference, and that compute is [7:54] currently not playing any productive role in helping improve the model. [7:58] This seems like a huge waste. [8:00] And it's even worse than it sounds, because it is only in deployment [8:03] that the most valuable bits of information which your model could [8:06] learn from are actually revealed. [8:08] What's actually happening in the organizations where I'm being used? [8:11] What are they using me for? [8:13] And what kinds of mistakes do I tend to make in the real world? [8:16] We've got some genius grad student who's never been allowed to take a real [8:19] internship, and we keep giving it more and more classroom case studies in the [8:22] form of RL training on environments. [8:25] It's so bizarre that we have AIs that are broadly deployed through the [8:29] economy already, and are participating in so many different kinds of tasks, [8:33] and are privy to so much domain- and organization-specific tacit knowledge, [8:39] and they're not able to make use of it. [8:41] But this kind of continual learning requires going back to the weights. [8:45] AIs can't just keep building up a bigger and bigger KV cache as they [8:48] learn from more and more users. [8:50] That's just not scalable, and that's also not how humans do it. [8:53] There's no clean separation in our brain between parameters and activations, [8:56] and it's not like some part of your skull keeps expanding as you learn [9:01] more things throughout your lifetime. [9:03] When we learn stuff, there's clearly some kind of compression, and this [9:07] aids our generalization and grokking. [9:09] There are, in fact, some humans who have this autistic-savant-type ability [9:13] to recall random tables of numbers or nonsense syllables years later [9:19] — basically the kind of fidelity of information that models have in context. [9:23] And such sheer volume cripples these humans' ability to understand [9:27] abstractions and metaphors. [9:29] Human continual learning is less about having all your observations at the tip [9:32] of your tongue and more about chiseling the right intuitions and big-picture [9:36] knowledge back into the weights. [9:38] But the moment you move into the weights, you have to give up on [9:41] in-context learning's sample efficiency. [9:43] Because gradient updates are super sample-inefficient, all of the [9:46] successfully shipped online-learning models have had to learn the exact [9:50] same thing across millions of users. [9:52] For example, the Cursor Tab model online-learns by predicting the [9:57] same exact objective for over 400 million requests a day. [10:00] The objective here is which edits actually got accepted by the user. [10:04] At least so far, we haven't seen models online-learn different kinds of things for [10:09] different users, because while a single session may generate more than enough [10:13] data for a human to learn from, it's not enough to train a more capable AI. [10:18] Current online learning can work for a very limited number of use cases. [10:22] But the whole point of continual learning is that the world is very [10:26] complicated, and each job and company and problem is different, and you need [10:31] your intelligence to be able to learn the specific information related to a [10:36] particular deployment, which simply can't be stuffed into some shared training run. [10:41] These are all the things we're talking about when we talk about on-the-job [10:43] learning: things like how everything in your organization works and fits [10:47] together, how to cooperate with all the infrastructure and the other people [10:50] around you to make progress on some larger project, what the common failure modes [10:55] are, and many other things like this. [10:58] As the podcast has grown, I've had to deal with more and more operational overhead. [11:02] Take paying bills. [11:03] In the past, contractors would just email me their invoices. [11:06] Every few weeks, I'd dig through my inbox, create a folder with all the [11:09] bills, and manually pay each one. [11:11] At this point, though, I just give everybody an email address [11:14] that goes straight to Mercury, which is my banking platform. [11:17] Whenever anybody sends an invoice to that address, Mercury automatically [11:20] downloads it, scans it, and extracts all the relevant information — things [11:24] like the contractor name, address, payment amount, invoice number, [11:27] and due date — and then uses all of this to create a draft payment. [11:30] Mercury then stores a list of these drafts for me to review. [11:33] I just go through the list and double-check that [11:35] they've been built correctly. [11:36] I don't have to track anything or enter any information myself. [11:39] Mercury does all the fundamental things for your business extremely well, [11:43] and it puts them all in one place. [11:44] If you want to learn more, go to mercury.com. [11:48] Mercury is a fintech company, not an FDIC-insured bank. [11:52] Banking services provided through Choice Financial Group [11:54] and Column N.A., Members FDIC. [11:57] In this way, sample efficiency and continual learning are actually [12:01] deeply connected problems. [12:02] Relatively little data is available to the model on the job. [12:07] Now, to learn from this data requires sample efficiency, and models can do that [12:11] in context, but using the fast weights that are built on the fly by attention, [12:16] which allow for this sample efficiency, scales very poorly in terms of memory. [12:20] So we need architectural innovations that allow for some kind of [12:23] intermediate representation. [12:25] I talked before about how we already have many different working ideas [12:28] for this kind of thing, from sparse attention to KV cache compaction. [12:32] And every week, somebody releases a new paper suggesting some kind of [12:35] other architectural optimization. [12:37] It doesn't seem to me that architecture is fundamentally what [12:40] is bottlenecking continual learning. [12:42] So perhaps the bottleneck is the loss function. [12:44] How do we update the weights, AKA how do we improve the model itself, [12:49] based on information that was learned from one particular session? [12:53] Even here, naively, it seems like there are many ideas that ought to work. [12:56] A lot of people are talking about this technique called on-policy [12:59] self-distillation recently. [13:01] If you want to learn more about it, I recorded a little impromptu [13:03] blackboard lecture on my iPhone with Sasha Rush a couple weeks ago, and [13:08] it's in the link in the description. [13:09] But to summarize the explanation, the idea is that we encourage the base model [13:14] to make the same predictions when trying to solve some real-world problem as the [13:18] model with all the context accumulated after a long session would have made. [13:22] The whole point of this procedure is to distill what the model learned in a [13:26] session back into the weights themselves. [13:29] This is better than RLVR for two reasons. [13:31] One, OPSD doesn't require us to have some outer-loop verifiable reward. [13:36] We just need a model that can learn the right things within the context window. [13:40] And as long as we have that, we can train the base model to match our veteran [13:44] teacher model, which has built up all this experience during the session. [13:48] And two, OPSD provides a much denser supervision signal than naive RL. [13:54] Instead of projecting a single reward through the whole [13:56] trajectory, you can train on the per-token probability discrepancy [14:00] between the teacher and student. [14:02] For continual learning, OPSD is also superior to supervised fine-tuning. [14:06] The most naive version of SFT for this application that you can [14:09] imagine is just to train the base model to predict all the tokens that [14:12] are observed during the session. [14:14] But this makes no sense if you think about it as a learning target. [14:16] The way you get better at your job is not by recalling the transcript [14:19] of every single thing that happened every day with perfect fidelity. [14:23] Rather, it's by consolidating the handful of insights and pieces of [14:26] knowledge that are actually relevant to you getting better at your job. [14:30] RL training doesn't suffer from this failure mode. [14:32] RL is great at concentrating the update to only what is relevant [14:36] to getting the outcome right. [14:38] That's why [14:41] the updates from RL are incredibly sparse. [14:43] And this is a very important property for continual learning, because as [14:46] you're learning on the job, you don't want to overwrite and forget all the [14:49] other things that the base model knows. [14:51] I wrote a post a few months earlier arguing that RL learns [14:55] much less information per sample than supervised learning. [14:58] But this may be a good thing rather than a bad thing. [15:00] You only change the model as much as is absolutely necessary to [15:04] achieve the outcome, and no more. [15:07] OPSD preserves this property of RL, where instead of slingshotting towards [15:11] the teacher distribution as supervised learning would have you do, you only [15:15] extract the knowledge that is necessary to achieve the same results as the [15:19] teacher on actual real-world tasks. [15:22] OPSD is one way to attack the sample-efficiency problem. [15:26] You take this scarce real-world experience, and you squeeze all the [15:29] signal into a tiny, well-targeted update. [15:31] But there's also another much more speculative idea. [15:34] Let's call it dreaming. [15:36] If the AI can build a good simulation of reality against which to rehearse [15:40] new skills, or try alternative strategies and reinforce what actually [15:44] works, then AIs could experience orders of magnitude more simulated [15:48] samples in the same wall-clock time. [15:51] Let's go back into history a bit. [15:53] A couple years after DeepMind released AlphaZero, a group of researchers [15:56] trained a model called EfficientZero, and the whole point of this model [15:59] is to be very efficient with data. [16:01] So if this model and a human both got two hours to play against a simulator [16:06] of an Atari game that they hadn't seen before, this model would actually [16:10] probably beat the novice human. [16:11] Does this mean that the model was more sample-efficient than the humans? [16:14] Well, that was the goal of the training, but it depends on how [16:17] you measure sample efficiency. [16:19] Because for each step in the real game, EfficientZero is playing dozens [16:23] of simulated games in its head. [16:25] In a similar way, future LLMs might be able to consume far less [16:29] real-world data while practicing endlessly against environments [16:32] that they build for themselves. [16:34] The big difference, of course, is that it will be much harder to build [16:37] a simulation of the whole world than it is to emulate the game of Go. [16:41] That's why I said this is a much more speculative idea. [16:44] If it works, it would become a fourth axis of scaling alongside pretraining, [16:48] RL, and inference-time compute. [16:50] You could call it test-time training or dreaming. [16:53] The model spends compute writing up RL environments and then training [16:56] against them, and it's rehearsing all the skills that will actually be used [17:00] in production for a specific user. [17:03] So instead of hitting /compact in Codex or Cursor or Claude, which kindles a small [17:09] amount of compute to write up a summary, and which gives you the simulacrum of [17:13] continual learning, you hit /dream. [17:14] And this incinerates huge amounts of compute to build and train against a [17:20] video-game version of what the model is witnessing in the real world. [17:24] So what might continual learning look like by 2027 or 2028? [17:26] And how do we get there? [17:29] Here's one scenario. [17:30] All of this RLVR training is producing an agent that can get its bearings when [17:34] it's thrown at an unfamiliar problem, and it can try different strategies, and [17:38] it can iterate when it hits a roadblock. [17:40] This is the crucial thing that RLVR has given you: an AI that [17:44] is at least competent enough to start getting some real-world [17:47] experience, if it could learn from it. [17:50] And once you have that, you send it out into the world to do real [17:53] work, even on projects that are off the training distribution. [17:56] Now let's say at this point, the effective context lengths have expanded [17:59] such that AIs can jam and co-work with you for a full week of wall-clock time. [18:06] At the end of a week, you give it a thumbs up or a thumbs [18:08] down, you give it a work review. [18:10] And if you give it a thumbs up, the base model distills everything that [18:13] the AI learned during the session, and it may use OPSD, it may use dreaming, [18:17] it may use some other technique that we aren't even aware of, or it'll use [18:21] a combination of all of the above. [18:23] And AI can get better at domains that are adjacent to what it was explicitly [18:28] trained for beforehand with RLVR. [18:32] And in the next round it gets better at the thing adjacent to what it [18:35] was previously online learned. [18:37] In this way, the gamut of AI skills and knowledge and capabilities can [18:41] expand far beyond the verifiable domains that the model was originally [18:46] trained against before it was deployed. [18:48] Just as pretraining created a base intelligence that was smart enough to [18:52] become a competent agent with enough RLVR on top, so RLVR has created an agent [18:58] that is competent enough to actually be broadly deployed in the world, and [19:02] from this broad deployment to learn on the job once the training recipe for [19:06] continual learning actually arrives. [19:08] By this point, the main way that AIs get better is not from the [19:12] training they have received before they are released to the public. [19:14] Rather, it's from all this experience that they'll be accumulating from being broadly [19:18] deployed in the economy and engaging in so many different kinds of tasks. [19:22] Every time that you interact with an AI, it'll be smarter, not only because [19:26] it's been learning from your previous sessions, but also because it's been [19:28] learning from all its interactions with all the other users in the world. [19:33] And that's very scary and exciting and different from the [19:36] way that AI improves right now. [19:38] This was a narration of a blog post that I also released on [19:42] my website at dwarkesh.com. [19:44] Go there if you want to read all the footnotes, or if you want to [19:47] sign up so you can find out when I release the next blog post. [19:50] Otherwise, I'll see you on the next episode.