Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

斯坦福CME295：Transformer与大语言模型 | 2026秋季 | 第4讲 - 基于LLM的强化学习 本讲座涵盖了用于LLM对齐与推理的强化学习数学基础及系统工程。讲师梳理了从经典强化学习、基础策略梯度（REINFORCE）到近端策略优化（PPO）的发展脉络，详细讲解了优势估计、比率裁剪以及KL惩罚。他们分析了无Critic方法（如群体相对策略优化GRPO及其变体DAPO和Dr. GRPO）的内存效率优势与长度偏差缺陷。讲座最后展示了同策略蒸馏（OPD）如何利用来自冻结教师模型或自生成特权上下文的稠密Token级逆向KL监督，从而绕过信用分配难题和辅助模型的计算开销。 LLM的后训练对齐分为两类：针对主观输出、基于Bradley-Terry偏好模型的RLHF，以及针对确定性任务、基于程序化验证的RLVR。 (9:32) 近端策略优化（PPO）利用比率裁剪和Token级Critic来稳定训练，但由于需要与策略网络并行训练价值网络，因而消耗大量内存和算力。 (40:34) 群体相对策略优化（GRPO）通过计算相对于经验群体统计量的优势来省去辅助价值函数，但未经修正的变体会引入长度偏差的病态问题。 (67:16) 同策略蒸馏（OPD）与自蒸馏完全避开了奖励建模，通过引入来自高能力或具备上下文特权的教师前向传递的稠密Token级逆向KL惩罚来实现对齐。 (88:51)

In Stanford's CME295 Lecture 4, the instructors explain how post-training for large language models has evolved from unstable policy gradients and complex actor-critic frameworks toward critic-free methods like Group Relative Policy Optimization and dense on-policy distillation. Foundations and Bottlenecks of LLM Policy Gradients (6:35) Reinforcement learning adapts to language models by treating prompts as states and either single tokens or full completions as actions. In subjective tasks, Reinforcement Learning from Human Feedback (RLHF) optimizes models against a frozen reward model fitted using Bradley-Terry preference pairs, whereas Reinforcement Learning with Verifiable Rewards (RLVR) evaluates objective domains like mathematics and software engineering using deterministic execution checks. Training via policy gradient algorithms such as REINFORCE relies on the log-derivative trick to compute parameter updates based on expected rewards under sampled trajectories. Stabilizing Policy Gradients: From PPO to GRPO (40:34) Vanilla policy gradients suffer from high gradient variance, an absence of baselines, and a lack of token-level credit assignment. Proximal Policy Optimization (PPO) resolves these instabilities by calculating advantages through an auxiliary value function, clipping the importance sampling ratio between 1 - ε and 1 + ε, and imposing a Kullback-Leibler (KL) divergence penalty against a frozen reference policy. However, maintaining and jointly training a separate value network introduces substantial computational and GPU memory overhead. Structural Pitfalls in Outcome-Supervised Group Optimization (67:16) Group Relative Policy Optimization (GRPO) bypasses the need for a critic by sampling a group of completions per prompt at a non-zero temperature, standardizing rewards against the group empirical mean and standard deviation to form baseline advantages. While this reduces the GPU footprint to the policy, reference model, and evaluation mechanism, it introduces unintended failure modes. Normalizing by sequence length creates length bias, where models generate bloated reasoning chains because short incorrect outputs receive harsher gradient penalties than verbose ones. Furthermore, outcome-supervised GRPO lacks token-level credit assignment, assigning uniform credit across entire sequences and reinforcing flawed intermediate reasoning steps that accidentally arrive at correct answers. Dense Token Supervision via On-Policy Distillation (88:51) To achieve fine-grained token-level credit assignment without training critic networks, instructor Shervine Amidi presents On-Policy Distillation (OPD). In this setup, the student generates responses autoregressively on-policy, while a frozen teacher model evaluates the student's output tokens in parallel via reverse KL divergence. This directional divergence creates mode-seeking behavior, strongly penalizing student hallucinations while tolerating instances where the student omits low-probability alternatives. Where an external teacher is unavailable, On-Policy Self-Distillation (OPSD) uses the identical base model conditioned with privileged context, such as ground-truth reasoning chains, to provide token-level target distributions. Commercial deployment relies on tiered workflows, using verifiable RL to establish capability frontiers in flagship models, while on-policy distillation transfers reasoning performance into lower-latency architectures. Nevertheless, single-teacher setups remain limited by coverage gaps and catastrophic forgetting, prompting systems to employ multi-teacher distillation where distinct domain-specialized models supervise individual verticals.

斯坦福大学CME295课程指出，强化学习已成为现代大语言模型后训练的核心环节；讲座系统阐述了策略梯度优化从传统PPO的高显存批评模型演进至无价值网络的GRPO，再延伸到基于反向KL散度的策略内蒸馏等前沿对齐机制。 经典强化学习映射与奖励范式划分 (6:35) 在将经典强化学习映射至大语言模型时，通常存在词元级（token-level）与序列完成级（completion-level）两种粒度。由于在自然语言生成中为每个词元设计稠密且合理的即时奖励极其困难，工程实践多采用序列级建模：将提示词作为初始状态，模型生成的整段输出序列视为单个宏观动作，并由环境对该序列赋予标量奖励。在计算实现上，生成过程需要自回归逐词元采样，无法在时间维度并行，存在显著的计算瓶颈；但在评估与梯度更新时，模型可以通过掩码自注意力并行处理完整序列，计算开销相对较低。 奖励信号主要划分为两类：一是针对聊天语气、安全性与帮助性等无客观标准答案的主观任务，采用人类反馈强化学习（RLHF），借助人类成对偏好数据并通过Bradley-Terry模型预先训练出冻结的标量奖励模型作为代理评估者，以避免实时人工介入；二是面向数学与编程等确定性领域的可验证奖励强化学习（RLVR），该范式绕过容易产生噪声的学习型奖励模型，直接依托编译器、单元测试或数值断言等可程序化执行的基准事实判定输出正确性。 原始策略梯度的缺陷与PPO优化机制 (33:22) 基于REINFORCE的原始策略梯度通过对数导数技巧推导期望奖励的参数梯度，但在语言模型训练中面临严重缺陷。首先，若奖励缺乏基线对比，即便是正向但劣质的生成也会获得参数增加激励，引发高方差梯度震荡；其次，序列级单一奖励无法实现词元级的精确信用分配（Credit Assignment），极易使包含错误逻辑步骤却碰巧得出正确结论的中间推理词元受到错误奖励；此外，连续梯度步可能引发策略灾难性漂移。 近端策略优化（PPO）通过引入与当前策略联合训练的词元级价值模型（Critic）以及广义优势估计（GAE），以词元前后的状态价值差计算优势函数，使优化重心转向动作相对于期望基线的超额收益。为提高样本效率，PPO允许在旧策略生成的单批次数据上重复执行多步更新，并通过重要性采样比率修正分布偏差。为了防止更新幅度过大导致策略崩溃，PPO设计了比率截断机制构建零梯度平坦区，同时引入针对原始SFT参考模型的KL散度惩罚，在局部步进与长期轨迹双重维度约束策略漂移。 GRPO剥离价值网络及其长度偏差缺陷 (67:16) 尽管PPO有效降低了梯度方差并实现了词元级评估，但在超大模型上同时部署并实时同步一个参数规模相当的在线价值网络带来了沉重的显存与工程负担。为解决这一痛点，群组相对策略优化（GRPO）完全摒弃了显存占用高昂的批评模型。GRPO对同一提示词以非零采样温度并行生成多个候选回答，计算该组候选输出在奖励模型或代码测试下的经验均值与标准差，进而通过组内标准化直接推导样本的相对优势，在保留比率截断与参考模型KL惩罚的同时显著精简了系统架构。 然而，授课者强调标准GRPO存在明显的内在病理。在数学与复杂推理任务中，GRPO会诱发模型生成思考链，但模型的平均响应长度往往持续膨胀，甚至在准确率见顶后依然失控。其数学根源在于损失函数中对完成长度取倒数的归一化因子：当输出较短时梯度权重被放大，导致负优势惩罚过重；而长文本的单词元惩罚被过度稀释，反向激励模型通过拉长文本以规避惩罚。为此，后续研究提出了针对整批次所有词元聚合归一化的DAPO、直接移除序列长度归一化因子的Dr. GRPO，以及消除低方差任务标准差失真和采用非对称截断边界等改进方案。 基于反向KL散度的策略内蒸馏演进 (88:51) 由于GRPO通常采用基于最终结果的监督方式，所有词元均分相同的宏观优势标量，仍无法根本解决中间幻觉推理步骤的局部信用分配难题。为在不重新引入辅助价值网络的前提下实现密集的词元级监督，讲座探讨了策略内蒸馏（On-Policy Distillation, OPD）。与离线静态数据集蒸馏不同，OPD保持严格的策略内生成，即学生模型自回归生成词元序列，冻结且更强大的教师模型则针对累积上下文评估每个生成词元的概率分布。 由于优化建立在学生模型生成的轨迹分布之上，目标函数采用反向KL散度。这种形式具有强烈的模式寻求（Mode-Seeking）特性：当教师认为某词元出现概率极低而学生模型赋权较高时，比率会急剧增大引发严重惩罚，从而强力抑制推理幻觉；反之若学生未覆盖教师的全部广度，则受罚较轻。该方法不仅消除了强化学习中由于代理奖励缺陷造成的奖励作弊漏洞，还在参数高效微调（如LoRA）场景下展现出极高的FLOP利用效率，有效缩小了低秩与高秩配置之间的性能鸿沟。 自蒸馏机制与多教师工业落地架构 讲授者进一步剖析了策略内蒸馏在工业级落地中的边界条件与工程拓扑。OPD的根本瓶颈在于依赖外部更优教师模型；当研发前沿基准模型时，往往不存在水平更高的外部指导者，且教师本身的幻觉与特定领域盲区也会误导学生。为突破此限制，讲座提出了策略内自蒸馏（OPSD）：学生在无特权提示下自回归生成回答，而同一基座模型在教师通道中则被直接投喂包含标准答案与完整推导链的特权上下文，通过教师强制技术输出更精确的词元分布，实现自我进化。 在现代AI企业的梯队产品体系中，通常采用分层混合部署：旗舰主力模型（如Pro系列）直接运行基于可验证奖励的底层强化学习（如GRPO或硬性事实校验）以探索前沿能力边界；轻量低延迟推理模型（如Flash系列）则通过策略内蒸馏快速承接旗舰模型的能力迁移；自蒸馏技术则常用于新旧检查点对齐以防止灾难性遗忘。此外，为防止单一通用教师在复杂多学科上产生信号干扰，前沿团队采用多教师策略内蒸馏，让学生模型在形式化数学证明、代码执行与通识推理等不同垂类中，分别接受经各领域可验证奖励专项训练的专家教师指导，确保全局训练的稳定性。 强化学习与大模型对齐技术已从复杂的在线Actor-Critic体系分化为两大互补演进路径：一条是以GRPO及其修正变体为代表的无价值模型强化学习，依托可验证环境奖励持续拓展前沿逻辑边界；另一条则是借助反向KL散度的策略内自蒸馏与多教师协同蒸馏，在免除额外评估模型的同时提供密集词元监督。但在缺乏外部更优指导者且任务缺乏确定性判别机制的非形式化领域，如何在不依赖庞大批评网络的前提下实现高质量中间信用分配，依然是工业界与学术界面临的核心开放课题。

[67:16] GRPO stands for group relative policy optimization, and what it does is it tries to maximize advantages, but without a value function. [49:25] Instructor: The reward model is trained beforehand and the value model is trained at the same time as when you're training your policy.

[0:06] Hello everyone and welcome to lecture 4 [0:10] of CM295. [0:12] So today is a very special day because [0:14] we'll be talking about reinforcement [0:17] learning based training methods that are [0:20] actually getting quite popular [0:22] especially when it comes to postraining [0:25] LLMs. And the reason why today's session [0:28] is a little bit special is because [0:32] contrary to the previous ones there's [0:34] going to be a little bit of math [0:36] involved [0:37] which is why last time we said all the [0:40] RL based methods we will just reserve [0:43] them for today. So today is the day and [0:46] yeah so I wanted to start with this [0:48] disclaimer [0:50] and the second thing I wanted to say is [0:53] we will try to start from intuition [0:55] first and then stay rigorous on the math [0:59] but please don't be intimidated by the [1:02] big formulas. Yeah, I hope it's going to [1:04] be a good one. [1:06] With that we're going to start as usual [1:09] by recapping what we saw last time. So [1:13] if you remember last time we were [1:15] talking about training LLMs and the [1:19] first step of training LLMs is called [1:23] pre-training. [1:24] It is a step that is meant to teach the [1:27] model how to understand text, how to [1:30] understand code. And this step typically [1:33] involves training your model on a lot of [1:37] data. By a lot of data I mean trillions [1:40] of tokens. So there's a lot of work [1:43] around finding the right mixtures. How [1:47] much web should it be in there? How much [1:49] books should be in there? And so on. And [1:52] the loss function for this one is the [1:55] cross entropy loss which tries to [1:58] penalize any deviation from the token [2:02] that the model needs to predict. So it's [2:04] next token prediction. And we saw some [2:08] emerging stages such as mid training [2:11] which involves higher quality data and [2:14] also long context training which [2:17] involves training your model on longer [2:20] context data as well. So that's the [2:23] first step. The second step that we saw [2:26] was called supervised fine-tuning aka [2:29] SFT. And in this step, we're trying to [2:34] take our model that understands text, [2:36] that understands code, and tune it to be [2:40] useful. So what do I mean by useful? For [2:43] instance, if you want to have an [2:44] assistant, you would want your model to [2:48] give answers to questions. And that is [2:51] one of the examples where people would [2:54] use SFT4. [2:56] So in ter in terms of the loss function, [2:58] we still have the cross entropy except [3:01] here we're only trying to penalize [3:04] deviations for token predictions that [3:07] are on the output side. So it's next [3:10] token prediction but for output [3:13] at the end of this stage [3:16] our model understand text our model is [3:19] useful but here we may want to inject [3:22] human preferences and we saw this third [3:25] step called preference tuning and this [3:29] is the step that involves RL based [3:32] methods that we're going to see today. [3:35] We did not see any specifics about the [3:39] exact loss formula. So we're going to [3:42] see today. [3:44] And the other piece that we saw was the [3:47] fact of increasing the model's reasoning [3:51] capabilities. So reasoning which also [3:55] happens to be performed with these [3:57] RLbased techniques. And we're also going [4:00] to see that today. [4:03] And this is where we left off last time. [4:06] And today we're going to focus on these [4:09] last two stages and in particular [4:13] emphasize how we come up with the loss [4:16] which right now we don't know what the [4:18] loss is and see how we can use that. [4:24] Cool. Um I want to again have a big [4:28] disclaimer here that these methods [4:30] they're typically mathematically [4:32] involved not just for this lecture but [4:35] if you were to read papers [4:37] on that topic and so if you feel like [4:42] it's too much here is the proposed steps [4:46] for remediation. So first one focus on [4:49] the intuition on why a certain thing is [4:52] needed. Second one is if there are [4:55] formulas or symbols that you are not [4:58] familiar with just make sure to review [5:00] them. And then the third one is just to [5:03] interpret the formulas from first [5:04] principles. [5:06] And I actually have a second disclaimer [5:10] which is that on top of all of that [5:13] papers they tend to choose conventions [5:16] that are different from one paper to [5:18] another. So that may add to the [5:21] confusion. So we will see that for [5:23] instance we're going to talk about [5:26] rewards. Rewards are usually noted R but [5:30] sometimes in papers R could also mean a [5:33] ratio ratio of probabilities. [5:36] So here to make sure we distinguish them [5:40] properly we will not them row in this [5:42] class. Second one is the policy and we [5:46] will see what a policy is. Um there are [5:50] several kinds of policy. Is it the old [5:53] policy? Is it the reference policy? And [5:55] sometimes these notations are also [5:58] mixed. And then the third one to call [6:00] out is that so far we've been focusing [6:05] on minimizing a loss function and we [6:08] noted the loss function L. [6:12] Well, here most of the time we will want [6:14] to maximize quantities. But the problem [6:18] is sometimes these quantities are also [6:20] noted L which usually you think of as [6:23] being a loss. But just make sure that [6:26] you know you know what you want to do. [6:28] Do you want to minimize? You want to [6:29] maximize. So that's also another thing [6:31] to keep in mind. [6:34] Cool. Well, with all of that said, let's [6:38] start. And I wanted to first start with [6:41] a refresher on what reinforcement [6:43] learning is. So, I'm not sure if any of [6:46] you has ever taken a reinforcement [6:48] learning based class, but at a very high [6:51] level, the way it works is you have an [6:55] agent that is interacting with an [6:58] environment [6:59] that is in a given state and that [7:03] wonders what action to take to go to [7:05] another state. [7:08] So the way it tries to decide on which [7:12] action to take is something that is [7:16] leveraging a policy. So pi theta is [7:20] typically how it's denoted [7:23] and after the agent performs an action [7:28] it receives a reward. [7:30] And the goal of RLbased problems is to [7:34] maximize the sum of rewards that your [7:37] agent is collecting. [7:39] Now you may be wondering well how does [7:42] that have to do with LLMs? [7:45] So here we're going to see a little bit [7:47] how things map to the LLM world. So the [7:52] agent here is the LLM [7:55] that takes some input [7:59] as a state and tries to decide what to [8:02] do next. So what token to sample next. [8:05] So the action is a token [8:08] and in order to decide what token to go [8:13] next to. What it does is it relies on [8:16] the probability distribution that it [8:18] outputs given an input. And so here that [8:22] one will be noted pi theta. So back to [8:24] the notation last last time. So this is [8:26] where the pi is coming from. Pi as in [8:29] policy [8:31] and it collects rewards. [8:35] So one note here is that it can be [8:38] challenging to define token level [8:41] rewards. [8:43] So a lot of the time we will also [8:46] encounter this setup not at the token [8:49] level but at the response level or at [8:53] the completion level. And so here you [8:56] can also consider an input as being your [8:59] prompt, [9:01] an action as being your response [9:05] and pi theta as being just a probability [9:08] of you outputting a response given your [9:10] prompt [9:12] and then as a result of that you will [9:14] get a reward. [9:18] So, we're going to spend a few minutes [9:19] on the rewards part and I know we spent [9:24] uh maybe 10 minutes last time talking [9:27] about this in different settings. So, [9:29] here I just want to summarize what we [9:31] saw. [9:33] So, [9:35] in order for you to quantify how good [9:38] the response is, [9:40] you actually have typically two [9:42] different categories of situations you [9:45] can be in. [9:47] The first one, the first category is [9:49] where there is no absolute ground truth [9:53] to the answer, where actually judging [9:55] your answer is a little bit more [9:57] subjective. [9:58] And this is where where we saw in the [10:02] preference tuning case where we could [10:05] have human preferences that try to [10:09] uprank let's say responses that have [10:13] let's say better tone compared to ones [10:16] that have a a worse tone. [10:19] And this is something that you would [10:21] typically model. You would model with a [10:25] reward model that leverages what we saw [10:29] last time the Bradley Terry formulation [10:34] which is a way to express your loss in a [10:36] parise fashion [10:39] and train a reward model. [10:42] So here the first category is you don't [10:44] have a ground truth [10:46] and you try to come up with a way to [10:48] quantify how good responses are for [10:50] instance with human preferences and you [10:53] would use the Bradley ter formulation [10:56] and these methods [10:59] would be called RLHF because you would [11:02] use human feedback as a way to quantify [11:06] how good the responses are. [11:10] And then the second big category of [11:14] situations you can be into is where [11:17] you can actually [11:20] verifiably tell if your answer is [11:23] correct or not. And so here some [11:26] examples of situations could be, you [11:28] know, if you want to solve a math [11:29] problem, you know what the solution is. [11:31] Or if you want to write a piece of [11:33] software, you know if your solution is [11:37] correct or not. if let's say it passes [11:39] the the test cases. [11:42] So you have a a deterministic way to [11:45] check whether your answer is correct or [11:48] not. And so these methods you will also [11:52] find them under the name RLVR [11:56] reinforcement learning with verifiable [11:58] rewards. [12:00] And these are typically in the field of [12:03] reasoning so math problems and and [12:06] coding problems. [12:09] So this is how you can have your rewards [12:13] and so it would be more at the [12:15] completion level not at the token level [12:16] at the completion level. [12:20] So given this definition of reward you [12:24] may want to tweak the parameters of your [12:28] model to maximize the rewards that [12:32] you're getting. So coming back to the [12:34] examples that we saw last week in the [12:36] preference tuning case, what you want is [12:39] to align your model with human [12:41] preferences. So you would start with a [12:43] prompt and then you would generate a [12:47] completion with your LLM [12:50] and then given the completion that you [12:52] would generate, you would pass in the [12:54] prompt and the completion into a rework [12:57] model that is frozen that tells you [13:00] whether the answer is good or not. with [13:02] respect to the the prompt and depending [13:06] on how the reward model responds [13:10] you would tweak the parameters of your [13:12] LLM in order to maximize the rewards the [13:16] expected rewards that it is generating. [13:20] So this is the preference tuning case [13:22] and you have similarly [13:24] for the reasoning case when you have a [13:28] prompt uh that is maybe more of a [13:30] reasoning prompt or a coding prompt and [13:33] here you have a deterministic way to [13:37] evaluate the quality of your answer. And [13:39] so here you would only look at the final [13:43] answer and you would use that in order [13:46] to tweak [13:48] the parameters of your LLM which is here [13:51] your policy. [13:55] Okay. So now we talked about the [13:58] setting. Now let's go into the math. [13:59] Yeah. [14:01] So the question is would the probability [14:04] of a full response be very low? Uh yes [14:07] indeed it would. That is the reason why [14:09] people use logarithm in order to have [14:11] some numerical stability. But yeah, [14:13] great point. [14:14] Yeah. [14:17] So the question is, do you need a human [14:18] every time you generate a completion? So [14:21] the answer is no, which is the reason [14:22] why you're actually training a reward [14:26] model firsthand. [14:28] You're training it at human preferences. [14:31] And the reason why you're doing that is [14:33] because you don't want to have a human [14:34] in the loop every time you generate a [14:37] completion. And what you would do is to [14:39] use that reward model for any responses [14:42] that you're generating. And you're using [14:44] that reward model in order to assess how [14:47] good the response is. So yeah, you are [14:50] actually not leveraging humans in this [14:52] loop. [14:55] Okay, perfect. So I was saying let's go [14:59] into the math. So [15:02] here our objective function is to [15:06] maximize the expected rewards that [15:08] you're getting. So in mathematical [15:11] terms, what you want to do is to find [15:15] parameters theta of your model that [15:19] maximize the objective function, which [15:22] is the expectation of rewards [15:27] as you're sampling trajectories from [15:30] your policy pi theta, which is your [15:33] model. [15:36] And so I know what you're thinking. [15:41] In machine learning when you want to [15:43] optimize an objective function typically [15:45] what you want to get is the gradient [15:48] because you want to perform your [15:49] updates. So the first question that we [15:52] want to tackle here is how do you get [15:55] the gradient of this objective? [15:58] And so I know a lot of us are coming [16:00] from different backgrounds and I want to [16:02] make sure we're speaking the same [16:04] language. So, a little refresher on [16:08] gradients. [16:10] You can think of gradients as a way for [16:13] you to get the direction and the [16:17] magnitude [16:19] of the steepest increase. [16:22] So, in the 1D case where your function [16:28] has an input x that is just one number, [16:33] you all know the derivative. So if the [16:35] derivative is positive, what that means [16:37] is your function increases as x [16:39] increases. If it's negative, it [16:41] increases as x decreases. [16:45] So the generalization of that for the n [16:48] dimensional case is the gradient and the [16:52] gradient of f in this case is the [16:56] partial derivative of that function with [16:59] respect to all the dimensions of your [17:03] input x which is in Rn. [17:08] Are we all good on this? Okay. So with [17:12] that being said, let's go back in to our [17:15] original problem which is we have an [17:17] objective function to maximize and what [17:20] we want to do is to compute the gradient [17:24] and so I will just do that. [17:28] So [17:31] what we want is for us to maximize the [17:35] expected rewards. [17:38] And so in mathematical terms what that [17:40] means is we want to maximize [17:44] uh the rewards [17:51] such that the trajectory is sampled from [17:54] your policy. So this is exactly what [17:57] what is on the slide means. [18:00] And so if you want to take the gradient [18:02] of that [18:06] well you would just compute what is in [18:10] between uh the integral. So what you [18:14] would do is [18:18] just wonder for a second what is a [18:20] function of theta. So is r a function of [18:22] theta? No it is frozen. Is pi theta a [18:26] function of theta? Yes. So what you [18:29] would do is actually go and put the [18:32] gradient in there and of course you have [18:36] like some regularity that you assume [18:41] is in place. [18:44] Okay, mathematical trick. What is the [18:46] gradient of pi theta? So you may know [18:51] that the gradient of log of a function [18:55] is the gradient of that function [18:58] over that function. [19:00] So this a mathematical results. So what [19:03] that means is that the gradient of a [19:06] function is equal to the gradient of the [19:08] log of that function times [19:12] that function. [19:14] So here this is equal to the gradient of [19:18] the log [19:20] of pi theta time [19:24] pi theta. [19:28] And so what does that mean? [19:32] I'm going to just rewrite everything so [19:34] that we're all clear. [19:38] So it's integral of r toao gradient of [19:44] log [19:46] of pi theta [19:50] pi theta of toao [19:53] and d to. [19:57] So this is nothing else. [20:03] This is nothing else than the expected [20:07] value [20:10] of your rewards [20:14] times the gradient [20:18] of the logarithm [20:21] of pi theta [20:25] where thou is sampled [20:28] from your policy of parameters theta. [20:35] Does everyone agree with the math? Yes. [20:38] Uh so the question is how did you [20:40] introduce the log? So I just [20:44] used the fact that the gradient of the [20:49] log of a function let's say a fa is [20:51] equal to the gradient of [20:56] that function over [21:00] that function. So it's just a [21:01] mathematical identity. [21:07] Yes, exactly. So what we did is we just [21:10] put that here. [21:14] >> Yes. Exactly. [21:15] >> Got you. [21:18] >> Great. Does everyone agree with the [21:19] math? [21:23] >> Okay, cool. [21:25] So the reason why I'm doing the math on [21:27] the blackboard is not because I like [21:29] writing on the blackboard, but I mean I [21:31] do. But here you will see this formula [21:34] quite a lot. And it's just important to [21:37] know where this comes from. [21:40] And so what that means is that the [21:42] gradient of your objective function is [21:46] incentivizing [21:48] your model to increase [21:52] probabilities of actions that have a [21:55] positive reward [21:57] and decrease the probability of actions [22:00] that have a negative reward. [22:03] Yep. [22:05] So a question is what does the tilda in [22:08] the to tilda pi of theta mean? So that [22:12] means that you're sampling your [22:14] trajectory from the probability [22:18] distribution given by your policy of [22:20] parameters theta. So tilda means sampled [22:23] from. [22:28] So again I will just [22:31] repeat the intuition. So what you can [22:34] take away from this formula is that if [22:38] the reward is positive, [22:41] the gradient will incentivize your model [22:44] to increase the probability [22:47] of such actions and it will incentivize [22:50] your model to decrease the probability [22:54] of actions that have a negative reward. [23:00] Okay. So based on that [23:03] there's one little thing that I want to [23:05] tell you about which is that we talked [23:07] in terms of trajectories [23:10] and in the LLM case what we typically do [23:14] is we first have a set of prompts the [23:17] X's and then we sample the responses [23:22] from [23:23] the policy of our model given these [23:27] prompts. [23:30] So the reason why we only did the tow [23:33] sample from pi theta is just because I [23:36] didn't want to uh to make it more [23:39] confusing on the blackboard. But just [23:41] think of expectations as being over [23:44] prompts and then over responses given [23:46] these prompts. [23:50] Okay. So now you may wonder when how can [23:54] you update the weights of your model [23:56] given what we saw. So here what we would [24:00] do is we would have a prompt X and we [24:03] would generate a completion Y sampled [24:07] from [24:09] our LLM given the input X. [24:14] What we would do is we would evaluate [24:16] the reward that is associated with the [24:19] prompt and the response [24:22] and then we would compute the gradient [24:25] which we saw is r times the gradient of [24:30] the log of pi. And so here's exactly [24:32] what's on the slide. [24:36] And then you would update your [24:37] parameters. [24:38] So you may be familiar with gradient [24:41] descent. So it's theta minus. But here [24:43] we want to maximize the objective. We [24:46] want to maximize our model generating [24:50] high rewards. So here it's actually [24:52] theta plus [24:55] alpha times the gradient because it's [24:56] gradient ascent. [25:00] So one thing that I want to say [25:03] is that contrary to [25:07] pre-training contrary to SFT [25:10] here we need to always generate the [25:13] response because here we're in an on [25:16] policy setting we're generating the [25:17] response and then performing the update [25:20] on the response generated by the model. [25:22] This is not a fixed response as is [25:24] generated by the model. So the problem [25:27] with that is you're asking your model to [25:30] generate responses every time you want [25:31] to make an update [25:34] and that step is very expensive. You [25:38] cannot parallelize it because you need [25:40] to ask your model okay what is the next [25:42] token and then you take that token and [25:43] you ask what is the next token and one [25:45] at a time you're generating this [25:48] responses [25:50] and so one thing that I want to tell you [25:53] is a method [25:57] that people can use [25:59] in order to limit the number of times [26:03] they need to generate completions [26:09] And I want to write that down. [26:14] So here what we're saying is that we [26:17] want to maximize the expected rewards [26:21] when the trajectory is sampled from the [26:24] current policy. [26:26] So we want to maximize this. [26:32] So we want to maximize the expectation [26:36] of r of toao when r sorry when toao is [26:42] sampled from the current policy [26:48] which is equal to the integral of r of [26:50] toao pi theta of toao [26:55] d to [27:00] another way of writing this could be to [27:02] say if we have let's say another [27:08] policy [27:09] to fixed [27:13] we could express this quantity [27:17] as a function of to fixed by [27:22] letting this be popped up. So here we do [27:25] pi pi theta of toao over pi [27:30] theta fixed [27:33] of toao time pi theta fixed [27:42] of toa da which is equal to the [27:44] expectation [27:47] of r [27:51] of toa [27:54] times pi [27:57] theta of toao [28:00] over pi pi theta fixed [28:06] of toao [28:07] where toao is sampled from [28:11] pi theta fixed. [28:15] So I want to just highlight. [28:19] So we went from maximizing the expected [28:23] value when TOAO is sampled from pi theta [28:27] of a quantity [28:31] over to a quantity which samples the [28:34] trajectory from [28:37] a policy of fixed like given parameters. [28:43] So why did I do that? [28:46] So let's imagine for a second that we're [28:49] tired of always generating completions [28:51] every time we want to perform an update. [28:54] Let's assume we generate completions [28:58] let's say once [29:00] and then we perform updates [29:04] using the completions [29:07] of that one time [29:10] but by reweing [29:13] the rewards for given trajectory [29:16] with the ratio of the probability [29:20] of your current policy over your fixed [29:23] policy. [29:26] So the reason why I wrote that is [29:28] because sometimes we don't want to [29:30] always do that. We want to have a fixed [29:32] policy that does that once and then just [29:35] rewe the terms in a way that makes it [29:38] still mathematically valid because this [29:41] is what we care about. [29:46] So one thing that I want to add is this [29:50] term [29:52] is called importance [29:56] sampling ratio. [29:58] Yes. [30:00] So the question is I don't get the [30:02] benefit. We're still computing pi theta [30:04] of toao like what is the benefit? So the [30:07] the trick is [30:09] evaluating the policy is different with [30:13] generating a rollout because evaluating [30:16] the policy you're taking everything as [30:19] input in parallel and computing [30:21] everything all the probabilities. But [30:23] here when you do the roll out as in when [30:25] you generate a response you need to take [30:28] the first token to generate the token [30:30] and then take that token and generate [30:33] the next token [30:35] over and over. [30:37] So this is expensive. This is cheap. [30:43] Yeah. So the question is uh please [30:46] explain a bit more what you mean by this [30:48] is cheap. So here you know the full [30:50] response. You know the full response. So [30:52] you're putting all the response as [30:54] inputs [30:57] all in parallel. So it's all being [30:59] computed in in a nice parallel fashion. [31:04] So maybe I can [31:07] I can do something else that could be [31:09] more helpful. [31:12] >> Do you think [31:18] Yeah, exactly. So um [31:23] so here [31:25] so generation [31:32] what you're doing is you're putting so [31:35] this is your model [31:37] putting a token in [31:40] token out [31:43] that token [31:46] in [31:51] token out and so on. But for evaluation, [32:00] what you're doing is you already know [32:02] the full completion. You already know [32:03] it. You know all the tokens. [32:09] So you're putting everything as input [32:11] all at once. [32:15] And you know like your LLM is a decoder [32:17] only LLM. So it has the masked self [32:20] attention. So it's able to nicely only [32:23] attend to itself and the ones before. [32:24] All of that is very parallel. And you [32:27] can get the corresponding probabilities [32:30] as output. [32:34] This is expensive. This is cheap. [32:40] Okay. Great question. Yep. [32:43] So the question is we're still doing the [32:45] expensive calculation with theta fix. [32:47] Yes, but you can do it once and reuse [32:49] that several times. You don't need to do [32:52] that every time. [32:55] Question is for multiple training steps. [32:56] Yes. So we will see that more in detail. [32:59] I just didn't want this to come out of [33:00] the blue. This is the reason. [33:05] So now what I want to do is to go [33:09] through some challenges that such a [33:12] vanilla approach can exhibit. So this is [33:16] this is not one challenge. This is just [33:18] a like a small FYI. [33:24] The first challenge that I want to [33:26] mention is that if we use a reward [33:31] function [33:32] that does not have a con concept of [33:36] relative compared to some baseline, [33:40] we may get gradients that are of high [33:44] variance. So I'll give you an example. [33:47] Let's assume we have a prompt and we [33:50] have two answers. Let's say one is good [33:52] and the other one is bad. [33:55] In the first case, the good answer has a [33:57] reward of 101 [34:00] and the bad answer has a reward of 100. [34:05] So you know that 101 is greater than 100 [34:10] but both are very high and you don't [34:12] exactly get the difference relative to [34:14] one another. [34:18] The problem is that this will need this [34:21] will lead to noisy gradients because you [34:24] know that your model will update [34:28] parameters as a function of R [34:32] and the gradient of the log of the [34:34] policy. [34:36] So if R is always big, it will make big [34:40] updates. [34:42] It will make big updates to get good [34:45] answers [34:47] pop up more. It will make big updates to [34:50] make the bad answer pop up more but in a [34:52] in a lower magnitude because the reward [34:55] is lower than the one for the than for [34:57] the good one. [34:59] So at the end of the day if you do that [35:01] enough times you will get the parameters [35:04] that you want to get but you will have [35:06] very high variance gradients as in you [35:10] will have big updates for good then big [35:13] updates for for bad and so on. [35:17] So what you want is to actually get a [35:21] sense of what is the expected reward [35:24] that you should receive for a given [35:27] state [35:28] and then contrast [35:31] the reward of each action with respect [35:33] to that. [35:36] So if you say that okay on average 100.5 [35:41] is you know roughly okay then if you [35:44] have point plus.5 [35:47] then you know it's good and if you have [35:50] minus.5 then you know it's not good [35:54] and you can show on a toy example [35:58] that the gradients that you get as a [36:00] result of that are of much lower [36:03] variance when you do have that baseline [36:06] when you do have that sense of [36:10] relative to something. [36:13] So that's the first challenge. [36:16] The second challenge is that the two [36:19] types of reward that we saw are at the [36:22] completion level. Meaning that you [36:25] either know that everything is good or [36:27] either know that everything is bad. You [36:29] either know that everything is correct [36:31] or you know that everything is not [36:33] correct. Well, I'll tell you this [36:35] example. So, let's assume your prompt is [36:38] about Teddy who's in a sealed box and [36:41] you want to you're actually moving the [36:43] box to the kitchen. And now you're [36:44] asking yourself, where is Teddy? [36:47] And let's assume your model says, "Okay, [36:49] step one, Teddy climbs out of the box. [36:52] Step two, Teddy walks to the kitchen. [36:54] And step three, answer kitchen." [36:57] So, you know that the answer is correct [36:59] because you [37:02] move the box to the kitchen, [37:04] but what is not correct is Teddy who's [37:07] working because Teddy is in the box the [37:09] whole time. [37:12] And in the vanilla case, you would [37:17] incentivize all tokens of this response [37:19] to happen more frequently. And this is [37:22] not what you want to do. [37:24] So you're lacking an explicit token [37:27] level credit assignment. That is [37:31] something that we need to work on. [37:34] And then the third one, I think we [37:36] briefly mentioned it last time, [37:40] the policy that you already have at the [37:42] beginning of this training stage is [37:45] already at a good state. You already did [37:48] pre-training, you already did your SFT, [37:50] your model is useful, and you don't want [37:53] your trained model to deviate too much [37:57] from the model that you trained because [37:59] it's already good. [38:02] And on top of that, sometimes the [38:04] rewards that you have, they're not [38:07] exactly aligned to your end goal. So [38:10] I'll give you the following example. So [38:13] let's assume me I come here and in order [38:18] to [38:20] in order to know if I did a good job at [38:23] giving this lecture I would rely on the [38:26] intensity of the claps that you would [38:28] make at the end of the lecture. So if I [38:31] were to only optimize for that [38:34] then that would not be ideal because [38:36] maybe I would just spend hour or two [38:39] hours making jokes that are not related [38:41] to the class. I would still receive [38:43] clubs maybe, but it would not be aligned [38:46] with my objective. [38:48] And so there is this risk of reward [38:51] hacking where your reward model may [38:55] actually be imperfect [38:57] and you may have your model who's just [38:59] trying to game the reward model and you [39:02] don't want that. [39:04] So I hope I convinced you that we don't [39:07] want the policy we don't want the model [39:10] to be too far from the model that you [39:13] already have. So we want to have some [39:17] sense of constraints in there and we [39:21] will see how we can quantify the [39:23] difference between distribution. Yeah. [39:27] So the question is uh what do we mean by [39:29] don't want to regress on previous tasks. [39:30] So your model was trained for specific [39:33] things and you're now asking it to be [39:35] better at something else. And so the [39:40] capabilities that it has acquired can be [39:44] something that it loses if you emphasize [39:47] too much on the task of interest here. [39:51] And so you also want your model to not [39:53] deviate too much because of that. [40:00] Okay, cool summary. [40:02] What are the three challenges? The first [40:04] one is gradients can be noisy. [40:07] Second one is we don't have out of the [40:10] box an explicit token level credit [40:13] assignment. [40:15] And the third one is we don't have any [40:19] explicit constraints on how far the [40:22] policy can go with respect to the policy [40:24] that we're starting from. [40:27] So with these three challenges in mind, [40:30] what we will do is talk about the most [40:34] common RL algorithms that are out there, [40:37] which I'm sure you've heard of. [40:40] We will start with the first one, PO. [40:45] So PO stands for proximal policy optim [40:49] optimization. [40:51] And what it tries to do is it tries to [40:55] maximize your objective function [40:58] but with two differences compared to the [41:00] first one. [41:03] The first difference is it tries to [41:05] maximize advantages and not rewards [41:10] and the second difference is it tries to [41:12] not deviate too much from the policy [41:15] that you had before. [41:18] And we will see exactly how these two [41:20] terms are obtained. But just one note, [41:24] PO the P in PO is for that second term. [41:28] You don't want you want a bit of [41:29] proximity [41:32] to that policy of reference. [41:35] Okay. So with that being said, let's go [41:38] into the first term which is the [41:41] advantage term. [41:43] So the reward function that you have is [41:46] wondering how good is this answer with [41:50] respect to the prompt. [41:52] But what we really want to ask is [41:55] compared to what we would expect for [41:57] this prompt, how good is this response? [42:02] So we want to have some form of relative [42:05] to a baseline and that is what [42:08] advantages are aiming for. [42:12] And in order to illustrate that, let's [42:15] go through an example. So let's assume [42:18] that we have the following prompts. [42:20] Teddy's in a box. Where is Teddy? And [42:23] then you have your response in the box. [42:27] So the reward for that one, let's assume [42:30] you it's a correct response. So the [42:32] reward for the whole completion relative [42:34] to that prompt is let's say plus one. [42:38] But you want to know [42:40] how these different tokens [42:43] are actually receiving the credits [42:45] relative to what they are expecting. [42:50] So [42:52] the question that we want to ask [42:53] ourselves is given a partial completion, [43:00] what is the expected final reward [43:04] if you were to continue sampling from [43:07] the policy for the rest of the tokens? [43:10] So I'll give you an example. [43:12] So let's assume you have the whole input [43:15] and then you're decoding let's say in [43:19] here you want to ask yourself okay if I [43:21] were to continue like this on average [43:24] what would be the expected final rewards [43:28] and here let's suppose it's 6 then you [43:31] decode you go to the okay 7 [43:36] box it's the correct answer so it should [43:38] have a big bump so let's say 0 95 [43:42] and one. [43:44] So what did we do here? [43:47] What we tried to do is to ask ourselves [43:51] given a partial completion [43:54] what the expected final reward would be. [43:58] And the nice thing with this approach is [44:02] we have a sense of a baseline. And the [44:05] baseline here is [44:07] compared to [44:10] how it would have completed if you were [44:13] to sample the policy [44:15] and you can exactly pinpoint [44:18] the contribution [44:21] of a given token when you do that. So in [44:24] this example, if you were to compute the [44:28] very simple [44:31] version of the advantage for the token [44:33] box, you would take the value with box [44:40] and subtract the value without box. So [44:42] it's 0.95 - 7. So plus [44:46] 25. [44:50] So [44:51] I hope this example motivates the fact [44:54] that we're introducing [44:56] a new function which here we call the [45:00] value function [45:03] that is a function that is defined at [45:05] the token level. [45:07] So what that function does, so the the [45:10] math formula on the slide, what it what [45:12] it says is it tries to determine what is [45:17] the expected future rewards if you were [45:21] to start from the input [45:24] and the partial completion. [45:27] And if you were to continue decoding [45:30] by sampling from your LLM, by sampling [45:32] from the policy, [45:34] what would be the expected value of [45:36] that? [45:38] that is your baseline. So in this case, [45:43] the model is actually trained along with [45:46] the policy because you need to ask [45:49] yourself what would be the expected [45:52] reward if you were to sample from the [45:54] policy. So you need to leverage [45:58] the parameters of the policy that you're [45:59] training [46:02] here. The way you would train such a [46:03] model is [46:06] you would try to predict the reward at [46:08] the end of the completion given a [46:11] partial completion along with the prompt [46:16] and this quantity would be defined at [46:18] the token level. So for each token of [46:21] the output you would have a value. [46:25] So this is the value function and the [46:29] way you would compute the advantage [46:32] is by taking the value [46:36] up to that token [46:39] and do minus the value up to the token [46:41] right before [46:44] and here you're able to tease out [46:47] the impact of that token itself. [46:52] So one thing that I want to say is [46:56] it is actually more complicated than [46:58] that. So this that's why it's in quotes. [47:01] So it's quote unquote. The reason why [47:04] it's a little bit more complicated than [47:06] that is if you were to generate a token, [47:11] you may increase the likelihood of you [47:14] getting a high reward, let's say, but [47:17] you may also put yourself in a position [47:20] where future tokens would benefit from [47:23] the token that you generated. [47:26] So, let me give you an example. [47:28] Um, let's assume you're teaching a [47:32] concept to someone. You're teaching it [47:34] some math concepts. [47:38] You teaching the concepts may lead to [47:42] your friend acing the math exam. So, [47:45] that's your short-term reward. But you [47:48] can also assume that the fact of them [47:54] succeeding the exam is also placing them [47:56] on a good path for succeeding later [48:00] exams and maybe being successful [48:04] at work and then starting a company, [48:07] being a CEO, being a billionaire. So you [48:09] cannot completely attribute you teaching [48:11] the concept to your friend as being the [48:13] sole reason for them being successful in [48:16] the future. But there is some part of it [48:19] that should be attributed to that [48:22] and that is the idea behind the [48:25] generalized advantage estimation paper [48:27] which is not part of the exam. It's not [48:29] part of the scope of this class but in [48:32] case you're interested feel free to look [48:35] at it. The idea is to also look at how [48:40] future values also perform and link that [48:43] back to a given token. [48:50] Cool. So, so far does that make sense? [48:52] So, here what we saw was a way to [48:55] compute [48:57] advantages which are a quantity that [49:00] tells you how [49:03] a given token predicted token is [49:06] impacting your purpose of getting let's [49:09] say high rewards. So you're getting [49:11] token level signal and you also have a [49:15] sense of a baseline [49:17] by comparing your value with the value [49:21] without that given token. Yes. [49:25] So the question is is there any [49:26] interaction between the value function [49:29] and the policy. So these two are [49:31] actually two models. In practice you [49:35] train them jointly. So you will have one [49:38] loss function and have a penalization [49:41] for the value and the penalization for [49:45] uh the actual policy. But in this case [49:49] these two are completely disjointed. The [49:50] value function is not being fed as input [49:52] anywhere. [49:57] Yeah. So the question is the reward [50:00] model and the value model are both [50:02] trained. So the reward model is trained [50:04] beforehand and the value model is [50:06] trained at the same time as when you're [50:08] training your policy. And the main [50:11] reason why you want to do that this way [50:13] is because the definition of your value [50:16] model is to determine the expected [50:21] rewards if you were to continue [50:23] according to the policies that you're [50:25] training. [50:30] Cool. [50:32] Yeah. [50:34] So, uh the question is uh value function [50:36] is a function of theta. So, yes, there's [50:38] a backbone that is shared and the [50:40] gradients are determined by your loss [50:42] function which contains a value loss [50:46] function and then like a policy the [50:48] actual LM loss function. [50:52] So in case you're interested in the [50:54] details, actually the PO paper does have [50:56] some technical details there [51:00] interesting to look at. [51:04] So now what we're saying is we're not [51:07] trying to optimize the expected value of [51:09] the rewards, we're trying to optimize [51:11] the expected value of the advantages. [51:15] Well, not exactly. [51:18] So if you remember [51:21] these, so assuming you can all see the [51:24] blackboard at the top, [51:28] we said that generating completions were [51:32] expensive. [51:35] So do we want to generate completions [51:36] every time? No. So what you want to do [51:42] is to leverage completions for multiple [51:46] updates. And here you can do that by [51:49] generating a batch using pi theta [51:54] and then update pi theta [51:58] on that same batch and the pi theta [52:00] would become pi old because you it's not [52:03] the current policy anymore. [52:06] And so that is the reason why you would [52:09] see in papers that ratio pop up. So [52:13] you're not optimizing for the [52:15] expectation of the advantage. You're [52:17] optimizing for the expectation of the [52:19] ratio times the advantage. And this is [52:23] because you want to adjust for the fact [52:26] that you're actually sampling from the [52:28] old policy, but you want to make sure [52:30] that this is corresponding to the [52:32] current policy. So you have this [52:34] corrective term, which is the ratio [52:37] between the current and the old policy. [52:43] And this is where I want to you know [52:45] give a disclaimer. So some papers they [52:47] note this ratio R. So this can be [52:49] confusing with respect to the rewards. [52:51] So here we will note this row [52:54] and it is called importance sampling [52:57] ratio. [52:59] You have a question? No. Okay. Great. So [53:06] what can happen if you do things this [53:08] way? Well, you can run into the issue [53:13] that we discussed previously where [53:16] you may run into a situation where [53:19] you're updating your theta such that it [53:22] deviates too much from previous [53:24] policies. So what you actually want to [53:27] do is to have some form of a control [53:31] over that ratio [53:33] and you don't want that ratio to be [53:35] let's say too far from one. [53:39] So there is a formula for that and I [53:41] don't want you to look at the formula [53:42] right now. I want us to just take a look [53:46] at it from an intuitive perspective. [53:50] So here is what I want us to do. [53:54] So let's assume that we're interested in [53:59] how much [54:02] your model is updating the policy. [54:05] So here we said [54:08] it's something like you know the ratio [54:10] times the advantage. [54:14] So it's linear as a function of the [54:15] advantage. [54:18] So if you were to draw I'm going to just [54:22] replicate what we have here. [54:25] um the evolution of this objective [54:29] function which I will note J as a [54:32] function of the ratio. [54:36] So if a is positive so if your advantage [54:38] is positive meaning you want your model [54:40] to increase the probability of a given [54:43] action from occurring then you will have [54:47] something like this [54:49] like a is positive. [54:53] Does everyone agree with it? [54:56] So let's assume here we have one. [55:01] So one so I'm just going to recap. So p [55:04] row is pi theta over pi old. [55:09] So one means that [55:13] your current policy and the old policy [55:15] are saying the same thing. [55:18] And so if you're here [55:20] then what you want to do in order to [55:22] perform your updates [55:24] is to maximize J. So how do you maximize [55:28] J? You maximize J by increasing [55:33] the probability of your policy [55:35] performing that action. So you would go [55:38] more somewhere there. [55:41] You would have your new policy be [55:43] updated and and correspond here. [55:46] The problem is if you don't do anything [55:49] to that expression, [55:52] the gradient is always non zero and [55:55] incentivizing you to increase row to [55:59] increase the probability of your policy [56:02] generating that action more and more. [56:06] So what you want to do is to break that [56:10] incentive [56:11] and clip [56:14] that function in order for you to have a [56:18] zero gradient re region where if you [56:21] were to let's say perform an update to [56:23] your policy and it would become here [56:26] you would have no incentive to [56:29] increase row [56:32] and this is exactly ly what is on the [56:35] slides. What we want to do is to clip [56:38] that function to remove the incentive [56:43] from the update to increase the [56:46] probability of a of an action being [56:49] generated and you want to avoid your [56:53] policy from increasing that probability [56:56] even more. So in other words, you don't [56:58] want your policy to deviate too much [57:00] from the old policy. So this is when the [57:03] advantage is positive. Similarly, very [57:06] similar fashion when it's negative. [57:10] Here [57:12] the objective function will incentivize [57:14] your model to decrease your ratio to [57:18] decrease the probability that your model [57:19] has from generating an action that leads [57:24] to negative advantage. So you will have [57:26] something like this. So if you come back [57:29] to the formula, this is what the formula [57:31] is doing. [57:33] Yes. [57:36] So uh the question is uh you had [57:39] introduced the K divergence. Why don't [57:41] you use that? Well, it's my next slides. [57:43] So I will I will say a little bit about [57:45] how that works. But this is one way of [57:49] constraining your model from being [57:51] updated too much. So to your point, [57:54] let's go to the K divergence. So what is [57:56] the K divergence doing? [57:58] So it is an operator that is taking in [58:01] two probability distributions. Let's [58:03] assume P and Q. So here this operator is [58:09] assuming that P is true. [58:13] And it is waiting that by log of PI over [58:16] QI. [58:19] You have also the continuous version [58:21] which is just take the integral. [58:26] So you have a second way of quantifying [58:28] deviations of your model [58:31] by saying you actually want a [58:34] penalization term in your loss function [58:39] that compares the current policy [58:42] and some other policy. [58:45] So here t typically what people do is [58:49] they actually penalize your current [58:52] policy from being too different from [58:54] your SFT policy from your base policy [58:57] your reference policy [58:59] which is why I want you to pay attention [59:01] to the subscripts of every pi that you [59:04] see out there. So here it's pi theta [59:06] with respect to pi ref. Pyreff is [59:09] reference as in the base policy and this [59:13] contrasts with pi old which is the [59:16] policy that you use to generate the [59:19] completions. [59:22] So I'll give you a full story. You start [59:25] with your pyreff. You train your model. [59:28] So you generate a batch. This becomes pi [59:32] old. You perform several updates. And [59:34] then you generate another batch. This [59:36] becomes your pi old. several updates, [59:39] pyreff stays fixed. [59:43] And this is what you're comparing your [59:45] current policy to. [59:49] And so [59:51] I want to very quickly mention a scary [59:55] formula we saw last week, which was the [59:58] one from DPO. [60:00] And I'm not sure if you remember, but we [60:04] had said that there was a supervised way [60:09] to express the RL objective [60:13] by using the Bradley theory formulation. [60:18] And what we said was that the objective [60:21] function was following something that [60:24] was very much into the line of what Brad [60:26] Literary was saying which is something [60:28] like so this one is like L [60:32] minus expectation of [60:36] log of [60:38] sigma and sigma is your sigmoids which [60:41] is sigma of let's say RW minus RL [60:48] And we had said that R was defined as [60:53] beta log of [60:58] your policy [61:00] compared to some other policy [61:03] and here let's say it's a reference [61:05] and we had said that beta was a quantity [61:09] some quantity [61:11] so I just want to [61:13] draw a connection here that the beta [61:16] that you're seeing in the kale [61:17] divergence. The beta that is actually [61:19] penalizing [61:21] for your model being too different from [61:22] your reference model is actually the [61:25] same beta. [61:27] So the higher the beta, [61:30] the more you will try to not deviate too [61:33] much from your reference policy. And I [61:37] just want to show you mathematically why [61:38] that is the case intuitively. [61:41] So [61:43] sigma if you were to draw the function [61:48] is something like this. [61:50] Whoop like this. It's like one and and [61:54] zero. [61:56] So sigma of x [61:58] is 1 / 1 + exponential of min - x. So [62:04] when x is zero it's.5. [62:08] If x goes to plus infinity this goes to [62:10] one and when x goes to minus infinity [62:13] this goes to zero. [62:17] So there is a region [62:19] where the sigmoid evolves very quickly [62:25] and so we had said that the incentive of [62:29] your loss is for this difference to be [62:34] as big as possible. [62:37] And how do you do that? You want to do [62:40] RW as big as possible. And how do you do [62:42] that? You do that by increasing the [62:45] probability of your policy generating [62:47] the winning answer. [62:51] But [62:53] if the beta that you have is a big beta, [62:58] then that means that even a slight [63:00] deviation [63:02] from the reference policy [63:05] will make this whole [63:08] value big [63:10] and you will already be in a region [63:13] where you're already close to one. [63:17] So in other words, if you have a big [63:19] beta, [63:22] your policy will have a lower incentive [63:25] to be far from the reference policy. [63:29] And this contrasts with the lower beta. [63:32] So if if you have a low beta, then you [63:34] need this quantity to be much higher [63:37] than this quantity for this to be big. [63:43] So, I'm not sure if that helps, but all [63:46] that to say that beta is [63:50] actually linked to your [63:54] incentive to want to make this close to [63:58] the reference policy or not. And this is [63:59] the same beta that is in the Dplas. Yes. [64:05] So, the question is roti is it using pi [64:07] old? Typically, yes. And here is this [64:10] pyre ref. Yes. [64:12] So just to confirm there are three [64:14] policies. Yes, there are three policies [64:15] though. So pi theta is the current [64:17] policy that you're updating. Pi old is [64:20] the one that you use to generate the [64:22] batch and then pyrf is your base model. [64:25] Yes. Was it the same question? [64:28] Yeah. So the question is if you only [64:30] care about how a policy deviates from [64:33] its old value, how do you make sure that [64:35] it doesn't deviate too much from the [64:37] reference one? Typically what people do [64:39] is they have these two constraints [64:41] simultaneously. [64:43] They have the clipping of the ratio with [64:45] respect to the old policy. So the [64:48] iteration constraints [64:51] along with the absolute respect to the [64:55] reference policy one. [64:57] So they're both penalizing for deviation [65:01] with the old policy but also with [65:03] respect to the absolute base policy. [65:08] Yeah, [65:10] the the question is how do you compute [65:12] scale? So it's a great question because [65:13] scale is expensive because it's over the [65:15] whole vocabulary. You have some [65:16] estimator for that. Yeah, [65:20] you had a question. No. Okay, great. [65:23] Let's move on. [65:26] So if I were to recap what we saw, we [65:28] saw a way to maximize a quantity that is [65:33] actually as a function of let's say [65:35] baseline which is called advantage. And [65:37] here we used the value function to do [65:40] that. And we also mentioned that we had [65:45] some things in place in order for us to [65:47] not deviate too much from what we had [65:49] before. So we saw the clipping mechanism [65:52] with respect to the old policy but we [65:56] also saw the kale penalty term between [65:59] the current policy and the reference [66:01] policy. So these are ideas but some [66:05] methods they mix both some methods they [66:07] use one but not the other. Some some [66:09] methods mix them. [66:12] There are multiple flavors of it. But I [66:14] think what is important for you to [66:15] remember is the different methods that [66:18] you can use in order to achieve these [66:21] goals. [66:23] And I wanted to have a dedicated slide [66:26] to say if you were to read the PO paper, [66:29] please be careful. L is not a loss. It [66:33] is an objective function. You want to [66:35] maximize that quantity. You don't want [66:37] to minimize. And the second one is you [66:39] will see R for the ratio. Be careful. It [66:42] is not a reward. It is a ratio. And with [66:46] that, I will just recap what we saw with [66:49] respect to the challenges that we [66:50] mentioned. Gradients being noisy. Well, [66:54] we have a way to compute a baseline [66:57] which also turns out to give token level [67:02] credit assignments. And we said we could [67:05] also use either clipping or the k [67:08] divergence methods in order to ensure [67:10] that the policy is not deviating too [67:12] much from what we had. [67:16] But the big problem with this approach [67:19] is that you need to train a value [67:23] function. [67:26] That is a lot of hassle. [67:28] And so this is the reason why this is [67:31] not the only method we're seeing in this [67:33] lecture. There's a second method which [67:36] has gained a lot of importance in the [67:38] past couple of years which is called [67:40] GRPO. [67:42] So GRPO stands for group relative policy [67:47] optimization [67:48] and what it does is it tries to maximize [67:52] advantages [67:54] but without a value function. [67:58] that is the big added value but without [68:02] the value function of gRPO is it doesn't [68:05] need a value function in order to work [68:09] and we're going to see exactly what it [68:11] does instead. [68:13] So back to the intuition that we had [68:17] around [68:18] having something to compare a reward to. [68:23] The intuition behind DRPO [68:26] is given a prompt, you're going to [68:29] sample multiple responses in order to [68:32] get that expected value. [68:36] So you take some prompt, you take a [68:39] positive temperature and when you [68:40] generate several responses, they [68:42] hopefully are all different and you will [68:44] get some responses that are better than [68:46] others. And if you do that and if you [68:50] compute the reward for each of these [68:54] outputs with respect to the prompts then [68:58] if you compare that to the mean of [68:59] reward and you standardize them can get [69:02] a sense of which response is better [69:07] to other one with respect to other ones [69:09] and you don't need a value function for [69:11] that [69:13] and [69:15] of course you also want to not deviate [69:18] too much from what you had before. So [69:20] the same tricks apply. So you have the [69:23] clipping mechanism which is making sure [69:27] that you don't have non-zero gradients [69:29] if you're too far from the old policy [69:32] and you also have that kale divergence [69:34] term which allows you to penalize any [69:37] big deviations from the reference model. [69:42] So it's a lot of math, a lot of formula. [69:45] So let's relax for a second and just [69:48] walk through that illustration together. [69:51] So how would we go about updating the [69:55] weights of our LLM if we were to follow [69:59] GRPO? [70:02] So first we take a prompt [70:06] and ask our LLM which is the policy [70:09] model to generate responses. [70:12] So it would generate one output, it [70:14] would generate two outputs actually g [70:16] outputs [70:19] and what you want is to know how good [70:22] these outputs are. [70:24] So each of these outputs with respect to [70:27] the input would be evaluated with [70:29] respect to your reward model which can [70:32] be a model or or something [70:34] deterministic. [70:37] And then you would compute advantage [70:40] values by subtracting the reward with [70:44] the mean of the rewards of that group [70:46] and dividing this with the standard [70:48] deviation of the rewards. [70:51] So here you get a relative sense of how [70:55] each output performs with respect to the [70:59] group. [71:01] And what you would do is you would use [71:04] these advantages [71:07] in order to compute your gradient [71:11] but you would also make sure to penalize [71:14] for any deviations with respect to your [71:16] reference model. So you would also have [71:18] a kale divergence term along with the [71:20] clipping [71:23] and you would not have any value [71:25] function. No value function needed. You [71:28] only need your policy model. You need a [71:30] reward model and a reference model. And [71:33] by the way, the reward model is optional [71:34] too because if you're doing let's say [71:37] reasoning or math or or coding, you can [71:40] leverage verifiable rewards. [71:44] Okay. So how does that compare with PPO? [71:48] So PPO you took some input, you put that [71:53] into your policy model. You generate one [71:55] response [71:57] and what you do is you input that prompt [72:01] and response into a reward model tells [72:04] you how good how well you're doing. [72:07] There are many variations of how to [72:10] incorporate the kale penalty [72:14] but one of the variations is to [72:16] incorporate as part of the rewards. [72:21] So you would do that [72:24] and in parallel to that you would use [72:27] your value model in order to get a token [72:31] level prediction [72:34] of what is the expected final reward if [72:36] you were to follow that policy [72:39] and you would compute an advantage [72:43] quantity which we saw is if we want to [72:47] be extremely vanilla it can be the [72:50] difference between the [72:51] functions, but it's actually more [72:53] complicated than that. And you have a [72:54] method that does that is called the [72:57] generalized advantage estimation that is [72:59] called GA. [73:02] So you can do that [73:04] and then take your advantage and use [73:07] that in order to compute the gradient [73:10] that will update your policy. [73:15] So I want us to [73:17] by the way, does that make sense so far? [73:22] Yeah. Okay. Um, so with that being said, [73:28] I just want to do a recap of what are [73:29] the models that are involved and which [73:31] ones are the ones that we're training [73:32] versus not. [73:34] So if you remember, the reference model [73:36] is fixed. So it's the one that you [73:38] started from. And you have optionally a [73:42] reward model which you train beforehand [73:46] and that you're leveraging here. you're [73:48] not training your reward model, but what [73:51] you're training is obviously your policy [73:54] model and in the case of PO you're also [73:58] training a value model [74:01] and you're training it jointly with the [74:03] policy model because as we saw the value [74:07] model tries to estimate the expected [74:10] rewards if you were to follow the policy [74:14] and you're actually training the policy. [74:15] So you need to make sure that these two [74:17] are in sync [74:19] and that is the core reason why PPU is a [74:22] hassle because you need to have all the [74:25] bandwidth to [74:27] to host the value model training [74:32] and at the end you obtain these formulas [74:34] and you know I'm glad we didn't start [74:36] from these formulas because they can be [74:38] a little bit intimidating. [74:41] So something that is similar between the [74:44] two formulas the ratios. [74:46] So here we have the ratio between the [74:48] probability given by the policy that [74:50] we're training [74:52] over the probability of the policy that [74:55] is the one that generated the batch. So [74:57] the old policy [74:59] that one is the quantity that you see [75:01] across both formulas. [75:05] You also see the clipping and here we [75:08] saw that the clipping was meant for us [75:11] to make sure that our policy is not [75:15] going too far from the old policy. [75:20] And then we also have a kale divergence [75:22] term. So I mentioned that there are many [75:25] variations in here. GRPO actually [75:28] expresses the K divergence in the loss [75:31] function, but you can very well express [75:34] it as part of the advantage that you're [75:36] computing, which is the case of BPU [75:43] in terms of the differences. [75:46] So the the previous thing that we saw [75:48] was one difference by the way. So where [75:50] you put the K penalty is something that [75:52] can vary from method to method. [75:56] The second difference is the way you [75:59] compute the advantage. [76:01] So in the GRPO case, what you do is you [76:05] compute the advantage at the completion [76:07] level [76:09] and you compare that with the other [76:12] outputs. [76:13] And so by the way if you look at the [76:16] gRPO formula you will see that T denotes [76:20] the token [76:21] and you see that the advantage is a [76:24] function of the output I and the token [76:27] T. And you may wonder, well, why is [76:28] there a T if it's at the completion [76:30] level? You can define the advantage [76:33] either at the completion level or the [76:35] token level, which is why there is this [76:38] more generic formula, but people [76:39] typically use the completion level one [76:42] just because it's easier because the [76:44] token level one you need to have some [76:46] way of distinguishing token level [76:49] contributions, which you're typically [76:51] not doing. [76:53] So anyway going back you see a i and t [76:56] but it's actually a a function of i for [76:58] gpo. So you compute that by comparing [77:02] your reward with respect to the mean of [77:04] the rewards of your group [77:06] and you divide that by the standard [77:08] deviation. But for po [77:11] you're computing the advantage at the [77:12] token level [77:15] and you're using the value function for [77:17] that. And as a reminder, the value [77:19] function is also at the token level [77:22] and it's trained jointly with the [77:24] policy. [77:28] And that's a lot. So I'll just uh pause [77:31] for maybe one minute to see if there is [77:33] any questions on these two big formulas. [77:37] Now if you see this slide, does that [77:38] make sense? [77:43] Yeah. [77:45] Yep. [77:48] So the question is you added in your [77:50] previous slide the K live versions for [77:51] PO. Yes I did. Um but these were more [77:56] meant to illustrate two types of [77:59] strategies to penalize but in the [78:03] general PO implementation you would [78:05] actually incorporate that as part of the [78:06] advantage. [78:11] And by the way it's a common thing. [78:13] Every paper has a different way of [78:15] implementing this. So yeah, don't be [78:18] surprised. [78:21] Okay, cool. Well, everyone is good with [78:24] these big formulas. Great. So let's [78:26] assume you do that for GRPO. It was [78:29] actually something that the authors [78:30] performed on [78:33] math problems on reasoning problems and [78:36] what they saw was so if you remember [78:38] last week we said that this was used to [78:42] incentivize the model to produce chains [78:45] of reasoning before producing an answer [78:48] and it allowed the performance to [78:50] increase. And one thing that is also [78:53] something that the authors saw [78:56] was that the length of the response [79:00] tended to increase as you were [79:02] performing the training [79:05] and it even tended to increase even when [79:08] the performance of the model was [79:10] reaching a plateau. So on the left side [79:13] the y-axis is the performance of the [79:15] model for a math problem. And then on [79:18] the right side it is the average [79:21] response length. [79:25] And so you may wonder well why is that [79:27] happening? Why would the model [79:30] incentivize [79:31] the response from getting longer and [79:34] longer even when the performance is [79:37] reaching a plateau? [79:40] So this has to do with again this big [79:43] formula and how you are performing your [79:46] normalization. [79:48] So here I want you to focus on just one [79:53] piece of the slide. So do you see the [79:56] one over the response of the outputs? Do [79:58] you see this term? [80:02] So when responses are short, [80:06] this term is is it big or small? [80:13] Big. Yes. So when O is small, one / O is [80:17] big. [80:19] How about when O is big? It's small. So [80:23] this coefficient is actually something [80:25] that is multiplied with the contribution [80:29] of a token to the objective function. [80:32] And here we're saying that if you have a [80:35] token that happens to be in a short [80:39] response [80:41] then the contribution will be bigger [80:43] than if it's in a long response. [80:48] Okay. So far nothing incredible. But now [80:51] let's assume you are thinking of it in [80:54] terms of when the advantage is positive [80:56] versus the advantage is negative. [80:59] So if you are in a short output [81:02] then if [81:06] your advantage is positive [81:09] it will be a big contribution upwards. [81:13] If you're in an in a case where it's [81:16] negative it will be a big contribution [81:19] downwards. [81:21] And if you're in a long output, if it's [81:23] uh advantage positive, then it's a [81:25] slight upward increase. And if you're in [81:29] a negative advantage output, it's a [81:31] slight downward contribution. So what [81:34] does that mean? So if you look at the [81:38] cases where you're actually outputting [81:41] bad answers, [81:44] relatively speaking, the model will [81:47] prefer a bad answer that is longer [81:51] compared to a bad answer that is [81:53] shorter. [81:56] So if we go back here, what that means [81:58] is when the model comes to a situation [82:02] where okay, it cannot do anything about [82:04] the bad answers, it just generates bad [82:07] answers, it will then prefer generating [82:11] longer bad answer compared to shorter [82:13] bad answers. [82:16] And that is the reason why in this graph [82:20] you are seeing a plateau of performance [82:23] but the length that is increasing [82:25] because the model prefers longer bad [82:28] responses compared to shorter shorter [82:31] ones. [82:34] Does everyone agree with this? And so [82:38] why is this a problem? Well, it's a [82:39] problem because you don't want that. [82:42] And that is one of the motivations where [82:44] there are some papers that followed GRPO [82:46] that tried to tweak some little parts. [82:49] This included the normalization term. So [82:52] here uh in the case of DAPO, [82:55] it wanted to have the same contribution [82:56] for all tokens. So the normalization is [82:59] with respect to the sum of the outputs. [83:02] And another one GRPO done right or Dr. [83:05] GRPO, I don't know how you would [83:07] pronounce it, but they would just forgo [83:09] that normalization term. [83:12] And I want to talk about other problems [83:15] that can occur. So the one other that I [83:18] want to mention is the way the [83:20] advantages are computed. They're divided [83:23] by the standard deviation of the [83:25] rewards. You will have varying standard [83:28] deviations depending on how complex your [83:30] input is and that may introduce a bias [83:34] and that's not good. So you may want to [83:37] remove the standard deviation. [83:40] And the second one which I wanted to [83:42] come back to your point around epsilon [83:46] is that if you have an action [83:50] that is of a very small probability [83:55] and if you use the same epsilon for both [83:59] the lower bound and the upper bound, [84:02] you're actually not giving a chance for [84:06] smaller probability outputs to be [84:09] generated. even like significantly more. [84:11] So let's assume your probability is like [84:13] let's say 0.001 [84:15] and your epsilon is I don't know 2. [84:18] You only are giving a small amount of [84:20] margin for this to be actually more [84:23] discovered and so it is not encouraging [84:26] diversity. So you may want to choose [84:28] epsilons that are different for the [84:31] lower bound and the upper bound. [84:35] Cool. Okay. wanted to recap JRPO. We saw [84:38] that the gradients are less noisy [84:42] because there are computing advantages [84:44] with respect to the group average and we [84:48] saw that the magnitude of policy updates [84:51] are also handled by the regular suspects [84:54] when it comes to the methods. So, [84:56] clipping and kale penalty, [85:00] but we saw that we didn't have explicit [85:04] token level credits assignments [85:07] and that's not good. We don't have a [85:10] value function anymore, but we don't [85:12] have token level credit assignments and [85:14] we want to solve that and with that gate [85:18] to Shervin. [85:20] Thank you, Ashin. So to illustrate these [85:23] points, we're going to go back to our [85:26] favorite example that we saw earlier in [85:28] the lecture regarding a teddy bear that [85:31] moves within a sealed box. And we're [85:34] going to recap what GRPO and verifiable [85:38] rewards incentivize. So here, regardless [85:41] of the reasoning steps, what matters was [85:45] only the answer. [85:47] So why is it a problem? As Afin [85:49] mentioned, you are going to miss on the [85:54] intermediary steps. So here I showed [85:56] what G completions could give in terms [86:00] of intermediary steps. So you could have [86:01] a combination of steps that go both [86:04] well, steps that go both wrong, or just [86:07] one of the two. [86:10] And as you can see, every token when you [86:14] look at the formula is going to be [86:16] associated to the same end reward. So [86:20] you're going to attribute the same [86:22] advantage as the end even though the [86:24] step is wrong which is one downside. and [86:29] in the longer run. So you might argue [86:32] okay you might have completions for [86:34] which the intermediary steps don't make [86:37] sense but on average they will and on [86:41] average the model will learn exactly [86:43] which ones are good. [86:46] But the thing is it's highly inefficient [86:49] because you see you're not leveraging [86:51] what you can leverage at each step. So [86:54] each of your terms of your loss [86:57] functions might not be correctly [87:00] rewarding the right things. So it's [87:02] noisy locally. And then this phenomenon [87:06] of learning the correct behavior is one [87:08] that is based on the long term. So this [87:12] is not great. [87:15] And Ashin just mentioned a few minutes [87:18] ago that in the GRPO formula you had an [87:21] advantage that was actually written out [87:24] as a function of T. So let's just [87:27] suppose that we're not following the [87:29] common practice of being based on [87:31] outcome supervision. Let's say you're on [87:34] the process supervision methodology and [87:36] you do follow that path. Then as we [87:40] mentioned, you need dedicated labeling [87:43] for these token level advantage to make [87:46] sense. And if you look at the paper, [87:49] they're they're slicing it at the step [87:52] level, which is here what truly makes [87:54] sense. So you need to label each of them [87:57] separately. [87:59] And you might argue if you go back to [88:03] each step [88:05] that you could similarly have an [88:06] ambiguity of exactly what goes wrong in [88:10] that step. So let's say you talk about [88:13] Teddy who walks in the kitchen is the [88:16] issue that Teddy walks or that he walks [88:19] to the kitchen. [88:21] So it could be [88:23] a recursive problem down to the token [88:26] level. would be bounds to label token at [88:30] the ends and a reward on each token [88:33] might be a hard model to train [88:37] and as we mentioned here one of the [88:39] reasons why we moved away from ppu in [88:42] the first place is to avoid maintaining [88:44] the weights of multiple models to train [88:47] so this is not what we want to do [88:51] so if I had to recap the benefits that [88:54] we have in these reinforcement learning [88:57] stages that we want to keep. So we want [88:59] the on policy part to stay so we like to [89:04] train off of the outputs of the model at [89:06] stake. [89:08] Then GRPO had this downside of not [89:12] crediting each token individually but [89:15] rather the overall outcome and hoping [89:17] for the best on the things that we learn [89:19] out of it. And we mentioned just now [89:22] that we do not want to train any [89:24] additional model to remedy that. So I [89:28] wrote down a proposal of what we could [89:31] do to remedy it. So on policy training [89:35] is [89:36] typically us interested in the model's [89:39] own generation. So that is set. One [89:42] proposal is to train next token [89:45] prediction regarding the loss [89:48] and for the lack of additional training [89:53] as a constraint maybe we could look [89:55] around at any strong models around there [89:58] and then see what we could learn from [90:01] it. [90:03] So I want to ask you a question. I [90:06] talked about next token prediction and a [90:10] model that is stronger than the one at [90:11] stake. So is there a concept that comes [90:13] to mind as potentially attractive here? [90:22] Yep. [90:23] Yeah. So yeah, the comment is [90:25] distillation. Yeah, exactly. Because [90:27] distillation is about next token [90:29] prediction, transfer of information from [90:32] one model to the other. Yeah, spot on. [90:34] but not the kind that maybe you know [90:37] it's a new one. Um so before we do that [90:41] I'm going to do a quick recap on the [90:44] loss that we use to compare [90:46] distributions of next token predictions [90:48] which is the one that we're going to be [90:50] interested in. So the scale divergence [90:53] and I wanted to frame it in very simple [90:56] terms. So as you know this loss function [91:00] is not symmetric. So kale divergence [91:03] between P and Q is not the same as the [91:07] one between Q and P. And if you want a [91:11] way to know instinctively what it means, [91:16] so you look at the sum and you see that [91:18] you sum based on weights of the [91:21] probability distribution of the first [91:23] argument. So you're supposing here that [91:26] the world is governed by that [91:28] probability distribution and then you [91:31] have this sum of logs of ratios [91:35] that will translate in penalization when [91:40] the probability weights of the two [91:43] differ. [91:45] So the way I propose we interpret this [91:48] so KL of P and Q is in a world governed [91:52] by P. How bad is it to approximate with [91:55] Q? [91:57] So typically when you do distillation, [91:59] you would [92:02] govern the world with the teacher model. [92:05] So T would be the first argument and the [92:06] student would be the second argument. [92:09] But here I said we want to do it on [92:12] policy with respect to the students. [92:15] So one natural way to formulate this is [92:18] not to do KL divergence of T and S but [92:23] rather the opposite and this is called [92:25] the reverse KL divergence [92:29] and it has some interesting properties [92:33] attached to it. So when you look at the [92:37] sum and [92:40] you look at the distribution of the [92:42] student and the teacher, you see that [92:46] when t is not equal to zero and if s is [92:51] close to zero, then the contributions of [92:54] these terms to the sum are not that big. [92:58] So we say they are forgiving to these [93:02] mismatches between these two [93:03] distributions. [93:06] But on the other hand, if [93:11] ti for a given term is small but si is [93:15] non zero, then you're going to have this [93:18] ratio that is going to be very big and [93:22] that is associated to a weight that is [93:24] non zero. are going to get a huge turn. [93:29] And this latter point combines with a [93:34] typical student models architecture to [93:37] not have as many degrees of freedom as [93:39] the teacher one to make it be mode [93:42] seeking meaning that the peaks in the [93:44] distribution of the teacher will be [93:46] those that will be learned by the the [93:49] students. [93:51] So these did this make sense here the [93:56] fact that we use reverse scale [93:58] divergence. [94:01] So I'm asking the question because even [94:03] though it's what people use today the [94:05] original paper did a mix of both. So I [94:08] think it's not it's not obvious that [94:10] this is obviously the better one. But [94:15] so now let's go back to our favorite [94:17] example and apply this way of measuring [94:20] the loss based on the distributions of [94:23] student and teacher. So here what we [94:25] want to do in on policy distillation is [94:28] that we want the student to generate a [94:32] completion based on a prompt and then we [94:35] ask the teacher based on the context how [94:39] much does the token generated by the [94:41] student make sense to you and we ask [94:44] this question at each new generated [94:47] token. So let's say you have this prompt [94:50] with a problem statement of Teddy in a [94:52] sealed box and you ask where is Teddy. [94:56] So let's say the sentence starts with [94:58] Teddy. So Teddy makes sense to the [95:01] teacher. I would generate also Teddy. [95:03] But then the student says Teddy climbs [95:06] and the teacher would say okay wait [95:09] climbs is not a high probability token [95:13] in my distribution. So this is what the [95:17] teacher would penalize here and what the [95:19] reverse scale divergence would penalize [95:23] and so on. Do you see anything that is [95:26] not being penalized here and that is a [95:29] surprise to you in this example? So the [95:31] things that are penalized in the in this [95:33] toy example are things in red. [95:36] Yeah. So the suggestion is wax. So in O2 [95:39] why is walks not penalized either? So, [95:43] do you have an explanation for it or [95:45] does it make sense that that it's not [95:47] penalized? [95:48] Yep. [95:51] Yeah. So, yeah, the intuition is is [95:53] great. So, uh the answer was maybe the [95:56] teacher interprets the the teddy bear as [95:58] being a human. So, as we said, [96:01] once you generate the next token, all [96:04] the tokens generated before come into [96:06] the context. So, let's say you are a [96:08] teacher model. Now you know in your [96:11] context that Teddy can do things like [96:13] climbing then walking might be in the [96:17] span of possibilities. So it's not [96:18] something that is super surprising. [96:22] So that is one property that is [96:24] interesting. [96:27] And uh yeah that's all I had to say [96:30] here. So when we look at on policy [96:33] distribution on policy distillation we [96:36] see that we have this supervision signal [96:39] that is dense because it comes at the [96:42] token level instead of the outcome level [96:46] and contrary to SFT it's as the name [96:50] suggests on policy meaning that you're [96:52] training the model to be better at with [96:56] respect to its own completions [97:00] and you see in practice great results [97:02] coming from it. And when you go even [97:04] further than that with the same amount [97:07] of compute, you see that you get more [97:09] bang out of your buck from this method. [97:13] And also it is more friendly to [97:18] parameter constrained situations. So [97:21] there is this great blog post by [97:23] thinking machine that did some toy [97:26] ablation studies and for example if you [97:30] play with the rank of the weights that [97:32] you want to tune and you do SFT first [97:36] and on policy distillation next you see [97:39] that the delta between the performance [97:42] that you get after SFT [97:45] is shortened greatly after you apply on [97:48] policy distillation which shows that the [97:51] method is great at operating under [97:54] compute constraints. [97:59] Good. Great. [98:01] And also uh since there is no reward [98:04] model, there is nothing to hack. And if [98:06] there were something to hack, we would [98:07] be more than happy to because if on [98:10] policy decision wants to bring the loss [98:13] down to zero, then you would align with [98:15] the teacher distribution, which is a [98:16] good edge case. That is great. [98:21] But then [98:23] when I mentioned we need a teacher [98:25] model, then you need to have that in the [98:27] first place. Let's say you're you're [98:30] you're training like a model, then you [98:33] need to have that and you're training a [98:35] frontier model. Maybe you're the best. [98:37] So there is no one that could teach you. [98:40] And then the second thing is that [98:44] contrary to gRPO where the signal that [98:47] you're basing your updates on is true [98:51] labels now you're comparing yourself to [98:55] a teacher models output distribution and [98:58] it might not be perfect nor might it be [99:02] complete in terms of coverage on the [99:04] domains that you're interested in which [99:07] is why it it might also come as a [99:09] limitation. [99:11] Does that make sense? [99:15] Okay, great. So now I'm going to see [99:18] each of these limitations more in [99:19] detail. So do you see a way where we [99:22] could avoid [99:24] the need for a teacher? [99:27] Okay, I know it's it's not obvious, but [99:29] once we're going to see it, it sounds [99:30] magical. So you just need to recall that [99:34] everything that you're inputting for you [99:38] to obtain the output distributions is [99:40] done by teacher forcing way. So the [99:44] figure that AIT had I think here between [99:48] generation and training you would be on [99:50] the right side where once you generate [99:53] the tokens from your student model you [99:57] put all of them the prompts plus the [99:59] tokens from the student model as input [100:01] of your teacher and you look at the [100:04] output next token prediction of your [100:07] teacher model. So nothing prevents you [100:10] from injecting things that could help a [100:14] potential teacher as part of the prompt [100:17] including the answer and everything that [100:20] leads up to the answer. So this is what [100:22] the unpoicy self distillation paper [100:24] does. So you have a prompt then you have [100:28] the complete answer and you ask the [100:32] model to come back with an answer once [100:34] again. And this is just a trick for you [100:37] to get these nice probability [100:39] distributions ready to compare with a [100:42] student. And a nice thing here is that [100:44] you can do that with the same model. [100:49] Does that look magical? [100:51] Yeah. Yeah, I know. Looks great. So, [100:55] okay. So, the second thing that we [100:57] mentioned is the Oh, yeah. There was a [100:59] question. [101:01] So the question is if I had to summarize [101:04] is when would you benefit most from [101:07] unpolicy dissolation? So yeah, you're [101:08] right that let's say you're the best [101:11] model out there. Maybe you will have a [101:14] lesser benefit from this uh from this [101:17] method and maybe you would use on policy [101:20] for other needs. the like GLM5 paper [101:25] that we saw last week is one example [101:27] where you would use it to calibrate the [101:30] fact that you do not lose knowledge that [101:32] you have learned in the past. So you [101:34] would distill yourself on previously [101:36] gained knowledge but uh you're right you [101:38] would typically use it on faster models [101:42] of your [101:45] of your series. So let's say you have [101:47] like your frontier lab, you have a pro [101:49] and a flash model. Maybe unpolicy [101:51] distillation would be what you use on [101:53] your flash series and then um and then [101:55] your pro series might be more [101:58] foundational gpo and then like these [102:02] core techniques that might need more [102:03] compute but that are based off like [102:06] hardcore labels. [102:09] >> Y awesome. So the last thing that I [102:13] wanted to mention is on the fact that [102:15] you might be bounded by the teacher's [102:17] abilities. So one approach is to [102:24] take a model and then use these [102:26] verifiable rewards across domains and [102:30] then have experts for each domain and [102:33] then when you want to do this on policy [102:35] distillation do it against the experts [102:38] of that domain. [102:41] So this is why there is a multi-teer and [102:45] policy distillation that comes in the [102:46] picture. So now is it a surprise to [102:48] anyone that we need multiple teachers [102:50] here and not just one? [102:53] Well empirically if you do this [102:56] post-raining RL on just a single one you [103:00] might have things like forgetting that [103:02] happens and you might use a given set of [103:05] parameters towards multiple objectives. [103:08] So this is a method that is that that [103:11] has shown to be more efficient. [103:16] Okay. Awesome. And with that, I hope you [103:18] have a great weekend.