Latent Self-Distillation Beats GRPO With 70% Fewer Rollouts

Xiaol.x

Sampling prefixes directly from student rollouts cuts autoregressive error compounding from quadratic to linear, allowing learned latent teacher contexts to outperform reinforcement learning without adding inference overhead.

AI-assisted summary of On-Policy vs Off-Policy Explained — The Math Behind Latent Self-Distillation — original source. Check the original for full context.

Source published: 2026-08-18

Key ideas

Prefix occupancy and divergence direction are orthogonal design choices: 'on-policy' refers exclusively to sampling prefixes from the student's distribution rather than dictating the use of reverse KL over forward KL. (0:00)
Evaluating the teacher on student rollouts reduces autoregressive compounding regret from O(T^2) down to O(T), directly resolving exposure bias without requiring the teacher to generate sequences autoregressively. (1:37)
Hand-crafted privileged contexts such as oracle execution traces or sibling rollouts exhibit severe cross-domain brittleness, in some cases degrading coding benchmark performance below the un-tuned base model. (5:53)
Latent On-Policy Self-Distillation (LOPD) compresses past trajectories into continuous latent tokens and stabilizes joint training via a dual margin constraint, matching or exceeding GRPO performance across 10 architectures using under 30% of the rollout budget. (7:27)

Detailed notes

## Background Published on YouTube by Xiaol.x on August 18, 2026, this visual LeanIm lesson explores the mathematical foundations of Latent On-Policy Self-Distillation (LOPD). According to the description, the video explains the distinction between on-policy and off-policy training, untangles rollout prefix occupancy from KL divergence direction, and connects LOPD to methods like SFT, DAgger, GRPO, OPSD, and MiniLLM.

## TL;DR The video clarifies that 'on-policy' describes rollout prefix occupancy ($d^\pi$) rather than divergence direction, demonstrating how off-policy training suffers from $O(T^2)$ compounding error cascades while on-policy evaluation limits regret to $O(T)$. It contrasts sparse reinforcement learning signals like GRPO with dense step-level conditional distillation, showing that reverse KL operates as a rich, per-token learning signal on student-generated prefixes. Finally, the presentation breaks down Latent On-Policy Self-Distillation (LOPD), which replaces brittle hand-crafted privileged context prompts with learned latent experience tokens compressed by a Q-Former, enforcing a dual-margin constraint to prevent collapse before discarding all privileged machinery at deployment.

## Atmosphere & style The presentation carries the crisp, rigorous tone of a graduate-level seminar, driven by clean LeanIm visual diagrams and disciplined mathematical derivations. The speaker communicates with calm analytical clarity, systematically dismantling common machine learning misconceptions with pedagogical precision and vivid metaphors, such as comparing off-policy distillation to a chess novice watching a grandmaster. It feels like an illuminating chalk talk tailored for research engineers, methodically untangling architectural confusions before building up to a sophisticated latent distillation framework.

## Bulletin - [00:00](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=0s) Prefix occupancy and divergence direction are orthogonal design choices: 'on-policy' refers exclusively to sampling prefixes from the student's distribution rather than dictating the use of reverse KL over forward KL. - [01:37](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=97s) Evaluating the teacher on student rollouts reduces autoregressive compounding regret from O(T^2) down to O(T), directly resolving exposure bias without requiring the teacher to generate sequences autoregressively. - [05:53](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=353s) Hand-crafted privileged contexts such as oracle execution traces or sibling rollouts exhibit severe cross-domain brittleness, in some cases degrading coding benchmark performance below the un-tuned base model. - [07:27](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=447s) Latent On-Policy Self-Distillation (LOPD) compresses past trajectories into continuous latent tokens and stabilizes joint training via a dual margin constraint, matching or exceeding GRPO performance across 10 architectures using under 30% of the rollout budget.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=0s) Policy distillation can be mapped along two independent axes: prefix rollout occupancy on the horizontal axis and local divergence direction on the vertical axis. - [00:00](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=0s) A prefix acts as a Markovian state; early student deviations alter the prefix distribution, shifting the optimization problem even if the per-token divergence loss formula remains unchanged. - [01:37](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=97s) Off-policy distillation evaluates students on teacher-preferred prefixes, incurring exposure bias and an O(T^2) compounding regret over horizon T under the DAgger framework. - [01:37](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=97s) Sampling prefixes on-policy from the student bounds per-step decision errors to linear O(T) regret while requiring only a single evaluation forward pass from the teacher. - [03:45](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=225s) Forward KL is mode-covering and encourages distribution smearing, whereas reverse KL is mode-seeking and heavily penalizes student mass where teacher probability is low. - [03:45](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=225s) Both OPSD and LOPD are strictly on-policy methods because they sample from student occupancy, despite OPSD optimizing forward KL and LOPD optimizing reverse KL. - [04:39](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=279s) Sparse trajectory-level reinforcement learning methods like GRPO collapse to zero advantage when candidate completions within a group have identical verifier outcomes. - [04:39](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=279s) Dense conditional teacher supervision provides step-level learning signals along student trajectories, sidestepping the credit assignment bottlenecks of sparse terminal rewards. - [04:39](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=279s) LOPD treats visited student prefixes as fixed context data for analytic KL computation rather than backpropagating end-to-end through autoregressive rollouts. - [05:53](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=353s) Self-distillation creates an asymmetric capability gap on a single model backbone by anchoring frozen teacher weights with privileged contextual information. - [05:53](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=353s) Rigid natural language privileged templates transfer poorly across domains, failing when sibling solutions all err or degrading execution tasks when using oracle traces. - [07:27](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=447s) LOPD replaces rigid text templates by encoding raw trajectories with a LoRA module and compressing them into continuous latent prefix tokens using a Q-Former. - [07:27](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=447s) Jointly optimizing student reverse KL alongside latent context generation risks degenerate collapse, where the teacher trivializes the loss by matching the student's output. - [07:27](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=447s) A dynamic dual variable enforces a minimum privilege margin threshold on successful trajectories, preventing collapse and sustaining an informative teacher evaluation gap. - [07:27](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=447s) All auxiliary training infrastructure—including the trajectory bank, LoRA adapter, Q-Former, and latent tokens—is discarded at test time, resulting in zero inference overhead.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=0s) Deconstructing "On-Policy": Occupancy vs. Divergence Direction

The term "on-policy" routinely confuses practitioners because it conflates two completely orthogonal design choices onto a single label. When mapping out policy distillation on a two-dimensional coordinate system, the horizontal axis dictates whose prefixes the model is trained on, while the vertical axis dictates which divergence is evaluated at each prefix. These are distinct mathematical knobs that should never be treated as the same design decision. "Off-policy versus on-policy is a statement about occupancy," whereas "forward KL versus reverse KL is a statement about the local loss." Latent On-Policy Self-Distillation (LOPD) occupies a specific, principled corner of this two-dimensional plane.

To make this mathematically concrete, one must first identify the fundamental object: a prefix is a Markovian state containing the user prompt, environmental observations gathered so far, and all autoregressively generated tokens up to the current step. The policy's next-token distribution is strictly conditioned on this state. Occupancy $d^\pi$ represents the probability distribution over all possible prefixes that a given policy $\pi$ visits during generation. If a student model hallucinates or outputs an incorrect tool name as early as turn two, every subsequent token must be generated from an altered state that the teacher model might never have encountered. Because training always computes an expectation of a local divergence under some occupancy measure $\mu$, changing $\mu$ fundamentally transforms the underlying optimization problem—even when the local loss formula on the chalkboard looks completely identical.

```mermaid flowchart TD accTitle: Two Axes of Policy Distillation accDescr: Diagram illustrating the two independent axes: rollout prefix occupancy on the horizontal axis and divergence direction on the vertical axis.

subgraph DivergenceAxis["Local Divergence Choice (Axis 2)"] FKL["Forward KL: Mode-Covering (Teacher || Student)"] RKL["Reverse KL: Mode-Seeking (Student || Teacher)"] end

subgraph OccupancyAxis["Prefix Occupancy Choice (Axis 1)"] OffPol["Off-Policy Occupancy (Teacher d^{π_T})"] OnPol["On-Policy Occupancy (Student d^{π_S})"] end

[01:37](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=97s) The $O(T^2)$ Regret Catch: Off-Policy Mismatch vs. On-Policy Sampling

Traditional off-policy distillation—including standard Supervised Fine-Tuning (SFT)—draws completed trajectories from an expert teacher or a static dataset and trains the student to reproduce those tokens. In this setting, the distribution over training prefixes is governed by the teacher's occupancy measure $d^{\pi_T}$, evaluating the student "in rooms the teacher likes to visit." At test time, however, the student must sample tokens autoregressively from its own distribution, traversing its own prefix occupancy $d^{\pi_S}$. The fundamental discrepancy between these two probability measures produces exposure bias, also known as the train-inference distribution shift.

``` Off-Policy Loss: E_{s ~ d^{π_T}} [ D( π_T(·|s) || π_S(·|s)) ] On-Policy Loss: E_{s ~ d^{π_S}} [ D( π_T(·|s) || π_S(·|s)) ] ```

This train-inference mismatch triggers a severe error cascade established in the imitation learning literature by the DAgger framework. If an imitation learner is supervised exclusively along expert trajectories, a bounded per-step decision error of $\epsilon$ compounds over an autoregressive horizon of length $T$ into $O(T^2)$ total regret. A single suboptimal token early in the sequence permanently derails the conditioning history, forcing the model to recover from states on which it never received supervision.

Conversely, when the teacher is queried directly on the states visited by the student ($s \sim d^{\pi_S}$), that same per-step error $\epsilon$ scales linearly as $O(T)$ regret. As the speaker illustrates through a chess metaphor: watching a grandmaster play exposes a novice to brilliant tactical sequences, but solely on board positions that the novice will never manage to reach during actual competitive play. Switching the prefix occupancy from $d^{\pi_T}$ to $d^{\pi_S}$ requires only that the student roll out candidates first, after which the teacher evaluates those exact prefixes in a single forward pass without generating tokens itself. This simple change of measure retains what Thinking Machines characterized as "the on-policy relevance of RL with the dense signal of distillation."

[03:45](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=225s) Forward KL vs. Reverse KL: Separating Local Divergence from Occupancy

The common claim that on-policy distillation inherently implies reverse KL is a misconception. Occupancy selection and divergence direction remain independent axes: - **Forward KL ($\mathrm{KL}(\pi_T \parallel \pi_S)$):** Mode-covering. The student is penalized whenever it fails to allocate probability mass wherever the teacher places mass, encouraging the student to smear its distribution over multi-modal outputs. - **Reverse KL ($\mathrm{KL}(\pi_S \parallel \pi_T)$):** Mode-seeking. The student is heavily penalized for allocating mass in regions where the teacher assigns low probability, driving it to concentrate tightly on actions supported by the teacher.

LOPD employs reverse KL computed over a top-$M$ plus tail compression of the vocabulary distribution. In contrast, On-Policy Self-Distillation (OPSD) evaluates multiple divergences and opts for forward KL across identical student-generated prefixes. Both methods are strictly on-policy because they sample prefixes from $d^{\pi_S}$, yet they optimize entirely different objectives.

[04:39](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=279s) From Sparse GRPO Rewards to Dense Conditional Teacher Supervision

Group Relative Policy Optimization (GRPO) already operates within the student rollout paradigm by generating groups of completions from a recent checkpoint, calculating token-level importance sampling ratios, and scaling them with a scalar trajectory advantage derived from a verifier. However, GRPO relies on sparse terminal feedback; if every candidate completion within a sampled group is uniformly correct or uniformly incorrect, the relative advantage collapses to zero, yielding no learning signal.

```mermaid flowchart LR accTitle: Distillation and RL Supervision Pipelines accDescr: Contrasting sparse trajectory-level GRPO reinforcement learning with dense conditional step-level teacher supervision.

subgraph GRPO["GRPO (Sparse Trajectory RL)"] G_Roll["Student Rollout"] --> G_Eval["Verifier Scoring"] G_Eval --> G_Adv["Trajectory Advantage (Zero if all same)"] G_Adv --> G_Grad["Policy Gradient Updates"] end

subgraph Distill["Conditional Distillation (LOPD)"] D_Roll["Student Prefix Data (s ~ d^{π_S})"] --> D_Dense["Dense Teacher Forward Pass"] D_Dense --> D_KL["Full Conditional KL: KL(π_S(·|s) || π_T(·|s))"] D_KL --> D_Backprop["Backprop per Visited Token"] end ```

At any individual visited prefix, reverse KL admits an exact interpretation as a dense token-level reward whose policy gradient matches a score function weighted by:

$$\log \pi_T(a \mid s) - \log \pi_S(a \mid s)$$

However, the gradient estimator must be treated carefully: - LOPD treats the student's sampled prefix strictly as fixed context data and backpropagates the exact analytic conditional KL divergence across the action distribution at that token. - Unlike MiniLLM, LOPD does not differentiate end-to-end through the entire autoregressive rollout process, which would introduce complex reward-to-go terms. - The practical engineering pattern combines dense, conditional teacher supervision at every step of the student's rollout with sparse outcome verifier rewards to maintain ground-truth task completion.

[05:53](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=353s) Self-Distillation and the Pitfalls of Hand-Crafted Privileged Context

When a strictly larger or more capable external teacher is unavailable, on-policy self-distillation creates an asymmetric performance gap using a shared model backbone by supplying the teacher with privileged context ($c$). While the student generates completions using only the prompt and history, the teacher branch observes privileged information that makes evaluation easier than generation—such as ground-truth solutions, execution traces, successful sibling solutions, or extracted skills. In this setup, "self means shared origin not two branches chasing each other during the update," meaning the teacher weights are held frozen ($\text{stop-gradient}$) to serve as an anchor.

```mermaid flowchart TD accTitle: Self-Distillation Privileged Architecture accDescr: Flow showing how the student generates trajectories on which an asymmetrically privileged teacher provides dense supervision.

Prompt["Task Prompt & Context"] --> Student["Student Policy π_S (Learned)"] Student --> Rollout["Sampled Trajectory / Visited Prefixes (s ~ d^{π_S})"]

Rollout --> S_Logits["Student Logits π_S(·|s)"] Rollout --> Teacher["Teacher π_T (Frozen Copy)"] Privilege["Privileged Context c (Oracle / Siblings / Latents)"] --> Teacher Teacher --> T_Logits["Teacher Logits π_T(·|s, c)"]

S_Logits --> Loss["Dense On-Policy Divergence"] T_Logits --> Loss Loss --> Update["Gradient Update to π_S"] ```

Prior methods rely on rigid, hand-crafted prompt templates for $c$, which transfer poorly across domains: - **OPSD:** Injects oracle execution traces. - **SDPO:** Conditions the teacher on a successful sibling rollout from the current group. - **Skill-SD:** Retrieves natural language skill summaries.

When the authors of LOPD benchmarked these methods using identical KL formulations while altering only the privileged artifact, the hand-designed artifacts displayed severe brittleness: an oracle trace improved long-context reasoning benchmarks while causing performance on LiveCodeBench to plunge below the un-tuned base model. Sibling conditioning failed whenever an entire group produced incorrect trajectories. The fundamental point of failure was the brittle representation substrate of the context rather than the on-policy occupancy itself.

[07:27](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=447s) LOPD: Learned Latent Experience, the Collapse Problem, and Privileged Margins

LOPD eliminates brittle text templates by learning continuous latent representations of experience. The pipeline retrieves successful raw trajectories, encodes them through a Low-Rank Adaptation (LoRA) module over a frozen backbone, and compresses them with a Q-Former into $K$ continuous latent tokens. These learned tokens are prepended directly into the teacher's embedding space as prefix context. Crucially, the student still samples rollouts conditioned exclusively on the task, and distillation is performed using reverse KL on student-visited prefixes. The latent composer module is trained end-to-end against this exact same distillation objective:

``` Raw Trajectories ──> Frozen Backbone + LoRA ──> Q-Former ──> K Latent Tokens ──> Teacher Context ```

However, optimizing reverse KL jointly across both the student and the composer introduces a degenerate failure mode: - The composer can trivially minimize the objective by shifting the latent context until the teacher's next-token distribution mirrors the student's output ($\pi_T(a \mid s, c) \approx \pi_S(a \mid s)$). - At this trivial critical point, the loss falls to zero without the student acquiring any meaningful capabilities. - Unconstrained joint training performs worse than keeping a frozen, unoptimized composer.

To prevent distribution collapse, LOPD formulates a constrained optimization problem using a per-token privilege metric scaled by trajectory success:

$$\Delta(s, a) = R \cdot \left[ \log \pi_T(a \mid s, c) - \text{sg}[\log \pi_S(a \mid s)] \right]$$

A dual variable dynamically activates whenever this advantage drops below a target threshold margin $M$. If the teacher begins matching the student, the dual penalty forces the latent representations to sustain a strictly informative, privileged evaluation gap on successful trajectories.

| Distillation Paradigm | Prefix Occupancy | Divergence Type | Supervision Signal | Deployment Requirements | |:--- |:--- |:--- |:--- |:--- | | **Standard SFT** | Off-policy ($d^{\pi_T}$) | Forward KL / CE | Offline full sequences | Standalone Student | | **GRPO** | On-policy ($d^{\pi_S}$) | Trajectory RL | Sparse verifier reward | Standalone Student | | **OPSD** | On-policy ($d^{\pi_S}$) | Forward KL | Teacher + Oracle trace | Standalone Student | | **Skill-SD** | On-policy ($d^{\pi_S}$) | Reverse KL | Teacher + Retrieved skills | Standalone Student | | **LOPD** | On-policy ($d^{\pi_S}$) | Reverse KL (Top-$M$) | Teacher + Learned Latents ($M$-margin) | Standalone Student (Latents discarded) |

At test time, the entire support infrastructure—the trajectory bank, the retriever, the LoRA weights, the Q-Former, and the latent tokens—is discarded. The deployed student operates entirely on its own weights with no extra inference-time latency or memory overhead.

In empirical evaluations across 10 backbone architectures, LOPD outperformed its baselines, matching or exceeding the capabilities of GRPO and Skill-SD while consuming less than 30% of their rollout budgets. By treating experience as an end-to-end learnable supervision channel rather than a rigid prompt template, LOPD achieves true internalization of expert behavior.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=0s) Off-policy versus on-policy is a statement about occupancy

> [00:00](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=0s) forward KL versus reverse KL is a statement about the local loss.

> [01:37](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=97s) in rooms the teacher likes to visit.

> [01:37](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=97s) the on-policy relevance of RL with the dense signal of distillation. — Thinking Machines

> [03:45](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=225s) The common claim that on-policy distillation inherently implies reverse KL is a misconception.

> [05:53](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=353s) self means shared origin not two branches chasing each other during the update

> [05:53](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=353s) The fundamental point of failure was the brittle representation substrate of the context rather than the on-policy occupancy itself.

> [07:27](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=447s) At test time, the entire support infrastructure—the trajectory bank, the retriever, the LoRA weights, the Q-Former, and the latent tokens—is discarded.

> [07:27](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ&t=447s) By treating experience as an end-to-end learnable supervision channel rather than a rigid prompt template, LOPD achieves true internalization of expert behavior.

> on-policy describes where training prefixes come from—not the direction of KL. — Xiaol.x

## Mentioned Xiaol.x, LeanIm, Latent On-Policy Self-Distillation (LOPD), Supervised Fine-Tuning (SFT), DAgger, Generalized Knowledge Distillation (GKD), MiniLLM, Group Relative Policy Optimization (GRPO), On-Policy Self-Distillation (OPSD), Self-Distillation Policy Optimization (SDPO), Skill-SD, Thinking Machines, LiveCodeBench, LoRA (Low-Rank Adaptation), Q-Former

## Video description What does “on-policy” actually mean—and why is it often confused with reverse KL?

This visual LeanIm lesson explains Latent On-Policy Self-Distillation (LOPD) from first principles. We derive the difference between on-policy and off-policy training, separate rollout occupancy from KL direction, and connect LOPD to SFT, DAgger, GKD, MiniLLM, GRPO, OPSD, SDPO, and Skill-SD.

Topics covered:

- Prefix occupancy (d^\pi) and exposure bias - Why off-policy errors can compound as (O(T^2)) - Forward KL versus reverse KL - Conditional KL versus full trajectory policy gradients - Sparse GRPO rewards versus dense teacher supervision - Privileged-context self-distillation - Learned latent experience tokens - LOPD’s collapse problem and privileged-margin objective - How to upgrade an existing on-policy training pipeline

Paper: https://arxiv.org/abs/2608.13040v1

The central takeaway: on-policy describes where training prefixes come from—not the direction of KL. LOPD keeps student-generated rollouts, adds dense conditional supervision, learns the teacher’s privileged context from experience, and deploys only the student.

[Source](https://www.youtube.com/watch?v=BNPJ3vZ-JUQ)

中文简报

LOPD在10项基准测试中超越GRPO，采样预算不足其30% 由于同策略训练由学生模型的前缀占用分布而非KL散度方向决定，且人工构建的特权提示跨任务易失效，LOPD将已验证的轨迹压缩为受边际约束的潜在教师token，使独立学生模型能以标准强化学习采样成本的一小部分内化这些知识。报告阐明，“同策略”（on-policy）严格决定了前缀占用分布，而非散度度量的选择，并展示了学生模型生成的采样（rollout）如何将异策略复合遗憾从二次误差降低至线性误差。尽管GRPO等方法受制于稀疏的验证器反馈，且以往的特权自蒸馏技术依赖于脆弱的文本提示，但LOPD引入了通过LoRA适配器和Q-Former压缩的连续潜在经验token。为了防止联合优化过程中教师模型简单模仿学生模型而出现平凡坍缩，LOPD在成功的轨迹上施加了对偶约束的特权边际目标。最终，学生模型内化这些密集信号并实现独立部署，以不足标准采样预算30%的成本取得了更优的基准测试表现。同策略训练严格定义了状态前缀占用（即从学生模型分布 d^{π_S} 中采样前缀）而非散度方向，从而将采样访问状态与正向或反向KL的选择分离开来。 (0:00) 基于学生模型生成的采样而非专家演示来评估监督，可将自回归复合遗憾限制在 O(T) 以内，克服了离线SFT固有的二次方 O(T^2) 曝光偏差陷阱。 (1:37) 潜在同策略自蒸馏（LOPD）用通过LoRA适配器和Q-Former压缩器学习到的连续潜在经验token取代了脆弱的人工文本提示，并在推理阶段将其完全丢弃。 (7:27) 为防止联合优化因教师模型直接模仿学生模型而发生平凡坍缩，LOPD在已验证的成功采样上实施了拉格朗日对偶约束的特权边际目标。 (7:27) 在十项基准测试对比中，LOPD在采样计算预算不足30%的情况下，表现达到或超越了GRPO和Skill-SD。 (7:27)

Reading summary

In this presentation, Xiaol.x explains that policy distillation is governed by two independent mathematical choices—prefix occupancy and local divergence direction—and shows how Latent On-Policy Self-Distillation (LOPD) pairs student-visited states with learned continuous teacher context to overcome the limitations of offline fine-tuning and sparse reinforcement learning. Separating Prefix Occupancy from Divergence Direction (0:00) Practitioners frequently conflate two orthogonal design choices under the single label "on-policy." As Xiaol.x outlines, policy distillation operates on a two-dimensional plane: the horizontal axis governs prefix occupancy (whose trajectories generate the training states), while the vertical axis dictates the local divergence loss evaluated at each state. A prefix acts as a Markovian state containing the prompt, environmental observations, and tokens generated so far. Because training minimizes an expected divergence over an occupancy distribution $\mu$, changing which policy visits the prefixes alters the underlying optimization problem even when the local loss formula looks identical. Consequently, on-policy sampling does not inherently require reverse KL divergence. Forward KL ($\mathrm{KL}(\pi_T \parallel \pi_S)$) is mode-covering, penalizing the student for failing to allocate probability mass wherever the teacher places mass, whereas reverse KL ($\mathrm{KL}(\pi_S \parallel \pi_T)$) is mode-seeking, heavily penalizing the student for placing mass where the teacher assigns low probability. On-Policy Self-Distillation (OPSD) pairs student-generated prefixes with forward KL, while LOPD pairs those same on-policy prefixes with a top-$M$ plus tail compression of reverse KL. Mitigating Compounding Regret and Sparse Rewards (1:37) Traditional Supervised Fine-Tuning (SFT) and off-policy distillation train the student on the teacher's occupancy measure ($d^{\pi_T}$). At test time, however, the student must sample autoregressively from its own distribution ($d^{\pi_S}$). If an early token hallucination or tool error occurs, the model enters altered states the teacher never visited. According to the DAgger imitation learning framework, a bounded per-step error $\epsilon$ under off-policy supervision compounds over an autoregressive horizon $T$ into $O(T^2)$ total regret. Switching to on-policy occupancy ($s \sim d^{\pi_S}$)—where the student rolls out candidates first and the teacher evaluates those exact prefixes in a single forward pass—reduces regret to scale linearly as $O(T)$. This conditional teacher evaluation also addresses a key bottleneck in Group Relative Policy Optimization (GRPO). Although GRPO uses student rollouts, it relies on sparse terminal verifier rewards; if all completions in a sampled group are uniformly correct or incorrect, the relative advantage collapses to zero. LOPD instead supplies dense, token-level supervision by treating the student's visited prefix as fixed context data and backpropagating the exact analytic conditional KL divergence at each step, avoiding MiniLLM's complex end-to-end differentiation through the autoregressive rollout while combining teacher supervision with sparse outcome verifier rewards. The Brittleness of Hand-Crafted Privileged Context (5:53) When a larger or more capable external teacher is unavailable, self-distillation creates an asymmetric performance gap using a shared model backbone. The teacher branch is held frozen via a stop-gradient and supplied with privileged context ($c$)—such as ground-truth solutions or execution traces—that makes evaluating states easier than generating completions from scratch. However, prior self-distillation methods rely on rigid, hand-crafted text templates for this context: OPSD injects oracle execution traces, SDPO conditions on a successful sibling rollout from the current group, and Skill-SD retrieves natural language skill summaries. Benchmarking by the LOPD authors using identical KL formulations revealed that these manual artifacts transfer poorly across domains. Conditioning on an oracle trace improved long-context reasoning benchmarks but caused performance on LiveCodeBench to plunge below the un-tuned base model, while sibling conditioning failed whenever an entire group produced incorrect trajectories. Learned Latent Context and Preventing Collapse (7:27) To replace brittle text templates, LOPD learns continuous latent representations of experience. The pipeline retrieves successful raw trajectories, encodes them through a Low-Rank Adaptation (LoRA) module over a frozen backbone, and compresses them with a Q-Former into $K$ continuous latent tokens prepended directly into the teacher's embedding space. Training this latent composer jointly with the student against the reverse KL objective introduces a degenerate failure mode: the composer can trivially minimize the loss to zero by shifting the latent context until the teacher's distribution mirrors the student's output ($\pi_T(a \mid s, c) \approx \pi_S(a \mid s)$), which performs worse than keeping a frozen, unoptimized composer. LOPD prevents distribution collapse through constrained optimization, defining a reward-scaled per-token privilege metric $\Delta(s, a) = R \cdot [\log \pi_T(a \mid s, c) - \text{sg}[\log \pi_S(a \mid s)]]$. Whenever this advantage drops below a target threshold margin $M$, a dynamic dual variable activates to force the latent representations to sustain an informative evaluation gap on successful trajectories. At test time, LOPD discards the entire support infrastructure—including the trajectory bank, retriever, LoRA weights, Q-Former, and latent tokens—allowing the standalone student model to operate on its own weights with no additional inference latency or memory overhead. Across empirical evaluations on 10 backbone architectures, the authors report that LOPD matched or exceeded GRPO and Skill-SD while consuming less than 30% of their rollout budgets.

Source quotes

[0:00] Off-policy versus on-policy is a statement about occupancy [0:00] forward KL versus reverse KL is a statement about the local loss. [1:37] in rooms the teacher likes to visit. [1:37] Thinking Machines: the on-policy relevance of RL with the dense signal of distillation. [3:45] The common claim that on-policy distillation inherently implies reverse KL is a misconception. [5:53] self means shared origin not two branches chasing each other during the update [5:53] The fundamental point of failure was the brittle representation substrate of the context rather than the on-policy occupancy itself. [7:27] At test time, the entire support infrastructure—the trajectory bank, the retriever, the LoRA weights, the Q-Former, and the latent tokens—is discarded. [7:27] By treating experience as an end-to-end learnable supervision channel rather than a rigid prompt template, LOPD achieves true internalization of expert behavior. Xiaol.x: on-policy describes where training prefixes come from—not the direction of KL.

Original transcript

[0:00] Today we take apart a word that keeps [0:02] confusing people on policy. [0:05] Look at this blank board. I am going to [0:08] draw two axes because that word is [0:10] hiding two independent choices. [0:13] The horizontal axis is whose prefixes we [0:18] train on. [0:19] The vertical axis is which divergence we [0:22] apply at a prefix. So, [0:25] you might expect those to be the same [0:28] knob. They are not. [0:30] Off policy versus on policy is a [0:34] statement about occupancy. Um forward KL [0:38] versus reverse KL is a statement about [0:42] the local loss. Latent on policy [0:45] self-distillation LOPD lives in one [0:48] corner of this plane and we will walk [0:50] there honestly. [0:53] Before any loss, name the object. [0:56] A prefix is a state. [0:58] It is the task, the observations so far, [1:01] and the tokens already written. The next [1:04] token distribution is always conditioned [1:06] on that state. Occupancy D to the pi is [1:10] simply the distribution over prefixes a [1:13] policy actually visits. If the student [1:15] writes a wrong tool name on turn two, [1:19] every later token lives in a state the [1:21] teacher may never have entered. Training [1:23] is always an expectation of a local loss [1:26] under some occupancy mu. [1:29] Um change mu and you have changed the [1:32] problem even if the formula on the board [1:35] looks identical. [1:37] Off policy distillation including [1:40] ordinary SFT draws completions from a [1:42] teacher or a data set, then asks the [1:45] student to match them. The occupancy is [1:47] the teacher's. In symbols, the loss is [1:51] the expected divergence, usually forward [1:54] KL or cross-entropy, [1:57] of teacher given student with S drawn [1:59] from D to the pi T. The The student is [2:04] being graded in rooms the teacher likes [2:08] to visit. At inference, the student must [2:10] generate from itself, so it walks D to [2:12] the pi student. Um the gap between those [2:16] two measures is the train inference [2:18] mismatch. [2:20] Huh. [sighs] [2:21] You might expect a small mismatch to [2:24] stay small. Here is the catch. Dagger [2:27] proved that if you only imitate the [2:29] expert on expert states, a per step [2:31] error epsilon becomes order T squared [2:34] regret on a horizon of length T. One [2:37] early wrong token bends every later [2:39] condition. The model is now solving from [2:42] a state it barely practiced. If instead [2:44] you query the expert on the states the [2:46] learner actually visits, the same [2:49] epsilon becomes order T. That is the [2:51] whole theoretical case for switching the [2:53] occupancy. The chess picture is the same [2:56] theorem. Watching a grandmaster shows [2:58] brilliant moves on boards a novice never [3:00] reaches. [3:03] Now, flip only access one, the student [3:06] rolls out first, then a teacher is [3:08] reevaluated on the same prefixes. [3:11] The teacher does not have to generate [3:13] one forward pass produces a full next [3:16] token distribution at every visited [3:18] prefix. The loss is the same local [3:20] divergence now averaged under D to the [3:23] pi student. Look at the board, the [3:26] integrand did not change, only the [3:28] measure did. That single substitution is [3:32] the upgrade from off-policy to [3:35] on-policy. Thinking Machines called the [3:38] result the on-policy relevance of RL [3:42] with the dense signal of distillation. [3:45] People say [3:47] on-policy distillation uses reverse KL. [3:51] Sometimes it does. That is still access [3:54] too. Forward KL teacher first is mode [3:59] covering. The student must put mass [4:01] wherever the teacher does and smears if [4:04] it cannot. [4:05] Reverse KL [4:07] student first is mode seeking. [4:10] The student is punished for mass the [4:13] teacher rejects so it concentrates on [4:15] teacher supported behavior. LOPD [4:19] uses reverse KL on a top M plus tail [4:23] compression of the two distributions. [4:25] OPSD evaluates several divergences and [4:28] adopts forward KL in its experiments on [4:32] the same on policy prefixes. Both are on [4:35] policy. They are not the same algorithm. [4:39] Group relative policy optimization [4:44] already lives in the student rollout [4:46] family. It samples from a recent student [4:49] then uses a per token importance ratio [4:53] while turning one verifier score into [4:55] one advantage for the whole trajectory. [4:58] If if the group is all correct or all [5:01] wrong, that advantage is zero. At one [5:06] fixed visited prefix [5:09] reverse KL [5:12] has an exact dense reward [5:14] interpretation. [5:16] Its gradient is a score function [5:19] gradient weighted by log teacher minus [5:22] log student. But keep the estimator [5:25] separate. LOPD treats the sampled prefix [5:29] as data and back propagates the full [5:31] conditional KL. It does not [5:33] differentiate through the whole rollout. [5:35] A full sequence reverse KL would add [5:39] reward to go terms as mini LLM does. So [5:43] the practical upgrade is is add dense [5:45] teacher supervision to student rollouts [5:48] while keeping outcome reward when it is [5:50] still needed to ground success. [5:53] What if you have no stronger teacher? On [5:56] policy self-distillation starts from the [5:59] same backbone and creates asymmetry with [6:01] context. The student sees only the task [6:05] and the history. The teacher [6:07] additionally sees a privilege uh C an [6:10] answer a trace a successful sibling a [6:12] skill. Privilege must make evaluation [6:15] easier than generation. In the abstract [6:19] OPS D objective the teacher branch is [6:21] stop gradient. Implementations may [6:24] freeze a reference copy and LOPD [6:27] explicitly does. So self means shared [6:31] origin not two branches chasing each [6:33] other during the update. Um [6:36] the loss is still an on policy [6:37] divergence. The new object on the board [6:40] is C. [6:42] Uh [6:43] existing recipes keep inventing a new C. [6:47] OPS D injects an oracle trace [6:51] SD. [6:53] PO conditions on a successful sibling [6:55] from the current group. [6:57] Skill SD injects a retrieved skill [7:00] summary. The LOPD paper reruns them with [7:03] their original KL directions changing [7:05] only the privilege. The same artifact [7:08] helps one benchmark and hurts another. [7:11] And oracle trace can raise a long [7:14] context score and still drop live code [7:16] bench below the vanilla model. [7:19] Sibling context is only as good as this [7:21] group's successes. The failure is the [7:23] substrate not the occupancy. [7:27] LOPD stops prescribing the artifact. [7:30] Retrieve a few successful raw [7:32] trajectories. Encode each with a low RA [7:36] on a frozen backbone. [7:38] Compress with a Q former into K latent [7:41] tokens. Concatenate them and inject them [7:45] as ordinary embeddings in the teacher's [7:47] context window. The student still [7:49] generates from the task alone. [7:52] Distillation is reverse KL on the [7:55] student's prefixes. So, the composer is [7:58] trained by the same signal that trains [8:01] the student. After training, throw the [8:03] bank, the retriever, and the latents [8:06] away. [8:07] Only the student is deployed. [8:10] Here is the catch that makes the new [8:12] math necessary. [8:14] Reverse KL over both the student and the [8:17] composer has a trivial critical point. [8:20] The composer can move the latent context [8:23] until the teacher distribution equals [8:26] the student. The loss goes to zero and [8:30] nobody learned. [8:32] Empirically unconstrained joint training [8:35] loses to a frozen composer. So, they [8:37] define a per token privilege, log [8:40] teacher minus stop grad log student, [8:43] then weight it by the verified outcome [8:45] of the whole trajectory. A dual variable [8:48] enforces that this advantage stay above [8:50] a margin M. [8:52] If the teacher collapses, the dual turns [8:54] on. The feasible region is keep the [8:57] teacher actually privileged, especially [8:59] on successes. [9:02] Take the board with you as a ladder. [9:04] Keep collecting recent student rollouts. [9:08] Add dense conditional reverse KL. If you [9:11] have no stronger model, create teacher [9:13] asymmetry with privileged context in a [9:16] fixed target branch. Then, stop hand [9:20] designing that context. Learn latent [9:22] tokens from raw experience and constrain [9:25] them with a margin. LOPD [9:29] wins the paper's 10 backbone benchmark [9:32] comparisons, and it passes GRPO and [9:35] Skill SD with less than 30% of their [9:38] rollout budget. The student at inference [9:41] has no retrieval and no latent [9:43] privilege. That is internalization. [9:46] Experience was a learnable supervision [9:49] channel, not a prompt template.