Scalable Agent Alignment with GRPO: Methods, Reward Design and Training Dynamics With Vanguard Group

Ai4

AI-assisted summary of Scalable Agent Alignment with GRPO: Methods, Reward Design and Training Dynamics With Vanguard Group — original source. Check the original for full context.

Source published: 2026-08-18

Detailed notes

## Background Published by Ai4 on YouTube on August 18, 2026, this presentation features Ashish Bansal, Principal Machine Learning Engineer at The Vanguard Group. The session presents a production-grade framework for fine-tuning agents with Group Relative Policy Optimization (GRPO) by converting expert feedback into structured quality criteria, crafting calibrated rewards, and maintaining stable training dynamics.

## TL;DR Ashish Bansal outlines an end-to-end methodology for aligning retrieval-augmented generation (RAG) query rewriters using GRPO driven by minimal subject matter expert (SME) feedback. Demonstrating that naive binary rewards cause gradient starvation and degrade performance, Vanguard introduced a blended dense reward model combining 70% direct retrieval accuracy with 30% SME-calibrated qualitative rubrics. Stabilized by strict KL divergence tracking, gradient norm capping at 0.5, and reward variance monitoring, this approach increased accurate retrieval rates from 28% to 42–45%. The resulting framework offers an efficient, domain-agnostic blueprint that eliminates the need for expensive critic models or exhaustive reward hyperparameter searches.

## Atmosphere & style The presentation carries a methodical, pragmatic engineering tone focused on solving concrete enterprise failure modes rather than theoretical abstractions. Bansal's delivery is structured and diagnostic, walking through experimental setbacks—such as reward collapse and gradient starvation—with candid clarity before detailing the calibrated remedies. It feels like an authentic post-mortem and blueprint from the engineering trenches, conveying rigor, disciplined optimization practices, and high operational relevance.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=23T-THc_pDE&t=4s) The framework distills a small, representative sample of domain expert (SME) feedback into structured LLM-executable rubrics to train agents via GRPO without requiring massive manual annotation. - [01:52](https://www.youtube.com/watch?v=23T-THc_pDE&t=112s) In standard retrieval pipelines, poor initial user queries form the primary bottleneck, degrading downstream synthesis because the fixed retrieval index cannot compensate for ambiguous prompts. - [01:52](https://www.youtube.com/watch?v=23T-THc_pDE&t=112s) A fine-tunable Qwen 1.7B query rewriter is deployed upstream of the vector database to adapt user prompts to domain-specific vocabulary and search intent. - [05:40](https://www.youtube.com/watch?v=23T-THc_pDE&t=340s) GRPO eliminates the need for a heavyweight critic network by computing baseline means and relative advantages dynamically across sampled groups of candidate outputs. - [08:43](https://www.youtube.com/watch?v=23T-THc_pDE&t=523s) Sparse binary rewards cause gradient starvation and performance degradation (falling from 28% to 26%) when entire sampled groups evaluate to zero reward, yielding zero variance and no learning signal. - [11:06](https://www.youtube.com/watch?v=23T-THc_pDE&t=666s) Replacing binary rewards with dense, multi-criteria rubrics based on categorized SME error modes provides granular gradient signals even for flawed candidate outputs. - [11:06](https://www.youtube.com/watch?v=23T-THc_pDE&t=666s) Automated LLM judges must be cross-validated against an SME-annotated golden evaluation dataset before deployment to ensure high-fidelity rubric scoring. - [13:25](https://www.youtube.com/watch?v=23T-THc_pDE&t=805s) A blended reward strategy balancing objective task correctness (70%) with qualitative rubric criteria (30%) prevents the policy model from reward hacking. - [13:25](https://www.youtube.com/watch?v=23T-THc_pDE&t=805s) Enforcing training diagnostics—including KL divergence tracking, gradient norm capping at 0.5, and reward variance monitoring—stabilizes optimization and prevents policy collapse. - [16:03](https://www.youtube.com/watch?v=23T-THc_pDE&t=963s) Transitioning from naïve feedback to SME-calibrated dense rewards raised retrieval accuracy from a 28% baseline to 42%–45%, demonstrating that alignment success depends on reward quality rather than model scale. - [18:53](https://www.youtube.com/watch?v=23T-THc_pDE&t=1133s) Multi-step reasoning tasks can be stabilized under GRPO by assigning individualized step-level credit, preventing downstream reasoning errors from penalizing valid preliminary steps. - [21:49](https://www.youtube.com/watch?v=23T-THc_pDE&t=1309s) The distillation process preserves SME bandwidth by turning a compact, one-off sample of expert reviews into programmatic weak supervision that scales reinforcement learning autonomously.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=23T-THc_pDE&t=4s) Distilling SME Feedback into Structured Rewards for GRPO

Ashish Bansal, Principal Machine Learning Engineer at Vanguard, introduces an end-to-end framework for aligning agents using Group Relative Policy Optimization (GRPO) driven by a compact set of subject matter expert (SME) feedback. The core motivation is addressing the high cost and scarcity of domain-expert annotations: if an organization only has access to a small sample of expert evaluations, how can those ratings be transformed into a scalable, structured reward function that guides reinforcement learning?

The framework relies on three essential pillars: 1. **SME-Graded Feedback**: Domain experts evaluate and score a small representative sample of agent behaviors, uncovering nuanced error modes. 2. **Distillation into LLM Judge Rewards**: Expert feedback is distilled into structured, LLM-executable evaluation rubrics that act as automated judges during training. 3. **GRPO Policy Optimization**: The resulting structured rewards provide the supervisory signal for GRPO fine-tuning, scaling up optimization across thousands of training steps.

```mermaid flowchart LR accTitle: End-to-End Expert Distillation and GRPO Alignment Architecture subgraph S1["1. Expert Feedback"] A["SME Graded Sample"] end subgraph S2["2. Reward Distillation"] B["Structured LLM Rubric"] C["Golden Dataset Verification"] A --> B B --> C end subgraph S3["3. Policy Optimization"] D["GRPO Fine-Tuning"] C --> D end ```

---

[01:52](https://www.youtube.com/watch?v=23T-THc_pDE&t=112s) The Architecture and Bottleneck of Retrieval Agents

To make this concrete, Bansal walks through a production retrieval-augmented generation (RAG) agent. In a standard pipeline, an incoming user query is fed into a semantic or hybrid search engine over a vector database to fetch relevant text chunks. The retrieved context and the user query are then passed to a large language model (LLM) to synthesize the final response. However, the initial user query is frequently the primary failure point. Users often provide poorly structured prompts, ambiguous phrasing, or isolated keywords.

Because downstream synthesis depends entirely on the relevance of the retrieved context, poor queries directly degrade response quality: "the worse the query is, the worse the retrievals are." To alleviate this bottleneck, a query rewriter powered by a Qwen 1.7B model is introduced upstream of the retrieval step. The rewriter maps conversational or disjointed queries into coherent search prompts.

While an off-the-shelf, non-fine-tuned base Qwen 1.7B rewriter succeeds at fixing grammatical coherence and vocabulary gaps, it struggles in specialized settings: - It fails to capture specific user search intent. - It misses domain-specific vocabulary and key financial entities unique to the internal corpus. - The retrieval index itself (the vector database and search algorithms) remains fixed and non-trainable, meaning performance gains must be achieved entirely by fine-tuning the trainable rewriter.

```mermaid flowchart LR accTitle: Retrieval Agent Pipeline with Trainable Query Rewriter Q["User Query"] --> QR["Trainable Query Rewriter\n(Qwen 1.7B)"] QR -->|"Rewritten Query"| DB[("Non-Trainable\nVector Database")] DB -->|"Retrieved Chunks"| SYN["LLM Synthesis"] SYN --> RES["Final Response"] ```

---

[05:40](https://www.youtube.com/watch?v=23T-THc_pDE&t=340s) GRPO Fundamentals and Group Relative Advantage

Group Relative Policy Optimization fine-tunes the rewriter by rewarding rewrites that produce superior downstream retrievals. Rather than relying on a separate critic network (as in standard PPO), GRPO computes baseline statistics dynamically within sampled output groups:

- An incoming query is passed to the current policy $\pi_\theta$ (the Qwen 1.7B rewriter). - The policy samples a group of $G$ alternative rewrites (typically $G = 4$ to $64$, depending on available compute budget). - Each candidate rewrite executes retrieval, and an evaluation mechanism assigns an absolute scalar reward $R_i$ to each generated response. - The group baseline mean $\mu$ is computed across all sampled outputs: $$\mu = \frac{1}{G} \sum_{i=1}^G R_i$$ - For each response, the relative advantage $A_i$ is determined by subtracting the group mean from the response reward: $$A_i = R_i - \mu$$

The policy update reinforces candidate outputs with positive relative advantage ($A_i > 0$) while penalizing those that underperform the group average ($A_i < 0$). In the query rewriting context, rewrites that retrieve correct context chunks receive positive advantage updates, while unhelpful rewrites are suppressed.

```mermaid flowchart TD accTitle: GRPO Group Sampling and Advantage Computation Q["Query Input"] --> P["Policy Model πθ"] P --> G1["Rewrite 1"] P --> G2["Rewrite 2"] P --> Gk["Rewrite G"] G1 --> R1["Reward R1"] G2 --> R2["Reward R2"] Gk --> Rk["Reward RG"] R1 & R2 & Rk --> M["Compute Group Mean (μ)"] R1 & M --> A1["Advantage A1 = R1 - μ"] R2 & M --> A2["Advantage A2 = R2 - μ"] Rk & M --> Ak["Advantage AG = RG - μ"] A1 & A2 & Ak --> UPD["Policy Update"] ```

---

[08:43](https://www.youtube.com/watch?v=23T-THc_pDE&t=523s) The Failure of Sparse Binary Rewards

Bansal details a naive baseline experiment: assigning a binary reward ($1$ if the retrieved chunks are correct, $0$ if incorrect). Despite running fine-tuning, performance degraded rather than improved:

| Model Configuration | Correct Retrieval Rate | |:--- |:--- | | **Zero-Shot Baseline (Base Qwen 1.7B)** | 28% | | **Fine-Tuned with Binary Rewards** | 26% |

Analyzing the training dynamics revealed why optimization went backwards: - **Gradient Starvation**: $72\%$ of the training samples received a flat reward of $0$. - **Zero Variance within Groups**: When an entire group of sampled rewrites generates incorrect retrievals, all rewards evaluate to $0$, producing a group mean of $0$ and relative advantages of $0$. - The policy received zero gradient signal, leaving the model "like in a black room" with no direction to climb.

Error analysis on failed retrievals highlighted distinct failure modes that binary feedback could not distinguish: - **Misclassified / Missed Intent**: The rewrite shifted the underlying goal of the query. - **Missing Specific Key Entities**: Omission of proprietary product names, account types, or financial metrics. - **Underspecification**: Generating overly broad search terms that match high volumes of irrelevant content.

---

[11:06](https://www.youtube.com/watch?v=23T-THc_pDE&t=666s) Dense, Graded Rubric Formulation and Judge Verification

To resolve gradient starvation, Vanguard replaced single-bit binary metrics with a dense, multi-criteria reward system derived from SME feedback. Instead of a hard-coded binary signal, the team categorized the frequency and severity of SME-identified error modes to assign proportional weighted penalties and scores.

A crucial requirement in this methodology is the **Verification Step**: - Before deploying an LLM judge to guide GRPO training, its rubric outputs must be cross-validated against an SME-annotated golden evaluation dataset. - The alignment team verifies that the automated judge reproduces expert decisions with high fidelity across edge cases. - Once calibrated, the structured judge runs at training scale, generating dense reward signals across sampled groups so that even flawed candidates provide gradient information for relative policy updates.

[13:25](https://www.youtube.com/watch?v=23T-THc_pDE&t=805s) Blended Reward Design and Training Stability Diagnostics

To train the query rewriter effectively without falling into optimization traps, a composite dense reward function is constructed. The objective is balanced across two core signals: a 70% weight (0.7) assigned directly to actual retrieval correctness (verifying whether the retrieved documents are objectively accurate) and a 30% weight (0.3) assigned to weighted qualitative rubrics derived from subject matter feedback. This ratio keeps the model firmly grounded in actual task performance. Relying solely on qualitative SME feedback risks triggering "reward hacking," where the model chases superficial proxy rewards rather than generating queries that fetch the correct documents.

```mermaid flowchart TD accTitle: Blended Reward and GRPO Stability Architecture subgraph S1["Reward Formulation"] A["Retrieval Accuracy (0.70 Weight)"] B["Weighted Rubric Criteria (0.30 Weight)"] A --> C["Blended Dense Reward"] B --> C end

subgraph S3["Evaluation Outcomes"] D --> H["Production Retrieval Benchmark"] end ```

Training reinforcement learning policies on top of large language models carries inherent instability, often leading the policy model to drift away from the base model, generate degraded outputs, or exploit reward artifacts. To keep the training dynamics strictly controlled, three core diagnostic and stabilization mechanisms are enforced:

- **Kullback–Leibler (KL) Divergence Monitoring**: Anchors the active policy model to the reference/base model. This prevents the agent from drifting too far into degenerate generation patterns and mitigates catastrophic forgetting, ensuring the rewriter preserves its fundamental language modeling capability and continues to output clear, coherent search queries. - **Gradient Clipping & Norm Bounds**: The gradient norm is strictly capped at `0.5`. This prevents policy updates from becoming excessively spiky or driving the optimization "off a cliff" when particular groups exhibit extreme advantage swings. - **Reward Variance Monitoring**: Keeps optimization within a stable operating envelope. If reward variance collapses to near-zero, the model stops learning; if variance spikes too high, optimization becomes chaotic and unstable.

---

[16:03](https://www.youtube.com/watch?v=23T-THc_pDE&t=963s) Benchmark Results and Method Portability

A performance comparison across experimental setups demonstrates that alignment quality is directly dictated by reward quality rather than model scale alone:

Using identical GRPO algorithmic mechanics, transitioning from naïve feedback rewards to rubric-calibrated dense rewards elevated the retrieval success rate from a baseline of 28% to between 42% and 45%. The central takeaway of this framework is that "your GRPO training is better as your rewards are."

By capturing a small, high-leverage set of expensive domain-expert feedback and converting it into structured reward rubrics, teams can guide reinforcement learning reliably. Furthermore, the methodology is completely use-case-agnostic and agent-agnostic—providing a repeatable blueprint for aligning any domain-specific agent workflow.

---

[17:49](https://www.youtube.com/watch?v=23T-THc_pDE&t=1069s) Q&A: Deriving Reward Rubrics and Weak Supervision

The audience Q&A begins with an inquiry regarding how specific reward numbers and weights were determined without resorting to prohibitively expensive hyperparameter grid searches across the reward space.

In the initial implementation of the retrieval agent, domain experts analyzed query failures and successes to explain *why* specific outputs succeeded or failed. These qualitative insights were translated into programmatic weak supervision metrics (such as evaluating whether relevant target passages appeared within the top-$k$ retrieved documents). Once these scoring layers were validated against expert assessments, the scoring distributions were codified as fixed training rubrics for the rewriter agent, removing the need for combinatorial hyperparameter search loops.

---

[18:53](https://www.youtube.com/watch?v=23T-THc_pDE&t=1133s) Q&A: Credit Assignment in Multi-Step Reasoning Trajectories

An attendee raises the challenge of multi-step reasoning: if an agent executes steps 1 and 2 correctly across several iterations but errs on step 3, how can a practitioner penalize the final incorrect answer under GRPO without destroying the positive signal for the valid preliminary reasoning steps?

The response highlights that the multi-metric dense reward formulation allows intermediate reasoning steps to receive individualized, step-level credit assignment. Each step in the multi-step chain can have distinct weights assigned to its intermediate output:

- If step 1 succeeds but step 2 fails, step 3 will almost certainly diverge because subsequent token generations depend on flawed intermediate context. - By evaluating the sequence on a per-step basis, the relative advantage calculated in GRPO assigns a strong negative advantage specifically to the deviating trajectory from the point of failure. - This ensures the policy update suppresses the faulty reasoning path during optimization while maintaining positive reinforcement for the valid upstream steps.

---

[21:49](https://www.youtube.com/watch?v=23T-THc_pDE&t=1309s) Q&A: Managing Subject Matter Expert Time Commitments

The final question addresses the practical operational overhead and time commitment required from subject matter experts (SMEs), asking whether their involvement was a one-off engagement or an ongoing burden on business operations.

SME time is recognized as highly constrained and expensive. The design of this GRPO alignment methodology is specifically architected to minimize expert labor:

- Instead of demanding comprehensive manual dataset labeling, engineering teams extract a minimal, highly representative sample of domain scenarios designed to span the task universe as compactly as possible. - The small corpus of SME evaluations is distilled into programmatic rules, structured rubric criteria, and reward judges. - Once these reward judges reflect expert standards, the GRPO training pipeline scales autonomously across large datasets without requiring continuous manual supervision from operational teams.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=23T-THc_pDE&t=4s) Scalable Agent Alignment with GRPO: Methods, Reward Design, and Training Dynamics presents a practical framework for fine-tuning agents with Group Relative Policy Optimization (GRPO) to reliably match domain-specific human preferences at production scale. — Ashish Bansal

> [01:52](https://www.youtube.com/watch?v=23T-THc_pDE&t=112s) the worse the query is, the worse the retrievals are. — Ashish Bansal

> [08:43](https://www.youtube.com/watch?v=23T-THc_pDE&t=523s) like in a black room — Ashish Bansal

> [13:25](https://www.youtube.com/watch?v=23T-THc_pDE&t=805s) reward hacking — Ashish Bansal

> [13:25](https://www.youtube.com/watch?v=23T-THc_pDE&t=805s) off a cliff — Ashish Bansal

> [16:03](https://www.youtube.com/watch?v=23T-THc_pDE&t=963s) your GRPO training is better as your rewards are. — Ashish Bansal

> [18:53](https://www.youtube.com/watch?v=23T-THc_pDE&t=1133s) If step 1 succeeds but step 2 fails, step 3 will almost certainly diverge because subsequent token generations depend on flawed intermediate context. — Ashish Bansal

> [21:49](https://www.youtube.com/watch?v=23T-THc_pDE&t=1309s) Instead of demanding comprehensive manual dataset labeling, engineering teams extract a minimal, highly representative sample of domain scenarios designed to span the task universe as compactly as possible. — Ashish Bansal

## Mentioned Ashish Bansal, The Vanguard Group, Ai4, Group Relative Policy Optimization (GRPO), Proximal Policy Optimization (PPO), Qwen 1.7B, Retrieval-Augmented Generation (RAG), Kullback–Leibler (KL) Divergence

## Video description Ashish Bansal - Principal Machine Learning Engineer, The Vanguard Group

Scalable Agent Alignment with GRPO: Methods, Reward Design, and Training Dynamics presents a practical framework for fine-tuning agents with Group Relative Policy Optimization (GRPO) to reliably match domain-specific human preferences at production scale.

The summary covers three pillars: (1) methods for converting expert feedback into structured, judgeable quality criteria, (2) reward design that blends rubric-level objectives with targeted penalties for critical failures such as hallucination, misattribution, and compliance leakage, and (3) training dynamics that keep optimization stable through KL control, clipping behavior, and reward-variance monitoring.

It highlights how GRPO’s group-based relative advantages improve learning efficiency without a heavyweight critic, enabling scalable alignment across large datasets and diverse tasks. The key takeaway is that high-quality agent behavior emerges not from a single reward metric, but from a calibrated reward system plus disciplined training diagnostics that preserve both capability and safety.

[Source](https://www.youtube.com/watch?v=23T-THc_pDE)

中文简报

基于GRPO的可扩展智能体对齐：Vanguard的方法、奖励设计与训练动态 Ashish Bansal 概述了一种基于极少领域专家（SME）反馈、利用 GRPO 对齐检索增强生成（RAG）查询重写器的端到端方法。针对朴素二元奖励会导致梯度匮乏并降低性能的问题，Vanguard 引入了一种混合稠密奖励模型，将 70% 的直接检索准确率与 30% 经 SME 校准的定性评分标准相结合。在严格的 KL 散度追踪、0.5 的梯度范数上限截断以及奖励方差监控的稳定保障下，该方法将准确检索率从 28% 提升至 42%–45%。由此构建的框架提供了一个高效且与领域无关的蓝图，无需昂贵的 Critic 模型或穷举式的奖励超参数搜索。该框架将少量具代表性的领域专家（SME）反馈提炼为大语言模型（LLM）可执行的结构化评分标准，从而通过 GRPO 训练智能体，无需海量人工标注。 (0:04) 在标准检索流程中，质量较差的初始用户查询是主要瓶颈，由于固定的检索索引无法弥补模糊的提示词，导致下游综合生成效果下降。 (1:52) 向量数据库上游部署了一个可微调的 Qwen 1.7B 查询重写器，以使初始用户提示词适配特定领域的词汇和搜索意图。 (1:52) GRPO 通过在候选输出的采样组内动态计算基准均值和相对优势，免去了对重量级 Critic 网络的需求。 (5:40) 当整个采样组的评估奖励均为零时，稀疏的二元奖励会导致零方差且无法提供学习信号，从而引发梯度匮乏并导致性能下降（从 28% 跌至 26%）。 (8:43)

Reading summary

Ashish Bansal, Principal Machine Learning Engineer at Vanguard, demonstrates that Group Relative Policy Optimization (GRPO) for agent alignment depends primarily on reward density, showing that distilling scarce domain-expert feedback into blended, multi-criteria rubrics dramatically outperforms naive binary rewards. The Upstream Retrieval Bottleneck and GRPO Mechanics (1:52) In production retrieval-augmented generation pipelines, initial user queries frequently suffer from ambiguity, disjointed phrasing, or missing context. Because downstream synthesis relies strictly on context fetched from an immutable, non-trainable vector index, poor queries create an end-to-end performance bottleneck. Vanguard addresses this by deploying a trainable Qwen 1.7B query rewriter upstream of the retrieval index. While an off-the-shelf Qwen 1.7B model corrects basic syntax, it misses proprietary financial entities and user intent without targeted domain alignment. To fine-tune the rewriter without an auxiliary critic network, Vanguard applies GRPO. The policy samples a group of candidate rewrites ($G = 4$ to $64$) per query, executes retrieval, and scores each rewrite. After calculating the group's mean baseline score, relative advantage is derived by subtracting the mean from each candidate's reward. The policy update reinforces rewrites that retrieve correct context chunks while penalizing outputs that fall below the group average. Sparse Binary Penalties Cause Gradient Starvation (8:43) A naive baseline experiment using binary rewards—assigning 1 for correct document retrieval and 0 for incorrect—caused retrieval accuracy to degrade from an unaligned baseline of 28% down to 26%. Bansal attributes this regression to gradient starvation, as 72% of the training samples scored 0. When an entire sampled group returned incorrect retrievals, the group mean and all relative advantages evaluated to 0, providing no mathematical direction for policy updates. Error analysis revealed distinct failure modes that single-bit grading could not reflect, including query intent shift, omitted domain entities, and underspecification that retrieved irrelevant content. Vanguard resolved this by converting subject matter expert (SME) error frequencies into graded, multi-criteria rubrics. Before training, automated LLM judges executing these rubrics were cross-validated against an SME-annotated golden evaluation dataset to ensure automated evaluations matched human judgment. Blended Dense Rewards and Training Diagnostics (13:25) To prevent reward hacking—where an agent satisfies stylistic rubric requirements without retrieving the correct documents—Vanguard deployed a composite reward function. The objective balances an objective 70% weight on actual retrieval accuracy with a 30% weight on qualitative SME rubrics, anchoring the model to practical task execution. Training dynamics were stabilized using three core diagnostics: monitoring Kullback–Leibler divergence against the base model to prevent policy drift and catastrophic forgetting, capping gradient norms at 0.5 to suppress volatile advantage swings, and tracking reward variance to avoid optimization collapse. Under these controls, SME-calibrated GRPO elevated retrieval accuracy from the 28% baseline to between 42% and 45%, showing that reward formulation rather than algorithmic modification drove the optimization leap. Minimizing Expert Overhead and Isolating Faulty Steps (17:49) Addressing the scarcity and high cost of domain experts, Bansal explains that the framework minimizes ongoing labor by extracting a compact, highly representative sample of domain scenarios. Experts evaluate this small sample to uncover why specific queries fail, which engineers translate into programmatic weak supervision metrics—such as checking whether target passages appear in top-$k$ retrievals. Codifying these distributions into static rubrics eliminates the need for expensive combinatorial hyperparameter searches and allows the training pipeline to run autonomously. For complex multi-step reasoning agents, this dense rubric structure enables step-level credit assignment. Rather than penalizing an entire trajectory when a final answer is wrong, GRPO evaluates intermediate steps individually. The policy update assigns negative relative advantage specifically to the deviating trajectory from the point of failure, suppressing faulty reasoning while preserving positive reinforcement for valid upstream steps. While calibrated GRPO raised retrieval accuracy to 42%–45%, the retrieval index itself remained fixed, meaning systemic performance gains depended entirely on rewriter alignment. Vanguard presents the framework as an agent- and use-case-agnostic blueprint, though its ultimate ceiling remains bound to how accurately the initial golden dataset captures complex domain-specific edge cases.

精炼阅读摘要

先锋领航集团（Vanguard）主任机器学习工程师 Ashish Bansal 提出了一套基于群体相对策略优化（GRPO）的智能体对齐框架，通过将少量领域专家的精细评估提炼为密集的结构化混合奖励，在显著降低专家标注成本的同时，突破了检索增强生成管道中查询重写器的性能瓶颈。检索瓶颈与GRPO优势评估机制 (1:52) 在生产级检索增强生成（RAG）管道中，下游大语言模型的合成质量高度受制于检索上下文的相关性，而用户输入的原始提示词往往模糊简略或仅包含孤立关键词，构成了系统的主要失效瓶颈。针对底层向量数据库与检索索引不可训练的限制，Vanguard 在检索上游引入了一个由 Qwen 1.7B 驱动的可训练查询重写器。尽管未经微调的基础模型能够修复语法瑕疵，但由于缺乏专有语料认知，它在捕捉用户特定搜索意图及金融专有实体方面表现乏力。为了定向优化该重写器，团队采用 GRPO 替代了依赖独立评论家网络（Critic Network）的传统 PPO 算法。针对输入的每道查询，GRPO 从当前策略中采样一组规模为 G（通常在 4 到 64 之间）的候选重写并执行检索，评估机制为每个输出赋予绝对标量奖励值。系统随后计算组内奖励均值，并通过单个重写的奖励减去该均值得到相对优势。该机制正向强化检索效果高于组均值的候选输出，同时惩罚表现落后的重写，完全通过组内横向比对驱动策略更新。稀疏奖励的崩溃与梯度匮乏 (8:43) 在早期对齐实验中，团队尝试采用朴素的二元奖励策略（检索正确赋 1，错误赋 0）。实验表明，该方式训练后的正确检索率不仅未提升，反而由零样本基线的 28% 下降至 26%。深入的训练动态分析揭示了“梯度匮乏”的根本原因：全流程中高达 72% 的训练样本获得了扁平的 0 奖励；一旦某组采样的全部候选重写均检索失败，组内方差即降为零，导致所有候选的相对优势皆计算为零，模型如同置身于无方向指引的暗室中完全无法学习。针对检索失败用例的错误分析进一步显示，单一比特的二元信号完全抹杀了不同失败模式间的严重度差异。模型常见的退化路径包括核心搜索意图偏移、关键金融实体与账户类型丢失，以及生成过于宽泛的搜索词导致检索出海量无关文档。由于无法给这些处于中间状态的次优输出提供梯度指引，稀疏二元机制彻底失效。密集混合奖励构建与专家知识蒸馏 (11:06) 为消除梯度匮乏，Vanguard 转向从领域专家反馈中提取密集的多维度评分细则，作为训练时自动化 LLM 裁判的执行标准。该流程包含一个关键的验证环节：在部署 LLM 裁判指导 GRPO 训练前，必须将其评分输出与经专家标注的“黄金评估数据集”进行交叉比对，严格校验自动化裁判在边界案例中是否能高保真复现专家的打分分布。为兼顾任务目标与策略稳定性，团队构建了 7:3 权重的复合密集奖励函数：70%（权重 0.7）直接挂钩实际检索结果的客观准确率（如目标段落是否出现在前 k 项召回中），30%（权重 0.3）分配给源自专家反馈的定性多维细则。Bansal 强调，这种权重配比至关重要：如果完全依赖定性专家细则，模型极易产生“奖励黑客”（Reward Hacking）现象，即片面迎合评分指标的表面特征却生成无效的检索查询；而结合客观弱监督信号则能稳固锚定任务性能，避免了成本极高的超参数网格搜索。训练稳定性控制与分步归因诊断 (13:25) 利用强化学习对大语言模型进行微调容易出现策略漂移与灾难性遗忘。Vanguard 在训练流水线中设立了三重诊断机制：首先，实时监控策略模型相对于基座参考模型的 KL 散度，防止生成退化并保留基础语言建模连贯性；其次，实施严格的梯度截断，将梯度范数上限锁定为 0.5，抑制因组内相对优势剧烈波动造成的非稳态更新；最后，持续跟踪奖励方差，避免因方差归零导致学习停滞或因方差失控引起训练震荡。针对多步推理智能体的信用分配问题，该框架支持对中间推理步骤进行细粒度评估。若智能体在长轨迹的前期步骤正确而在最终步骤偏离，GRPO 的优势计算机制能够在偏离发生的具体步骤赋予强烈的负相对优势，同时保留上游有效步骤的正向强化，从而有效抑制错误轨迹而不抹杀先前的正确推导。基准表现突破与专家成本最小化 (16:03) 在保持底层 GRPO 算法与模型规模完全不变的前提下，通过引入经专家校准的密集混合奖励与稳定性诊断，智能体的正确检索率从基线模型的 28%（朴素二元奖励对齐后为 26%）大幅跃升至 42%–45%。实验结果直接印证了“GRPO 的训练上限取决于奖励函数设计质量”的核心论断。针对领域专家时间成本极高的现实，该方法在架构层面最大限度减少了人工负担。研发团队无需构建海量人工标注集，仅需提取能紧凑覆盖业务维度的极小规模代表性场景样本，便可将其提炼为结构化评估细则与弱监督逻辑。自动化校验完成后，GRPO 即可在数千训练步中自主扩展。Bansal 指出，该流程完全独立于具体业务用例与智能体结构，为各类垂直领域智能体对齐提供了通用的工程范式。先锋领航集团的实践证明，在领域智能体缺乏大规模标注的情况下，将极少量专家评估提炼为经黄金数据集校验的密集混合奖励，是激活 GRPO 并避免梯度匮乏的核心路径。尽管该框架在查询重写场景中显著提升了检索准确率并压低了人工介入成本，但如何在逻辑分支更为复杂的非线性多步工作流中自动化扩展定性细则，仍是未来需要持续探索的方向。

Detailed notes — zh

## Background 本内容由频道 Ai4 于 2026 年 8 月 18 日在 YouTube 平台发布，主讲人为来自先锋领航集团（The Vanguard Group）的首席机器学习工程师 Ashish Bansal。视频介绍了在工业生产规模下，利用群体相对策略优化（GRPO）将领域专家反馈转化为结构化奖励，从而实现智能体可靠对齐与微调的实践框架。 ## TL;DR 演讲以企业级 RAG 检索智能体中的查询重写器（Query Rewriter）微调为切入点，剖析了粗暴二元奖励导致训练陷入“暗室”与性能倒退的痛点。团队通过将少量领域专家的定性研判提炼为加权评分准则（Rubric），设计了 7:3 的真实检索命中与准则打分混合奖励机制，并配合 KL 散度约束、梯度裁剪及方差监控等训练动力学手段。实验表明该方法在保持基座语言能力的同时将检索表现大幅提升至 42%~45%，展示了如何以极小样本的专家认知撬动大规模智能体对齐。 ## Atmosphere & style 整场演讲呈现出浓厚而严谨的企业级工程实践风格，讲者态度从容沉稳，语言精炼且直击工业落地痛点。叙事节奏层层递进，从剖析朴素微调失败的真实踩坑经历，自然过渡到精巧的混合奖励设计与算法调优细节。问答互动环节针对超参数设计、多步推理信用分配与专家时间成本等尖锐问题展开了高质量探讨，给听众带来兼具实战指导性与启发性的技术分享体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=23T-THc_pDE&t=4s) 企业级智能体对齐的核心在于利用极少量的领域专家（SME）反馈，提炼为结构化奖励信号并结合 GRPO 进行大规模微调。 - [02:05](https://www.youtube.com/watch?v=23T-THc_pDE&t=125s) 在 RAG 智能体中，底层向量数据库为不可训练基础设施，而前端查询重写器（如 Qwen 1.7B）是策略优化的核心训练边界。 - [05:40](https://www.youtube.com/watch?v=23T-THc_pDE&t=340s) GRPO 通过对单条查询采样一组响应并计算组内均值 baseline，得出各候选的相对优势值来更新策略，无需重型 Critic 网络。 - [08:43](https://www.youtube.com/watch?v=23T-THc_pDE&t=523s) 朴素的 0/1 二元奖励会导致 72% 的时间无梯度信号（即'暗室'困境），导致模型召回率从 28% 反向退步至 26%。 - [11:06](https://www.youtube.com/watch?v=23T-THc_pDE&t=666s) 对齐框架（AR Framework）要求将专家定性归因提炼为 LLM 裁判准则，并在黄金验证集上验证一致性后再投入密集加权奖励计算。 - [13:25](https://www.youtube.com/watch?v=23T-THc_pDE&t=805s) 采用 70% 真实检索准确度加 30% 专家多维度规（Rubric）的混合奖励，能有效防止模型发生强化学习中的奖励作弊（Reward Hacking）。 - [13:25](https://www.youtube.com/watch?v=23T-THc_pDE&t=805s) 通过 KL 散度约束、梯度裁剪（截断至 0.5）以及奖励方差监控三大手段，可以有效防止模型漂移与训练崩溃。 - [16:03](https://www.youtube.com/watch?v=23T-THc_pDE&t=963s) 在相同 GRPO 算法下，借助专家加权准则与混合奖励，查询重写质量显著跃升至 42%~45%，证明奖励质量决定了 GRPO 训练上限。 - [18:53](https://www.youtube.com/watch?v=23T-THc_pDE&t=1133s) 面对多步推理链路，可通过单步稠密奖励（Step-Level Rewarding）与负向优势度控制误差级联，避免错误污染整条策略更新。 - [21:49](https://www.youtube.com/watch?v=23T-THc_pDE&t=1309s) 通过抽取覆盖业务全景的极小代表性样本集获取专家反馈，能够以最小的专家工时投入撬动大规模且可扩展的工业级智能体对齐。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=23T-THc_pDE&t=4s) 智能体对齐的核心命题：从小样本专家反馈到 GRPO 训练来自先锋领航集团（Vanguard）的首席机器学习工程师 Ashish Bansal 带来了关于利用群体相对策略优化（GRPO, Group Relative Policy Optimization）对齐智能体的实践分享。在工业级落地场景中，团队常常面临一个根本性的核心问题：“如果我们只有极少量的领域专家（SME）反馈，是否能将这些反馈转化为结构化的奖励信号，并借助这些奖励来对齐和微调智能体？”为了跑通这套方法论，整个对齐体系必须紧扣三大支柱： 1. **专家分级反馈获取**：由领域专家对一小部分具有代表性的智能体行为样本进行精准评审与打分。 2. **结构化评判奖励蒸馏**：将专家的定性经验与反馈提炼为可由大语言模型（LLM）执行的结构化裁判准则（Rubrics）。 3. **GRPO 策略优化与规模化泛化**：将结构化裁判奖励无缝接入 GRPO 的训练动力学中，探索如何将小规模反馈转化而来的评分体系在微调过程中进行大规模推广应用。为了具象化展示这套方法的威力，演讲以企业级应用中极其经典的检索增强生成（RAG）智能体为例，全面复盘如何优化智能体以针对各类复杂查询给出最优解。 --- ### [02:05](https://www.youtube.com/watch?v=23T-THc_pDE&t=125s) 经典检索智能体的痛点与查询重写器机制在标准检索智能体架构中，用户的查询（Query）输入系统后，首先会通过语义检索或混合检索算法从底层文档库及文本块（Chunks）中召回相关资料；随后，检索到的上下文与原始查询一同输入大语言模型完成最终回答的综合生成。然而，这套链路中最脆弱的瓶颈往往就在于原始查询本身——用户输入的查询通常极不规范，可能只是一堆零散关键词，或者表意模糊残缺。检索系统的规律显而易见：“输入的查询质量越差，召回的文档质量就越差；而糟糕的召回结果必然直接导致大模型生成幻觉或劣质回答。” 为了消除输入质量带来的性能瓶颈，业界通行的做法是在检索前引入查询重写器（Query Rewriter）。在此方案中，团队选用了一个轻量级的开源小模型（Qwen 1.7B）专门承担重写任务，其目标是将口语化、碎片化的用户输入重构为语法通顺、词汇丰富且表意严密的检索词，从而显著拉近用户词汇与知识库索引之间的语义距离。 ```mermaid flowchart LR accTitle: 检索智能体架构与训练边界 subgraph Pipeline["端到端推理链路"] direction LR U["用户原始 Query"] --> QR["Query Rewriter (Qwen 1.7B)"] QR --> RQ["重写后 Query"] RQ --> VDB["向量数据库 / 检索系统"] VDB --> C["检索到的 Chunks"] C --> LLM["生成模型 (LLM)"] LLM --> A["最终回答"] end classDef trainable fill:#e1f5fe,stroke:#0288d1,stroke-width:2px; classDef nonTrainable fill:#f5f5f5,stroke:#9e9e9e,stroke-width:1px; class QR trainable; class VDB,LLM nonTrainable; ``` 然而，一个未经领域微调的朴素 Qwen 1.7B 模型虽然具备基础的语法润色和词汇扩充能力，但由于它并未深入学习过企业专属语料库，重写时往往会曲解用户意图，甚至彻底丢掉语料库强相关的专属关键词与核心业务实体。在剖析整个智能体系统时，Ashish 明确划分了系统的可训练边界： - **不可训练模块（基础设施层）**：底层的向量数据库及固化的检索索引，属于不可直接微调的企业 AI 基础设施。 - **可训练模块（智能体策略层）**：前端的查询重写器模型。团队的核心思路是运用 GRPO 微调机制，“以重写最终引发的检索结果优劣为依据，反向优化重写器”，从而在运行时精准驱动底层检索系统召回高价值文本块。 --- ### [05:40](https://www.youtube.com/watch?v=23T-THc_pDE&t=340s) GRPO 算法原理与优势计算机制在深入实际调优前，Ashish 系统回顾了 GRPO 的工作流。作为一种无需独立重型 Critic 网络的强化学习微调技术，GRPO 极其适合对齐轻量级生成策略。当一条原始查询进入策略模型 $\pi_\theta$（即 Qwen 1.7B 查询重写器）后，模型并不会只生成单一候选，而是依据算力预算一次性采样输出一组响应序列（Group of Responses，通常设定在 4 到 64 之间）。 ```mermaid flowchart TD accTitle: GRPO 策略优化机制 Q["输入 Query"] --> P["策略模型 π_theta (Rewriter)"] P --> G1["响应 Rewrite 1"] P --> G2["响应 Rewrite 2"] P --> Gdots["..."] P --> Gn["响应 Rewrite G (4~64个)"] G1 --> R1["计算奖励 r_1"] G2 --> R2["计算奖励 r_2"] Gdots --> Rdots["..."] Gn --> Rn["计算奖励 r_G"] R1 & R2 & Rdots & Rn --> Mean["计算组内均值 Baseline (Mean)"] Mean & R1 --> A1["计算相对优势 A_1 = r_1 - Mean"] Mean & R2 --> A2["计算相对优势 A_2 = r_2 - Mean"] Mean & Rn --> An["计算相对优势 A_n = r_G - Mean"] A1 & A2 & An --> Update["策略更新 Policy Update"] ``` 每条生成的重写候选都会被分配一个量化奖励，系统随后计算出该组所有候选奖励的算术平均值作为组基线（Group Baseline Mean）。随后，用各个候选的独立奖励减去组均值，便可得出该候选在当前组内的相对优势值（Relative Advantage）： - **正相对优势**：代表该重写质量优于组内平均水平，策略梯度会大幅强化该行为，促使模型未来更倾向于生成此类高质量重写。 - **负相对优势**：代表该重写劣于平均水平，模型将在梯度反向传播中受到抑制，主动远离此类低效生成模式。在查询重写场景下，这一机制的终极目标非常纯粹：只要重写出的查询能检索出命中了正确答案的高质量 Chunks，该重写就能获得更高的正优势；反之，若导致了劣质召回，则会在策略更新中被严厉惩罚。 --- ### [08:43](https://www.youtube.com/watch?v=23T-THc_pDE&t=523s) 朴素二元奖励的失效与“暗室”困境在早期的基线实验中，团队尝试采用了一种最直观的朴素策略——二元奖励（Binary Rewards）。即重写后的查询触发检索后，若召回的文本块命中了正确答案，则赋予奖励 1；若未命中，则赋予奖励 0。然而，训练结果却令人大跌眼镜： - **未微调基线（Base Qwen 1.7B）**：未经微调的重写模型配合检索，正确召回率为 **28%**。 - **二元奖励微调后**：模型在投入了大量训练后，召回率不仅没有提升，反而倒退到了 **26%**。深入分析训练监控数据后，团队发现了症结所在：在整个训练周期中，高达 **72% 的时间内奖励全为 0**。这意味着策略梯度彻底陷入了信号真空——由于缺乏连续的反馈阶梯，模型“没有任何梯度可以攀爬，就像被关在一个完全漆黑的房间里”，重写器完全无法分辨哪种生成略有改善、哪种生成彻底失败。对召回失败案例的错误归因分析进一步表明，模型的错误并非简单的“对与错”，而是呈现出多维度、多类别的失败模式： - **意图偏离（Mis-intents）**：误解了用户真实意图，重写到了完全无关的方向。 - **核心实体缺失（Missing Key Entities）**：遗漏了业务语料库中至关重要的专有名词或产品术语。 - **信息欠指定（Underspecified）**：生成的查询过于宽泛笼统，无法在海量向量空间中精准定位切片。粗暴的 0/1 二元信号完全抹杀了这些错误背后的细微梯度，必须转向更高级的密集奖励体系来牵引策略走向正确轨道。 --- ### [11:06](https://www.youtube.com/watch?v=23T-THc_pDE&t=666s) 专家经验蒸馏、黄金集验证与密集加权奖励设计为了突破稀疏奖励造成的性能倒退，团队提出了一套结构化的对齐框架（AR Framework）。该框架旨在将业务领域专家的定性研判系统化地转化为训练动力学中的可执行信号： 1. **专家经验提炼**：梳理领域专家对各类典型召回错误（如实体遗漏、意图偏离等）的打分标准。 2. **构建结构化裁判准则（LLM Executable Rubrics）**：将定性规则转化为大模型裁判可自动化运行的结构化评估提示词。 3. **黄金数据集（Golden Dataset）严格验证**：Ashish 特别强调，这是整个方案中最关键的闭环节点。当结构化细则制定完成后，绝不能盲目直接上线微调，必须预先在专家标注的高质量黄金验证集上评估该 LLM 裁判的打分结果，确保大模型裁判的输出与专家的真实偏好高度一致。 4. **密集加权奖励（Dense Weighted Rewards）注入**：通过在真实运行环境中统计各类错误发生的频率，为不同维度的错误赋予差异化的权重惩罚，将原本非黑即白的 1-bit 二元奖励全面升级为细粒度的多维加权评分矩阵。通过这一系列设计，策略模型在每一次探索中都能获得清晰的细粒度奖惩反馈，从而彻底告别“暗室摸索”，在 GRPO 训练中实现真正稳定、高效的策略对齐。 ### [13:25](https://www.youtube.com/watch?v=23T-THc_pDE&t=805s) 混合奖励设计与防止奖励作弊（Reward Hacking）为了让模型朝着正确的方向优化，团队设计了一种由两部分构成的混合奖励（Blended Reward）评分机制：其中 70%（权重 0.7）取决于检索的实际准确性（即检索结果是否正确），剩余的 30%（权重 0.3）则分配给基于多维度量规（Weighted Rubric）的打分。之所以采用这种 7:3 的加权方式，核心目的在于确保模型不会仅仅盲目追逐专家评分的标准，从而掉入强化学习中常见的“奖励作弊”（Reward Hacking）陷阱。通过将 70% 的权重牢牢锚定在最终真实的检索命中上，模型得以在具备实用价值的硬指标约束下稳定探索。这种稠密奖励（Dense Rewards）与加权评分准则的结合，有效引导了查询重写器（Rewriter）的训练走向。然而，仅靠奖励设计还不够，强化学习训练极易发生模型漂移（Drift），即策略模型偏离基座模型太远，导致输出质量严重退化或产生不可预测的异常行为。为了实时掌控并稳定训练动态，团队引入了三大监控指标： ```mermaid flowchart TD accTitle: 混合奖励设计与训练动态监控 subgraph Reward["混合奖励计算 (Blended Reward)"] R1["实际检索准确度 (70%)"] R2["加权评分准则 Rubric (30%)"] R_Total["综合奖励分数"] R1 --> R_Total R2 --> R_Total end subgraph Dynamics["训练动态控制 (Training Dynamics)"] D1["KL 散度约束 锚定基座模型，防止灾难性遗忘"] D2["梯度裁剪 (Clip / Grad Norm = 0.5) 抑制尖峰震荡，防止梯度失控"] D3["奖励方差监控 (Reward Variance) 保持健康区间，确保有效学习"] end R_Total --> Dynamics ``` - **KL 散度（KL Divergence）**：将训练中的策略模型严格锚定在原始基座模型（Reference Model）附近，防止模型跑偏太远，同时避免灾难性遗忘（Catastrophic Forgetting），确保重写器不会丧失生成通顺连贯语句的基本语言能力。 - **梯度裁剪与范数控制（Clip and Grad Norm）**：将梯度范数上限严格截断在 0.5，避免特定批次（Group）回传时产生极端尖锐的训练尖峰，杜绝“掉下悬崖”（Off a cliff）式的训练崩溃。 - **奖励方差（Reward Variance）**：密切关注奖励分布的波动。如果奖励方差过低，说明区分度不足，模型学不到任何有效特征；如果方差过高，则会导致优化过程剧烈震荡。必须将其维持在健康的平衡区间。 --- ### [16:03](https://www.youtube.com/watch?v=23T-THc_pDE&t=963s) GRPO 对齐效果评估与核心结论通过这套奖励与训练控制体系，检索对齐取得了显著的性能跃升。在最初仅使用基线模型（Baseline）时，检索准确率仅有 28%；若引入朴素奖励（Naive Rewards），表现甚至出现反噬，准确率下滑至 26%；而在引入专家反馈转换而成的加权准则与混合奖励后，查询重写质量大幅攀升至 42%（讲者提到跃升至 42% 到 45% 的区间）。这一飞跃是在完全使用相同 GRPO 算法的前提下实现的。讲者特别强调：“你的 GRPO 训练效果取决于你的奖励质量”（"your GRPO training is better as your rewards are."）。整套方案的精髓在于如何把极度昂贵、体量极小的专业领域专家（SME）反馈转化为高质量、可量化的奖励规则。这套对齐框架完全独立于具体用例或智能体架构（Agent Agnostic），可以广泛迁移到各类特定场景的智能体训练中。 | 策略版本 | 检索成功率 / 良好率 | 方案特征 | |:--- |:--- |:--- | | **基线模型 (Baseline)** | 28% | 无特定强化微调的初始状态 | | **朴素奖励 (Naive Rewards)** | 26% | 缺乏精细加权的粗粒度反馈，出现性能倒退 | | **GRPO + 专家反馈加权准则** | 42% ~ 45% | 7:3 混合稠密奖励、KL 控制与梯度平滑约束 | --- ### [17:49](https://www.youtube.com/watch?v=23T-THc_pDE&t=1069s) 现场问答：奖励准则的构建与超参数探索演讲结束后进入问答环节。第一位观众提问：在实际操作中，如果把各类奖励权重当作超参数来进行网格搜索（Hyperparameter Search）会异常昂贵且耗时，团队究竟是如何确定这些具体的奖励指标与权重分配的？讲者解答道，这并非纯粹靠暴力搜索超参数得来。在最初使用朴素检索智能体时，团队安排了一组领域业务人员（Theme/SME 团队）深入排查样本，逐个分析为什么某些改写后的查询表现更好、为什么某些会失败。随后，工程团队将这些定性的人工归因提炼为弱监督指标（Weak Supervisor Metrics）——例如“检索出的文档是否位于 Top-K 列表中”、“内容相关度得分是否达标”等。当这些维度的打分系统成型后，它们便直接被固化为评分准则（Rubric）并注入到训练动态中。这一过程本质上是将业务专家的定性经验“发现”并转化为训练动力学的一环。 --- ### [18:53](https://www.youtube.com/watch?v=23T-THc_pDE&t=1133s) 现场问答：多步推理（Multi-Step Reasoning）中的逐步奖励与信用分配现场第二位观众提出了关于多步推理对齐的深层技术问题：在多步推理链路中（例如进行了三到四步），假设第一步和第二步都是完全正确的，但第三步由于一个小错误导致最终结果全错；而另一个候选生成可能全盘皆错。在这种端到端结果均为“错误”的情形下，GRPO 机制如何在指出最终错误的同时，不抹杀模型在前两步推理中的正确性？ ```mermaid flowchart LR accTitle: 多步推理的逐步稠密奖励归因 subgraph StepRewards["多步推理轨迹分解"] S1["步骤 1：正确 (赋予正向阶段权重)"] --> S2["步骤 2：正确 (赋予正向阶段权重)"] S2 --> S3["步骤 3：错误 (触发惩罚机制)"] end subgraph Advantage["GRPO 优势度计算"] S3 --> ADV["生成高负向优势度 (Negative Advantage) 阻止错误前向传播并抑制策略更新"] end ``` 讲者解释称，面对具备多步推理能力的领域模型，正是稠密奖励机制（Dense Rewards）大显身手的场景： - **逐步评分与独立赋权（Step-Level Rewarding）**：系统不必等到最终结果输出才给出一个单一的稀疏信号，而是可以拆解推理链路，为每个单独的推理步骤赋予不同的权重。 - **误差级联控制与负向优势度**：模型在生成后续 Token 时高度依赖上文，一旦第二步发生错误，后续第三步必然受到污染。通过监控具体步骤的表现，系统可以在出错节点立刻介入，使该轨迹在 GRPO 计算中获得更强的负向优势度（Negative Advantage）。 - **策略保护**：这种负优势度能确保错误的单步操作不会正面影响策略更新，精准实现基于单步（Step-by-step）粒度的奖励与惩罚分配。 --- ### [21:49](https://www.youtube.com/watch?v=23T-THc_pDE&t=1309s) 现场问答：业务专家（SME）的时间投入与数据杠杆最后一位观众提出了工程落地的现实痛点：在向业务专家寻求支持时，对方最关心的往往是“你们到底需要我们投入多少工时”。在日常业务繁重的情况下，这种专家介入是一次性的还是长期的？讲者坦言，领域专家（SME）的反馈成本确实极度昂贵，时间也极难协调。正因如此，团队才选择 GRPO 作为算法支点——因为 GRPO 能够通过组内相对优势度最大化利用优质奖励的引导能力。实践中，团队并不需要海量的全量人工标注，而是采取“小样本高杠杆”策略： - 抽取一个规模极小、但代表性极强的微型数据子集（Small Sample Subset）。 - 该样本集必须尽可能完整覆盖业务场景的问题全集（"cover the whole universe but keep it as small as possible"）。 - 专家仅需对这一极小规模的样本提供深度归因反馈，工程团队随即将其提炼为可复用的自动化评估规则与奖励函数。这种做法的核心在于利用昂贵且小规模的专家认知，撬动大规模、可扩展（Scalable）的模型对齐训练，从而以最小的业务专家工时投入实现工业级智能体的稳定对齐。讲者在掌声中结束了演讲。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=23T-THc_pDE&t=4s) 如果我们只有极少量的领域专家（SME）反馈，是否能将这些反馈转化为结构化的奖励信号，并借助这些奖励来对齐和微调智能体？ — Ashish Bansal > [02:05](https://www.youtube.com/watch?v=23T-THc_pDE&t=125s) 输入的查询质量越差，召回的文档质量就越差；而糟糕的召回结果必然直接导致大模型生成幻觉或劣质回答。 — Ashish Bansal > [02:05](https://www.youtube.com/watch?v=23T-THc_pDE&t=125s) 以重写最终引发的检索结果优劣为依据，反向优化重写器 — Ashish Bansal > [08:43](https://www.youtube.com/watch?v=23T-THc_pDE&t=523s) 没有任何梯度可以攀爬，就像被关在一个完全漆黑的房间里 — Ashish Bansal > [13:25](https://www.youtube.com/watch?v=23T-THc_pDE&t=805s) 掉下悬崖 — Ashish Bansal > [16:03](https://www.youtube.com/watch?v=23T-THc_pDE&t=963s) 你的 GRPO 训练效果取决于你的奖励质量 — Ashish Bansal > [16:03](https://www.youtube.com/watch?v=23T-THc_pDE&t=963s) your GRPO training is better as your rewards are. — Ashish Bansal > [21:49](https://www.youtube.com/watch?v=23T-THc_pDE&t=1309s) 你们到底需要我们投入多少工时 > [21:49](https://www.youtube.com/watch?v=23T-THc_pDE&t=1309s) cover the whole universe but keep it as small as possible — Ashish Bansal ## Mentioned Ashish Bansal, 先锋领航集团 (The Vanguard Group), Ai4, GRPO (Group Relative Policy Optimization), Qwen 1.7B, RAG (检索增强生成), LLM (大语言模型), KL 散度 (KL Divergence) ## Video description Ashish Bansal - Principal Machine Learning Engineer, The Vanguard Group Scalable Agent Alignment with GRPO: Methods, Reward Design, and Training Dynamics presents a practical framework for fine-tuning agents with Group Relative Policy Optimization (GRPO) to reliably match domain-specific human preferences at production scale. The summary covers three pillars: (1) methods for converting expert feedback into structured, judgeable quality criteria, (2) reward design that blends rubric-level objectives with targeted penalties for critical failures such as hallucination, misattribution, and compliance leakage, and (3) training dynamics that keep optimization stable through KL control, clipping behavior, and reward-variance monitoring. It highlights how GRPO’s group-based relative advantages improve learning efficiency without a heavyweight critic, enabling scalable alignment across large datasets and diverse tasks. The key takeaway is that high-quality agent behavior emerges not from a single reward metric, but from a calibrated reward system plus disciplined training diagnostics that preserve both capability and safety. [Source](https://www.youtube.com/watch?v=23T-THc_pDE)

Source quotes

[0:04] Ashish Bansal: 如果我们只有极少量的领域专家（SME）反馈，是否能将这些反馈转化为结构化的奖励信号，并借助这些奖励来对齐和微调智能体？ [2:05] Ashish Bansal: 输入的查询质量越差，召回的文档质量就越差；而糟糕的召回结果必然直接导致大模型生成幻觉或劣质回答。 [2:05] Ashish Bansal: 以重写最终引发的检索结果优劣为依据，反向优化重写器 [8:43] Ashish Bansal: 没有任何梯度可以攀爬，就像被关在一个完全漆黑的房间里 [13:25] Ashish Bansal: 掉下悬崖 [16:03] Ashish Bansal: 你的 GRPO 训练效果取决于你的奖励质量 [16:03] Ashish Bansal: your GRPO training is better as your rewards are. [21:49] 你们到底需要我们投入多少工时 [21:49] Ashish Bansal: cover the whole universe but keep it as small as possible

Source quotes

[0:04] Ashish Bansal: Scalable Agent Alignment with GRPO: Methods, Reward Design, and Training Dynamics presents a practical framework for fine-tuning agents with Group Relative Policy Optimization (GRPO) to reliably match domain-specific human preferences at production scale. [1:52] Ashish Bansal: the worse the query is, the worse the retrievals are. [8:43] Ashish Bansal: like in a black room [13:25] Ashish Bansal: reward hacking [13:25] Ashish Bansal: off a cliff [16:03] Ashish Bansal: your GRPO training is better as your rewards are. [18:53] Ashish Bansal: If step 1 succeeds but step 2 fails, step 3 will almost certainly diverge because subsequent token generations depend on flawed intermediate context. [21:49] Ashish Bansal: Instead of demanding comprehensive manual dataset labeling, engineering teams extract a minimal, highly representative sample of domain scenarios designed to span the task universe as compactly as possible.

Original transcript

[0:04] Thank you for joining me for this session. Uh this side Ashish Bansil. Uh I'm working [0:08] as a principal machine learning engineer at Vanguard. And uh today I would like to have [0:13] these 20 minutes to walk through a solution that can uh help align the agents through a smaller set [0:19] of feedback uh using gRPO methods. So uh how how we uh start thinking from a objective of [0:30] we do have a smaller set of theme feedback. So I want to take you through a core question of uh if [0:37] we if we have a smaller set of theme feedback can we convert them into more of a structured [0:44] rewards and can these rewards help you align and train the uh agents. So for this particular method [0:52] we do need three main components. Uh first one is the theme uh graded feedbacks. So basically [1:00] what happens here is we need some of the experts that can grade a small sample of agent behavior. [1:07] The next one is to uh see this uh behavior and feedback could be distilled into a structured [1:14] LLM executable judge rewards and these rewards could be used in the training dynamics of gRPO. [1:21] So the third one after the structured uh rewards are been constructed these structured rewards are [1:29] used to optimize the agent uh for the rewards. And the main question here is [1:35] can these smaller uh feedbacks could be converted into a rubric that could be applied across [1:43] at scale in this gpo fine-tuning process. So for the uh presentation purposes, I'll take a [1:52] very classic uh retrieval agent and walk through how we are training this agent [1:59] to do uh an objective of finding the best solution for that particular query. [2:05] Just taking through a classic example of how the uh retrieval agent looks like. So [2:13] here the first piece is uh the query comes in. The query is used to retrieve the uh [2:21] proper documents and the chunks from the whole document basis and it can use either the semantic [2:27] search or the hybrid search from uh either of the algorithms. Once the retrievalss are done [2:34] that retrievalss are used with the query to synthesize an answer from an LLM. [2:38] But if you see the ma major problem here is the query itself. Sometimes what happens as a user you [2:44] just put some of the keywords or the query is very much uh not constructed in a very well mannered. [2:52] So there are a lot of techniques in uh industry what practitioners are using to [2:58] boost the performance of the query itself. For this one, we we will use a query rewriter as a [3:03] mechanism to rewrite the query that is coming from the user base. And can we make it more coherent [3:10] so that the retrievalss are better because the bottleneck here if you see the worse the query is [3:15] the worse the retrievalss are. And if you have very bad retrievalss, the synthesize of that [3:20] particular response would be bad as well. So we will utilize a query rewriter here which is a quen [3:26] 1.7B uh model. And what happens is once we get a query uh we will use that query to rewrite back [3:34] and uh just do the retrievalss on that particular query. What it will give as a mechanism here is [3:40] we are able to close the gap of that keywords or a bad query into a more of a coherent language [3:46] or we are just trying to fill in the vocabulary of that particular query. [3:54] So if if I just use that as a nave example of a non-finetune based model, what happens here is [4:02] it is able to do the coherence and able to close the gap of the vocabulary. But uh because it's [4:09] a uh it's not trained on the corpus that we have. So it is still missing the intent of the query [4:15] and it's still missing the uh keywords that are very much corpus specific and some of the key [4:21] entities that this uh corpus includes. So there's a very big room to improve or this whole agent [4:32] how we want to improve this one. So similar uh architecture here. Uh what what we want to see is [4:41] what are the components that we can touch in this whole architecture to improve the whole retrieval [4:46] agent and what we can't touch. So if you see the first uh piece that is a retrieval piece it is [4:53] it is a non-trainable piece. It is a part of an whole AI infrastructure [4:57] that your whole corpus are setting and you're trying to retrieve from that uh vector database. [5:03] But the second one is the trainable piece which is a uh uh like you can train that rewriter using [5:12] uh different mechanism. So here here is the whole idea. So we would use a gRP [5:18] fine uh fine-tuning mechanism to optimize the rewrites rewrites by the retrieval they cause. [5:26] So this is a very subtle but a very nuance piece which would help us to uh get a query that can [5:34] give the better retrievalss at the runtime and that's what we are trying to chase. [5:40] Before going uh into more of like how we would improve this system I just want to uh give a [5:45] refresher on what gpo exactly does and how does this apply to this use case. So this is this [5:53] is a whole uh architecture of gRPO. What happens here is uh GRPO stands for group relative policy [6:01] optimization and it's a reinforcement learning technique which we would use to fine-tune our [6:08] retrieval agent. So I'll just walk through how how uh a particular query comes in and how we will [6:16] try to optimize our query rewriter through this mechanism. So if you see from the left hand side [6:22] a query comes in and then it goes into a policy uh model which is our query rewriter [6:29] that is a pi theta and what it does is it tried to generate a group of responses and the group of [6:37] responses are from 1 to g. So it's not only just producing one result. So this is the uh this is [6:43] the main crux of gRPO is it's tried to generate a group of responses and uh in the whole gRPU [6:50] architecture you can uh start from either 4 to 64 kind of group uh and depending on how much compute [6:56] you have you try to generate those responses once these group of responses are generated there is a [7:03] there's a mechanism where we assign rewards to all these responses and uh once the rewards [7:09] are assigned what we do is we take a mean of all these responses which is our group baseline mean [7:16] using this mean. So we do have a reward and we do have a mean. So what we start doing is [7:22] for for each response there is a reward and there is a mean that we have seen. So what we do is for [7:28] each reward we try to subtract the mean and try to get the advantage that is a relative advantage [7:34] of that particular response that we are getting here. So after assigning all the advantages to all [7:40] these responses, we try to do the policy update of that particular model. And what is our goal here [7:47] for the GRPO? What we want to do is we want to uh see the better the advantage, [7:53] it has a more uh reinforcement on this should be the response generated by the model again and [8:00] less the advantage the model should not and stay away from generating those kind of responses. [8:06] Going back to our use case, what we are trying to do is we are trying to train a rewriter here. [8:11] And what does that mean for our rewriter in this case? So if I use gRPO, my main objective was to [8:19] write a query which is able to retrieve the better [clears throat] chunks [8:23] not uh not the worst chunks which are able to answer that particular uh query. So [8:30] using this advantage my goal would be to get the right retrievalss as a better advantage and if [8:37] it is not doing the right retrievalss I'll just uh deny that in the uh training of that policy. [8:43] So as a first step what we did is uh we took a very naive approach of using binary rewards. [8:50] So my query comes in it rewrites and it retrieves some of the documents. So what [8:57] I did over there is u every time a query comes in it retrieves some chunks. So if the chunks [9:04] are correct according to the query we say it's a yes like it's a one and either it's not correct we [9:10] say it's not correct we give a zero reward and if you see the results here the baseline uh what I'm [9:16] trying to present here is baseline is just query as it is being rewritten by my base quen model [9:23] and then a retrieval happens. What we have seen is we are able to do 28% uh correct retrievalss [9:31] from the whole set but when I trained the model using just the binary rewards it went backwards [9:37] to 26%. Why that is happening? Even though we have uh did the so much of feedback into it [9:44] we did the training but we are going backwards. What is the why this is happening? So if you see [9:50] the red bar uh on the chart it it depicts like 72% of the time u my rewards are always zero [9:59] what that means for gRP training so the uh the policy uh is not getting any signal it's it [10:06] doesn't have any gradient to climb and the model is not learning anything it's like in a black room [10:12] so it's just sitting over there and saying okay I don't know because all the rewrites happening [10:17] from the rewriter is had. So looking into more into details of how can we improve this, [10:24] we went into look for what are the different errors that are coming through this uh retrievalss [10:31] and what we have seen is uh there are different classes uh that are that are getting [10:38] wrong here. Some of them are like um mis intents. Some of them are missing the key key uh specific [10:45] entities and uh and others are more of like they they are not underspecified. Um so [10:52] doing this what we have seen is we can't just use a very naive uh smaller you know [10:59] binary rewards. We need something more advanced in the sense that could help us shift the policy into [11:06] a right direction. So here is the proposed AR framework that I was talking before. So [11:14] if we have a subject matter expert uh feedback on these kind of errors like how much times it is [11:21] mclassifying missing the key entities uh it's underspecified and all those criterias. Can we [11:27] use this feedback convert these feedback into a LLM executable rewards? Once we have those things [11:37] um so once we have those things uh it is it is distilled into a uh into a structured uh responses [11:46] and can we use this structure at scale to uh get more rewards at the training time [11:53] once we distill it. This is the main key piece that I really want to emphasize on is [11:59] once we try to create a structure rubrics there is no way to verify it. So we should have a [12:05] mechanism in between once this theme smaller set of theme feedback comes in you create a structure [12:11] you should verify it using uh a golden data set and see are your rewards coming from that [12:18] particular judge is equivalent to theme feedback or not. Once you have those kind of rewards [12:25] verified, you can use those rewards in the gRP fine tuning uh tuning uh mechanism. [12:31] So this is the whole proposed framework. Going back to what we have seen is we have seen that [12:36] our nave u you know binary ones are failing. So can we do into a dense revolts? Uh so here is the [12:46] here's the fix that what we are trying to come at is we went from one blind bit to a graded score [12:52] uh uh whole metrics and what we have seen is uh these are the errors that we are seeing over there [13:00] instead of just taking these as a standard weight what we did is when we are trying to [13:05] do the distillation of that SM feedback we also are getting the weight how much time this kind of [13:12] scenario is happening in our uh runtime. So we assigned the particular weights to [13:17] each of these errors and replace the uh you know um the binary rewards into a more of [13:25] uh um a dense rewards and what we have kept here is we kept a blended score of [13:31] uh two things one is the 0.7 times the actual retrieval that is happening is it correct or not [13:37] and the next one is the.3% of all these uh weighted uh rubric and why we are doing it um [13:45] The main reason we want to do that kind of blended reward here is that we are not u [13:52] we are trying to make the model learn in a right direction. It's not getting the signal [13:58] from the theme feedback and I don't want my model to chase those rewards only. That that goes into [14:04] a dynamics of uh you can say reward hacking. So that's where I was keeping it grounded towards [14:10] that in the 70% of the time uh it has to be always depending on the retrievals it's doing. [14:18] So what we have seen is uh the dense and uh dense and this weighted rubrics is able to help [14:26] train my model in the right direction and uh we have seen uh because of the blended score [14:32] I just want to keep a track on how the model is getting uh stabilized during my training because [14:37] when you try to train models through reinforcement learning sometimes model tend to uh you know drift [14:45] from the base model itself and it start giving bad responses and try to hack the rewards in many of [14:51] these training dynamics. So here is the some of the uh uh kind of like metrics that we were going [14:59] against and we are trying to make sure that our uh training model is not drifting from the base. [15:05] So first is the KL divergence which helped us uh you know we anchor the model to its reference [15:12] uh that is the original model so it can't drift too far as I was talking before [15:17] and it also prevents the catastrophic forgetting the rewriter should not forget how [15:23] to write a coherent query. Next one is the clip and the grad norm. uh this is capped at 0.5 so [15:29] that when a particular group is coming it's not try to do a lot of like spiky kind of uh training [15:36] and uh it's not going to go off a cliff uh kind of scenario. The third one is the uh reward variance. [15:44] So we don't want to keep our rewards very high or very low. So if it is very low it's it's not [15:51] learning anything. If it's too high it's instable. So we want to keep a check on that rewards and is [15:57] trying to come in a healthy range. That's where our model is trying to have a better training. [16:03] Um the this is the this is the part what um what we are seeing here is going back to how we have [16:12] seen when we have only the baseline we were able to get the 28% of the times we were able to get [16:18] the right retrieval from those uh documents. when we used the nave rewards we uh were able to get [16:24] only 26% of the times that retrievals were good but after doing this uh training using feedback we [16:32] were able to get for uh 42% of that particular uh times our model uh uh the query was looking good [16:41] um and so this is a huge jump from our 28% to 45% and this is using the exact same algorithm of gRPO [16:53] but what what I want to emphasize is your gRPO training is better as your rewards are. [16:59] So the main crux of this particular uh uh scenario was if you have a better rewards [17:08] your model would be better and where we are getting these rewards are getting from a smaller [17:13] set of feedback. So the whole intent is if we have a very smaller and expensive set of theme feedback [17:22] can we convert them into rewards and this whole mechanism is uh your use [17:27] case agnostic or agent agnostic I can say you can use this uh methodology across any agents [17:34] and um train those things in a very in a very beautiful way which is aligned to your use cases. [17:41] So that's all from my side. if you have any questions yet. [applause] [17:49] Hello. Yeah. How did you get uh to the specific rewards? Um you know [17:55] typically you could treat them like hyperparameters and do like a hyperparameter [17:59] search and that would be kind of expensive just to exactly. So in this uh in this study what we [18:05] have seen is when we have the knive implementation of a retrieval agent. So we do have a certain set [18:11] of theme people who were trying to u you know see why this question is better why this question is [18:18] wrong. So we create a metrics that's a kind of a weak supervisor metric is it like uh are the [18:24] retrievalss in the top K or not? Is it relevant or not? So once we see all those scoring coming in [18:31] we use those scoring as a rubric to uh like further tune the rewriter so that we are able [18:37] to retrieve those agents. So that's a discovery. What we have done is the rated from the uh subject [18:44] matter experts are used as the part of like it should be a part of training dynamics. [18:53] Um with uh with multi-step reasoning um were you doing much multi-step reasoning with that? [19:00] Sorry I'm not able. Multi-step reasoning. Yeah. Mhm. So in your multi-step reasoning, [19:05] say that uh the first step is correct, the second step is correct, but the third step is wrong. [19:12] Um and then you do that over the course of I [snorts] don't know three or four iterations. [19:17] How do you how do you grade those uh when you are inducting your your uh GRPO um in [19:27] in essentially telling it that it's wrong without saying you know while still encouraging it and [19:32] telling that it telling the model that it was good on the first two steps. So uh if [19:36] I understood your question correctly is uh when you are doing um alignment of a agent which is [19:42] trying to do multi-step reasoning and you want to validate if it the third step is correct or wrong. [19:49] Uh and how would you validate in your GRP training is is that right understanding? [19:56] Well, um, so the final result is either correct or incorrect, right? But you could have two two [20:03] models and and the first, second, and third steps to get to the result could be correct on one, [20:10] but the first and second step may be correct on one, but the the final step is incorrect, giving [20:16] second one the wrong answer. So, so when you have situations like that where it's mostly [20:21] right going through, then the final result is wrong. Yeah. Just because of this one small piece, [20:28] how do you how do you go about grading that and also rewarding it through those first few steps? [20:35] Yeah. So, it's a it's a very good question. So what we have seen in my experience when we [20:40] are trying to train domain specific models and that has a reasoning capabilities of going uh [20:46] multi-steps. So using this dense kind of mechanism where you can have multiple metrics to assign a [20:53] weight. So you can think of like each step has a different weightage how important that step [20:59] is because let's say if your step one is correct, step two is wrong. So most uh probably your step [21:05] three would be also wrong because that's the model trying to generate the next token from there. [21:10] So uh as soon as we see that we give them a uh weightage and a reward in that sense that the [21:17] advantage that we are calculating in the gRP step is more negative on that side and it's not able to [21:23] you know uh uh impact our policy model. It's not regarded when it's getting trained during the [21:30] the policy optimization. So you're able to reward on a first step basis. Exactly. [21:37] Yes. Yeah. That's where the dense kind of when you say different metrics are coming in, you can [21:42] rate each step in that reasoning and you can put a weight how how much weightage you want to put it. [21:49] Yeah. Thanks. This will be our last question. I hope it's a good one. Uh you had mentioned these [21:55] subject matter experts that would come and weigh in and and try to optimize the responses from [21:59] these agents. What kind of like time commitment was that for them? Was it a once- off? You get [22:04] asked all the time like how much time do you need from us? Yeah. Off the line so to speak. Uh [22:11] thank you for the question. uh I think uh these SME feedbacks are very expensive [22:16] because of the time commitment over here and given uh we are trying to use the gRPO method here [22:23] and why GRPO is uh we are using here GRPO because we know the better the rewards GRPO [22:30] would tend to do much better. So if you are investing that time uh using theme feedback [22:36] what we try to do is we get get a very smaller subset you know very small sample of our data [22:43] basically we want to cover the whole universe but keep it as small as possible. [22:47] So depending on the domain and the specificity specificity of the model that you're training you [22:53] try to see how much data points you need from the subject matter experts. So it depends on [22:59] the use case how much time they want to spend but uh it's more of like can we use these smaller and [23:05] expensive set to uh you know scale these training paradigms basically. Thanks. Thanks everyone.