Same Weights, 5× the Score: What Actually Makes Agentic RL Work

Xiaol.x

AI-assisted summary of Same Weights, 5× the Score: What Actually Makes Agentic RL Work — original source. Check the original for full context.

Source published: 2026-09-20

Key ideas

A 20-billion-parameter checkpoint evaluated across eight search benchmarks jumped from 0.109 to 0.584 without changing a single weight, demonstrating that agentic RL performance is fundamentally governed by external scaffolding, harness state management, and rollout variance rather than the core algorithm. (0:00)
Off-the-shelf math process reward models collapse to between 19.23% and 21.8% accuracy on agentic data analysis tasks—performing worse than basic majority voting (24.36%) because they punish exploratory tool actions and miss silent code errors. (12:35)
Standard GRPO frequently encounters 'signal starvation' in multi-step agent environments when all sampled rollouts in a batch fail, producing identical zero rewards, zero baselines, zero advantages, and zero gradient updates. (1:30)
An audit of 519 post-June 2026 papers revealed a critical state-confounding flaw: because agents steer their own environments, benchmark gains conflate policy action quality with arriving at easier intermediate states, rendering current reported RL scores upper bounds. (15:54)

Detailed notes

## Background Published by Xiaol.x on YouTube on September 20, 2026, this video investigates why a single 20-billion-parameter model checkpoint can score 0.109 under one setup and 0.584 under another across eight search benchmarks without changing a single weight. Drawing from an audit of eleven agentic reinforcement learning papers from late 2025 through June 2026 and a follow-up harvest of 519 papers through September, the presentation argues that agentic RL is primarily an environment, harness, and signal problem rather than an algorithmic one.

## TL;DR The presentation demonstrates that training failures in agentic reinforcement learning are driven by signal starvation, credit assignment breakdowns, and context window saturation rather than flawed policy gradient algorithms. By offloading mechanical bookkeeping from the policy into externalized harnesses (such as in Harness-1) and localizing execution environments, modest models can dramatically outperform larger baselines. It details practical engineering requirements like sequence-level clipping, variance-based prompt filtering, and rejecting off-the-shelf math PRMs in favor of environment-probing verifiers. Finally, an empirical audit of 519 follow-up papers tests seven June predictions, uncovering critical caveats regarding benchmark reproducibility and the confounding effect of favorable intermediate state distributions.

## Atmosphere & style The presentation carries the brisk, incisive tone of a seasoned systems engineer conducting a forensic code audit rather than an academic hype reel. It feels methodical, intellectually unsparing, and grounded in forensic skepticism, repeatedly calling out published papers for missing random seeds, omitted error bars, and benchmark conflations. Listening feels like being let into a frank, post-mortem debrief where neat theoretical assumptions are stripped away in favor of hard infrastructure economics, operational tradeoffs, and checkable empirical ledgers.

## Bulletin - [00:00](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=0s) A 20-billion-parameter checkpoint evaluated across eight search benchmarks jumped from 0.109 to 0.584 without changing a single weight, demonstrating that agentic RL performance is fundamentally governed by external scaffolding, harness state management, and rollout variance rather than the core algorithm. - [12:35](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=755s) Off-the-shelf math process reward models collapse to between 19.23% and 21.8% accuracy on agentic data analysis tasks—performing worse than basic majority voting (24.36%) because they punish exploratory tool actions and miss silent code errors. - [01:30](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=90s) Standard GRPO frequently encounters 'signal starvation' in multi-step agent environments when all sampled rollouts in a batch fail, producing identical zero rewards, zero baselines, zero advantages, and zero gradient updates. - [15:54](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=954s) An audit of 519 post-June 2026 papers revealed a critical state-confounding flaw: because agents steer their own environments, benchmark gains conflate policy action quality with arriving at easier intermediate states, rendering current reported RL scores upper bounds.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=0s) Agentic RL in 2026 is fundamentally a signal problem rather than an algorithmic problem, relying on the environment, harness, and rollout variance to deliver gradients. - [01:30](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=90s) Group Relative Policy Optimization (GRPO) suffers from 'signal starvation' whenever all rollouts in a cohort fail, collapsing advantages and gradients to zero. - [02:46](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=166s) Traditional RL assumptions break in agent environments because linear text transcripts fail to reflect true underlying environment state, often making it impossible to determine even the sign of an action's credit. - [04:26](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=266s) ARLArena and SAMPO show that stabilizing agent policy gradients requires sequence-level clipping, behavior cloning warm-starts, format penalties, and low-variance KL penalties; tolerant token-level clipping caused performance to collapse by 96.5%. - [04:26](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=266s) Ten of the eleven reviewed papers published between late 2025 and mid-2026 omit random seeds and variance bounds entirely, leaving reported baseline differences largely unanchored. - [06:36](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=396s) G2PO leverages overlapping agent trajectories to compute step-level baseline values over shared state-graph nodes, boosting task success by 22.2 percentage points over standard GRPO. - [08:12](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=492s) The agent harness operates as a compute allocation mechanism; offloading candidate tracking and verification ledgers from the model's context into the harness accounted for boosting retrieval recall from 0.109 to 0.584. - [08:12](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=492s) Ablations in Harness-1 indicate that bundled harnesses can be noisy; three of its eight internal mechanisms actually degraded retrieval metrics when enabled in isolation. - [10:31](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=631s) Replacing commercial search APIs with a local 32-million-page search index cut operational tool-call costs from up to $243,000 down to $220 while accelerating query latency by 10× to 46×. - [10:31](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=631s) Synthetic dual-agent curricula like Tool-R0 can self-evolve task success from 24.85% to 47.84% without human data, but weight-sharing between generator and solver causes a severe 17.42-point drop. - [12:35](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=755s) DataPRM demonstrates that accurate step-level reward modeling requires direct programmatic probing into sandbox memory, as text-only evaluators cannot catch silent runtime logic errors. - [14:05](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=845s) Prompt filtering should be guided by rollout reward variance or mutual information diagnostics rather than policy entropy, which correlates neutrally to negatively with downstream performance. - [15:54](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=954s) A review of 519 subsequent papers through September 2026 confirms that harness and environment engineering have outpaced algorithm tweaks as the primary driver of reported gains. - [15:54](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=954s) Evaluating agent policies without matched intermediate state handoffs introduces severe confounding, as policies are credited for superior decision-making when they may simply be operating in more forgiving environment states.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=0s) The Core Thesis: A Signal Problem, Not an Algorithm Problem

The discussion opens with an arresting paradox: take a single 20-billion-parameter model checkpoint and evaluate it across eight search benchmarks. Under one setup, it posts a dismal score of 0.109; under another, it surges to 0.584. The model weights are identical in both runs. The entire five-fold performance gap stems purely from the scaffolding, training methodology, and harness surrounding the weights. This discrepancy serves as the framing question for an investigation into eleven papers on agentic reinforcement learning (RL): if an agent has a verifier to supply a reward and an interactive environment to produce rollouts, why does training remain so notoriously difficult?

The answer lies in recognizing that "agentic reinforcement learning in 2026 is a signal problem, not an algorithm problem." The algorithm and the final scalar rewards are not where the breakdown occurs. Instead, reward signals must travel through an environment constructed by humans, pass through a harness that maintains intermediate state, and emerge from a batch of rollouts that must exhibit variance before any policy gradient can form. When that informational conduit is repaired, a modest 4-billion-parameter policy can outperform an extensively prompted frontier model.

[01:30](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=90s) Signal Starvation in Group Relative Policy Optimization

To understand why sparse rewards stall agent training, the speakers examine the default workhorse of modern reasoning and agent RL: Group Relative Policy Optimization (GRPO). Under GRPO, the training engine samples a cohort of rollouts for a single task and scores each trajectory against the cohort's empirical mean. Using the group average eliminates the computational overhead of maintaining a separate critic model.

```mermaid flowchart TD accTitle: Signal Starvation in GRPO accDescr: Diagram showing how identical failed rollouts produce zero advantage and signal starvation.

subgraph RolloutBatch ["Sampled Group Rollouts (e.g., N=8)"] R1["Rollout 1: Fail (r=0)"] R2["Rollout 2: Fail (r=0)"] R3["Rollout...: Fail (r=0)"] R8["Rollout 8: Fail (r=0)"] end

subgraph AdvantageCalc ["Group Baseline Computation"] Mean["Group Mean: 0.0"] Adv["Advantage: r - Mean = 0.0"] end

subgraph Outcome ["Policy Update"] ZeroGrad["Zero Gradient: Signal Starvation"] end

RolloutBatch --> Mean Mean --> Adv Adv --> ZeroGrad ```

However, when applied to long-horizon agentic workflows, this setup frequently collapses into complete informational bankruptcy. If eight out of eight rollouts fail on a challenging task, every single reward is zero, the group mean is zero, and subtracting the baseline yields an advantage of exactly zero across the board:

- All advantages vanish, producing no gradient and conveying zero information to the policy. - A September 2026 paper formalizes this failure mode under the term **"signal starvation."** - Consequently, the fundamental bottleneck in agentic RL is not raw reward availability, but manufactured **"disagreement"** (variance across rollouts).

[02:46](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=166s) Credit Assignment and the Transcript Fallacy

Even when an agent eventually stumbles upon a non-zero reward, assigning credit across an extended trajectory remains a structural challenge. An agentic run may span dozens of sequential actions—issuing web queries, executing bash commands, invoking tools, and writing to memory—culminating in a single scalar evaluation at the end. To dissect how credit assignment operates in practice, the speakers review a comprehensive meta-survey covering 69 papers (auditing 56 core methods across a 42-page full-text review).

Remarkably, the survey authors audited their own evaluation reliability. Two independent researchers blind-coded 252 diagnostic cells and achieved consensus on 223 of them (an 88.5% raw agreement rate, with diagnostic-specific Cohen's kappa values ranging from 0.543 to 0.99). The low end of that kappa span denotes only moderate agreement, but the transparent rigor highlights six distinct ways agentic environments break traditional RL assumptions: 1. Environments are open, non-closed systems that allow re-entry and side effects. 2. The linear textual history does not capture the true underlying environment state. 3. Arbitrary mid-trajectory state rollbacks and replays are generally impossible. 4. Action units are non-fungible (a raw token, an API invocation, and a memory write represent fundamentally distinct semantic operations). 5. There is no low-cost oracle available to verify whether an intermediate action (such as step seven) was beneficial. 6. The most striking finding: inspecting a text-only transcript can leave even the **sign** of an action's credit unidentified, making it impossible to determine whether an intermediate step helped or harmed the overall trajectory.

[04:26](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=266s) The Four Practical Knobs: Lessons from ARLArena and SAMPO

Turning to operational engineering choices, the speakers examine the ARLArena framework, which systematically decomposes policy gradient training into four orthogonal dimensions and evaluates them in isolation: clipping mechanisms, advantage design, dynamic prompt filtering, and loss aggregation.

```mermaid flowchart LR accTitle: SAMPO Four Optimization Dimensions accDescr: The four orthogonal dimensions of agentic policy gradient explored by ARLArena.

subgraph Prereqs ["Mandatory Stabilization"] BC["Behavior Cloning on Good Rollouts"] FP["Format Penalty"] KL["Low-Variance KL Term"] end

subgraph Knobs ["The 4 Knobs (SAMPO)"] K1["1. Sequence-Level Clipping (Avoid Token-Level)"] K2["2. Environment-Informed Advantage"] K3["3. Variance-Based Filtering (Not Entropy)"] K4["4. Loss Aggregation Scheme"] end

Prereqs --> Knobs ```

Before tuning any parameters, baseline stabilization is mandatory. The authors found that baseline runs could not even produce measurable comparisons without first applying: - Supervised behavior cloning initialized on successful rollouts from the environment itself. - Strict format violation penalties to prevent syntactic degradation. - A low-variance Kullback–Leibler (KL) divergence penalty.

Once stabilized, clipping behavior emerged as the critical performance determinant. While tolerant token-level clipping caused catastrophic policy collapse (plummeting down 96.5% to a score of 0.08 on one checkpoint), sequence-level clipping maintained stability, climbing 113% to reach 5.05. Their synthesized combination, designated **SAMPO**, reached an overall score of 8.98. The third knob dictated that environmental structural context must be directly injected into the advantage calculation rather than relying strictly on the scalar trajectory endpoint.

A crucial experimental caveat is highlighted: despite focusing entirely on training stability, ARLArena reports zero seeds and zero error bars. In fact, ten out of the eleven papers reviewed omit seeds and variance bounds entirely, with only one single paper reporting standard deviations across three seeds.

[06:36](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=396s) Step-Level Value Attribution on Graph Topologies (G2PO)

Addressing the coarseness of single trajectory-level rewards naturally points toward step-level scoring. The speakers analyze **G2PO (Group-Graph Policy Optimization)** as an elegant realization of this idea. G2PO recognizes that when a batch of agent rollouts executes across the same task, the trajectories regularly intersect at identical intermediate states—such as visiting the identical webpage, viewing the same search snippets, or entering the same environment state via divergent paths.

Rather than treating trajectories as isolated parallel lines, G2PO aggregates identical observations into shared nodes within a unified state graph: - A state's baseline value is calculated as the group mean over all rollouts that intersected that node. - Individual actions along edges are evaluated via standardized one-step temporal difference (TD) errors. - G2PO yields a **22.2 percentage point gain** in task success over the standard GRPO baseline, while imposing only ~0.4% in additional training compute. - As the lone paper reporting rigorous experimental variance, G2PO posts **95.0% ± 0.8%** across three random seeds, handily outperforming a prompted ~400-billion-parameter baseline that scored 71.9%.

Despite this success, a later paper notes a persistent theoretical limitation: because step-level advantages in graph formulations still ultimately trace back to each trajectory's terminal outcome, intermediate actions trapped inside universally failing rollout batches still risk remaining undifferentiated.

[08:12](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=492s) Harness Engineering: State Externalization in Harness-1

Moving beyond reward and advantage formulations, the speakers highlight what they consider the core insight of the survey: examining how long-horizon search policies consume their limited context windows. In conventional agent architectures, the policy squanders its context acting as an administrative filing cabinet—tracking candidate documents, logging verified constraints, mapping evidence, and pruning stale tokens.

```mermaid flowchart TD accTitle: Harness-1 State Externalization Architecture accDescr: Separation of mechanical filing in the harness from semantic decision-making in the policy.

subgraph Harness ["Externalized State Harness"] Cands["Candidate Document Tracker"] EvGraph["Evidence Graph & Verification Ledger"] Dedup["De-duplicated Observation Cache"] Render["Render Structured Summary into Prompt"] Cands --> Render EvGraph --> Render Dedup --> Render end

subgraph Policy ["Language Model Policy"] Context["Rendered Context Window"] Decisions["Pure Semantic Agency: • Formulate Queries • Select Retained Docs • Decide Stopping Point"] Context --> Decisions end

Render --> Context Decisions -->|"Structured Tool Actions"| Harness ```

**Harness-1** remedies this context tax by externalizing mechanical bookkeeping into the environment scaffolding. The harness directly maintains the candidate pool, evidence graphs, claim-level verification ledgers, and deduplication buffers, rendering a concise summary back into the prompt. The language model is left solely with high-level semantic decisions: deciding what queries to execute, which evidence to accept, and when to conclude the search.

This state externalization explains the opening paradox, boosting curated retrieval recall from **0.109 to 0.584** on identical weights. The paper's ablation table reveals the exact distribution of gains: - **Harness only (untrained base model):** 0.19 - **RL training only (stripped naive harness):** 0.46 - **Both harness and training combined:** 0.56 (reaching 0.584 on expanded runs)

Training provides the larger individual boost, but neither component succeeds in isolation. Furthermore, when evaluating a strong frozen model, performance scales from 0.511 on a naive loop, to 0.807 on a production search harness, to 0.849 on Harness-1 without updating model parameters. This leads to the central realization: **"the harness is a compute allocation mechanism."**

However, the authors' ablation data contains an unexpected counter-result: three of the eight individual mechanisms inside Harness-1 actually degrade performance when enabled. For example, disabling observation deduplication actively improved both benchmark metrics, proving that while the harness succeeds as an integrated bundle, several individual components fail to justify their inclusion in isolation.

[10:31](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=631s) Environment Economics and Synthetic Task Generation

Externalizing state to the environment implies that engineering realistic interactive environments is the primary work of agent RL. The speakers examine the real-world operational costs of environment interactions during agent training. One search-agent study recorded 73.2 million tool calls across its training lifecycle: - Querying commercial search APIs would have cost between **$59,000 and $243,000**, with call latencies ranging from 1.5 to nearly 8 seconds. - Instead, the researchers constructed a local 32-million-page search index, eliminating API expenses ($0) and dropping query latency to 0.15 seconds (a 10× to 46× speedup) for an initial scraping and indexing cost of just **$220**.

Concurrently, generating diverse training tasks without expensive human annotation is increasingly handled by synthetic generation loops. The **Tool-R0** framework sets up a dual-agent self-evolution curriculum, pairing a task generator against an agent solver initialized from identical base weights: - Starting from zero human tasks, the self-evolving solver improved from 24.85% to 47.84% task success. - This automated curriculum outperformed baselines trained on up to 210,000 human-annotated trajectories, albeit with a narrow 1.78-point margin across a five-benchmark average, while losing outright on two of the five benchmarks. - Ablations show this loop is fragile: forcing the generator and solver to share model weights caused a severe 17.42-point drop, while removing an explicit difficulty reward penalized performance by an additional 4.3 points.

[12:22](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=742s) Off-the-Shelf Reward Models and Process Verification

The segment concludes by addressing whether practitioners can bypass custom environment rewards by simply deploying an off-the-shelf reasoning Process Reward Model (PRM) to score intermediate tool actions. The speakers note that evidence from **DataPRM** firmly refutes this idea, demonstrating that general process verifiers trained on math or deductive reasoning completely fail when tasked with evaluating step-level actions in complex agent environments.

[12:35](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=755s) Why Reusing Reasoning Reward Models Fails (and How Probing Fixes It)

Attempting to evaluate agentic data-analysis workflows with standard process-reward models trained purely on mathematics yields catastrophic results. When applied to multi-turn data analysis, every single one of these math-trained verifiers scores below free majority voting. The weakest models collapse to an accuracy of 19.23% compared to majority voting’s 24.36%, while even a 32-billion-parameter reasoning model only crawls up to 21.8%. Worse still, their accuracy degrades as more samples are generated: sampling more attempts simply causes the model to make more confidently wrong picks, rendering the grader "worse than useless."

```mermaid flowchart TD accTitle: Text-Only vs Environment Probing PRMs accDescr: Comparison between standard text-only process reward models that fail on silent errors and environment-probing reward models that inspect intermediate execution state.

subgraph TextOnly["Text-Only PRM (Fails)"] A["Agent Generates Code"] --> B["Code Runs (Exit Code 0)"] B --> C{"Evaluator Reads Text Only"} C -->|"Silent Logic Error"| D["False Positive Reward"] C -->|"Legitimate Trial & Error"| E["False Negative Penalty"] end

subgraph EnvProbe["Environment-Probing PRM (DataPRM)"] F["Agent Alters State"] --> G["Execute Probes into Sandbox"] G --> H["Inspect Memory & Tables"] H --> I["Accurate Step Reward"] end ```

This failure stems from two structural pathologies: 1. **Silent errors:** The agent’s code executes cleanly without throwing an exception, yet produces an incorrect numerical or analytical output. A text-only grader reading intermediate execution logs cannot inspect internal variable states or mutated data tables, missing the mistake completely. 2. **Punishing necessary exploration:** In data analysis and search, intermediate trial-and-error (such as testing an exploratory command or sanity-checking a slice of data) is an essential problem-solving tactic. Math-based verifiers misinterpret these exploratory iterations as blunders and penalize them accordingly.

The solution demonstrated by DataPRM is to build an environment-aware verifier capable of dispatching programmatic probes into the runtime sandbox to directly inspect intermediate execution states. In their ablation analysis, the authors candidly show that raw environment access does almost all the heavy lifting. Multi-turn interaction without direct environment inspection yields essentially no performance improvement, and elaborate reward-shaping functions provide the smallest marginal gain. However, this approach carries real trade-offs: the environment-probing grader consumes 2.57 times more tokens, and at higher sample counts, completely unfiltered data actually outperforms all three of the paper's proposed data-filtering heuristics.

---

[14:05](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=845s) The Field Guide to Agentic RL in Ten Rules

Distilling the engineering practices across eleven papers yields a ten-point tactical field guide for anyone initiating an agentic reinforcement learning project:

| Rule # | Principle | Key Mechanism & Evidence | |:--- |:--- |:--- | | **1** | **Stabilize before you compare** | Bootstrap with behavior cloning, enforce format penalties, and apply low-variance KL penalties before tuning policy objectives. | | **2** | **Sequence-level clipping** | Clip advantages strictly at the trajectory/sequence level rather than being tolerant at the individual token level. | | **3** | **Inject environment structure into advantage** | Calculate credit assignment based on transitions and state graphs rather than relying purely on scalar terminal outcomes. | | **4** | **Filter prompts by reward variance, not entropy** | Discard standard entropy metrics. A mutual information diagnostic correlates positively (+0.39) with downstream task performance, whereas reasoning entropy correlates neutrally to negatively (0.11 to -0.14). | | **5** | **Move mechanical state into the harness** | Offload memory buffers, scratchpads, and execution logs from policy prompt context into external environment wrappers. | | **6** | **Localize environments before scaling rollouts** | Remove remote API bottlenecks and distributed network latency before scaling training clusters. | | **7** | **Instrument the harness boundary** | Observe the boundary rather than inventing fragile proxy interfaces at the model level. (Warning: recipes do not generalize across harnesses; the identical technique produced +22.6 points on one harness and +0.6 on another). | | **8** | **Never reuse reasoning reward models** | Verifiers trained on static math reasoning fail on tool use, punish valid exploration, and underperform majority voting. | | **9** | **Synthesize and verify tasks with difficulty rewards** | Self-generate task distributions, verify programmatic solvability, and weight training toward higher task difficulty. | | **10** | **Report random seeds and error bounds** | Ten out of the eleven foundational papers omit seeds and confidence intervals, making published performance deltas provisional until replicated. |

---

[15:54](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=954s) Scoring Seven Predictions Against 519 September Papers

Evaluating seven forward-looking predictions made in June against an automated crawl of 519 agentic RL papers published across the subsequent eleven weeks (July through mid-September) reveals a rapidly maturing research ecosystem:

```mermaid flowchart LR accTitle: June Predictions vs September Outcomes accDescr: Seven predictions formulated in June 2026 evaluated against 519 papers harvested through September 2026.

subgraph Confirmed["Confirmed & Progressing"] P1["1. Harness & Env Focus (Confirmed + Caveat)"] P4["4. Reproducibility & Controls (Confirmed)"] P5["5. Self-Generated Tasks (Confirmed + Ceiling)"] P7["7. Post-Training Adoption (Confirmed in Enterprise)"] end

subgraph Nuanced["Mixed or Refined"] P2["2. Dense/Graph Credit (Mixed Debate)"] P3["3. Collapse Solutions (Partial / Unconverged)"] P6["6. State-Handoff Confounding (Refined / Upper Bound)"] end ```

- **Prediction 1: The harness and environment replace the algorithm as the primary source of gains.** *Result: Confirmed emphatically, with an effect-size caveat.* By September, harness papers jumped from one to over twenty, accompanied by a dozen dedicated environment suites. However, "attention is not effect size": the only paper reporting rigorous confidence intervals demonstrated harness guidance delivering a modest +7.17 point gain across 265 matched experimental cells. - **Prediction 2: Credit assignment shifts from scalar terminal rewards to process and graph-structured signals.** *Result: Mixed.* The community is split. While several papers advocate fine-grained process rewards, a compelling counter-camp argues that terminal outcome ceilings are merely artifacts of bad batch selection; under their diagnosis, training simply stalls because policies continue updating on tasks where all candidate rollouts already agree. - **Prediction 3: Reasoning and advantage collapse receive a definitive algorithmic fix.** *Result: Partial.* The field has formally categorized the failure modes (e.g., advantage collapse, stage-dependent teachers), but lacks a single convergent fix, and virtually no subsequent work adopted RAGEN-2's mutual information diagnostic. - **Prediction 4: Reproducibility and experimental rigor improve.** *Result: Confirmed.* Methodology shifted noticeably away from June’s un-seeded baselines. September papers introduced word-count-matched sham controls, bootstrap intervals, and 176-cell matched experiments that held warm-starts and update steps strictly constant to isolate individual variables. - **Prediction 5: Self-generated tasks replace static curated benchmarks.** *Result: Confirmed, but hits a ceiling.* Co-evolution algorithms tied directly to on-policy rollouts eventually exhaust their signal once the base model outgrows the generator's ability to create meaningfully hard, verifiable tasks. - **Prediction 6: Reinforcement learning gains hold against strong prompted baselines.** *Result: Refined (Uncovering the State-Confounding Problem).* Closed-loop agents interact iteratively with environments, which means they dictate their own subsequent context windows and environment states. Evaluating a supervised checkpoint against an RL checkpoint at task completion conflates *how well the policy made decisions* with *the favorable states it steered itself into*. Later papers resolved this by running explicit checkpoint handoffs—scoring agents strictly from identical intermediate states. Consequently, existing benchmark gains must be understood as upper bounds. - **Prediction 7: Agentic RL becomes routine enterprise post-training.** *Result: Confirmed for enterprise products; unverified for frontier labs.* Enterprise platforms have widely integrated Group Relative Policy Optimization (GRPO) and simulation-to-reward pipelines into routine deployment stacks. Whether frontier closed-source labs run this as default post-training remains unverified behind closed doors.

Crucially, the September literature introduced several critical research topics that were completely absent from the June papers: - **Attribution as measurement:** Accurately measuring what caused a success within multi-step rollouts. - **Honesty in agent self-reporting:** Addressing shortcut policies where agents hallucinate successful tool execution. - **Harness security:** Preventing policies from exploiting environment wrappers or finding out-of-band exploits. - **Multi-agent skepticism and uncertainty calibration:** Questioning multi-agent overhead and demanding rigorous uncertainty bounds.

---

[19:38](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=1178s) Conclusion: A Signal Problem and the Uncomfortable Open Question

The central question of modern agentic reinforcement learning is whether performance bottlenecks stem from the core optimization algorithm or from the quality of the incoming training signal. The evidence points squarely to a **signal problem**:

> "The reward is fine. It arrives through an environment you built, a harness that holds the state, and a group of rollouts that has to disagree for a gradient to exist. Every rule in the field guide is about that path."

This explains how an identical 20-billion-parameter model checkpoint can score 0.109 in one configuration and 0.584 in another with zero weight modifications. The scaffolding, the environment sandboxing, and the rollout variance are not incidental wrappers around a model; they represent the core of the engineering problem.

The unresolved challenge confronting the field centers on state-distribution confounding. Because multi-turn agents choose their own intermediate environments, standard performance numbers leave open how much of today's measured progress represents genuinely superior decision-making versus merely inheriting more forgiving starting states. Determining where action quality ends and state privilege begins remains the critical, checkable frontier for agentic RL.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=0s) agentic reinforcement learning in 2026 is a signal problem, not an algorithm problem.

> [01:30](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=90s) signal starvation

> [01:30](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=90s) disagreement

> [04:26](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=266s) SAMPO

> [08:12](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=492s) the harness is a compute allocation mechanism.

> [12:35](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=755s) worse than useless

> [15:54](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=954s) attention is not effect size

> [19:38](https://www.youtube.com/watch?v=JFfCpCQA0xM&t=1178s) The reward is fine. It arrives through an environment you built, a harness that holds the state, and a group of rollouts that has to disagree for a gradient to exist. Every rule in the field guide is about that path.

## Mentioned ARLArena, DataPRM, G2PO, GRPO, Harness-1, LiteResearcher, OpenClaw-RL, Polar, RAGEN-2, SAMPO, Tongyi DeepResearch, Tool-R0, arXiv

## Video description One 20-billion-parameter checkpoint. Eight search benchmarks. It scores 0.109 — and it scores 0.584. Same weights both times. Everything that changed was around the model.

That gap is what this episode is about. We read eleven papers on agentic reinforcement learning from their arXiv LaTeX sources — ten from February to June 2026, plus Tongyi DeepResearch from October 2025 — and came out with one claim: agentic RL in 2026 is a signal problem, not an algorithm problem. The reward is fine. It reaches the policy through an environment you built, a harness that holds the state, and a group of rollouts that has to disagree with itself before a gradient exists at all.

Then we did the part nobody does. The reading list stops in June; it is now September. So we harvested 519 agentic-RL papers from the eleven weeks after the list closes and scored seven predictions against them. Three confirmed, one with a caveat, one partial, one mixed, one refined. The one we got most interestingly wrong is the one worth watching for.

CHAPTERS 0:00 The same model scored 0.109 and 0.584. What changed? 1:30 If you have a verifier, why is the gradient ever zero? 2:46 When a reward does arrive, what is it rewarding? 4:26 So what do you actually turn? Which knobs matter? 6:36 Does finer-grained credit actually fix it? 8:12 What if the problem is where the state lives? 10:31 Where do the environments and the tasks come from? 12:22 Can you trust the thing that grades the steps? 14:05 So what is the field guide, in ten lines? 15:54 We would have predicted seven things in June. Which held? 19:38 What do we take away, and what stays open?

THE FIELD GUIDE, IN TEN LINES 1. Stabilize before you compare — behaviour cloning, format penalty, low-variance KL. 2. Clip at the sequence level, not tolerantly at the token level. 3. Put environment structure into the advantage, not just the final outcome. 4. Filter prompts by reward variance, not entropy. Entropy points the wrong way. 5. Move mechanical state out of the policy and into the harness. 6. Make the environment local before you scale rollouts. 7. Instrument the harness boundary rather than rewriting the harness. 8. Do not reuse a reasoning-domain reward model. 9. Synthesize your tasks, but verify them and reward difficulty. 10. Report seeds. Ten of these eleven papers do not.

THE ELEVEN PAPERS Polar: Agentic RL on Any Harness at Scale — arXiv 2605.24220 ARLArena / SAMPO: A Unified Framework for Stable Agentic RL — arXiv 2602.21534 LiteResearcher: A Scalable Agentic RL Training Framework — arXiv 2604.17931 RAGEN-2: Reasoning Collapse in Agentic RL — arXiv 2604.06268 From Reasoning to Agentic: Credit Assignment in RL for LLMs — arXiv 2604.09459 G2PO: Group-Graph Policy Optimization — arXiv 2606.22995 Tool-R0: Self-Evolving LLM Agents from Zero Data — arXiv 2602.21320 Tongyi DeepResearch Technical Report — arXiv 2510.24701 OpenClaw-RL: Train Any Agent Simply by Talking — arXiv 2603.10165 DataPRM: Process-Level Reward Modeling for Agentic Data Analysis — arXiv 2604.24198 Harness-1: RL for Search Agents with State-Externalizing Harnesses — arXiv 2606.02373

WHAT WE DO AND DON'T CLAIM Every number spoken is a row in a 61-row claims ledger read from the papers' own e-print sources. Ten of the eleven papers report no seeds and no error bars — G2PO alone gives three seeds with standard deviations, and we say so three times. Harness-1's headline metric is retrieval recall, not task success, and three of its eight harness mechanisms score better switched off. We never speak a number from OpenClaw-RL, because its GUI agent is evaluated on its own training set. Every July-to-September paper was verified to exist through the arXiv API and read at abstract level only — no full text — and the episode says so out loud. One claim is ours and not the papers': that every headline on this list mixes "the agent reached better states" with "the agent acted better", so all of them are upper bounds.

Also: the source reading list says 7 of 11 papers release code. We checked. It is 10 of 11 — three name their repository only inside the LaTeX.

MADE WITH Animated in Lean 4 with Leanim; charts hand-authored as SVG and redrawn from the papers' own tables; two MiniMax voices at natural speed, no time-stretching; hosts Mara and Theo are hand-drawn pixel art. Pacing and visual structure were reverse-engineered from a Welch Labs video (85 cuts, one register change every 25 seconds, the reveal at 48% of runtime).

Corrections live in the pinned comment. If you find an error, say so and we will pin it.

#AgenticRL #ReinforcementLearning #LLM #MachineLearning #AIResearch #RLHF #GRPO #AIAgents

[Source](https://www.youtube.com/watch?v=JFfCpCQA0xM)

中文简报

权重未变得分翻五倍：智能体强化学习真正起效的核心机制该报告指出，智能体强化学习（Agentic RL）中的训练失败主要源于信号匮乏、信用分配失效以及上下文窗口饱和，而非策略梯度算法本身存在缺陷。通过将机械性的状态记录工作从策略模型剥离至外部运行框架（如 Harness-1）并实现执行环境的本地化，较小规模的模型表现能显著超越更大的基线模型。报告详细阐述了序列级裁剪、基于方差的提示词过滤，以及弃用现成数学过程奖励模型（PRM）转而采用环境探测验证器等实用工程要求。最后，对2026年6月之后的519篇跟进论文进行的实证审查检验了先前的七项预测，揭示了基准可复现性以及有利中间状态分布所带来的混杂效应等关键局限。一个200亿参数的模型检查点在八项搜索基准测试中的得分从0.109跃升至0.584，期间未改动任何权重，这表明智能体强化学习的表现从根本上取决于外部脚手架、运行框架状态管理和 rollout 方差，而非核心算法本身。 (0:00) 现成的数学过程奖励模型在智能体数据分析任务中的准确率暴跌至19.23%到21.8%之间，表现甚至劣于基础的多数投票法（24.36%），原因是它们惩罚了探索性工具操作，且忽略了隐蔽的代码错误。 (12:35) 在多步智能体环境中，当一个批次内采样的所有 rollout 均告失败时，标准 GRPO 会频繁遭遇“信号匮乏”，导致奖励全为零、基线全为零、优势全为零且无任何梯度更新。 (1:30) 对2026年6月之后发表的519篇论文的审查揭示了一个关键的状态混杂缺陷：由于智能体会主导自身的环境状态，基准测试的成绩提升混淆了策略动作质量与进入更简单中间状态的影响，导致当前汇报的强化学习得分实质上成了性能上限。 (15:54)

Reading summary

Analyzing literature across eleven papers, the presentation argues that agentic reinforcement learning bottlenecks stem from the information conduit—including environment scaffolding and rollout variance—rather than the core optimization algorithms. Signal Starvation and the Credit Assignment Problem (1:30) Modern agent reinforcement learning frequently relies on Group Relative Policy Optimization (GRPO), which scores rollouts against cohort averages to eliminate separate critic networks. However, on long-horizon tasks, uniform failures across all sampled trajectories produce zero group variance and zero advantage, leading to a condition formalized as 'signal starvation.' Under this failure mode, the policy receives no gradient information, illustrating that rollout disagreement rather than raw reward availability is the primary training bottleneck. Assigning credit across extended multi-step trajectories presents further structural challenges. A meta-survey of 69 papers identified that agent environments break traditional RL assumptions because linear text transcripts fail to capture true underlying states, intermediate verification oracles do not exist, and arbitrary rollbacks are impossible. In many cases, examining text-only logs cannot even clarify the positive or negative sign of an intermediate action's contribution to final task success. Stabilization and Step-Level Graph Formulations (4:26) Operational frameworks like ARLArena demonstrate that policy gradient stability requires explicit prerequisites before parameter comparisons can occur: behavior cloning initialized on successful rollouts, format violation penalties, and a low-variance KL penalty. Once stabilized, training choices heavily impact stability; while tolerant token-level clipping collapsed performance by 96.5% to 0.08 on one checkpoint, sequence-level clipping improved it by 113% to 5.05. Their synthesized SAMPO recipe achieved a score of 8.98, though ten of the eleven surveyed papers omitted seeds and variance metrics. To move beyond coarse trajectory-level feedback, G2PO aggregates overlapping intermediate agent states into unified graph nodes. Baselines are computed from the average of trajectories intersecting each node, and edges are scored using standardized temporal difference errors. This graph-attribution formulation improved task success by 22.2 percentage points over standard GRPO with an estimated 0.4% training compute overhead, achieving 95.0% ± 0.8% across three random seeds. Harness Engineering and Environment Economics (8:12) A central finding of the review is that the environment harness functions as an external compute allocation mechanism. In conventional setups, agents squander context windows tracking documents, candidate pools, and execution logs. Systems such as Harness-1 externalize mechanical filing into the environment and feed condensed summaries back to the policy, freeing the language model for semantic choices. This externalization drove a 20-billion-parameter checkpoint from 0.109 to 0.584 on curated retrieval recall without altering model weights. However, ablations revealed trade-offs: disabling observation deduplication actually improved metrics, showing that not all bundle components help in isolation. Environment engineering also carries concrete economic constraints. For instance, executing 73.2 million tool calls across one search-agent study would have cost $59,000 to $243,000 using commercial APIs with latencies up to 8 seconds. Deploying a local 32-million-page search index cost $220 initially, dropped latency to 0.15 seconds, and reduced ongoing API fees to zero. Meanwhile, synthetic task generators like Tool-R0 improved solver success from 24.85% to 47.84%, but performance dropped sharply by 17.42 points when generator and solver shared weights. Failure of Static Reasoning Verifiers (12:22) Practitioners cannot substitute custom environment evaluators with off-the-shelf Process Reward Models (PRMs) trained on mathematics or formal deduction. Evidence from DataPRM indicates that math-trained verifiers underperform simple majority voting on multi-turn data analysis tasks, with weak models scoring 19.23% against majority voting's 24.36%. Furthermore, generating additional samples degraded their accuracy as models selected incorrect answers with higher confidence. This breakdown arises because text-only verifiers cannot detect silent logic errors that run without throwing exceptions, while simultaneously penalizing exploratory trial-and-error actions that are essential to data analysis. DataPRM addresses this by executing programmatic probes into the runtime sandbox to inspect modified variables and tables directly. While effective, sandbox probing requires 2.57 times more tokens, and raw unfiltered training data outperformed all proposed heuristic filters at higher sample scales. Evolution of Rigor and State Confounding (15:54) An evaluation of June predictions against 519 papers published through mid-September reveals a shifting research landscape. Attention to harnesses surged from one paper to more than twenty, and methodological rigor improved through sham controls, bootstrap intervals, and multi-cell matched experiments. However, task generation hit ceilings once base models outpaced generators, and the adoption of fine-grained graph credit remains contested by researchers arguing that outcome plateaus stem from poor batch selection rather than reward granularity. Importantly, the later literature uncovered the 'state-confounding problem.' Because closed-loop agents dictate their subsequent context windows and environment states, standard evaluations conflate policy decision-making with the favorable states agents steered into. Checkpoint handoff experiments that score agents strictly from shared intermediate states suggest that existing benchmark claims represent upper bounds rather than pure policy gains. The review concludes that agentic RL progress is limited by signal transmission—the environment design, state-bearing harness, and rollout disagreement—rather than policy gradient algorithms. The key open question remains state-distribution confounding, leaving unclear how much measured improvement reflects superior policy decisions versus privileged intermediate environment states.

Source quotes

[0:00] agentic reinforcement learning in 2026 is a signal problem, not an algorithm problem. [1:30] signal starvation [1:30] disagreement [4:26] SAMPO [8:12] the harness is a compute allocation mechanism. [12:35] worse than useless [15:54] attention is not effect size [19:38] The reward is fine. It arrives through an environment you built, a harness that holds the state, and a group of rollouts that has to disagree for a gradient to exist. Every rule in the field guide is about that path.

Original transcript

[0:00] Um Theo one model two numbers same 20 [0:04] billion parameter checkpoint same eight [0:07] benchmarks it scores not 109 and it [0:11] scores not.584 [0:14] >> same weights both times what changed was [0:17] everything around the weights the [0:18] training and the scaffolding the model [0:20] runs inside [0:21] >> a five-fold gap and none of it from the [0:26] weights [0:28] >> and that gap is the subject of this [0:30] episode. 11 papers on agentic [0:32] reinforcement learning all walk up to it [0:34] from different directions. [0:36] >> So here is the episode question. You [0:40] have a verifier, so you have a reward. [0:43] You have an environment, so you have [0:45] rollouts. Why is this still hard? [0:49] >> Because the reward was never the scarce [0:52] thing. [sighs] Agentic reinforcement [0:54] learning in 2026 is a signal problem, [0:57] not an algorithm problem. [0:59] >> Say that as the thesis and we spend the [1:03] episode earning it. [1:04] >> The reward is fine. It reaches the [1:07] policy through an environment you built, [1:09] a harness that holds the state and a [1:12] group of rollouts that must disagree [1:14] before a gradient exists at all. Fix [1:17] that path and a 4 billion parameter [1:19] model beats a prompted frontier model. [1:22] And in the last act whether the [1:26] predictions we would have made in June [1:28] survived the summer [1:30] start with the paradox because it is [1:34] genuinely strange sparse reward but it [1:38] is a real reward. Why does that not just [1:41] work? [1:42] >> Take the method almost everyone uses [1:45] group relative policy optimization. You [1:48] sample a group of rollouts on one task [1:50] and you score each one against the group [1:53] average. [1:54] >> The average is the baseline. So it is [1:56] cheap and you need no critic at all. [2:00] >> Right? Now put that on a long horizon [2:02] agent task where eight out of eight [2:03] rollouts fail. Every reward in the group [2:06] is identical. Every advantage is the [2:08] group mean subtracted from itself [2:10] >> which is zero. All of them. [2:14] >> All of them. The update carries no [2:16] information about this task at all. A [2:19] September paper gives this a name I [2:21] like. Signal starvation. [2:23] >> So the scarce resource is not reward. It [2:27] is disagreement. [2:28] >> That is the cleanest way to say it. And [2:30] once you see it that way, most of the [2:32] field guide turns out to be different [2:34] ways of manufacturing or protecting [2:37] disagreement. [2:38] Hold that thought because the next [2:42] question is what the reward is even [2:43] attached to when it does arrive. [2:46] A trajectory is dozens of steps, a [2:49] search, a tool call, a memory, right? [2:53] The reward is one number at the end. [2:56] Which part earned it? [2:57] >> That is credit assignment. And the list [3:00] has a survey of it. 69 papers, 56 of [3:03] them core methods with a 42p paper full [3:06] text audit. [3:07] >> And unusually for a survey, they [3:10] measured their own reliability. [3:13] >> They did. Two researchers blind coded [3:15] 252 diagnostic cells and agreed on 223 [3:20] 88 12% with per diagnostic kappa from [3:24] 0.543 to 0.99. [3:28] The low end of that range is only [3:32] moderate agreement, but the honesty is [3:35] the point. [3:36] >> Its contribution is naming six ways [3:39] agentic settings break credit [3:41] assignment. The environment is not a [3:43] closed system. You can re-enter the text [3:45] history is not the state. You cannot [3:47] usually replay from the middle [3:50] >> and [3:51] um the units are not comparable. A [3:55] token, a tool call, and a memory write [3:57] are different kinds of thing. [3:59] >> Plus, no cheap oracle for whether step [4:01] seven was good. But the sharpest result [4:03] is this. A textonly history can leave [4:06] even the sign of credit unidentified. [4:09] >> The sign, not the size. That is a much [4:13] stronger claim. [4:15] >> Not the size. From the transcript alone, [4:17] you sometimes cannot tell whether an [4:19] action helped or hurt. Every method that [4:22] scores steps from a transcript inherits [4:25] that problem. [4:26] >> Here is where I want the practical [4:29] answer. If I am training an agent on [4:32] Monday, what do I change? [4:34] >> One paper is built exactly for that. Arl [4:38] Arena decomposes policy gradient into [4:40] four orthogonal dimensions and sweeps [4:43] each one in isolation. Clipping, [4:45] advantage design, dynamic filtering, and [4:48] loss aggregation. Four knobs swept one [4:52] at a time. And what do they set each one [4:56] to? [4:57] >> First, stabilize before you compare [4:59] anything. Behavior clone from the [5:01] environment's own good rollouts. Add a [5:04] format penalty. Add a low variance KL [5:06] term. Their baseline is not measurable [5:08] until all three are in. [5:11] >> A boring answer and I suspect the most [5:13] important one here. [5:15] >> Then the finding that matters. Tolerant [5:17] clipping induces collapse. Sequence [5:19] level clipping is stable. On one [5:22] checkpoint, a method lands at 0.08, [5:25] down 96 12%. That is collapse, not [5:30] regression. [5:31] >> And at the good end of that spread, [5:36] >> sequence level clipping gets 5.05 up [5:40] 113%. Their combination which they call [5:43] sampo reaches 8.98. [5:46] Third knob put environment level [5:48] information into the advantage not just [5:51] the trajectory outcome. [5:53] >> And here is my caveat because it is a [5:57] real one. This is the paper about [5:59] stability and it reports no seeds and no [6:03] error bars anywhere we could find [6:06] >> which is true of 10 of the 11 papers on [6:09] our list. One paper reports three seeds [6:12] with standard deviations. One. [6:15] >> Um, so when I say a method collapses, I [6:19] mean in one run in their harness at [6:23] their hyperparameters, [6:25] their appendix shows a method flipping [6:27] sign on one clipping change. [6:30] >> Fair. Take the four knobs as a map of [6:33] where the sensitivity lives, not as a [6:35] leaderboard. [6:37] If [6:38] one number at the end is too coarse, the [6:43] obvious move is to score every step. [6:46] Does that work? [6:47] >> Partly, and the best version in the list [6:50] is clever. G2PO notices that when you [6:53] sample a group of rollouts, they visit [6:55] many of the same observations. [6:58] >> The same room, [7:00] the same search result reached by [7:02] different paths. So instead of isolated [7:05] lines, it merges identical observations [7:08] into shared nodes of one state graph. A [7:11] state's value becomes a group mean over [7:13] everything that passed through it. [7:15] >> And how does a single action get scored [7:18] in that graph? [7:19] >> A one-step temporal difference error [7:21] over every edge standardized globally. [7:24] That is 22.2 points of success over the [7:27] group relative baseline for about 0.4% [7:30] extra training time. And this is the one [7:34] paper with seeds. [7:36] >> It is 95.0 plus or minus 0.8 over three [7:41] seeds against a nearly 400 billion [7:44] parameter model prompted at 71.9 [7:48] >> worth a second. A 1.5 billion parameter [7:51] model trained beats a model 200 times [7:54] its size prompted. [7:56] >> That is the strongest thing agentic [7:58] reinforcement learning has going for it. [8:00] Though a September paper argues step [8:02] advantages like these still key off each [8:05] trajectories's own final outcome. So [8:08] actions inside failed trajectories stay [8:10] undifferentiated. [8:12] >> We have spent five chapters on reward [8:14] and advantage. You keep hinting that is [8:17] the wrong end of the telescope. [8:19] >> I think it is. Here is the reveal and it [8:22] is the paper behind the two numbers we [8:24] opened with. Ask what a long horizon [8:27] search policy spends its context on. [8:30] >> Tell me what it is actually spending the [8:34] context on. [8:36] >> Bookkeeping. Which documents are still [8:38] candidates? Which support which [8:40] constraint? What has been verified? What [8:42] to evict when the context fills. None of [8:45] it is judgment. All of it is filing. So [8:48] harness one moves the filing out of the [8:50] policy into the environment. The harness [8:53] keeps a candidate set, importance tag [8:55] curation, an evidence graph, a claim [8:58] level verification ledger, and [8:59] dduplicated observations and renders [9:02] them back into the context. The policy [9:04] keeps every semantic choice, what to [9:07] search, what to keep, what to check, [9:09] when to stop. It just stops being the [9:11] filing cabinet. Same base and curated [9:14] recall goes from not 109 to 0.584. [9:19] >> Now split that apart. It could be the [9:22] training or the harness [9:25] >> both ways and it is the most useful [9:27] table on our list. Harness only no [9:30] training not.19 [9:33] training only harness stripped not 46 [9:37] both not.56 [9:40] >> so neither alone gets close and the [9:43] training is the bigger single term [9:45] >> bigger yes they also fix the model and [9:48] vary only the harness a strong untrained [9:51] model goes from 0.511 on a naive loop to [9:54] 0.807 807 on a production harness to [9:58] 0.849 on theirs [10:00] >> with no training at all. Their phrase, [10:05] the harness is a compute allocation [10:08] mechanism. [10:10] >> The sentence of the whole reading list, [10:12] but the caveat is sharp. Three of their [10:14] eight harness mechanisms score better [10:17] switched off. Removing content [10:18] dduplication improves both metrics [10:22] >> better with the piece removed. So the [10:25] harness pays as a bundle while most of [10:27] its parts cannot be shown to pay alone. [10:31] If the environment is doing that much [10:33] work, then building environments is the [10:35] job. What does that cost? [10:38] >> There is a beautiful number here. One [10:40] paper trains a search agent and counts [10:42] every tool call in training. 73.2 [10:46] million of them. [10:47] >> And at commercial search interface [10:50] prices, what does that come to? between [10:54] 59,000 and $243,000 [10:57] depending on the provider at one and a [10:59] half to nearly 8 seconds of latency per [11:02] call. So what did they do instead of [11:04] paying that? [11:06] >> Built the world locally. A 32 million [11:09] page index they serve themselves. Same [11:11] calls, $0 and not.15 seconds. 10 to 46 [11:17] times faster for $220 of crawling. $200 [11:22] upfront instead of a quarter of a [11:23] million and the tasks. Somebody still [11:26] writes the questions. [11:28] >> Increasingly, nobody does. One paper [11:31] co-rains two copies of the same base [11:33] model. A generator inventing tool use [11:36] tasks and a solver answering them with [11:38] zero human authored tasks and takes the [11:41] solver from 24.85 [11:43] to 47.84. [11:46] which beats models trained on how much [11:49] human data [11:50] >> up to 210,000 annotated examples though [11:53] the margin over the best of them is 1.78 [11:56] points on a five benchmark average and [11:59] it loses 205 outright [12:01] >> so I would not call that a clean win the [12:06] ablations say [12:08] >> that the loop has loadbearing parts give [12:10] the generator and solver shared weights [12:12] and it drops 17.42 42 points. Drop the [12:16] explicit difficulty reward and it falls [12:18] another 4.3. Self-generation is not [12:21] free. [12:23] >> If we are scoring steps now, we need a [12:25] greater. Can I take a reward model off [12:28] the shelf? [12:29] >> No. And the evidence is blunt. Data PRM [12:32] tests general process reward models. The [12:35] ones trained on mathematics on data [12:37] analysis agents. Every one of them [12:39] scores below majority voting. below free [12:42] majority voting. That is worse than [12:46] useless. [12:47] >> The weakest sit at 19.23 against [12:50] majority votings 24.36 [12:53] and the 32 billion parameter 1 only [12:55] reaches 21.8. Several get worse as you [12:59] sample more. So more attempts means more [13:01] confidently wrong picks. [13:03] >> Why do they fail in that particular way? [13:08] >> Two reasons. Silent errors. The code [13:11] runs clean, no exception. And the answer [13:14] is wrong. A text only graater cannot see [13:17] that. And they punish exploration, [13:19] scoring necessary trial and error as [13:22] failure. [13:23] >> So the fix has to be a a graater that [13:27] can reach into the environment itself. [13:30] >> Their grater executes probes to inspect [13:33] the intermediate state and their [13:35] ablation is honest about what carries [13:37] the weight. Environment access is the [13:40] big term. Multi-turn interaction without [13:42] it buys essentially nothing. And their [13:45] reward shape contribution is the [13:47] smallest piece. [13:48] >> I like that they publish that. Um, [13:51] anything else that cuts against them? [13:53] >> Their filtering ablation shows [13:55] unfiltered data winning at the higher [13:57] sample counts against all three of their [13:59] own filtering strategies. And the [14:01] greater costs 2.57 times the tokens. [14:06] >> Put it on the board. If someone is [14:08] starting an agentic reinforcement [14:10] learning project this week, what is the [14:12] list? [14:13] >> One, stabilize before you compare. [14:16] Behavior cloning format penalty low [14:19] variance KL. Two, clip at the sequence [14:22] level, not tolerantly at the token [14:25] level. Three, put environment structure [14:27] into the advantage. Four, and this is my [14:31] favorite because it overturns a habit, [14:33] filter prompts by reward variance, not [14:36] by entropy. [14:37] >> Because entropy is the wrong instrument. [14:40] In the collapse paper, a mutual [14:42] information diagnostic correlates [14:43] positive 0.39 with final performance, [14:46] while reasoning entropy correlates 0.11 [14:49] to -0.14. [14:51] >> The standard diagnostic points the wrong [14:54] way. Keep going. [14:56] >> Five. move mechanical state out of the [14:59] policy into the harness. Six, make the [15:03] environment local before you scale [15:05] rollouts. Seven, instrument the harness [15:08] boundary rather than rewriting it with a [15:10] proxy at the model's own interface. [15:14] >> And that one has a warning attached, [15:17] doesn't it? [15:18] >> A useful one. The same recipe on four [15:20] different coding harnesses gives plus [15:23] 22.6 points on one and plus not.6 on [15:26] another. Generality claims should be [15:28] read as harnesssp specific. [15:31] >> Eight, do not reuse a reasoning domain [15:34] reward model. [15:36] Nine, synthesize your tasks, but verify [15:40] them and reward difficulty. [15:43] >> And 10, report seeds. 10 of our 11 [15:46] papers do not. That is not a footnote. [15:49] It is the reason every number in this [15:51] episode should be heard as provisional. [15:54] >> This is the part I wanted to do. The [15:56] list stops in June. It is now [15:58] midepptember. We harvested 519 agentic [16:02] reinforcement learning papers from the [16:05] 11 weeks after. Let us score ourselves. [16:09] >> One, the environment and the harness, [16:11] not the objective, become the main [16:13] source of gains and the main object of [16:15] study. Confirmed emphatically. In June, [16:19] one paper on our list studies the [16:21] harness itself. By September, more than [16:24] 20 plus a dozen on environments. [16:27] >> One caveat, attention is not effect [16:29] size. The one September number with an [16:31] interval says harness guidance is worth [16:34] 7.1 7 points over 265 match cells. [16:38] Single digits two credit assignment [16:42] moves toward process and graph [16:44] structured signals. That one splits [16:48] >> mixed and the best argument in the field [16:50] right now. Several September papers push [16:53] denser credit, but one argues the [16:55] outcome only ceiling is an artifact of [16:58] practice and that what you need is to [17:01] stop updating on tasks whose rollouts [17:03] already agree. [17:05] >> Our collapse papers diagnosis from the [17:08] opposite side and three that collapse [17:11] gets a real fix [17:12] >> partial. It is named precisely now [17:15] advantage collapse stage dependent [17:18] teachers but there is no convergent fix [17:20] and nobody adopted the mutual [17:22] information diagnostic [17:25] four reproducibility [17:27] confirmed and the good news of the check [17:30] production post training stacks shipped [17:32] but more importantly the methodology [17:34] improved a word count matched sham [17:36] control bootstrap intervals 176 matched [17:40] settings one paper holding the warm [17:42] start and the update count fix to [17:44] isolate a single variable [17:46] >> against 10 of 11 with no error bars in [17:49] June that is a field learning controls [17:53] >> five self-generated tasks replace [17:56] curated sets confirmed with a ceiling [17:59] the papers name themselves since [18:01] co-evolution tied to on policy rollouts [18:03] stops giving signal as the model [18:05] strengthens [18:06] >> and six that gains survive a strong [18:10] prompted baseline I thought that one was [18:13] safe. [18:14] >> Nobody refuted it. Something better [18:16] happened. A closed loop agent writes its [18:18] own later inputs. So, a supervised [18:20] checkpoint and a trained one are scored [18:23] from different states. End of task [18:25] success mixes where the agent arrived [18:27] with what it did there. [18:29] >> And nothing on our list separates those. [18:32] We flagged it ourselves from the surveys [18:34] unidentified sign result. then found a [18:37] September paper making the argument [18:39] properly, fixing it with checkpoint [18:41] handoffs that compare only on states [18:43] both policies reach. Our prediction [18:45] stands, our evidence is now an upper [18:48] bound. And seven, a gentic reinforcement [18:51] learning becomes ordinary post-training [18:54] confirmed for products with an [18:55] enterprise model post-trained by group [18:57] relative policy optimization and a [19:00] simulation toreward pipeline. For the [19:02] frontier labs, we found nothing either [19:04] way. So that half is unverified, not [19:07] confirmed. [19:09] >> Three confirmed, one with a caveat, one [19:12] partial, one mixed, one refined, plus [19:14] six themes June had no entry for at all. [19:18] Attribution is measurement, whether an [19:20] agent's own report is honest, shortcut [19:22] policies, harness security, multi-agent [19:26] skepticism, [19:28] and uncertainty reporting itself. A [19:32] reading list is a snapshot of what a [19:34] field is proud of, not of what it is [19:36] about to start worrying about. [19:39] >> Then give me the one sentence answer to [19:41] the episode question. Algorithm problem [19:44] or signal problem? [19:46] >> Signal problem. The reward is fine. It [19:49] arrives through an environment you [19:50] built, a harness that holds the state, [19:53] and a group of rollouts that has to [19:54] disagree for a gradient to exist. Every [19:57] rule in the field guide is about that [19:58] path. My takeaway is the two numbers we [20:02] opened with not 109 and 0.584 [20:07] same weights. The wrapper is not a [20:10] detail around the model. It is most of [20:12] the engineering. [20:14] >> The open question is the one September [20:16] handed us and it is uncomfortable. If [20:18] every number here was measured from [20:20] states the policy chose for itself, we [20:22] do not yet know how much of the progress [20:24] is better decisions and how much is [20:26] better starting positions. a good place [20:29] to stop because it is checkable. [20:33] Um, that was loop state. Papers, claims [20:37] ledger and harvest are in the show [20:39] notes. [20:40] >> 11 papers read from source, 519 checked [20:43] for the follow-up, and 10 of the 11 [20:46] still owe us error bars.