SoL-Pi Harness Cuts AI Agent Costs 33% in EdgeBench Tests

Xiaol.x

Filtering automated research proposals down to four context and execution mechanisms lowers AI agent operating costs by one-third while retaining roughly 94% of baseline capability, though compounding recursive self-improvement remains unproven.

AI-assisted summary of SoL-Pi: Scaling Auto-Research Loops for Efficient Agent Harnesses — original source. Check the original for full context.

Source published: 2026-09-11

Key ideas

SoL-Pi filters 152 automated research proposals down to four validated harness mechanisms that optimize agent efficiency without altering underlying model weights. (1:45)
On 51 held-out EdgeBench tasks, SoL-Pi cuts execution costs by approximately one-third and token consumption by 45% to 49% while retaining roughly 94% of native Pi's capability score. (8:10)
The four surviving mechanisms—Action Fusion, ObservationPack, Evidence-Preserving Reducer, and Online Context Compact—mitigate redundant API round-trips, repetitive log replaying, and context debt. (4:13)
True compounding, runaway recursive self-improvement remains an unverified hypothesis; SoL-Pi relies on a hybrid research workflow guided by human-defined constraints. (11:47)

Detailed notes

## Background Published on YouTube by Xiaol.x on September 11, 2026, this two-host explainer examines "SoL-Pi: Scaling Auto-Research Loops for Efficient Agent Harnesses." According to the video description, the program tracks how 152 proposed research directions were narrowed down to four core mechanisms—Action Fusion, ObservationPack, the Evidence-Preserving Reducer, and Online Context Compact—evaluating whether an agent can lower its operational costs while retaining roughly 94% of its benchmark capabilities.

## TL;DR The episode analyzes SoL-Pi, an approach that optimizes autonomous agent harnesses—prompts, tools, and memory management—without altering model weights to curb runaway token waste. By filtering 152 candidate hypotheses down to four surviving mechanisms, the harness introduces batched command execution, disk-backed observation paging, receipt-verified log reduction, and economically calculated context compaction. Evaluated across 51 held-out EdgeBench tasks, SoL-Pi achieves roughly a one-third reduction in cost and a 45% to 49% drop in total token consumption while preserving approximately 94% of the native Pi baseline's accuracy. Finally, the hosts draw a strict line between these validated efficiency gains and speculative claims, clarifying that a fully compounding, autonomous self-improvement loop remains an unproven hypothesis.

## Atmosphere & style The presentation carries an analytical, disciplined, and engineering-minded tone that deliberately counters runaway AI self-improvement hype. Driven by an explanatory two-host dialogue supported by technical animations and concrete debugging walkthroughs, the atmosphere feels like an objective technical audit. Its narrative moves from an enticing premise about slashing research bills to granular architectural scrutiny, concluding with a grounded reality check on the boundaries of recursive agent discovery.

## Bulletin - [01:45](https://www.youtube.com/watch?v=x9r85EtVFM8&t=105s) SoL-Pi filters 152 automated research proposals down to four validated harness mechanisms that optimize agent efficiency without altering underlying model weights. - [08:10](https://www.youtube.com/watch?v=x9r85EtVFM8&t=490s) On 51 held-out EdgeBench tasks, SoL-Pi cuts execution costs by approximately one-third and token consumption by 45% to 49% while retaining roughly 94% of native Pi's capability score. - [04:13](https://www.youtube.com/watch?v=x9r85EtVFM8&t=253s) The four surviving mechanisms—Action Fusion, ObservationPack, Evidence-Preserving Reducer, and Online Context Compact—mitigate redundant API round-trips, repetitive log replaying, and context debt. - [11:47](https://www.youtube.com/watch?v=x9r85EtVFM8&t=707s) True compounding, runaway recursive self-improvement remains an unverified hypothesis; SoL-Pi relies on a hybrid research workflow guided by human-defined constraints.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=x9r85EtVFM8&t=0s) Autonomous agent research often wastes significant token budgets on repetitive loops and bloated contexts, making harness efficiency a critical bottleneck. - [00:00](https://www.youtube.com/watch?v=x9r85EtVFM8&t=0s) SoL-Pi optimizes the agent harness—including prompts, tool wrappers, memory layouts, and delegation rules—leaving foundation model weights untouched. - [01:45](https://www.youtube.com/watch?v=x9r85EtVFM8&t=105s) An automated discovery loop funneled 152 candidate research hypotheses down to four production mechanisms under a strict requirement to preserve capability. - [01:45](https://www.youtube.com/watch?v=x9r85EtVFM8&t=105s) Evaluation relies on EdgeBench, a suite of 51 long-horizon, real-world executable tasks held out entirely from the discovery process to prevent train-set memorization. - [04:13](https://www.youtube.com/watch?v=x9r85EtVFM8&t=253s) Action Fusion bundles predictable dependent command pairs, such as file edits and subsequent test suites, into a single round trip to eliminate unnecessary model calls. - [04:13](https://www.youtube.com/watch?v=x9r85EtVFM8&t=253s) ObservationPack limits terminal outputs larger than 10 KiB to two full provider requests before offloading them to disk and referencing them via lightweight handles and byte offsets. - [04:13](https://www.youtube.com/watch?v=x9r85EtVFM8&t=253s) The Evidence-Preserving Reducer uses a secondary model to extract structured error receipts from massive logs, verifying quoted lines against source hashes to prevent hallucinations. - [04:13](https://www.youtube.com/watch?v=x9r85EtVFM8&t=253s) Online Context Compact dynamically recalculates break-even economics at plan boundaries, ensuring context rewrites save more tokens than they consume in cache invalidation. - [08:10](https://www.youtube.com/watch?v=x9r85EtVFM8&t=490s) The screening dataset comprised 535 executable tasks, including 495 historical GitHub PR/issue pairs and 40 synthetic verification scenarios. - [08:10](https://www.youtube.com/watch?v=x9r85EtVFM8&t=490s) Empirical testing showed financial cost reductions of ~33% ($894 on the GPT backend and $1,158 on Claude Opus 5) alongside minimal drops in task accuracy. - [08:10](https://www.youtube.com/watch?v=x9r85EtVFM8&t=490s) Overall token usage dropped by 45% to 49%, translating to API-equivalent savings of 50% to 54% over model-native harness baselines. - [11:47](https://www.youtube.com/watch?v=x9r85EtVFM8&t=707s) The demonstrated breakthrough is a disciplined empirical process of broad exploration and strict evidence preservation, rather than a science-fiction open-ended recursive loop.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=x9r85EtVFM8&t=0s) The Core Problem: Agent Token Waste and Harness Self-Improvement

The discussion opens with a provocative premise: "What if the next breakthrough in AI research is not a larger model, but a smaller bill?" The hosts point out that an agent can successfully execute a coding task on its first attempt, only to spend the next hour trapped in repetitive loops, burying its original insight under an ever-growing pile of context. This waste introduces the core motivation behind SoL-Pi. When AI is tasked with recursive self-improvement—using AI to refine the very systems that build AI—every single exploratory experiment burns tokens, whether it succeeds or fails. If researchers merely scale up the search space without constraining waste, the research process looks remarkably busy rather than genuinely productive.

Crucially, the hosts clarify that this optimization does not touch the underlying model weights. Instead, the optimization target is the *harness*: the prompts, tools, memory layouts, validation checks, and delegation rules that govern how a foundation model behaves as an autonomous researcher. The challenge is establishing trust: an idea might seem clever in theory, but it cannot be accepted if it degrades core capability, fails on unseen tasks, or never even activates during evaluation. To avoid this, the research loop enforces a strict rule: any surviving proposal must reduce token consumption or monetary cost while respecting a predetermined capability threshold.

```mermaid flowchart TD accTitle: SoL-Pi Recursive Harness Search Loop accDescr: 152 initial proposals funnel through multi-task checks to produce 4 validated mechanisms. subgraph S1["Search Pool"] A["152 Proposed Directions"] end subgraph S2["Evaluation and Screening"] B["Trajectory Analysis & Review"] C["Screening on 535 Training Tasks"] D["Validation on 51 Held-out EdgeBench Tasks"] end subgraph S3["Final Integration"] E["4 Surviving Mechanisms"] F["SoL-Pi Harness"] end A --> B B --> C C --> D D --> E E --> F ```

- **Harness Scope**: Covers prompts, tool wrappers, memory caches, delegation rules, and verification pipelines, leaving model weights completely untouched. - **Recursive Nature**: Discovered improvements reshape future iterations of the search space, using evidence from earlier runs to inform subsequent experiments while maintaining a fixed meta-research protocol.

---

[01:45](https://www.youtube.com/watch?v=x9r85EtVFM8&t=105s) The 152-to-4 Funnel and EdgeBench Validation

The hosts walk through the full SoL-Pi system pipeline. Starting from an initial baseline harness, the system branches out into 152 distinct proposed research directions. Each branch represents an unverified hypothesis rather than an assumed optimization. Agents analyze where current execution patterns burn unnecessary effort, implement specific candidate changes, and evaluate them first on known trajectories before running them across held-out environments.

As empirical evidence is gathered, weak candidates are discarded, and the branching tree aggressively narrows down until only four mechanisms survive. The hosts emphasize the distinction between what has actually been shown versus what remains theoretical: 1. **Demonstrated Phase**: Searching an expansive hypothesis space and empirically discovering four durable efficiency mechanisms within an agent harness. 2. **Proposed Compounding Phase**: Using the newly optimized, lower-cost harness to recursively fund and run subsequent generations of harness searches—a self-funding loop that remains an unverified hypothesis.

To ensure candidates do not simply overfit or memorize their testbeds, the evaluation setup uses EdgeBench, an evaluation suite featuring 51 long-horizon, executable agent tasks. Because these long-horizon runs take hours to execute, uncurbed context expansion and redundant API queries quickly become prohibitively expensive.

- **Initial Funnel**: 152 proposed research directions filtered down to 4 final production mechanisms (roughly a 1-in-40 acceptance rate). - **EdgeBench Benchmark**: 51 long-horizon, real-world executable tasks held out entirely from the discovery loop to prevent train-set memorization.

---

[04:13](https://www.youtube.com/watch?v=x9r85EtVFM8&t=253s) The Four Surviving Efficiency Mechanisms

The conversation shifts to a granular code and workflow walkthrough, using a concrete debugging scenario to explain how each of the four surviving mechanisms eliminates waste.

```mermaid flowchart LR accTitle: The Four Surviving Harness Mechanisms accDescr: Step-by-step application of Action Fusion, ObservationPack, Evidence-Preserving Reducer, and Online Context Compact. subgraph Step1["Execution"] AF["Action Fusion<br/>(Batch Edit + Test)"] end subgraph Step2["Ingestion"] EPR["Evidence-Preserving Reducer<br/>(Receipt Verification)"] end subgraph Step3["Storage"] OP["ObservationPack<br/>(Handles & Byte Offsets)"] end subgraph Step4["Lifecycle"] OCC["Online Context Compact<br/>(Break-Even Compaction)"] end AF --> EPR EPR --> OP OP --> OCC ```

#### 1. Action Fusion In typical agent workflows, an agent edits a file, waits for the environment output, calls the model back to decide what to do next, and then issues a test command (such as `npm test`) that it already knew it had to run. Action Fusion merges predictable command pairs into a single round trip: - The agent combines an edit or write operation with a dependent follow-up test in one payload. - The runtime executes the edit first; if it succeeds, it triggers the follow-up command immediately and returns a combined observation. - If the initial edit fails, the dependent test command is aborted, and the failure is reported without burning an extra model call.

#### 2. ObservationPack When command executions return voluminous output logs, agents typically keep replaying the identical output in full across every subsequent prompt. ObservationPack changes this paradigm from "replay every byte" to "keep access to every byte": - Text payloads larger than 10 KiB are delivered in full for only two provider requests. - For all subsequent queries, the raw payload is replaced by a lightweight, stable handle accompanied by brief excerpts, while the full log is archived locally on disk. - If the agent needs to inspect specific missing lines, it invokes `observation_recall` using the handle and a byte offset, returning a targeted page alongside the next offset.

#### 3. Evidence-Preserving Reducer Massive build or test failure traces often hide just a few critical lines of root-cause information inside thousands of lines of noise. The Evidence-Preserving Reducer delegates the initial parsing of giant logs to a cheaper secondary model: - The secondary model summarizes the output into a structured "receipt," noting status, uncertainty, and exact verbatim quotations of error lines. - The harness guards against hallucinations by validating the receipt against the source hash, ensuring every quoted line exists verbatim in the local archive. - If the reducer fails verification, the harness falls back to the original raw output, guaranteeing provenance even though it cannot guarantee the cheap reader picked every single relevant line.

#### 4. Online Context Compact Even when an issue is resolved, past debugging transcripts persist in the active context window, incurring ongoing costs. However, summarizing context too aggressively can invalidate prefix cache hits and trigger expensive prompt re-encodings. Online Context Compact evaluates whether to rewrite context at plan boundaries based on a strict financial break-even calculation: - Compaction eligibility is calculated as: $$\text{Break-Even Requests} = \frac{\text{Rewrite Cost}}{\text{Savings Per Request}}$$ - The harness factors in outstanding cache debt, contextual safety margins, and context window pressure before triggering native Pi compaction. - If compaction is economically justified, the active task continues seamlessly into a fresh turn with reduced context debt.

---

[08:10](https://www.youtube.com/watch?v=x9r85EtVFM8&t=490s) Benchmark Results, Score Tradeoffs, and Boundaries

The hosts address the critical question of whether these optimizations represent a true efficiency breakthrough or merely a "very elegant shortcut" where an agent cuts costs by giving up on hard tasks. To prevent degradation, SoL-Pi enforces two strict gating criteria: 1. Every capability metric must remain within a pre-declared tolerance margin. 2. At least one efficiency metric (cost or token count) must show marked improvement.

The training and development environment spans 535 executable scenarios. Of these, 495 were extracted from historical GitHub pull request and issue pairs (where an environment is rolled back, confirmed to fail on a regression test, and checked for a pass after applying the patch), while 40 are synthetic tasks paired with programmatic verifiers. A candidate that fails on the 51 held-out EdgeBench tasks is discarded immediately, without feeding that failure trace back into the repair loop.

The financial and performance data show significant token savings alongside a slight drop in accuracy: - **Cost & Token Reductions**: SoL-Pi cuts financial costs by approximately one-third compared to the native Pi baseline, reducing total token consumption by 45% to 49%. Compared to model-native harness baselines, API-equivalent costs drop by 50% to 54%. - **Capability Preservation**: Across both backends, SoL-Pi retains roughly 94% of native Pi's EdgeBench benchmark score (dropping from 44.88 to 42.00 on the GPT backend, and from 44.76 to 42.22 on Claude Opus 5).

The hosts conclude with important caveats regarding the project's scope. The research was a hybrid effort rather than an end-to-end autonomous discovery system: human researchers established initial architectural principles, curated directional proposals, and refined the final implementations. Furthermore, while the public repository provides the four Pi extensions, it does not include the full large-scale automated research pipeline used to discover them, leaving the vision of a fully autonomous, compounding recursive loop as a hypothesis for future work.

[11:47](https://www.youtube.com/watch?v=x9r85EtVFM8&t=707s) Demonstrated Reality vs. Compounding Self-Improvement

The discussion wraps up with a grounded reality check regarding recursive self-improvement and the vision of an agent loop that can indefinitely "discover an even cheaper harness." The hosts make it clear that an infinite, runaway "compounding cycle has not been demonstrated here." Rather than validating science-fiction scenarios of open-ended recursive self-upgrades, the actual achievement of SoL-Pi is far more practical, disciplined, and concrete.

What the project truly proves is an empirical recipe for building efficient research agents without sacrificing task capability: "Search broadly, preserve evidence, and make each saved token pass an exam." Rather than cutting context blindly or assuming optimization loops can endlessly feed into themselves, real progress comes from casting a wide net over potential harness improvements, strictly protecting task-critical findings, and subjecting every single efficiency gain to rigorous benchmark evaluation.

- **Unproven claims vs. reality:** An infinite, compounding loop of self-discovered harnesses remains unproven. - **The core demonstrated principle:** Broad exploration paired with strict evidence preservation, ensuring any token savings pass direct evaluation exams.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=x9r85EtVFM8&t=0s) What if the next breakthrough in AI research is not a larger model, but a smaller bill?

> [00:00](https://www.youtube.com/watch?v=x9r85EtVFM8&t=0s) harness

> [04:13](https://www.youtube.com/watch?v=x9r85EtVFM8&t=253s) replay every byte

> [04:13](https://www.youtube.com/watch?v=x9r85EtVFM8&t=253s) keep access to every byte

> [04:13](https://www.youtube.com/watch?v=x9r85EtVFM8&t=253s) receipt

> [08:10](https://www.youtube.com/watch?v=x9r85EtVFM8&t=490s) very elegant shortcut

> [11:47](https://www.youtube.com/watch?v=x9r85EtVFM8&t=707s) discover an even cheaper harness

> [11:47](https://www.youtube.com/watch?v=x9r85EtVFM8&t=707s) compounding cycle has not been demonstrated here

> [11:47](https://www.youtube.com/watch?v=x9r85EtVFM8&t=707s) Search broadly, preserve evidence, and make each saved token pass an exam.

## Mentioned SoL-Pi, Pi, EdgeBench, Xiaol.x, NVlabs, Action Fusion, ObservationPack, Evidence-Preserving Reducer, Online Context Compact, GPT-5.6, Claude Opus 5, GitHub, Lean, Leanim, Manim, MiniMax, Aura

## Video description Can AI make its own research process cheaper—and still get the work done?

In this two-host explainer, we explore SoL-Pi: Scaling Auto-Research Loops for Efficient Agent Harnesses. Follow 152 proposed research directions down to four surviving mechanisms, using official animations, code walkthroughs, and visual explanations.

We unpack Action Fusion, ObservationPack, the Evidence-Preserving Reducer, and Online Context Compact—and explain what “recursive self-improvement” means in this project.

The central tradeoff: approximately one-third lower cost than Pi while retaining roughly 94% of its average EdgeBench score. We examine both the savings and the limitations.

Chapters: 00:00 Why does agent research waste tokens? 01:14 What is recursive harness improvement? 01:45 Complete official SoL-Pi animation 02:46 Demonstrated research versus future possibilities 04:13 Four mechanisms that reduce wasted work 08:10 Can we trust the savings?

Sources: https://nvlabs.github.io/SoL-Pi/ https://github.com/NVlabs/SoL-Pi

Created with Lean/Leanim and Manim. AI-generated narration via MiniMax/Aura. Official visuals credited to NVlabs.

#SoLPi #AIAgents #AutoResearch #MachineLearning #Manim

[Source](https://www.youtube.com/watch?v=x9r85EtVFM8)

中文简报

SoL-Pi框架在EdgeBench测试中将AI智能体成本降低33% 通过将自动化研究提议筛选为四项上下文与执行机制，该方法在保持约94%基线能力的同时将AI智能体运行成本降低了三分之一，但复合递归自我改进仍未得到证实。本期节目分析了SoL-Pi，该方法在不改变模型权重的前提下优化自主智能体框架（包括提示词、工具和内存管理），以遏制失控的token浪费。通过从152个候选假设中筛选出四项保留机制，该框架引入了批量命令执行、基于磁盘的观察分页、带回执验证的日志精简以及经经济学测算的上下文压缩。在51项留出EdgeBench任务的评估中，SoL-Pi实现了约三分之一的成本降低以及45%至49%的总token消耗降幅，同时保留了原生Pi基线约94%的准确率。最后，主持人明确区分了这些经过验证的效率提升与投机性说法，澄清完全复合的自主自我改进闭环仍是一个未经证实的假设。 SoL-Pi将152项自动化研究提议筛选为四项经过验证的框架机制，在不改变底层模型权重的情况下优化智能体效率。 (1:45) 在51项留出EdgeBench任务中，SoL-Pi将执行成本降低约三分之一，token消耗减少45%至49%，同时保留了原生Pi约94%的能力评分。 (8:10) 保留的四项机制——Action Fusion、ObservationPack、Evidence-Preserving Reducer和Online Context Compact——缓解了冗余的API往返、重复的日志回放以及上下文债务。 (4:13) 真正的复合、失控性递归自我改进仍是一个未经证实的假设；SoL-Pi依赖于人类定义约束指导下的混合研究工作流。 (11:47)

Reading summary

Xiaol.x reports on SoL-Pi, a research initiative demonstrating that optimizing an AI agent's operational harness—rather than modifying model weights—can cut execution costs by roughly one-third while preserving about 94% of task capability. Targeting Harness Architecture Over Model Weights (0:00) Autonomous AI research agents frequently waste tokens by getting stuck in repetitive debugging loops, where early progress is buried under expanding context windows. SoL-Pi targets this inefficiency by optimizing the harness—prompts, tool wrappers, memory caches, validation checks, and delegation rules—leaving foundational model weights entirely untouched. To prevent hollow optimizations, any proposed modification must demonstrably reduce token consumption or financial cost while maintaining performance above a strict capability threshold. The 152-to-4 Screening Funnel and Validation (1:45) The discovery process began with 152 candidate research directions generated as unverified hypotheses. These were screened across 535 development tasks (495 derived from historical GitHub pull request and issue pairs, and 40 synthetic verification tasks). The search space was narrowed down to four durable mechanisms, which were then evaluated against EdgeBench—a held-out benchmark of 51 long-horizon, executable tasks. Candidates that failed on EdgeBench were discarded immediately to prevent overfitting and training-set memorization. Four Concrete Efficiency Mechanisms (4:13) The surviving harness architecture relies on four interlocking components. Action Fusion merges predictable command pairs (such as editing code and running an immediate regression test) into a single API round trip, aborting the follow-up step if the first fails. ObservationPack retains outputs larger than 10 KiB for only two provider requests before replacing them with a lightweight handle, archiving full logs to disk with byte-offset recall via `observation_recall`. The Evidence-Preserving Reducer delegates large error traces to a cheaper secondary model that generates a structured summary receipt; this receipt is checked against the source hash to verify exact verbatim quotations of failure lines before acceptance. Finally, Online Context Compact evaluates whether to rewrite context at plan boundaries using a formula that divides rewrite costs by savings per request, triggering compaction only when economically justified by cache debt and context pressure. Cost Reductions and Benchmark Tradeoffs (8:10) On the EdgeBench benchmark, SoL-Pi reduced total token usage by 45% to 49% and cut monetary costs by approximately 33% relative to the native Pi baseline. On a GPT-5.6 backend, cost dropped to $894 while the benchmark score shifted from 44.88 to 42.00. On Claude Opus 5, cost dropped to $1,158 while the score went from 44.76 to 42.22. In both backends, the harness retained roughly 94% of baseline capability while cutting API-equivalent costs by 50% to 54% compared to model-native harness baselines. The hosts emphasize that an infinite, self-funding compounding research loop remains an unverified hypothesis, not a demonstrated result. The project relied on human researchers to set initial principles and refine implementations, and the public release provides the four runtime mechanisms rather than the full autonomous discovery pipeline.

Source quotes

[0:00] What if the next breakthrough in AI research is not a larger model, but a smaller bill? [0:00] harness [4:13] replay every byte [4:13] keep access to every byte [4:13] receipt [8:10] very elegant shortcut [11:47] discover an even cheaper harness [11:47] compounding cycle has not been demonstrated here [11:47] Search broadly, preserve evidence, and make each saved token pass an exam.

Original transcript

[0:00] What if the next breakthrough in AI [0:02] research is not a larger model, but a [0:05] smaller bill? An agent can solve one [0:07] coding task beautifully, then spend the [0:09] next hour repeating the same exploration [0:11] until the useful idea is buried in its [0:14] own transcript. [0:16] >> That is the uncomfortable question [0:17] behind soul pie. If recursive [0:20] self-improvement means asking AI to [0:23] improve the systems that make AI, every [0:26] experiment spends tokens whether it [0:28] succeeds or not. Scaling the search [0:31] before controlling the waste can make [0:33] progress look busy instead of useful. [0:36] >> So what exactly is being improved? Not [0:38] the model weights. The target is the [0:41] harness, the prompts, tools, memory, [0:44] checks, and delegation rules that turn a [0:47] model into a working researcher. [0:50] >> Here is the pressure test. [0:53] Imagine one environment, one task, and [0:55] dozens of proposed fixes. A proposal can [0:58] sound clever, but if it steals [1:00] capability, fails on a hidden task, or [1:03] never activates during its test, it has [1:05] not earned our trust. The loop needs a [1:08] hard promise. Reduce cost or tokens [1:10] while preserving a declared capability [1:12] check. Before we watch, why call this [1:16] recursive self-improvement? [1:18] Are the models changing their own [1:20] weights or is something else learning? [1:22] >> The target is the harness, the tools, [1:25] memory, and control rules surrounding a [1:28] model. Agents study where that harness [1:30] wastes effort, then test changes to it. [1:34] Successful experiments suggest new [1:36] research directions. The output of one [1:39] search becomes input to another. The [1:42] research procedure itself stays fixed. [1:45] >> Now watch the complete animation. [1:48] One starting harness opens 152 proposed [1:51] directions. Each branch represents a [1:54] possible change, not a proven [1:55] improvement. [1:57] >> A preliminary analysis screens one [1:59] direction. Agents run tasks, combine [2:02] observations, propose a change, [2:04] implement it, and review it. They check [2:06] the same trajectory, then held out [2:08] tasks. Each box supplies evidence for [2:11] whether that change deserves to survive. [2:14] Now, the search repeats across branches. [2:16] Accepted results suggest more directions [2:19] while weak candidates are dropped. That [2:21] feedback is the recursive part. Evidence [2:24] from one iteration shapes the next [2:26] search. Watch the active branches narrow [2:29] toward four survivors. [2:33] >> Finally, the four surviving mechanisms [2:35] merge into SOPI. This diagram shows the [2:38] research process. We will read the [2:40] actual benchmark costs and scores [2:43] separately. After explaining the [2:44] mechanisms, so there are two loops to [2:47] distinguish. Searching for harness [2:50] improvements is demonstrated here. Using [2:52] the cheaper harness to fund its own [2:54] successor is the proposed next cycle. [2:57] That compounding payoff has not been [2:59] demonstrated yet. The scale is [3:02] deliberately visible. The project page [3:05] shows 152 proposed directions entering [3:08] the pool. Only four mechanisms survive [3:11] into the final harness. That funnel is [3:14] the point. Search widely, then demand [3:17] evidence before a mechanism earns a [3:18] place in the system. And the benchmark [3:21] is not a victory lap on a single toy. [3:23] Edgebench holds out 51 long horizon [3:26] executable agent tasks. So the search [3:28] cannot simply memorize its training [3:30] environments. The real question is [3:32] whether an efficiency trick survives a [3:34] new task and still gets the work done. [3:36] >> The reported costs fall, but that is [3:39] only half the story. A cheaper bill [3:42] matters alongside task score. We will [3:45] check both because a predeclared [3:48] preservation criterion does not mean [3:50] every measured score stays identical. [3:54] >> Then let us follow one debugging task [3:57] through the four surviving changes. [4:00] Where does a model make a decision it [4:02] did not need to make? Which byes should [4:05] stay available without being replayed [4:07] forever? And when does forgetting old [4:09] context actually pay for itself? [4:12] Give me one real debugging task. Where [4:16] does the waste actually enter? [4:18] >> You edit a function. The model gets the [4:20] edit result, thinks again, then asks to [4:23] run the test it already knew it needed. [4:25] Action Fusion lets one edit or write [4:27] request include a follow-up command. The [4:30] edit happens first, the test second, and [4:33] one combined observation comes back [4:35] >> so faster because it skipped the check. [4:40] >> The check still runs. The extra model [4:42] round trip disappears here. Then run [4:46] carries npm test. If the edit fails, the [4:50] command is skipped and that fact is [4:52] reported. If the test fails, the failure [4:55] comes back. Fusion joins a predictable [4:58] sequence. It does not turn failure into [5:00] success. [5:01] >> Fine. The test returns a huge log. We [5:05] read it once. Why keep paying for it? [5:08] >> That is observation pack's target. [5:10] repeated large text results in the [5:12] public code. Eligible successful text [5:15] results larger than 10 kibbytes travel [5:17] in full for two provider requests. Later [5:20] requests receive a stable handle and [5:22] short excerpts. The complete original is [5:25] archived locally. [5:27] >> A handle is just an address. What if the [5:31] crucial line was in the middle? [5:33] >> The agent calls observation recall with [5:35] that identifier and a bite offset. It [5:38] gets an exact page plus the next offset [5:40] to continue. This changes what is sent [5:42] to the model, not the stored session [5:44] history. The key distinction is access [5:47] versus replay. Keep access to every [5:49] bite. Stop replaying every bite. [5:53] >> But the first reading can still be [5:55] expensive. A giant failed build log [5:59] might have only three useful lines. [6:02] >> The evidence-preserving reducer [6:03] delegates that first reading to a [6:05] cheaper configured model. It asks for a [6:08] compact receipt, the observed success or [6:11] failure, uncertainty, and exact [6:13] quotations. The main agent still owns [6:16] diagnosis, repair, rerunning the check, [6:19] and the final decision. [6:21] >> And if our cheap reader confidently [6:25] invents the important line, [6:27] >> the receipt is rejected. Code checks the [6:30] source hash, the status, and whether [6:32] every quote appears exactly in the [6:34] archive log. A recognizable failing log [6:37] must retain failure evidence. If [6:40] reduction fails, the original result [6:42] survives. This verifies provenence of [6:45] the selected evidence. It cannot prove [6:47] the reader selected every important [6:49] line. [6:50] >> We we fix the test. Why is the whole [6:54] investigation still sitting in every [6:56] later prompt? [6:58] Online context compact treats a [7:00] completed plan step as a candidate [7:02] moment to summarize old work. But [7:05] compaction has a bill of its own. [7:08] Rewriting context can disturb cached [7:10] prefixes. So a shorter prompt does not [7:13] automatically mean a cheaper run. [7:15] >> How does it know whether that rewrite [7:18] will pay back? [7:20] >> It estimates requests remaining. [7:22] compares the rewrite's incremental cash [7:24] cost with tokens saved on each later [7:26] request and checks context window [7:28] pressure. Think of break even requests [7:30] as rewrite cost divided by savings per [7:33] request. The actual code adds margins [7:36] and outstanding cash debt. A profitable [7:38] rewrite or urgent window protection can [7:41] trigger native PI compaction. [7:43] >> And after that fresh context, the task [7:47] keeps going. After successful [7:50] compaction, Pi continues the active task [7:52] in a new turn. Put the four moves [7:54] together. Fewer redundant decisions, [7:57] less repeated output, cheaper evidence [7:59] reading, and economical context renewal. [8:02] They optimize the machinery around [8:04] useful work. Their interfaces make clear [8:07] what must survive each shortcut. [8:11] >> Four clever mechanisms sound promising, [8:14] but I can make any agent cheaper with [8:16] one line. stop working. How do we tell [8:20] an efficiency breakthrough from a very [8:22] elegant shortcut? [8:23] >> That is why the search has two gates. [8:26] Every capability metric must remain [8:28] within a tolerance declared in advance. [8:30] Then at least one efficiency metric must [8:33] improve. A candidate that saves money by [8:36] skipping necessary verification is [8:37] supposed to fail the first gate. [8:39] >> So where did the candidates come from? [8:42] The headline is 152 proposed directions [8:46] and only four surviving mechanisms. That [8:50] is roughly one survivor for every 40 [8:52] starting ideas. A lot of the research [8:55] [snorts] is learning which attractive [9:00] ideas do not survive contact with [9:03] evidence. [9:04] >> The loops learned in 535 executable [9:08] environments. 495 came from GitHub issue [9:11] and pull request pairs. The repository [9:13] was restored to before the fix. A [9:16] regression check had to fail there and [9:18] pass after the accepted patch. Another [9:20] 40 environments were synthetic tasks [9:23] defined by executable verifiers [9:25] >> and the final exam [9:28] >> 51 edgebench tasks were held out. The [9:31] research loop does not see their results [9:33] and a failed heldout candidate is [9:35] rejected without using that failure for [9:37] repair. These tasks run for hours, which [9:40] gives repeated context and extra model [9:42] turns enough time to become expensive. [9:46] >> Read the cost chart first. The left [9:49] group uses GPT5.6 [9:51] Soul. The right uses Claude Opus 5. In [9:55] each group, compare the native harness [9:57] PI and Soulpie. Soul PI costs $894 on [10:02] the first backend and $1,158 on the [10:06] second. Against Pi, that is about 1/3 [10:10] less cost with 45 to 49% fewer tokens. [10:14] Against the model native harnesses, the [10:16] API equivalent cost is 50 to 54% lower. [10:20] These are totals from this benchmark [10:22] using official pricing, not a guarantee [10:24] for every subscription or future [10:26] workload. [10:27] >> But the score chart complicates the [10:30] victory lap. With the sole backend, PI [10:34] scores 44.8 8 to3 while soul pie scores [10:38] 42. On the claude back end, pi scores [10:42] 44.7 to6 while soli scores 42.2 to2. Um, [10:50] cheaper does come with a small average [10:52] score loss here. [10:54] >> Exactly. The assembled harness retains [10:56] roughly 94% of pi's average score. Small [11:00] losses allowed for individual mechanisms [11:02] can accumulate when they combine. So the [11:04] supported conclusion is a substantial [11:06] efficiency trade-off under capability [11:08] tolerances. It is not evidence that [11:11] every task gets the identical answer at [11:13] half the price. One more boundary. [11:17] This was a hybrid research process. [11:20] Humans supplied early principles, [11:22] filtered directions, and cleaned up [11:24] surviving implementations. [11:27] The release repository contains the four [11:30] PI extensions. It does not by itself [11:34] give us the complete largecale research [11:37] pipeline that produced them. [11:40] >> And the exciting recursive story is [11:42] still a hypothesis. A cheaper harness [11:45] might fund more experiments which might [11:47] discover an even cheaper harness. That [11:50] compounding cycle has not been [11:51] demonstrated here. What has been [11:53] demonstrated is more concrete. Search [11:55] broadly, preserve evidence, and make [11:58] each save token pass an exam.