Measuring Autonomous AI Research: 18 Frontier Models on the nanoGPT Speedrun

Xiaol.x

AI-assisted summary of Measuring Autonomous AI Research: 18 Frontier Models on the nanoGPT Speedrun — original source. Check the original for full context.

Source published: 2026-08-23

Key ideas

Fable 5 outperformed 18 frontier models on Prime Intellect's autonomous nanoGPT speedrun, reaching the validation threshold in 2,726 training steps compared to the 3,290-step baseline and the 2,600-step human record. (2:16)
Despite expending 2.9 billion reasoning tokens over 6.1 days of autonomous runtime with Codex xhigh, GPT-5.6 Sol plateaued at 3,042 steps, demonstrating that raw reasoning compute does not directly yield scientific discovery. (0:01)
The benchmark isolates pure optimizer and schedule discovery on a frozen 124-million-parameter GPT architecture, evaluated deterministically on an eight-GPU H200 sandbox with zero network access. (0:01)
Top-performing models succeeded by managing experimental noise through rigorous multi-seed screening, iterative re-probing, and compound hyperparameter testing, whereas lower-ranking agents prematurely abandoned viable ideas after a single unfavorable seed. (2:16)

Detailed notes

## Background Published by Xiaol.x on YouTube on August 23, 2026, the video covers Prime Intellect's evaluation of 18 frontier AI models across 153 autonomous experiments on the nanoGPT optimizer speedrun benchmark. The task challenges models to optimize a fixed 124M-parameter GPT training run under strict GPU and validation constraints.

## TL;DR The benchmark challenges frontier models to modify only the optimizer, hyperparameters, learning rate schedule, and initialization of a frozen 124M-parameter GPT model to cross a target validation loss in the fewest training steps. Despite consuming 2.9 billion tokens and working autonomously for 6.1 days, GPT-5.6 Sol plateaued at 3,042 steps, trailing behind runner-up Opus 5 at 2,920 steps and leader Fable 5 at 2,726 steps, while the human record stands at 2,600. The performance gap reveals that autonomous discovery is driven by experimental hygiene—managing noise, running ablations, and re-probing discarded candidates—rather than sheer reasoning compute. Through a rigid evaluation harness utilizing frozen validation scripts and multi-seed testing, the study reframes autonomous AI research as a feedback control problem rather than a token-generation contest.

## Atmosphere & style The presentation carries an analytical, sober, and methodical tone that strips away the hype often surrounding autonomous AI and massive token generation. It delivers a grounded reality check, contrasting sophisticated software scaffolding with the unforgiving realities of experimental noise in deep learning. Watching the breakdown feels like inspecting a detailed engineering audit, marked by disciplined skepticism and an appreciation for rigorous empirical feedback.

## Bulletin - [02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) Fable 5 outperformed 18 frontier models on Prime Intellect's autonomous nanoGPT speedrun, reaching the validation threshold in 2,726 training steps compared to the 3,290-step baseline and the 2,600-step human record. - [00:01](https://www.youtube.com/watch?v=rNobPqwvjaE&t=1s) Despite expending 2.9 billion reasoning tokens over 6.1 days of autonomous runtime with Codex xhigh, GPT-5.6 Sol plateaued at 3,042 steps, demonstrating that raw reasoning compute does not directly yield scientific discovery. - [00:01](https://www.youtube.com/watch?v=rNobPqwvjaE&t=1s) The benchmark isolates pure optimizer and schedule discovery on a frozen 124-million-parameter GPT architecture, evaluated deterministically on an eight-GPU H200 sandbox with zero network access. - [02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) Top-performing models succeeded by managing experimental noise through rigorous multi-seed screening, iterative re-probing, and compound hyperparameter testing, whereas lower-ranking agents prematurely abandoned viable ideas after a single unfavorable seed.

## Key takeaways - [00:01](https://www.youtube.com/watch?v=rNobPqwvjaE&t=1s) The nanoGPT speedrun benchmark strictly bounds the search space to optimizer logic, learning rate schedules, hyperparameters, and weight initialization while freezing the 124M-parameter architecture, dataset, batch size, and sequence length. - [00:01](https://www.youtube.com/watch?v=rNobPqwvjaE&t=1s) Success requires driving an eight-seed validation loss below 3.27859 in the fewest possible training steps relative to an unoptimized 3,290-step baseline. - [00:01](https://www.youtube.com/watch?v=rNobPqwvjaE&t=1s) Evaluation relies on a frozen sandbox harness with eight H200 GPUs and no network access, measuring the joint performance of the model and its search environment. - [02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) Fable 5 led the frontier model rankings at 2,726 steps, followed by Opus 5 at 2,920 steps, while GPT-5.6 Sol variants landed in the upper middle tier at 3,042 to 3,058 steps. - [02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) Model rankings remain unchanged even when standardizing for total runtime, experiment count, or raw token budgets, proving that search quality rather than compute volume drives outcomes. - [02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) In stochastic machine learning environments, a major failure mode is abandoning promising research directions due to the noise of a single bad random seed. - [02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) Fable achieved superior results through rigorous experimental hygiene, including screening candidate pools, systematic re-ablations after code merges, and re-testing previously discarded ideas that yielded compound gains. - [02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) GPT-5.6 Sol generated sophisticated engineering scaffolds—such as recursive worker contracts and modular ablation generators—but struggled to interpret noisy empirical feedback into optimal parameter recipes. - [04:21](https://www.youtube.com/watch?v=rNobPqwvjaE&t=261s) The benchmark protocol enforces inspectability by logging agent scratchpads, monitor telemetry, and requiring candidate recipes to pass a deterministic eight-seed verification script. - [04:21](https://www.youtube.com/watch?v=rNobPqwvjaE&t=261s) Autonomous scientific research functions as a feedback control problem where success is determined by the quality of the loop between hypothesis generation and subsequent empirical measurement.

## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=rNobPqwvjaE&t=1s) The nanoGPT Speedrun Benchmark and Setup

The benchmark yields an immediately counterintuitive result: GPT-5.6 Sol ran with Codex xhigh reasoning, consumed a staggering 2.9 billion total tokens, and executed 6.1 days of continuous autonomous work, yet it failed to take first place. Its best validated recipe plateaued at 3,042 steps, whereas Fable 5 reached 2,726 steps. This outcome demonstrates that autonomous scientific discovery is not simply a competition to generate the longest chain of thought; rather, it is a contest over how an agent manages uncertainty and empirical feedback.

The concrete experimental challenge centers on `train_gpt_simple.py`, a standard training script for a 124-million-parameter GPT model. Crucially, the model architecture, dataset, batch size, and sequence length remain completely frozen. The search space is strictly bounded: the agent is permitted to edit only the optimizer, its hyperparameters, the learning rate schedule, and the weight initialization scheme. The objective is to drive the validation loss below the target threshold (crossing 3.28, specifically an eight-seed mean under 3.27859) in the fewest training steps possible. The unoptimized baseline requires 3,290 steps, while the existing human record stands at 2,600 steps. The benchmark therefore tests pure "optimizer discovery, not a model redesign."

Evaluating these agents requires a rigid evaluation harness that acts as a referee. Running on a sandbox equipped with eight H200 GPUs and zero network access on this tier, the harness enforces a deterministic rulebook: experiments are driven through `run.sh` with fixed random seeds, and candidate solutions are evaluated against a frozen `verify.py` script to decide what counts as an official, fair record. Because different harness implementations can alter search efficiency, the leaderboard strictly measures the combined behavior of the model and its harness. It does not mean the harness itself discovered the optimizer, but rather that it governed the environment in which the model navigated the search space.

```mermaid flowchart TD accTitle: Autonomous Optimizer Search Loop accDescr: Flowchart illustrating the cycle of modifying the optimizer script, screening with run.sh, and certifying via frozen verify.py.

subgraph SearchSpace ["Constrained Search Space"] A["train_gpt_simple.py"] A1["Frozen: Architecture, Data, Batch Size, Seq Length"] A2["Editable: Optimizer, Hyperparameters, LR Schedule, Init"] end

subgraph Execution ["Execution & Screening"] B["Agent Proposes Changes"] C["Screen via run.sh<br/>(8x H200 GPUs, No Network)"] end

subgraph Validation ["Evaluation & Certification"] D["Multi-Seed Check"] E{"Frozen verify.py<br/>Validation Loss < 3.27859?"} F["Certified Record PR"] G["Discard / Re-probe / Re-ablate"] end

A --> B B --> C C --> D D --> E E -->|"Pass (Fewer Steps)"| F E -->|"Fail / Inconclusive"| G G --> B ```

[02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) Leaderboard Results and Handling Experimental Noise

On the speedrun leaderboard, lower step counts represent superior convergence speed. Against the human record of 2,600 steps, Fable 5 leads the models at 2,726 steps, followed by Opus 5 at 2,920 steps. GPT-5.6 Sol Pro sits around 3,100 steps (3,058 steps), while the detailed run table lists GPT-5.6 Sol at 3,042 steps. This places the Sol models squarely in the upper middle tier rather than on the frontier. Critically, this ordering remains identical even when standardizing across equal run time, total experiment count, or raw token budgets—proving that throwing more reasoning tokens at the problem does not explain the performance gap.

The reason a model with massive reasoning compute can lose comes down to experimental noise. In stochastic deep learning optimization, an individual run can easily disguise a lucky seed, a silent crash, or an internal scaling bug as genuine scientific progress. Weaker agent traces fall prey to this variance by completely abandoning promising ideas after encountering a single unfavorable random seed. In contrast, stronger traces employ rigorous experimental hygiene: they screen candidates across a small pool of seeds, rigorously re-test borderline performance gains, run fresh ablation passes after every code merge, and systematically revisit previously discarded negative results whenever the underlying recipe evolves. Fable even explored combinations of hyperparameter knobs that degraded performance when applied individually but yielded significant compound gains when toggled together; one such late-stage re-probe recovered 31 steps.

GPT-5.6 Sol was neither passive nor structurally incompetent. An audit of its public traces reveals sophisticated engineering workflows: it synthesized modular ablation generators, established recursive worker contracts, and orchestrated highly controlled experiments, actively converting distinct research roles into code. Yet despite this software architectural elegance, its verified outcome stalled at 3,042 steps. Sol exhibited immense organizational "effort, not experimental judgment"—it built impressive scaffolds but struggled to preserve faint signals amid noise and translate ambiguous evidence into an optimal recipe.

[04:21](https://www.youtube.com/watch?v=rNobPqwvjaE&t=261s) The Inspectable Autonomous Research Loop

The repository's strict operational protocol makes these metrics meaningful and scientifically inspectable. The autonomous loop operates under clear checkpoints: - The agent iteratively edits the core optimization parameters inside `train_gpt_simple.py`. - It performs rapid initial screening of each candidate recipe using `run.sh`. - Candidates demonstrating consistent progress are promoted to full eight-seed validation runs. - The frozen `verify.py` script automatically evaluates the validation loss across the eight seeds alongside a strict significance margin. - Every run publishes its full reasoning traces, agent scratchpads, system monitor logs, and a unified record pull request per model.

This infrastructure converts the fuzzy concept of autonomous machine learning into a concrete, auditable control loop: propose a hypothesis, run the code, measure the outcome, verify against rigorous statistical thresholds, and decide whether to keep or discard the change.

GPT-5.6 Sol with Codex xhigh fell behind not because it lacked reasoning capacity or "thought too little," but because autonomous scientific research is ultimately a feedback control problem. Success requires an agent to dynamically select informative experiments, accurately model environmental stochasticity, retain ambiguous signals instead of prematurely discarding them, and correctly discern when an incremental gain warrants the compute expenditure of eight verification seeds. This benchmark does not demonstrate recursive self-improvement or reveal a miraculous, fundamentally novel optimizer class, but it provides a grounded mental model for evaluating autonomous systems: true research ability is not measured by the volume of tokens burned, but by "the quality of the loop between a hypothesis and the next measurement."

## Notable quotes > [00:01](https://www.youtube.com/watch?v=rNobPqwvjaE&t=1s) autonomous scientific discovery is not simply a competition to generate the longest chain of thought; rather, it is a contest over how an agent manages uncertainty and empirical feedback.

> [00:01](https://www.youtube.com/watch?v=rNobPqwvjaE&t=1s) The search space is strictly bounded: the agent is permitted to edit only the optimizer, its hyperparameters, the learning rate schedule, and the weight initialization scheme.

> [00:01](https://www.youtube.com/watch?v=rNobPqwvjaE&t=1s) optimizer discovery, not a model redesign

> [00:01](https://www.youtube.com/watch?v=rNobPqwvjaE&t=1s) the leaderboard strictly measures the combined behavior of the model and its harness.

> [02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) throwing more reasoning tokens at the problem does not explain the performance gap.

> [02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) In stochastic deep learning optimization, an individual run can easily disguise a lucky seed, a silent crash, or an internal scaling bug as genuine scientific progress.

> [02:16](https://www.youtube.com/watch?v=rNobPqwvjaE&t=136s) effort, not experimental judgment

> [04:21](https://www.youtube.com/watch?v=rNobPqwvjaE&t=261s) autonomous scientific research is ultimately a feedback control problem.

> [04:21](https://www.youtube.com/watch?v=rNobPqwvjaE&t=261s) the quality of the loop between a hypothesis and the next measurement

## Mentioned Prime Intellect, Xiaol.x, nanoGPT, Fable 5, Opus 5, GPT-5.6 Sol, GPT-5.6 Sol Pro, GPT-5.5, Codex xhigh, Kimi K2.7, NVIDIA H200, train_gpt_simple.py, run.sh, verify.py

## Video description Prime Intellect ran 153 autonomous experiments across 18 frontier models on the same nanoGPT optimizer speedrun.

The benchmark:

- Fixed 124M-parameter GPT training task - Baseline: 3,290 steps - Human record: 2,600 steps - Record threshold: eight-seed mean validation loss less than 3.27859 - Up to eight H200 GPUs per run - Runs lasting up to eight days

Selected results:

- Fable 5: 2,726 steps - Opus 5: 2,920 steps - GPT-5.6 Sol Pro: 3,058 steps - GPT-5.6 Sol, Codex xhigh: 3,042 steps - GPT-5.5: 3,234 steps - Kimi K2.7: 3,240 steps

This video visualizes the benchmark task, leaderboard, evaluation harness, public traces, and reproducibility protocol. It does not claim that the ranking proves a general reason for model performance.

Sources:

Prime Intellect: https://www.primeintellect.ai/blog/measuring-autonomous-research

Public repository: https://github.com/PrimeIntellect-ai/frontier-automated-speedrun

[Source](https://www.youtube.com/watch?v=rNobPqwvjaE)

中文简报

评估自主AI科研能力：18款前沿模型角逐nanoGPT竞速基准该基准测试要求前沿模型仅修改冻结的1.24亿参数GPT模型的优化器、超参数、学习率调度和初始化方案，以最少的训练步数达到目标验证损失。尽管消耗了29亿token并在无人工干预下自主运行了6.1天，GPT-5.6 Sol的表现最终停滞在3,042步，落后于位列第二的Opus 5（2,920步）和榜首Fable 5（2,726步），而人类纪录为2,600步。该性能差距表明，自主科研发现的核心驱动力在于实验严谨性——控制噪声、执行消融实验以及重新探查被放弃的备选方案，而非单纯堆叠推理算力。通过结合固定验证脚本和多种子测试的严格评估框架，该研究将自主AI科研重新定义为一个反馈控制问题，而非一场生成token的竞赛。在Prime Intellect的自主nanoGPT竞速测试中，Fable 5的表现超越了其余18款前沿模型，仅用2,726个训练步数便达到验证阈值，相比之下基准线为3,290步，人类纪录为2,600步。 (2:16) 尽管搭配Codex xhigh在6.1天的自主运行中消耗了29亿个推理token，GPT-5.6 Sol最终停滞在3,042步，这表明单纯的推理算力并不能直接转化为科学发现。 (0:01) 该基准测试将研究范围严格限定在冻结的1.24亿参数GPT架构上对优化器与调度策略的纯粹探索，并在完全无网络访问的8卡H200沙箱中进行确定性评估。 (0:01) 表现优异的模型之所以成功，是因为它们通过严谨的多随机种子筛选、迭代重探和复合超参数测试来控制实验噪声；而排名靠后的智能体往往在单次随机种子表现不佳后便过早放弃了可行方案。 (2:16)

Reading summary

Findings from the nanoGPT Speedrun benchmark demonstrate that autonomous scientific discovery functions as an empirical feedback control problem rather than a contest of raw reasoning compute. While GPT-5.6 Sol consumed 2.9 billion tokens across 6.1 days of autonomous work, Fable 5 won the model leaderboard by managing experimental noise and stochastic feedback more effectively. Bounded Benchmark Setup and Deterministic Harness (0:01) The nanoGPT Speedrun challenge restricts agent autonomy strictly to optimizer discovery rather than full model redesign. Working with `train_gpt_simple.py` to train a 124-million-parameter GPT model, the benchmark keeps the architecture, dataset, batch size, and sequence length frozen. Agents may modify only the optimizer algorithm, its hyperparameters, the learning rate schedule, and the weight initialization scheme. The objective is to push validation loss below a target threshold—specifically an eight-seed mean under 3.27859 (crossing 3.28)—in the fewest training steps possible, starting from an unoptimized baseline of 3,290 steps toward a human record of 2,600 steps. Evaluation runs in an isolated sandbox equipped with eight H200 GPUs and no network access. A deterministic harness coordinates the search: candidate recipes are screened via `run.sh` with fixed random seeds, and official records are certified exclusively through a frozen `verify.py` script. Because variations in harness implementation directly impact search efficiency, the leaderboard assesses the combined capability of the model and its harness rather than optimizer discovery conducted by the harness independently. Leaderboard Results Defy Raw Compute Budgets (2:16) The speedrun leaderboard reveals that scaling reasoning compute does not automatically secure optimization breakthroughs. Fable 5 led all models by reaching the validation threshold in 2,726 steps, followed by Opus 5 at 2,920 steps. By comparison, GPT-5.6 Sol running Codex xhigh plateaued at 3,042 steps, while GPT-5.6 Sol Pro reached 3,058 steps (summarized at 3,100 on the leaderboard overview), situating the Sol family in the upper middle tier. Lower entries included GPT-5.5 at 3,234 steps and Kimi K2.7 at 3,240 steps. This relative ranking remained unchanged even when standardizing for identical run time, total experiment count, or raw token expenditure. Despite spending 2.9 billion tokens over 6.1 continuous days, GPT-5.6 Sol could not close the performance gap, showing that high token volume alone does not determine convergence efficiency. Experimental Hygiene Versus Software Scaffolding (2:16) The performance divide stemmed largely from how agents managed stochastic noise in deep learning optimization. Random fluctuations can disguise lucky seeds, silent failures, or scaling bugs as apparent progress; weaker agent traces frequently discarded viable ideas after a single unfavorable seed. In contrast, Fable 5 practiced strict experimental hygiene: evaluating candidates over small multi-seed pools, re-testing borderline metrics, running fresh ablation passes after code merges, and re-evaluating discarded negative results as the baseline recipe shifted. Fable also tested hyperparameter combinations that failed individually but succeeded jointly, recovering 31 steps in one late-stage re-probe. Conversely, GPT-5.6 Sol demonstrated sophisticated software engineering rather than strong experimental judgment. Public trace audits show Sol built modular ablation generators, recursive worker contracts, and code-based research divisions of labor. Despite this architectural elegance, Sol struggled to interpret faint empirical signals amid noise, expending immense organizational effort without translating ambiguous data into an optimal recipe. The Inspectable Feedback Control Protocol (4:21) The benchmark protocol enforces transparency by organizing autonomous machine learning into an inspectable, auditable control loop. Models iteratively edit `train_gpt_simple.py`, conduct initial screening through `run.sh`, promote promising candidates to eight-seed validation, and submit to automated threshold checks via `verify.py`. Each run records full reasoning traces, scratchpads, system monitor logs, and unified record pull requests. This framework treats autonomous scientific inquiry as a closed feedback loop: agents must dynamically select informative experiments, model stochastic variance, preserve borderline signals, and accurately assess whether an incremental gain justifies the compute cost of multi-seed verification. The benchmark does not claim to show recursive self-improvement or the discovery of a fundamentally new optimizer class. Instead, it indicates that autonomous scientific competence is governed not by token consumption, but by the quality and precision of the feedback loop connecting a hypothesis to empirical measurement.

Source quotes

[0:01] autonomous scientific discovery is not simply a competition to generate the longest chain of thought; rather, it is a contest over how an agent manages uncertainty and empirical feedback. [0:01] The search space is strictly bounded: the agent is permitted to edit only the optimizer, its hyperparameters, the learning rate schedule, and the weight initialization scheme. [0:01] optimizer discovery, not a model redesign [0:01] the leaderboard strictly measures the combined behavior of the model and its harness. [2:16] throwing more reasoning tokens at the problem does not explain the performance gap. [2:16] In stochastic deep learning optimization, an individual run can easily disguise a lucky seed, a silent crash, or an internal scaling bug as genuine scientific progress. [2:16] effort, not experimental judgment [4:21] autonomous scientific research is ultimately a feedback control problem. [4:21] the quality of the loop between a hypothesis and the next measurement

Original transcript

[0:01] Here is the confusing part. GPT-5 [0:04] 6-soul ran with codex x-high reasoning, [0:08] 2.9 billion total tokens, and 6.1 days [0:14] of autonomous work, yet it did not win. [0:17] Its best validated recipe reached 3,042 [0:21] steps. Fable 5 reached 2,726. [0:27] So, the result is not a contest for the [0:29] longest chain of thought. It is a [0:31] contest for what the agent does with [0:35] uncertainty. [0:38] Here is the concrete case. The agent [0:40] starts from train_gpt_simple.py, [0:46] a small 124 million parameter GPT [0:50] training recipe. The model architecture, [0:53] data, batch size, and sequence length [0:57] are frozen. [0:58] The editable part is the optimizer, its [1:01] hyper parameters, the learning rate [1:03] schedule, and initialization. [1:06] The job is to make validation loss cross [1:10] 3.28 [1:11] in fewer training steps. The baseline is [1:15] 3,290. [1:17] The human record is 2,600. [1:20] This is optimizer discovery, not a model [1:24] redesign. [1:26] >> [snorts] [1:26] >> So, is this a harness problem? Partly, [1:30] but in a precise sense. The harness is [1:33] the referee, a sandbox, 8H, 200 GPUs, no [1:39] network on this tier, a rulebook, run.sh [1:43] for experiments, fixed seeds, and frozen [1:46] verified.py. [1:48] It decides what counts as a fair record. [1:51] Um [1:53] the case being solved is still the [1:54] optimizer recipe inside [1:56] train_gpt_simple.py. [2:01] Different harnesses can change how [2:02] effectively a model searches, so the [2:04] leaderboard measures model plus harness [2:07] behavior. [2:09] It does not mean the harness itself [2:11] discovered a better optimizer. [2:16] Now the leaderboard becomes readable. [2:19] Lower is better. Human record, 2,600. [2:24] Fable 5, 2,726. [2:27] Opus 5, 2,920. [2:30] GPT-5, [2:33] 6-Soul Pro, 3,100. [2:36] GPT-5, [2:38] 6-Soul, 3,042 [2:41] in the detailed run table. [2:43] That places Soul in the upper middle, [2:46] not at the frontier. The page reports [2:49] the same ordering under equal time [2:51] experiment or token budgets. More tokens [2:54] alone do not explain the lead. [2:57] Why can a model with enormous reasoning [3:00] budgets still lose? Because the score is [3:03] noisy. [3:04] A single training run can confuse a [3:06] lucky seed, a crash, or a scaling bug [3:10] with a scientific result. Weaker traces [3:14] kill a whole idea after one bad seed. [3:18] Stronger traces screen on a few seeds, [3:20] retest borderline gains, re-ablate after [3:23] every merge, and revisit old negatives [3:25] when the recipe changes. [3:27] Um um Babel even tested pairs of knobs [3:31] that were individually worse but jointly [3:32] better. [3:33] One late re-probe was worth 31 steps. [3:40] Um [3:41] Soul was not idle or incompetent. Its [3:45] trace built reusable ablation [3:47] generators, recursive worker contracts, [3:50] and controlled experiments. The public [3:53] page shows Soul turning research roles [3:55] into code, but the validated result [3:58] still stopped at 3,042 steps [4:02] versus Fables 2,726. [4:06] The honest reading is narrow. Soul can [4:09] organize a serious search, yet this run [4:11] preserved fewer weak signals and [4:13] converted less noisy evidence into a [4:15] winning recipe. [4:17] Soul is effort, not experimental [4:19] judgment. [4:21] The GitHub rulebook explains why the [4:23] numbers mean something. The agent edits [4:25] train GPT simple.py. [4:28] It screens a recipe with run.sh, [4:31] then promotes it through an eight-seed [4:33] record run. Frozen verify.py checks the [4:36] mean loss and the significance margin. [4:39] The repository publishes traces, [4:41] scratchpads, monitor reports, and one [4:44] record pull request per model. That [4:46] turns autonomous research into an [4:49] inspectable loop. [4:51] Propose, run, measure, verify, keep or [4:54] discard. [4:57] So, [4:58] is Soul X high behind because it thought [5:01] too little? No. It is behind because [5:05] autonomous research is a feedback [5:09] control problem. The agent must choose [5:12] experiments, model noise, preserve [5:16] ambiguous evidence, and know when a [5:18] result deserves eight more runs. This [5:22] benchmark [5:23] does not prove recursive [5:26] self-improvement, and it does not claim [5:28] a fundamentally new optimizer. It gives [5:31] us a sharper mental model. [5:33] Researchability is not tokens spent. It [5:36] is the quality of the loop between a [5:38] hypothesis and the next measurement.