OpenAI GPT-6 Skips Equilibrium Solvers for Finite Recurrent Loops

Is GPT-6's "Recurrent Depth" Just a DEQ? Looped Transformers & Their Scaling Laws|Loop State EP.01 — Xiaol.x

Xiaol.x

Recurrent-depth architectures treat depth as an unrolled test-time compute budget rather than a fixed-point equilibrium problem, delivering approximately half the performance gain of unique layers per iteration.

AI-assisted summary of Is GPT-6's "Recurrent Depth" Just a DEQ? Looped Transformers & Their Scaling Laws|Loop State EP.01 — original source. Check the original for full context.

Source published:

Key ideas

  • Recurrent-depth transformers are fundamentally distinct from Deep Equilibrium Models (DEQs): they execute finite unrolled iterations with backpropagation rather than relying on numerical root-finders to solve for fixed points, enabling them to naturally represent multi-step periodic orbits. (1:09)
  • The iso-depth parameter exchange rate converges to φ = 0.46, meaning one recurrent pass yields roughly half the utility of a brand-new unique layer at matched FLOPs, explaining why production architectures cap loops between 2 and 4 iterations. (10:52)
  • In test-time recurrence sweeps, Huginn's 3.5B-parameter model matched the forward FLOPs of a 50B dense model at 32 recurrent passes, driving GSM8K Chain-of-Thought accuracy from near 0% to the 40s. (6:41)
  • Pre-training laws from the Ouro project (7.7T tokens) show that recurrent loops expand algorithmic manipulation capacity—matching an 8B dense model on reasoning using a 2.6B looped architecture—while factual memory capacity remains bound to physical parameters at ~2 bits per parameter. (9:11)

Detailed notes

## Background Published on YouTube by Xiaol.x on September 4, 2026, this debut episode of the research podcast 'Loop State' examines rumors surrounding the 'recurrent depth' architecture behind OpenAI's GPT-6 (codename Astra). The page description outlines an inquiry into whether this design is merely a rebranded Deep Equilibrium Model (DEQ), tracing the intellectual lineage of looped transformers and analyzing three empirical scaling laws that govern their performance.

## TL;DR Hosts Theo and Mara establish that recurrent depth is fundamentally distinct from a DEQ: instead of relying on numerical solvers and implicit differentiation to find a static fixed point, looped transformers unroll weight-tied blocks across a finite compute budget, naturally supporting periodic orbits as well as fixed points. Drawing on architectures like Huginn and findings from Saunshi et al. and Ouro, they show that accuracy scales linearly with log depth and fits clean Chinchilla-style power laws during pre-training. Crucially, iso-depth scaling benchmarks establish an exchange rate of φ ≈ 0.46, meaning a recurrent pass yields about half the utility of a fresh layer and explaining why industry applications stop at two to four loops. The hosts conclude by separating verified facts from speculation, noting that while the mechanics are proven at smaller scales, frontier-level scaling metrics and inference trade-offs remain closely guarded proprietary secrets.

## Atmosphere & style The discussion carries the sharp, caffeinated rhythm of an intense chalkboard seminar between two researchers who have combed through every pre-print on the topic. Rendered cleanly on a Leanim board with natural MiniMax voice synthesis, Mara's focused cross-examinations balance Theo's encyclopedic recall of papers and empirical formulas. The mood is intellectually grounded and refreshingly skeptical, refusing to indulge in frontier AI hype while methodically dissecting the raw mathematics and engineering trade-offs of modern architectures.

## Bulletin - [01:09](https://www.youtube.com/watch?v=V03zLD8JKuU&t=69s) Recurrent-depth transformers are fundamentally distinct from Deep Equilibrium Models (DEQs): they execute finite unrolled iterations with backpropagation rather than relying on numerical root-finders to solve for fixed points, enabling them to naturally represent multi-step periodic orbits. - [10:52](https://www.youtube.com/watch?v=V03zLD8JKuU&t=652s) The iso-depth parameter exchange rate converges to φ = 0.46, meaning one recurrent pass yields roughly half the utility of a brand-new unique layer at matched FLOPs, explaining why production architectures cap loops between 2 and 4 iterations. - [06:41](https://www.youtube.com/watch?v=V03zLD8JKuU&t=401s) In test-time recurrence sweeps, Huginn's 3.5B-parameter model matched the forward FLOPs of a 50B dense model at 32 recurrent passes, driving GSM8K Chain-of-Thought accuracy from near 0% to the 40s. - [09:11](https://www.youtube.com/watch?v=V03zLD8JKuU&t=551s) Pre-training laws from the Ouro project (7.7T tokens) show that recurrent loops expand algorithmic manipulation capacity—matching an 8B dense model on reasoning using a 2.6B looped architecture—while factual memory capacity remains bound to physical parameters at ~2 bits per parameter.

## Key takeaways - [00:01](https://www.youtube.com/watch?v=V03zLD8JKuU&t=1s) Reports that GPT-6 (Astra) utilizes recurrent depth stem from a single anonymous leak, with OpenAI Chief Scientist Jakub Pachocki only confirming that compute-graph depth is within a factor of two of GPT-4. - [01:09](https://www.youtube.com/watch?v=V03zLD8JKuU&t=69s) Recurrent depth differs from DEQs in its mathematical contract: DEQs train for fixed-point stationarity via implicit differentiation, whereas looped transformers optimize representations over a bounded iteration budget. - [01:09](https://www.youtube.com/watch?v=V03zLD8JKuU&t=69s) Looped transformer hidden states exhibit complex dynamical behaviors—settling into fixed points for simple tokens and stable periodic orbits for complex reasoning—which DEQ root-finders cannot represent. - [02:02](https://www.youtube.com/watch?v=V03zLD8JKuU&t=122s) The Huginn architecture guarantees path independence by initializing with Gaussian noise and freshly re-injecting static input embeddings at every recurrent iteration. - [03:25](https://www.youtube.com/watch?v=V03zLD8JKuU&t=205s) Replacing looped execution with explicit DEQ fixed-point solvers slows down training and degrades final model quality, demonstrating that equilibrium convergence is unnecessary for latent reasoning. - [05:31](https://www.youtube.com/watch?v=V03zLD8JKuU&t=331s) Looped transformers trace a distinct lineage from 2018 Universal Transformers through theoretical algorithmic formulations to large-scale pre-training efforts like Huginn and Ouro. - [06:41](https://www.youtube.com/watch?v=V03zLD8JKuU&t=401s) Test-time recurrence acts as a dynamic compute knob, where harder benchmarks like GSM8K and ARC-Challenge continue extracting accuracy gains at high loop budgets. - [06:41](https://www.youtube.com/watch?v=V03zLD8JKuU&t=401s) Early-exit heuristics based on token KL divergence enable adaptive inference, allowing straightforward tasks to exit early while difficult dilemmas utilize extra recurrent passes. - [07:58](https://www.youtube.com/watch?v=V03zLD8JKuU&t=478s) Accuracy scales linearly with log effective depth, with looped models capturing 60% of unshared layer performance on math word problems and 120% on algorithmic reasoning primitives. - [07:58](https://www.youtube.com/watch?v=V03zLD8JKuU&t=478s) Weight tying in looped blocks acts as a helpful inductive bias for iterative computation, successfully executing internal multi-step reasoning without emitting chain-of-thought tokens. - [09:11](https://www.youtube.com/watch?v=V03zLD8JKuU&t=551s) Ouro established a modified Chinchilla scaling law incorporating an explicit recurrence term, achieving an R² of 0.96 across a 7.7-trillion-token pre-training regime. - [09:11](https://www.youtube.com/watch?v=V03zLD8JKuU&t=551s) Recurrent depth does not expand factual knowledge storage—which remains fixed at approximately 2 bits per stored parameter—but instead increases reasoning manipulation applied to stored knowledge. - [10:52](https://www.youtube.com/watch?v=V03zLD8JKuU&t=652s) Iso-depth scaling laws establish an exchange rate of φ = 0.46, meaning unrolling a single block 4 times yields the computational utility of roughly 1.9 unshared unique layers. - [10:52](https://www.youtube.com/watch?v=V03zLD8JKuU&t=652s) The parameter exchange rate φ is sensitive to training architecture, dropping to 0.38 under truncated backpropagation and rising to 0.65 when dense hyper-connections are introduced. - [12:14](https://www.youtube.com/watch?v=V03zLD8JKuU&t=734s) Recurrent depth trades stored weights for inference latency and KV cache memory bandwidth, as every recurrent pass recomputes attention and appends cache states. - [13:30](https://www.youtube.com/watch?v=V03zLD8JKuU&t=810s) Looped transformers reframe transformer depth from a rigid structural geometry into an allocateable, per-token compute budget.

## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=V03zLD8JKuU&t=1s) Astra, GPT-6, and the "Recurrent Depth" Leak

The episode opens with Mara and Theo digging into the mystery surrounding OpenAI's newly shipped frontier model, GPT-6 (codenamed Astra), which dropped on September 3, 2026. Two days prior to launch, *The Information* cited a single anonymous source claiming Astra relies on a "constrained form of recurrent depth"—essentially taking a single block of transformer layers and reusing it across an iterative loop. When asked about it, OpenAI offered neither a confirmation nor a clear denial; Chief Scientist Jakub Pachocki merely noted that the depth of Astra's computation graph is within a factor of two of GPT-4, and the official release post avoided the term entirely. Reported, not confirmed.

```mermaid flowchart LR accTitle: Astra Recurrent Depth Speculation accDescr: Pipeline from the anonymous leak about Astra's recurrent depth to the community hypothesis comparing it to DEQs. A["Leak (The Information)<br/>'Constrained Recurrent Depth'"] --> B["Jakub Pachocki<br/>'Compute graph depth within 2x GPT-4'"] B --> C{"Community Question<br/>Is this just a DEQ?"} ```

The immediate question sweeping the machine learning community was whether this supposedly novel architecture was merely a Deep Equilibrium Model (DEQ) repackaged under marketing buzzwords. Theo and Mara tackle this head-on: rather than saving the conclusion for the end, they settle the DEQ comparison immediately before tracing the lineage and the empirical scaling laws governing looped transformers.

---

[01:09](https://www.youtube.com/watch?v=V03zLD8JKuU&t=69s) The Fundamental Difference: DEQ vs. Looped Transformer

Theo gives an unambiguous verdict: recurrent depth is *not* a DEQ, despite both paradigms reusing a single weight-tied layer block. A DEQ is strictly defined by an equilibrium condition; it solves for an explicit fixed-point hidden state $z^*$ satisfying $z^* = f(z^*, x)$, utilizing numerical root-finders and computing gradients via the implicit function theorem. A looped transformer operates under an entirely different contract: it simply unrolls the block for a pre-allocated or sampled budget of steps and backpropagates through that unrolled trajectory.

As Mara summarizes, "One is find the equilibrium. The other is iterate and stop when you've had enough." While a DEQ trains a model to produce a stationary state, recurrent depth trains the representation to be maximally useful after a bounded compute budget. Furthermore, empirical observations reveal that hidden states in looped models exhibit complex dynamical behaviors: while simple tokens may settle into fixed points, difficult reasoning tokens routinely fall into stable periodic orbits. Because a DEQ's root-finder can only resolve a single fixed point per input, it cannot naturally represent an orbit.

| Property | Deep Equilibrium Model (DEQ) | Recurrent-Depth / Looped Transformer | |:--- |:--- |:--- | | **Mathematical Contract** | Solves for equilibrium: $z^* = f(z^*, x)$ | Finite iteration: $s_i = R(E, s_{i-1})$ | | **Training Gradient** | Implicit differentiation at $z^*$ (IFT) | Truncated backpropagation through depth | | **Solver Required** | Yes (e.g., Broyden, Anderson acceleration) | No numerical solver | | **Memory Footprint** | $O(1)$ constant in depth | Proportional to unrolled backprop window | | **Representable Dynamics** | Fixed points only | Fixed points AND multi-step orbits |

---

[02:02](https://www.youtube.com/watch?v=V03zLD8JKuU&t=122s) Mechanics of a Recurrent Depth Pass: Huginn Architecture

To make the mechanics concrete, Theo walks through the forward pass of Huginn, the open 3.5-billion-parameter reference model published by Geiping et al. in February 2025. The architecture is split into three modular stages:

1. **Prelude:** A shallow prefix of initial transformer layers that embeds the input sequence into a base latent tensor, $E$. 2. **Recurrent Block ($R$):** The core weight-tied transformer block unrolled across depth. 3. **Coda:** A shallow postfix of final layers that unembeds the latent representation into output token logits.

```mermaid flowchart TD accTitle: Huginn Recurrent Depth Forward Pass accDescr: Dataflow diagram of the Huginn architecture showing prelude embedding, recurrent block iterations with fresh injection, and coda projection. subgraph S1 ["1. Embedding"] Tokens["Input Tokens"] --> Prelude["Prelude Layers"] Prelude --> E["Latent Vector E"] end

subgraph S2 ["2. Recurrent Loop (Steps 1 to r)"] Noise["s₀ ~ Gaussian Noise"] --> Loop E -->|"Fresh injection at step i"| Loop["sᵢ = R(E, sᵢ₋₁)"] Loop -->|"Iterate r times"| Loop end

subgraph S3 ["3. Prediction"] Loop -->|"Final state sᵣ"| Coda["Coda Layers"] Coda --> Logits["Output Token Logits"] end ```

Crucially, the starting state $s_0$ is initialized as pure Gaussian noise rather than the token embedding itself. At every recurrence step $i \in \{1, \dots, r\}$, the static embedding $E$ is injected fresh alongside the prior recurrent state:

$$s_i = R(E, s_{i-1})$$

Theo identifies this continuous re-injection as the "deep thinking" trick originated in work by Bansal and Anil. Feeding $E$ into every iteration guarantees path independence, stabilizing recurrence so the final trajectory does not degenerate or drift based on initial noise. Mara draws the direct analogy: the recurrence operator functions like iterative optimization, repeatedly re-reading the conditioning data at every step just as gradient descent repeatedly queries a dataset.

---

[03:25](https://www.youtube.com/watch?v=V03zLD8JKuU&t=205s) Solvers, Orbits, and Truncated Unrolling

Comparing the training pipelines highlights the architectural divergence between classical DEQs and recurrent depth. Bai, Kolter, and Koltun (2019) designed DEQs to find $z^* = F(z^*, x)$ using root-finding algorithms such as Broyden's method or Anderson acceleration. Because gradients are derived purely at the fixed point via the implicit function theorem, DEQs never store intermediate activations, yielding $O(1)$ constant memory overhead with respect to depth. The heavy penalty is paying for an iterative numerical solver during *both* the forward and backward passes, under the strict mathematical assumption that a stable fixed point exists.

Huginn discards these assumptions completely: - **Randomized Recurrence:** During pre-training, the recurrent budget $r$ is sampled dynamically from a log-normal Poisson distribution with a mean of 32. - **Truncated Backpropagation:** Gradients are backpropagated solely through the final 8 unrolled steps, mirroring truncated backpropagation through time (BPTT) in recurrent neural networks, but executed down the depth dimension. - **Non-Equilibrium Dynamics:** Examining latent trajectories demonstrates that simpler tokens converge toward stationary fixed points, whereas complex tokens drift into multi-step periodic orbits.

Subsequent investigations have tested whether explicit fixed-point solvers could be reintroduced into recursive architectures. In the Tiny Recursive Model (Jolicoeur-Martineau, 2025), replacing looped execution with a DEQ solver made training markedly slower and harmed model quality, reinforcing their thesis that "converging to a fixed point is not essential." While recent attractor architectures continue to explore implicit differentiation to outperform standard transformers, the consensus remains that recurrent depth treats depth as a flexible compute budget rather than a mathematical destination.

---

[05:31](https://www.youtube.com/watch?v=V03zLD8JKuU&t=331s) Lineage: From Universal Transformers to GPT-6

The hosts map the intellectual lineage that elevated looping from an obscure architectural tweak into a primary frontier pre-training axis:

```mermaid flowchart TD accTitle: Lineage of Looped Transformers accDescr: Historical progression from Universal Transformers in 2018 up to Astra and GPT-6 in 2026. UT["2018: Universal Transformers<br/>(Dehghani et al.)<br/>Recurrent layer + ACT halting"] DEQ["2019: Deep Equilibrium Models<br/>(Bai et al.)<br/>Infinite depth fixed-point limit"] PROG["2023: Algorithmic Foundations<br/>Giannou et al. (programmable computer)<br/>Yang et al. (iterative algorithms)"] SCALE["2025: Scaling Explosion<br/>• Huginn (Geiping et al., 3.5B)<br/>• Saunshi et al. (Latent Thoughts law)<br/>• Ouro (Zhu et al., 7.7T tokens)<br/>• Mixture-of-Recursions (Bae et al.)"] GPT6["2026: Frontier Deployment<br/>GPT-6 / Astra<br/>Constrained recurrent depth"]

UT --> DEQ UT --> PROG DEQ --> SCALE PROG --> SCALE SCALE --> GPT6 ```

- **2018 (Universal Transformers):** Dehghani et al. pioneer recurrently applying a single transformer layer alongside adaptive computation time (ACT) for dynamic halting. - **2019 (Deep Equilibrium Models):** Bai, Kolter, and Koltun take the looped block to its mathematical limit at infinity, using numerical solvers to find fixed points. - **2023 (Algorithmic Expressivity):** Theoretical works demonstrate the distinct computational advantages of looping: Giannou et al. prove looped transformers act as programmable computers, while Yang et al. demonstrate that loops learn iterative routines far better than fixed-depth networks. - **2025 (The Scaling Explosion):** - *Huginn* (Geiping et al.): Validates stable pre-training of a 3.5-billion-parameter recurrent model over 800 billion tokens. - *Saunshi et al.:* Formalize looped transformers as internal "latent thoughts" and derive the first depth scaling law. - *Ouro* (Zhu et al.): Pre-trains recurrent language models on a massive 7.7-trillion-token corpus. - *Mixture-of-Recursions* (Bae et al.): Implements dynamic token-level routing, allowing individual tokens to selectively choose their own recurrent loop count. - **2026:** Frontier model implementations culminate in OpenAI's GPT-6 (Astra).

---

[06:41](https://www.youtube.com/watch?v=V03zLD8JKuU&t=401s) Scaling Law 1: Test-Time Recurrence (Huginn)

The first scaling law governs what happens when you hold pre-trained weights constant and adjust the number of recurrent iterations $r$ during inference. Huginn evaluated test-time recurrence sweeps across $r \in [1, 64]$ on their 3.5-billion-parameter checkpoint trained on 800 billion tokens.

The performance gains scale dramatically with task difficulty: - **GSM8K (Chain-of-Thought):** Starts near 0% accuracy at $r = 1$, surpasses 25% around $r = 8$, and climbs into the 40s by $r = 32$. - **ARC-Challenge:** Rises steadily from 22% up to 44%. - **OpenBookQA:** Improves from 25% to approximately 41%.

Harder tasks continue consuming depth compute productively far longer than easier tasks. Geiping et al. plot a secondary horizontal axis labeled "materialized parameters": unrolling the 3.5B core block for 32 recurrent passes executes the forward FLOPs equivalent to a 50-billion-parameter dense model, all while retaining the GPU memory footprint of 3.5 billion stored weights.

To prevent simple prompts from wasting FLOPs, Huginn deploys a zero-shot early-exit heuristic. If the Kullback–Leibler (KL) divergence between token representations across two consecutive recurrent steps drops below $5 \times 10^{-4}$ (5 in 10,000), the loop halts immediately. Under this rule, straightforward high school mathematics problems exit early, whereas intricate ethical and moral dilemmas require an average of 3.5 additional recurrent iterations before stabilizing.

---

[07:58](https://www.youtube.com/watch?v=V03zLD8JKuU&t=478s) Scaling Law 2: Accuracy is Linear in Log Depth (Saunshi et al.)

Saunshi et al. (2025) investigate the expressivity of recurrent depth by comparing a 4-layer core block looped $L \in \{1, 2, 3, 6, 9, 12\}$ times against an unshared baseline transformer featuring $4L$ unique layers. While both setups share the identical effective depth ($4L$), the looped variant uses $L$ times fewer total parameters.

Their empirical evaluation establishes a universal log-depth scaling relationship that cleanly fits both architectures:

$$\text{Accuracy} = \alpha \log(\text{Effective Depth}) + \beta$$

The critical metric is the efficiency ratio $\alpha_{\text{loop}} / \alpha_{\text{base}}$, which measures how much of the unshared depth benefit is retained when weight-tying the layers: - **Math Word Problems & OpenBookQA:** The ratio reaches roughly 0.6, meaning looping provides 60% of the accuracy gains of adding unshared parameters. - **Reasoning Primitives:** The ratio hits 1.2—meaning looping a single block actually *outperforms* stacking fresh, unconstrained layers.

When an underlying task requires iterative algorithmic computation, parameter sharing acts as an inductive bias rather than a penalty. Saunshi et al. prove theoretically that a single-layer transformer block looped $\log N$ times can compose $N$ permutation group elements, demonstrating that looped execution faithfully simulates internal chain-of-thought reasoning without emitting intermediate tokens.

---

[09:11](https://www.youtube.com/watch?v=V03zLD8JKuU&t=551s) Scaling Law 3: Chinchilla for Loops (Ouro)

In the Ouro study, Zhu et al. extend the Chinchilla pre-training scaling laws by introducing an explicit term for recurrent depth. Fitting evaluation loss across five model sizes (53 million to 1.36 billion parameters) and recurrence counts ($T_{\max} \in \{1, 2, 4, 8\}$) over a 7.7-trillion-token budget, they model loss as:

$$L = E + \frac{A}{(N + t_1)^\alpha} + \frac{B}{(D + t_2)^\beta} + \frac{C}{(T_{\max} + t_3)^\gamma}$$

The fit achieves an $R^2$ of 0.96 with a positive exponent $\gamma$, confirming that increasing loop budgets yields predictable power-law reductions in pre-training perplexity.

However, Theo highlights the necessary caveat: at equal FLOP budgets, a dense transformer composed of fresh layers still outperforms a looped model by 1.5 to 4 benchmark points. This performance gap widens as the loop count increases and narrows as model parameter size expands. Looping confers sheer parameter efficiency, not "free" accuracy.

Under heavy pre-training, Ouro demonstrates that a 2.6-billion-parameter model looped 4 times matches standard 8-billion dense models on reasoning-heavy tasks. Probing the hidden activations reveals the underlying mechanism: recurrent loops do not expand factual memory storage—which remains fundamentally pegged at roughly 2 bits per parameter—but instead expand the computational manipulation capacity applied to those stored facts. Bae et al.'s Mixture-of-Recursions reinforces this at isoFLOP frontiers, showing that routing compute dynamically per token maximizes reasoning efficiency without bloating parameter counts.

---

[10:52](https://www.youtube.com/watch?v=V03zLD8JKuU&t=652s) The Exchange Rate: How Much is One Recurrence Worth? ($\phi = 0.46$)

Closing the loop on parameter efficiency, Theo cites an April 2026 paper (*Iso-Depth Scaling Laws*) that computes an exact exchange rate between repeated blocks and fresh layers. Fitting 116 experimental runs to a modified Chinchilla formula, the authors express effective parameter count by weighting recurrent blocks with an exponent $\phi$:

$$N_{\text{eff}} = N_{\text{unique}} + r^\phi \cdot N_{\text{recurrent}}$$

Across the benchmark sweeps, $\phi$ converges with tight confidence bounds to **0.46**: - At matched compute, one recurrence pass is worth roughly half of a unique, unshared transformer block. - Unrolling a single block through 4 loops behaves mathematically like 1.9 distinct layers, rather than 4 full layers.

This exchange rate explains why industrial implementations converge on conservative recurrence budgets: - **Nambe:** Loops 2 times. - **Loopy:** Loops 2 times. - **Ouro:** Loops 4 times. - **OpenAI (Astra):** Jakub Pachocki's comment that graph depth is within a factor of two implies an internal recurrence budget of 3 to 4 loops.

Industry practitioners deliberately cap recurrence where the marginal return curves flatten. Finally, Theo points out that $\phi$ is sensitive to the underlying optimization recipe: truncating backpropagation drops $\phi$ to 0.38, whereas introducing dense hyper-connections across the loop elevates $\phi$ to 0.65. Tuning this parameter exchange rate represents the precise optimization frontier that top labs now compete over.

[12:14](https://www.youtube.com/watch?v=V03zLD8JKuU&t=734s) What the Evidence Actually Shows: GPT-6 Rumors vs. Academic Limits

Mara pulls back to ground the discussion in what is verified versus what is pure speculation, asking point-blank what the community actually knows about GPT-6 versus what is merely being inferred. When stripped down to confirmed public facts, the record is remarkably sparse: there is only a single anonymous report citing "recurrent depth," a post from OpenAI Chief Scientist Jakub Pachocki bounding the compute-graph depth at roughly twice that of GPT-4, and a system card noting that a drop in chain-of-thought monitoring capability was not differentially caused by architecture. Everything else—the actual loop count, the training distribution over recurrence factor $r$, and the empirical scaling curve—remains completely hidden behind closed doors.

```mermaid flowchart TD accTitle: Recurrent Depth Inference Trade-off accDescr: Diagram illustrating the operational cost and latency trade-offs during test-time recurrence passes. subgraph LoopStage["Recurrent Block (Pass 1 to r)"] Attn["Recompute Attention"] Attn --> KV["Write KV Cache Entry"] KV --> Exit{"Early Exit Gate?"} end In["Token Representation"] --> Attn Exit -->|"Continue Loop"| Attn Exit -->|"Exit / Max Steps"| Out["Output Token"] ```

Theo points out that the published academic literature has severe boundaries of its own. Huginn represents an academic exploration evaluated on a single 3.5-billion-parameter checkpoint trained on 800 billion tokens without a learning rate cooldown. On the other end, Ouro pushed training out to 7.7 trillion tokens, but its model sizes capped out at 2.6 billion parameters. Simply put, no one in the open research community has ever published a frontier-scale scaling law incorporating an explicit loop term.

Compounding these empirical gaps is the unresolved question of serving and inference costs. Recurrent depth fundamentally trades stored parameter weights for wall-clock latency: every single loop recomputes attention across the sequence and writes a fresh entry into the KV cache. While architectural mitigations like early-exit routing gates and reusing only the deepest cache entry can ease the burden, running recurrent iterations inflates latency per token. The actual economic exchange rate between memory bandwidth, latency, and parameter reuse at frontier scale is completely unpublished. As Theo and Mara summarize, the only defensible baseline claim is that "GPT-6 loops. The loop is a proven third scaling axis at small scale, and the frontier numbers are unknown. That's the whole claim."

- **Public evidence on GPT-6 (Astra):** One anonymous report in *The Information*, an informal upper bound from Jakub Pachocki stating compute-graph depth is within twice GPT-4's depth, and an official system card clarification regarding chain-of-thought monitor degradation. - **Limits of open benchmarks:** Huginn trained a single 3.5B model on 800B tokens with no learning rate schedule cooldown; Ouro trained across 7.7T tokens but strictly on sub-3B scale models (topping out at 2.6B). - **Inference overhead:** Recurrence imposes severe compute penalties by continually appending to the KV cache and recomputing self-attention layers unless managed by early-exit gating heuristics.

---

[13:30](https://www.youtube.com/watch?v=V03zLD8JKuU&t=810s) Final Takeaways: Depth as a Per-Token Budget and the Mystery of $\varphi$

Wrapping up the inaugural episode, Mara provides her core synthesis of the architectural debate: recurrent depth is fundamentally not a Deep Equilibrium Model (DEQ). Rather than seeking a fixed point through root-finding algorithms and implicit differentiation, it is a practical looped transformer that treats computational depth as an allocateable compute budget spent dynamically per token—a budget governed in open papers by a predictable logarithmic law.

Theo frames the ultimate open research frontier around the parameter substitution metric $\varphi$ (phi). In recent iso-depth scaling literature, the exchange rate between a stored unique parameter and an unrolled, recurrent parameter hovers around $\varphi = 0.46$, meaning a looped iteration buys roughly half the utility of a brand-new unique weight block. Whether frontier-scale pre-training alters this exchange rate remains the primary secret guarded by frontier labs: "That's the number GPT-6 knows and we don't." With that lingering mystery, the hosts sign off with the show's signature motto: "Iterate until it makes sense."

## Notable quotes > [00:01](https://www.youtube.com/watch?v=V03zLD8JKuU&t=1s) constrained form of recurrent depth — The Information

> [01:09](https://www.youtube.com/watch?v=V03zLD8JKuU&t=69s) One is find the equilibrium. The other is iterate and stop when you've had enough. — Mara

> [02:02](https://www.youtube.com/watch?v=V03zLD8JKuU&t=122s) deep thinking — Theo

> [03:25](https://www.youtube.com/watch?v=V03zLD8JKuU&t=205s) converging to a fixed point is not essential. — Alexia Jolicoeur-Martineau

> [06:41](https://www.youtube.com/watch?v=V03zLD8JKuU&t=401s) materialized parameters — Theo

> [07:58](https://www.youtube.com/watch?v=V03zLD8JKuU&t=478s) latent thoughts — Saunshi et al.

> [12:14](https://www.youtube.com/watch?v=V03zLD8JKuU&t=734s) GPT-6 loops. The loop is a proven third scaling axis at small scale, and the frontier numbers are unknown. That's the whole claim. — Theo and Mara

> [13:30](https://www.youtube.com/watch?v=V03zLD8JKuU&t=810s) That's the number GPT-6 knows and we don't. — Theo

> [13:30](https://www.youtube.com/watch?v=V03zLD8JKuU&t=810s) Iterate until it makes sense. — Mara and Theo

## Mentioned OpenAI, GPT-6 (Astra), GPT-4, Jakub Pachocki, The Information, Mara, Theo, Loop State, Deep Equilibrium Models (DEQ), Huginn, Ouro, Universal Transformers, Tiny Recursive Model, Mixture-of-Recursions, Path Independent Equilibrium Models, Geiping et al., Saunshi et al., Zhu et al., Bai, Kolter & Koltun, Alexia Jolicoeur-Martineau, Bae et al., Giannou et al., Yang et al., GSM8K, ARC-Challenge, OpenBookQA, Nambe, Loopy, Leanim, MiniMax

## Video description OpenAI's GPT-6 (codename Astra) shipped on September 3, 2026, and one architecture phrase leaked out with it: "recurrent depth." One block of layers, reused in a loop. The first question everyone asked: is this just a deep equilibrium model wearing a new name?

Loop State is a two-host research podcast rendered as a Leanim board. Mara asks the questions, Theo brings the papers. We answer the DEQ question first, then trace the looped-transformer lineage and the three scaling laws people have measured for it.

Short answer: no. A DEQ solves for a fixed point z* = f(z*, x) with a root-finder and trains through it with implicit differentiation. Recurrent-depth transformers unroll a weight-tied block a finite number of times and backpropagate through the unrolled steps. No solver, no fixed point required. The tell: looped models show fixed points AND orbits, and a DEQ can only represent the former.

Chapters 00:00 Cold open: a frontier model that loops 01:09 The answer: not a DEQ, and why 02:02 One forward pass: prelude, recurrent block, coda (Huginn) 03:25 DEQ vs recurrent depth, precisely 05:31 The lineage: Universal Transformers to GPT-6

Chapters 00:00 Cold open: a frontier model that loops 01:09 The answer: not a DEQ, and why 02:02 One forward pass: prelude, recurrent block, coda (Huginn) 03:25 DEQ vs recurrent depth, precisely 05:31 The lineage: Universal Transformers to GPT-6 06:41 Scaling law 1: test-time recurrence (Huginn 3.5B) 07:58 Scaling law 2: accuracy is linear in log depth (Saunshi et al.) 09:11 Scaling law 3: a Chinchilla term for loops (Ouro, 7.7T tokens) 10:52 How much is one recurrence worth? φ = 0.46 12:11 What the evidence does not show 13:30 Sign-off

On GPT-6: the "recurrent depth" claim comes from a single anonymous source (The Information, Sept 1). OpenAI has not confirmed it. Jakub Pachocki said Astra's compute-graph depth is withinpapers, and we label it that way.

Papers discussed - Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (Huginn), Geiping et al. 2025: https://arxiv.org/abs/2502.05171 - Reasoning with Latent Thoughts: On the Power of Looped Transformers, Saunshi et al. 2025: https://arxiv.org/abs/2502.17416 - Scaling Latent Reasoning via Looped Language Models (Ouro), Zhu et al. 2025: https://arxiv.org/abs/2510.25741 - Iso-Depth Scaling Laws: How Much Is One Recurrence Worth?, 2026: https://arxiv.org/abs/2604.21106 - Deep Equilibrium Models, Bai, Kolter & Koltun 2019: https://arxiv.org/abs/1909.01377 - Mixture-of-Recursions, Bae et al. 2025: https://arxiv.org/abs/2507.10524 - Tiny Recursive Model, Jolicoeur-Martineau 2025: https://arxiv.org/abs/2510.04871 - Universal Transformers, Dehghani et al. 2018: https://arxiv.org/abs/1807.03819 - Path Independent Equilibrium Models, Anil et al. 2022: https://arxiv.org/abs/2211.09961

Key numbers - Huginn: 3.5B params, 800B tokens; GSM8K CoT goes from ~0% at r=1 to the 40s at r=32; r=32 is the compute of a ~50B dense model. - Saunshi: Acc = α log D + β; α_loop/α_base = 0.6 on math word problems, 1.2 on reasoning primitives. - Ouro: L = E + A/(N+t₁)^α + B/(D+t₂)^β + C/(T_max+t₃)^γ, R² = 0.96; 2.6B × 4 loops ≈ 8B dense on reasoning benchmarks; knowledge capacity stays ~2 bits/param. - Iso-depth law: φ = 0.46, so at matched FLOPs one recurrence is worth about half a unique block. That's why industry stops at 2 to 4 loops.

Production: two MiniMax voices at natural 1.0× speed, every turn measured before a single frame was drawn; rendered with Leanim (Lean 4). Source figures are rasterized from the arXiv e-prints and credited on screen.

#GPT6 #OpenAI #LoopedTransformer #RecurrentDepth #DeepEquilibrium #ScalingLaws #LLM #AIResearch #Huginn #Ouro

[Source](https://www.youtube.com/watch?v=V03zLD8JKuU)

中文简报

OpenAI GPT-6弃用平衡求解器，转向有限循环递归 循环深度架构将深度视为展开的测试期计算预算，而非不动点平衡问题，每次迭代大约带来独立层一半的性能增益。 主持人Theo和Mara指出，循环深度与深度平衡模型（DEQ）有着本质区别：循环Transformer并不依赖数值求解器和隐式微分来寻找静态不动点，而是在有限的计算预算内展开权重绑定的模块，从而天然支持周期轨道与不动点。结合Huginn等架构以及Saunshi等学者与Ouro项目的研究发现，他们展示了准确率随对数深度呈线性扩展，并在预训练期间符合清晰的Chinchilla式幂律。关键在于，等深度扩展基准确定了φ ≈ 0.46的兑换率，这意味着一次循环计算的效用大约是全新独立层的一半，这也解释了为何工业应用通常止步于两到四次循环。主持人最后对已证实的客观事实与推测进行了区分，指出尽管这些机制在较小规模下已得到验证，但前沿级别的扩展指标和推理权衡仍属严密保护的商业机密。 循环深度Transformer与深度平衡模型（DEQ）有着本质不同：它们通过反向传播执行有限的展开迭代，而不是依赖数值求根器求解不动点，这使其能够自然地表征多步周期轨道。 (1:09) 等深度参数兑换率收敛于φ = 0.46，意味着在FLOPs匹配的情况下，一次循环迭代带来的效用大约是全新独立层的一半，这解释了生产级架构为何将循环限制在2到4次迭代之间。 (10:52) 在测试期循环扫描中，Huginn的3.5B参数模型在进行32次循环计算时达到了50B稠密模型的前向FLOPs，并将GSM8K思维链准确率从接近0%提升至40%多。 (6:41) 来自Ouro项目（7.7T token）的预训练法则表明，循环迭代扩展了算法处理能力——使用2.6B循环架构在推理任务上媲美8B稠密模型——而事实性记忆容量仍受限于物理参数，约为每个参数2比特。 (9:11)

Reading summary

Theo and Mara argue that the 'recurrent depth' architecture reported in OpenAI's GPT-6 is not a Deep Equilibrium Model (DEQ), but a looped transformer that treats computational depth as an elastic per-token compute budget. Open academic literature shows that weight-tied recurrence obeys predictable logarithmic scaling laws, trading parameter storage efficiency against wall-clock inference latency. Architectural Divergence from Deep Equilibrium Models (1:09) Theo and Mara emphasize that recurrent depth differs fundamentally from DEQs despite both utilizing weight-tied layer blocks. DEQs rely on numerical root-finders like Broyden's method or Anderson acceleration and implicit differentiation to solve for an exact stationary fixed point ($z^* = f(z^*, x)$) at $O(1)$ memory cost. In contrast, looped transformers execute a bounded number of iterations ($s_i = R(E, s_{i-1})$) trained via truncated backpropagation. Empirical observations reveal that while simple tokens may converge to fixed points, complex tokens routinely fall into stable multi-step periodic orbits, which a DEQ root-finder cannot naturally resolve. To illustrate the mechanics, the hosts cite Geiping et al.'s 3.5-billion-parameter Huginn model (February 2025). Huginn processes inputs through a prelude into a latent tensor $E$, initializes recurrent state $s_0$ with Gaussian noise, and continuously re-injects $E$ alongside the prior state at each recurrent step before coda layers project to logits. Originating in work by Bansal and Anil, this fresh injection stabilizes recurrence against initial noise and prevents representation drift, behaving like an iterative optimization routine that re-queries input data at every pass. Empirical Scaling Laws and Log-Depth Expressivity (6:41) Empirical benchmarks demonstrate that looped recurrence provides significant accuracy gains on complex reasoning. In Huginn's test-time sweeps across iteration counts $r \in [1, 64]$, GSM8K chain-of-thought accuracy climbed from near 0% at $r = 1$ to over 25% at $r = 8$ and past 40% at $r = 32$, while ARC-Challenge rose from 22% to 44%. To manage computational expenditure, Huginn applies an early-exit heuristic that halts iteration when consecutive inter-step KL divergence drops below $5 \times 10^{-4}$. Unrolling the 3.5B core block 32 times matches the forward FLOPs of a 50B dense model while retaining the GPU memory footprint of 3.5B stored weights. Saunshi et al. (2025) further proved that accuracy scales linearly with log depth ($\text{Accuracy} = \alpha \log(\text{Effective Depth}) + \beta$). Comparing looped blocks to unshared baselines, the efficiency ratio $\alpha_{\text{loop}} / \alpha_{\text{base}}$ reached 0.6 on math word problems and OpenBookQA, but rose to 1.2 on reasoning primitives. For tasks requiring iterative computation, parameter sharing acts as a beneficial inductive bias, enabling a single transformer block looped $\log N$ times to theoretically compose $N$ permutation group elements without outputting tokens. The Parameter Exchange Rate and Industrial Optimization (10:52) Pre-training experiments reveal that looping delivers parameter efficiency rather than superior accuracy over unconstrained models. In the Ouro study, Zhu et al. evaluated recurrence over 7.7 trillion tokens across models up to 1.36 billion parameters, demonstrating power-law loss scaling ($R^2 = 0.96$). However, at matched FLOP budgets, dense transformers composed of fresh layers still outperform looped models by 1.5 to 4 benchmark points. Recurrence expands the computational manipulation applied to facts, but does not increase underlying factual memory storage, which remains bounded around 2 bits per parameter. An April 2026 study (*Iso-Depth Scaling Laws*) formalized this relationship through an effective parameter exponent $\phi$, where $N_{\text{eff}} = N_{\text{unique}} + r^\phi \cdot N_{\text{recurrent}}$. Across 116 runs, $\phi$ converged near 0.46, meaning one recurrent step provides roughly half the benefit of an unshared layer. Because marginal gains taper quickly, industrial implementations cap loops conservatively: Nambe and Loopy loop twice, Ouro loops four times, and OpenAI's Astra appears to target an internal budget of three to four loops based on Jakub Pachocki's statement that compute graph depth is within twice that of GPT-4. Frontier Speculation Versus Published Academic Limits (12:14) Mara and Theo stress that public evidence regarding GPT-6 remains extremely narrow: one anonymous report in *The Information*, Pachocki's comment bounding compute graph depth, and an official system card clarification stating architecture did not differentially cause monitor degradation. Astra's exact loop count, pre-training recurrence distributions, and scaling frontiers remain undisclosed. Published academic research also faces clear limits. Huginn evaluated a single 3.5B checkpoint without a learning-rate schedule cooldown, and Ouro capped models at 2.6B parameters. Furthermore, recurrence introduces severe inference costs: every additional loop recomputes self-attention and writes fresh entries into the KV cache. Mitigations like early-exit gates reduce waste, but the real-world economic exchange rate between memory bandwidth, latency, and parameter reuse at frontier scale has never been published. Theo and Mara conclude that recurrent depth redefines model scaling by treating depth as an elastic per-token budget rather than a fixed layer count. However, whether frontier-scale pre-training alters the academic exchange rate of $\phi \approx 0.46$ or overcomes inference latency bottlenecks remains an unanswered industry secret.

Source quotes

[0:01] The Information: constrained form of recurrent depth [1:09] Mara: One is find the equilibrium. The other is iterate and stop when you've had enough. [2:02] Theo: deep thinking [3:25] Alexia Jolicoeur-Martineau: converging to a fixed point is not essential. [6:41] Theo: materialized parameters [7:58] Saunshi et al.: latent thoughts [12:14] Theo and Mara: GPT-6 loops. The loop is a proven third scaling axis at small scale, and the frontier numbers are unknown. That's the whole claim. [13:30] Theo: That's the number GPT-6 knows and we don't. [13:30] Mara and Theo: Iterate until it makes sense.

Original transcript

[0:01] Welcome to Loop State. I'm Mara and [0:03] today we're talking about the one [0:05] architecture word that escaped from [0:08] OpenAI's new model, Theo. Set it up. [0:12] >> The model was reported under the code [0:14] name Astra and shipped on September 3rd [0:16] as GPT6. 2 days earlier, the information [0:20] reported from one anonymous source that [0:22] Astra uses a constrained form of [0:24] recurrent depth, one block of layers [0:27] reused in a loop. and OpenAI's answer [0:32] >> not a confirmation. Their chief [0:34] scientist said the depth of Astra's [0:36] computation graph is within a factor of [0:39] two of GPT4 and the launch post never [0:42] uses the word. So reported not [0:45] confirmed. Hold that thought. [0:47] >> But the moment the word got out, [0:49] everyone asked the same question. Is [0:51] this just a deep equilibrium model? A [0:53] DEEQ wearing a new name? That's the [0:56] question and we're going to answer it [0:58] first, not last. Then the papers this [1:02] idea actually comes from and the three [1:04] scaling laws people have measured for [1:06] it. [1:07] >> Let's do the answer. [1:09] >> No, recurrent defaf is not a deeq even [1:13] though both reuse one block. A deq is [1:16] defined by a fixed point. It solves for [1:19] a state that the block maps back onto [1:21] itself and it trains through that fixed [1:23] point with implicit differentiation. [1:26] >> And the looped transformer, [1:29] >> the looped transformer just runs the [1:31] block a fixed or sampled number of times [1:33] and back propagates through the unrolled [1:35] steps. No solver, no fixed point [1:37] required, no implicit function theorem. [1:40] It's trained to be useful after our [1:42] steps, not to be stationary. So, same [1:46] shape, different contract. One is find [1:48] the equilibrium. The other is iterate [1:50] and stop when you've had enough. [1:53] >> Exactly. And the nice twist is that the [1:55] looped models often drift toward fixed [1:57] points anyway without being told to. [1:59] We'll get there. First, the mechanism. [2:03] >> Walk me through one forward pass of a [2:05] recurrent depth transformer. [2:08] The public reference is the Hugan model [2:10] from Gying and colleagues February 2025. [2:15] >> Three pieces. A prelude of a couple of [2:17] layers embeds the tokens into a latent [2:19] vector, call it E. A recurrent block R [2:22] is the core. A cod of a couple of layers [2:24] unmbeds and predicts [2:26] >> and the loop. [2:29] >> You start the state zero as pure [2:31] Gaussian noise. Then for I from 1 to R, [2:35] the new state is R applied to the [2:37] embedding E and the previous state. [2:39] Finally, the KOD reads SR read as an [2:43] equation SI equals R of E and SI -1. [2:49] >> Noise as the starting state, not the [2:51] embedding. [2:53] >> The embedding is injected fresh at every [2:55] step instead. That's the deep thinking [2:57] trick from Bonsel and Anneil's work. It [3:00] stabilizes the recurrence and makes the [3:02] endpoint independent of where you [3:04] started. They call that path [3:05] independence. [3:07] >> So the loop is an iterative operator [3:12] that keeps rereading the input like [3:14] gradient descent, rereading the data [3:17] every step. [3:19] >> That's the author's own analogy. And the [3:21] depth R is a knob you can turn at test [3:24] time. [3:26] >> Okay. Okay, now do the DEEQ side [3:27] properly cuz I want the difference to be [3:30] sharp. [3:31] >> By Coulter and Coltune 2019, a DEEQ has [3:35] one weight tide block F. Instead of [3:38] running it L times, you solve the [3:40] equation Z star equals F of Z star and [3:43] X. A root finder does it. Broden or [3:47] Anderson acceleration, whatever [3:48] converges. And [3:51] the gradient [3:53] >> comes from the implicit function theorem [3:56] at the fixed point. You never store the [3:59] trajectory. So memory is constant in [4:02] depth. But you pay for a solver in every [4:05] forward and backward pass. And the whole [4:07] thing assumes the fixed point exists and [4:09] is stable. [4:11] >> And Hugan does none of that. [4:14] >> None of it. Hugan samples are from a log [4:17] normal poson distribution with mean 32 [4:20] during training and back propagates only [4:23] through the last eight steps truncated [4:25] unrolling like an RNN through time but [4:28] through depth. [4:29] >> Do the authors say this out loud? [4:33] >> They do. Their words equilibrium methods [4:36] solve the direct problem and their work [4:38] suggests truncated unrolling is a [4:40] scalable alternative. Then they look at [4:43] the latent trajectories and find that [4:45] easy tokens converge to fixed points and [4:48] hard tokens fall into orbits. [4:50] >> Has anyone tried putting the solver back [4:52] in? [4:53] >> Yes. The tiny recursive model swapped [4:56] its loop for a deeq solver and found it [4:58] slower and worse. Their line is that [5:01] converging to a fixed point is not [5:03] essential. And this spring attractor [5:06] models put implicit differentiation back [5:08] and beat bigger transformers. So the [5:11] field is still arguing. [5:14] >> So the DEEQ is a destination, recurrent [5:17] depth is a budget, [5:19] >> and the orbits are the tell. A DEEQ [5:22] can't represent an orbit. It only has [5:25] one answer per input. That's the [5:27] cleanest single reason the two are [5:29] different models. [5:31] >> Give me the family tree. Where does [5:33] looping a block come from? [5:35] >> Universal transformers Deani and [5:37] colleagues 2018. one transformer layer [5:40] applied recurrently with adaptive [5:42] halting. That's the ancestor. [5:45] >> Then DEEQ in 2019 is the branch that [5:48] says take the loop to infinity and solve [5:51] it. [5:52] >> Right? Then 2023, GO and colleagues show [5:55] a looped transformer can act as a [5:57] programmable computer and Young and [5:59] colleagues show loops learn iterative [6:01] learning algorithms better than fixed [6:02] depth. [6:04] >> And 2025 is the year it scales. three [6:09] papers gying hugan at three and a half [6:11] billion parameters soni and colleagues [6:14] on looped transformers as latent [6:16] thoughts with the first depth scaling [6:18] law andro from zoo and colleagues which [6:21] pre-trains looped models on 7.7 trillion [6:24] tokens [6:25] >> plus mixture of recursions which lets [6:28] each token pick its own loop count [6:31] >> by and colleagues also 2025 and then a [6:35] frontier lab ships it that's the ark [6:38] from a halting trick to a pre-training [6:40] axis. [6:42] >> Scaling law number one. What happens [6:45] when you just turn the loop knob at test [6:47] time? [6:48] >> Hugan's figure 1. 3 1/2 billion [6:50] parameters trained on 800 billion [6:53] tokens. On the x-axis, the number of [6:56] recurrences at test time from 1 to 64. [6:59] And the curves [7:02] >> GSM 8K with Chain of Thought starts at [7:05] essentially zero accuracy at one [7:07] recurrence, crosses 25% around 8, and [7:10] climbs into the 40s by 32. Ark challenge [7:13] climbs from 22 to 44. Open book UA from [7:16] 25 to about 41. [7:19] >> So the hard task keeps eating compute [7:23] the longest. [7:25] >> Yes. And the authors put a second axis [7:28] on top materialized parameters. 32 [7:31] recurrences of that block is the compute [7:33] of a 50 billion parameter fixed depth [7:35] model coming out of 3 1/2 billion stored [7:38] weights. [7:39] >> What does the model do when it doesn't [7:42] need all that [7:44] >> zero shot early exit? If the KL [7:46] divergence between two successive states [7:48] drops below 510,000 [7:51] stop. High school math exits early. [7:54] Moral scenarios take about three and a [7:56] half more steps on average. [7:58] >> Scaling law number two, the one from [8:01] Sani and colleagues. [8:03] >> They pre-train a fourlayer model looped [8:06] L times for L in 1 2 3 6 9 12 against a [8:13] baseline with 4 L fresh layers. Same [8:16] effective depth, but the looped model [8:18] has L times fewer parameters. [8:21] >> And the law Accuracy equals alpha * log [8:26] of effective depth plus beta. It fits [8:29] both families. The interesting number is [8:31] the ratio of the looped alpha to the [8:34] baseline alpha, [8:35] >> which tells you how much of the depth [8:38] benefit you keep when the layers are [8:40] shared. [8:41] >> On math, word problems, and openbook QA, [8:44] the ratio is about 0.6. On reasoning [8:47] primitives, it's 1.2. Looping beats, [8:50] adding parameters. So the more the task [8:52] looks like iterating an algorithm, the [8:55] less the loop costs you. [8:57] >> That's the reading. The same paper [8:59] proves a one layer block looped log end [9:02] times can compose n groupoup elements [9:04] and that looped models can simulate [9:06] chain of thought. Depth via loops is [9:09] real depth for reasoning. [9:11] >> Um scaling law three. This is the [9:14] pre-training one from Oro. They fit [9:17] total loss as a chinchilla style sum. An [9:20] irreducible e plus a / n to the alpha [9:23] for parameters plus b over d to the beta [9:26] for data plus a new term c over t max to [9:29] the gamma for the maximum recurrent [9:32] step. [9:32] >> How well does it fit? [9:34] >> R^ 2 of 0.96 across five model sizes [9:38] from 53 million to 1.36 billion and loop [9:42] counts of 1 2 4 and 8. The gamma is [9:47] positive. More loops, lower loss [9:50] predictably. [9:51] >> But there's a catch because I've seen [9:55] the table. [9:57] >> The catch is honest. At the same loop [9:59] count, a standard transformer with fresh [10:02] layers still beats the looped one by 1 [10:04] and 1/2 to four points. The gap grows [10:06] with loops and shrinks with model size. [10:09] >> So looping is parameter efficiency, not [10:13] free accuracy. [10:15] Right. Their headline, a 2.6 billion [10:18] looped model on 7.7 trillion tokens [10:21] matches 8 billion dense models on [10:24] reasoningheavy benchmarks. And a [10:26] knowledge probe shows why looping does [10:29] not raise capacity. Still about two bits [10:32] per parameter. It raises manipulation [10:35] >> and mixture of recursions. [10:39] >> Same story at isoflops. The recursive [10:42] paro frontier sits below vanilla at [10:44] smaller sizes and token level routing [10:46] pushes it further. Loops buy you compute [10:49] per token, not memory per token. [10:52] >> So if loops aren't free, what's the [10:55] exchange rate? Is there a law that says [10:57] how much a repeated block is worth [10:59] compared to a fresh one? There is now [11:02] Schwithm Rukert and Kais April 2026 fit [11:06] a chinchilla style law where the [11:08] parameter term is the once used [11:09] parameters plus r the^ 5 * the recurrent [11:13] parameters. [11:14] >> So um fi is the price of a loop [11:18] >> fi came out at 0.46 with a tight [11:21] confidence interval over 116 runs. Read [11:25] it as at matched compute, one recurrence [11:28] is worth about half a unique block. Four [11:30] loops of a block act like 1.9 blocks, [11:34] not four. [11:35] >> That's a lot less than four. [11:38] >> And it explains the industry number. [11:40] Nambbe loops twice, loopy loops twice, [11:44] or loops four times, and the inference [11:46] from OpenAI's factor of two comment is [11:49] three or four. Everyone stops where Fee [11:52] says the returns flatten. Does the [11:55] training recipe move fi? [11:58] >> Yes. Truncated back prop drops it to [12:00] 0.38. Hyper connections on the loop push [12:03] it to 0.65. [12:05] So fi is a property of the recipe and [12:08] that's the number a frontier lab would [12:10] optimize. [12:12] >> Boundary time. [12:14] Um what do we actually know about GPT6 [12:18] and what are we inferring? What's public [12:20] is one anonymous report of recurrent [12:23] depth, a chief scientist postbounding [12:25] the compute depth at twice GPT4, and a [12:28] system card that says the drop in chain [12:31] of thought monitor is not differentially [12:34] due to architecture, no loop count, no [12:37] training distribution over R, no scaling [12:39] curve [12:40] >> and the papers have limits of their own. [12:43] >> Hugan is an academic run. 800 billion [12:46] tokens, no learning rate cool down, one [12:49] model. Oro is 7.7 trillion tokens but [12:52] tops out at 2.6 billion parameters. [12:56] Nobody has published a frontier scale [12:58] law with a loop term. [12:59] >> Then there's the serving cost. [13:02] >> Every recurrence recomputes attention [13:04] and writes a KV entry. Early exit gates [13:07] and reusing the deepest cache entry [13:09] help, but recurrent depth trade stored [13:11] weights for wall clock latency and the [13:13] exchange rate at frontier scale is [13:15] unpublished. [13:16] >> So the safe claim is GPT6 loops. The [13:21] loop is a proven third scaling axis at [13:24] small scale and the frontier numbers are [13:27] unknown. [13:28] >> That's the whole claim. [13:30] My takeaway, recurrent depth isn't a [13:34] deeq. It's a transformer that treats [13:37] depth as a budget you spend per token. [13:40] And the open papers say that budget [13:42] follows a log law. [13:44] >> My open question at Frontier Scale, what [13:47] is FI? The exchange rate between a [13:49] stored parameter and a repeated one is [13:51] 0.46 in the open literature. That's the [13:55] number GPT6 knows and we don't. This has [13:59] been loop state. Iterate until it makes [14:02] sense.