The Goose and the Heron: Why RWKV Forgets, and How ROSA Brings It Back

Xiaol.x

AI-assisted summary of The Goose and the Heron: Why RWKV Forgets, and How ROSA Brings It Back — original source. Check the original for full context.

Source published: 2026-09-29

Key ideas

RWKV-7's fixed-size recurrent state causes complete long-range recall failure, scoring 0/20 on needle retrieval at 16,000 tokens even with parameter scaling up to 1.5B. (0:00)
Pairing a frozen RWKV-7 with ROSA—a parameter-free, CPU-based suffix automaton from RWKV-8—via a 100k-parameter output 'ear gate' boosts 32k-token needle recall up to 95–100%. (1:54)
Successful gate training requires a synthetic copy-paste curriculum; natural prose training leaves the gate timid, and input-side residual stream injection degrades code perplexity. (2:34)
To combine behavioral formatting with verbatim memory, researchers propose splitting the work: state tuning (~1.5M initial values) controls 'how' while ROSA supplies 'what', though preliminary scene-break tests remain inconclusive and uncombined. (6:10)

Detailed notes

## Background Published by Xiaol.x on YouTube on September 29, 2026, this hand-painted explainer examines long-context retention in RWKV architectures. The show notes describe how a frozen RWKV-7 model loses ancient context due to its fixed-size memory satchel, how BlinkDL's parameter-free ROSA suffix automaton restores verbatim retrieval, what alternative approaches failed, and an untested proposal to combine state tuning with ROSA.

## TL;DR RWKV-7's constant-sized recurring state ensures fast inference but causes severe memory decay, resulting in zero recall for passcodes planted 16,000 tokens upstream. By pairing the frozen model with ROSA—a zero-parameter CPU suffix automaton—and training a tiny 100k-parameter 'ear gate' on a synthetic copy-paste curriculum, needle recall reaches 95% to 100% at up to 64,000 tokens. Interventions like input-side injection, hashed phrasebooks, and soft semantic memory failed, prompting the author to propose splitting labor: using state tuning to govern behavioral habits ('how') while ROSA supplies exact historical memory ('what'). The video candidly acknowledges that this hybrid approach remains untested, closing with a disciplined 2x2 ablation plan to determine whether state tuning actually enhances ROSA.

## Atmosphere & style The presentation possesses a poetic, meditative, and artisanal quality, blending an allegorical fable of a goose swimming a river alongside a watchful heron with concrete machine learning engineering. The measured synthetic narration and hand-painted visual aesthetic create a tranquil, reflective viewing experience devoid of commercial hype. It strikes an unusual balance between lyrical storytelling and strict empirical modesty, concluding with refreshing candor about preliminary shortcomings and the need for fair, decisive testing.

## Bulletin - [00:00](https://www.youtube.com/watch?v=x_RLOw2luNM&t=0s) RWKV-7's fixed-size recurrent state causes complete long-range recall failure, scoring 0/20 on needle retrieval at 16,000 tokens even with parameter scaling up to 1.5B. - [01:54](https://www.youtube.com/watch?v=x_RLOw2luNM&t=114s) Pairing a frozen RWKV-7 with ROSA—a parameter-free, CPU-based suffix automaton from RWKV-8—via a 100k-parameter output 'ear gate' boosts 32k-token needle recall up to 95–100%. - [02:34](https://www.youtube.com/watch?v=x_RLOw2luNM&t=154s) Successful gate training requires a synthetic copy-paste curriculum; natural prose training leaves the gate timid, and input-side residual stream injection degrades code perplexity. - [06:10](https://www.youtube.com/watch?v=x_RLOw2luNM&t=370s) To combine behavioral formatting with verbatim memory, researchers propose splitting the work: state tuning (~1.5M initial values) controls 'how' while ROSA supplies 'what', though preliminary scene-break tests remain inconclusive and uncombined.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=x_RLOw2luNM&t=0s) RWKV-7 uses a fixed-size recurrent state that keeps inference fast and constant-memory, but inevitably discards long-range factual details. - [00:00](https://www.youtube.com/watch?v=x_RLOw2luNM&t=0s) At 16,000 tokens upstream, RWKV-7 fails completely (0/20) on needle-in-a-haystack passcode retrieval, despite language modeling loss remaining completely flat. - [00:35](https://www.youtube.com/watch?v=x_RLOw2luNM&t=35s) Scaling the base architecture to 1.5B parameters does not resolve early-context memory decay, showing the failure is architectural rather than capacity-bound. - [01:05](https://www.youtube.com/watch?v=x_RLOw2luNM&t=65s) ROSA (Rapid Online Suffix Automaton) acts as an exact text ledger on CPU with zero parameters and ~12 µs/token latency, achieving 20/20 needle recall. - [01:35](https://www.youtube.com/watch?v=x_RLOw2luNM&t=95s) ROSA cannot generate text independently because its greedy next-token prediction fails roughly 90% of the time on unstructured prose. - [01:54](https://www.youtube.com/watch?v=x_RLOw2luNM&t=114s) A tiny learned 'ear gate' of ~100k parameters arbitrates between RWKV-7 logits and ROSA proposals based on match length, frequency, and base model confidence. - [02:34](https://www.youtube.com/watch?v=x_RLOw2luNM&t=154s) A synthetic copy-paste curriculum is necessary to train the gate, as exact repeated phrases are too rare in natural prose to build gating confidence. - [03:15](https://www.youtube.com/watch?v=x_RLOw2luNM&t=195s) With the copy-paste trained gate, needle recall at 32k tokens jumps to 85–100% and holds at 90%+ out to 64k tokens, even amidst decoy codes. - [03:32](https://www.youtube.com/watch?v=x_RLOw2luNM&t=212s) Injecting ROSA representations directly into the input residual stream degraded code perplexity from 2.9 to 3.5 nats/token. - [04:05](https://www.youtube.com/watch?v=x_RLOw2luNM&t=245s) Static hashed phrasebooks (Engram) and soft multi-hop continuous attention fail to match the reliability of exact string matching via suffix automata. - [05:49](https://www.youtube.com/watch?v=x_RLOw2luNM&t=349s) ROSA only handles exact verbatim copies; it cannot infer implicit structures, enforce JSON schemas, or guide behavioral tasks. - [06:10](https://www.youtube.com/watch?v=x_RLOw2luNM&t=370s) State tuning freezes all model weights and tunes only ~1.5M initial state scalars, encoding behavioral formatting ('how') without burdening context capacity. - [06:56](https://www.youtube.com/watch?v=x_RLOw2luNM&t=416s) Separating 'how' (state-tuned initial satchel) from 'what' (ROSA exact external recall) is proposed as a synergistic long-context architecture. - [07:42](https://www.youtube.com/watch?v=x_RLOw2luNM&t=462s) Initial empirical evaluations of state tuning alone yielded an F1 of 0.22 on 4k scene-break detection, trailing a standard 0.24 classifier baseline. - [08:10](https://www.youtube.com/watch?v=x_RLOw2luNM&t=490s) State tuning and ROSA have not yet been evaluated simultaneously, leaving the combined hybrid architecture an open hypothesis awaiting a 2x2 ablation test.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=x_RLOw2luNM&t=0s) A Goose That Forgets: RWKV-7's Memory Bottleneck

The video opens with an allegorical portrait of RWKV-7 as a wild goose swimming down an endless river of text. As an RNN-based architecture, the model carries everything it knows in a single, compact satchel of memory—its hidden state—which never expands regardless of how much text flows past. This fixed-size state is the exact reason RWKV is lightning-fast and computationally efficient, but it is also the fundamental reason it forgets.

To demonstrate this structural limitation, a simple needle-in-a-haystack test is run: a single arbitrary sentence is planted early in the stream: "The secret code for Heron is 734912." When prompted for this passcode 16,000 tokens later, the goose recalls it exactly zero times out of twenty attempts. At 32,000 tokens, the decay is even starker; only a fact placed right at the tail end of the context survives, and even then only about one-third of the time. Scaling up model capacity provides no relief: a larger 1.5-billion-parameter goose fares no better on early-context recall. Crucially, the model's language modeling loss remains completely flat between 1,000 tokens and 32,000 tokens, demonstrating that the failure is not linguistic: "It has not forgotten how to write. It has forgotten what it read."

- **The Memory Satchel:** RWKV-7 relies on a constant-sized recurring state, keeping inference speed high at the cost of long-range factual retention. - **Passcode Recall Rate:** A code planted 16,000 tokens upstream yields a 0/20 success rate. - **Scale Invariance:** Upgrading to the 1.5B parameter variant does not resolve the retrieval failure.

---

[01:05](https://www.youtube.com/watch?v=x_RLOw2luNM&t=65s) Enter the Heron: ROSA and the Exact Ledger

To remedy the goose's selective amnesia, the video introduces the heron: ROSA (Rapid Online Suffix Automaton), an algorithmic companion developed by BlinkDL for the upcoming RWKV-8 architecture. While the goose swims forward holding its small satchel, the heron flies overhead maintaining an exact, uncompressed ledger of the entire text stream. At every single token step, ROSA queries its deterministic data structure with one straightforward question: "The last time this exact run of words appeared what came next?"

Because ROSA is a suffix automaton rather than a neural network, it contains zero parameters and requires no floating-point arithmetic. It processes the stream on a standard CPU at an ultra-low latency of roughly 12 microseconds per token. On synthetic needle retrieval benchmarks, this pure string-matching mechanism achieves a flawless 20 out of 20 score across all token depths. However, ROSA cannot operate as an independent language generator: on ordinary prose, its greedy next-token prediction is incorrect nearly nine times out of ten. As the narrator notes, "A Heron that always talks is useless. The goose has to learn when to listen."

- **ROSA Definition:** Rapid Online Suffix Automaton, introduced in the context of BlinkDL's RWKV-8 research. - **Computational Footprint:** Zero learned weights, zero floating-point operations, executing at approximately 12 µs/token on CPU. - **Needle vs. Prose Disparity:** Scores 100% (20/20) on exact needle lookups, but fails ~90% of the time when predicting unstructured prose.

---

[01:54](https://www.youtube.com/watch?v=x_RLOw2luNM&t=114s) What Works: A Gate at the Ear and Copy-Paste Curriculum

The architecture that succeeds leaves the underlying RWKV-7 goose completely frozen and perches the heron on its shoulder. Rather than altering internal representations, a tiny learned gate containing roughly 100,000 parameters is placed directly at the model's output—whispering into the "goose's ear" to blend ROSA's string-matched candidate directly into the next-token probability distribution. The gating mechanism evaluates three specific signals: the length of the matching suffix, how many times that sequence has appeared upstream, and the baseline confidence the goose already assigns to that token.

```mermaid flowchart LR accTitle: ROSA Gating Architecture at the Ear accDescr: Diagram showing frozen RWKV-7 and CPU-based ROSA suffix automaton feeding into an ear gate for next-token prediction.

subgraph Stream ["Input River"] Tokens["River of Tokens"] end

subgraph Models ["Dual Processing"] Goose["Frozen RWKV-7\n(Fixed Hidden State)"] Heron["ROSA Suffix Automaton\n(Exact Ledger on CPU)"] end

subgraph Integration ["Output ear gate (~100k params)"] Gate["Learned Gate\n(Match len, count, goose prior)"] Output["Next Token Prediction"] end

Tokens --> Goose Tokens --> Heron Goose -->|"Base Logits"| Gate Heron -->|"Candidate Match"| Gate Gate --> Output ```

Initially, training this gate on ordinary natural text causes it to remain overly timid: long repeated phrases seen only once are extraordinarily rare in clean prose, so the gate never develops confidence in ROSA's isolated predictions. At 32,000 tokens, naive gate training matches the baseline model's failure. The breakthrough requires a synthetic "copy-paste curriculum": in 50% of the training sequences, earlier spans of text are artificially duplicated and pasted further downstream. Faced with guaranteed rewards for identifying long repetitions, the gate learns to swing wide open when a familiar sequence appears, allowing the secret code to pass through before snapping shut immediately afterward.

- **Ear Gate Scale:** Approximately 100,000 trainable parameters arbitrating between base model logits and ROSA proposals. - **Needle Recall at 32k Tokens:** Improves dramatically to 95%, 85%, and 100% across various insertion depths. - **Robustness to Distractors:** Retains 95% or higher accuracy even when four decoy passcodes are embedded in the haystack. - **Context Generalization (64k Tokens):** Evaluated at 64,000 tokens (eight times the training context), recall holds at 90% or higher. - **Perplexity on Long Copies:** Copying text from 30,000 tokens prior reduces cross-entropy from >3 nats/token down to 0.16 nats/token. - **Larger Model Impact:** On the 1.5B parameter RWKV-7 processing code, overall perplexity drops ~4%, expanding to a 5% improvement beyond 16,000 tokens.

---

[03:32](https://www.youtube.com/watch?v=x_RLOw2luNM&t=212s) What Fails: Throats, Phrasebooks, Prose, and Paraphrasing

The video contrasts the successful "ear gate" with four alternative configurations that failed, noting that each misstep attempted to insert the heron's notes somewhere other than the goose's ear.

| Failed Approach | Mechanism / Design | Outcome & Root Cause | |:--- |:--- |:--- | | **Input-Side Injection** ("Down the Throat") | BlinkDL's native proposal: injecting ROSA retrieval embeddings directly into the residual stream. | Perplexity on source code degraded from 2.9 to 3.5; needle recall plummeted. (Caveat: loss was still trending down when training compute was exhausted). | | **Engram Hashed Phrasebook** | 17M parameter lookup table indexing hashed token pairs and triples. | Static prior with no awareness of the current document; resulted in 0% improvement on verbatim copy benchmarks. | | **Standard Prose Integration** | Deploying the ear gate over conventional fiction / narrative text. | Baseline model already predicts natural text well; evaluation loss improved only in the third decimal place due to zero remaining headroom. | | **Soft Multi-Hop Memory** | Replacing exact string matching with continuous attention over cached hidden states for paraphrased prompts. | On novel scene-break detection, 1-hop memory performed no better than a baseline adapter (flipping ranking across seeds); 2-hop performed worse. |

The central takeaway from these experiments forms a clear boundary: exact, verbatim retrieval grafted externally onto a frozen model via a calibrated gate works exceptionally well. Conversely, any intervention that attempts to modify internal representations on a limited compute budget, or demands semantic comprehension of paraphrased text rather than strict string matching, breaks down.

---

[05:45](https://www.youtube.com/watch?v=x_RLOw2luNM&t=345s) The Packed Satchel: Splitting "How" from "What"

Because ROSA is strictly an exact copy mechanism, it can only highlight words that have literally appeared upstream. It is fundamentally incapable of teaching a model behavioral nuances: identifying where an implicit scene break occurs, enforcing structured JSON schemas, or adhering to complex task instructions. To solve this without fine-tuning the entire model, the video proposes a separation of labor: "split the work."

The proposed method leverages *state tuning*. Every neural weight inside the goose remains completely frozen; instead, backpropagation optimizes solely the initial hidden state vectors—roughly 1.5 million scalar values representing a small baseline tensor for every attention head and layer. This initialized state acts as a pre-packed satchel that provides task instructions and formatting conventions before the river of text begins. While RWKV's running updates gradually overwrite this initial state over tens of thousands of tokens, the packed satchel only needs to guide the model's structural style ("how"), while ROSA's external suffix ledger reliably provides historical facts ("what") all the way out to 64,000 tokens.

```mermaid flowchart TD accTitle: Division of Labor: State Tuning vs ROSA accDescr: State tuning dictates the behavioral format, while ROSA provides long-context factual recall through the ear gate.

subgraph Start ["Trip Origin"] Init["State Tuning (~1.5M values)\nPre-packed Initial Satchel"] end

subgraph Journey ["Downstream Execution"] Behavior["Controls the 'HOW':\nFormatting, Schema, Habits"] ROSA["ROSA Ledger Controls the 'WHAT':\nExact Verbatim Recall up to 64k"] end

subgraph Ear ["Output Stage"] Gate["Gated Ear Whisper"] Result["Correctly Formatted Long-Context Output"] end

Init --> Behavior Behavior --> Gate ROSA --> Gate Gate --> Result ```

- **State Tuning Footprint:** Approximately 1.5 million trainable parameters, confined strictly to the initial layer-head states. - **The "How" vs. "What" Split:** State tuning instills formatting habits and output behavior; ROSA supplies verbatim factual memory. - **Safety from Degradation:** The satchel does not need to preserve ancient facts over long sequences, liberating it from the context-length bottleneck.

---

[07:42](https://www.youtube.com/watch?v=x_RLOw2luNM&t=462s) The Honest Verdict: Early Benchmarks and the Fair Test

The presentation concludes with total transparency regarding the status of the hypothesis: the packed satchel combined with ROSA remains an untested bet. Preliminary evaluations on identifying novel scene transitions within 4,000-token windows yielded underwhelming metrics under a brief 500-step training regime across two random seeds. State tuning achieved an F1 score of only 0.22, trailing a basic classification head which scored 0.24. Furthermore, introducing soft semantic memory introduced severe variance, causing the two seeds to drift wildly apart (0.26 versus 0.19).

The video cautions that this early experiment was severely limited—testing only a single learning rate and, crucially, evaluating state tuning in isolation without ROSA attached. To rigorously validate whether the hybrid architecture holds merit, the project outlines a disciplined roadmap: first conducting a proper learning rate sweep restricted entirely to development splits, followed by an unambiguous 2x2 ablation benchmark on a synthetic task requiring both strict structural adherence and retrieval across 30,000 tokens.

- **Scene Boundary Baseline:** State tuning scored 0.22 F1 versus 0.24 F1 for a simple classifier baseline across 500 steps. - **Seed Volatility:** Soft semantic memory caused significant stability issues between seeds (0.26 vs. 0.19). - **The Four-Way Test Plan:** 1. Base model alone (Neither satchel nor heron). 2. State-tuned satchel only. 3. ROSA heron only. 4. State-tuned satchel combined with ROSA heron. - **The Elimination Rule:** "If both cannot beat the heron alone, the satchel goes." The video closes as the goose wades back into the stream with its pre-packed satchel and the heron perched at its ear, awaiting empirical confirmation.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=x_RLOw2luNM&t=0s) A frozen RWKV-7 swims a river of text with one small satchel of memory that never grows.

> [00:00](https://www.youtube.com/watch?v=x_RLOw2luNM&t=0s) The secret code for Heron is 734912.

> [00:00](https://www.youtube.com/watch?v=x_RLOw2luNM&t=0s) It has not forgotten how to write. It has forgotten what it read.

> [01:05](https://www.youtube.com/watch?v=x_RLOw2luNM&t=65s) The last time this exact run of words appeared what came next?

> [01:05](https://www.youtube.com/watch?v=x_RLOw2luNM&t=65s) A Heron that always talks is useless. The goose has to learn when to listen.

> [01:54](https://www.youtube.com/watch?v=x_RLOw2luNM&t=114s) Keep the goose frozen and train only a tiny gate at its ear, using a copy-paste curriculum.

> [03:32](https://www.youtube.com/watch?v=x_RLOw2luNM&t=212s) Down the Throat

> [05:49](https://www.youtube.com/watch?v=x_RLOw2luNM&t=349s) The heron only copies; it can't teach behaviour. So the bet is to split the work.

> [07:42](https://www.youtube.com/watch?v=x_RLOw2luNM&t=462s) The honest part: this bet has not been shown yet.

> [07:42](https://www.youtube.com/watch?v=x_RLOw2luNM&t=462s) If both cannot beat the heron alone, the satchel goes.

## Mentioned Xiaol.x, BlinkDL, JohnHeibel, RWKV, RWKV-7, RWKV-8, RWKV-Long-ROSA, ROSA (Rapid Online Suffix Automaton), Engram, p5.js, p5.brush, MiniMax TTS, PDoomVideo

## Video description A frozen RWKV-7 swims a river of text with one small satchel of memory that never grows. That's why it's fast, and why it forgets. This painted explainer walks through RWKV-Long-ROSA: what worked, what failed, and the next bet, which is not yet tested.

PART 1: The Goose and the Heron Plant a code early in a long document and the goose recalls it 0 times out of 20. Then comes the heron: ROSA, the Rapid Online Suffix Automaton from BlinkDL's RWKV-8. It keeps an exact ledger of the whole river, has no parameters, and runs on a CPU. Keep the goose frozen and train only a tiny gate at its ear, using a copy-paste curriculum. Needle recall at 32k tokens reaches 95 / 85 / 100%. What failed: pushing ROSA's notes in on the input side, an Engram hashed phrasebook, ordinary prose, and paraphrased (semantic) questions.

PART 2: The Packed Satchel The heron only copies; it can't teach behaviour. So the bet is to split the work. • State tuning: keep every weight frozen and learn only the starting state (~1.5M numbers). It carries the "how": format, habits, the task. • ROSA: exact long-range recall. It carries the "what", with recall of 1 / .9 / 1 at 64k tokens. The honest part: this bet has not been shown yet. On a short 4k scene-break test, state tuning scored .22 F1 against .24 for a plain classifier. Adding semantic memory made the two seeds disagree (.26 vs .19). State tuning and ROSA have never been tested together. The video ends with the fair test that would settle it.

CHAPTERS 0:00 A goose that forgets 1:05 Enter the heron (ROSA) 1:54 What works: a gate at the ear 3:32 What fails 5:00 The ledger: works / fails / next 5:45 Part 2: The Packed Satchel 5:49 The heron only copies 6:10 The bet: state tuning 6:56 Split the work: how vs what 7:42 The honest part: not yet 8:43 Off again

LINKS Code & results: https://github.com/xiaol/RWKV-Long-ROSA ROSA comes from BlinkDL's RWKV-8. Visual style inspired by https://github.com/JohnHeibel/PDoomVideo

Hand-painted in p5.js + p5.brush. Narration is AI voice (MiniMax TTS).

#RWKV #LongContext #MachineLearning #LLM #StateSpaceModels #AIResearch

[Source](https://www.youtube.com/watch?v=x_RLOw2luNM)

中文简报

鹅与苍鹭：RWKV为何遗忘，以及ROSA如何帮其找回记忆 RWKV-7的固定大小循环状态确保了快速推理，但会导致严重的记忆衰减，对于植入在16,000个token之前的密码，召回率为零。通过将冻结的模型与ROSA（一种零参数CPU后缀自动机）配对，并在合成复制粘贴课程上训练一个仅10万参数的微型“耳门”，在长达64,000个token时的“大海捞针”召回率可达95%至100%。输入端注入、哈希短语手册和软语义记忆等干预措施均告失败，促使作者提出分工方案：使用状态微调来控制行为习惯（“如何做”），而由ROSA提供精确的历史记忆（“是什么”）。视频坦承这种混合方法尚未经过充分测试，并以一份严谨的2x2消融实验计划作结，以确定状态微调是否真能增强ROSA。 RWKV-7的固定大小循环状态会导致完全的长距离召回失败，即使参数规模扩大到1.5B，在16,000个token下的寻针检索得分仍为0/20。 (0:00) 将冻结的RWKV-7与来自RWKV-8的无参数、基于CPU的后缀自动机ROSA结合，通过一个10万参数的输出“耳门”，可将32k token下的寻针召回率提升至95%–100%。 (1:54) 成功的门控训练需要合成复制粘贴课程；自然散文训练会使门控表现保守，而输入端残差流注入则会恶化代码困惑度。 (2:34) 为了将行为格式化与逐字记忆相结合，研究人员提出进行分工：状态微调（约150万个初始值）负责控制“如何做”，而ROSA负责提供“是什么”，尽管初步的场景分割测试尚无定论且尚未结合。 (6:10)

Reading summary

RWKV-7’s fixed-size hidden state achieves high computational efficiency at the expense of long-range factual retention. As documented by Xiaol.x, pairing the frozen recurrent model with ROSA—a CPU-based suffix automaton—via an external output gate restores verbatim long-context recall without altering the base neural weights. The Memory Satchel Bottleneck (0:00) RWKV-7 relies on an RNN-style fixed-size hidden state that remains constant regardless of how much text flows past. While this design ensures rapid inference and keeps language modeling loss completely flat between 1,000 and 32,000 tokens, it introduces severe memory decay. In needle-in-a-haystack benchmarks, a passcode placed 16,000 tokens upstream resulted in zero successful recalls out of twenty attempts. At 32,000 tokens, the model retained facts only when they were positioned at the very end of the stream, and even then only roughly one-third of the time. Scaling up to a 1.5-billion-parameter variant failed to alleviate the issue. Because the loss curve remains stable despite these retrieval failures, Xiaol.x notes that the model has not forgotten how to write, but has structurally forgotten what it previously read. Integrating ROSA via Ear Gating (1:54) To counter this amnesia, BlinkDL developed ROSA (Rapid Online Suffix Automaton) for the upcoming RWKV-8 architecture. Operating entirely on CPU without learned weights or floating-point operations, ROSA executes at approximately 12 microseconds per token. While it achieves 100% accuracy on synthetic needle lookups, its greedy next-token predictions fail nearly 90% of the time on ordinary prose, meaning it cannot function as an independent generator. The successful hybrid architecture leaves RWKV-7 frozen and attaches a 100,000-parameter learned "ear gate" at the output layer. This gate blends ROSA's string-matched proposals into the next-token distribution using match length, recurrence count, and the base model's confidence. When trained on a synthetic "copy-paste curriculum"—where 50% of sequences duplicate upstream text—recall at 32,000 tokens reached 85% to 100%, and generalization held at 90% or higher at 64,000 tokens. On 30,000-token-old copied text, cross-entropy dropped from over 3 nats/token to 0.16 nats/token. Limits of Internal and Semantic Interventions (3:32) The success of output gating contrasts with several failed integration methods that attempted to modify the model elsewhere. BlinkDL’s native proposal of injecting ROSA retrieval embeddings directly into the residual stream degraded source code perplexity from 2.9 to 3.5 and damaged needle recall before training compute was exhausted. An engram hashed phrasebook with 17 million parameters provided zero improvement on verbatim copy tasks due to its lack of document-specific awareness, while standard prose integration yielded negligible gains because base models already predict natural language well. Attempts to replace exact string matching with soft continuous attention over cached states for paraphrased prompts also broke down. On novel scene-break detection, 1-hop memory failed to outperform a basic adapter across different seeds, while 2-hop memory degraded performance further. The tests demonstrate that while exact string matching grafted externally via a calibrated gate works, internal representational modifications and semantic multi-hop approximations fail under limited compute. Decoupling Behavioral Style from Factual Storage (5:45) Because ROSA only performs literal string matching, it cannot guide qualitative behavior, such as identifying implicit scene transitions, enforcing JSON schemas, or following instructions. To address this limitation without full fine-tuning, the program proposes dividing responsibilities between "how" and "what." Under this approach, all model weights remain frozen, and backpropagation optimizes only the initial hidden state vectors—roughly 1.5 million scalar values across all layers and attention heads. This pre-packed state instills structural habits and formatting conventions from the start of generation. Although RWKV's running updates gradually overwrite this initial state over long sequences, it only needs to govern output style, allowing ROSA's external ledger to handle historical factual recall up to 64,000 tokens. The hybrid combination of state tuning and ROSA remains an unproven hypothesis. Early 500-step benchmarks on scene-break detection saw state tuning trail a simple classification head (0.22 versus 0.24 F1), while semantic memory variants suffered substantial seed volatility (0.26 versus 0.19 F1). Xiaol.x outlines a planned four-way ablation benchmark on a 30,000-token task to determine whether state tuning adds value alongside ROSA, noting that if the combined model cannot outperform the ROSA heron alone, the state-tuned satchel will be discarded.

Source quotes

[0:00] A frozen RWKV-7 swims a river of text with one small satchel of memory that never grows. [0:00] The secret code for Heron is 734912. [0:00] It has not forgotten how to write. It has forgotten what it read. [1:05] The last time this exact run of words appeared what came next? [1:05] A Heron that always talks is useless. The goose has to learn when to listen. [1:54] Keep the goose frozen and train only a tiny gate at its ear, using a copy-paste curriculum. [3:32] Down the Throat [5:49] The heron only copies; it can't teach behaviour. So the bet is to split the work. [7:42] The honest part: this bet has not been shown yet. [7:42] If both cannot beat the heron alone, the satchel goes.

Original transcript

[0:02] This is a goose. [0:04] RWKV7 [0:07] It swims down a river of text and it [0:10] carries everything it knows in one small [0:12] satchel that never gets any bigger. [0:15] That satchel is why it is fast. It is [0:19] also why it forgets. [0:21] Plant one sentence upstream. [0:24] The secret code for Heron is 734912. [0:29] Then 16,000 tokens later ask for it. Um [0:34] planted early [0:36] the goose gets it zero times out of 20. [0:40] At 32,000 tokens only a fact planted [0:43] near the very end survives about a third [0:46] of the time. The bigger goose [0:49] 1.5 billion parameters does no better. [0:53] And yeah, it's prose is fine. It's [0:56] losses flat from 1,000 tokens to 32,000. [1:01] It has not forgotten how to write. It [1:03] has forgotten what it read. [1:05] Enter the Heron. [1:07] Rosa, the rapid online suffix automaton [1:11] from BlinkDL's RWKV8. [1:15] The Heron keeps a ledger of the whole [1:17] river. [1:18] At every step it asks one question. [1:22] The last time this exact run of words [1:24] appeared what came next? No parameters [1:29] no floating point 12 microseconds a [1:32] token on a CPU [1:35] on the needle test the Heron alone [1:37] scores 20 out of 20 at every depth. But [1:42] on ordinary prose it's best guess is [1:46] wrong nearly nine times in 10. A Heron [1:48] that always talks is useless. The goose [1:51] has to learn when to listen. [1:54] So here is what works. Freeze the goose, [1:58] put the heron on its shoulder, train [2:00] only a tiny gate about 100,000 [2:03] parameters that mixes the heron's [2:05] candidate into the goose's next word [2:09] guess. [2:10] The gate looks at how long the match is, [2:14] how often it was seen, and how much the [2:16] goose already believes it. Trained on [2:18] ordinary text, the gate stays timid. [2:21] Long repeats seen only once are rare [2:23] there, so it never learns to trust one. [2:25] At 32,000 tokens, it is no better than [2:28] the goose alone. The fix is a [2:30] curriculum. [2:32] In half the training samples, copy a few [2:35] earlier spans and paste them back in [2:37] later. [2:38] Now trusting a long match pays off. [2:41] And um [2:43] the gate opens. [2:45] As the code comes out, it swings wide, [2:48] then snap shut right after. [2:51] Needle at 32,000 tokens, 95, 85, and [2:55] 100% across depths with four decoy codes [2:59] in the haystack still 95 or better. At [3:02] 64,000, eight times its training length, [3:06] nine in 10 or better. [3:08] Copy a passage from 30,000 tokens back, [3:11] and the cost falls from over three nats [3:14] a token to 0.16. [3:18] The bigger goose gets the same fix. Um [3:21] on source code, perplexity drops about [3:24] 4% overall [3:27] and 5% past 16,000 tokens. [3:32] Now the failures. [3:34] Each one puts the heron somewhere other [3:36] than the goose's ear. [3:38] First, push the heron's notes down the [3:40] goose's throat, BlinkDL's own input side [3:43] design. Embeddings added to the residual [3:46] stream on a frozen goose, it hurts. Code [3:49] perplexity climbs from 2.9 to 3.5, and [3:52] even with a gate on top, the needle gets [3:55] worse. [3:56] One honest caveat, its loss was still [3:59] falling when training stopped. That is a [4:02] budget limit, not a proof it cannot [4:04] work. [4:05] Second, N-gram, a 17 million parameter [4:09] phrasebook of hash token pairs and [4:11] triples. It knows the language, but it [4:14] never reads this river. On the copy [4:16] test, it changes nothing. [4:19] Third, pros. [4:21] The goose already models what it can [4:23] there. Novel loss moves in the third [4:25] decimal place. There was never any [4:28] headroom to win. [4:29] Fourth, and still open, meaning [4:33] a paraphrased question gives the heron [4:35] no exact match. So, the newest [4:38] experiment swaps the ledger for soft [4:40] multi-hop memory over hidden states. [4:43] Against match controls on scene [4:46] boundaries, one hop ties a plain local [4:49] adapter of the same size, and their [4:51] order flips between seeds. Two hops do [4:55] slightly worse. No retrieval win yet. [5:00] So, here is the ledger. [5:02] What works? [5:04] Exact, verbatim, retrieval from the [5:08] outside through a tiny gate that was [5:11] taught to trust. [5:12] What fails? [5:14] Anything that rewires a frozen goose on [5:16] this budget, and anything that needs [5:19] meaning instead of matching strings. [5:21] Next, learned bit string Rosa on a [5:24] backbone that can train, and memory [5:27] tests where two facts truly have to be [5:29] linked. Back on the river, 32,000 tokens [5:33] down, the goose is asked again. This [5:36] time the heron leans in. [5:39] 734912 [5:49] >> But, the heron only copies. It can point [5:54] at words the river already carried. It [5:57] cannot teach the goose how to behave, [6:00] where a scene ends, what shape an answer [6:04] takes, what the task wants. [6:07] That has to come from somewhere else. [6:10] Here is the bet. Keep every weight [6:12] frozen and learn just one thing. [6:16] What is already in the satchel before [6:18] the trip begins, the starting state. [6:21] That is state tuning. For this goose, [6:25] that is about 1.5 million numbers, one [6:28] small grid for every head and every [6:30] layer. Every trip starts from the same [6:34] packed satchel. Nothing in the goose [6:37] itself changes. [6:39] A packed satchel shapes the first [6:41] stretch of river. The habits, the [6:43] format, the job. As the river runs, the [6:46] goose keeps writing into the same small [6:48] bag, and early habits can fade. That is [6:51] fine if something else carries the far [6:54] past. [6:56] So, split the work. [6:59] State tuning packs the satchel, it knows [7:01] how. Rosa keeps the ledger, it knows [7:04] what. Near the start, the packed habits [7:08] steer. [7:09] Far downstream, [7:11] when a question needs a fact from 30,000 [7:14] tokens [7:16] back, the heron finds it in the ledger [7:18] and whispers it through the gate at the [7:20] ear. Nothing goes down the throat, the [7:24] path that failed last time. [7:27] The satchel never grows, the ledger [7:29] grows with the river. Behavior is [7:31] learned once on short stretches. Recall [7:35] stays exact as far as anyone has [7:37] measured, out to 64,000 tokens. [7:42] Now the honest part. [7:44] Has the packed satchel worked yet? Not [7:46] yet. The first test asked the goose to [7:49] mark scene breaks in novels inside 4,000 [7:52] tokens with 500 training steps and two [7:54] seeds. The packed satchel scored .22 [7:58] um [7:59] Training only a small classifier scored [8:02] .24. Adding semantic memory made the two [8:06] seeds disagree. [8:08] That test was short. It tried one [8:10] learning rate and it never put the [8:12] satchel together with the heron's [8:13] ledger. That pairing is the whole bet [8:16] and it has not been tried. So, [8:19] first a proper learning rate sweep on [8:22] development data only. [8:24] Then the fair test, a task that needs a [8:27] learned answer format and a fact from [8:30] 30,000 tokens back. Four geese, neither [8:34] satchel only, heron only, and both. If [8:38] both cannot beat the heron alone, the [8:41] satchel goes. [8:43] So, the goose sets off again, satchel [8:46] packed, heron at its ear, state tuning [8:50] knows how, Rosa knows what. Next time we [8:53] find out whether that is enough.