MARCH Router Halts Recurrent State Overwrite, Lifts Retrieval 25%

MARCH Explained: Scaling Recurrent Memory with Content-Routed State Anchors — Xiaol.x

Xiaol.x

Content-conditioned routing to historical state checkpoints resolves recurrent memory overwrite without sacrificing decoding speed, provided models retain a learned null option to reject stale history.

AI-assisted summary of MARCH Explained: Scaling Recurrent Memory with Content-Routed State Anchors — original source. Check the original for full context.

Source published:

Key ideas

  • MARCH introduces content-routed state anchors to resolve temporal interference in recurrent models, preserving historical checkpoints without abandoning fast native recurrence. (0:00)
  • Evaluations on Gated DeltaNet pretrained on 50 billion tokens show a 25% relative improvement on LongBench (11.9 to 14.9) and a 14% gain on in-context retrieval. (6:14)
  • Ablations confirm that a learned null route is essential, giving the model an explicit zero-payload option to avoid injecting stale or irrelevant historical context. (7:09)
  • The framework enables a unified 'anchored RWKV-MS' design that routes across both time (historical snapshots) and capacity (parallel state slots) in a two-level hierarchy. (8:44)

Detailed notes

## Background Published on YouTube by Xiaol.x on August 27, 2026, this video explains the MARCH (Memory-Anchor Routing across Context History) framework. According to the description, the explainer addresses the temporal overwrite failure mode in recurrent neural networks by periodically saving cumulative recurrent-state checkpoints and routing queries back to historical states, while also detailing practical connections to multi-state RWKV.

## TL;DR Recurrent models decode efficiently with fixed-size states but suffer from temporal interference, where ongoing updates overwrite older associations. MARCH resolves this through temporal expansion, preserving cumulative recurrent checkpoints and generating content-conditioned routing keys that allow queries to selectively read historical residuals alongside the current state. The framework incorporates strict causality, Top-K efficiency optimizations, and a vital learned null route to prevent irrelevant historical injection. The video reviews reported benchmark gains on LongBench with Gated DeltaNet and demonstrates how MARCH can be synthesized with multi-state RWKV to create a two-dimensional time-and-slot routing architecture.

## Atmosphere & style The presentation carries the disciplined, analytical atmosphere of an advanced graduate systems seminar or technical paper walkthrough. The delivery is focused and methodical, methodically deriving architectural mechanics, routing equations, and computational trade-offs without unnecessary hype. Listening to the breakdown feels lucid and intellectually rigorous, especially as the speaker carefully demarcates verified empirical results from exploratory prototype implementations.

## Bulletin - [00:00](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=0s) MARCH introduces content-routed state anchors to resolve temporal interference in recurrent models, preserving historical checkpoints without abandoning fast native recurrence. - [06:14](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=374s) Evaluations on Gated DeltaNet pretrained on 50 billion tokens show a 25% relative improvement on LongBench (11.9 to 14.9) and a 14% gain on in-context retrieval. - [07:09](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=429s) Ablations confirm that a learned null route is essential, giving the model an explicit zero-payload option to avoid injecting stale or irrelevant historical context. - [08:44](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=524s) The framework enables a unified 'anchored RWKV-MS' design that routes across both time (historical snapshots) and capacity (parallel state slots) in a two-level hierarchy.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=0s) Fixed-size recurrent states suffer from temporal interference, where ongoing state updates overwrite earlier associations needed for later retrieval. - [00:00](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=0s) Unlike transformers where past tokens remain individually addressable in the KV cache, standard recurrent architectures maintain history solely within the latest state. - [01:33](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=93s) Recurrent memory can scale along two distinct axes: capacity expansion (parallel present slots) and temporal expansion (preserving historical checkpoints over time). - [02:19](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=139s) MARCH maintains continuous recurrence across chunk boundaries rather than resetting, allowing each checkpoint anchor to summarize the entire prefix context up to that point. - [03:08](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=188s) Anchors are indexed using content-conditioned routing keys derived from shared anchor tokens, decoupling routing identities from simple temporal positions. - [03:55](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=235s) Causal masking restricts routing exclusively to prior checkpoints, while a learned null route provides a mechanism to bypass historical retrieval entirely. - [04:42](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=282s) The model decouples routing queries from content extraction queries, adding a content-routed historical residual directly onto the base recurrent state read. - [05:28](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=328s) Computation is kept practical through token and anchor tiling, fused online softmax, and Top-K retrieval, avoiding materializing the full token-by-anchor routing matrix. - [06:14](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=374s) Gated DeltaNet with MARCH achieves significant retrieval gains on LongBench, with Top-4 routing doubling dense training throughput at 128k context lengths. - [07:09](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=429s) A checkpoint interval of 512 and routing width of 64 provided the best trade-off between checkpoint density and computational overhead in ablations. - [07:09](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=429s) Removing the learned null route degrades performance across all aggregate metrics, proving that an explicit mechanism to decline historical retrieval is required. - [07:58](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=478s) Multi-state RWKV and MARCH operate orthogonally, with RWKV-MS distributing information across capacity slots and MARCH distributing across temporal snapshots. - [08:44](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=524s) An anchored RWKV-MS architecture implements hierarchical two-level routing by first selecting a historical snapshot and then routing into a specific memory slot. - [10:26](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=626s) Promising future extensions include adaptive anchoring triggered by prediction surprise or scene boundaries, anchor consolidation, and hierarchical multi-timescale routing.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=0s) The Fixed-State Paradox of Recurrent Models Recurrent models offer a distinct structural advantage by processing arbitrarily long sequences while carrying a single fixed-size matrix, making decoding extremely efficient. However, this creates a fundamental paradox: every new piece of information must share that same small carrier with the entire past. Unlike transformers, where old tokens remain individually addressable in the key-value cache, a recurrent model's past survives solely through its latest state. If a later update weakens an old association, the earlier version is completely gone.

To resolve this limitation, MARCH—standing for Memory-Anchor Routing across Context History—proposes preserving recurrence as the fast native path while adding a selective route back to earlier versions of its memory. For example, if the state initially records the association that "project code name maps to orbit", a much later token might write a new value along a similar key direction. While this local delta update is useful, it can overwrite the exact coordinates that previously carried "orbit". When a query eventually asks "What was the code name?", the current query reads only the latest matrix, leading to a failure not because the query is flawed, but because the required version of memory no longer has a direct read path. This core issue is known as temporal interference.

- **Transformers:** Old tokens remain individually addressable in the key-value cache. - **Recurrent models:** The past survives only through the latest state, making earlier overwritten associations irretrievable. - **Temporal interference:** Occurs when ongoing updates evolve a single state, overwriting historical points needed for retrieval.

```mermaid flowchart LR accTitle: Recurrent State Overwrite vs. Historical Preservation accDescr: Diagram contrasting standard recurrent state evolution with MARCH checkpointing. A["Incoming Token Stream"] --> B["Recurrent Model (Fixed Matrix)"] B --> C["Latest State Only (Vulnerable to Overwrite)"] B --> D["MARCH Checkpoints (Historical Retention)"] ```

### [01:33](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=93s) Capacity Expansion versus Temporal Expansion When scaling recurrent memory, there are two fundamentally different expansion directions. Capacity expansion makes the present state wider or splits it into parallel partitions, whereas temporal expansion preserves multiple historical versions of a state. Multi-state RWKV work represents capacity expansion, where several RWKV states coexist at the current time and routing distributes reads and writes across those slots. In contrast, MARCH performs temporal expansion by following one continuous recurrent trajectory and preserving selected cumulative checkpoints.

At the end of a chunk marked by boundary $B_M$, MARCH does not reset recurrence; instead, it simply copies the current state. Consequently, anchor $A^M$ at layer $L$ equals state $S$ at boundary $B_M$. Because the recurrence remains continuous, this anchor summarizes the whole prefix through that boundary rather than merely the latest chunk, creating a sequence of anchors that records the evolution of one cumulative memory.

- **Capacity expansion:** Makes the present state wider or splits it into parallel partitions (e.g., multi-state RWKV). - **Temporal expansion:** Follows one continuous trajectory and preserves historical checkpoints (e.g., MARCH). - **Checkpoint definition:** Anchor $A^M$ at layer $L$ equals state $S$ at boundary $B_M$.

### [03:08](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=188s) Content-Conditioned Anchors and Causal Routing A bank of old matrices is useless unless the model can identify what each matrix remembers. MARCH solves this by inserting a shared anchor token after each checkpoint, which reads only its aligned state anchor at a given layer. This anchor representation is then projected to a compact routing key, $\kappa_M$. Because the representation already contains information read from the aligned checkpoint across layers, the key is conditioned on memory content rather than time alone, allowing the same learned anchor embedding to produce different keys at different boundaries.

For a text token $T$, a small routing query $\rho_T = W_R X_T$ computes a score for anchor $M$ via the dot product $\rho_T^T \kappa_M$. Only anchors with boundary $B_M$ strictly before $T$ are visible to preserve strict causality. Furthermore, MARCH introduces a learned null route whose state payload is zero. Taking a softmax over visible anchors and the null route produces probabilities $\pi$, ensuring the model is never forced to inject stale history and can choose to do nothing when no checkpoint is relevant.

- **Content conditioning:** Keys ($\kappa_M$) reflect memory content rather than just temporal position. - **Causality constraint:** Only anchors with boundaries strictly preceding token $T$ are visible. - **Learned null route:** Provides a zero-payload option so the model can ignore irrelevant history via softmax probabilities ($\pi$).

### [04:42](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=282s) Deriving the Historical Residual and Computational Costs MARCH cleanly separates two distinct jobs: the routing query chooses which historical state matters, while the ordinary state read query $Q_T$ extracts content from the chosen matrix. The native recurrent output is given by $S_T Q_T$, and historical memory contributes a weighted sum over $\pi_{T, J} A^J Q_T$. Combining these terms yields the current state read plus a content-routed historical residual, preserving the base recurrent update without hiding it behind a hard switch.

Trade-offs accompany these anchors, as they trade memory for temporal resolution. With checkpoint interval $C$, candidate states grow roughly as sequence length divided by $C$, with each retained anchor storing a recurrent state matrix. To manage this efficiently, the paper's reader tiles tokens and anchors, fuses routing scores, computes online softmax, and performs weighted state reads without materializing the full token-by-token anchor routing matrix. Top-K routing reduces work further, turning the architecture into a controllable point between a single compressed state and token-level attention.

- **Read separation:** Routing queries select the relevant historical state, while ordinary state queries ($Q_T$) extract the content. - **Combined output:** Current state read plus content-routed historical residual. - **Optimization:** Token-and-anchor tiling, online softmax fusion, and Top-K routing avoid massive matrix materialization.

### [06:14](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=374s) Empirical Results and Ablation Insights The authors pretrained MARCH models from scratch on 50 billion tokens using 16,000-token training sequences, with their main setting utilizing a checkpoint interval of 512 and a routing width of 64. On LongBench, the reported Gated DeltaNet average score rises from 11.9 to 14.9, representing a 25% relative gain, while in-context retrieval increases from 20.5 to 23.3 (a 14% relative gain). At 128k tokens, top-4 routing more than doubles dense MARCH training throughput and cuts core runtime by roughly an order of magnitude, though vanilla Gated DeltaNet remains faster due to performing no historical retrieval.

Ablation studies reveal crucial design lessons. Checkpoint interval $C = 512$ provides the best overall balance between fine history preservation and anchor sparsity, and routing width 64 serves as the balanced default. Crucially, removing the null route degrades every reported aggregate, demonstrating that selective memory requires an explicit, learnable mechanism to output "none of these."

- **LongBench performance:** Gated DeltaNet average rises from 11.9 to 14.9 (25% relative gain). - **In-context retrieval:** Increases from 20.5 to 23.3 (14% relative gain). - **Ablation finding:** Removing the null route degrades every reported aggregate, proving the necessity of an explicit "none of these" option.

### [07:58](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=478s) Bridging MARCH and Multi-State RWKV While both MARCH and multi-state RWKV (RWKV-MS) share the core motivation of rejecting the idea that one undifferentiated recurrent state is enough for long-range memory, their mechanisms differ fundamentally. RWKV-MS expands across slots at one time (asking which current slot should write or answer a token), whereas MARCH expands across time (asking which earlier version of recurrent memory contains information erased by later updates).

The cleanest conceptual extension combines both axes into a two-dimensional "anchored RWKV-MS" design. By letting $\mathcal{S}_{T, S}$ represent current RWKV-MS slot $S$, and defining the entire slot bank at a boundary snapshot as $\mathcal{A}^M = \mathcal{S}$ at $B_M$, the model can route first over historical snapshots with $\pi$ and then reuse the slot router inside each selected snapshot. This creates a two-level router that first chooses when, and then chooses which slot inside that timeframe.

```mermaid flowchart TD accTitle: Two-Level Routing in Anchored RWKV-MS accDescr: Hierarchical flowchart showing routing across historical time snapshots followed by slot selection. A["Incoming Token"] --> B["Level 1: Historical Snapshot Router (pi)"] B --> C["Selected Historical Snapshot Bank (A^M)"] C --> D["Level 2: RWKV Slot Router (gamma)"] D["Level 2: RWKV Slot Router (gamma)"] --> E["Final Memory Output"] ```

### [09:31](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=571s) Repository Additions and Future Extensions The accompanying repository incorporates this optional historical anchor branch, where interval zero disables it to preserve existing behavior, and positive intervals create configurable snapshots, capacity bounds, routing width, top-K settings, residual scales, and null biases. Operating strictly online and causally, tokens read prior anchors, update the live RWKV-MS state, and only then append a boundary snapshot, with streaming sessions persisting anchor states, keys, and validity masks. Regression tests validate mechanics such as causality, capacity, chunked streaming equivalence, and router gradients.

Looking forward, three strong extensions are proposed: - **Adaptive anchoring:** Saving snapshots when update magnitude, state novelty, prediction surprise, or semantic scene boundaries cross a threshold, giving stable regions fewer anchors and rapid change finer resolution. - **Byte-budget consolidation:** Evicting redundant anchors, merging similar snapshots, or demoting information already answerable by the live slot bank. - **Hierarchical routing:** Choosing memory type or timescale, then historical snapshot, then RWKV slot, accompanied by practical experiments matching memory bytes between more current slots versus fewer slots plus anchors.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=0s) Unlike transformers, where old tokens remain individually addressable in the key-value cache, a recurrent model's past survives solely through its latest state. — Xiaol.x

> [00:00](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=0s) To resolve this limitation, MARCH—standing for Memory-Anchor Routing across Context History—proposes preserving recurrence as the fast native path while adding a selective route back to earlier versions of its memory. — Xiaol.x

> [01:33](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=93s) Capacity expansion makes the present state wider or splits it into parallel partitions, whereas temporal expansion preserves multiple historical versions of a state. — Xiaol.x

> [01:33](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=93s) Because the recurrence remains continuous, this anchor summarizes the whole prefix through that boundary rather than merely the latest chunk, creating a sequence of anchors that records the evolution of one cumulative memory. — Xiaol.x

> [03:08](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=188s) A bank of old matrices is useless unless the model can identify what each matrix remembers. — Xiaol.x

> [03:08](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=188s) Taking a softmax over visible anchors and the null route produces probabilities $\pi$, ensuring the model is never forced to inject stale history and can choose to do nothing when no checkpoint is relevant. — Xiaol.x

> [04:42](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=282s) MARCH cleanly separates two distinct jobs: the routing query chooses which historical state matters, while the ordinary state read query $Q_T$ extracts content from the chosen matrix. — Xiaol.x

> [04:42](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=282s) Top-K routing reduces work further, turning the architecture into a controllable point between a single compressed state and token-level attention. — Xiaol.x

> [06:14](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=374s) Crucially, removing the null route degrades every reported aggregate, demonstrating that selective memory requires an explicit, learnable mechanism to output "none of these." — Xiaol.x

> [07:58](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=478s) RWKV-MS expands across slots at one time (asking which current slot should write or answer a token), whereas MARCH expands across time (asking which earlier version of recurrent memory contains information erased by later updates). — Xiaol.x

> [07:58](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=478s) This creates a two-level router that first chooses when, and then chooses which slot inside that timeframe. — Xiaol.x

> [09:31](https://www.youtube.com/watch?v=BuK5yF7eV9s&t=571s) Operating strictly online and causally, tokens read prior anchors, update the live RWKV-MS state, and only then append a boundary snapshot, with streaming sessions persisting anchor states, keys, and validity masks. — Xiaol.x

## Mentioned MARCH (Memory-Anchor Routing across Context History), Xiaol.x, Gated DeltaNet, LongBench, Multi-state RWKV (RWKV-MS), Anchored RWKV-MS, arXiv, GitHub, YouTube

## Video description Recurrent models decode efficiently by compressing history into a fixed-size state—but that compression can overwrite facts needed later. MARCH (Memory-Anchor Routing across Context History) addresses this problem by periodically preserving cumulative recurrent-state checkpoints and routing queries back to the historical states most relevant to the current token.

This explainer covers:

- Why fixed recurrent states suffer from temporal interference - MARCH’s content-conditioned state anchors - Causal anchor routing, Top-K retrieval, and the learned null route - Current-state reads plus historical residual memory - The trade-off between checkpoint density and computation - Reported Gated DeltaNet results on LongBench and in-context retrieval - How MARCH differs from multi-state RWKV - A proposed two-dimensional “anchored RWKV-MS” design - Adaptive anchoring, consolidation, and hierarchical routing

Paper: https://arxiv.org/abs/2608.12435 refs: https://github.com/xiaol/Multi-state-RWKV-online-memory

Important scope note: the reported quality and throughput results come from the MARCH paper’s Gated DeltaNet experiments. The anchored RWKV-MS mechanism discussed here is a tested prototype; equivalent model-quality gains still require matched training and long-context evaluation.

Chapters: 00:00 The fixed-state paradox 00:45 What overwrite really means 01:33 Two ways to scale recurrent memory 02:19 The checkpoint move 03:08 Why an anchor needs a content key 03:55 Routing, causality, and the null option 04:42 Deriving the residual historical read 05:28 Where the cost goes 06:14 What the paper actually shows 07:09 The ablations teach the design 07:58 Is this like multi-state RWKV? 08:44 The anchored RWKV-MS deduction 09:31 What was added to the repository 10:26 Strong next extensions 11:25 The honest conclusion

[Source](https://www.youtube.com/watch?v=BuK5yF7eV9s)

中文简报

MARCH 路由机制遏制循环状态覆写，检索性能提升 25% 基于内容的路由机制将查询导向历史状态检查点，在不牺牲解码速度的前提下解决了循环记忆覆写问题，但需配合可学习的空选项以拒绝陈旧历史信息。 循环模型虽能凭借固定大小的状态实现高效解码，但面临时间干扰难题，即后续更新容易覆写原有的记忆关联。MARCH 通过时间扩展机制解决了这一痛点，在保留累积循环检查点的同时生成基于内容的路由键，允许查询在当前状态之外选择性读取历史残差。该框架融合了严格的因果性限制、Top-K 效率优化，以及一项关键的可学习空路由机制以避免注入无关的历史信息。视频回顾了 Gated DeltaNet 在 LongBench 评测基准上的提升表现，并展示了如何将 MARCH 与多状态 RWKV 相融合，构建兼顾时间与槽位的二维路由架构。 MARCH 引入了基于内容路由的状态锚点，旨在解决循环模型中的时间干扰问题，既保留了历史检查点，又未舍弃其高效的原生循环特性。 (0:00) 在基于 500 亿 token 预训练的 Gated DeltaNet 上的评测显示，其在 LongBench 上的表现相对提升了 25%（从 11.9 升至 14.9），上下文检索性能提升了 14%。 (6:14) 消融实验证实，可学习的空路由必不可少，它为模型提供了明确的零有效载荷选项，从而避免注入陈旧或无关的历史上下文。 (7:09) 该框架支持统一的“锚定 RWKV-MS”设计，在双层分层结构中跨越时间（历史快照）和容量（并行状态槽）进行综合路由。 (8:44)

Reading summary

The MARCH (Memory-Anchor Routing across Context History) framework resolves recurrent models' susceptibility to temporal interference by preserving content-addressable historical state checkpoints along a continuous sequence trajectory. By pairing selective routing with native recurrence, the architecture bridges the gap between fixed-size recurrent states and full transformer key-value caches. Resolving Temporal Interference Through Historical Anchors (0:00) While recurrent neural networks decode efficiently by maintaining a single fixed-size state matrix across long sequences, incoming token updates inevitably overwrite earlier information. When subsequent updates weaken prior associations—such as overwriting a previously stored code name mapping—standard recurrence fails because it lacks a direct read path back to earlier matrix states. Unlike transformers, which maintain individual token addressability within a key-value cache, pure recurrent models leave overwritten data irretrievable due to temporal interference. To address this limitation, MARCH introduces temporal expansion, contrasting with capacity expansion models like Multi-State RWKV (RWKV-MS) that widen or partition the active state at the current time step. Instead of resetting memory, MARCH follows a single continuous recurrent trajectory and periodically copies the recurrent state matrix $S$ at boundary $B_M$ to create an anchor $A^M$. Each anchor captures a cumulative prefix summary through that boundary, maintaining a historical record of state evolution without disrupting native recurrent updates. Content-Conditioned Routing and Residual Memory Reads (3:08) To identify relevant historical matrices, MARCH inserts a shared anchor token after each boundary checkpoint. This token reads its aligned state across layers and projects it into a compact routing key $\kappa_M$, conditioning the key on memory content rather than chronological step alone. During text generation, a token query $\rho_T$ computes routing scores against visible prior anchors via dot products, enforcing strict causality by masking any boundary occurring after the current token. Crucially, routing incorporates a learned null route with a zero-state payload into the softmax distribution, allowing the model to deliberately output no historical state when past context is irrelevant. The system decouples state selection from content extraction: the model reads its current state $S_T Q_T$ and adds a content-routed historical residual calculated from the weighted sum of selected anchors. To control computational and memory overheads, the implementation employs token-and-anchor tiling, online softmax fusion, and Top-K routing to avoid materializing dense routing matrices. Benchmark Gains, Ablation Lessons, and Computational Costs (6:14) Evaluated on models pretrained from scratch on 50 billion tokens with 16,000-token sequence lengths, MARCH demonstrated consistent empirical improvements. Using a primary configuration with checkpoint interval $C = 512$ and routing width 64, Gated DeltaNet's average score on LongBench increased from 11.9 to 14.9 (a 25% relative gain), while in-context retrieval improved from 20.5 to 23.3 (a 14% relative gain). At a 128k context length, top-4 sparse routing more than doubled dense MARCH training throughput and reduced core runtime by roughly an order of magnitude, though vanilla Gated DeltaNet remained faster because it bypasses historical retrieval entirely. Ablation experiments indicated that an interval of $C = 512$ balanced temporal preservation against state sparsity, while routing width 64 served as an effective default. Furthermore, ablations confirmed the architectural necessity of the null route: removing it degraded performance across every reported aggregate metric, demonstrating that selective retrieval depends on an explicit mechanism to reject irrelevant historical states. Multi-Dimensional Scaling and Future Memory Architectures (7:58) Recognizing that multi-state RWKV scales memory across current slot capacity while MARCH scales across time, the authors propose a unified "anchored RWKV-MS" framework. In this two-dimensional configuration, snapshots of the entire RWKV-MS slot bank are saved at boundary checkpoints. A two-level router first selects which historical snapshot to access using routing probabilities $\pi$, and then deploys a slot router to identify the relevant memory slot within that snapshot. The accompanying repository implements causal, streaming-compatible snapshotting with configurable capacity bounds, Top-K settings, and null biases. Looking ahead, the authors outline three primary extensions: adaptive anchoring, which triggers snapshots based on prediction surprise or state novelty rather than fixed intervals; byte-budget consolidation to prune or merge redundant anchors; and hierarchical routing that allocates capacity dynamically between concurrent slots and temporal anchors. MARCH demonstrates that selective temporal checkpointing mitigates state overwriting in recurrent architectures, but it introduces trade-offs: anchors scale memory linearly with sequence length, and retrieval adds computational overhead compared to pure recurrence. Unresolved questions center on whether dynamic, surprise-driven checkpointing and memory consolidation can maintain retrieval quality under constrained hardware budgets.

Source quotes

[0:00] Xiaol.x: Unlike transformers, where old tokens remain individually addressable in the key-value cache, a recurrent model's past survives solely through its latest state. [0:00] Xiaol.x: To resolve this limitation, MARCH—standing for Memory-Anchor Routing across Context History—proposes preserving recurrence as the fast native path while adding a selective route back to earlier versions of its memory. [1:33] Xiaol.x: Capacity expansion makes the present state wider or splits it into parallel partitions, whereas temporal expansion preserves multiple historical versions of a state. [1:33] Xiaol.x: Because the recurrence remains continuous, this anchor summarizes the whole prefix through that boundary rather than merely the latest chunk, creating a sequence of anchors that records the evolution of one cumulative memory. [3:08] Xiaol.x: A bank of old matrices is useless unless the model can identify what each matrix remembers. [3:08] Xiaol.x: Taking a softmax over visible anchors and the null route produces probabilities $\pi$, ensuring the model is never forced to inject stale history and can choose to do nothing when no checkpoint is relevant. [4:42] Xiaol.x: MARCH cleanly separates two distinct jobs: the routing query chooses which historical state matters, while the ordinary state read query $Q_T$ extracts content from the chosen matrix. [4:42] Xiaol.x: Top-K routing reduces work further, turning the architecture into a controllable point between a single compressed state and token-level attention. [6:14] Xiaol.x: Crucially, removing the null route degrades every reported aggregate, demonstrating that selective memory requires an explicit, learnable mechanism to output "none of these." [7:58] Xiaol.x: RWKV-MS expands across slots at one time (asking which current slot should write or answer a token), whereas MARCH expands across time (asking which earlier version of recurrent memory contains information erased by later updates). [7:58] Xiaol.x: This creates a two-level router that first chooses when, and then chooses which slot inside that timeframe. [9:31] Xiaol.x: Operating strictly online and causally, tokens read prior anchors, update the live RWKV-MS state, and only then append a boundary snapshot, with streaming sessions persisting anchor states, keys, and validity masks.

Original transcript

[0:00] Here is the paradox. A recurrent model [0:03] can process an arbitrarily long stream [0:05] while carrying one fixed-size matrix. [0:08] That is excellent for decoding [0:10] efficiency, but every new fact must [0:12] share the same small carrier with the [0:14] entire past. [0:16] In a transformer, old tokens remain [0:19] individually addressable in the [0:21] key-value cache. In a recurrent model, [0:24] the past survives only through the [0:26] latest state. If a later update weakens [0:28] an old association, the earlier version [0:31] is simply gone. [0:33] March asks a precise question. Can we [0:36] preserve recurrence as the fast native [0:39] path while adding a selective route back [0:42] to earlier versions of its memory? [0:44] Imagine the state first stores the [0:47] association project code name maps to [0:50] orbit. [0:51] A much later token writes another value [0:55] along a similar key direction. The delta [0:58] update is useful locally, yet it can [1:00] revise the very coordinates that carried [1:02] orbit. [1:04] When the question, "What was the code [1:06] name?" [1:07] finally arrives, the current query reads [1:10] only the latest matrix. [1:13] The failure is not that the query is [1:15] bad. The required version of memory no [1:18] longer has a direct read path. [1:21] So, the central problem is temporal [1:24] interference. One state keeps evolving, [1:27] but retrieval sometimes needs an earlier [1:30] point on that trajectory. [1:32] There are two fundamentally different [1:36] expansion directions. Capacity expansion [1:39] makes the present state wider or splits [1:42] it into parallel partitions. Temporal [1:45] expansion keeps multiple historical [1:47] versions of a state. [1:49] Your existing multi-state RWKV work is [1:54] mainly capacity expansion. [1:57] Several RWKV states coexist at the [2:00] current time and routing distributes [2:03] reads and writes across those slots. [2:06] March is temporal expansion. It follows [2:09] one continuous recurrent trajectory and [2:11] preserves selected cumulative [2:13] checkpoints. These ideas are related, [2:15] but they are not the same geometry. [2:18] Let boundary B sub M mark the end of a [2:22] chunk. [2:23] March does not reset the recurrence [2:25] there. It simply copies the current [2:27] state. So, anchor A superscript M at [2:31] layer L equals state S at boundary B sub [2:36] M. [2:37] Because the recurrence remains [2:40] continuous, [2:42] this anchor summarizes the whole prefix [2:44] through that boundary, not merely the [2:47] latest chunk. [2:49] A sequence of anchors therefore records [2:53] the evolution of one cumulative memory. [2:58] This is the first key move. Preserve old [3:01] state versions before future decay and [3:03] correction can erase their useful [3:05] associations. [3:07] Um a bank of old matrices is useless [3:11] unless the model can identify what each [3:13] matrix remembers. March inserts a shared [3:17] anchor token after each checkpoint. At a [3:20] layer that token reads only its aligned [3:22] state anchor. [3:24] The anchor representation is then [3:26] projected to a compact routing key, [3:29] kappa sub M. Across layers, the [3:33] representation already contains [3:35] information read from the aligned [3:36] checkpoint, so the key becomes [3:38] conditioned on memory content rather [3:40] than only time. [3:42] The same learned anchor embedding can [3:44] therefore produce different keys at [3:46] different boundaries. [3:49] The address says in effect, [3:51] "This is what that historical state can [3:53] answer. [3:55] For text token T, a small routing query [3:58] row sub T equals W sub R times X sub T. [4:03] Its score for anchor M is the dot [4:05] product row sub T transpose kappa sub M. [4:10] Only anchors with boundary B sub M [4:13] strictly before T are visible. [4:17] March then adds one more candidate, a [4:20] learned null route whose state payload [4:23] is zero. [4:25] A softmax over visible anchors and null [4:28] produces probabilities pi. This matters. [4:32] The model is not forced to inject stale [4:34] history. When no checkpoint is relevant, [4:37] it can spend its probability on doing [4:40] nothing. [4:41] Now separate two jobs. The routing query [4:44] chooses which historical state matters. [4:47] The ordinary state read query Q sub T [4:50] extracts content from the chosen matrix. [4:53] These are different learned geometries. [4:56] The native recurrent output is S sub T Q [5:00] sub T. Historical memory contributes the [5:03] weighted sum over pi sub T {comma} J [5:06] times A superscript J Q sub T. [5:10] Add the two terms. The result is current [5:13] state read plus a content routed [5:16] historical residual. March does not [5:19] replace the recurrent update or hide it [5:21] behind a hard switch. It preserves the [5:24] base path and learns when history should [5:26] help. [5:28] Anchors trade memory for temporal [5:30] resolution. [5:32] With checkpoint interval C, the number [5:33] of candidate states grows approximately [5:36] like sequence length divided by C and [5:40] each retained anchor stores a recurrent [5:42] state matrix. [5:45] The paper's reader tiles tokens and [5:47] anchors fuses, routing scores, online [5:50] softmax, and weighted state reads, and [5:54] avoids materializing the full token by [5:57] anchor routing matrix or every candidate [6:00] readout. [6:01] Top K routing reduces work further. The [6:04] design is not constant memory anymore. [6:07] It is a controllable point between a [6:09] single compressed state and token level [6:12] attention. [6:13] The authors pretrain match models from [6:17] scratch on 50 billion tokens with 16,000 [6:21] token training sequences. Their main [6:23] merge setting uses checkpoint interval [6:26] 512 and routing with 64. [6:31] On long bench, the reported gated [6:33] DeltaNet average rises from 11.9 to [6:36] 14.9, a 25% relative gain. In-context [6:40] retrieval rises from 20.5 to 23.3, [6:44] a 14% relative gain. [6:47] At 128k [6:51] tokens, top four merge more than doubles [6:55] dense merge training throughput and cuts [6:58] core runtime by roughly one order of [7:01] magnitude. Vanilla gated DeltaNet [7:04] remains faster [7:05] because it performs no historical [7:07] retrieval. [7:09] Checkpoint density is a real trade-off. [7:11] Smaller intervals preserve finer history [7:14] but create more anchors. Larger [7:17] intervals are cheaper, yet checkpoints [7:19] may be too sparse. In the reported [7:22] sweep, [7:23] C = 512 gives the best overall balance. [7:28] Routing with 64 is the balanced default. [7:32] With 192 improves some retrieval but [7:36] hurts general performance. Top four [7:39] nearly matches dense routing on several [7:41] aggregates, although it trails on [7:43] needle-in-a-haystack tests. [7:46] Removing the null route degrades every [7:49] reported aggregate. That is a strong [7:51] lesson. Selective memory needs an [7:53] explicit learnable way to say, "None of [7:56] these." [7:58] Yes in motivation, no in mechanism. [8:02] Both reject the idea that one [8:04] undifferentiated recurrent state is [8:06] enough for long-range memory. [8:08] Both use routing, so extra memory does [8:11] not become an equally expensive dense [8:13] read. [8:14] But, RWKV-MS [8:17] asks, "Which current slot should write [8:20] or answer this token?" March asks, [8:23] "Which earlier version of recurrent [8:25] memory still contains what later updates [8:28] erased?" [8:30] So, RWKV-MS expands across slots at one [8:33] time, while March expands across time. [8:37] Um the cleanest extension is not to [8:39] choose one, it is to compose both axes. [8:42] Um [8:44] Let calligraphic S sub T {comma} S be [8:46] current [8:48] RWKV-MS [8:49] slot S. The existing read is the sum [8:52] over slot weights gamma sub T {comma} S [8:55] times S sub T {comma} S R sub T. [8:58] At a boundary snapshot, the entire slot [9:00] bank calligraphic A superscript M equals [9:04] calligraphic S at B sub M. [9:07] Route first over historical snapshots [9:09] with pi, then reuse the slot router [9:12] inside each selected snapshot. [9:15] The final memory output is current [9:17] multi-slot read plus lambda sub A times [9:21] the weighted historical snapshot reads. [9:24] This is a two-level router. Choose when, [9:27] then choose which slot inside that time. [9:31] The repository now contains this [9:33] optional historical anchor branch. [9:36] Interval zero disables it, preserving [9:39] existing behavior. [9:41] Positive intervals create snapshots, [9:43] capacity bounds the bank, routing width, [9:46] top K, residual scale, and null bias are [9:49] configurable. [9:51] The online order is strictly causal. [9:55] A token reads only prior anchors, [9:58] updates the live RWKVMS state, and only [10:01] then appends a boundary snapshot. Um [10:04] streaming sessions persist anchor [10:06] states, keys, and validity masks. [10:11] Regression tests cover causality, [10:14] capacity, chunked streaming equivalence, [10:17] state round trips, router gradients, and [10:20] trainer state scatter. This validates [10:23] mechanics, not model quality. [10:26] Um first, replace fixed intervals with [10:29] adaptive anchoring. [10:31] Save a snapshot when update magnitude, [10:34] state novelty, prediction surprise, or a [10:36] semantic scene boundary crosses a [10:38] threshold. [10:40] Um stable regions then consume fewer [10:43] anchors, rapid change receives finer [10:46] resolution. [10:48] Second, consolidate under a byte budget. [10:51] Evict redundant anchors, merge similar [10:53] snapshots, or demote information already [10:56] answerable by the live slot bank. This [11:00] connects naturally to your whole style [11:04] surprise and consolidation ideas. [11:07] Third, make routing hierarchical. [11:10] Choose memory type or time scale, then [11:13] historical snapshot, then RWKV slot. A [11:17] practical experiment should compare more [11:19] current slots against fewer slots plus [11:21] anchors at exactly matched memory bytes. [11:24] March's central insight is simple and [11:26] powerful. [11:28] A recurrent state is not only a memory [11:30] object, its trajectory is also a memory [11:32] resource. Preserve useful points on that [11:37] trajectory, give them content keys, and [11:39] retrieve them only when needed. [11:42] For your work, [11:44] the closest description is [11:47] two-dimensional recurrent memory. [11:49] RWKV-MS provides current capacity across [11:53] slots and anchors provide historical [11:55] coverage across time. [11:57] The paper gives strong evidence on gated [12:01] DeltaNet. [12:02] The repository now gives a tested [12:06] RWKV-MS [12:08] prototype. The next scientific step is [12:11] control training and long context [12:13] evaluation not assuming the paper's [12:15] gains transfer automatically.