Nvidia Cuts Video Diffusion Compute by Approximating Dropped Attention

Xiaol.x

Replacing hard token-dropping with dynamic Gaussian thresholds and zeroth-order tail voting prevents visual degradation while speeding up attention by up to 5.4 times without retraining.

AI-assisted summary of Sol-Attn Explained: NVIDIA’s Training-Free Sparse Attention for Faster Video Generation — original source. Check the original for full context.

Source published: 2026-08-19

Key ideas

NVIDIA researchers introduced Sol-Attn, a training-free sparse attention method that alleviates the quadratic compute bottleneck of video diffusion transformers without requiring backbone retraining. (0:00)
By observing that standardized block-proxy logits follow a Gaussian distribution, Sol-Attn sets dynamic per-query thresholds (τᵢ = μᵢ + βσᵢ) and replaces hard token dropping with a low-cost zeroth-order Taylor tail approximation. (3:10)
Sol-Attn fuses routing, exact attention, and tail correction into a single online softmax kernel, outperforming FlashAttention-3 by up to 5.41× at 128k context lengths and streaming routing 11.5× faster than top-k. (7:05)
As part of the wider Sol-Engine runtime (which stacks Sol-Attn with step caching, NVFP4 quantization, and token pruning), end-to-end video generation accelerates up to 5.08× on NVIDIA Blackwell systems. (10:14)

Detailed notes

## Background Published by the channel Xiaol.x on YouTube on August 19, 2026, this video provides a visual, teacher-led breakdown of Sol-Attn, NVIDIA's training-free sparse attention mechanism for accelerating diffusion transformers. The presentation explains how Sol-Attn mitigates the quadratic compute tax of video attention without discarding tail information, contrasting the technique with FlashAttention, detailing the broader Sol-Engine stack, and evaluating its implications for WKVM and RWKV model serving.

## TL;DR While FlashAttention resolved memory I/O bottlenecks, video diffusion transformers still face a quadratic arithmetic tax that severely limits generation speed. Sol-Attn tackles this without model retraining by exploiting an empirical Gaussian score distribution to set dynamic per-query density thresholds, routing blocks entirely within SRAM rather than materializing memory-heavy proxy maps. Instead of hard-dropping sub-threshold blocks—which leads to smeared textures and visual drift—the method incorporates a zeroth-order Taylor approximation of the skipped tail directly into a unified online softmax accumulator. This design yields up to 5.41× kernel speedups and over 2× end-to-end gains on models like HunyuanVideo, serving as a core primitive within NVIDIA's Sol-Engine while providing calibrated routing insights for hybrid serving architectures like WKVM.

## Atmosphere & style The presentation carries the methodical, measured cadence of an advanced systems engineering seminar, balancing visual flowcharts with granular kernel mechanics and mathematical formulations. Rather than indulging in generative AI hype, the speaker maintains a grounded, pedagogical tone that systematically demystifies performance metrics, carefully distinguishing standalone kernel speedups from full-stack framework accelerations. Watching it feels intellectually rigorous and pragmatic, guiding the viewer through low-level hardware realities with clean, analytical clarity.

## Bulletin - [00:00](https://www.youtube.com/watch?v=arf7aLT8TzA&t=0s) NVIDIA researchers introduced Sol-Attn, a training-free sparse attention method that alleviates the quadratic compute bottleneck of video diffusion transformers without requiring backbone retraining. - [03:10](https://www.youtube.com/watch?v=arf7aLT8TzA&t=190s) By observing that standardized block-proxy logits follow a Gaussian distribution, Sol-Attn sets dynamic per-query thresholds (τᵢ = μᵢ + βσᵢ) and replaces hard token dropping with a low-cost zeroth-order Taylor tail approximation. - [07:05](https://www.youtube.com/watch?v=arf7aLT8TzA&t=425s) Sol-Attn fuses routing, exact attention, and tail correction into a single online softmax kernel, outperforming FlashAttention-3 by up to 5.41× at 128k context lengths and streaming routing 11.5× faster than top-k. - [10:14](https://www.youtube.com/watch?v=arf7aLT8TzA&t=614s) As part of the wider Sol-Engine runtime (which stacks Sol-Attn with step caching, NVFP4 quantization, and token pruning), end-to-end video generation accelerates up to 5.08× on NVIDIA Blackwell systems.

## Key takeaways - [00:48](https://www.youtube.com/watch?v=arf7aLT8TzA&t=48s) FlashAttention reduced memory IO from quadratic to linear, but video diffusion models still incur full O(L²) matrix arithmetic across exploding frame token counts. - [01:33](https://www.youtube.com/watch?v=arf7aLT8TzA&t=93s) Aggressive keep-or-drop sparse attention truncates the long tail of attention probabilities, leading to smeared textures, soft details, and visual drift. - [02:24](https://www.youtube.com/watch?v=arf7aLT8TzA&t=144s) Conventional top-k routing enforces rigid compute budgets regardless of distribution sharpness, while top-p introduces high variance and costly global scan/sorting overhead. - [03:10](https://www.youtube.com/watch?v=arf7aLT8TzA&t=190s) Standardized block-proxy attention scores empirically follow a Gaussian distribution, enabling a global hyperparameter β to reliably control sequence-wide sparsity. - [03:52](https://www.youtube.com/watch?v=arf7aLT8TzA&t=232s) Query thresholds are calculated dynamically as τᵢ = μᵢ + βσᵢ, allowing sharp distributions to select few blocks and flat distributions to allocate more capacity. - [04:36](https://www.youtube.com/watch?v=arf7aLT8TzA&t=276s) Row-wise means and variances are derived via pre-aggregated linear and quadratic projections of pooled keys, computing thresholds in O(LD + ND²) time without materializing an intermediate proxy score matrix. - [06:10](https://www.youtube.com/watch?v=arf7aLT8TzA&t=370s) Non-selected blocks are retained via a zeroth-order Taylor expansion centered on mean key vectors, adding a cheap approximate vote directly into the online softmax statistics. - [07:05](https://www.youtube.com/watch?v=arf7aLT8TzA&t=425s) A unified kernel architecture simultaneously evaluates streaming routing, exact full tiles, and tail approximations within the same fused online softmax loop without auxiliary HBM buffers. - [08:07](https://www.youtube.com/watch?v=arf7aLT8TzA&t=487s) Threshold-based streaming routing is 11.5× faster than top-k and 32.7× faster than top-p while avoiding multi-gigabyte memory buffer overheads. - [09:07](https://www.youtube.com/watch?v=arf7aLT8TzA&t=547s) Sol-Attn achieves a 2.02× to 2.12× end-to-end speedup on HunyuanVideo at ~85% sparsity with visual quality and VBench scores meeting or exceeding dense baselines. - [10:14](https://www.youtube.com/watch?v=arf7aLT8TzA&t=614s) The broader Sol-Engine stack combines Sol-Attn with NVFP4 quantization, step caching, and token pruning to achieve up to 5.08× end-to-end acceleration on NVIDIA Blackwell B200 GPUs. - [11:13](https://www.youtube.com/watch?v=arf7aLT8TzA&t=673s) While Sol-Attn assumes bidirectional diffusion and cannot replace WKVM's fixed-size recurrent decoding state, its calibrated thresholding and tail correction offer viable patterns for routed hybrid models.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=arf7aLT8TzA&t=0s) The Leftover Quadratic Tax of Video Diffusion

Generating video with a diffusion transformer turns every video frame into patches, and those patches into an explosion of tokens. Increasing resolution or generating longer clips causes sequence lengths to balloon rapidly. Because standard self-attention requires every token to attend to every other token, the compute cost scales quadratically ($O(L^2)$): doubling the sequence length quadruples the required arithmetic operations.

While FlashAttention solved memory I/O bottlenecks by computing attention in SRAM tiles and maintaining running online softmax statistics (running maximum, running sum, and output accumulator) to avoid writing the full $L \times L$ attention matrix to High Bandwidth Memory (HBM), it did not alter the fundamental arithmetic complexity. Every single query tile still has to multiply against every key tile. For modern video diffusion models, this remaining quadratic compute represents "the tax you actually pay." Sol-Attn is introduced as an acceleration method that cuts down this compute tax without requiring any retraining or fine-tuning of the diffusion backbone.

- **FlashAttention's scope**: Reduced memory complexity from $O(L^2)$ to $O(L)$, but kept computational complexity strictly at $O(L^2)$. - **Block-level granularity**: In hardware-friendly sparse attention architectures, tokens are grouped into blocks (typically 64 tokens). Skipping a block bypasses an entire tile of matrix multiplication.

---

[01:33](https://www.youtube.com/watch?v=arf7aLT8TzA&t=93s) The Flaws of Keep-or-Drop Sparse Attention

Standard training-free sparse attention approaches typically rely on a two-stage heuristic: query and key blocks are pooled into mean summary vectors, and a coarse proxy score is computed between these pooled vectors to evaluate whether the corresponding token blocks are relevant. Based on these proxy scores, winning blocks are retained while losing blocks are completely discarded.

```mermaid flowchart TD accTitle: Standard Sparse Attention vs Information Loss accDescr: Diagram illustrating standard two-stage proxy routing and the consequence of hard discarding.

subgraph ProxyStage["1. Proxy Scoring"] Q["Query Block"] -->|"Pool Mean"| QP["Pooled Query"] K["Key Block"] -->|"Pool Mean"| KP["Pooled Key"] QP & KP --> Dot["Dot Product Proxy Score"] end

subgraph HardGate["2. Hard Selection"] Dot --> Decision{"Score > Cutoff?"} Decision -->|"Winner"| FullTile["Exact Full-Tile Attention"] Decision -->|"Loser"| Drop["Zero Contribution (Discarded)"] end

Drop --> Artifacts["Soft Details, Smeared Textures, Drift"] ```

When blocks are dropped entirely, their contribution to both the softmax numerator and denominator is forced to zero. Under mild sparsity, this hard truncation produces minimal noticeable error. However, under aggressive sparsity targets (such as 80% to 90%), the collective discarded tail still holds substantial probability mass. Completely eliminating the tail causes video frames to become soft, destroys fine surface textures, and induces noticeable drift relative to full dense attention. This exposes two primary challenges: making routing cheaper without materializing intermediate routing maps in HBM, and accounting for the information in the skipped attention tail.

---

[02:24](https://www.youtube.com/watch?v=arf7aLT8TzA&t=144s) Top-k vs. Top-p and the Gaussian Score Discovery

Conventional routing algorithms fail to balance efficiency and stability across varying attention distributions:

| Routing Policy | Mechanism | Strengths & Weaknesses | Memory & Kernel Overhead | |:--- |:--- |:--- |:--- | | **Top-$k$** | Selects a fixed number ($k$) of highest-scoring blocks for every query row. | Rigid budget allocation. Enforces the same cost regardless of whether a query has an ultra-sharp peak or a uniform/flat distribution. | Requires materializing proxy maps into HBM and executing global sorting passes. | | **Top-$p$** | Accumulates normalized probabilities until reaching cumulative threshold $p$. | Flexible but unpredictable. May retain almost zero blocks for sharp distributions and nearly all blocks for flat distributions. | High budget variance; requires expensive sorting and scan passes across full rows. |

Empirical analysis across diffusion timesteps, transformer layers, attention heads, and query blocks reveals that standardized block-proxy logits consistently follow a bell-shaped Gaussian distribution. In a standard normal distribution, choosing a cutoff threshold $\beta$ standard deviations above the mean corresponds directly to a predictable upper-tail area ($1 - \Phi(\beta)$). This property allows a single global hyperparameter $\beta$ to act as a calibrated "density knob" that maintains the model's target sparsity while letting individual query rows adapt their raw thresholds dynamically based on their local variance.

---

[03:52](https://www.youtube.com/watch?v=arf7aLT8TzA&t=232s) Dynamic Density Thresholding and Low-Cost Moments

The routing threshold $\tau_i$ for a given query block $i$ is formulated mathematically around the mean ($\mu_i$) and standard deviation ($\sigma_i$) of its proxy scores:

$$\tau_i = \mu_i + \beta \sigma_i$$

A key block $j$ is retained for exact calculation only when its proxy score exceeds $\tau_i$. Consequently, queries with sharp attention distributions select fewer blocks, whereas queries with flatter distributions retain more, maintaining an average sparsity that closely tracks the theoretical Gaussian tail.

Calculating $\mu_i$ and $\sigma_i$ by first computing all proxy dot products would inadvertently reconstruct the very proxy score matrix the system seeks to avoid storing. Because each proxy score is merely the dot product of a pooled query vector $\bar{q}_i$ with a pooled key vector $\bar{k}_j$, the row-wise mean and variance can be expressed as linear and quadratic projections of the pooled keys:

- The mean pooled key vector $\mathbb{E}[\bar{k}]$ and the second-moment matrix $\mathbb{E}[\bar{k} \bar{k}^T]$ are accumulated once across all key blocks. - Each pooled query block is then projected through these pre-aggregated moments to obtain $\mu_i$ and $\sigma_i$ in $O(L D + N D^2)$ time with only $D^2$ additional storage overhead. - In the Sol-Engine Triton implementation, a preprocessing kernel pools keys, reduces moments across the sequence, and directly writes out a single scalar threshold $\tau_i$ for each query block.

---

[05:24](https://www.youtube.com/watch?v=arf7aLT8TzA&t=324s) Streaming Routing and Approximate Tail Voting

Once $\tau_i$ is established, global sorting is eliminated. Pooled keys are processed as a short sequence traversed in streaming chunks directly inside the kernel:

```mermaid flowchart LR accTitle: Streaming Routing and Dual-Path Online Softmax accDescr: Streaming execution flow showing thresholding inside SRAM and dual paths for exact and approximate attention.

subgraph StreamEngine["SRAM Tiling & Online Routing"] Chunk["Pooled Key Chunk"] --> FormProxy["Token-to-Block Proxy Score"] FormProxy --> Check{"Score > tau_i?"} end

subgraph DualExecution["Unified Accumulator"] Check -->|"Yes (Above Threshold)"| ExactTile["Exact Full Tile (Q * K_full^T)V_full"] Check -->|"No (Below Threshold)"| ApproxTail["Zeroth-Order Taylor Tail (Cheap Exp * Sum V)"] ExactTile --> SoftmaxMerge["Single Shared Online Softmax Accumulator"] ApproxTail --> SoftmaxMerge end ```

To prevent accuracy degradation when aggressively pruning blocks, Sol-Attn applies a zeroth-order Taylor expansion of the exponential scores centered around the block's mean key vector. Instead of dropping non-selected blocks, the system treats all tokens within a skipped block as if they shared the score of the pooled key vector. Multiplying this low-cost exponential scalar by the block's pre-summed values produces an approximate contribution that feeds directly into the running online softmax numerator and denominator.

Ablation studies confirm that at identical sparsity ratios, traditional hard keep-or-drop schemes exhibit rapid growth in relative $L_2$ error and noticeable drops in cosine similarity relative to dense attention. By retaining an approximate vote for the tail, Sol-Attn preserves structural fidelity without incurring full tile matrix multiplications.

---

[07:05](https://www.youtube.com/watch?v=arf7aLT8TzA&t=425s) Sol-Attn Unified Kernel Architecture and Synthetic Benchmarks

Rather than splitting routing, exact attention, and tail correction into isolated kernels, Sol-Attn executes all three within a single, fused online softmax loop. A query tile meets a tile of pooled keys; columns exceeding $\tau_i$ load original $K$ and $V$ tiles into registers for full-precision attention, while sub-threshold columns are masked and processed via the low-overhead tail approximation.

- **Unified State**: The exact inner loop and approximate outer loop update the exact same online softmax registers (running maximum $m$, normalizer $l$, and accumulator $O$) simultaneously. - **Minimal API Footprint**: In the official code release, the interface accepts contiguous BF16 tensors for $Q$, $K$, and $V$ with head dimension 128, a threshold parameter $\tau$ (or $\beta$), and an optional exact sink for text tokens. - **Routing Speedup**: Threshold-based streaming routing runs **11.5× faster** than top-$k$ and **32.7× faster** than top-$p$ implementations. - **Memory Overhead**: Avoids external routing buffers, remaining on par with dense attention memory consumption, whereas systems like SparseVideoGen2 require approximately **8× more** buffer memory. - **H100 Throughput**: Achieves a **5.41× speedup** over FlashAttention-3 at a sequence length of 128,000 tokens with 90% sparsity. - **cuDNN Comparison**: Compared against cuDNN block-sparse attention executing over identical indices, the integrated tail approximation adds only a negligible overhead inside the kernel, while finishing faster end-to-end by eliminating standalone index generation passes.

---

[09:07](https://www.youtube.com/watch?v=arf7aLT8TzA&t=547s) Video Generation Quality and Real-World Speedups

Qualitative evaluations on text-to-video diffusion demonstrate that Sol-Attn preserves visual stability across challenging scenes. In comparative side-by-side frames from HunyuanVideo depicting a planetary rover traversing terrain, high-frequency elements—including robotic chassis components, ground shadows, and the distant horizon—remain anchored without blurring or spatial drift.

- **HunyuanVideo (14B parameter model, 81 frames at 720p)**: Yields a **2.02× end-to-end speedup** at roughly 85% sparsity, while achieving VBench evaluation scores slightly above the dense attention baseline. - **HunyuanVideo (13B parameter model, 129 frames)**: Delivers a **2.12× end-to-end speedup**. - **Long LTX (Stage 2)**: Achieves approximately **2.4× speedup**. - **Sana World Model Refiner (Video-to-Video)**: Achieves a **3.04× speedup**. - **Bernini Video Editing**: Demonstrates a **2.34× speedup**.

The authors emphasize that these results represent faithful similarity to dense attention outputs alongside task-specific benchmarks, rather than a claim of training an entirely new generative architecture.

---

[10:14](https://www.youtube.com/watch?v=arf7aLT8TzA&t=614s) The Complete Sol-Engine Stack vs. Kernel Speedup

Sol-Attn serves as one component within NVIDIA's broader **Sol-Engine** inference stack (hosted on the Sana repository's `sol-engine` branch). Built on top of SGLang's multimodal runtime, Sol-Engine combines five complementary optimization techniques:

1. **Step Caching**: Reusing diffusion states across adjacent denoising intervals. 2. **NVFP4 Quantization**: Accelerating linear projections with 4-bit floating point hardware support. 3. **Kernel Fusion**: Collapsing elementwise and normalization operations. 4. **Sparse Attention (Sol-Attn)**: Pruning attention computation at the block level. 5. **Token Pruning**: Eliminating uninformative spatial tokens outright.

Because Sol-Attn operates at the attention primitive layer, it stacks multiplicatively with caching and weight quantization. On NVIDIA Blackwell (B200) hardware, the full Sol-Engine stack reaches **3.48× end-to-end acceleration** on Hunyuan 14B and up to **5.08×** on larger Hunyuan pipelines, with further speedups posted for MiniMax Hailuo and LTX 2.5. The speaker notes the distinction clearly: headline 5× end-to-end system numbers reflect the entire Sol-Engine suite, not the isolated sparse attention kernel.

---

[11:13](https://www.youtube.com/watch?v=arf7aLT8TzA&t=673s) Implications for WKVM and Hybrid Model Serving

The video concludes by addressing whether Sol-Attn can be applied to WKVM (a system hypervisor designed for stateful recurrent and hybrid architectures such as RWKV).

The operational models between the two architectures differ fundamentally: - **WKVM's Design**: Operates on fixed-size recurrent state slots for autoregressive decoding, deliberately avoiding dynamically expanding key-value caches, session forks, and unconstrained memory growth. - **Sol-Attn's Scope**: Designed for bidirectional video diffusion where all query and key tokens are known up front. It stores every key and value in memory, pruning only the matrix arithmetic during dense full-sequence passes. It does not address causal decoding, state hibernation, or sequence branching.

Despite these differing execution models, two core conceptual takeaways from Sol-Attn translate directly to WKVM's routed-span guest paths (such as approximate Gemma attention segments): 1. **Calibrated Thresholding over Rigid Top-$k$**: Dynamically setting bounds based on standardized score statistics rather than hard-coded token counts. 2. **Tail Approximation over Hard Dropping**: Providing an aggregated, low-cost summary vote for skipped contexts instead of zeroing them out completely.

As summarized in the closing contrast: "Do not import a DiT kernel and call it a serving architecture. Sol-Attn tames the attention multiply; WKVM changes what you are serving."

## Notable quotes > [00:00](https://www.youtube.com/watch?v=arf7aLT8TzA&t=0s) Because standard self-attention requires every token to attend to every other token, the compute cost scales quadratically ($O(L^2)$): doubling the sequence length quadruples the required arithmetic operations. — Xiaol.x

> [00:00](https://www.youtube.com/watch?v=arf7aLT8TzA&t=0s) Every single query tile still has to multiply against every key tile. For modern video diffusion models, this remaining quadratic compute represents "the tax you actually pay." — Xiaol.x

> [01:33](https://www.youtube.com/watch?v=arf7aLT8TzA&t=93s) Completely eliminating the tail causes video frames to become soft, destroys fine surface textures, and induces noticeable drift relative to full dense attention. — Xiaol.x

> [02:24](https://www.youtube.com/watch?v=arf7aLT8TzA&t=144s) Empirical analysis across diffusion timesteps, transformer layers, attention heads, and query blocks reveals that standardized block-proxy logits consistently follow a bell-shaped Gaussian distribution. — Xiaol.x

> [03:52](https://www.youtube.com/watch?v=arf7aLT8TzA&t=232s) Consequently, queries with sharp attention distributions select fewer blocks, whereas queries with flatter distributions retain more, maintaining an average sparsity that closely tracks the theoretical Gaussian tail. — Xiaol.x

> [05:24](https://www.youtube.com/watch?v=arf7aLT8TzA&t=324s) Instead of dropping non-selected blocks, the system treats all tokens within a skipped block as if they shared the score of the pooled key vector. — Xiaol.x

> [05:24](https://www.youtube.com/watch?v=arf7aLT8TzA&t=324s) By retaining an approximate vote for the tail, Sol-Attn preserves structural fidelity without incurring full tile matrix multiplications. — Xiaol.x

> [07:05](https://www.youtube.com/watch?v=arf7aLT8TzA&t=425s) Rather than splitting routing, exact attention, and tail correction into isolated kernels, Sol-Attn executes all three within a single, fused online softmax loop. — Xiaol.x

> [10:14](https://www.youtube.com/watch?v=arf7aLT8TzA&t=614s) Because Sol-Attn operates at the attention primitive layer, it stacks multiplicatively with caching and weight quantization. — Xiaol.x

> [11:13](https://www.youtube.com/watch?v=arf7aLT8TzA&t=673s) Do not import a DiT kernel and call it a serving architecture. Sol-Attn tames the attention multiply; WKVM changes what you are serving. — Xiaol.x

> Sol-Attn is not a replacement for WKVM’s fixed recurrent state, but its calibrated thresholding and tail-correction ideas may be valuable for WKVM’s approximate Gemma routed-span guest path. — Xiaol.x

## Mentioned NVIDIA, Sol-Attn, Sol-Engine, FlashAttention, FlashAttention-3, WKVM, RWKV, Gemma, Triton, SparseVideoGen2, H100, cuDNN, HunyuanVideo, VBench, Long LTX, LTX 2.5, Sana, Sana World Model Refiner, Bernini, SGLang, NVFP4, NVIDIA Blackwell (B200), MiniMax Hailuo, Xiaol.x

## Video description Why is video generation still slow after FlashAttention?

This visual, teacher-led explanation breaks down Sol-Attn, NVIDIA’s training-free method for accelerating diffusion-transformer inference by skipping unnecessary attention blocks—without simply discarding their information.

You’ll learn:

- Why video attention remains quadratically expensive - Why top-k routing is rigid and top-p routing is unpredictable - How τᵢ = μᵢ + βσᵢ creates a dynamic density threshold - How routing happens inside online softmax without materializing a proxy map - How the skipped attention tail receives an approximate contribution - How Sol-Attn differs from the complete Sol-Engine stack - Whether these ideas are useful for WKVM and RWKV serving

The key WKVM conclusion: Sol-Attn is not a replacement for WKVM’s fixed recurrent state, but its calibrated thresholding and tail-correction ideas may be valuable for WKVM’s approximate Gemma routed-span guest path.

Chapters

00:00 Video tokens make attention expensive 00:48 What FlashAttention did—and did not—solve 01:33 Why keep-or-drop sparsity loses information 02:23 Top-k versus top-p routing 03:10 The surprising Gaussian score distribution 03:52 A threshold for every query 04:36 Computing thresholds without the score map 05:24 Routing while attention streams 06:10 Giving the dropped tail a vote 07:05 One kernel performing three jobs 08:07 Understanding the speed results 09:07 Dense versus Sol-Attn video quality 10:14 Sol-Engine’s complete acceleration stack 11:13 Is Sol-Attn useful for WKVM?

Sources

Paper: https://arxiv.org/abs/2607.24027 Code: https://github.com/NVlabs/Sana/tree/sol-engine Project page: https://nvlabs.github.io/Sana/Sol-Attn/

#MachineLearning #VideoGeneration #SparseAttention #NVIDIA #DiffusionModels #FlashAttention #RWKV #WKVM

[Source](https://www.youtube.com/watch?v=arf7aLT8TzA)

中文简报

NVIDIA通过近似被舍弃注意力降低视频扩散计算开销用动态高斯阈值和零阶尾部投票替代硬性token丢弃，在无需重新训练的情况下防止画质受损，同时将注意力计算速度提升高达5.4倍。尽管FlashAttention解决了内存I/O瓶颈，但视频扩散Transformer仍面临严重制约生成速度的二次方算术开销。Sol-Attn在无需重新训练模型的情况下解决了这一问题，它利用经验高斯分数分布设定每个query的动态密度阈值，并完全在SRAM内完成块路由，避免实体化高显存占用的代理图。该方法没有硬性丢弃低于阈值的块（硬丢弃会导致纹理模糊和视觉漂移），而是将跳过尾部的零阶泰勒近似直接整合到统一的在线softmax累加器中。这一设计在HunyuanVideo等模型上实现了高达5.41倍的内核加速和超过2倍的端到端收益，不仅作为NVIDIA Sol-Engine的核心原语，还为WKVM等混合推理架构提供了经过校准的路由见解。 NVIDIA研究人员推出了Sol-Attn，这是一种免训练的稀疏注意力方法，无需重新训练主干网络即可缓解视频扩散Transformer的二次方计算瓶颈。 (0:00) 通过观察到标准化块代理logits遵循高斯分布，Sol-Attn设定了动态的逐query阈值（τᵢ = μᵢ + βσᵢ），并以低成本的零阶泰勒尾部近似替代硬性token丢弃。 (3:10) Sol-Attn将路由、精确注意力和尾部校正融合到单个在线softmax内核中，在128k上下文长度下性能领先FlashAttention-3高达5.41倍，流式路由速度比top-k快11.5倍。 (7:05) 作为更广泛的Sol-Engine运行时的一部分（该系统将Sol-Attn与步骤缓存、NVFP4量化和token剪枝相结合），在NVIDIA Blackwell系统上端到端视频生成加速高达5.08倍。 (10:14)

Reading summary

NVIDIA's Sol-Attn accelerates video diffusion transformers without retraining by replacing hard-pruned sparse attention with dynamic thresholding and a zeroth-order Taylor approximation for the skipped attention tail. Resolving Quadratic Arithmetic and Pruning Degradation (0:00) Video diffusion transformers generate massive token sequences, causing self-attention compute to scale quadratically ($O(L^2)$). While FlashAttention mitigated memory bandwidth bottlenecks by tiling in SRAM, it maintained $O(L^2)$ computational complexity. Conventional training-free sparse attention approaches address this by grouping tokens into blocks, computing coarse proxy scores from mean-pooled queries and keys, and dropping low-scoring blocks. However, standard keep-or-drop policies zero out sub-threshold blocks entirely. Under high sparsity targets (80% to 90%), discarding the collective tail strips significant softmax probability mass. This information loss leads to soft video frames, destroyed surface textures, and visual drift relative to dense attention baselines. Gaussian Routing and Dynamic Density Thresholding (2:24) Traditional routing mechanisms introduce high runtime overhead: top-$k$ forces a rigid budget regardless of attention sharpness and requires sorting, while top-$p$ produces unpredictable budget variances. Empirical analysis across layers, heads, and timesteps revealed that standardized block-proxy logits follow a Gaussian distribution. This distribution allows a global cutoff parameter, $\beta$, to act as a density control that adapts to local row variance. The threshold is calculated as $\tau_i = \mu_i + \beta \sigma_i$. To avoid reconstructing the proxy score matrix in high-bandwidth memory (HBM), Sol-Attn pre-accumulates the pooled keys' mean $\mathbb{E}[\bar{k}]$ and second-moment matrix $\mathbb{E}[\bar{k} \bar{k}^T]$. Query blocks are projected through these moments in $O(LD + ND^2)$ time with $D^2$ auxiliary storage, enabling dynamic threshold generation without global sorting. Unified Streaming Kernel with Tail Approximation (5:24) Rather than discarding sub-threshold keys, Sol-Attn computes a zeroth-order Taylor expansion centered around the pooled key vectors. Skipped tokens are assigned their block's pooled proxy score, which is scaled by pre-summed values and accumulated alongside exact tiles inside a unified online softmax loop. This preserves structural fidelity and mitigates $L_2$ error without running full matrix multiplications. Implemented as a single fused kernel, streaming routing executes 11.5× faster than top-$k$ and 32.7× faster than top-$p$ while matching dense attention memory usage. On an NVIDIA H100 GPU, the kernel reaches a 5.41× speedup over FlashAttention-3 at 128,000 tokens with 90% sparsity, outperforming cuDNN block-sparse implementations by eliminating separate indexing passes. Real-World Benchmarks and System Integration (9:07) In text-to-video tests on HunyuanVideo (14B parameter model, 81 frames at 720p), Sol-Attn achieved a 2.02× end-to-end speedup at roughly 85% sparsity, alongside VBench scores slightly exceeding the dense baseline. Additional benchmarks include a 2.12× speedup on HunyuanVideo (13B), 2.4× on Long LTX Stage 2, and 3.04× on the Sana World Model Refiner, all while maintaining high-frequency spatial details. Sol-Attn forms the sparse attention primitive within NVIDIA's broader Sol-Engine inference framework, which stacks step caching, NVFP4 quantization, kernel fusion, and token pruning. Running the complete stack on NVIDIA B200 hardware provides a 3.48× to 5.08× end-to-end acceleration on Hunyuan models, though these larger multi-fold speedups stem from the cumulative system optimizations rather than the attention kernel alone. While Sol-Attn optimizes dense bidirectional passes in video diffusion models by pruning arithmetic across known sequences, its full-cache design does not directly replace stateful autoregressive serving systems like WKVM; its broader applicability rests primarily on the concepts of statistical thresholding and tail approximation.

Source quotes

[0:00] Xiaol.x: Because standard self-attention requires every token to attend to every other token, the compute cost scales quadratically ($O(L^2)$): doubling the sequence length quadruples the required arithmetic operations. [0:00] Xiaol.x: Every single query tile still has to multiply against every key tile. For modern video diffusion models, this remaining quadratic compute represents "the tax you actually pay." [1:33] Xiaol.x: Completely eliminating the tail causes video frames to become soft, destroys fine surface textures, and induces noticeable drift relative to full dense attention. [2:24] Xiaol.x: Empirical analysis across diffusion timesteps, transformer layers, attention heads, and query blocks reveals that standardized block-proxy logits consistently follow a bell-shaped Gaussian distribution. [3:52] Xiaol.x: Consequently, queries with sharp attention distributions select fewer blocks, whereas queries with flatter distributions retain more, maintaining an average sparsity that closely tracks the theoretical Gaussian tail. [5:24] Xiaol.x: Instead of dropping non-selected blocks, the system treats all tokens within a skipped block as if they shared the score of the pooled key vector. [5:24] Xiaol.x: By retaining an approximate vote for the tail, Sol-Attn preserves structural fidelity without incurring full tile matrix multiplications. [7:05] Xiaol.x: Rather than splitting routing, exact attention, and tail correction into isolated kernels, Sol-Attn executes all three within a single, fused online softmax loop. [10:14] Xiaol.x: Because Sol-Attn operates at the attention primitive layer, it stacks multiplicatively with caching and weight quantization. [11:13] Xiaol.x: Do not import a DiT kernel and call it a serving architecture. Sol-Attn tames the attention multiply; WKVM changes what you are serving. Xiaol.x: Sol-Attn is not a replacement for WKVM’s fixed recurrent state, but its calibrated thresholding and tail-correction ideas may be valuable for WKVM’s approximate Gemma routed-span guest path.

Original transcript

[0:00] Imagine [0:02] you are generating a video with a [0:04] diffusion transformer. Every frame is [0:07] cut into patches. Those patches become [0:10] tokens. Raise the resolution. Add more [0:13] frames and the token list becomes [0:16] enormous. Self attention asks every [0:18] token to look at every other token. That [0:21] is a square. If the sequence doubles, [0:24] the arithmetic roughly quadruples. Um, [0:27] flash attention already taught the chip [0:30] not to write the full score matrix into [0:32] memory, but it still computes every [0:35] interaction. For modern video models, [0:38] that leftover square is the tax you [0:40] actually pay. Solatn is a training free [0:44] way to skip most of that tax without [0:46] retraining the model. [0:48] Let us separate two costs that people [0:51] often mix. Softmax attention produces an [0:54] L byl matrix of scores. then mixes the [0:57] values. Materializing that matrix is an [1:00] IO disaster. Flash attention tiles the [1:03] work on chip and keeps only a running [1:05] maximum, a running sum, and a running [1:08] output. Memory becomes linear. Compute [1:11] does not. Every query still meets every [1:13] key. Sparse attention is the next [1:16] bargain. Keep the online softmax [1:18] bookkeeping, but refuse to visit most [1:21] key value tiles. The hardware friendly [1:24] unit is a block, typically 64 tokens. [1:28] Skip a block and you skip a whole tile [1:31] of matrix multiplies. [1:34] Almost every training free sparse method [1:36] now does the same two-stage dance. [1:39] First, compress each query block and [1:43] each key block to a mean vector and [1:46] score those means. That cheap score is a [1:51] proxy for whether the whole block [1:53] matters. Then keep the winners and drop [1:56] the losers. The dropped block [1:58] contributes exactly zero to both the [2:00] softmax numerator and the denominator. [2:03] Under mild sparity, that can be fine. [2:06] Under aggressive sparsity, those [2:08] discarded tails still carry mass. The [2:11] picture goes soft, textures smear, and [2:13] the output drifts from dense attention. [2:16] The paper's question is therefore [2:18] double. Can routing become cheaper and [2:20] can the drop tail still count for [2:22] something. [2:24] Look at the paper's routing figure. Two [2:26] query rows, same original scores. Top K [2:31] sorts the proxies and keeps a fixed [2:33] count. Every query spends the same [2:36] budget even when one row is peaky and [2:40] the other is almost flat. Top P instead [2:44] walks a cumulative probability until it [2:47] hits P. On a peaky row, it may keep [2:50] almost nothing. On a flat row, it may [2:53] keep almost everything. The budget [2:55] becomes unpredictable and both methods [2:58] first write the entire proxy map into [3:01] high bandwidth memory. That right then [3:03] select pass is not free. At long [3:06] sequence length, routing itself becomes [3:09] attacks. [3:11] Here is the empirical gift. If you [3:14] collect those block proxy logets across [3:17] dnoising steps, layers, heads, and query [3:20] blocks, the standardized distribution is [3:22] near a bell curve. For a Gaussian, a [3:25] cutoff beta standard deviations above [3:28] the mean corresponds to a unique upper [3:30] tail density. One shared beta can [3:33] therefore set the model's average [3:35] sparsity while each query maps that same [3:38] standardized cutoff back onto its own [3:40] raw scale. The paper's calibration plot [3:43] shows the empirical density tracking 1 - [3:46] 5 beta. The threshold is not a magic [3:50] number. It is a density knob. [3:53] Write it as a teacher would on a board. [3:56] For query block I, compute the mean mui [4:00] and the standard deviation sigma i of [4:03] its proxy scores. Then set to i equal to [4:08] mui plus beta * sigma i. Keep block j [4:14] only when its proxy score exceeds to i. [4:17] Queries with sharp peaks keep fewer [4:19] blocks. Queries with flatter scores keep [4:22] more. The average density stays near the [4:25] Gaussian tail. So the budget is dynamic [4:27] and still controllable. That is the [4:30] missing third option between rigid top K [4:33] and runaway top P. [4:36] The obvious implementation would form [4:38] every proxy score then take mean and [4:42] variance that rebuilds the map you were [4:45] trying not to store. Because a proxy [4:47] score is just a pulled query dotted with [4:50] a pulled key. The row mean and variance [4:52] are linear and quadratic moments of the [4:54] pulled keys. You can accumulate the mean [4:57] pulled key and the second moment matrix [4:59] once, then project each pulled query [5:02] through those summaries. The cost is [5:05] order LD +N D squared with only D [5:09] squared extra storage in um the released [5:15] soul engine code that is a Triton [5:17] pre-process pool keys reduce moments [5:20] write one towo per query block [5:24] once tow I exists you no longer need a [5:28] global sort treat the pulled keys as a [5:31] short sequence and walk them in chunks. [5:34] For each chunk, form the tokentob block [5:37] scores, average them down to one proxy [5:39] per block, and compare against the [5:41] threshold on chip. Selected blocks [5:43] immediately enter an inner loop that [5:46] reads the original key value tile and [5:48] does exact attention. Unselected blocks [5:51] never become a materialized index tensor [5:53] in high bandwidth memory. Streaming [5:56] thresholding over chunks recovers the [5:58] same selected set as full row [6:00] thresholding. routing becomes a nested [6:03] loop inside online softmax not a [6:07] separate kernel that writes a map. [6:10] Now the second idea and it is the [6:14] accuracy idea. A block you skip is not a [6:18] block of zeros. Expand the exponential [6:21] scores around the block's mean key. The [6:24] zeroth order term says pretend every [6:28] token in that block scored like the [6:30] pulled key. Multiply that cheap [6:33] exponential by the summed values and you [6:36] have an approximate contribution to both [6:38] the softmax numerator and the [6:40] denominator. Exact blocks still do the [6:43] full tile. Approximate blocks keep a [6:46] cheap vote. The paper's ablation is [6:49] blunt at the same selected indices. Keep [6:53] or drop error grows quickly with [6:55] sparsity. Adding the approximation [6:58] lowers relative L2 error and keeps [7:00] cosine similarity higher. You are no [7:02] longer throwing the tail in the bin. [7:05] The paper's pipeline figure is the whole [7:08] method. A query tile meets a tile of [7:11] pulled keys. Averaging the columns gives [7:14] the routing scores. Columns above [7:17] threshold dispatch the original key [7:20] value tiles into the exact inner loop. [7:24] Columns below threshold stay in the [7:26] outer loop as the approximation with the [7:29] selected columns masked so they are not [7:31] counted twice. Routing sparse compute [7:34] and correction share one online softmax [7:38] state in registers. That is why the [7:41] authors call it sparsifying online [7:45] attention. In Saul engine, the public [7:47] call is almost embarrassingly small. [7:51] Contiguous Bflat 16, Q, K, and V. Head [7:57] dimension 128, a tow knob, and an [8:02] optional exact sync for text tokens. [8:08] Kernel first, not marketing first. On an [8:12] H100, [8:13] speed up over flash attention three [8:16] grows with both sequence length and [8:18] sparity, reaching 5.41 times at 128,000 [8:23] tokens and 90% sparity. Um, [8:27] uh, threshold routing is 11.5 times [8:31] faster than top K and 32.7 times faster [8:35] than top P. Processor memory stays near [8:39] dense attention while sparse video gen [8:42] 2's routing buffers need about eight [8:44] times more. Against CUDNN [8:48] block sparse attention on identical [8:51] indices, the extra approximation costs [8:54] only a few% inside the kernel and the [8:59] unified pass is a few% faster end to end [9:02] because the separate routing stage [9:05] disappears. [9:07] A faster kernel is worthless if the [9:09] video falls apart. Compare the papers [9:12] Huan Rover frames. Dense attention on [9:15] the left. Solatin on the right. The [9:18] machine, the shadow, the horizon, [9:22] stay put on Juan 2.114 billion, 81 [9:28] frames at 720p. [9:31] Soul Aton reports 2.02 02 times end to [9:34] end speed up at about 85% sparsity with [9:38] VBench slightly above dense. Um, Hunuen [9:44] 13 billion at 129 frames reaches 2.1 two [9:49] times. Long LTX stage 2 reaches about [9:53] 2.4 times. Video to video is harsher. [9:58] The Santa World Model Refiner hits 3.04 [10:01] 0 four times Bernini editing 2.34 times [10:06] these are similarity to dense numbers [10:09] plus task scores not a claim of a new [10:12] generative model [10:15] sole aten is one technique inside [10:17] Nvidia's soul engine on the sauna soul [10:21] engine branch the engine is a video [10:23] diffusion inference stack built on [10:25] sglang's multimodal runtime it composes [10:29] five reusable acceleration s step cache, [10:33] NVFP [10:35] for quantization, [10:36] kernel fusion, sparse attention, and [10:40] token pruning. Because Solatin [10:44] sparsifies the attention operator [10:45] itself, it stacks with cash infusion. On [10:49] B200, the paper reports 3.48 times end [10:53] to end on Juan 14 billion and 5.08 times [10:58] on Hunuan. The repo later posts still [11:01] larger stacks on Miniax H3 and LTX 2.5. [11:06] Remember the contract? Those larger [11:09] numbers are full stack, not the sparse [11:11] kernel alone. [11:13] [sighs] [11:14] Now the honest question. WKVM is a [11:17] hypervisor for model state. Its native [11:20] bet is that recurrent and hybrid [11:22] language models should allocate a fixed [11:24] size state slot, not a growing key value [11:26] chain. Um, so Latin still stores every [11:31] key and every value. It only skips most [11:34] of the arithmetic of looking at them and [11:37] only for birectional video diffusion. [11:41] All queries present at once. No causal [11:43] decode, no session fork, no hibernate. [11:46] That is not WKVM's physics. The adjacent [11:52] lesson is real. Your Gemma guest path [11:55] already does content routing and refuses [11:57] to pretend dropped memory is zero. Um, [12:01] Solaten's better idea is the same [12:03] spirit. Threshold instead of a rigid top [12:06] K and approximate the tail instead of [12:08] deleting it. Still those two thoughts [12:10] for routed span if they earn their keep. [12:13] Do not import a DIT kernel and call it a [12:16] serving architecture. Soul at tames the [12:19] attention multiply. WKVM changes what [12:23] you are serving.