Cursor Megakernel Bypasses CPU Bottlenecks in Blackwell MoE Training

Xiaol.x

Shifting token routing and buffer reuse entirely onto GPU multiprocessors eliminates host-synchronization stalls, raising distributed training throughput by up to 1.41 times on tuned Blackwell clusters.

AI-assisted summary of Mixture of Kittens Explained: How Cursor’s GPU Megakernel Accelerates MoE Training — original source. Check the original for full context.

Source published: 2026-08-13

Key ideas

Cursor developed Mixture of Kittens (MoK), a GPU-resident megakernel for NVIDIA Blackwell systems that unifies inter-GPU communication, scheduling, and expert FFN computation into a single persistent launch to eliminate MoE pipeline bubbles. (0:00)
Using pull-based dispatch, push-based combine via PyTorch symmetric memory, and an on-chip macrobatch ring buffer, MoK dynamically routes tokens across GPUs without dropping tokens or synchronizing with the host CPU. (7:21)
On an NVIDIA GB300 NVL72 rack running Qwen 3.5 (EP64), MoK delivered up to a 2.37× speedup in forward MXFP8 over Megatron baselines, alongside a 1.41× end-to-end speedup on Cursor's internal 512-GPU training cluster. (19:24)
MoK is tailored specifically for high-bandwidth Blackwell NVLink domains and is not universally faster, exhibiting a 1.5% slowdown compared to DeepEP on DeepSeek V4 Pro BF16 backward microbenchmarks. (19:24)

Detailed notes

## Background Published on YouTube by Xiaol.x on August 13, 2026, this technical breakdown examines Cursor's Mixture of Kittens (MoK), a GPU-resident megakernel built to accelerate distributed Mixture-of-Experts training. The video explores how MoK tackles communication, scheduling, and host synchronization stalls on NVIDIA Blackwell architectures by unifying token routing, grouped GEMM execution, and combine phases into a persistent device-side runtime.

## TL;DR While Mixture-of-Experts models reduce theoretical arithmetic, distributed token routing and host-side CPU orchestration routinely turn sparse computation into a severe systems bottleneck. Cursor resolves this with Mixture of Kittens, an architecture-specific megakernel on NVIDIA Blackwell systems that partitions GPU Streaming Multiprocessors into dedicated asynchronous communication workers and persistent grouped-GEMM compute clusters. Utilizing symmetric memory for pull-based dispatch and push-based combine, alongside a macrobatch ring buffer, MoK handles dynamic token loads on-device without dropping tokens or stalling for host synchronization. Microbenchmarks demonstrate up to a 2.37× forward speedup on Qwen 3.5 and a 1.41× end-to-end training boost on Cursor's 512-GPU stack, although results are workload-dependent and can yield minor regressions on models with mismatched communication-to-compute profiles.

## Atmosphere & style The presentation carries a rigorous, systems-first engineering tone that dissects hardware traces and execution pipelines with meticulous precision. It feels like an advanced systems architecture seminar, replacing superficial optimization claims with an unvarnished examination of why modern tensor cores sit idle waiting on host control planes. The delivery is pragmatic and analytical, celebrating the ingenuity of autonomous device-resident scheduling while maintaining intellectual honesty about tuning complexities, proprietary baseline limitations, and hardware-specific boundaries.

## Bulletin - [00:00](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=0s) Cursor developed Mixture of Kittens (MoK), a GPU-resident megakernel for NVIDIA Blackwell systems that unifies inter-GPU communication, scheduling, and expert FFN computation into a single persistent launch to eliminate MoE pipeline bubbles. - [07:21](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=441s) Using pull-based dispatch, push-based combine via PyTorch symmetric memory, and an on-chip macrobatch ring buffer, MoK dynamically routes tokens across GPUs without dropping tokens or synchronizing with the host CPU. - [19:24](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=1164s) On an NVIDIA GB300 NVL72 rack running Qwen 3.5 (EP64), MoK delivered up to a 2.37× speedup in forward MXFP8 over Megatron baselines, alongside a 1.41× end-to-end speedup on Cursor's internal 512-GPU training cluster. - [19:24](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=1164s) MoK is tailored specifically for high-bandwidth Blackwell NVLink domains and is not universally faster, exhibiting a 1.5% slowdown compared to DeepEP on DeepSeek V4 Pro BF16 backward microbenchmarks.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=0s) Despite activating only a fraction of parameters, distributed MoE layers routinely consume over half of total training runtime because token transport across GPUs creates a dense systems bottleneck. - [00:00](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=0s) On ultra-fast accelerators like NVIDIA GB300, GPU streams outpace host CPU execution, causing compute engines to sit idle waiting for host-side dispatch orchestration. - [01:50](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=110s) Standard pipelining approaches suffer from high launch overhead and barrier latency, while dynamic routing forces undesirable compromises between convergence-harming token dropping and serialization-inducing host synchronization. - [05:14](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=314s) MoK eliminates host-side scheduling bottlenecks by launching a persistent megakernel (`dispatch_mlp_swiglu_combine`) that manages dispatch, grouped GEMMs, SwiGLU, and combine inside a single GPU runtime context. - [05:14](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=314s) Rather than relying on CUDA streams or green contexts, MoK enforces deterministic SM partitioning directly in software, dedicating specific SMs to communication while reserving the rest for tensor core compute clusters. - [07:21](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=441s) Following a router all-gather, an on-device scheduler tallies expert assignments, aligns token buffers to 256-row grouped GEMM tile boundaries, and builds local itinerary arrays directly in GPU memory. - [07:21](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=441s) PyTorch symmetric memory enables a pull-based dispatch and push-based combine architecture where communication SMs use TMA to directly read and write peer buffers without cross-rank signaling. - [10:34](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=634s) Fine-grained device counters (`x_routed_ready` and `y_routed_ready`) coordinate communication and compute SMs, allowing next-minibatch token pulls to overlap completely with current-minibatch GEMM arithmetic. - [10:34](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=634s) Compute clusters dynamically pull work tiles using Blackwell Cluster Launch Control (CLC), balancing workload distribution across grouped GEMM projections and elementwise SwiGLU activations. - [12:20](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=740s) MoK implements a pre-allocated macrobatch ring buffer on the GPU to handle dynamic token count variations without stalling for runtime host memory reallocations. - [12:20](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=740s) The host CPU is kept out of steady-state execution: the megakernel reads a one-element `num_tokens` device tensor, computes its own loop boundaries, and self-terminates excess thread block clusters. - [15:40](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=940s) MoK exposes five primary tuning knobs—forward/backward comm SM counts, minibatch size, macrobatch size, and schedule capacity—allowing engineers to systematically shift bottleneck placement across hardware limits. - [15:40](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=940s) Deterministic memory schedules guarantee bitwise reproducible accumulation orders across training runs, supporting both pure BF16 and fused MXFP8 precision formats. - [19:24](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=1164s) In EP64 benchmarks on a GB300 NVL72 rack, MoK achieved up to 531.1 TFLOPs/s on Qwen 3.5 MXFP8 forward passes (a 2.37× speedup over Megatron baselines) and 1.92× in BF16 forward. - [19:24](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=1164s) Speedups are workload-dependent: MoK trailed DeepEP by 1.5% on DeepSeek V4 Pro BF16 backward passes, demonstrating that efficiency gains depend strictly on the ratio of arithmetic intensity to communication volume. - [19:24](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=1164s) Cursor reported an end-to-end training throughput increase from 760.9 to 1,070.2 tokens/sec/GPU (1.41×) across 512 GB300 GPUs, though specific production model code and cluster setups remain proprietary. - [23:14](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=1394s) The project conceptualizes the megakernel not as an oversized math operator, but as an on-device factory that coordinates memory transport, scheduling, and compute without CPU intervention.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=0s) The Sparse-Compute Paradox

A Mixture-of-Experts (MoE) architecture sounds deceptively inexpensive on paper because it activates only a tiny subset of its experts for any given token. In Cursor's real-world production training stacks, however, reality contradicted theory: the expert layer routinely ate up more than half of the total end-to-end training runtime. This discrepancy highlights a fundamental systems dilemma: "sparse arithmetic creates a dense systems problem." While the floating-point operations drop dramatically compared to a dense model, the tokens now have to be routed dynamically across physical GPUs, packed into alignments that hardware tensor cores can ingest efficiently, and eventually shipped back to their home ranks.

The conventional distributed MoE pipeline appears rational at first glance. A router module identifies each token's top-$k$ experts. A dispatch phase issues copies of those tokens to the ranks hosting the chosen experts. Once received, each expert executes its feed-forward network—consisting of a gate projection, an up projection, a SwiGLU activation, and a down projection. Finally, a combine step routes the calculated activations back to each token's originating rank. The breakdown occurs because inter-GPU communication takes roughly as long as the expert computation itself. When these stages are treated as independent, sequential software steps, massive hardware-idling gaps appear.

Pipelining the workload helps, but chopping operations into fine chunks introduces severe launch overhead, barrier latency, and host-side orchestration bottlenecks. On state-of-the-art platforms like NVIDIA GB300 systems, the GPU execution stream runs so fast that it overtakes the host Grace CPU, sitting completely idle while waiting on host instructions. Tackling this bottleneck requires moving beyond isolated kernel optimizations like writing a faster matrix multiplication. The entire routed token journey must remain resident on the GPU and overlap at fine granularity. Cursor’s Mixture of Kittens (MoK) addresses this with a device-resident megakernel—a single persistent GPU program coordinating communication, routing, and expert computation from within the device itself.

[01:50](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=110s) The Broken Overlap Staircase and Dynamic Routing

When an MoE layer is distributed across an accelerator cluster, expert parallelism introduces a second machine to coordinate: the interconnect network. To see where the execution trace fractures, consider a simplified toy batch consisting of four tokens and four routed experts distributed across two GPUs. If the gating router sends Token A to Expert 3 and Token B to Expert 1, and those expert parameters are hosted on separate ranks, an all-to-all dispatch transfer is required to move the token activations. Once the feed-forward math completes, an all-to-all combine transfer returns the resulting vectors to their origin rank before a final router-weighted sum is computed.

```mermaid flowchart TD accTitle: Sequential MoE Execution Pipeline vs Overlap accDescr: Diagram contrasting naive sequential MoE phase stalls with the chunked pipelining staircase. subgraph Sequential["Naive Sequential Pipeline (Idle Gaps)"] direction LR S1["All-to-All Dispatch\n(Compute Idle)"] --> S2["Expert FFN GEMMs\n(Network Idle)"] S2 --> S3["All-to-All Combine\n(Compute Idle)"] S3 --> S4["Weighted Sum Reduce"] end subgraph Pipelined["Pipelined Overlap Staircase"] direction LR P1["Transfer Chunk 0"] --> P2["Compute Chunk 0\n& Transfer Chunk 1"] P2 --> P3["Combine Chunk 0,\nCompute Chunk 1,\n& Transfer Chunk 2"] end ```

In a standard sequential implementation, each stage halts until the prior stage finishes across all participating devices. The hardware trace displays glaring inefficiencies: compute units sit completely dark while the network runs dispatch, and the network sits dark while tensor cores churn through expert feed-forward networks. The textbook systems response is pipelining: slice the batch into chunks, compute on chunk zero while chunk one is in transit, and overlap transfers in a staircase pattern. Yet this staircase demands constant coordination handoffs—checking whether a chunk has arrived, launching a matrix multiply, and triggering the combine communication the instant an expert output clears. Implementing this minibatching and ring-buffer macrobatching without kernel fusion introduces numerous separate kernel launches, compounding launch boundary latency.

Dynamic routing further complicates pipelined execution because a destination rank cannot know a priori how many tokens will be routed to it. Standard implementations resort to one of two undesirable compromises: - **Fixed-capacity buffers with token dropping:** Destination buffers are statically allocated, and any routing overflow is simply discarded. Cursor explicitly rejects this strategy because dropping tokens degrades training quality and convergence. - **Host-synchronized buffer allocation:** The GPU gathers dynamic per-rank token counts and synchronizes with the host CPU so it can allocate exact-fit buffers. This synchronization serializes execution. On high-throughput GB300 systems, the GPU stream rapidly catches up to the Grace CPU thread and stalls, proving that "the control plane can starve the data plane."

[05:14](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=314s) Defining the Megakernel and SM Partitioning

Even if the host CPU is removed from the critical scheduling loop, overlapping communication with computation presents a delicate balance. If chunk transfers are partitioned too finely, tensor cores repeatedly hit pipeline synchronization barriers before reaching full arithmetic saturation. If chunk transfers are sized too coarsely, the execution engine experiences long pipeline bubbles at the front of the schedule and during the final combine phase.

```mermaid flowchart TD accTitle: Device-Resident SM Partitioning Architecture accDescr: How Mixture of Kittens partitions GPU streaming multiprocessors into dedicated communication and compute workers. subgraph GPU["Device-Resident Megakernel (dispatch_mlp_swiglu_combine)"] subgraph CommSMs["Communication SMs (Tunable Allocation)"] C1["TMA Async Pull Dispatch"] C2["TMA Async Push Combine"] end subgraph Sync["Fine-Grained Readiness Counters"] RC1["x_routed_ready"] RC2["y_routed_ready"] end subgraph ComputeClusters["Compute SM Clusters (Persistent Grid)"] W1["Gate & Up Grouped GEMMs"] W2["SwiGLU Elementwise"] W3["Down Grouped GEMM"] W4["Blackwell CLC Work Stealing"] end C1 -->|"increments"| RC1 RC1 -->|"unblocks"| W1 W1 --> W2 --> W3 W3 -->|"increments"| RC2 RC2 -->|"unblocks"| C2 end ```

To coordinate these tasks, Mixture of Kittens abandons traditional execution boundaries and assigns dedicated roles to hardware Streaming Multiprocessors (SMs), mediating data readiness through localized atomic counters. Cursor attempted using separate CUDA streams isolated via green contexts to enforce this division, but found them unreliable for deterministic, hardware-level SM partitioning. Instead, MoK relies on strict software-level ownership inside a single persistent kernel launch. In the public PyTorch bindings, this fused forward operation is exposed as `dispatch_mlp_swiglu_combine`.

- A megakernel is not a single, monolithic matrix multiplication; it is a long-running, persistent GPU program that houses dispatch, expert feed-forward calculations, and combine communication inside one runtime context. - Instructions from dispatch, grouped GEMMs, SwiGLU activations, and combine routines are interleaved at the SM task level rather than separated by distinct kernel launch barriers. - The megakernel is only necessary when three distinct pressures collide: token activations must cross physical GPU boundaries, fine-grained pipelining creates prohibitive handoff overhead, and dynamic token routing makes host-side synchronization too costly. - Latency-critical scheduling is moved onto the GPU silicon, while operations outside the primary communication-compute overlap loop—such as the final weighted sum epilogue—remain decoupled as separate functional steps.

[07:21](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=441s) Constructing the Device-Side Schedule and Symmetric Memory

MoK implements a persistent grid pattern on NVIDIA Blackwell architectures. Rather than retiring when an operation concludes, resident thread block clusters query Blackwell Cluster Launch Control (CLC) to fetch the next unfinished task block, staying active until the device-side task grid is fully drained. Before entering the megakernel, all GPU ranks participate in an all-gather exchange of the router's top-$k$ expert assignments. Once these gating choices are globally visible, an on-device scheduler runs three preparatory stages:

1. **Per-Expert Token Accounting:** Each rank tallies the precise number of routed token copies headed toward each local expert from every peer rank in the cluster. 2. **Grouped GEMM Tile Alignment:** The scheduler calculates any necessary padding to round every expert’s assigned token segment up to an exact multiple of 256 rows, satisfying hardware tile requirements for high-efficiency grouped GEMMs. 3. **Data-Plane Itinerary Generation:** It constructs two parallel scheduling arrays: `peer_rank` and `peer_token_index`. A given index $R$ in these tables explicitly dictates which peer rank and source index routed row $R$ must be pulled from during dispatch, and where its final result must be pushed back during combine. Structural padding entries are assigned a sentinel rank value of `-1`.

```mermaid flowchart LR accTitle: Symmetric Memory Direct Pull Dispatch and Push Combine accDescr: Token activations moving via TMA directly between symmetric memory buffers without intermediate local copies. subgraph RemoteRank["Peer GPU Rank"] R_In["Input Buffer\n(Token Activations)"] R_Out["Combine Slot\n(Final Accumulation)"] end subgraph LocalRank["Local GPU Rank"] CommSM["Communication SM\n(TMA Engine)"] LocalBuf["Contiguous Expert Rows\n(256-Row Aligned)"] ComputeSM["Compute SMs\n(Grouped GEMMs / SwiGLU)"] end R_In -->|"Direct Pull (TMA Load)"| CommSM CommSM --> LocalBuf LocalBuf --> ComputeSM ComputeSM -->|"Local Intermediate"| LocalBuf LocalBuf --> CommSM CommSM -->|"Direct Push (TMA Store)"| R_Out ```

This device-side itinerary functions seamlessly because all expert-parallel ranks allocate identically sized symmetric memory buffers via PyTorch’s symmetric memory rendezvous mechanisms. This exchange provides every GPU with direct peer memory pointers into both the input and combine buffers across the entire cluster.

By taking advantage of symmetric memory, MoK adopts a **pull-based dispatch** and **push-based combine** architecture: - **Pull Dispatch:** A local communication SM uses remote pointers to pull token vectors directly from the source rank's memory into a contiguous, 256-row-aligned local expert buffer, completely bypassing any intermediate local reordering steps. - **Push Combine:** When calculation concludes, the communication SM writes the completed activation row straight into the remote origin rank’s target combine buffer slot. - **Elimination of Cross-Rank Signaling:** Because the local GPU initiates both the pull and the push, it maintains immediate local visibility over data readiness. It does not need to wait for or process a flood of incoming completion messages from remote ranks; it simply updates an internal register or shared-memory counter to notify local compute clusters that a segment is ready.

[10:34](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=634s) Communication vs. Compute SMs in the Minibatch Pipeline

Inside the launched megakernel, the GPU’s physical Streaming Multiprocessors are split into two explicit operational groups. The first group consists of a user-tunable number of dedicated Communication SMs. These multiprocessors execute network-bound dispatch and combine tasks using Blackwell's Tensor Memory Accelerator (TMA) for asynchronous loads and stores. The remaining multiprocessors form Compute Clusters dedicated entirely to high-throughput tensor core arithmetic: the gate and up grouped GEMMs, the elementwise SwiGLU activation, and the down grouped GEMM.

```mermaid flowchart TD accTitle: Minibatch 0 Step-by-Step Execution Trace accDescr: Chronological walkthrough of Minibatch 0 from token pull to combine push. T1["1. Comm SM reads schedule:\nToken 17 (Copy 1) on Peer Rank 2"] --> T2["2. TMA Pull into local routed row 0"] T2 --> T3["3. Comm SM increments x_routed_ready for Minibatch 0"] T3 --> T4["4. Compute Cluster unblocks:\nExecutes Gate & Up Grouped GEMMs"] T4 --> T5["5. SwiGLU elementwise multiply:\n(swish(gate) * up)"] T5 --> T6["6. Compute Cluster executes Down Grouped GEMM"] T6 --> T7["7. Compute Cluster increments y_routed_ready for Minibatch 0"] T7 --> T8["8. Comm SM unblocks:\nTMA Push result to Peer Rank 2 combine slot"] T8 --> T9["9. Epilogue: Weighted sum across top-k copies"] ```

Work distribution among the compute clusters is handled dynamically via Blackwell Cluster Launch Control, enabling clusters to steal whichever task tile is next in line across the device-resident task grid. Sizing the communication SM pool requires precise balancing: allocating too few communication SMs causes the interconnect to starve, leaving tensor cores waiting for input, while dedicating too many drains the hardware capacity needed for intensive matrix operations.

The lifecycle of an individual execution unit illustrates how this coordination operates without host intervention: - **Schedule Lookup & TMA Pull:** On Rank 0, a communication SM inspecting Minibatch 0 reads its device schedule and identifies that top-$k$ copy 1 of Token 17 resides on Peer Rank 2 and targets a local expert. The SM pulls that vector across NVLink directly into local routed row 0. - **Signaling Compute Readiness:** Once the minibatch transfer completes, the communication SM increments the device counter `x_routed_ready` for Minibatch 0. - **FFN Computation:** A persistent compute cluster waiting on `x_routed_ready` wakes up and launches the expert's gate and up projections. The SwiGLU activation multiplies the activated gate branch with the up branch, and the down projection maps the representations back down to the model's hidden width. - **Signaling Combine Readiness:** As soon as all required output tiles for Minibatch 0 are written into local buffers, the compute worker increments `y_routed_ready`. - **TMA Push Combine:** The waiting communication SM detects the updated counter and pushes the finished activation vector across NVLink directly into Peer Rank 2's combine buffer slot for Token 17. - **Decoupled Epilogue:** Finally, an independent epilogue applies the gating router weights and accumulates the finished top-$k$ copies.

Because readiness tracking is managed through fine-grained, device-local counters, the megakernel achieves full hardware overlap: while compute clusters process Minibatch 0, the communication SMs are already pulling activations for Minibatch 1.

[12:20](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=740s) Minibatch Pipelining and Macrobatch Ring-Buffer Reuse

The core pipeline operates on a producer-consumer model: communication SMs produce ready minibatches, while compute SMs consume them to perform expert work and yield finished minibatches. As soon as minibatch 0 finishes its down-projection, the combine phase can begin immediately without being gated on minibatch 1's expert computation. Setting the minibatch size requires balancing two distinct failure modes: making minibatches too tiny causes frequent signaling overhead and underfills tensor core waves, whereas making them too huge delays both the first compute wave and the ultimate combine step. The project recommends sizing minibatches to provide roughly enough work for two full compute waves.

Because expert routing is dynamic, the host CPU cannot know in advance the exact number of arriving tokens. To eliminate dynamic runtime host allocations, Mixture of Kittens (MoK) pre-allocates a fixed routed token buffer called the macrobatch. When the routed rows exceed this buffer's physical capacity, the storage is reused as a ring buffer. The forward path processes macrobatches in reverse order: at a ring boundary, the communication group combines macrobatch $M$ while simultaneously dispatching macrobatch $M-1$ into the released memory region. Release and acquire counters in GPU memory prevent data overwrites until the grouped GEMM and combine reads are fully completed. By draining and immediately refilling each slot, fixed memory capacity is preserved without requiring host round-trips or dropping tokens.

```mermaid flowchart LR accTitle: Macrobatch Ring Buffer Lifecycle accDescr: Pipeline showing cyclic buffer reuse between communication dispatch, compute, combine, and slot release without host synchronization.

subgraph RingSlot["Macrobatch Ring Slot"] D["Dispatch (Pull Tokens)"] C["Compute (Grouped GEMM)"] CB["Combine (Push Tokens)"] end

Sync["Counters (Release/Acquire)"]

D --> C C --> CB CB -->|"Drain slot"| Sync Sync -->|"Refill slot (M-1)"| D ```

- Steady-state execution completely bypasses the host: the CPU never queries token counts to allocate memory. - The router writes total token counts (`num_tokens`) as a one-element CUDA tensor directly on the device. - The megakernel reads this single-element tensor on-chip, calculates macrobatch and minibatch counts, and self-exits excess thread block clusters. - The host CPU enqueues only high-level steps (scheduler, megakernel barrier, weighted epilogue) and never synchronizes across dispatch, compute, and combine stages. - Minibatch pipelining replaces the sequential latency sum ($\text{Comm} + \text{MLP}$) with an overlapped maximum: $\text{Fill} + (M - 1) \times \max(\text{Comm Service}, \text{MLP Service}) + \text{Drain}$.

---

[15:40](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=940s) Tuning Controls and the Five Knobs of Bottleneck Placement

MoK converts the entire distributed MoE pipeline—scheduling, dispatch, grouped expert FFNs, combine, and router-weight application—into a single schedule-driven megakernel. This architectural shift exposes five critical tuning knobs that allow engineers to shift bottleneck placement rather than just adjusting cosmetic configurations:

```mermaid flowchart TD accTitle: Megakernel Resource Partitioning Controls accDescr: Diagram illustrating how MoK splits streaming multiprocessors and manages buffers via five tunable parameters.

subgraph SM_Allocation["SM Allocation"] CSM["Comm SM Count (Forward & Backward)"] KSM["Compute SMs (Tensor Core GEMMs)"] end

subgraph Data_Chunking["Granularity & Buffer Sizing"] MB["Minibatch Size (Handoff Grain)"] MCB["Macrobatch Size (Ring Window)"] SC["Schedule Capacity (Imbalance Headroom)"] end

- **Forward Communication SM Count**: Allocates dedicated SMs to drain network traffic; adding SMs accelerates transfers but robs compute capacity from tensor cores. - **Backward Communication SM Count**: Independently tuned because backward passes involve reverse communication, replay decisions, three activation gradient GEMMs, three weight gradient GEMMs, and router weight gradient calculations. - **Minibatch Size**: Controls handoff granularity; small sizes allow earlier compute overlap but incur barrier overhead, while large sizes ensure full tensor core waves at the expense of pipeline latency. - **Macrobatch Size**: Configures the reusable memory window for the ring buffer, preventing out-of-memory errors on skewed distributions. - **Schedule Capacity**: Reserves buffer headroom to absorb dynamic token routing imbalances across ranks without dropping tokens.

The CUDA scheduler deterministically fixes which source rank and token index fills each padded expert slot, matching an exact reference visitation order. In addition, floating-point accumulation order is deterministic across runs, preserving bitwise reproducibility for ablations and on-policy RL training. Precision modes support pure BF16 or MXFP8 (where expert weights are pre-quantized with microscopic block scales, activation quantization is fused directly into the pipeline, and shared experts remain in BF16 for numerical stability).

---

[19:24](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=1164s) Benchmark Findings: GB300 NVL72 and Internal Cluster Evidence

Evaluated on a single Nvidia GB300 NVL72 rack at expert parallel degree 64 (EP64) with 48 tokens per GPU prior to routing, MoK was benchmarked against the fastest public baselines (NCCL + PyTorch, DeepEP + PyTorch, DeepEP + Transformer Engine, and Hybrid EP + Megatron):

The data confirms that MoK is a specialized workload optimization rather than an unqualified win across every possible architecture. On the DeepSeek V4 Pro BF16 backward benchmark, MoK runs roughly 1.5% slower than the DeepEP baseline, illustrating that performance gains hinge directly on compute-to-communication balance and expert sizing.

Beyond standalone layer microbenchmarks, Cursor tested an end-to-end production setup across 512 GB300 GPUs spanning multiple NVL72 racks. Swapping out their prior DeepEP-based custom MXFP8 pipeline for MoK elevated training throughput from 760.9 tokens/sec/GPU to 1,070.2 tokens/sec/GPU—a 1.41× end-to-end acceleration. However, because production model code, run configurations, and cluster logs remain proprietary, external practitioners cannot independently replicate this 1.41× end-to-end metric from the public repository alone.

- The public codebase requires Python 3.12+, PyTorch 2.10+, and CUDA 13+ compiled for Nvidia Blackwell SM100 or SM103 architectures (e.g., GB200, GB300 NVL72). - The functional implementation relies fundamentally on PyTorch symmetric memory across the expert parallel domain.

---

[23:14](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=1394s) The Resident Factory: Eliminating Three Architectural Bubbles

The architectural thesis behind Mixture of Kittens is best understood as a "resident factory, not an oversized operator." Rather than repeatedly launching discrete kernels and synchronizing through the host CPU, a single GPU-resident megakernel orchestrates worker groups: one team of SMs pulls and pushes network tokens while another team runs GEMMs on tensor cores, passing minibatches back and forth via hardware counters.

MoK systematically tackles three primary sources of pipeline bubbles:

1. **Scheduling & Cross-GPU Overhead**: Pull-based dispatch and push-based combine ensure destination ranks independently fetch needed inputs and push outputs, eliminating cross-GPU completion signals and host barriers. 2. **Compute Wave Starvation**: Granular, tunable minibatches prevent tensor core underutilization while avoiding idle delay waiting for large transfers. 3. **Dynamic Host Allocations**: The macrobatch ring buffer dynamically recycles symmetric memory on-chip, processing fluctuating token loads without dropping tokens or stalling for host reallocations.

MoK demonstrates that "the megakernel achievement is not 'fusion makes everything faster.'" Instead, its strength lies in co-designing the scheduling, transport, GEMM compute, and memory buffers into a unified device-resident system. By making SM ownership explicit and streaming dynamic tokens through ring buffers within high-bandwidth NVLink domains, MoK shows how to remove boundary overheads when standard distributed pipelines stall.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=0s) sparse arithmetic creates a dense systems problem. — Xiaol.x

> [01:50](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=110s) the control plane can starve the data plane. — Xiaol.x

> [05:14](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=314s) dispatch_mlp_swiglu_combine — Xiaol.x

> [07:21](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=441s) peer_rank — Xiaol.x

> [07:21](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=441s) peer_token_index — Xiaol.x

> [10:34](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=634s) x_routed_ready — Xiaol.x

> [10:34](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=634s) y_routed_ready — Xiaol.x

> [23:14](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=1394s) resident factory, not an oversized operator. — Xiaol.x

> [23:14](https://www.youtube.com/watch?v=hlzBiTt6h7M&t=1394s) the megakernel achievement is not 'fusion makes everything faster.' — Xiaol.x

## Mentioned Cursor, Mixture of Kittens, NVIDIA, NVIDIA GB300, NVIDIA GB200, NVL72, Blackwell, SM100, SM103, Grace CPU, Tensor Memory Accelerator (TMA), Cluster Launch Control (CLC), CUDA, PyTorch, NCCL, DeepEP, Transformer Engine, Hybrid EP, Megatron, Qwen 3.5, DeepSeek V4 Pro

## Video description Mixture-of-Experts models activate only a small fraction of their experts, but routing tokens across GPUs can still consume a large share of training time. Cursor’s Mixture of Kittens addresses this with a GPU-resident megakernel that coordinates communication, expert computation, scheduling, and result combination.

This English explainer walks through:

- Why ordinary MoE pipelines stall at CPU, launch, and synchronization boundaries - How device-side routing schedules are constructed - Pull-based dispatch and push-based combine - Communication-SM and compute-SM partitioning - Minibatch overlap and macrobatch ring-buffer reuse - Public GB300/NVL72 benchmark results and their limitations - Why the approach targets specific Blackwell workloads rather than being a universal GPU optimization

Chapters

00:00 The Sparse-Compute Paradox 01:50 Why Ordinary MoE Pipelines Stall 07:21 Inside the MoK Megakernel 15:40 Evidence, Tuning, and Boundaries 23:14 A Device-Resident Factory

Sources

Repository: https://github.com/cursor/mixture-of-kittens Technical article: https://cursor.com/blog/mixture-of-kittens

The reported speedups are workload-dependent. The 1.41× end-to-end comparison comes from Cursor’s private 512-GPU production stack, while the released implementation targets Blackwell SM100/SM103 systems and requires workload-specific tuning.

#MixtureOfExperts #Megakernel #GPUComputing #CUDA #DeepLearning #AIInfrastructure

[Source](https://www.youtube.com/watch?v=hlzBiTt6h7M)

中文简报

Cursor Megakernel 规避 Blackwell MoE 训练中的 CPU 瓶颈将 token 路由和缓冲区复用完全转移至 GPU 多处理器，消除了主机同步停顿，在调优的 Blackwell 集群上将分布式训练吞吐量提升高达 1.41 倍。尽管混合专家（MoE）模型减少了理论计算量，但分布式 token 路由和主机侧 CPU 编排往往会使稀疏计算成为严重的系统瓶颈。Cursor 针对 NVIDIA Blackwell 系统推出了特定架构的 megakernel——Mixture of Kittens (MoK)，通过将 GPU 流式多处理器划分为专用异步通信工作单元和持久化 grouped-GEMM 计算集群来解决这一难题。MoK 利用对称内存进行基于拉取的分发与基于推送的合并，并结合 macrobatch 环形缓冲区，在设备端直接处理动态 token 负载，既无需丢弃 token，也无需因主机同步而发生停顿。微基准测试显示，其在 Qwen 3.5 上的前向传播速度最高提升 2.37 倍，并在 Cursor 的 512-GPU 训练集群上实现了 1.41 倍的端到端提速；不过测试结果取决于具体工作负载，在通信与计算配置不匹配的模型上可能会出现轻微的性能回退。 Cursor 开发了面向 NVIDIA Blackwell 系统的 GPU 驻留 megakernel——Mixture of Kittens (MoK)，将 GPU 间通信、调度和专家 FFN 计算统一整合至单次持久启动中，以消除 MoE 流水线气泡。 (0:00) 借助基于拉取的分发、通过 PyTorch 对称内存实现的基于推送的合并，以及片上 macrobatch 环形缓冲区，MoK 能够在跨 GPU 动态路由 token 的同时，无需丢弃 token 或与主机 CPU 进行同步。 (7:21) 在运行 Qwen 3.5 (EP64) 的 NVIDIA GB300 NVL72 机架上，相比 Megatron 基线，MoK 在前向 MXFP8 计算中实现了高达 2.37 倍的加速，并在 Cursor 内部的 512-GPU 训练集群上实现了 1.41 倍的端到端提速。 (19:24) MoK 专为高带宽 Blackwell NVLink 域量身定制，并非在所有场景下均表现更优，在 DeepSeek V4 Pro BF16 反向微基准测试中相比 DeepEP 出现了 1.5% 的性能下降。 (19:24)

Reading summary

Cursor's Mixture of Kittens (MoK) architecture addresses distributed Mixture-of-Experts training bottlenecks by moving routing, transport, and computation into a single device-resident GPU megakernel. Evaluated on NVIDIA Blackwell hardware, the system overlaps communication and tensor math by partitioning Streaming Multiprocessors directly on-chip rather than relying on host CPU orchestration. The Sparse Compute Bottleneck and Host Starvation (0:00) In Cursor's production training stacks, distributed Mixture-of-Experts (MoE) layers routinely consumed over half of the end-to-end execution time despite theoretical reductions in floating-point operations. The bottleneck stems from systems-level overhead: dynamic token routing across GPUs requires all-to-all dispatch and combine steps that take roughly as long as the expert feed-forward calculations themselves. Standard sequential pipelines leave tensor cores idle during network transfers and idle the interconnect during matrix multiplication. Traditional attempts to pipeline transfers into smaller chunks encounter severe barrier overhead. On high-throughput architectures like NVIDIA GB300 systems, the GPU execution stream outpaces the host Grace CPU. When frameworks rely on CPU-side synchronization to count tokens and allocate buffers dynamically, the control plane starves the data plane, stalling high-speed hardware while waiting on host instructions. Hardware SM Partitioning and Persistent Scheduling (5:14) MoK eliminates host-side pipeline management by consolidating execution into a device-resident megakernel, exposed in PyTorch as dispatch_mlp_swiglu_combine. Rather than launching separate sequential kernels, MoK establishes a persistent thread block grid on the GPU and enforces software-level ownership over physical Streaming Multiprocessors (SMs), partitioning them into dedicated communication workers and compute clusters. Communication SMs utilize Blackwell's Tensor Memory Accelerator (TMA) for asynchronous data movement, while compute clusters execute grouped GEMMs (gate, up, and down projections) and SwiGLU activations, using Blackwell Cluster Launch Control (CLC) for dynamic work stealing. Coordination between the two groups occurs entirely on-chip through localized readiness counters (x_routed_ready and y_routed_ready), allowing compute clusters to process one minibatch while communication SMs concurrently transfer the next. Symmetric Memory Transport and Ring-Buffer Reuse (7:21) Data movement in MoK relies on PyTorch symmetric memory allocations across the expert parallel domain, establishing direct cluster-wide peer pointers into input and combine buffers. MoK implements a pull-based dispatch and push-based combine strategy: local communication SMs directly pull assigned tokens from remote GPUs into contiguous, 256-row-aligned local expert buffers and push finished outputs straight back into origin combine slots. Because the local device initiates both operations, cross-rank signaling and completion messages are entirely bypassed. To accommodate dynamic routing without dropping tokens or incurring host reallocation pauses, MoK pre-allocates a fixed macrobatch memory window managed on-chip as a ring buffer. The GPU router writes total token counts to a single-element device tensor, and thread blocks cyclically drain and refill macrobatch slots in reverse order using on-device release and acquire counters, maintaining a steady-state pipeline without CPU intervention. Empirical Gains and Workload Trade-offs (19:24) MoK exposes five structural tuning knobs: forward and backward communication SM counts, minibatch size, macrobatch size, and schedule capacity headroom. The CUDA scheduler enforces deterministic token assignment order and floating-point accumulation, ensuring bitwise reproducibility across runs in pure BF16 or block-scaled MXFP8 precision. Benchmarked on an NVIDIA GB300 NVL72 rack at expert parallel degree 64 (48 tokens per GPU), MoK accelerated Qwen 3.5 MXFP8 forward passes by 2.37× over a Hybrid EP and Megatron baseline (achieving 531.1 TFLOPs/s versus 224.4 TFLOPs/s) and achieved a 1.92× speedup in BF16 forward execution. However, the system is not universally superior across all configurations: on a DeepSeek V4 Pro BF16 backward microbenchmark, MoK ran 1.5% slower than the DeepEP baseline, demonstrating that performance hinges directly on specific expert dimensions and compute-to-communication balances. Cursor reports that deploying MoK across a 512 GB300 cluster increased production training throughput from 760.9 to 1,070.2 tokens per second per GPU (a 1.41× improvement); however, external verification of this end-to-end figure is constrained by proprietary model code and cluster setups. Additionally, MoK requires NVIDIA Blackwell SM100 or SM103 architectures and PyTorch symmetric memory, limiting its utility strictly to specialized, modern NVLink environments.

Source quotes

[0:00] Xiaol.x: sparse arithmetic creates a dense systems problem. [1:50] Xiaol.x: the control plane can starve the data plane. [5:14] Xiaol.x: dispatch_mlp_swiglu_combine [7:21] Xiaol.x: peer_rank [7:21] Xiaol.x: peer_token_index [10:34] Xiaol.x: x_routed_ready [10:34] Xiaol.x: y_routed_ready [23:14] Xiaol.x: resident factory, not an oversized operator. [23:14] Xiaol.x: the megakernel achievement is not 'fusion makes everything faster.'

Original transcript

[0:00] A mixture of experts model activates [0:03] only a few experts for each token. That [0:05] sounds cheap. Yet in Cursor's production [0:08] workloads, the expert layer could [0:09] consume more than half of total training [0:12] time. The paradox is that sparse [0:14] arithmetic creates a dense systems [0:16] problem. Tokens must travel to the right [0:18] GPUs, arrive in an order the tensor [0:21] cores can use, and then travel home. The [0:24] ordinary pipeline is sensible. A router [0:26] chooses each token's top experts. [0:29] Dispatch sends copies to the ranks that [0:32] own those experts. Each expert runs its [0:34] gate. Up projection, swiggloo [0:37] activation, and down projection. Combine [0:40] returns the results to the token's [0:42] original rank. But when communication [0:45] takes about as long as computation, [0:48] running those stages as separate islands [0:50] leaves expensive gaps. Pipelining helps, [0:55] but every small chunk can introduce [0:57] another launch boundary, another [0:59] barrier, or another host decision. On [1:02] GB300 systems, the fast GPU stream can [1:07] even catch the integrated grace CPU and [1:10] sit idle waiting for CPU side work. The [1:13] real target is therefore larger than a [1:15] faster matrix multiply. The entire [1:18] routed expert journey must stay on the [1:20] GPU and overlap at a useful granularity. [1:24] Mixture of kittens answers that systems [1:27] question with a mega kernel. Not one [1:30] giant matrix operation, but one [1:32] persistent GPU program that coordinates [1:34] communication and expert computation [1:36] from inside the device. We will follow [1:39] one token through the baseline, the mega [1:42] kernel schedule, the ring buffer, and [1:44] the evidence. Then mark exactly where [1:46] the design does and does not apply. [1:50] A mixture of experts layer promises more [1:54] model capacity without running every [1:56] expert for every token. Yet cursor [1:59] reports that this layer remained [2:01] composer's major bottleneck, sometimes [2:04] taking more than half of endtoend [2:06] training time. The paradox is that [2:08] sparse expert choice reduces arithmetic [2:11] but expert parallelism adds a second [2:13] machine a communication network. Once [2:17] communication can take as long as [2:20] computation faster expert matrix [2:23] multiplies alone no longer fix the [2:26] layer. Start with a toy batch of four [2:29] tokens and four routed experts split [2:32] across two GPUs. The router sends token [2:35] A to expert 3 and token B to expert one. [2:39] Expert parallelism means those expert [2:42] weights live on different GPU ranks. [2:45] Dispatch is the alltoall transfer that [2:48] moves each token activation to its [2:50] selected expert. The expert runs its [2:53] feed forward network. Combine is the [2:56] alltoall transfer that returns each [2:58] expert result to the token's original [3:00] rank. A separate weighted sum uses the [3:04] router weights. The straightforward [3:07] distributed pipeline is easy to [3:09] understand. Dispatch every routed token. [3:13] Wait. Run the expert feed forward [3:16] networks. Wait. Combine every result. [3:20] Wait. Then reduce the weighted outputs. [3:24] Each phase is useful. The problem is the [3:27] empty space between them. The official [3:30] cursor explanation says communication [3:34] can take as long as computation. So [3:37] executing the two sequentially is [3:39] inefficient. In our toy trace, the [3:42] compute units sit idle during dispatch [3:44] and the network sits idle during expert [3:46] compute. The standard remedy is [3:48] pipelining. Transfer one chunk of [3:51] tokens, compute on that chunk while the [3:54] next chunk travels, and repeat. This [3:57] turns the three large blocks into an [3:59] overlap staircase. But the staircase [4:02] needs frequent handoffs. Chunk arrived. [4:05] Matrix multiply may start. Expert output [4:08] finished. Combine may start. Cursor's [4:11] mega kernel section states that [4:13] implementing mini batching and ring [4:15] buffer macro batching without fusion [4:17] would require multiple kernel launches. [4:20] Increasing launch boundary overhead. [4:23] More chunks create more opportunities to [4:27] overlap but also more um boundaries to [4:31] coordinate. Routing is dynamic. A [4:34] destination rank does not know in [4:35] advance how many tokens it will receive. [4:38] Cursor names to conventional responses. [4:41] A fixed buffer can drop overflow tokens [4:43] which they reject for training quality. [4:46] or the GPU can send per rank token [4:49] counts to the CPU which allocates exact [4:53] size buffers. Um that synchronization [4:57] prevents the GPU stream from running [4:59] ahead of the CPU thread. On GB 300 [5:03] systems cursor observed GPU streams [5:06] catching CPU side work and becoming [5:08] completely idle. The control plane can [5:11] starve the data plane. Even after [5:14] removing the CPU, overlap has a tuning [5:18] problem. Communicate too finely and [5:20] tensor cores hit barriers before they [5:23] are fully fed. Communicate too coarsely [5:25] and they wait for the first tokens in [5:27] the final combine. Moch assigns some [5:31] streaming multipprocessors to [5:32] communication and the rest to expert [5:35] computation with local counters between [5:37] them. Cursor reports that separate [5:39] streams with green contexts were [5:41] unreliable for partitioning those [5:43] multiprocessors. Exactly. The schedule [5:45] therefore needs precise device side [5:48] ownership of both kinds of work. This is [5:51] the concrete meaning of a mega kernel [5:53] here. One long GPU kernel contains [5:56] dispatch expert feed forward work and [5:59] combine and overlaps their instructions [6:02] at the streaming multi-processor task [6:05] level instead of separating them by [6:08] kernel launch boundaries. The public [6:11] binding names the fuse forward operation [6:14] dispatch MLP swigloo combine [6:18] inside the CUDA source communication [6:20] tasks and compute tasks share readiness [6:22] counters while a tunable number of [6:24] multiprocessors is reserved for [6:26] communication. [6:28] It is one device resident traffic [6:30] controller for the routed layer. So the [6:33] mega kernel is not necessary merely [6:36] because one matrix multiply is slow. It [6:39] becomes compelling when three pressures [6:41] coincide. Tokens must cross GPUs. Fine- [6:45] grained overlap creates many handoffs [6:47] and dynamic token counts make host [6:50] synchronization dangerous. Mo's answer [6:53] is a device-built schedule plus fused [6:56] forward and backward mega kernels. The [6:59] final weighted sum epilogue remains a [7:01] separate operation in the public [7:03] functional path. Keep that boundary in [7:06] mind. A mega kernel is not put [7:09] everything in one function. It is moving [7:12] the latency critical coordination onto [7:14] the GPU so communication and computation [7:18] can behave like one pipeline. [7:21] A mixture of experts layer contains fast [7:26] operations yet the whole layer can still [7:28] stall. A conventional implementation [7:31] launches dispatch communication, waits [7:34] for enough remote tokens, launches [7:36] expert matrix multiplies, launches the [7:39] swiggloo activation, launches the down [7:41] projection, then launches combined [7:44] communication. Each piece may be [7:48] optimized, but every boundary is another [7:50] chance for the network, tensor cores, or [7:53] host thread to wait. The embarrassing [7:56] result is a GPU with excellent kernels [7:59] and visible empty gaps between them. MK [8:02] attacks those gaps not by changing the [8:05] expert equation but by changing who [8:07] orchestrates the layer. Here a mega [8:10] kernel is one GPU kernel launch that [8:13] contains the routed dispatch expert feed [8:16] forward computation and routed combine [8:20] instructions overlap at the streaming [8:22] multiprocessor task level instead of [8:25] meeting only at kernel boundaries. It is [8:28] also a persistent grid design. Resident [8:32] thread block clusters finish one task, [8:34] ask Blackwell cluster launch control for [8:37] another task and keep working until the [8:40] device side task grid is drained. So [8:43] mega does not mean one giant matrix [8:46] multiply. It means the GPU itself owns [8:50] the schedule for many different [8:53] operations while the kernel is alive. Um [8:56] before the mega kernel, every rank all [8:58] gathers the router's top expert choices. [9:01] The scheduler then performs three GPU [9:03] stages. First, it counts for each local [9:07] expert and peer rank how many routed [9:09] token copies will arrive. Second, it [9:12] pads every expert segment to a multiple [9:15] of 256 rows. The grouped GM tile [9:19] alignment. [9:21] Third, it writes two parallel arrays, [9:24] peer rank and peer token index. Row R in [9:27] these arrays says exactly where routed [9:30] row R must be pulled from and later [9:33] where its result must be pushed back. [9:35] Padding rows carry rank minus one. This [9:39] compact table is the data plane [9:41] itinerary reused by dispatch and [9:43] combine. The itinerary works because [9:47] every expert parallel rank allocates [9:49] identically shaped symmetric buffers. [9:52] PyTorch's symmetric memory rendevu gives [9:56] each GPU the peer pointers for the input [9:58] and combine buffers. A communication SM [10:03] can therefore pull token data directly [10:05] from the source rank into a contiguous [10:08] local expert row, then push the finished [10:10] row directly into the original ranks [10:12] combine slot. There is no intermediate [10:17] local reorder. Pull dispatch and push [10:20] combine also avoid a fan out of cross [10:23] rank completion messages. The GPU that [10:25] initiates the transfer knows locally [10:27] when its transfer is usable, then [10:30] updates a local counter for compute. [10:34] Inside the launch, MOK gives streaming [10:37] multiprocessors two professions. The [10:40] first configurable group is [10:41] communication SMS. They run dispatch and [10:44] combine tasks using tensor memory [10:47] accelerator loads and stores. The [10:50] remaining clusters are compute workers [10:51] for gate and upgroup gems, swiggloo, and [10:54] the downgroup gem. The split is exact [10:57] software partitioning, not a hope that [10:59] two CUDA streams will divide the chip [11:01] evenly. Compute clusters use cluster [11:04] launch control to steal the next [11:06] unfinished task. The communication count [11:09] is tunable because too few communication [11:13] SMS starve the network while too many [11:17] leave too little matrix capacity. Follow [11:21] mini batch zero on rank zero. Its [11:24] schedule row says token 17 top K copy [11:27] one lives on pure rank two and belongs [11:30] to a local expert. Here a communication [11:33] SM pulls that vector into routed row [11:36] zero and increments the X routed ready [11:38] counter for mini batch zero. A compute [11:41] cluster waits on that counter then runs [11:43] the experts gate and up projections. [11:46] Swigglu multiplies the activated gate [11:49] with the up branch and the down [11:51] projection returns to model width. When [11:54] every required output tile for this mini [11:56] batch is written, compute increments Y [11:59] routouted ready. A communication SM then [12:03] pushes the result to pure rank 2's [12:06] combine slot for token 17. The final [12:08] epilog applies router weights and sums [12:11] the top K copies. The counters make [12:14] readiness local and fine grained. While [12:18] compute works on mini batch zero, [12:20] communication can dispatch mini batch [12:23] one. When mini batch zeros down, [12:25] projection finishes, combine can start [12:27] without waiting for mini batch 1's [12:29] expert work. This is the core pipeline. [12:32] Communication produces ready mini [12:34] batches. Compute consumes them and [12:37] produces finished mini batches. MO [12:39] exposes mini batch size because [12:42] granularity has two opposite failure [12:44] modes. Tiny mini batches signal often [12:47] and underfill tensor core waves. Huge [12:51] mini batches delay the first compute and [12:53] the final combine. The project [12:56] recommends tuning this boundary uh with [12:59] a practical heruristic of enough work [13:01] for roughly two full compute waves. [13:04] Routing is dynamic. So the exact number [13:07] of arriving tokens is not known to the [13:09] CPU in advance. Moch avoids exact size [13:13] allocation by pre-allocating a fixed [13:16] routed token buffer called the macro [13:18] batch. If all routed rows do not fit at [13:21] once, the same storage is reused like a [13:24] ring. The forward code walks macro [13:26] batches in reverse order. Um at a ring [13:29] boundary, communication combines macro [13:32] batch m while dispatching macrobatch m [13:35] minus one into the corresponding [13:37] released region. counters prevent [13:39] overwrite until the old grouped GMM and [13:42] combined reads are done. The slot is [13:45] drained, then immediately refilled, so [13:47] fixed memory capacity does not force a [13:49] host round trip or token dropping. [13:52] Notice what the host never needs in the [13:56] steady forward path. It never reads the [13:59] routed token count to size a new tensor. [14:03] The scheduler writes numbum tokens as a [14:06] oneelement CUDA tensor. The mega kernel [14:10] reads that value on the device, computes [14:12] the true number of macro and many [14:14] batches and exits excess clusters [14:17] itself. Symmetric buffers were allocated [14:20] earlier and dependencies are release and [14:23] acquire counters in GPU memory. Uh the [14:26] CPU still in cues the schedule mega [14:29] kernel barrier and weighted epilogue but [14:31] it does not synchronize between dispatch [14:33] expert computation and combine setup and [14:36] workspace destruction may synchronize [14:39] the repeated mode data path does not [14:41] require host paced stage boundaries. [14:45] Now the overlap can be summarized [14:48] um for M mini batch rounds. Think of [14:51] layer time as fill plus m minus one [14:54] times the slower of per round [14:56] communication service and expert MLP [14:59] service plus drain and any ring weight. [15:02] Sequential launches pay communication [15:04] plus MLP for every round. The mega [15:08] kernel tries to replace that sum with a [15:10] maximum. This is a lens not an exact [15:14] benchmark model. Dispatch and combined [15:17] share communication SMS tails are [15:20] uneven. Expert routing is imbalanced and [15:23] CLC may yield to higher priority [15:26] interact work. The durable idea is GPU [15:30] resident orchestration. Predeclare [15:32] memory encode dependencies as counters [15:34] and let specialized SM groups advance [15:37] whenever their data is ready. [15:40] Mixture of kittens advertises [15:43] up to 2.37 [15:46] times the throughput of the fastest [15:49] public baseline. Yet one disclosed row [15:52] goes the other way on DeepSeek 54 Pro [15:56] BF16 backward mo is about 1 and a.5% [15:59] slower than deep plus pietorch. That is [16:03] not a contradiction. It is the key to [16:06] reading the evidence correctly. These [16:08] are workload specific maxima not a [16:10] promise that one kernel wins everywhere. [16:13] Uh a conventional distributed mixture of [16:16] experts layer schedules routes [16:19] dispatches tokens across GPUs runs [16:22] grouped expert feed forward networks [16:25] combines the results and then applies [16:27] router weights. Modular baselines [16:30] execute those responsibilities through [16:32] multiple operators and synchronization [16:34] points. MO turns the central dispatch, [16:37] expert compute, and combined path into a [16:40] scheduled driven mega kernel. The gain [16:43] is not merely fewer launches. One [16:47] running program can decide when network [16:49] work and matrix work hand tokens to each [16:52] other without asking the CPU to size [16:55] every dynamic buffer. Um, that [16:58] coordination exposes five controls [17:02] forward and backward. each choose a [17:06] communication SM count. More [17:09] communication SM can drain the network [17:12] faster, but every SM given to networking [17:14] is unavailable to the tensor core work. [17:17] Too few communication SMS leave compute [17:20] waiting for tokens. Mini batch size sets [17:23] the handoff grain. Tiny chunks overlap [17:26] early but pay frequent barriers and may [17:28] create partial compute waves. Huge [17:31] chunks fill the tensor cores but delay [17:33] first token arrival and leave a longer [17:35] tail. Macrobatch size sets the reusable [17:38] token ring window. Schedule capacity [17:41] reserves room for expert imbalance. [17:44] Tuning is therefore bottleneck [17:46] placement, not cosmetic configuration. [17:49] The schedule fixes which source rank and [17:52] token index fills each padded expert [17:55] segment. Tests compare the CUDA schedule [17:58] against an exact reference visitation [18:00] and interle order. During forward, MO [18:04] pulls dispatch data, computes expert [18:07] mini batches, then pushes combined [18:10] outputs. Backward reverses the traffic [18:13] and separately computes activation [18:16] weight and router gradients. Um, a ring [18:19] buffer lets dispatch refill slots after [18:22] combine drains them, avoiding token [18:25] dropping and CPU allocation round trips. [18:28] Cursor describes the floating point [18:30] order as fixed. So hardware scheduling [18:34] should not change the bits useful for [18:36] ablations and on policy training. BF16 [18:40] keeps routed activations and weights in [18:43] a 16bit format. In MXFP8 mode, routed [18:47] weights are pre-quantized to eight bit [18:49] values with block scales. Activation [18:51] quantization is fused into the data path [18:54] and the shared expert stays in BF-16 for [18:57] stability. Backward is not forward [18:59] played in reverse. It adds reverse [19:01] communication, forward replay decisions, [19:04] three activation gradient matrix [19:06] multiplications, [19:08] three weight gradient multiplications, [19:10] and router weight gradients. That is why [19:14] the implementation exposes separate [19:16] forward and backward communication SM [19:19] controls and why the benchmark reports [19:21] four different headline ratios. The [19:24] standalone test covers global [19:26] scheduling, dispatch all to all, expert [19:28] feed forward networks, combine all [19:30] toall, and the final weighted sum. It [19:34] runs inside one GB300 NVL72 rack at [19:39] expert parallel degree 64 with 48 tokens [19:44] per GPU before routing. Four shapes are [19:47] tested. Kimmy K27, [19:51] GLM 5.2, 2 Quen 3.5 [19:57] 397BA [20:00] BA17B [20:02] and Deep Seek V4 Pro. Each is compared [20:06] with NCCL plus PyTorch, Deep EP plus [20:09] PyTorch, Deep EP plus Transformer [20:12] Engine, and Hybrid EP plus Megatron. The [20:16] fastest baseline is selected separately [20:20] for every row. All four maxima occur on [20:23] the quen shape. MXFP8 forward is 531.1 [20:29] teraflops per second at 1.068 [20:32] milliseconds versus hybrid EP plus [20:35] Megatron at 224.4 [20:38] 2.37 times. MXFP8 [20:42] backward is 1.78 times that same [20:45] baseline family. BF-16 forward is 1.92 [20:49] times DBP plus PyTorch and BF-16 [20:54] backward is 1.58 times it. Across other [20:57] shapes the ratios are smaller and [20:59] deepseek BF16 backward slightly loses. [21:03] The chart supports strong specialization [21:05] not universal dominance. Cursor also [21:08] reports an internal end-to-end [21:11] comparison on 512GB [21:14] 300GPUs across several NVL72 racks. [21:19] Replacing its previous deepbased custom [21:22] MXFP8 [21:24] MOI path with MO raises tokens per [21:27] second per GPU from 760.9 [21:31] to 170.2 [21:33] a 1.41 times result. [21:37] This is more operationally meaningful [21:40] than a layer microbenchmark [21:44] but the production stack model run logs [21:48] and sweep are private. So the public [21:50] repository cannot independently [21:52] reproduce that number. The released [21:55] package targets Nvidia Blackwell SM100 [21:59] or SM1003 [22:01] GPUs such as GB200 or GB300 NVL72. [22:07] It requires Python 3.12 or later, [22:11] PyTorch 2.10 or later built for the [22:13] matching CUDA version and CUDA 13 or [22:16] later. The functional API also depends [22:19] on PyTorch symmetric memory across the [22:21] expert parallel group. Those constraints [22:24] are part of the result. This design [22:27] exploits one fast 72 GPU NVLink domain. [22:33] Portability beyond that platform is an [22:36] engineering goal, not evidence already [22:38] supplied. Um so the mega kernel [22:42] achievement is not fusion makes [22:44] everything faster. It is more precise. [22:47] Own the route schedule, network [22:49] transfers, expert math, and token [22:52] buffers together. Then tune how scarce [22:55] SMS and token chunks divide the waiting [22:58] time. The deterministic schedule makes [23:01] experiments easier to compare. The [23:04] benchmark proves large gains for [23:06] disclosed NVL 72 workloads, while its [23:09] losing row and private production stack [23:11] tell us exactly where confidence should [23:13] stop. The useful mental model is a [23:16] resident factory, not an oversized [23:19] operator. One group of streaming [23:22] multiprocessors moves tokens. Another [23:25] group runs the expert MLP. Local [23:28] counters hand each mini batch from one [23:30] crew to the other while the program [23:32] remains inside a single forward or [23:34] backward kernel. Mo removes three [23:38] different bubbles with three different [23:40] moves. Pull, dispatch, and push combine. [23:43] Simplify scheduling and avoid crossGPU [23:46] completion signals. A tunable mini batch [23:50] finds the middle ground between starving [23:52] tensor cores and waiting too long for [23:55] data. A macrobatch ring reuses fixed [23:59] symmetric memory so dynamic expert loads [24:02] do not force token dropping or CPUs [24:05] sized allocations. [24:08] On the reported GB300 NVL72 [24:12] layer benchmarks, MK reaches up to 2.37 [24:16] times the fastest public baseline for [24:18] MXFP8 forward with smaller but still [24:23] reported gains for the other precision [24:26] and direction cases. In Cursor's [24:29] internal 512GPU [24:31] training comparison, throughput rose [24:34] from 760.9 [24:36] to 1,070.2 [24:39] tokens per second per GPU, a 1.41 [24:44] times result. Those numbers are evidence [24:48] for this workload and platform, not a [24:52] universal GPU law. The release code [24:55] targets Blackwell SM 100 or SM 103 needs [25:00] tuning and depends on a fast NVLink [25:03] domain where communication can overlap [25:05] useful compute. The transferable idea is [25:08] narrower and stronger when a pipeline is [25:11] dominated by boundaries. Move theuler to [25:14] the device make resource ownership [25:17] explicit and reuse memory as a flowing [25:20] ring.