Arcee Halts 400B Blackwell Model Collapse With Dynamic Router Biases

How Not to Blow Up: Training a 400B MoE to 17T Tokens Without Loss Spikes - Presentation by Lucas — Ai4

Ai4

Extreme mixture-of-experts sparsity triggers severe expert starvation that eludes standard loss metrics, requiring momentum-driven router bias updates rather than static penalties to prevent catastrophic late-stage training collapse.

AI-assisted summary of How Not to Blow Up: Training a 400B MoE to 17T Tokens Without Loss Spikes - Presentation by Lucas — original source. Check the original for full context.

Source published: 2026-08-19

Detailed notes

## Background Published by the Ai4 channel on YouTube on August 19, 2026, this presentation features Lucas Atkins, CTO of Arcee AI. The video description outlines an 'anti-loss-spike' playbook detailing how Arcee trained an ultra-sparse 400-billion-parameter Mixture-of-Experts model across 17 trillion tokens without loss spikes by addressing router drift, MaxVio divergence, and numerical stability.

## TL;DR Lucas Atkins details the high-stakes operational reality of training Trinity Large—a 400B MoE with only 13B active parameters—on 2,048 Nvidia Blackwell B300 GPUs under a rigid 33-day cluster reservation. After discovering a severe early load-balancing divergence via the MaxVio metric that standard loss curves concealed, the team mitigated the crisis through private contingency planning, strict technical governance, and the creation of SMEBU (Scaled Momentum Expert-Bias Updates). Paired with precision fallbacks to BF16, z-loss regularization, and intra-document masking, these interventions stabilized the run and established foundational architectures for upcoming trillion-parameter models like Genesis Science 1.

## Atmosphere & style The presentation carries the gripping, authentic energy of a frontier engineering war story, blending transparent post-mortem analysis with genuine startup grit. Atkins delivers his retrospective in a calm, humble, yet compelling style, vividly conveying the immense psychological weight of managing a high-stakes $20 million compute bet against immovable cluster deadlines. It feels refreshingly candid and pragmatic, taking viewers inside the quiet panic of early telemetry anomalies and the triumph of disciplined technical triage.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=4s) Arcee AI executed a high-stakes training run, scaling from a 4B dense baseline to a 400B MoE with a 12-person team under a hard $20 million budget cap across six months. - [02:08](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=128s) Trinity Large achieved extreme architectural sparsity with 400B total parameters but only 13B active per token across 256 experts, trained within a strict 33-day rental window. - [02:08](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=128s) Executing the workload on 2,048 Nvidia Blackwell B300 GPUs required expert parallelism (EP) and context parallelism (CP) under production conditions, including live firmware updates. - [04:54](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=294s) MaxVio (Maximum Violation) measures routing load imbalance between the most and least utilized experts, where ratios above 6:1 threaten to produce dead parameters and collapse model capacity. - [04:54](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=294s) MaxVio acts as a critical early warning signal that diverges sharply hundreds of billions of tokens before loss curves or gradient norms show any sign of instability. - [08:15](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=495s) Under high-stakes pressure, leadership contained team panic by establishing a confidential fallback plan (pivoting to a 200B model at Day 12) and absorbing all operational blame. - [08:15](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=495s) Side-cluster ablations on a 200B model isolated the search space, verifying that extreme routing sparsity was the sole driver of router divergence rather than network depth or tokenization. - [13:45](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=825s) Cluster triage and governance prevented chaotic interventions by partitioning the 2,048 GPUs into controlled tracks and requiring approval from a four-lead committee for any config change. - [13:45](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=825s) The core technical breakthrough was SMEBU (Scaled/Momentum Expert-Bias Updates), which applies dynamic, momentum-based bias updates to correct lagging experts relative to others across layers. - [13:45](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=825s) Load balancing in massive MoEs is a delicate trade-off: routers must balance sufficiently to prevent numerical collapse without over-balancing and erasing expert specialization. - [17:30](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1050s) Loss stability required a suite of companion fixes: falling back from MXFP8 to BF16 in attention layers, adding z-loss for logit regularization, sequence auxiliary loss, and intra-document masking. - [19:15](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1155s) MaxVio telemetry provides real-time detection of out-of-distribution (OOD) data during continued pre-training and SFT, spiking above 6 when encountering novel data distributions. - [19:43](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1183s) Arcee's upcoming architecture targets over 1 trillion parameters with greater sparsity, underpinning the Department of Energy-partnered Genesis Science 1 model for automated research workflows.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=4s) The $20 Million YOLO Bet: Scaling to a 400B MoE

Lucas Atkins introduces himself as the CTO of Arcee AI, an American company developing open-weight language models. While the company publishes technical reports detailing architectures, benchmarks, and benchmark scores, Atkins frames this talk around the operational reality of running one of their highest-stakes "YOLO training runs." Rather than an abstract paper review, the focus is on how a lean startup team of approximately 12 researchers navigated immense compute constraints to execute a massive leap in model scale.

In mid-2025, the team made an aggressive strategic bet: scale from a 4-billion-parameter dense model all the way to a highly competitive 400-billion-parameter Mixture-of-Experts (MoE) architecture by early 2026. This had to be accomplished under strict capital boundaries, with a hard ceiling of $20 million budgeted across four models over a six-month span. In the frontier training landscape where frontier labs deploy hundreds of millions of dollars, this budget left virtually zero margin for wasted compute or prolonged cluster experimentation.

- **Team Size:** ~12 researchers executing the full training stack. - **Budgetary Constraints:** $20 million maximum across four distinct models over six months. - **Scaling Target:** Scaling from a 4B dense baseline up to a 400B MoE.

---

[02:08](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=128s) The Setup: 2,048 Blackwells and Extreme Sparsity

To de-risk the flagship run, Arcee released a 26-billion-parameter model and a 6-billion-parameter MoE in December to prove they could manage the intricate engineering and routing dynamics of MoEs over trillions of tokens. The day those proof-of-concept models shipped, the team transitioned immediately to their flagship: Trinity Large. They secured a cluster of 2,048 Nvidia Blackwell B300 GPUs for a non-negotiable rental window of exactly 33 days. The mandate was to push 20 trillion tokens through a 400B architecture within that single, fixed-duration hardware reservation.

Because they had neither the funding nor the time for extensive pre-run cluster burns, the team entered production on day one with numerous untested variables. They embraced extreme sparsity: Trinity Large scaled to 400B total parameters with only 13B active parameters per token—making it the sparsest open-weight MoE ever released. By comparison, standard sparsity ratios (such as DeepSeek's) would have activated 26B to 27B parameters. Beyond the 15-to-16-fold parameter jump from their 26B predecessor, this run marked the first large public workload on Nvidia Blackwell silicon. The infrastructure required moving beyond simple data parallelism into sophisticated expert parallelism (EP) and context parallelism (CP), compounded by cluster-wide pauses when Nvidia had to update GPU firmware mid-flight.

```mermaid flowchart LR accTitle: Trinity Large Architecture and Compute Setup accDescription: Flow showing the compute cluster feeding into the Trinity Large sparse MoE architecture.

subgraph Compute["Infrastructure Constraints"] A["2,048x NVIDIA B300 (Blackwell)"] B["33-Day Fixed Window"] A --> B end

subgraph Architecture["Trinity Large MoE Specs"] C["400B Total Parameters"] D["13B Active Parameters / Token"] E["256 Total Experts"] C --> D C --> E end

subgraph Parallelism["Parallelism Strategy"] F["Expert Parallelism (EP)"] G["Context Parallelism (CP)"] end

B --> Architecture Architecture --> Parallelism ```

- **Hardware Allocation:** 2,048 Nvidia B300 GPUs reserved for exactly 33 days. - **Token Target:** 20 trillion tokens at a 400B parameter scale. - **Extreme Sparsity:** 400B total parameters with only 13B active per token. - **Parallelism Strategy:** Migration from single-node data parallelism to multi-node Expert Parallelism and Context Parallelism.

---

[04:54](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=294s) The MaxVio Metric and the Day-4 Crisis

When operating a sparse MoE with 256 experts, tracking load balancing across the router is critical to prevent expert collapse. Atkins defines `MaxVio` (Maximum Violation)—a core metric measuring the imbalance between the most-routed-to expert and the least-routed-to expert. If routing collapses, unrouted experts become "dead parameters," effectively reducing model capacity by half or more. Under ideal conditions, a MaxVio ratio below 3:1 is excellent, while a ratio around 6:1 is passable towards the tail end of training; anything higher indicates severe routing pathology.

```mermaid flowchart TD accTitle: MaxVio Routing Pathology Escalation accDescription: Flowchart illustrating how early MaxVio divergence leads to unrecoverable loss spikes if untreated.

M["Router Distributes Tokens to 256 Experts"] M --> V["MaxVio Divergence (Ratio > 6:1)"] V --> D["Dead Experts Emerge (>50% Unused)"] D --> S["Loss & Grad Norm Look Deceptively Flat Early On"] S --> F["Catastrophic Loss Spikes at 15T Tokens"] ```

A few hundred billion tokens into the run (around day four), the team observed a concerning divergence: `MaxVio` climbed sharply, even though the standard telemetry—loss curves and gradient norms—remained completely smooth and healthy. Experience dictated that allowing this imbalance to compound unaddressed would lead to catastrophic loss spikes and gradient explosions later in the run (e.g., at the 15-trillion token mark). Because cluster time could not be extended, an unresolved routing failure threatened to waste the entire compute budget. Initial emergency interventions failed entirely: adjusting auxiliary-free load-balancing coefficients (derived from DeepSeek techniques) and inserting dense layers offered minor stabilization but failed to arrest the divergence.

- **MaxVio Definition:** Ratio of token volume assigned to the highest-utilized expert versus the lowest-utilized expert. - **Target Thresholds:** $\le 3.0$ is ideal; $\le 6.0$ is acceptable at late stages; $>6.0$ indicates dangerous imbalance. - **Early Failure Signal:** MaxVio exploded while loss and gradient norm remained deceptively stable. - **Initial Fixes Attempted:** Auxiliary-free loss coefficient tuning and dense layer additions failed to arrest the imbalance.

---

[08:15](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=495s) Leadership Under Fire: Bounding the Problem and Protecting the Team

With the entire company's capital and market reputation on the line, anxiety among the engineering team surged. Atkins stepped in to manage both the psychological pressure and the technical search space. To contain organizational panic, he established a private, unannounced "drop-dead day" at Day 12: if the 400B sparsity bug was not fixed within five to six days, he would unilaterally pivot the cluster to train a safer, 200B fallback architecture using previously proven sparsity settings for the remaining 18 days of runtime. Keeping this contingency confidential prevented the team from losing focus on solving the 400B challenge.

To eliminate analysis paralysis, Atkins reduced the diagnostic search space by isolating variables. The team spun up a 200B run on a small side-cluster using their old sparsity ratios; the run exhibited perfectly stable MaxVio metrics, immediately proving that the failure was driven strictly by the extreme sparsity rather than tokenizers, network width, or layer depth. Concurrently, Atkins absorbed all operational blame: he explicitly told the team that success belonged to them, but failure and any resulting fallout would rest solely on his shoulders. This psychological safety stopped engineers from making erratic, fear-driven config modifications and redirected all energy toward structured ablations.

- **Private Fallback Plan:** A secret trigger at Day 12 to drop to a 200B model taking 18 days of compute. - **Search Space Reduction:** Side-cluster ablations on 200B verified that extreme sparsity was the sole cause of router divergence. - **Leadership Decoupling:** Complete removal of career risk from researchers to curb impulsive, panic-driven interventions.

---

[13:45](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=825s) Cluster Triage, Governance, and SMEBU: The Core Breakthroughs

To prevent researchers from stepping on each other's toes or killing runs overnight, Atkins established a strict 24/7 technical governance committee: every config change required explicit approval from at least one of four core leads (Atkins, the principal researcher, the lead infrastructure engineer, or a team lead). The physical 2,048-GPU cluster was split to run controlled, parallel ablations: - A conservative track on a 1,024-GPU partition for large-scale validations. - Two 512-GPU partitions dedicated to rapid small-scale experimentation.

```mermaid flowchart TD accTitle: Cluster Triage and SMEBU Mechanism accDescription: Diagram showing cluster partitioning for ablations and how SMEBU applies dynamic momentum bias updates.

subgraph ClusterPartition["2,048 B300 Hardware Partitioning"] P1["1,024 Nodes: Large-Scale Validation"] P2["512 Nodes: Small Ablation Track A"] P3["512 Nodes: Small Ablation Track B"] end

subgraph SMEBUMechanism["SMEBU Router Balancing"] R["Router Routing Drift Detected"] B["Calculate Relative Distance Across Layers"] M["Apply Scaled Momentum Expert-Bias Update"] R --> B --> M end ```

The breakthrough came with the invention of **SMEBU** (Scaled/Momentum Expert-Bias Updates). Traditional load balancing applies a static, fixed coefficient to penalize over-routed experts and boost under-routed ones. In an architecture with 256 compact experts, standard static biases were too sluggish to prevent routing drift. SMEBU introduced a momentum-based mechanism: when an expert falls behind in token allocation relative to others across different layers, its routing bias updates dynamically proportional to how far out of sync it is. This instantly stabilized routing across the network. Atkins emphasizes that load balancing is a "razor's edge": the model must balance enough to remain numerically stable without over-balancing to the point of wiping out token-level expert specialization.

- **Cluster Allocation:** Divided into $1 \times 1,024$ nodes for primary tests and $2 \times 512$ nodes for fast parallel ablations. - **Change Governance:** Mandatory sign-off from one of four designated technical leads before any cluster alteration. - **SMEBU Innovation:** Momentum-driven router bias updates calculated dynamically relative to expert imbalance across layers. - **Specialization Trade-off:** Maintaining sufficient balance for stability without suppressing token-level specialist routing.

---

[17:30](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1050s) Precision Fallbacks, Loss Stabilization, and MaxVio as Data Telemetry

Alongside SMEBU, the team executed a suite of stabilizing architectural fixes. They executed a numerical precision fallback, pulling attention layers out of MXFP8 and back into robust BF16. They introduced **z-loss** to penalize large router logits and keep activations bounded, incorporated a **sequence auxiliary loss**, and later implemented **intra-document masking** during sequence packing to prevent cross-contamination across concatenated documents. Together, these interventions brought MaxVio back under control, where it remained flat throughout the entire multi-trillion token run.

| Stabilization Technique | Implementation Mechanism | Problem Solved | |:--- |:--- |:--- | | **SMEBU** | Momentum-based dynamic router bias updates | Router drift and expert starvation at extreme sparsity | | **Precision Fallback** | Attention computations moved from MXFP8 to BF16 | Micro-scaling floating-point numerical instability | | **z-loss** | Logit regularization ($\mathcal{L}_z = \log^2 \sum e^{x_i}$) | Runaway router logit growth and softmax saturation | | **Sequence Auxiliary Loss** | Auxiliary load balancing evaluated per sequence | Sequence-level expert load skew | | **Intra-Doc Masking** | Attention masking across packed document boundaries | Cross-document token interference and attention pollution |

Atkins highlights that MaxVio also serves as an ultra-sensitive real-time diagnostic for data distribution shifts and data diversity. Because MoE routers develop specialized pathways for specific document formats and domains, sudden shifts in the training distribution trigger immediate spikes in MaxVio. Monitoring this metric provides an early-warning signal that reveals whether an MoE has encountered a specific data distribution before—even when fine-tuning or auditing third-party models without access to their original pre-training corpora.

[19:15](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1155s) Monitoring MaxVio to Detect OOD Data in Post-Training

Beyond pre-training stability, router behavior provides a practical diagnostic signal during downstream fine-tuning and adaptation. Lucas explains that teams should set up tracking for MaxVio (maximum violation) inside monitoring platforms like Weights & Biases. When introducing fresh data—whether during supervised fine-tuning (SFT) or continued pre-training—the metric acts as an early warning system for distribution shift.

If the router’s MaxVio spikes to extreme readings such as 12, 15, or 17, or generally exceeds a baseline threshold of around 6, it offers clear evidence that the model has never encountered that domain or style of data before. Recognizing these severe routing imbalances allows engineers to proactively step in and smooth over the incoming distribution before it destabilizes the run.

- **MaxVio Anomaly Baseline:** Any router violation score exceeding 6 (climbing to 12, 15, or 17) indicates strongly out-of-distribution (OOD) data. - **Monitoring Integration:** Standard experiment trackers like Weights & Biases can be used to log and catch MaxVio surges during SFT and continued pre-training phases.

```mermaid flowchart LR accTitle: MaxVio Diagnostic and Distribution Shift Detection subgraph InputData["Data Ingestion"] D1["New Data Batch (SFT / Continued Pre-Train)"] end

subgraph Monitoring["Router Diagnostics"] MV["Measure MaxVio Metric (Weights & Biases)"] Check{"MaxVio > 6? (e.g., 12, 15, 17)"} end

subgraph Actions["Intervention"] OOD["Identify Out-of-Distribution Data"] Smooth["Smooth & Balance Data Distribution"] Stable["Safe Convergence"] end

D1 --> MV MV --> Check Check -- "Yes" --> OOD OOD --> Smooth Check -- "No" --> Stable ```

[19:43](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1183s) Scaling to Trillion+ Parameters: Genesis Science 1 and the Next-Gen Trinity

Lucas reflects on the trajectory of Arcee's Trinity model family—progressing from Trinity Nano and Trinity Mini up to the 400B-parameter Trinity Large. Over the past four to six months, the team has turned its focus toward their next-generation architecture. The flagship deployment of this upcoming generation is Genesis Science 1, developed through a direct partnership with the U.S. Department of Energy (DOE). The initiative is designed to automate complex, end-to-end scientific research workflows across DOE facilities, ranging from high-performance computing (HPC) utilization to automated experimental design and rapid scientific iteration.

Slated for release later in the year, the next Trinity architecture aggressively scales both capacity and efficiency: Arcee is pushing the model footprint past the one-trillion-parameter boundary while making the expert routing substantially sparser. Lucas jokes that he hopes he will not have to return the following year to give another post-mortem lecture on loss spikes, quipping, "I hope it's a celebratory talk." He closes the presentation on an emotional note about the ethos driving the engineering team, stating that the work he feels most proud of is "choosing extremely difficult things to do that we have no business trying to accomplish, working with very motivated people, and then having to create our way out of it."

```mermaid flowchart TD accTitle: Trinity Model Lineage to Genesis Science 1 subgraph Predecessors["Trinity Model Lineage"] T_Small["Trinity Nano & Trinity Mini"] T_Large["Trinity Large (400B MoE, 17T Tokens)"] T_Small --> T_Large end

subgraph Frontier["Next-Generation Scaling"] T_Next["Next-Gen Trinity Base (>1 Trillion Parameters, Ultra-Sparse)"] GS1["Genesis Science 1 (DOE Partnership)"] HPC_App["Automated Scientific Workflows (HPC Orchestration & Experiment Design)"]

T_Large --> T_Next T_Next --> GS1 GS1 --> HPC_App end ```

- **Next-Generation Scale:** The upcoming architecture scales past 1,000,000,000,000 (1 trillion) parameters with significantly higher sparsity than Trinity Large. - **DOE Partnership:** Genesis Science 1 targets Department of Energy HPC systems to automate experimental design, code execution, and scientific workflow iteration. - **Release Timeline:** Built on top of the next-generation Trinity series and scheduled to debut later in the year.

## Notable quotes > LLM progress now depends heavily on one practical issue: training stability at scale. — Lucas Atkins

> Sparse Mixture-of-Experts (MoE) models are especially sensitive, since routing drift can overload experts, collapse utilization, and stall learning. — Lucas Atkins

> You will leave with a concrete checklist for stability instrumentation and first-response fixes to keep large open-weight runs on track. — Lucas Atkins

> [00:04](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=4s) YOLO training runs — Lucas Atkins

> [04:54](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=294s) dead parameters — Lucas Atkins

> [08:15](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=495s) drop-dead day — Lucas Atkins

> [13:45](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=825s) razor's edge — Lucas Atkins

> [19:43](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1183s) I hope it's a celebratory talk. — Lucas Atkins

> [19:43](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1183s) choosing extremely difficult things to do that we have no business trying to accomplish, working with very motivated people, and then having to create our way out of it. — Lucas Atkins

## Mentioned Lucas Atkins, Arcee AI, Ai4, Nvidia, Nvidia Blackwell B300, DeepSeek, U.S. Department of Energy (DOE), Weights & Biases, Trinity Large, Trinity Mini, Trinity Nano, Genesis Science 1, SMEBU, MXFP8, BF16

## Video description Lucas Atkins - CTO, Arcee

LLM progress now depends heavily on one practical issue: training stability at scale. Sparse Mixture-of-Experts (MoE) models are especially sensitive, since routing drift can overload experts, collapse utilization, and stall learning.

In this talk, I will share an "anti-loss-spike" playbook from a recent open-weight run: a 400B-parameter MoE with 13B active parameters per token, trained for 17T tokens with an unsmoothed loss curve and zero loss spikes. I will start with the failure pattern we saw, router drift, overload, MaxVio divergence, and plateau, then cover the fixes that restored steady convergence: bounded and momentum expert-bias updates (SMEBU), z-loss for logit stabilization, a precision fallback from MXFP8 to BF16, better balancing objectives, and data/packing choices that reduced step-to-step variance.

You will leave with a concrete checklist for stability instrumentation and first-response fixes to keep large open-weight runs on track.

See less

[Source](https://www.youtube.com/watch?v=V-9TscGJj-Y)

中文简报

Arcee 采用动态路由偏置阻止 400B Blackwell 模型崩溃极致的专家混合稀疏性引发了标准损失指标无法察觉的严重专家饥饿现象，需要依靠动量驱动的路由偏置更新而非静态惩罚，以防止训练后期发生灾难性崩溃。 Lucas Atkins 详细阐述了在严格受限的 33 天集群租用期内，利用 2,048 块 Nvidia Blackwell B300 GPU 训练 Trinity Large（一个总参数达 400B、激活参数仅 13B 的 MoE 模型）的高风险实战过程。在通过 MaxVio 指标发现了标准损失曲线所掩盖的严重早期负载均衡偏差后，团队通过内部应急预案、严格的技术治理以及研发 SMEBU（缩放动量专家偏置更新）化解了这场危机。结合回退至 BF16 精度、z-loss 正则化以及文档内掩码等措施，这些干预手段稳定了训练过程，并为 Genesis Science 1 等即将推出的万亿参数模型奠定了基础架构。 Arcee AI 开展了一项高风险训练任务，其 12 人的团队在 6 个月内、在 2000 万美元的严格预算上限下，从 4B 稠密基线扩展至 400B MoE。 (0:04) Trinity Large 实现了极高的架构稀疏性，总参数达 400B，涵盖 256 个专家，但每个 token 仅激活 13B，并在严格的 33 天租用窗口内完成了训练。 (2:08) 在 2,048 块 Nvidia Blackwell B300 GPU 上运行该工作负载需要在生产环境下采用专家并行（EP）和上下文并行（CP），其中包括进行在线固件更新。 (2:08) MaxVio（最大违规度）用于衡量使用率最高与最低专家之间的路由负载不平衡，当比例超过 6:1 时，便存在产生死参数并导致模型容量崩溃的风险。 (4:54) MaxVio 扮演了关键的预警信号角色，在损失曲线或梯度范数显现出任何不稳定迹象的数千亿个 token 之前，它便会剧烈偏离。 (4:54)

Reading summary

Arcee AI CTO Lucas Atkins details how a 12-person research team navigated tight compute and capital constraints to train Trinity Large, a 400-billion-parameter sparse Mixture-of-Experts (MoE) model, without experiencing catastrophic loss spikes. Operating within an inflexible 33-day cluster window, the team stabilized the run by developing dynamic momentum-based routing updates and pairing them with structured experimental governance and precision fallbacks. Extreme Sparsity Under Fixed Hardware Deadlines (0:04) In mid-2025, Arcee AI committed to scaling from a 4-billion-parameter dense model to a 400-billion-parameter MoE under a hard $20 million budget cap spread across four models over six months. To execute the flagship Trinity Large run, the team secured 2,048 Nvidia Blackwell B300 GPUs for a non-negotiable rental window of 33 days, targeting a throughput of 20 trillion tokens. Atkins noted the run marked the first large public workload on Blackwell silicon, which required adapting to cluster-wide pauses for mid-flight GPU firmware updates. Without capital or time for extensive pre-run test burns, the team adopted extreme sparsity: 400 billion total parameters with only 13 billion active per token across 256 experts. This configuration made Trinity Large roughly twice as sparse as architectures like DeepSeek, which activate 26 to 27 billion parameters. Managing this scale forced the 12-person team to implement multi-node Expert Parallelism and Context Parallelism under zero margin for wasted compute. Detecting Routing Collapse Through MaxVio Telemetry (4:54) To monitor load balancing across 256 experts, the team tracked MaxVio (Maximum Violation), defined as the ratio of tokens assigned to the most-routed-to expert versus the least-routed-to expert. Under healthy conditions, a MaxVio ratio remains below 3:1, while ratios up to 6:1 are tolerable late in training. Atkins explained that higher ratios indicate severe expert collapse, where unrouted experts turn into dead parameters that effectively halve model capacity. On day four, only a few hundred billion tokens into the run, MaxVio climbed sharply even though standard telemetry—including loss curves and gradient norms—remained completely flat. Drawing on past runs, Atkins warned that ignoring this divergence would trigger catastrophic loss spikes around the 15-trillion-token mark. Initial emergency measures, such as tuning auxiliary-free loss coefficients borrowed from DeepSeek and inserting dense layers, failed to arrest the imbalance. Containment Governance and Experimental Search Control (8:15) To counter mounting team anxiety and avoid erratic decision-making, Atkins established an undisclosed drop-dead milestone at Day 12: if the 400B routing bug could not be resolved within several days, he planned to unilaterally pivot the remaining 18 days of compute to a safer 200B fallback model. Atkins also absorbed all public blame to remove career risk from the researchers, curbing panic-driven configuration tweaks. To isolate the root cause, the team tested a 200B model on a separate side cluster using proven sparsity settings; the test exhibited stable MaxVio metrics, confirming that extreme sparsity alone was causing the router drift. The physical 2,048-GPU cluster was then partitioned into a 1,024-GPU slice for large-scale validation and two 512-GPU slices for fast parallel ablations. All configuration changes were restricted to mandatory approval from a four-person technical governance committee. Stabilizing MoE Routing with Dynamic SMEBU (13:45) The primary operational breakthrough was SMEBU (Scaled/Momentum Expert-Bias Updates). Standard load balancing relies on static coefficients that reacted too slowly for 256 compact experts. In contrast, SMEBU introduced a momentum-based mechanism that dynamically adjusts an expert's routing bias proportional to its cumulative routing drift across layers. Atkins described load balancing as a delicate trade-off: enforcement must maintain numerical stability without blunting specialized token routing. Alongside SMEBU, the team deployed several targeted architectural adjustments. They executed a precision fallback, moving attention layers from MXFP8 back into BF16, applied z-loss to regularize router logits and avoid softmax saturation, introduced sequence auxiliary losses, and implemented intra-document masking during sequence packing to prevent cross-document attention pollution. Together, these interventions stabilized MaxVio across trillions of tokens. Diagnostic Extensions and Future DOE Implementations (19:15) Beyond pre-training stability, Atkins emphasized that MaxVio serves as an early-warning diagnostic during downstream fine-tuning and continued pre-training. Tracking MaxVio in experiment monitors reveals distribution shifts: readings exceeding baseline thresholds of 6, or surging to 12, 15, or 17, provide immediate evidence of out-of-distribution data, allowing practitioners to smooth incoming mixtures before runs destabilize. Building on Trinity Large, Arcee is directing development toward a next-generation architecture scaling past one trillion parameters with even higher sparsity ratios. The flagship deployment is Genesis Science 1, created in partnership with the U.S. Department of Energy to automate scientific research workflows, experimental design, and high-performance computing execution across DOE facilities. Atkins demonstrates that aggressive MoE sparsity can achieve frontier model scale within startup compute budgets, provided teams monitor router-level metrics rather than relying solely on flat early loss curves. However, whether dynamic momentum balancing like SMEBU remains stable when scaling beyond the one-trillion-parameter threshold remains an untested question for Arcee's upcoming DOE deployments.

精炼阅读摘要

Arcee AI 首席技术官 Lucas Atkins 阐述了团队如何在严苛的算力与资金约束下，通过首创的 SMEBU 动量路由偏置算法及工程治理体系，成功在 2,048 张 Blackwell GPU 上训练 400B 规模的超稀疏 MoE 模型，并解决了早期路由失衡引发的潜在训练崩溃。高风险算力豪赌与极端稀疏架构 (0:04) 2025 年中，Arcee AI 确立了一项激进的训练规划：在 6 个月内利用总计不超过 2000 万美元的预算训练 4 款模型，并将模型架构由 4B 稠密模型直接跃迁至 400B 参数的混合专家（MoE）模型 Trinity Large。整个核心执行团队仅约 12 名研究员。为了在 33 天不可延长的租约窗口内完成 20 万亿 token 的训练任务，团队租用了由 2,048 张 Nvidia Blackwell B300 组成的集群，并首次在 Blackwell 芯片上承载大规模公开训练负载。 Day-4 路由危机与 MaxVio 失衡指标 (4:54) Trinity Large 拥有 400B 总参数，但每个 token 仅激活 13B 参数（包含 256 个专家），其稀疏度远超当时常规开源模型。在训练推进到数百 billion token（约第 4 天）时，团队监测到核心路由均衡指标 MaxVio（最大违规比，即分配 token 最多与最少的专家负载比值）急剧攀升。理想状态下该比值应低于 3:1，超过 6:1 则意味着严重的路由病态；未分配到 token 的专家会沦为死参数，导致模型有效容量折损过半。尽管早期的损失曲线和梯度范数表面上维持平稳，但经验表明，该隐性失衡若累积到 15 万亿 token 阶段必然诱发灾难性 loss 尖刺。起初，团队尝试微调源自 DeepSeek 的无辅助损失负载均衡系数以及插入稠密层，但均未能遏制路由发散趋势。危机隔离与 24/7 工程治理 (8:15) 面对高昂算力可能全部报废的压力，Atkins 设定了保密的第 12 天止损红线：若无法解决 400B 的稀疏路由问题，将单方面在剩余 18 天转向训练经过验证的 200B 备用模型。同时，团队在辅助集群上测试 200B 架构，确认其 MaxVio 表现完全正常，从而将排障范围精准锁定在极端稀疏度设置本身，排除了分词器、网络宽度与层深度等无关变量。在组织管理上，Atkins 明确由自己独自承担技术失败责任，以此消除工程师因恐慌而胡乱修改配置的心理压力。集群硬件被拆分为一个 1,024 节点的大规模验证分区和两个 512 节点的快速消融分区，并实行全天候变更审批制，所有配置调整必须经过包括 Atkins 在内的 4 位技术负责人之一签字许可。 SMEBU 突破与多维度数值稳定方案 (13:45) 针对传统静态偏置惩罚在 256 个微型专家架构中响应迟缓的问题，团队开发了 SMEBU（动态缩放动量专家偏置更新）。当某个专家在各层间的分配明显落后时，SMEBU 依据其滞后程度动态施加带动量的路由偏置更新，既维持了数值平衡，又防止了过度均衡抹杀 token 级别的专家特化能力。此外，团队实施了一系列工程与算法兜底措施：将注意力层计算从存在微缩浮点不稳定的 MXFP8 回退至稳健的 BF16；引入 z-loss 正则化 logits 以防止 softmax 饱和；加入序列辅助损失以平衡序列级负载；并在序列打包时应用文档内掩码，消除跨文档干扰。这一组合方案使得 MaxVio 指标彻底恢复并平稳贯穿了数万亿 token 的全流程。 MaxVio 分布监控与万亿级后续演进 (19:15) Atkins 指出，MaxVio 亦可作为监测数据分布偏移的高灵敏度遥测指标。在指令微调（SFT）或持续预训练过程中，若该指标异常激增至 12、15 乃至 17，即清晰表明模型遇到了此前从未学习过的域外（OOD）数据，工程师可据此在训练崩溃前提前平滑数据分布。目前，Arcee 团队正与美国能源部（DOE）合作研发 Genesis Science 1，基于下一代 Trinity 架构自动化处理实验设计与超算工作流。该系列计划突破 1 万亿参数量级，并进一步提高专家路由的稀疏程度，预计将于年内正式发布。 Lucas Atkins 总结强调，极端受限资源下的前沿大模型工程，其核心在于平衡严谨的变量隔离机制与大胆的技术容错边界。尽管 SMEBU 与配套稳定措施化解了 400B 稀疏 MoE 的训练危机，但其下一代万亿参数架构将面临更极端的稀疏度挑战，路由稳定性与自动化科学应用仍有待实际发布验证。

Detailed notes — zh

## Background 本内容由 YouTube 频道 Ai4 于 2026 年 8 月 19 日发布，主讲人为 Arcee AI 首席技术官 Lucas Atkins。视频主要复盘了其团队在资源极端受限的情况下，如何在 2048 块 NVIDIA B300 集群上成功将 400B 参数（单 Token 激活 13B）的极致稀疏 MoE 模型训练至 17T Tokens，并实现全周期零 Loss Spike 的实战经验与稳定训练策略。 ## TL;DR Arcee 团队仅由 12 名核心研究员组成，在 33 天固定租期和 2000 万美元预算约束下孤注一掷挑战 400B 极致稀疏 MoE（Trinity Large）的训练。训练初期团队遭遇了 MaxVio（专家路由最大失衡度）急剧恶化的隐形危机，随时面临模型崩溃与项目报废的灭顶之灾。通过压缩搜索空间、制定止损底线、设立 4 人决策门控机制并屏蔽外部焦虑，团队自主研发了基于动量的专家偏置动态更新算法 SMEBU，并配合 BF16 精度回退、Z-loss、序列辅助损失与数据掩码等组合拳，成功平息危机完成 17T Tokens 的平稳训练。最后 Lucas 还分享了利用 MaxVio 探测数据分布漂移的技巧，并公布了与美国能源部合作的下一代万亿级超稀疏科学模型计划。 ## Atmosphere & style 整场演讲充满硬核实战色彩与跌宕起伏的危机叙事感，既有初创团队押上全公司身家豪赌的紧张窒息，又有管理者直面崩溃时冷静扛责的担当魄力。Lucas 的讲述真诚且富有感染力，将高压绝境下的心理攻防、算力资源调度与深层算法创新交织在一起，给观众带来如同亲临万卡集群救火前线的沉浸感与热血感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=4s) 极限资源下的小团队豪赌：Arcee AI 仅凭约 12 名核心研究员和 2000 万美元总预算，在半年内完成了从 4B 密集模型到 400B 混合专家模型（MoE）的跨越式训练。 - [02:08](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=128s) 极致稀疏架构与算力窗口：Trinity Large 拥有 400B 总参数但单 Token 仅激活 13B，采用专家并行与上下文并行，租用 2048 块 NVIDIA B300 GPU 且训练窗口严格锁定在 33 天。 - [04:54](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=294s) 隐形危机与 MaxVio 飙升：训练初期路由失衡指标 MaxVio 剧烈攀升，而此时 Loss 与梯度范数依然平稳正常，表明大量专家可能陷入无法学习的“死专家”状态并潜藏崩溃风险。 - [07:21](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=441s) 降维排查与止损底线：通过小规模 200B 对照实验将根因锁定在极致稀疏度上，CTO 秘密设定第 12 天止损降级底线，并主动承担全部责任以消除团队的恐慌情绪。 - [11:55](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=715s) 危机应对的精力聚焦原则：在面临巨大压力时，全面屏蔽投资人担忧、舆论评判及沉没成本等不可控因素，将全员精力 100% 投入到可掌控的消融实验与算法攻坚中。 - [13:45](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=825s) 算力分流与 24/7 门控机制：设立 4 人核心决策组严格审批主集群变更，并将 2048 卡集群划分为 1024 卡主线与两个 512 卡消融支线，杜绝无序的随意中断与算力浪费。 - [15:25](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=925s) 突破性算法 SMEBU：针对细粒度海量专家自研动量自适应偏置更新机制 SMEBU，在保证路由全局动态平衡的同时，避免过度平均化以保留专家的 Token 专精能力。 - [15:25](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=925s) 多维度稳定性组合拳：通过将 Attention 从 MXFP8 回退至 BF16、引入 Z-loss 抑制 Logit 极大值、加入序列级辅助损失及文档内掩码，最终实现 17T Tokens 训练全程零 Loss Spike。 - [18:25](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1105s) MaxVio 的衍生价值：MaxVio 可作为探测数据分布漂移的敏感指标，在微调和持续预训练中能够提前识别模型未见过的全新数据分布。 - [18:25](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1105s) 未来迈向万亿超稀疏模型：Arcee 正与美国能源部（DOE）合作开发科学计算大模型 Genesis Science 1，并规划总参数突破 1T 且稀疏度更激进的下一代 MoE 架构。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=4s) 资源受限下的小团队豪赌：从 4B 到 400B MoE Arcee AI 的首席技术官 Lucas Atkins 带来了关于大模型稳定训练的实战复盘。这场分享源于团队在 2026 年初进行的一次极具“孤注一掷”（YOLO）色彩的训练任务。Lucas 坦言，这并不是一次照本宣科的技术报告复述——尽管团队已经发布了详尽阐述架构决策的技术白皮书，但他更想分享的是：一个极小规模的团队如何在资源极端受限的情况下完成这一壮举。在 2025 年年中做出决策时，Arcee 团队仅有约 12 名研究人员，整个公司押注在半年内从 40 亿参数（4B）的 Dense 模型直接跃升至 4000 亿参数（400B）的混合专家模型（MoE），并要求模型在开源生态中保持极强的竞争力。整个项目的财力约束同样非常严苛。团队在六个月内训练四款模型的总预算上限仅为 2000 万美元。虽然这笔资金对普通团队是一笔巨款，但在当代前沿大语言模型预训练的资金消耗面前，这点预算显得极其微薄。为了验证自身掌控 MoE 固有复杂性及万亿 Token 级别扩展性的能力，团队在 2025 年 12 月先后推出了 26B 密集模型和 6B MoE 模型。这两次测试验证了技术路径的可行性，也为最终的 Trinity Large 旗舰模型训练奠定了基础。 - **团队规模**：仅约 12 名核心研究人员。 - **总预算限制**：6 个月内 4 个模型共计最多 2000 万美元。 - **技术跨越路径**：2025 年中期的 4B Dense $\rightarrow$ 2025 年底的 6B MoE 与 26B 模型 $\rightarrow$ 2026 年初的 400B MoE。 --- ### [02:08](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=128s) 极限算力窗口：Blackwell 集群与极致稀疏度设计在前两款验证模型发布的当天，团队便马不停蹄地开启了 Trinity Large 的训练。由于资金限制，他们仅租用了 2048 块 NVIDIA B300（Blackwell 架构）GPU，租期严格锁定在 33 天，没有任何超期余量。这意味着团队只有“一次开火的机会”（one shot），此前从未在如此庞大的集群规模上进行过预先验证。原本制定的训练目标是在 33 天内吞吐 20 万亿（20T）Token，这在 400B 参数级别上堪称疯狂，但得益于 Blackwell 芯片的算力优势，理论上吞吐速度能够达标。然而，Trinity Large 在架构上做出了大幅度的激进改动。团队拥抱了极致的稀疏性（Extreme Sparsity），将专家数量翻倍但保持每次激活的专家数不变：总参数量达 400B，但每个 Token 仅激活 13B 参数。相比之下，若遵循类似 DeepSeek 的稀疏比例，激活参数通常会在 26B 到 27B 左右，因此 Trinity Large 成为了当时公开发布的稀疏度最高的 MoE 模型。不仅如此，这是 Blackwell GPU 首次在公开领域进行的大规模集群训练，中途 NVIDIA 甚至不得不让集群停机一天以刷入新固件。为了支撑这一规模，模型切分方案也从过去的单卡数据并行彻底重构为复杂的专家并行（Expert Parallelism）与上下文并行（Context Parallelism）。 ```mermaid flowchart TD accTitle: Trinity Large 架构与训练约束 subgraph Compute["算力与时间约束"] A["2048 块 NVIDIA B300 GPU"] --> B["33 天固定租期"] B --> C["目标：吞吐 20T Tokens"] end subgraph Architecture["极致稀疏 MoE 架构"] D["总参数量 400B"] --> E["单 Token 仅激活 13B"] E --> F["专家总数翻倍 / 极高稀疏比"] end subgraph Parallelism["并行架构升级"] G["从单纯 Data Parallelism"] --> H["引入 Expert Parallelism"] H --> I["引入 Context Parallelism"] end ``` - **GPU 算力规模**：2048 块 NVIDIA B300 GPU。 - **训练时间窗口**：固定 33 天（无延期预算）。 - **参数规模**：400B 总参数，13B 活跃参数（Active Parameters）。 - **并行策略**：采用专家并行（EP）与上下文并行（CP）等多维混合并行方案。 --- ### [04:54](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=294s) 隐形危机：MaxVio 路由失衡指标的飙升训练刚开始的前几天一切看似风平浪静，但吞吐到数千亿 Token 时，一个关键指标出现了异常——MaxVio（最大失衡度/路由违规率）开始急剧攀升。MaxVio 是用于衡量 MoE 路由负载不均衡程度的核心指标，定义为分配到最多 Token 的专家所接收的 Token 数量，与分配到最少 Token 的专家所接收数量之间的比值。如果专家负载严重失衡，就会导致大量“死专家”（Dead Experts）——例如在 256 个专家中只有 100 个被路由使用，剩下 156 个专家将完全丧失学习能力，相当于模型超过一半的参数彻底报废。通常情况下，MaxVio 保持在 3 以内是绝佳状态，在训练后期达到 6 勉强合格，而一旦超过 6 就会引发严重故障。令人不安的是，在 MaxVio 持续恶化的同时，Loss（损失值）和 Grad Norm（梯度范数）依然表现得极其平稳并正常下降。这种表面平稳极具欺骗性，因为团队深知一旦放任路由漂移恶化，迟早会在数万亿 Token 后引发灾难性的 Loss 爆炸和梯度失控。对于一个只有 33 天预算且不可重来的项目而言，如果在 15T Token 时遭遇崩溃，公司将面临灭顶之灾。 - **MaxVio 定义**：最高负载专家接收的 Token 数量 $\div$ 最低负载专家接收的 Token 数量。 - **健康指标基准**：理想比率 $\le 3$；训练后期可接受上限 $\approx 6$；$>6$ 则预示严重崩溃风险。 - **异象表现**：训练数百亿 Token 后 MaxVio 急剧飙升，而 Loss 与 Grad Norm 依然极其平滑。 --- ### [07:21](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=441s) 危机应对：收敛搜索空间与“止损日”底线管理在训练进行到第 4 天左右，团队尝试了各种常规救援手段，包括反复调整源自 DeepSeek 的无辅助损失（Auxiliary-loss-free）负载均衡系数，甚至尝试增加密集层（Dense Layers），但都未能根治 MaxVio 的发散。由于押上了全公司的命运和银行账户上的全部积蓄，团队内部开始蔓延恐慌情绪。面对这种极端危机，Lucas 制定了关键的“降维排查”与心理防御策略。首先，Lucas 秘密设定了“最后止损日”（Drop Dead Day）。他私下计算出：如果放弃 400B 极致稀疏方案，退回到团队此前熟练掌握的 200B 稀疏度，训练完 20T Token 需要 18 天。因此，若在第 12 天仍无法解决该问题，就必须退回 200B 方案以保全项目交付。Lucas 将这个期限深藏于心，没有向团队透露，从而给团队争取到了 5 至 6 天专注攻关的窗口。随后，团队通过“压缩搜索空间”（Lowering the Search Space）的方法，在小规模节点上单独运行 200B 旧稀疏度实验，发现该问题并未复现，从而迅速排除了分词器、网络深度、专家宽度等干扰项，精准锁定根因就在于“极致稀疏度”本身。面对人心惶惶的局面，Lucas 明确向团队表态：如果问题最终成功解决，功劳属于团队每一个人；如果方案失败甚至被迫降级，所有责任由他一人承担，团队成员绝不会因此受到惩罚。这一举措有效阻断了因恐惧而做出非理性决策的恶性循环，稳住了团队的阵脚。 ```mermaid flowchart TD accTitle: 危机排查与决策流程 A["MaxVio 异常飙升 (第 4 天)"] --> B{"常规修复尝试"} B -->|"调负载均衡系数 / 增密集层"| C["未能解决"] C --> D["CTO 制定第 12 天止损底线 (秘密预案：退回 200B)"] D --> E["小集群对照实验：压缩搜索空间"] E --> F["验证 200B 旧稀疏度无异常"] F --> G["精准锁定病因：极致稀疏度引发路由漂移"] G --> H["CTO 揽下全责，阻断恐慌决策"] ``` - **应急排查实验**：在部分节点并行测试 200B 参数的旧稀疏度模型，迅速确认问题与分词器、模型层数无关，完全由超高稀疏度引发。 - **秘密止损节点**：在第 12 天前若未解决，将启动备用方案在剩余 18 天内跑完 200B 模型。 - **管理原则**：成功归功全员，失败由管理者承担全责，杜绝团队在恐慌状态下做变形决策。 ### [11:55](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=715s) 危机管理与心态调整：聚焦于能够掌控的事情在训练出现严重不稳定时，整个团队陷入了极大的焦虑之中。Lucas 提到，当时团队成员普遍感到压力巨大，“仿佛全世界的重量都压在他们肩上”。作为技术负责人，Lucas 主动对大家表示：“不，这些压力由我来扛，你们只需要专注于自己最擅长的事情——做顶尖的研究员和优秀的工程师。” 面对此类危机，人们往往容易分心去担忧那些无法掌控的外界因素： - 投资人对训练失败的负面反应与质问； - 已经烧掉巨额资金带来的沉没成本； - 市场上可能出现的唱衰舆论（“看吧，他们根本做不出来”）； - 整个行业对公司的评价，乃至公司是否能在这场危机中存活下来； - 团队内部滋生的恐慌与相互推诿指责。 Lucas 强调，在危机时刻必须坚定地执行一个原则：大脑只聚焦于“自己能够掌控的事情”。虽然这说起来容易做起来难，但通过反复训练和极度专注，团队可以将精力完全投入到实际行动中：进行大量消融实验（ablations）、调整模型配置、探索全新的负载均衡机制，并明确决策流与下一步测试计划。保持团队成员始终处于充实且目标明确的攻坚状态，避免因无所事事的“真空期”而陷入内耗与恐慌。正如 Lucas 所总结的：“只去解决你当下能改变的事。因为如果问题解决了，外面的担忧自然烟消云散；如果没解决，那也是之后才需要去面对的事。” ```mermaid flowchart TD accTitle: 危机处理与精力分配策略 subgraph S1["无法控制的外部干扰（忽略）"] A1["投资人质问与沉没成本"] A2["外界舆论与公司生存危机"] A3["恐慌情绪与相互指责"] end subgraph S2["可以控制的实际行动（聚焦）"] B1["设计新负载均衡算法"] B2["快速参数配置与消融测试"] B3["建立 24/7 核心决策机制"] end S1 -.->|屏蔽干扰| S2 ``` --- ### [13:45](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=825s) 算力分流与决策机制（Gatekeeper）为了应对混乱局面并高效利用算力，团队迅速确立了严格的决策流程与集群切分机制。当时团队拥有庞大的计算集群，很多成员在夜间看到训练指标稍显不佳时，会擅自停掉任务以测试自己的方案，这导致了严重的资源浪费与研发混乱。为了规范实验，团队制定了严密的决策门控机制并重组了算力资源： - **核心决策小组（4人）**：由 Lucas、核心研究员、基础设施工程师以及另一位团队主管组成。规定任何对主集群配置的修改或任务中断，必须获得至少一名核心成员的明确同意。为了应对高压状态，确保 24/7 随时有人值守并做出权威裁决，避免 12 位天才工程师各自发起无序的短平快测试。 - **2048 卡集群切分策略**： - **保守主线（1024 卡）**：用于运行核心模型方案与大规模验证实验。 - **消融支线（两个 512 卡子集群）**：用于并行验证各类小规模消融实验与激进探索。 --- ### [15:25](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=925s) 核心突破：SMEBU 与多项稳定性组合拳在尝试了学术界现有的多种负载均衡方法均告失败后，团队自主研发了名为 **SMEBU（Smooth/Bounded Momentum Expert-Bias Updates）** 的机制，彻底扭转了局势。 ```mermaid flowchart LR accTitle: SMEBU 动量专家偏置更新机制 subgraph Traditional["传统负载均衡"] T1["固定系数偏置"] --> T2["路由无法应对大量细粒度专家"] end subgraph SMEBU_Flow["SMEBU 动态平衡"] S1["监测跨层专家失衡程度"] --> S2["动量自适应调整偏置"] S2 --> S3["保持适度平衡并保留 Token 专精"] end ``` 常规的负载均衡主要依赖在路由损失（Router Loss）上添加固定系数偏置，对冷门专家施加激励。然而在当前架构中，专家数量极多且单个专家参数量较小，固定偏置的路由更新根本无法跟上漂移速度。SMEBU 引入了**基于动量（Momentum-based）**的动态调整机制：系统会监测全局（例如相隔 20 层的不同专家）的路由失衡程度，根据专家的脱靶（out of sync）比例自适应更新偏置幅度，该机制上线后立竿见影地解决了路由崩溃。在应用 SMEBU 时，团队也特别注意平衡专家的“稳定性”与“专精性（Specialization）”： - **避免过度平衡**：如果专家利用率被强行拉得过于绝对平均，专家就会丧失在特定 Token 类型或特定问题上的专精能力，从而失去 MoE 架构固有的容量优势； - **走好钢丝**：必须在维持路由稳定收敛与保留专家 Token 级专精特性之间找到精妙的平衡点。除了 SMEBU 外，团队还同步实施了多项系统级与算法级改进： - **计算精度回退**：将 Attention 部分原本采用的 MXFP8 混合精度回退至 BF16； - **Logit 稳定化**：引入 Z-loss 抑制 Logit 极大值漂移； - **辅助损失**：添加序列级辅助损失（Sequence Auxiliary Loss）； - **数据呈现优化**：后续引入了文档内掩码（Intra-doc Masking）等数据排布与打包机制。在这套组合拳之下，训练完全恢复正常，MaxVio 指标在整个 17T Tokens 的训练周期内始终保持平稳收敛。 --- ### [18:25](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1105s) MaxVio 指标的额外价值与未来展望（DOE 合作与万亿 MoE） Lucas 进一步分享了 MaxVio（最大违规度/路由失衡指标）的一项重要衍生用途：**评估数据分布漂移与数据多样性**。 MoE 模型对数据分布的变化极为敏感。当输入数据类型发生剧烈变化时，路由器原本形成的激活习惯会被打破，导致 MaxVio 剧烈震荡。因此，在引入新数据集进行 SFT（微调）或持续预训练（Continued Pre-training）时，可以通过在 Weights & Biases 中监控 MaxVio 指标来提前探测模型是否“见过”该类数据： - 如果 MaxVio 指标飙升至 12、15 甚至 17 以上（正常应保持在 6 以下），则明确表明模型从未见过该领域的数据分布，团队需要提前做好平滑过渡准备。在演讲尾声，Lucas 回顾了团队从 Trinity Nano、Trinity Mini 到本次 Trinity Large 的演进历程，并公布了下一阶段的重磅研发计划： - **Genesis Science 1**：与美国能源部（DOE）达成合作，基于下一代 Trinity 架构构建前沿科学模型，用于自动化 DOE 高性能计算（HPC）系统上的科学工作流、实验设计与快速迭代，预计于年内发布。 - **超越万亿参数**：下一代模型总参数量将突破 1 万亿（1T+ Parameters），并采用更极端的稀疏度（Way more sparse）设计。 Lucas 幽默地表示，希望明年的演讲不再是关于“如何救火排障”，而是一场纯粹的庆功分享。他在结语中动情地总结道：“我最引以为傲的工作，就是选择那些极其艰难、看似根本不可能完成的挑战，与充满斗志的伙伴并肩作战，然后用创造力杀出一条血路。” ## Notable quotes > [00:04](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=4s) 一个极小规模的团队如何在资源极端受限的情况下完成这一壮举 — Lucas Atkins > [02:08](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=128s) 一次开火的机会 — Lucas Atkins > [07:21](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=441s) 如果问题最终成功解决，功劳属于团队每一个人；如果方案失败甚至被迫降级，所有责任由他一人承担，团队成员绝不会因此受到惩罚。 — Lucas Atkins > [11:55](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=715s) 仿佛全世界的重量都压在他们肩上 — Lucas Atkins > [11:55](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=715s) 不，这些压力由我来扛，你们只需要专注于自己最擅长的事情——做顶尖的研究员和优秀的工程师。 — Lucas Atkins > [11:55](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=715s) 大脑只聚焦于自己能够掌控的事情 — Lucas Atkins > [11:55](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=715s) 只去解决你当下能改变的事。因为如果问题解决了，外面的担忧自然烟消云散；如果没解决，那也是之后才需要去面对的事。 — Lucas Atkins > [18:25](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1105s) 希望明年的演讲不再是关于‘如何救火排障’，而是一场纯粹的庆功分享。 — Lucas Atkins > [18:25](https://www.youtube.com/watch?v=V-9TscGJj-Y&t=1105s) 我最引以为傲的工作，就是选择那些极其艰难、看似根本不可能完成的挑战，与充满斗志的伙伴并肩作战，然后用创造力杀出一条血路。 — Lucas Atkins ## Mentioned Lucas Atkins, Arcee AI, Trinity Large, Trinity Mini, Trinity Nano, Genesis Science 1, NVIDIA, Blackwell B300, DeepSeek, 美国能源部 (DOE), Weights & Biases, SMEBU ## Video description Lucas Atkins - CTO, Arcee LLM progress now depends heavily on one practical issue: training stability at scale. Sparse Mixture-of-Experts (MoE) models are especially sensitive, since routing drift can overload experts, collapse utilization, and stall learning. In this talk, I will share an "anti-loss-spike" playbook from a recent open-weight run: a 400B-parameter MoE with 13B active parameters per token, trained for 17T tokens with an unsmoothed loss curve and zero loss spikes. I will start with the failure pattern we saw, router drift, overload, MaxVio divergence, and plateau, then cover the fixes that restored steady convergence: bounded and momentum expert-bias updates (SMEBU), z-loss for logit stabilization, a precision fallback from MXFP8 to BF16, better balancing objectives, and data/packing choices that reduced step-to-step variance. You will leave with a concrete checklist for stability instrumentation and first-response fixes to keep large open-weight runs on track. See less [Source](https://www.youtube.com/watch?v=V-9TscGJj-Y)

Source quotes

[0:04] Lucas Atkins: 一个极小规模的团队如何在资源极端受限的情况下完成这一壮举 [2:08] Lucas Atkins: 一次开火的机会 [7:21] Lucas Atkins: 如果问题最终成功解决，功劳属于团队每一个人；如果方案失败甚至被迫降级，所有责任由他一人承担，团队成员绝不会因此受到惩罚。 [11:55] Lucas Atkins: 仿佛全世界的重量都压在他们肩上 [11:55] Lucas Atkins: 不，这些压力由我来扛，你们只需要专注于自己最擅长的事情——做顶尖的研究员和优秀的工程师。 [11:55] Lucas Atkins: 大脑只聚焦于自己能够掌控的事情 [11:55] Lucas Atkins: 只去解决你当下能改变的事。因为如果问题解决了，外面的担忧自然烟消云散；如果没解决，那也是之后才需要去面对的事。 [18:25] Lucas Atkins: 希望明年的演讲不再是关于‘如何救火排障’，而是一场纯粹的庆功分享。 [18:25] Lucas Atkins: 我最引以为傲的工作，就是选择那些极其艰难、看似根本不可能完成的挑战，与充满斗志的伙伴并肩作战，然后用创造力杀出一条血路。

Source quotes

Lucas Atkins: LLM progress now depends heavily on one practical issue: training stability at scale. Lucas Atkins: Sparse Mixture-of-Experts (MoE) models are especially sensitive, since routing drift can overload experts, collapse utilization, and stall learning. Lucas Atkins: You will leave with a concrete checklist for stability instrumentation and first-response fixes to keep large open-weight runs on track. [0:04] Lucas Atkins: YOLO training runs [4:54] Lucas Atkins: dead parameters [8:15] Lucas Atkins: drop-dead day [13:45] Lucas Atkins: razor's edge [19:43] Lucas Atkins: I hope it's a celebratory talk. [19:43] Lucas Atkins: choosing extremely difficult things to do that we have no business trying to accomplish, working with very motivated people, and then having to create our way out of it.

Original transcript

[0:04] Um yeah, briefly to introduce myself, Lucas Adkins. I'm the CTO over at RCAI. We make [0:11] open-way language models here in the United States. And when it comes to this presentation, [0:20] uh which I've I I I started doing it in in some fashion around April of this year [0:27] based off of one of our kind of biggest YOLO training runs. [0:32] I I didn't want to make it an overly technical talk. Like I didn't want it just to be a [0:37] rehash of our tech report. Like we have a pretty good tech report that kind of goes into [0:42] everything, like all the technical architectural decisions behind our last series of models. [0:48] Um and at the end I can kind of go through [0:51] um you know, the scores and the benchmarks there. But I thought what would actually be useful [0:58] for you know, a general audience might be kind of how we managed that with a very small team. [1:06] Um we have we had at the time about 12 researchers. [1:13] Um we kind of made a a big bet that at the beginning, you know, in mid 2025, [1:20] that by the beginning of 2026, we could scale all the way from like a 4 billion parameter [1:25] dense model to a 400B mixture of experts model. Um and have it be really highly competitive. Um and [1:34] so not just were we a small team, we're also very resource constrained. We had um at max $20 million [1:41] we could spend on four models over the course of those six months. Now, that's a lot of money, [1:47] yes, but in the grand scheme of of training language models, that's um it's not a lot. So, [1:54] uh I've cut this talk down a bit. Um usually I'll do it over the course of 40 minutes, [1:59] so bear with me if I'm if I'm speeding through it, but I'll try to hit the highlights. Then [2:03] if I have any time at the end, maybe I'll take some questions cuz I sometimes find that that's [2:08] where some of the real alpha comes out. So, we went from uh we did a 26 billion parameter [2:16] model in December of last year. We did a 6 billion parameter MOE, and those were purely to prove out [2:22] Trinity large, uh that we knew how to how to do MOEs, that we could scale them at many, you know, [2:27] trillions of tokens, and we could handle the inherent complexity that comes with MOEs. Because [2:33] yes, they are more efficient, uh but they are a lot more There's a lot more you have to solve for [2:39] uh and optimize for while you're training. So, we moved immediately, the day that those two models [2:43] were released, we moved on to doing Trinity large. Um and we had uh 33 days to do the whole thing. [2:50] We had rented uh 2048 B300s. Our budget allowed us to have those for 33 days. And uh we couldn't go [3:00] over that. Uh what that also meant was we had one shot. We'd never tested at this scale before. We [3:06] couldn't afford it. We couldn't afford to have a cluster up for several months. We kind of had to [3:09] do small-scale work and then go. So, um we started the run. The plan was uh 20 trillion tokens [3:17] um in 33 days, which is already crazy at 400B, but they were Blackwells. You can get pretty fast with [3:23] them. We changed a few things. We made the model twice as sparse, meaning um we had twice as many [3:30] experts than we previously had in with any of our other MOEs, but kept the same number of active. [3:37] Uh that's one of the things that allowed us to do that was embracing sparsity. So, Trinity large is [3:41] 400 billion parameters um with only 13 billion of those active. If you're following like a deep seek [3:49] sparsity, uh that it probably would have been somewhere around 26, 27 billion. [3:54] So, we were very sparse. Um this is actually still the sparsest uh MoE that's ever been released. [4:00] We were going way up in parameters. So, going from 26 billion all the way to 400 billion, [4:05] um you know, almost uh 15, 16 times as much I mean, it's it's a drastic change. [4:11] Um and we'd never worked on a cluster of this size. We'd also never worked [4:14] with Blackwells before. This was the first large public run of Blackwells. [4:20] Um I'm certain there've been some with the closed labs, but publicly this was the first one and [4:25] Nvidia even had to have us take the cluster down for a day so they could install new firmware. [4:30] Um and then we had to completely shard the model differently. Um when you're doing 26B, [4:35] kind of fit a lot of those just on a single GPU and do data parallelism, which is a lot easier, [4:41] uh but this was a lot more sophisticated. Uh we had to get into like expert parallelism [4:45] and context parallelism. So, there were a lot we were changing. Ideally, when you're going [4:49] into a big run, you're not changing anything. Uh you're just scaling it up. We were scaling it up [4:54] and changing a lot. Um and then this is kind of uh important for the whole talk. So, this is uh [5:02] max uh VIO or, you know, violence, if you will, is a a measure of how [5:10] unbalanced or um you know, imbalanced your your experts are. So, if you have 256 experts, [5:18] um you have a a router and you're you're doing a a loss over that router to try to make it balance, [5:26] send uh an equal number of tokens to your total amount of experts. And the reason [5:30] this is important is because if they're not balanced, you'll end up having dead experts, [5:34] right? You'll have uh maybe you have 256, but only 100 ever get routed to. You have 156 dead experts, [5:41] you know, more than half your entire model is just dead parameters. Um so, this is a very important [5:47] metric you have to track when you're doing MOEs. Um There's many ways to do it. There's many [5:52] different names for it. This is the one we do. Uh, but overall, when you're looking at this number, [5:57] it is the difference between your most routed to, the number of tokens that your most routed [6:01] to expert gets, and your least routed to expert. Um, and ideally, you're keeping that under, [6:08] I mean, three would be a grade um, as far as the ratio, like your your most routed to expert is [6:16] only three times more routed to than your least. Um, anything less than that would be ideal. [6:22] Six is passable if you're at the end of training. Anything above that, you're probably going to have [6:27] a bad time. So, we started the run and everything was going great for the first couple days. [6:33] Uh, except uh, about a couple hundred billion tokens into the run, our max VO started to go way [6:40] up, but our loss didn't and our grad norm didn't, which those are usually the two things you need to [6:45] pay attention to. If those are super smooth, if they're not spiking, if they're continuing to go [6:49] down, generally everything's fine. But this this worried us because we knew that if we let this [6:55] keep going, if it kept going at the rate that it was, if it kept uh, if the if the if the balance [6:59] stopped being uh, or kept getting really bad, um, that we would eventually have catastrophic loss [7:07] and grad norm. And we did not want that to happen 15 trillion tokens in. We wanted that to happen [7:15] never, but the the worst thing is we could if if we couldn't fix it because we only had 33 days, [7:21] this worried us a lot. So, we were about four days into the run and um, [7:28] well, so we basically had to make a decision. We had to fix this. But everything we tried didn't [7:33] work. We tried changing the load balancing coefficients. We were using like signal free [7:37] or auxiliary free load balancing um, that came from deep seek. We kept changing those. It didn't [7:42] fix it. Um we tried adding more dense layers. That helped but didn't fix it. Uh we did a lot and and [7:50] and everyone kind of started to panic, right? Cuz the stakes really couldn't have been higher. We'd [7:54] bet the whole company um on that we would be able to do this. And since we were using [7:59] a massive amount of our our bank account to to kind of prove to the world that we were able to [8:04] do this in a in a you know, cheap and uh sped up and efficient manner. Um yeah, the stakes [8:10] couldn't have been higher. It was basically like the company was on the line. And and one of the [8:15] the reasons I wanted to frame the talk this way is kind of like how do you manage the [8:21] uh uh your team in in times of extreme crisis. Um and I there was a a couple ways. The first one [8:32] um was that I chose to myself what our um drop dead day was. By that I mean [8:43] at what point would we need to like cut the model size in half, go back to a sparsity that we'd used [8:48] before. And and and it'll still allow us to get through the 20 trillion tokens. None of us wanted [8:55] to do that. We'd planned for 400 B. We told people we were doing 400 B. A 400 B on paper to us felt a [9:01] lot more impressive than a 200 B. Um so what what I kind of did was I calculated how long [9:07] it would take us to train at 200 B assuming that sparsity was the only problem. Us going [9:13] twice as sparse is the reason that we had this max vio issue. Um and so I calculated that I had [9:20] uh 18 days I could train the whole model at 200 B at our previous sparsity. So in my head [9:25] I knew that once we hit day 12 if we still hadn't solved this problem that that's what [9:30] we were doing. I didn't tell anyone that. Um I didn't I didn't I didn't want people to think [9:35] of that as an option. Um but I personally had that in my mind. I made that decision and and [9:41] um, that's kind of the stakes with which I was dealing with. We basically had [9:46] you know, a five or six day window where we could solve this. [9:50] Um, one of my favorite things to do in in general when you're problem solving, but especially when [9:55] it comes to like research or engineering problems like this is like lowering the search space. [10:02] Um, rule out as many things as possible as to what the problem could be early on. So, [10:06] you're you're focusing on not just this massive um, piece of infrastructure, but the actual [10:13] the actual problem. So, we uh, did a run at our old sparsity. Right? We did it at 200 B [10:19] on the side uh, on a small subset of nodes to see if we had the same problem. We didn't. [10:25] Um, the only thing changing there was sparsity. So, we kind of we were able to immediately kind [10:31] of rule into the fact that it was sparsity was the issue. We were this us going this sparse. [10:36] There weren't papers of other people going this sparse. So, we we reasonably believe the closed [10:40] labs were going this sparse if not way more sparse, but there wasn't a lot of literature [10:44] about how to handle this level of sparsity at this size. Um, and so we had to figure that out. [10:51] And by lowering the search space early instead of looking at tokenizers or number of layers [10:57] or width or, you know, depth of the the the experts themselves, it allowed us to really [11:02] zero in on what that problem was. Um, and this is something for me as a leader that was extremely [11:06] important for me to let everyone know was was that if if this works, like if we solve this, [11:13] it's everyone on the team's it's because everyone on the team. And nothing to do with me. Uh, but [11:20] if it goes wrong, if we have to do that 200 B or even if that 200 B doesn't work, that's my fault. [11:27] Um, and there won't be consequences for them as a result of that. Uh and that was really important [11:34] for me. I think this is just a good lesson in general as being a leader and a manager, but [11:40] um this kind of came out of necessity because I saw people starting to make uh stupid decisions [11:48] out of fear. Right? Like you know, um pardon my French, French, [ __ ] was hitting the fan. [11:55] Uh they were nervous. Uh it felt like the entire weight of the world was on their shoulders. [12:00] And so I kind of had to say, "No, I'll put them on mine. Y'all just focus on doing what you do best, [12:05] which are excellent researchers and excellent engineers." Um and then [12:10] what ended up happening was um how do you focus on what's important? It is is is you could focus [12:18] on all these different things, right? There was um we had investor reactions to this not going well, [12:24] the sunk cost of like, "Oh, we've spent all this money." The narrative in the market of like, [12:28] "Oh, they couldn't actually do it." Um what the wider industry would say, whether our company can [12:33] survive this. Um and then people in their minds starting to panic and then blaming other people. [12:41] Those are all things people will worry about and they'll start to think about. [12:45] And you know, I'm a huge believer in like I can only I should only really focus in my brain on [12:51] things that I can control. Easier said than done, but uh with repetition and with a lot of work, [12:57] you can kind of really zero in on that. And that was my approach here. What could we actually do? [13:03] Uh we could do a ton of ablations. We could switch up the configurations. We could try [13:08] to come up with a new way of load balancing. Um and then we could figure out who decided that [13:13] and then what would we test next if that didn't work. Do that. And making sure that my team was [13:18] focused on these things and had an ample amount of things to do that felt like they were meaningfully [13:23] working towards the solution of this problem as opposed to having a lot of dead time or time by [13:28] themselves uh where it it'll obviously lead into that area of what you shouldn't focus on. Um so [13:35] really like what can you control? The rest of it doesn't matter. Because if you if you fix it, that [13:40] stuff doesn't matter. And if you don't fix it, you can deal with that afterwards. But deal deal with [13:45] what you can change in that moment. Um and then we also had to set up some some like ground rules of [13:50] like well how were decisions made because people started to throw a bunch of solutions out there. [13:54] Um and we had a big cluster which meant that uh people wanted to try their work. [13:59] Um and we ended up having situations where people like overnight would stop a run because [14:04] they thought it looked poor and they wanted to like try one of their thing you know solutions [14:09] and that wasn't you know conducive. Um we needed to kind of have a group of people [14:15] um which we ended up being four of us, me and my core researcher and and my infrastructure engineer [14:20] and another one of my team leads. We one of us at least had to agree to a change to the cluster to [14:27] the config. Um ideally we were all up but we have you know we had people up at all times of night [14:34] um super stressed so we wanted to make sure we at least had someone who we trusted to make that call [14:40] uh 24/7. That was really important was like who who makes the decisions um or who's the the gate [14:46] for whether or not something gets tried cuz otherwise again if you have 12 [14:50] really talented people all trying to solve this problem you can't be having you can't be doing 12 [14:57] uh very short tests. You need you need to be doing the ones that show the most fruition. [15:01] We also set up some side clusters. We split that 2048 um into like two [15:08] uh one 1024 cluster for larger ablations and then two smaller 512 clusters for smaller ablations. Um [15:17] and yeah that's exactly what this is. We had a conservative track which was using [15:21] like the larger cluster and then a smaller one, which were using their smaller machines. [15:25] Um and ultimately what we came up with uh the big one was something called SmeBu, which was [15:30] um the typical load balancing approaches. Uh you they they they go by a fixed coefficient. [15:36] So, if you have um uh an expert who's uh not being routed to as much, you will add a bias [15:44] to the router or to the loss function of the router, where you basically say, "Hey, for this [15:48] expert next time um it's incentivizing you to route to it more." And that's generally how you [15:55] do load balancing. Uh but that wasn't working for us, cuz we had so many experts. And and they were [16:00] smaller than you'd usually have. And so, the route This alone, the router wasn't able to keep up. So, [16:05] we tried a bunch of different techniques that were from papers. You know, there were some newer load [16:09] balancing approaches. None of those really solved it, but SmeBu did. So, the way SmeBu [16:13] works is it's momentum-based. So, if you have one expert who's um not getting as many tokens, [16:21] but then, you know, 20 layers over on a completely different expert, you have one [16:26] that's getting routed to 10 times less, you you you update the the the coefficient, like the bias, [16:33] relative to how out of sync they are. Um and that basically solved it immediately. And that was a [16:41] nice thing to have happen. Um and it kind of it makes sense. Um the risk there is that you over [16:47] uh route to the experts. Like, you don't want the experts to be too balanced, because then you don't [16:52] get any specialization, and that's one of the benefits of MoEs is that you have specialists. [16:57] Um token level specialists, not like, you know, one's like a physics specialist, and another one's [17:02] a coding specialist. But uh for like different uh types of tokens and different types of problems, [17:08] you you ideally specialize them. Um and if you're too balanced, you don't get that. You'll still [17:13] have a good model, but there's a it's a weird, you know, uh a razor's edge you're kind of trying to [17:20] to to to follow there where you want it uh balanced enough to be stable, [17:24] but not so balanced to the point where you're not getting the inherent benefits of doing an MOE. Um [17:30] we we switched to BF16. We were doing some of our expert uh some of our attention in MXFP8. [17:36] Um we added something called Z loss. Um and uh we did a we also added a sequence auxiliary loss. [17:44] Um and then we also did some stuff around, again I said adding dense layers. We also um [17:49] came up with like intra-doc masking. We didn't add that until after this fix went in. Um but that was [17:57] there's a whole there's a whole line of research that gets very exciting when you get into not just [18:01] what data you're showing the model and how much, but how you're showing that the data to the model [18:06] and how much. Um so I should probably update my next talks to be about that cuz I could talk at [18:12] length about that. But what ended up happening was uh we it was completely fixed. Um you know, on the [18:18] left there is is kind of what it was before. Uh on the right is where it sat uh for the entire run. [18:25] Uh another cool thing about Max ViO, by the way, is it's also a really really great indicator of um [18:34] uh data distribution and like data diversity. So MOEs are really [18:39] uh sensitive to shifts in data distribution because again the the router gets used to [18:44] hitting experts for a given type of document, a given type of input. When you change that, [18:50] it doesn't know what to do with those experts. So it's it's a really great uh way of seeing like [18:55] telling really early if your your MOE has seen this kind of data before. Like even if you're [18:59] using uh someone else's MOE or ours and you don't have access to that pre-training database or data, [19:06] if you want to know whether some data that you're introducing or a way of [19:10] visualizing that that information is introduced to the model. Has it seen that before? Is to add [19:15] some logging for max violence. Any model could help you set that up in like weights and biases. [19:20] Um and then introduce that data whether in SFT or even like doing like a continued pre-train. [19:25] And if it's really high, if you're like 17 or 15 or 12, if you're like anything above six, [19:31] uh it's a pretty good um it's a pretty good way of of knowing that that it has not seen [19:37] that data before. Um which is helpful for us in making sure that we smooth that over. [19:43] Okay, yeah. So we had Trinity Nano, Trinity Mini. That was Trinity Large and we spent [19:46] the last six months, well, four months um working on our next series of models, the first of which [19:53] will be uh a part of a a partnership we're doing with the DOE called Genesis Science 1, [19:58] which will be built on our next generation of models. So we're going at it again. Uh [20:04] this time we're going above a trillion parameters. We're going way more sparse. So I hope to not have [20:09] another talk about this next year. Um uh I hope it's a it's a it's a celebrate celebratory talk. [20:17] Um and that'll be working through to automate like the the workflows that the DOE does [20:21] um when it comes to using their HPC systems, experiment design, and iteration of that sort. [20:27] Um that'll come out later this year and that'll be trained on top of our next generation of Trinity. [20:32] So um very excited uh to go through that. And I'll close it out with this, [20:38] which is like the the work that I'm most proud of doing is like choosing extremely difficult [20:43] things to do that we have no business trying to accomplish, uh working with very motivated people, [20:48] and then having to create our way out of it. Um so thank you. Thanks for coming.