Crusoe Pre-Trains Frontier AI Models to Weed Out Faulty Cloud GPUs

From Stack to Symphony: Building Infrastructure That Performs as One System With Crusoe — Ai4

Ai4

Because large-scale AI training runs stall whenever a single processor underperforms, operators must validate full clusters with frontier model pre-training to catch marginal hardware defects before handoff.

AI-assisted summary of From Stack to Symphony: Building Infrastructure That Performs as One System With Crusoe — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 14, 2026, this presentation features Alex Akesson, Director of Product Management at Crusoe. According to the page description, the session examines how Crusoe provisions capacity, burn-in tests every GPU node, and validates cluster health through large-scale frontier model pre-training to build AI infrastructure that operates as a cohesive system.

## TL;DR Alex Akesson outlines Crusoe's four-stage infrastructure pipeline—pre-deployment, provisioning, burn-in, and cluster multi-node validation—designed to prevent single-node performance bottlenecks across massive AI training clusters. The framework utilizes digital twin auditing and an ephemeral RAM-based runtime (CURE) to eliminate configuration drift, followed by over 30 tiered stress tests that catch marginal thermal and electrical defects. To guarantee cluster readiness before customer handoff, Crusoe benchmarks Model FLOPs Utilization (MFU) by executing frontier foundation model pre-training runs across up to 1,000+ GPUs.

## Atmosphere & style The presentation delivers a technical, highly methodical, and authoritative overview of modern data center reliability engineering. Akesson speaks with structured clarity, systematically breaking down the chaotic failure modes of thousands of interconnected GPUs into repeatable, automated orchestration workflows. Watching the talk feels like an insider look into frontier AI hardware operations, balancing pragmatic low-level hardware debugging with large-scale distributed computing architecture.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=nunApaZBtgg&t=4s) Large-scale AI model training operates synchronously across thousands of GPUs, making the entire cluster strictly bottlenecked by any single underperforming or jittery node. - [00:04](https://www.youtube.com/watch?v=nunApaZBtgg&t=4s) Modern compute trays (such as NVIDIA GB200) integrate multiple CPUs, GPUs, DPUs, NICs, and BMCs, requiring precise synchronization across hundreds of sub-systems per rack and multi-rack InfiniBand/RoCE fabrics. - [01:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=115s) Crusoe employs a structured four-stage infrastructure pipeline—Pre-Deployment, Provisioning, Burn-In, and Cluster Multi-Node Validation—to transition hardware deterministically from physical delivery to production. - [03:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=185s) During Pre-Deployment, automated out-of-band Redfish discovery audits physical server hardware against a software 'digital twin' specification before host OS boot, blocking any non-compliant machines. - [05:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=305s) Fleet configuration drift is eliminated through declarative, idempotent SKU manifests and CURE (Crusoe Ubuntu Runtime Environment), an ephemeral RAM-only OS that executes updates without inheriting disk state. - [07:11](https://www.youtube.com/watch?v=nunApaZBtgg&t=431s) Firmware flashing follows a strict dependency order, updating the BMC and host BIOS out-of-band before systematically flashing GPUs, NICs, DPUs, and NVMe controllers prior to OS deployment. - [08:21](https://www.youtube.com/watch?v=nunApaZBtgg&t=501s) Burn-in testing targets marginal hardware defects that pass basic functional checks but throttle, corrupt data, or fail under heavy thermal and electrical load, executing inside customer-identical virtual environments. - [09:58](https://www.youtube.com/watch?v=nunApaZBtgg&t=598s) Burn-in incorporates sustained and pulsating TDP power stress cycles to mimic training dynamics, alongside tri-node ring topology tests that mathematically isolate interconnect versus node interface issues. - [12:31](https://www.youtube.com/watch?v=nunApaZBtgg&t=751s) Cluster health and throughput are validated by pre-training frontier AI models at scales exceeding 1,000 GPUs, measuring Model FLOPs Utilization (MFU) and catching thermal and performance outliers prior to handoff. - [14:27](https://www.youtube.com/watch?v=nunApaZBtgg&t=867s) Crusoe combats ongoing infrastructure entropy through continuous fleet audits and vertical integration spanning clean energy, data center design, and cloud software to maintain single-point operational accountability. - [15:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=955s) Participation in NVIDIA's Exemplar Cloud program verifies reference-grade performance for demanding customer production workloads across voice generation, video diffusion, and interactive world models.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=nunApaZBtgg&t=4s) AI Infrastructure as a Symphony: The Complexity of Modern GPU Clusters

Building artificial intelligence infrastructure requires coordinating a vast hierarchy of interconnected components so that they behave not as an unruly collection of disjointed parts, but rather as "one big symphony." Modern AI accelerators are far from single monolithic chips; a single compute tray comprises dozens of independent computing elements working together, each possessing its own firmware, software stack, operational settings, and low-level tunings. In a modern NVIDIA GB200 compute tray, the architecture combines four GPUs, two Grace CPUs, high-speed Network Interface Cards (NICs), Data Processing Units (DPUs), NVMe storage drives, and a Baseboard Management Controller (BMC) orchestrating the entire board.

```mermaid flowchart TD accTitle: AI Cluster Hierarchy accDescr: Visual hierarchy from single components up to a synchronous multi-rack cluster

subgraph Tray["GB200 Compute Tray"] CPUs["2x Grace CPUs"] GPUs["4x GPUs"] Peripherals["NICs, DPUs, NVMe & BMC"] end

subgraph Rack["Rack Scale (1 of 16+)"] Trays["18x Compute Trays"] Interconnect["NVLink & NVSwitches"] Support["Power & Liquid Cooling Units"] end

subgraph Cluster["Cluster Scale"] Fabric["InfiniBand / RoCE Network"] Switches["Switches, Transceivers & Optical Cabling"] end

Tray --> Rack Rack --> Cluster ```

When individual compute trays are aggregated into a complete rack, the complexity scales dramatically. An entire rack brings together 18 compute trays linked through NVLink interconnects and NVSwitches, alongside dedicated liquid cooling distribution units and redundant power delivery systems, totaling hundreds of distinct sub-systems per rack. Scaling beyond a single rack into a full cluster introduces 16 or more racks tied together through high-bandwidth InfiniBand or RoCE (RDMA over Converged Ethernet) fabrics complete with their own switches, optical transceivers, and cabling. Because large frontier AI training runs operate synchronously across thousands of processors, the entire job is strictly bound by its weakest link: if a single GPU experiences performance degradation or latency jitter, every other node across the cluster sits idle waiting for it. Delivering the reliability demanded by AI workloads requires engineering a finely tuned "Goldilocks environment" where every component is optimized for peak performance.

- **NVIDIA GB200 Compute Tray Components**: 4 GPUs, 2 Grace CPUs, multiple NICs, DPUs, NVMe storage drives, and an out-of-band BMC. - **Rack-Level Complexity**: 18 compute trays interconnected by NVLink and NVSwitches, backed by power supplies and cooling distribution units. - **Cluster Networking**: Multi-rack fabric leveraging InfiniBand or RoCE networking, high-radix switches, and optical transceivers. - **The Synchronous Bottleneck**: AI training operates synchronously; a single underperforming GPU stalls all other GPUs participating in the workload.

---

[01:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=115s) The Four-Stage Validation and Provisioning Pipeline

To transform thousands of disparate hardware components into a unified, deterministic compute platform, Crusoe developed a specialized four-stage deployment pipeline. This framework governs every step of a server's lifecycle from the moment physical hardware arrives at a data center until it runs production AI workloads.

```mermaid flowchart LR accTitle: Crusoe Four-Stage Infrastructure Pipeline accDescr: Sequence showing Pre-Deployment, Provisioning, Burn-in, and Cluster Validation

A["1. Pre-Deployment<br/>(Physical to Software-Ready)"] --> B["2. Provisioning<br/>(Baseline SKU Harmonization)"] B --> C["3. Burn-In<br/>(Stress & Subsystem Auditing)"] C --> D["4. Cluster Multi-Node Validation<br/>(Frontier Model Pre-training)"] ```

The pipeline begins with pre-deployment, taking physically racked servers and establishing their network identity, baseline connectivity, and hardware integrity. Once verified, the server enters the provisioning phase, where automated systems eliminate hardware drift and drive every node to a uniform, optimal state. Next, the burn-in phase subjects every subsystem to intense thermal and electrical stress to expose marginal hardware defects before workloads touch the machine. Finally, cluster multi-node validation evaluates thousands of GPUs concurrently by pre-training frontier models at scale, proving that the collective fabric achieves the established "known-good baseline" prior to customer handoff.

- **Phase 1 (Pre-Deployment)**: Bridges physical racking to software provisioning by discovering identity and verifying component presence. - **Phase 2 (Provisioning)**: Flashes firmware and configures tunings across every node to match a strict baseline manifest. - **Phase 3 (Burn-In)**: Applies stress tests across electrical, thermal, and compute subsystems to detect latent hardware defects. - **Phase 4 (Cluster Multi-Node Validation)**: Runs large-scale frontier model pre-training across thousands of GPUs to certify cluster-wide synchronization and throughput.

---

[03:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=185s) Pre-Deployment: Hardware Inventory and Digital Twin Verification

Pre-deployment begins the moment data center site technicians physically slide and secure a chassis into a server rack, operating before the host OS ever attempts its first boot. The initial "waiting stage" aggregates foundational vendor asset metadata, including chassis serial numbers, asset IDs, BMC MAC addresses, and initial authentication credentials, while waiting for the machine to request and receive a dynamic DHCP lease.

```mermaid flowchart TD accTitle: Pre-Deployment Stage Gating Flow accDescr: Flowchart illustrating the BMC inventory, Digital Twin comparison, and validation states

A["Chassis Racked & Powered"] --> B["Capture DHCP Lease & Reserve BMC IP"] B --> C["Redfish Inventory Query via BMC"] C --> D["Compare Against Digital Twin Specification"] D --> E{"10-20 Validation Checks"} E -- "Any Failure / Blocked" --> F["Automated Remediation or Tech Dispatch"] F --> C E -- "All Checks Green (Passed)" --> G["Promote to Provisioning Stage"] ```

Once the BMC IP is reserved, the automation framework communicates out-of-band via Redfish to inventory all internal subcomponents, controllers, and current firmware versions. The system cross-checks this real-time inventory against Crusoe's internal "digital twin" specification to verify that the physical machine precisely matches its planned architectural bill of materials. The system also verifies out-of-band administrative access. The server remains gated in this pre-deployment loop until 10 to 20 granular validation checks register a clean pass status; any stage flagged as failed, not run, or blocked triggers automatic remediation or dispatches a technician before software provisioning is allowed to begin.

- **Zero-Touch Discovery**: Ingests vendor metadata, binds the BMC MAC address to a reserved IP, and queries hardware through Redfish before host booting. - **Digital Twin Auditing**: Performs deep reconciliation between the physical inventory and the modeled software blueprint. - **Strict Gating Logic**: Hardware tracks across four explicit states—*Pass*, *Fail*, *Not Run*, or *Blocked*—and cannot proceed to provisioning until all checks are clear.

---

[05:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=305s) Provisioning: Fleet Consistency, Declarative Manifests, and CURE

When thousands of servers arrive across multiple manufacturing batches, hardware vendors inevitably ship machines with disparate firmware revisions, settings, and factory configurations. A cluster containing even minor configuration divergence—such as a single host operating on a mismatched NIC firmware version—risks degrading collective training performance. The provisioning phase solves this by harmonizing every node to an exact, qualified baseline using a declarative model that makes zero assumptions about the prior state of the machine.

```mermaid flowchart LR accTitle: Three Tenets of Crusoe Provisioning accDescr: Three foundational architectural pillars of the provisioning phase

T1["1. Inherit Nothing<br/>Ephemeral RAM-only OS (CURE)"] T2["2. Safe to Rerun<br/>Idempotent diff against SKU manifest"] T3["3. Verify, Don't Trust<br/>Validate against single source of truth"] ```

The provisioning architecture rests on three core pillars: 1. **Inherit Nothing**: Nodes boot into an ephemeral, RAM-resident root filesystem called CURE (Crusoe Ubuntu Runtime Environment), ensuring that every single server executes its provisioning steps within a pristine, bit-identical environment without residual state from disk. 2. **Safe to Rerun**: Every hardware SKU is paired with an idempotent declarative manifest defining its exact firmware revisions, software packages, and performance tunings. The automation continuously compares the node's detected state against the manifest and applies only the diff. 3. **Verify, Don't Trust**: Rather than trusting installation exit codes or vendor assertions, the orchestrator re-queries the system post-update to validate all reported version strings directly against the declared single source of truth.

- **Fleet Skew Mitigation**: Eliminates cross-batch variance caused by hardware vendors shipping different firmware revisions over time. - **CURE (Crusoe Ubuntu Runtime Environment)**: Ephemeral RAM-based OS providing an identical, immutable execution layer for maintenance and flashing. - **Idempotent Reconciliation**: Declarative SKU manifests allow automated updates to be executed repeatedly without causing side effects or configuration drift.

---

[07:11](https://www.youtube.com/watch?v=nunApaZBtgg&t=431s) The Firmware and Software Update Sequencing

Executing hundreds of component updates on complex compute trays demands a precise, staged dependency order. The provisioning workflow initiates with pre-boot setup and powers on the server to boot into the CURE RAM environment. Update routines begin with low-level out-of-band subsystems before touching high-bandwidth accelerators and storage controllers.

```mermaid flowchart TD accTitle: Sequential Firmware and OS Update Order accDescr: Step-by-step firmware flashing hierarchy from out-of-band BMC down to OS deployment

subgraph Stage1["1. Out-of-Band Phase (via Redfish)"] BMC["Baseboard Management Controller (BMC) Firmware"] --> BIOS["Host BIOS Firmware"] BIOS --> BIOS_Set["BIOS Settings (Power & Fan Profiles)"] end

subgraph Stage2["2. In-Band Subsystem Phase (via CURE)"] GPU["GPU Firmware"] --> NIC["NIC & DPU Firmware"] NIC --> NVME["NVMe Storage Firmware"] end

subgraph Stage3["3. Operating System Phase"] OS["Host OS Installation & Customer Preparation"] end

Stage1 --> Stage2 Stage2 --> Stage3 ```

The BMC firmware is flashed first because it acts as the primary orchestrator for the entire server chassis. Once the BMC is updated, the engine flashes the host BIOS and enforces critical BIOS-level settings out-of-band via Redfish, establishing optimized power delivery curves and fan thermal profiles. With out-of-band management secured, the system systematically flashes the GPU firmware, NIC and DPU firmware, and NVMe controller firmware. Only after every low-level component matches the SKU manifest does the workflow proceed to the final operating system installation, leaving the node configured in its highest-performing state.

- **Orchestration Precedence**: Prioritizes BMC firmware updates first to ensure out-of-band communication stability throughout subsequent steps. - **Out-of-Band BIOS Tuning**: Configures system-level fan profiles, thermal headroom, and power profiles via Redfish prior to host initialization. - **Subsystem Flashing**: Systematically updates GPU, network interface card, DPU, and storage controller firmware before host OS deployment.

---

[08:21](https://www.youtube.com/watch?v=nunApaZBtgg&t=501s) Burn-In: Detecting Marginal Errors Through Escalated Stress Testing

Once provisioning ensures that the entire fleet is structurally identical, the burn-in phase auditions every server to uncover hardware defects. Blatantly broken machines are trivial to identify because they generate fatal initialization faults, populate System Event Logs (SEL), or emit driver error codes like NVIDIA XIDs. The true objective of burn-in is uncovering "marginal errors"—subtle hardware flaws where a node enumerates cleanly and passes light functional checks, but stumbles, throttles, or introduces data corruption as soon as it experiences heavy thermal and electrical load.

```mermaid flowchart TD accTitle: Burn-In Escalation Hierarchy accDescr: Three-tier burn-in testing hierarchy spanning Node, GPU, and Fabric stress runs

subgraph Tier3["Tier 3: Fabric Tests"] F1["Multi-node Interconnect Congestion"] F2["RoCE / InfiniBand Throughput Under Load"] end

subgraph Tier2["Tier 2: GPU Tests"] G1["High-Power Compute Workloads"] G2["Thermal Throttling & Power Limit Sweeps"] end

subgraph Tier1["Tier 1: Node Tests"] N1["Host Memory & CPU Stress"] N2["PCIe Bus Integrity & Storage I/O"] end

Tier1 --> Tier2 Tier2 --> Tier3 ```

To replicate production conditions faithfully, burn-in suites run directly inside the same virtualized environments exposed to end users. The framework executes over 30 stress tests organized in an escalating hierarchy: basic node-level sanity checks progress into intense single-node GPU benchmarks, which subsequently escalate into large-scale network fabric stress tests. The burn-in suite incorporates a continuous feedback loop that regularly updates test parameters, diagnostic scripts, and failure thresholds based on emerging hardware patterns observed in production.

- **Targeting Marginality**: Specifically engineered to trigger intermittent hardware anomalies that only manifest during sustained thermal and electrical saturation. - **Virtualized Parity**: Executes diagnostics inside the identical virtualization layer used by end customers to ensure realistic performance profiles. - **Over 30 Multi-Layer Tests**: Progresses through escalating tiers spanning Node Tests (CPU/RAM/PCIe), GPU Tests (matrix engines/thermals), and Fabric Tests (high-throughput interconnects). - **Dynamic Test Evolution**: Continuously tunes diagnostic test thresholds and adds test cases based on real-world cluster telemetry and failure analysis.

[09:58](https://www.youtube.com/watch?v=nunApaZBtgg&t=598s) Tiered Burn-In Testing: Sanity Checks to Power Stress and Fabric Ring Topologies

To maximize operational efficiency and avoid wasted testing time, Crusoe structures its burn-in pipeline around a "fail fast" methodology. Quick sanity checks take only seconds to run, whereas comprehensive stress tests require multiple hours. Filtering out defective hardware immediately prevents long testing queues from being consumed by failing nodes, accelerating the overall delivery of healthy capacity to customers.

```mermaid flowchart TD accTitle: Crusoe Node and Fabric Testing Pipeline subgraph NodeLevel["1. Node-Level Validation"] A["Sanity Checks<br/>(CPU, Memory, Storage)"] --> B["Storage Benchmarking<br/>(NVMe Bandwidth, IOPS, Latency)"] B --> C["GPU Diagnostics<br/>(NVIDIA DCGM & AMD Field Diagnostics)"] C --> D["Power Stress Testing<br/>(Sustained TDP & Pulsating Cycles)"] end

subgraph FabricLevel["2. Fabric Validation"] E["Single-Node Fabric Tests"] --> F["Multi-Node Collective Tests<br/>(NCCL / RCCL)"] F --> G["Ring Topology Isolation<br/>(A to B, B to C, A to C Link Checks)"] end

NodeLevel --> FabricLevel ```

The burn-in suite tests standard CPU, memory, and storage subsystems, benchmarking local NVMe and block storage for bandwidth, IOPS, and latency to catch degraded drives. GPU testing relies on vendor diagnostic suites, including NVIDIA's DCGM suite run at varying intensities and AMD's GPU field diagnostic tests.

Beyond standard compute diagnostics, Crusoe tests GPU power dynamics by pushing accelerators to their thermal design power (TDP) for sustained durations, as well as executing pulsating power tests that cycle up and down to emulate the bursty power draw of production training jobs.

- **Storage validation**: Continuously benchmarks bandwidth, IOPS, and latency across local NVMe and network block storage to eliminate degraded disks. - **Vendor-specific GPU diagnostics**: Deploys NVIDIA DCGM test suites at varying intensities alongside AMD GPU field diagnostic test suites. - **Power characterization**: Validates thermal and power stability under sustained maximum TDP loads and pulsating power cycles mimicking real training step phases. - **Fabric-level validation**: Supports InfiniBand and RoCE fabrics using multi-node collective communication benchmarks via NCCL and RCCL. - **Ring topology testing**: Employs tri-node loop diagnostics ($A \to B$, $B \to C$, $A \to C$) to mathematically isolate whether a degradation stems from a specific node interface or an interconnect link.

---

[12:31](https://www.youtube.com/watch?v=nunApaZBtgg&t=751s) Large-Scale Cluster Testing and Model FLOPs Utilization (MFU)

Once individual nodes and network links pass burn-in, Crusoe validates the environment through end-to-end multi-node pre-training workloads across the entire cluster. These validation runs span from 8 up to 1,000+ GPUs (representing multi-megawatt loads) across multiple numerical precision levels, executing frontier open-source foundation models. Cluster health is judged primarily by Model FLOPs Utilization (MFU)—the percentage of theoretical peak compute that directly translates into model training progress.

```mermaid flowchart LR accTitle: Cluster Validation and Anomaly Resolution A["Multi-Megawatt Pre-Training Run<br/>(512+ GPUs on Frontier Models)"] --> B["Telemetric Anomaly Detection<br/>(e.g., Node 40 GPU 7 +20°C Delta)"] B --> C["Node Eviction & Hardware Replacement"] C --> D["Pre-Training Re-run & Target MFU Verification"] D --> E["Production Customer Handoff"] ```

During a representative 512-GPU pre-training run on a frontier model, telemetry revealed an anomalous blue trace: GPU 7 on node 40 consistently ran 20°C hotter than the rest of the cluster under identical load. Although the cluster was functioning, the thermal outlier indicated impending degradation. Crusoe took the affected node out of the pool, replaced it with a fresh node, and re-executed the pre-training workload to confirm that the full cluster achieved its expected MFU benchmark before customer handoff.

- **Scale of validation**: Stress tests range from small 8-GPU slices up to multi-megawatt footprints exceeding 1,000 GPUs. - **Primary benchmark metric**: Evaluates real Model FLOPs Utilization (MFU) to ensure customers receive maximum usable compute per dollar. - **Thermal anomaly mitigation**: Demonstrates that early detection of a 20°C single-GPU delta prevents cluster-wide thermal throttling or unexpected mid-run job failures.

---

[14:27](https://www.youtube.com/watch?v=nunApaZBtgg&t=867s) Continuous Fleet Auditing, Combatting Entropy, and Vertical Integration

Cluster qualification represents a snapshot in time. Production environments continuously face infrastructure entropy as new kernel versions, firmware revisions, and CUDA combinations are released over weeks and months. To maintain baseline performance, Crusoe conducts continuous fleet audits and works directly with hardware vendors to qualify and deploy updates across the fleet's lifecycle.

```mermaid flowchart TD accTitle: Crusoe Full-Stack Accountability Model subgraph VertIntegration["Crusoe Vertically Integrated Model"] direction TB P["Clean Energy Generation"] D["Data Center Design & Operations"] C["Cloud Platform & Cluster Orchestration"] P --- D --- C end VertIntegration -->|Single Layer of Accountability| R["Consistent Lifecycle Performance & Zero Drift"] ```

Crusoe's vertical integration across energy generation, data center operations, and cloud orchestration provides a single operational owner for every hardware, thermal, power, and software layer. This structure eliminates finger-pointing between data center providers and cloud operators when diagnosing complex distributed anomalies.

- **Managing system entropy**: Continuously audits live clusters against configuration drift caused by new firmware, drivers, and CUDA dependencies. - **Lifecycle performance**: Maintains initial handoff performance standards across the entire lifetime of the cluster. - **Vertical integration**: Combines clean energy sourcing, custom data center builds, and cloud software under a unified operational model.

---

[15:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=955s) NVIDIA Exemplar Validation and Production AI-Native Workloads

To provide external verification of cluster performance, Crusoe participates in NVIDIA's Exemplar Cloud validation program. This program subjects cloud infrastructure to transparent reference benchmarking using real foundation model training recipes across various frameworks, parameter scales, and precisions. Crusoe demonstrated that its fleet—including NVIDIA B200 GPU infrastructure—matches NVIDIA's exact reference performance targets.

In production, this infrastructure supports several AI-native companies running mission-critical workloads:

| Customer | AI Domain | Primary Workload / Model | Infrastructure Requirement | |:--- |:--- |:--- |:--- | | **Cartisia** | Voice AI | Sonic real-time voice generation model | Ultra-low latency and consistent GPU throughput | | **Linum** | Video Generation | Text-to-video diffusion pipelines for animated films | High-bandwidth multi-node compute | | **Odyssey** | World Simulation | Interactive, large-scale generative world models | Massive cluster reliability and synchronized fabric |

Alex concludes that Crusoe’s extensive validation and burn-in investment exists to ensure customers receive an integrated, synchronized system rather than a collection of disjointed GPUs:

> "We want to make sure that when you come to us, you get a whole system that works as a symphony. You're not just getting one single GPU... Our goal is really firm. It is to make AI infrastructure that is as performant as possible for our customers."

## Notable quotes > [00:04](https://www.youtube.com/watch?v=nunApaZBtgg&t=4s) one big symphony — Alex Akesson

> [00:04](https://www.youtube.com/watch?v=nunApaZBtgg&t=4s) Goldilocks environment

> [01:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=115s) known-good baseline

> [03:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=185s) digital twin

> [05:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=305s) Inherit Nothing

> [05:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=305s) Safe to Rerun

> [05:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=305s) Verify, Don't Trust

> [08:21](https://www.youtube.com/watch?v=nunApaZBtgg&t=501s) marginal errors

> [09:58](https://www.youtube.com/watch?v=nunApaZBtgg&t=598s) fail fast

> [15:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=955s) We want to make sure that when you come to us, you get a whole system that works as a symphony. You're not just getting one single GPU... Our goal is really firm. It is to make AI infrastructure that is as performant as possible for our customers. — Alex Akesson

## Mentioned Alex Akesson, Ai4, AMD, AMD GPU Field Diagnostics, Cartisia, Crusoe, CURE (Crusoe Ubuntu Runtime Environment), Grace CPU, InfiniBand, Linum, NCCL, NVIDIA, NVIDIA B200, NVIDIA DCGM, NVIDIA Exemplar Cloud, NVIDIA GB200, NVLink, NVSwitch, Odyssey, RCCL, Redfish, RoCE, Sonic

## Video description Alex Akesson - Director, Product Management, Crusoe

At scale, one bad node can stall an entire training run. From Stack to Symphony takes you inside the journey every GPU node makes from rack to production: how we provision capacity, burn-in test every node, and prove cluster health by pre-training frontier models at scale. You'll leave with a clear framework and proven strategies for building infrastructure that performs as one system, with the reliability and predictability modern AI workloads demand.

[Source](https://www.youtube.com/watch?v=nunApaZBtgg)

中文简报

Crusoe预训练前沿AI模型以剔除故障云GPU 由于只要单个处理器性能不佳就会导致大规模AI训练停滞，运营商必须利用前沿模型预训练验证完整集群，以便在交付前捕获边缘硬件缺陷。 Alex Akesson概述了Crusoe包含部署前、配置、老化测试和集群多节点验证的四阶段基础设施管线，旨在防止大规模AI训练集群出现单节点性能瓶颈。该框架利用数字孪生审计和基于RAM的临时运行时（CURE）来消除配置漂移，随后通过30多项分层压力测试捕获边缘热与电气缺陷。为确保在向客户交付前集群已完全就绪，Crusoe通过在多达1,000多颗GPU上运行前沿基础模型预训练，对模型FLOPs利用率（MFU）进行基准测试。 现代 AI 大规模训练高度依赖数千张 GPU 的同步协作，任何单一节点的抖动或降温降频都会拖慢整个集群，因此计算托架与整机架必须作为协同系统统一调优。 (0:04) Crusoe 采用四阶段基础设施流水线（预部署、配置部署、老化测试与集群多节点验证），确保硬件从机房上架到生产交付实现全流程确定性落地。 (1:55) 在预部署与配置阶段，系统通过带外 Redfish 审计物理资产并与“数字孪生”规范比对，随后借助基于内存的临时操作系统 CURE 与声明式 SKU 清单执行幂等固件刷写，彻底杜绝配置漂移。 (3:05) 老化测试设计了持续满载与脉冲式 TDP 功耗压测，并运用三节点环形拓扑诊断，在逼近客户真实虚拟化环境中精准暴露潜在硬件隐患与网络互联瓶颈。 (8:21) 集群在正式交付前会运行多兆瓦级的前沿模型预训练压测，通过模型 FLOPs 利用率（MFU）与精细遥测排查细微温差等异常，确保算力集群达到 NVIDIA 参考性能标准。 (12:31)

Reading summary

Crusoe manages the operational complexity of large-scale GPU clusters through a four-stage provisioning and validation pipeline designed to prevent synchronous training bottlenecks. By treating multi-rack AI hardware as a unified, vertically integrated system, the company aims to eliminate configuration drift, isolate marginal hardware defects, and sustain target compute efficiency. Synchronous Bottlenecks in Modern Accelerator Clusters (0:04) Modern artificial intelligence infrastructure scales from individual multi-chip compute trays up to clusters spanning 16 or more racks connected by InfiniBand or RoCE fabrics. A single NVIDIA GB200 tray integrates four GPUs, two Grace CPUs, network interface cards, DPUs, NVMe drives, and a Baseboard Management Controller (BMC), while an entire rack contains 18 trays alongside NVLink switches and liquid cooling units. Because large-scale frontier AI model training operates synchronously across thousands of processors, a performance drop or latency spike in a single component causes the entire cluster to wait. Delivering the necessary reliability requires orchestrating hundreds of subsystems per rack into a deterministic environment that eliminates the risk of an individual weak link halting the broader collective workload. Pre-Deployment Verification and Declarative Provisioning (3:05) To handle incoming hardware from disparate manufacturing batches without inherited state, Crusoe utilizes a pre-deployment and provisioning framework. Before host booting, the system queries physical assets out-of-band via Redfish, cross-referencing components against an internal digital twin model across 10 to 20 gating checks. Unverified machines are held in explicit gating states—Pass, Fail, Not Run, or Blocked—triggering automated remediation or physical technician dispatches before proceeding. Once cleared, nodes boot into the Crusoe Ubuntu Runtime Environment (CURE), an ephemeral, RAM-resident OS that ensures updates execute within an identical baseline. Provisioning adheres to declarative, idempotent manifests defining exact firmware revisions and system tunings. To avoid initialization errors, updates follow a strict hierarchy: out-of-band BMC and BIOS updates occur first to lock in fan and power curves, followed sequentially by GPU, NIC/DPU, NVMe controller firmware, and the operating system. Tiered Burn-In and Latent Defect Detection (8:21) Rather than focusing only on immediate component failures, Crusoe's burn-in stage targets marginal errors that only emerge under heavy electrical and thermal saturation. Diagnostics run inside the same virtualized environments exposed to end users and employ an escalating fail-fast design across more than 30 tests. Rapid sanity checks screen out blatantly broken nodes before time-intensive procedures tie up diagnostic queues. Validation covers CPU, memory, and NVMe throughput before testing GPUs with NVIDIA DCGM and AMD diagnostic suites. To replicate real-world training conditions, tests apply sustained maximum Thermal Design Power (TDP) as well as pulsating power cycles to emulate bursty training phases. At the network tier, multi-node NCCL and RCCL benchmarks evaluate interconnects, utilizing tri-node ring topologies (A to B, B to C, A to C) to isolate whether communication anomalies stem from a single node interface or the network fabric. Cluster-Scale MFU Validation and Lifecycle Auditing (12:31) Final cluster qualification evaluates end-to-end multi-node foundation model pre-training across scales from 8 to over 1,000 GPUs under multi-megawatt loads. Cluster health is assessed using Model FLOPs Utilization (MFU), which measures the proportion of theoretical peak compute translated into active training. In one cited 512-GPU run, telemetry flagged an anomalous single-GPU temperature delta of +20°C on node 40, prompting eviction and replacement to prevent future thermal throttling. To counter system entropy caused by ongoing driver, kernel, and firmware updates post-deployment, Crusoe uses continuous fleet auditing. Company leadership asserts that vertical integration across clean power generation, proprietary data center design, and cloud software creates a single layer of operational accountability, preventing finger-pointing between separate infrastructure and facility providers. Crusoe tests its B200 and accelerator clusters against NVIDIA's Exemplar Cloud program benchmarks to verify parity with vendor reference standards. While production deployments support workloads like real-time voice, video generation, and world simulation models, maintaining high Model FLOPs Utilization relies on the constant upkeep of diagnostic test profiles against ongoing software and driver revisions.

精炼阅读摘要

Crusoe指出，现代AI基础设施由复杂的计算托盘、机架互联及网络织构层叠而成，由于前沿模型训练具有严格的同步依赖性，集群必须如交响乐般协同运作，任何单点性能退化都会迫使全局等待。 现代GPU集群的架构复杂性与同步瓶颈 (0:04) 现代AI加速器已演变为由多重独立单元构成的高度复合体。以NVIDIA GB200计算托盘为例，单一托盘内集成了4个GPU、2个Grace CPU、高速网卡（NIC）、数据处理器（DPU）、NVMe存储以及负责底板管理的基板管理控制器（BMC）。当18个托盘组合成机架时，需要依托NVLink与NVSwitch进行互联，并辅以专用液冷分配单元与冗余供电系统；当规模进一步扩展至由16个以上机架组成的集群时，则必须经由InfiniBand或RoCE高带宽网络、高基数交换机及光收发模块实现互通。在千卡级别的大规模前沿AI训练中，计算节点间以同步方式运行，整个工作负载由最薄弱环节决定。集群内只要有一块GPU遭遇性能降级或延迟抖动，其他所有节点都会陷入空闲等待，因此基础设施必须提供高度确定性且调优至最佳状态的“金发女孩环境”（Goldilocks environment）。 预部署与固件编排消弭硬件配置漂移 (3:05) 为使成千上万来自不同制造批次的服务器达到统一基线，Crusoe构建了标准化的四阶段流水线。在宿主机操作系统启动之前的预部署阶段，自动化框架在获取DHCP租约并保留BMC IP后，通过Redfish带外协议清点内部组件，并与内部“数字孪生”（Digital Twin）规范进行比对；服务器必须通过10至20项验证检查，处于通过（Pass）状态后才被允许进入下一阶段。进入配置阶段后，为解决供应商固件版本不一导致的集群配置偏移，服务器会引导进入名为CURE（Crusoe Ubuntu Runtime Environment）的内存瞬态运行环境。该体系遵循“不继承任何磁盘历史状态”、“基于SKU清单实现幂等比对与差异化更新”以及“更新后直接重查版本而非盲信安装退出码”三大原则，并严格按照依赖顺序依次刷写底层固件：首先升级作为整机编排核心的BMC固件，随后通过Redfish更新宿主机BIOS并设定功耗与风扇温控配置，继而烧录GPU、NIC、DPU及NVMe固件，最终才部署宿主机操作系统。 阶梯式老化测试排查边际硬件缺陷 (8:21) 服务器完成一致性配置后将进入老化测试（Burn-In）阶段，旨在揭露那些能顺利枚举并能通过轻度功能检查、但在重度热与电力负载下才会发生节流或数据损坏的“边际错误”（marginal errors）。测试套件直接在暴露给最终客户的同等虚拟化环境中运行，包含30多项测试，并严格遵循“快速失败”（fail fast）的递进层级：先以数秒内完成的节点级CPU、内存与NVMe存储基准测试（包含带宽、IOPS和延迟）筛选明显故障；随后运行不同强度的NVIDIA DCGM或AMD GPU现场诊断套件，同时施加持续达到热设计功耗（TDP）的满载压力与模拟突发训练脉冲的交替功率循环；最后上升至多节点织构测试，使用NCCL与RCCL执行集合通信基准测试，并借助三节点环形拓扑测试（A至B、B至C、A至C）在数学层面上精准隔离单节点接口与互联链路的劣化问题。 集群MFU验证与全栈垂直整合治理 (12:31) 在单机与链路测试达标后，Crusoe通过运行跨越8卡至1000卡以上（对应数兆瓦电力负载）的前沿开源基础模型预训练任务进行全集群端到端验证，并将模型FLOPs利用率（MFU）作为判定集群健康度的核心指标。在一次512卡GPU的实际测试中，遥测系统发现节点40的GPU 7在同等负载下温度持续高出20°C，团队随即剔除该节点并更换硬件，确认整网达到预期MFU基准后再交付。针对生产环境中因内核、固件及CUDA版本更迭引发的基础设施熵增，Crusoe持续执行集群审计，并借助整合清洁能源生成、数据中心设计与云平台编排的垂直一体化架构，确立单一责任主体，避免运营商与数据中心之间的推诿。此外，Crusoe参与了NVIDIA Exemplar Cloud验证项目，证明其包含B200 GPU在内的设施达到了NVIDIA基准目标，支撑了Cartisia（低延迟语音生成）、Linum（高带宽文本到视频扩散管线）以及Odyssey（世界模拟模型）等生产级AI负载。 Crusoe的工程重点在于通过严苛的带外验证、受控刷写和实网预训练，将散乱的算力组件整合为高度确定的统一系统；但在实际生产中，如何随着底层加速芯片与网络规模持续激增，长期维持跨组件的无漂移状态，仍依赖于云厂商对全栈基础设施的持续审计与软硬件强力协同。

Detailed notes — zh

## Background 本视频由 Ai4 频道于 2026 年 8 月 14 日发布在 YouTube 平台。视频由 Crusoe 产品管理总监 Alex Akesson 主讲，介绍了 Crusoe 如何将 GPU 节点从物理上架逐步交付至生产环境，通过全流程配置拨正、严苛老化测试以及千卡级前沿大模型预训练验证，构建高可靠、高一致性的 AI 算力集群。 ## TL;DR 演讲深入剖析了现代 GPU 集群因高度同步分布式训练而面临的“木桶效应”挑战，并详细拆解了 Crusoe 的四阶段节点准入交付框架。在预部署与配置拨正阶段，系统通过数字孪生校验和基于内存的 CURE 临时系统实现无继承、幂等的确定性固件刷写；在压力老化测试中，结合快速失败、动态功耗与环形网络拓扑测试筛查隐蔽故障；最后在千卡规模上通过真实大模型预训练以 MFU（模型算力利用率）为标准进行整体验证，确保软硬件如交响乐般协同运作。 ## Atmosphere & style 整场演讲呈现出严谨、沉稳且高度工程化的专业硬核风格，Alex Akesson 结构清晰地将原本繁杂的数据中心运维工作解构为确定性的自动化流水线。叙述语调冷静自信，充满对系统底层细节的掌控力，既有对“亚健康单卡温度异常”等实际工程排障案例的生动还原，又展现出对抗系统熵增的宏大系统思维。观众在聆听过程中能直观感受到现代 AI 超级计算机在庞大规模背后所要求的极致秩序感与工业美学。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=nunApaZBtgg&t=4s) 现代 AI 集群属于高度协同的复杂系统，大规模分布式同步训练对短板节点极度敏感，单张滞后 GPU 即可拖垮整个万卡集群的效率。 - [01:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=115s) Crusoe 建立了标准化的硬件上线四阶段管线：预部署（Pre-deployment）、配置拨正（Provisioning）、压力老化测试（Burn-in）及集群多节点验证（Cluster Multi-node Validation）。 - [03:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=185s) 预部署阶段利用 Redfish 协议枚举物理元器件，并与资产系统的“数字孪生”模型逐一比对，确保硬件拓扑无误后方可通电开机。 - [05:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=305s) 配置拨正阶段遵循“不继承旧状态、幂等可重跑、永不盲信严格验证”三大原则，通过内存临时系统 CURE 实现异构出厂批次的确定性基线配置。 - [08:16](https://www.youtube.com/watch?v=nunApaZBtgg&t=496s) 压力老化测试着重捕获轻载下正常、但高并发热应力与极端电气负载下偶发崩溃的“边缘故障”（Marginal Errors）。 - [09:58](https://www.youtube.com/watch?v=nunApaZBtgg&t=598s) 测试套件遵循“快速失败”设计，在轻量健全性检查后开展数小时的 GPU DCGM/AMD 诊断、持续满载及脉冲式功耗测试，并通过环形拓扑排查 Fabric 网络互联瓶颈。 - [12:31](https://www.youtube.com/watch?v=nunApaZBtgg&t=751s) 集群级测试直接在千卡规模上预训练开源前沿大模型，以 MFU（模型算力利用率）为核心准绳，实测捕获并剔除潜在的过热或亚健康节点。 - [14:27](https://www.youtube.com/watch?v=nunApaZBtgg&t=867s) 面对持续的“系统熵增”，Crusoe 依托绿色能源、数据中心与 AI 云平台的端到端垂直整合架构，持续进行集群审计与驱动/CUDA 环境认证。 - [15:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=955s) 集群通过 NVIDIA Exemplar Cloud 官方性能基准验证，成功支撑 Cartisia 语音大模型、Linum 视频生成及 Odyssey 物理世界仿真等尖端 AI 原生业务。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=nunApaZBtgg&t=4s) 从堆叠走向交响：现代 AI 基础设施的协同挑战 在现代数据中心中，构建 AI 基础设施绝不是将成堆的硬件部件简单堆叠在一起，而是要让它们像一支出色的“交响乐团”（Symphony）一样高度协同运作。现代 GPU 服务器本身就是一个高度复杂的微型系统，由数十台独立计算机与部件组合而成，各自运行着独立的固件、软件，包含成百上千项配置与参数调优。 以当前一代 NVIDIA GB200 计算托盘（Compute Tray）为例，仅一个托盘内就集成了 4 颗 GPU、2 颗 Grace CPU、多块网卡（NIC）、DPU、NVMe 固态硬盘，以及用于监控和编排整个系统的基板管理控制器（BMC）。放眼整个机架，规模更呈指数级放大：一个机架包含 18 个计算托盘，通过 NVLink 和 NVLink Switch 互联，并配备专门的供电模块与冷却单元，涉及数百个相互协作的子系统。而这仅仅是 16 个机架中的其中之一；当扩展到整个集群时，还会引入 InfiniBand 和 RoCE（RDMA over Converged Ethernet）网络层，包含数以万计的交换机、光模块与线缆。 大规模分布式前沿模型的训练是完全同步的（Synchronous），所有计算节点之间存在严格的时序依赖。如果集群中出现哪怕一块性能较慢的 GPU，其他所有 GPU 都会陷入等待，直接拖垮整体效率。因此，Crusoe 必须在整个集群中打造一个严苛的“黄金环境”（Goldilocks environment），让每一个组件都调优至巅峰性能状态。 ```mermaid flowchart TD accTitle: AI集群多层依赖架构图 subgraph Cluster["集群与网络互联 (Cluster & Fabric)"] SW["InfiniBand / RoCE 交换机"] --- TR["光模块与线缆"] end subgraph Rack["整机架互联 (GB200 NVL72 机架，共18个托盘)"] NVSW["NVLink & NV Switch"] --- PDU["供电与冷却单元"] end subgraph Tray["单计算托盘 (GB200 Compute Tray)"] BMC["BMC 管理控制器"] GPU["4x GPU + 2x Grace CPU"] IO["NIC / DPU / NVMe 存储"] BMC --> GPU BMC --> IO end Cluster --- Rack Rack --- Tray ``` - GB200 计算托盘由 4 颗 GPU、2 颗 Grace CPU、网卡、DPU、NVMe 和 BMC 构成。 - 18 个计算托盘在单机架内通过 NVLink/NV Switch、冷却及供电系统组合运行。 - 集群扩展依赖大规模 InfiniBand/RoCE 网络、交换机与光纤互联。 - 同步训练任务对短板节点极度敏感，“木桶效应”会导致万卡集群性能被单点拖累。 --- ### [01:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=115s) 硬件上线管道：端到端四阶段交付框架 为了让海量离散硬件步调一致地工作，Crusoe 研发并持续迭代了一套标准的节点准入管道（Pipeline）。该管道划分为四个关键阶段，覆盖从物理服务器入架到最终交付生产的完整生命周期： 1. **预部署（Pre-deployment）**：服务器物理装架后的第一道防线。主要负责建立设备网络身份、校验连通性，并对照数字孪生（Digital Twin）核实物理硬件配置。 2. **配置拨正（Provisioning）**：消除不同出厂批次的软硬件差异，将每个 GPU 节点的固件、操作系统和内核参数自动化调优至官方认证的最佳基线。 3. **压力老化测试（Burn-in）**：对单节点及其子系统施加严苛的电气与热应力负载，排查隐藏在轻载背后的边缘故障。 4. **集群多节点验证（Cluster Multi-node Validation）**：跨数千颗 GPU 联合预训练前沿模型（Frontier Models），实测多机协同下的真实性能表现，验证集群是否达到预期的“基准良好状态”（Base Known Good）。 --- ### [03:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=185s) 预部署阶段：在首次开机前建立数字身份与硬件校验 预部署阶段始于服务器首次上电开机之前。当全球各数据中心的现场技术人员将物理服务器推入机架、接通电缆的那一刻，预部署自动化流程便立即启动。该阶段由等待、采集与验证等离散而严密的步骤构成： 首先，系统进入“等待”阶段，全面收集供应商数据，包括设备资产编号（Asset ID）、BMC MAC 地址及登录凭证；同时监控网络链路，等待 DHCP 租约下发以便建立带外管理通信。在成功锁定 BMC 的 IP 地址后，管道利用 Redfish 协议穿透底层，枚举并采集服务器内部每一个元器件的具体型号与固件版本。 随后进入关键的交叉比对环节。系统将实时扫描到的物理硬件清单与内部资产系统的“数字孪生”模型进行逐一校验，确认实际硬件与规划采购完全吻合，并验证系统登录链路通畅。整个过程设计了严格的防护卡点（Gatekeeping）：必须通过 10 至 20 项细粒度的健康检查，且状态全部亮起绿灯后，节点才被允许脱离此阶段；一旦任何一项检查异常，系统将自动进入阻断或报修流程。 - 节点状态由四种明确标准管理：通过（Pass）、失败（Fail）、未运行（Not run）以及阻塞（Block）。 - 数据采集依托带外网络及行业标准 Redfish 协议进行深度硬件探查。 - 仅当全部硬件拓扑与数字孪生完全匹配且管理网络连通后，才释放进入软件安装阶段。 --- ### [05:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=305s) 固件与软件配置拨正：无继承、幂等性与零信任 通过预部署的节点随即进入配置拨正（Provisioning）阶段，其核心目标是打造全集群的软硬件一致性。Crusoe 经常同时接收来自多个硬件供应商成千上万台服务器，不同批次的出厂发货时间不同，内置固件往往参差不齐。即使是一台一个月前出货的机器与今天刚出厂的相同型号机器，其出厂固件差异也可能非常显著。如果集群中哪怕只有一台机器运行了错误的网卡（NIC）固件版本，都将直接影响整个分布式训练作业。 为了在异构出厂状态中实现确定性，Crusoe 的拨正系统不依赖任何先验假设，而是基于声明式清单（Declarative Manifest）运作，由三大核心原则驱动： 1. **完全不继承任何旧状态（Inheriting nothing）**：节点通过网络引导进入完全驻留在内存中的临时根文件系统（RAM-based ephemeral OS），确保所有机器处于完全一致且干净的位对齐环境中执行初始化。 2. **幂等且可安全重跑（Safe to rerun）**：为每个服务器型号（SKU）维护一套声明式规范文件，定义该机型所需的确定固件版本、驱动软件与调优参数。拨正流程持续对比实际节点状态与规范，仅对差异项执行幂等写入，随时可以无害重新执行。 3. **永不盲信，严格验证（Verify, don't trust）**：拒绝假设操作指令成功。拨正完成后，必须通过独立的探针重新读取所有组件的版本字符串与特征码，强制与系统唯一的“事实来源”（Source of Truth）进行二次闭环比对。 在具体执行流程中，系统上电后首先通过 PXE 引导进入 Crusoe 专有的临时操作系统环境 **CURE**（Crusoe Ubuntu Runtime Environment）。随后遵循从底层管理层到上层计算层的严格顺序刷写固件：首先通过 Redfish 带外网络更新 BMC 固件，紧接着刷新 BIOS 固件并下发预定义的 BIOS 设置（如特定的风扇曲线策略与能耗模式 profiles）；随后在系统内部依次刷写 GPU 固件、网卡固件、DPU 固件及 NVMe 存储固件。只有在所有固件和底层调优全部达到基线后，系统才会开始安装生产操作系统，将其交付给上层服务。 ```mermaid flowchart LR accTitle: 固件与配置拨正流水线 subgraph Boot["引导准备阶段"] P1["开机与带外初始化"] --> P2["进入 CURE (RAM 临时环境)"] end subgraph OOB["带外固件更新 (Redfish)"] F1["1. BMC 固件"] --> F2["2. BIOS 固件"] --> F3["3. BIOS 能耗/风扇配置"] end subgraph InBand["带内部件固件与系统交付"] I1["4. GPU / NIC / DPU / NVMe 固件"] --> I2["5. 生产操作系统安装交付"] end Boot --> OOB OOB --> InBand ``` --- ### [08:16](https://www.youtube.com/watch?v=nunApaZBtgg&t=496s) 压力老化测试：捕获静默与边缘故障 当所有节点在固件与配置上达到完全统一后，便进入第三阶段——压力老化测试（Burn-in）。如果说前两阶段是硬件的对齐与排齐，那么老化测试就是对机器的“残酷试镜”（Audition），旨在找出那些在行为上偏离标准的异常节点。 在实际运维中，完全损坏（Broken）的硬件往往最容易处理：如果节点无法初始化、上电报错，或者在系统事件日志（SEL）及 NVIDIA XID 报错日志中产生大量严重告警，监控系统可以瞬间发现。真正棘手且致命的是“边缘故障”（Marginal Errors）——这类服务器在初始化枚举时指标完全正常，在轻量负载下也能顺畅运行，但一旦叠加高并发热应力（Thermal Stress）或极端电气负载（Electrical Stress），它们就会偶发性崩溃，或是展现出不可预测的性能抖动与异常图样。 Crusoe 的老化测试策略具有两个显著特征：第一，测试完全在模拟最终客户使用方式的虚拟化环境中运行，确保测试与真实生产环境 100% 同构；第二，测试系统具备自我进化能力，所有测试过程中采集的指标与异常都会回流到测试套件中，不断更新检测算法与阈值。整套老化测试方案包含 30 多项高度定制的测试用例，从单节点基础测试、GPU 专项计算测试，逐步递进升级至整机 Fabric 互联网络通信测试，层层加压筛除隐患。 ### [09:58](https://www.youtube.com/watch?v=nunApaZBtgg&t=598s) 节点与网络老化压测策略：快速失败与硬件全方位检验 在节点基础检查通过后，测试流程会立即转向高强度的压力测试（Stress Tests）与高性能验证。之所以严格分层，是因为基础健全性检查（Sanity Checks）只需数秒即可完成，而深度老化压测（Burn-in Tests）则需要持续数小时；团队遵循“快速失败”（Fail Fast）原则，在投入大量时间成本之前，先以极低代价剔除明显异常的节点。进入节点级压测后，除了覆盖标准的 CPU、内存与存储测试外，GPU 层面则紧密配合芯片原厂的测试套件——针对 NVIDIA GPU 运行不同强度的 DCGM（数据中心 GPU 管理器）诊断套件，针对 AMD GPU 则运行 AMD 专用的 GPU 现场诊断工具，从而将整机与计算核心的负载推向极限。 当单机内部测试达标后，流程会推进到网络互联（Fabric）层面，涵盖单节点网络与跨节点网络测试，不仅支持针对 InfiniBand 架构的 NCCL 基准测试，也支持针对 RoCE 架构的 RCCL 测试。在整个老化测试过程中，团队能够捕捉到各种深层硬件隐患：在存储端，通过对本地与块存储持续压测带宽、IOPS 与延迟，精准捕获性能衰退的 NVMe 固态硬盘；在 GPU 供电端，既会将 GPU 长时间推至 TDP（热设计功耗）极限以观察其能否稳定承载，又会执行“脉冲式功率测试”（Pulsating Power Tests），通过反复拉升与骤降功耗来模拟真实大模型训练时的功耗骤变波动；在网络互联端，团队格外青睐环形拓扑测试（Ring Topology Tests，如三节点环境下的 A-B、B-C、A-C 互联校验），这种方式能快速定位究竟是哪一个具体节点或哪一条物理链路存在潜在瓶颈，从而精准进行单点调试与修复。 ```mermaid flowchart TD accTitle: GPU 节点老化与网络压测流水线 subgraph S1["快速失败阶段 (数秒)"] A["基础健全性检查 (Sanity Checks)"] --> B{"是否有异常?"} B -- "是" --> C["快速剔除/返修"] end subgraph S2["深度单机压测 (数小时)"] B -- "否" --> D["计算与存储测试<br/>(CPU/内存/NVMe IOPS与延迟)"] D --> E["GPU 极限压测<br/>(DCGM / AMD 诊断套件)"] E --> F["电源特性测试<br/>(持续 TDP / 脉冲式功耗波动)"] end subgraph S3["网络 Fabric 验证"] F --> G["多节点通信基准<br/>(NCCL / RCCL)"] G --> H["网络拓扑测试<br/>(InfiniBand / RoCE 环形链路诊断)"] end ``` - **快速失败设计**：轻量级检查仅耗时数秒，耗时数小时的高强度测试排在后段，大幅缩短整体交付周期并最大化检测效率。 - **存储性能基准**：持续监控本地及块存储的带宽、IOPS 与延迟指标，筛除性能降级的 NVMe 固态硬盘。 - **动态功耗测试**：结合持续满载 TDP 压力与脉冲式功耗起伏，提前暴露 GPU 供电异常与电压不稳等潜在隐患。 - **环路网络排障**：借助环形拓扑流量测试，在多节点互联中精准隔离故障网卡、光模块或交换机端口。 --- ### [12:31](https://www.youtube.com/watch?v=nunApaZBtgg&t=751s) 全集群预训练验证与 MFU 达标：排除亚健康节点 单节点及链路验证完成后，系统进入集群级测试（Cluster Testing）阶段。集群测试的核心在于验证由成百上千张显卡组成的庞大集群能否“作为一个整体协同运转”。在此阶段，团队会在整个系统上直接运行全规模的多节点前沿大模型预训练任务，涵盖从 8 卡到 1,000+ GPU（功耗达数兆瓦级别）的不同规模，并在多种计算精度下评估开源前沿大模型的表现。这套测试最核心的衡量指标是 MFU（Model FLOPs Utilization，模型算力利用率），即用户实际支付的算力中有多少真正转化为了有效模型计算；只有当整个集群的 MFU 表现达到严格设定的基准线后，该集群才被允许交付给客户。 以最近一次 512 颗 GPU 的前沿大模型实跑为例，整个集群在运行大规模预训练任务时，绝大部分显卡都表现出整齐划一的运行曲线，犹如一个紧密配合的整体。然而监控数据显示，40 号节点上的 GPU 7 在运行过程中温度始终比其他显卡高出整整 20°C，存在明显的过热隐患。尽管整个作业尚未崩溃，但这颗过热的“亚健康”显卡随时可能导致训练降频甚至中断。团队果断将该节点整体下线并替换为全新节点，随后重新运行完整的预训练测试负载，最终成功达到既定的 MFU 目标。这种在交付前以真实大模型负载进行全量排查的机制，确保了交到客户手中的基础设施具备最高水准的可靠性与性能表现。 - **多兆瓦级真实负载压测**：在 8 至 1,000+ GPU 规模上运行主流开源前沿大模型预训练作业，覆盖各种计算精度。 - **核心评估指标 MFU**：以模型算力利用率为准绳，确保硬件性能无隐性折扣。 - **过热节点实测案例**：在 512 卡集群测试中成功捕获单卡温度异常高出 20°C 的隐蔽故障，通过节点替换消除长尾隐患。 --- ### [14:27](https://www.youtube.com/watch?v=nunApaZBtgg&t=867s) 持续对抗系统熵增与垂直整合交付 从机架上电、固件与软件版本对齐、内核参数调优，到多节点集群级预训练验证，这一整套标准化的交付管线确保了集群交付瞬间的高品质。然而，基础设施本身处于持续的“系统熵增”（System Entropy）之中——上游不断发布新的固件补丁、驱动程序、操作系统补丁以及不同版本的 CUDA 组合，今天交付的集群在几周或几个月后所面临的软硬件生态将大不相同。因此，Crusoe 建立了持续的集群审计机制，不断监控在役算力集群的状态，并与硬件芯片合作伙伴紧密协作，对最新版本的驱动与软件进行持续验证与资格认证，确保集群在全生命周期内始终保持极致性能。 Crusoe 的核心竞争力之一在于其高度垂直整合的业务架构。从绿色能源供给、自建数据中心基础设施，到专为 AI 构建的云平台，所有层级均由同一个高度协同的团队端到端运营。这种全栈贯通的模式消除了传统多供应商之间的推诿扯皮，一旦底层硬件、电力或网络出现任何异常信号，单一责任链能够以最快速度响应并完成全链路调优，确保基础设施整体发挥出最大效能。 - **动态应对系统熵增**：通过持续的舰队审计与最新固件、CUDA 环境认证，保障集群长期运行中的性能不衰退。 - **端到端垂直整合**：覆盖能源供给、数据中心物理设施与云软件栈，具备单点责任归属（Single Layer of Accountability）优势。 --- ### [15:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=955s) NVIDIA 卓越云认证与前沿 AI 原生客户应用 为了对外证明集群的实际性能达到行业最高水平，Crusoe 积极参与了 NVIDIA 的 Exemplar Cloud 验证计划。这是一套面向云服务商的透明性能基准，通过在真实集群上运行 NVIDIA 官方标准训练配方，在不同参数规模和不同主流框架下进行严格比对。测试结果表明，Crusoe 基于 NVIDIA Blackwell B200 等 GPU 舰队构建的集群，完全达到了 NVIDIA 官方的参考基准性能（Reference Performance），使客户在获得算力时无需担心任何隐性性能折损。 这些严苛的质检流程与性能保障，直接体现在了 AI 原生创新企业的实际业务成效中。例如，实时语音 AI 领军企业 Cartisia 利用该云平台训练其旗舰 Sonic 语音大模型；AI 视频生成公司 Linum 依托高性能算力实现了文本到动画电影的高效渲染；Odyssey 则在其上构建高度复杂的大规模物理世界仿真系统。这些尖端负载对算力基础设施的吞吐量与稳定性有着极致要求。Alex Akesson 总结道，Crusoe 的目标从来不是简单交付一块块独立的 GPU，而是打造一套能够像交响乐团般和谐协同运转的统一系统（"Infrastructure That Performs as a Symphony"），为前沿 AI 负载提供无与伦比的性能底座。 - **NVIDIA Exemplar Cloud 认证**：在 B200 等 GPU 舰队上严格复现 NVIDIA 官方参考性能基准。 - **前沿 AI 原生客户赋能**： - **Cartisia**：基于集群训练 Sonic 等超低延迟实时语音生成模型。 - **Linum**：运行文本生成高质量动画电影视频大模型。 - **Odyssey**：依托多节点高吞吐算力进行大规模世界仿真（World Simulation）。 - **核心理念**：超越单卡硬件堆砌，打造由能源、机房、网络到计算高度协同的“交响乐式”AI 基础设施。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=nunApaZBtgg&t=4s) 构建 AI 基础设施绝不是将成堆的硬件部件简单堆叠在一起，而是要让它们像一支出色的“交响乐团”（Symphony）一样高度协同运作。 — Alex Akesson > [00:04](https://www.youtube.com/watch?v=nunApaZBtgg&t=4s) 如果集群中出现哪怕一块性能较慢的 GPU，其他所有 GPU 都会陷入等待，直接拖垮整体效率。 — Alex Akesson > [05:05](https://www.youtube.com/watch?v=nunApaZBtgg&t=305s) 如果集群中哪怕只有一台机器运行了错误的网卡（NIC）固件版本，都将直接影响整个分布式训练作业。 — Alex Akesson > [08:16](https://www.youtube.com/watch?v=nunApaZBtgg&t=496s) 如果说前两阶段是硬件的对齐与排齐，那么老化测试就是对机器的“残酷试镜”（Audition），旨在找出那些在行为上偏离标准的异常节点。 — Alex Akesson > [08:16](https://www.youtube.com/watch?v=nunApaZBtgg&t=496s) 真正棘手且致命的是“边缘故障”（Marginal Errors）——这类服务器在初始化枚举时指标完全正常，在轻量负载下也能顺畅运行，但一旦叠加高并发热应力（Thermal Stress）或极端电气负载（Electrical Stress），它们就会偶发性崩溃，或是展现出不可预测的性能抖动与异常图样。 — Alex Akesson > [09:58](https://www.youtube.com/watch?v=nunApaZBtgg&t=598s) 团队遵循“快速失败”（Fail Fast）原则，在投入大量时间成本之前，先以极低代价剔除明显异常的节点。 — Alex Akesson > [12:31](https://www.youtube.com/watch?v=nunApaZBtgg&t=751s) 集群测试的核心在于验证由成百上千张显卡组成的庞大集群能否“作为一个整体协同运转”。 — Alex Akesson > [12:31](https://www.youtube.com/watch?v=nunApaZBtgg&t=751s) 这套测试最核心的衡量指标是 MFU（Model FLOPs Utilization，模型算力利用率），即用户实际支付的算力中有多少真正转化为了有效模型计算；只有当整个集群的 MFU 表现达到严格设定的基准线后，该集群才被允许交付给客户。 — Alex Akesson > [14:27](https://www.youtube.com/watch?v=nunApaZBtgg&t=867s) 基础设施本身处于持续的“系统熵增”（System Entropy）之中——上游不断发布新的固件补丁、驱动程序、操作系统补丁以及不同版本的 CUDA 组合，今天交付的集群在几周或几个月后所面临的软硬件生态将大不相同。 — Alex Akesson > [15:55](https://www.youtube.com/watch?v=nunApaZBtgg&t=955s) Crusoe 的目标从来不是简单交付一块块独立的 GPU，而是打造一套能够像交响乐团般和谐协同运转的统一系统（"Infrastructure That Performs as a Symphony"），为前沿 AI 负载提供无与伦比的性能底座。 — Alex Akesson ## Mentioned Alex Akesson, Crusoe, Ai4, NVIDIA, NVIDIA GB200, Grace CPU, NVLink, NVLink Switch, InfiniBand, RoCE, BMC, Redfish, CURE (Crusoe Ubuntu Runtime Environment), DCGM, AMD, NCCL, RCCL, NVIDIA Blackwell B200, NVIDIA Exemplar Cloud, Cartisia, Sonic, Linum, Odyssey ## Video description Alex Akesson - Director, Product Management, Crusoe At scale, one bad node can stall an entire training run. From Stack to Symphony takes you inside the journey every GPU node makes from rack to production: how we provision capacity, burn-in test every node, and prove cluster health by pre-training frontier models at scale. You'll leave with a clear framework and proven strategies for building infrastructure that performs as one system, with the reliability and predictability modern AI workloads demand. [Source](https://www.youtube.com/watch?v=nunApaZBtgg)

Source quotes

[0:04] Alex Akesson: 构建 AI 基础设施绝不是将成堆的硬件部件简单堆叠在一起，而是要让它们像一支出色的“交响乐团”（Symphony）一样高度协同运作。 [0:04] Alex Akesson: 如果集群中出现哪怕一块性能较慢的 GPU，其他所有 GPU 都会陷入等待，直接拖垮整体效率。 [5:05] Alex Akesson: 如果集群中哪怕只有一台机器运行了错误的网卡（NIC）固件版本，都将直接影响整个分布式训练作业。 [8:16] Alex Akesson: 如果说前两阶段是硬件的对齐与排齐，那么老化测试就是对机器的“残酷试镜”（Audition），旨在找出那些在行为上偏离标准的异常节点。 [8:16] Alex Akesson: 真正棘手且致命的是“边缘故障”（Marginal Errors）——这类服务器在初始化枚举时指标完全正常，在轻量负载下也能顺畅运行，但一旦叠加高并发热应力（Thermal Stress）或极端电气负载（Electrical Stress），它们就会偶发性崩溃，或是展现出不可预测的性能抖动与异常图样。 [9:58] Alex Akesson: 团队遵循“快速失败”（Fail Fast）原则，在投入大量时间成本之前，先以极低代价剔除明显异常的节点。 [12:31] Alex Akesson: 集群测试的核心在于验证由成百上千张显卡组成的庞大集群能否“作为一个整体协同运转”。 [12:31] Alex Akesson: 这套测试最核心的衡量指标是 MFU（Model FLOPs Utilization，模型算力利用率），即用户实际支付的算力中有多少真正转化为了有效模型计算；只有当整个集群的 MFU 表现达到严格设定的基准线后，该集群才被允许交付给客户。 [14:27] Alex Akesson: 基础设施本身处于持续的“系统熵增”（System Entropy）之中——上游不断发布新的固件补丁、驱动程序、操作系统补丁以及不同版本的 CUDA 组合，今天交付的集群在几周或几个月后所面临的软硬件生态将大不相同。 [15:55] Alex Akesson: Crusoe 的目标从来不是简单交付一块块独立的 GPU，而是打造一套能够像交响乐团般和谐协同运转的统一系统（"Infrastructure That Performs as a Symphony"），为前沿 AI 负载提供无与伦比的性能底座。

Source quotes

[0:04] Alex Akesson: one big symphony [0:04] Goldilocks environment [1:55] known-good baseline [3:05] digital twin [5:05] Inherit Nothing [5:05] Safe to Rerun [5:05] Verify, Don't Trust [8:21] marginal errors [9:58] fail fast [15:55] Alex Akesson: We want to make sure that when you come to us, you get a whole system that works as a symphony. You're not just getting one single GPU... Our goal is really firm. It is to make AI infrastructure that is as performant as possible for our customers.

Original transcript

[0:04] Hey everyone, I'm Alex. I'm from the Cruso team  and I'm here today to talk about what it takes to [0:09] build AI infrastructure to just not act like one  many stack of parts, but instead to be one big [0:15] symphony. We're going to walk through the journey  every GPU makes from the moment it lands in one of [0:20] our racks to the moment it's available for one of  our customers and everything we do along the way [0:25] to make sure that it has the performance and  the reliability that AI native customers demand. [0:31] So if you look at a modern GPU today, it's  made out of many components, right? It's [0:35] dozen of computers working together to  create this larger system and they each [0:39] have their own firmware, their own software,  their own settings, and their own tunings. [0:43] Uh if you look at a current generation Nvidia  GB200 compute tray, you have the four um [0:49] you have the four GPUs, you have the two gray  CPUs, you have all the nicks, you have the DPU, [0:56] you have the NVMEs, and then you have the  baseboard management controller, right? That's [0:59] orchestrating the whole system. And that's just  one of the compute trays. If you look at the whole [1:03] rack, that all gets added up, right? So you have  18 of those trays working with NVLink, NV switches [1:09] to create that larger rack. And you also have all  the cooling units. You have the power supplies. [1:14] So it's hundreds of systems working together in  that rack. And that's at one rack, one of out of [1:19] 16 of those. Once you go beyond that, you're  looking at the whole cluster. So you're going [1:23] up to the Infiniband and the Rocky layer, which  has its own switches, transceivers, and cables. [1:29] All of that needs to be orchestrated as one larger  system. And when you're doing one of these larger [1:34] training runs, they're synchronous. So all  the components are dependent on one another. [1:38] If you have one GPU that's slower than all the  others, they're all waiting up on that one. So, we [1:44] need to make sure that we create this Goldilocks  environment across our cluster that is for peak [1:49] performance where all components are optimized to  give you the best uh performance and reliability. [1:55] So, how do you get all of those components to  work together as one unit? Um, our answer is a [2:01] pipeline that we have built uh and perfected over  time. It's made out of four key layers, right? [2:06] So it starts with pre-eployment. Pre-eployment is  the first step we do when the server is racked in [2:12] one of our data centers and when it goes from  physically installed to ready for software [2:16] provisioning. It's all the steps where we look for  identity. We look for connectivity. We look for is [2:21] the hardware validated on the system. Next we do  provisioning. Provisioning is where we drive every [2:27] node, so every GPU to its optimal state. And then  we have burn-in. Burn-in is where we stress every [2:33] system and subsystem on the server to make sure  and catch any type of issue that could happen. So [2:40] when when when it's under load and then finally  we have cluster multi-node validation and that is [2:45] the final sets of tests we do across a larger  cluster. So we take thousands of these GPUs, [2:51] we pre-train frontier models on them and  we see if they're hit the base known good [2:56] that we want to have for that cluster. So all  of these steps work together and we do them [3:01] so that we can ensure that we can have the  most performant cluster for our customers. [3:05] So if we jump right in to the first step that is  pre-eployment. Pre-eployment is everything we do [3:11] to prepare the node to make we're making you know  to even just provision it right. We want to make [3:16] sure that it's ready for software layering on top.  Um pre-eployment starts before first boot. So when [3:22] one of our site technicians or one of our global  data centers puts the server into the rack that [3:27] is when pre-provisioning pre-eployment starts.  Um you can there's a lot of independent steps [3:32] that need to happen. So the first stage there the  waiting uh the waiting stage is when we collect [3:37] all the vendor data. So all the asset ids from  our servers all the the BMC MAC address all the [3:43] credentials we need to log into that system. It's  also when we wait for the device to be online. [3:48] So the server is in the rack but we want to wait  for that DHCP lease to kick off so we can start [3:53] our uh provisioning flows. We then reserve the BMC  IP. We then what we do through the BMC and Redfish [4:00] is that we collect all the different subcomponents  and components on the server. So we look at at [4:05] what are the different um not only components but  also versions that are existing in that system. [4:10] And then finally we do all a series of validation  checks based on that. So we compare what's on the [4:15] system to what's in our digital twin. Uh we  see is the actual hardware we predicted to be [4:20] on that server actually what's being reflected on  it and then we also make sure can we even log in [4:25] successfully to this server one once that's all  done only then do we move to provisioning this [4:31] this node. Uh and you can see that dashed line  there that means we keep being in that phase [4:36] until everything clears. So until all these 10  or 20 different checks are all check mark green [4:41] only then do we move and that continues  happening. If anything is flagged, [4:45] you know, either automatically or through one of  our technicians, it gets repaired. Um, once that's [4:50] done, we move uh all all those steps kind of  follow four main things. Either they pass all this [4:57] all this pre-eployment steps or  they fail, not run or block. So, [5:00] we have this very simple way to know why any  of those underlying steps haven't proceeded. [5:05] So, now we have gone through pre-eployment which  means the node is ready for software provisioning. [5:11] So, we move to our provisioning phase. uh  provisioning is basically where we ensure [5:16] fleet consistency across all of those those  nodes. So earlier I talked about how there's [5:20] so many different firmware software and different  tunings. So in provisioning we want to bring them [5:25] to one cohesive unit across all of the systems.  Um we get thousands of systems in at a time from [5:32] different hardware vendors and because of that  there's they're always shipping on their own [5:37] schedule. So uh a software that was shipped on the  system a month ago could be very different from [5:42] one that gets shipped on the same system today. So  what we want to do here is bring all the systems [5:47] to that qualified optimal performance baseline.  Um if you have a mismatched in your cluster, say [5:53] one of your nodes is on the wrong nick firmware,  it's going to impact your entire training job. [5:58] So our goal here is to bring that consistency  across everything. Um what the provisioning [6:03] system does is that it's it assumes nothing about  the system and then it uses a declarative manifest [6:09] to go to look at what's on the system, compare  it and then um bring it to the correct um update [6:15] state. Um and then finally we prove that we  actually completed this successfully. So there are [6:21] three main components that make this provisioning  successful. First we have this idea of inheriting [6:27] nothing. So the node boots into a root file system  that lives entirely in RAM. So every node is [6:32] in this bit identical environment. The second  thing it's safe to rerun. So it's always what [6:38] we're doing for every skew is that we have  a corresponding manifest that declares all [6:42] the uh firmware version, software versions  and tuning parameters we need on that system [6:46] and we're constantly looking at what's on the  node, looking what's in the manifest and then [6:50] acting on the difference. So it's very safe to  rerun. And finally, we verify we don't trust. [6:56] So uh there are so many different components and  so many different strings we're trying to compare. [7:00] So what we do in our provisioning system  is that we always go back to what is the [7:04] declared source of truth. One source of truth  for every uh system in our infrastructure. [7:11] If you look at this how it goes end to end you  can start on the left side here with the pre-boot [7:16] setup. This is when we power on the system in  our provision in in our provisioning flow. And [7:21] what we then do is that we boot into cure. Cure  is our Crusu Ubuntu runtime environment. It's our [7:26] ephemeral OS environment. Once we're into that,  we can start doing all the firmware and software [7:31] updates. We start with the firmware updates. At  the top left, you see their BMC firmware. So, [7:35] the baseboard management controller gets updated  first. That's the thing that manages the whole [7:39] system. So, we want to go with that first. Once  we've updated the BMC, we proceed with the BIOS [7:44] firmware. We then go with the BIOS settings. So,  these are like the fan, uh the power profiles on [7:49] the system, and we do all those threes out of band  through Redfish. After that, we move to the GPU [7:54] firmware, the nick firmware, the DPU firmware, and  the NVME firmware. Only once all that is complete [7:59] do we move to the operating system install phases  where we then set up the operating system and get [8:04] the node prepared for our customers. So throughout  this flow, we've been able to take that node [8:09] and bring it to its optimal most performant state.  We have it um on the qualified versions and tuning [8:16] parameters for it. Um, with that being done, we  move to the third phase, which is the burn-in [8:22] phase. This is when we take all of these different  nodes that are now all the same and audition them [8:27] to see if any of those aren't behaving as they  should be. You know, burn-in has there's so much [8:33] you can test in burn-in and we're there's easy  things you can catch, right? So, broken is really [8:37] easy to catch. the if it if if the node fails to  initialize or it has errors or if there's a lot [8:42] of events in uh if you look at the critical event  logs like the system event logs or the x ids you [8:47] can see if the node isn't healthy. So what we're  really focusing more on in burnin is to find those [8:52] marginal errors. So these are nodes that enumerate  clean they pass under light load but as soon as [8:57] you start putting more stress you know either  thermal or electrical on it they start failing [9:02] or displaying strange patterns. So that's really  what we want to catch during our burn-in phases. [9:08] Something that we also want to make sure to do is  that we test it in a virtualized environment. We [9:11] test it in the end user in the same environment  our end users would have. Um and the second thing [9:17] that we do is that we're always learning from  all that these tests that we do. Uh everything [9:21] is feeding back into our suite. We're constantly  adding new tests or changing the thresholds. [9:26] So we want to learn and improve over time  as systems and system behaviors change. [9:32] Looking at burn-in there's over 30 tests that we  do on our systems uh where we stress them in many [9:37] different ways. At the bottom layer you have node  tests then you have GPU tests and then you have [9:42] fabric tests. So it's kind of following the same  system that I mentioned earlier of how you go from [9:47] the specific node to the fabric as a whole. Um  another thing we do with tests is that we look [9:52] at doing them in order series of escalation.  So we start with simple uh sanity checks [9:58] and [snorts] then we move to more stress tests  and high performance tests. The reason for that is [10:02] that those sanity checks take seconds to run. The  more intense ones take multiple hours. So we want [10:08] to fail fast and see if a node displays anything  wrong before we invest in much longer tests. [10:13] That allows us to be as efficient as possible in  running these tests. So getting the nodes to our [10:18] customers as quickly as possible while ensuring  we have the most performance system and we catch [10:23] any possible strange um signals that we see on  those nodes. Um so at the bottom on the node [10:28] test you see the standard ones. We do a bunch of  CPU memory storage tests. And then on the GPU side [10:34] we partner very closely with Nvidia and AMD and  run their test suites. So from Nvidia it's a DCGM [10:40] suite that runs at varying different intensities  and really stresses the system and the GPUs. [10:45] And then for AMD, we run the AMD uh GPU field uh  uh diagnostic test as well. Uh both of those um [10:52] make sure that we stress the GPUs and the system  as a whole. Finally, once we've completed all [10:56] those tests, do we move to the fabric and at  the fabric level, we test both single node tests [11:02] and then we move to multi-node tests as  well. Um multi-node tests both being nickel [11:08] and rickle. So we support both the infidaban type  of cluster and rocky type clusters as well. Um [11:14] once we have completed all of those tests,  we get the node ready for the next step. [11:20] And there's so many different things we can  catch during those uh during these types of [11:24] burn-in. For example, on the node side, we can  catch degraded NVMe. So what we do is that uh [11:30] we keep through and we we benchmark bandwidth,  IOPS, and latency on both local and block storage. [11:35] And that allows us to catch those types of issues.  Uh on the GPU side, something that's really cool [11:40] is that we look at how power can impact GPU  performance. So we will push the GPUs to the [11:45] uh TDP power for a sustained period of time and we  see if they're able to hold or display any strange [11:50] behaviors. We also do pulsating power tests where  we repeat repeatedly go up and down in power and [11:56] that kind of mimics how a training job would look  like on that uh on that GPU and allows us to catch [12:02] uh any issues that that you might face on  it. And then finally on the fabric side um [12:06] what we really enjoy doing there are these ring  topology type tests. So say you have three nodes [12:12] you test A to B, B to C and then A to C. And  that can help you identify uh which node and [12:18] which link is potentially causing an issue. And  then it can help you either debug a specific node [12:23] or it can help you debug a specific part of your  fabric. Um if we keep going uh to the next slide [12:31] we can talk about the final step which is after  we complete burn-in we move to cluster testing [12:37] and cluster testing is everything we do on the  entire cluster. So we've now validated that all [12:43] the nodes are completely and working correctly.  So now we want to make sure how does that system [12:47] work as a whole. So in cluster testing what we do  is that we run full-scale multi-node pre-training [12:55] workloads across all of our systems. Um we run  it at varying different scales. So we do it from [13:01] 8 to a,000 plus GPUs. A thousand plus that's  multimegawatt type of workloads. Um and we do [13:07] it at different precision levels. Um while we're  doing this we're able to uh test all the larger [13:13] all the largest uh open source frontier models.  And what we're really going for is seeing the MFU. [13:19] So what is the model flops utilization rate? How  much of the compute you're paying for is actually [13:24] going to your model. So we're always measuring  that and we're making sure that we're hitting [13:28] um the scores that we want for that cluster. Only  then do we give that cluster to our customers. [13:33] And you can kind of see that here um how on a  recent run that we did uh this is one 512 GPUs [13:40] that we were running through one of those large  frontier models. And what you can see here is that [13:44] all the GPUs are acting as one cohesive system.  But on the blue line there on top, you see one of [13:49] one of the GPUs, GPU 7 uh on I think it's on on  node 40 has that 20° up every single time. So it's [13:58] it's overheating compared to all the other GPU.  So that's telling us something's off in that GPU. [14:03] So what we did is that we took that GPU and that  node out. We then basically we then replaced it [14:10] with a new node. We then reran the whole  pre-training workload and we were able to [14:15] hit our MFU scores. So by catching that we were  able to hand over to our customers a cluster [14:20] that was as performant as possible and that show  just shows how important each of those steps are. [14:27] Now if you look at the handoff standard as a  whole you can see that there's many steps uh that [14:33] that we've walked through throughout throughout  this throughout this pipeline. So we started off [14:37] with making sure the node power is on. We talked  about everything we need to do to make sure the [14:41] firmware, software, and tunings match. And then  we talked about um all the different multi-node [14:46] cluster testings we do. Only once that's all  done can we say that the cluster is ready for [14:50] our customers. And what's also important to  note is that that is at one point in time, [14:55] right? So that cluster uh is just there when  we hand it off. But we have constant entropy in [15:00] our infrastructure systems, right? New firmware,  new software, new CUDA combinations get released. [15:04] So the node that's the the cluster that's shipped  today it's going to look very different uh weeks [15:09] and months from now. So what we do is that we're  constantly auditing our fleet making sure that [15:13] our customers are on the most performance setup  and then we work really closely with our hardware [15:17] partners to um release and qualify the latest  versions. So our goal here is to make sure that um [15:25] the the cluster stays fast from the beginning and  throughout its lifetime. Um as a company Crusoe is [15:32] vertically integrated. So this allows us to really  be intentional at each layer. So from um our [15:38] cloud to our data centers to our energy, it's one  cohesive organization that runs it all. And that's [15:43] that makes sure that if we see any issue u there's  one layer of accountability to make sure that we [15:48] get those clusters as performant as possible.  Now that we've completed all of those steps, it's [15:55] you know how does this all work in production? And  um we want to then then make sure that this is all [16:00] behaving in a great way. So one of the things  Cruso does is that we participate in Nvidia's [16:04] exemplar cloud validation. This is Nvidia's um  kind of um transparent benchmarking standard [16:10] for clouds. So what what they do is that they've  set up um real training workloads that you run [16:16] against Nvidia's um uh basically Nvidia's recipes.  So we do this across large frontier models. We [16:23] do varying parameters and different frameworks.  And what we've shown as a cloud is that we're [16:27] able to hit the NVIDIA reference performance. So  because of all the steps we do throughout this [16:33] throughout this journey to make sure that  the cluster is as performant as possible, [16:36] we can rest assured that our customers are getting  the exact reference performance on the clusters [16:41] that Nvidia is looking for. And we did this  on our Nvidia uh B200 uh fleet of GPUs. [16:48] Another thing that we also helped us validate  that all this work really matters is just seeing [16:53] the amount of work that all of our uh customers  do on Cruso cloud. So we have lots of different [16:58] AI natives that are using uh these clusters to  do great things. We have Cartisia that does VO [17:04] voice AI. So um they do real-time voice and they  have uh their sonic model. We have Linum that does [17:10] video generation. So text to video so anyone  can make an animated film. And we have Odyssey [17:16] that does its world simulation. All these  customers really rely on the performance and [17:20] the reliability that Cruso cloud delivers. And  that's why we really are intentional about all [17:25] those steps we do in our process to hand over  the best GPUs to them. And and that really is [17:31] why we invest so much in this process at Cru. So  we want to make sure that when you come to us, [17:36] you get a whole system that works as a symfony.  You're not just getting one single GPU, [17:41] right? Uh our goal is really firm. It is to  make AI infrastructure that is as performant [17:47] as possible uh for our customers. Um and with  that, thank you all for your time. [applause]