Sail Targets 10x AI Cost Cuts by Trading GPU Speed for Agent Workloads
Colossus | Investing & Business Podcasts
As long-running background agents replace interactive chatbots, inference providers can slash token costs orders of magnitude by trading low latency for scavenged power, unoptimized silicon, and saturated batch throughput.
## Background
Published on August 25, 2026, by Colossus on the podcast *Invest Like the Best* (EP.488), this episode features host Patrick O'Shaughnessy in conversation with Neil Movva, founder of Sail. According to the show description, the discussion explores how Sail is building a 'token factory' optimized for cost rather than speed to serve long-running, background AI agents across the entire stack of software, chips, and power.
## TL;DR
Neil Movva breaks down the architectural and economic playbook for driving AI inference costs down by 10x to serve autonomous, long-horizon agents that prioritize batch throughput over interactive low latency. He provides a deep technical tour of silicon trade-offs—such as SRAM versus DRAM memory bandwidth, the KV cache bottleneck in transformer attention, and custom kernel optimization across Nvidia and alternative chips. Movva also outlines Sail's scavenger infrastructure model, which aggregates discarded, intermittent power drops and heterogeneous hardware with fault-tolerant software to profitably undercut frontier lab pricing.
## Atmosphere & style
The dialogue is deeply technical, fast-paced, and grounded in operational realism, marked by Movva's infectious enthusiasm and granular command of hardware physics. Rather than indulging in abstract AI hype, the conversation feels like a masterclass in industrial systems engineering, blending sharp supply-side contrarianism with whiteboard-level silicon analysis.
## Key takeaways
- `02:20` Sail operates as a 'token factory' optimized specifically for ultra-low-cost inference, targeting autonomous AI background agents where low latency is secondary to cheap compute.
- `05:32` The market demand for AI tokens is shifting from human-interactive low-latency tasks to virtually unbounded long-running background workloads like deep web research and cybersecurity red-teaming.
- `10:00` Cybersecurity and verifiable domains act as a software 'proof of work' where scaling reasoning compute and token volumes directly correlates with discovering system vulnerabilities and solving complex problems.
- `20:03` Inference efficiency is fundamentally split between low-latency demands requiring costly NVLink multi-GPU setups and throughput-oriented workloads that bypass interconnect bottlenecks via pipeline or expert parallelism.
- `24:58` Hardware architectures present severe memory trade-offs: fast on-die SRAM achieves extreme memory bandwidth but lacks capacity, while high-capacity HBM/DRAM suffers from strict memory bandwidth limitations.
- `31:38` The fundamental tension in Transformers stems from combining memory-bound attention layers with compute-bound MLP layers, driving a future need for hybrid chip architectures.
- `36:43` Frontier AI lab focus is pivoting from static human text data toward reinforcement learning (RL) gyms and execution sandboxes that deliver verifiable automated feedback.
- `40:00` Manual low-level GPU kernel development is evolving into high-level whiteboard architecture design, with frontier AI models handling the direct code generation.
- `45:00` Under a 'no bad chips, only bad pricing' philosophy, unoptimized or alternative silicon creates significant margin opportunities for infrastructure providers with custom software compilation capabilities.
- `50:00` Unlike training, which requires giant co-located clusters, inference compute can be decentralized across lean, low-cost distributed sites operating on cheap, scavenged power drops with minor uptime trade-offs.
- `01:01:04` The primary algorithmic inefficiency in current models lives in the Key-Value (KV) cache, where raw memory consumption vastly exceeds actual token entropy requirements.
- `01:06:08` The premium frontier labs charge for a 3-to-6-month model lead is fragile, as public code generation continuously distills flagship capabilities down to open-source models.
Patrick O'Shaughnessy introduces Neil Movva, the founder of Sail, framing the conversation around a fundamental question: what happens when artificial intelligence is optimized for cost rather than speed? Sail operates as what Movva terms a "token factory"—an inference provider specifically engineered for a future where autonomous agents run persistently in the background for hours, days, or weeks at a time, rather than answering a human waiting on a screen in real time.
The episode serves as an end-to-end architectural tour through what O'Shaughnessy calls the "full stack of intelligence," spanning the software layer, specialized silicon, data center design, and raw electrical power. Movva’s core thesis is contrarian in an ecosystem currently fixated on delivering sub-second response times: because background agents do not require instant feedback, latency can be traded away to radically compress the cost of computing a token. By stripping down inference systems and executing a scavenger strategy for undervalued hardware and power, Sail aims to make intelligence cheap enough to become an ubiquitous commodity.
`03:22` Building a Token Factory for Autonomous Workloads
Movva describes Sail's core offering as a lean token factory designed to serve open-source large language models through an API at prices that undercut standard market rates. Alongside raw model serving, Sail hosts "Sailboxes"—dedicated cloud-based virtual machines engineered to maintain persistent, long-running agent environments that execute uninterrupted over days or weeks. Movva anchors the mission of the company in the economics of abundance: historically, whenever a foundational technology or resource becomes ten times cheaper, it ceases to be an incremental optimization and instead unlocks an entirely new product category.
```mermaid
flowchart LR
accTitle: Transition from Interactive Tokens to Asynchronous Outcomes
subgraph Interactive["Interactive Paradigm"]
A["Human Prompt"] --> B["Low-Latency GPU Serving"]
B --> C["Real-Time Token Response"]
end
subgraph Background["Agentic Token Factory"]
D["High-Level Task Mandate"] --> E["Sailbox Environment"]
E --> F["Cost-Optimized Batch Inference"]
F --> G["Multi-Hour Autonomous Execution"]
G --> H["Finished Work / Deliverable"]
end
```
While token cost serves as Sail’s operational North Star today, Movva notes that tokens themselves are merely a temporary accounting metric for work. The market is already shifting away from human-capped token budgets toward outcome-driven execution. In this evolving paradigm, autonomous agents self-administer their own computational allowances—dynamically determining how many reasoning steps, tool calls, and trial-and-error attempts are required to achieve a verified outcome.
- **The Abundance Thesis**: Lowering inference costs by an order of magnitude fundamentally transforms language models from metered productivity software into a universal industrial input.
- **Sailboxes**: Virtual machines optimized for persistence, allowing agents to maintain operational memory and tool access over extended multi-day horizons.
- **Dynamic Reasoning Budgets**: A shift away from rigid enterprise quotas on tokens toward letting autonomous systems take as many independent "shots on goal" as a given task demands.
`05:32` The Shift from Low Latency to Long-Horizon Agents
When evaluating why the broader market has overlooked this structural opening, Movva highlights two critical tailwinds. The first is enterprise demand for open-source model sovereignty. Organizations increasingly refuse to tether their core operations to proprietary black-box APIs, preferring open-weight models where weights are fully owned, customizable, and immune to sudden platform deprecation. The second is an industry-wide over-indexing on low latency. Competing inference providers—such as Baseten, Fireworks AI, and Together AI—built their architectures around high-throughput, low-latency serving to satisfy interactive coding tools like Cursor, where millisecond response times directly dictate the user experience.
Movva argues that human interactive speed is ultimately a bottleneck. As long as a human sits in the loop waiting for an answer, the total ceiling of work is restricted by the human's attention span and cadence. Instead, Movva advocates that true leverage emerges when agents operate on human organizational timescales. Rather than micro-managing an agent every five minutes, managers assign high-level goals and review output on daily or weekly cycles. In this framework, "the best latency is no latency at all"—the computation executes overnight while the user sleeps, making interactive tokens-per-second irrelevant compared to raw compute efficiency. This operational shift is reinforced by the validation of test-time compute scaling: spending more compute and extended time during inference allows models to think through edge cases and deliver vastly superior answers.
`07:45` Unbounded Demand: Deep Research and Cybersecurity
Because autonomous background tasks do not depend on human attention, Movva views the potential volume of background inference as effectively unbounded. A software engineer at a desk can only consume a finite number of real-time tokens using interactive tools like Codex or Claude Code. By contrast, background workloads operate without cognitive ceilings. Movva projects that while background workloads might end the current year splitting inference traffic 50/50 with real-time tasks, long-running agent workloads will ultimately capture 90% of total inference market share.
Movva points to two standout categories driving massive background compute consumption:
- **Deep Exhaustive Research**: Moving beyond skimming 10 or 100 documents to systematically processing tens of thousands of sources. For instance, companies like Parallel Web Systems monitor internet-scale updates in real time to build authoritative knowledge indexes—an exabyte-scale challenge requiring massive, low-cost analytical throughput.
- **Asymmetric Cybersecurity Red-Teaming**: Writing code is linear, but discovering vulnerabilities is combinatorial. Specialized background agents run continuous penetration testing across every repository, proactively scanning for memory errors, business logic flaws, and network vulnerabilities before malicious actors can exploit them.
`10:00` Cybersecurity as "Proof of Work" and the Jagged Frontier
When deploying models for automated cybersecurity, the strategy shifts from having a model read source code once to spinning up active environments where autonomous agents continuously penetration-test the applications. Within the security community, this dynamic has sparked an inside joke: software security has effectively turned into "proof of work." A company's security posture is increasingly measured by how many dollars they spend running Anthropic's APIs against their own infrastructure trying to break in, simply because frontier models represent the most potent offensive tools available.
```mermaid
flowchart TD
accTitle: Autonomous Security and Verifiable Problem Solving
subgraph OffensiveTesting["Autonomous Pen-Testing"]
A["Target Application / Environment"] --> B["Diverse Model Sampling"]
B --> C["Claude Haiku & Smaller Models"]
B --> D["Frontier Flagship Models"]
C & D --> E["Discovered Vulnerabilities"]
E --> F["Automated Patches"]
end
subgraph EconomicUnlock["Cost Trajectory for Verifiable Tasks"]
G["Historical Cost: Millions"] --> H["Current Cost: Thousands"]
H --> I["Target Cost: $10s to $100s"]
end
```
At the same time, the frontier of model capability in vulnerability research is notoriously jagged. Larger flagship models do not simply capture a clean superset of all software bugs; smaller, faster models like Claude Haiku will uncover specific vulnerabilities that giant models completely overlook, and vice versa. This reality necessitates a broad, diverse sampling strategy where developer teams run swarms of heterogeneous models against target systems to uncover edge cases and patch them before malicious actors can exploit them.
- **Security as economic proof:** Robustness is directly proportional to the compute and API capital spent autonomously attacking the target system.
- **Jagged intelligence:** Bug discovery is non-linear across model scales, making diverse multi-model ensembles superior to relying on a single frontier model.
---
`11:20` Proactive Personal Agents and Spending "Speculative Tokens"
Looking beyond enterprise workflows into speculative consumer applications, radical cost reductions in inference unlock entirely new product categories. The most compelling consumer unlock is the shift from reactive tools to continuously running, proactive intelligent agents. Today's digital assistants, like Siri, operate as isolated point solutions requiring explicit human prompting because compute is too precious to waste. With abundant, low-cost inference, an agent could run constantly in the background, reading every email and text message received throughout the day to build a comprehensive, encyclopedic understanding of a user’s life.
If the underlying system is reliable and provably private, the assistant can model user habits so deeply that it anticipates actions before they happen. Opening a smartphone could immediately surface the exact next task or context needed. Achieving this vision requires a fundamental economic shift: developers and platforms must be willing to spend tokens speculatively without any guaranteed return on investment.
- **Continuous context:** Background agents ingest daily communications to maintain an active model of personal context.
- **Speculative execution:** The core economic barrier to ambient intelligence is the cost of running inference cycles that might produce no immediate user-facing output.
---
`13:15` Solving Verifiable Problems vs. Human Taste
On a macro level, long-running agentic intelligence applies directly to any problem that is verifiable. This includes software engineering, formal mathematical proofs, and scientific research. Historically, tackling long-horizon verifiable problems carried an astronomical hidden compute cost. That cost is rapidly compressing: complex research tasks that once required millions of dollars in compute are entering the thousands, with the near-term potential to drop into the hundreds or tens of dollars for definitive answers to scientific questions.
Once reasoning costs collapse, the bottleneck shifts entirely to human curiosity and question formulation. Models will autonomously ingest a high-level query, construct hypotheses, and chase down every branch of follow-up inquiry on their own, bounded only by the allocated token budget. However, this dynamic applies strictly to quantitative, verifiable domains. Neil distinguishes verifiable problem-solving from tasks rooted in "human taste" — such as artistic beauty or stylistic quality in writing — emphasizing that Sail focuses entirely on driving down costs for objective, verifiable computation while leaving taste to humans.
- **Cost compression:** Long-horizon scientific and computational inquiry is shifting from millions of dollars down to tens of dollars.
- **Verifiable vs. non-verifiable:** Objective reasoning tasks scale with token volume, whereas human aesthetic judgment remains distinct and non-algorithmic.
---
`15:10` Introducing the Intelligence Stack: Software, Chips, and Power
To enable a future of unbounded speculative compute and autonomous scientific discovery, Sail is designed as an industrial-scale "token factory" engineered specifically to supply the lowest-cost inference possible. Achieving this order-of-magnitude cost reduction requires re-architecting the entire vertical stack of computing infrastructure. The technical approach bridges three distinct layers: specialized inference software orchestration, semiconductor hardware selection, and physical power procurement at the datacenter level.
`15:00` The Software Foundation and the Genesis of Tensor Cores
Sail's master plan begins strictly at the software layer, targeting how today's data centers can dramatically improve efficiency. Because modern AI infrastructure relies heavily on NVIDIA GPUs, the initial goal was building an entire large language model software stack optimized purely for peak GPU efficiency—squeezing more tokens out of each chip than anyone else in the world. This approach is rooted in the lowest level of programming: GPU kernels. Neil Movva spent his early professional career working on GPUs and low-level kernels, landing his first role at NVIDIA while still in college, right as the concept of the Tensor Core was fighting to justify its existence on silicon.
A Tensor Core is a dedicated, specialized hardware unit on the GPU built specifically to accelerate matrix multiplication. While there may not be a single divine law explaining why matrix multiplication became the atomic unit of modern computation, it serves as an exceptionally succinct way to combine two blocks of numbers and allow them to interact, making linear algebra an extraordinarily compact representation for arbitrary relationships in data. Historically a dominant gaming graphics company, NVIDIA began exploring skunkworks projects in the mid-2010s to adapt graphics processors for machine learning. Engineers and managers attended small academic conferences like ICML and NeurIPS, jotting down notes in lab notebooks about how deep learning researchers were repurposing consumer gaming GPUs to train large models.
By 2015 and 2016, Jensen Huang developed the conviction that machine learning workloads would explode, deciding to commit precious silicon die area to hardware acceleration for AI. In any semiconductor company, asking for silicon real estate means competing directly against existing core business units—in this case, the graphics teams who guarded die area fiercely. Machine learning advocates had to fight tooth and nail to secure just 5% to 10% of the die area on early generations to accelerate convolutions, the foundational operation for computer vision models at the time.
```mermaid
flowchart TD
accTitle: Evolution of GPU Silicon Allocation
subgraph EarlyEra["Mid-2010s: Gaming GPU"]
G1["Graphics Pipeline & Shaders (90-95% Die Area)"]
G2["Experimental ML Silicon / Convolutions (5-10% Die Area)"]
end
subgraph Transition["Jensen Huang's Conviction (2015-2016)"]
T1["Academic ML traction at ICML / NeurIPS"]
T2["Dedicated Matrix Multiplication Units"]
end
subgraph ModernEra["Modern Architecture"]
M1["Dedicated Tensor Cores"]
M2["Peak GPU Throughput Engine for LLMs"]
end
`17:35` Chasing the "Speed of Light" and NVIDIA's Frugal Culture
Working on NVIDIA's low-level software optimization team shaped Movva's engineering philosophy through an internal benchmark known as the "speed of light." In NVIDIA parlance, the speed of light represents the absolute theoretical edge of physical possibility for a piece of hardware. If a chip is designed to run at a specific clock frequency and execute a certain number of multiply-accumulate operations per cycle, software engineers are expected to eliminate every software bottleneck until the chip hits that theoretical ceiling. Movva carried this exact standard over to Sail, demanding that his team focus strictly on reaching 100% of the hardware's absolute physical potential rather than settling for relative performance leads over competitors.
Beyond the technical rigor, NVIDIA's corporate culture left a lasting impression through its exceptional engineering retention and deep-seated frugality. Many of the silicon engineers who joined in 2015 and 2016 remain at the company today, dedicating their entire careers to parallel computing architectures. Alongside this technical passion was an intense operational frugality dating back to corporate cutbacks following the 2008 financial crisis. While Silicon Valley became known for lavish perks, NVIDIA eliminated free lunches and did not even supply free milk in communal breakroom refrigerators. Employees who wanted milk for their morning coffee had to join an internal "milk club," chipping in one dollar a month so a designated organizer could buy jugs of Costco milk for the fridge—a level of discipline Movva remembers distinctly, even if Sail chooses not to replicate the milk club.
- **NVIDIA Retention:** High proportion of long-tenured silicon architects who view parallel computing as their lifelong mission.
- **The "Speed of Light" Metric:** An uncompromising engineering mandate to maximize raw hardware throughput to the theoretical mathematical ceiling rather than benchmarking against external market rivals.
- **Post-2008 Frugality:** The legendary internal $1/month employee-funded Costco "milk club" exemplified NVIDIA's disciplined operational expenditure.
`20:03` Throughput Machines vs. Interactive Chatbots
A fundamental tension exists between how GPUs are physically designed and how consumer AI applications currently utilize them. At its architectural core, a GPU is a throughput engine designed to process massive streams of parallel data simultaneously. It achieves maximum efficiency and optimal power utilization when kept completely saturated with heavy, batched computational workloads.
However, the AI boom over the past several years has steered industry infrastructure toward conversational, interactive chatbots. Serving interactive humans at keyboards forces the hardware into an ultra-low-latency paradigm where individual tokens must be returned as fast as possible in real time. This requirement fragments compute batches and prevents the underlying silicon from running at its natural, peak-throughput efficiency.
Here is a detailed, faithful walkthrough of the provided transcript section.
---
`20:00` Throughput Versus Latency: The Unbreakable Hardware Trade-off
Neil Movva breaks down the fundamental architectural tension inside GPUs between optimizing for low **latency** (spitting out single tokens to a human as fast as possible) versus high **throughput** (processing vast amounts of total tokens across long-running background tasks). Because early AI applications were entirely chatbot-centric, the entire industry chose to optimize hardware and software stacks for low latency. However, Movva argues that as AI shifts toward proactive, background agents, throughput-oriented architectures will dominate.
```mermaid
flowchart LR
accTitle: Trade-off between Latency and Throughput on GPUs
subgraph LAT["Latency Optimization (Chatbots)"]
direction TB
L1["Small Batches / Single User"] --> L2["Tensor Parallelism via NVLink"]
L2 --> L3["Low Latency / High Hardware Cost"]
end
subgraph THRU["Throughput Optimization (Agents)"]
direction TB
T1["Large Batches / Multiple Users"] --> T2["Pipeline / Expert Parallelism"]
T2 --> T3["High FLOPS / Low Cost per Token"]
end
```
He explains that this trade-off is foundational to computer science: you can choose to be "narrow and fast" or "wide and slow." To keep a GPU's compute units fully saturated, you must group work together into large batches. However, running a large batch means each individual request stays on the chip longer, taking a slower path as it shares processing cycles with other users' traffic.
To illustrate this, Movva offers a transportation analogy:
* **Latency Optimization (Private Car):** Takes a direct, targeted path as the crow flies from point A to point B to get an individual downtown as quickly as possible.
* **Throughput Optimization (City Bus):** Carries many people at once. It stops, waits for passengers to hop on and off, and takes a slower overall route, but moves far more total people per hour per dollar.
To achieve maximum low latency, NVIDIA relies heavily on **NVLink**—an ultra-fast interconnect that allows a single large matrix multiplication to be split (sharded) across up to eight GPUs simultaneously (**tensor parallelism**). While this cuts latency by letting each GPU execute one-eighth of the work, it suffers from sublinear scaling due to communication overhead. Using eight times the hardware might only yield a 4x or 5x speedup.
Because Movva's company, Sail, focuses strictly on throughput, they skip aggressive tensor parallelism and NVLink dependencies in favor of alternative approaches like **expert parallelism** or **pipeline parallelism**, where communication delays can be overlapped and hidden behind massive compute workloads.
- **Core Metric Shift:** Chatbots require low latency (fast time-to-first-token and high human reading speeds); background agents require high throughput (maximum total FLOP utilization per dollar).
- **Sublinear Scaling:** Sharding matrix operations across 8 GPUs via NVLink yields ~4x–5x speedups rather than 8x due to tile-size efficiency drops and inter-GPU communication overhead.
- **NVIDIA’s Moat:** NVLink is mandatory for ultra-low latency inference, but less critical for throughput-focused, bus-style batching strategies.
---
`24:58` The Future of AI Chips: SRAM vs. DRAM Memory Architecture
Movva discusses alternative hardware architectures, comparing traditional HBM/DRAM-equipped GPUs against ultra-fast, SRAM-centric chips designed by specialized startups like **Cerebras** and **Groq**.
To explain why companies make different hardware choices, Movva contrasts the two fundamental memory technologies:
1. **SRAM (Static RAM):** Built using standard transistor configurations (typically a **6T transistor cell**) directly on the logic die. It holds its bit state stably as long as power is applied without needing active refreshing. Because it sits directly next to the arithmetic logic units (ALUs), it delivers staggering bandwidths. However, it takes up massive amounts of physical silicon area. An entire top-tier $800\text{ mm}^2$ NVIDIA Blackwell logic die made entirely of SRAM would only contain single-digit gigabytes of storage.
2. **DRAM / HBM (Dynamic RAM):** Manufactured on entirely different process lines by specialized vendors like Micron, SK Hynix, and Samsung. DRAM stores bits as electrical charges on microscopic capacitors. Because these charges naturally leak, a memory controller must actively read and refresh every single bit roughly every 50 milliseconds—"constantly juggling billions of balls in the air." In exchange for this complexity, DRAM delivers three orders of magnitude higher storage density.
```mermaid
flowchart TD
accTitle: Comparison of SRAM and DRAM Memory Architectures
subgraph SRAM_BOX["SRAM Architecture (Cerebras / Groq)"]
direction TB
S1["6T Transistor Cells on Logic Die"] --> S2["Extreme Bandwidth: ~21 PB/s"]
S2 --> S3["Low Density: ~50 GB per Wafer"]
end
subgraph DRAM_BOX["DRAM / HBM Architecture (NVIDIA Blackwell)"]
direction TB
D1["Capacitor Charge Storage (50ms Refresh Cycle)"] --> D2["Moderate Bandwidth: ~10 TB/s"]
D2 --> D3["High Density: 288 GB HBM3e"]
end
```
Movva evaluates **Cerebras Systems** and their Wafer-Scale Engine 3 (WSE-3). Instead of cutting silicon into individual dies constrained by TSMC’s $\sim 800\text{ mm}^2$ reticle limit, Cerebras leaves an entire silicon wafer intact, bridging across the scribe lines between dies to create a single massive chip.
By packing around 50 gigabytes of SRAM per wafer—and stitching multiple wafers together into pipeline clusters—Cerebras achieves up to a terabyte of memory operating at an astounding **21 petabytes per second** of bandwidth (compared to $\sim 10\text{ terabytes per second}$ for HBM on an NVIDIA Blackwell). This enables them to pass full parameter sets of massive models (like Kimi) through the compute cores in roughly a millisecond, driving throughput speeds upwards of **1,000 tokens per second**.
Despite these impressive stats, Movva highlights a critical bottleneck for pure SRAM architectures: the **KV Cache** (Key-Value Cache). Every token processed in an ongoing conversation remains in the context window so the attention mechanism can reference past history. As conversations scale to 100,000+ tokens across thousands of simultaneous users, the KV Cache grows dynamically and unpredictably, eventually overwhelming SRAM's limited capacity. He predicts the market will ultimately move toward hybrid approaches that pair SRAM's extreme processing bandwidth with DRAM/HBM's massive capacity.
- **NVIDIA Blackwell Specs:** ~500 MB on-die SRAM alongside 288 GB stacked HBM3e capacity; ~10 TB/s memory bandwidth.
- **Cerebras WSE-3 Specs:** ~50 GB SRAM per whole-wafer unit; ~21 PB/s memory bandwidth; enables speeds reaching ~1,000 tokens/sec.
- **SRAM Mechanics:** Uses a 6-transistor (6T) cell design directly on logic silicon; fast but area-expensive.
- **DRAM Mechanics:** Uses capacitor-based storage requiring refresh cycles every ~50 ms; manufactured off-die by SK Hynix, Samsung, and Micron.
---
<ElicitationsGroup message="To explore these technical topics further:">
<Elicitation label="Compare Tensor, Pipeline, and Expert Parallelism" query="Explain the differences between Tensor Parallelism, Pipeline Parallelism, and Expert Parallelism in distributed AI inference."/>
<Elicitation label="Analyze the economics of SRAM vs. HBM for LLMs" query="Analyze the cost and memory architecture trade-offs between SRAM-based wafer-scale chips and HBM-based GPUs for deep learning models."/>
</ElicitationsGroup>
`30:00` The KV Cache Dilemma and Long-Context Limitations
Every token processed through a large language model contributes to the key-value (KV) cache, an ever-expanding structure that frequently ends up requiring far more memory than the static weights of the model itself. The static model weights represent "crystallized knowledge," while the KV cache holds the "dynamic knowledge" of the specific ongoing conversation. Although the KV cache preserves an exact mathematical record of preceding context, conversational performance often degrades over lengthy interactions. This happens because models are fundamentally trained on shorter sequences—typically 8,000 to 16,000 tokens—with only sparse training at outer extremes like 200,000 tokens. Maintaining consistent intelligence across expanding context windows remains an ongoing technical struggle for frontier labs.
```mermaid
flowchart TD
accTitle: Transformer Memory and Compute Balance
subgraph AttentionStage["Attention Layer (Dynamic Context)"]
A["KV Cache: Exact Conversation History"] --> B["Memory-Bound Execution"]
B --> C["Needs High-Capacity Off-Chip Memory (GPU HBM)"]
end
subgraph MLPStage["MLP Layer (Crystallized Knowledge)"]
D["Model Weights: Static World Knowledge"] --> E["Compute-Bound Matrix Multiplications"]
E --> F["Needs Fast On-Chip SRAM (Groq / Cerebras)"]
end
```
This dynamic creates a major structural bottleneck for ultra-fast, low-latency specialized AI hardware. While architectures relying on massive on-chip SRAM deliver unbeatable speed for weight access and matrix multiplications, their limited on-chip memory capacity makes handling massive KV caches an immense operational challenge. Even though labs like Anthropic pioneered 1-million-token context windows, developers in production still actively trigger manual compaction commands (like running `/compact` in Claude Code) long before approaching those theoretical limits.
- **Crystallized vs. Dynamic Knowledge:** Model weights store frozen training data; the KV cache dynamically stores conversational context and can easily surpass weight size in memory footprint.
- **Training Disparity:** Models exhibit performance degradation at 200k tokens because the bulk of training distribution centers on 8k–16k token lengths.
- **Hardware Bottleneck:** On-chip memory accelerators struggle with large KV caches due to strict physical memory limits compared to off-chip storage.
---
`31:38` The "Original Sin" of Transformers and Hybrid Chip Architectures
Looking three to five years into the future, specialized chipmakers like Cerebras and Groq will likely function primarily as accelerators operating alongside traditional GPUs with off-chip high-bandwidth memory (HBM). Maximizing inference efficiency requires hybridizing two distinct paradigms: fast on-chip SRAM for processing speed and high-capacity off-chip memory for context depth. As transformers push toward million-token contexts, this division maps directly onto the internal split of transformer layers.
The fundamental architectural dilemma—what can be called the "original sin" of Transformers—lies in placing an inherently memory-bound operation directly alongside a compute-bound operation within every layer:
| Layer Component | Primary Function | Computational Bottleneck | Ideal Hardware Placement |
|:--- |:--- |:--- |:--- |
| **Attention Layer** | Dynamically adapts to conversational history / KV cache | **Memory-bound** (in the limit of long context) | GPUs with massive off-chip HBM |
| **MLP (Feed-Forward) Layer** | Encodes static world knowledge via matrix multiplications | **Compute-bound** (at high batch sizes) | Accelerators with fast on-chip SRAM (Cerebras, Groq) |
Because building a single monolithic silicon architecture that excels equally at both memory bandwidth and raw compute density is exceptionally difficult, the GPU remains a versatile compromise. However, future high-efficiency topologies will increasingly split model workloads, routing the memory-heavy attention mechanism to GPUs while offloading the compute-intensive MLP layers to high-speed accelerator chips.
---
`33:19` Unsupervised Learning, Sequence Modeling, and the Sponge of Scaling Laws
The transformer architecture revolutionized artificial intelligence by enabling highly efficient unsupervised learning over arbitrary sequential data. At its core, the headline innovation is the attention mechanism, which allows the network to dynamically look back across every previous token in a sequence, re-weighting their significance to determine what is most relevant for predicting the next token. Because human language is the most expressive and information-dense sequence data generated by civilization, transformers naturally came to dominate natural language processing.
Crucially, transformers succeed because they eliminate brittle, human-designed priors. Where older computer vision and classical machine learning models imposed rigid structural assumptions—limiting support vector machines (SVMs) to thousands of parameters and deep vision networks to roughly 150 million parameters—transformers assume only that latent patterns exist within sequences. They function as massive empirical "sponges," smoothly absorbing compute and data. Increasing available compute by an order of magnitude yields predictable, logarithmic performance improvements, confirming the enduring power and elegance of empirical AI scaling laws.
- **Dynamic Attention:** Re-evaluates entire past context sequences at every single predictive step without rigid structural constraints.
- **Parameter Explosion:** Bridged the historical leap from classical models (thousands of parameters) and early computer vision (tens to hundreds of millions) to modern multi-trillion parameter frontiers.
- **Compute Absorption:** Operates without domain-specific priors, allowing performance to scale reliably as a direct function of compute and data.
`35:00` Why Transformers Work: Overfitting and Compression
Transformers remain dominant because they function as exceptionally powerful general learners that can scale across virtually any dataset thrown at them. Their decisive advantage over alternative attention-replacement techniques is that they model unconstrained pairwise relationships: any single token in a sequence can attend to any other token. While real-world problems might not always require dense, all-to-all connectivity, the architecture guarantees that if an underlying relationship exists in the sequence, the model can capture it. Until researchers discover more disciplined ways to prune that interaction space and make information retrieval selective, dense attention remains unbeatable.
This behavior mirrors a classic heuristic from the computer vision era popularized by Andrej Karpathy: when training on a novel dataset, the initial objective should always be to drastically over-parameterize the model and intentionally overfit the training data. Overfitting serves as definitive proof that a learnable relationship exists, that the optimization algorithm functions properly, and that knowledge can be instilled into network weights. Generalization only happens afterward through compression—working backwards from an overfitted baseline to distill those generalizable patterns into the smallest feasible parameter count rather than rote-memorizing the corpus.
`36:43` The Future of Data: Gyms and Verifiable Tasks
The historical pool of human knowledge can be understood as a "one-time subsidy on data" that the AI industry received for free. That high-quality corpus amounts to roughly 30 trillion tokens of clean text, expanding to approximately 300 trillion tokens under a more permissive definition of acceptable data. Frontier models have already digested the entire public internet multiple times over, exhausting the utility of passive human scrapings. Furthermore, the era of mining generic chat interactions from everyday users has hit severe diminishing returns. Because today’s median model is already significantly more capable than the average consumer, unconditioned feedback from the "everyday Joe" provides negligible signal; models now require either niche expert preference or automated evaluation environments.
```mermaid
flowchart TD
accTitle: Evolution of AI Data Sources
subgraph S1["Phase 1: Internet Subsidy"]
A["Static Web Scrapes\n30T-300T Tokens"] --> B["Full-Corpus Saturation\n(Seen Multiple Times)"]
end
subgraph S2["Phase 2: RL Gyms & Verifiable Environments"]
C["Verifiable Domains\n(Code, Math, Tool Use)"] --> D["Agentic Gym\n(Computer Sandbox)"]
D --> E["Automated Self-Grading\n(Progress Metric)"]
E -->|Recursive Self-Improvement| D
end
B -. Exhausted.-> C
```
The future of data therefore centers on reinforcement learning (RL) "gyms" focused on verifiable tasks. Instead of absorbing static text, models are placed into isolated execution sandboxes where they tackle objective challenges—such as writing complex software, proving mathematical theorems, or operating a personal computer like a knowledge worker—and receive clear programmatic feedback on whether their actions achieved the goal. Frontier lab balance sheets reflect this transformation, reallocating spend from acquiring raw web data toward engineering high-fidelity RL environments. Achieving artificial general intelligence will likely not stem from a single unified epiphany, but from iteratively stacking these specialized, verifiable intelligences until no operational gaps remain.
- **Total Internet Token Pool:** ~30 trillion tokens of strictly high-quality text; ~300 trillion tokens under a broader definition.
- **Diminishing Signal from General Users:** Raw human preference data from standard conversational prompts has ceased to move frontier model capabilities forward.
- **Verifiability as Core Prerequisite:** Self-improvement loops only function when the agent has access to an objective, automated grading mechanism.
`39:45` The Kernel Frontier and FlashAttention
Optimizing model execution at the hardware boundary requires granular control over GPU kernels—the discrete computational programs executed directly on the graphics processor. While external observers often assume low-level GPU programming is a solved problem thanks to foundational breakthroughs like Tri Dao's FlashAttention, modern production workloads constantly fall off the "happy path." Even minor architectural tweaks—such as introducing an alternative Rotary Position Embedding (RoPE) formulation—instantly invalidate standard open-source kernels, forcing teams to patch or rewrite low-level code to maintain peak hardware utilization.
Historically, GPU software architectures have relied on discrete, siloed libraries where individual kernels handle narrow, single-purpose primitives: one specialized kernel executes a general matrix multiply (GEMM), while an entirely separate kernel might perform a basic element-wise tensor addition. The emerging frontier of efficiency centers on kernel fusion—merging these isolated compute steps into compound operators that process multiple mathematical transformations in a single pass to eliminate expensive round-trips to high-bandwidth GPU memory.
`40:00` Kernel Engineering, AI Code Generation, and Whiteboard Programming
The conversation explores low-level optimization, beginning with kernel fusion—combining sequential operations like a matrix multiplication and an addition so intermediate data stays in on-chip registers rather than round-tripping out to high-latency DRAM. Given how mechanical this work appears, the host asks why human specialists like Tri Dao (the pioneer behind FlashAttention) are still required, rather than automated AI models handling the engineering end-to-end. Neil explains that while human kernel writing is evolving rapidly, the workflow has shifted toward conceptual design:
> "I like to say we write kernels on the whiteboard. We go to the whiteboard. We describe what we think the machine should be doing. Then we succinctly describe that in natural language to a model. And then the model is able to do the execution of: here is my input and output, here is the strategy of how we want to dispatch this work onto the GPU, I'm going to go implement it."
Neil cautions that custom kernel engineering is unlikely to serve as a long-term defensible moat. As frontier models like GPT-5 and next-generation reasoning systems improve, their baseline ability to write near-optimal low-level GPU code will lift the entire industry. While specialized engineering teams can inject domain knowledge or privileged prompts to guide code generation today, everyone is ultimately downstream of frontier model capabilities.
```mermaid
flowchart LR
accTitle: Whiteboard-Driven Kernel Engineering Flow
subgraph Human["Human Architecture"]
A["Whiteboard Design\nStrategy & memory layout"] --> B["Natural Language Prompt\nDispatch & I/O specs"]
end
subgraph Model["Model Execution"]
B --> C["Frontier AI Model"]
C --> D["Fused GPU Kernel Code\nTargeted dispatch"]
end
```
---
`41:48` Hardware Utilization: Extracting Energy from the Blackwell "Piece of Coal"
Drawing a historical parallel to the energy industry—where steam and coal engines evolved from capturing 10% of thermal energy to over 90%—the discussion turns to how much raw performance is actually extracted from cutting-edge silicon like NVIDIA's Blackwell architecture. Neil notes that in the chip's ideal state—performing massive, high-dimensional matrix multiplications—it can achieve around 70% of peak theoretical utilization. Pushing beyond that limit is constrained by physics and hardware thermals rather than software inefficiency:
- **Thermal and Power Throttling:** NVIDIA’s quoted peak FLOPS represent an idealized ceiling that real-world silicon cannot sustain continuously without hitting thermal and power envelopes.
- **The "Happy Path" Challenge:** Standard transformer inference rarely stays in that continuous, large-batch compute matrix multiplication regime.
- **The Factory's Core Task:** Sail's infrastructure is specifically architected as an "engine around the chip" designed to continuously feed GPUs maximum-sized batches of work so they never drop out of their highest-efficiency operating state.
---
`43:08` The Rack-Scale Paradigm and the High-Stakes Allocation of Frontier Chips
A fundamental architectural shift has taken place over the past year: computing is no longer programmed one GPU or one motherboard at a time, but across entire racks and data centers. With systems like the NVL72 (liquid-cooled racks integrating 72 Grace Blackwell GPUs), NVIDIA dictates the complete hardware topology. Neil emphasizes that mastering this rack-scale paradigm is the key to unlocking both the lowest absolute latency and the highest possible token throughput.
```mermaid
flowchart TD
accTitle: Evolution of GPU Computing Scale
subgraph Historical["Historical Paradigm"]
G1["Single GPU Card"] --> M1["Single Server / Motherboard"]
end
subgraph Modern["Modern Paradigm"]
M1 --> R1["Rack-Scale Compute\n(e.g., NVL72 / GB200)"]
R1 --> D1["Data Center-Scale Cluster"]
end
```
When asked whether the ravenous scramble for Blackwell chips resembles an illicit supply-constrained market, Neil outlines why NVIDIA does not simply run an uncapped price auction to clear the market:
- **Strategic Ecosystem Management:** Letting the deepest-pocketed tech giants buy up all available compute risks creating an overly concentrated customer base that could eventually dictate terms to NVIDIA. Because "compute is power today," allocation is handled strategically to balance the ecosystem.
- **Counterparty and Financial Risk:** Renting tens of thousands of next-gen GPUs involves multi-year commitments worth tens or hundreds of millions of dollars. Cloud providers and suppliers are deeply hesitant to allocate scarce hardware to early-stage startups that have only operated for a few months, making deep industry relationships and financial credibility indispensable for securing capacity.
`45:00` "No Bad Chips, Only Bad Pricing": Exploiting Alternative Silicon
While NVIDIA commands unprecedented financial backing due to sky-high demand and severe scarcity, Neil approaches the broader chip ecosystem with a pragmatic, value-oriented philosophy: "There's no bad chips. There's only bad pricing." As long as the cost matches the compute output, Sail is willing to make any silicon work. The primary roadblock for non-NVIDIA chips like AMD is rarely the raw silicon; it is that the broader industry simply does not know how to program them effectively. NVIDIA invests massive software resources into making their kernels blazing fast out of the box, leaving only modest optimization "alpha" on the table. In contrast, alternative vendors do less out-of-the-box kernel tuning. For a team with elite kernel engineers, this creates a massive opportunity: unoptimized chips combined with a widespread perception that AMD is inferior to NVIDIA is absolute "music to my ears," allowing Sail to buy up neglected supply at attractive valuations.
```mermaid
flowchart TD
accTitle: Heterogeneous Silicon Arbitrage Workflow
subgraph Sourcing["1. Silicon Sourcing & Arbitrage"]
A["Alternative Hardware<br/>(AMD Instinct, AWS Trainium, Google TPU, Startups)"] --> B["Low Acquisition Cost / High Availability"]
end
subgraph Engineering["2. Software Optimization"]
B --> C["In-House Kernel Team<br/>Custom Low-Level Microcode"]
C --> D["Comparative Advantage Tuning<br/>(Tailored to Specific Model Layers)"]
end
subgraph Serving["3. Heterogeneous Serving Stack"]
D --> E["Sail Token Factory"]
E --> F["Ultra-Low Cost Inference Margins"]
end
```
That dynamic of neglected hardware is already shifting at the high end. Hyperscalers like Meta and OpenAI have secured enormous volumes of AMD chips, rapidly tightening AMD's open-market allocations. Beneath them, a long tail of chip startups—such as Etched, SambaNova, and d-Matrix—is pushing into the ecosystem. For these net-new hardware ventures, the existential bottleneck is manufacturing scale: securing enough wafer allocation from TSMC to reach commercial volume. Sail aggressively monitors these new entrants to capture early supply, operating an architectural arbitrage where they squeeze performance out of overlooked silicon and pass the savings on at healthy margins.
- **Comparative advantage over standardization:** Rather than locking into homogeneous data center deployments, Sail embraces heterogeneous architectures across Google TPUs, AWS Trainium, AMD Instinct, and bespoke ASICs, pairing each chip to the specific workload where its economics shine.
- **Fast-moving data center partnerships:** Deploying non-standard silicon requires working with unconventional data center operators who are agile enough to support new server form factors and creative power architectures without bureaucratic delays.
---
`47:56` Is the AI Boom Different? Speculation vs. Immediate Utility
Investor sentiment around the semiconductor run-up has grown increasingly skittish. Historically, semiconductors accounted for roughly 2% to 4% of the S&P 500; today, they represent nearly a fifth of the entire index's market cap. With cyclicals like Micron and SK Hynix riding explosive rallies, many market historians fear a brutal reversion to the mean under the assumption that compute will inevitably commoditize. Neil views this anxiety through personal history rather than abstract financial charts. Born in 1997, he grew up watching his mother work at Intel through the frenzy and aftermath of the dot-com bubble, an era when Cisco briefly stood as the most valuable company on Earth with Intel close behind.
The critical distinction between the 2000 telecom bust and today’s AI buildout lies in the nature of the demand. The dot-com crash followed speculative overinvestment in dark fiber and networking hardware built for hypothetical internet traffic that failed to materialize for years. Token consumption today is fundamentally non-speculative: "People buy tokens because they're immediately valuable to them. You don't hoard tokens, you use them immediately."
This dynamic also diverges sharply from the 2023–2024 supply crunch for NVIDIA Hopper chips. That earlier phase was dominated by training runs, which are inherently speculative research bets that may never yield a commercially viable product. In contrast, modern inference represents direct utility in production software, underscored by users hitting rate limits and companies actively placing budget caps on autonomous developer tools like Claude Code. Because inference serves immediate, real-time work, Neil sees aggregate inference spend trending monotonically upward rather than collapsing under speculative overcapacity.
`50:00` Training vs. Inference and the Diseconomy of Data Center Scale
The infrastructure players—from physical data center developers to foundries like TSMC—are historically the most conservative actors in the entire AI ecosystem. Up to this point, data centers have been engineered primarily for training, which represents a strict superset workload over inference. While any high-bandwidth training cluster can effortlessly run inference workloads, the inverse is not true. Training demands extreme inter-chip networking bandwidth and monolithic cluster sizes where hundreds of thousands of GPUs must communicate continuously, creating severe diseconomies of scale. It is exponentially more expensive and logistically punishing to assemble a single facility housing 100,000 GPUs than it is to build clusters of 10,000 or 1,000 GPUs.
This bottleneck is ultimately dictated by aggregate power availability on the grid. Constructing a gigawatt-scale data center in the United States today is virtually impossible, and securing a 100-megawatt interconnection is increasingly out of reach for anyone outside a tiny circle of hyperscalers. Even 10 megawatts sits on the outer edge of feasibility. Conversely, 1-megawatt power drops are plentiful throughout the country, creating a wide floor of decentralized, stranded capacity. While training cannot tolerate the latency of cross-data center communication, inference does not require massive co-located clusters. Thanks to modern liquid cooling, power density has surged to the point where an entire megawatt of compute no longer requires a massive warehouse footprint—it can now fit into roughly eight refrigerator-sized racks lined up side by side.
```mermaid
flowchart TD
accTitle: Training vs Distributed Inference Cluster Requirements
subgraph Training["Monolithic Training Facility"]
T1["Gigawatt / 100 MW Megacluster"] --> T2["100,000+ Unified GPUs"]
T2 --> T3["Ultra-dense NVLink / InfiniBand"]
T3 --> T4["Zero Fault Tolerance / High Redundancy Overhead"]
end
subgraph Inference["Distributed Token Factory (Sail)"]
I1["Plentiful 1 MW Local Power Drops"] --> I2["~8 High-Density Liquid-Cooled Racks"]
I2 --> I3["Single Fiber Line / Diverse & Cheap Chips"]
I3 --> I4["Decoupled Async Workloads via Software Control Plane"]
end
```
- **Power availability tiers**: Gigawatt facilities are nearly impossible; 100 MW is extraordinarily rare; 10 MW is on the boundary of feasibility; 1 MW sites are abundant nationwide.
- **Physical footprint evolution**: Advances in direct-to-chip liquid cooling allow a 1 MW compute deployment to compress down to approximately eight server racks.
---
`52:45` The Scavenger Strategy: Buying Unwanted Chips and Distributed Compute
The roadmap for radically lowering the cost of intelligence relies on combining two structural arbitrage opportunities: harvesting heterogeneous compute at deep discounts and running it inside hyper-efficient, small-scale physical facilities. The foundational ethos is radically simple: "we will buy any chip anywhere in the world for any duration of time." By offering complete liquidity and flexibility to take on capacity that hyperscalers and frontier labs reject, a token factory can acquire raw FLOPs far below prevailing market prices and stitch them into a unified, functional fleet.
```mermaid
flowchart LR
accTitle: Low-Cost Token Generation Flywheel
A["Discounted & Heterogeneous Compute"] --> C["Ultra-Low Cost per Token"]
B["Low-Overhead 1 MW Distributed Sites"] --> C
C --> D["Background Agent Workloads"]
```
To maximize this cost advantage over the long run, infrastructure must be stripped of the expensive enterprise overhead that burdens traditional tier-3 and tier-4 data centers. Building an army of a thousand distributed 1 MW facilities means deliberately eliminating costly backup diesel generators, redundant power conditioning systems, and multi-path fiber trenches. Instead of paying exorbitant premiums for triple-redundant network transit and tight enterprise service level agreements (SLAs), these lean facilities run on a single local power feed and a single trenched fiber line, drastically cutting capital expenditures.
---
`54:30` Embracing 95% Uptime for Asynchronous Background Agents
Stripping away power and network redundancy means facilities will inevitably experience outages, driving availability down to levels that traditional enterprise buyers would consider catastrophic. A data center with 95% uptime would be completely fatal for a real-time web application or a synchronous, chat-oriented consumer product, leaving traditional operators with essentially zero buyers for such capacity. However, a specialized token factory can become the very first buyer for 95% uptime by designing the software layer around two core assumptions: a resilient distributed control plane and the asynchronous nature of long-horizon AI agents.
```mermaid
flowchart TD
accTitle: Resilient Task Relocation for Background Agents
A["Agent Running Long Background Task"] --> B{"Transient Site Failure (95% Uptime)"}
B --> C["Control Plane Detects Drop"]
C --> D["Task Re-routed to Available GPU Fleet"]
D --> E["Agent Resumes (Adds 2-10 Min Latency)"]
E --> F["Zero Impact on Asynchronous End User"]
```
Because failures across geographically scattered 1 MW sites are uncorrelated, the central control plane can instantly detect an offline facility and migrate in-flight requests to another GPU in the global fleet. When an agent running an autonomous multi-hour workflow loses its compute node mid-turn, rerouting the task might inject an extra two to ten minutes of latency while a new GPU is allocated. For an interactive chatbot this would break the user experience, but for an asynchronous background agent performing complex research or deep analysis while the user is asleep, a minor transient delay is entirely imperceptible:
> "My argument is that my customers don't care because their agent was running for hours. They're sleeping. It doesn't matter."
- **Fault tolerance dynamics**: A robust control plane treats hardware availability as a linear cost curve, where 95% uptime yields a proportional, acceptable discount in operational cost.
- **Latency insensitivity**: Relocating tasks during unexpected downtime introduces a 1-to-10-minute delay, which is negligible against workloads designed to run unattended for hours or days.
`55:00` Scavenging Intermittent Power for Background Agents
The economic thesis behind serving background agents rests entirely on breaking the standard Service Level Agreements (SLAs) that traditional cloud providers guarantee. When selling inference, Sail explicitly sets expectations with customers: while average throughput remains highly competitive, the 99th percentile (P99) latency is intentionally left uncontrolled. For background tasks running over hours or days, an unpredictable spike in latency or a delay is completely acceptable if it delivers unbeatable cost economics.
This willingness to compromise on uptime transforms how a data center can approach energy. While modern facilities demand five-nines reliability and 24/7 baseload power, a token factory designed for delayed batch workloads can operate comfortably on 95% uptime, or even 80% if the power is cheap enough. As a "son of California" with a natural fondness for solar and wind, Neil sees renewable energy as drastically undertapped in the United States, blocked primarily by the problem of intermittency. Traditional data centers consider solar and wind unviable because a constant baseload cannot easily run on sources that fluctuate with the weather.
Sail circumvents this by embracing what would normally be considered a catastrophic outage—having a site go dark for days or weeks due to heavy fog hanging over a valley or calm, windless skies. Because weather systems are highly predictable, workloads can simply be modeled and routed away to another facility across the globe ahead of time. Tolerating these deep outages grants access to cheap, orphaned power that nobody else can touch because it is too logistically annoying for conventional operators to manage. The trade-off requires cheaper hardware: if an operator fills a facility with cutting-edge, top-of-the-line Nvidia racks, the capital depreciation on idle chips during a power lull is ruinous. But with inexpensive, depreciated, or alternative silicon, the capital cost of sitting idle becomes negligible.
```mermaid
flowchart TD
accTitle: Scavenger Arbitrage for Compute and Power
subgraph Power["Power Scavenging"]
direction TB
RE["Cheap Intermittent Renewables<br>(Solar, Wind, Stranded Power)"]
Forecast["Weather Modeling & Forecasting"]
RE --> Forecast
end
subgraph Compute["Hardware Arbitrage"]
direction TB
CheapChips["Low-Cost Silicon<br>(Alternative / Depreciated ASICs)"]
IdleTolerance["Low Capital Cost<br>Tolerates Extended Downtime"]
CheapChips --> IdleTolerance
end
subgraph Routing["Dynamic Orchestration"]
direction TB
Failover["Global Workload Migration<br>(Shift to Sunny/Windy Regions)"]
Workload["Background Agent Inference<br>(Relaxed P99 Latency SLA)"]
Failover --> Workload
end
`56:43` The Scavenger Strategy: Mini-Mills vs. Monolithic Plants
This operational model forms what Neil calls a "scavenger strategy"—first scavenging underutilized chips, and then scavenging the stranded power to feed them. The fundamental rule is never to bid against well-funded frontier labs like OpenAI or Anthropic for hyper-scarce, top-tier compute capacity. Entering a head-to-head bidding war against hyperscalers is a losing battle; instead, the goal is to target supply that is completely illegible to them.
Rather than chasing a massive, concentrated supply at a single gigawatt site, Sail pieces together an aggregate fabric out of disparate, overlooked sources. Neil draws an analogy to the industrial history of steel manufacturing: rather than constructing an enormous, monolithic Bessemer steel mill, Sail operates like an agile network of mini-mills. Over time, amassing these unconventional fragments of power and hardware results in an aggregate token factory whose unit economics are impossible for traditional hyperscalers to beat.
- **Non-competitive procurement:** Avoids bidding against frontier labs by targeting supply they cannot legibly consume.
- **Aggregate vs. concentrated scale:** Builds massive capacity through widely distributed, uncoordinated nodes rather than gigawatt megasites.
- **The steel mini-mill analog:** Displaces massive, capital-heavy legacy plants by aggregating flexible, cheaper local alternatives.
---
`58:30` Vertical Integration, Pragmatism, and Real-Life Factorio
When evaluating how vertically integrated a token factory should be, an operator faces a vast spectrum. At the absolute extreme sits total vertical integration: owning the primary energy generation assets, constructing custom data centers, fabricating proprietary silicon, writing the orchestration software, and selling the finished token straight to the end customer. At the opposite end lies an ultra-capital-light model: owning zero physical assets and existing purely as a software-defined coordination layer across third-party infrastructure.
Neil admits that his reaction to this choice reveals a tension between his two inner personas:
- **The Founder:** An imaginative engineer whose entire life is consumed by chips, power, and thermodynamics, who wants to touch every single variable: "I will never stop until I have built the most efficient system from soup to nuts." Host and guest joke that he is essentially playing a real-life game of *Factorio* for intelligence.
- **The CEO:** A pragmatic operator who recognizes that owning the entire physical stack from day one requires an astronomical amount of capital that would sink an early-stage company.
Because software offers the highest structural leverage, the pragmatic path starts with pure software orchestration. Rather than buying out power plants or utilities immediately, Sail pursues favorable Power Purchase Agreements (PPAs) and lets domain experts handle what they have historically mastered. Neil views deep vertical integration as a privilege that must be earned: you reach massive scale first, earning the right to bring individual layers of the stack in-house as inefficiencies become unbearable.
Because the industry is underwriting what will become the largest market for compute in the history of civilization—trillions of dollars destined for inference—breaking the legacy assumptions embedded in hardware and power becomes economically mandatory. Neil looks first to infrastructure partners to build custom, inference-optimized solutions that shatter those old assumptions; if those partners cannot or will not build them, Sail will step in and build them directly.
---
`01:01:04` Stack-Ranking Inefficiencies: Compute vs. Memory and Sparsity
When zooming out across the full technological stack—software algorithms, chip architectures, data center infrastructure, and energy generation—the natural question is where the system remains most glaringly inefficient at producing useful intelligent tokens.
Contrary to conventional assumptions, raw compute scaling is not the primary bottleneck. As Neil highlights, the industry is already remarkably effective and judicious at utilizing raw floating-point operations (FLOPs): "You give me more FLOPs and I will use more FLOPs." In fact, modern algorithmic architectures have already embraced aggressive compute pruning. In contemporary Mixture-of-Experts (MoE) architectures, very few models run at more than 10% density—meaning that out of all available expert sub-networks baked into the weights of the model, less than 10% are actually activated for any given token during inference. Compute efficiency has advanced rapidly, forcing the real systemic bottlenecks elsewhere into data movement, memory bandwidth, and power distribution.
`01:00:00` Inefficiencies in Attention, KV Cache, and Global Compute Orchestration
Examining where modern compute architectures waste the most resources, the bottleneck is rarely inside Mixture of Experts (MoE) routing. Frontier models already achieve roughly 1% sparsity, and the industry has spent years mastering how to squeeze every drop of efficiency out of MoE layers. The glaring inefficiency lives inside the attention mechanism and its memory footprint—specifically the key-value (KV) cache. Currently, systems store multiple kilobytes of uncompressed data per token in the KV cache, an information density that fails to "earn its keep" relative to its underlying entropy. That storage requirement is likely off by one or two orders of magnitude. While closed labs keep their methods proprietary, DeepSeek has published breakthrough research on aggressive KV cache compression, proving that an order-of-magnitude efficiency gain can still be unlocked year after year.
Zooming out from silicon architecture to global compute logistics reveals an even starker problem: compute orchestration across the planet is deeply broken. With Nvidia projected to produce 5 million Blackwell chips in a single year, vast swaths of this hardware disappear into private enterprise pools, dedicated customer allocations, or warehouses where GPUs sit entirely idle. While public scrutiny often targets xAI's total FLOP utilization challenges on massive clusters, the reality across the broader enterprise landscape is far worse. Silicon and power sit stranded behind rigid allocations, which causes literal physical pain for efficiency-minded engineers. Sail's mission attacks this orchestration failure directly by treating global compute as a dynamically packed, shared utility.
subgraph Algorithmic["Algorithmic Layer"]
A["MoE Routing<br/>(~1% Sparsity, highly optimized)"]
B["Attention & KV Cache<br/>(Kilobytes/token, uncompressed)"]
B -->|Order of Magnitude Compression| C["DeepSeek-Style KV Compression"]
end
subgraph Infrastructure["Global Infrastructure Layer"]
D["5M+ Nvidia Blackwell Chips"]
D --> E["Private Enterprise Pools"]
D --> F["Stranded Warehouse Capacity"]
E & F -->|Low FLOP Utilization| G["Idle Silicon & Wasted Power"]
G -->|Dynamic Orchestration| H["Shared Token Factory Utility"]
end
```
- Frontier models operate at roughly 1% sparsity in MoE layers, leaving minimal room for low-hanging optimization gains there.
- The KV cache stores uncompressed data that is an estimated 10x to 100x larger than the true entropy of the tokens requires.
- Nvidia's output of ~5 million Blackwell chips highlights the massive scale of underutilized hardware locked in private, unshared silos.
---
`01:02:44` Semiconductor Fabs, Process Corners, and Embracing Imperfect Silicon
Addressing the supply constraints of foundries like TSMC, Intel, and memory manufacturers requires looking past simplistic silver bullets. Simply doubling fab throughput or magically expanding wafer capacity by 100x does not instantly yield 100x cheaper tokens; semiconductor ecosystems grow in delicate equilibrium, and clearing a TSMC fab constraint immediately shifts the bottleneck 20% down the line to packaging, memory, or power delivery. A far more fertile optimization lies in re-evaluating the rigid invariants foundries impose on silicon delivery.
Every silicon manufacturing run produces substantial variance between the best-performing and worst-performing dies on a wafer. Foundries like TSMC invest immense time, engineering controls, and capital expenditure into tightening "process corners"—ensuring that the bottom-tier chips behave as closely as possible to the top-tier chips. While conventional cloud providers demand this near-uniformity, an agile inference provider does not need foundries to discard or over-control imperfect dies. By pairing lower-binned chips or factory rejects with dramatically expanded, low-cost power scavenged across the United States, imperfect silicon that would otherwise fail to earn a spot in a tier-one data center can be run profitably and efficiently.
- Semiconductor supply chains are tightly coupled; fixing a front-end foundry bottleneck simply surfaces downstream bottlenecks after a 20% production increase.
- Tightening fab "process corners" adds significant manufacturing cost to force uniformity between the fastest and slowest dies on a wafer.
- Sail’s strategy trades silicon uniformity for cost by deploying rejected or lower-binned dies into ultra-cheap power envelopes.
---
`01:05:08` Culture, Engineering Principles, and Hiring for Performance at Sail
Building a company centered on radical cost reduction requires an engineering culture that rejects fixed constraints. When Sail's engineering team begins optimizing a new model, initial baseline performance is often poor, but the team operates on a multi-month and multi-year trajectory rather than accepting the immediate status quo. Even when working with state-of-the-art silicon like Nvidia's Blackwell architecture, hardware limits are never treated as unchangeable facts of nature. Engineers systematically profile, characterize, and document underlying hardware bottlenecks, sharing actionable architectural feedback directly with Nvidia while building an internal institutional playbook for evaluating future silicon purchases.
The internal dynamic at Sail is deeply collegial, built around collaborative whiteboard sessions and peer instruction. The company actively recruits individuals who embody both great students and exceptional teachers, frequently hiring engineers who served as college teaching assistants and genuinely enjoy demystifying complex concepts. When evaluating which human skills will remain durable over the next several years as autonomous machines absorb routine software tasks, raw technical trivia matters far less than intrinsic mindset. The core prerequisite is insatiable curiosity combined with an innate, unteachable "love for performance."
- Hardware behavior is treated as malleable; bottlenecks on new chips are exhaustively benchmarked, documented, and fed back to hardware partners like Nvidia.
- Team hiring strongly favors former academic TAs who cultivate an ego-free environment where everyone alternates between teaching and learning.
- The ultimate filter for long-term engineering talent in an AI-accelerated world is pure curiosity paired with a deep passion for system optimization.
`01:05:00` What Makes a Great Performance Engineer
When evaluating talent for low-level systems work, the single most critical attribute is an obsession with digging into every single microsecond the hardware is running and understanding exactly what is taking place on the silicon. Extensive prior AI experience or deep familiarity with CUDA is considered a red herring rather than a prerequisite. Because GPUs and the broader CUDA paradigm have evolved so drastically over the past five years, demanding a decade of legacy experience is largely pointless. Technical frameworks can be taught on the job, but an intrinsic passion for performance engineering cannot.
- **Mindset over pedigree:** Prioritizes engineers who are driven to inspect hardware behavior at the microsecond level over those with extensive resumes in AI or CUDA.
- **Evolving toolchains:** Given the rapid architectural shifts in GPUs over recent years, historical platform-specific experience matters far less than raw systems debugging capability.
---
`01:06:08` The Closed-Source Premium and Latent Distillation
Frontier labs such as Anthropic spend an enormous financial and computational premium simply to remain three to six months ahead of the open-source ecosystem. While this dynamic makes sense for frontier research labs, the debate around model distillation often overlooks how capability inherently diffuses across the broader software ecosystem. Although distillation is sometimes labeled as intellectual property theft, an ever-increasing share of internet artifacts is already AI-generated. A massive proportion of new GitHub repositories are authored using tools like Claude Code. If users own their AI-generated code and publish it openly, open models can simply train on high-quality public repositories, creating a continuous stream of "latent distillation."
```mermaid
flowchart TD
accTitle: The Latent Distillation Cycle
subgraph Frontier["Frontier Labs"]
A["Frontier Models<br/>(e.g., Anthropic / Closed Labs)"] -->|Assists Developers| B["AI-Generated Code<br/>(e.g., Claude Code artifacts)"]
end
subgraph PublicEcosystem["Open Web & Repositories"]
B -->|Pushed to Public Repos| C["Open Source Artifacts<br/>(GitHub & Web Data)"]
end
subgraph OpenSource["Open Source Ecosystem"]
C -->|Trains On High-Quality Data| D["Open Frontier-Class Models"]
D -.->|Narrows Capability Gap| A
end
```
It is fundamentally impossible to prevent the diffusion of model capabilities across the internet over time. Furthermore, the commercial durability of charging high premiums for a 3-to-6-month lead remains questionable because enterprise adoption cycles move far slower than frontier lab release cadences. Most enterprises operate conservatively, remaining on older model versions rather than integrating bleeding-edge releases overnight. Because open source will continuously fill any vacuum left behind, and the baseline cost to train frontier-grade models drops every day, open models will remain a permanent fixture of the AI landscape.
- **Latent distillation:** Frontier capabilities diffuse naturally into open datasets as developers publish AI-generated code and content to open-source platforms.
- **Enterprise inertia:** Large enterprises do not update architectures every three to six months, mitigating the immediate real-world economic value of short-lived frontier leads.
- **Persistent open source:** Lowering training costs and strong open incentives ensure that open-source alternatives will continuously track frontier capabilities.
---
`01:08:45` The Vision of Abundant Tokens and Diverse Harnesses
The overarching objective is to build a future characterized by radically cheap, abundant tokens paired with highly diverse agent harnesses. Rather than relying on a centralized platform like a "new Google" to control the entire stack, every company and individual user should be empowered to construct and own their own customized agents. Tailoring model intelligence will likely occur primarily through rich in-context learning rather than heavy weight fine-tuning.
Realizing this world requires treating intelligence not as a scarce, expensive consultant that is queried sparingly for difficult questions, but as an abundant, ubiquitous resource. Driving the marginal cost of a token as close to zero as physically possible demands leveraging every available supply-side efficiency across hardware, power sources, and geographical footprint:
> "My job is to make the tokens as cheap as humanly possible. I will achieve that and I will do it through every layer in the stack available to me. I love the supply-side levers. I will use every chip, I will use every source of power, and I will use every piece of land in the United States that's suitable for this."
- **Democratized harnesses:** Users and enterprises own customized intelligence frameworks adapted via extensive context rather than proprietary black-box fine-tunes.
- **Shifting mental models:** Transitions AI from an expensive, reserved oracle into a high-volume, continuous utility.
---
`01:10:15` Divergent Views on Chips, Flash Memory, and a Trillion Tokens a Day
The most contrarian perspectives center on chip architectures and memory hierarchies for inference. While the industry remains fixated on High Bandwidth Memory (HBM) supply bottlenecks, the alternative path involves designing custom silicon that sidesteps HBM entirely by aggressively offloading the KV cache onto flash memory. Realizing this requires investigating model architectural adaptations that make heavy flash offload viable.
Within the inference engineering community, optimizing hardware specifically around throughput rates of 1 to 10 tokens per second unlocks radical cost efficiencies that low-latency systems cannot achieve. The ultimate objective is establishing the infrastructure required for the ecosystem to affordably consume a trillion tokens per day.
- **HBM bypass:** Explores architectural modifications to offload KV cache directly to flash memory, escaping high-bandwidth memory constraints.
- **Targeted throughput:** Re-engineers the inference stack specifically for background agent workloads running at 1 to 10 tokens per second.
- **Scale target:** Architecting systems to sustainably support daily consumption volumes exceeding a trillion tokens.
`01:10:00` Trillion-Token Economics and Contrarian Tech Takes
At OpenAI pricing levels, a trillion tokens costs at least $5 million to $5.6 million. Scaling AI requires a three to six order-of-magnitude reduction in token costs so that a job currently costing millions can eventually run for tens of thousands of dollars. Pushing down these unit economics is essential so product developers never feel constrained from offering expansive token usage to free-tier users. Contrary to doubts about whether human demand for intelligence is capped, market demand for intelligence is virtually infinite; the primary hurdle lies in building accessible product on-ramps.
A notable contrarian viewpoint involves NVIDIA, TSMC, and the geopolitical panic surrounding chip manufacturing. Looking closely at hardware generations—from Hopper to Blackwell to Rubin—the performance-per-watt for BFloat16 matrix multiplication has not improved dramatically. A similar trend appears across TSMC's 5nm, 4nm, 3nm, and 2nm process nodes. If Western supply chains lost access to TSMC, the resulting supply shock would be manageable because Western fab capabilities like Intel are at worst only 2x behind in performance-per-watt.
- **Current Trillion-Token Cost:** ~$5 million to $5.6 million at baseline frontier pricing.
- **Cost Reduction Target:** 3 to 6 orders of magnitude to make massive background agentic tasks economically viable.
- **TSMC Process Node Curve:** Incremental performance-per-watt gains across 5nm $\rightarrow$ 4nm $\rightarrow$ 3nm $\rightarrow$ 2nm.
- **Western Fab Gap:** Intel's process nodes trail TSMC's performance-per-watt by roughly 2x at worst, making the geopolitical chip narrative overblown.
---
`01:13:19` Model Architectures and Hardware Bottlenecks
Serving infrastructure sits completely downstream from model developers like OpenAI and Anthropic. Upstream architectural choices—such as opting for sparse attention over dense attention, or transitioning training data formats from BFloat16 down to FP8 or FP4 low-precision formats—drastically dictate physical serving requirements and chip selection.
```mermaid
flowchart TD
accTitle: Upstream Model Choices to Hardware Execution
subgraph Upstream["Model Architecture Decisions"]
A["Attention Strategy<br/>(Sparse vs. Dense)"]
B["Data Precision<br/>(BF16 to FP8 / FP4)"]
end
subgraph Bottlenecks["Hardware Supply Chain Constraints"]
C["TSMC Wafer Capacity"]
D["HBM Capacity"]
E["Advanced Packaging"]
F["Power & Rack Real Estate"]
end
subgraph Downstream["Serving & Infrastructure"]
G["Custom Systems & Hardware Arbitrage"]
end
A --> Bottlenecks
B --> Bottlenecks
Bottlenecks --> G
```
Entrepreneurs starting compute hardware companies must address four major supply chain bottlenecks: TSMC wafer capacity, High Bandwidth Memory (HBM) capacity, advanced packaging, and power access. Because NVIDIA's general-purpose architectures are well-balanced but hard to change quickly, new hardware startups must be "spiky"—picking a specific bottleneck to attack directly.
- **Primary Memory Bottleneck:** Micron and other memory manufacturers in Boise are cautious about massive capital expenditure for cyclical HBM capacity, prolonging memory shortages.
- **System-Wide Cost Rebalancing:** Consumer electronics like iPhones will likely see price increases or reduced memory capacity as market demand reroutes scarce memory to enterprise AI chips.
---
`01:14:38` NVIDIA's Strategy and the Full-Stack Perspective
NVIDIA deliberately refrains from competing directly with its customers by building its own hyperscale neoclouds or selling tokens directly. CEO Jensen Huang has successfully cultivated an ecosystem of billion-dollar neocloud partners like CoreWeave. Creating a diverse, competitive landscape of buyers ensures that if any single vendor integrates vertically or adopts rival silicon like AMD, multiple alternative providers remain eager to purchase NVIDIA GPUs.
Reflecting on formative influences highlights the importance of taking a holistic view of technology systems. Early academic guidance pointed away from narrow early specialization in favor of mastering the entire computing stack—from gate-level silicon engineering up through operating systems, networking, and internet-scale software services. Enduring technical leadership requires holding a deep appreciation for every layer in the full stack of intelligence.
- **NVIDIA Business Philosophy:** Preserve ecosystem goodwill and avoid competing with client clouds to maintain buyer-side competition.
- **Full-Stack Competency:** True technical depth requires mastering every layer from silicon gates to distributed software architecture.
## Notable quotes
> `05:32` the best latency is no latency at all — Neil Movva
> `40:00` I like to say we write kernels on the whiteboard. We go to the whiteboard. We describe what we think the machine should be doing. Then we succinctly describe that in natural language to a model. And then the model is able to do the execution of: here is my input and output, here is the strategy of how we want to dispatch this work onto the GPU, I'm going to go implement it. — Neil Movva
> `45:00` There's no bad chips. There's only bad pricing. — Neil Movva
> `47:56` People buy tokens because they're immediately valuable to them. You don't hoard tokens, you use them immediately. — Neil Movva
> `52:45` we will buy any chip anywhere in the world for any duration of time. — Neil Movva
> `54:30` My argument is that my customers don't care because their agent was running for hours. They're sleeping. It doesn't matter. — Neil Movva
> `58:30` I will never stop until I have built the most efficient system from soup to nuts. — Neil Movva
> `01:01:04` You give me more FLOPs and I will use more FLOPs. — Neil Movva
> `01:08:45` My job is to make the tokens as cheap as humanly possible. I will achieve that and I will do it through every layer in the stack available to me. I love the supply-side levers. I will use every chip, I will use every source of power, and I will use every piece of land in the United States that's suitable for this. — Neil Movva
## Mentioned
Neil Movva, Patrick O'Shaughnessy, Sail, Sailboxes, Baseten, Fireworks AI, Together AI, Cursor, Codex, Claude Code, Parallel Web Systems, Anthropic, Claude Haiku, Siri, NVIDIA, Jensen Huang, ICML, NeurIPS, Cerebras Systems, Groq, TSMC, Micron, SK Hynix, Samsung, Kimi, Blackwell, Wafer-Scale Engine 3 (WSE-3), Costco, Andrej Karpathy, Tri Dao, FlashAttention, Rotary Position Embedding (RoPE), NVL72, GB200, Grace Blackwell, AMD Instinct, AWS Trainium, Google TPU, Etched, SambaNova, d-Matrix, Meta, OpenAI, Intel, Cisco, Factorio, DeepSeek, Hopper, Rubin, CoreWeave, GitHub, xAI
## Video description
My guest today is Neil Movva, founder of Sail. Sail is building what Neil calls a token factory, an inference company designed for a specific kind of future, one where AI agents run in the background for hours or days at a time rather than answering a human in real time.
In that world, latency matters less and cost matters more, and Neil has built the whole company around driving the cost of a token as low as it can possibly go.
What makes this conversation special is that it is one of the most detailed tours I have ever done through the full stack of intelligence, the software, the chips, and the power, and how all three connect.
Along the way we cover the trade-off between speed and cost that lives inside every GPU, his scavenger strategy for buying the chips and power nobody else wants, his contrarian view on Nvidia, and why the premium the frontier labs charge for being three to six months ahead may not last.
Please enjoy my conversation with Neil Movva.
For the full show notes, transcript, and links to mentioned content, check out the episode page here.
-----
Become a Colossus member to get our quarterly print magazine and private audio experience, including exclusive profiles and early access to select episodes. Subscribe at colossus.com/subscribe.
-----
Ramp’s mission is to help companies manage their spend in a way that reduces expenses and frees up time for teams to work on more valuable projects. Go to ramp.com/invest to sign up for free and get a $250 welcome bonus.
-----
Trusted by thousands of businesses, Vanta continuously monitors your security posture and streamlines audits so you can win enterprise deals and build customer trust without the traditional overhead. Invest Like the Best listeners get a special offer of $1,000 off Vanta when you go to vanta.com/invest.
-----
WorkOS is the infrastructure B2B and AI-native companies use to sell to enterprise. It covers everything enterprise security requires: SSO, SCIM, RBAC, Audit Logs, AI governance, and more. Trusted by 2,000+ fast-growing companies, including OpenAI, Anthropic, Cursor, and Vercel.
-----
Rogo is the AI platform for finance. They're building agents for Wall Street that are trained to understand how bankers and investors actually do work: from diligence and modeling, to turning analysis into deliverables. To learn more, visit rogo.ai/invest.
-----
Ridgeline has built a complete, real-time, modern operating system for investment managers. It handles trading, portfolio management, compliance, customer reporting, and much more through an all-in-one real-time cloud platform. Visit ridgeline.ai.
-----
Editing and post-production work for this episode was provided by The Podcast Consultant.
Neil Movva, founder of Sail, argues that artificial intelligence inference should be optimized for cost rather than low latency by catering to autonomous background agents instead of human-facing chatbots. By pairing low-cost, unoptimized hardware with decentralized power sources, he contends inference costs can fall by several orders of magnitude.
Trading Latency for Throughput in Background Workloads (2:20)
Movva posits that the artificial intelligence industry over-indexes on sub-second response times because current infrastructure was designed for interactive chat. While human-in-the-loop tasks like interactive coding are inherently bottlenecked by human attention spans, background agentic workloads—such as cybersecurity red-teaming and deep research—can run asynchronously over days or weeks. Movva forecasts that long-running background tasks could ultimately grow from roughly 50% to 90% of total inference market share.
Arbitraging Alternative Silicon Through Low-Level Software (45:00)
Rather than competing for hyper-scarce, top-tier chips like NVIDIA's Blackwell, Movva advocates a scavenger strategy centered on the principle that there are "no bad chips, only bad pricing." Alternative hardware from vendors like AMD often sells at lower valuations primarily because the broader industry lacks the custom kernels needed to run it efficiently. Movva claims an inference provider with elite kernel engineers can target this neglected supply, optimize microcode in-house, and match NVIDIA's compute output at significantly lower acquisition costs.
Decentralized Megawatts and Tolerating Imperfect Infrastructure (50:00)
Large-scale AI training requires monolithic, gigawatt- or 100-megawatt-scale data centers with extreme inter-chip networking bandwidth, which face severe power and grid diseconomies. Inference workloads, by contrast, can be distributed across plentiful 1-megawatt drops. Movva proposes operating an army of decentralized 1 MW sites that eliminate expensive redundancies like backup diesel generators and multi-path fiber trenches.
Embracing Low Uptime and Intermittent Power (54:30)
Traditional enterprise cloud services require high availability, but Movva argues background agents can tolerate 95% or even 80% uptime. Because failures across geographically scattered small sites are uncorrelated, a distributed control plane can detect transient outages and re-route in-flight agent tasks to other nodes. This tolerance allows operators to scavenge cheap, intermittent energy like solar and wind, treating weather-driven dropouts as acceptable latency spikes rather than catastrophic service failures.
Tackling KV Cache Inefficiencies and Memory Constraints (60:00)
Movva identifies the attention mechanism and its key-value (KV) cache as the primary architectural inefficiency in modern large language models, rather than raw compute or Mixture-of-Experts routing. Systems currently store kilobytes of uncompressed data per token in the KV cache, an information density Movva argues is misaligned with its underlying entropy. Addressing this bottleneck will require aggressive KV cache compression or architectural shifts, such as offloading cache data from expensive High Bandwidth Memory (HBM) directly onto flash storage.
Movva targets an eventual three-to-six order-of-magnitude reduction in token costs to bring trillion-token workloads from millions of dollars down to tens of thousands. However, this strategy hinges on customer willingness to accept relaxed latency service level agreements, as well as overcoming upstream semiconductor bottlenecks across advanced packaging and memory.
## Background
本期节目由 Colossus 旗下的知名播客《Invest Like the Best》（第 488 期）于 2026 年 8 月 25 日在 RSS 及各大播客平台发布。节目由主持人 Patrick O'Shaughnessy 对话 Sail Research 创始人兼首席执行官 Neil Movva，深入探讨了其正在构建的“Token 工厂”构想以及为后台长程自主 Agent 打造的超低成本推理基础设施。节目简介强调，这是一场贯穿底层软件算子、芯片硬件架构到边缘电力基础设施的“完整智能技术栈”全景拆解。
## TL;DR
Neil Movva 认为未来 90% 的 AI 推理将转向运行数小时乃至数天的后台自主 Agent，此时系统核心矛盾从实时低延迟转向极致的单位 Token 成本。为了让推理成本降低 10 倍，Sail 采取了独特的“算力清道夫策略”：放弃对前沿顶级集群的昂贵竞价，转而低价搜刮被市场低估的边缘芯片与 95% 可用性的间歇性绿电，通过自研异构调度与底层 Kernel 融合技术榨干硬件效能。同时，他深入剖析了 GPU 吞吐与延迟的权衡、Transformer 架构中注意力层与 MLP 的“原罪”、HBM 供应链短缺危机，并提出了对 NVIDIA 生态分配策略及前沿闭源实验室溢价不可持续的反共识研判。
## Atmosphere & style
整场对话洋溢着高密度的硬核技术极客气质与深刻的商业理性，宛如一场在大学白板前展开的深度系统架构推演。Neil Movva 展现出从晶体管门电路、底层 CUDA 算子到吉瓦级电网全栈贯通的透彻理解，既有对追求理论物理极限“光速”的工程师浪漫，又有拒绝重资产整合、精准进行商业套利的审慎务实。Patrick 的提问层层递进、直击底层机制，两人在冷静拆解行业狂热与浮躁叙事的同时，流露出对技术必然性与计算系统纯粹之美的发自内心的敬畏。
## Bulletin
- `03:22` Sail 正在通过超低成本 API 和专用虚拟机 Sail Boxes 构建“Token 工厂”，押注未来 AI 推理的主流场景将从即时交互的 Chatbot 转向后台自主运行数小时甚至数天的长程 Agent，届时推理成本将成为唯一决定性指标。
- `10:00` 大模型在网络安全中展现出“锯齿状智能边界”，前沿模型与 Claude Haiku 等轻量模型的漏洞发现集合并不重合，安全审计正演变成以异构模型密集攻击为形式的“工作量证明”。
- `20:00` 硬件利用面临吞吐量与低延迟的根本权衡：追求首字极速的张量并行不仅通信开销高昂且仅能带来 4 到 5 倍的次线性加速，而批处理流水线虽然拉长单请求延迟，却能彻底释放 GPU 算力并大幅压低单位 Token 成本。
- `36:43` AI 训练数据中的互联网公开文本红利（约 30T 至 300T Tokens）已基本耗尽，前沿实验室正将开支转向可客观自动验证的强化学习环境（RL Gyms）以支撑模型递归自我提升。
- `56:43` Sail 实施“算力清道夫策略”，利用液冷技术将 1 兆瓦机房压缩至 8 个机柜大小，主动采购市场上被低估的边缘芯片与 95% 乃至 80% 可用性的廉价间歇性绿色电力，通过上层软件调度构建抗周期的超低成本推理网络。
## Key takeaways
- `05:32` 未来三年内，长程自主智能体（Long-Horizon Agents）预计将占据整个 AI 推理市场份额的 90%，人类注意力的有限性决定了人机交互将收敛于天或周尺度的时间周期。
- `08:47` 深度研究与网络安全自动化是长程智能体的杀手级落地场景，通过调度异构模型集群展开全天候渗透测试，使系统安全性直接量化为“承受住了多少美元的 API 攻击”。
- `11:22` 只要推理成本持续下降数个数量级，一切具备客观自我校准机制的“可验证问题”（如软件工程、数学证明与科学发现）都将迎来通用机器解法。
- `19:15` GPU 底层设计本质上是“高吞吐量机器”，过去行业为了 Cursor 等工具的即时响应而强推极低延迟架构，造成了昂贵 GPU 算力的大量内耗与错配。
- `21:45` 英伟达 NVLink 是解决延迟的利器，但在高并发长程任务中，采用流水线并行和专家并行掩盖通信延迟，比依赖高成本硬件互连的张量并行具备更优的单位美元性价比。
- `24:58` 芯片架构正面临 SRAM（极速、极低容量、不可承受的 KV Cache 膨胀）与 DRAM/HBM（高密度、微秒级刷新延迟）的物理权衡，片上加速器与大显存 GPU 的异构协同是必然趋势。
- `31:39` Transformer 架构存在“原罪”，即把访存密集型的 Attention 层与计算密集型的 MLP 层强行绑定，未来异构芯片分工将趋向于由专用加速器承载静态权重、传统 GPU 承载长上下文缓存。
- `40:48` 底层算子（Kernel）优化正从人类手写 CUDA 代码过渡到“白板架构设计+前沿模型自动生成”，局部软件工程优势难以构成基础设施初创企业的绝对护城河。
- `45:00` 芯片市场不存在“劣质芯片”只有“错误定价”，非英伟达芯片因开箱即用软件调优较弱而被市场低估，反而为具备深层 Kernel 调优能力的厂商创造了丰厚的算力套利窗口。
- `47:56` 本轮 AI 繁荣与 2000 年互联网泡沫及早前大模型预训练存在本质区别：推理 Token 的消耗属于即产即销、带来确定性业务价值的生产性开支，而非面向远期虚幻预期的投机性基建。
- `50:00` 推理负载无需训练集群所需的 100MW 到 1GW 集中式超算网络，依托液冷技术将 1MW 算力微型化为 8 台机柜并接入碎片化电网，有效规避了数据中心的“规模不经济”。
- `55:00` 通过放弃对 P99 尾部延迟的严格承诺，系统能够接纳 80%~95% 在线率的廉价间歇性能源与次品晶圆，结合天气预报实现算力任务跨地域动态无缝漂移。
- `01:00:00` 当前大模型前向传播中的 MoE 参数稀疏激活已达极限（普遍低于 10% 乃至 1%），注意力机制中显存占用严重过载的 KV Cache 是下一阶段最具数量级压缩潜力的低效瓶颈。
- `01:07:02` 闭源前沿实验室支付的极高研发成本仅能维持 3 到 6 个月的微弱能力领先，伴随开源社区对 AI 生成代码的“隐性蒸馏”，前沿 Token 的超高溢价难以长期维持。
- `01:10:00` 将每日消耗万亿 Token 的门槛从 500 万美元压缩至数千美元，需要摆脱对高昂 HBM 的依赖，通过修改模型架构将 KV Cache 极限卸载至廉价的 Flash 闪存介质中。
- `01:12:42` 英伟达在纯 BF16 算力的每瓦性能演进已逼近物理天花板，而其选择扶持 CoreWeave 等 NeoCloud 而不亲自下场卖 Token，展现了避免与客户直接竞争以维系生态霸权的高超商业策略。
## Detailed walkthrough
### `00:00` 引言与嘉宾介绍：Token 工厂的构想
在节目开场，主持人 Patrick O'Shaughnessy 首先介绍了多家赞助平台（Ramp、Felix by Rogo 以及 WorkOS 在企业级 AI 基础设施中的核心地位），随后引出了本期嘉宾——Sail Research 创始人兼首席执行官 Neil Movva。Patrick 指出，这是一场针对“完整智能技术栈”（涵盖软件层、芯片架构到底层能源基础设施）极其详尽的解构。
当前市场正掀起一场 AI 军备竞赛，但 Neil 与 Sail Research 押注的却是一个截然不同的未来：AI Agent（智能体）不再只是在聊天框里向人类提供毫秒级实时回复，而是在后台独立、持续运行数小时、数天甚至数周。在这样的世界中，用户对低延迟的敏感度大幅降低，反而是**推理成本（Cost per Token）**决定了一切。Sail 的定位正是一家为这一范式打造的“Token 工厂”（Token Factory），其核心目标是在全行业实现无与伦比的极低 Token 成本。
- 本期核心主题：揭示 GPU 内部算力与成本的权衡抉择、边缘闲置算力与电力的捡漏策略（Scavenger Strategy）、对 NVIDIA 的反共识思考，以及前沿闭源实验室微弱领先优势难以维系的底层逻辑。
---
### `03:22` 重新定义推理：Sail 的核心使命与“丰沛”哲学
谈及 Sail Research 的现实业务，Neil Movva 用最纯粹的语言给出了定义：Sail 是一家面向开源大语言模型（LLM）的 Token 制造工厂。他们提供标准 API 服务，以极具杀伤力的击穿价向全球开发者输送 Token 算力；同时提供被称为“Sail Boxes”的长程 Agent 专用云端虚拟运行环境，专门适配那些需要连续执行数小时甚至数周的自主任务。
```mermaid
flowchart TD
accTitle: Sail Research 商业与技术架构
accDescr: 展示 Sail Research 如何通过超低成本推理与 Sail Boxes 虚拟机向开发者提供底座支持。
subgraph Infrastructure["底座支撑能力"]
A["超低成本推理 API<br/>（开源 LLM 极低单价供应）"]
B["Sail Boxes 虚拟机<br/>（专为数小时/数天长程 Agent 优化）"]
end
subgraph Mission["核心愿景"]
C["以 10 倍降价催生全新产品品类<br/>（Abundance 丰沛理念）"]
end
subgraph Downstream["下游范式转变"]
D["从'人类设定预算'<br/>转向'Agent 自主调度 Token 预算'"]
E["从'购买每秒 Token 产出'<br/>转向'按最终任务结果（Outcomes）交付'"]
end
A --> C
B --> C
C --> D
C --> E
```
Neil 强调，公司的核心哲学是**“丰沛”（Abundance）**。在商业演进史上，任何将生产要素成本降低 10 倍的技术，最终都会催生出完全独立的新产品范式。人类如今拥有了让机器进行“思考”的能力，而 Sail 的天职就是调动全球尽可能多的物理计算资源，以几乎无法被拒绝的成本将思考能力普惠化。对于“Token 成本是否仍是衡量智能的最优指标”，Neil 坦言这只是当下的衡量方式，未来计价必定逐步转向“交付结果”（Outcomes）。未来的长程 Agent 会像人类员工一样，根据任务难度自己“决定尝试多少次射门、调用多少次工具，并自主分配 Token 预算”，而非由人类工程师硬性限制每月的额度上限。
---
### `05:32` 长程 Agent 的崛起与“低延迟”迷局
当被问及为何市场尚未大规模转向该领域时，Neil 指出，整个行业目前正受两大浪潮驱动：
1. **开源力量的崛起与主权诉求**：越来越多的客户与企业极度渴望对核心智能底座拥有掌控权（Sovereignty），他们希望拥有模型权重、自主部署，不用担心闭源 API 随时被切断或更改协议。
2. **推理侧的盲目路径依赖**：过去两年中，如 Baseten、Fireworks、Together AI 等推理托管平台，普遍将所有研发重心倾斜在极限“低延迟”（Low Latency）推理上。Neil 直言，这种行业策略主要是被 Cursor 等核心代码补全客户的实时交互需求一手带偏的。
在 Neil 看来，尽管一年前这种策略是正确的，但当下逻辑已经完全不同。当人类盯着光标等待补全时，高吞吐（如 100 Tokens/秒）与极低延迟确实至关重要；但一旦任务交由后台自主处理，“最完美的延迟其实是零延迟（The best latency is no latency at all）”。当用户早晨醒来时，长程 Agent 已经在夜间彻底完成了复杂研究与工程落地，人类根本不需要经历实时等待的焦虑。
> “在协作流程中，如果人类必须不断提示、不断等待并给予反馈，那么人类自身就成为了限制 Agent 发挥更大价值的最大瓶颈。”
人类管理团队成员时，绝不会每隔 5 分钟去查岗一次，而是布置目标并以天或周为单位验收进展。人机协作的终局必然收敛于这种“人类尺度的时间周期（Human time scales）”。支持这一转变的技术支柱正是**测试期计算扩展（Test-time Compute Scaling）**——即向模型注入更多思考时间，其推理质量便会呈指数级提升。从早期初露端倪但表现平平的 Claude 3.5 Opus（实指具备长程思考能力的代表性架构），到当今能够在开源生态下稳健连续运行一小时且不出轨的模型，任务时长正在迅速拉长。Neil 预测，尽管目前长程后台任务仅占整体工作量的极小比例，但在未来三年内，长程自主任务将占据整个 AI 推理市场份额的 **90%**，因为人类的注意力是有限的瓶颈，而后台消耗 Token 的天花板是无限的。
---
### `08:47` 深度研究与网络安全：无限长程推理的杀手级场景
在具体落地场景上，Neil 列举了极具说服力的两大核心赛道：
* **超大规模深度研究（Deep Research）**：
如果用户需要针对某个复杂命题得出确定性结论，传统的 RAG 或实时检索最多只能遍历 100 到 1000 个网页源。而在长程背景下，Agent 可以通读并交叉比对 10,000 个甚至更多的数据源。例如其客户 Parallel Web Systems 正在推进的任务：针对全互联网构建动态实时权威索引并监测其变化。这种艾字节（Exabyte）规模的信息清洗、分析与提取，是传统的即时对话交互根本无法涉足的。
* **自主网络安全（Cybersecurity）与漏洞挖掘**：
在软件工程中，编写代码的逻辑是线性的，但攻破一段代码的组合路径却呈指数级暴增。网络安全领域正全面转向由专用 Agent 主导的进攻性渗透与防御性修补：
- 针对内存安全漏洞（Memory Errors）进行动态模糊测试；
- 深度分析代码中的业务逻辑漏洞（Business Logic Errors）；
- 全天候模拟并扫描网络协议栈脆弱点（Network Vulnerabilities）。
当行业出现新型安全评估框架时，安全团队倾向于调集海量背景算力，对历史沉淀的所有代码库运行数十种不同维度的自动化渗透测试。这需要部署大批高度定制、不知疲倦的专用 Agent，也正是廉价 Token 工厂最天然的施展舞台。
### `10:00` 网络安全演变为“工作量证明”与智能的锯齿状边界
在网络安全与自动化渗透测试领域，长程运行智能体正在引发范式转变。过去的自动化安全检测往往只是让工具扫描一遍源代码，而现在的做法则是直接让智能体动态搭建起真实的靶场环境，对应用程序发起持续不断的渗透测试。业界甚至由此流传开一个黑色幽默的论断——“网络安全本质上已经变成了一种工作量证明（Proof of Work）”。衡量一套软件系统到底有多安全，最真实的指标已经变成了“你在 Anthropic 的 API 上砸了多少美元去尝试黑入自己的系统”；只要你肯花大价钱调用世界上最顶级的模型攻击它而始终无法攻破，它就经受住了当前最严苛的检验。
与此同时，研究人员发现大语言模型在安全攻防中表现出极为鲜明的“锯齿状智能边界”（jagged frontier of intelligence）。前沿大模型找到的 Bug 集合，并不必然包含小模型所能发现的漏洞。在实际测试中，像 Claude Haiku 这样轻量级、低成本的模型经常能揪出前沿大模型完全忽略的盲区漏洞，反之亦然。这种能力上的不对称性，催生了一种采用异构模型进行密集采样测试的全新策略：通过调度大小各异的多样化模型组成自主攻击智能体集群，全天候全方位地“围殴”软件系统，以便工程师在黑客发现破绽之前抢先打好补丁。
```mermaid
flowchart TD
accTitle: 异构模型渗透测试循环
accDescr: 展示不同梯度的 AI 模型协同攻击软件系统以发现互补漏洞并生成补丁的闭环流程
subgraph Models ["异构模型集群 (采样攻击)"]
M1["轻量模型 (如 Claude Haiku)<br/>擅长捕捉特定边缘漏洞"]
M2["前沿大模型<br/>擅长复杂逻辑渗透"]
end
subgraph Target ["自动化靶场环境"]
ENV["动态搭建的应用测试环境"]
end
subgraph Defense ["防御与修复"]
BUG["汇总锯齿状漏洞视图"]
PATCH["自主生成并部署补丁"]
end
M1 -->|"发起多维度渗透"| ENV
M2 -->|"发起深度攻击"| ENV
ENV -->|"暴露独特缺陷"| BUG
BUG -->|"闭环修复"| PATCH
```
### `11:22` 极致廉价推断的终局：主动型智能体与可验证问题
当谈及超低成本、长程运行的推断能力一旦完全实现会催生哪些全新产品品类时，Neil 描绘了一个从“被动响应”跃迁至“主动出击”的个人计算未来。今天诸如 Siri 这类语音助手依然只是单点工具，极其被动且完全依赖人类的指令提示。而在推断成本极度低廉的世界里，智能体可以在后台 24 小时无休止地运行，静默通读用户一整天收到的每一封电子邮件和短信，建立起一部关于用户生活的百科全书式认知图谱。只要用户对机器的可靠性与隐私保护建立起足够信任，智能体甚至能在用户点亮手机屏幕的瞬间，精准预判其下一步意图并主动呈现在面前。Neil 明确指出，解锁这一体验的关键在于拥有极其廉价的算力底座：“你必须甘愿在没有任何明确回报承诺的情况下，肆意消耗 Token（spend tokens without any promise of return）。”
从更广阔的宏观视角来看，只要推断成本持续下降，人类就拥有了一种能够征服一切“可验证问题”（verifiable problems）的通用智能载体。可验证问题涵盖了大部分软件编写、形式化数学证明以及前沿科学发现，这些领域目前都受制于一个隐形的美元成本门槛——即解决它们到底需要调用多少 Token。如今这一门槛正被拉入可承受的现实区间：攻克复杂长程任务的推断开销正在从数百万美元锐减至数千美元，未来更有望压缩到几十甚至几百美元。当主持人提问这是否意味着未来人类的唯一局限只剩下“提出问题的水平”时，Neil 表示赞同，他认为模型正处在能够根据一个高阶抽象问题自主追查所有后续分支的临界点上，唯一的掣肘就是“Token 预算”，而这正是 Sail 决心从根本上解决的痛点。不过，对于文学写作质量、艺术美感等人类主观品味问题，Neil 明确划清了界限，认为主观品味根本不存在标准答案，Sail 会彻底放手交给人类，自身则全力聚焦于可量化的硬核问题。
- **工作量证明安全观**：系统安全性正被量化为“承受住了多少美元的 API 攻击”，安全审计预算直接等同于系统防御强度。
- **锯齿状能力互补**：大模型与轻量级模型在代码漏洞挖掘上并非简单的包含关系，多样化采样能大幅提高漏洞检出率。
- **主动式个人智能**：背景常驻运行的智能体需要消耗海量“无确定产出保证”的 Token，来实时建模用户行为并主动提供辅助。
- **可验证领域通解**：软件开发、数学定理推导与科学发现均属可验证问题，未来获取确定性科研结论的推断开销有望降至百元乃至十美元级。
- **边界划分**：明确放弃攻克主观审美与文学品味等“不可验证问题”，纯粹聚焦量化与硬逻辑场景。
### `13:50` 赞助商插播与“代币工厂”全栈架构的序幕
在进入深度架构探讨之前，节目插播了两个企业级产品赞助。首先是自动化安全与合规平台 Vanta，该平台目前为 Ramp、Cursor 和 Harvey 等超过 16,000 家快速成长的企业提供 24 小时全天候的审计就绪支持，其自主运行的 GRC 智能体能够监控供应商及 AI 工具引入的新风险，将供应商安全评估时间缩短多达 50%。随后是面向投资管理行业的统一运营平台 Ridgeline，该系统将投资组合会计、对账、报表、交易和合规整合在单一底层平台上，利用其深度嵌入的 AI 能力帮助资产管理公司淘汰落后的陈旧系统。
插播结束后，对话正式切入 Neil 创业的核心命题。主持人将话题引向 Neil 正在构建的宏大技术构想——建造一座能够以极低成本源源不断产出智能的“代币工厂”（Token Factory）。为了将推理 Token 的成本打到极限，Neil 并没有走常规的单点优化路线，而是将整个智能栈（full stack of intelligence）拆解为三个紧密交织的核心支柱：上层的软件架构（Software）、中层的芯片硬件（Hardware/Chips），以及底层的电力基础设施（Power）。从下一阶段开始，对话将逐层剖析这三者如何相互咬合以实现数量级的成本颠覆。
### `15:00` 从底层算子起步：最大化榨干 GPU 性能与 Tensor Core 的诞生
Neil Movva 谈及 Sail 打造“Token 工厂”的总体规划时强调，一切必须从软件端破局。想要在当今的数据中心里挖掘效率潜力，首要任务就是围绕 GPU 的峰值极限效率重新构建整套大语言模型（LLM）软件栈。目标非常明确：在使用相同 NVIDIA GPU 的前提下，压榨出远超全球其他任何机构的 Token 产出。而这一极限制程的核心起点，就是最底层的算子（Kernels）编程。这正是 Neil 的老本行，他的整个职业生涯都专注于 GPU 与底层算子开发，大学期间的第一份工作便是在 NVIDIA。
在此期间，Neil 亲眼见证了 Tensor Core（张量核心）是如何在芯片上“挣得立足之地”的。所谓 Tensor Core，本质上就是 GPU 上专门用来加速矩阵乘法的专用计算单元。至于为何矩阵乘法会成为现代 AI 计算的原子单位，Neil 直言宇宙中或许没有某种先验的宿命法则规定必须如此，但有一种精辟的解释是：矩阵乘法提供了一种极其简练的方法，能将两组数字混合在一起并产生有趣的相互作用；线性代数恰好成为了在数据中紧凑表达任意复杂关系的绝佳载体。
回顾历史，NVIDIA 长期以来是一家以游戏显卡为主导的图形巨头。直到 2010 年代中期，内部开始出现一些秘密的“臭鼬工厂”（Skunkworks）实验项目，探索如何让图形处理器适配机器学习任务。Neil 翻阅过当时主管的工作笔记，他们去参加 ICML、NeurIPS 等当时规模尚小的机器学习学术会议，敏锐地记录下前沿动向：深度学习正在兴起，研究人员正用游戏显卡来训练庞大模型。到了 2015 至 2016 年间，黄仁勋（Jensen Huang）展现出极强的战略定力，坚信这一用途未来必将爆发，果断决定将极其宝贵的芯片硅片面积（Silicon Die Area）划拨给这种新兴算力。在任何芯片设计公司，争夺硅片面积都是一场白刃战，设计团队极度谨慎，绝不愿冒将宝贵面积分配给错误技术的机会成本风险。经过激烈博弈，团队最终为第一代专用架构争抢到了大约 5% 至 10% 的硅片面积，用于加速当时在计算机视觉中占据统治地位的卷积（Convolution）运算，初代 Tensor Core 由此诞生。
```mermaid
flowchart TD
accTitle: GPU 芯片设计演进与软件栈优化
accDescr: 展示从传统游戏渲染到机器学习张量核心，再到底层算子极致调优的演化路径。
subgraph Hardware["硬件硅片演进 (Silicon Die Area)"]
A["传统游戏 GPU<br/>(专注于像素渲染)"] --> B["2015-2016 战略转向<br/>(关注 ICML/NeurIPS 论文)"]
B --> C["争夺 5-10% 硅片面积<br/>(第一代 Tensor Core 诞生)"]
C --> D["硬件硬化加速<br/>(矩阵乘法 / 卷积加速)"]
end
subgraph Software["底层软件调优 (Software Stack)"]
E["底层算子 Kernel 编写"] --> F["消除所有微架构瓶颈"]
F --> G["逼近 100% 光速 (Speed of Light)"]
end
D --> E
G --> H["从同一颗芯片上榨取出更多 Token"]
```
---
### `17:15` 追逐“光速”极限与极度节俭的 NVIDIA 文化
在硬件成型之后，软件团队的核心使命就是彻底榨干硬件潜能。Neil 所在团队的工作经历，让他深刻领会了 NVIDIA 内部奉为圭臬的工程信条——追逐“光速”（Speed of Light）。所谓“光速”，指的就是硬件在物理理论极限下所能达到的性能天花板：只要一块芯片在特定工作频率下理论上每周期能完成特定数量的乘法运算，工程师就必须打破一切软件架构与调度瓶颈，逼近这个理论极值。Neil 至今仍把这套工程准则带给 Sail 的每一位工程师：“我们追逐的是 100% 的光速极限。我根本不在乎相对竞争对手赢了多少百分比，我只在乎绝对性能——这块芯片理论上到底能做到什么，我们如何把它完全释放出来。”
谈到 NVIDIA 早期的公司氛围，Neil 分享了两个鲜明特质：极高的员工留存率与深入骨髓的节俭文化。
- **超强的团队稳定性与信念感**：他在 2015、2016 年合作过的许多硅片设计专家至今仍留在 NVIDIA。这是一群极其狂热且富有才华的工程师，他们坚信并行计算（Parallel Computing）这一范式，见证了芯片历代形态的演进，将其视作终身事业。
- **硬核的“牛奶俱乐部”轶事**：经历了 2008 年金融危机后的削减开支，硅谷科技公司常见的免费福利在 NVIDIA 被压缩到极致。不仅没有免费午餐，冰箱里甚至连咖啡用的免费牛奶都不提供。工程师若想喝咖啡加奶，必须自发加入内部的“牛奶俱乐部”（Milk Club），每人每月凑 1 美元，由俱乐部统一去 Costco 采购牛奶放进冰箱冷藏。这种极度节俭、不把资源浪费在与工程交付无关事情上的文化，深刻烙印在早期团队中（尽管 Neil 笑称 Sail 并没有把节俭做到不给牛奶的地步）。
---
### `19:15` 算力本性与现实错配：吞吐量机器被迫做低延迟交互
从深入理解芯片与底层软件调优的经验中，Neil 提炼出了对当下整个 AI 算力行业格局的本质洞察：GPU 从底层架构逻辑上说，纯粹是一台**高吞吐量机器（Throughput Machine）**。当给它喂入海量、并行的海量数据批次时，它的算力利用率最高、运行状态最舒适。
然而，过去几年整个 AI 行业落地演进的方向却与 GPU 的硬件本性背道而驰。行业目前最主流的落地形态是将 AI 当作需要即时响应的“交互式聊天机器人”（Interactive Chatbot）。在交互场景下，系统的核心指标变成了端到端低延迟，计算集群不得不为了向坐在键盘前的人类用户实时回传字符，而牺牲大批处理的高吞吐效率，这直接导致昂贵的 GPU 算力无法跑在最擅长、最经济的运行模式下。这也是 Sail 决心重新思考大模型系统架构、押注长程 Agent 场景的深层技术动因。
### `20:00` 吞吐量与延迟的权衡：公交车与私家车隐喻
谈及用户体验时，人们的第一反应往往是“别让用户等待，越快越好”。但对于 GPU 而言，拼命追求极速吐出 token 实际上是一种巨大的内耗。要让 GPU 处于算力被完全榨干的“快乐路径（happy path）”，与追求超低延迟的响应之间存在着根本性的技术冲突。整个硬件底层都在面临“吞吐量导向（throughput-oriented）”与“延迟优化（latency-optimized）”之间的残酷权衡。过去整个行业之所以毫不犹豫地选择牺牲算力利用率去死磕延迟，完全是因为大模型的应用形态被牢牢限制在 Chatbot（聊天机器人）模式中。Neil 预判，未来一年最具颠覆性的转变，将是行业从实时聊天机器人转向后台运行数小时乃至数天的前瞻性智能体（background agents）；而在那个世界里，围绕吞吐量来构建整套基础设施才是最具商业价值的选择。
计算机科学中有一条不可逾越的基础法则：要么“窄而快（narrow and fast）”，要么“宽而慢（wide and slow）”。GPU 本质上是为高度并行而生的巨型计算单元，如果想要追求极高吞吐量，最核心的手段就是批处理（batching）——将大量不同用户的计算请求打包成一个批次一次性塞给 GPU 运行。虽然这种做法能把 GPU 内部的所有计算单元彻底填满，但每一个处理步骤实际承担的净工作量也成倍增加。结果便是：单个批次中的每一个 token 或每位用户的请求，在 GPU 上逗留的时间都会显著拉长，因为它的数据必须和无数陌生人的流量绑在一起逐步推进。
为了让这个底层逻辑更加通俗，Neil 抛出了一个极为生动的交通隐喻：如果你想去旧金山市中心，你可以选择私家车，也可以选择乘坐公共汽车。私家车就像超低延迟服务，可以走最近的直达路线，随叫随走；而公交车为了同时运送几十上百人，必须走一条满足所有人的全局路线，一路上不得不反复停靠站台、等待乘客上下车，单趟耗时自然更长。但从整个城市的运力输送成本来看，公交车的单位人次能耗与开销远低于私家车。Neil 创办的 Sail，第一步目标就是在英伟达的 GPU 之上打造一辆极致高效的“超级公交车”。
```mermaid
flowchart TD
accTitle: 延迟优化与吞吐量优化的技术路线对比
accDescr: 对比了私家车模式的低延迟张量并行路径与公交车模式的高吞吐量流水线批处理路径
subgraph LatencyPath ["延迟优化路线（私家车模式）"]
A1["单个用户请求"] --> B1["小批次 / 零等待"]
B1 --> C1["张量并行 (NVLink) 切分大矩阵"]
C1 --> D1["高通信开销 + 次线性扩展 (4-5x)"]
D1 --> E1["超低单 Token 延迟 (Chatbot)"]
end
subgraph ThroughputPath ["吞吐量优化路线（公交车模式）"]
A2["海量并发请求"] --> B2["大批次组包 (Batching)"]
B2 --> C2["流水线 / 专家并行掩盖通信"]
C2 --> D2["算力核心 100% 满载饱和"]
D2 --> E2["极低单 Token 成本 (Agent 离线任务)"]
end
```
---
### `21:45` NVLink 的高昂代价与 Sail 的异构策略
深入到底层互连技术，英伟达最引以为傲的杀手级创新莫过于 GPU 之间的 NVLink 互连系统。它的带宽与速度极其恐怖，如果开发者想让一个超大矩阵乘法以最快速度完成，甚至可以直接将矩阵切成两半、乃至分摊到多达 8 张英伟达 GPU 上协同运算（即张量并行 Tensor Parallelism），最后再通过 NVLink 将各卡的结果归约（reduce）到一起。这确实是压榨出系统极限最低延迟的有效法门，但代价却是肉眼可见的边际效益递减（次线性扩展 Sublinear Scaling）。
把单张卡的工作量降到八分之一，整体速度绝不会快上 8 倍，实际往往只能换来 4 到 5 倍的提速。造成这种浪费的核心原因在于两点：其一是多卡频繁同步带来的通信开销（communication overhead）；其二是每一颗 GPU 在处理被切碎的极小计算瓦片（tile of work）时，硬件流水线的利用效率远不及处理整块大算力任务。对于执着于“首字秒出”的低延迟聊天场景，企业不得不咬牙买单；但 Neil 直言，这绝对不是他会走的技术路径。在 Sail 的体系中，他们倾向于采用专家并行（Expert Parallelism）或流水线并行（Pipeline Parallelism），通过算法层面的精巧编排，将通信延迟完全重叠并掩盖在庞大的计算耗时之中。
- **NVLink 的技术定位**：NVLink 本质上是专门用来拯救延迟性能的技术枷锁，对于追求超低延迟的推理任务来说不可或缺。
- **算力性价比（FLOPs per dollar）倒挂**：行业内许多其他芯片厂商虽然在高速互连协议上难以望英伟达项背，但其芯片在纯粹的基础算力矩阵乘法上表现卓越，单位美元能买到的 FLOPs 远高于英伟达。
- **异构接入逻辑**：由于 Sail 放弃了极度依赖 NVLink 的张量并行，转向了不依赖紧密互连的并行范式，这使得他们能够跳出英伟达生态，将那些缺乏顶级互连但单位小时拥有与运营成本（cost per hour to own and operate）极低的高性价比第三方芯片纳为其用。
---
### `24:58` 内存层级之争：SRAM 与 DRAM 的物理局限
在硬件设计的前沿地带，诸如 Cerebras、Groq 等跳出传统 GPU 范式的挑战者们，正在押注截然不同的内存层级架构（Memory Hierarchy）。它们不再遵循传统的计算与显存解耦路线，而是试图通过在芯片上疯狂堆叠 SRAM，来为大模型的权重（weights）和 KV Cache（键值缓存）构建极致带宽的极速内存。Neil 借此深入拆解了芯片存储技术的两条根本分支：集成在逻辑晶圆上的 SRAM 与独立工艺制造的 DRAM。
SRAM 直接使用台积电（TSMC）等代工厂的标准逻辑工艺制造。如果开发者需要，可以直接在主逻辑裸片上调用标准单元库（standard cell library）印刷出 SRAM 阵列。SRAM 采用经典的 6 晶体管单元（6T transistor cell）结构，这是一种静态自锁的晶体管排布，写入一个 bit 后，只要保持供电，无需任何外部干预即可永久保持状态，读取速度快如闪电。然而 SRAM 的死穴在于占用硅片面积极其惊人。即使是一颗达到台积电光罩极限（reticle limit）、尺寸高达 800 平方毫米的顶级逻辑芯片（如英伟达 Blackwell），若将整张硅片全部做成 SRAM，撑死也只能提供几 GB 的存储空间。
```
【SRAM 6T 单元】逻辑工艺直接集成，6 晶体管静态维持状态 -> 极快读取、超低延迟，但面积巨大、容量上限极低（MB~GB 级）
【DRAM 1T1C 单元】存储专用工艺（美光/海力士），电容充放电 -> 密度极高（GB~TB 级），但电荷持续泄漏，须每 50ms 动态刷新
```
与此相反，美光（Micron）、SK 海力士（SK Hynix）和三星（Samsung）等存储巨头走的是 DRAM 路线。DRAM 不靠晶体管锁存状态，而是依赖微型电容器中存储的微小电荷来表达二进制。电容存在不可逆的物理漏电，因此写入电荷后必须由内存控制器每隔大约 50 毫秒扫描并刷新（refresh）一次所有数据。Neil 用极为形象的场景描述了这一过程：内存控制器就像杂技演员一样，必须在半空中永不停歇地同时玩耍并抛接数十亿个球（"juggling billions of balls in the air"）。
尽管 DRAM 控制复杂且速度慢于 SRAM，但其电容结构带来的存储密度高出了几个数量级。通过将 DRAM 多层垂直堆叠形成高带宽内存（HBM），英伟达 Blackwell 能在主逻辑裸片周围围上一整圈、高达 288 GB 的 HBM 显存；而与其紧挨着的 Blackwell 核心计算裸片内部，SRAM 实际只有区区 500 MB 左右。两者的存储密度相差了整整近三个数量级。
---
### `27:30` 晶圆级引擎与不可预测的 KV Cache 难题
SRAM 拥有一项 HBM 永远无法企及的绝技：它与芯片内部执行矩阵乘法的算术逻辑单元（ALU）在物理层面零距离接触。这种微米级的物理邻近性赋予了它骇人听闻的吞吐速度。英伟达 Blackwell 的顶级 HBM 内存带宽大约在 10 TB/s 量级，而 Cerebras 最新的 Wafer Scale Engine 3（WSE-3，晶圆级引擎）从 SRAM 抓取数据的聚合带宽则高达不可思议的 21 PB/s（拍字节/秒），整整高出几个数量级。
Cerebras 解决 SRAM 容量缺陷的手段极度激进。他们打破了台积电单颗芯片 800 平方毫米的光罩面积壁垒，不将整块 300 毫米晶圆切割成数百颗独立芯片，而是利用划片槽（scribe lines）直接将晶圆上的所有计算单元全数相连，把整张晶圆变成单一的一颗超级大芯片。这样一张巨型晶圆能集成约 50 GB 的超高速 SRAM；若将多张晶圆串联成流水线，整套系统就能拥有多达 1 TB 的极速内存池。凭借 21 PB/s 的超大带宽，系统能在约 1 毫秒内将类似 Kimi 这样万亿参数大模型的所有权重完整刷入逻辑核心一遍，从而轻易打出每秒 1000 个 token 的神级生成速度。
- **混合架构的必然宿命**：Neil 预测，纯 SRAM 方案虽然惊艳，但这类硬件的终局必然是走向某种“混合架构（Hybrid）”——必须用 Cerebras 的超高速计算核心搭配具备更大容量的外挂存储。
- **KV Cache 内存黑洞**：虽然 Cerebras 可以靠堆叠晶圆硬生生塞下万亿参数模型的静态权重，但它对动态且无法预知上限的 KV Cache 束手无策。
- **上下文膨胀的物理代价**：在大语言模型的交互中，用户输入的每一个 token 在整个会话周期内都必须被固化保存在上下文窗口中。当对话进行到 100,000 个 token 时，预测第 100,001 个 token 依然需要对前面 10 万个 token 的表征进行注意力寻址。伴随并发用户数的激增与对话长度的伸展，KV Cache 的显存开销会呈爆炸式动态增长，这在寸土寸金的超高速 SRAM 上是无法承受之重。
### `30:00` KV Cache 与长上下文退化的挑战
在大型语言模型的运行过程中，用户发送的每一个 token 都会生成并追加到 KV Cache（键值缓存）中。随着对话不断变长，KV Cache 的体积迅速膨胀，往往会变得比模型自身的静态权重（Weights）还要庞大。Neil Movva 将模型权重比作“结晶化的知识”（crystallized knowledge），而 KV Cache 则代表当前对话所特有的“动态知识”（dynamic knowledge）。很多人会发现在长时间多轮对话之后，模型的表现开始出现退化甚至混乱，这往往就是长上下文带来的技术瓶颈所导致的。
更深层的原因在于模型训练的分布差异：虽然前沿模型声称支持极长的上下文窗口，但在预训练阶段，模型绝大多数时候处理的依然是相对较短的序列（例如 8,000 或 16,000 个 token）。即使在长序列微调时覆盖过 20 万 token，这也绝非它的核心强项。因此，前沿实验室（Frontier Labs）一直面临着一个长期的持久战：如何确保模型在 20 万 token 长度下的智能水平与在 1 万 token 时完全一致。尽管 Anthropic 等团队率先将上下文窗口推到了 100 万 token，但 Neil 直言自己在日常使用（如编写代码）时，早在达到上限之前就会频繁输入 `/compact` 来压缩上下文，因为强行撑满全长体验并不好。这种极速、超低延迟的纯片上架构，终究会撞上 KV Cache 这个巨大痛点：即使能在静态权重存储和访存上做到极致性能，无法承受的上下文缓存开销也会成为芯片架构致命的“肉中刺”。
### `31:39` 异构芯片生态：片上加速器与 GPU 的混合协同
展望未来三到五年异构芯片市场的格局，Neil 认为像 Cerebras、Groq 等主打超大片上 SRAM（On-chip Memory）的芯片，其本质角色应当是“加速器”（Accelerators）。它们最理想的部署模式并非单打独斗，而是与搭载离片大容量内存（Off-chip Memory，如 HBM）的传统 GPU 协同工作。这种组合兼具两者的长处：离片内存提供海量容量以容纳上下文和 KV Cache，片上内存则提供极致带宽与运算速度。
Neil 提出了一个尖锐的观点，称之为 **Transformer 的“原罪”**（the original sin of Transformers）：它将一个本质上受限于内存带宽的层（Attention 注意力层）与一个受限于算力瓶颈的层（MLP 多层感知机层）强行并置在一起。在足够大的 Batch Size 极限下，存储世界知识的 MLP 主要是计算密集型（Compute-Bound）的矩阵乘法，而动态捕捉上下文关系的 Attention 层在长序列下则是典型的访存密集型（Memory-Bound）。要用单一架构兼顾这两种截然相反的负载极其困难。相对而言，GPU 在两者之间维持了平衡，但异构分工是更优的解法：让拥有超高带宽访问能力的 Cerebras 或 Groq 芯片专门负责托管 MLP 权重，而让容量充沛的 GPU 专门负责承载长序列 Attention 和 KV Cache。
```mermaid
flowchart TD
accTitle: Transformer 架构与异构硬件匹配示意图
accDescr: 展示 Transformer 的原罪以及如何将 Attention 与 MLP 分别适配到 GPU 与专属加速器
subgraph TransformerLayers["Transformer 的内在矛盾（原罪）"]
Attn["Attention 注意力机制<br/>（动态适应当前对话，极致长序列下 Memory-Bound）"]
MLP["MLP 多层感知机<br/>（编码世界知识与矩阵乘法，大批次下 Compute-Bound）"]
end
subgraph HardwareMapping["异构硬件分工"]
GPU["传统 GPU / 离片内存 (HBM)<br/>- 优势：大容量<br/>- 负责：承载注意力层与膨胀的 KV Cache"]
Accelerator["Cerebras / Groq / 片上 SRAM<br/>- 优势：极速带宽、低延迟<br/>- 负责：承载 MLP 权重与矩阵计算"]
end
Attn --> GPU
MLP --> Accelerator
```
- **GPU 的角色**：凭借庞大的 HBM 显存容量支撑超长上下文窗口和海量 KV Cache，化解内存容量瓶颈。
- **SRAM 架构芯片（Cerebras/Groq）的角色**：凭借无与伦比的访存带宽和低延迟专攻 MLP 计算与静态权重。
- **业界风向**：英伟达与 Groq 等硬件体系的发展态势正在印证这种将计算密集型与访存密集型拆解异构的融合趋势。
### `33:19` Transformer 为何能横扫一切与扩展律的胜利
回顾 2017 年以来的变革，Transformer 架构之所以具有统治地位，核心在于它能够以极高的效率处理大规模无监督数据。Transformer 的精髓在于将任意序列数据输入模型后寻找其内在规律，而其中的标志性创新——注意力机制（Attention），允许模型在推理的每一步动态回溯整个输入序列，重新评估所有历史信息，精准挑选出与当前预测最相关的部分。由于它对学习任意序列模式极其包容，而人类创造的最核心、最精妙的序列数据正是自然语言，这直接促成了 Transformer 在语言领域的霸权。
更宏观地看，Transformer 最伟大的优势在于其近乎无限的**可扩展性（Scalability）**。传统的机器学习算法（如支持向量机 SVM 或经典线性模型）通常只有几千到几十万个参数；后来的计算机视觉虽然进入深度学习时代，但当时 1.5 亿参数就已算得上庞然大物。传统方法往往依赖大量的人工先验设定，而 Transformer 摒弃了人为设定的归纳偏置（Priors），信奉“只要序列里有规律，我就通过堆砌参数和算力把规律找出来”。它是连接百万级参数跃迁至数万亿级参数的关键桥梁。Neil 形象地将 Transformer 称作一块“巨大的海绵”：你只要将提供给它的计算量增加 10 倍，它就能忠实地吐出相应的对数级性能提升。只要数据和算力两大要素仍在增长，这套优雅而强悍的扩展律（Scaling Laws）就会继续牢不可破地统治下去。
### `35:00` Transformer 架构的威力：全对全注意力与“先过拟合后压缩”
谈及 Transformer 为何能横扫一切机器学习领域，Neil Movva 指出其核心优势在于极强的通用学习能力：几乎任何你能喂给它的数据集，它都能有效吸收。过去业界尝试过各种替代注意力机制（Attention）的新方案，但 Transformer 最无可替代的特性在于它能够对序列中任意两个 Token 之间的关系进行建模（Pairwise Relationship）。序列中的每一个 Token 都可以直接关注到其他任何 Token，这意味着序列中只要潜藏着某种模式或关联，Transformer 就必然能将其捕捉出来。尽管在很多实际任务中，也许并不真的需要如此奢侈的“全对全”（All-to-all）建模，但 Transformer 提供了这种保底能力；在人类找到更优的剪枝方法或更具选择性的信息建模范式之前，注意力机制依然是现存最强大的算子。
这种建模逻辑也呼应了计算机视觉早期摸索出的工程智慧。Neil 引用了前特斯拉与 OpenAI 科学家 Andrej Karpathy 的经典名言：面对一个全新的数据集，你的第一反应应该永远是用一个严重过参数化（Over-parameterized）的模型去强行过拟合（Overfit）现有数据。过拟合并非终点，而是试金石——它证明了数据中确实存在可被建模或记忆的规律，证明了你的学习算法是奏效的，并且模型有能力容纳这些知识。一旦成功实现过拟合，下一步才是“压缩”（Compression）。真正的泛化能力（Generalization）正是在压缩的过程中诞生的。训练模型从来不是为了让它死记硬背眼前的样本，但在能够泛化之前，必须先通过倒推的方式，在尽可能精简的参数量中提炼出普适的通用规律。
### `36:43` 数据的终局：从互联网“单次红利”走向强化学习环境（RL Gyms）
在讨论数据的未来走向时，Neil 抛出了一个极为精辟的论断：“互联网对人类而言，是数据上的一次性补贴（A one-time subsidy on data）。”人类免费获得了这笔庞大的高阶数字资产，其质量极高。以文本量计，互联网上最顶级的优质文本大约在 30 万亿（30 Trillion）Tokens 左右；如果把标准放宽，广义的高质量文本大约有 300 万亿（300 Trillion）Tokens。但现实是残酷的：前沿大模型已经把整个互联网的数据来回翻看了无数遍，来自于人类在互联网上留存的原生数据红利已经彻底见顶。
数据的下一个阶段，是模型在强化学习虚拟环境（RL Environments / Gyms）中实现自我进化。过去那种依赖普通用户与 ChatGPT 互动、点赞点踩的传统偏好数据（RLHF）正在迅速贬值。Neil 直言，当下部署的主流模型平均智力水平已经远远超过了提供反馈的普通人（"Everyday Joe"）。面对超越自己的智慧体，未经挑选的普通人类偏好信号已经毫无价值，业界现在只需要真正顶尖专家的指导。正因如此，未来的数据生成形态变成了为模型设定一个极其严苛但“结果可被自动验证”（Verifiable）的任务环境：
- 将模型置于一个隔离的虚拟训练场中，让其独立攻克难题；
- 系统能够以客观标准实时量化其进展（例如代码是否跑通、单元测试是否通过、数学定理证明是否成立）；
- 甚至直接给智能体配备一套完整的虚拟电脑环境，模拟人类白领进行真实工作流操作。
在这一范式下，“环境本身就变成了源源不断的数据”。面对这是否只是另一个阶段性红利的疑问，Neil 认为这远比互联网数据更为深刻：实现 AGI（通用人工智能）的最现实路径，就是不断把一个个垂直领域的专用智能堆叠起来，直至所有能力空白被填满。只要任务具备客观的自我打分与自校准机制，模型就能在验证环境中实现无限递归的自我提升。这也是为什么顶尖前沿实验室（Frontier Labs）的开支结构发生了剧烈转变——他们不再砸巨资购买静态的人类数据，而是将预算排山倒海般倾注于构建这些能够支撑模型自主迭代的强化学习环境。
```mermaid
flowchart TD
accTitle: Paradigm Shift in AI Training Data
accDescr: A diagram contrasting the exhausted Internet static data subsidy with verifiable RL environments for recursive self-improvement.
subgraph Phase1["第一阶段：互联网一次性补贴（已见顶）"]
A["互联网公开数据<br/>~30T 至 300T Tokens"] --> B["预训练与监督微调<br/>（模型已多次通读整个网络）"]
C["普通用户反馈 / 点赞点踩<br/>（已低于模型基线水平）"] --> B
end
subgraph Phase2["第二阶段：可验证的强化学习环境（自我迭代）"]
D["严苛且可验证的任务<br/>（写代码 / 数学定理 / 虚拟电脑操作）"] --> E["隔离的 RL Gym 训练场"]
E --> F["客观自动评分系统<br/>（Pass/Fail 单元测试与度量）"]
F -->|"递归式自我提升<br/>（Recursive Self-Improvement）"| E
end
Phase1 -.->|"数据枯竭与质量瓶颈"| Phase2
```
### `40:48` 深入底层算子：打破标准内核与算子融合（Kernel Fusion）
随着对话转回底层系统，主持人引导 Neil 深入拆解如何通过底层软件层面的深度掌控来挤压硬件效率。很多人误以为 GPU 内核（Kernel）的优化已经是一个被解决的封闭领域——学术界与工业界已经诞生了像 Tri Dao（FlashAttention 算法的发明人）这样的顶尖天才，其打造的 FlashAttention 内核早已成为现代深度学习与 Transformer 模型的绝对基石。
然而，现实工程环境远比理想状态破碎。一旦模型架构脱离了标准黄金路径（Happy path），哪怕只是引入极其细微的改动——例如微调旋转位置编码（RoPE, Rotary Position Embedding）的实现方式——现存高度优化的标准内核就会瞬间失效或不兼容，迫使工程师必须亲自下场对底层 CUDA 代码打补丁。
Neil 解释道，内核（Kernel）本质上就是在 GPU 上运行的一个独立程序。在传统硬件架构中，内核通常以库（Library）的形式提供，每个内核各司其职，范围极小：
- 矩阵乘法（Matrix Multiply）对应一个专用内核；
- 就连两个张量相加（Tensor Addition）这样基础的算术操作，也要单独占用一个内核；
- 当计算流水线由无数分散的小算子拼凑而成时，GPU 将在显存搬运（Memory Bandwidth）上浪费海量时间，这也正是“算子融合”（Kernel Fusion）登上舞台的核心动因——将多个连续的计算步骤熔铸为一个紧凑内核，彻底消灭不必要的显存来回读写开销。
### `40:00` 编写 GPU Kernel 与算子融合：AI 时代的自动化工程趋势
在深度学习推理中，算子融合（Operator Fusion）是一个极其核心的概念——例如将一个矩阵乘法的结果直接与另一个矩阵乘法的结果相加，把原本需要在高带宽显存（DRAM）中来回写入与读取的数据流，直接合并在片上 SRAM 中作为一个统一的 Kernel（算子核心）一次性执行完毕。主持人为此提出疑问：既然这种逻辑如此清晰，为什么当前仍然有大量人类工程师在手动编写优化 Kernel？甚至像 FlashAttention 的发明者 Tri Dao 这样顶级的 Kernel 工程师在业内拥有如此崇高的声望？如果 AI 极度擅长代码工程，这难道不应该早已被自动化了吗？
Neil Movva 解释道，虽然他不便代表 Tri Dao 发言，但他从 Tri Dao 身上学到的核心认知是：**现代工程师其实已经不应该纯粹手动手写底层 Kernel 了**。在 Sail，团队的工作方式被称为“在白板上写 Kernel”：工程师首先在白板上做概念设计，明确底层机器的硬件资源调度与数据流路径，随后用极其凝练、精确的自然语言向大模型描述这些策略（包括输入输出格式、硬件线程块与 Tensor Core 的分派策略等），最终由前沿大语言模型直接生成并落地具体的底层 CUDA 代码。
Neil 坦言，模型在 Kernel 工程上的表现正在飞速进化，这绝不会成为任何一家基础设施初创公司不可动摇的技术护城河。就像 GPT-5 或同级别前沿模型的推出会“水涨船高，托起所有的船”（A rising tide lifts all boats）一样，最顶尖的 AI 实验室内部早已在大量利用全自动化流程生成 Kernel。试图单单在“编写算子”这一局部软件层面建立绝对竞争壁垒是不切实际的，整个行业最终都处于前沿大模型能力释放的下游。
```mermaid
flowchart LR
accTitle: Kernel Engineering Automation Workflow
accDescr: 展现从白板概念设计到自然语言提示词再到模型自动生成GPU Kernel的代码流水线
subgraph Design ["人类工程师（概念架构）"]
A["白板架构设计<br/>数据流与片上缓存规划"] --> B["精确自然语言描述<br/>调度策略与 I/O 规约"]
end
subgraph Automation ["模型工程执行"]
B --> C["前沿代码大模型"]
C --> D["高性能融合 Kernel<br/>CUDA / 算子代码落地"]
end
subgraph Hardware ["硬件执行"]
D --> E["GPU 物理调度<br/>避免反复读写 DRAM"]
end
```
- **白板到代码的范式转移**：工程师角色从逐行敲写底层寄存器与共享内存指令，转变为高维度的系统概念架构师，底层具体实现完全交付给大模型。
- **算子融合消除显存墙**：通过将相依的操作融合成单个执行 Kernel，大幅减少数据进出 DRAM 的带宽开销与延迟。
- **软件护城河的侵蚀**：底层极其贴近硬件物理极限（Speed of light）的软件工程优势，正迅速被更强的前沿大模型通用编程能力抹平。
---
### `42:00` 煤炭与 Blackwell：GPU 真实利用率瓶颈与机架级计算
主持人以能源发展史打比方：人类利用一块煤炭所蕴藏的能量，其转化利用率在几百年来从 10% 提升到了 95% 以上；那么对于像 NVIDIA Blackwell 这样的尖端 AI 芯片而言，我们当前的硬件利用效率究竟处于什么水平？Neil Movva 指出，评价 GPU 的效率需要拆开来看：当 GPU 处于它“最开心的舒适区”——即处理超大维度的矩阵乘法时，芯片能够跑到标称峰值利用率的 70% 左右。而无法触碰 100% 纸面算力的瓶颈其实已经不再是软件调度，而是严格受制于物理层面的供电限制与发热节流（Power and Thermal Throttling）。
然而在现实世界的大模型推理中，Transformer 模型绝大部分时间根本不会自然处于这种大 Batch 的矩阵计算舒适区。Neil 强调，Sail 这类推理引擎的核心使命，就是在芯片周围构建一套庞大的吞吐聚合架构，源源不断地向 GPU 灌入超大规模的并发任务批次，强行将硬件保持在最高效率的计算区间。
与这一趋势相伴的，是过去一年底层计算范式的深刻演进：**人们不再针对单张独立 GPU 或单一主板进行编程，而是必须将整个机柜（Rack）、乃至整个数据中心视为单台计算机进行统筹编程**。以 NVIDIA 最新的 GB200 NVL72 为例，它直接以配备 72 颗 GPU 的完整高密度液冷机架系统交付。在当前格局下，无论是追求绝对的最低时延，还是压榨极限的最高吞吐量，NVL72 都是统治级的计算单元形态，而如何以最高效的方式对其全机柜级互连总线与算力进行整体编程，已成为基础设施领域的全新竞赛。
- **70% 峰值算力物理墙**：在最优的大规模矩阵运算下，GPU 能发挥约 70% 的峰值性能，剩下的空间受限于供电墙与散热温控极限，而非单纯的软件缺陷。
- **打破单卡视角**：从过去针对单张 PCIe/SXM 显卡编写算子，转变为必须针对 NVLink 全机架互联拓扑进行统一的机架级计算（Rack-scale Computing）编程。
- **GB200 NVL72 的双重优势**：全机柜级紧密互联架构使其同时成为目前获取超低延迟（单请求多卡切分）与超高吞吐（超大并发吞吐处理）的最前沿硬件标准。
---
### `44:05` “毒品交易”般的芯片市场：NVIDIA 的分配策略与初创企业的信任门槛
针对当前最顶尖 AI 算力芯片（如 Blackwell 系列）供不应求的现状，业界流传着一种比喻：最尖端芯片的流通生态几乎就像是一个极度饥渴、高度紧缩的地下“毒品黑市”，所有玩家都在不计代价地四处寻觅货源。在被问及这种市场感知是否属实时，Neil 揭示了隐藏在狂热背后的两重深层商业现实。
首先，NVIDIA 对自身芯片的分配拥有极长期的战略考量。面对井喷式的市场需求，NVIDIA 完全可以像传统大宗硬件供应商那样肆意拉高售价，让供需曲线在更高的价格均衡点相交。然而 NVIDIA 心知肚明，“算力就是当今世界的终极权力”（Compute is power today）。如果单纯遵循价高者得的逻辑，放任极少数现金储备极其雄厚的超大规模云厂商（Hyperscalers）扫空所有最先进芯片，这些巨头将在未来积累足以反噬 NVIDIA 定价权的恐怖生态话语权。因此，NVIDIA 在芯片分发配额上表现得极具政治手腕与战略克制，绝不会让单一买家垄断未来。
其次，在算力租赁市场中，“信任与商业关系”（Relationships）起到了决定性的风控作用。算力中心与重资产机房极度厌恶风险，当一家成立仅几个月的全新初创公司声称“希望签署一份长达 3 年至 5 年、租赁 10,000 张 Blackwell 显卡的巨额订单”时，出租方第一反应是对方随时可能破产违约。在动辄数亿美元的资本承诺面前，没有深厚的信誉背书与极为过硬的行业关系网络，仅凭纸面意向几乎不可能撬动任何规模化的顶级算力配额。
> "Compute is power today. They're quite strategic about how they allocate compute."
> （在今天，算力就是权力。NVIDIA 在如何分配这些算力上有着非常深刻的战略考量。）
- **防范下游巨头坐大**：NVIDIA 拒绝纯粹通过激进加价将所有 Blackwell 算力出清给出价最高的单一巨头，而是通过精细的配额分配维持生态制衡。
- **初创公司的信用赤字**：面对数年期、千万至数亿美元级的芯片长租合约，算力持有方极度忌惮初创公司的履约能力，使得芯片获取严重依赖传统信任纽带与深层行业人脉。
- **两极化的算力市场**：最尖端的先进制程算力被战略管制与信用壁垒层层封锁，迫使非巨头玩家必须在次级芯片市场或长尾异构硬件上寻求差异化突围。
### `45:00` “没有劣质芯片，只有错误定价”：非英伟达生态的套利机会
谈及芯片市场的竞争格局，Neil Movva 提出了一个鲜明而务实的核心哲学：“没有劣质芯片，只有错误的定价”（"There's no bad chips. There's only bad pricing."）。只要价格合适，Sail 愿意设法让任何芯片发挥效用。以 AMD 为例，Neil 认为其硬件本身性能出色，真正的症结在于外界普遍缺乏对其编程和调优的能力。英伟达不仅硬件强劲，其自研算子（kernel）和软件栈也极为完善，外部团队能榨出的额外超额收益（alpha）相对有限；而 AMD 等芯片厂商在开箱即用的底层 kernel 优化上投入较少，市场对 AMD “性能不如英伟达” 的固有偏见，在 Neil 眼中反而是天籁之音——别人越是低估、弃用这些芯片，Sail 就越能以极具吸引力的价格大举扫货，并依靠顶级 kernel 团队榨取极致性能，以此实现算力套利。
不过，这种套利窗口正在悄然演变。Meta 和 OpenAI 等大买家已开始大批量采购 AMD 芯片，导致 AMD 的供应同样日趋紧张。与此同时，市场上涌现出大量新一代芯片初创公司，例如 Etched、SambaNova 以及 d-Matrix。Neil 指出，这些新兴厂商面临的最大命门在于规模化能力：它们能否在台积电（TSMC）争夺到足够的晶圆配额以实现量产？Sail 的策略是保持极端敏锐，一旦市面上出现新芯片，便立刻评估并尝试锁定其大量供给。
与背负沉重历史包袱的传统云厂商不同，Sail 没有被特定芯片绑定的数据中心基础设施惯性，团队乐于拥抱 Google TPU、AWS Trainium 等各类专用硅片。Neil 坚信每种硬件架构都具备独特的比较优势，Sail 的核心竞争力在于通过构建异构推理服务系统（heterogeneous serving system），无论硬件多么棘手，都能将其嵌入最契合的工作负载并把效率榨取到极致。
```mermaid
flowchart TD
accTitle: 芯片套利与异构推理策略
accDescr: 展示 Sail 如何将市场低估的替代芯片转化为高利润推理算力的流程。
subgraph HardwareSourcing ["芯片采购与市场认知"]
A["非英伟达/新兴芯片<br/>(AMD, TPU, Trainium, Etched 等)"] --> B["市场偏见与编程门槛<br/>(认为不好用 / 定价被低估)"]
B --> C["Sail 低价批量锁定供给<br/>(寻找价格洼地与比较优势)"]
end
subgraph SoftwareOptimization ["底层软件工程重构"]
C --> D["顶级底层 Kernel 优化"]
D --> E["定制异构调度与推理栈<br/>(Heterogeneous Serving)"]
end
subgraph Commercialization ["算力交付与商业变现"]
E --> F["压低单 Token 生产成本"]
F --> G["对外输出极具价格竞争力的推理服务"]
end
```
- **核心采购原则**：任何算力只要单价足够低，就能通过深度优化转化为正向毛利，关键在于挖掘硬件本身的相对优势。
- **新兴 AI 芯片玩家面临的瓶颈**：初创芯片公司能否立足，根本上取决于在台积电（TSMC）能够切下多大的先进晶圆产能配额。
- **异构推理架构**：不拘泥于单一芯片标准，针对 TPU、Trainium 及各家定制加速芯片分别适配特定算子与推理管线。
---
### `47:56` AI 繁荣与互联网泡沫的区别：投机性基建 vs. 确定性推理消耗
面对资本市场对半导体行业的恐慌情绪——半导体在标普 500 指数中的权重已从历史常态的 2%~4% 飙升至 19%~21%，诸多投资人担心重演类似当年思科（Cisco）与英特尔（Intel）见顶崩盘的“均值回归”——Neil 给出了截然不同的历史观察。出生于 1997 年的 Neil 直言，自己的母亲在 2000 年互联网泡沫前夕就在英特尔工作，他见证了当年网络硬件繁荣与破灭的轨迹。
在 Neil 看来，当前 AI 浪潮与 25 年前互联网泡沫有着根本差异：当年的网络基础设施投资充斥着纯粹的投机色彩，厂商基于“对未来海量用户需求的虚幻预期”铺设了巨额冗余带宽，但实际需求长期未至；而当下对 Token（AI 生成内容的基本单位）的消耗是完全非投机性的（"it's no longer speculative"）。用户购买并消耗 Token 是因为它们能立即产出实际价值——“没有人会囤积 Token，买来就是为了即时使用”。
这种本质差异甚至打破了前两年的 AI 投资逻辑。在 2023 至 2024 年英伟达 Hopper 架构（H100/H200）严重断货期间，大部分开支用于模型预训练，而预训练投资本身具有投机性；但现阶段，如 Claude Code 等编程智能体工具的火爆，甚至迫使服务商对用户的调用支出设限。Neil 强调，推理开支（Inference spend）正在呈现单调递增态势，这种来自终端生产力工具的即时刚需，构成了当前算力需求坚不可摧的基本盘，也直接驱动着底层数据中心、机柜与集群架构走向重构。
- **标普 500 半导体权重**：从长期历史均值的 2%~4% 激增至 19%~21%，引发市场对“算力大宗商品化”与周期性崩盘的疑虑。
- **投机性支出 vs. 生产性支出**：
- 2000 年代光纤与网络设备：面向远期未经验证的用户假设，属于投机性超前基建；
- 2023—2024 年大模型预训练：押注未知模型能力跃迁，兼具科研与投机属性；
- 当前推理（Inference）支出：企业与开发者为即时可用的自动化工作流买单，即产即销，不留过剩库存。
- **推理需求的刚性增长**：长程智能体与编程 Agent 的普及，正将推理需求推向以“每天万亿 Token”为尺度的指数级单调增长轨道。
### `50:00` 训练与推理的基建鸿沟：从集中式超大型集群到分布式 1 兆瓦站点
AI 行业中最保守的角色莫过于基础设施层参与者——无论是数据中心运营商，还是更上游的台积电（TSMC）等芯片制造基础设施方。过去两年的数据中心完全是围绕“训练”构建的。训练工作负载是推理的超集：任何能够用于大模型训练的算力集群都能轻松胜任推理，但反过来却无法成立。这两者最核心的区别在于互连网络——芯片之间需要投入多少网络通信带宽，以及整个集群必须做多大。
超大规模数据中心实际上存在显著的“规模不经济”（diseconomy of scale）。在单一数据中心塞进 10 万颗 GPU，其工程难度与单位成本远比建一个 1 万颗或 1,000 颗 GPU 的机房高昂得多。如今行业在讨论数据中心时，衡量的单位直接变成了“兆瓦（MW）”甚至“吉瓦（GW）”。现实情况是，在美国几乎已经不可能轻松建设一座 1 吉瓦的数据中心，甚至 100 兆瓦也变得极端艰难，唯有极少数顶级巨头才拿得到指标；10 兆瓦大致处于目前开发能力的边界；但如果是 1 兆瓦的电力规模，供应则极为充沛。
市场上存在大量碎片化、非集中的总电力底仓。过去做训练数据中心的人对此毫无兴趣，因为训练要求所有算力高度集中在物理同一处，没有人愿意处理跨数据中心训练的超高网络延迟和通信损耗。当前市场依然存在惯性认知滞后，许多数据中心开发者仍到处搜寻 10 兆瓦或 100 兆瓦的大型站点。然而，推理负载根本不需要这种集中度，分布式、小型化的 1 兆瓦数据中心网络完全可以满足需求。
```mermaid
flowchart TD
accTitle: 训练集群与分布式推理架构对比
accDescr: 对比传统高带宽集中式训练数据中心与分布式低成本推理网络的结构差异
subgraph Training["传统训练集群：集中式高壁垒"]
T_Power["100 MW ~ 1 GW 极难获取"]
T_Net["超高内部网络互连带宽"]
T_Scale["10 万卡单体集群 / 规模不经济"]
T_Power --> T_Net --> T_Scale
end
subgraph Inference["分布式推理网络：碎片化低成本"]
I_Power["海量 1 MW 碎片化电力"]
I_Net["普通公网互连 / 独立推理节点"]
I_Scale["8 机架/微型机房 / 全美广泛分布"]
I_Power --> I_Net --> I_Scale
end
```
---
### `52:45` 8 台“冰箱”装下 1 兆瓦与“全球扫货”的算力清道夫策略
随着液冷技术的引入，数据中心的物理形态发生了戏剧性的变化。过去人们脑海中 1 兆瓦的计算中心通常是一座巨大的仓库式建筑，但现在凭借极高密度的液冷机架，1 兆瓦的计算设备实际上只需要大约 8 个物理机架就能容纳。每个机架的体积大约就相当于一台普通家用冰箱，整整 1 兆瓦的算力，只要 8 台“冰箱”排成一排即可实现。
这种微型化和高密度的物理现实，完美支撑了 Sail 独特的算力商业模式。这种模式的核心逻辑在于两步：第一，去搜刮全球市场上那些被低估、价格远低于合理水平的各种“杂牌”与分散算力；第二，凭借自研软件能力从这些芯片中压榨出最大性能，并将它们部署在极小规模的微型数据中心里只做推理。
Neil Movva 将自己的打法概括为极具进攻性的“扫货清道夫”策略：“我们会在全球任何地方、以任何时长、收购任何芯片。”这种极致的灵活性与流动性是目前市场上任何传统云厂商都不具备的。无论是别家挑剩下的算力切片，还是冷门闲置的算力集群，Sail 都会直接真金白银地买下来，并纳入自己的推理计算网络中，这构成了公司当前核心的成本壁垒。
- **1 兆瓦物理占地极度收缩**：凭借先进液冷技术，传统庞大的机房被压缩至仅约 8 个机柜，整体体积仅等同于 8 台排列整齐的家用冰箱。
- **全生命周期流动性承接**：不拘泥于长期合约或固定规格，以极度自由的租期和价格吞吐不同规格的 GPU 算力。
---
### `54:30` 砍掉冗余：95% 可用性与异步长程智能体的协同优势
从长远来看，要将成本优势拉到极致，就必须投资于那些让主流资本望而却步的边缘数据中心。由 1,000 个小型数据中心组成的“分布式算力军团”，与单体 1 吉瓦的算力巨无霸相比，最大的不同在于彻底砍掉了高昂的冗余成本：
1. **无电力冗余**：不配备极其昂贵的柴油备用发电机，仅依靠单一市电来源。
2. **无网络冗余与苛刻 SLA**：只拉一条光纤走线到机房，绝不建设三路光纤热备和昂贵的故障转移链路。
这种极简架构的结果是机房必然会偶尔宕机。甚至当部分站点的正常运行时间（Uptime）掉到 95% 时，Neil Movva 也完全不在意。对传统的企业级云服务来说，95% 的可用性意味着“致命的灾难”，在市场上几乎零买家，但他直言：“我就是这批资产的第一个买家，我非常乐意买下 95% 可用性的机房。”
这种敢于接盘“垃圾资产”的底气源于两点：
- **高鲁棒性的控制平面**：底层的软件调度系统极为健壮。由于各个分布式数据中心之间的硬件故障互不相关，单点宕机发生时，控制平面只需无缝将任务迁移到其他机房即可，成本与可用性之间呈现出完全线性的健康折算。
- **异步长程智能体（Long-Horizon Agents）的工作特性**：面向后天世界的推理工作不是与真人进行毫秒级实时对话，而是代理人在后台独立运行数小时乃至数天去完成复杂任务。当某个 GPU 掉线导致单个轮次失败并重新调度时，可能会额外增加 1 分钟、3 分钟甚至 10 分钟的延迟；但对于那些在睡觉、等待第二天查看长篇研究报告的用户来说，这一小段延迟根本无关痛痒。牺牲对人类无感的延迟，换取低至数倍的基础算力成本，正是这套“Token 工厂”的核心杀手锏。
### `55:00` 放宽 P99 延迟约束与利用间歇性绿色电力
在面向后台长时间运行智能体（Background Agents）的场景中，系统的性能评价维度发生了一次彻底的转向。Sail 会明确告知客户：平台的平均吞吐量（Throughput）极具竞争力，但 P99 延迟（99 分位延迟）则完全不予受控保证。这种对尾部延迟的放弃，直接换取了在经济成本上“无可匹敌的绝对优势”。对于动辄需要运行数小时甚至数天的离线任务而言，任务究竟是在一秒内开始还是延迟了十分钟才被调度，对最终用户体验几乎毫无影响，但它为底层数据中心的基础设施设计释放了巨大的套利空间。
这种弹性的核心落脚点在于能源结构的重塑。传统数据中心将风能和太阳能等间歇性能源视为不合格的供电方案，因为数据中心需要的是 24 小时恒定的基准负荷（Base Load），而天气变幻莫测。作为一名加州本地人，Neil Movva 始终认为全美的风光清洁能源潜力被严重低估了。对于 Sail 而言，数据中心即使仅维持 95% 甚至是 80% 的正常运行时间（Uptime），只要电价足够便宜就是完全可以接受的。传统托管厂商闻之色变的长达数天乃至数周的停电事故——比如持续无风、阴云遮蔽或山谷大雾——在 Sail 的模型中反而是高度可预测的气象事件。通过精准预测天气，调度系统可以在某地电网停摆前，提前将推理负载无缝迁移到全球其他有充沛算力冗余的节点。由于部署的不是极其昂贵的英伟达前沿机架，计算硬件的资本折旧压力相对可控，系统完全能够承受芯片在无电可用时的暂时闲置。
- **尾部延迟与成本兑换**：放弃严苛的 P99 实时性约束，换取无与伦比的单位 Token 生产成本。
- **拥抱间歇性电力**：传统数据中心要求 99.999% 的高可用供电，而该架构甚至能接受 80%~95% 的在线率，只要电力价格足够低廉。
- **气象建模与动态迁移**：利用长周期停电的高度可预测性，结合天气预报提前跨地域漂移智能体负载。
- **折旧与空闲博弈**：避开高昂的英伟达前沿旗舰芯片，以极低采购成本使得“硬件因断电而闲置”在经济账上完全成立。
---
### `56:43` “拾荒者策略”：搜刮边缘芯片与废弃电力构建代工厂
这种反共识的系统构建逻辑被形象地定义为“拾荒者策略”（Scavenger Strategy）。该策略的运作链路分为两步：首先在市场上搜刮那些被忽视的次级算力芯片，接着为这些芯片搜刮那些无人问津的弃风、弃光等廉价边缘电力。Neil 坦言，自己绝不会去与 Anthropic 或 OpenAI 这样的前沿巨头公开竞价争夺集中的算力集群配额——在这类高强度资金博弈中初创公司毫无胜算，也毫无必要去卷入。相反，Sail 专注于开掘那些在巨头眼中“不具备可读性”（Not legible）、碎片化且难以规模化直接利用的非标供给。
```mermaid
flowchart TD
accTitle: 拾荒者分布式代工厂架构
accDescr: 展示 Sail 如何通过聚合碎片化边缘资源并动态调度，以低成本向客户输出推理 Token。
subgraph Input["低价值边缘供给 (Scavenged Supply)"]
A["次级/非英伟达推理芯片<br/>(低资本沉没成本)"]
B["间歇性绿色电力<br/>(弃光/弃风低价电网)"]
end
subgraph Platform["分布式代工厂系统 (Sail Token Factory)"]
C["气象与断电预测模型"]
D["全局负载动态调度网络<br/>(跨区域任务无缝迁移)"]
E["高容错分布式微型算力磨坊<br/>(非单体巨型超算中心)"]
C --> D
D --> E
end
subgraph Output["终端产品交付"]
F["面向长时间运行 Agent 的<br/>极低成本推理 Token"]
end
A --> E
B --> E
E --> F
```
通过这种模式，Sail 积攒起的是广阔而分散的“聚合供给”（Aggregate Supply），而非单点集中的“集中供给”（Concentrated Supply）。这就像在钢铁工业史上，最终打败或补充传统大型单体钢铁联合企业的，往往是遍布各地的废钢电炉微型钢厂（Mini-mills）。通过将无数细小的非标资源编织在一起，系统最终凝结成一座具有超强价格防御壁垒的 Token 制造厂。
- **避开正面竞价**：彻底放弃竞标主流一线 GPU 集群，避开与前沿实验室的资本内卷。
- **聚合非标供给**：依靠工程和调度能力，将分散、异构且不稳定的碎片化资源打包转化为标准化算力。
- **钢铁微型厂隐喻**：如同电炉炼钢微型厂取代大型高炉，未来的低成本推理网络依靠的是无数分布式节点协同，而非单个庞大脆弱的超级数据中心。
---
### `58:20` 垂直整合的抉择：创业者的浪漫野心与 CEO 的商业务实
面对“究竟要在这条价值链上做到多深”的垂直整合抉择，Neil 坦承自己内心存在着“创业者”与“CEO”两种截然不同的人格撕裂。作为一个骨子里痴迷芯片、电力与能源架构的技术狂热者，他的浪漫野心渴望掌控全栈：自建发电设备、独立设计推理专用芯片、打造专属数据中心，甚至把整个系统玩成现实版的工业自动化游戏《异星工厂》（Factorio for intelligence），不榨干从硬件物理层到底层软件的最后一滴效率誓不罢休。
然而，作为必须对商业现实与现金流负责的 CEO，他深知盲目追求重资产整合等同于自杀。拥有整套物理基础设施所需的资本规模是天文数字，因此公司必须从高商业杠杆率的软件层切入。当前的战略重点是与既有能源企业和公用事业公司建立长周期的购电协议（PPA），让传统领域的专家去干他们擅长的事。只有随着规模不断扩张，Sail 才能“赢得逐步将重资产环节纳入麾下的权利”。这种激进的乐观主义建立在对推理市场的宏大预判之上：随着全球数十万亿美元被倾注进 AI 推理基础设施，整个行业的底层假设都将被重写；Sail 的使命是识别所有存在定制优化可能性的结构性缝隙，当合作伙伴无法响应这些极端需求时，再亲自下场自研替代。
- **创始人情怀与《异星工厂》梦想**：希望从芯片自研、电厂建设到推理软件全栈垂直打通，追逐物理极限级的系统效率。
- **CEO 的商业理性**：全链条重资产投入将带来无法承受的资本消耗，必须依托高杠杆软件切入并以 PPA 协议撬动第三方基建。
- **获取整合权利的渐进路线**：规模未达临界点前，依托专业分工；一旦规模突破且现有供应商无法配合定制，再果断走向纵向自研。
---
### `01:01:04` 算力堆叠与 MoE 架构下的真实效率审视
当被问及若审视整个软件、硬件和能源系统，当今生产有用智能 Token 的过程中哪一部分的低效问题最严重时，Neil 给出了一套反直觉的技术判断。他认为，当前外界对计算规模化（Compute Scaling）的指责大多并不中肯，FLOPs（浮点运算次数）本身的扩展效率其实相当之高。硬件给到多少 FLOPs，软件系统就能扎实地吃透并释放出对应的有效计算。
从模型架构的角度来看，当前工业界在计算利用上已然表现得十分克制与审慎。以现代主流的 MoE（Mixture of Experts，混合专家）架构为例，几乎没有哪家前沿模型的网络激活密度能够超过 10%。这意味着在任一特定 Token 的前向传播运算中，模型只会动态激活全部专家参数中的极小一部分。算法在计算资源层面的精打细算，表明真正的效率瓶颈往往不在于 FLOPs 本身的空转浪费，而深埋在其他未被充分解耦与优化的外围环节之中。
- **FLOPs 扩展效率被低估**：计算扩展本身具有高度确定性，算力供给增加能直接被软件转化为等比例的吞吐输出。
- **MoE 的低密度激活特征**：前沿混合专家模型运行时的参数激活密度普遍被严格限制在 10% 以下，计算调度极为节制。
### `01:00:00` 算力瓶颈与注意力机制低效：KV Cache 的压缩潜力
谈及模型层面的低效时，Neil Movva 指出前沿模型的稀疏度已经接近 1%，在 MoE（混合专家模型）架构上的算力挤压已经相当极致，行业在此并没有浪费太多空间。真正尚未被攻克、甚至极其低效的环节在于注意力机制（Attention）及其对显存的消耗，尤其是键值缓存（KV Cache）。目前业内对 KV Cache 的压缩程度极其原始，每个 token 仍然在显存中占用数千字节（kilobytes）的空间；如果从信息熵的角度来审视，它所承载的信息量完全“对不起它消耗的资源”，其内存占用很可能存在一到两个数量级的冗余。
虽然前沿实验室的内部细节外界不得而知，但 Neil 对 DeepSeek（深度求索）的研究赞不绝口。DeepSeek 持续公开发布关于 KV Cache 压缩的突破性进展，且几乎每年都能推动一个数量级的压缩跨越。在 Neil 看来，一家机构能够如此高频地取得数量级进展，恰恰证明了这一领域还留有巨大的优化空白，底层的算法与系统重构仍有极其广阔的潜力可挖。
- **MoE 稀疏度现状**：前沿模型激活参数占比已降至 1% 左右，结构本身已被压榨得很充分。
- **KV Cache 内存低效**：单 token 占用数 KB 显存，实际信息熵远低于物理占用，存在 10 到 100 倍的潜在压缩空间。
- **DeepSeek 的技术标杆**：通过每年推进一个数量级的压缩工作，证实了注意力机制显存优化的巨大余量。
---
### `01:01:04` 全球算力的调度悲剧：闲置 GPU 与“算力孤岛”
如果将视野进一步放大，人类对全球算力的组织与调度方式堪称灾难。英伟达单在今年就将出货 500 万张 Blackwell 架构芯片，但这些算力是否每分每秒都在全力运转？Neil 极其怀疑。现实情况是，海量算力直接“蒸发”在了各类对外界不公开的私有算力池（private pools）中，这些被锁定的 GPU 绝大部分时间都处于凄凉的空转状态。Neil 坦言：“看到那些由硅和电力组成的昂贵 GPU 就那样白白闲置着，这在生理上让我感到痛苦（It pains me physically）。”
外界往往喜欢拿 xAI 集群在浮点运算利用率（MFU）上的挑战打趣，但 Neil 强调，全球其余机构的真实利用率甚至更加糟糕得多。成千上万的 GPU 被整箱囤积在仓库里，或者被某些大客户长期独占却很少调度。Sail 的核心理念就是直接向这种低效宣战，将全球算力视作一种亟待高效调配的共享资源，通过极致的编排与切片机制提升全局利用率，这比单纯依赖硬件扩产要有效得多。
```mermaid
flowchart TD
accTitle: Compute Allocation Inefficiency vs Global Orchestration
accDescr: Comparison between fragmented private GPU pools and Sail's shared compute factory orchestration.
subgraph S1["传统算力分配现状"]
A["英伟达 500 万张 Blackwell 出货"] --> B["私有集群与企业独占池"]
B --> C["仓库囤积与离线闲置"]
C --> D["极低 MFU 与算力浪费"]
end
subgraph S2["Sail 调度模式"]
E["全球/异构算力汇聚"] --> F["跨集群共享与动态打包"]
F --> G["长任务与推理流水线"]
G --> H["极致算力与电力利用率"]
end
```
---
### `01:02:50` 晶圆代工与工艺公差：接纳“次品芯片”的异构思路
面对是否能通过台积电（TSMC）、英特尔或存储巨头扩产直接让算力激增的问题，Neil 提出了冷静的技术制约视角：半导体供应链是整体动态平衡的。即便打个响指让晶圆厂产能翻倍，解决了台积电的产能瓶颈，系统也会在产量提升 20% 时立刻撞上封装、存储或配电等下一道瓶颈。然而，晶圆制造中更耐人寻味的议题是代工厂与下游客户之间的博弈与边界定义。
在传统芯片代工流程中，一条生产线上下来的“体质最差”与“体质最优”的晶粒（die）存在极大的性能离散度（variance）。台积电等顶级代工厂耗费了极高昂的成本、时间并引入极其严苛的制程控制，试图收窄所谓的“工艺角”（process corners），将最差芯片的电气特性死死锚定在最优芯片附近。Neil 认为，这种绝对均一性对 Sail 来说并非必要：“如果代工厂能对我透明开放更多工艺权衡，我愿意接纳那些性能较差、甚至是代工厂准备报废剔除的芯片（rejects）。”通过将低成本晶圆供应与全美广泛寻获的廉价电力进行全系统端到端优化，Sail 致力于为那些原本“不配进入一线数据中心”的芯片构筑安身立命之所。
- **产能瓶颈的动态转移**：晶圆代工单点扩产无法根本解决问题，系统会在增产 20% 后迅速遭遇新瓶颈。
- **工艺角（Process Corners）的严苛控制**：代工厂为追求芯片一致性付出了极高的控制成本与良率代价。
- **Sail 的逆向策略**：放宽对晶粒一致性的要求，大规模接纳次品或离散度大的芯片，靠廉价电力与系统调度消化硬件瑕疵。
---
### `01:04:45` Sail 的工程文化：不把机器当成既定现实与对性能的热爱
谈到 Sail 的组织构建与工程师文化，Neil 透露团队始终坚持长线演进的心智模型。每当团队开始为一个新模型做底层优化时，早期的系统效率往往非常糟糕，但团队绝不为此感到焦虑，而是着眼于在 1 个月、6 个月甚至 1 年的时间跨度内系统能演进到何种高度。Sail 最核心的工程师信条是：**绝不把所面对的硬件机器视为不可改变的固定现实（fixed）**。即使像 Blackwell 这样极其先进的芯片，如果团队察觉到某种制约性能释放的硬件或微架构瓶颈，也绝不妥协，而是会对其进行极其严密精确的定量表征与记录；这些沉淀下来的认知不仅会直接反馈给合作伙伴英伟达，更会作为 Sail 长期决策的不变知识资产，指引未来每一次芯片选型与系统搭建。
在团队氛围上，Sail 崇尚高频的白板推演（whiteboard sessions）与极具大学学院风的学术切磋。团队很多核心成员在大学期间都曾担任助教（TA），骨子里兼具“优秀求知者”与“优秀布道者”的双重性格。当被问及未来 3 年当大量工作被机器自动化接管时、什么样的工程师画像最能对抗时代的淘汰，Neil 毫不犹豫地指出：必须是“100% 的纯粹好奇心”。在他看来，几乎所有软硬件工程技能都可以后天传授，唯独一件事情是绝对教不会的——那就是对极致性能刻入骨髓的热爱（"The one thing I cannot teach is love for performance"）。
### `01:05:00` 顶级性能工程师的核心特质与 CUDA 的“障眼法”
在谈到工程团队的选拔标准时，Neil Movva 明确指出，他最看重的核心特质是对机器运行细节的极致好奇心——能否深入探究机器运转的每一个微秒（microsecond），搞清楚在那一微秒内硬件上究竟发生了什么。在他眼中，这是衡量一名优秀性能工程师（Performance Engineer）最重要的品质。相反，他从不刻意筛选拥有丰富 AI 经验的候选人，甚至完全不在意对方是否有 CUDA 开发经验。Neil 直言，迷信 CUDA 经验实际上是一个巨大的“障眼法”（red herring）。
在过去五年里，无论是 GPU 硬件架构还是 CUDA 本身的编程理念，都经历了翻天覆地的演变，要求候选人拥有十年所谓的“老兵经验”毫无意义。具体的工具框架和硬件知识可以在入职后传授，但那种对性能调优发自内心的热爱与执着，是无论如何也教不出来的，这才是他在招募人才时苦苦寻觅的东西。
```mermaid
flowchart TD
accTitle: 性能工程师甄选逻辑
accDescr: 对比传统招聘看重的内容与 Neil Movva 真正看重的核心特质
subgraph Trap["传统招聘误区（障眼法）"]
T1["盲目要求 10 年 CUDA 经验"]
T2["过分看重特定 AI 框架年限"]
end
subgraph Reality["底层实际情况"]
R1["GPU 架构与 CUDA 概念近 5 年剧烈迭代"]
R2["具体框架技术完全可以入职后传授"]
end
subgraph Core["核心考核标准"]
C1["对性能工程的纯粹热爱"]
C2["探究机器每一微秒执行细节的执着"]
end
Trap -. 错判.-> Reality
Core ==> Yield["构建极限低成本推理架构的人才基石"]
```
### `01:07:02` 闭源与开源之辩：前沿实验室的溢价与无法阻挡的“隐性蒸馏”
针对闭源前沿实验室与开源生态的关系，Neil 给出了一句极为精炼的总结：“前沿实验室（Labs）正支付极其高昂的溢价，仅仅为了在能力上领先整个行业 3 到 6 个月。”尽管如此，他认为对于像 Anthropic 这样的头部公司而言，这种巨额投入目前依然是合理的商业选择。随后，对话深入到了模型“蒸馏”（Distillation）这一极为敏感的话题。外界普遍有一种论调，认为利用前沿模型的输出来训练小模型是一种“窃取”或剽窃行为；但 Neil 提出了一个更贴近现实的逆向视角：信息与模型能力的扩散根本不可能被阻止。
即使开发团队从未主动去爬取 Anthropic 的数据，互联网上涌现出的人工智能生成内容（AIGC）也正在呈指数级增长。以 GitHub 为例，过去一年中新建的代码仓库里，有极大比例直接由 Claude Code、GitHub Copilot 等前沿大模型自动生成。如果开源社区仅使用 GitHub 上公开认可的优质代码来训练一个一流的新模型，这算不算蒸馏？如果秉持“用户拥有自己与 AI 交互产生的内容所有权”这一立场，当海量用户自愿将这些 AI 生成的代码公开发布到网络上时，长期的“隐性蒸馏”（Latent Distillation）便不可避免地发生了。信息扩散是大势所趋，唯一的变数只是扩散的速度有多快。
面对“模型缩放定律（Scaling Laws）若能长期成立，前沿实验室是否就能永久靠 3 到 6 个月的领先期维持超高 Token 溢价”的疑问，Neil 持怀疑态度。他指出，现实中的企业级部署（Enterprise Deployments）根本跟不上 3 到 6 个月的迭代节奏；许多大型企业到现在依然在跑更早期的稳定版本（例如老版本的 Opus 系列等），在面对任何底层变更时都慎之又慎。当前整个行业才刚刚触及大模型应用的皮毛，断言输赢为时尚早，甚至这根本不是一场能分出最终胜负的比赛，而是一个持续的演进过程。开源不仅永远不会消失，而且每当有领头羊退出，巨大的利益和推动力就会促使新的挑战者迅速填补空白，如今训练一个前沿级别模型的门槛和难度每天都在显著降低。
- **前沿溢价的脆弱性**：企业客户对模型更新的评估和验证极其保守，无法迅速消化前沿实验室按季度迭代的微弱领先优势，高昂的溢价很难无休止地维系。
- **开源生态的自我修复力**：训练前沿水准模型的成本持续下行，即使某一家开源领军机构转向或停滞，社区与新玩家也会在生态激励下立刻接棒。
### `01:10:37` 丰沛的 Token、个性化 Harness 与颠覆性的 Flash 内存架构
展望未来生态的平衡格局，Neil 描绘了他所追求的愿景：“极致丰沛廉价的 Token，以及极大多元化的交互支架（Diverse Harnesses）。”他坚信，未来的核心方向不是指望所有用户都被圈养在单一全栈巨头的生态里，而是让每家企业、甚至每一个终端用户都能构建专属于自己的 Harness，彻底拥有并定制自己的智能体。这种定制化未来大概率不再依赖昂贵低效的权重微调（Weight Fine-tuning），而是依靠更强大的上下文学习（In-context Learning）。而要让这一设想落地，底层的绝对支柱就是“让 Token 便宜到极限”。
为了实现这一目标，Neil 专注于撬动所有的供给侧杠杆（Supply-side levers）：压榨每一块可用的旧芯片、寻找廉价未被充分利用的废弃电力，甚至利用美国境内所有适宜部署算力的边角土地。他感叹道，今天的人们依然把 AI 智能体当成一位按小时收费的昂贵咨询顾问，只有在遇到极其棘手的难题时才舍得去提问；但事实上，机器能够产生思考是一件不可思议的奇迹，整个行业应当通过无限压低成本，让这种智能流淌进每个人的手中。
当被问及有哪些观点会让身边的行业好友觉得他“脑子坏掉了”（look like you have three heads）时，Neil 直言是他对底层 AI 芯片与推理架构的激进构想：
- **绕开 HBM 垄断与闪存极限制**：整个行业都在为昂贵且短缺的高带宽内存（HBM）焦虑，而 Neil 带领团队探索的核心思路是通过修改底层模型架构，将 KV 缓存（KV Cache）大规模、极限地卸载（Offload）到便宜的 Flash 闪存介质中。
- **逆向优化低吞吐场景**：主流硬件设计都在拼命追求高并发与超高每秒 Token 输出，但 Sail 的北极星指标却是围绕每秒 1 到 10 个 Token（1 to 10 tokens/sec）的平价长时推理来重构系统，这种取向在主流推理界显得离经叛道。
- **支撑单日万亿级 Token 吞吐**：打造全新的底层计算范式，让普通企业和开发者真正有能力以极低成本每天消耗一万亿个 Token（a trillion tokens per day），从供给端彻底释放长程智能体的应用潜力。
### `01:10:00` 每日万亿 Token 的愿景与人类对智能的需求
当谈到“每天消耗一万亿（1 Trillion）Token”到底意味着什么时，Neil Movva 给出了一组直观的财务参照：按照当前 OpenAI 针对前沿模型（如 GPT-4 等级）的官方定价，生成一万亿 Token 至少需要耗费 500 万美元。如果希望未来实现普通人每人每天都能消耗如此庞大的计算量，整个行业就必须将每个 Token 的单位成本压低 3 到 6 个数量级。如果能将万亿 Token 的成本压缩到 5,000 美元左右，就会立即涌现出一大批极具商业价值的客户；而对于某些特定尺寸的较小模型，目前万亿 Token 的成本已经开始接近数万美元的区间，这意味着单个复杂任务或工作流已经完全有能力调用如此量级的计算。
面对主持人关于“普通人自身大脑并不消耗这么多思考量，世界上是否真的存在如此庞大的智能需求”这一质疑，Neil 坚定地表示自己绝不怀疑人类对智能的需求上限。他认为“世界上对智能的渴求永远存在”，真正的瓶颈在于产品界如何为用户构建接入这些智能的通路（on-ramps）。Neil 坦言自己并非产品经理，无法描摹出所有的具体产品形态，但他致力于赋能那些产品开发者，让他们摆脱算力成本的桎梏——他经常听到客户抱怨“我的免费层用户太多，我根本负担不起为他们提供这么多 Token 的成本”，而 Sail 的目标正是彻底消除这种顾虑。
- **万亿 Token 当前成本**：按现有顶级商业模型定价，一万亿 Token 成本至少在 500 万美元以上。
- **降本目标**：需要实现 3 到 6 个数量级（1,000x 至 1,000,000x）的成本下降，降至数千美元时将迎来爆发式采用。
- **需求哲学**：智能本身不存在需求天花板，核心挑战在于构建足够友好的产品交互通路。
---
### `01:12:42` 对 NVIDIA 与台积电地缘政治的逆向思考
在探讨业内有哪些看似共识实则错误的认知时，Neil 提出了针对 NVIDIA 和半导体制造的逆向视角。虽然短期内他绝不看空 NVIDIA，并承认“永远不要做空 NVIDIA，他们总能重塑自我”，但他指出业内忽视了一个底层物理事实：如果严格按照同等条件（like-for-like）对比，从 Hopper 到 Blackwell 再到下一代 Rubin 架构，进行 BF16（BFloat16）乘法运算时的“每瓦性能”（Performance per watt）提升其实并不显著。类似地，追踪台积电（TSMC）从 5 纳米、4 纳米、3 纳米到 2 纳米制程的演进，芯片底层每瓦性能的跃升也远没有大众想象的那样戏剧化。
这一物理现实直接引出了他对地缘政治风险的独特研判：外界常常陷入一种极端恐慌，认为一旦西方因地缘局势失去台积电，整个 AI 产业就会彻底停摆。Neil 认为这种冲击主要体现在供应链产能的短期休克，但在技术物理本质上“并没有那么糟糕”。西方本土拥有的最先进工艺（例如 Intel 的先进制程）与台积电之间的能效差距，最坏情况下大约也就落后 2 倍左右。在芯片战的舆论喧嚣之下，真实的技术物理代差远比媒体描绘的要小得多。
---
### `01:13:30` 处于下游的推理服务与模型架构的巨变
对于整条产业链中未直接涉足但最令其着迷的环节，Neil 指向了处在上游的基础模型架构研发。作为推理服务商，Sail 完全处于模型层下游：OpenAI 或 Anthropic 等前沿实验室决定了神经网络的基础结构，Sail 无法干预其底层架构设计，只能密切预测研究方向，并据此设计自身的软硬件服务系统。
Neil 强调，模型架构师所做的选择极具深远影响，甚至关系到机器思考的本质。例如，选择使用稀疏注意力机制（Sparse Attention）还是密集注意力机制（Dense Attention），或者在数值精度上从原先的 BF16 训练转向 FP8 甚至 FP4 等更低精度的数值格式。这些在研究人员看来似乎只是算法层面的常规探索，但传导到系统底层时，却会从根本上颠覆推理基础设施对特定芯片的选择、服务器机架的工程搭建以及对未来算力形态的全部预判。
---
### `01:14:00` 给 AI 硬件创业者的建议：聚焦四大供应链瓶颈与 HBM 危机
如果面对 100 位试图通过打造定制芯片、系统或机架来切入计算市场的创业者，Neil 认为他们最核心的任务不是死磕某个孤立的技术路线，而是必须证明自己彻底吃透了主导现代芯片供应的底层瓶颈，并在商业模式上完成套利（arbitrage）：
```mermaid
flowchart TD
accTitle: 现代 AI 芯片供应链四大核心瓶颈
accDescr: 展示制约现代 AI 芯片供应的四大物理瓶颈：台积电晶圆产能、HBM 内存产能、先进封装以及数据中心电力。
subgraph Bottlenecks["现代 AI 芯片四大供应链瓶颈"]
W["1. 台积电晶圆制造产能<br/>(Wafer Capacity)"]
H["2. 高带宽内存产能<br/>(HBM Capacity)"]
P["3. 先进封装技术<br/>(Advanced Packaging)"]
E["4. 电力与基础设施<br/>(Power & Racks)"]
end
W --> S["硬件创业者破局之道:<br/>寻找 NVIDIA 难以快速转向的设计权衡进行定点突破"]
H --> S
P --> S
E --> S
```
Neil 指出，NVIDIA 的芯片并非完美无缺，而是做到了极其高超的“全局均衡”。由于巨头身躯庞大，很多设计抉择一旦确立就极难快速转向；因此新兴创业公司必须展现出“尖锐的特化属性”（spiky）。在这四大瓶颈中，Neil 认为最值得正面发起突击的是 HBM（高带宽内存）。因为建设全新的内存晶圆厂极为耗时且资本开支极其沉重，博伊西的内存厂商（指美光 Micron 科技等）在经历多次半导体强周期洗礼后，对周期性重资产扩张极为审慎。Neil 预测，短期内 HBM 的短缺不仅无法缓解，反而会向消费电子外溢，导致未来 iPhone 等消费级终端削减内存配置或直接推高终端售价。
---
### `01:14:38` NVIDIA 为何不直接下场卖 Token
针对“NVIDIA 为何不索性沿着价值链向下走到底、直接向客户出售 Token”以及“为什么不建立自己的 NeoCloud 直接参与终端竞争”的疑问，Neil 盛赞了 NVIDIA 和创始人黄仁勋（Jensen Huang）的高超商业智慧。NVIDIA 的长期战略原则是“绝不与自己的客户直接竞争”。
黄仁勋非常擅长“让身边的合作伙伴成为亿万富翁”——最典型的例子就是他一手扶持了 CoreWeave 这样的新型云厂商（NeoCloud），使其迅速跻身数十亿美元估值的独角兽行列。NVIDIA 刻意不去破坏这种建立在生态利益共享基础上的信任与商誉。通过培育一个由众多 NeoCloud 和推理服务商构成的多元生态，让各家厂商相互角逐以持续为 NVIDIA 创造旺盛的芯片需求；一旦其中某一家试图搞垂直整合或转向 AMD 等竞品，生态中立刻会有三家更饥渴的厂商顶替其位置。在买家之间维持健康的竞争态势，正是 NVIDIA 稳固霸权的最高境界。
---
### `01:15:20` 终极感悟：导师的善意与贯通全局栈的追求
在全剧终的经典提问环节——“别人为你做过的最善良的事情是什么”，Neil 毫不犹豫地提到了多年来指引他的导师们。他感慨道，在这个世界上，能有人愿意从百忙之中抽出大量时间，将帮助你理解世界、传授核心认知当成其个人责任，并推着你迈过顿悟的门槛，这是极其罕见的幸运。
这其中包括前文提及的 NVIDIA 前辈们，他们将对极致性能工程的热爱注入了 Neil 的血液；更包括他大学二年级时的学术导师。Neil 回忆当时自己是一个心浮气躁的本科生，冲进导师的办公室宣称：“我想做 AI 芯片！我清楚知道自己要什么，为什么还要浪费时间去学计算机网络和操作系统这些基础课程？”那位导师没有批评他的傲慢，而是耐心地在白板前为他完整梳理了计算机体系的全局架构栈（Full Stack），展现了底层每一个拼图相互咬合的系统之美。导师告诉他：
> “能够从晶体管门电路级别的硅片（gate-level silicon）一路理解到搭建互联网级大规模服务的人，在这个世界上凤毛麟角。你应该立志成为那种在一生中真正参透整个计算栈的人，去追求这种深度的专业造诣是一件无比崇高的事情。”
这段对话彻底扭转了 Neil 试图在局部走捷径的心态，并成为深深烙印在他此后系统工程与创业生涯中最宝贵的财富。
## Notable quotes
> `05:32` The best latency is no latency at all — Neil Movva
> `05:32` 在协作流程中，如果人类必须不断提示、不断等待并给予反馈，那么人类自身就成为了限制 Agent 发挥更大价值的最大瓶颈。 — Neil Movva
> `10:00` 网络安全本质上已经变成了一种工作量证明（Proof of Work） — Neil Movva
> `11:22` spend tokens without any promise of return — Neil Movva
> `17:15` 我们追逐的是 100% 的光速极限。我根本不在乎相对竞争对手赢了多少百分比，我只在乎绝对性能——这块芯片理论上到底能做到什么，我们如何把它完全释放出来。 — Neil Movva
> `44:05` Compute is power today. They're quite strategic about how they allocate compute. — Neil Movva
> `45:00` There's no bad chips. There's only bad pricing. — Neil Movva
> `36:43` 互联网对人类而言，是数据上的一次性补贴（A one-time subsidy on data）。 — Neil Movva
> `47:56` it's no longer speculative — Neil Movva
> `52:45` 我们会在全球任何地方、以任何时长、收购任何芯片。 — Neil Movva
> `01:01:04` 看到那些由硅和电力组成的昂贵 GPU 就那样白白闲置着，这在生理上让我感到痛苦（It pains me physically）。 — Neil Movva
> `01:07:02` 前沿实验室（Labs）正支付极其高昂的溢价，仅仅为了在能力上领先整个行业 3 到 6 个月。 — Neil Movva
> `01:04:45` The one thing I cannot teach is love for performance — Neil Movva
> `01:15:20` 能够从晶体管门电路级别的硅片（gate-level silicon）一路理解到搭建互联网级大规模服务的人，在这个世界上凤毛麟角。你应该立志成为那种在一生中真正参透整个计算栈的人，去追求这种深度的专业造诣是一件无比崇高的事情。 — Neil Movva 的学术导师
## Mentioned
Neil Movva, Patrick O'Shaughnessy, Sail Research, Ramp, Felix by Rogo, WorkOS, Vanta, Ridgeline, Cursor, Baseten, Fireworks, Together AI, Parallel Web Systems, Anthropic, Claude, Claude Haiku, Claude 3.5 Opus, Claude Code, NVIDIA, Jensen Huang, Blackwell, GB200 NVL72, Hopper, Rubin, CoreWeave, Cerebras, Groq, Etched, SambaNova, d-Matrix, AMD, Google TPU, AWS Trainium, Meta, OpenAI, GPT-4, GPT-5, TSMC, Intel, Cisco, Micron, SK Hynix, Samsung, Tri Dao, FlashAttention, Andrej Karpathy, DeepSeek, GitHub Copilot, xAI, Factorio
## Video description
My guest today is Neil Movva, founder of Sail. Sail is building what Neil calls a token factory, an inference company designed for a specific kind of future, one where AI agents run in the background for hours or days at a time rather than answering a human in real time.
In that world, latency matters less and cost matters more, and Neil has built the whole company around driving the cost of a token as low as it can possibly go.
What makes this conversation special is that it is one of the most detailed tours I have ever done through the full stack of intelligence, the software, the chips, and the power, and how all three connect.
Along the way we cover the trade-off between speed and cost that lives inside every GPU, his scavenger strategy for buying the chips and power nobody else wants, his contrarian view on Nvidia, and why the premium the frontier labs charge for being three to six months ahead may not last.
Please enjoy my conversation with Neil Movva.
For the full show notes, transcript, and links to mentioned content, check out the episode page here.
-----
Become a Colossus member to get our quarterly print magazine and private audio experience, including exclusive profiles and early access to select episodes. Subscribe at colossus.com/subscribe.
-----
Ramp’s mission is to help companies manage their spend in a way that reduces expenses and frees up time for teams to work on more valuable projects. Go to ramp.com/invest to sign up for free and get a $250 welcome bonus.
-----
Trusted by thousands of businesses, Vanta continuously monitors your security posture and streamlines audits so you can win enterprise deals and build customer trust without the traditional overhead. Invest Like the Best listeners get a special offer of $1,000 off Vanta when you go to vanta.com/invest.
-----
WorkOS is the infrastructure B2B and AI-native companies use to sell to enterprise. It covers everything enterprise security requires: SSO, SCIM, RBAC, Audit Logs, AI governance, and more. Trusted by 2,000+ fast-growing companies, including OpenAI, Anthropic, Cursor, and Vercel.
-----
Rogo is the AI platform for finance. They're building agents for Wall Street that are trained to understand how bankers and investors actually do work: from diligence and modeling, to turning analysis into deliverables. To learn more, visit rogo.ai/invest.
-----
Ridgeline has built a complete, real-time, modern operating system for investment managers. It handles trading, portfolio management, compliance, customer reporting, and much more through an all-in-one real-time cloud platform. Visit ridgeline.ai.
-----
Editing and post-production work for this episode was provided by The Podcast Consultant.
Timestamps:
(00:00:00) Welcome to Invest Like The Best
(00:02:20) Neil Movva
(00:03:22) Building a Token Factory
(00:05:32) The Rise of Long-Running Agents
(00:08:47) Deep Research and Cybersecurity
(00:15:12) The Full Stack of Intelligence
(00:20:03) Throughput Versus Latency
(00:24:58) The Future of AI Chips
(00:33:19) Why Transformers Work
(00:36:43) The Future of Data
(00:44:05) The Market for AI Chips
(00:47:56) Is the AI Boom Different?
(00:51:08) Reinventing the Data Center
(00:56:43) Scavenging Power
(01:01:04) Where Compute Is Most Inefficient
(01:07:02) Open Versus Closed Models
(01:10:37) A Trillion Tokens a Day
(01:12:42) The Contrarian Case on NVIDIA
(01:14:38) Advice for AI Hardware Founders
[Source](https://traffic.megaphone.fm/CLS1341199916.mp3)
Source quotes
[5:32] Neil Movva: The best latency is no latency at all
[5:32] Neil Movva: 在协作流程中，如果人类必须不断提示、不断等待并给予反馈，那么人类自身就成为了限制 Agent 发挥更大价值的最大瓶颈。
[10:00] Neil Movva: 网络安全本质上已经变成了一种工作量证明（Proof of Work）
[11:22] Neil Movva: spend tokens without any promise of return
[17:15] Neil Movva: 我们追逐的是 100% 的光速极限。我根本不在乎相对竞争对手赢了多少百分比，我只在乎绝对性能——这块芯片理论上到底能做到什么，我们如何把它完全释放出来。
[44:05] Neil Movva: Compute is power today. They're quite strategic about how they allocate compute.
[45:00] Neil Movva: There's no bad chips. There's only bad pricing.
[36:43] Neil Movva: 互联网对人类而言，是数据上的一次性补贴（A one-time subsidy on data）。
[47:56] Neil Movva: it's no longer speculative
[52:45] Neil Movva: 我们会在全球任何地方、以任何时长、收购任何芯片。
[61:04] Neil Movva: 看到那些由硅和电力组成的昂贵 GPU 就那样白白闲置着，这在生理上让我感到痛苦（It pains me physically）。
[67:02] Neil Movva: 前沿实验室（Labs）正支付极其高昂的溢价，仅仅为了在能力上领先整个行业 3 到 6 个月。
[64:45] Neil Movva: The one thing I cannot teach is love for performance
[75:20] Neil Movva 的学术导师: 能够从晶体管门电路级别的硅片（gate-level silicon）一路理解到搭建互联网级大规模服务的人，在这个世界上凤毛麟角。你应该立志成为那种在一生中真正参透整个计算栈的人，去追求这种深度的专业造诣是一件无比崇高的事情。
Source quotes
[5:32] Neil Movva: the best latency is no latency at all
[40:00] Neil Movva: I like to say we write kernels on the whiteboard. We go to the whiteboard. We describe what we think the machine should be doing. Then we succinctly describe that in natural language to a model. And then the model is able to do the execution of: here is my input and output, here is the strategy of how we want to dispatch this work onto the GPU, I'm going to go implement it.
[45:00] Neil Movva: There's no bad chips. There's only bad pricing.
[47:56] Neil Movva: People buy tokens because they're immediately valuable to them. You don't hoard tokens, you use them immediately.
[52:45] Neil Movva: we will buy any chip anywhere in the world for any duration of time.
[54:30] Neil Movva: My argument is that my customers don't care because their agent was running for hours. They're sleeping. It doesn't matter.
[58:30] Neil Movva: I will never stop until I have built the most efficient system from soup to nuts.
[61:04] Neil Movva: You give me more FLOPs and I will use more FLOPs.
[68:45] Neil Movva: My job is to make the tokens as cheap as humanly possible. I will achieve that and I will do it through every layer in the stack available to me. I love the supply-side levers. I will use every chip, I will use every source of power, and I will use every piece of land in the United States that's suitable for this.
Original transcript
[0:00] Ramp is the only platform built to make your finance team leaner, faster, and better, saving businesses 5% annually on average so you can stay focused on growth. Ramp customers grow revenue 3.2 times faster than the average American business. Visa, Vercel, Kerser, Stripe, Notion, Eleven Labs, Shopify, and 70,000 other businesses all now run on Ramp. Mine does too, and so should yours. Learn more at ramp.com slash invest. Felix by Rogo is a personal finance agent that turns a single prompt into finished client-ready work using your firm's own templates, context, and standards. Send Felix an email like, take these comments and turn them for me, or update my tracker with the context of these emails, and Felix sends back finished PowerPoint decks, Excel models, and sourced research. Felix works the way your team already does, delivering work quickly and accurately around Learn more at rogo.ai slash Felix. OpenAI, Cursor, Anthropic, Perplexity, and Vercel all have something in common. They all use WorkOS. To achieve enterprise adoption at scale, you have to deliver on core capabilities like SSO, SCIM, RBAC, and audit logs. Instead of spending months building these mission-critical capabilities yourself, you can just use WorkOS APIs to gain all of them on day zero That why so many of the top AI teams you hear about already run on WorkOS WorkOS is the fastest way to become enterprise ready and stay focused on what matters most your product Visit workos.com to get started. Hello and welcome everyone. I'm Patrick O'Shaughnessy and this is Invest Like the Best. This show is an open-ended exploration of markets, ideas, stories, and strategies that will help you better invest both your time and your money. If you enjoy these conversations and wanna go deeper, check out Colossus, our quarterly publication with in-depth profiles of the people shaping business and investing. You can find Colossus along with all of our podcasts at Colossus.com. in the securities discussed in this podcast. To learn more, visit PSUM.VC. My guest today is Neil Mova, the founder of SAIL Research. SAIL is building what Neil calls a token factory an inference company designed for a specific kind of future one where AI agents run in the background for hours or days at a time rather than answering a human in real time In that world latency matters less and cost matters much more, and Neil has built the entire company around driving the cost of a token as low as it can possibly go. What makes this conversation special is it's one of the most detailed tours I've ever done through the full stack of intelligence, the software, the chips, the power, and how the three connect. Along the way, we cover the trade-off between speed and cost that lives inside of every GPU, his scavenger strategy for buying the chips and power no one else wants, his contrarian view on NVIDIA, and why the premium the Frontier Labs charge for being three to six months ahead may not last. Please enjoy my conversation with Neil Mova. I think it's important early in these conversations to just say the thing, literally what you're building and what it does today. So maybe just orient us there with a brief description, like literally what the system is that you're building and why it should exist. Sail Research is a token factory. We have an API where anyone can send us requests, where they can use large language models, open source large language models for any tasks they want. We will serve those tokens to them at a price that is unbeatable in the market. We also support their ability to build agents on top of this. We host what we call sale boxes, which are long running agent virtual machines hosted in the cloud that are designed for agents that run for hours days or weeks So I should think about you as a peer company to others that serve different kinds of inference You serving one specific kind of inference and your goal is to be the absolute cheapest provider and enabler of a certain kind of use of intelligence. Exactly. The theme of our company is abundance. We want to deliver this new commodity of intelligence to as many people as possible at a cost that is sustainable for almost every industry. We think that whenever you make something 10 times cheaper, it's a new product category. We aspire to do that for tokens. We think it's so profound that the machine can think, and now our job is to make as many machines as possible in the world work towards thinking. So if you think about the theme of the day being token costs, is token costs the right way to think about this? Is there some other way you'd put it? To start with, absolutely, token costs. Today, my North Stars, I want to have the lowest cost per token in the industry and do that by a mile. I don't think tokens are the final unit of work or intelligence, but they are what we use today. After tokens, you start to move more towards more outcomes, which is like a vague direction. You can imagine, for example, today when you consume tokens through an agent, you don't actually control how many tokens the agent reasons for. It can reason for a certain amount of time or it can call a certain number of tools. And increasingly, I think we will have agents do some unit of work, take as many shots on goal as they can. And however,
[5:00] tokens they use to get there is going to be a dependent variable depending on the task. So you think about agents that self-administer a token budget as opposed to a company setting a budget for how many tokens engineers can spend per month. Why is there an opportunity that you can tackle? It seems like the entire world is oriented around more, better, faster, cheaper tokens right now. It seems like the world is trying to solve this problem very aggressively. What was the unique opening that you saw that's maybe the market's not being efficient in its attempt to tackle this? So I think there's two things that are tailwinds for our company. One has got to be the rise of open source. I had to talk about that first. I think we're starting to see an increasing number of our customers and the broader market care about owning intelligence. They want to have control, sovereignty over the thing that they depend on. That created a much more robust market for our customized models or even just like these vanilla open source models that no one can ever take away from you. You always have the weights. You always have the right to deploy them however you like. In that world, there's been a reasonably robust market for the past couple of years serving these models at large scale. The challenge is all those companies, you could take your pick, base 10, fireworks together, they all focus on low latency inference And they were pulled in that direction by one very important customer Cursor I think that that was the right choice about a year ago And as of six months ago it started to look like maybe low latency wasn the only thing you wanted from an agent. You wanted more persistence, more long horizon tasks. And now it's, to me, very obvious that the future of an analytic inference is long horizon tasks. You're going to run the machine for hours or days at a time. It doesn't matter if it's been set tokens at 100 tokens per second. Maybe 10 is just fine. if that comes with corresponding advantages and efficiency. Why are you so confident in that? To me, it seems like I want everything as fast as possible. When you're waiting on it, you absolutely deserve the fastest sensor possible. My trick is I don't want you to be waiting on it. I want it to be proactive. I want it to be in the background. One way to say it is like the best latency is no latency at all. When you wake up in the morning, the work's already been done overnight. You didn't even have to ask for it. That's the dream. We're not quite there yet. More importantly, I think the more you're in the loop as you prompt agents and wait for a response, In fact, you're the bottleneck in having the agent do more or less work. What we'd like is the agent to operate on more human time skills. You don't manage your colleagues every five minutes. You ask them to do a high-level task and you come back and check in maybe every day, but more likely once a week. And that to me is the future of human agent collaboration, more like human time skills. Say more about the early indications that this is happening and therefore you should be building this company So the first and most important thing is the idea of test time compute scaling The idea that you can give an agent more time and it will give you a better answer So that was theorized about two years ago now. But it wasn't really something that we could actually bet on until I would say late last year with Opus 4.5. Opus 4.5 was the first agent that was at all suitable for longer horizon tasks. It was pretty mediocre when it first came out. But you look at the more recent models and what we've done on open source as well. And you see that agents are capable of running for an hour at a time. I wouldn't say it's days, but definitely an hour is quite suitable today. Seeing that average task length get longer and longer, it doesn't take many points to have you draw out the exponential and see that agents are worth running for a longer period. What do you think will be the market share of long-running agents in three years or something like this? I love this market because it's unbounded. There's no human in the loop, so you can consume as many tokens as you like in the background. versus human attention span. If you tell me to consume 10x as many tokens at Codex or at Cloud Code, I'm actually not sure if I can anymore. I'm already in the loop and locked in coding for most of the day that I'm at the laptop. What is unbounded is how many tokens can be consumed in the background or proactively. Long-term, I think we're going to end this year at maybe 50 background and real workloads but I see this going to 90 in favor of background What are your favorite examples of something that gets accomplished much better as a background task than as a human in the loop task? Most deep research. Most questions where you want to have a definitive answer over not 100 sources, not 1,000 sources, but 10,000 sources or more. If you want to build an authoritative index of information, like for example, one of our customers, Parallel Web Systems, seeks to do. They want to build an index over the whole internet, and they want to monitor the internet in real time for changes. That is the kind of crazy exabyte scale task that you need a very different kind of intelligence or scale of intelligence to achieve. Deep research is a top category for us. And then increasingly, we see cybersecurity following this direction. If you think about, yes, there's so much code you can generate, but there's exponentially more ways to break that same code than is to generate that code. There are some great customers out there who are working very hard to find agents that can break any piece of software and proactively patch them. When Fable first came out, for example, or Mythos first came out, basically there was this push in the cybersecurity community to run Fable against every line of code we've ever written and look for bugs in 20 different ways. Meaning you're looking for both memory errors, you're looking for business logic errors, and looking for network vulnerabilities. All these things. And these are all actually things that you would write specialized agents for. You wouldn't just
[10:00] To have Fable look at the source code once, you'd have it actually set up environments where you can pen test these applications. At some point, people started to make this joke that security has become proof of work. When you want secure software, it's really a question of how many dollars did you spend on Anthropix APIs? trying to break into your software. That is the best indication for how secure it is, because that's the best tool in the world. And increasingly, we found that open source models Well, the frontier of intelligence here is quite jagged. It's not the case that Fable finds a superset of all bugs in software. You would find some bugs with a very small model that you don't find with a large model. You'd find some bugs with Haiku that you would find with Hable and vice versa. So it encouraged this very diverse approach to sampling and trying to build cybersecurity agents that break software autonomously. such that you can patch them. - If you were to get speculative and imaginative about the sorts of things that very cheap, very long-running agents can enable, We talked about some very practical examples, deep research. cybersecurity, et cetera. If you get a little bit dreamier about the use cases, new product category that this sort of inference will unlock, I guess the question is just like, so what? If you're maximally successful, dream a little bit about what that might enable. I think for individual users, what I'm excited about most is this idea of proactive, intelligent agents. You can imagine a Siri that is running in the background all the time to understand all the emails you received in a day, all the text messages you receive in a day, and has a much more encyclopedic view of your life and how to be helpful in that life. Right now, there's still point solutions. You end up doing a lot of prompting. Siri is not very proactive. Something we can fix with abundant inference. If you trust the machine enough that it's reliable and also trustworthy as in private, You might even imagine the machine can understand how you interact with it and proactively surface your next action. Whenever you open your phone, can we build a good model of what you're going to do next? My estimation is yes, we totally can. And the key to that is incredibly cheap intelligence. You have to be willing to spend tokens. without any promise of return. That is the unlock. The long lens view to take on this is that We have a form of intelligence that can tackle any verifiable problem. Any verifiable problem means most software It means a lot of formal... math proofs and similar. And it could also mean scientific discovery. These are all relatively verifiable problems. And all those things currently have a dollar cost attached to them, essentially, that's a hidden one. It's like how many tokens could you possibly harness to make this work? we have started to bring it within view. A dollar cost for these long horizon tasks that is reasonable. It's not millions, it's thousands. And maybe it could be hundreds or even tens of dollars in the near future to have a definitive answer to any scientific question, to any research problem. So if we dream about that future, we then become limited just by the questions that people can ask, basically? Pretty much. The questions we can ask, the models are on the cusp of basically taking even a high-level question and chasing it down. Every possible follow-up, you can have the model essentially take that on its own. And the question is, what is your token budget? And we will solve the token budget problem. What about non-verifiable? tasks. Those are basically the entire category of human taste into that category. We have not solved human taste yet, and I don't know that it fundamentally can be. I'm excited to be surprised here, but we are focused on very quantitative problems. We leave the quality of writing, we leave the beauty of art to people. Vanta automates security and compliance for over 16,000 fast-moving companies like Ramp, Cursor, and Harvey, keeping them audit-ready around the clock. It's the number one agentic trust platform, and it now helps companies like yours watch for the risks that show up between audits across your vendors, your AI tools, and your whole environment. Every new tool your team signs up for, every vendor that turns on AI features, is an opportunity for something to go wrong. and most security programs weren't built for AI's pace of growth. The Vanta agent works like a 24/7 GRC engineer in the background, finding issues, drafting fixes for you, and cutting vendor assessment time by up to 50%. Whether you're a fast-growing startup or a global enterprise, Vanta helps you earn and prove trust. Invest like the best listeners get a special offer for $1,000 off. at vanta.com slash invest. Ridgeline is the first end-to-end system of record with embedded AI for investment management firms running portfolio accounting, reconciliation, reporting, trading, and compliance on one unified platform. Firms are moving off legacy technology and onto Ridgeline because of how far ahead Ridgeline's AI features are compared to anything else in investment management software. which is why I believe that firms that come out ahead in the AI era will be the ones running on Ridgeline's unified platform. If you're serious about your firm's AI strategy, Ridgeline should be part of that conversation. You can request a demo at Ridgeline.ai. you All right, now let's talk about the very clever stack of solutions that you hope to build. Ultimately, they have this giant token factory supplier of extremely low-cost intelligence. I think you think about this in terms of software, hardware, and power.
[15:00] Talk through what your master plan is to approach this challenge that's so different from what others are thinking about doing. We always had to start with software. Where is the opportunity on today's data centers to improve efficiency? And the first thing we did was we tried to build the entire LLM software stack around peak GPU efficiency, meaning we're using NVIDIA GPUs. We wanted to squeeze out more tokens from the same chip than anyone else in the world. And that starts with the lowest level of programming, kernels. It's actually my background. I spent my whole professional life working on GPUs and kernels. NVIDIA was my first job while I was in college. I got to see how the Tensor Cores got to earn their rights. to be on the chip. What does that mean? Like, what is the tensor core? Tensor core is a specialized unit on the GPU that accelerates matrix multiplication. Simple as that. There's been a long history of how we evolved that tensor core over time that we'll get into. And why is matrix multiplication so important? I cannot say that there is a divine truth of the universe that explains why matrix multiplies seem to be the atomic unit of computation. But one way I've heard it described to me is, well, it's a really succinct way to mix two blocks of numbers together and have them interact in some interesting way. That's as much as I can say about it. It is really convenient that linear algebra turns out to be a very compact representation of arbitrary relationships in data. So NVIDIA... Great graphics company. Had market share dominance in GPUs and gaming graphics for quite some time. And then starting in like the mid-2010s, they started to actually start these like Skunkworks projects to make the graphics processor more suitable for machine learning tasks that they were tracking. I remember actually reading some of the lab notebooks of some of my managers when I was at NVIDIA. They would visit these small ML conferences like ICML or NeurIPS at the time. They would just take note of these papers like, oh, this deep learning thing seems to be catching on. using gaming and video GPUs. in order to train their large models. We should double look on this and figure out what's going on here. By 2015-2016, At least Jensen had the conviction to kind of double down on, hey, this usage of our chips is only going to grow. Let's start allocating more and more precious silicon die area to this capability that seems to be emerging. Let's put the first version of Tensor Cores on the chip. So we're talking about taking this gaming chip, which is designed for painting pixels on a screen. and adapting it to do mission-to-do-play. It was early. You would be competing against the graphics teams, essentially, when you ask for more silicon area at any chip company. There's always competition for that. It is something that the designers guard so carefully. You don't ever want to invest in the wrong technology because that's opportunity cost that you could have allocated to some other functionality. We fought tooth and nail and got just a tiny bit of diarrhea, maybe like 5-10%, something like that, for the first generation of these chips. to get some amount of acceleration for basic which were the fundamental operation for computer vision models of the day. And then we had a software team that was trying to squeeze all the performance we could out of the chip. And I think on that software team, which is where I work, that's what actually taught me the most about the ethos that NVIDIA has around this term called speed of light. They always chase the speed of light for any piece of hardware that they make. It is so ingrained in every engineer's mind that if the machine can do it, we're going to push the machine to the frontier until it does what we think is. And the speed of light is the edge of what's possible. The speed of light is the edge of what's possible, exactly. If we think the chip can run at this frequency and produce this many multipliers per cycle, we're going to get there. We're going to break every bottleneck and get to that peak level of performance. To this day, I tell all my engineers, we're chasing 100% speed of light. I don't care about relative numbers versus the competition. I only care about absolute numbers. What are we able to do on the chip? How do we achieve that? Before we leave that chapter of your time at NVIDIA, anything else beyond that cultural touch point that changed the way you think about things or... that stood out the most about how the business ran back then or its culture? I have a ton of stories about NVIDIA. I can tell you a few of them. One of my favorites is that on the tenure side, a lot of people I worked with in NVIDIA in 2015, 2016 are still there today. That company has incredible retention, and these are the best engineers, on the silicon side at least, I've worked with them my whole career. They're extremely, extremely motivated and passionate. They believed in parallel computing as a concept. through its various incarnations and have loved seeing the chip evolve. This is their life's work and they're extremely competent in that direction. They're also a very frugal company. NVIDIA and all its Silicon Valley companies, after 2008, they had some cutbacks and perks. So no free lunch, for example. NVIDIA took it one step further. There was no free milk in the fridge. So if you wanted to drink coffee at NVIDIA, and you wanted some milk, you'd actually have to chip in a dollar every month to the milk club, and the milk club would stock Costco milk in the fridge. And I remember that distinctly. We don't do that at sale. It's a frugality that permeates the company. So coming out of this time there, you get this experience of what it's like to develop more efficient usage of the underlying hardware through software. So link that to today's environment. The GPU is fundamentally a throughput machine. The GPU is happiest when you give it a lot of that's actually not the way that we've taken AI in the last couple of years. We've pushed AI to be an interactive chatbot tool is the most common form of AI usage today. In that world, you care a lot about actually spitting answers out to the person at the keyboard as
[20:00] as possible. To your point about don't make the user wait, I want things as fast as possible. That's actually quite interesting for the GPU. It's very difficult to put the GPU in its happy path of being fully compute utilized when you're trying to spit out tokens quickly. There's a fundamental trade-off on the GPU between being throughput-oriented or latency-optimized. And everyone has chosen latency optimization because the shape of usage was chatbot-oriented. I believe that's the most profound change we're going to see in the next year. We're going to move away from chatbots to more proactive or background agents. In that world, it makes a lot more sense to build a stack around throughput. Can you explain technically why the trade-off between throughput and latency is unbreakable? Why can't we have both from the same hardware? It's quite foundational in almost every system that you could ever possibly look at. There's always a trade-off between getting a small amount of data through the system as quickly as possible and leaving a lot of buffer room for that, or trying to run wide and slow. Narrow and fast or wide and slow is like a classic trade-off in all computer science. But for GPUs specifically, I think there's one thing to focus on, which is there's this concept of like batching on the GPU. We want to group many users' work together into a batch that we can run all at once on the GPU. That the parallel processing of the GPU We like to have a lot of parallel work to do The thing is though you doing net more work when you run a large batch of compute together So you might be filling all the units but every step along the way as you carry a batch of work through the GPU there more work to be done. Any individual token or any individual user's request in that batch, it's going to spend a longer time on the GPU being carried with other people's traffic. Maybe the way to say it is, if you want to get downtown in SF, you can take the bus or you can take a private transit. And The private transit is going to have its own direct path as the crow flies or using exactly the roads that you want from point A to point B. A bus, it's going to have to serve many more people and it has to fundamentally do something that works for everyone. So it takes a slower path and it stops and waits for other people to get on and off. I think the bus versus car analogy is pretty accurate. It's a great analogy. And step one for what you're trying to do is create the best possible bus on top of NVIDIA GPUs. Like that's step one of your optimization. optimization. That's exactly right. It means we explore things like different parallelism schemes. And maybe that's another example I can give you is with NVIDIA GPUs, one of the things that they've really innovated on and done a great job with is the NVLink interconnect between GPUs. And in fact, that NVLink system is so good that if you have a large matrix multiply that you want to perform faster you can actually cut that matrix multiply in half and shard it across two or more up to eight let say NVIDIA GPUs and have them all work on pieces of that larger matrix multiply and have them connect to their results together at the end, reduce their results back together at the end. This is a great, great way to cut the minimum latency of an operation. Each GPU is now doing one-eighth as much work, let's say, and therefore can finish faster, but not eight times faster. It's sublinear scaling. You'll use eight times more hardware, but you won't get eight times the speed. You might get like four to five X the speed. You're not going to get strong scaling. This is because of communication overhead. It's because every GPU is going to be a little less efficient working on a smaller tile of work than a larger tile of work. It's the only way to speed up. If you want the minimum latency possible, you can do that. But it is not the choice I would make. For example, I would prefer to use a different parallelism scheme like expert parallelism or pipeline parallelism. We may do interesting things to overlap and hide the communication latency in a way that you would have less ability to do that for a low latency service. So is there a way to think about NVLink as a technology which improves latency performance? Yes. And only latency performance? Which will segue into the next segment of what we do differently as a company. But yes, NVLink is mandatory for low latency inference. So NVIDIA is excellent at low latency inference, and I telling you that we don really care that much about low latency inference So where does that leave us I not holding my breath for other companies broadly to figure out NVLink quickly It challenging technology to figure out It's hard to scale. It's hard to productionize. If I do have some other vendors chip and it is good at the foundational compute components, it can still be much multiplies really well. It just can't communicate those results across its peers quickly. Well, maybe there's room for that other chip in my stack as a really, really good compute per dollar option. That's what I actually optimize for in most cases is how many flops does this chip have and how much is it going to cost me per hour to own and operate? There are other chips that definitely rank higher than NVIDIA on flops per dollar, but they may not have as much interconnect. So it's my job to figure out what parallelism scheme am I going to use that's going to make this chip suitable for inference. It's not going to be tensor parallelism. NVIDIA is basically mandatory for that. But other techniques may work well for me. So before we leave the latency part of the story, Can you comment on companies like Cerebris or others that can perform incredibly fast operations? I'm curious, like what you think about those approach to those companies, what might happen in the future? What is your prediction for the future of very low latency focused hardware? Cerebris, Grok, and a couple others that are coming out of stealth now, I think have made a very interesting bet on not just building another GPU,
[25:00] actually building a different kind of accelerator that focuses on a different memory hierarchy. They want to maximize the amount of SRAM on the chip and use that as very, very fast memory for weights and kv cache. So SRAM versus DRAM, there's two ways to make memory for a chip. One is to integrate the memory on the logic die itself, meaning you tell TSMC, I want this many megabytes of storage on my chip, and there's a way to build that. TSMC has a standard cell library you can use, and you can just print out a bunch of cells of SRAM. The problem with SRAM is it It takes a lot of area on the silicon die. So if you want to build a large die, like let's say the NVIDIA Blackwell at 800 millimeters square, if you made that whole die SRAM, it would be in the maybe like single digit gigabytes, it feels like. It's not a crazy amount of data storage. Compare that to if you're willing to take a different process technology entirely. So not TSMC anymore, but now Micron, SK Hynix, Samsung. They build DRAM, which is a whole different way to build memory that's more focused on capacitors than transistor cells. The standard way to build SRAM is what's called the 6T transistor cell. It's a stable transistor arrangement that allows you to write a bit to it, and then it holds that state in that bit regardless of whether you keep applying Well you had to apply some power but it holding that bit without any sort of like active management It static Now dynamic RAM DRAM it dynamic because what you do to write some data is you write a charge onto a capacitor And as soon as you write that charge into that capacitor, the charge is dissipating. The dynamic part of DRAM is that you must, every 50 milliseconds or so, refresh every bit you've written. So you're constantly juggling billions of balls in the air, essentially billions of bits. had to be managed by a memory controller, which is reading and refreshing every bit on the DRAM. Now, the benefit of that is you can get much, much higher density. And it's a whole different process technology. There's a ton of different trade-offs. Hence, why we split the DRAM manufacturing into an entirely different company like Micron, SK Hynix, and Samsung. These are the best companies in the world to do this. They build DRAM. And if you take DRAM from those companies and you stack it into many layers and you kind of print them or solder them around the main logic die that you get from NVIDIA, you can now get hundreds of gigabytes. Blackwell has 288 gigabytes of HBM capacity around the logic die. And the logic die itself maybe only has like 500 megabytes of SRAM. So it's possibly multiple orders of magnitude, three orders of magnitude difference in density for DRAM versus SRAM. So let's go back to Cerebris. What are they doing? They see this problem There not really an obvious way to increase SRAM density on the chip But the with SRAM is because it so physically close to the logic gates that actually do the computation the arithmetic logic units are right next to the SRAM that they going to pull from The compute units that are doing the matrix multiplies can pull data from SRAM at mind-boggling speeds. Cerebus quits petabytes per second, 21 petabytes per second for their wafer scale engine 3. Compare that to HBM on an NVIDIA Blackwell is 10 terabytes per second or so. in that range. So once again, many orders of magnitude difference, more capacity, but proportionally less bandwidth, essentially. What Cerebrus does is they say that we're going to take as many of these dyes as we can. We're not going to limit ourselves to the 800 millimeter reticle limit, the TSMC, 800 square millimeter limit that TSMC imposes on us. We're going to take the entire wafer and have every dye connect to every other dye over scribe lines. And we're just going to try to get as much SRAM as we can on the whole wafer. And we can get to like, let's say 50 gigabytes of SRAM per wafer. And then we're going to stack many wafers together in a pipeline or similar. And now we can have up to a terabyte of memory, very, very fast memory. You do all that work just to get to the ability to read data from SRAM at 21 petabytes per second per wafer Therefore you can now serve these language models at extremely high tokens per second because you can move the entire parameter count of a large model like Kimi You can move all that data in and off the logic cores in about a millisecond or something like that So there you go. You have a path to a thousand tokens per second. So what is your prediction for like that segment of the market? What happens to them is some hybrid. We had to pair the Cerebris chip where it's very strong. It's very, very good at fast access to memory with something that has more capacity for memory. It's true that you can take a 1 trillion parameter model like Kimi and fit it on a large number of cerebrous diet wafers, but you can't do something about the KV cache very easily. The KV cache is something that grows as people use the model more, and that is always dynamic. You don't even know how much KV cache you're going to need. It depends on how many users you have and how many users you want to serve. Can you explain KV cache? KV cache, when you ever use a language model, Every token you send through the language model stays in the context window of the language model for as long as you're having a conversation. We talk for 100,000 tokens, the 100,000th and 1th token is still in the conversation behind us. And the model is referencing all the past conversation history in order to make better predictions about what the next thing we're going to say is. That KVCache, it's a bunch of memory. You have to store a representation forever.
[30:00] token that we send through the language model. And it frequently gets to be larger than the weights of the model themselves. You have this crystallized knowledge in the model weights, and you have the dynamic knowledge of the exact conversation we're having in the KB cache is the way I like to think about it. And this is why sometimes people would observe deep in a conversation, things start to degrade because there's some sort of technical problem. So the KB cache is quite interesting in that regard. The KB cache is an exact representation of everything that came before. We store all the information that we've seen in the conversation. However, during training, the model did not get trained primarily on very long context conversations. It got trained primarily on, let's say, 8,000 token conversations or 16,000 token conversations. So if you take the model to 200,000 tokens, there was some training that happened at that context length, but it's not the model's core strength. And so there's always been a challenge for the Frontier Labs to figure out how do we make the model exactly as intelligent at 10,000 tokens as we expect it to be at 200,000 tokens. And it's going to be a perennial battle for us. We've had 1 million context windows as a concept for years now. Anthropica was, I think, the first to hit that 1 million context window length. I still use slash compact in my quad code well before 1 million context length. I don't think it's actually great to hit the full length. These extremely fast extremely low latency approaches ultimately are limited by this factor Yes You can do whatever you want for the weights It very possible to have much unbeatable performance on weight storage However the KV cash is going to be a big thorn in your side. So three years from now, five years from now, what role do you think these kinds of chips play? What sort of market share do they have in the heterogeneous chip market? Cerebris and Grok and maybe a couple others, you should think of them as accelerators. What they are really good at is being used in conjunction with a more traditional GPU-like device that critically has this off-chip memory built in. You want off-chip memory for capacity and on-chip memory for speed. We want to hybridize these two things. So if you take transformers in the limit, you take a transformer to a million context length, what ends up happening is you have this compute bound stage, which is the actual matrix multiplies for what we call the MLP, which is where most of the models' world knowledge is encoded. and then you have the attention layer, which is where we're dynamically adapting to the current conversation. Attention in the limit is usually memory bound. The MLP in the limit is compute bound at large enough batch size. I would say the original sin of Transformers is that you've taken this extremely fundamentally memory bound layer and juxtaposed it right next to a compute bound layer It is very difficult to have a single chip that is good at both compute operations and memory operations The GPU is quite balanced in this regard but you had to choose one or the other Cerebrus has a very fast memory access for something like a matrix multiply and it's really good to host the MLP, the weights, essentially, on the Cerebrus chip. But the GPU has the capacity to scale to really long context lengths. You would like to put the attention possibly on the GPU and the MLP on the Cerebrus chip. And I believe this is what's happening with NVIDIA and Grok. Can you riff for a minute just on Transformers? For people that, again, aren't deeply familiar with what this innovation was in 2017, what its strengths and weaknesses are, and whether or not you think it will remain the dominant architecture or a dominant architecture for the future of AI. What it did, it allowed us to learn unsupervised data really effectively. Because Transformers, what they're all about at the end of the day, is taking any sequence, any arbitrary sequence of data, and trying to find patterns in that data. And they critically, the attention operation, which is the headline component of transformers, it allows the model to dynamically adapt to what it thinks is the most relevant component of the sequence. Every step you take through a transformer you are essentially like re the input that you looked at before and figuring out which is most relevant for your next prediction It extremely amenable to learning arbitrary sequence data And the most interesting sequences of data that we produce on a regular basis is language. And that's how we got to dominance in the language regime. To zoom out even further, I think what transformers really did well is that they scaled. Transformers make no such human prior. Transformers just say, well, there's going to be a pattern in the sequence of data. And if there is a pattern, I'm going to find it. I'm going to throw more and more parameters at this problem until it works. Transformers benefit from a lot of the computer vision work too. For example, one of the challenges in computer vision was we had a hard time going from hundreds of thousands of parameters, which you get for linear models like support vector machines or other legacy machine learning models. Those had thousands of parameters. Then we got to deep learning and got to tens of millions of parameters with computer vision. The biggest models were 150 million parameters, was a huge model for computer vision. And now we routinely talk about trillions of parameters. And transformers are the link to go from millions to trillions of parameters. So if I think about the important units of scaling being data and compute, does it stand to reason that you think transformers will just stick around because that's the thing that we're good at getting more of, those two things? Transformers are such great sponges. You increase the compute available to a transformer by 10x and you'll get some log improvement somewhere. And so far, the scaling laws really work. They're really quite beautiful.
[35:00] And to the point about what do transformers do really well, they extend to almost any data set you can throw at them. They're extremely powerful general learners. And I think what's especially useful about transformers over other techniques that we've tried to replace attention is transformers represent any pairwise relationship that you want. Any token in the sequence can attend to any other token in the sequence. So if there's any relationship that's in the sequence at all, you're going to find it with a transformer. Now, it may be the case that you don't need all-to-all modeling. You don't need every token to look at every other token. But if you need to, transformers give you that option. And until we know a better way to prune that space down, a better way to kind of have information modeling be more selective, attention is a very, very good operation. This is another trick that we learned in the computer vision days. One of the old Karpathy sayings is that if you have a new data set that you want to train a model for, your first goal should be to over-parameterize the model and try to overfit the data that you have, to prove that there is a relationship that you can model or memorize, that your learning algorithm works, that you can instill knowledge into the model. Once you can overfit, then you can compress. And the compression is how you get generalization. You don want to actually memorize the data that you have in front of you You want to generalize and therefore once you overfit the data set then you can kind of work backwards and try to find the general patterns that fit into the smallest parameter count possible What your prediction for the future of data and riff on the importance of data in this whole story? I like the phrase that internet was a one-time subsidy on data. We got it for free. It's extremely high quality. There are about 30 trillion tokens of high quality text, 300 trillion tokens if you take a wider view on what qualifies This is good text. And we've basically looked at it all already. Models have seen the entire internet many times over at this point. And there is not a whole lot more to be done on human data from the internet. The next phase of data in my mind is model self-improvement through RL environment, gyms, basically. Now, in fact, we don't even benefit from getting more random user interactions with AI. It used to be that the new type of data that we cared about a lot was the interaction data from people using ChatGPT and giving ChatGPT signals on what they liked and didn't like. I like the argument now that the median model that we serve is so much more advanced than like a random human giving feedback that the signal you get from random human preference, unconditioned human preference is not actually worth anything anymore. You want expert human preference at this point The model has outgrown Everyday Joe Yeah everyday Joe Exactly So the future of data to me is giving the model a hard verifiable task and letting it run in this gym where it kind of isolated and it just has a problem that it can make progress on and get measurement of whether it made progress on that problem or not. You can imagine coding problems are in this category. Math problems are also in this category. More and more, we can just give the agent a computer, essentially, and have it act like it's a human worker and give it feedback on whether it's making progress towards the target outcome. That environment becomes the data. I think this is not a super differentiated take, but it's been really, really productive from what I've seen so far. And you think that just goes on for a really long period of time? Or is that another, like, if I think about the internet as this one big block, like this is another big block that will have its day in the sun and we'll kind of get it all and then we'll have to move on to something else. I think it's actually more profound than that. Basically the idea is that if you want artificial general intelligence, the best way to get there is to just keep stacking specialized intelligences until you have no more gaps to fill. The only thing you need to make sure you do to make this work is you must make sure that your task is verifiable. You need to give the model a self grading system. If you have that, you have the recipe for self improvement on any task you like. I think you've seen this held up by the way Frontier Labs spend. They spend that much on data. Now they spend a lot more on RL environments These environments absolutely capture that relationship of recursive self improvement on a verifiable task Coming back to your initial task of making existing hardware more efficient by being more in control of what going on at the hardware level through software Keep going on what you've done so far and what you want to do. And then we're going to jump to hardware and then jump to energy finally. I mentioned kernels. It's surprising. People think kernels are done. There are great people like Tredow who write excellent kernels, and they form the bedrock of all of our modern deep learning is built on flash attention. Modern transformers are built on flash attention. But if you deviate from the happy path at all, if there's a new model that comes out that has a slightly different way to embed positional information, like the change of the rope system, suddenly the kernel that we had is not suitable for this new model. And we may have to make a patch to this kernel. I wouldn't say we're in the phase where we had to invent new kernels from scratch, but having the ability to quickly modify existing GPU kernels, sorry, a kernel, by the way, is a general term for any program you run on the GPU. Historically, kernels tend to be put into a library where every kernel has a very, very scoped purpose. Typically, you have a kernel for a matrix multiply. You have another kernel for even something as simple as addition. You want to add two tensors together, that's another kernel. And then increasingly, we've started to fuse those kernels together. So if I do a matrix
[40:00] apply, and then I want to add it to another matrix that I've also multiplied, maybe those two become one kernel, and I just fuse the operations where sort of writing the data out to DRAM, and then reading it back in just to do the addition, maybe I can just do this easily. Why are humans still doing this? Like, it seems like the sort of thing that AIs would be exceptionally good engineering, more efficient kernels. Maybe that's where we're going, and we're just not quite there yet. But if we aren't there yet, is that where we're going? If we're not there yet, why humans still doing this? Why is TreeDow so well-known? You know, it's a name I know. I don't want to speak for Tree, but what he taught me was you shouldn't write kernels by hand anymore, necessarily. I like to say we write kernels on the whiteboard. We go to the whiteboard. We describe what we think the machine should be doing. Then we succinctly describe that in natural language to a model. And then the model is able to do the execution of here is my input and output. Here is the strategy of how we want to dispatch this work onto the GPU. I'm going to go implement it. So we're doing the conceptual design. Exactly. I'm not sure exactly why models are not superb at doing this. I don't think this is like our moat or anything like that. I'm sure in six months' time, we'll have much better models on kernel engineering. And I'm sure the labs would tell you that they already do a lot of their kernel engineering in a fully automated way. So software as an edge if I think about software as maximally near speed of light efficient usage of an underlying piece of hardware is going to trend towards not being an advantage for a company like yours over time I think that right The rising tide of something like Mythos or GBD 5 Sol that lifts all boats. It really does. I actually don't think there's a point in specializing to say we work on making the model better for just kernel engineering. I think that's actually not the most meaningful subset of coding in general, kernel engineering in particular. Maybe there's some privileged information you inject into the prompt that's like a useful way to steer the model to be better at writing kernels. But broadly speaking, yes, we're all downstream of the frontier in terms of this capability. I always love this from the history of energy. There's always this pendulum between the raw source, let's say coal. And then if there's a certain amount of energy available inside of a hunk of coal, what percent of it we can harness and use? A big part of the history of energy was getting that number from 10% to 95% or whatever. If I just think about a Blackwell or something, and Blackwell is the piece of coal, what percent do you think we're at? How efficiently can we use an existing piece today? There's a lot of different ways to analyze that. In some level, we are really efficient at optimizing the performance when the GPU is doing the thing that it most happy doing which is a large dimension matrix will apply That operation runs at 70 of peak utilization and it limited not by software but by power The way NVIDIA quotes peak flops is a little optimistic. You never hit that because of power throttling, because of heat. Exactly, thermals. In practice, you don't spend the majority of your time in a transformer in that happy path where you're doing a large batch matrix-level deploy. Our job is to basically build the engine around the chip such that we are feeding the GPU these large batches of work at all times. One of the most profound transitions we've had in the GPU world in the last year has been this moving of, you don't program one GPU at a time anymore. You should think about the whole rack, and maybe you should think about the whole cluster, the entire data center at a time. With NVIDIA, they've started shipping not just a single GPU or a single motherboard, but actually the whole rack system is something that they prescribe. They call it NBL72, their latest chip, the Grace Blackwell 300. That ships as a rack of 72 units, and it's an open race to figure out who can program the whole rack scale computer as efficiently as possible. And my belief is that that shape of compute is the future of efficiency and speed. In fact, the video does a great job of if you want the lowest possible latency, you should be using that chip. And if you want the highest possible throughput, you should probably also be using that chip as of right now. And it all comes down to like, this is a very new paradigm of programming One of the things you hear is that the market for the best chips Blackwell let say is like a drug market or something right now There all sorts of fascinating things happening to get as many of them as possible because everyone so short I hope you react to that analogy. Is that what it feels like? Then also to talk about what the market is like for not the bleeding edge chips. If I am willing to accept a slightly or moderately inferior chip, what's that market like? Let us into that world. Media has a long-term view on all their chips. They see this immense demand for the black gold chips. They can do what other suppliers have done in the past, which is crank prices, meet the market, supply and demand curves will correct. They'll intersect at some point and everyone will be technically happier. But NVIDIA sees if they just let the most deep pockets buy all the chips, that maybe hurts them in the long term. If that customer ends up accruing a lot of more power, they understand the compute is power today. They're quite strategic about how they allocate compute. That's the first thought. The Second thought is that relationships matter a lot. Nobody wants to have a huge order of a chip rental come in from this new startup that says, oh yeah, I'm going to rent 10,000 blackwells for three years or five years. This startup has only been operating for months, typically. Who knows if they're good for the money? The way you convince someone to give you access to compute is quite challenging these days and requires some pretty great relationships for...
[45:00] for just incredible financial backing to make this happen on the Nvidia side. And it's all because the scarcity is so high and demand is just off the charts. Now for other chips, I wouldn't even call them inferior. I like to say there's no bad chips. There's only bad pricing. I will make any chip work at the right price. That's one of the ethoses of the company. Let's talk about AMD. AMD, I think great chips overall. The challenge is that people don't understand how to program them very well. I've been talking to you about how we have such a great kernel team. We're so serious about squeezing the performance out of the hardware. NVIDIA is pretty good at doing that for their own chips, frankly. There's some alpha that we can squeeze out, but there's a lot more to be done other chips because the vendor does a little bit less work than NVIDIA does to make the best kernels out of the box. Or even better, there's alpha and just other people have this perception that AMD is not as good as NVIDIA. That's music to my ears. I'm very happy for them to sleep on this chip and for me to buy as much as I can. Now, I think that that's not actually super true anymore. I think AMD is actually somewhat popular amongst some large buyers. I think publicly Meta and OpenAI have bought a ton of AMD chips We increasingly seeing that all the AMD supply is also being allocated But there a long tail of other companies that are popping up Net new companies are great such as Hatch or SambaNova or Dmatrix All these companies are popping up I think the main challenge for them is scale Can they actually get enough wafer allocation for TSMC to pump out chips to make it into the market? If there's a new chip on the market, I'd like to know about it as quickly as possible and evaluate whether we can buy a good fraction of that supply. And so it's fundamentally an arbitrage for you. If you can be much better at eking out performance from chips that have received less attention, you can then resell that at a margin and it can be a great business. Exactly. And I think that it's not the case that everyone else has a skill issue that they can't make these chips work as well. I think we're quite competent in this. I think we're probably one of the best teams in the world to use multiple silicon architectures and be quite aggressive in chasing down performance in unlikely places. But yeah, I think it's the speed at which we're willing to kind of build our stack around a new chip. We don't have a huge amount of incumbency around, well, our data center providers are only stuck with this class of chip, and it's going to be a huge pain for us to deploy these net new chips. We have some very creative data center partners who are willing to move very quickly. And there's a new class of those that we can talk about. Most importantly, we don't shy away from the challenge. That's frankly a big part of this is just saying, yes, we love TPUs. We're going to make TPUs work. Yes, we love Tranium. We're going to make Tranium work. And if it doesn work easily we going to find a way to fit it in with a heterogeneous serving system It will have a place Every chip has a comparative advantage We had to find that advantage and then squeeze it in that direction Just as an interlude before we get to hardware data centers energy et cetera which will be a really fun part of the conversation, I'd love you to talk about your perception of the investor class's worry. If you look at memory stocks, or my current favorite is you look at the chart that plots the percent of the S&P 500 that's semiconductors. Historically, it was like 2%, 3%, 4%. Now it's 19%, 20%, 21%. And it just sort of looks like if you're a student of market history, you get all these things through time that have sort of reached some crazy near-term peak and then collapsed back to long-term norms. That has all investors worried. A lot of people have made a lot of money in Micron and SK Hynix and companies like this, but everyone feels like long-term like computes a commodity, it will not represent a quarter or fifth of the entire market capitalization of the world. They're scared and that's the setup. Everyone acknowledges that there's a huge shortage. Everyone sort of feels like, ah, we'll figure it out and these things will revert back down to their normal place in capital markets. I'm curious what you think about that narrative. I'm less of a student of history as more of a member of history. I was born in 1997 and my mom worked at Intel in the run up to the year 2000 and the dot boom and crash I remember the time where Cisco was the most valuable company in the world and Intel was close behind I mostly draw parallels to that period of history from 25 years ago to today And I think the main difference is that a lot of the investment in networking equipment historically was speculative. We anticipated this future demand for users that never came. And I think what's interesting about token consumption or AI consumption broadly is that it's no longer speculative. People buy tokens because they're immediately valuable to them. You don't hoard tokens, you use them immediately. This is also even different from what we had two years ago, where there was a supply crunch for Hopper generation chips in 2023, 2024. In that period, it was all training-oriented spend, and training is inherently speculative. Now it's ever instituting caps on how much you can spend on cloud code. It's a very, very different world to be talking about inference spend and predicting inference spend to go up. I do think inference spend monotonically increases. There's no speculation on inference. Coming back now to your take on hardware, the unit level is interesting to me. Talk about chips, talk about racks, talk about clusters. I'd love to talk about data centers. You said you've had some interesting partners doing some cool things. Talk to us about the present and future of data centers as you see it, because this seems like obviously a critical thing for being able to serve all this inference.
[50:00] So there's lots of innovation in this part of the world, and obviously you're focused on it. I think one of the themes in our conversation has come back to what is training versus inference? What is the difference between these two categories? And what was different about two years ago being training-oriented and today being inference-oriented? And I think the most conservative players in the entire AI stack have got to be the infra players, whether they're data centers or even more conservatives, TSMC, the chip infra people. Data centers, they were built for training. Training is a superset workload over inference. You can make any training cluster work for inference, but maybe not vice versa. The difference there is networking. How much do you invest in bandwidth between chips? And how large of a cluster do you need? There's actually a diseconomy of scale to data centers in some way. It's way more expensive and difficult to build 100,000 GPUs in one data center than it is to build 10,000, than it is to build 1,000. Now we just talk about how many megawatts or gigawatts do you have. And basically, there's no way to build a gigawatt data center in the United States easily anymore. Even 100 megawatts is increasingly hard. It's basically impossible unless you're a very special set of customers. 10 megawatts is probably on the edge of what's possible today. And one megawatt, I would argue, is plentiful. So there this incredible floor on the market where you can find lots of aggregate power but it will not be concentrated And that was not interesting to anyone who building training data centers because you just assume at least all be in one spot for no one wants to deal with cross data center training The market has some lag in it I think that the market still assumes that we had to go shake down those 100 megawatt and 10 megawatt data centers wherever we can bind them. It's still the attitude I hear from a lot of data center developers, but increasingly we're seeing a few new thinkers realize that inference is going to be suitable for these distributed one megawatt data centers. We're quite in agreement with that. And we are very happy to buy small pools of compute across the United States and use that as our inference fleet. Give us a sense of literal physical size of one megawatt versus 10. Yeah. Well, so this got really wonky with the advent of liquid cooling. Now you can pack insane levels of power density into a single physical rack, a megawatt of compute. You'd imagine this like massive data a huge warehouse basically. And now you can actually pack that into around like eight racks for the compute. Each rack is about the size of a refrigerator. You can just imagine eight of them lined up. That's a megawatt. Your view would be that the future that you want to help build is a whole bunch of different chips that can be used together that you can buy your buyer to eke out the most per chip and that those chips can then be coupled in very small data centers to just do inference Those two steps of a whole bunch of random compute some of which is cheaper than it should be, your ability to eke more out of it, and then small units of expression in the data center equals way cheaper intelligence. I certainly think so. Yes, there's a lot of ways to access cheaper flops if you're able to be creative with what you take. And so one of the ways that I describe what we do is we will buy any chip anywhere in the world for any duration of time. That is a level of flexibility and liquidity that no one else has right now. We're very aggressive about putting our money where our mouth is, and we will take any capacity and find a way to make it work in our fleet. And that is a big part of our advantage today. Long term, we had to create more of that advantage by investing in these data centers that other people are going to be skeptical of. Because what's going to happen when you set this army of a thousand small data centers versus the one big gigabot data center. A few things. You're not going to have power redundancy. Quite often, you're not going to have backup diesel generators on site. Those are all very expensive. We cut all that overhead. We're not even going to have redundant networking in a lot of cases. We're going to put these in facilities where we have good access to power, a single source of power, and we're going to trench one line of fiber to these data centers, but we not going to have three lines of fiber with redundancy and failover in SLAs It just going to go down sometimes I won be surprised if some of them get down to like 95 uptime Which is bad Very bad That fatal atrocious for anyone else Couldn survive in a big gig You have this be zero buyers for a data center that has 95% uptime. I'm that first buyer. I will buy 95% uptime. And the reason for that is because of this background engine thing. If things are running in the background, you don't care? Partially. It's actually two things. One is that we have a really robust control plane that is going to be fine handling any single failure in any single data center, as long as it's not correlated with other data centers and I can just move the workload somewhere else. I'm cool with that. The failures happen at some rate, and I am basically linearly happy with a data center that is 95% uptime versus 98% versus 99%. It's just linearly good or bad for me. You do need that async piece that I mentioned. We serve these long horizon agents because what happens when a request fails is that I'm going to have to go find a new GPU to put that request on. And that means that for that single turn of the agent's work, it's working for an hour, but then hits a roadblock because its GPU got pulled away. In that moment in time, that agent is going to experience maybe like an extra minute or two or three, maybe even 10 of latency. My argument is that my customers don't care because their agent was running for hours. They're sleeping. It doesn't matter. It's
[55:00] longer. We tell our customers, look, our average throughput is going to be very competitive, but our P99, our 99 percentile latency, is not going to be controlled. It cannot be. And in return, I'll give you unbeatable economics. And I think that's the right fit for background agents. Talk about power as a category. What the innovation that you're seeing is, where you think it goes from here? What are you seeing that's interesting, innovative? Where do you think this goes? Okay. So I said I want 95% uptime on my data centers. Could I even take 80% uptime at the right price? Probably. And what does that mean? Well, I'm a son of California. I love solar and wind. I think solar and wind power is way undertapped in the United States. And the challenge has always been this intermittency. You would even consider solar and wind unsuitable for data centers because you have a persistent base load and the intermittent power source. What are you going to do? Well, I think we're actually not that far from solving that problem. I am totally capable of tolerating a outage from a data center that's measured in even days or weeks, which is like the worst case nightmare scenario for a data center is that we're going to have a long-term outage because the wind doesn't blow and the clouds are in the sky. Fog is hanging over the valley for some time. That's the worst case scenario. It's in fact highly predictable and I can just call in capacity in some other place in the world whenever that happens I just model the weather and figure out when my data is going to be offline move my work with somewhere else and it fine The trick is that it going to give me better access to power that no one else is going to touch because it is so annoying to deal with that kind of outage And if my chips are cheap enough, they're probably not going to be NVIDIA racks. If my chips are cheap enough, I don't mind the capital cost of having idle chips. I've heard you describe this entire system as like a scavenger strategy. Yeah, that's right. Unpack that analogy a little bit. Well, first we scavenge chips and then we scavenge power for those chips. The idea is in both cases, I do not want to be bidding against Anthropic or OpenAI for a compute capacity. I'm not going to win against them and I don't want to. I want to be more creative and use the supply that they don't find legible today. And over time, I amass enough aggregate supply. I'm never going to get concentrated supply. I will only get aggregate supply. And over time, I build my aggregate factory that is unbeatable in economics. We are building a factory. We're trying to build the best steel factory in the world, but it will come through many mills, not through large monolithic steel plants. And if I imagine the different versions of this, how vertically integrated you can be, the extreme would be you own everything. So that's a very capital intensive business. You own the power source, you build the data centers, you design your own chips you control the software that eats the most out of those chips and you sell the end finished token to your user Your user is me and you just own the whole stack But you can imagine many other permutations of the business where draw the line anywhere. You could be incredibly capital light, own nothing and just be like the coordination plane across all this stuff, the virtual scavenger. How do you think about that question of which type of these businesses to be? You know, there's actually two parts of me to receive that question. One is the CEO of a company that needs to work every day and grow sustainable and as quickly as it possibly can. The other is the founder. And the founder is much more imaginative and loves this stuff. The founder in me wants to do everything. This is my entire life. I spent my entire life thinking about chips, power, energy. All I care about is this stuff. So of course, I want to be maximally ambitious. I don't want to stop ever. I will never stop until I have built the most efficient system from soup to nuts. You're doing real life factorial, basically. Very much so. For intelligence. Very much so. That's like the emotional from the heart answer. on the CEO side, I think we have to be more pragmatic. The capital we're looking at for owning everything is insane. Software has high leverage, so we have to start with software. But ultimately, do we own power generation or can we get great power purchase agreements with utilities? I'm more inclined to pursue letting other people specialize in the things that they historically good at and then see if we can get to this scale I think of it as like I want to get to the scale where I earn the right to take this under our wing I absolutely think that there efficiencies to be gained everywhere in the stack If you can break the assumption that people I would be buying from, they made assumptions about who their customers would be, and I maybe break those assumptions. It's a pretty optimistic view. I think it's only possible because we're actually trying to underwrite the largest market for compute in the history of computing. We're actually going to build so many millions, trillions of dollars of investment into inference. Because of that focus, it makes sense to build a lot of things that are custom for inference. And it's my job to seek all the places where that's possible. And then as they become obvious to me and my partners, I'll look at my partners to build custom things for me. And if they can't do it for me, I will do it myself. If you had to just zoom out on this entire system, software, hardware, energy, et cetera, and stack rank the places that you think that we are the most inefficient today at producing useful intelligent tokens. What does that list look like? I think compute scaling is actually very efficient. You give me more flops and I will use more flops. I would say we're actually fairly judicious already with our use of flops. If you look at a modern MOE model, there are very few models that are more than 10% dense, meaning 10% of the possible number of experts you can activate are activated.
[60:00] I think the frontier models are close to like 1%. Fairly sparse already. I don't think that we're wasting too much on the MOE side. People have been working with MOEs for quite some time. They're pretty good at squeezing MOEs. Where we are not good is attention and its use of memory specifically. The KV cache is quite uncompressed right now. I think if you look at the entropy in a KV cache, it's not earning its keep. Like we're storing many kilobytes of data in the KV cache per token. And that's probably off by an order of magnitude or two. I don't know the frontier labs do, but DeepSeek certainly publishes really interesting work to compress that further and further. And they're making good progress. And I think the fact that they're able to make an order of magnitude and progress here every year or so signals that there's a lot more room to go. If you zoom out further, I think that we actually don't marshal our compute effectively at all. We have all this compute in the world. NVIDIA is pumping out 5 million Blackwell chips this year. Where are they all going? Are they all being used at all the time? I certainly doubt it. I think that at some level, we just need better orchestration of compute across the world. This is very difficult to do because a lot of the compute disappears into private pools of compute that will never see the light of day. And those GPUs sit very sadly idle. It pains me physically to see that those GPUs are just silicon and power going into that and it just sitting idle And I want to fix that How we organize and orchestrate the world compute as a shared resource and pack it more efficiently I would estimate that we all make fun of XAI for having some challenges with total flop utilization on its clusters The reality for the rest of the world is it's far worse. A ton of GPUs just sit in warehouses or sit in private pools allocated to a specific customer, just don't get utilized. You're attacking the efficiency of that very directly. That's way more effective, yeah. What about FAPS? What do you think is the future of fabs themselves. I think everyone is wondering, will the memory companies, will TSMC, will Intel and others, how will they expand capacity basically? Will we do it here in the US? Riff on fabrication of chips themselves. Like if we could just snap our fingers and have a hundred times the chips in the stock today, we'd probably have way cheaper tokens. That seems like an important part of the universe to hear your view on. Everything grows in balance with each other. If we snap our fingers and double all those things, you might fix a TSMC bottleneck. You're just going to run into another bottleneck. You make 20% more chips than you have another bottleneck immediately. I will say, though, it is interesting what they consider to be a must deliver, like an invariant that their customers me are always going to want versus what I think is like a more fluid relationship I think that if a fab exposes more of their trade to me I able to make more intelligent decisions about what I can do One of the most interesting examples here is that any fab has a lot of spread in their worst chip that comes out of the production line and the best chip that comes out of the production line. There's a lot of variance in how chips are made. And the question is, if you have a company like TSMC, they work very, very hard to tighten what we call these process corners. We want to keep the worst chip as close in characterization to the best chip. And they've got a great lens to make that possible. But that means that they are adding a lot of controls in the process that maybe I don't need. Maybe I'm actually willing to find a place for that worst chip. You don't need to tighten the process control as much, which takes more time and cost. Maybe I'm willing to take a lot more rejects. And I think for us, it's like a more holistic optimization around cost of the dyes, supply of the dyes, and then the cost of power and places we can put them. My whole goal is to so dramatically expand the supply of power across the United States that I have a home for a lot of ships that otherwise would not have earned their place in a data center. Can we talk about how you design the system of your own business? What lessons have you learned? You talked about some interesting NVIDIA lessons Bring me into the culture and how you structure a team and a business where this is the North Star There a lot of in thinking We don worry about the immediate nature of when we start working on a model the efficiency is not going to be very good We think about where we could end up in a month, six months, or a year's time. We don't accept the state of the machines we work on as fixed. Even something like the Blackwell chip, if we think that there's some bottleneck that is holding us back from achieving this performance, it's very important to me that we understand and characterize that very well and write it down so we can both tell nvidia about it who are friends and also to basically keep this in mind for future chips that we buy we want to learn things that are invariant for us or the company long term and kind of fold that into future decisions that we make we're very collaborative i think one of the most important traits that we look for are people who either who are both good students and great teachers a lot of our people on the team were tas in college and love the experience of sharing knowledge in this way we do whiteboard sessions all the time. The collegial environment where everyone has something to teach and something to learn is extremely important for us. What are the attributes of people that you would want to hire that you think will be resilient to the work environment three years from now when more stuff is handled by machines? Curiosity, 100% curiosity. The one thing I cannot teach is love for performance.
[65:00] for digging into every microsecond the machine is working and understanding what's happening on the machine at that time. That to me is the most important trait for a performance engineer. It's what I look for. I don't look for lots of AI experience. I don't look for CUDA experience at all. That's actually a huge red herring. I mean, CUDA as a concept or GPS as a concept have evolved so much in the last five years. There's no point asking for 10 years of experience. I want to teach that, but I cannot teach the love for performance engineering. That is what I seek. Can you give your assessment of the major labs one by one, but also then the relationship of closed source as a category to open source, what you think is happening and will happen? In a line, I would say the labs pay an immense premium to be three to six months ahead of everything else. I think that's probably still worth it. I think it makes perfect sense for an anthropic to do what they do. There's a sensitive topic around distillation, which I think is a very core piece of the relationship between closed and open frontier. I'd like to offer an alternative view on that, which is there is the sense that distillation is theft, that you are taking something from the frontier models when you distill on their outputs. And in fact, even if that's not your intent, even if you don't ever try to scrape data from Anthropic, one thing I offer is that an increasingly large percentage of the artifacts we put out on the Internet are AI generated You just look at GitHub alone What percentage of repos created in the last year do we think were created by cloud code Do we consider that to be distillation Because it probably all we need I would not be surprised if you could train a fable class model only on the outputs of code you consider good on GitHub that's open source. And certainly if we take the position that users own the outputs of their interaction with AI, and they choose to put that up on GitHub, which a lot of them do, we're going to have latent distillation for a long time. It seems fundamentally impossible for me. I don't think it's fundamentally possible to prevent the diffusion of information or model capabilities. It will happen. The question is just how fast. And so then the question becomes, do scaling and improvement laws hold forever or for a really long period of time? And if they do, then there's value to being three or six months ahead and that will just last as long as it lasts and they can charge a huge premium for those tokens relative to a very cheap open source token. Is that the right way to think about it? I think it's possible. I don't know that the premium for being three to six months ahead is going to last that long. I mean, if you look at like enterprise deployments, they don't move at three to six months speed. A lot of enterprises are probably still on like four, six, Opus four, six or Opus four, seven. They don't adopt the bleeding edge rapidly. There a lot of questions that people have around rolling out any change at all We just so early in scratching the surface that I don think there any way to call a winner in this race Certainly I don even think this is a race that can be decided ever It a continual process Finally I don think open source ever goes away. If there's a vacuum because one leader steps out, a new leader will step in. There's too much incentive and too much tailwinds too. It gets easier every day to train a frontier class model. So your hope of what the future looks like is what balance between closed and open, What balance between model companies doing everything because they have the advantage of owning the stack or whatever? Anthropic can do that. It's like the new Google will just do that or something. What do you hope the future looks like? I want abundant tokens and diverse harnesses. I want everyone to build their own harness. Every company, every user, even. Make the agent your own. We're not that far away from that level of customization and capability. I want people to own their intelligence, and I want that intelligence to be customized. probably not through weight fine-tuning, but probably through more in-context learning. That's a more technical detail. But the underlying input to this abundance future is about cheap tokens. My job is to make the tokens as cheap as humanly possible. I will achieve that and I will do it through every layer in the stack available to me. I love the supply-side levers. I will use every chip I use every source of power and I will use every piece of land in the United States that suitable for this In return people will have the incentive to explore what it like to have abundant intelligence We still treat the agent as a person that is expensive to consult and you should ask them when you have a hard question That's not the way to think about intelligence. It's incredible that the machine can think, and we should try to get that into as many hands as as many people as possible. You sit in such a unique seat, and you have such a unique perspective on what you're trying to do to make this feature a reality. What do you think are your most divergent views of the world versus your friends who are really well-informed and interested in this stuff? What ideas of yours make your friends look at you like you have three heads? Most of the ideas on chips, I would say. You know, when I talk about building custom chips and they ask me, oh, so what's different? It's about sidestepping the HBM shortage and focusing on more extreme offload to other forms of memory, such as flash. I'm quite passionate about that idea. Everyone on my team knows that I keep banging the drum around like, What would we have to change about the model architecture to make offloading KVCache to Flash work at a much greater level? That's in the community of inference people. We have some divergent views on what you can do if you design a system around serving at 1 to 10 tokens per second, which is our whole North Star. More broadly, I think there is this larger sense around how do people consume a trillion tokens per day? That's the world we want to create, the capability for them to do that.
[70:00] What's a trillion tokens like ground us and how much that is? A trillion tokens. Well, OK, at OpenAI pricing, that's at least $5 million at the very least for 5.5 or 5.6. Yeah, I think the dollars is probably the most. Yeah, it's a good metric. Yeah. Yeah. It's millions of dollars. Yeah. So what's the world in which we consume what currently costs $5 million per person per day? We were asking for at least three to six orders of magnitude improvement in cost per token. Get that into 5,000, you probably have some customers. And in fact, I would argue that for some size of model, we are approaching a trillion tokens being measured in tens of thousands of dollars. And that's something that you could imagine running for a single job. Are you at all worried that just like the average person just can't and won't do that? Doesn't do that now with their own brain? There actually isn't that much demand for intelligence in the world? I never will believe in that. There is always demand for intelligence in the world. The on ramps to that intelligence are our challenge as a product community. I'm not a product person, so I cannot say I had the best vision. You want to enable those people. I want to enable those people. I want them to never be held back by the sense that my free tier users, I can't afford to give them this many tokens. And I hear that from my customers all the time. We want to fix that. What about the inverse question Not what you think is craziest but like what consensus thing you think is wrong One of the things I keep coming back to is this question of NVIDIA I am bullish on NVIDIA in the short term and NVIDIA you should never bet against them They always going to reinvent themselves Fundamentally I think one thing that surprises people is when I tell them that, hey, if you look at Hopper to Blackwell to Rubin, and you compare like for like, what is the performance per watt of BFloat 16 multiply? It hasn't improved all that much. Or you take that one step further and go to TSMC. If you look at TSMC, 5 nanometer versus 4 versus three versus two. The performance per watt on these chips doesn't change like a dramatic amount. The consequence of this is people lose their minds over geopolitics. Like what would happen if we lost access to DSMC for any reason? My contrarian take is that it wouldn't be that bad. Supply would take a shock for sure. But the best processes that we have in the West, like Intel, not that far behind, at worst, like maybe 2x, worst performance per watt. The gap is just far smaller than you would make it out to be if you follow like the chipboard dialogue. What else is happening in the AI world that is not in your path, meaning it's not like a component of this whole system that you would end up doing something in that interests you most? Well, we're fully downstream of models. So the model people get to decide how to design their architectures I don have any input to OpenAI or Anthropic but I can only pray that they go in the direction that is amenable to me or I have to like do my best to predict where I think they going to go and build my serving architecture accordingly both software and hardware choices They have I think the most interesting game in some ways to play Once again, this is going to be back to the profundity of the machine thinking and how consequential it is to decide to use something like sparse attention versus dense attention, or how consequential it is to use a different data type. We were training in BFLD16, but now we can train in FP8 or FP4, lower precision data types. that it's just an arbitrary choice, it feels like, but it has profound implications for what chips I can use and how I should build my hardware and think about the future of compute. If you had 100 entrepreneurs in a room, all of whom wanted to create some new compute startup, and let's say they were specifically wanting to make hardware chips or systems or racks or whatever, what advice would you give them on how to orient their companies or the type of company, not the specific choice they're making on a tech bet or something like this? Because it seems like we're going to try everything and that would be great for the world. Some stuff will work. But if you had to give them advice on how to orient their business to be successful in this coming world, what advice would you give them? It's all about the bottlenecks on supply chain. So you need to first convince me or convince an investor that you understand the like three to five bottlenecks that dictate modern ship supply. There TSMC wafer capacity there HPM capacity and there advanced packaging Maybe a fourth one would be power Where will you get the power How will you build these racks You should have a great answer to each of those four bottlenecks and how you going to work around them because it's all arbitrage at the end of the day. You're building a chip because you think that NVIDIA has made some choices that are difficult for them to change, which is true. NVIDIA makes a lot of choices that are difficult for them to change. They're not perfect. They're just really well balanced. You want to be spiky. You want to pick something and say, I think they've underpriced the impact of how short we're going to be on HBM. We're going to push really hard in this other direction instead, which I do think is probably the thing to attack most. Why? There's no easy way to bring on a lot more fabs of memory. So it's just going to be a while until we have... It's going to be a while. Yeah. The boys in Boise don't love huge capex for cyclical. They've been burned on that many times. Conceivably, because of that shortage, the world is just going to route around it by making everything else in the system more efficient? I think they're going to make everything else more expensive. I think that iPhones will cut their memory. iPhones are going to go up in price. We're just going to deal with it. Why doesn't NVIDIA go all the way to the end and sell tokens, do you think? NVIDIA is really smart about this. They don't compete with their customers. NVIDIA takes a long view on everything. Why don't they even start with the NeoCloud? Why don't they just sell, compete at the back door? Well, NVIDIA is really good. Jensen is really good at making his friends billionaires. He's made CoreWeave, a billion dollar company.
[75:00] company. And there's no need for him to destroy that goodwill. He wants to create a diverse community of neoclouds and inference providers who are all jockeying to create demand for NVIDIA, such that if any one of them decides to, I don't know, vertically integrate or go with AMD or any other option, he's got three more people hungry to fill that position. It's great to have competition amongst his buyers. My favorite closing question for everyone is, what is the kindest thing that anyone's ever done for you? My immediate first thought is like all my mentors that I've had over the years. It's a rare person who takes a lot of time out of their schedule and makes it like their personal interest, essentially, to make sure that you understand something or teach you something or ingrain some value in you that they think that you're on the cusp of understanding, but just push you over the line for understanding. A lot of the people in NVIDIA that I mentioned earlier, it was still that like love of performance engineering in me, but also my professors in college who I remember like my advisor in like sophomore year, I was very impatient student as I show up at his office hours and say, I want to build AI chips. I know what I want to do. Why am I wasting time taking all these like other basic classes and networking and operating systems? And he just laid out basically like the whole stack and showed me the beauty of understanding every piece in the puzzle. He took my entire path of like trying to focus on one piece of the system and said that it's so rare that someone can actually understand the entire stack from the gate level silicon all the way to building a great internet scale service. you should aspire to be someone who over the course of your lifetime achieves that level of understanding. It is such a rare trait. That level of expertise is so noble to chase. And that stays with me quite a bit. Amazing conversation. Thanks for your time. Thank you so much for having me If you enjoyed this episode visit Colossus You find every episode of this podcast complete with hand edited transcripts You can also subscribe to Colossus our quarterly print digital and private audio publication featuring in-depth profiles of the founders, investors and companies that we admire most. Learn more at Colossus.com slash subscribe. Thank you. source of truth for security and risk. Learn more at vanta.com slash invest. The best AI and software companies from open AI to cursor to perplexity use WorkOS to become enterprise ready overnight, not in months. Visit WorkOS.com to skip the unglamorous infrastructure work and focus on your product. Ridgeline is redefining asset management technology as a true partner, not just a software vendor. They've helped firms 5x in scale, enabling faster growth, smarter operations and a competitive edge. Visit ridgelineapps.com to see what they can unlock for your firm. Every investment firm is unique and generic AI doesn't understand your process. Rogo does. It's an AI platform built specifically for Wall Street, connected to your data, understanding your process and producing real outputs. Check them out at rogo.ai slash invest.
![Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488] — Colossus | Investing & Business Podcasts](https://reducer.xgoose.org/media/thumbnail/3341.jpg)