SemiAnalysis's Patel Sees AI Inference Rivals Global Commodities Market
Sequoia Capital
Co-designing models, software, and silicon together delivers up to a 100-fold increase in AI efficiency compared to hardware upgrades alone, driving inference costs down by 60x annually.
## Background
Published by Sequoia Capital on RSS / web on June 30, 2026, and hosted by Shaun Maguire and Sonya Huang, this episode features Dylan Patel, founder of SemiAnalysis. The discussion covers how hardware-software co-design drives a 100x improvement in AI performance, the shift from CUDA to model-level optimizations, and the economic scale of AI inference.
## TL;DR
Dylan Patel breaks down why the biggest gains in AI efficiency stem from co-designing models, software kernels, and silicon together rather than relying solely on faster hardware. He examines the hardware splits between Nvidia GPUs, Google TPUs, and specialized ASICs, explaining how model architectures like DeepSeek, OpenAI, and Anthropic are shaped around specific hardware constraints. Patel also explores the economics of living benchmarks like InferenceX, data center power bottlenecks, the rise of Neoclouds, and Jensen Huang's strategy to maintain a multipolar hardware ecosystem.
## Atmosphere & style
The conversation is fast-paced, highly technical, and packed with deep industry insider knowledge, blending quantitative rigor with an informal, candid tone. Dylan Patel speaks with sharp confidence and intellectual energy, drawing on his unique path from forum moderator and small-business operator to top semiconductor analyst. The dynamic between Patel and the Sequoia hosts feels like an energetic whiteboard session among peer experts, moving effortlessly between high-level economic forecasts, low-level chip topography, and entertaining historical anecdotes.
## Key takeaways
- `00:00` SemiAnalysis built its unique edge by combining deep semiconductor supply chain engineering expertise with rigorous hedge fund financial analysis.
- `10:00` The market for AI inference will eventually rival or exceed major global commodities, accounting for multiple percentage points of world GDP.
- `15:00` AI benchmarking requires living systems like InferenceX because constant software and model updates drive down cost for equivalent quality at roughly 60x annually.
- `18:15` Hardware, model architecture, and infrastructure decisions are strictly downstream of the trade-off between interactivity latency and system token throughput.
- `20:15` By 2030, OpenAI and Anthropic will consume over 100 gigawatts of power, and while space-based compute will remain under 1% initially, land constraints will push orbital compute to over 50% of incremental expansion by 2040.
- `23:45` The true 100x efficiency gains in AI come from software-hardware co-design—simultaneously tuning model architectures, kernels, and silicon together rather than in isolated silos.
- `28:15` Physical memory limits are the ultimate technological bottleneck, as basic DRAM and NAND flash cell architectures have remained fundamentally unchanged for decades while compute density scales rapidly.
- `30:00` Modern silicon power density is approaching 1,400W to 4,000W, forcing engineers to adopt pragmatic infrastructure workarounds like converted natural gas truck engines for on-site power.
- `33:00` Nvidia GPUs and Google TPUs represent a fundamental co-design split: OpenAI's sparser architectures lean toward GPUs, whereas Anthropic and Google co-design denser models around TPU topologies.
- `36:00` The so-called CUDA moat has largely unraveled due to AI-assisted coding, but Nvidia retains lock-in because open-source models are explicitly co-designed around Nvidia hardware dimensions.
- `47:00` Hyper-specialized custom ASICs risk getting trapped in local minima and becoming obsolete if fundamental model paradigms shift, maintaining a permanent market for general-purpose compute.
- `51:00` The global compute crunch persists because new frontier models expand the economic TAM of useful tasks faster than physical silicon supply can double or triple.
- `54:00` Not all data center capacity is equal: pricing, yields, and hardware utility vary drastically based on power smoothing, optical switching, and execution capabilities.
- `01:05:00` Traditional hyperscaler architectures were optimized for CPU-based isolation, creating an opportunity for hyper-leveraged neoclouds to move faster on massive GPU clusters.
- `01:05:00` Jensen Huang strategically allocates GPUs and balance sheet support to neoclouds and independent labs to cultivate a multipolar world and prevent hyperscaler monopsony.
## Detailed walkthrough
### `00:00` Internal Culture, Background, and the First "Neural Network"
SemiAnalysis maintains a uniquely dynamic internal culture split between heavy technical experts across the semiconductor supply chain and former hedge fund professionals. This structure frequently creates lively internal debates where financial analysts argue over costs while engineers push for the most exciting technologies. The firm operates with an informal atmosphere heavily influenced by founder Dylan Patel’s background as a former internet forum moderator.
Patel's early life was rooted in small family businesses, growing up in and living at a family-run motel, which later expanded to include a gas station across the street. Lacking the height to reach cigarettes kept above the counter, Patel learned to anticipate customer orders visually as they walked through the door. By observing demographics—such as age, race, and style—he would pre-position a step stool next to the correct brand (like Camels or menthols) before the customer even asked. Patel lightheartedly calls this practical pattern-recognition task the first "neural network" he ever trained.
His introduction to computer hardware occurred following his eighth birthday. After asking for an Xbox 360 for Christmas months in advance, the console suffered from the infamous "Red Ring of Death" hardware defect shortly before his cousin visited. Determined to fix it, Patel disassembled the console and bypassed the issue by shorting its temperature sensor. By age 12, he was deeply immersed in tech communities, eventually moderating major forums across platforms like Reddit covering Android, Apple, Google, Intel, NVIDIA, and AMD.
- Patel monitored the architectural evolution of smartphones as they rapidly surpassed PCs in technological complexity.
- His small-business background gave him an economic perspective, leading him to analyze financial metrics alongside technical specs.
- While internet users often favored AMD GPUs for price-to-performance, Patel argued NVIDIA was superior due to smaller chip sizes, better power efficiency, higher performance, and stronger profit margins.
---
`05:00` Early Career, Loss, and the Launch of SemiAnalysis
Beyond his early interest in hardware, Patel was a competitive StarCraft player, reaching the Grandmaster rank on the North American ladder. After completing university degrees outside the semiconductor field, he worked for two years as a quantitative risk analyst. During this time, he identified and exploited a market risk discrepancy that generated over $10 million in risk-free revenue for his firm. However, after a colleague claimed credit for the work and he was denied his bonus, Patel lost trust in the company's social contract.
A series of personal hardships followed in early 2020: his grandmother, who had lived with his family in the motel, developed dementia and passed away after a tragic fall, alongside other personal setbacks. When COVID-19 lockdowns hit, Patel moved in with his brother in Nashville. During the lockdown, he traded stocks heavily—profiting from short and long positions around the pandemic—and posted extensively online about the emerging semiconductor shortages.
After getting doxed by an online opponent who revealed his real identity, Patel initially paused posting for three weeks. Realizing he had no reason to hide, he shifted from anonymous forum posts to writing in-depth, long-form content under his real name. On his 24th birthday, he launched the SemiAnalysis blog with two comprehensive essays. The detailed nature of these posts rapidly drew industry attention and inbound consulting inquiries. Patel then packed his belongings into a truck with a bed-tent and air mattress, spending six months traveling across American national parks—living out of cheap $30-a-night motels during the week to write analysis, and listening to technical audiobooks and reading textbooks while hiking on weekends.
---
`10:00` Supply Chain Conferences and the Scale of AI Inference
From late 2020 through 2024, Patel maintained a mobile lifestyle, attending over 40 industry conferences globally every year to cover every node of the supply chain. He noted a sharp contrast between AI conferences like NeurIPS—which feature younger researchers and fast-moving software paradigms—and deeply technical hardware events. Because the semiconductor industry skew older, veteran experts were often eager to share detailed knowledge with a younger, enthusiastic analyst.
Specific specialized conferences, such as SPIE Advanced Lithography and SPIE Photomask, require years of immersion to fully comprehend due to their extreme technical depth. Patel built an extensive network across chemical suppliers, toolmakers, and fabrication plants, uncovering hidden supply chain bottlenecks and single-point-of-failure vulnerabilities that are never published publicly.
Looking toward the future of computing, Patel views the production and utilization of AI tokens as one of the largest future economic drivers globally. He projects that the market for AI inference will eventually rival or exceed major traditional global commodities, accounting for multiple percentage points of world GDP.
- Patel recalled an anecdote from a Japanese chemical conference where a veteran worker described a single factory fire in the 1980s that destroyed the world's sole supply of a key chemical, causing global memory prices to double or triple overnight.
- Deep technical events like IEEE and SPIE offer insights into niche supply chain dependencies, such as precise tool chemical consumption and sole-source material providers.
- SemiAnalysis developed **InferenceX** as an industry benchmark to evaluate real-world AI inference performance and cost metrics across diverse hardware platforms.
Here is a detailed, faithful walkthrough of this section of the interview with Dylan Patel.
---
`15:00` The Genesis of InferenceX and Living Benchmarks
Dylan Patel explains that SemiAnalysis initially tackled hardware testing by figuring out how to run benchmark suites and then publishing the results publicly to scale their research. While point-in-time benchmarks work for static systems, traditional testing falls short for AI because inference models, frameworks, and low-level software evolve almost daily. Software libraries like PyTorch, vLLM, and SGLang update roughly twice a week, and new models (such as Mythos 5 or Fable) release constantly. These rapid software optimizations and driver updates drive down model costs for equivalent quality at a rate of roughly **60x per year**. To capture this moving target, benchmarking must be a "living, breathing" system continuously executing on the latest silicon and models.
To build **InferenceX**, SemiAnalysis leveraged its industry reputation to rally broad ecosystem support. Cloud providers and hyperscalers—including CoreWeave, Crusoe, Nebius, Oracle, Microsoft, Amazon, Google, and OpenAI—donated compute, while open-source software teams and specialized startups (such as SGLang, vLLM, Radix Arc, and Infraact) collaborated on runtime integration. Hardware vendors like Nvidia, AMD, Google, and Amazon contributed access to their latest processors.
The benchmark suite runs automated, daily tests across over **$50 million in donated hardware** (projected to pass **$100 million** with the addition of TPUs and AWS Trainium chips), spanning approximately **15 distinct chip architectures**.
- **Daily Model Coverage:** Runs tests on top Chinese models (from Moonshot, Alibaba, and four other major Chinese labs) alongside leading US open-source models like GPT-LSS and Nemotron.
- **Config & Optimization Sweeps:** Evaluates across vast matrix configurations to avoid flawed comparisons (such as pitting one vendor's unoptimized setup against another's tuned configuration).
- **Open-Source Artifacts:** InferenceX publishes all raw results alongside reproducible container images for optimal operating points.
`18:15` Navigating the Throughput vs. Interactivity Pareto Curve
Patel highlights that nearly all decisions across hardware, infrastructure, model design, and application layers are downstream of the **throughput vs. interactivity (latency) curve**. Today's infrastructure often treats AI workloads with a "one-size-fits-all" approach, but optimal deployment depends heavily on where a workload falls along this curve:
1. **High Interactivity / Low Latency:** Required for real-time user experiences (e.g., Anthropic's Claude Code Fast Mode or OpenAI's priority queue). Uses low batch sizes and aggressive acceleration techniques like speculative decoding or multi-token prediction.
2. **High Throughput / Batch Processing:** Used for offline document processing or background jobs. Maximizes batch sizes to saturate hardware, trading individual user latency for maximum cost efficiency.
When host Shaun Maguire asks how cost factors into this, Patel breaks down a concrete numerical example:
- **Batch Size 100:** Yields 10 tokens per second per user, generating a total system throughput of **1,000 tokens/sec** on a single piece of compute.
- **Batch Size 1:** Yields **250 to 500 tokens/sec** for a single user, but total system output drops to 250 tokens/sec.
- **Cost Differential:** Choosing the batch-optimized setup delivers a **4x cost decrease** for the exact same unit of hardware. However, power users or time-sensitive feedback loops often justify paying 4x more per token to eliminate latency.
`20:15` Terawatts in Orbit: Forecasting Compute Scaling to 2040
When asked to forecast the share of inference compute running in space over a 10- to 20-year horizon, Patel outlines a stark contrast between short-term land constraints and long-term orbital infrastructure. While expressing strong admiration for SpaceX (joking that he would instantly buy their IPO stock, clarifying it is "not investment advice"), he argues that space-based data centers will remain negligible in the immediate future due to launch costs and terrestrial power availability.
```
TERRESTRIAL VS. SPACE COMPUTE EXPANSION
2030 Horizon 2040 Horizon
┌─────────────────────────────────┐ ┌─────────────────────────────────┐
│ Combined OpenAI + Anthropic: │ │ Global Scale: │
│ > 100 GW power demand │ │ Terawatts of inference compute │
│ │ │ │
│ Space Compute Share: │ │ Space Compute Share: │
│ < 1% total capacity │ │ > 50% of incremental additions │
└─────────────────────────────────┘ └─────────────────────────────────┘
```
Patel projects that power demands for inference will grow exponentially on Earth:
- **2030 Power Horizon:** OpenAI and Anthropic alone will consume over **100 gigawatts (GW)** of power combined, with Hyperscalers like Meta and Google adding substantially more. Space compute will account for **less than 1%** of total capacity.
- **2040 Power Horizon:** Total global inference capacity will scale into **terawatts (TW)**. At that point, land acquisition, terrestrial grid constraints, and power generation bottlenecks on Earth will flip the economics, causing **more than 50% of incremental compute additions** to be deployed in space.
`22:15` Intelligence Per Watt vs. Human Biology
The conversation shifts to measuring "intelligence per watt" and "intelligence per dollar." Patel acknowledges that specialized silicon—like a 30-year-old TI-84 calculator—delivers exponentially higher intelligence per watt for basic arithmetic than a human brain, though general intelligence remains a vastly different challenge.
Using power meters and cost tracking on InferenceX hardware, SemiAnalysis tracks both efficiency vectors:
- **Historical Efficiency Gains:** While model cost per quality unit dropped **60x annually**, intelligence per watt improved by roughly **40x per year** (as some gains come from non-power algorithmic optimizations).
- **The Human Brain Gap:** AI hardware remains **many orders of magnitude less efficient** than human biology.
- **The Practical Trade-off:** Patel notes that bridging this gap is not strictly necessary to scale AI. Unlike human brains—which suffer from sickness, disease, specific food preferences, and the need for sleep—silicon clusters can be directly continuously powered at gigawatt scales.
`23:45` The Myth of Isolated Layers: Software-Hardware Co-Design
Shaun Maguire poses a thesis: over the past three years, most performance improvements came from hardware leaps and model-level algorithmic changes, leaving little remaining headroom in low-level kernel optimizations. Dylan Patel strongly disagrees, arguing that treating hardware, software kernels, and model architectures as isolated silos misses where the primary gains occur.
Between Nvidia's Hopper and Blackwell generations, hardware improvements alone delivered roughly a 30x gain on optimized deployments (such as DeepSeek). However, the overall performance jump was vastly larger because the model architecture and underlying kernels were explicitly shaped around specific silicon quirks:
- **DeepSeek Co-Design:** DeepSeek V3 and V4 achieved massive efficiency breakthroughs because the shapes of their Mixture-of-Experts (MoE) layers, memory bandwidth routing, network I/O, collectives, and arithmetic intensity were tailored specifically for Hopper and Blackwell architectures (and Huawei silicon).
- **Hardware Mismatches:** Despite Google's TPUs being exceptional chips for pre-training Anthropic and running DeepMind workloads, they struggle significantly to run DeepSeek effectively because DeepSeek's expert shapes were built around Nvidia's architecture.
- **Lab-Specific Silos:** Anthropic trains heavily on TPUs but inferences largely on AWS Trainium and GPUs. OpenAI transitioned its model designs from Hopper-optimized shapes (e.g., GPT-4o, which features sparse expert dimensions similar to DeepSeek V3) to Blackwell-optimized configurations. Meanwhile, Google's Gemini 2 and Gemini 3 were co-designed specifically around TPU v6e and TPU v7.
- **The Co-Design Multiplier:** Combining independent 2x gains at the hardware, software, and model layers sequentially yields an 8x boost ($2 \times 2 \times 2$). Co-designing all three layers simultaneously yields up to a **100x overall performance jump**.
Patel highlights that this deep co-design philosophy mirrors top-tier hardware companies. Nvidia co-designs from model architecture all the way down to silicon, while TSMC co-designs raw manufacturing processes, tools, and consumables directly with customer chip designs.
`28:15` Memory Bottlenecks and Cell Architecture Limits
When asked to identify the most critical technological bottleneck facing the industry over the next year (excluding supply chain or raw scaling limits), Patel points directly to physical memory limitations.
- **Slow Capacity and Bandwidth Scaling:** Memory capacity and bandwidth improvements have lagged far behind compute density scaling.
- **Stagnant Fundamental Physics:** The core NAND flash cell was invented 25 years ago, and the standard DRAM cell was invented over 40 years ago. Despite process node shrinks, there has been no fundamental architectural redesign of the underlying storage and memory cell mechanisms to keep pace with modern AI bandwidth demands.
<FollowUp label="Want to dive into how Dylan Patel compares OpenAI's sparse models to Anthropic's dense ones?" query="How do OpenAI's sparse model architectures compare to Anthropic's dense models in terms of hardware requirements and inference costs?"/>
Here is a detailed walkthrough of this portion of the transcript, capturing the facts, arguments, nuances, and banter.
---
`30:00` Memory Bandwidth, Power Density Limits, and Power Solutions
Hardware innovation over the last five years has largely focused on making HBM (High Bandwidth Memory) stacks faster. However, upcoming architectures will shift toward stacking memory directly on top of the processor chip rather than side-by-side, causing memory bandwidth to explode. Beyond memory, data center and desktop chips have hit a two-decade-long power density wall: historically, power consumption can be reliably predicted at roughly $1\text{ W/mm}^2$. Modern silicon like Nvidia's current architectures and Google's TPUs are approaching 1,400 watts, with Nvidia's upcoming Rubin generation targeting 2,000 watts and Rubin Ultra projected around 4,000 watts.
To bypass the limits of expanding silicon area, emerging engineering designs are pushing power delivery beyond the $1\text{ W/mm}^2$ threshold. While driving more power into smaller silicon areas reduces overall silicon footprint, it presents significant thermal and electrical interference challenges. Meanwhile, on the data center infrastructure side, power constraints can be addressed through pragmatic workaround engineering:
* **Converted Truck Engines:** Millions of mass-produced US diesel truck engines can be converted to natural gas on assembly lines and back-driven through electrical motors to generate electricity on-site.
* **Labor Force Utilization:** Instead of waiting for specialized power grid buildouts, auto mechanics can be hired from local repair shops to maintain and service these engine fleets on data center grounds.
* **Historical Context:** Western focus shifted away from hardware innovation toward software and ad sales over the last two decades, leaving simple physical engineering solutions underutilized until recent compute demands forced new approaches.
---
`33:00` Nvidia GPUs vs. Google TPUs: The Hardware-Software Co-Design Split
The debate over Nvidia GPUs versus Google TPUs comes down to hardware-software co-design rather than simple superior specs. Estimates project Google producing over 10 million TPUs annually while Nvidia produces tens of millions of GPUs, creating multi-billion-dollar hardware pipelines for both ecosystems. While Nvidia relies on NVLink switches and general-purpose programmability, Google leverages custom Inter-Chip Connect (ICI) networking that connects up to 8,000 chips without traditional switches, albeit requiring data to pass through intermediate chips.
Rather than one platform being objectively superior, key architectural differences direct labs toward specific hardware:
* **Matrix Multiply Units & Attention:** Differences in matrix multiply unit shapes, attention mechanisms, and expert structures dictate which chips run specific models efficiently.
* **Model Divergence:** OpenAI's model designs lean significantly more sparse, making them less suitable for TPUs, whereas Anthropic and Google build models that are denser and tailored for TPU clusters.
---
`36:00` The "CUDA Moat" Shift and Model Architecture Locks
The classic narrative that Nvidia's software ecosystem and CUDA form an unassailable moat has shifted over recent months. With AI coding assistants (like Claude and Codex) aiding in low-level optimization and only a few dozen major frontier model labs existing globally, labs are willing to write custom kernels for multiple hardware targets.
> "And certainly the CUDA moat and software moat is at least partially disentangled because models are just great at coding and all software gets commoditized in that case."
The true lock-in is not CUDA itself, but rather how open-source models are tailored to specific hardware features:
* **Downstream Model Optimization:** Chinese frontier models (such as DeepSeek, Kimi, Zhipu AI, Alibaba, Tencent, and Xiaomi) along with Nvidia's Nemotron are explicitly co-designed and optimized for Nvidia GPU dimensions (such as hidden dimensions and $d_{\text{model}}$ parameters).
* **Ecosystem Downstream Effect:** Because open-source weights are tuned for Nvidia silicon, downstream application developers, inference providers, and RL tuning teams default to buying or renting Nvidia GPUs to avoid performance degradation.
* **Lab Independence:** Major labs (like OpenAI, which forked PyTorch early on) do not rely on off-the-shelf open-source software stacks; they co-design custom infrastructure directly against their target hardware.
---
`38:30` Cerebras, Fast-Mode Inference, and Memory Scaling Limits
Cerebras has carve-outs in high-speed, low-latency inference, particularly for fast-mode applications where rapid token generation is crucial. However, reliance on SRAM-based wafer-scale engines introduces significant architectural tradeoffs:
* **Context Window & Size Bottlenecks:** Ultra-large models (scaling into multi-trillion parameters) combined with long context lengths (e.g., 1 million tokens) struggle to fit onto SRAM-based architectures like Cerebras or Groq without HBM.
* **Economic Drivers:** Across commercial usage, the vast majority of lab revenue comes from frontier-tier models rather than smaller, cheaper variants. Even as higher-tier model costs increase, market spending follows raw model quality over token volume.
---
`40:30` Semiconductor ROI, Industry Misconceptions, and Long-Term Trends
SemiAnalysis combines heavy engineering expertise with financial analysis to track daily token spend and compute unit economics. Common industry misconceptions include claiming AI lacks return on investment (ROI) or asserting that model capabilities have hit a plateau. Saturation on older benchmarks reflects test ceiling effects rather than a halt in model reasoning capabilities.
Looking toward a 10-year horizon, long-term infrastructure scaling explores frontier concepts like space-based data centers and orbital silicon deployments. However, near-term supply chain shifts—such as timing adjustments in co-packaged optics or critical chemical inputs across the 1,000+ semiconductor manufacturing steps—continue to dictate commercial outcomes.
[045:00] Analog Compute and Naveen Rao’s Long-Term Silicon Bets
The conversation opens with a discussion about long-term bets in hardware innovation, specifically highlighting Naveen Rao's new venture. Dylan Patel notes that Rao understands this is not a quick two- or three-year effort, but a fundamental, long-term bet aiming to innovate across silicon, software abstraction, and model layers simultaneously—including concepts like analog compute paired with energy-based models. While Dylan quips that "it definitely won't work quickly," he respects Rao deeply, recounting how he first met him around 2019/2020 after baiting him on the internet while still posting anonymously, taking the exchange from public replies to DMs to a phone call.
```
Naveen Rao / Mosaic ML / Silicon Strategy
├── Analog Compute & Energy-Based Models (Long-term silicon/software bet)
└── Dylan's Early Connection (Baiting Naveen online in 2019/2020 as an anonymous poster)
```
Sonya Huang and Shaun Maguire reflect on Rao’s track record, noting his willingness to help younger generations and identify talent, alongside his forward-thinking work at MosaicML.
* **Naveen Rao's Approach:** Simultaneously redesigning the silicon layer, software abstraction layer, and model architecture (e.g., exploring analog compute and energy-based models).
* **Industry Trajectory:** Long-term architecture bets take many years to mature and are unlikely to deliver fast, short-term turnarounds.
---
[047:00] End State Ecosystem: Custom ASICs vs. General-Purpose NVIDIA Compute
Addressing the future of hardware, Shaun asks if every hyperscaler and AI lab will eventually rely entirely on custom chips like Trainium or TPUs for inference while using NVIDIA strictly for training. Dylan emphasizes that everyone will try to build custom silicon, driven by supply chain diversification. Currently, most chips look structurally identical—a central logic compute die surrounded by High Bandwidth Memory (HBM), with networking on top and PCIe/IO on the bottom (with niche exceptions like Groq and Cerebras doing "weird shit"). However, as hardware and model architectures diverge, co-design will push various players into different optimizations.
Dylan warns that specialized ASICs risk getting trapped in "local minima." If a fundamental breakthrough alters the core model architecture—such as replacing the attention mechanism—a hyper-specialized chip becomes obsolete. Because AI labs cannot predict their model architectures even one year out, general-purpose compute will maintain a massive, enduring market. NVIDIA benefits directly from this because its broad customer base continuously feeds diverse architectural requirements into its design pipeline.
* **Hyperscaler TPU Strategy:** Google runs three distinct TPU design programs: one co-designed with Broadcom, one with MediaTek, and a third unannounced architecture that differs radically from the first two.
* **Google's GPU Usage:** Despite owning TPUs, Google pays roughly $11 per hour per GPU to xAI for compute access, while certain internal non-Gemini teams (such as specialized science or Waymo projects) selectively utilize GPUs over TPUs.
* **Local Minima Risk:** Highly specialized ASICs optimize heavily for current model paradigms but risk obsolescence if architectural paradigms shift rapidly.
---
[051:00] The Compute Crunch, Economic TAM, and Model Improvement Loops
Dylan dismisses the idea that compute build-outs are slowing down, describing an unprecedented, ongoing compute crunch caused by demand outstripping supply. Even accounting for real-world delays in hardware deployment, global data center build-outs are scaling aggressively from 20 gigawatts this year to over 30 gigawatts next year. The crunch persists because new frontier models (like "Mythos 5" or "Fable 5") expand the Total Addressable Market (TAM) of economically valuable work far faster than physical compute can double or quadruple.
> "The model is so much better and it can do so many more tasks that the TAM for it is way larger... The world's compute did not double in that same timeframe, but the demand for useful tasks that can be done by AI has."
This economic expanding envelope allows labs to print massive profits on tokens even while paying premium rates for hardware. Anthropic, for instance, turned net income profitable (excluding stock-based compensation) in Q2, with expectations to be profitable including SBC by Q3. On API pricing for models like Opus, their token gross margins sit above 80%. Because margins are so high, labs can easily afford to rent GPUs at above-market rates from third parties (including renting capacity from SpaceX) while maintaining solid corporate margins.
* **Deployments:** 20 GW expected this year, rising to over 30 GW next year, despite structural supply chain and construction delays.
* **Anthropic Financials:** Achieved net income profitability (ex-SBC) in Q2 with >80% gross margins on primary API tokens (though blended corporate margins are slightly lower due to Amazon Bedrock and Google Vertex revenue-share deals).
* **Recursive Engineering Loops:** Models are increasingly writing infrastructure code and automating their own deployment, accelerating the release cadence of subsequent models.
* **Capital Raises:** To fund massive infrastructure outlays, labs and tech giants are raising immense capital; Larry Page's early $1B investment in SpaceX for a ~10% stake (later diluted to ~5%) stands out as an asset Google can selectively liquidate for capital.
---
[054:00] Data Center Heterogeneity and Hardware Yield Disparities
Sonya brings up concerns around high leverage in data center build-outs, citing news that Crusoe was asked by a customer to halt construction on a facility. Shaun counters with a private equity perspective—noting high leverage with high growth concentrates equity upside—before asking whether data center capacity varies widely in actual performance and yield.
Dylan agrees that not all gigawatts are created equal. Enterprise-grade execution, advanced optical switching, power smoothing, and infrastructure quality create drastic differences in pricing and performance across operators. Specialized providers and top hyperscalers derive significantly more utility per megawatt than lower-tier NeoClouds struggling with operational complexities.
* **GPU Rental Valuations:** Standard GPU clusters historically rented for $12 million to $13 million per gigawatt (or $12–$13 per watt). Premium short-term crunch deals, such as Google renting SpaceX's GPU capacity, spiked to nearly $25 million per gigawatt ($25 per megawatt-year).
* **Trainium Rental Pricing:** Amazon Trainium capacity rents below $10 million per gigawatt to core partners like Anthropic and OpenAI.
* **Contract Structures:** Anthropic's Trainium agreement included a custom floor mechanism: if performance fell short, rates adjusted lower with cancellation options, but as performance proved exceptionally strong, pricing scaled upward.
* **Infrastructure Disparities:** The divide between top-tier data center builders and inexperienced operators is widening, leading to customer cancellations and stalled construction for lower-efficiency facilities.
<ElicitationsGroup message="Explore related hardware and model strategy topics:">
<Elicitation label="Examine the software & hardware architecture of Amazon Trainium" query="Break down the hardware and software architecture of Amazon Trainium chips and how Anthropic optimizes for them."/>
<Elicitation label="Compare local vs. global minima in custom AI silicon designs" query="Explain how custom AI ASICs get trapped in local minima when model architectures shift, using technical examples."/>
</ElicitationsGroup>
Here is a detailed, faithful walkthrough of the provided transcript segment:
`01:00:00` Data Center Economics, Power Management, and Compute ROI
The discussion opens with an analysis of data center co-location dynamics, specifically focusing on buying raw power rather than pre-installed compute. Power pricing in co-location facilities traditionally hovered around $60 per kilowatt-month, but current transactions range wildly from $120 to $160 per kilowatt-month. Lower-tier data centers in places like India can cost as little as $80 due to grid unreliability and poor internet infrastructure, while higher-risk tiering pushes prices up to $200 per kilowatt-month for customers with weaker credit profiles. In the data center construction market, many projects fail or experience severe delays—often led by inexperienced teams who put money down on turbines without understanding the execution lag. To account for this, SemiAnalysis uses a probability-weighted data center model tracking hardware and equipment across every site globally.
The conversation pivots to how major tech players like Google optimize gigawatt-scale facilities. Instead of suffering from typical data center power utilization rates (where hardware draws only 60% to 70% of peak allocated power), Google fits over a gigawatt of actual hardware into a one-gigawatt envelope by dynamically shifting power workloads. Furthermore, hyperscalers negotiate specialized utility contracts: if a power grid can handle two gigawatts for 362 days a year but only one gigawatt for three peak days, operators will negotiate for two gigawatts with an agreement to shed load or turn off when prompted. Utilizing on-site gas generators, battery storage, and active workload management allows operators to sell significantly more effective gigawatts.
At the compute layer, value creation varies depending on who receives the hardware. A gigawatt of compute allocated to Anthropic yields higher revenue than the same power allocated to OpenAI, especially following the impact of release iterations like Claude 3.5 Sonnet. Similarly, if SpaceX received a gigawatt, they would deploy it with exceptional efficiency. SpaceX draws on immense networking expertise from Starlink and power management know-how from Tesla, backed by leaders like Brett Mayo. Additionally, while specialized providers like CoreWeave deliver superior GPU compute performance and reliability over Amazon, Google, or Microsoft, hyperscalers like Google hold a financial advantage: they can pre-sell compute capacity six months before deployment, taking signed customer contracts to debt markets to fund their initial purchase orders (POs).
* Data center co-location power pricing spans $80/kW-month in less reliable grids up to $200/kW-month for lower-credit customers.
* Standard data centers run at roughly 60% to 70% power utilization; advanced workload sloshing allows full capacity usage.
* Peak-shaving tactics utilize battery storage, natural gas, and site generators to sell higher effective capacity under utility constraint agreements.
---
`01:05:00` The Rise of Neoclouds and Nvidia’s Multipolar Strategy
The speakers address why Neoclouds emerged despite historical expectations that traditional hyperscalers would monopolize AI compute. Drawing on a 2023 report titled *Amazon Cloud Crisis*, Dylan explains that traditional cloud architecture was optimized for CPU-based infrastructure. Custom hypervisors like Amazon Nitro NICs provided excellent tenant isolation and cost-efficient CPU/storage scaling, but actively degraded AI performance. Furthermore, AI workloads do not require individual core or single-GPU slicing; customers lease entire 8-GPU servers or full 72-GPU racks under long-term contracts. Traditional hyperscaler custom networking, while superior for legacy enterprise traffic, struggled under heavy AI cluster demands, forcing companies like Microsoft to turn to Neocloud capacity when internal data center expansions fell behind schedule.
```
+-----------------------------------------------------------------------+
| HYPERSCALER vs. NEOCLOUD |
+-----------------------------------------------------------------------+
| Traditional Cloud (Hyperscalers) | AI Cloud / Neoclouds |
+-------------------------------------+---------------------------------+
| • Optimized for CPU workloads | • Tailored for massive GPU AI |
| • Individual core/socket slicing | • Full-rack / multi-rack leases|
| • Custom Nitro NICs (Isolation) | • Low-latency raw interconnect |
| • Slow, bureaucratic rollouts | • High-leverage, fast execution|
+-------------------------------------+---------------------------------+
```
Time-to-market and financial incentives heavily favor Neocloud execution. Unlike corporate employees at legacy hyperscalers, Neocloud leadership teams—such as Chase and the team at Crusoe—hold hyper-leveraged equity stakes, giving them intense motivation to build out power and hardware faster. Many of these builders originated in the volatile Bitcoin mining industry, bringing fast-paced infrastructure deployment experience to the space.
Overarching this dynamic is Nvidia CEO Jensen Huang’s deliberate strategy to cultivate a "multipolar world." Nvidia actively avoids an ecosystem dominated entirely by a few hyperscalers (Google, Amazon, Microsoft) or top-tier closed model labs (OpenAI, Anthropic). A market dominated exclusively by Google TPUs or Amazon Trainium chips directly threatens Nvidia's market power. To counter this, Jensen leverages capital allocation, balance sheet backing, and GPU allocations to foster Neoclouds (CoreWeave, Crusoe) and independent AI labs. Even as newly formed "NeoLabs" experience management churn, emerging teams like Thinking Machines (Tinker) have rapidly scaled to hundreds of millions of dollars in annualized recurring revenue (ARR) within months. Jensen's long-term strategy ensures a diverse, competitive buyer base for non-proprietary AI hardware. The segment concludes with mutual congratulations on the impact of SemiAnalysis and the dedication required to navigate the AI hardware landscape.
* Legacy hyperscaler features like tenant isolation and hypervisor offloading (e.g., AWS Nitro) created bottlenecks for distributed GPU cluster training.
* Crusoe and CoreWeave leveraged crypto-mining operational backgrounds to outpace traditional data center buildout timelines.
* Jensen Huang strategically backs alternative cloud providers and independent labs to prevent market consolidation around proprietary silicon like TPUs and Trainium.
* Thinking Machines achieved several hundred million dollars in ARR within six months of launch.
## Notable quotes
> `36:00` And certainly the CUDA moat and software moat is at least partially disentangled because models are just great at coding and all software gets commoditized in that case. — Dylan Patel
> `51:00` The model is so much better and it can do so many more tasks that the TAM for it is way larger... The world's compute did not double in that same timeframe, but the demand for useful tasks that can be done by AI has. — Dylan Patel
Dylan Patel of SemiAnalysis argues that achieving order-of-magnitude efficiency gains in AI requires holistic hardware-software co-design rather than siloed optimizations. He asserts that the AI hardware landscape is defined by severe physical bottlenecks, diverging architectural choices, and an ongoing compute crunch that reshapes data center economics.
Co-Design and the Multiplicative Gains Myth (23:45)
Patel disputes the notion that low-level optimization headroom has dried up, contending that hardware, kernels, and model shapes cannot be treated as isolated silos. Combining individual 2x improvements across silicon, software, and model layers sequentially produces an 8x gain, whereas co-designing them concurrently can drive up to a 100x improvement. For instance, DeepSeek V3 and V4 tailored their Mixture-of-Experts (MoE) shapes, memory bandwidth routing, and collectives specifically to Nvidia's Hopper and Blackwell architectures. Conversely, despite their strength in dense workloads, Google TPUs struggle to run DeepSeek efficiently because the model's dimensions are mismatched with TPU execution units.
This co-design dynamic also shifts the nature of Nvidia's moat. While coding assistants have partially commoditized software like CUDA, lock-in now persists because open-source models from major Chinese labs and Nvidia's Nemotron are hardcoded around Nvidia GPU dimensions. Downstream developers default to Nvidia hardware to avoid performance degradation.
Living Benchmarks and the Interactivity-Throughput Tradeoff (15:00)
Traditional static benchmarks fail in AI because rapid updates to libraries like PyTorch, vLLM, and SGLang drop inference costs for equivalent quality by roughly 60x annually, while intelligence per watt improves by about 40x per year. SemiAnalysis established InferenceX as an automated benchmarking system spanning 15 chip architectures and tens of millions of dollars in compute to sweep configurations continuously across leading US and Chinese models.
Patel emphasizes that all hardware and infrastructure decisions trace back to the throughput versus interactivity trade-off. Optimizing for high throughput using large batch sizes (such as batch size 100 yielding 1,000 tokens per second total) delivers a 4x cost reduction on the same hardware compared to batch size 1, which tops out at 250 to 500 tokens per second for real-time responsiveness. Power users often pay that 4x premium per token specifically to eliminate latency.
Physical Limits, Memory Walls, and Terrestrial Power (28:15)
Physical limitations in memory and power delivery represent the most immediate barriers to scaling. NAND flash and DRAM cell designs have seen no fundamental architectural overhauls in decades, causing bandwidth to lag far behind compute density. Silicon power density is also colliding with a historical 1 W/mm² limit as upcoming chips approach 2,000 to 4,000 watts, forcing architectures to move toward vertical 3D memory stacking.
To bypass terrestrial grid delays, operators are deploying unconventional physical engineering, such as converting diesel truck engines to run on natural gas on-site with local mechanics. Patel forecasts that OpenAI and Anthropic will jointly require over 100 gigawatts by 2030, with orbital space-based compute remaining under 1% of capacity due to launch costs. By 2040, however, as total compute scales into terawatts, he expects terrestrial constraints will push more than 50% of incremental capacity into space.
ASIC Risks, the Compute Crunch, and Neocloud Execution
Patel warns that hyper-specialized ASICs risk obsolescence in 'local minima' if core paradigms like the attention mechanism change, preserving a massive enduring market for adaptable general-purpose GPUs. Despite construction bottlenecks, global data center buildouts are projected to expand from 20 gigawatts to over 30 gigawatts next year. Demand continues to outstrip supply because frontier models expand the economic addressable market for tasks faster than physical compute can double, allowing labs like Anthropic to maintain high token gross margins above 80%.
This demand has fueled the rise of Neoclouds over legacy hyperscalers. Hyperscaler architectures optimized for CPU isolation (like AWS Nitro) degraded distributed GPU training, whereas specialized builders like CoreWeave and Crusoe scaled bare-metal clusters faster. Patel notes that Nvidia deliberately funds Neoclouds and independent labs to cultivate a multipolar ecosystem, preventing the market from consolidating around proprietary chips like Google TPUs or Amazon Trainium.
Patel concludes that hardware divergence will deepen as labs balance general-purpose silicon against custom ASICs, but navigating physical constraints in memory, power delivery, and site engineering will remain the defining operational challenge for the foreseeable future.
[36:00] Dylan Patel: And certainly the CUDA moat and software moat is at least partially disentangled because models are just great at coding and all software gets commoditized in that case.
[51:00] Dylan Patel: The model is so much better and it can do so many more tasks that the TAM for it is way larger... The world's compute did not double in that same timeframe, but the demand for useful tasks that can be done by AI has.
Original transcript
[0:00] I think it's really fun inside of Semi-Analysis because we have 90 people and like a big chunk of them are technologists, engineers across the whole supply chain. And then a big chunk is people who are formerly at hedge funds. And you see these arguments like people are like, oh, well, that doesn't matter. And it's like that someone's like, well, but cost. And then the engineer is like, no, no, no, but this technology is the coolest. And you see this organically like fight it out. And we're pretty informal. And, you know, given the fact that I was a forum moderator, you can imagine what this internal track looks like. Well, you're enjoying it. you don't wrestle with a pig because the pig enjoys it. We're here in the semi-analysis office with Dylan Patel. I'm Sean from Sequoia. I'm a partner, or Sonia Huang. It's pretty insane what you've done. Semis five years ago were not very sexy in the West. They were sexy in the East, but people here in the West had kind of forgotten about them. You did not forget about them though You went very long You created probably the premier research company in the space that been educating the world and you know the state of the art from very technical details to supply chain you know to the bigger picture There's rumors that Semi-Analysis recently passed $100 million of revenue. I don't know how accurate those are. Whatever the numbers are, you guys are crushing. It's as accurate as the information is, you know. You know, you never know. There's also rumors that you might start a venture fund. like you know i i hear all the time in the ecosystem people wanting you know affiliation with semi-analysis you've you've built this trusted brand and so whatever you do it's working it's clearly like just the beginning of the journey for you congratulations on that but how did this happen like how did you first question is like what is the background how did you kind of get to where you are now well when i was a young boy and you know coming out of the womb No, no. So, okay. So I grew up in like a small business. My parents had a motel. We lived in the motel. We laid our gas station. So, you know, I was selling, you know, I joke a lot of times the first neural network I trained was racially and visually profiling people based on when they entered the gas station, which cigarette to pick. Basically you know the cigarettes were all extrude across the top And I was too short to actually like you know reach them And technically it wasn legal to sell cigarettes at that age but whatever I had to move the stepstool over to the right area I started working my first office before it was legal too So it was a good experience. Well, I didn't get paid, right? It's a family business. But yeah, we have a motel and then across the street was our gas station. So, you know, sometimes, you know, someone would walk in. And so like if an old white lady with curly hair walked in, I'd move the ladder or the stepstool over to where the camels are. And if, you know, And, you know, different age, demographic, profession, you know, race, et cetera. I would move the steps to all over. And I joke this is the first neural network I trained because if I waited for them to tell me, I'd have to, like, move it over. And then I'd step up versus, like, just being ready. So, you know, menthols versus, you know, 100 slims and all these things. You know, I think I joke this is the first neural network I trained. But I grew up in family businesses, lived in a motel. And it all really goes back to when I was, like, you know, it was my eighth birthday. My birthday is in May. and it was April when the Xbox 360 was announced. For my birthday, I didn't ask for the Xbox or I didn't ask for a birthday gift. My parents asked what I wanted. I asked for it for Christmas. We celebrated Christmas, but there was no way, at least at the time, I thought there was no way they would ask, would give me the Xbox 360 for Christmas. And so I got it for, asked for my birthday for tab for Christmas Anyways Christmas comes around I get it You know fast forward a couple months my cousin who lives in Alabama they also lived in a motel was going to come over for spring break for his spring break and we were going to hang out at my house. And he's in between me and my older brother in age. Brother's a bit more jockey, so he didn't really care too much about the Xbox. He played sometimes, but didn't really care. But my cousin, I wanted him to think I was cool, right? So I bragged many times on the phone. I was like, yeah, I got an Xbox. And then the Xbox broke. There was something, there's a hardware defect called the red ring of death. But long story short, I had to open it up and, you know, short the temperature sensor and it fixed it. But there was many other tricks I tried first and then other than worked. And so that's sort of how I like got into hardware is like open Pandora's box. By the time I was 12, I was like on these forums a lot, reading, posting a lot. And this is around the time when Reddit ate all other forums. And so I became a moderator of, you know, Android and Apple and Google as well as like hardware and was watching, you know, looking at Intel, NVIDIA and AMD and all these other forms. Right. So I was built a PC, all these forms. I was watching, reading, posting a lot, but some of them I was moderating a lot. And so, you know, smartphones, watching smartphones develop from like very simple to speed racing to being technologically more advanced than PCs in many ways, architecturally. And same with like, you know, all the GPUs, like just tracking and watching that, reading every content.
[5:00] I'm always having the economic tinge because I grew up in a small business. So I was always looking at the economics, right? There was a time where all the, like, let's say neckbeards on the internet loved AMD GPUs. And, like, I personally had bought an AMD GPU, too, because price performance. But then when it came down to, like, what's technically better, I'd always be like, no, no, no. NVIDIA is better because they use a smaller chip to get, you know, better performance and better power efficiencies and their margins better. And so I would always talk about how NVIDIA's margins were better than AMD's in the GPU landscape. And so it was very fun. And you were 12 at the time? I started moderating when I was 12, but this is all through my tween age and high school years, right? Do you have any other weird hobbies or was it just semis? I played a ton of StarCraft at one point. I was a grandmaster on the North American ladder. Oh, wow. I'm serious. So you've been just obsessively good at multiple things. Yeah. I mean, obsession is good. How were your grades? They were decent. I would say like I had mostly A's, but they're classes that I like were thought were really boring or, you know, just didn't enjoy. Like Spanish I got like not the greatest grades you know but it was like I speak fluent Spanish by the way so it really dumb But like it just sort of But maybe that why you didn get a good grade No I didn learn Spanish until later to be fair But yeah, so it was sort of my grades were fine, right? Like, I mean, like, they were fine enough for Asian parents. I was better than most of school, but, you know, it wasn't like, you know, try hard maxing for like, you know, all A's. Okay, so you're very much a student of the internet, Ben. This is how you develop this expertise. At what point did you decide to start semi-analysis? And what's been the biggest surprise since starting the company? Yeah, so I went to school. I got a few degrees in stuff that wasn't related to semiconductors. Was a quant for two years at a small quant risk firm. And then basically, you know, there was a culmination of events that happened, right? One was that my, you know, sort of like I got screwed out of a bonus. I'd made my company many millions of revenue, of risk-free revenue, because I exploited like a risk thing in the market. I think well over 10 million. And then someone else took credit for my work and all this sort of stuff. But eventually I did get right-sized, but I lost the social contract with the company I was working with. Add some, my grandparents grew up in my house with us, or in the motel with us. They lived with us and so very close to them. And my grandmother got dementia and she forgot who I was And she fell down some stairs and had like a tragic accident and passed away So all of that happened in early 2020 Additionally there were some like you know girl things And so you know there a few things that happened that made me like kind of very sad And so all of those things sort of culminated. Then COVID happened. And my brother's like, dude, just come stay with me. He lived in Nashville. So I came and stayed with him in Nashville. We were like, oh, lockdowns will be a few weeks. You can stay with me while they happen. And then you can go back home and, you know, whatever. Famous last words, lockdowns lasted much longer. But living with my brother for a few months, it was sort of like, okay, I didn't know what I was doing. I was now at my brother's home. Everything was his rules, sort of like him and his fiancee at the time, now wife, were like there. And so I basically had to tiptoe around, but I didn't care about my job. And so I was like posting even more than normal. I'd always been posting a lot on the internet. I'd always been trading stocks a lot. But I made a lot of money shorting COVID and long in COVID and all this stuff. Semiconductor shortages happened around then too. And anyways, I was like very much obsessed with posting and things like that. And eventually, around that time, someone, I got into an argument with someone on the internet and they doxed me, right? They publicly revealed my identity for my anonymous account. And at the time, I was like, oh, no, I'm scared. I stopped posting for like three weeks. And I was like, what am I doing? Why do I care So then I just started posting under I had like blogs and stuff as well I made a real blog semi And on my 24th birthday I posted um you know two blogs And, and then from there, it just like, it was not a newsletter, but I got so much traction because now instead of posting on anonymous name as a real name, and I put a lot more effort into those two posts than I usually did instead of like shit posting on the internet, it was like real effort into the blog. Um, you can actually go back and read those if you want. they're not that great, but you know, they were, they were good for the time. They're the best stuff you can find on the internet about some of these. Um, and, and I just kept posting, posting, posting. I started getting a lot of consulting business, you know, 2020. I also sort of, I was again, crashing out, didn't know what I wanted to do. So I, uh, packed everything up or sort of, I, I took my truck. I bought a tent that fits on the back of the tent truck, um, bought an air mattress, whatever, and would like, and drove around all these national parks all around America. And so like two or three or four days of the week, I'd stay in a random motel where I negotiated the price to be like $30 a night for a room. And I was working on semi-analysis stuff. And then the weekends I'd read books and oftentimes read textbooks while in some random national park or hiking and listen to audio books about semiconductors, about AI, about all the things that I cared a lot about and got way more educated over these six months where I'm just like going to every national park. And then the whole time I was alone. I was alone the whole time. I was posting blogs.
[10:00] Everyone was like, Dylan, what the f*** are you doing? Pre-Star Link or the very early days of Star Link? Pre-Star Link, pre-Star Link. Yeah, so it was like very much like, what are you doing? I travel around LATAM again like for a year, initially with my friend and then with my ex, you know, for, you know, about a year. And then I go, then 22, 23, 24, end of 21, 22, 23, and 24. I'm completely, I'm still completely homeless since mid-2020, right? But I'm traveling around to every conference in the world. I go to 40 plus conferences a year, no matter where in the supply chain it is. I'm like, oh, that looks interesting. I guess I'll go to that. And I'm like, I went to one conference like, wow, this is amazing. You get to talk to the experts and they just like, they're going to talk to you because, and then you're so excited. And in the case of semiconductors, everyone's a boomer. So it's like, it's great to like, you know, they're like, they don't see young people who are like excited about it. So they're really happy to tell stuff. And so you just have to ask on this, was there like a part of the supply chain or one of these conferences that, that particularly changed your view of the semi-world or that you felt then or feel now is particularly underrated? I think the trade shows and conferences range really widely Obviously the ones I have the most fun at include NeurIPS Why is that Because it 20 AI researchers and they generally in my distribution of age range So it like a lot of fun but they also like leading AI researchers and it a lot of fun and you learn a lot. Um, there's also a lot of parties and then it ranges all the way to like, you know, there's random chemical conference in Japan where it's 300 Japanese dudes. It's like 20 guys from ASML, 20 guys from TSMC, 20 guys from Intel. And those are the only people who speak English. everyone else speaks only Japanese and you're like, I guess there's still pretty interesting and fun. I think, I think like one thing that I have a, like a skill set of is like, I'm able to bond with anyone regardless of their background and like who they are. I'm able to talk to them, find something interesting to talk about. Oftentimes it's the tech stuff, but you know, it's, it's, and so I think like the most interesting conferences are oftentimes like, you know, the really big ones because that's where the biggest stuff is happening. But I think the niches that are really, really exciting is like, you know, SPIE. so there's IEEE which is International Electrical Engineering something and there's SPIE which is another ecosystem SPIE conferences are super super deep in details every single one that I went to especially like SPIE Advanced Lithography or SPIE Photo Mask I went to them the first time I didn't even understand 90 what I heard and then I read read read read I made some contacts of course and the next time I went I understood like half of what I went to Third time I went I understood like 75 of what I went to Even now I went and I was like I still don understand everything that going on Whereas like you go to like NeurIPS, you know, a couple of times you can understand, okay, what's neuro symbolic reasoning? Okay, what's this? What's that? Like you can kind of get a mapping of what everything is pretty quickly. But some parts of the supply chain are so arcane and so deep and so technical. It takes a lot of times for you to even understand what's happening on everything, right? for every research paper. It doesn't necessarily mean you didn't, you know, you go to a conference for a few reasons, right? You understand the research, you understand, but like, it's all the research that's being published. But what you really care about is understanding how does that research intersection, intersect with technology. Also, how does that research differ from what's there today? And none of these research papers tell you what's happening today. But then you just ask people and you build contacts and you learn, and then you like learn about the supply chain and, oh, this company supplies this company, even though it's not publicly stated anywhere. Like, you know, you learn that the, the, this chemical is like cost about this much and a tool uses about this much. And you look, you hear the horror stories of like this chemical had a shortage and it totally threw off this part of the supply chain. And then it turns out there's only three companies in the world that make that chemical And it like my favorite one is I learned uh a Japanese guy at that specific Japanese uh conference that I went to to where almost no one spoke English In very broken English he told me about how his father worked in this industry in the 1980s that the only factory in the world that built this chemical burned down, and that caused memory prices to double or triple. And I was like, wow, not too different from today. Not at all. That's crazy. inference going to be the biggest market on earth, the biggest market beyond earth. Agree or disagree? I mean, obviously use of tokens is going to be the biggest market. And the value that's created from tokens is going to be the biggest market. But I think tokenomics, sort of the use of tokens, adoption of AI sort of is the most important thing that's happening. And inference, whether it's open models or closed models, will be like one of the biggest markets in the world. Much bigger than oil, I think much bigger than like, you know, many other parts, like inference of AI will be, you know, many percentage points of the GDP. Yeah. Right. What you've done with inference X, I think is, you know, industry standard, maybe say a word on why you started it, what it does. And, you know, what do people misunderstand about performance benchmarking on inference? Yeah. So, so to zoom back, right. Like semi-analysis, we do a lot of stuff. That's like, you know, a lot of it is like research for institutional clients and, and our subscription first products, but a lot of it is also like, Hey, you know, this would just be
[15:00] to figure out. Let's figure out how to figure it out and just post it publicly. And that gets, you know, more and more scale. And so we've done this with a lot of GPU benchmarking and testing and training performance and inference performance. But, you know, ultimately we saw like inference benchmarking was like point in time, you know, you test it and you take some time and you release it and it's like slow and arcane and outdated because models change all the time. Every, I feel like every week there's a new model, whether it's a Chinese model or, you know, today Mythos 5, Fable dropped. And new models are coming out all the time. On the software layer, PyTorch, VLM, SGLang, new drivers, new something drops. In fact, the update cycle for most of these libraries is twice a week. So you basically have the software updating all the time and therefore performance changing. New inference optimizations are coming out, and those get updated. And so I feel like it's a relentless breakthrough after breakthrough after breakthrough that keeps driving efficiency and cost down, which is why we've seen model costs drop for equivalent quality by 60x a year. It's incredible. But to stay on top of that, you can't have point-in-time benchmarking. You need to have benchmarks be living and breathing, i.e., constantly running on the latest hardware, on the latest models. And so we embarked on a project and we got a lot of buy from the ecosystem This was only possible because we had enough aura with some of the ecosystem where we were able to get CoreWeave and Crusoe and Nebius and Oracle and Microsoft and Amazon and Google and OpenAI to contribute to us compute And then we were able to work with SGLang and VLM and now Radix Arc and Infraact, which are the private companies who are sort of leading those efforts, the open source efforts, to collaborate with us. We were able to get NVIDIA and AMD and Google and Amazon now because we're adding TPUs and Tranium to collaborate. Now we've got all these people collaborating. We've got over $50 million of hardware donated to us. Once we launched TPUs and Tranium, it would actually be over $100 million of hardware, maybe about like 15 different chip types, all running these benchmarks. every single day on all the latest model, right? The best model for Moonshot, the best model from Alibaba, the best model from, there's about five different Chinese models, the best open source models, the best Chinese labs there. We run benchmarks on their models every day, and then also the best US open source models, GPT-LSS, Nemotron, etc. So we're running these benchmarks every day in an automated fashion, and they run on these servers that are dedicated to us for inference benchmarking, and we sweep across so many different configurations and optimization types. And then what it creates is and all the results are public and all the configurations are public So now we have the Pareto optimal curve because a lot of times when people are comparing inference performance they like taking a suboptimal curve or point for someone else and comparing it to their optimal one It's like, well, yeah, I can make I can stick. You know, if I drove a Porsche versus like some race car driver, obviously I'd drive it slower. The same thing with inference benchmarking. And so what we did is we created open source, basically containers for the optimal points across every point on the interactivity, i.e. how fast is it responding to me versus batch size, i.e. how many users am I simultaneously serving curve. And so now anyone who wants the optimal point can just go to inference X, download it, and run that as the optimal point. And they can check every day if they want, or they can even auto-download the most optimal point for that model. And their inference performance will be near peak. Is that curve the most important curve, in your opinion? The throughput interactivity curve? Yeah. I think most things in hardware, infrastructure, model, application layer, everything is downstream of that curve, right? Is it something that needs to be super, super fast, super low latency, and I don't really care about the cost, so I make batch size very low and I use techniques like speculative decoding or multi-token prediction heavily and there so many possible techniques there Or is it something where actually I batch processing a ton of documents and I don really care about all these things I don use these techniques that actually are worse on cost efficiency to help you with speed for an individual user because I just want to pack a bunch of users. I don't care if the document takes all night to process, right? And right now, the way we treat AI infrastructure is it's like one size fits all. But over time, we're going to get to the point where there's stuff where you have batch workloads or you need instant response. And there's the whole curve that's going to matter for users. And so we see this with Anthropic, right? Cloud code, fast mode costs way more than regular mode. And same with OpenAI's priority queue thing. Sorry, dumb question. How does cost factor into the start? So if I, let's say imaginary example, I have a batch size of 100 and I can do 10 tokens per second per user. So in total, I'm doing a thousand tokens per second off of that one piece of compute. That's one side of the curve, super slow, 10 tokens per second. Other side is I have 500 tokens per second, but I can only have one user. And so maybe 250 tokens per second, one user. And then there's points on the middle that are more freudal optimal, right? The average person actually wants like 50 or 100 tokens a second, and maybe the number of users I can batch together. So the curve is, okay, 1,000 tokens total per second or 250 tokens per second,
[20:00] per second, depending on how many users I batch and there's a curve in the middle. And so ultimately some workloads will actually want the 4x cost decrease because the same unit of hard work can do a thousand versus 250. And some users, I'll pay 4x more because I don't care about the price. I care about time because the person using the tokens is expensive or the feedback loop that I have here is expensive. If you had to guess, you choose the timeframe, 10 years or 15 years, what percent of inference compete do you think will happen in space? It can be 0%, 50%, 99%. This is a tough one. So I think the non-consensus, or at least against SpaceX thing, I love SpaceX, by the way, and I totally would buy the IPO if I could buy stocks. Non-investment advice. Thank you. Thank you. Not investment eyes. From Sequoia. I don't think that space data centers will really matter in the next, three to five years. With that said, I think in 20 years, I think the vast majority of compute will be going in space And so the real factor there is sort of what the cost It the timeframe It the cost of building power on terrestrial land and how much power you going to be able to do on terrestrial land And I think obviously my views of where inference you know you know how many gigawatts or terawatts are devoted to inferences It a crazy curve for me personally. What's your forecast? How many gigawatts are? Yeah, I think, I think by, you know, 2030, just open ion anthropic will have over a hundred gigawatts combined. And then you'll add, you know, meta and Google and, you know, so on and so on and so forth. It's a humongous amount of compute that will be dedicated to inference. And by 2040, it'll be terawatts, right? The curve of productivity that we're going to get, and so inference deployments is going to be huge. And so if you look at 2040, I think probably more than half of the incremental compute will be going in space. But if you look at 2030, I think it's sub 1%. Do you think intelligence per watt has been increasing? And then it seems like there's still a giant gap between where we are, intelligence per watt versus human biology. And so if we are do you think we are to close that gap? And if so, where is that game going to come from? Yeah, I think it often depends on what you're doing too, right? Like a TI-84 is way more intelligence per watt in terms of doing math than us. That's like 30 years old, right? So obviously this is like a dumb, dumb, you know, sort of- General intelligence there. Yeah, but general intelligence wise. So one of the things InferenceX does is we also measure the power and cost of all of these this hardware And so we offer not just you know throughput versus interactivity we offer cost versus interactivity We offer power versus interactivity And so as far as has intelligence per watt been increasing, I mentioned it's been a 60x cost decrease for same benchmark level. We've also seen the same on intelligence per watt. It's not been exactly 60x, it's been closer to like 40x. Some of the efficiencies are non-power ways. But there's been a humongous improvement in intelligence per watt on an annual basis, at least so far this year, last year, year before, year before. And I expect that to continue. As far as where we are from the human brain, we're many orders of magnitude away. Thankfully, it doesn't really matter. We can devote a lot of power to computers, much easier to power computers than human brains. We have sickness, disease, and food preferences. Sleep. Exactly. Let me just ask one more question on on this general theme, in my opinion, in terms of like, you know, intelligence per water, intelligence per dollar, like any of these metrics, I think there's kind of three levels of input. You can get hardware improvements where the hardware more efficient You can get low low systems optimizations like kernel improvements major multiplication libraries things like that Or you can get high-level, model-level algorithmic improvements at the highest level. To me, it seems like in the last three years, most of the gains have come from hardware level and some from the model level. do you agree with that? Do you think that's what it will look like in the future? Do you think there's a bunch of juice to squeeze in a kernel level? Yeah, Sean, I completely disagree with you, by the way. Great, that's why I'm asking the question. Okay, so I think one way is to look at it as these three different layers. And in that sense, from Hopper to Blackwell, which is all we've had over the last three years, roughly 30x improvement on DeepSeek, on the most optimized deployment, which is, you can see on inference X, there's about a 30X improvement. But over the last three years, we've had way more improvement intelligence per watt. A lot of that coming from the model layer, right? If you look back three years, it's GPT-4. Now it's like maybe like one of the smaller QN models that's like 27B parameters total and like 2 billion active is like way better. And so you've got this huge improvement.
[25:00] on model layer. You've got this pretty sizable improvement on hardware, but it's that co-design layer. And I think that's what's important, right? If you look at the architecture of any of these models, but DeepSeq is the most famous one, at least that's public and people have seen it. DeepSeq got huge efficiency gains from co-optimization or kernel level, optimizing memory. Yes, I think it's like kernels, of course, but it's actually you build the hardware architecture for the chip. So if you look at the shapes of all the experts in DeepSeq, V3, they were all optimized for Hopper. And if you look at for V4, they're optimized for Blackwell and Huawei's chip. And what's interesting is despite the fact that TPUs are objectively an amazing chip, you know, and they run all of DeepMind and they do all the training for Anthropic as well, on the pre-training side, at least, TPUs suck at running DeepSeq. But they are really, really great at running other kinds of models that don't run well on NVIDIA. There is some level of such deep optimization that has been done, whether it be shapes, uh, network IO, uh, patterns, you know, how you do the collectives, how you do, um, things around, you know, the, the arithmetic intensity of the attention mechanism, all these different things are co between the model and the and the and the hardware and the infrasoftware in between And it it hard to say you can disentangle the games Do you think that my understanding is that China has done this a lot better than the West the last few years? And the Deep Sea was one of the first models to really do this? I don't necessarily think so. I think it's more so that the West doesn't tell people what they do. OpenAI didn't tell people that GP40 was how sparse it was, what the shape size was, all these things. But GPT-4.0 is roughly the same size, slightly smaller than DeepSeq v3. And 4.0 came out a little bit earlier, if I recall correctly. So is your view that all three of these things have been happening simultaneously at roughly the same rate and the biggest gains are when you just co-optimize? Yeah, I would say there's been more gains on the model layer than on the software infrastructure layer and the hardware layer. But there's been innovations on every layer and really the biggest gain and the beauty of the best labs is when they co-optimize all three. You know, and that's what like, you know, when Anthropic is, you know, even though they use many different kinds of hardware, they don't really inference too much on TPUs. They mostly train on TPUs and they inference a lot on Tranium and GPUs and GPUs are more the jack of all trades but they optimized their hardware they optimized their model they optimized everything so they can do that Whereas OpenAI their prior models were optimized for Hopper more now they more optimized for Blackwell and you step forward through time these labs, and the same with Google, right? They've optimized, you know, Gemini 2 was really optimized for the TPU V6E, or Gemini 3 was, and then Gemini, you know, the next Gemini that's coming out is really optimized for TPU V7. And so sort of like a lot of these things are being co-optimized. And actually, when you pull that model and put it running on the old hardware, it's really not that great. And so I think a lot of this co-optimization is the most important thing. It's called software hardware co-design. And that's, what's like really exciting about like, you know, sort of what, what, you know, I think my day-to-day is like, you know, great. You get to look at one layer. There's all these innovations happening here. There's all these innovations happening on every layer, but the real breakthrough innovation is when you leapfrog a few layers, you co-optimize and co-design them. And now all of a sudden you've, you've taken what could have been a two X here, two X here, two X here. And instead of being multiplicative to eight X, it's actually a hundred X because you've optimized it across all three layers. And so that's, So it's really exciting about sort of like what you see at the labs, what you see at like a company like NVIDIA, who not co on the model layer per se but a little bit from the model layer all the way downstream to you know silicon or you look at a company like TSMC they co not just you know fabrication but all the way from the components and the consumables and the tools all the way upstream to what the designs their chips or the customers are telling them is this co across many layers of the abstraction stack There will always be bottlenecks somewhere in that optimization that they're like lagging behind and then need to get pulled forward. you know and band-aids to you know if you had to predict like what are at any level of the stack it can be literally anywhere what are some of the bottlenecks your most like you're kind of tracking most acutely the next year and not necessarily in the supply chain not in like scale but in terms of the actual um and it can it can be in the supply chain too but just like is it memory improvements? Is it that just scaling loss? Memory is an easy one that everyone's talked about, but I'm not going to talk about it from a supply chain angle. I'm talking about it from a technology angle. Memory capacity and bandwidth have been improving very slowly. The NAND cell was invented 25 years ago. The DRAM cell was invented 40 years ago. There's been no major breakthrough in cell like what a NAND cell is. Obviously, NAND is a very simple gate. Or a DRAM cell.
[30:00] There is stuff that could come down the pipeline that could be hugely innovative. But even over the last five years, all we've really done is make the HBM more stacks faster. But actually, there's new innovations coming in the next few years where instead of stacking the HBM separately from the chip, you stack the memory directly on the chip, and that makes your bandwidth explode. And so there's interesting companies in that space and interesting POCs that companies are trying to do there. I think memory bandwidth is one of the biggest. Another one is for the history of silicon, basically over the last two decades at least, how many watts a chip is can be easily predicted just by looking at it for a data center or desktop chip. It peaks up at one watt per millimeter squared. And so if a chip is 100 millimeters squared, generally the power consumption is around 100 or a little bit less. And if you look at the newest NVIDIA silicon, the newest TPU silicon, it's still on that range of one watt per millimeter squared. So chips are now getting to 1,400 watts. The next generation is 2,000 watts for NVIDIA with Rubin and such. And you move forward to Rubin Ultra, it's going to be like 4,000 watts or something like that. But really, there's increasing the amount of silicon. What exciting is we now finally doing things and it in development right now where you actually can pump the amount of power into the silicon to be way more more than one watt per millimeter squared And now that all of a sudden means you need less silicon. Obviously, it's running at higher power and it's less efficient in some cases, but you reduce the amount of silicon and you're able to like overcome certain problems. Thermal issues, there's interference of like electrical interference issues. There's all sorts of different issues that crop up and that's why it's a hard engineering problem. That's why we've stuck at about one. But what's exciting is the world is trying to change these things. I think interesting, like in a different part of the supply chain is sort of like, you know, people, people will talk about like energy is hard and, you know, we have energy bottlenecks and it's like, yeah, but there's actually like very simple solutions, you know, one could think of, right. Take the millions of diesel engines for trucks that the U.S. has the capacity to make. You can very trivially convert them to be using for gas in the assembly line and then stick them up to a electrical motor, like back driving it. So the electrical motor generates electricity rather than the electrical motor causing the rotation of the wheel, for example, but doing it the opposite direction. And now you've generated electricity by pumping gas into something that US can make millions of And then okay well that sounds like a pain in the ass to service right Because now you have to have hundreds of these on a data center site Well actually you can just pull people out of car mechanic shops and have them run around and repair truck engines Actually, it's actually pretty trivial. I don't want to say it's trivial. I couldn't do it. I think you're making a really good point, which is that because the West wasn't really thinking about semi-guns even harder more broadly the last 20, 30 years, we didn't have much innovation. We've got the best minds thinking about how to improve these things. Why would you want to go work in hardware when you can make ads, sell ads? Yeah, exactly. Okay, I'm dying to ask. NVIDIA versus TPU. What are your thoughts? I think everyone wants to pick one or the other for this, but it's really a function of like, look, you look two years from now, Google is going to make 10 plus million TPUs through their supply chain, and NVIDIA is going to make many more tens of millions of GPUs and both are going to be $100-plus billion. Well, Google's going to be $100-plus billion of TPU created a year, and NVIDIA will be $500-plus or whatever it is. I'm not making a specific estimate. It's not a revenue forecast. This is just a thought experiment. Yeah, our research has that. Sean, you've been media trained so well. Absolutely. Getting ready for the SpaceX IPO Are you guys big in SpaceX Yeah Okay so that makes sense We very lucky to be very large investors Awesome awesome So I would say the case of sort of like Google TPUs versus NVIDIA GPUs they both have like points that are really like in their favor, right? You know, NVIDIA will be like, oh, we have switches and we're general purpose. And TPUs will be like, well, we're more optimized. We're actually more energy efficient. and our network is actually more optimized for certain types of network architectures. And so you have these counterpoints that both would really get into. And I could, with a straight face, argue with you that GPUs are way better than TPUs or TPUs are way better than GPUs, but it comes down to hardware software co-design. So actually the way OpenAI's models are headed, it would be a terrible decision for them to use TPUs potentially. And the way that Anthropic and Google's models are headed, it's actually a terrible decision potentially for them to train with GPUs. I mean, it'd be fine for them to try. How wide is that? What's the fundamental difference there? There's various things, right? Like the size of the matrix multiply unit is different as a very simple thing. And therefore the shape of the matrix multiply you do, the attention mechanism you use, the way that attention mechanism is structured, the way the experts are structured. So you think, so OpenAI and Anthropic are converging to very different model architectures? I think they have quite different model architectures, in fact. Interesting.
[35:00] are much more sparse and that has benefits. And then Anthropics are, you know, they're still sparse, but more dense in general. And that has different benefits. And there's many other things, right? The network topology, right? NVIDIA, all of their chips are connected to switches, NVLink switches. For Google, they have no switch. But what they've done is they've been able to, you know, NVIDIA, the NVLink can only connect 72 GPUs. For Google, their ICI can connect 8,000 chips at super high bandwidth, but you have to pass through other chips to get there because there's no switch. And so there's like, there's trade-offs there, there's positives and negatives, and that influences the model architecture. It's not necessarily that you should, you know, claim one is better than the other, because at the end of the day, how do you say that this is better than that when you can't measure them in isolation, because it also extends up to the model layer, right? But I remember for a long time thinking, you know, one, the programmability of NVIDIA and then just CUDA as such a big moat. It seems to me that narrative has kind of changed, at least in my mind, for the last three or six months. Like model companies no longer care about, if we have to write custom kernels for this other chip, so be it, we'll work with four or five chips if we have to. Cloud and Codex are actually quite good at doing a lot of that optimization work And so it seems like some of the and then it you know it not like there 10 model companies that are each you know each need programmability There's on the order of tens maybe model companies. And so it seems to me that like if you, the fundamental premise of like tens of thousands of big customers that need CUDA compatibility, like it seems that kind of thesis is changing. Yeah. I mean, And certainly the CUDA mode and software mode is at least partially disentangled because models are just great at coding and all software gets commoditized in that case. I do think there is some level of open source. And what people call the CUDA mode is not actually anything to do with CUDA. But it's like the fact that DeepSeek, Kimi, and Zipu.ai, and Alibaba, and Tencent, all these companies. Xiaomi had an awesome model recently. their models are a co-design for GPUs. And therefore, if I want to run them on TPUs, actually, in some cases, they don't run really well on TPUs. Now, Google just has to create their own open source model ecosystem or open source models themselves. So they have the Gemma models. And so you end up with like, well, that's not really CUDA as a moat. It's that the downstream product is more optimized for NVIDIA. And in these cases, these companies are open sourcing them or like Nemotron is just open sourcing it And then the users of it for example to open the inference API providers the RL companies that are trying to take open models and customize them for companies business use cases All these different companies are downstream of the fact that like, okay, well, I guess I need to use NVIDIA. Because the ecosystem uses NVIDIA, even though I don't particularly care about writing CUDA kernels, because the models are great at that. But it's like the shape of like, well, this expert, the DMOD is this, and the hidden dimension, blah, blah, blah, is this, right? And so therefore, it's better to run on NVIDIA GPUs than it is on TPUs, and vice versa, right? If Google were to actually open source really good models, this would be the same thing, right? People would take their models, and they'd be like, oh, wow, these don't run that well on NVIDIA GPUs. I should actually just rent TPUs or buy TPUs and do it on there. For small teams, you're going to want to use all the open source software like VLMSG laying, PyTorch, all that stuff. But the big labs they don necessarily need to use all that right OpenAI forked PyTorch long ago and Anthropic and all these other people don necessarily rely heavily on the open source implementation of these things They forked things or built it on their own already And so they don need to rely on the open source And therefore now it more like I choose the best hardware and I co my model and infrastructure software through and through for that hardware that is the best and most cost efficient And I have AI help me write all that software What do you think of Cerebris I think Cerebris is a really innovative company I think in some spots of the market they're really, really good. Very fast inference. I think that's a big market. We use fast mode almost exclusively at Semi Analysis. But I love how disciplined you've been about accounting for, I don't know if that was one exhibit you did or if you do it consistently, but accounting for the dollar spent in the ROI on each task. It's an awesome analysis. Yeah, we do it pretty diligently. So thank you. That was the dark GDP article that we wrote. And also track everyone's token spend by day. And if someone's spiked up, I'm like, what did you do? It's like, okay, thank you for telling me that that seems worth it. Cool. On with my day. I think fast mode is obviously worth a lot for high-end tasks, right? I can just see so many different use cases where super fast tokens are worth it. I can also see the flip side where there's a lot of use cases where super fast tokens aren't needed and therefore the market won't pay for them and they'll use GPUs and TPUs instead. I think the big risk for Cerebrus is I mostly think the best models are the ones that you want to use fast mode on and small models you necessarily might not use fast mode on. I could see that being wrong with financial markets maybe or something like that, like a Jane Street HIF.
[40:00] frequency training or something like that, or medium frequency training. But ultimately, you know, running really large models at really long context is very difficult on SRAM-based chips like Cerebris, like Grok. And so now it all of a sudden is like, you know, what happens then if like the models get too big, right? If OpenAI's model is not, you know, on the order of, you know, hundreds of billions of parameters or, you know, low trillion parameters, but it's actually 10 plus trillion parameters. Now, all of a sudden, I don't think that that will fit on Cerebris, And then if that doesn't – with a long contacts length, if you have a million contacts length, now that makes it really difficult to justify. And so far, we've seen the bulk of revenue and usage at the labs be on their best model. Even when the model price has gone up, we've seen that. There's some data that shows that even though Fable just released today, they've had incredible amounts of people switch to Fable and Mythos, sort of that next-tier model, even though it's way more expensive. And so, um, that's volume by dollars totally, but without volume by tokens. Well, I guess who cares about volume by tokens is about the dollars, right? If I don't care that there's, you know, uh, you know, I don't know, 200,000 mini coopers or Toyota Camry sold if, if, uh, you know, I don't know, four, four, one fifties are five X ASP and they sell only half as much Okay Right And therefore the most lucrative market is pickup trucks in America right Mostly being facetious but like I think this is one of the things that you done so well and differentiates you from almost everyone else is that you care so much about the economics in addition to the technology. And I think very few people bridge those two things well. Yeah, thank you. I think it's really fun inside of Semi-Analysis because we have 90 people and like a big chunk of them are technologists engineers across the whole supply chain. And then a big chunk is people who are formerly at hedge funds. And you see these arguments, like people are like, oh, well, that doesn't matter. And it's like, then someone's like, well, but cost. And then someone, the engineer's like, no, no, no, but this technology is the coolest. And you see this, you see this organically like fight it out. And we're pretty informal. And, you know, given the fact that I was a forum moderator is, you can imagine what the, the, the, you're enjoying it. You don't wrestle with the pig because the pig enjoys it. Exactly. On this topic, before I go into the next question, are there like trigger topics in semis for you? You know, like if someone's, which is like such a meme, you think this person must be a moron. Like if, you know, if it's like, oh you like memory is the bottleneck I mean it true but like I think moreover the one that really gets me is people are like AI has no ROI It infuriates me, right? Like there's like, what's the ROI? Or like denying model progress, right? There's these people that are like, models aren't getting better. They're not reasoning. They can't think. They're going to dead end and plateau. And it's like, bro, the line has been up and to the right in terms of capabilities this entire time. And they're like, look, this benchmark didn't improve. That's because it said 90%. percent. Look at the new benchmarks. Yeah. You saturated. Now they're skyrocketing, right? It's like, I think that's more so the issue and challenge. Like I think semis are really complex and I don't fault people for, um, lacking like understanding of it. Like I learned stuff every day about the semiconductor supply chain from people. And I've been studying it for, you know, arguably 18 years since I started moderating the forms when I was 12, right? Like, you know, I've arguably been studying it for that long, but even then, it's like live, breathe, and that's all I care about. But there's so many layers of the abstraction stack. It's like, I learned about a new chemical that does like $100 billion of sales yesterday. And I'm like, whoa, didn't know this one existed and what process it did. But it's like, you learn about things all the time. It like okay billion sales in a couple hundred billion dollar industries whatever But like you know it like But it essential It essential And it like actually every chip requires it It like wow I guess there are a thousand process steps It's like, oh yeah, you like semiconductors name every process step. It's like, no, come on. What I think is the most funny is when people have all the facts in front of them and then they get the conclusion completely wrong. And that's what happens in our job all the time too. Yeah, yeah. I mean, I think my attitude is not to be mad that you do that. to do it as fast as possible. I think the industry, because it's so, it's just like AI is the most important thing in the world right now. And there's so many near-term bottlenecks. We talk a lot about the near-term. Are there longer-term things that you're really excited about? Like say on a 10-year time frame, we talked about orbital data centers, but like Silicon Metallinex, you think they're underrated or overrated on a 10-year time frame? Are there other things that on a 10-year time frame- Yeah, I mean, I think space is like super crazy awesome in the 10-year time frame that I'm, for space data centers and all these sort of mining asteroids and all these things, which is super excited about the vision of SpaceX, right? Again, not investment in advice before you hop in. I think on the semiconductor side, tremendous market movements and tremendous things can happen just when things happen one year later or sooner. And so that's all technology that in terms of co-packaged optics, like, well, everyone knows it's going to happen,
[45:00] the end of the decade. The debate is like 27, 28, 29, 2030. But some point along there, it's going to happen. I think the more interesting thing is like, there's companies like, did you guys invest in Naveen Rao's company? We did. Okay. Yeah. So I think like he's trying to innovate on like the silicon layer, on the software abstraction layer and the model layer simultaneously. And he fully understands that it's not like a, you know, we're going to do this in a few years. It's not a two-year timeframe. Yeah. It's not a few-year timeframe. It's a long-term bet. And like stuff like that is like, okay, we're going to bring like potentially like analog compute with energy-based models and like all this crazy shit all at once. It's like, that's exciting. Probably won't work, but you know, that's exciting. And I like really look forward to it. And it definitely won't work quickly. Yeah. It definitely won't work quickly is what I should say. I believe in Naveen and like, you know, I, I, I met him very, you know, I think he's one of the first people I met in the industry. Funnily enough, like in 2020 or 2021, actually 2020, it says something about him. I think he's someone in my experience, he's always trying to- I baited him on the internet. I baited him on the internet, that's the case. He's always trying to help the younger generation. He's trying to identify talent. He's also so ahead of his time with Mosaic. I remember getting pictures No he was 2019 I was still anonymous then actually I baited him on the internet and he started replying and then I just took it to DMs and then took it to a call And that was the first person who was really important that I talked to in the entire semiconductor industry. That's funny. But yeah, sorry to interrupt. That's funny. What do you think is the end state of the ecosystem? Do you think every lab, every hyperscaler just has its own chips? Like, Tranium seems like it's now working, right? So do you think we end up with every lab, every Hyperscale has own chips, at least for inference, and then maybe for training you go to NVIDIA or whoever? What do you think is the end state? I think everyone will try and they won't stop trying. I think ultimately, you know, supply chains matter. What technology you can bring in matters. And more and more as the industry gets bigger, supply chain diversification happens. You know, right now, everyone's chip more or less looks the same. It's a big logic compute die in the center. And there's some HBM on the right and left and on the top and bottom. Top side is networking. And then the bottom side is PCIe and other IO. And that is the exact same structure for Tranium, TPU, NVIDIA chips, and most of the startups. Not Grok and 3Risk. Those are doing weird shit, but that's cool. I think as you step forward we going to get more bifurcation of hardware architecture and model architecture And therefore people are going to co them And you know some of them will end up in local minimas right You know as we you know if it like gradient descent like people are trying to go to the most optimized solution Some people will race to a local minima, and then the question is, like, how do you scoot back over to, like, the absolute minima? And to some extent, like a general, NVIDIA will always be more general purpose than anyone else's chip in general, at least on a parallel AI compute basis because they have so many customers who care about different things. They will always give them feedback in the design. You know, the minima will always be better than them. But is that minima a local minima? Like is the TPU or Tranium or Grok or Cerebris or whoever's design optimized awesomely for here, but in the end state, actually, you got to go over here. And so they're the wrong. and maybe they make a great time. They're great for a little bit of time, but then they end up being wrong. It's like, that's the real question. And so I think there will be a big market for general purpose AI compute because you talk to people at labs, they don't even know what architecture they're going to be doing in a year. Like, right? Like they literally don't know what architecture they're going to be doing in a year. They have bets. They have many research bets and that's this exciting thing, but they don't know where it's going. Generally they like know what hardware they have and they trying to co But ultimately like if a new breakthrough happens on model architecture it like replace the tension mechanism with something else right Who knows Or you know all of a sudden you know something happens the best hardware will change And therefore, like, are people going to make five-year investments on hardware solely on, you know, an ASIC that is more specialized or are they going to have some bucket of more general purpose compute? And so you see this with like Google is paying $11 an hour per GPU to XAI for GPUs, right? Like that's insane, right? That's a very high amount of – obviously compute is limited and so on and so forth. But it's like very like insane, but at the same – despite the fact that they have TPUs. And so there's like some questions there like why do they do that? Google actually has three different design programs for TPUs. They're making a TPU with Broadcom. That's a different architecture than a TPU with MediaTek. That's a different TPU than the architecture that is – I won't disclose by research. But they're making different architectures. It's not just like, oh, they're making TPUs with a couple vendors and it's the same architecture. It's different architectures. And the third one is a very different architecture from the first two. And so I think people recognize that the local minima can happen. And therefore, I think everyone will have their own ASIC program. I think everyone will deploy billions of dollars of their own ASICs, tens of billions of dollars. In the
[50:00] dollars a year of their own A6. But ultimately, they're also going to have workloads that don't use TPUs, right? Some of the Google bets that are not Gemini DeepMind actually primarily use GPUs. They don't use TPUs. Some of them also primarily use TPUs, right? It's a bit of a broad thing, but maybe for drug discovery or for Waymo, you might not want to use TPUs. I don't say which one it is, but there's different architecture bets and different paths for AI. AI for science may have different algorithmic patterns than general intelligence AGI models. And so I think we'll see diversity continue to proliferate. Yeah. And because the market has gotten so big, niches will be carved out. And so that makes it possible for companies to have their niche and actually make money, even if the majority of the pie goes to NVIDIA and TPU and Tranium. Okay. Love that. Can we talk about the data center build out? Like, one, it seems like, I mean, by all accounts, if you look at the charts, like dollars per computer hour, we are in the middle of a crazy compute crunch. And it seems like it's both a demand and supply side crunch, right? Like demand for long-rise agents skyrocketing, supply, all these data center build-outs are delayed. Do you think this is going to compute crunch for the foreseeable future or do you think it alleviates at some point Yeah so every quarter we deploying vastly more compute than the prior quarter and there more data centers built than the prior quarter This year there going to be 20 gigawatts even accounting for the delays And next year there's going to be more than 30 gigawatts accounting for the delays. Of course delays happen on everything, right? Anything hardware can have a delay. That's just the reality of life. Are we going to have a compute crunch for the rest of our lives? It depends on what happens with models, but like the TAM for Mythos, you know, Mythos 5, Fable 5 is not just like 2x that of Opus, right? The model is so much better and it can do so many more tasks that the TAM for it is way larger than that. And yet computing the world did not double in the last, you know, six months, right? From, you know, Opus or maybe like seven or eight months since Opus 4.5 would launch to now. Huge, you know, 4.6, 4.7, 4.8 were improvements, but Fable and Mythos were like a huge step function improvement. The world's compute did not double in that or quadruple or whatever in that same timeframe. But the demand for useful tasks that can be done by AI, the number of useful tasks and the value of them that can be done by AI has. And so now the question is, what happens? Well, obviously, Anthropic in Q2 is profitable. They net income profitable excluding stock compensation And I think by Q3 they may even be profitable including stock based compensation That like how profitable they getting and their margins on an Opus token at least Opus 4.8 token is like north of 80% for the API price. They've got a lot of deals where their total corporate gross margins gets clawed down a little bit because of like how they do bedrock deals and vertex deals and things like that. But ultimately, their per token margin is so high, well, then if you don't have the capability, they have the capability to pay. Ultimately, every GPU they buy at above market rate, you know, they also bought GPUs at above market rate from SpaceX, which is below the rate of Google, but that's because they signed earlier. You know, it's something that, you know, other companies, maybe a venture-backed company or company that's not really got positive margins, can't necessarily do, right? What is the cost benefit ratios? Like every GPU I rent, because I'm out of compute capacity, I can immediately turn around and sell tokens on it, or every TPU or every Tranium, I can immediately sell tokens on it at a positive margin. And if I'm running 75% gross margin and I double the cost of the compute, it's fine. I'm still running 50% gross margin. And spinning up more compute nodes is not really necessarily a human requiring task for them if they're renting them. And so ultimately it's like, well my NOI still goes up right And so I going to rent GPUs at whatever price at some level whatever price I can pay I have almost a reverse question of like at some point does this compute build go bump in a night Earlier today, I think there was a tweet like Crusoe publicly said one of their customers had asked to halt construction on one of the data center build-outs. It seems like everybody in the ecosystem is so levered right now to like, we got to build, we got to build, we got to build. High leverage, high growth to me is like, makes me very, very nervous as an investor. Wait, hold on. high leverage, high growth means small amount of equity has huge upside. You're not a debt investor. You're an equity investor. You got to go to the school of private equity. Levered buyouts only. I actually come from a school of private equity. Oh, awesome. She forgot the school. She's been a VC for too long. No, I just do revenue multiples. Do you see any signs of that? Are you worried about that? I see what you mean, right? And that sort of goes back to the model point, right? Obviously, if the models expanding the total economic valuable work, that's sort of the dark GDP report that we did and that you mentioned earlier. If the work that these models can do does not expand faster than the compute capacity, then that tide turns, right? And over the last six months, that tide has been very much levered in the direction of
[55:00] You know, the models can do more work or are expanding their TAM of work they can do faster than the compute is increasing. And so prices go up. It's very possible that all of a sudden model progress stops. You talk to anyone at Anthropoc or OpenAI, maybe they're drinking the Kool-Aid, but you talk to basically all of them, they're like, no, no, no, no, model progress still go up. And so, you know, ultimately, you know, current methods could stall somewhere. I'm not sure where that would be. it seems like we have line of sight to model improvement, rapid model improvement. And in fact, models are improving faster than they were six months ago or a year ago because there's, I wouldn't call it recursive self-improvement, but basically the engineer, the models are helping write all the info and launch the next model sooner and sooner and sooner. So you've got this pseudo-recursive self-improvement loop going. And so the models are getting better and better and better, faster. But ultimately, capital is a big problem, which is why Google raised capital. They've got a godly amount of SpaceX, right? They own like 5% of the company. I think a little more, but yeah. I think at one point, they had like 10%. Larry Page invested a billion dollars at a $10 billion valuation, got 10% of the company. It got diluted, like all this. But that was one of the greatest investments of all time. Good job Larry They got a good missionary So they know they have like billion in the bank that they can sell in nine months or whatever from the lockup And they have all the gross profit they do And yet they still modeled that. And they're like, we need to raise capital. And so they did an offering. And it's like, that's insane. So that tells you how much they think they need to spend. But capital is like really, you know, Meta did announce that they're going to do a raise. Stock tanked. People don't like it. But, you know, that's all these companies are going to raise capital, whether it be debt or equity, at some point, money spigots will have to slow down. But right now, every GPU that Amazon adds, they're making higher revenue or every TP or Tranium, whoever anyone adds, is making gross profit. I'm going to do a little bit of a T up on this to turn it into a question for you. But as we talk about this, for me, the thing that's going through my head is almost an alternative hypothesis for like the Crusoe example. I'm using an analogy in oil. Like in oil, Saudi Arabia has way lower costs per barrel to produce oil than a lot of other countries. There's also like the purity of the oil. A lot of, you know, Saudi has generally like very low contaminants in their oil which makes the refining easier all of this The question for me is like when you look at for every gigawatt that being put in the ground I called the 20 gigawatts coming online today How much homogeneity do you see in those gigawatts Is it something like, and you can tell me whatever metric you think is right, but are Google's gigawatts two times more valuable than, say, most NeoClouds because they have optical switches and they've been doing it for a long time and they know how to do power smoothing because i think this could be the alternative hypothesis that some of the people that are it's like the people that are good at building data centers they should they should just do it to the max because there's so much demand and there's so much better than it but then maybe we're starting to see the early signs the people that are like not as good at it kind of getting hit a little so i don't know the reality i'm just curious yeah how do you think about that. So far, there are metrics for this, right? So Tranium sells at sub $10 billion per gigawatt rental rate to Anthropic and to OpenAI. GPUs, at least before the craziness of the last six months, usually went around $12 to $13 billion per gigawatt. So the rental rate, and this is from a NeoCloud versus Amazon even And now when Amazon sells GPUs they also be 13 or so And my understanding of that also is that those numbers like Amazon subsidized that a little bit So that it I actually think the numbers were even like I think the disparate is even more It's less than 10, it's less than 10, but there's like some weird, basically how much. And like, look, my understanding, like Anthropic played a big role in making Trainium useful in terms of, you know, writing all the libraries, etc. And so everything I hear is that Tranium is really freaking good hardware, and it's getting way better. And obviously, Anthropik is now using it a lot. So hopefully, we would see that price go up. The deal they did was actually there was a floor mechanism. And if it didn't do well, it would be cheaper to the point where it's cancelable. And if it did really well, the price is kind of higher. But effectively, less than $10 is where Tranium shakes out at. Whereas GPUs, I mean, the SpaceX deal again was like 25 or something crazy, billion dollars per gigawatt or $25 million per megawatt, right? A year rental rate with Google. I was like, that's a crazy divergence. Now, obviously, if Amazon was selling Tranium today, it'd probably be more expensive than 10 because of the compute shortages. But you do see this already in the sense of with data centers. Oftentimes, the rental price of a data center.
[60:00] doing co-location, right? Not compute in there, but just power. Here's the data center. You price it generally on a dollars per kilowatt per month. And so they used to be $60 per kilowatt hour per month. And now you see things transacting at anywhere from like 120 to 160. But different quality data centers, this actually, I've seen data centers go as high as 200 when the customer is not such a great credit rating. And then the data centers are pretty good one. And I've seen stuff go as low as a hundred still, or in like India go like as low as 80 because the grid's not reliable. The internet connection is not great. And it's a pretty mid data center, but at least it's a data center. And so you see this huge discrepancy there already. In the case of like data center construction, usually the pitfalls, they just fail. There's a lot of people who have failed, you'll claim they're like four guys. They're like, yeah, we heard I bought some turbines. I put the money down for them. I'm going to build a data center and then they get delayed, delayed, delayed and fail. Um, so you have to like probability, wait, time, wait, time lag, the teams that suck versus don't. Um, and, and sort of, you know, our data center model does that. We kind of track every data center, uh, and try and do this for every single one based on you know equipment that they using and all these things One of the things you mentioned about Google is you know in a gigawatt data center they actually put like 1 gigawatts of hardware And because they have such understanding all the way from workload to you know they able to slosh the power around And so instead of you know, constantly, you know, a gigawatt of compute, which typically runs at like 60 or 70% utilization in terms of power consumption, not utilization of the hardware, someone's always renting it. They're now running it at like, you know, you know, that 60 to 70% means it's at a gigawatt and you're using the full gigawatt. You see people doing deals with, including Google, with utilities where they're like, oh, well, I know this grid can sustainably take a gigawatt, but except for three days of the year, you can actually do two gigawatts. So give me two gigawatts and then just tell me to turn off. And so they'll do that. And so these sorts of tricks, and then you need to have supreme management of workload, backup power, all these things, generators on site to figure out how to actually keep it two gigawatts sustainably. When people do this, they're able to charge more. Whether it be I'm actually selling two gigawatts despite only having one gigawatt because those three days I'm able to deal with via battery, gas, etc. Or I figured out how to build power on site. Now I have a gigawatt where no one else does. And so I'm able to do it quickly. It's not necessarily transacting for a higher price. It's that I'm selling more gigawatts. And sometimes there are levers where you selling more gigawatts where each gigawatt is selling at a different price I think it more on the data center and energy layer It more about just having it versus not and then that being delayed or not It more binary But on the compute side, I do think there's a lot more interesting work there, right? A gigawatt given to Anthropic is objectively worth more revenue than a gigawatt given to OpenAI. And it seems that both of them could sell every gigawatt that they have right now, given rate limit problems and token max limit and all these sorts of things that open the ananthropic, especially since Codex 5.5 came out, it's much better. And then likewise, if you gave a gigawatt to SpaceX, they'd turn it on. My guess, my suspicion is that they probably make better use of the hardware than most people. I think people underestimate how much networking experience they have from Starlink in particular, and also how much power management experience they have from Tesla. Yeah, people like Brett Mayo are incredible. Pretty good. Yeah. And so for me, that's actually probably the thing that might – I don't actually know the answer, but I think that might be missing from the analysis a lot of people have. I think it also the fact that when CoreWeave builds a gigawatt even though their GPU compute is objectively better than Amazon or Google or Microsoft in terms of performance We tested the performance and reliability The problem is Google sells it six months before they have it up and they need to turn around and take that paper that they signed to get debt with that credit backing and then turn around so they can actually pay for the PO that they've already issued, you know, for the order that they've already issued. Whereas SpaceX was like, no, no, no, this is running now. Buy it, right? And it's a big discrepancy when you have a balance sheet to do that versus not. And that also helps your revenue per megawatt be much higher. Why does the NeoCloud opportunity even exist? Because if you had asked me five years ago, I would have said the hyperscalers are going to own this. And you mentioned just now CoreWeed has better performance than the hyperscalers. Why does this opportunity exist maybe at the macro level and then in the execution level? Yeah. So in 2023, I wrote a report that had Amazon really hate me. It was called Amazon Cloud Crisis. So I talked about how Amazon was the best because they had their Nitro NICs, which offered like tenant isolation. All the hypervisor ran on the NIC, and then you could sell all the cores. And they had custom SSDs that they made and they'd buy the raw NAND and they'd have lower cost because they'd buy the raw NAND and build their own SSDs. And they had their custom Graviton CPUs and that drove down cost per core. And so they had all these things
[65:00] That enabled them to sell more cores, have better security, good networking. But this was all for the traditional CPU, better storage, for the traditional cloud world. But in the AI cloud, a lot of this stuff hurt performance, right? These Nitro NICs were bad for performance, still are worse performance, although they've caught up a lot because they've had a couple iterations to improve them. But they're still worse for performance. A lot of the security stuff doesn't matter because it's not like I'm time-splicing users or splicing a socket into many users, right? It's like no one rents a single GPU in an 8GPU server. No one rents a single GPU in a 72GPU rack. They rent the whole rack. And in fact, they rent many of the racks. And then there's no like, oh, I rent for six hours and I give it back. It's everyone has these long-term contracts. So the mechanics of the GPU rental market meant that a lot of the expertise of the hyperscalers fell away. And a lot of the expertise that they did have were actually some of them were detrimental, right? Network performance. For Google and Amazon, they had custom networks that were better for traditional CPU and for the stuff that they were doing, but actually worked for AI. And then in other cases it like well Microsoft would save money by building their own data centers but their data center teams were not actually that great And so when it came time to run when it was predictable building it was like fine When it came to like actually double your forecast for the year it like they fell on their face and they had to go get a bunch of NeoCloud capacity I think so performance, I think, you know, I think time to market's another one, right? You know, these massive organizations, no one's getting rich from building this data center faster, right? But you look at Crusoe, for example, Chase and all the other people at the team, I was going to name some people on the team, but I'd rather not. All these people are getting rich if they fucking deliver this compute faster. They're hyper-levered equity owners. They're also all coming from Bitcoin, and you're not supposed to say that. I mean, a lot of the data centers, like their main data center guy came from Microsoft. I don't know. I'm just teasing, but it's like you learn a lot when you're in a very high fluctuation market. How much do you think was Jensen playing 4D chess? Jensen absolutely hates a world where all the hyperscalers have all the power. There's a reason he's blowing money on random AI labs that I don't even know if it makes sense to, but he's blowing money and pumping them up and going to everyone around the world and saying you should invest in this company because he wants to create a multipolar world That why he loves Chinese labs because he wants to create a multipolar world A world where open philanthropic and Google models are the only models is one in which he screwed Yep Right A world in which the hyperscalers are the only ones building compute is one he screwed in. And so, of course, he needs to point the allocation gun at neoclouds, help backstop their clusters, do anything and everything. Because while today, a GPU sold to Crusoe and a GPU sold to to CoreWeave and a GPU sold to Google and Amazon are all the same price for him. Five years from now, Crusoe and CoreWeave existing means Google TPU will be weaker and means Amazon Tranium will be weaker. And more inference being done with non-closed source model labs is better for him. So I think the NeoCloud ecosystem is these people that are Wild West, these NeoLabs as well, a lot of them have investments from NVIDIA. It's the Wild West. Some will fail. Many will fail. but you know some will emerge as really great teams whether it be you know oddly crusoe who's a bunch of crypto guys who then started building data centers and doing flare gas stuff or you know core weave who initially was a bunch of hedge fund guys they were also and then they were crypto guys but then they they like built you know there were a lot of people who didn't bubble up like them started around the same time just failed right and so i think you know i gotta say both those teams are phenomenal yeah they a lot of credit and it like that your Yeah I mean my point is like he you know you throw it like you throw a bunch of like bait into the water and the best fish will figure out and survive right And sort of the same way with the NeoClouds and he hopes the NeoLabs as well. We'll see if any of the NeoLabs really bubble up. But like Thinking Machines has a few hundred million dollars of ARR, right? That's pretty impressive even though they've had – in the media, it's like, oh, they've lost all this talent. It's like, well, but Tinker is doing a few hundred million dollars of ARR. Like that's pretty impressive for out of the gate, a product that's less than six months old or whatever. And, you know, we hope the same happens to other Neo labs. And so, you know, he wants a multipolar world. Truly, congratulations on the success. Thank you. Thank you. Just the last thing I'll say is I've seen a little bit of this. I think the public, they can probably tell from listening to you how hard you work. But, like, it's clear you've just been working your ass off for more than a decade. And it, you know, led to the last few years of being in the right place, right time. but it's unbelievable what you've accomplished. And I know it's just the beginning. So thank you so much for doing this. Awesome.
[70:00] Thank you.
