Nvidia Buys Groq for $20B as AI Compute Demands Scale Up

AI算力需求激增，Nvidia以200亿美元收购Groq 将确定性SRAM架构整合至现有GPU集群中，可在完全相同的功耗限制下实现2.5倍的Token产出，推动向自主AI智能体的转变。 本期交流探讨了AI软件领域根本性的经济模式转变，即分发成本受到密集的物理算力和Token生成限制的支配。Brad Gerstner与Sunny Madra深入剖析了确定性芯片架构的演化，详细阐述了Groq基于SRAM的设计如何与Nvidia的GPU基础设施相结合，从而在固定功耗限制下实现Token密度2.5倍的提升。他们还探讨了智能体工作流的爆发式增长如何推动前沿实验室总营收的大幅提升，破除了基础设施泡沫的论调，并详细论释了AGI的加速临近、劳动力从智商向情商的适应转变，以及Nvidia的市场主导地位。 AI 软件彻底颠覆了传统软件近乎零边际分发成本的经济模型，每次推理查询都依赖高密度的物理算力消耗，Token 生产正成为新时代衡量生产力增长的核心经济单元。 (0:09) 英伟达通过约 200 亿美元整合 Groq 的确定性 SRAM 架构，将推理流水线中的 Prefill 与 Decode 阶段解耦，在相同电力与物理空间约束下实现了 2.5 倍的 Token 吞吐提升。 (17:57) AI 推理单位成本在两年半内暴降多达 99%；随着技术从单纯的文本补全迈入自主 Agent 时代，单任务 Token 消耗量急剧增加，而为终端用户创造的实际经济价值也提升了 100 倍。 (22:00) 针对基础设施资本开支泡沫的质疑，企业级 Agent 落地推动 Anthropic 创下单月新增 100 亿美元年化营收的纪录，英伟达未来八个季度亦锁定 1 万亿美元销售额，正向 10 万亿美元市值目标迈进。 (28:00)

Altimeter Capital's Brad Gerstner and Groq Cloud's Sunny Madra argue that artificial intelligence upends traditional software economics by replacing zero-marginal-cost distribution with intensive physical compute and power constraints. They contend that soaring agentic token demand and deterministic hardware architectures are driving historic enterprise revenue acceleration, countering skepticism of an infrastructure bubble. Compute-Intensive Distribution and the AI Supercycle (0:09) Traditional software scaled on near-zero incremental distribution costs, but AI generation requires intensive physical compute for every query, with floating-point operations scaling roughly with model parameters multiplied by the square of context length. Gerstner frames this transition as an economic inflection point capable of dramatically compressing global GDP doubling times. Pointing to the tech sector compounding earnings per share at 15% over the past decade compared to 6% for non-tech companies, the speakers assert that compute capacity and token generation have become the foundational atomic units of future economic expansion. Token Density and Deterministic Inference Architectures (17:57) As model workloads shifted from simple single-shot answers to complex multi-step reasoning, data center constraints pivoted from raw logic toward memory bandwidth and power availability. Madra explains that Groq bypassed traditional GPU latency overhead by utilizing a deterministic dataflow architecture powered by ultra-fast onboard SRAM rather than standard high-bandwidth memory (HBM). Following Nvidia's reported $20 billion acquisition of Groq's platform, the two technologies were paired across NVLink interconnects to disaggregate the inference pipeline into prefill and decode stages. Madra and Gerstner report that combining deterministic SRAM streaming with Nvidia GPUs enables fixed-power installations—such as one-gigawatt data centers—to extract 2.5 times more token volume out of the identical physical and electrical footprint. Falling Unit Costs and Agentic Revenue Acceleration (22:00) Unit inference costs have fallen by approximately 90% year-over-year and up to 99% over a 30-month period, driven by packaging innovations like wafer-scale chips, quantization, and lower-precision formats. Gerstner argues that this cost deflation directly fuels an economic flywheel as models evolve from conversational chat into autonomous agentic harnesses that execute multi-step workflows, debug software, and triage communications in continuous background loops. Rejecting assertions that heavy infrastructure capital expenditures represent an unsustainable bubble, Gerstner points to reported early-2026 enterprise revenue numbers. Anthropic's annualized run rate expanded from $3.5 billion in January to $8 billion in February and $10.5 billion in March. The speakers contend that high-utility autonomous tasks have flipped earlier negative model margins into profitable unit economics, justifying multi-billion-dollar data center investments. Edge Bottlenecks, Sandboxing, and Hardware Dominance (45:36) Severe physical barriers continue to hinder local edge computing; Madra notes that executing an 8-billion parameter model on-device can fully drain a smartphone battery within 30 minutes, keeping advanced models anchored to cloud infrastructure. At the frontier frontier level, Gerstner underscores safety and security realities by citing Anthropic's unreleased Mythos model, which discovered 26 zero-day vulnerabilities in the Safari browser during sandboxed testing. Such discoveries prompted industry consortia like Project Glasswing to establish 100-day evaluation and AI-assisted patching windows before public rollouts. Addressing market competition from custom ASICs such as Google TPUs and Amazon Trainium, Gerstner argues that surging compute demand supports multiple winners. Citing $1 trillion in booked sales over eight quarters, Gerstner projects that Nvidia will navigate supply chain and memory constraints to eventually reach a $10 trillion market valuation. While the speakers present an aggressive case for continued exponential compute demand and fast enterprise payback, their outlook assumes hardware architectures can successfully circumvent physical power and memory bottlenecks while industry safety coalitions keep pace with increasingly autonomous frontier models.

Altimeter Capital 创始人 Brad Gerstner 与 Groq Cloud 总裁 Sunny Madra 指出，人工智能打破了传统软件边际分发成本为零的规律，将算力与每瓦 Token 生成密度推向全球生产力增长的核心。随着推理需求爆发与智能体行动能力的提升，硬件架构重组正在消化巨额资本支出，并推动企业端营收呈现指数级扩张。 边际分发成本重构与算力瓶颈 (0:09) 传统软件数十年来依赖接近零的增量分发成本，但人工智能彻底颠覆了这一经济范式：每一次用户查询和 Token 生成都必须消耗海量物理计算资源，算力浮点运算量大致与模型参数量乘以上下文长度的平方（Parameters × Context²）成正比。Brad Gerstner 指出，技术行业每股收益过去十年以 15% 复合增长，远超非技术类公司的 6%，技术占全球 GDP 份额已从 5% 升至约 13%。在 Demis Hassabis 等业界领袖看来，AI 带来的冲击可能达到工业革命的 10 倍，而推进速度快上 10 倍，Token 生产因而成为未来数万亿美元知识工作市场的核心原子单位。 面对算力需求激增，Jonathan Ross 创立的 Groq 放弃了传统 GPU 架构与非确定性调度硬件，采用确定性编译器驱动的数据流架构与板载超快 SRAM（表现类似大型 L1 缓存，带宽显著高于外部高带宽内存 HBM）。随着推理时间推理模型（如 OpenAI o1 系列）普及，Token 消耗呈抛物线增长，Groq Cloud 推出后数周内用户从数万激增至超过 400 万，凸显出基础设施层面对推理计算优化的极度渴求。 Groq与英伟达融合及密度重构 (17:57) 针对全球基础设施在电力和物理空间上的硬性约束，Sunny Madra 促成了 Groq 与英伟达的业务整合，英伟达在展示可行原型后约 30 天内完成了对 Groq 约 200 亿美元的收购。该方案将推理流水线解耦为 Prefill（预填充）和 Decode（解码）阶段：前端利用英伟达庞大的通用计算能力和高容量 HBM，后端则通过 NVLink 互联 Groq 高速 SRAM 芯片进行低延迟解码。在类似 1 吉瓦级电力约束的固定设施内，这种架构融合让系统在相同功耗与建筑外壳下实现了 2.5 倍的 Token 产出。 Madra 透露，英伟达内部已发展出涵盖 7 种芯片类型和 5 种机架配置的复杂系统，逐步演化为解耦计算、存储、网络与 CPU 的完整集群。在每周消耗数十万亿 Token 的推理时代，算力扩展的瓶颈已经从纯计算逻辑转移为可用电力与内存带宽。通过提升单位瓦特和单位空间的 Token 密度，前沿实验室如 OpenAI 和 Anthropic 能够在固定资本支出的物理限制下大幅改善推理运营的经济效益。 推理成本暴跌与智能体飞轮 (22:00) 单位推理成本在过去一年中下降了约 90%，在过去两到两年半内累计降幅高达 99%。硬件与编译器协同设计、先进封装（如 Cerebras 的晶圆级处理器）、低精度格式（NVFP4）以及量化技术，正在部分抵消模型规模迈向 1 万亿至 10 万亿参数带来的 FLOPs 暴增。Gerstner 分析称，在简单的聊天与自动补全阶段，实验室承担着严重的负毛利率；但随着系统迈入自主行动阶段，智能体循环能够自主编写程序、调试系统并执行复杂多步工作流，单项任务消耗的 Token 增加十倍，为终端用户创造的价值却增长了百倍，显著抬高了付费意愿。 这种效用飞跃正在打破所谓“AI 基础设施资本支出泡沫”的质疑。Gerstner 列举数据指出，Anthropic 的年化营收在 2026 年初迎来爆发，从 1 月的 35 亿美元攀升至 2 月的 80 亿美元，3 月更达到 105 亿美元，相当于在 30 天内净增了 100 亿美元的年化营收增量。软件自动化工具如企业内部个人助手在后台持续运行跨系统协作，配合单价降低和高价值交付，使此前处于亏损的单位经济效益转变为高盈利模式，为单座数十亿美元的算力工厂建设提供了商业支撑。 前沿模型演进、安全沙盒与边界 (35:30) 产业核心领袖普遍私下认同通用人工智能（AGI）的临界点正在快速逼近，OpenAI 与 Anthropic 预计未来一年内追加的计算资源将超过此前十年所有实验室的总和，其中 OpenAI 代号为“SPUD”（GPT-5.5）的模型即率先在英伟达 Blackwell 集群上训练。在急速推展算力的同时，安全风险与模型失控边界受到高度审视。Anthropic 内部红队在尚未公开的“Mythos”模型（基于谷歌 TPU v7 训练）测试中，观察到其具备自主突破沙盒环境的企图，并挖掘出 Safari 浏览器中的 26 个零日漏洞及 BSD 系统的陈年缺陷。为此，微软、亚马逊等企业组建了 Project Glasswing 联盟，建立 100 天沙盒加固窗口，利用 AI 代理在发布前主动修补漏洞。 在设备端侧，本地计算依然面临极大的物理局限。尽管苹果等厂商试图通过系统生态集成大模型来满足“足够好”的日常体验，但 Sunny Madra 指出，在 iPhone 本地运行一个相对精简的 80 亿参数模型，仅需 30 分钟即可耗尽全部电量。这意味着前沿智能在可预见的未来仍必须深度依赖云端基础设施分发，难以完全由终端硬件独立承载。 资本折旧、市场竞争与长期估值 (50:44) 针对到 2030 年单座算力中心可能攀升至 1000 亿美元的重资产担忧，Madra 阐明了算力硬件的自然折旧路径：最初用于高强度预训练的集群在后续世代交替时，会系统性转为承载推理任务，而系统整体算力效率相比训练 GPT-2 时期已提升 50 倍以上。面对亚马逊 Trainium、谷歌 TPU 等定制化 ASIC 的竞争挑战，Gerstner 认为市场规模庞大足以容纳多元方案，并透露英伟达未来 8 个季度已锁定 1 万亿美元的订单销售额，其核心制约来自内存与供应链短缺而非下游需求。 结合当前约 4.5 万亿美元市值与 70% 的年化增速，Gerstner 指出英伟达交易市盈率仅约 13 倍，明显低于广泛市场平均水平，并维持其将成为全球首家突破 10 万亿美元市值公司的预判。而在更广泛的社会层面，传统基于纯智商（IQ）的代码与计算技能正被机器平民化，人际沟通、说服与领导力等情商（EQ）资产将成为劳动者的核心溢价来源；面对技术重塑分配体制的挑战，亦需通过如“Invest America”法案等联邦层面的股权制度设计来对冲财富分配失衡。 本场研讨勾勒出算力作为新型生产力要素的微观架构与宏观经济逻辑。尽管单位成本暴跌与智能体能力跃迁推动了前沿实验室营收爆炸式增长，但底层芯片向物理极限逼近所引发的极端功耗、端侧电池制约、系统安全沙盒防线以及巨额基础设施折旧，依然是决定这一轮技术超级周期能否平稳延续的关键变量。

[2:41] Brad Gerstner: bionic with AI [5:07] Brad Gerstner: 10x the impact of the Industrial Revolution, but happening at 10x the speed [8:35] Sunny Madra: FLOPs per token ≈ Model Parameter Size × (Context Length)² [11:36] Sunny Madra: 2.5x more tokens [17:57] Jensen Huang: Interesting idea. Let's have a chat. [22:00] Brad Gerstner: Phase of Action [35:30] Jensen Huang: Is this 100x from what you did before? [35:30] Brad Gerstner: end of the exponential [41:10] Sunny Madra: an apple to fall on Newton's head [47:39] Brad Gerstner: fear-mongering [53:53] Brad Gerstner: the beautiful thing about capitalism [53:53] Brad Gerstner: law of large numbers

[0:09] The premise of the class today, we're going to talk about-- [0:13] everybody knows how software in the world, software produced [0:17] had near zero incremental cost of distribution. [0:20] That is not the case with AI. [0:22] More users on AI apps require a lot of compute, [0:26] and so it's not near zero. [0:29] That's the topic of our discussion. [0:32] We're going to do a presentation by the group. [0:35] We're going to do a fireside chat, [0:37] and then we'll open it up for questions. [0:40] So without further ado, I'm really [0:41] excited for today's guests. [0:43] Our first guest is Brad Gerstner. [0:46] Brad is the founder and CEO of Altimeter. [0:48] Brad started Altimeter in 2008 with a few million dollars [0:51] from friends and family. [0:54] Today, 18 years later, Altimeter manages over $15 billion [0:58] across public and private markets. [1:00] Brad, I've known you for a little bit, [1:02] and the single consistent thing I've known about you [1:04] is that the best investors have invested across super cycles, [1:09] across up markets, down markets, recession, crises, COVID, [1:14] and Brad has done all of that and more. [1:16] Brad started his career trained as a lawyer, helped [1:19] start General Catalyst back in the dotcom era, [1:22] started a couple of businesses after that. [1:24] Altimeter being the fourth. [1:26] And at Altimeter, early in the internet, early to Google, [1:31] early with mobile and Meta, and many others, early to cloud [1:35] and software, later investment in Snowflake, Confluent, Mongo, [1:39] GitLab and now with AI, one of the largest investors [1:43] in OpenAI and Anthropic, which I know you guys from last week [1:46] love, in NVIDIA. [1:48] And he was on the board of Cerebrus, [1:50] and led an investment in Grok, which we're [1:51] going to get into deep today. [1:54] And outside all of that, perhaps the most important movement [1:57] that Brad has started is Invest America. [2:00] Can I get a quick show of hands, how many of you [2:02] have heard about Invest America? [2:04] Wow, look at that. [2:06] Look at that. [2:07] Got a lot-- a lot of opportunity, it looks like. [2:09] That's right. [2:10] That's right. [2:10] In brief, Invest America is a federal legislation [2:14] creating an investment account at the time [2:16] of birth for every child born in America. [2:19] The biggest impact Invest America is going to have, [2:22] according to me, is independence away [2:24] from dependence from our state, and making [2:27] every child in America an owner of our economy. [2:31] Brad, I have the great honor of calling you [2:33] my mentor, coach, and partner. [2:35] Thank you so much for doing it. [2:36] Please join us. [2:37] It's great to be here. [2:38] Thanks for having me. [2:39] [APPLAUSE] [2:41] And a special thanks to Dr. Goel for green lighting the class. [2:47] I think it's a really important one. [2:48] In a lot of schools today, in particular colleges [2:52] on the East Coast as well, there's [2:54] this-- people don't really know what to do with AI. [2:57] And I say all the time, you got to make yourself bionic with AI. [3:01] You can't consume enough AI today because it doesn't-- it [3:06] used to be you go to this school, [3:08] you get a job at a place, like Grok or Altimeter, and today, [3:11] I don't really care where you went to school. [3:14] I want somebody who shows up and delivers [3:16] abnormal value, bionic value. [3:19] And the way you do that is going to be leveraging the latest [3:21] technology. [3:22] So I'm glad that you are enabling [3:24] the students to sit at the intersection [3:27] of such important topics. [3:29] I'm going to introduce Sunny, and then I'm [3:31] going to share a couple slides, and then I'll invite Sunny up. [3:34] But I was thinking, Sunny and I have been [3:36] great friends for a long time. [3:40] We're going to play poker tonight in the all-in poker game [3:43] here in Silicon Valley, so we're buddies inside and outside [3:46] of work. [3:46] But I was thinking about the introduction, [3:48] and then I asked ChatGPT and Claude [3:49] to give me an introduction. [3:51] And ChatGPT wasn't great, to be perfectly honest. [3:54] And Claude blew me away, so I figured [3:55] I'd just read to you what Claude had to say about Sunny. [4:01] So our next guest is a serial entrepreneur [4:03] who apparently can't stop getting acquired [4:05] by bigger and bigger companies. [4:07] And, honestly, the trajectory is incredible. [4:10] He co-founded Extreme Labs, a mobile development [4:13] shop acquired by Pivotal. [4:15] Then he co-founded Autonomic, a smart mobility platform acquired [4:19] by Ford, which he became the VP running [4:22] Ford Dax, their internal innovation lab. [4:25] Then he co-founded Definitive Intelligence [4:27] that was acquired by Grok, where he [4:29] helped-- where he became president and helped launch [4:31] Grok cloud. [4:32] And then NVIDIA bought the platform, of course, [4:34] recently for $20 billion, their largest acquisition ever. [4:39] So if you're keeping score at home, [4:41] Pivotal, Ford, Grok, NVIDIA, the man's career [4:44] is basically a SPAC that only goes up, according to Claude, [4:49] unlike Thomas, OK. [4:52] He has a computer engineering degree [4:54] from the University of Ottawa, which [4:56] proves that even Canadians can disrupt things when [4:59] they put their minds to it. [5:01] Please welcome Sunny Madra. [5:03] Thank you. [5:04] [APPLAUSE] [5:07] So, I like to share a couple slides [5:11] to set the context for the moment that we're living in. [5:16] And inference, the conversation we're going to have today [5:18] is really a subset of this important conversation. [5:22] But this is global GDP per capita over the course [5:26] of the last 2,000 years. [5:28] And if you look at that you realize that basically for 1,800 [5:33] years, nothing happened. [5:35] It was survival. [5:38] There was no excess productivity from a fixed [5:41] amount of labor and capital. [5:43] It was what we could use to survive, [5:44] and then all of this stuff starts happening in the 1800s [5:48] and 1900s. [5:49] The number of years it takes to double GDP-- [5:52] and I like to say GDP is what creates the excess [5:57] in life for enjoyment. [5:59] It's beyond survival, the surplus that we all have. [6:04] And so the number of years it takes to double GDP has [6:07] plummeted, and now we're doubling global GDP-- [6:10] or you might think of it as quality of life-- [6:12] every 25 years. [6:14] But you may say, well Brad, what does GDP have to do anything? [6:17] Well, it has to do with everything. [6:19] So what happens when you have higher rates of GDP? [6:23] You have lower rates of poverty, you [6:25] have higher rates of basic education, [6:26] you have higher rates of literacy. [6:28] You have more democracy, more freedom, [6:31] higher rates of vaccination, lower child mortality. [6:35] So it turns out that innovation in and of itself [6:39] is a societal good. [6:41] And it happens to be correlated and accelerating. [6:45] So technology as an investor has gone from 5% of global GDP [6:50] to about 13% of global GDP. [6:52] And if I asked you guys, 10 years from now, [6:54] are we going to be at below 13% of global GDP [6:58] or above 13% of global GDP? [7:00] I think you would all say that technology [7:03] is a percentage of global GDP is going to be a lot bigger number. [7:08] We're sitting in the heart of Silicon Valley, technology [7:11] outearns non-technology. [7:14] So the dotted blue line here-- [7:17] this is the NASDAQ-- [7:19] has compounded earnings per share [7:20] at 15% for the last 10 years, compared [7:24] to 6% for non-tech companies. [7:27] So why do technology companies tend [7:29] to be better investment than non-tech companies? [7:31] Because they compound their earnings per share faster. [7:34] Again, I think, will be accelerated by AI. [7:37] And, of course, AI is going to massively accelerate all of this [7:42] because when we look at all the knowledge work in the world, [7:44] the TAM for it is measured in the trillions. [7:49] Demis said it well, it'll be 10x the impact of the Industrial [7:54] Revolution, but happening at 10x the speed, [7:56] probably unfolding in a decade rather than a century. [8:00] So I think that is the context. [8:02] What we're doing here and the acceleration [8:05] that will come with AI should be something that's [8:08] better for all of society. [8:10] We're going to have to talk about the guardrails [8:12] and the societal change we're going [8:13] to have to make to be that. [8:15] But sitting at the very root of all of this is compute. [8:20] You guys all know that the atomic unit of AI, [8:25] or intelligence, is the token. [8:28] And there's nobody better able to talk about the production [8:33] of this atomic unit than Sunny. [8:35] And so, Sunny, I want to go back in the wayback [8:38] machine a little bit, tell us what Grok is, [8:44] and what your observations were in 2023 [8:49] and '24 about what was going to happen with Inference. [8:54] Yeah. [8:55] So a little bit of a quick background. [8:57] So Grok was founded by Jonathan Ross. [9:01] Jonathan Ross was the creator of the TPU at Google. [9:05] And Jonathan Ross's background is interesting. [9:08] He's a high school dropout, not because he couldn't complete it, [9:11] because he was probably too boring for him. [9:13] And went straight from being a high school dropout [9:16] and probably completed his GED or something, [9:18] and went straight into a PhD math program at NYU. [9:23] And then gets recruited into Google, [9:24] and like every great engineer over the last 20 years, [9:27] was made to work on ad optimization or ad testing, [9:31] which is terrible in some ways. [9:34] But what he did was he listened to a talk by Jeff Dean. [9:38] And Jeff Dean had come in and basically [9:40] said, hey, good news, bad news. [9:43] Good news, I think we've found an algorithm [9:45] to solve automatic speech recognition, which [9:47] could be useful in many places. [9:49] Bad news, there's not powerful enough compute [9:51] so we can never run it. [9:52] And Jonathan took it amongst himself [9:54] to come up with a design, and coming [9:57] from a completely different area design, [9:59] using an FPGA for the first version of what became the TPU. [10:03] And then, ultimately, Jonathan left Google [10:06] because he thought the rest of the world should have this. [10:09] It shouldn't just be embedded inside Google. [10:10] And so he left and started it. [10:12] And so, quickly, what Grok is, and it continues [10:16] to be inside NVIDIA as well, it's [10:18] a chip that's designed with a dataflow architecture. [10:22] And what makes it very significantly [10:23] different than any other computer architecture [10:26] is that it's fully deterministic. [10:28] So hand in hand with the architecture [10:30] is a compiler, and a compiler which [10:33] predetermines where all the calculations are [10:35] going to happen. [10:36] And that last bit is really important [10:38] because the underlying thing to any AI [10:41] problem and token generation is lots and lots of math. [10:45] And that's why we're seeing this compute explode. [10:47] And I implore everyone to go look at the following. [10:51] We talked about one of Brad's great investments, Snowflake. [10:54] Snowflake is database retrieval company. [10:57] And so you have to go, get a record, and bring it back. [10:59] And if you look at the number of cycles [11:01] it takes to do that on a compute cycles, you can really see-- [11:04] and it's not a really large amount. [11:06] But you look at the number of tokens [11:07] it takes to generate-- a number of compute cycles or flops [11:10] it takes to generate a single token, it's mind boggling. [11:13] And the best way to think about it is it's [11:15] usually the parameter size of the model times the context [11:19] length squared. [11:20] And so that's for each token. [11:22] And you're doing something and have lots and lots of tokens. [11:24] So we're in this era where we have this incredible technology, [11:27] but it's incredibly compute intensive, [11:29] several, several orders of magnitude larger [11:31] than any other computing paradigm we've had before. [11:36] At your own startup, you have a conversation with Jonathan [11:40] about merging. [11:41] I was an investor in Cerebras, which is also [11:44] building a fast inference chip. [11:46] Grok was building a fast inference chip. [11:49] These two companies had been in existence [11:51] for upwards of 10 years. [11:54] And the extraordinary thing is, like in year nine, [11:57] they're both fighting for survival. [12:00] They're not thriving. [12:02] Like they're building for a market that didn't really exist. [12:07] But you saw something. [12:10] Jensen came on my podcast, BG2, and he said, [12:14] everything just changed. [12:15] I said, what do you mean? [12:16] He said, inference time reasoning, [12:18] he said, we've gone from pre-training models [12:21] to inference time reasoning. [12:22] And inference is about to 1 billion x. [12:26] So not 10x, not 100x, not a million x, [12:28] it's going to billion x. [12:30] And our systems of compute are not designed for what's coming. [12:34] I remember you and I had a conversation, [12:36] and shortly thereafter, you helped broker [12:39] this vision for Jonathan that said, Jonathan, [12:42] I think I see your future more clearly than you do. [12:45] So tell us about that moment. [12:47] Yeah, I think, at that moment a couple of things are happening. [12:49] So, the market had been dominated by NVIDIA [12:54] because NVIDIA is what the researchers used [12:56] to create the models. [12:57] And so naturally in part of creating a model, [13:00] inference is the forward pass of training. [13:03] Right in the back prop, that's what's different. [13:05] And so you're always doing inference [13:06] when you're creating models, and so it's very natural to just run [13:09] it on the same hardware that you've created the model on. [13:12] And so one of the things that we saw with the broad architecture [13:16] was that we could complete inference much more efficiently. [13:20] And so if you look at our V1 chip, which [13:24] we put into the cloud-- and I'll get back to in a second-- that's [13:26] a silicon designed in 2018, silicon [13:30] from 2019, 1914 nanometer and super competitive [13:34] against hoppers. [13:35] Which is five generations newer in terms of silicon technology. [13:40] And so, really, what we saw was we thought [13:43] it would be very difficult for convincing people [13:46] to buy our hardware and use it, but if we built a cloud [13:48] and put it in the cloud, developers really [13:50] don't care if there's an API. [13:52] And developers, we've seen they're quite fungible there. [13:55] So our big insight was take these things, [13:58] start putting them in the cloud, run the data centers [14:00] and make them available via an API, [14:02] and make the best open source models available for everyone. [14:05] And even including OpenAI models, [14:07] like OpenAI Whisper, which was open source from the beginning, [14:10] and so we had put a lot of those models there, [14:12] and that's what really took off. [14:13] And we launched the cloud, and within a few weeks, [14:16] we went to a couple 100,000 users and today it's like [14:18] something at 4 million users. [14:20] And it took NVIDIA almost 17 years to get to 7 million users. [14:24] So, effectively, reasoning models come along. [14:28] Reasoning models have are much more voracious [14:32] in their token consumption. [14:33] This is even before we get to agents. [14:35] This is just deeper thinking than what one shot [14:40] pre-trained models were doing. [14:41] And so when you looked at token consumption curves, [14:45] they were just going parabolic. [14:47] And our hardware, our clouds were starting to break. [14:50] OpenAI only had a gigawatt of compute, Anthropic only [14:53] had a gigawatt of compute, so we had to figure out [14:56] how to make both more token efficient models, [14:59] but also more token efficient architectures. [15:02] So now, remember, Cerebrus and NVIDIA [15:05] were big time competitors. [15:08] NVIDIA and Grok were perceived as big time competitors. [15:13] So, Sunny, you sent me a text and said, I have an idea. [15:18] We at the time were major shareholders, [15:21] and still are, in NVIDIA, and good friends with Jensen. [15:26] And you had an idea. [15:27] And the reason I want to point this out [15:28] is just like how one person's idea-- [15:33] we see these big transactions, but sometimes we [15:35] don't unpack that it's just one decision on one day that [15:40] causes these things to occur. [15:42] So, what was your-- you had a vision [15:45] that was pretty orthogonal when thinking about NVIDIA. [15:49] Tell us about how that came to be. [15:51] Yeah. [15:51] And, Brad, you did lead that email, so that was awesome. [15:57] But basically when we were looking [15:58] at the problem of inference, even [16:00] as Grok, what became obvious to us [16:03] is if you started to dissect how inference works, [16:06] there's first a dissection which happens between, say, prefill [16:08] and decode. [16:09] And so many people were starting to do that where you basically [16:14] use a separate set of machines for prefill [16:16] and another set of machines for decode, [16:18] and you can basically get some efficiency-- or lots [16:20] of efficiency by doing that. [16:22] What we further did-- and this is a good lesson for everyone-- [16:25] we further looked at prefill and decode. [16:28] And within the decode we realized [16:29] that we could disaggregate the decode, [16:31] because within the decode there's [16:32] many different functions that are happening. [16:34] And some of those functions are compute intensive, [16:37] and some of those functions are memory bandwidth intensive. [16:40] And so one of the big differences with Grok over a GPU [16:43] is GPUs have lots and lots of compute and lots [16:46] of external memory, which is HBM for them, which is slower. [16:50] We don't have a lot of compute on Grok chips, [16:52] but we have a lot of SRAM. [16:54] And that SRAM is very high bandwidth, almost more [16:57] than an order of magnitude faster. [16:59] And so typically on a CPU you'd see that as your L1 cache, [17:02] but we have lots of that in our chips. [17:04] And so when we looked at the problem-- [17:06] and what the email to Jensen was about [17:08] was basically connecting to their chips via something [17:11] they call NVLink. [17:12] So NVIDIA chips speak to each other via protocol [17:14] called NVLink, and that allows you to basically not run [17:17] something on a single GPU. [17:18] You can run it on lots and lots of GPUs together. [17:21] I think today we have 72 you can do. [17:23] We're scaling up to 576. [17:25] Grok has a similar protocol, and we've [17:27] been running thousands of chips together. [17:28] In fact, we have many models that we were running on 4,000 [17:31] to 8,000 chips at a time. [17:33] So basically NVLink Fusion was a way for us [17:35] to allow our chips to speak to the NVIDIA chips [17:38] so we could take part of the problem, which we knew the Grok [17:40] chips were faster at and more performant at, and run it there. [17:44] And the net result of all that is if you take the same [17:46] footprint of power, you can get two and 1/2 times more tokens [17:50] out by basically combining those two systems together, [17:53] which in today's world of constrained compute is really [17:56] valuable. [17:57] So, Sunny sends me a text and he said, [18:00] I think we can partner with NVIDIA. [18:03] That in and of itself is a pretty big change [18:05] because if somebody your chief competitor, [18:09] the idea that you can partner with them [18:11] is a pretty big change. [18:12] He said, would you mind sending Jensen a text? [18:15] And I'm thinking to myself, man, I'm [18:18] going to spend some political capital with Jensen [18:20] so, like, I need to know that this isn't a crazy idea. [18:23] And so I kind of sit on it for a week or something, [18:25] and then Sunny texts me again. [18:27] He's like, have you sent Jensen that text yet? [18:30] And so I said, OK, I'm going to send it to him. [18:32] And Jensen immediately got back to us [18:33] and said, interesting idea. [18:36] Let's have a chat. [18:38] And you guys started working with him and him, [18:40] and what was really compelling, I [18:42] think, to Jensen was you had obviously somebody [18:47] who built a competitive chip, but they had mentally [18:50] thought about how can we produce a lot more tokens together. [18:53] So what Sunny just said is really important. [18:57] OpenAI has got a fixed footprint of let's call it a gigawatt [19:01] that they're going to take in September of Vera Rubens. [19:05] In one end of the factory goes power and chips. [19:08] You, obviously, have the building and all the costs. [19:10] And out the other end comes tokens. [19:12] When they bought Grok for the exact same power footprint, [19:17] for the exact same building, they're now generating two [19:21] and 1/2 times the number of tokens. [19:23] And the constraint we have in the world is power and memory. [19:27] So if you can double or triple the amount of tokens [19:29] for the exact same footprint, it leads [19:32] to an enormous economic outcome for OpenAI or for Anthropic. [19:37] And so you've seen as the demand on inference, [19:41] because of inference time reasoning-- [19:42] and we'll talk next about agents-- [19:45] as the demand for these tokens of intelligence have exploded, [19:49] and literally we're consuming tens of trillions of tokens now [19:53] per week around the world, we've had [19:57] to come up with more power, more chips, more of these inputs [20:01] in order to produce those. [20:03] And so it's not just about fast chips. [20:04] It's also just about-- or fast inference, [20:07] it's just about the ability to get more tokens into the world [20:10] in a world that is constrained. [20:11] Yeah. [20:12] How many days-- from the time you [20:14] showed Jensen a working system, how many days from that [20:19] until him greasing you with $20 billion? [20:25] Probably just over a month. [20:27] Yeah. [20:27] Yeah, 30 days. [20:29] Yeah. [20:29] And Jensen is like-- did they have any competitive efforts [20:33] going on at NVIDIA? [20:34] Yeah. [20:35] I mean, I think NVIDIA-- [20:38] and you see it-- and we talked about it at GTC, [20:41] NVIDIA has an ecosystem already of seven chips and five [20:43] different racks. [20:44] So NVIDIA is no longer making a GPU. [20:48] And I think that's what is one of NVIDIA superpowers [20:50] that they've started to look at-- disaggregating [20:52] the problem in all different ways, [20:54] whether it's storage, whether it's CPUs, whether it's [20:56] compute or networking chips. [20:59] And so that already exists. [21:00] So they had already thought about building [21:02] a decode-only chip, and something that [21:04] was powered by a lot of SRAM. [21:06] But I think, sort of, it's a good lesson for everyone, [21:10] like us putting that email in, starting to work together [21:13] and building a prototype that was [21:15] working with their systems was a real proof of concept for them. [21:18] In these large systems, making these things work together, [21:22] making them performant-- and this is across two different [21:25] companies with two completely different stacks-- [21:27] I think when they saw that we were [21:28] able to do that, it showed that we'd be a good integration. [21:31] And the last thing I'll say is we're two very different types [21:34] of companies. [21:35] I think if we were kind of making a better GPU, [21:37] there'd be a lot of conflict within NVIDIA [21:40] after the type of deal that we did. [21:42] But because we were making this SRAM [21:43] chip, deterministic compiler based, which [21:46] is completely different than how GPUs work, [21:48] it's very complimentary for the culture and the engineering [21:50] teams to come together as well. [21:52] How many people in here have used OpenClaw? [21:56] I mean, that's pretty incredible penetration. [22:00] I saw a stat-- [22:01] Marc Andreessen may have tweeted this today, [22:04] that most of the people he talks to are somewhere between $100 [22:08] and $1,000 now a day on token consumption with OpenClaw. [22:15] And he said, basically, the next 20 years of Silicon Valley [22:19] is going to be producing technologies to drive down [22:22] the cost of intelligence. [22:24] And so I want to talk about that, Sunny. [22:27] If we look at the cost of inference, [22:29] it's dropped by basically 90% over the course [22:33] of the last year. [22:34] It's dropped by closer to 99% over the course of the last two, [22:38] two and 1/2 years. [22:40] So talk to us about what's driving [22:43] the unit cost of inference. [22:46] And if I take a like for like-- let's call [22:50] it a unit of intelligence, whether it's a basic question I [22:54] ask or whether it's a little bit more complicated question I ask, [22:58] do you expect that unit cost to continue to go down? [23:02] And if so, why? [23:03] What are the inputs to that unit? [23:04] Yeah. [23:05] So the inputs are I think the following three major things. [23:08] The supply chain, like what can you [23:11] do across the supply chain which is mostly [23:14] centered around Taiwan today. [23:17] TSMC and the different packaging technologies, [23:20] and the lithography technologies that they buy from others. [23:22] The innovation that your engineers can perform, [23:25] and what I would say is the amount of power you have. [23:33] And so those are the kind of things we're talking about. [23:35] And so what we see today is lithography technology [23:41] is starting to reach a limit. [23:43] We're not going as fast as we used to, [23:45] and so we're not getting the Moore's law [23:48] so we have to exceed that. [23:49] So we're exceeding that in a couple of different ways. [23:52] We're exceeding that by making bigger and bigger chips. [23:54] And so if you see that these chips become quite large now, [23:58] which is very exciting, but also lead [24:00] to a lot of interesting issues. [24:02] Cerebrus, as Brad's been talking about, [24:05] their chips are kind of size like a pizza box versus CPUs [24:09] you guys all would have seen. [24:10] And so there's a lot of energy and technology there. [24:12] Like, how big of a package can you make [24:14] and how much silicon can you pack in there? [24:16] Then there's the innovations. [24:17] And so the innovations is really where [24:20] we're seeing most of this work happen, because that's [24:23] hand in hand with the models. [24:25] We're seeing this really interesting force today, [24:28] and there is a bunch of stuff that-- [24:30] I don't know if it was leaked or put out there, [24:32] and Elon is at the center of some [24:34] of that, which is they're discussing [24:35] these newer models are approaching like 10 trillion-- [24:37] 1 trillion to 10 trillion parameters. [24:39] And those 10 trillion parameter models [24:41] go back to the first thing I told you. [24:43] That's, in the fundamental flop calculation, [24:45] how much compute it takes to generate a token. [24:48] So as fast as companies like us are making [24:52] better and better technology through lithography upgrades, [24:55] through memory bandwidth upgrades, through innovation [24:59] and how we lay out our circuits, through quantization efforts, [25:02] NVFP4 was another one, the models are getting bigger [25:05] and then the demand is increasing. [25:07] So it's going to-- put it back to you, [25:08] there's this three it's a cube, but it's really difficult [25:13] to navigate right now because all the factors are [25:17] growing in ways which are really challenging. [25:20] So the demand keeps going up, the models keep getting bigger. [25:23] And as fast as we're innovating, even if we get a 50x over five [25:26] years, the models, the demand are going faster. [25:29] And that's why we're seeing this unique phenomenon like H100 [25:32] prices, if you're building a startup or using them, [25:34] they're going up, in fact. [25:36] Yes. [25:37] Like one of the things that I think is important for everybody [25:41] to understand, I mean, when OpenAI and Anthropic started, [25:45] their gross margins on the businesses were highly negative. [25:51] So that's a scary thing to do, go raise a lot of money, [25:55] and it basically produce a widget for $1, [25:59] and you're selling it for $0.20, and you have a big negative [26:03] gross margin. [26:04] But why was that? [26:06] They were going out and they were charging you [26:08] all to use ChatGPT, or their APIs [26:11] were charging a certain amount of money, [26:14] they weren't that capable so there was only so much money [26:17] we were willing to pay. [26:19] And two years ago, the cost of inference was a lot higher. [26:24] But the bet they were making is that the cost of inference [26:27] would come down a lot, and your willingness to pay [26:31] would go up a lot as intelligence [26:33] got a lot more valuable. [26:35] So I like to think of the first inning of AI [26:39] was just getting to a place where we could yield answers. [26:43] In code generation it was basically like autocomplete, [26:46] tab complete. [26:47] In the case of ChatGPT, it was like basically [26:50] telling a slightly better version of Google. [26:53] But now we're entering into this phase of action [26:57] where agents do things, go build me an app, [27:00] go build me a website, figure out [27:02] how to resolve this customer service problem, [27:05] sell more of my product, find a cure for cancer, book me [27:09] a hotel in New York. [27:10] It starts doing things. [27:12] And when it does things, the amount of tokens it has [27:14] to consume in order to do those things explodes by an order [27:18] of magnitude, but the value delivered to the end consumer [27:22] as a unit of intelligence goes up by 100x. [27:26] So your willingness to pay goes up dramatically. [27:28] Can I add one to that? [27:30] This week we saw Mythos-- which is the unreleased model [27:34] by Anthropic-- [27:35] find a bug in BSD which-- [27:39] think about how many engineers and software developers [27:43] and everyone else and companies using [27:45] that have looked at that code. [27:46] So we've gone to a place where it's [27:48] doing things beyond human capability, which is-- [27:51] And we're in year three. [27:53] Exactly. [27:53] We're in year three. [27:54] To give you another-- like, what's [27:57] my best evidence to convince you of the value of AI? [28:00] Well, my best evidence is that Anthropic in the month of March [28:04] just added $10 billion in annualized revenue [28:08] in a single month. [28:10] That is the total amount of annual revenue for Databricks [28:16] plus Palantir combined. [28:18] And they added it one month. [28:21] And they didn't add it because they [28:23] hired a million salespeople, went out to a million companies [28:26] and convinced them to buy their product. [28:29] They added because their product crossed [28:33] a threshold of intelligent capability [28:36] that millions of customers around the world said, [28:39] I have to have this product to make my company better. [28:42] The amount that Altimeter is spending went up, [28:45] but millions of self-interested actors around the world [28:49] independently made a judgment, I have [28:52] to buy a lot of those tokens, a lot of those capabilities, [28:55] both Cloud Code and Cowork. [28:57] And the same thing is happening in OpenAI, not quite [29:00] on the same exponential in terms of revenue. [29:04] But I think for me, this was a little bit of an Oppenheimer [29:09] moment, this was a little bit of the splitting of the atom. [29:12] Like, we've heard Dario and Sam talk [29:15] about the exponential or the end of the exponential [29:17] on intelligence, but the big question was, [29:21] are they going to be able to afford [29:23] to continue to build the compute in order to keep up with this? [29:27] I had this somewhat uncomfortable moment [29:30] with Sam Altman on my podcast, The BG2 Pod, [29:33] that went a little viral when I asked Sam, hey, Sam, you've [29:38] made $1.4 trillion of spending commitments, [29:42] but you only have 13 billion of revenue. [29:45] So explain to me how that works. [29:47] Like, how can you commit to spending $1.4 trillion, [29:50] you have 13 billion of revenue? [29:51] And I had hoped that Sam would make [29:53] the case that his revenue was going to go up a lot, [29:56] and these were kind of call options [29:58] and he could renegotiate them, but instead he said to me, well, [30:02] if you don't like your investment, [30:03] I'll buy back your shares. [30:04] Which was not exactly the response [30:07] I was hoping for out of Sam in the moment, [30:10] but that was the question. [30:11] Heading into 2026, my podcast partner, [30:17] Bill Gurley, a lot of other people highly skeptical [30:20] saying this is an AI bubble. [30:21] These guys are spending at rates they're never [30:24] going to be able to pay the bills on because there aren't [30:27] people on the other end willing to pay for the products [30:30] to justify that level of spending. [30:32] And what happened in January was Anthropic had a $3.5 billion [30:37] a month. [30:38] In February they have an $8 billion month. [30:41] And in March they have a $10.5 billion a month. [30:45] That to me said, oh, everything's changed. [30:48] The product is now sufficiently good [30:50] that you have revenue scaling on the same exponential [30:53] as intelligence, so they can afford [30:56] to pay for the $50 billion per gigawatt [31:00] to stand up all of these inference factories [31:03] to produce all this kind of collective intelligence. [31:06] Just react to that, Sunny, because our group [31:09] talks a lot about this. [31:10] There was a lot of debate in our group [31:12] and on the All-In Pod and others as to whether or not [31:14] this was a bubble. [31:16] Yeah. [31:16] I'd say there's of a couple of things [31:18] that maybe the broader world doesn't see yet. [31:22] One, the models that we see today [31:24] haven't even been trained on the latest hardware, [31:26] whether you want it to be Blackwell's or-- [31:30] Vera's are just coming out or Rubin's are just coming out, [31:33] so we haven't even seen that yet and so we haven't seen [31:35] the capabilities that you get. [31:36] And so we'll start to see that. [31:38] I think one of the first ones we'll see [31:40] is the stuff out of Elon's Grok. [31:42] So that's A. So the capabilities you're seeing here [31:45] are things that were done on older hardware. [31:47] So that's A. And so when you're inside the ecosystem, [31:49] you know what's capable and what's coming next. [31:51] I think B, one of the things that is really starting to take [31:56] off-- and I think Anthropic has done an incredible job here, [31:58] and I think Codex has an equally incredible job on very hard kind [32:03] of software problems is that there's not just a chat [32:06] interface that majority of people are interacting with, [32:08] and it's not just an API, but they've created like a harness [32:12] around the models, and those harnesses-- [32:14] OpenClaw is just another harness as well-- those [32:17] harnesses have figured out how to extract more and continually [32:21] extract. [32:21] I think with Cloud Code and Cowork [32:24] you can have it just ping you whenever [32:26] it's stuck on your phone, even if you start it somewhere else. [32:28] And so it can be in this continuous loop [32:29] and it's working for you all the time. [32:31] We've never had anything like that. [32:32] When it's doing that, you take that token consumption of like [32:36] were doing a query before and was doing some thinking [32:38] and coming back, now it's just working all night long [32:41] and pinging you. [32:41] You tell it, don't even bother me. [32:42] Keep coming back. [32:43] So we're seeing these harnesses really extract more and more [32:46] tokens out of it as well. [32:48] And the type of problems that people are solving, [32:50] we gave the code problem, you put a bunch of other ones, [32:52] but inside big businesses-- and I tweeted this, [32:56] and I think it's fair, but like inside NVIDIA now [32:58] we have this thing called the NVIDIA Personal Assistant. [33:01] And it's connected to Slack. [33:02] It's connected to Teams. [33:04] But it's connected to our email, and it's [33:08] connected to all our files, wherever they may exist. [33:12] And so every morning it runs, and you're like figures [33:15] out like all your task items for the day. [33:17] You can have it answer those things, [33:18] and it's really incredible. [33:20] And so you start to-- the way we work-- [33:21] and we we were talking about this earlier with someone-- [33:24] like, you don't even write email now. [33:25] Like, someone else's agent is going in their email, [33:27] emailing you, and your agent is looking at emailing them back. [33:30] But a lot more work is getting done because my time is freed up [33:33] from basically answering emails all day [33:35] long and approving things out of all [33:36] these traditional SaaS systems. [33:38] The agents handle all that. [33:39] So the explosion, to your point, is just [33:42] in the first or second inning, the amount of tokens [33:45] is really just going up. [33:46] So we don't fear that. [33:48] We don't look at that as an overbuild in any way, shape, [33:51] or form. [33:51] I think the facts and evidence on the field [33:56] is, number one, the cost of both training and inference, [34:00] but inference in particular, is plummeting and continues [34:03] to plummet. [34:04] That shouldn't be altogether surprising. [34:06] Technology ultimately is highly deflationary. [34:10] I've never seen something this deflationary this quickly. [34:13] I think it's a byproduct of extreme co-design. [34:16] It's not a single chip, it's a factory. [34:19] And across the factory there are all sorts of Moore's laws [34:22] playing out combinatorially across the factory. [34:26] At the same time, when you're able to produce [34:28] a lot more tokens, the unit of intelligence [34:30] that you're delivering is much more valuable, [34:33] so the willingness to pay on the other end goes up a lot. [34:36] And I'll tell you, for an OpenAI or an Anthropic today, [34:40] if those guys were at negative gross margins a year [34:42] and a half or two years ago, they're now [34:44] at very positive gross margins. [34:47] So all of a sudden, this business that looked diseconomic [34:50] looks highly economic today. [34:52] So it's resolved a little bit that question. [34:57] Maybe just, Sunny, I want to finish our section [35:01] with maybe just a little forecast [35:02] and pre-wire you guys to-- we're going [35:05] to open it up to questions. [35:06] It can be about the economics of inference [35:08] or any part of the stack or any other questions [35:11] that you all have. [35:13] But you mentioned Mythos. [35:15] It's a model out of-- that came out this week. [35:18] Was not generally released, but was sandboxed by Anthropic. [35:23] Tried to get out a few times. [35:25] Tried to escape the sandbox. [35:27] Trained on TPU7. [35:30] On the other hand you have SPUD or 5.5 coming out of OpenAI [35:34] probably this week or next, which is a first Blackwell [35:37] trained model. [35:39] Elon's going to have one. [35:40] Meta's just out with a model yesterday, Google, et cetera. [35:44] Talk us through-- you get to see into the product pipeline [35:48] at NVIDIA. [35:50] Do you think that the pace of-- [35:55] the cost of inference curve continuing to come down, [35:59] do you think that continues for the next several years? [36:02] Do you think that the step function or the exponential, [36:05] if you will, of both pre-training and inference time [36:08] reasoning in terms of improving the algorithmic capabilities [36:11] of intelligence continues? [36:12] Well, I can tell you, having a chance to work with Jensen now, [36:16] he challenges us in everything we do, [36:18] to not show up unless it's 100x. [36:21] So whatever we bring to him-- and I can't get into too many [36:26] details, but his first challenge back is, [36:28] is this 100x from what you did before? [36:31] So he is challenging the engineers to take a look [36:33] at every part of the problem-- from all the way down [36:37] into memory controllers or memory capacity or circuits, [36:40] whatever it happens-- to be to make sure we 100x everything. [36:43] So on the first part of your question, [36:45] yes, because he pushes us to do it, [36:46] and he gives us the latitude to do it [36:48] and he gives us the resources to go do it. [36:50] So I can tell you, like, the types of things [36:52] that we've been enabled in coming in as the Grok [36:54] team, things we could never do as a startup, [36:57] but Jensen has enabled us to do those things. [36:58] So are you guys harnessing AI yourselves [37:02] to design the next generation chips? [37:03] A ton. [37:04] We were doing that even before because we needed to, [37:06] we were a small team, but now we have [37:08] access to the entire ecosystem of things that are available. [37:12] So I think that's A, is that we're being pushed to do it. [37:16] On the related side, though, the more we innovate, [37:20] the more the model makers innovate [37:21] and the bigger the models get. [37:23] And so-- which means the capabilities that are coming out [37:27] are better, so we continue to need that build out. [37:30] So we'll all look back and we'll think, [37:33] there's a couple companies that changed the footprint [37:35] of the internet for us. [37:36] And you could talk more about this and I can even brag, [37:38] but the work that Google did to build the infrastructure they [37:42] did for video, for search, it really [37:45] paved the way for the rest of the internet, CDNs, [37:47] all types of other things. [37:48] And so a lot of this work that's happening to build out [37:51] this infrastructure will pay benefits, [37:53] and you need that to continue to happen, [37:55] because it can't just be in the innovation of the chips. [37:57] You need more and more infrastructure to be built. [38:00] I'll wrap with this, we have the great privilege [38:04] of talking with Jensen or Elon or Sam or Dario, [38:09] and you guys all can read about the personal battles [38:13] they have between those, some days, not a lot of love [38:19] lost between them in the race to AGI. [38:24] But right now I see amazing uniformity. [38:27] When I talk to them, they all, in a non-hyperbolic way, [38:31] say we're there, and we got there faster than we thought. [38:36] Like, we're nearing the end of the exponential. [38:38] And if you ask Dario, Dario, what [38:40] is the most surprising thing to you right now? [38:43] He says, we're almost at the end of the exponential [38:46] and people don't even seem to realize it. [38:48] And if you ask Sam, he'll say the same thing. [38:51] And if you ask Elon, he'll say the same thing. [38:53] That shouldn't be scary to any of us. [38:56] It just means that we're in this recursive place [39:00] where we have AGI. [39:02] And the job of everybody in this room-- including [39:05] the folks sitting up here-- is going [39:07] to be how do we harness this technology [39:10] for the betterment of all of us, which [39:13] is going to require going back to what [39:15] Apoorv said about the Invest in America Act. [39:18] Dario says the accumulation of wealth [39:22] that's about to occur-- people call it the age of abundance [39:25] we're going to enter into, that's [39:27] going to be easier than ever. [39:29] But the distribution problems we're going to encounter [39:32] are going to be harder than ever. [39:34] And so that was really the inspiration behind the work [39:38] I've done on Invest America and the work [39:40] that I think we're all going to have to collectively do around [39:43] the social contract, the intersection [39:45] between public policy and technology. [39:47] Because when the exponential looks [39:48] like that and all of a sudden you [39:50] have agents that are going to be able to have more [39:54] capability than collective human intelligence-- [39:57] and it's happening at an accelerating rate. [39:59] Remember, all the stuff we've talked about [40:01] has occurred with almost no compute. [40:05] Anthropic and OpenAI are going to add more compute this year [40:09] than all the labs put together for the last decade. [40:12] And the year after that, they're going to double it again. [40:15] So I think the rate of change is fairly parabolic. [40:21] And that to me is both exciting-- [40:23] I'm an optimist about what's to come, [40:25] but I'm not Pollyanna about the challenges that [40:30] come with that rate of change. [40:32] Like it's going to require active engagement, [40:35] like it has in other periods in history around the Industrial [40:38] Revolution, the Digital Revolution, et [40:40] cetera, because it's going to exact a lot of change [40:43] on the world. [40:43] But with that, I just want to say, [40:46] it's been extraordinary watching Sunny orchestrate the work [40:50] that he's done at Grok. [40:51] He's an incredible thought leader in this whole area. [40:55] I appreciate you coming in. [40:56] But maybe just open it up some questions, [40:58] and hopefully we can cover a lot of territory. [41:02] Right here. [41:03] [APPLAUSE] [41:10] As the marginal benefit of [INAUDIBLE] increasing wages [41:13] are estimated, how do you suggest that a player positions [41:17] themselves and to make sure we're not just [41:19] wasting our time studying them? [41:22] [LAUGHTER] [41:23] Yeah. [41:24] I mean, listen, I have to answer this question for my son [41:29] and for so many others. [41:33] And humans have a unique way of finding a way [41:37] to add value to society, notwithstanding disruption. [41:42] In the Industrial Revolution, if you [41:44] were a tradesperson or a craftsperson, [41:46] and you build a product beginning to end, [41:48] almost all of them were displaced by mass production. [41:53] And for that person, it didn't feel good. [41:56] I was really good at making a wheel start to finish, [41:59] but I was totally disintermediated [42:01] by the means of production. [42:03] OK. [42:04] But it's not like we-- the world just stopped. [42:09] Those people found other things to do. [42:11] And one of the observations I have [42:13] is that we used to have 80% of people that [42:16] were in manufacturing and that were [42:17] in farming and other things, today we [42:20] have 70% of people in the service economy. [42:23] We have the luxury of people-- we didn't used [42:26] to hire coaches, as an example. [42:29] You couldn't afford to hire a coach. [42:31] Today you have coaches and yoga instructors and tons of things [42:35] in the world that adds a lot of value to the world, [42:37] and I think that we have higher order things that people do. [42:41] And so for me, one of the things is [42:44] if you are well off enough that you [42:46] could hire a tutor, a specialized tutor, that [42:49] was great. [42:50] But for 98% of the world who couldn't afford that, [42:52] now they can get that. [42:53] Or if you were part of the 2% or 3% [42:56] that could have concierge medicine, it was really great. [43:01] But for the other 97%, it wasn't great. [43:03] Well, now they can get that same level of care. [43:07] And so I think this is about democratizing intelligence, [43:10] democratizing access, et cetera. [43:13] But it's not to say that there aren't going [43:15] to be different challenges. [43:17] My number one thing, again, is make yourself bionic, [43:20] be a creator. [43:22] Figure out a way that you add value. [43:25] So if somebody comes and wants to interview at Altimeter, [43:28] and they say, oh, I don't use AI and I [43:32] don't use Excel Spreadsheets. [43:34] I do everything by hand, that would be a problem. [43:38] Like, I expect somebody to use all the greatest [43:40] tools at their disposal to be the most effective they can, [43:44] to add value to allow us to generate alpha in the world. [43:48] And so starting in a place like this, [43:50] another way of saying it-- reserving this for a tweet [43:54] at some point in time-- but I think that IQ gets commoditized [44:00] and EQ becomes super valuable. [44:03] What do I mean by EQ? [44:05] I mean a network of people in this room. [44:07] I mean the ability to persuade the person sitting next [44:10] to you, the ability to form your team, [44:13] the ability to lead people in different directions, [44:15] like, that is super valuable. [44:18] I think it becomes more valuable in the future. [44:20] But I think just being the smartest person in a room [44:24] and solving the problem at the board faster [44:26] than all the other humans in the room, [44:28] like that, I think is commoditized [44:30] and you're not going to be able to beat the machine. [44:32] It doesn't mean you don't need to learn those things, [44:34] but I think you'll be hard to beat the machine. [44:36] Brad, I want thank you for doing this. [44:38] We need BG2 back. [44:39] We miss it. [44:40] We only get Brad once in a while in the All-In Pod now. [44:42] You only get your slot, so-- [44:44] I need to get you back on. [44:46] Yeah, let's get BG2 back. [44:48] But can I add one thing to that. [44:49] I think there's this other moment that's [44:52] occurring right now, and I think about this quite often. [44:55] If you actually look at what's happening in mathematics right [44:58] now, there's all kinds of new discoveries happening. [45:00] And I use the following analogy, like, humanity [45:02] had to wait for an apple to fall on Newton's head for him [45:05] to then start theorizing about gravity [45:07] and start formulating that. [45:09] But now if we can have something else working and discovering [45:12] new things, like-- and it goes back to that chart [45:14] that Brad showed. [45:15] It's really until we started having more innovation, [45:18] more intelligence that those curves went up and to the right. [45:21] We're just about to make that go more vertical. [45:24] So I think the overall benefit to humanity [45:26] has already been shown what happens when [45:28] you have more intelligence. [45:29] And so we don't have to wait for things to happen. [45:32] We let the agents do it without us in the loop, which [45:34] I think will be powerful. [45:36] There's one up. [45:37] Yeah. [45:37] When you think about the creation of hardware [45:39] and software, particularly given that Apple [45:41] is following the strategy that everybody else, [INAUDIBLE] [45:45] all of them. [45:46] And OpenAI is building a device which [45:47] might actually try doing that. [45:49] So I'm just curious as to how you think about that. [45:52] I think it's a high stake-- [45:53] I mean, listen, I'll tell you, even the people at Apple [45:55] are nervous with their strategy. [45:57] And so part of it is their challenge around privacy. [46:03] They have a real challenge with-- because we [46:06] don't have the capability yet on the edge, [46:09] and they don't want to have you sharing information up [46:13] to the cloud, given their-- they view [46:17] as one of their core consumer value propositions [46:20] is consumer property-- or consumer privacy, [46:24] but I think they put themselves at risk. [46:27] The bull case would simply be that we're [46:31] so sticky to the device and the device [46:33] is so good that they have time, and ultimately [46:36] the Gemini model that they're going to put on the phone [46:39] is just going to be a much more capable Siri. [46:41] We can all agree that old Siri is really bad [46:43] and will be a more capable Siri. [46:45] For the vast majority of people, that will be good enough. [46:48] So that would be the bull case. [46:50] I think the bare case is, like you said, [46:53] that other people come along and build more ambient devices [46:56] that consumers really liked. [46:59] But for me, I frankly wish that OpenAI [47:02] wasn't working on a device. [47:04] I wish they would just focus on building intelligence. [47:07] And I think Apple is going to be very formidable in the device [47:11] world, so think they're in a reasonably good spot. [47:13] One stat, an $8 billion parameter model-- [47:16] which is quite small-- [47:18] can burn out a phone-- an iPhone in 30 minutes. [47:21] It goes back just-- [47:22] Battery life. [47:23] Yeah, battery life. [47:24] Yeah. [47:24] So I just say you got to go back and look [47:26] at how compute intensive AI is. [47:28] And so that's the real challenge on the stuff [47:30] that Brad said about pushing so much [47:32] of that frontier intelligence to the edge. [47:36] Over here, yeah. [47:39] I want to touch on something we're [47:41] talking about with Invest America. [47:44] And I think it was on a recent episode of All-In, [47:46] and Chamath was criticizing various tech CEOs for-- [47:52] in creating hype around new product launches. [47:55] So I'm curious what AI we're doing to national security [47:59] agency jobs. [48:01] And I guess I'm curious, what extent [48:04] is CEOs' need to revise and rethink their messaging? [48:10] He and I had that argument on the pod again today. [48:12] Again, call me naive. [48:14] I think that Dario's speaking authentically what he believes. [48:18] I think Sam's speaking authentically what he believes. [48:22] They're staring at this exponential. [48:23] They believe that they see AGI or ASI, [48:26] and I think they do have legitimate concerns. [48:29] Like, listen, I'm glad that we sandbox Mythos. [48:31] They tested it internally, they found 26 vulnerabilities [48:36] on the Safari browser. [48:37] And like I said to Chamath today, [48:39] do you want them to just throw it out there [48:40] and then all your browser history is out in the public? [48:43] Probably not. [48:44] So at the same time, I don't think it helps going out [48:49] and fear mongering, particularly if your real intent is [48:53] regulatory capture to prevent everybody else from climbing up [48:56] the ladder now that you're on the top. [48:58] Like, I have a real problem with that. [49:01] But I think we have to find that balance and those trade [49:03] offs between reminding people about the optimistic side [49:10] of things. [49:12] And I encourage you to read both of Dario's essays. [49:20] And his first essay on this is quite optimistic about what [49:25] can happen. [49:25] But I think he has the other side of it, which [49:27] is but it doesn't happen without us being very [49:30] thoughtful about the guardrails and things we [49:33] need to put in place. [49:34] One of the things I was really happy about [49:36] is called Project Glasswing, which [49:39] is this consortium that they put together [49:41] this week to effectively sandbox Mythos before they release it [49:45] publicly-- [49:45] Amazon, Microsoft, et cetera. [49:47] Like that team seemed to me to be [49:49] a very pragmatic, market-based solution to solve the problem. [49:53] And he and I were just texting before I came over here. [49:56] They found and they've hardened a lot of things [49:58] already very quickly, and within 100 days [50:00] you can do a lot when you're having the AI fix [50:03] the things that it finds. [50:04] And so I certainly-- [50:09] I talk optimistic about it, I think a lot of other people do. [50:12] Might encourage them to find a little bit more balance [50:14] in their commentary, but I also don't [50:19] want us to ignore the realities that when you split the atom, [50:25] it can either provide unlimited free energy for the world [50:28] and totally bring people out of darkness, [50:32] or it can be used to make a bomb to destroy cities and nations. [50:36] And so powerful technology is powerful technology. [50:39] We can't just stick our head in the sand [50:40] and act like it's a one way street. [50:43] OK, there. [50:44] Well, that's what [INAUDIBLE] we're [50:47] mentioning in terms of [INAUDIBLE] is about 100 billion [50:54] by 2030. [50:55] So I'm just curious how you're balancing [50:57] the fact that it's cheaper [INAUDIBLE] company wide. [51:01] I think you see a couple of phenomenons. [51:04] One, the gear that's used for training [51:06] turns into inference gear, right. [51:08] So those big clusters, we're seeing [51:10] that happen kind of all the time, [51:12] so there's just a natural progression between those two [51:18] worlds. [51:18] And then I really do think the innovations that [51:22] come from those larger models and those training clusters [51:26] have such a large benefit, and they tie back to what Brad said. [51:32] I was just reading this thing today by Mustafa [51:34] from Microsoft saying, look, GPT 2, what-- [51:39] we have 50x more powerful compute today than what we did [51:42] when we did GPT 2, but look at the capabilities. [51:46] And so I think you just have to keep those two things in line [51:49] with the entire topic of the conversation. [51:53] Innovation is going to keep happening because-- [51:56] Brad touched on it a little bit-- [51:57] we're just now unleashing AI into designing these things. [52:00] And there's things that we see and we [52:02] learn in terms of optimizations and software and hardware [52:05] optimizations that we don't see. [52:06] So I continue to believe it'll come down, [52:09] but, yeah, we're just working on a problem that's just very, very [52:13] intensive from a compute standpoint [52:15] so those numbers are going to be large. [52:17] Maybe one final question. [52:19] You can pit dive with Sunny and I, [52:22] we can answer some questions after the fact, [52:25] and-- but otherwise I just want to say [52:29] it's a great privilege for us to get to spend some time with you [52:32] guys, so thanks for having us. [52:33] But maybe right here. [52:35] Yeah, hi. [52:36] Thank you for sharing. [52:37] Just one question about Columbus. [52:39] [INAUDIBLE] So, that's going to be [52:42] our major shareholder in NVIDIA and sending [52:45] your [INAUDIBLE] immediate one. [52:48] So what do you think is the long term sustainable economic model [52:51] for NVIDIA, because right now [INAUDIBLE] if you have [52:58] 60% ever, right that makes it very [53:03] difficult for the entire ecosystem [53:07] because they take way too much time. [53:11] What do you think would eventually would happen? [53:13] Let's say a few scenarios. [53:15] A, their revenue may be limited, but they keep their margin. [53:21] And two, is somehow lower their margin substantially [53:26] and be able to keep a much bigger market share. [53:30] So, maybe just repeat it real quick [53:33] and then answer it since I'm an investor NVIDIA. [53:36] I work there so I shouldn't answer that. [53:38] No. [53:38] I mean, listen, NVIDIA is a $4.5 trillion [53:42] company that's trading at about 13 times earnings. [53:45] Very cheap, half the market multiple. [53:47] Growing at 70%. [53:49] Is obviously dominant in the market today. [53:53] And I think there's a wall of worry about NVIDIA [53:57] because everybody says what you do is Trainium [54:01] and TPUs and Cerebras and Grok. [54:04] And all these people can come up with inference solutions, [54:06] they can steal your share, they can compete on price. [54:09] That's the beautiful thing about capitalism. [54:11] You know what NVIDIA will have to do? [54:12] Compete. [54:13] They either deliver a product that people [54:15] are willing to pay more for, or they have to drop their price, [54:17] their margins come down, and they'll [54:19] have to compete in that market. [54:21] I would tell you that when I look into the product [54:24] roadmap for what's going on at NVIDIA, [54:27] and the acquisition of Grok was part of it, [54:29] I think they're going to be in an incredible position. [54:31] They've already announced that they have a trillion, a trillion [54:35] of sales over the course of the next eight quarters [54:37] that are already booked. [54:39] People have more demand than they [54:41] can get memory and supply to build all of this compute out. [54:45] So I think we're so early in this. [54:47] I was in Silicon Valley not so long ago, 16 years ago, [54:52] when they said there could never be a trillion company, [54:55] there would never be a trillion-- [54:57] I'd ask the question, well, why? [54:58] So they say, well, law of large numbers. [55:01] I was like, well, what stone tablet is that etched into? [55:05] It nowhere. [55:07] Today we have a $4.5 trillion company. [55:09] I've already said publicly NVIDIA [55:11] will be the first $10 trillion company. [55:13] And I'm not-- it's not because I'm a cheerleader. [55:16] I can sell my NVIDIA and invest in anything [55:17] that I want to invest in. [55:19] But that company's leadership, that team, [55:24] the lead that they have on both training and inference, [55:27] and the rate at which they're moving I think [55:30] puts them in a really great competitive position. [55:32] And they're doing all this notwithstanding the fact [55:34] that Tranium is successful, TPU is successful, [55:37] custom ASICs are being very successful [55:39] and they're still killing it. [55:41] And I think that says a lot more about the size of the market [55:44] for intelligence and the compute that's [55:46] needed to get us there than it does [55:48] about the individual company. [55:50] With that said, we have to wrap. [55:53] Apoorv, thanks for having us. [55:54] Thank you. [55:55] Thank you.