MatX Combines SRAM and HBM to Break AI Chip Latency Bottleneck

MatX融合SRAM与HBM，打破AI芯片延迟瓶颈 这家初创公司计划通过在单颗芯片上融合两种内存类型来超越科技巨头，消除大语言模型在速度与经济吞吐量之间的棘手权衡难题。 在本次访谈中，Reiner Pope探讨了Google自研TPU硬件的历史演进，并讨论了MatX如何通过设计纯粹专为大语言模型定制的芯片，来解决整个行业在延迟与吞吐量之间的权衡难题。他详细阐述了芯片设计的高风险瀑布流特性、在台积电代工产线争取产能份额的运营现实，以及数据中心基础设施面临的严苛物理要求。此外，Pope还分享了系统工程中Rust相较于其他语言的技术优势见解、他个人的周末优化项目，以及通过分离预填充（pre-fill）与解码（decode）模型架构所能开拓的全新效率空间。 Google近期凭借Gemini 3实现的AI复兴得益于其自研TPU硬件，这一结构性优势始于2016年的最小可行产品TPU v1。 (0:15) 实现硬件协同需要“机械共鸣”（mechanical sympathy），与传统CPU不同，AI硬件优化的目标是达到峰值性能的70%至80%。 (3:22) MatX的创立旨在打造纯粹针对Transformer架构优化的LLM芯片，摆脱了Google和NVIDIA等巨头必须兼容传统工作负载的约束。 (7:55) AI硬件面临着一个棘手的权衡难题：依赖HBM的芯片倾向于高吞吐量却承受高延迟，而纯SRAM芯片虽能实现超低延迟但吞吐量极低。 (14:00) MatX正在通过在单颗芯片上集成这两种内存类型来化解延迟与吞吐量的权衡，将模型权重存放于SRAM中，并将活跃的推理数据存放于HBM中。 (15:20)

Reiner Pope, co-founder of MatX and former Google engineer, argues that maximizing large language model performance requires purpose-built hardware that overcomes fundamental memory latency and throughput bottlenecks. He contends that shifting economic dynamics at frontier AI labs make bespoke non-CUDA architectures viable, despite severe capital requirements and physical supply chain constraints. Reconciling Memory Latency and Throughput for LLMs (14:00) Hardware for deep learning has historically forced a trade-off between latency and throughput. High-throughput chips from NVIDIA, Google, and Amazon rely on High Bandwidth Memory (HBM) to maximize concurrent inferences, yet they incur latencies of roughly 20 milliseconds per token to read through memory. Conversely, architectures using on-chip Static Random-Access Memory (SRAM), such as Cerebras or Groq, achieve 1-millisecond latencies but suffer from poor throughput and unfavorable unit economics. MatX attempts to resolve this divide by placing model weights in SRAM while storing active inference data in HBM, targeting both low latency and high throughput. To execute this, MatX pairs its memory configuration with a reconfigurable systolic array that can split into smaller segments to process Transformer attention mechanisms without losing the efficiency required for Mixture of Experts (MoE) layers. Pope highlights that the company relies on 4-bit precision arithmetic and intentional "sloppy numerics"—sacrificing strict IEEE rounding standards to cut hardware gate complexity—a compromise validated internally by training small models from scratch. The Strict Waterfall and Physical Bottlenecks of Production (27:11) Unlike iterative software deployment, chip manufacturing follows an unforgiving waterfall lifecycle: architecture, Verilog logic design, verification, and physical layout synthesis. A single initial tape-out costs approximately $30 million—compared to $1,000 for subsequent chips—while industry first-pass success rates hover around 50%. Serious logic errors require restarting this multi-million-dollar process. Production is further gatekept by leading-edge foundries like TSMC, where advanced nodes are essential for power efficiency rather than density gains, and constructing a competitive fabrication plant requires an estimated $10 billion. Beyond silicon, Pope emphasizes that rack-level integration presents severe mechanical hurdles. High-speed interconnects rely on rigid, heavy copper cabling that resists bending while needing to preserve signal integrity under intense power draw and heat dissipation constraints. For a startup, procuring scarce components like HBM and wafers requires proving survival to conservative suppliers; MatX navigates this dynamic by raising $500 million and securing advance commercial purchase contracts. Rethinking the CUDA Moat Through Co-Design (33:19) Pope challenges the conventional assumption that NVIDIA’s CUDA software ecosystem forms an insurmountable barrier in frontier AI. While backward compatibility remains critical for PC and gaming ecosystems, the frontier AI market is concentrated among a handful of well-capitalized labs. For organizations committing $10 billion to a compute cluster, Pope argues that employing a 50-person engineering team to write custom software for alternative chips is a minor operational cost if it doubles hardware throughput. To bridge the gap between silicon and software, MatX incorporates an internal machine learning research team that co-designs hardware directly with model structures. On the systems programming side, the team favors Rust over Haskell or Go, utilizing Rust's type system to define irregular hardware types—such as 17-bit or 19-bit integers—while avoiding the latency and memory overhead introduced by garbage collection headers. Engineers rely on mental arithmetic and internal cost heuristics like `go/gates` to estimate gate and SRAM bitcell trade-offs before committing designs to simulation. Navigating the Multi-Year Architectural Lag (40:26) The multi-year lag between chip architecture and tape-out prevents hardware teams from optimizing for specific production models. Instead, designers must evaluate a probability distribution of future model architectures two years in advance, engineering hardware versatile enough to capture 90% of expected workloads. Looking forward, Pope expects major gains from decoupling current model constraints, specifically splitting pre-fill computation from token decoding and designing architectures where inference compute scales independently from training configurations. These physical constraints also influence long-term scaling concepts like space data centers. While terrestrial data centers absorb a modest 10% redundancy tax by routing around chip failures and deploying technicians within a day, space-based installations cannot be physically serviced. Assuming a 3- to 5-year chip lifespan, space clusters would incur an estimated 100% redundancy tax in extra deployed silicon, making such facilities viable only if terrestrial power limitations become sufficiently severe to justify massive capital trade-offs. MatX targets its first tape-out within a year, expecting early deployments in inference by 2027 where customer adoption risks are lower than in model training. However, the company's trajectory hinges on navigating TSMC production queues, executing first-pass silicon verification, and proving that specialized architectural co-design can outpace entrenched incumbents in an era of rapidly evolving foundation models.

MatX联合创始人兼CEO Reiner Pope指出，现有芯片因兼顾旧有工作负载而牺牲了LLM效率，初创企业通过软硬件协同设计、混合内存架构与低精度计算，能够在吞吐量和延迟上显著突破大模型计算瓶颈。 专为LLM解耦的芯片架构与内存突破 (14:00) 传统计算体系在处理LLM推理时面临延迟与吞吐量的严苛权衡。基于HBM的系统由于遍历内存需要约20毫秒，导致单Token延迟偏高；而仅依赖SRAM的架构遍历虽只需约1毫秒，却受制于容量和吞吐量，经济性较差。为打破这一限制，MatX在单芯片上结合了两种内存：将模型权重置于高速SRAM中，活跃推理数据放入大容量HBM中，以此在维持极高吞吐量的同时降低延迟。大模型算力需求还促使矩阵运算维度从过去的128扩展到数千，MatX设计了可切分且不损失效率的大型脉动阵列，使Transformer模型的注意力机制与专家混合架构（MoE）均能高效映射。 算法定制与硬件协同降低计算成本 (21:48) 硬件设计本质上需要与底层计算物理特性的“机械同感”。MatX并不盲从IEEE或Open Compute的标准规范，而是采取“不拘泥于特定数值精度”的策略，放弃不必要的舍入模式与边界情况处理，以此大幅削减硬件门电路开销。其专职机器学习团队通过从头训练小型LLM来验证这些低精度方案，并预测4-bit精度将成为未来能效的最佳平衡点。这种软硬件协同设计打破了传统AI芯片公司仅由ML团队编写算子内核的局限。在语言工具选择上，MatX采用Rust替代Go或Haskell，既获得了类型系统支持，又规避了垃圾回收器的内存开销，能够精准控制如17位、18位或19位整数等自定义硬件数据类型。 制程垄断与全栈工程交付瓶颈 (19:22) 构建AI硬件的壁垒远不止芯片硅片本身，机架级物理工程同样极具挑战。机架必须承载巨大的供电输入、散热导出，并在严苛的信号完整性要求下布设难以弯折的高速重型铜缆。在制造端，台积电等先进制程代工厂在AI能效上占据不可替代的地位，但其新建晶圆厂耗资高达百亿美元。初创公司若想在与科技巨头的产能争夺中突围，必须依托雄厚资金与下游客户的刚性采购合同来确保供应链配额。芯片从架构构想到流片下线的开发周期往往长达数年，团队需要预测未来两年的模型分布，并通过严格的瀑布流验证流程以规避高达3000万美元的单次流片失败风险。 先锋AI市场对CUDA生态的重构 (33:19) 市场普遍认为英伟达凭借CUDA生态构筑了牢不可破的护城河，但Pope认为这一壁垒在顶尖AI领域被高估了。与拥有数万款游戏需要向后兼容的PC和游戏市场不同，前沿AI领域的买家主要集中在少数几家顶尖实验室。当一个实验室在单一算力集群上投入达到百亿美元级别时，花费数千万美元聘请50名顶尖工程师为MatX等新型非CUDA硬件完全重写底层软件，仅是一笔微不足道的商业成本，且能换取性能的翻倍提升。这也为摆脱传统硬件生态锁定的专用芯片打开了市场空间。 模型架构演进与落地时间表 (52:05) 当前模型设计仍存在诸多人为设定的性能桎梏。例如，Prompt处理（Pre-fill）与逐步Token生成（Decode）的计算特性截然不同，训练阶段的高算力消耗与推理阶段的高内存带宽需求也存在错配，未来拆分这些阶段的模型架构将释放巨大效率。在产品节奏上，MatX预计在一年内完成芯片流片并在当年底提供芯片，终端用户有望在2027年通过A/B测试体验到其硬件驱动的模型服务。针对更长远的太空数据中心设想，Pope分析指出，由于无法实现地面技术人员在一天内完成芯片检修的运维闭环，太空无维护环境将使硬件冗余成本从地面的10%跃升至接近100%。 MatX展示了专为LLM定制芯片的技术可行性与高吞吐低延迟优势，但其商业成功仍高度依赖首次流片的良率表现、复杂机架供应链的交付能力，以及顶尖AI实验室为新型专用架构投入重写软件资源的实际意愿。

[1:00] Reiner Pope: pretty much anyone who's over 30 and at a large lab has been at Google Brain at some point. [3:22] Reiner Pope: mechanical sympathy [3:22] Reiner Pope: A CPU is like a motorcycle navigating a complex obstacle course... A GPU is like a massive truck carrying a huge payload [7:55] Reiner Pope: the best chips physically possible for LLMs. [21:48] Reiner Pope: Using Float32 is like having an image with a billion-color bit depth; it is far more efficient to have more pixels (parameters/compute) and fewer colors (precision). [25:33] Reiner Pope: Minecraft 3D [40:26] Reiner Pope: how much advantage do you get from vertical integration versus how much advantage do you get by concentration of R&D work. [41:23] Reiner Pope: tax of reliability [44:55] Reiner Pope: mutation is good [44:55] Reiner Pope: special sauce [47:35] Reiner Pope: tape-out a chip in one month [52:05] Reiner Pope: super hacky backdoor [57:12] Reiner Pope: We want to cut corners anywhere we can. [61:01] Reiner Pope: numbers every programmer should know [62:57] Reiner Pope: you care about every single bit.

[0:01] Reiner Pope is the co-founder and CEO of MatX. [0:03] He's a former math whiz and Haskell programmer who became [0:06] a TPU architect for Google. [0:07] Now he's teamed up with Google's former chief chip architect to design [0:10] a better chip for AI. [0:16] A year ago, everyone was saying Google is canceled. [0:21] AI is going to eat their search. [0:22] No one's going to search for things, and therefore, the business won't do well. [0:26] Obviously, that sentiment has really shifted, in part helped by [0:30] Gemini 3 is really good. [0:32] Then also it's really fast. [0:34] It's powered by the custom chip hardware Google has. [0:37] You were inside Google for a lot of the foundational period, [0:42] laying the groundwork for that stuff. [0:44] What do people not appreciate about what Google did right [0:49] to lay all the groundwork for their current AI success? [0:52] They started with the research. [0:54] The transformers came from there. [0:55] Pretty much anyone who's over 30 and at a large lab has [0:59] been at Google Brain at some point. [1:02] I think there was and has been a lot of talent there. [1:06] TPUs are pretty good. [1:11] We think there's better you can do, of course, but they at least had [1:14] the option, the opportunity to design the TPUs for neural nets at least, rather [1:19] than graphics applications like NVIDIA. [1:23] The overall architecture, starting with single core doing [1:27] what was at the time reasonably large systolic arrays, by today's standards, [1:30] nowhere near as much, but I think those were a lot of really good decisions. [1:35] When did the TPU project start? [1:37] TPU v1 was announced in 2016, I think. [1:41] That was what led to the creation of all of those 2016–2017 startups. [1:46] Cerebras, Groq, Graphcore, SambaNova, all of those. [1:50] TPU v1 is a really impressive project. [1:53] It was done on a very short timeline. [1:56] I don't know the full details, but maybe about a year or so, [1:58] maybe a year and a half, with a skeleton team of 20-30 people. [2:03] Really, really minimal viable product. [2:06] More recent TPUs and more recent AI chips in general [2:10] can't do that because the market has moved, and the table [2:15] stakes are much higher. [2:17] The first-generation product, they just… [2:20] One big systolic array, stick a memory next to it, we're done. [2:24] It was really simple, nice, elegant product. [2:26] Obviously, TPU v1 predates the Transformer. [2:29] Is that just a coincidence that they happened at very similar times, or [2:35] are they related in some way? There was a period of maybe about four [2:39] years of a lot of ML research, or neural [2:46] net research, prior to Transformer. [2:49] What was popular? [2:50] LSTMs and ConvNets and ResNet and Inception. [2:54] The big thinking at the time was to adapt it to be used for LSTMs. [3:02] It's a reasonable fit there. [3:04] But I think there was just a huge flurry of activity. [3:09] Why did it all happen then and not later? [3:10] It's probably just because people stopped publishing. [3:13] 2022 was about the time when Google completely stopped [3:17] publishing its research. [3:20] All the good papers are from before that as a result. [3:22] But is there some hand-wavy story you can tell about parallelization where both [3:26] Transformers and TPUs are [3:31] about really internalizing [3:34] the importance of parallelization? [3:37] Definitely, I put it somewhat on people, actually. [3:42] It is just true. Hardware is massively parallel. [3:45] You've got tens of billions, hundreds of billions of transistors [3:49] on your chip, and it takes maybe 100 clock cycles to get from one [3:53] side of the chip to the other. [3:55] You can't do a sequential computation involving transistors [3:58] on both sides of the chip. [3:59] The hardware is just fundamentally parallel, and you have to [4:02] take advantage of that. [4:03] TPU v1 and all later TPUs naturally took advantage of that. [4:08] Matrix Multiply is really nice because it is so parallel. [4:12] I think on the hardware side that's generally understood. [4:15] I think most ML researchers, especially of the time, [4:21] were not super deep in what hardware wants, and what is… [4:25] Mechanical sympathy is sometimes a term that's used for that. [4:29] What is the term? It kind of— It speaks for itself. [4:34] Think about the poor machine and what does it want? [4:37] What does it want? [4:39] The term actually, I think, originates in maybe high-frequency trading [4:44] in areas like that, which is, I haven't worked in. [4:46] I've just, I like reading about the software [4:48] that people have built from there. [4:51] It's like, for them, what does the machine want? [4:52] It wants a lot of instruction-level parallelism. [4:54] This is CPUs, not TPUs. [4:56] Once a lot don't branch, so unpredictable [5:02] branches kill your performance. [5:04] Think about the things that CPUs do and how to use them best. [5:07] Can I get to peak performance on a CPU? [5:09] It's that idea. [5:10] I think the whole idea of peak performance on a CPU is kind of crazy. [5:15] No one even says, "What is peak performance? [5:17] What is my percentage of peak on a CPU?" Because the performance [5:21] of software running on CPUs is really bad, but running on GPUs, or TPUs, [5:25] or AI chips in general, actually that is the main focus. [5:27] It's like, "What is my percentage of peak? [5:28] Can I get 70% or 80%?" I feel like many people listening to this [5:33] know that GPUs perform better for AI workloads than CPUs. [5:38] It's a funny history when you think about it, where just [5:42] one day we woke up with all these very mathematically intensive workloads, [5:47] first crypto mining, and then AI, so then NVIDIA is extremely [5:52] well-positioned because they've been making GPUs for gamers [5:56] that you would plug into... [5:57] You'd buy your Dell PC back in the day and maybe upgrade [6:02] the graphics card by plugging in a better NVIDIA graphics card than the one [6:07] the stock Dell computer came with. [6:10] They were incredibly well-positioned to capture that. [6:13] I think people know that. [6:14] What is the intuitive explanation as to why GPUs are better [6:19] for AI workloads than CPUs? [6:22] People say, "They're better for these mathematical computations." But that's [6:26] a tautological answer, basically. [6:28] Is there some way you can have a mental model for why that is the case? [6:31] Because software instruction sets also involve doing math. [6:38] Intuitions, I'm not sure. [6:39] Let me try and just go to some of the big differences, which is, [6:45] really wide vector instructions is the hallmark of a GPU, [6:51] which I think it's maybe… [6:54] If you want some intuition, it's like how much is spent on controlling the thing? [6:59] Maybe control means, if I'm driving a truck, how much… [7:02] Is the driver versus the payload? [7:04] A truck has a huge payload in it. [7:06] That's more like the GPU, whereas maybe a motorcycle is more [7:09] like the CPU, where you've got… [7:11] Actually just processing the instructions, reading, "What do I have to do next? [7:16] How do I do that?" That is most of the cost on a CPU, whereas [7:21] if you just keep the same instructions but make the payload 100 times bigger, [7:24] then you can shift most of the cost to be in the actual work that you want to do. [7:31] CPUs have been optimized for very complex instruction sets, whereas [7:37] GPU is optimized for— Complex instruction sets and [7:42] fine-grained changing what you want to do. [7:46] Like steering, in this analogy, a CPU can steer an obstacle course, [7:49] no problem, whereas, on a GPU, you're just going to go [7:52] straight line for a really long time. [7:55] This is getting us into... [7:58] What is MatX? [7:59] How did you guys start it, and which part of this [8:01] space are you attacking? [8:02] MatX is making the best chips physically possible for LLMs. [8:13] What led us into MatX… [8:16] Mike is the other founder. [8:18] Mike and I were both working at Google. [8:22] I was working on the inference stack for running LLMs, and I was saying, [8:28] "How can we make the best software on TPUs for running LLMs?" Then what we really [8:34] wanted out of hardware was support much, much larger matrices. [8:37] The matrices have grown from maybe 128 in dimension into the many thousands. [8:41] Truck goes to many trailers. [8:48] Much larger matrices and much lower precision arithmetic. [8:55] We tried to move the TPUs in this direction. [8:58] TPUs have been moving in this direction, but they're constrained [9:00] by a lot of other workloads. [9:03] There was a big ads workload at the time. [9:06] Back in '22, before ChatGPT was released, there was this idea that LLMs were going [9:12] to be a big thing, but not conviction, and really hard to make a big bet on that. [9:19] I think a startup is more of the right place to make a big bet on a workload. [9:23] If you fail, it's fine. [9:25] Another startup will succeed. [9:26] Whereas I think a company like Google or NVIDIA, the next chip [9:31] has to work for sure. [9:34] You can take more technical risks as they turn up. [9:37] Actually, I would say we were taking product risks rather than technical risks. [9:41] But is there actually product risk? [9:42] Because it seems like LLMs are going to work. [9:45] I think now we understand it. [9:47] Two or three years ago, I think it was just like a— And when [9:51] you say the best chips for LLMs... [9:52] I can think of multiple ways to measure best. [9:54] It could be the best performance per watt. [9:56] It could be the lowest latency, capable of handling the largest models. [9:59] What is best? [9:59] In general, there are two metrics which LLM workloads care about, which is [10:05] throughput, which is really just an economics thing. [10:08] I buy a chip for $30,000, and then can I do 10,000 tokens a second [10:13] or 100,000 tokens per second of throughput? [10:15] That determines the dollars per token. [10:18] Throughput and then latency, how fast does a thing respond? [10:21] As I see the market, the economics seems to be most important. [10:28] Ultimately, the quality of the AI you can train [10:31] and serve is constrained by, "I have only a $10 billion budget, [10:35] and I want to train and serve the best model I can on that budget." If I can have [10:41] more tokens per dollar, then I can get a better quality out. [10:47] The product we aim to build is far ahead on throughput, [10:52] but then, actually, the surprising thing is we're competitive [10:56] with the best on latency as well. [10:58] I think that is a unique thing in offering both in the same place. [11:01] Is this for… [11:01] Obviously in AI, there's training the models and then [11:03] running the models' inference. [11:05] Is this most interesting for inference, or is there any training angle? [11:09] Incidentally, is it useful for trading, but you are trying to win inference, [11:14] is that how you think about it? [11:15] I think that's a reasonable way to look at it. [11:17] I think the best inference chip today will be a really good training chip as well. [11:24] Our product is both training and inference, but I think the first [11:27] sales will be an inference. [11:28] That's mostly just a market effect where it's easier to buy, [11:33] it's not as big of a risk to go to buy an inference cluster as a training cluster. [11:38] I think the product is really compelling for training as well. [11:40] I think it should be the best training product. [11:43] You guys just raised a big new round of financing. [11:46] That's right. [11:49] We've raised a series B round, it's led by Jane Street and Situational Awareness. [11:56] Situational Awareness, that is, Leopold Aschenbrenner's fund. [11:59] He wrote the definitive book on AGI and where it's going. [12:04] Then Jane Street, they're real technical experts. [12:07] They understand all the details really well. [12:09] Very happy to be having them lead the round. [12:13] It's a $500 million round, helps us actually ramp the manufacturing and supply [12:18] chain for our chip so we can bring our chip to market. [12:21] That's a lot of money. It is. [12:24] I think roughly, I would say it costs a ballpark [12:28] $100 million to produce a chip in small volumes. [12:32] But then you see the orders that are going around, like OpenAI, Anthropic, Google. [12:37] They're going around buying multi-gigawatt clusters, they cost [12:41] tens of billions of dollars of chips, and you want to deploy all [12:44] of that in a year or so. [12:46] You just need a massive supply chain behind you. [12:50] Assuming everything works technically, what rate of production [12:55] could you start to see? [12:57] We have some estimates of where we would like to be on this. [13:02] Ramping to very large volumes is a huge challenge for anyone. [13:06] Obviously, for the large players, they've had some practice at it. [13:08] Getting to a very large volume for a startup is hard. [13:12] We would like to be at a place where we're shipping multiple gigawatts a year. [13:16] Multiple gigawatts per year? [13:18] Speaking of the metrics, you talked about tokens per second. [13:18] Yeah. [13:21] We used to measure chips in FLOPS, and I guess there's some kind [13:24] of custom FLOP thing for AI chips. [13:26] But is everyone just using tokens per second these days? [13:28] Is the industry aligning on that as the chip metric? [13:32] I guess it's like an application metric versus the chip itself. [13:37] FLOPS of the chip is the key chip metric. [13:42] There's a little bit of, if I go and say, [13:44] "I've got an exaFLOP chip," to you then the appropriate suspicion is to say, [13:50] "But can I actually use those FLOPS effectively?" Then you need [13:54] to map the application to that. [13:56] This is kind of telling you the usable FLOPS for your purposes. [14:00] As a consumer of AI, we have known for a long time [14:05] that lower latency products succeed. [14:10] Google talked about their internal testing where the differences were down to… [14:17] Was it 50 milliseconds? Something like that. [14:19] In result times where they noticed more Google engagement, the faster the results [14:23] were, and you'd think that 50 milliseconds is imperceptible to a human. [14:27] It almost is, but turns out it's not. [14:30] I think Amazon has, certainly, they've optimized the latency [14:33] of the Amazon experience quite a lot. [14:35] I don't know if they've talked about this stuff publicly, but you know that their [14:39] internal metrics similarly show that the faster the product page loads, [14:42] the more people buy it. Yet in AI, Google has carved out a [14:48] meaningful advantage via Gemini just being [14:53] really fast for its level of intelligence. [14:57] As far as I can tell, ahead of most of the other labs [15:01] at a latency, at a fixed high level of intelligence. [15:06] Why have you guys or Groq or better chips not been adopted faster [15:12] to give this product latency? [15:14] It's just that this will happen and you guys will be powering all the AI products, [15:18] but I note that Google has an interesting lead there. [15:22] I think there's ultimately… [15:24] At least for existing chips in the market, there's a really uncomfortable trade-off [15:27] between latency and throughput. [15:29] The chips that are best at throughput have historically been the chips [15:32] that are based on HBM as the memory. [15:35] That is Google, Amazon, NVIDIA. [15:39] In order to have very large throughput, you need a lot of inferences [15:44] in flight simultaneously. [15:45] That needs the large memory. [15:47] But that hasn't been so good at latency. [15:48] Then, there's the Groq and Cerebras that are much better at latency because [15:51] they've got this, the SRAM, weights are in SRAM, very low latency. [15:56] The problem is, and the challenge when you go to a Groq or Cerebras system [15:59] is that the throughput you get there, it just is not very good. [16:04] The fundamental dollars per token is just not competitive with Google [16:09] or NVIDIA or Amazon. [16:11] It is actually possible to do both in the same chip. [16:15] It's kind of an obvious thing. [16:16] You say you take the HBM, you take the SRAM, [16:17] put them together on the same chip, you put the weights in SRAM, and you put [16:21] all of the inference data in HBM. [16:24] That is what we are doing, in fact. [16:26] I think that actually hits a really nice sweet spot where you can get low [16:29] latency and also be very cheap. [16:32] I think that's a really attractive point to be. [16:33] It hasn't happened in the market yet, just because of product decisions that [16:36] have been made by the different chips. [16:38] But we should expect all the AIs we're using to get [16:43] significantly faster over the coming 3–5 years? [16:45] Order of magnitude faster, I'd say. [16:47] Generally, HBM-based chips tend to be about 10 milliseconds or 20 milliseconds [16:52] per— I'm sorry, HBM-based chips are things like TPUs? [16:55] That's right. [16:57] There's just some simple math of, how long does it take you [17:00] to read through all of HBM? It takes about 20 milliseconds. [17:02] That's the amount of time per token it runs, whereas the amount of time to read [17:06] through all of SRAM is much faster. [17:08] You get about 1 millisecond, so they are managed pretty faster. [17:11] Famously, software used to be… [17:15] Old-fashioned deterministic software, the kind that's now out of favor, [17:20] used to be very easy and quick to scale. [17:23] You had a social networks that have some Southwest, Northwest moment, [17:26] and they can scale through 10, 100, 1,000 orders of magnitude of adding users [17:32] because it's just a few rows in a database, and it's [17:35] a very underutilized CPU. [17:37] What's interesting about the AI world [17:39] is there are very real bottlenecks. [17:45] Elon spent lots of time talking about power, but it's not just [17:48] bringing power online. You mentioned HBM is reminding me of… [17:51] It seems like there's a view that maybe there's going to be some HBM [17:56] supply chain crunch. [17:59] Where do you see… [18:01] Are we in for just a crunched world where some limiter is [18:07] pacing the rate of AI buildout over the coming few years, where [18:11] the economics work, and the products and everything like that, [18:14] but ultimately, we just can't bring the components online [18:17] fast enough because we have to build out the factories, things like that. [18:20] What are those crunched components? [18:24] I think so. [18:24] I'll just comment, by the way, this is a great time to be a supplier in this place. [18:29] Or just really— You should have started an HBM company. [18:31] I know. [18:33] I think it's also just a fun time to be someone who optimizes software. [18:36] That's always what I like doing. [18:38] Always the challenge is, "Why am I optimizing this if no one [18:41] cares?" But finally, there's a place where you can… [18:46] It's actually very meaningful in a very tangible sense. [18:49] If I can make this 20% more efficient, then it can save that 20% of the buildout. [18:54] The supply chain, we're going to have crunches [18:57] on all of the supply chain really. [18:59] If you look at the big components of what any company, like us for example, [19:05] build out, there is dependency on logic ties [19:08] from typically TSMC, maybe Samsung, or HBM, which are the big three HBM vendors. [19:13] Hynix, Samsung, and Micron. [19:17] Then there's also just the whole rack manufacturing, which includes [19:22] literally just sheet metal and so on that builds the rack, [19:24] but also cables and connectors because of all the high-speed interconnect. [19:29] That's what we—Racks don't sound hard. [19:31] Are they sneaky hard? [19:33] The big challenge is that you want to bring in a huge amount of power, [19:37] get a huge amount of heat out, and also have phenomenal interconnect, [19:41] which has very high signal integrity requirements. [19:44] Pack a lot of cables in with… [19:46] The cables don't bend too much, they have to have enough copper in them, [19:49] and so on, that you don't lose data rate on the interconnect. [19:54] If you push it to a limit, it's sound. [19:56] Wafers, racks, HBM. What else? [19:59] Data centers, which I think is power, primarily a little bit of buildout, [20:03] but primarily power and good infrastructure there. [20:08] How do you then, as a startup that is looking to acquire [20:12] all these components, elbow your way in amongst the giants [20:17] of the Googles and the NVIDIAs and all these people who [20:21] have long-running relationships and have been buying for much longer? [20:27] Ultimately what all of these suppliers care about, they do somewhat care about [20:31] a diversity of their own customers. [20:34] It's not a great position to be in. [20:35] They don't want monotony. That's right, yeah. [20:39] But then, what is their hesitation? [20:43] Or the calculus for one of these large suppliers is, [20:47] if I reserve some of my capacity for you, a startup, are you going [20:51] to be around in a year? Is anyone going to even buy your product? [20:53] Our approach has been to just actually find buyers for the product, [20:59] and then the buyers answer that question, ultimately. [21:04] If you show up with a bunch of fairly ironclad contracts to a supplier, [21:08] then that has helped. [21:09] That's the nature of it, yeah. [21:11] I presume also the round you just raised really helps [21:15] there, where showing that you are incredibly well-capitalized and not going [21:20] anywhere also helps from a supplier validation point of view. [21:25] Absolutely. [21:26] It helps just to say that we are around. [21:28] We, in some cases, are actually… [21:31] It depends on which part of the supply chain, but some parts of the supply chain, [21:35] some are fungible. Logic ties are typically pretty fungible. [21:38] But other parts of manufacturing, you actually need something [21:42] specifically set up for you. [21:44] We're also able to cover the capital costs of that. [21:47] That makes sense. [21:48] Coming back to the MatX architecture. [21:51] You want to build the best chip for LLMs. What is that? [21:54] Yeah, exactly. Sounds great. [21:58] There's a few aspects to that. [22:00] I think the first one is pick your memory system right. [22:05] I said, we've seen this HBM family, we've got the SRAM family, [22:08] put them both together is actually, the most obvious idea, [22:12] but you can actually do it. [22:14] There are a lot of details to make that work well. [22:16] We've done that work. [22:17] One of the things that shows up there is you've spent all of this [22:20] area on your chip on SRAM. [22:22] How do you fit in the matrix multipliers, which are the other big thing you really [22:25] need to do, and somehow create a much more [22:28] efficient matrix multiply engine? [22:31] There is a gold standard for that. [22:32] That is called the systolic array. [22:33] Make a really large systolic array. [22:35] You can't beat that in area or power efficiency. [22:37] Provably so? Practically? [22:39] Practically. It has not known a better approach there. [22:43] The main thing is, where are the inefficiencies typically? [22:45] The inefficiencies show up when you leave the systolic array. [22:49] If you make a systolic array really big, then you just don't leave it as often. [22:54] That's the idea. [22:56] Make a really big systolic array. [22:57] That is sort of the theme of several of the 2023-era startups, including us. [23:06] But one of the challenges there is, now, there is this part of the neural network, [23:10] as part of the Transformer, which is this attention that doesn't [23:13] map well onto a large systolic array. That's the tension. [23:18] The mixture of expert layer maps really well, but the attention does not. [23:23] What we came up with, which is quite different than some [23:25] of the other startups in this space, is say, take a really large historical [23:29] array, but have a way to split it up into pieces without losing efficiency. [23:34] That is the core of the design for us. [23:37] Then, the third component. [23:39] First was HBM and SRAM, second is the systolic array, [23:42] third component is an interesting new approach on low-precision arithmetic. [23:48] Low-precision arithmetic, in general, we've seen number formats [23:51] get narrower and narrower. [23:53] They get faster and faster as you make them less precise. [23:57] Number formats get narrower. What does that mean? [24:00] Float32 was how people used to train neural nets. [24:04] That's just too much precision. It's wasteful. [24:06] Too much precision, yeah. [24:07] It's like saying, "I've got an image with a billion [24:12] color bit depth." It's too many colors. [24:15] You'd rather have more pixels and fewer colors. [24:18] That trend seems to go almost all the way down to one bit even, [24:23] where just have very few colors but a huge number of pixels. [24:30] In net seems to be a better, just more efficient way to train models. [24:35] Sorry, literally what precision are you dealing with in this sense? [24:42] We have a range. [24:48] We actually have an ML team who we hired specifically to research [24:52] different forms of numerics and how to make them all work together really well. [24:58] We have a range of precisions. [24:59] It's not just one precision. [25:01] We think probably the main thing will be similar to where NVIDIA is at, [25:04] which is 4-bit precision. [25:06] But I think a mix of different precisions is useful for just when you look [25:10] at the research, sometimes you want some layers in higher precision or [25:12] lower precision, and so on. 4-bit is 16. [25:15] Yeah, you get 16 choices. That's it. [25:17] That's it. It's pretty imprecise. [25:21] That's really interesting. [25:21] I didn't know about that dynamic, but it makes sense. [25:24] Half of them are positive, half of them are negative. [25:26] It's even— How do you design a chip? [25:33] Is that a whiteboard? [25:34] What software are you working in? I'd just love to know… [25:37] I understand how you design software and what that process looks like. [25:40] I have actually no sense for what chip design looks like. [25:43] The way that you actually type a chip into a computer is similar to software. [25:47] You write Verilog. [25:49] Verilog is a programming language. [25:50] It is a very parallel programming language, which makes it different [25:53] than C or Python or something. [25:56] But it is a programming language. [25:59] The mechanics of how you express the design are the same as software, [26:02] and we have continuous integration, Git, all of those things. [26:05] But a program executes... [26:07] Like your Verilog program... [26:09] We don't really run it. [26:11] Exactly. How does it run? [26:13] We synthesize it. [26:15] Synopsys and Cadence provide EDA tools. EDA? [26:18] You have to remember, I'm just— I don't even know what it means, really. [26:24] I think it's Electronic Design Automation. [26:27] It takes the Verilog and says… [26:30] First turns it into a description of what are the logic gates that are involved, [26:33] ANDs, ORs, NOTs, and then the wires between them. [26:37] Then it runs for days doing some really difficult algorithms, [26:42] and then eventually produces… [26:46] Gates are the first thing, and then even below that, [26:48] it literally just produces polygons. [26:50] It says like, P-type semiconductor here, N-type [26:53] semiconductor here, and polysilicon. [26:57] You write Verilog, and then that compiles down into gates [27:02] and ultimately the Minecraft 3D. [27:07] "This is where your elements should go." [27:11] But then, what is the iteration loop? [27:16] When we write code at Stripe, we build a first version of something, [27:20] and then we try it out, and then we refine it, and we add more [27:25] functionality over time. [27:26] We're going to write some tests at some point. [27:27] We'll ship that, we'll find product market fit, and then we'll refine it in market. [27:32] Do you just sit down and write the completed chip, and it works really well? [27:35] Every year we tape-out a chip, and if there's a bug, [27:37] we just wait till next year. [27:38] It's not really how we do it. What's the iterative process? [27:41] How do we actually do it? It's much more waterfall than software is. [27:44] Waterfall is almost a bad word in software development. [27:47] But it's a fact of life in general. [27:51] The waterfall goes from architects to logic designers who are writing Verilog. [27:57] Then, there's this design verification, and then physical design. [28:02] There's this really big architecture phase which happens before even writing any [28:05] Verilog, which is, "What do I want the organization [28:10] of my chip to be?" In some sense… [28:13] I came to hardware after doing almost 10 years in software. [28:18] I really like the blank slate you get in hardware. [28:21] You've got all of the raw materials, you have a much more [28:24] varied in what you have available. [28:27] What is the organization of your chip? [28:28] Do I have 100 cores? Do I have one core? [28:31] Do I have systolic arrays? Do I have vector units? [28:33] All of those things. [28:36] Then we spend a long time coming up with that general principle [28:39] and then saying, "Okay, now I've got these applications I want to run. [28:42] I want to run a transformer of a particular shape. [28:44] I want to map that onto this architecture that I've got in my head." [28:48] We do a lot of iteration. [28:50] Well, I've got this architecture in my head. [28:51] I write it down to communicate to other people, but that's just [28:53] like a markdown file. [28:57] Then, still actually a lot in my head, but maybe with Python simulation and so [29:01] on, I'll see, do my applications map well to it? [29:05] Can I run LLMs? [29:07] You have a simulator where you write your chip, you can then simulate its [29:12] performance, and you have some battery of tests [29:15] that you see how this chip design works. [29:18] Is it like an industry standard... [29:22] Is it the X-Plane of chip testing? [29:28] There's an industry standard thing for the Verilog once you've done the design. [29:32] They're just Verilog simulators that you can test against. [29:37] But you've already invested a huge amount of work by the time you've got [29:39] to that point, and so you sure hope you haven't made a big mistake at that point. [29:44] The thing that everyone does prior to that is, [29:49] we'll write our own performance simulator, which, I mean, [29:53] it is very specific to your particular architecture, and you can write it quite [29:56] concisely in just a normal programming language. [29:59] That is where most of the architecture work is done. [30:01] Then the simulation on Verilog is more, "I know what I'm doing. [30:03] I just want to make sure I didn't have any bugs when I implemented it." But I presume [30:07] it's a game of inches where different people are trying different [30:10] things, and then you... [30:13] Do you simulate it to see if it runs 1% better across the battery of tests? [30:18] Or is that not how it works? [30:20] In this space, not so much. [30:22] Just to characterize what performance of an AI chip is, it is how many, really… [30:29] First thing you care about is FLOPS. [30:31] How many FLOPS have I got? [30:32] That's a product of how many multiplies, like, "I've got a grid of a certain size, [30:37] 1000 by 1000." Can do a million multiplies in a clock cycle. [30:41] Then I have a certain clock frequency, a gigahertz. [30:43] I multiply them out. That is the speed of it. [30:47] I don't even need to write that and test it to see how fast it is. [30:50] It just is. [30:52] What I plan in advance is it's going to be this fast. [30:55] What I can then optimize on, maybe a little bit, is clock speed. [30:58] There's not a lot I can do there. [31:00] Then, I can optimize a bit on area as well. [31:03] There is some room for optimization, but actually a lot of it gets set. [31:06] Actually, just the speed of the chip gets set very much upfront. [31:11] Then how many chips do you fab? [31:13] Is it only the ones going into production, or is it just build a few to throw away, [31:19] or how does it work? [31:21] The ideal, which companies tend to hit about 50% of the time, [31:24] is that your first tape-out… [31:27] Tape-out costs $30 million. [31:28] Your first— Tape-out is just production run? [31:30] That's right. [31:33] The actual manual, the first chip costs $30 million, the second chip costs $1,000. [31:37] Yes. [31:38] Tape-out is that first chip. Okay. [31:40] The ideal is that your first tape-out is actually your production thing. [31:44] You do a tape-out, you make maybe a thousand chips and test them, [31:48] and then you do production volume. [31:50] In the unlucky 50% of the time, you need to redo some or all of your tape-out. [31:58] In good cases, and in many cases, you can redo just the metal layers [32:02] which costs you only like $100,000. [32:05] As opposed to the— Pay the $30 million again. [32:09] But in bad cases, if you've made something serious and you [32:12] can't fix it at the metal layers, you have to do the whole thing again. [32:15] Why can't that be solved? [32:18] Is that definitionally an error in simulation, where it turns out these [32:21] two gates were too close together, and it just led to some reliability issues? [32:30] Yeah. [32:30] What you're describing is physical, the physical implementation [32:35] of the chip is wrong. That's one class. [32:38] The other class is that the logical specification of the chip is wrong. [32:41] But shouldn't that be— Shouldn't you have caught that before? [32:44] Yeah. [32:44] Before you spent $30 million on manufacturing it. [32:46] Yeah. [32:49] We do a lot of testing. We try not to ship these things. [32:53] I hear software companies also ship bugs to production as well. [32:56] Fair. [32:57] Sometimes things— It's a very good retort. [33:00] Shouldn't you not be shipping bugs? [33:03] But there is a real trade-off in, you can spend more and more [33:06] time on design verification. [33:10] There's always this question of, when do you stop? [33:14] You stop when your coverage metrics ever hit a certain point, but maybe not 100%. [33:19] Then if [33:22] Apple has to discretize the iPhone release [33:27] cycle, and they have settled on once per year, they'll decide, [33:31] "We've got this better camera, but it's got to wait for the next [33:34] version," or, "We're going to improve the waterproofing, but that's got to wait [33:38] for the iPhone 8 or whatever." They have taken a continuous process [33:42] of always coming up with ways to make the iPhone better and discretized [33:46] it into annual iPhone releases. [33:48] What will your discrete cadence be? What's our vision of that? [33:52] Yeah. [33:52] Many chip vendors have this sort of tick-tock model, which is [33:56] you'll do on one generation, maybe you're trying to release every year. [34:01] On even numbered years, you'll do a physical technology upgrade, [34:05] so new transistor technology, new memory technology, and you interconnect. [34:09] Then on odd numbered years, you might do an architecture overhaul. [34:12] I think that's a pretty good fit because you have different parts of your company [34:16] that are skilled at different areas, and it allows you to keep both of them [34:19] occupied without having instead every 2 years doing a massive risk release. [34:25] Yeah. [34:26] So you think that's probably likely for you? [34:28] That's right. [34:29] You mentioned interconnect. [34:31] There's a narrative out there that in NVIDIA, [34:34] a huge part of the defensibility comes not from the chips, which are good, [34:37] but from the software layer, and the ability for engineers to write [34:42] these really parallel workloads, and the fact that they've been refining [34:46] CUDA for whatever number of years. A decade or so. [34:49] Exactly, a long time. [34:52] How do you think about parallelization, and is that narrative true? [34:56] It's true for sure. [34:58] It's true in many areas of the market. [35:03] I think, and especially where you look at where [35:06] NVIDIA entered the market, they're doing [35:11] PC devices, lots of gaming, and so on. [35:15] There are thousands of games, maybe tens of thousands of games released, [35:19] and they all need to be programmed against CUDA, and so [35:23] there's such a huge investment in the software that this is really [35:27] important, the compatibility. [35:28] There are not thousands of LLMs. [35:30] There's one LLM per frontier lab, and there's maybe five frontier [35:33] labs or something like that. [35:37] Just the economics of that is different. [35:39] The calculation for a frontier lab roughly goes as, I just bought a $10 billion [35:45] compute cluster, I have hired 50 of the best [35:50] people who can write optimized GPU, or TPU, or Trainium software. [35:56] I pay them less than $10 billion, a lot less. [36:01] Let's put them to work optimizing the compute. [36:07] Good work there, depends on what your baseline is, [36:10] but it can very easily double the performance of the software you write. [36:15] There is a huge amount of custom software written for every generation of chip. [36:19] When a new chip comes out, software is substantially rewritten [36:22] to optimize for that specific chip. [36:24] That's just the right trade-off given the relative costs of these things. [36:28] What that means for us is that ecosystem already exists, [36:33] and that way of operating, where you say, "I'm just going to staff a 50-person team [36:39] to write software for this chip," works really well if you're [36:42] trying to sell to frontier labs. [36:44] You're saying CUDA is way more important for the games environment, [36:49] where it just does a lot of games than this top-heavy AI market that we're [36:55] in, where if people say, "You need to then customize [37:02] your workload for a MatX chip," it's like, "Well, fine, do that." Cost of business. [37:07] Yeah, that makes a lot of sense. [37:11] Where will you fab the chips? TSMC. [37:16] Why is TSMC so durable? [37:22] It's interesting. [37:23] They don't charge a lot as well. [37:24] You'd think that if they're a monopoly provider, they should [37:26] charge a lot of money. They don't. [37:29] I think that is a big aspect of why they're so durable. [37:32] It's like this cyclical conservatism [37:36] crossed with Taiwanese business conservatism, means you're at the most [37:41] conservative part of the matrix. [37:43] But I mean, it does… [37:48] I mean, an American capitalist might say, "Well, they're just screwing up. [37:51] They could have extracted more money from the market." But you could also say [37:54] that this is actually the long-term sustaining advantage because they will [37:59] just stay ahead for a really long time. [38:01] They don't incur the creation of competitors. [38:03] But isn't the creation of competitors priced in because [38:06] of the geopolitical risk? [38:10] It's not like everyone's fat, dumb, and happy with their TSMC dependence. [38:13] They're actually thinking a lot about it. [38:15] So there is real technical advantage there as well. [38:17] It's not just the discouragement. [38:19] But designing chips seems really hard, building airplanes seems really hard. [38:23] There are so many areas where competitive market forces create multiple options. [38:31] Yet, that has not occurred here. [38:34] There are multiple options. [38:35] You can buy from Intel or Samsung. [38:37] But at leading-edge nodes. [38:40] What do we even care about in leading-edge nodes, I guess? [38:42] The big advantage is on power. [38:44] The advantage on area is smaller. [38:46] The leading edge nodes, the density doesn't go up as much as it used to. [38:52] When you are really, really sensitive to power, it is a good [38:54] idea to be on leading edge nodes. [38:56] That is AI chips and mobile phone chips. [38:59] But there's a lot of the market where you don't actually like [39:02] the devices in your car. Sure. [39:04] Car chips, yeah, that's fine. [39:05] But you're saying, if you exclude the two most interesting [39:09] parts of the market— That's true. [39:16] For this super high-growth area of the market, it's interesting to me, [39:17] again, there's a lot of other really complex business problems out [39:22] there that competition has solved. Chip design is a… [39:25] Like, why has someone not left TSMC and gone and built a new fab? [39:29] I don't know. [39:32] The cost of a fab is extremely expensive. [39:35] I recognize that also the cost of a lab is extremely expensive, too. [39:41] I don't really understand the technical details of why it's so hard. [39:45] I mean, there is some amount of just a $10 billion fab versus $100 million [39:51] tape-out and chip development. [39:52] There's a huge difference there. [39:53] But beyond that, I'm not sure. [39:55] What's TSMC like to deal with? They're very big. [39:58] As a startup, we tend to work with, not directly with TSMC, but with [40:04] an ASIC vendor who, firstly, does a huge amount of the actual [40:07] backend work for us to interface with them, but then also has their [40:10] existing relationships with them. [40:13] TSMC cares a lot about diversity of their customer pool. [40:18] It gets back to that conservatism. Yeah. [40:21] They're great to work with from that perspective. [40:24] They want to encourage startups. That's right. [40:26] That's very cool. Why don't the labs design their own chips? [40:29] Google does. Google does. [40:31] OpenAI is starting. [40:32] It's really a trade-off of how much advantage do you get from vertical [40:35] integration versus how much advantage do you get by concentration of R&D work. [40:41] You take the five labs, and if they all buy from one player, [40:43] then you can put five times as much R&D into that chip. [40:48] Does that beat the advantage you get from saying, "I know exactly what my model is"? [40:52] Because of the several-years delay from designing a chip to being [40:57] in production, you can't actually say, [40:59] "I know exactly what my model is," because models change much faster than that. [41:06] Even the labs are forced into this position where they have to make [41:09] predictions, and they have to hedge against what they might [41:11] do two years from now. [41:12] The calculus is, what is the probability distribution of what my model might look [41:16] like and then design a chip that gets 90% of that [41:20] probability distribution or something? [41:23] Elon is excited about data centers in space. [41:27] The two criticisms I've heard are that cooling is very hard and then [41:34] repairing the chips is hard. [41:37] But I know nothing about chips. You do. [41:41] The repair is really interesting. [41:43] When you look at how NVIDIA deploys their racks, how we do something pretty [41:47] similar to what NVIDIA does. [41:50] In general, you always need to design for the fact that some of your [41:52] chips are going to be down. [41:53] Mean time between failure of chips is not that large. [41:58] In a cluster of 100,000 chips, there are going to be chips [42:01] that are down all the time. [42:02] One way you can do that is you can make a rack where one rack has spare chips in it. [42:07] NVIDIA has eight spare chips in a rack of 64. [42:13] That's pretty good. [42:15] The combinatorics works really well for you there. [42:20] Because you can pick which ones to avoid, you can with very high probability [42:27] tolerate a lot of failures. [42:27] And then the other just for the other family of things is to say my rack has to [42:31] work, but I have some spare racks as well. [42:35] You can math that out with the tax of reliability here is [42:38] only like 10%, that's pretty good. [42:41] But that relies on someone coming and servicing the part within [42:45] a day or something like that. [42:46] If you say they're going to service it never, then [42:50] I think you actually can get where you want to be, but maybe with 100% [42:53] tax on reliability rather than 10%. [42:55] For example, if you think the average lifetime of a chip is in the range [42:59] of three to five years. [43:01] That means if I deploy twice as many chips, then three to five years from now, [43:05] half of them will still work. [43:06] Also, the burn-in is particularly failure-y. [43:10] How about the cooling? Most of the challenge... [43:14] I guess this is actually really a data center design aspect. [43:18] At the rack level, the challenge of cooling is just getting [43:20] the heat out as quickly as possible out of the rack into the cooling network. [43:26] How you get it out of the spaceship, other people would know [43:30] that better than I do. [43:33] That seems to be the main objection, but I don't know. [43:37] I think it's if you think the cost of repair is [43:40] that you need to have deployed twice as many chips, then it's a trade-off [43:43] of the capital of the chips versus the power saving. [43:46] Exactly, the repair thing, it feels like, can be solved, because also, [43:51] I think part of the bet, part of Elon's claim, [43:54] is that we will just be so power-limited that you have no option [43:58] but to go to space. [44:00] People can argue about that, but were that to be the case, then yes, [44:05] it's like you can get power in space, and you cannot on Earth, [44:08] so you might as well go there. [44:10] Whereas like the cooling is a more fundamental, "Does the product actually [44:13] work at all?" Reiner thinks about AI [44:19] the unglamorous way: compute, [44:20] systems architecture, and what it takes to run models reliably at scale. [44:24] If you're building an AI product, the business model similarly has [44:27] a ton of unglamorous complexity. [44:29] You're not just selling AI, you're monetizing consumption across [44:32] API calls, tokens processed, GPU hours. [44:36] Stripe Billing is a scalable system for usage-based billing. [44:39] It lets you launch token-based pricing, subscriptions, credits, [44:41] hybrid models, whatever you want. [44:43] You can create revenue models based on usage without rebuilding your [44:46] pricing system every six months. [44:48] If you're building an AI product, Stripe Billing is worth a look. [44:55] What are your AI predictions for 2026? [44:58] What I'm really excited about is just being able to… [45:04] I'm still excited about the coding. [45:06] This is what we do as a company. [45:07] It's what many others do as a company as well. [45:10] The one aspect of this is expanding into more domains. [45:16] For example, in where we spend our time. [45:21] We as a company, we write Rust, we write Verilog, we write Python. [45:26] No Haskell? No, there's a story there. [45:28] I used to love Haskell. [45:30] Rust is my current favorite. [45:32] Mutation is good. [45:36] The models are extremely good at Rust and Python. [45:40] They've done a lot of RL on them. [45:42] They have not done as much RL on Verilog. [45:46] They've done almost none on, "Write me a markdown file that describes [45:51] a chip architecture." How do you even RL on that? [45:55] You have to say, what is a good chip architecture? [45:58] You have to somehow say whether that's a good result or not. [46:02] I think one of the things the labs are doing is trying to broaden what they've [46:05] done RL on, source it from customers [46:08] and so on in order to make it less spiky, [46:15] fill out the gaps between the spikes. [46:17] I presume the labs would love to work with you on improving the models [46:23] by doing RL on this specific task. [46:28] However, it's also special. [46:29] How does that make sense for us? [46:31] You're a special sauce. [46:33] Do you want to come up with some AI approaches but keep them proprietary? [46:40] We've looked at a few different aspects here. [46:42] There's the… [46:43] What we're able to do by ourselves, our business is not training models. [46:46] We do it in order to do the research on numerics, but actual [46:50] production models we don't do. [46:54] The biggest mileage is on the RL, and it's not something we can really do ourselves. [46:59] We'd love it if we could have a custom model just for us, but that doesn't [47:02] seem to be— Well you could, right? [47:05] The terms we've been offered by labs so far have not been on those terms, but— [47:08] Because you have to share the IP back. [47:10] The way they prefer to do it is that they put it into their mainstream [47:15] model because it's good for them. [47:17] Which, obviously, you don't want to do. [47:19] How do you think… [47:21] What does you using AI to design a model do you think look like? [47:25] Because this is actually an interesting sight glass into a weak version [47:29] of recursive self-improvement where we're using the AIs to develop better AIs. [47:35] I'm curious, what you think that looks like? [47:37] Is it your own proprietary recursive models? [47:40] What else? [47:41] Is there day-to-day AI usage that's load-bearing? [47:45] The stuff that is available today and I think will become even better, [47:49] very quickly is just the stuff that looks most like software. [47:54] Writing Verilog, running tests, running continuous integration, and so on. [48:00] That is a big fraction of the development time in a chip. [48:03] It's probably 9, 12, 15 months. [48:05] There's some stuff that's downstream of that, which is physical design, [48:10] which is, you take that Verilog and you generate the gates and the polygons. [48:17] We don't have a clear path for… [48:19] It's not at least the most obvious thing is not clear for how to compress that. [48:24] The goal, can you tape-out a chip in one month? [48:26] One month would be the goal. [48:28] In theory, you could compress all of the logic design and design [48:30] verification down to a short amount of time just by continuing [48:35] on the same path we're doing now. [48:36] But if you wanted to take the physical design down, [48:39] that has to leave code, you're now doing like graphical interfaces [48:42] and saying I want to play stuff and so on. [48:45] Actually, there has been work on this even [48:47] prior to LLMs, which is a specific [48:53] model trained for that particular problem. [48:56] I think the vendors, which is like Synopsys and Cadence, [49:00] probably should move in that direction. [49:03] Most of the focus has not been, do it faster; it's been, [49:05] do it with higher quality. [49:08] But that is a big bottleneck on, can I have a new chip every month? [49:13] There's just the practical thing of a new chip every month doesn't really make [49:16] sense because then if I'm deploying... [49:19] If it takes me a year to populate a data center, that means I'm going to have [49:21] different chips in different corners of the data center. [49:23] Yes. [49:24] When you talk about one month to tapeout, so you do all this work [49:28] to ultimately produce a file. [49:34] Everything TSMC then does, it's not entirely in software. [49:38] Is there some type-setting that has to happen of moving stuff around? [49:42] But what happens when you send your files to TSMC? [49:45] Then what? They create a mask. [49:47] That is where the ASML tools come in. [49:52] A mask is really just a stencil. [49:55] You shoot the lasers through the mask or the x-rays through the mask and then [50:00] that produces the different P-type and N-type semiconductors. [50:05] They produced the mask. That is the expensive part. [50:11] They're building up these 15 or so metal layers. [50:14] They placed on the silicon and then there are different layers of metals [50:17] which connect all the transistors together. [50:20] They do that on a wafer. [50:22] It happens on a stepping basis. [50:25] There's a maximum size of chip you can build which is constrained [50:28] by this machinery. [50:30] The wafer stepper is part of the ASML special sauce, right? [50:34] There are probably some important alignment requirement there. [50:38] I think I remember that being quite like the classic manufacturing [50:41] throughput problem. [50:42] I think they've done a lot of work on optimizing that. [50:45] They tape that, so then you just produce hundreds of copies of your chip. [50:49] You have to test it because there are defects. [50:52] You typically, I think the average rates really depends on process and so on, [50:58] but small single digit number of defects per chip. [51:03] You test the chip and see whether it has any defects in it. [51:06] Many chips are designed to be able to tolerate a few defects, and so you need [51:10] to configure it to tolerate the defects. [51:12] Now you have a die that by itself works. [51:15] Then you need to package it. [51:16] You put it in a package together with memories, [51:19] typically that's the HBM, and then maybe you escape the wires [51:24] to connect to other chips. [51:25] How long does it take to make a mask? [51:29] What we see is time from tape-out to first chips back. [51:34] Again, depends on node, but it's ballpark four or five months. [51:37] So tape-out is just sending the file? [51:42] We consider tape-out as sending the file, and then there's a whole process of you [51:45] make the masks for all the layers, and then, actually just producing the chips. [51:49] Producing the masks and producing the chips happens after tape-out. [51:51] That's right. I see. [51:53] Is the term tape-out from like you send a magnetic tape with the [51:56] instructions or something? It could be. [51:58] I was in software when the software was created. [52:02] I was curious about the tape actually means. [52:04] It feels like… [52:04] When I think about AI predictions, one thing I'm really struck by is how, [52:11] still in 2026, every time you open a [52:16] chat window, it's contextless. [52:19] It's got no memory. [52:20] Now, to be fair, it's like, guys, it's been four years. [52:23] Not even four years. It's been three and a half years. [52:25] Just calm down, we'll get there. [52:27] But I also interpret a lot of the current enthusiasm [52:32] for OpenClaw and all that stuff as [52:36] this super hacky backdoor into state [52:40] management where your little claw will write a markdown file of what it's doing [52:45] and then look at that markdown file the next time and things like that. [52:48] But it just feels like state management and memory is going to be [52:53] a huge deal and that will really change the character of AI products. [52:58] It's really interesting. [53:01] Long context is one of the biggest bottlenecks on speed of the model. [53:09] Every single token you generate, it reads through all of the previous [53:12] tokens, or maybe it reads through a subset of them, but reads through a lot [53:15] of the previous tokens you've written. [53:19] Memory bandwidth for that is really constraining. [53:21] You can think of model-level ways to solve that problem, which is to say [53:26] maybe I can compress it into fewer bytes or something like that. [53:29] But it's interesting that the most effective way to solve it has been—it's [53:33] really a combination of everything—but the most effective way to solve it has [53:36] been once you hit your 300,000 token limit, [53:40] have the model go back through it and compact. [53:44] This is kind of what OpenClaw is doing. [53:46] It's compacting everything you've done. [53:49] But it's funny that it's so manual. [53:53] I think— Manual is the wrong word. [53:55] It's so primitive. [53:57] It's maybe because it's so controllable. [54:00] If you want to iterate on how you compact, you give a different prompt, and you say, [54:04] "Compact this way, compact that way." You can iterate that on that [54:06] in seconds or minutes. [54:08] Whereas if you're trying to do some iteration on the model level, [54:11] where you say, "Now I've got a different model architecture," it's going to take [54:14] you months to train and launch something. Yes. [54:17] Any other AI predictions? [54:18] I'm generally just interested in what makes models cheaper and faster. [54:25] Just at the model architecture level, really tied into this context thing. [54:28] I think the context size will stay ballpark the same where it is, [54:32] maybe a few times larger, but the parameter count will go up. [54:36] Parameter count should grow much faster than context length actually, [54:38] just because of the underlying physics of what's available. [54:42] Though, has that been the story? [54:46] Would that be a reacceleration of parameter count? [54:48] Because it feels like we've leveled off slightly in the last year or two, [54:52] and instead we've been focusing on more and better RL. [54:56] Parameter count or thinking tokens, I guess. [55:00] Those are available, but the context length is struggling to grow. [55:07] But you think we… [55:08] When you say context length is struggling to grow, but you're saying we [55:10] keep context the same length. [55:12] Keep context the same length. [55:13] But we're better at working with large context. [55:15] Is that what you're saying? [55:17] Just have application-level interventions to manage large context, like compacting. [55:22] Because I think everyone's had the experience [55:24] currently of the chat conversation and the further down in the chat you get. [55:28] It just gets looser and—Yeah, sloppy. [55:30] It's just like really sloppy by the end, and it's like making mistakes, whatever. [55:33] You're saying we start to do better with large contacts. [55:36] I buy that. [55:37] When will I be typing into a chat window, and it is a MatX chip [55:43] underneath it, powering it? [55:45] Tape-out in under a year. [55:47] That means chips available end of year. Ballpark. [55:52] That's exciting. [55:52] So in 2027, I will be seeing very high performing chats as [55:56] a result of your chips. [55:57] In the 1% experiment of the users or something like that. [56:00] Exactly. [56:00] I need to find a way to finagle myself into the A/B test. [56:04] MatX is 100 people? That's right. [56:09] How have you gone about building the team, the culture? [56:16] What we have on the team is hardware, mostly hardware, but a big software [56:20] team and also a big ML team. [56:23] I think the ML team is quite unusual in what we ask them to do. [56:27] When you look at a typical ML team in an AI chip company, it will be [56:33] what I might say ML engineering or ML performance. [56:37] They're writing kernels that actually just use your hardware [56:43] as well on a given model. [56:44] There's a missed opportunity there. [56:45] If you're saying all we do is we take other people's models, [56:48] and we write kernels for them, you're optimizing this, but you [56:52] can't optimize this at the same time. [56:53] We want to optimize the whole thing at the same time, real code design. [56:57] Our ML team is actual real ML research. [57:00] What they do every day is they train small LLMs from scratch focusing [57:07] on numerics and attention. [57:12] This has really, really helped us make an interesting product. [57:18] It's shown up most strongly in our numerics. [57:23] Often what you see when people design numerics is they say, [57:27] back when Float32 was popular, it would be, "I'm going to follow the IEEE [57:30] standard." Now it is like, follow the Open Compute standard. [57:34] There are lots of little details where you say things like [57:38] maybe, "What's the rounding mode I'm going to use?" Like, [57:41] "Round to the nearest even," or something like that, which is like [57:45] the best known standard for how to round. [57:46] We want to cut corners anywhere we can. [57:49] Maybe don't do the best rounding, maybe don't do the… [57:54] Don't get all the corner cases correctly. [57:56] That's a very scary proposition if you're just making those choices blind, [57:59] but if you have the benefit of a research team who can back you up as you do that, [58:05] it's really powerful, and it's really interesting that we can [58:08] make some sloppy choices in these cases. [58:11] I feel like often technical advances come through better iteration loops. [58:17] A favorite example of this I found recently was that the Wright brothers [58:20] actually had a failed season before first flight. [58:25] First flight was in 1904, and they were down in Kitty Hawk in 1903 [58:29] and not making that much progress. [58:31] They went back to Ohio, and they had a wind tunnel, [58:34] and they were testing their design in the wind tunnel. [58:37] You can imagine not a lot of wind tunnels in 1904. [58:42] They did a lot of wind tunnel testing and their successful flight was after that. [58:46] Is this something you're focused on where, to get better chips, [58:50] you allow for a better testing and iteration loop, [58:53] and what does that look like? [58:56] I think this mostly happens in the architecture and product definition stage. [59:02] Maybe even more generally, I think AI chips seem to live or die [59:06] by product definition and architecture. [59:08] What is the most extreme form of fast iteration is doing it in your head. [59:12] Can you map a model to hardware in your head? [59:15] Can you estimate the performance of what it is in your head? [59:18] You're not going to be 100% perfect, but maybe you can prove some [59:21] lower bound in performance. [59:24] The simplest possible thing is my model has a trillion parameters, my device [59:31] can do a billion multipliers per second, so it takes 1,000 seconds to run or [59:34] something like that, just to do that simple division. [59:37] But then there are much more complicated things like we tend to look at [59:40] resource balances, like how many memory fetches do I need [59:43] to do per a multiplier or something like that. [59:49] At least the way I like to do design and architecture and optimization is to [59:55] be able to estimate the performance to within about 30-40% [59:59] before even typing anything in at all. [60:02] We've tried to do that a lot. [60:05] A lot of our architecture comes from there. [60:08] Then the next stage of iteration is… [60:11] That's on the performance side. [60:13] This also happens on the circuit design side as well. [60:16] Can you take a circuit and say what is the gate count on that? [60:20] A 16-bit multiplier has approximately 16 squared mini-gates, [60:24] and you can do that for more complicated things by sorting networks and so on. [60:29] We already have a pretty good idea of the costs and speeds of things at that [60:33] point after doing these calculations. [60:36] Then what we tend to do as the next step of iteration is [60:40] on the ML side, we run model experiments. [60:42] You get iteration speed just by having small models mostly. [60:48] On the hardware side, we use performance simulators to do [60:53] the next level of detail to make sure we're seeing all [60:55] the things we want to see. [60:56] This idea that the best iteration is in your head is reminding [61:01] me of Jeff Dean's numbers... [61:04] Do you have your equivalent of that numbers every MatX? [61:06] We have go/gates in our company, which says, what is the cost of an XOR [61:11] gate, an AND gate, a full adder, SRAM bitcell, and so on. [61:15] You want people to be working with that stuff in their head and have [61:17] an intuitive sense for it because it leads to better iteration. [61:22] What is the pitch to someone joining MatX? [61:26] I think if you are someone who likes optimizing, [61:30] just optimize something—software, hardware, factorial, [61:36] whatever—if you're trying to fit something into the smallest budget possible, I [61:41] think it's a pretty exciting place to be. [61:44] I think hardware companies in general are really exciting because you have such a [61:50] broad range of skills of people on the team. [61:52] You have software people, you have hardware people, [61:54] you've got physical design, you've got people who are just looking at [61:57] the insertion force of a card into a rack. [62:02] So there's so much discussion and learning you can do. [62:06] I think MatX in particular, we really care about this and I think we [62:11] extended it all the way up into the application [62:13] and the machine learning as well. [62:20] Really, really interesting technical problems, and I think just generally [62:25] there's lots of interesting people to talk to. [62:26] Yes. [62:27] Presumably in terms of impact, if you can design a meaningfully higher [62:32] throughput chip, a 20% higher throughput chip means 20% more AI is happening. [62:37] If the bottleneck is elsewhere, like power or something like that or cost, [62:41] you actually just are meaningfully increasing the amount [62:44] of intelligence in the world, which is presumably exciting to people. [62:49] I think this shows up both as just can it apply in more applications, [62:54] as well as just how smart is the model. Yes. [62:57] How about Rust? [62:59] A previous project I worked on at Google, we did a lot of Haskell. [63:04] I did Haskell when I was at school. I loved it. [63:08] Very principled, very interesting. [63:10] I like Haskell, but I also like making stuff fast. [63:14] The question is, what is the first thing you want to do? [63:17] You want to be able to modify your memory. [63:19] Haskell, you jump through hoops to do that. [63:21] Maybe I just want a language that is functional programming [63:24] that lets me modify my memory. [63:26] I think Rust has a lot of the nice things which are like type classes or traits [63:32] and a rich type system. [63:35] One of the things that we have done, interesting ways we use it at MatX are [63:41] the range of data types that you express on software. [63:45] What are the integer types? [63:46] Int32, int64, int8, maybe that's all you care about. [63:50] But it turns out in hardware, you care about every single bit, and so [63:55] you want to use 17, 18, 19-bit integers. [64:01] That is quite natural to express, and we build up a whole ecosystem of [64:07] rich hardware data types in Rust as well. [64:11] Has Rust beaten Go for the position of performant type programming language [64:17] with modern features or do they actually address different? [64:22] There's what the Rust marketing will say, which is "safe without a garbage [64:28] collection," which I think is a real… [64:31] It is the objective thing that you can say is different, but sort of buries the lead, [64:37] which is also just it's got nice type system features that Go doesn't have. [64:42] Why is garbage collection… [64:44] Why does it matter at all? [64:46] People often focus on the time it takes to run a garbage collector, [64:49] but the other thing is that every time you allocate an object, you've got the object, [64:52] and then you've got the garbage collector header at the beginning. [64:55] It uses a lot more memory as well. [64:57] If you want to design some, I don't know, data structure that uses [65:01] the right amount of memory rather than a bit more than… [65:04] I hadn't realized that in Rust you're allocating your memory manually [65:07] versus in Go you have— That's right. [65:10] I didn't realize that. And you prefer that for what you're doing. [65:13] I just really like dealing with the details. [65:15] Like, you give me a puzzle, and I'll be like, [65:16] "Let me solve every single piece of it." That tickles that part [65:20] of my mind with Rust. [65:21] It seems like you're a fan of optimization generally. [65:24] Is that a fair characterization? Yeah. [65:26] Where else have you… [65:27] Chip optimization is one domain, where else? [65:36] When one of the really exciting things I found about working at Google is [65:39] that the whole Google code base is available, and you can look at [65:43] how does a memory allocator work, how does a mutex work, [65:46] how does a HashMap work, any of those things, and you can go [65:49] and look inside the implementations. [65:51] Google has excellent implementations of those, some of the best you could write. [65:59] One of the things I did on my nights and weekends when I was at Google was just [66:02] to go find those implementations, write a benchmark. [66:07] How many nanoseconds does it take to allocate eight bytes of memory? [66:11] Can I make that faster? [66:13] Can I maybe I inline this function? [66:15] Maybe I look at the assembly and say, "It looks like [66:18] there's a few memory moves here or there are some registers that are being [66:21] used that I don't need in the fast path and in the slow path. [66:25] Can I do something there?" I don't know, that was always my [66:30] just fun and learning activity. [66:32] Being outside of Google, I feel, I probably could have done this inside [66:36] of Google as well, but outside of Google, I felt the luxury to be able to talk about [66:42] these results as well. [66:43] One of the things I've looked at recently is just hash tables are used so much. [66:51] One prompt for me was, if I wanted to design custom CPU [66:57] instructions for accelerating hash tables… [66:58] Hash tables are one of the most common things. [67:00] I'm looking at them up and writing them all the time. [67:02] What would the optimal CPU be for that? [67:07] Following down that chain is like, what is the best hash table [67:12] implementation in the first place? [67:14] I spent some time looking at different SIMD implementations. [67:18] There's this really cool technique called 'cuckoo hashing' where you [67:22] hash into two different locations, and then you use the bucket [67:26] which is less full. [67:30] It's been in the literature for decades and yet the best hash table [67:35] implementations don't use it because it's somehow not practical. [67:39] Why is it not practical? [67:40] Practical hash tables are these days considered to be ones that use [67:46] SIMD vector instructions to scan eight buckets at a time. [67:54] The way cuckoo hashing is normally described is I look up one [67:57] bucket here and one bucket there. [67:59] I'm not using the vector instructions. [68:00] Vector instructions are much faster than scalar instructions and so [68:03] there's a missed opportunity. [68:05] Again, just take the two good ideas and stick them [68:08] together, do vector instructions on cuckoo hashing. [68:13] You have to be careful to get the details right, but if you get it right, [68:15] you can actually just wing it. [68:16] Sorry, is your claim that one could design a custom CPU that has way better hash [68:22] table performance, or even on current chips, you could [68:26] get way better hash table performance? [68:28] Both. [68:31] I'm interested in what you can design in custom hardware, [68:33] but MatX doesn't make CPUs. [68:35] We're not going to make CPUs. You could. [68:36] New line of business. [68:39] We just want to focus on shipping one product well for the time being. [68:43] Fair. Good answer. [68:45] I think it's an interesting exercise, but I don't get to feel the endorphins [68:49] of seeing the number go down. [68:52] I first did this on just Intel CPUs. [68:55] You can get better performance than some of the best hash table [68:59] implementations available using cuckoo hashing on Intel CPUs. [69:05] What are examples of workloads that are really hash table read intensive? [69:11] I know everything, JavaScript, I guess, but… [69:18] It's a tricky exercise because when you really think about it, you're like, [69:22] "Did I really need a hash table there? [69:23] I probably didn't, but I just reach for it all the time." But you can go [69:26] to the Google JavaScript team and probably help them eke out better performance [69:30] in the Chrome JavaScript engine? Potentially. [69:33] I'm not going to spend my time on that. [69:35] If they're listening to this podcast, just a free idea from Reiner. [69:39] Explain the dragon. [69:41] This is from a book that, when I was working on the JAX team… [69:44] The JAX team is one of the ML infrastructure teams at Google. [69:47] I was there as the most recent team before I left. [69:50] I'm sorry, what does the JAX team do? [69:52] The JAX team develops… [69:54] This is Google's new, more modern version of TensorFlow or a competitor to PyTorch. [70:00] It's how you write models in Python to run on TPUs. [70:03] A big part of the JAX team, however, is to say, we have JAX, the technical artifact. [70:08] Can we help enable users to actually use it really well and get high performance? [70:14] Ultimately that became, who are the users? [70:17] It's people writing LLMs. [70:18] How do you get good performance on LLMs? [70:20] Really, really strong team, the JAX team at Google, [70:24] although as with a lot of brain people are now elsewhere as well. [70:28] We developed a lot of the different techniques for how to lay out [70:31] models efficiently on many chips. [70:34] Ultimately, some people at Google—and I contributed after I left Google—wrote this [70:39] guide called How To Scale Your Model, how to run an LLM as fast as possible. [70:44] It is the main reference for how to get high performance on TPUs. [70:47] There is now also a GPU version of this as well. [70:49] It's a dragon because it's How to Train Your Dragon. [70:52] I see. [70:55] Last question. [70:57] People might not have thought that there's room for new chip companies. [71:02] It might have seemed unusual or very hard. [71:05] You guys, it seems like a very good approach with that. [71:09] Where do you think are other opportunities for companies to be started here in 2026? [71:14] Where do you think people should be looking for entrepreneurial opportunities [71:17] or just technical challenges that haven't been properly addressed? [71:21] More labs, I think, is still interesting. [71:24] Just can we do more on model architecture is always interesting. [71:27] You think we have not fully explored model architecture space? [71:30] The Frontier Labs have done a pretty good job of exploring it, but I think, [71:36] as the hardware changes, the shape of the model should change for sure. [71:41] Presumably you're not thinking like, yet another Frontier Lab [71:44] pursuing the same architecture. [71:45] You think there's probably off the wall looking architectures that will actually [71:49] make a lot of— I think there's a little bit off the wall for sure. [71:53] Do you have a specific architecture in mind? [71:55] My mentality is always sticking within the transformer family, but [72:00] what are the constraints that are currently available, [72:02] currently imposed that you could lift? [72:05] For example, one of the things is there's this idea, when you're doing transformer [72:10] inference, you do pre-fill that is processing what the user [72:13] said to you, and then there's decode which is generating the response to that. [72:17] Those are totally different in pretty much every aspect of how they actually run. [72:22] One runs a step at a time, the other one runs really in parallel. [72:26] There is this somewhat artificial constraint today that those are [72:29] the same model that's doing both. [72:31] Maybe lift that constraint. [72:33] Another example would be, there's this idea that the model that you… [72:37] This is more fundamental constraint that you have to train [72:40] the same model as you serve. [72:42] But again, training is very different from serving. [72:45] At training, it's very compute intensive. [72:47] At serving, it's more memory bandwidth intensive. [72:50] Maybe, is there a way you can make a model that when you use it at inference time, it [72:56] increases the amount of computing it does to use some of the available resources? [72:59] Makes sense. Reiner, thank you. [73:01] Pleasure.