a16z Backs Gimlet to Solve AI Inference Bottlenecks Through Heterogeneous Compute

a16z Backs Gimlet to Solve AI Inference Bottlenecks Through Heterogeneous Compute Facing severe power, facility, and hardware constraints in AI inference, venture firm a16z announced its investment in Gimlet Labs, a company developing a multi-silicon inference cloud to optimize compute across diverse hardware architectures. AI inference faces physical constraints across power, land, and hardware, even as five U.S. hyperscalers prepare to spend $1 trillion in capex next year. Expanding physical facilities cannot match software-speed inference demand, necessitating heterogeneous architectures combining GPUs, CPUs, and custom silicon. Gimlet Labs coordinates multi-silicon systems across compiler, runtime, and physical facility levels, exposing them through a single inference API. Gimlet reports delivering up to 10X gains in throughput and interactivity on frontier models under the same power envelope, with customers including a frontier lab and a hyperscaler. Physical Constraints on AI Inference Inference demand is outstripping the availability of physical inputs, including powered land, turbines, transformers, data centers, GPUs, advanced-node wafers, and high-bandwidth memory. Even with five U.S. hyperscalers projected to invest $1 trillion in capital expenditures next year and NVIDIA mobilizing over $500 billion for AI factories, physical buildouts cannot keep pace with software-driven demand, creating bottlenecks that restrict model providers. The Shift Toward Heterogeneous Architectures AI workloads have grown diverse: voice applications depend on low latency, batch processing prioritizes throughput, and agentic workflows route between specialized models, CPUs, and external tools. Because single model calls split into compute-heavy prefill and memory-bound decode stages, no single processor optimizes for every metric. Maximizing intelligence per watt requires heterogeneous systems that combine GPUs with CPUs, memory-optimized systems, and purpose-built silicon. Gimlet's Full-Stack Orchestration Integrating diverse processors introduces major compiler and physical challenges, including differing networking topologies, power densities, and cooling needs like varying inlet-water temperatures. Gimlet Labs resolves these issues by analyzing workloads to balance latency, throughput, and cost, splitting tasks at model or operation levels behind a single API. Extending into physical data center operations, the platform unites multi-silicon capacity, achieving reported throughput and interactivity improvements of up to 10X within identical power envelopes.

a16z宣布投资多芯片异构推理云初创公司 Gimlet Labs 随着 AI 推理市场对电力和算力基础设施的需求激增，a16z 宣布投资打造多芯片异构推理云的 Gimlet Labs，旨在打破同构基础设施瓶颈，在相同电力消耗下最大化推理性能。 AI 推理需求受限于电力、数据中心及 GPU 等物理资源，五家美国超大规模计算企业预计明年资本支出将达 1 万亿美元。 由于不同 AI 工作负载特性差异巨大，单一处理器架构无法兼顾成本、延迟与吞吐，异构计算成为必要方向。 Gimlet Labs 构建了首个多芯片推理云，可将不同任务拆解并路由至最合适的处理器，并统一封装在单个 API 之后。 Gimlet 深入处理网络、供电和散热等物理层差异，在相同功耗下实现高达 10 倍的吞吐量与交互性能提升，客户涵盖前沿实验室与超大规模厂商。 AI 推理的物理瓶颈与异构需求 AI 推理正在遭遇电力、用地、数据中心和先进芯片等物理资源的严重短缺，OpenAI 和 Anthropic 等机构的发展也受到算力扩张速度的限制。面对爆炸式增长的推理需求，单纯扩张硬件无法追平软件增速。不同的 AI 场景对延迟、吞吐量和成本的要求截然不同，即使单次模型调用也包含计算密集型的 prefill 和内存密集型的 decode 阶段，需要结合 GPU、CPU 及专用芯片组成异构计算架构来协同处理。 异构计算的整合挑战与 Gimlet 的方案 大规模基础设施历来遵循同构设计，引入多种硬件架构会在软件协调、网络拓扑、机架密度以及电力和冷却需求上带来巨大复杂度。Gimlet Labs 构建了多芯片异构推理云，其编译器与运行时能够为每个工作负载生成兼顾性能与成本的执行方案，并在物理数据中心层面将不同芯片整合成统一资源池，向开发者仅暴露一个简单推理 API。 落地成效与团队背景 Gimlet 的方案使前沿模型在相同电力预算下实现了高达 10 倍的吞吐量和交互性提升，目前已拥有前沿 AI 实验室和超大规模企业客户。其核心团队覆盖了从内核、编译器、高速网络到电力、冷却和数据中心建设的全栈能力。

[America](https://www.a16z.news/t/america) | [Tech](https://www.a16z.news/t/technology) | [Opinion](https://www.a16z.news/t/opinion) | [Culture](https://www.a16z.news/t/culture) | [Charts](https://www.a16z.news/t/charts) We are running out of watts. AI inference is one of the fastest-growing markets in the history of capitalism, and we are running out of nearly every physical input required to serve it: powered land, turbines, transformers, data centers, GPUs, advanced-node wafers, and high-bandwidth memory. GPU capacity is being leased at record prices, while OpenAI’s and Anthropic’s growth is constrained by how quickly they can expand their compute fleets. You feel that scarcity firsthand: rate limits tighten and responses slow as providers trade latency for throughput to produce more tokens from the same watts. This is why the largest capital investment cycle in contemporary history is underway. Seven of the ten most valuable companies in the world are hyperscalers or supply them. Five U.S. hyperscalers alone are expected to spend $1 trillion in capex next year. NVIDIA has gone a step further, working with capital partners to mobilize more than $500 billion for AI factories. But expansion alone will not be enough. Inference demand compounds at software speed; power plants, data centers, and semiconductor fabs do not. Serving the most consequential workload of our lifetime requires squeezing more intelligence from every watt. That starts with recognizing that inference is not one workload. [Without the GPU, today’s AI would not exist.](https://a16z.com/how-to-win-the-largest-market-in-ai/) Its flexibility enabled the early experiments with transformers. Then enormous clusters built around a single accelerator architecture powered the training runs that proved scaling laws and produced today’s frontier models. But the diversity of AI applications is exploding. A voice assistant lives or dies on latency; batch processing maximizes throughput; a research agent may prioritize cost per token; a coding agent must balance all three. Even a single model call splits into compute-intensive prefill and memory-bound decode. Agents multiply the variation by routing among small specialized and larger reasoning models, retrieving files, executing code on CPUs, and calling external tools. Inference is no longer a single workload. At the same time, hardware choices are proliferating. Some architectures optimize arithmetic throughput; others optimize memory capacity, bandwidth, or low-latency connectivity. No single processor can deliver the best combination of cost, throughput, and latency across every application and every step. GPUs will remain foundational, but they will increasingly operate alongside CPUs, memory-optimized systems, and other purpose-built silicon. The optimal system must match each piece of work to the architecture that handles it best, producing more useful work from the same resources. It must be heterogeneous. The challenge is making all of this hardware operate as one system. Large-scale infrastructure is designed around homogeneity with standardized servers, networking, power, cooling, and operating models. Introducing multiple architectures creates complexity across every layer. The software must coordinate processors that were never designed to work together, while the physical facility must accommodate different networking topologies, rack densities, power profiles, and cooling requirements. Cerebras and NVIDIA systems, for example, require different inlet-water temperatures. Sometimes heterogeneous compute is a compiler problem; sometimes it is a plumbing problem. Making heterogeneous compute work at scale is key to unlocking what we believe will be the world’s largest market. We believe Gimlet Labs has built the solution: the first multi-silicon inference cloud, designed to produce more intelligence from every watt. For each workload, Gimlet determines an execution plan that balances latency, throughput, and cost. It can route different models and tools onto different processors, separate prefill from decode, or divide a model at the layer or operation level. Its compiler optimizes each piece for the target hardware, while its runtime coordinates execution across the system. To developers, all of that complexity sits behind a single inference API. Gimlet extends that orchestration into the physical data center. It integrates GPUs, CPUs, and purpose-built accelerators into one pool of capacity, managing the differences in networking, power, and cooling that make heterogeneous infrastructure difficult to operate. This system is already delivering up to 10X gains in throughput and interactivity on frontier models within the same power envelope. In a world where scale matters, Gimlet has done this at the largest scale possible, counting both a frontier lab and a hyperscaler as customers. In a market starved for compute, efficiency is net-new capacity and latency is product differentiation. We believe there is no better team than Zain, Michelle, Natalie, Omid, and James to tackle this problem. They’ve seen this movie before and are one of the few teams that have the range to follow a problem through every layer of abstraction. When kernels and compilers were not enough, they pushed into high-speed networking. When software was not enough, they moved into power, cooling, and data center construction. Wherever the existing stack ends, the Gimlet team starts. The future of inference is heterogeneous, and Gimlet is building the infrastructure layer that makes it possible. We could not be more excited to partner with the entire Gimlet team. *This newsletter is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. Furthermore, this content is not investment advice, nor is it intended for use by any investors or prospective investors in any a16z funds. This newsletter may link to other websites or contain other information obtained from third-party sources - a16z has not independently verified nor makes any representations about the current or enduring accuracy of such information. If this content includes third-party advertisements, a16z has not reviewed such advertisements and does not endorse any advertising content or related companies contained therein. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z; visit [https://a16z.com/investment-list/](https://a16z.com/investment-list/) for a full list of investments. Other important information can be found at [a16z.com/disclosures](http://a16z.com/disclosures). You’re receiving this newsletter since you opted in earlier; if you would like to opt out of future newsletters you may unsubscribe immediately.*