How Specialized Hardware and Production Scale Are Set to Drive the AI Inference Market

How Specialized Hardware and Production Scale Are Set to Drive the AI Inference Market As AI workloads shift from fixed training costs to ongoing operational inference, general-purpose GPUs face memory and power bottlenecks, driving computing history toward specialized inference hardware. Venture firm a16z highlights startup Etched as an example of full-stack inference architecture aimed at this shift. AI compute is transitioning from capex-intensive training to opex-driven inference, where cost scales directly with token generation. GPUs face an architectural mismatch with inference because token generation is gated primarily by memory streaming and data-center power limits rather than flexible arithmetic. Techniques like batching exchange latency for throughput, which clashes with the low-latency requirements of coding agents and reasoning workloads. Hyperscalers develop internal custom silicon (such as Google's TPU), leaving a wide market open for merchant inference systems targeted at other labs and enterprises. The Specialization Turn in AI Workloads Computing history demonstrates that as workloads expand and stabilize, processing moves from flexible general-purpose processors to specialized hardware, just as graphics moved to GPUs and networking to custom switch chips. While training represents a one-time amortizable capex cost per model, inference is continuous operational expenditure. Generating tokens requires reading vast weight collections and context caches out of memory for minimal arithmetic per byte, turning tokens per watt into the critical operational metric. The Bottlenecks of General-Purpose GPUs GPUs enabled the modern AI wave through their adaptability, but they waste power on unused flexibility in inference environments constrained by electrical wattage. Common mitigation techniques such as batching pool user requests to amortize memory reads, but this comes at the cost of higher latency. This latency trade-off is increasingly unsustainable for responsive applications like interactive coding agents and long-context reasoning. Etched's Full-Stack Inference Approach Etched, founded by Harvard dropouts Gavin Uberti, Robert Wachen, and Chris Zhu, was established to build full-stack inference systems—covering silicon, boards, interconnects, and racks—for organizations lacking hyperscaler-grade custom chip programs. With more than 400 engineers focused substantially on manufacturing and logistics, the company built chips using TSMC's N4P process running on low voltage alongside cluster-scale memory interconnects. The firm is preparing to ship racks and deploy a 10-megawatt site to support architectures including mixture-of-experts, state-space models, and long-context agents.

a16z投资备忘：专用硬件时代来临与AI推理芯片初创公司Etched a16z发文指出，计算历史遵循着工作负载从通用向专用演进的规律。随着AI推理需求呈指数级增长，通用GPU在内存与功耗上的瓶颈愈发明显，专用推理系统成为必然趋势。为此，a16z宣布投资全栈推理硬件公司Etched。 AI推理本质是随使用量扩展的运营成本（opex），各大平台对推理算力的需求正急剧上升，促使其成为最大规模的计算工作负载。 GPU虽具通用性，但处理推理时存在内存受限及“批处理牺牲延迟”的缺陷；在数据中心受限于电力瓦数的当下，“每瓦生成Token数”成为关键衡量指标。 超大规模云服务商已开发自研芯片（如TPU），但非巨头生态（AI实验室、企业私有云等）亟需可直接采购并部署的专业推理系统。 Etched由哈佛辍学生创立，专注于构建包含芯片、机架、互联及供电的全栈系统，首款TSMC N4P制程芯片一次流片成功，并将于夏季交付客户机架。 计算的专业化演进与AI推理瓶颈 从CPU到GPU再到网络交换芯片，历史表明当某种关键计算负载规模足够大且形态稳定时，专用硬件比通用系统更快、更经济。模型训练属于一次性资本支出，而推理则是与使用量挂钩的运营成本。当前推理需求呈垂直攀升，但GPU生成Token的核心瓶颈在于内存数据搬运而非纯算力运算，其批处理机制虽然提升了吞吐量却牺牲了延迟，无法满足智能体编程和长上下文推理的实时要求。 从自研专用硅片到全行业机会 虽然Google研发TPU以及亚马逊、微软、Meta布局定制芯片验证了专用化路线，但这些方案仅服务于巨头内部平台，外部客户无法自由购买或在自有数据中心部署。对于大量没有芯片团队的AI实验室、推理云以及主权AI项目，独立的专业推理硬件存在巨大的市场空白。 Etched的系统级技术突破与量产交付 Etched组建了超过400名来自一线科技公司的工程团队，其技术核心在于两点：低电压推理（运行电压约为传统AI芯片的一半，大幅提升能效）和集群规模内存（通过超低延迟互联使整个机架在解码阶段如同单一机器协同工作）。其首款TSMC N4P工艺芯片一次流片成功并在不到两个月内完成点亮，首批机架预计将于夏季向客户发货，并正建立10兆瓦设施以支持万亿参数MoE、状态空间模型及长上下文智能体的高负荷推理。

[America](https://www.a16z.news/t/america) | [Tech](https://www.a16z.news/t/technology) | [Opinion](https://www.a16z.news/t/opinion) | [Culture](https://www.a16z.news/t/culture) | [Charts](https://www.a16z.news/t/charts) There’s a certain rhythm in the history of computing: we can call it *the specialization turn*. When a workload is varied and shifting—usually early in the lifecycle of that workflow—people pay for flexibility: nobody knows what tomorrow’s program will look like, so the work goes to a general-purpose system. That works for most things, most of the time. But every so often a workload grows so large, and coheres into a shape so stable, that the economics of the workload can be improved by specialization. The work can migrate into hardware built in its own image: into custom chips and, increasingly, entire custom systems built around them. A machine designed for *one* job, and only that one job, can do that job faster and cheaper than a machine that has to be ready for everything. That’s what we’ve seen through the history of computing. Graphics migrated from the CPU to a new class of accelerator, the GPU: the “graphics processing unit.” The same turn happened with networking: the internet’s trillions of daily packets long ago stopped moving through CPUs and started moving through custom switch chips from companies like Broadcom. In short: *the more important a workload, and the more stable its shape, the more sense it makes to build hardware in the image of the workload*. We are witnessing this now for perhaps the most consequential workload of our lifetime: AI inference. When people talk about AI compute, they typically mean training: training has been where all the drama (and dollars) lived these past few years. But training, in accounting terms, is just capex. You pay for it once per model, the way a studio pays to make a film: an enormous cost, typically, but a fixed and amortizable one. Inference, by contrast, is opex: a cost that scales with usage. Every time you ask ChatGPT a question or make an API call or have an agent write some code for you, the company that’s serving the model incurs some cost. To generate a single token, a model needs to read its weights—hundreds of gigabytes of parameters—out of memory, run them through the matrix arithmetic, produce the token, and then do it all over again for the next one. Every token, in other words, has a price in silicon time and electricity. Inference is the COGS of intelligence. And for the companies serving models at scale, the last few years have seen the demand for inference, and the cost of inference, go sharply vertical. In May 2026, Google announced that it was processing 3.2 quadrillion tokens per month across its products, roughly 300 times what it had been processing two years earlier. We see the same story with OpenAI, which now has roughly a billion monthly active users and a rapidly growing coding agent platform in Codex, and with Anthropic, which has seen enormous success with its Claude Code product—not to mention all the companies building on top of these platforms. AI inference is quickly becoming the largest workload computers have ever run; and we believe that it’s only going to grow much further. The machine serving nearly all of that demand today is the GPU. The reason for that is simple: the GPU is *flexible*. The GPU’s flexibility—the fact that it can support all sorts of demanding workloads—is the reason the AI era exists at all: when nobody knew which architectures would win, a processor that could run anything was exactly the right tool. But if you hold the workload up against the machine, the mismatch is hard to unsee. Generating a token is less about arithmetic and more about memory: to produce each token, the chip must stream the model’s entire weights, plus a growing cache of everything already in the context window, and performing only a couple of simple operations on each byte it reads. The standard patch for this is what’s called “batching”: serving many users’ requests at once, so that each expensive read of the weights from memory is amortized across dozens or hundreds of tokens instead of just one. Batching works well; but it’s not a panacea. It buys throughput, but it does so by selling latency and making every inference workload slower. That’s a trade that the fastest-growing workloads—coding agents in tight loops, long-context reasoning—can’t really afford: customers want things done *fast*. In short: GPUs are built for everything; inference is now big enough, and differentiated enough, that it requires custom hardware. And this is particularly true for AI, which is so compute- and energy-intensive. Data centers are now gated by watts rather than dollars, and every watt that a general-purpose chip spends on flexibility that the inference workload never touches is a watt not producing tokens. Tokens per watt is the real currency of inference. Over the last few years, the players with the most money at stake have more or less figured this out. For neural networks, this transformation started in the early 2010s, with Google’s development of the TPU, the “tensor processing unit,” and the entire infrastructure around it—custom interconnect, pods, and cooling. Other hyperscalers, like Amazon, Meta, and Microsoft, have followed; and now the frontier labs, like OpenAI, are following as well. This isn’t a mysterious development. It’s just the oldest pattern in computing reasserting itself: when a workload gets big enough and stable enough, it earns specialized hardware of its own. The machines that win a stable and huge market like AI inference are the ones designed as a whole, from the transistor to the data center floor, and that nail production scale to meet the demands of this new work. In 2022, months before ChatGPT and years before Cursor, Claude Code, and Codex, Gavin Uberti, Robert Wachen, and Chris Zhu dropped out of Harvard to make a bet that looked, at the time, nearly irresponsible: they would build an entire inference system—new chips, boards, interconnects, racks—from scratch. Their thesis was simple. Inference would be the largest market in AI, and the general-purpose GPU wouldn’t be the endgame of that market. Hyperscalers had realized that already. But their custom silicon programs were for *their* workloads: you can rent a TPU, but you can’t buy one, or rack it in your own data center, or build a business on it. For everyone who wasn’t a hyperscaler—for AI labs without silicon teams, inference clouds, sovereign AI programs, enterprises building their own fleets—there was a huge opportunity for chips designed specifically for inference. Gavin, Robert, and Chris set out to build exactly that. What began as a radical, contrarian wager has matured into a strong bet on the defining challenge of our era. Over the last few years, Etched has hired more than 400 engineers from Nvidia, Google’s TPU group, Broadcom, Apple, SK Hynix, and TSMC. A striking share of them work not on chip design but on production: supply chains, rack assembly, burn-in, and logistics. In a supply-constrained market, the scarce skill is not designing a fast chip; it is shipping complete, working systems by the thousands. As Gavin, Robert, and Chris like to put it: *production is the product*. Under the hood of that product are two core ideas. The first is *low-voltage inference*: running the math blocks at roughly half the voltage of a typical AI chip, packing several times more usable compute into the same power envelope—more tokens per watt, the currency that matters. The second is *cluster-scale memory*: an ultra-low-latency interconnect that pools memory across chips, so that during decode, an entire rack behaves like one enormous machine. Both of these technologies are not possible by only designing a chip: they require technical breakthroughs across the entire system across memory, power delivery, cooling and more—precisely the bottlenecks specialization predicts, in precisely the market where every multiple of tokens per watt converts directly into revenue. But as impressed as we are with the elegance of Etched’s first product, what’s blown us away is their tempo of execution. Etched’s first production chip taped out on TSMC’s N4P process and worked on the first attempt—a“first-pass silicon,” in industry shorthand—rare even for industry incumbents. As we sat in their 2 megawatt lab in their office, we heard stories again and again of the team defying the odds to achieve unprecedented timelines, flying across the world to unblock vendors and running company-wide day and night shifts to bring up their chip in under two months. (The industry norm is between six and nine months.) The result: those first racks ship to customers this summer; a 10 megawatt site is standing up, with customers already running workloads on Etched hardware remotely. Despite the origin story, what Etched has built is not a transformer-only machine, but a full-stack inference system to serve the most demanding workloads, whether many-trillion parameter MoEs, state-space models, or long-context agents. History is unsentimental on this point: accelerated hardware will be built for the most important workloads, but very few can build a machine capable of production at scale. It’s not enough to have a great architecture and demo rack; you have to ship at scale, with the speed and quality that the most demanding customers in the world require amid fierce competition. In a world where production is the product and the machine is the moat, we’re thrilled to partner with Etched. *This newsletter is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. Furthermore, this content is not investment advice, nor is it intended for use by any investors or prospective investors in any a16z funds. This newsletter may link to other websites or contain other information obtained from third-party sources - a16z has not independently verified nor makes any representations about the current or enduring accuracy of such information. If this content includes third-party advertisements, a16z has not reviewed such advertisements and does not endorse any advertising content or related companies contained therein. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z; visit [https://a16z.com/investment-list/](https://a16z.com/investment-list/) for a full list of investments. Other important information can be found at [a16z.com/disclosures](http://a16z.com/disclosures). You’re receiving this newsletter since you opted in earlier; if you would like to opt out of future newsletters you may unsubscribe immediately.*