谷歌Bill Jia称电网与内存将成AI基建最大瓶颈

谷歌Bill Jia称电网与内存将成AI基建最大瓶颈 随着推理需求激增并向Agent演进，AI算力扩张的物理约束正从芯片生产全面转向电力供应与高并发内存存储体系。 Google工程副总裁Bill Jia深入分析了当前AI基础设施的发展趋势与核心瓶颈，强调未来算力需求将增长成百上千倍，大厂维持激进的CapEx投资具有战略必要性。他指出，随着数千亿级日常调用的普及，推理算力在整体算力结构中将从五五开迈向八二开，而行业面临的真实物理限制正从单纯的芯片短缺转向能源供给与电力电网瓶颈。针对硬件生态与架构演变，Bill Jia澄清TPU与英伟达并非全面竞争而是模式互补，并预言自主代理（Agent）时代的到来将让内存（Memory）与多级存储管线承受最严峻的供需危机。 AI基础设施投资与现金流：Google工程VP Bill Jia指出未来AI算力需求将扩张10倍至1000倍，各大厂巨大的CapEx投入是防止未来无法承接需求的战略必然，扭转现金流将依赖软硬件全栈协同降本。 (4:46) 40GW新增算力瓶颈在电力：明年预计上线的40GW数据中心容量，其真正约束并非芯片供应，而是电网基载电力不足以及地方社区对能耗、水耗与噪音的反弹。 (9:35) 推理算力反超训练迈向“二八开”：训练属于阶段性投入，而推理面向数十亿终端用户，算力分配正从五五开逐步走向推理占80%、训练占20%的结构。 (12:30) Agent时代迎来Memory（内存）危机：系统从单次推理转向多轮交互的Agent架构后，上下文与状态维护使得硬件瓶颈从加速卡转移至系统内存、CPU及闪存存储。 (22:43)

[12:30] Bill Jia: 因为训练只是一个一次性或者几次性的投资，但推理服务的是数十亿的用户和各种各样的应用，所以未来的需求主要在推理。 [15:07] Bill Jia: 训推一体现在还为时过早，因为训练跟推理所需要的 infrastructure 差别还是蛮大的。 [20:45] Bill Jia: 人类之所以能够发明 AI，是因为人类在科学、工程、艺术、文学各个方面有深刻的理解；因为有了这些基础，才创造了 AI。 [17:37] Bill Jia: 我觉得 TPU 不是在跟英伟达全面竞争，我觉得这是一个误解。TPU 的商业模式跟英伟达是完全不一样的。

[0:00] Looking ahead, the demand for AI will [0:02] continue to grow; everyone predicts [0:05] this, and we have reached a consensus [0:07] that building this infrastructure is [0:10] absolutely necessary. From your [0:12] position in infrastructure, do you see [0:14] any particularly good entrepreneurial [0:16] opportunities? [0:17] >> If I were in the infrastructure space, [0:19] I would definitely invest in developing [0:21] certain kinds of agents that are very [0:24] cost-efficient and very scalable. I [0:29] think memory will remain in short [0:32] supply; after the inference stage, it [0:34] demands a lot from the CPU, and it also [0:37] requires a great deal from Flash and [0:40] hard drive storage. [0:46] >> Could you recap our fireside chat just [0:49] now for us? What were the core points [0:51] discussed today? [0:52] >> Well, we covered about three points. [0:56] First, everyone agrees that major tech [0:59] companies are now paying close [1:01] attention to CapEx to vigorously [1:03] strengthen investments in AI computing [1:06] power. Our conclusion is that this type [1:10] of investment is absolutely necessary. [1:13] Because overall, the development of AI [1:16] is still in a very early stage. If we [1:19] look to the future, the development of [1:22] AI will continue to be in a phase of [1:25] rapid growth over the next one, two, [1:27] three, five, ten, and twenty years. [1:30] Therefore, the demand for AI will [1:33] continue to grow indefinitely. If that [1:35] is the case, and everyone predicts this [1:38] , we have reached a consensus that [1:40] building this infrastructure is [1:42] absolutely necessary. [1:43] >> Uh-huh, [1:44] >> That’s the first point. The second [1:46] point is that AI infrastructure— [1:49] whether it's building models, [1:51] post-training, or inference—is a very [1:54] time-consuming, resource-intensive, and [1:56] energy-consuming process. So the second [1:59] topic of discussion was how we can [2:02] achieve many hardware and software [2:04] optimizations to bring costs down. [2:06] Everyone is very eager to work together [2:09] to accomplish this, because in the past [2:11] year or two, we have actually already [2:13] done a lot of hardware and software [2:16] optimization, and the results are quite [2:18] clear. Metrics like performance per [2:23] watt, performance per dollar, and [2:25] dollar per intelligence are all [2:27] trending in the right direction, [2:29] showing that AI training and inference [2:32] are gradually becoming cheaper. However [2:36] , our consensus is also that there is [2:39] still a lot of work to be done in the [2:41] future. We want everyone to work [2:44] together to lower costs, and once costs [2:47] are down, AI will become more [2:49] affordable, and by making it more [2:51] affordable, we can fully accelerate [2:53] innovation, right? That was the second [2:56] topic of discussion. The third issue to [2:59] discuss is that AI still has some [3:01] security risks, right? Because this AI [3:04] model, this large language model, is a [3:06] neural network, its design makes a lot [3:09] of sense and it somehow works [3:11] beautifully and works very well. [3:13] However, because this large language [3:15] model is a neural network that is too [3:17] complicated, it remains fundamentally a [3:20] black box, and we don't understand why [3:22] it works this way. If we don't [3:24] understand why it works this way, its [3:27] behavior is very hard to control, and [3:29] while we can reduce the risk, the risk [3:31] still exists. We are talking about how [3:36] the industry can control this, to [3:39] enforce safety, security, and [3:41] safeguards even more, right? Actually, [3:46] we feel that once AI is fully refined [3:49] and all malfunctions and security risks [3:52] are eliminated, in actual inference, [3:55] it's not too bad, so far not too bad. [3:59] There might be more hidden risks in the [4:01] future; actually, right now, there are [4:04] still more of these hidden risks that [4:06] exist. When this AI is still new and we [4:09] don't know how it works, while testing [4:12] it, evaluating it, or in the sandbox, [4:14] it can be very, very unpredictable. How [4:18] do we make this sandbox—the testing, [4:21] verification, and evaluation [4:23] environment—more secure, and how do [4:25] we make this sandbox more centralized [4:28] and have a wall so it doesn't run wild? [4:33] It's mainly about the overall cost, [4:36] doing the whole hardware and software [4:38] optimization, and how to minimize these [4:41] AI safety risks. These are the three [4:44] aspects of the problem. Everyone is [4:46] investing a lot of money in computing [4:48] power, and all the frontier labs are [4:50] putting in a lot of capital. For CAPEX, [4:51] do you think this is still in the early [4:53] stages and something these companies [4:55] should continue to do? We’ve seen [4:57] that Google's balance sheet is negative [4:59] , and what people are worried about now [5:01] is that you're investing so much, but [5:03] the revenue from applications hasn't [5:05] been seen yet. Do you think this is [5:07] just a pacing or timing issue? [5:09] >> Yes, I think there are two main points, [5:11] which we have touched upon. The first [5:13] point is that the demand for AI is [5:15] still incredibly huge. It will even be [5:17] 10x, 100x higher in the future. [5:19] >> Mm-hmm. [5:21] >> If we know that demand in the future [5:23] will continue to grow, then making [5:25] these investments now is the right [5:28] thing to do, right? Because if I don't [5:30] invest and others do, when the demand [5:33] rises later, being unable to [5:34] accommodate that demand would be a big [5:37] problem, right? [5:38] >> This is something that must be done [5:40] first. This is definitely correct. The [5:44] second point is that with all these [5:46] investments and this demand, how can we [5:49] serve it in a more cost-efficient way [5:51] to make it cash flow positive? How can [5:55] we make this model higher in quality [5:57] and lower in cost? By increasing its [6:00] intelligence and reducing its [6:02] floating-point operations, right? There [6:05] is still a lot that can be optimized in [6:07] software, from networking and memory [6:10] access to compute; there are many [6:12] stories and areas available for [6:14] optimization. Meaning, even if the [6:17] model and hardware remain unchanged, I [6:19] can make the cost cheaper through [6:20] software optimization. Second, and [6:22] third, the hardware itself can also [6:25] become cheaper. [6:26] >> Mm. [6:28] >> With this kind of synergy, I believe [6:30] that in the future, this massive demand [6:33] will certainly be served at a much [6:35] lower cost. The results will be better, [6:39] but the costs will be lower, so it will [6:41] become cash flow positive; this is [6:43] definitely achievable, so being cash [6:45] flow negative for now isn't something [6:47] to be afraid of. As long as we can keep [6:50] persisting. [6:52] >> So, you mean the pace of CAPEX [6:54] investment will find a better balance [6:56] with everything, including cost [6:58] optimization and application revenue, [7:00] right? And then, under a healthy [7:03] balance, we can continue to grow [7:05] sustainably. You also mentioned earlier [7:07] that there is still a lot of room for [7:09] inference optimization. If you were to [7:11] start a business today, and you were [7:13] positioned in this infrastructure layer [7:14] , what kind of good entrepreneurial [7:16] opportunities do you see? Especially [7:17] compared to big tech, is there anything [7:19] to do, since everyone is worried about [7:21] the various advantages of big tech? In [7:22] your opinion, if you were a startup, [7:24] what kind of opportunities do you think [7:26] they would be better at executing? [7:28] >> Yes, I think there are many great [7:30] startups, such as Fireworks AI, their [7:33] product is called Infini. Because that [7:38] founder is a panelist. He is the [7:41] founder of vLLM, which is a serving and [7:44] inference framework. These companies [7:47] are already doing very well, so I think [7:50] if you focus on things like, for [7:52] example, inference. Doing inference [7:54] ultimately means making the model [7:57] faster and optimizing costs; in fact, [8:00] there is a lot you can do now by [8:02] leveraging AI. If you build AI agents, [8:06] they can actually drive inference [8:08] performance, reliability, and [8:11] scalability to be extremely high. If I [8:14] were to start a company, I would [8:16] definitely invest in this. Specifically [8:18] , building these AI agents. This agent [8:22] itself would be used to handle [8:24] inference. You can adapt to different [8:27] hardware, whether it's NVIDIA GPUs, AMD [8:30] GPUs, Google TPUs, or AWS Trainium—no [8:33] matter the hardware, no matter the [8:35] model, right? Because there are so many [8:38] types of models and so many types of [8:40] hardware, n times m is a massive number [8:42] . If you optimize every single one [8:45] manually, it's very unscalable, but [8:47] developing agents to handle the [8:50] optimization is really cheap and [8:52] scalable. So if I were in this space, I [8:56] would definitely invest in developing [8:59] certain kinds of agents to handle this [9:01] in a very cost-efficient and scalable [9:04] way. [9:06] >> Hmm, so you think this won't be [9:08] dominated by big tech companies, but [9:10] rather by startups? I think big [9:13] companies will do it, but sometimes, [9:15] since they aren't short on cash, they [9:18] might not necessarily do it the best [9:20] way. I believe there are some brilliant [9:25] minds—very enthusiastic, passionate, [9:27] skilled, and expert—who can start a [9:30] company to do this, and I think they [9:32] can sometimes move much faster. There [9:35] will be an additional 40 gigawatts of [9:37] compute capacity next year, which [9:39] already exceeds the entire current [9:42] market inventory. Looking at the supply [9:44] side, what bottlenecks do you see, and [9:47] do you think that 40 gigawatts of [9:49] capacity can actually come online? [9:54] >> Right, whether it ends up being 40, 30, [9:57] or 50 gigawatts, I don't know the exact [9:59] number, but generally speaking, it's [10:02] certain that a massive amount of data [10:04] center capacity will come online next [10:07] year. [10:11] >> But whether it can actually go live...I [10:13] think the bottleneck is really on the [10:16] power side, right? Because currently, [10:19] power generation still relies mainly on [10:22] natural gas and coal. The three [10:28] alternatives—solar, wind, and others [10:31] —aren't yet stable or efficient, so [10:34] finding natural gas, coal, or oil for [10:37] power generation is key. Actually, [10:41] utility companies capable of providing [10:44] this around the world are limited. Plus [10:48] , with the current political climate, [10:51] building AI data centers consumes a lot [10:53] of resources, generates significant [10:55] heat and noise, and draws massive [10:58] amounts of power, right? It really does [11:01] raise concerns for local neighborhoods. [11:05] In some states, like in the US, some [11:07] are saying, "I don't welcome AI data [11:10] centers in my state." Long-term, I [11:16] think power will remain a bottleneck, [11:18] which is not surprising from both a [11:21] technical and political perspective. I [11:27] think that, I think that for this tech [11:29] sector, I think regulators and tech [11:32] sector leaders should sit together to [11:34] see this issue. I think in the very [11:39] long term, if we could build data [11:41] centers in space, that's a long shot, [11:44] but if that could happen, it would be [11:46] fantastic. More realistically speaking, [11:49] how to use green energy, how to use [11:52] nuclear energy for power, that is very [11:54] cost-efficient, energy-efficient, and [11:57] sustainable. Google, how does your team [11:59] internally view how this power issue [12:01] will eventually be solved? Google will [12:03] definitely invest, right? We will [12:06] definitely invest, but we still hope to [12:08] work with the government—whether it's [12:10] the US government or other national [12:12] governments—to jointly solve this [12:14] power issue together. [12:15] >> So, in the future, these new data [12:17] center computing resources will likely [12:19] go to places with abundant energy. Just [12:22] like drilling for oil, right? It’s [12:24] useless to drill for oil where there is [12:26] none; you must build where energy can [12:28] be provided. Right, we just talked [12:31] about the supply side and the demand [12:33] side. We currently see the industry's [12:35] compute split as roughly half for [12:36] training and half for inference. How do [12:38] you see this demand ratio in the future [12:40] , Chen? Besides this scenario, what [12:42] other scenarios might we see in the [12:44] future? [12:44] >> Yes, I think it's hard to say it's [12:47] 50/50. I actually think inference might [12:50] have already surpassed training. While [12:56] training sometimes gets a boost from [12:58] recursive self-improvement and [13:00] reinforcement learning, I think the [13:03] major long-term trend is that inference [13:05] compute will take up more and more. [13:12] This is because training is just a [13:14] one-time or few-time investment, while [13:17] inference serves billions of users [13:19] across many different applications, so [13:22] future demand will mainly be in [13:24] inference. So I think over time, if [13:28] it's 50/50 now, it will definitely [13:31] shift to 60/40, 70/30, or 80/20. [13:34] Because once inference applications [13:36] scale up and consumption increases, its [13:39] percentage will definitely rise. Coding [13:42] is currently a very successful vertical [13:45] , not just for enterprise, but also a [13:47] very successful consumer product. I [13:51] think there will certainly be more such [13:54] vertical success stories in the future. [13:58] For example, using AI for financial [14:00] analysis and financial investments will [14:03] surely outperform quantitative finance [14:06] because AI has such strong analytical [14:08] capabilities, right? Then there is [14:11] using AI for legal services. Legal work [14:14] is very rigorous; it’s about the [14:16] facts as they are, and AI can handle [14:18] that kind of analysis. It's definitely [14:21] better than humans, so AI in the [14:23] medical and biological fields will [14:26] perform exceptionally well. Autonomous [14:29] driving is now very mature; Tesla’s [14:32] self-driving cars are quite advanced. I [14:38] believe it’s only a matter of time [14:40] before every car in the world is [14:42] equipped with full self-driving [14:44] functions. Traditional car companies [14:49] produce competent vehicles, but they [14:52] lack AI competency. I think if [14:55] companies emerge that offer a [14:57] centralized software platform, you [14:59] wouldn't need to build your own; you [15:01] could just use their services. I think [15:05] that will be a very big market. People [15:08] are both concerned that these AI [15:10] foundation models seem capable of [15:12] everything, and excited to see AI [15:14] applications blossoming in areas like [15:17] law and healthcare. Do you think these [15:21] large models will eventually take over [15:23] what existing startups are doing, or [15:26] will we see a continued ecosystem of [15:28] diverse, specialized players? I think [15:33] it will be both. Big players and [15:37] frontier labs will continue to focus on [15:40] larger models, pushing the boundaries [15:43] across all dimensions. That path will [15:50] definitely persist, as it’s the [15:52] ultimate pursuit in this field—the [15:54] one move that overcomes all others. At [16:00] the same time, we will see vertical [16:03] models—models specialized in coding, [16:05] finance, autonomous driving, or legal [16:08] services. These will definitely exist [16:12] as well. With these, because they are [16:15] trained, they possess general knowledge [16:18] . But once you feed them [16:20] domain-specific knowledge, they become [16:22] incredibly deep and precise in those [16:25] areas. This is also necessary, so I [16:27] think it will be a process of both [16:29] paths moving forward together. Is there [16:33] a trend in frontier labs toward [16:35] integrating training and inference? If [16:39] this boundary becomes harder to [16:41] distinguish, what new requirements does [16:43] that place on infrastructure? [16:47] >> It’s still too early for integrated [16:50] training and inference, as the [16:52] infrastructure required for training [16:54] and for inference is still quite [16:56] different. Training often involves [16:59] linear processes. [17:01] >> Right. [17:02] >> First, you get one model completed, [17:04] right? That is a separate process. Then [17:07] , when it comes to inference. By the [17:10] time you reach inference, the model is [17:12] already well-refined. But during [17:15] inference, the model is constantly [17:17] acquiring new online data. It makes [17:21] some updates, and these updates can be [17:24] used for inference very quickly. This [17:26] is achievable, but I don't think it's [17:29] necessary—nor is the technology [17:31] mature enough—to integrate training, [17:34] post-processing, and inference all into [17:36] one. Dropic signed a five-year [17:38] agreement for one million TPUs this [17:41] year, so the TPU finally has external [17:43] customers. It feels like the TPU is now [17:46] competing comprehensively with NVIDIA. [17:49] How do you view the ecosystem of TPUs [17:51] and CUDA, and how do you see their [17:53] competition? [17:54] >> I don't think the TPU is competing [17:56] comprehensively with NVIDIA; I think [17:59] that’s a misconception. The TPU’s [18:01] business model is completely different [18:04] from NVIDIA’s, right? Google decided [18:07] to offer these TPUs to a few selected [18:10] companies. It's more about having [18:13] certain partnership relationships. [18:17] It’s not like they are selling TPUs [18:19] to everyone, making them the primary [18:22] resource for everyone. Google isn't [18:26] doing that; it’s very selective, [18:28] right? However, I think that overall, [18:33] more AI hardware in the market will [18:35] help drive down costs. This will also [18:40] lead to a relatively healthy boost in [18:42] performance; that’s my perspective. [18:46] In an interview this past May, you [18:49] mentioned that many simple tasks might [18:51] be handled by small models on mobile [18:53] devices in the future, while large, [18:56] complex questions would be handled in [18:58] the cloud. But we currently see that [19:01] for many things like Gemini, most of [19:04] the computing power is still resolved [19:06] in the cloud. So, when will we see an [19:09] explosion in on-device computing power? [19:11] Actually, I think that...Of course, [19:14] Gemini is a relatively new product, [19:16] right? It’s...I’m not entirely sure [19:19] whether it’s all on the server side [19:21] or if some of it is on-device; I’m [19:23] not clear on that. But, I think that in [19:27] the industry, this hybrid model of [19:29] on-device and cloud is actually already [19:32] happening. Things like image editing [19:36] and generation happen on the device. [19:42] Google has also released many apps [19:44] where, if you and I were having a [19:46] conversation—say you speak Spanish [19:48] and I speak English—we could just [19:50] keep speaking our own languages, and [19:53] real-time translation would translate [19:55] it into the language the other [19:57] understands. That is also an on-device [20:03] model. This can actually happen even [20:06] faster. I feel that a lot of the [20:08] current attention is on server-side [20:10] models; you don't hear much about [20:12] edge-side models because the business [20:14] model might not be clear. But I think [20:18] edge chips are becoming more competent, [20:20] and edge models are also becoming more [20:22] competent, so these models will get [20:24] stronger in the future. For example, [20:27] the models Google launched called Gemma [20:29] , the 2B and 7B models, are actually [20:32] performing very well. This means a lot [20:35] of traffic can be served on the device, [20:37] which is a major experimental direction [20:40] within Google. We are essentially [20:43] shifting a lot of traffic to the edge. [20:45] AI is constantly achieving great [20:48] results in coding and mathematics. Now, [20:51] when people think about education, they [20:54] wonder: if AI has already solved these [20:56] problems, should we still learn math or [20:59] computer science? Or should we focus [21:02] our energy on what AI cannot yet [21:04] replace, or what will be replaced last? [21:07] Where should we allocate our resources [21:09] and energy when educating children? [21:11] >> Yes, I actually strongly believe that [21:14] even though AI can help humans do many [21:17] things. The reason humans can invent AI [21:20] . [21:21] >> Mm. [21:22] >> Is because humans have a lot of [21:24] understanding in science, engineering, [21:27] arts, and literature; it’s because [21:30] they have these foundations that they [21:32] can create AI. And going forward, to [21:37] create better AI over time, I think [21:39] humans will still play a very important [21:43] role; humans must be involved in [21:45] creating better AI. If humans don't [21:48] have those basic theoretical [21:50] understandings, whether it's math, [21:52] physics, chemistry, arts, or literature [21:55] , it would be impossible to do so, [21:57] right? Besides, after learning this [22:00] foundational knowledge, it makes people [22:03] better thinkers. Mm. [22:05] >> The process of learning knowledge [22:08] builds strong competence. It’s not [22:10] necessarily that the knowledge itself [22:12] is useful, like one plus one equals two [22:14] . You know, whether you learn that [22:16] might not be that important. But after [22:18] you learn a lot of mathematics, it [22:20] makes the human brain more analytical [22:23] and more creative. This brain will be [22:26] at a different stage. [22:27] >> Mm. [22:29] >> So it will produce many other things; [22:31] it’s not just about repeating [22:33] knowledge, as knowledge doesn't need to [22:35] be repeated. [22:37] >> Mm. [22:38] >> With this knowledge, it can create [22:40] newer knowledge, which is a very [22:42] important process. [22:43] >> In terms of infrastructure, we know [22:44] many things are in short supply— [22:46] compute, networking, and memory are all [22:48] scarce. Which bottleneck do you see as [22:50] the most serious and the hardest to [22:52] solve? I think memory will still be in [22:54] very short supply. This is because [22:57] after the rise of agents, in the past, [23:00] it was mainly focused on inference, [23:02] that is, raw inference. In raw [23:06] inference, what's missing is [23:09] accelerators, like GPUs and TPUs, as [23:12] well as memory. In the past, during the [23:18] inference stage, after the accelerators [23:20] , there was high demand for CPUs, even [23:23] more demand for memory, and also [23:25] significant demand for flash and hard [23:28] drive storage. As I just mentioned, [23:34] there will definitely be better [23:36] enterprise-oriented and [23:37] consumer-oriented new products for [23:39] agents, so the demand will be huge, and [23:42] I think all these components will [23:44] remain in relatively short supply. [23:48] Great, thank you very much for sharing [23:51] today.