肖志斌：先进制程提速见顶 软硬件协同可追回三倍算力差距

肖志斌：先进制程提速见顶，软硬件协同可追回三倍算力差距 随着芯片物理缩放与SRAM微缩陷入停滞，AI算力竞争的核心瓶颈已从晶体管制造转向以虚拟仿真和软硬件协同重构系统级能效。 对话全面拆解了从 7nm 到 2nm 制程升级中控功耗重于提速的工程本质，以及芯片从架构定义、EDA 验证到台积电代工的完整产业链壁垒。肖志斌深度剖析了大模型推理在 Prefill 与 Decode 阶段遭遇的内存墙困境，对比了 GPU、Groq、Cerebras 与 SambaNova 等不同硬件架构路线的取舍，并指出纯软件与系统协同调优能追回先进制程 2 到 3 倍的差距。最后，他结合创办 ZFlow.AI 打造架构级仿真平台的实践，分享了从架构师到 CEO 的转变，以及大模型作为软件业“EDA 工具”对未来工程师职业分化的深远影响。 先进制程升级的核心目标已从提升速度转向降低功耗与发热量；从 3nm 到 2nm 单芯片运行速度提升仅约 3%~5%，而 SRAM 存储单元物理缩放停滞正逼迫行业将焦点转移至系统级架构设计。 (34:18) 大模型推理全流程呈现明显计算瓶颈分期：Prefill 阶段为算力密集型，Decode 阶段为极端显存带宽受限型；通过算存分离、动态调度等软硬件协同设计，软件优化可追回 2 到 3 倍的先进制程物理差距。 (40:55) 英伟达的核心护城河不仅是 CUDA 生态，更在于初创企业般的极致工程执行力（如将 2:4 稀疏概念硬化入量产硅片），以及对台积电 CoWoS 先进封装与全球 HBM 产能分配的供应链绝对主导权。 (62:35) ZFlow.AI 基于虚拟仿真在 8x NVIDIA B200 节点部署 DeepSeek-V3，通过仿真确定 Prefill-Decode 分离架构并注入运行时参数，在物理实测中实现峰值吞吐提升 1.5 倍、P99 尾部延迟降低 2 到 3 倍。 (94:13) AI 大模型正演变为软件行业的“EDA 工具”，传统代码编写被自动化接管导致普通开发者薪资曲线面临平准化，行业超额红利将进一步向打通底层物理计算与上层业务落地的系统架构师集中。 (111:44)

[25:50] 肖志斌: without these three, basically no one could design AI or chips. [23:34] 月球大叔: it's hard to disrupt one's own business [62:35] 肖志斌: NVIDIA is the company with the strongest execution [65:43] 肖志斌: as cheap as tap water [74:02] 肖志斌: 天下没有免费的午餐，你必须决定舍弃什么 [79:44] 肖志斌: 为所有人做嫁衣 [84:09] 肖志斌: 架构师的本质，就是从最初的任务定义出发，在不确定性中去寻找确定性，然后一步步把问题彻底解掉。 [91:39] 肖志斌: fundraising is just a natural byproduct [99:11] 肖志斌: AI Token Factory Simulator [108:57] 肖志斌: solve our own problem of how to optimize and how to design the architecture [114:03] 肖志斌: Everyone needs to become an architect. Put simply, it means you can bridge the gap from top to bottom.

[0:00] NVIDIA is the company with the [0:01] strongest execution in the chip [0:03] industry, and for instance, the annual [0:05] delivery volume of lithography machines [0:07] is also a bottleneck. Yes, people think [0:10] that chips going from 7nm to 5nm or 3nm [0:12] are getting faster, but the real focus [0:15] isn't on speed; it's actually on power [0:18] consumption. [0:19] I seem to recall hearing about a [0:20] company in Silicon Valley that even [0:21] wants to solder base models directly [0:23] onto the chip. [0:24] So I believe the next generation of [0:26] great chip companies will actually be [0:28] AI-native chip design firms. You really [0:31] know your stuff; it feels like your [0:33] brain contains the entire history of [0:35] chip development and a map of every [0:37] manufacturer—it's incredible. Because [0:40] once a chip tape-out fails, you might [0:42] lose hundreds of millions of dollars [0:44] and one or two years of...Present. [0:47] Interviewing 100 interesting people in [0:49] Silicon Valley, this is Uncle Yueqiu. [0:51] Today we have invited Mr. Xiao Zhibing, [0:53] a top-tier chip architect with over 20 [0:55] years of experience. He entered [0:57] Zhejiang University in 1999, later [0:59] pursued his PhD at UCD in the US, and [1:01] has been working in the chip industry [1:03] ever since. Let's welcome Mr. Xiao [1:05] Zhibing. [1:05] Thank you, Uncle Yueqiu, for the [1:07] invitation, and hello to all the [1:09] viewers. Let's dive straight in. Your [1:12] 20-year career path is truly [1:14] fascinating. You started by putting [1:17] accelerators into CPUs, then moved to [1:19] dedicated AI chips, and then to sparse [1:22] chips. Now you've founded a system [1:25] optimization company that doesn't build [1:27] chips. For me, seeing this transition [1:31] makes me wonder if the main battlefield [1:33] for AI performance is shifting from [1:36] transistors and chips to software [1:37] optimization and development. How did [1:41] you make this transition? [1:44] That is a very good question. To answer [1:47] first: yes, the battlefield for AI [1:49] chips has shifted from chip design [1:51] itself to system-level design, [1:53] including the design of software for [1:55] these heterogeneous systems. Actually, [1:59] one fortunate thing over my past 20- [2:01] plus years is that what I've done has [2:03] never strayed from my original field; [2:05] it has always been related to chip [2:07] design. To be more precise, it’s [2:11] hardware-software co-design and [2:13] architectural design. As for why I went [2:16] into architecture and chip design after [2:18] starting as an EE major as an [2:20] undergraduate at Zhejiang University? [2:25] It was because I took courses on [2:27] computer organization and computer [2:29] architecture, and there was a famous [2:31] professor at the time named Patt, who [2:34] invented the branch prediction [2:35] architecture in CPUs, who came to teach [2:38] at Zhejiang University. It just so [2:43] happened that I took his course that [2:45] summer, and from then on, I fell in [2:47] love with computer architecture. [2:51] Looking back at everything I’ve done [2:53] over the past 20 years—from my [2:55] undergraduate, master's, and doctoral [2:57] studies to joining Oracle for CPU [3:00] design, working on database [3:01] acceleration chips, and later at [3:03] Alibaba on the Hanguang 800 and AI chip [3:06] design. After that, I started my own [3:10] company making AI chips, and now I’m [3:12] running an AI infrastructure system [3:14] optimization company. In reality, [3:18] I’ve been doing just one thing all [3:20] along: finding ways to make systems [3:22] more effective and helping them perform [3:24] computations more efficiently. I [3:27] actually started out working on DSP [3:29] processors. Back then, while I was an [3:31] undergraduate and graduate student at [3:33] the university, I focused on [3:35] single-core DSP processors. That was [3:37] around 2002 or 2003, during the time of [3:40] projects similar to the 863 Program, [3:42] where there was a push in China to [3:44] support the independent research and [3:47] development of CPUs and DSP processors. [3:49] I was fortunate to be involved in that [3:51] project since my undergraduate years. [3:54] Then, I continued as a graduate student [3:56] at the same university. Participating [3:58] in the forward-design development of [4:00] such large-scale DSP processors while [4:03] being a graduate student there was [4:04] quite rare. So, at that time, my [4:08] graduate research topic was based on [4:10] MIPS—a traditional processor [4:12] architecture—where I worked on [4:14] extending the multimedia instruction [4:16] set at the MIPS processor instruction [4:19] level. Ultimately, this DSP processor [4:23] was designed for something quite simple [4:25] back then: MPEG encoding for VCDs, [4:28] including both audio and video encoding [4:30] . That was my graduate project. As for [4:33] my doctoral project, I pursued it at UC [4:36] Davis. It transitioned from single-core [4:40] DSPs to multi-core DSPs. The problem we [4:47] were trying to solve back then was [4:49] using a multi-core approach for stream [4:51] computing, known as data computing, to [4:54] handle communication problems, video [4:56] encoding tasks, and some big data query [4:58] and sorting issues. This stream-based [5:06] structure is actually very similar to [5:08] what we now call the dataflow [5:10] architecture used in AI processors. [5:14] It’s about letting the data flow [5:16] on-chip, instead of moving the data [5:18] outside of the chip. The data actually [5:24] flows between many multi-core DSP [5:26] processors, and each DSP processor has [5:29] very, very small instruction and data [5:31] memory, and they are connected together [5:33] through some form of interconnection. [5:39] That architecture was actually very [5:41] advanced at the time; each DSP [5:43] processor could adjust its own [5:45] frequency and voltage. This was my [5:47] doctoral research topic. Well, [5:49] Later at Oracle, we advanced to [5:51] supporting database acceleration [5:54] directly within the CPU. Oracle had [5:58] acquired Sun Microsystems around 2015, [6:00] and I joined Oracle in 2017 when we [6:03] were still developing SPARC processors. [6:08] Perhaps many of our listeners are not [6:10] familiar with SPARC processors. At that [6:13] time, SPARC, along with IBM’s POWER [6:16] and x86, were the primary instruction [6:18] set architectures for large-scale [6:21] server CPUs. These servers, combined [6:24] with the Solaris operating system, [6:26] could run for years without needing a [6:28] reboot; they were extremely stable. [6:31] After Oracle acquired Sun, we wanted to [6:34] innovate, but how? Since our focus was [6:39] databases, we thought about placing [6:41] database operations into the SPARC [6:43] processor as a coprocessor or an [6:45] acceleration unit. By sharing all [6:51] memory bandwidth with the CPU, we could [6:54] saturate the CPU's memory bandwidth to [6:56] enable in-memory databases. At the time [7:00] , we used TPC-H—those in the database [7:03] field would know it—which is a [7:05] benchmark containing 22 queries. We [7:12] hardened these operators into hardware, [7:15] designed an instruction set and runtime [7:17] , and integrated them with the Oracle [7:19] database, effectively creating an [7:21] accelerator for Oracle’s query [7:23] optimizer to speed up certain [7:24] operations. I spent about five or six [7:31] years doing this at Oracle. After that, [7:35] I went to Alibaba to work on their [7:37] first-generation AI chip, and later, at [7:40] my previous AI chip company, we worked [7:42] on sparse AI chips. Now, we are [7:49] building Flow.AI. Through all these [7:51] experiences, we ultimately saw that as [7:54] Moore’s Law slows down, improvements [7:56] in computational performance or power [7:58] efficiency at the single-chip level are [8:01] becoming a bottleneck. The true [8:10] bottlenecks are actually at the system [8:12] and software levels, so if what I’m [8:14] doing is no longer a bottleneck, or if [8:16] many companies are already solving that [8:18] single-chip problem, I should focus on [8:20] solving new bottleneck issues. That is [8:28] why we founded Flow; we use [8:30] architectural approaches and [8:32] simulations to guide AI chip design and [8:34] optimize AI systems. We integrated [8:41] model compilation, runtime scheduling, [8:44] and AI system simulation into a unified [8:46] platform, allowing our clients to [8:48] simulate various hardware optimization [8:50] methods even without the actual [8:52] hardware. Especially with AI clusters [8:59] growing larger, many systems engineers [9:02] often lack the necessary hardware [9:04] support. Therefore, our system [9:11] simulation provides them with an [9:13] excellent tool for system optimization, [9:15] which directly improves the efficiency [9:17] of AI models on hardware. We also [9:24] answered a crucial question: which [9:26] model runs most economically on which [9:28] hardware—hardware that might already [9:31] exist or might not even exist yet. By [9:35] combining simulation with automated [9:38] profiling, we address both sides. To [9:41] answer your question, the bottleneck of [9:44] AI systems today isn't really the AI [9:46] chip, but rather the system and [9:48] software. Oh, thank you, Teacher Xiao! [9:52] Listening to you for so long, it feels [9:54] like you've personally experienced it [9:57] all—from VCDs, multimedia, and big [9:59] data to databases, and now AI. Since I [10:02] studied multimedia, I really resonate [10:04] with that. I used VCDs when I was a kid [10:07] , too. Over the past 20 years, it feels [10:10] like you've been personally involved in [10:12] every major technology-related field [10:14] from the perspective of computer [10:16] architecture and chips. Yes, exactly. [10:19] It’s truly an epic history. Listening [10:22] to you, I get the sense of you working [10:24] from within this grand era. I’m a [10:28] software person, not hardware, but I [10:30] can certainly feel that. [10:32] Over the past 20 years, you’ve put so [10:34] much effort into all these different [10:36] areas of architecture and chips. Right, [10:38] especially what you mentioned about [10:40] transitioning now because Moore's Law [10:42] is nearing its end. With Moore's Law [10:44] failing, you focus more on software. We [10:47] can definitely dive deeper into the [10:49] stories of each stage later. [10:51] My company is actually a system [10:52] optimization firm. In reality, all AI [10:56] chip companies, or those planning to [10:58] make chips, need to do what I am doing [11:00] first. So, I wouldn't rule out jumping [11:03] into chip design if we see the [11:05] opportunity; perhaps we might even [11:07] incubate a chip company. Yes. I [11:09] understand what you just said, which is [11:11] if you run simulations and find that [11:13] certain models run better on a specific [11:15] architecture, you might just go and [11:16] build that architecture yourself, right [11:18] ? Exactly, and models have reached a [11:21] point of convergence. For instance, if [11:25] a model can satisfy our needs for a [11:27] specific task, right? Our needs, like [11:31] for a robot vacuum, or monitoring [11:33] vehicles in a parking lot—if these [11:36] specific tasks don't really change [11:38] anymore, then we can absolutely build a [11:40] chip, a dedicated chip, to accelerate [11:42] that process. Yes. I think I heard [11:46] about a company in Silicon Valley that [11:47] actually wants to hard-wire the model [11:49] directly into the chip. Is that what [11:50] they're aiming for? [11:52] That’s a Canadian company called [11:53] Tenstorrent, and Tenstorrent's founder [11:56] is also their founder. Tenstorrent is [11:59] another AI inference chip company, and [12:01] their current CEO is Jim Keller, the [12:03] legendary Silicon Valley architect. [12:07] Their approach is very radical, but I [12:09] think the reason investors still back [12:11] them is that it represents a trend. [12:17] It's possible that for specific tasks, [12:19] models are becoming more solidified, [12:21] and from an economic standpoint, it [12:23] makes sense to hard-wire them into the [12:25] chip. Of course, their architecture [12:31] actually has some flexibility; it's not [12:33] completely hard-wired. They have [12:37] different metal layers on the chip that [12:39] can be re-fabricated. So it's cheaper [12:43] than doing a full re-spin of the chip. [12:47] It's programmable, but the [12:49] programmability requires sending your [12:51] wafers back to the foundry to redo the [12:53] metal layers. Yes. Wow, the layperson [12:57] sees the excitement, but the expert [12:59] sees the real game. Your ability to [13:01] tell the stories of these chip [13:03] companies is incredible. I have a [13:04] feeling this interview is going to be [13:06] very, very fascinating. Since you [13:10] mentioned you might build chips again [13:12] in the future, as someone who really [13:14] has no idea how chips are made, I’d [13:16] love to understand how a chip is [13:18] actually designed and brought to mass [13:20] production. I see. Actually, chip [13:24] design has a very, very long [13:26] development cycle. When we used to make [13:30] CPU chips, it was basically a [13:32] two-to-three-year design cycle, plus [13:34] fabrication, and finally the system and [13:36] software; it could take three to five [13:38] years to reach the market. Recently, [13:43] Nvidia has really set the pace for [13:45] chips, releasing several new ones every [13:48] single year. Starting to build a chip [13:52] is like building a house; we must [13:54] decide what kind of chip to make. Will [13:57] it be an encoding/decoding chip, a [13:59] communications chip like a specific [14:02] Wi-Fi standard, or something more [14:04] general like a CPU or GPU? So, deciding [14:08] what to build is actually market-driven [14:10] . For instance, you might work with a [14:14] hyperscaler or cloud service provider [14:16] that identifies a specific use case, [14:19] such as requiring custom encoding/ [14:21] decoding chips. Take video streaming [14:26] companies, for example—it's quite [14:28] interesting. Since everyone’s phone [14:33] uses different encoding, incoming video [14:35] comes in all sorts of formats. They [14:42] need transcoding to convert these into [14:44] a unified format for server storage, [14:46] and then convert them again for [14:48] distribution based on individual device [14:50] performance and resolution. That [14:55] requires a specialized chip. This all [14:58] starts with identifying the need. If [15:00] you try to handle this with CPUs, [15:02] it’s extremely expensive. In fact, [15:04] many CPUs don't even have that native [15:07] encoding capability, so you’d need to [15:09] add a massive number of CPU servers [15:11] just for this task. So, it starts with [15:15] requirements development, and once [15:17] those are set, the architects step in. [15:20] Architects must balance the PPA—Power [15:23] , Performance, Area, and Cost. They [15:27] decide how large the chip should be, [15:29] what features to include, and how those [15:31] features compare to the competition. [15:35] Because chip competition is cutthroat, [15:37] the top player basically grabs the [15:39] majority of the market, the second gets [15:41] a small slice, and the third or fourth [15:43] might just get the scraps—or nothing [15:46] at all. That’s why you also need to [15:49] conduct market research on competitors [15:51] to ensure your chip won't be obsolete [15:53] in a year or two. This is the [15:56] architect’s job, and they eventually [15:58] produce the final chip spec. Next, we [16:02] have the micro-architect, who takes [16:04] that chip spec—whether it’s at the [16:06] instruction level or the high-level [16:08] block level—and defines the [16:09] structural implementation. Once these [16:13] structures and functions are clearly [16:15] defined, then the micro-architects take [16:17] the stage. How do you turn it into [16:19] realizable hardware? For the same [16:22] function, say an FFT encoder, there are [16:25] countless ways to implement it, and [16:27] each method has different trade-offs. [16:30] This is where microarchitecture comes [16:32] in; the architecture provides the [16:34] constraints, and the microarchitect [16:36] turns it into a microarchitecture. Many [16:40] microarchitects also write RTL, which [16:42] is the hardware description language, [16:45] translating it into hardware [16:46] programming. After that stage, it's [16:49] time for the verification engineers to [16:51] step in. Actually, before the AI era, a [16:57] chip design company typically had a [16:59] ratio of one design engineer to two or [17:02] three verification engineers, so a lot [17:04] of verification engineers were needed. [17:11] This involves functional verification, [17:14] formal verification, timing [17:15] verification, physical verification, [17:18] and simulation before the chip is [17:20] produced; you have to use FPGA-based [17:22] verification or buy large emulators [17:25] from Cadence, like the Palladium, for [17:27] validation. Verification is therefore [17:31] crucial, because if a chip tape-out [17:33] fails, it could mean hundreds of [17:34] millions of dollars are lost. Not to [17:37] mention one or two years of time wasted [17:39] , so verification is extremely [17:41] important. Moving on, after the RTL is [17:45] implemented, it undergoes synthesis, [17:47] then physical implementation, followed [17:50] by place and route, clocking, power, [17:52] and co-design with system vendors [17:54] regarding power delivery and packaging. [17:59] Once the packaging design is done, you [18:02] send it to the fab—like TSMC or SMIC [18:04] —which takes about three to four [18:07] months. What happens during those three [18:10] to four months? After you send them the [18:14] layout, they create masks, building it [18:16] up layer by layer. You can think of it [18:19] as drawing patterns or printing on [18:21] silicon. The complexity of a chip is [18:25] determined by the number of these [18:27] printed layers. Nowadays, it usually [18:31] requires dozens, or even nearly a [18:33] hundred, of these mask sets. Then, [18:36] after three or four months, your wafers [18:38] return, and you have to find a wafer [18:40] testing facility to check if there are [18:43] any defects, which is different from [18:45] functional verification. Finally, once [18:48] the wafer comes back, you have to dice [18:50] it into individual dies. This is to [18:53] ensure yield, meaning there are no [18:55] defects. The foundry guarantees the [18:58] yield of these dies, and then you send [19:00] them to the packaging house to be put [19:02] into a packaged format. Finally, the [19:05] packaged chip needs to be placed on a [19:07] system board by the system manufacturer [19:10] for testing. Then you conduct [19:12] functional verification and various [19:14] laboratory condition tests, and finally [19:16] , you send samples to the customer. [19:18] After sending the samples, the next [19:20] step is mass production. So, the entire [19:23] process is very long, and the entire [19:25] design cycle is also very long. [19:27] Wow, that sounds so long and so [19:29] complicated, oh my god. [19:30] Yes. [19:31] How long is this cycle, eventually [19:33] reaching mass production? Generally [19:35] speaking, we used to say three to five [19:37] years, but for Nvidia's chips, from [19:39] design to development, it takes about [19:42] two years or so. But some chips now [19:45] iterate every year, though in reality, [19:47] their personnel overlap. It means they [19:50] launch a new generation every year, but [19:52] the R&D process might have actually [19:54] started two to three years ago. Mhm. [19:57] So, what we see now, for example, [20:00] Nvidia's B300 or GB300, might have been [20:03] designed two or three years ago. [20:05] Oh, wow, this is really too long. It [20:08] feels like if there are any slip-ups in [20:10] the middle, it would cause huge delays [20:12] and problems. That's right. From design [20:15] to the entire wafer yield, the supply [20:17] chain, the thermal management of [20:19] packaging and testing, and finally, [20:21] your software having to match [20:23] simultaneously—the probability of [20:25] error is very high. This is why chip [20:29] verification is so important. Speaking [20:32] of verification, let me tell a story. [20:37] Back when Jensen Huang was making chips [20:39] , he had nearly run out of money, but [20:41] he spent the last of it to buy an [20:43] emulator. It was used to verify the [20:47] chip, specifically for verifying that [20:50] GPU at the time. This led to him doing [20:53] this before tape-out, because back in [20:55] the day, when making chips, [20:57] verification tools were still very [20:59] limited. So, chips in the past [21:01] basically had to be taped out several [21:04] times to be successful. But later, [21:06] thanks to these verification methods [21:08] and emulation tools, most chips can [21:10] succeed on the first tape-out, with [21:13] only a small number requiring metal [21:15] fixes. But that is also very normal in [21:18] the chip industry, yes. I’ve noticed [21:21] some companies in Silicon Valley [21:22] starting to use agents to intervene and [21:24] take on roles in design and [21:26] verification. What is your take on [21:28] these companies? [21:29] Oh, I’m very familiar with them, as [21:31] many of them are run by my friends. [21:33] They’re doing a lot, though I won't [21:34] name the companies, right. [21:36] Right, right. I think this is a trend. [21:39] Using AI to design chips is currently [21:41] focused on verification, because [21:43] verification is actually quite similar [21:46] to coding. So it’s a bit like a [21:50] coding agent; it has inputs and outputs [21:52] , and there are verifiable simulated [21:54] environments to check whether you've [21:56] made a mistake. That’s why using AI [21:59] for verification is very common, while [22:01] using AI for forward-looking design is [22:04] still relatively rare. As for using AI [22:07] for physical design—like routing or [22:10] floorplanning—companies like Synopsys [22:12] and Cadence are already using it [22:14] extensively. Yes, so I believe using AI [22:19] to design chips is an inevitable trend, [22:22] but there is still a missing piece: how [22:24] to choose the architecture, what kind [22:26] of chip to make, and how to define the [22:29] chip. Currently, such architectural [22:33] tools are still very scarce, and [22:35] architects still need to play a [22:37] significant role. The second point is [22:40] that chip design companies today face a [22:42] major problem when it comes to adopting [22:45] agents or AI agents. Most have their [22:50] own established workflows, so many [22:52] engineers are reluctant to use AI in [22:54] their current work. If you want to [23:00] replace things, you have to rethink the [23:03] entire process in an AI-native way. So [23:08] I think the next great chip company [23:10] will actually be an AI-native chip [23:11] design company. From architecture and [23:16] design to verification and physical [23:18] design, the entire process must use an [23:20] AI-native approach. Currently, I think [23:27] most chip companies are only adopting [23:29] AI as a tool, but to truly maximize the [23:32] value of AI, it must be a new, [23:34] AI-native chip organization. [23:39] Wow, you are very open-minded. You [23:41] firmly believe that there will be even [23:43] greater chip companies in the future. [23:45] Exactly. [23:46] Having seen the ups and downs of so [23:48] many chip companies and experienced so [23:50] many generations of R&D, you believe [23:52] there will be a huge AI-native company [23:54] later that will produce super, super [23:56] impressive chips. [23:57] Yes, or perhaps only such companies can [24:00] survive in the future. But chip [24:04] companies now have their established [24:06] logic, so it's very hard to do this on [24:08] the existing foundation. [24:09] It's hard to disrupt one's own business [24:11] . [24:11] Oh, right, exactly. Wow, looking at it [24:14] this way, there are so many [24:15] opportunities, right? So, we just [24:16] charge forward, and that's it? [24:17] Exactly, exactly. [24:19] So powerful, so powerful. It’s just [24:21] so inspiring. I don't know how many [24:23] students will go into chips after [24:24] watching this show. Good luck, everyone [24:26] . Right, you mentioned many, many [24:27] points earlier. Since I used to work on [24:29] video network transmission, I generally [24:31] know the process you described, but I [24:33] had never considered it from the [24:34] perspective of chip design. However, I [24:36] have heard from many friends [24:37] at YouTube, they said that in order to [24:39] save on YouTube's massive annual video [24:41] encoding and decoding costs, Right, [24:43] they definitely need to create a lot of [24:45] customized hardware for this, because [24:46] that demand is certainly there. [24:48] Demand, yes, and from a TCO perspective [24:50] , the math actually works out. [24:52] Right, it's too big, it's the world's [24:54] number one video company. And the [24:56] second point is, while I was doing my [24:58] PhD, I often heard the word FPGA. Hmm, [25:01] Right, you mentioned FPGA just now as [25:02] well. Yes, [25:03] but it doesn't seem to be as popular in [25:05] the last two years. [25:06] Oh, right. [25:06] Do you have that feeling? Anyway, I [25:08] feel it’s not that popular. Uh, [25:10] actually, it's not currently used for [25:12] mainstream AI data center inference; [25:14] it's just not as effective. But in [25:17] areas like chip verification and [25:19] emulation, it is still used extensively [25:21] . In some scenarios where there is a [25:25] high demand for programmability, it is [25:27] still widely used, even for [25:29] interconnects, such as chip-to-chip [25:31] connections. If the protocol is [25:35] constantly changing, then it might not [25:37] be...once the workload is fixed, it is [25:39] not the most efficient implementation. [25:44] However, for these highly fragmented [25:46] scenarios, FPGAs are still used in many [25:49] cases, yes. Understood. One more [25:51] question: you talked about many roles [25:53] earlier, like architects, [25:55] micro-architects, and verification [25:57] engineers. [25:57] I've often heard the claim that the US [25:59] possesses the world's most advanced [26:01] chip technology, [26:03] and that companies like NVIDIA and [26:05] Apple are exceptionally good at design [26:07] and architecture, [26:08] and yet, they have offloaded all the [26:10] manufacturing... [26:11] Mm-hmm. [26:12] It's outsourced to TSMC, and everyone [26:14] says the US excels at chip design, [26:16] and TSMC excels at chip manufacturing. [26:18] Is this statement correct? [26:20] I think it is correct. In fact, what [26:24] the US truly possesses is IP and EDA. [26:28] Second is architecture and design, like [26:31] RISC-V, which also started in the US. [26:34] Although ARM's architecture is British, [26:37] there's also the X86 architecture and [26:39] instruction-set IP. Then there are the [26:43] EDA companies like Synopsys, Cadence, [26:45] and Mentor Graphics, the three largest [26:47] EDA firms. Without these three, [26:50] basically no one could design AI or [26:52] chips. [26:53] What does EDA stand for? [26:55] EDA is Electronic Design Automation, [26:57] right. [26:58] Oh, what does it mainly do? [27:01] It helps with every step I mentioned, [27:03] which engineers accomplish by working [27:05] with EDA tools. [27:06] So, it's a specialized tool just like [27:09] MATLAB. Exactly. For example, there's [27:13] less done for architecture design, but [27:15] Synopsys has tools like Platform [27:17] Architect. Then, for converting RTL to [27:22] netlists, there are tools like [27:24] Cadence's Genus. For RTL verification, [27:27] there are simulation tools like VCS. [27:31] For IC design, there's Place&Route, [27:34] plus timing verification tools, formal [27:36] verification tools, and even EDA tools [27:39] for package design. Basically, for [27:42] every job function, there's a [27:44] corresponding tool helping the engineer [27:47] do that work, right. [27:48] Are these tools expensive? [27:50] These tools are sold at a very high [27:52] price, and some things are only [27:54] available from these few companies, yes [27:56] . [27:56] Mm. Is the reason they are the only [27:58] ones who have them because they have [27:59] accumulated a vast amount of knowledge [28:01] and intellectual property in these [28:03] tools? [28:03] Actually yes. The EDA industry has gone [28:06] through years of development with [28:08] large-scale mergers and acquisitions. [28:11] These companies acquired many, many [28:13] small startups, eventually [28:15] consolidating into these few giants. [28:18] That's the first point. The second [28:21] point is that the backend—the [28:22] physical design tools—are actually [28:24] designed in collaboration with the [28:26] foundries. So, your physical design [28:30] tools and the manufacturing nodes we [28:32] mentioned, like nanometers, are [28:34] developed together. It requires [28:37] cooperation from the foundries, with [28:39] everyone working together. So, this [28:45] includes the accumulation of capital [28:47] and industry M&A, as well as the [28:49] experience gained from collaborating [28:51] with these physical design and [28:53] manufacturing partners. Mm. [28:59] Wow, the ins and outs of this are so [29:01] profound. Are other countries, [29:03] including China, working on these [29:05] things as well? [29:07] Uh, yes, because the semiconductor [29:09] supply chain is very long. There's [29:13] foundry, which includes logic [29:14] manufacturing plants and memory [29:16] manufacturing plants. [29:18] What does "foundry" mean? [29:20] Foundry actually refers to chip [29:23] manufacturing plants, like TSMC. Yes, [29:26] they all—and in China, there's SMIC, [29:29] and YMTC which does DRAM, and there are [29:32] those doing NAND. There are many [29:34] companies in China, including some FPGA [29:37] companies. Between China and the US, [29:40] they have the most complete supply [29:42] chains, but other countries play a part [29:45] —like ASML in the Netherlands making [29:47] lithography machines, right? And Japan [29:51] has Tokyo Electron for semiconductor [29:53] equipment, and some companies in Japan [29:55] making materials. These are all [29:58] indispensable links in the [30:00] semiconductor industry. Uh, like the UK [30:03] has Arm, right? Arm provides IP. So, [30:07] the most complete categories are really [30:09] between China and the US, but Japan and [30:12] South Korea have their own specialized [30:14] areas, whether it's foundry, equipment, [30:17] or materials. Yes. [30:19] Wow, you really know this like the back [30:21] of your hand. It feels like your brain [30:23] contains the entire history of chip [30:24] development and a map of all the [30:26] various manufacturers. That’s [30:27] incredible. You also mentioned [30:28] something called RISC-V earlier. [30:30] Since I often look at the work coming [30:32] out of Berkeley, I know they are [30:33] particularly good at open-source, [30:35] open-source software, and various RISC [30:37] architectures. But I also know RISC-V [30:39] is an open-source instruction set [30:41] architecture for chips. Right. [30:42] Why do you think they wanted to create [30:44] an open-source instruction set? [30:47] Uh, I think—I remember having some [30:50] discussions with some of the founders [30:52] of RISC-V. At the time, they wanted to [30:57] do some architectural research or [30:58] development, but they had to use a [31:00] licensed architecture. Moreover, it was [31:06] aimed at mobile applications, and it [31:08] was quite expensive and not open enough [31:11] for academic research, so they decided [31:13] to promote this open-source ecosystem [31:15] and build a new instruction set. And a [31:20] new instruction set has many benefits. [31:24] You can design it from scratch, and [31:26] it's highly scalable, designed very [31:27] well, and doesn't need to be compatible [31:30] with a lot of legacy stuff. It’s [31:34] geared toward new workloads, so as for [31:36] this open-source instruction set, I [31:38] think the industry is already using [31:40] RISC-V processors extensively. Even in [31:43] NVIDIA's own GPU chips, there are [31:46] actually many controllers or processors [31:48] that make extensive use of RISC-V. Wow, [31:51] I really didn't know that. But I do [31:53] know there's a lot, both domestically [31:55] and over here, it feels like there’s [31:57] an industry alliance. Yes, there's [31:59] something called the RISC-V Summit, and [32:02] the RISC-V alliance. Many of the [32:05] founders of RISC-V startups are [32:07] actually our friends—our good friends [32:09] , both domestically and abroad. In the [32:12] US, there's SiFive, and other RISC-V [32:15] companies led by them. Then in Taiwan, [32:18] there's Andes, who also do RISC-V. And [32:20] domestically, there are quite a few [32:22] others; many people from the Alibaba [32:24] ecosystem have come out to make RISC-V [32:26] chips, and they are also our alumni or [32:28] friends. Right, it feels like this [32:30] circle is quite small. It’s like, [32:32] everyone more or less knows each other, [32:34] right? [32:34] Ah, yes. Because the chip industry is [32:37] relatively close-knit due to the [32:39] upstream and downstream relationships. [32:41] Even with the current tight US-China [32:43] geopolitical situation, everyone still, [32:45] well, there’s actually still a lot of [32:48] coordination and cooperation when it [32:50] comes to open source. Yes. [32:52] That’s really great, that feeling. [32:54] And I often see that at chip industry [32:56] conferences or academic conferences. [32:59] Hmm. [32:59] Most of the Chinese faces feel like [33:01] they're all schoolmates or colleagues. [33:02] Yes, yes, that’s right. [33:04] Right, and you just mentioned the [33:05] various nanometer processes. [33:06] Let’s talk about this issue of [33:08] nanometer processes. [33:08] Yes, yes, yes. [33:10] I see it in the news every day, all [33:12] kinds of reports: 2nm, 3nm, 5nm, 7nm. [33:14] What do these nanometers actually mean? [33:18] Uh, actually, in the past, a process [33:20] node—like a certain number of [33:22] nanometers—really corresponded to a [33:24] physical parameter in the manufacturing [33:27] process. That was roughly before the [33:31] 65nm node. We know that all our chips [33:37] are composed of transistors, and most [33:39] transistors now have three terminals. A [33:44] gate, a source, and a drain. The [33:48] nanometer measurement used to refer to [33:50] this gate length. Before 65nm, that was [33:54] the case. But after that, this " [33:56] nanometer" designation has become more [33:58] of a marketing term, meaning it [34:00] represents a generation of technology. [34:03] This is not a single-point technology, [34:05] nor does it represent just one physical [34:07] parameter. It could represent a new [34:13] type of transistor, like a FinFET, and [34:15] then its metal pitch, which is because [34:18] wires are stacked on the silicon— [34:19] that’s the metal, the copper [34:21] interconnects or metal interconnects. [34:27] The width of the metal interconnect at [34:29] the bottom-most layer is what's called [34:31] the metal pitch. This pitch is also a [34:36] technical parameter, as is the cell [34:38] size of standard cell gate-level [34:40] circuits, and the density of SRAM, [34:42] which is a type of on-chip memory. All [34:47] these parameters together define a [34:49] generation of technology. Manufacturers [34:52] label them 7nm, 5nm, or 12nm, but in [34:54] reality, there is no single physical [34:57] parameter corresponding to these. [35:00] Moreover, each manufacturer—like [35:02] Samsung, Intel, or TSMC—has 7nm [35:05] processes that are not truly comparable [35:08] to one another. What’s most important [35:13] to compare is, for instance, your [35:15] standard cell—a standard cell is a [35:17] gate, like an AND or OR gate—the area [35:20] of these basic logic units, or the [35:22] density of your SRAM, or the SRAM size, [35:24] or the PPA under a certain design. This [35:30] is what people compare; it’s hard to [35:32] compare processes directly, even if [35:35] they are both called 7nm. [35:37] For us ordinary people, after hearing [35:39] you list so many technical terms, we [35:41] still don’t really understand them, [35:43] but we know it’s very impressive. So, [35:46] regarding the progress of the process [35:48] node itself, for example, moving from [35:50] 7nm to 5nm, [35:51] or from 5nm to 3nm, [35:53] right. What exactly does it solve, what [35:56] has improved, and what problems remain [35:58] unsolved? [36:00] Right, there is actually a [36:01] counterintuitive issue here. People [36:04] think 7nm to 5nm or 3nm just means the [36:07] chip gets faster, but the real focus [36:10] isn't on speed; the real focus is on [36:13] power consumption. That is to say, at [36:17] the same performance level, power [36:19] consumption can potentially be cut in [36:21] half, or the changes—the benefits— [36:23] in power consumption between [36:25] generations are far greater than the [36:27] changes in speed. For example, for TSMC [36:32] , perhaps from N3 to N 2—the 3nm to [36:36] 2nm node—the power savings might be [36:39] 10%to 15%, while the speed increase [36:42] might only be around 3%to 5%. So, [36:46] between each chip generation, power [36:48] efficiency is actually the most [36:50] important factor; power consumption [36:52] decreases significantly while [36:53] maintaining the same performance. [36:55] This is truly the first time I've heard [36:57] this particular phrasing. [36:58] Right, and once power consumption drops [37:01] , in the same rack, for example, you [37:03] can pack in more compute, so the [37:05] computing density can increase. So [37:07] Because the heat dissipation [37:09] requirements also decrease, right? The [37:10] heat generated decreases. [37:12] Yes, the heat also decreases. [37:16] Previously, before reaching 5nm or 7nm, [37:19] power consumption and area actually [37:21] followed what is called Dennard scaling [37:24] , meaning your supply voltage—the [37:26] chip's voltage—would keep dropping [37:29] along with the area. But now, because [37:35] all gates and all transistors have a [37:37] baseline voltage. It can't go down any [37:40] further. So. As everyone knows, a [37:42] chip's power consumption is divided [37:44] into two. Parts: one is static power [37:46] consumption, and the other is dynamic [37:48] power consumption. Static power [37:50] consumption is, well, you can think of [37:52] all these logic circuits as a gate. [37:56] Static power consumption is what you [37:58] get when it is in a stable state, [37:59] whether it's on or off. The power it [38:01] generates then, and dynamic power [38:03] consumption is the power generated [38:05] during the switching state, which is [38:07] called dynamic power consumption. [38:09] Before, at 12nm or even above 16nm, it [38:11] was mainly dynamic power consumption, [38:13] but later static power consumption [38:16] became larger and larger, meaning [38:18] leakage current effects increased. So, [38:20] static and dynamic power consumption is [38:23] a bit complex. [38:24] Hmm, no worries, please continue. I [38:26] think everyone is trying their best to [38:27] follow your line of thought. It's truly [38:29] great because I feel you've already [38:30] tried your best to use the most [38:31] accessible language to help everyone [38:33] understand; people can just watch it a [38:35] few more times to digest it, right? Yes [38:38] , because the technology stack in this [38:40] chip industry is particularly deep, and [38:42] there are just too many technical terms [38:44] , so I'm explaining as much as I can. [38:48] But what everyone should remember is [38:50] that for the entire chip process, for [38:53] instance between 7nm and 2nm, what [38:55] people are mainly talking about is the [38:57] issue of power; it's really about [38:59] reducing power consumption, hmm. [39:01] Therefore, for a 2nm chip compared to a [39:03] 7nm chip, the 2nm chip's power [39:05] consumption might only be a fraction of [39:07] the other's. And because its power [39:09] consumption and heat are lower, you can [39:11] pack more into a single server rack; [39:12] this single rack can then generate [39:14] greater capability, and there's no need [39:16] to move as much data around since it's [39:18] all together. Exactly, exactly, but if [39:21] we go into detail, we could talk for a [39:23] long time. For example, taking AI chips [39:26] , you know that half of the chip is [39:28] logic and half is SRAM, and SRAM is a [39:31] type of on-chip memory. [39:34] And the scaling of this SRAM stops [39:37] somewhere between 7nm and 3nm. As [39:41] process technology goes further, the [39:43] cell size of SRAM doesn't change, which [39:45] is a point many people might not be [39:47] fully aware of. [39:49] Not aware. [39:50] Yes, and the logic portion can indeed [39:53] continue to scale down. Mm. [39:55] So, here is a question: what does this [39:58] mean? [39:58] What does this mean? [40:00] This is very interesting. It brings us [40:04] back to the perspective of AI models, [40:06] where AI model serving involves prefill [40:08] and decoding. Mm. [40:17] If it's compute-intensive, it's helpful [40:20] ; but if it's a memory-bound problem— [40:22] like for companies such as Groq or [40:24] Cerebras, which rely heavily on on-chip [40:27] SRAM—if SRAM cannot scale, then their [40:29] chip design philosophy must change. [40:39] Right. So they have to do other things. [40:43] Yes, or their architectural ratio will [40:45] be different, or they need to do [40:47] something else, such as 3D stacking or [40:49] other things. [40:51] It’s getting too deep; let’s see if [40:53] we can figure this out together later. [40:55] Yes, mm. [40:55] Before we go deep into the interaction [40:58] between these chips and AI, we can [40:59] first provide a brief overview for [41:01] everyone. [41:02] Mm. [41:02] For example, I host a large model in a [41:05] data center. [41:05] Yes. [41:06] Then a user sends a prompt. Yes, yes. [41:09] To this model. [41:10] Yes. [41:10] So, how do the chips in the entire data [41:13] center interact to finally spit this [41:15] out, and keep spitting it out [41:17] continuously? [41:18] Uh, that's actually a very good [41:20] question. We can walk through it; for [41:23] example, after a user says something, [41:25] you generate a prompt, right? Then this [41:27] prompt must pass through your device, [41:29] be sent over the network, and arrive at [41:31] the other side's gateway via the [41:33] network. Then after the network, the [41:37] first step on the server side is a [41:39] large scheduler, meaning your input or [41:41] your prompt, the question you raised, [41:44] has to go through a queuing process. [41:50] Then the scheduler places you into a [41:52] queue, based on the GPU or computing [41:54] resources available. The scheduler then [41:59] solves the problem of what to do next, [42:02] specifically where to send you—which [42:04] server or which GPU. As for the [42:11] question you asked, if there is context [42:13] , there is also an RAG process, meaning [42:16] the relevant data is retrieved from [42:18] your historical context. This data [42:26] might be in a database, on a disk, or [42:28] if it's an ongoing conversation rather [42:30] than one from days ago, it might be in [42:32] the GPU memory. You then combine that [42:39] part with your new question to organize [42:41] an input prompt. Once organized, this [42:46] context needs to be sent to the large [42:48] model. But where is the large model [42:51] actually located when it receives it? [42:55] There is actually a runtime—a piece [42:57] of software waiting for your input. [43:02] Once it receives your context from the [43:04] scheduler, this runtime begins the next [43:06] set of actions. What does it do? It [43:12] loads the model and performs the [43:14] prefill action, which involves sending [43:16] your entire context to the model. The [43:23] runtime loads your model from memory or [43:25] another location into the GPU, and at [43:27] the same time, it can perform batching [43:29] with your context, processing multiple [43:32] user requests together. This is called [43:36] the prefill stage. This prefill stage [43:39] is compute-intensive and allows for [43:41] batching. [43:43] Since there are no dependencies between [43:46] tokens, it is compute-intensive and [43:48] determined by the GPU's computing power [43:50] . After completing this part, it [43:54] generates a magical thing called KV [43:56] cache. Yes, we can refer to Mr. [43:59] Jincheng's explanation of KV cache in [44:02] the second video. [44:04] Right, and this KV cache will be stored [44:06] somewhere, potentially in the GPU, or [44:09] if it's too large, it might be stored [44:11] in storage like flash or an SSD. Once [44:15] the KV cache is ready, the next part is [44:18] the decode stage, where the large model [44:20] starts outputting answers. The answer [44:25] is generated one token at a time, so [44:27] this is called the decoding phase. This [44:33] decoding phase is actually quite tricky [44:35] ; it is a memory-bound, or [44:37] memory-intensive, task. It has to load [44:42] all these KV caches into the GPU and [44:44] process all the weights of the large [44:46] model, just to generate a single token. [44:52] And between tokens, after generating [44:54] one, it creates a new column in the KV [44:57] cache, which must be stored in the [44:59] original KV cache. Then this new KV [45:05] cache and the model must be re-read and [45:07] sent out again to generate the next [45:09] token. So just imagine the massive [45:14] amount of data movement involved; this [45:17] is why data transfer and memory [45:18] bandwidth are such a bottleneck. And at [45:22] the same time, there is another issue, [45:24] which is the interconnection problem. [45:27] This is because current large models, [45:29] such as open-source ones like DeepSeek [45:32] V3 with 671 billion parameters, or Qwen [45:35] 2.5 with 72 billion parameters, are so [45:38] huge that you basically have to use [45:40] distributed inference. So, these KV [45:44] caches or model weights must be sent to [45:47] different GPUs or servers, which then [45:49] requires interconnection and [45:51] communication. You also have to match [45:57] the interconnect communication time [45:58] with the computation time so that your [46:00] pipeline doesn't suffer from "starving, [46:03] " meaning it doesn't stall due to the [46:05] slowest link. So, there is a vast [46:09] amount of runtime work involved, which [46:11] is exactly what many AI inference [46:13] companies are focusing on now. Every [46:16] bit of such optimization can squeeze [46:18] out a little more performance, [46:20] resulting in faster inference or better [46:22] memory efficiency. [46:25] Exactly, so the scope for optimization [46:27] here is huge. For example, NVIDIA [46:31] claims that for their B200 and certain [46:33] models, like an open-source 120B model [46:36] ......they managed to improve [46:38] performance by five times through [46:40] software optimization after just two [46:42] months. Five times! This is... [46:46] Right, OpenAI also mentioned yesterday [46:48] that they improved the inference [46:50] performance of their own advanced GPT- [46:52] 4o model significantly as well, right? [46:56] Yes, there are actually many layers of [46:58] optimization involved here. There is [47:01] kernel-level optimization, runtime [47:03] scheduling optimization, compiler-level [47:06] model optimization, and also resource [47:08] scheduling optimization. Actually, to [47:11] do optimization well, you really need [47:13] to get the abstractions right. Once the [47:17] abstractions are solid, you can hand [47:19] many things over to AI to handle. [47:21] Exactly. [47:23] Then, what will be the career [47:24] development path for these optimization [47:26] engineers in the future? [47:28] Uh, I definitely think so, and that’s [47:31] exactly why we started the company we [47:33] have now. We really want to automate [47:36] this entire process, but we’re [47:38] actually missing a few pieces. For [47:41] example, we first built an AI system [47:43] simulator, and then integrated [47:45] simulation with optimization. We also [47:49] have an automated profiling tool. [47:53] It’s not just simulation—we [47:55] simulate pre-compile static behaviors, [47:57] dynamic behaviors, and aspects [47:59] involving underlying GPU hardware, like [48:02] kernel performance and kernel launch [48:04] overhead. We need all of this to be [48:08] automated. Then we take that data to [48:13] calibrate our simulation model. So, [48:15] we’ve built this complete suite of [48:17] tools that includes simulation, [48:19] compiler optimization, runtime [48:21] scheduling, and profiling. Our goal is [48:24] essentially to solve this optimization [48:26] problem. Current AI still often cannot [48:30] achieve full automation, or it can't [48:33] perform these tasks on virtual hardware [48:35] ; it requires real hardware for more [48:37] iterations. Mm. [48:40] Real hardware means the cost is too [48:42] high, right? [48:43] Real hardware is just too expensive, [48:45] and you also need a real system [48:47] deployed somewhere. And if you have [48:50] that, you can't really do this kind of [48:52] "if-else" style exploration either, [48:54] right? Yeah. Right, but based on your [48:57] description, it feels like... [48:58] Once AI develops for a few more [49:00] generations and captures these specs [49:02] accurately, it seems like it could [49:03] handle the optimization itself. [49:05] Uh, it could, because our company is [49:07] also using AI agents to chain together [49:09] the tools we've built, and it's [49:11] actually doable. [49:12] AI-native, right? [49:13] Exactly, that's right. [49:15] Right, and there’s another question: [49:17] in your description, you mentioned [49:19] large-scale computing, SRAM, DRAM, HBM, [49:21] and things like that are... [49:23] These are the recent "memory walls," [49:25] right? Right. [49:26] Could you explain to everyone which [49:28] part exactly causes this "memory wall"? [49:31] The memory wall is primarily caused by [49:33] models becoming increasingly larger. [49:35] Storage is divided into memory, such as [49:38] HBM and DRAM, and NAND, like the SSDs [49:41] made by companies like SanDisk. The [49:46] reason for the memory wall is, first, [49:48] the DRAM usage when serving models— [49:50] since models are getting bigger. Second [49:55] , the intermediate KV cache is also [49:58] growing, and the KV cache is positively [50:00] correlated with your context length and [50:02] the number of user concurrent requests. [50:07] This involves DRAM, because HBM is also [50:10] a type of DRAM. Another point is that [50:15] we humans are generating so much data, [50:17] and it has to be stored somewhere; plus [50:19] , we need many checkpoints for model [50:21] training. This data is actually growing [50:27] more and more, and content generated by [50:29] AI—including direct output and [50:31] intermediate KVC—may all need to be [50:33] offloaded and stored somewhere. This is [50:37] why storage is becoming increasingly [50:39] crucial. Right, although it has [50:41] plummeted again these past two days, we [50:43] won't go into those specifics, exactly. [50:45] Investing is actually quite interesting [50:48] ; it’s related to the industry, but [50:50] it has its own set of rules and issues [50:52] regarding capital flow. [50:54] This is not investment advice, so [50:56] everyone should manage their own [50:58] affairs. Also, as you just mentioned, [51:01] there's all this data movement, chips, [51:03] and bandwidth. [51:04] Right. [51:05] And so on, mhm. [51:06] A while back, NV acquired that company [51:09] that made TPUs, which is Groq. [51:12] They make what they call an LPU. [51:13] What do you think is the main problem [51:15] they are trying to solve? [51:18] Uh, it’s really about solving the [51:20] fact that GPUs aren't naturally suited [51:22] for this decoding mode. Right, it's too [51:25] obvious; it was designed for training [51:27] from the start. [51:28] Exactly, so a GPU is basically what you [51:30] could consider a high-throughput [51:32] streaming machine. [51:33] It's great at parallel processing. [51:35] Ah, it’s great at parallel processing [51:37] , so it’s very suitable for [51:38] prefilling. [51:39] Right. [51:39] But it’s not suitable for decoding. [51:42] The ratio of computing resources isn't [51:45] right because the GPU is mostly idle, [51:47] and it's constantly moving massive [51:49] amounts of data back and forth using [51:52] HBM. So why is Groq better at decoding? [51:58] Actually, the architecture of Groq is [52:01] very similar to the Manycore DSP [52:03] processor I worked on during my Ph.D. [52:05] Essentially, all task data resides [52:07] permanently inside the chip, and [52:09] through this dataflow pipeline approach [52:11] , a task passes through different parts [52:14] of the chip—let's call them cores— [52:16] and through continuous pipelining and [52:18] software scheduling, it naturally [52:20] avoids the need to frequently move data [52:23] back and forth between on-chip and [52:25] off-chip memory. When data resides [52:30] permanently inside the chip, it's [52:31] efficient. And while GPUs aren't really [52:37] suitable for this, people are now [52:39] trying to simulate these effects—this [52:42] so-called dataflow computation—so [52:44] they've made many optimizations like [52:46] recent FlashAttention; all of this is [52:48] actually to keep data inside the GPU as [52:50] much as possible and minimize movement. [52:58] But ultimately, the GPU is not a [53:00] dataflow machine, so NVIDIA needs an [53:02] architecture similar to Groq's dataflow [53:04] to assist it. Actually, just two days [53:11] ago, I attended AMD's Advanced AI event [53:14] , where AMD is collaborating with [53:16] another similar company, Cerebras, on [53:18] an inference cluster with a [53:20] pre-configured architecture. Right, so [53:25] what is the difference between Cerebras [53:27] and Groq? Actually, in terms of [53:30] architecture and logical design, they [53:33] are very similar; both use this kind of [53:35] on-chip SRAM dataflow architecture. [53:40] They try to keep data resident within [53:42] the chip as much as possible, allowing [53:44] tasks to be mapped onto a single chip [53:47] without needing to access external HBM [53:49] or other DRAM. Then, Cerebras is [53:52] actually doing this wafer-scale [53:54] integration. They’ve developed many [53:57] innovative technologies. Usually, when [54:01] we talk about producing chips, we map [54:03] them onto a wafer, cut them into [54:05] individual dies, and then package those [54:08] dies. Afterward, we connect them via [54:11] system-level interconnects. But [54:16] Cerebras says they don't need to do [54:18] that; they take the whole wafer for [54:20] packaging directly, and the chips are [54:22] interconnected through pre-designed [54:23] traces on the wafer, so each chip [54:25] doesn't need to be cut or packaged [54:27] individually. This so-called [54:35] wafer-scale approach has many benefits, [54:37] the main one being that chips don't [54:39] need to be re-packaged or connected [54:40] through board-level wiring outside the [54:42] package. Instead, they are directly [54:48] interconnected via the wafer-level die. [54:50] But this also brings many engineering [54:52] problems, such as power delivery. [54:55] Originally, you could power a chip from [54:57] its edges, but if you have a whole [54:59] wafer, you don't have enough edges to [55:01] provide power, so you have to innovate [55:03] in power delivery. Too thin, or what? [55:06] Yes, exactly. [55:07] I mean, it's too thin, just like that. [55:10] No, for a single chip, power is [55:12] supplied through the pins on the sides, [55:15] but for a whole wafer, you can only [55:17] supply power to the chips around the [55:19] periphery. So you are forced to [55:21] implement vertical power delivery. The [55:23] second challenge with vertical power is [55:25] that the wafer is very large, so you [55:27] have to handle heat dissipation— [55:29] specifically, how to manage the cooling [55:31] . And the third issue is that every [55:35] chip has a yield rate; traditionally, [55:37] we cut them into dies and discard the [55:39] defective ones, only packaging the good [55:41] dies or chips. But there are bound to [55:46] be some bad chips within that wafer. So [55:50] they have to design an interconnection [55:52] mechanism to bypass those bad chips. [55:56] There’s a lot of engineering effort [55:57] involved there. From my perspective, I [56:01] think Cerebras 'wafer-scale engine is a [56:04] very cool technology, but you don't [56:06] necessarily have to do wafer-scale, [56:08] right? [56:09] There are actually other ways to solve [56:11] this problem. [56:12] Exactly, exactly. [56:13] Are you interested in starting another [56:15] major company to do this? [56:17] Actually, there are some new [56:19] technologies, like 3D stacking or using [56:21] high-speed interconnects to solve the [56:23] scaling problem. So, you personally [56:25] aren't that optimistic about Cerebras, [56:28] right? [56:29] Uh, I think Cerebras has its advantages [56:31] too. Because many chip companies now [56:34] can find specific application scenarios [56:36] where the volume is large enough. At [56:40] the same time, the technologies they've [56:42] been working on are quite innovative, [56:44] like vertical power delivery; most [56:46] chips in data centers today are [56:47] starting to move toward vertical power [56:49] delivery. So, many of their [56:52] technologies are actually ahead of [56:54] others. Perhaps they might design [56:57] something new; we don't know yet, right [56:59] . [56:59] Right, because a lot of innovation [57:00] comes from these kinds of experimental [57:02] and disruptive approaches. [57:04] Exactly, exactly, exactly. Hmm. [57:06] So I think it’s a very great company, [57:09] or a very, very excellent company. [57:11] Designing like that under those [57:12] circumstances, and having TSMC [57:14] cooperate—meaning the manufacturing [57:17] side had to work with them to pull this [57:19] off—was really not easy. And didn't [57:22] it come out a while ago that Sam Altman [57:24] personally invested in a lot of these [57:26] companies early on? [57:28] Uh, that's right. They also saw the [57:33] situation, which follows a similar [57:35] logic to NVIDIA; the Cerebras [57:36] architecture is very well-suited for [57:39] decoding, specifically for this part of [57:41] large model decoding. Yes, we see these [57:46] new chip companies releasing many demos [57:48] where the decode speed is just [57:50] phenomenal. [57:50] Ah, yes, yes, yes. [57:52] Very. [57:53] Yes. It’s impressive. In Silicon [57:55] Valley, from my layman's perspective, [57:57] I've noticed three such chip companies: [58:00] Groq, Cerebras, and SambaNova. What [58:02] does SambaNova do? SambaNova was [58:05] actually founded by many of my former [58:07] colleagues from Oracle. After we left [58:12] Oracle, many people went to SambaNova, [58:14] including the CEO, who is a good friend [58:17] of mine. They were really looking to [58:21] build a dataflow company. It's about [58:25] using a dataflow architecture for AI. [58:28] The initial intent, besides Dataflow AI [58:30] , also included a database function. Of [58:34] course, that database feature wasn't [58:36] widely used later, so many people don't [58:38] mention it anymore. Secondly, its [58:42] architecture isn't as aggressive; it's [58:44] not pure SRAM. While SRAM takes up a [58:47] large portion, it also includes DDR or, [58:50] later on, HBM. It’s a compromise [58:53] architecture. It's not like Cerebras, [58:56] where when a model gets too big to fit, [58:59] it requires more chips. So, it is a [59:02] compromise—a dataflow architecture [59:04] that still incorporates DRAM and HBM. [59:08] Wow, these three companies really have [59:10] their own strengths. [59:11] Exactly. [59:12] Mm. Anyway, with so many chips [59:14] available, plus various manufacturing [59:17] processes, [59:17] and all kinds of different technologies [59:19] . [59:20] At the software level, the algorithm [59:22] level, the data level, [59:24] and this infrastructure level. [59:27] Right. [59:28] How much of this hardware performance [59:30] gap can actually be recovered? For [59:32] example, [59:32] a 2nm or 5nm process, [59:35] Right. [59:35] chip, [59:36] for a large model, [59:37] Right. [59:37] versus the other side that can only use [59:40] 7nm or 12nm. [59:41] Mm. [59:42] How much can the software level help [59:44] bridge that gap? Right. First, we need [59:47] to look at where 7nm actually differs [59:50] from [2nm/5nm], such as the power [59:52] consumption of logic transistors at the [59:55] same performance level—that is, [59:57] compute density—and second, its [60:00] interconnects, such as the supported [60:02] SerDes speed. So, for the difference [60:07] between 7nm and 2nm, I think the [60:09] performance gap is roughly two to three [60:11] times, around three times; you can [60:14] consider that the absolute physical [60:16] limitation. But there is a catch: as I [60:20] mentioned earlier, SRAM doesn't scale [60:23] well from 5nm to 3nm, so the benefits [60:25] of that part might need to be [60:27] discounted. From a purely software [60:35] perspective, if we look at things like [60:38] compute-memory separation, [60:39] low-precision quantization, speculative [60:42] decoding, multi-token prediction, and [60:44] real-time scheduling optimization, this [60:46] part can actually recover about two to [60:49] three times the performance. There is [60:56] another part, which is utilization. [60:59] Through software optimization, we can [61:02] improve the industry-average [61:04] utilization from, say, 30%to 50%, up to [61:07] 80%to 90%. This accounts for another [61:09] part of the gap. I think from a [61:13] software perspective, you can actually [61:15] catch up, but it's hard to say. For [61:18] instance, with the same AI, if I have a [61:21] 2nm chip and you have a 7nm process [61:24] chip, I can still apply the same [61:26] software optimizations. So, if you [61:31] haven't optimized your software well, [61:33] it’s entirely possible to boost the [61:35] performance of your hardware to match [61:37] the level of a process that hasn't been [61:39] optimized. [61:42] In that sense, from a positive [61:43] perspective, no matter what chip you [61:45] get, there is always plenty of room for [61:47] optimization. That's right. [61:48] But the downside is [61:50] Mm. [61:50] Both have room for optimization; even [61:52] the 2nm one has room to improve. [61:54] You can move forward. Yes. [61:56] Yes, both can move forward. It’s hard [61:58] to say. [61:59] Yes, or perhaps some people don't need [62:01] such refined software because their [62:03] chip iteration is too fast, and [62:05] software actually has a cost. So they [62:09] might not need such detailed [62:11] system-level co-design. But if your [62:15] process is restricted, you actually [62:17] need to do more of this system-level [62:19] co-design, and the finer you go, the [62:21] more benefits you get, though it comes [62:23] with higher human and resource costs. [62:26] Mm. [62:26] Mm, that is also why many [62:28] resource-constrained companies are able [62:30] to come up with some great innovations. [62:32] Uh, exactly, exactly, exactly. [62:35] After hearing your description of how [62:37] much can be caught up, I feel like [62:39] NVIDIA's biggest advantage in training [62:41] and inference is being targeted by more [62:44] and more people wanting a piece of the [62:46] pie. [62:47] That's actually correct. As model [62:51] workloads become more fixed, many [62:53] people feel they can customize their [62:55] own inference chips for their specific [62:57] application scenarios. [62:59] Just like back when video was hot and [63:01] everyone started making video chips. [63:03] Exactly, but if you think about it, [63:05] NVIDIA is the company with the [63:07] strongest execution in the chip [63:09] industry, and because [63:10] This is quite different from how most [63:12] people perceive them. For those of us [63:14] in software and AI, we might think NV [63:16] is the most...capable company, but from [63:18] a chip industry perspective, their [63:20] execution is incredibly strong—their [63:22] chip iteration speed is very fast, and [63:24] their pace of architectural innovation [63:26] is also very rapid. It’s a bit like a [63:29] startup. Take NVIDIA's GPU architecture [63:31] , for instance—things like doubling [63:33] sparsity—once they decide to do it, [63:36] they just implement it. Meanwhile, many [63:39] other companies are still writing [63:40] papers and haven't gotten around to it. [63:43] When they do specific optimizations [63:45] based on certain conditions, their [63:47] iteration speed is incredibly fast. As [63:51] soon as they get feedback from the [63:52] model side, like from someone like you, [63:54] they can evolve their architecture with [63:56] the speed of a startup. [63:59] Wow. [63:59] And NVIDIA’s decision to acquire [64:01] Mellanox was also an incredibly fast [64:04] move. Just think about it: acquiring a [64:06] company with a software stack [64:08] incompatible with your own, which could [64:10] potentially pose a threat—they could [64:12] have done it themselves, but they made [64:14] the decision so quickly. This is [64:16] exactly why founder-led companies move [64:19] so fast. That’s the first point. The [64:21] second point is the supply chain. Even [64:23] Jensen Huang mentioned in recent [64:25] interviews that a major advantage they [64:27] have is their control over the supply [64:29] chain. While everyone can certainly [64:31] design an AI inference chip [64:33] architecture that is well-suited for a [64:35] specific scenario, it's another matter [64:38] if you can't secure HBM capacity or [64:40] TSMC's capacity, and your internal [64:42] engineering iteration speed can't keep [64:44] up with NVIDIA's. There are other [64:46] layers of difficulty involved here. [64:49] Your perspective, coming from chip and [64:51] computer architecture, is completely [64:53] different from ours. Those of us in [64:56] software and modeling always felt that [64:58] NVIDIA was just moving slowly and [65:00] meticulously in terms of software. [65:03] I never imagined they had such [65:05] disruptive R&D capabilities on the chip [65:07] side. Exactly. From a software [65:09] perspective, NVIDIA’s pace seems [65:11] relatively slow. But from our [65:12] perspective in the chip industry, their [65:15] pace—or rather, their execution—is [65:17] second to none, I think. [65:18] If that's the case, NVIDIA’s moat in [65:20] chips seems bottomless. It really seems [65:23] that way now. [65:24] I believe that as long as Jensen is [65:26] still leading the company, this [65:28] advantage will continue, yes. Wow, that [65:31] is terrifying. It really goes to show [65:33] that you have to ask the insiders to [65:35] get the deepest insights into these [65:36] things. Oh my, that's fantastic. My god [65:39] , your ability to tell a story is just [65:41] incredible. I can't wait to see how [65:42] amazing this episode turns out. One [65:44] more question. Nowadays, most AI tasks [65:46] are about AI inference. Today, Claude's [65:49] API went down again; it's clear the [65:51] demand is just too high. [65:52] Exactly, exactly. [65:54] Yeah. In your opinion, what is the [65:56] biggest bottleneck for AI inference? Is [65:58] it compute chips, HBM, network systems [66:01] —those interconnects—or electricity [66:03] , or something else? [66:05] Uh, I think it still comes down to a [66:07] lack of compute power. Yes, a lack of [66:09] power. And if we look at it from a [66:11] bottleneck perspective, it's the lack [66:14] of system optimization, the ability for [66:16] continuous optimization. The ability to [66:20] keep lowering the token cost, because [66:22] there are more and more models, chip [66:24] architectures are evolving rapidly, and [66:26] more will follow. Not to mention the [66:31] multimodal models and agent-based [66:33] inference, which actually have huge [66:35] room for optimization. So, what's [66:38] really missing is the ability to [66:40] co-design from a system-level [66:42] perspective, and to do this co-design [66:44] very quickly, to accelerate the [66:46] iteration speed and bring down that [66:48] token cost. We must always pursue the [66:51] ultimate token cost, so that large [66:53] models become as cheap as tap water, [66:55] making AI more accessible. [66:56] Exactly. I believe AI should become a [67:00] utility-like infrastructure, just like [67:02] water and electricity. [67:06] For practitioners like us, this isn't [67:08] just a possibility; it's something that [67:10] will definitely happen. [67:11] It's hard to imagine that after all [67:13] this, [67:13] Yeah. ... [67:14] it will still be this expensive? It [67:16] will definitely come down. Yes, costs [67:18] will certainly go down. The [67:20] semiconductor industry is constantly [67:22] innovating and working hard, including [67:24] the KV cache I mentioned earlier, and [67:26] distributed inference. Besides the [67:29] software side, many new chip companies [67:31] are in the works. Memory is a [67:35] bottleneck, and many device companies [67:37] are developing new solutions, including [67:39] new interconnect solutions. As long as [67:43] there are bottlenecks, engineers are [67:45] very smart; we will solve the problems, [67:47] especially now with the help of AI. [67:51] However, there are things like [67:53] production capacity or infrastructure [67:55] that follow their own rules; they are [67:57] physical constraints. Take the US, for [67:59] example, there's a power shortage, [68:01] right? To build a power plant or a [68:04] transmission grid, it actually takes a [68:07] cycle of over five years—the timeline [68:09] is very long. Manufacturing has its own [68:13] rhythm too, like wafer production. [68:17] Ultimately, everything is stuck on the [68:19] execution speed of the physical world. [68:22] The execution speed of the physical [68:23] world, and for example, the annual [68:25] delivery volume of lithography machines [68:27] , is actually a bottleneck, yes. Mm. [68:30] Right, in the existing environment, our [68:32] brilliant engineers will definitely do [68:34] everything possible to solve the [68:36] problems one step at a time. [68:39] But limited by various production [68:41] capacities and manufacturing speeds, [68:43] the time it takes for costs to come [68:45] down will still be quite long. [68:47] Exactly, exactly. So that is why, if [68:50] the physical world is limited, what [68:52] about software? Currently, the waste of [68:55] computing resources in software is [68:57] actually massive. From my perspective, [69:00] the utilization rate of many GPUs is [69:03] quite low. [69:05] Secondly, there are many features; [69:07] because models change so quickly and [69:09] hardware evolves so fast, no one has [69:11] actually been doing those optimizations [69:14] more precisely. Only when costs or [69:17] physical limitations force everyone to [69:19] do so will these optimizations be done [69:21] with extreme precision, and these [69:23] optimizations must be automated. This [69:27] is also why our company wants to turn [69:29] our knowledge of AI chip systems and [69:31] architecture into something an AI agent [69:34] can invoke, enabling continuous [69:36] automated optimization. [69:39] I firmly believe that your company, [69:41] which is so AI-native, will definitely [69:43] have a bright future, truly. [69:45] Thank you, thank you. [69:46] Yes. Oh, I don't know, this might be [69:48] our first interview that started out so [69:51] hardcore; we've been talking about [69:52] hardcore stuff for over an hour [69:54] straight. [69:55] Let's switch gears a little bit and [69:56] return to your personal story. [69:58] Mm-hmm. [69:59] So, in 2012, you finished your PhD. [70:01] Yes. You could have chosen academia, [70:04] various chip companies, or software [70:06] companies; why did you end up going to [70:08] Oracle? What attracted you to Oracle? [70:10] What attracted you? [70:13] Uh, a main reason for Oracle was that [70:15] it was a new team doing something new, [70:18] and it was a benchmark project for true [70:20] hardware-software co-design. It [70:28] involved accelerating database [70:30] operations on CPU chips, which required [70:33] chip co-design, how to perform [70:35] hardware-software co-design with [70:37] software, and how to iterate with the [70:39] OLTP database. It was actually a very [70:46] complete and innovative project. You [70:49] know, in our chip industry, if you [70:51] really want to do a chip project like [70:53] that from start to finish, before AI, [70:55] it was actually very rare for anyone to [70:58] have such an opportunity. Usually, [71:02] architects are more senior, so if you [71:04] really join a chip company, you might [71:06] just work on a block. You work on [71:10] modules, or some people start out in [71:13] verification. You have to keep leveling [71:15] up like you're playing a game. Back [71:19] then, after Oracle acquired Sun [71:21] Microsystems, they wanted to build a [71:23] new accelerator based on Oracle's [71:25] business, which gave me an opportunity. [71:30] We called that team "Software in [71:32] Silicon," meaning we were building [71:34] software into silicon using hardware to [71:37] implement it. At the time, that team [71:40] was a very innovative project, which is [71:42] why I felt it was such a great [71:44] opportunity. [71:45] It really was rare. In fact, my [71:49] followers are always asking me how to [71:51] choose a company and how to make those [71:53] career decisions. [71:56] Personally, I feel you should choose [71:58] like you did. [72:00] Go to a place where you can see and [72:02] learn about the entire industry. When [72:04] you've just graduated, you're still [72:06] very young. Right, right. [72:08] If you immediately limit yourself to a [72:09] tiny role that is uninteresting and [72:11] lacks challenge, [72:12] that's not ideal. [72:14] Exactly, while you're young, if you can [72:16] find opportunities that give you a [72:18] global perspective, you should do more [72:20] of that. [72:22] For example, if you look at a famous AI [72:25] infrastructure company versus a big [72:27] tech firm working on legacy databases. [72:30] Right. My constant advice is to go to [72:33] that AI infrastructure company first to [72:35] learn the entire AI workflow, rather [72:37] than returning to a big firm for a [72:39] fancy title, [72:40] Yes. [72:41] where you only end up working on things [72:43] from the ancient past. Exactly, and the [72:46] processes there rarely change; you [72:48] might just be patching things based on [72:50] someone else's foundation. Of course, [72:53] that kind of work can be more relaxing, [72:55] as people have different pursuits. Some [72:57] people value stability, or they need to [73:00] solve visa issues, right? In that case, [73:02] the choice is understandable. But if [73:06] you don't have those worries, I think [73:08] it's best to work on innovative [73:09] projects where you can see the big [73:11] picture. That's why my career path has [73:15] basically been about taking on new, [73:18] challenging, and innovative projects. [73:20] Wow, you've maintained such a great [73:22] state of mind. Look at how you talk to [73:24] everyone; from the moment I met you [73:26] until now, you've always been smiling [73:27] and living every day so happily. It [73:29] looks like the kind of work that gives [73:31] you positive energy and helps you level [73:32] up. I really like your current state, [73:35] and [73:35] you're also very thoughtful. You've [73:37] been working for almost 20 years, right [73:39] ? [73:39] Yet you still think about how everyone, [73:41] while graduating, also needs to deal [73:43] with visa status issues, family matters [73:44] , and all sorts of other problems. [73:46] Yes. Oh, you are just truly happy and [73:49] thoughtful. My goodness, you really [73:50] kept yourself in such great shape. [73:52] Thank you, thank you, um. [73:54] So throughout this whole process, [73:56] you've mentioned many times that this [73:58] Oracle project is an innovative one, a [74:00] project designed to accelerate database [74:02] queries. [74:02] In this, what do you feel was the most [74:04] interesting or critical decision made? [74:06] Because in chip or system design, [74:08] anyone who has studied system PhD knows [74:10] it's always about tradeoffs. [74:12] There is no such thing as perfect, [74:14] there’s no free lunch; you have to [74:16] choose what to give up. What do you [74:18] think was the biggest decision that [74:19] helped you or the company grow the most [74:21] ? [74:23] Uh, actually, it's because the DEX is [74:26] composed of two parts: one is the [74:28] memory processing part, which fetches [74:30] data from memory, and the other is what [74:32] we call the query processor, the [74:34] database acceleration engine unit—you [74:36] can think of it as the compute unit. [74:45] The biggest decision we made at the [74:47] time was, we defined a database [74:49] compression format; that compressed [74:51] format sits in memory, and we read it [74:53] in, decompress it on the fly, compute [74:55] on the fly, extract only the data we [74:57] need for calculation, and then send the [75:00] result back to memory. So a huge [75:07] decision we made back then was [75:09] selecting this compression method for [75:11] computation, and we had to choose a [75:13] specific compression algorithm. And, [75:16] well, you know, there are some [75:17] tradeoffs involved. Once you've [75:21] selected this compression algorithm, if [75:23] the data type changes or something else [75:25] changes, and the compression efficiency [75:27] drops, then the risk is actually quite [75:29] high. So, I think the biggest thing we [75:34] did at the time, including the part I [75:36] was responsible for, was developing [75:38] that compression architecture algorithm [75:40] and the microarchitecture for the [75:41] compression. We had a compression [75:45] algorithm at Oracle called OZIP. We [75:49] compared OZIP with software, we did a [75:51] lot of software-based testing to see [75:54] what the compression efficiency could [75:56] reach, and how OZIP was implemented. [76:01] There are many ways to implement this, [76:03] but we ultimately used a lookup table [76:05] approach. When you have a database, [76:09] since I work with queries, the data [76:11] doesn't change too frequently. So, we [76:16] first scan the database and generate a [76:18] compression table, which we call a [76:20] codeword table. Then, this codeword [76:26] table can be preloaded into our [76:28] accelerator before the chip starts [76:30] running. When a query comes into the [76:38] chip, I use that codeword for lookups [76:40] and then decompress the uncompressed [76:42] data. This is followed by the rest of [76:50] the query processing. The advantage [76:54] here is that even if the data changes, [76:56] I can rescan the database to generate a [76:58] new code table for re-decompression. [77:03] Although there is some risk involved, [77:05] we have mitigated it in a certain way, [77:07] which is actually quite interesting. [77:10] Mm. Anyway, I don't quite understand it [77:13] , but I know you must have accomplished [77:15] a lot of extraordinary things over [77:17] these five years. [77:19] Mm. [77:19] So, do you think this project of yours [77:21] is very innovative? You must have been [77:23] quite happy, looking at how you are now [77:25] . [77:25] What do you think are some of the [77:27] biggest advantages of Oracle, and what [77:29] are its most notable shortcomings? [77:30] Well, the advantage of Oracle is [77:32] actually that they bought Sun [77:34] Microsystems back then, so Sun had many [77:36] incredible engineers and architects, [77:38] and we were... [77:39] Let me clarify this for everyone: Sun [77:41] is the company behind Java. Sun is the [77:44] company that created Java, Sun [77:46] Microsystems. In the 90s, they were the [77:49] hottest company, similar to NVIDIA or [77:51] Google today. They also supported many [77:55] open-source projects, but they were [77:57] sold to Oracle for about seven billion, [77:59] which was extremely cheap. Which parts [78:03] were sold? The SPARC CPU chip division, [78:07] the Solaris operating system, including [78:09] Java, MySQL, the whole system, and many [78:12] other things, as well as the storage [78:15] systems—it was all very cheap. [78:17] Looking at it from today's perspective, [78:19] it was extremely cheap, seven billion. [78:22] Seven billion, indeed. And this company [78:26] has excellent vertical integration [78:28] capabilities; we have CPU design [78:31] capabilities, databases, the Solaris [78:33] operating system, and runtime [78:35] scheduling. Anyway, you, you can always [78:39] find someone to ask about anything, so, [78:42] it is a vertically integrated [78:44] organization, which is why Oracle was [78:46] relatively good at that time, and also [78:48] the reason it wasn't. In the end, this [78:51] project wasn't particularly successful. [78:54] Why? It’s because the CPU was SPARC. [78:56] You know, the SPARC processor has its [78:59] own instruction set, and it is [79:01] incompatible with x86. Later, x86 [79:03] combined with Linux to create these [79:05] types of small servers, which became [79:07] the mainstream for data center servers. [79:12] Before, it was SPARC and Solaris, but [79:14] because Solaris wasn't open-sourced, [79:17] the company failed to make it open at [79:19] the time, which meant the ecosystem [79:21] wasn't prosperous enough, leading to [79:23] fewer and fewer developers building [79:25] applications on SPARC, and thus the [79:27] ecosystem failed to thrive, becoming [79:29] weaker and weaker, so the whole project [79:32] wasn't particularly successful. [79:37] Regarding the SPARC processor, later on [79:39] , Oracle actually laid off the entire [79:41] department, basically our whole team at [79:44] the time. [79:44] Let’s set aside the layoffs for a [79:46] moment, let’s talk about what you [79:47] mentioned earlier regarding open source [79:48] . [79:49] Right. [79:49] The biggest piece of news these past [79:51] two days. [79:51] Mm. [79:52] Jensen Huang is taking the lead, [79:53] Mm. [79:54] calling on everyone to embrace open [79:56] source and support [79:57] this open source initiative, openM, [79:59] right? [79:59] More and more companies are joining [80:01] this. [80:01] Right. [80:02] I am also a firm supporter of open [80:04] source. [80:04] Right. Looking back at your past career [80:08] , [80:10] what do you think open source [80:11] means for a country, for an enterprise, [80:14] and for an individual, respectively? [80:16] Wow, that is a rather big question, yes [80:18] . [80:18] It’s quite big, very big, but since [80:20] it’s so important, I feel I must ask, [80:21] even though we hadn't prepared this [80:23] question beforehand. [80:24] That’s true, that’s true. I think [80:26] from a hardware perspective, open [80:28] source is rare; open-source IPs or [80:31] instruction sets like RISC-V are [80:33] actually quite uncommon. And in [80:37] hardware, many IPs are actually not [80:39] open-source, like SerDes or various [80:41] other IPs, everyone has to license them [80:44] . Then, from a software perspective, [80:46] once a chip is designed, it needs [80:48] software and an ecosystem. So, software [80:51] open source is actually what created [80:53] the prosperity of the internet. In fact [80:56] , those few papers from Google, you [80:58] know, the papers on databases, [81:00] including Google’s use of x86 servers [81:02] plus open-source Linux, combined with [81:05] these open-source big data processing [81:07] systems, actually created the [81:09] prosperity of the previous generation [81:11] of the internet. For the entire [81:13] industry and for enterprises, open [81:15] source is absolutely—I am also a [81:17] supporter of open source. It’s [81:20] actually extremely helpful for [81:21] fostering the industry ecosystem, and [81:23] it’s also quite good for us [81:25] individual engineers. Once things are [81:29] open-sourced, we can contribute to more [81:31] complex projects, and at the same time, [81:33] your skills become transferable, right? [81:38] When you move from one company to [81:40] another, you can actually transfer your [81:42] technical skills or your own knowledge. [81:47] However, when it comes to open-sourcing [81:49] models, there is another issue: why [81:51] don't many companies choose to [81:53] open-source? It’s really because they [81:57] spent a huge amount of money training [81:58] these things. If they do open-source [82:02] them, how does the company itself [82:04] measure the ROI? In effect, you’re [82:06] just making a contribution to everyone [82:08] else, right? So, for many of these [82:14] model companies, if they decide to [82:16] open-source, especially for the early [82:18] movers—the leading companies in large [82:20] language models—the big question is [82:22] how they can recover the significant [82:24] initial investment they’ve made. [82:30] Right. [82:30] But latecomers can open-source. [82:33] Open-sourcing allows them to expand [82:35] their own ecosystem. However, [82:40] open-sourcing might not be a great [82:42] strategy for a specific company, so it [82:44] depends on where you stand—whether [82:46] you are a leading large model company [82:48] or a follower. So for companies, there [82:53] are pros and cons, depending on your [82:56] position. For individuals, open-source [82:58] is definitely a good thing. For us [83:01] personally, open-source means you can, [83:03] for example, deploy your own private [83:06] open-source agents, you can do a lot of [83:08] things with open models, and you can [83:10] keep control of your own data. For [83:16] enterprises, if an open-source solution [83:18] can be made more compliant or secure in [83:20] some way, it can lower costs and also [83:23] serve as leverage against those big [83:25] model companies. When it comes to [83:30] national strategy, the issue becomes [83:32] much bigger, and it’s again a [83:34] question of who moved first. If China [83:38] were in the position the U.S. is in, [83:40] China might not necessarily open-source [83:42] either, right? So, latecomers using [83:46] open-source as a way to catch up is a [83:48] fairly common model in the industry. [83:51] Hmm, I think your analysis is extremely [83:53] comprehensive; the whole open-source [83:55] topic is just incredibly complex. Yes, [83:57] exactly. [83:58] There are considerations from various [84:00] aspects, but in any case, for us as [84:02] individuals, I think it is definitely [84:05] beneficial. [84:05] As an individual, always stay [84:07] open-minded. [84:08] Right. Mm. [84:09] So from 2012 to 2017, you became a [84:12] Principal at Oracle in just five years. [84:16] Mm. Honestly, everyone in Silicon [84:18] Valley knows how difficult that is, [84:20] because when you graduate, you start as [84:23] a junior, then senior, and Principal is [84:25] a very, very high-level position. What [84:27] do you think... [84:29] I have to clarify this; at that time, [84:31] Oracle's titles were inflated. Our " [84:34] Principal" wasn't exactly like the [84:36] Principal role at a company like NVIDIA [84:39] , but it was still a pretty good [84:41] position, yes. Right, what do you think [84:45] helped you get promoted and grow so [84:47] quickly? [84:50] Uh, I think it’s mostly about making [84:52] actual contributions to projects [84:54] because the company can see your output [84:56] and your impact. So it's more about [85:01] taking on harder, more challenging work [85:03] . For example, the software-hardware [85:08] co-design for compression I mentioned [85:11] earlier, and to a greater extent, [85:13] coordinating resources and doing things [85:15] differently than traditional chip [85:17] engineers—who are just assigned a [85:19] spec to implement. I was doing more of [85:24] what you'd call architect-level work, [85:26] yeah. [85:28] Mm, you started doing architect work [85:30] very early on, from a higher [85:31] perspective. [85:32] Which naturally pushed you into a much [85:34] larger... [85:35] Position. Right, it really comes down [85:38] to the problems you tackle and the [85:40] projects you take on, yes. [85:43] After that, you joined a startup chip [85:46] team back in China. [85:47] Mm-hmm. [85:48] What made you make that career choice? [85:52] The situation with that company at the [85:54] time was, the background was that every [85:56] hyperscaler was looking into custom [85:58] chips, and every hyperscaler wanted to [86:01] do custom chips. And this company— [86:04] well, let's call it an internet company [86:06] —when we joined, we weren't even sure [86:09] exactly what kind of chip to make. So [86:13] for me, it was a great opportunity [86:15] because you go in, and you're not just [86:17] an architect; you become a product [86:19] manager, someone who has to figure out [86:21] from a business perspective what kind [86:23] of chip to build. In the past, that was [86:27] usually decided by the CTO or CEO of a [86:29] chip company. Then we had to conduct a [86:33] lot of research to figure out which [86:35] businesses were useful, what scenarios [86:37] existed, what the ROI and cost were, [86:39] and whether these business scenarios [86:41] could be distilled into dedicated chips [86:43] , so we did a lot of that. Second, it [86:47] was a zero-to-one process. We handled [86:53] many things back then, including [86:55] bringing in designs, deciding which IPs [86:57] to buy, signing the first partnership [86:59] agreements with foundries, and [87:01] evaluating numerous factors—we really [87:04] did a lot. These weren't things someone [87:08] in a technical role was originally [87:10] supposed to do. So, for an individual, [87:12] it’s helpful for entrepreneurship; [87:14] I’d say yes. Mm. [87:16] Right, your career choices from the [87:17] beginning to the second stage have been [87:19] quite consistent, as you wanted to... [87:21] Go from zero to one, allow yourself to [87:23] take on a broader scope, and complete [87:25] tasks end-to-end—it feels like an [87:26] internal startup within a big company. [87:28] Ah yes, it’s actually internal [87:30] entrepreneurship within a large company [87:32] . Back then, AI wasn't as prevalent yet [87:35] , and the GPU wasn't necessarily meant [87:37] for AI chips; it could have been for [87:39] something else. So here, because I [87:45] enjoy doing it, every time I make a new [87:47] choice, I ask myself if I can learn [87:49] things I hadn't been exposed to before, [87:52] while also ensuring the compensation is [87:54] decent, right? And your scope might [88:00] grow larger. It’s also possible that [88:03] in a large company, more people will [88:05] join later, so you have to figure out [88:07] how to collaborate with everyone to get [88:09] the job done. But at the very least, [88:13] when we joined, we were in a fantastic [88:15] position, basically a founding team [88:17] working in an internal startup state. [88:20] Right. It seems you really enjoy the [88:23] process of being given a vague problem, [88:25] making it clear step by step, [88:27] ultimately forming a complete path, and [88:29] solving the issue. Yes, because those [88:34] of us in architecture want to find [88:36] certainty amidst uncertainty, start [88:38] from the definition of a task, and then [88:40] solve the problem. [88:42] That’s beautiful. I think in terms of [88:44] career development and choices, I [88:46] really should learn more from you. I [88:47] feel I share that same personality; I [88:49] like being able to define a problem and [88:51] slowly build something from zero to one [88:53] . I don’t like going from one to 100, [88:55] but I especially enjoy going from zero [88:57] to one. Yeah, actually, some of these [88:59] opportunities are quite rare. So, the [89:02] experience of building AI chips at a [89:04] major internet company is truly unique, [89:06] because the culture is very different [89:08] from traditional chip firms. The entire [89:13] process allows you to start from [89:15] scratch with a blank slate, which was [89:17] very helpful for my later [89:18] entrepreneurial journey. Yes. [89:21] Right, and then you went on to start [89:23] your own business, and [89:24] participated in two startups. [89:26] Yes. At the previous company, you were [89:29] the Chief Architect, [89:30] and at this current company, ZFOW, you [89:33] chose to be the CEO. [89:35] Mm. Why did you make that choice? [89:39] Because at the previous company, as [89:41] Chief Architect, my work was still [89:43] similar to what I did before, though I [89:45] did take on some new tasks. For example [89:51] , at my previous company, besides chip [89:54] architecture, I also worked on system [89:56] software, involving myself with R-time [89:58] teams, and also brought in some of the [90:00] capabilities we had built back at [90:02] Alibaba. For instance, how we cooperate [90:09] with design services, how we work with [90:11] system vendors, and how to find [90:13] partners. However, in my current [90:17] company, because AI infrastructure is [90:19] quite different from just doing chips [90:21] or just doing software. When making [90:25] chips, the business model is fixed. You [90:30] build the chip, and if it's competitive [90:32] in two years, you just follow the steps [90:34] to sell the chips, boards, and software [90:37] services. So the go-to-market is, how [90:41] should I put it, very certain. But for [90:44] AI infrastructure, the entire industry [90:47] changes extremely fast. For example, we [90:50] have built many products, but how do we [90:52] sell them? That is a question—how [90:55] exactly do you price them? What exactly [90:57] are you providing? And are you doing [91:00] consulting, or a service, or a [91:02] subscription? For me, this is a whole [91:05] new challenge. So I felt I had to take [91:10] this on myself, because it's hard to [91:12] find someone who can combine technical [91:14] innovation with business model [91:16] innovation. That’s why it’s another [91:21] learning process for me. Yes, mm. [91:25] Challenging yourself, playing the game [91:27] of life. [91:28] Right, it is quite challenging, because [91:30] selling chips and selling software are [91:32] really very different. Yes. So, you [91:34] went from being a Chief Architect to [91:37] becoming a CEO. [91:39] How much time have you spent on product [91:41] , sales, marketing, fundraising, and [91:43] technology during this past period? How [91:46] has your role shifted overall? [91:50] Uh, actually, I spend quite a lot of [91:52] time on product technology, but the [91:54] majority of my time is spent on BD. Yes [92:00] , finding partners, talking to [92:02] potential clients, signing MOUs, and [92:04] defining the business model together— [92:06] that part of the work takes up a huge [92:09] portion of my time. Secondly, our burn [92:13] rate is relatively very low. Currently, [92:18] fundraising does take up a share of my [92:20] energy, but I want to keep fundraising [92:22] within a very short window rather than [92:24] having it occupy my time long-term, yes [92:26] . So, my goal now is to spend more time [92:30] on business and BD. I spend most of my [92:32] time there, while still continuously [92:34] following up on product definition. I [92:36] have a solid understanding of basically [92:38] every technical detail, yes. [92:40] Right, so you spend a lot of time [92:42] talking to clients, [92:43] determining the business model, [92:44] defining the product form, [92:46] and after a while, waiting until these [92:47] are all clearly discussed, with clients [92:48] in place and a business model ready, [92:50] Yes, [92:51] you spend a month getting it all done [92:52] in one go and then dive in. Yes, [92:54] that’s my own personal model. Of [92:56] course, as I learn in the process of [92:59] being a CEO, perhaps some people are [93:01] constantly fundraising; by raising more [93:03] capital, you can do more things. But [93:06] for us in the chip industry, we might [93:09] prioritize making the business case [93:11] work first, having paying users, and [93:13] proving your value. Subsequent [93:16] fundraising, I feel, is just natural— [93:18] as long as it serves to satisfy your [93:20] business development. Once I have a [93:23] significant business, my fundraising [93:26] will actually be much faster, yes, hm. [93:28] Well, I’ve heard many successful [93:30] entrepreneurs in Silicon Valley say [93:32] that fundraising is just a natural [93:34] byproduct. Once you have a real [93:35] business, many investors don’t even [93:37] need you to ask; they’ll line up to [93:39] offer you money to help expand your [93:41] business. Ah, yes, because the logic of [93:43] fundraising is different from the logic [93:45] of building products and running a [93:47] business. Fundraising is a different [93:49] skillset. Yes, and if you spend too [93:51] much time on it, it really does affect [93:53] the time you have to actually run the [93:55] business, hm. [93:56] Hm. But fundraising ability is also [93:58] very important. [93:59] It is important. Some companies really [94:00] do rely on the CEO’s strong [94:01] fundraising ability to keep overcoming [94:02] obstacles and winning battles. [94:03] Exactly, exactly. [94:04] And having a strong exit capability, [94:05] being able to...Exactly, exactly, that [94:07] is also a very important skill, I... [94:08] Everyone has different ways of [94:09] competing in the marketplace. [94:10] Yes, no, and I’m also learning, [94:12] learning, learning. [94:13] Over the past period, your company [94:15] announced some inference optimizations [94:17] on the B200, right? A 1.5 x increase in [94:19] peak throughput and two to three times [94:21] that of... [94:22] tail latency improvements, right? That [94:24] is, P99 latency. [94:26] Yes. [94:26] How did you go about developing this [94:28] entire process? [94:30] Well, for this, we were actually there [94:32] at Day Zero, or very early on, [94:35] supporting DeepSeek-V 3 on 8x B200s [94:37] through the vLLM framework, and we [94:39] found certain configurations by using [94:42] some simulation methods. Then, we used [94:47] those configurations to set the [94:48] parameters for the vLLM framework. For [94:53] example, our conclusion for 8x B200s [94:56] was whether to use P-D separation or [94:58] not, how many GPUs to allocate for [95:01] Prefill, and how many for Decode. Those [95:08] decisions were reached through [95:09] simulation and then applied to [95:11] configure the vLLM runtime. But we also [95:17] did a lot of engineering work in [95:19] between, because getting a concurrent [95:21] vLLM runtime to run stably—and there [95:23] were many issues with throughput—was [95:25] a real challenge. Actually, the [95:30] open-source framework was also evolving [95:33] at the time, so our approach was to use [95:35] simulation to quickly identify these [95:37] configurations, and then......use those [95:40] configurations to integrate directly [95:42] with the vLLM framework to achieve a [95:44] more optimal deployment. That was the [95:47] whole logic and train of thought. [95:50] However, it actually took a lot of time [95:53] to get the framework stabilized, [95:54] because when DeepSeek first came out, [95:56] there were many bugs and quite a lot of [95:59] issues. Yes, yes. Regarding the process [96:02] you just mentioned, I've seen on social [96:04] media that everyone is saying these [96:06] open-source runtimes are too complex [96:08] and troublesome, and it takes so much [96:10] manpower and resources to get good [96:12] results. They surely must be aware of [96:14] this issue themselves. [96:16] So, comparing their internal [96:18] optimizations to yours...What is your [96:20] specific advantage here? [96:23] Actually, we are also considering [96:25] collaborating with the open-source [96:27] ecosystem. Because our simulation can [96:30] provide a virtual backend for these [96:32] ecosystems, your runtime can run in a [96:34] virtual simulation environment. This [96:37] way, you don't need as many GPUs to [96:39] perform these optimizations, and you [96:41] can even start them earlier. So I [96:45] believe we are complementary. In a [96:48] sense, this is also a tool to help with [96:50] design and optimization. [96:52] Exactly. Actually, many clients come to [96:56] us for architecture design exploration. [97:04] This tool is also very helpful for chip [97:06] design, especially for AI chips, [97:08] because the first step in chip design [97:10] is simulation, so this tool can be [97:12] considered an architectural EDA tool. [97:21] However, we don't just use this tool [97:23] for chip design assistance and [97:25] architecture exploration; it can also [97:27] be used for data center optimization. [97:32] Many chip companies are actually trying [97:34] this internally, providing some of [97:36] their internal simulators to those [97:38] doing model inference, allowing for [97:40] rapid iteration and optimization [97:42] strategies. Many chip companies are [97:46] doing this. But we are different [97:49] because we want to build a neutral [97:50] platform. Ours is compatible with [97:53] different chip architectures and [97:55] systems. You can combine A and B, for [97:59] example, NVIDIA hardware with AMD [98:01] hardware, or hardware from companies [98:03] other than NVIDIA, to perform [98:05] simulation optimization and eventually [98:07] achieve low-cost inference or training. [98:11] We want to create this kind of [98:13] universal, neutral, and impartial [98:15] platform. [98:17] Yes, in this way, there isn't so much [98:19] pressure to be tied to one vendor, and [98:21] anyone can use it. [98:23] Exactly, exactly. And the industry also [98:27] lacks such tools, tools that can really [98:30] help you explore and tune without [98:32] needing actual GPUs. [98:35] Well, those results you mentioned [98:37] earlier, the 1.5 x peak performance, [98:39] those results. Are they simulated or [98:41] real? [98:42] Those are real. We collaborated with [98:45] Neocloud; they provided the GPU [98:47] resources, and we did the tuning. [98:50] Was there a big difference between [98:51] those and your previous simulation [98:53] predictions? [98:54] Uh, for our DeepSeek model, it's [98:56] divided into stages. Currently, our [98:59] predictor can reach over 90%accuracy [99:01] for the prefill stage compared to [99:04] actual runtime, and for the decode [99:06] stage, it's roughly between 80%and 90%. [99:11] The reason is that there are some [99:13] dynamic behaviors during actual runtime [99:15] . Since it's dynamic, when you simulate [99:22] , you can only provide various [99:24] scenarios; it's very hard to simulate [99:26] precisely based on the specific data [99:29] distribution. So, we can achieve [99:34] simulation accuracy close to 90%, yes. [99:38] My understanding is, does this dynamic [99:40] refer to when user requests come in, [99:42] their arrival time, and the resulting [99:44] system load or temperature? [99:48] Yes, that dynamic includes things like [99:50] batching, concurrency dynamics, as well [99:53] as expert scheduling. In MOE models, [99:58] there is expert scheduling; when your [100:00] data distribution changes, the expert [100:02] routing changes, so there are many such [100:04] dynamic behaviors. These dynamic [100:08] behaviors are actually quite hard to [100:10] simulate. You can only simulate the [100:13] data distribution, or use results [100:15] obtained through profiling to calibrate [100:18] the simulation model. Therefore, for [100:22] our simulations, we are able to [100:24] calibrate our results using profiling [100:26] data. This makes us quite unique in the [100:31] industry; we are chip and architecture [100:33] designers who use this approach to [100:35] drive optimization. This is quite [100:39] interesting, but the problem is [100:41] actually very challenging: how do you [100:43] achieve enough simulation accuracy [100:45] while still being able to optimize? You [100:48] have to perform a lot of abstraction; [100:50] you need to abstract your system [100:52] architecture and your network [100:54] simulation, right? For example, if you [100:57] are doing distributed inference, you [100:59] need to simulate the network. Network [101:04] simulation involves network topology, [101:06] the precision level of switch [101:08] simulation, how to simulate routing, [101:10] whether to simulate congestion control [101:12] algorithms, and to what extent it is " [101:14] good enough" for optimization. Without [101:18] having truly built complex real-world [101:20] systems, and then being able to [101:22] abstract them into this framework, it [101:24] is actually very difficult to achieve; [101:26] the challenge is extremely high. [101:29] It feels like you are building a [101:31] virtual data center. Actually, people [101:34] have recently brought up something [101:37] called DSAL, or "AI Token Factory [101:39] Simulator," and we are somewhat similar [101:42] to that concept; yes, Jensen mentioned [101:44] this at GTC. Called AI Token Factory [101:47] Simulator, exactly. [101:48] This stuff is indeed too difficult. You [101:50] mentioned the congestion algorithm— [101:52] how to send, how to receive, [101:54] and how to route it, roughly referring [101:56] to that kind of architecture. [101:58] Exactly. [101:59] Yes. That will affect the results, but [102:02] the expert strategies in MOE will too, [102:04] right? [102:05] That will also have an impact. Well, [102:09] but in this regard, I’m confident we [102:11] can definitely build abstractions, [102:13] because AI workloads are relatively [102:15] more predictable compared to algorithms [102:18] running on general-purpose CPUs, so we [102:20] can achieve this abstraction. After [102:25] hearing your explanation, on the Ant [102:27] Group website, besides the AI [102:28] infrastructure optimization we just [102:30] discussed, [102:31] you’ve also built an Open Cloud [102:33] AutoTuner, [102:34] and a memory manager. Yes. [102:36] Actually, linking these three together, [102:38] the path you described earlier is [102:40] becoming clearer, right? [102:41] Exactly. [102:42] My understanding is that the Open Cloud [102:44] handles the upper-layer applications, [102:45] user requests, and interactions, while [102:47] the memory part looks at how to manage [102:49] video memory and RAM to optimize [102:51] computation—is that how you see it? [102:53] That’s exactly how you’re thinking [102:54] about it. Yes, these are actually two [102:56] levels of optimization. What we talked [102:59] about earlier was model serving [103:00] optimization, and this is agent-level [103:02] optimization. It’s really about how [103:07] to choose the model based on user [103:08] behavior, how to select memory [103:10] allocation, and how to perform optimal [103:12] configuration at the Open Cloud agent [103:14] level. The point we’ve always wanted [103:20] to make is that we already have many AI [103:22] chips, many open-source frameworks, and [103:24] many agent inferences, but the [103:26] configuration for every case is [103:28] different. If you need a large number [103:32] of engineers to handle these things, it [103:34] gets quite troublesome. So, what we [103:36] want to do—including the serving [103:38] AutoTuner and the agent inference [103:40] AutoTuner—is based on the same [103:42] concept: we want to use automated [103:44] optimization methods to get this done. [103:46] It seems like historically, there have [103:48] been many "auto" companies in the [103:50] database field too. You hit the nail on [103:52] the head; when we worked on database [103:54] queries, there was something called a " [103:56] query optimizer." [103:59] And a lot of what it did, for example, [104:02] deciding whether to use the CPU or [104:04] where to offload tasks, or how to [104:06] allocate memory—the complexity was [104:09] actually similar to, or even greater [104:11] than, AI, because databases have many [104:14] random, dynamic, and data-dependent [104:16] behaviors. While AI has things like MOE [104:22] and batching, most of it is still [104:24] matrix operations, so it’s relatively [104:26] fixed. So, as we move AI operations [104:30] toward the database approach, we’ll [104:32] see more of these random behaviors [104:34] being introduced. Therefore, when [104:37] building an optimizer, some of those [104:39] algorithms can indeed be similar. [104:42] But isn't commercialization quite [104:43] difficult? Because from what I [104:45] understand, companies that build [104:47] database optimizers...Hmm. ... [104:48] have there really been that many that [104:50] actually survived? [104:50] You are absolutely right; database [104:53] acceleration companies rarely succeed. [104:58] The reason is that their business [105:00] models evolve very quickly, and for [105:02] those of us in software, the technology [105:04] stack is so deep—for example, those [105:06] of us working on optimizers find it [105:08] very hard to keep up with these trends [105:10] when hardware is involved. Also, the [105:17] people building databases often have [105:19] their own optimizers, so it becomes [105:21] very difficult to define a viable [105:23] business model. [105:26] Mm. [105:27] Exactly, because several database [105:29] experts I previously admired, [105:30] started similar companies, and, uh, [105:33] they all ended up struggling quite a [105:34] bit. True, the technology is impressive [105:38] , but the business model—and this is [105:40] why I am also constantly learning—is [105:42] that for an AI infra company, you [105:44] either need to bring clients who need [105:47] and trust you, believing you can [105:49] deliver, or you bring the resources— [105:51] the GPU resources. You have to lean on [105:54] one of those two sides. Otherwise, it [105:56] is very difficult; companies purely [105:58] focused on inference or optimization [106:00] find it hard to secure their place. [106:03] Right. Right, many that appear to be AI [106:05] infra companies are actually GPU [106:07] companies, or— [106:07] Right, many, uh, many AI infra [106:09] companies are actually GPU companies, [106:11] or they are AI infra companies that are [106:13] tied to a specific client, right? [106:15] Right, strongly bound. Yes, strongly [106:18] bound to a major client. [106:19] Right. [106:20] So, you are aware of this history. [106:23] Mm. [106:24] So the challenges for this company are [106:26] indeed quite significant. The [106:29] challenges are immense—there are [106:31] technical challenges, and there are [106:33] commercialization challenges—but I [106:35] think this is a crucial problem, and if [106:37] solved, the value would be immense. The [106:41] industry also lacks a neutral [106:43] simulation platform that can perform [106:45] optimization. And things like serving [106:47] runtimes are missing a virtual backend. [106:50] There is so much to be done here. I can [106:53] see that if we move away from GPUs, [106:55] open-source communities or various [106:57] vendors could use your backend to [106:59] perform all kinds of operations. [107:02] Right. [107:03] It would save a significant amount of [107:04] time and effort. [107:05] Exactly. To put it bluntly, our company [107:09] could become the architecture team for [107:11] any AI infra or chip company, as we [107:13] have the tools and the talent, right? [107:15] Right, this type of model is actually [107:17] very easy to be acquired, you know? [107:19] After hearing your description. That is [107:21] also a possibility, yes. [107:22] Yeah, and there’s another company [107:24] that just came to mind while you were [107:25] describing that. [107:26] Mm, VMware. [107:27] Ah, yes. [107:28] Right, that company back then was also [107:29] ...in terms of commercial— [107:31] Commercialization was probably quite [107:32] difficult, too. [107:32] Yeah, but then it suddenly exploded, [107:34] because they did virtual machines. [107:35] Yes, exactly. [107:36] Right, you were doing these virtual [107:37] machines, doing all sorts of virtual [107:39] abstractions. [107:40] Yes. [107:40] Right, maybe at the beginning, no one [107:42] could see the business model clearly. [107:44] But then one day, suddenly every cloud [107:46] provider had to deploy this set of [107:48] things. Yes, I think from my [107:50] perspective, Jensen has already seen [107:52] this. That’s why he’s building this [107:56] AI Token Factory Simulator, because AI [107:58] hardware and system software are so [108:00] complex that nobody can handle them [108:02] anymore. And then there’s the [108:05] hardware cost; to really deploy a [108:07] system and then optimize it, what’s [108:09] actually missing isn't optimization [108:11] engineers, many AI infra companies lack [108:13] the resources, yeah. [108:15] Well, I think there is a lack of [108:17] engineers too, because the current [108:18] level of complexity is something [108:20] graduates just can't handle, you know? [108:22] Uh, yes, that’s right, that's right. [108:24] It's just too complex. I mean, for a [108:26] graduate to be able to handle it, [108:28] everyone I’ve seen is either someone [108:29] who has worked in the industry for many [108:31] years and knows the underlying GPU and [108:33] architecture, or someone who is heavily [108:34] involved in the SD or LLVM communities [108:36] and has learned a vast amount of [108:37] knowledge there. [108:38] But for anyone else, to expect them to [108:40] just jump in and start optimizing, [108:42] Based on my personal experience and [108:44] current interviews, it’s just not [108:46] possible, honestly. Our team's [108:48] experience is also quite unique. [108:51] Everyone here is a practitioner; [108:53] we’ve deployed systems on GPU clouds [108:55] and built full-stack GPU cloud [108:57] solutions for providers. And we have [109:01] experts, like my Ph.D. peer, who has [109:03] published at HOT Chips and worked on [109:05] simulators for many years. I have that [109:09] background too, always working on [109:11] software-hardware co-design for chips. [109:15] With this understanding, we are [109:17] essentially solving our own problem of [109:19] how to optimize and how to design the [109:21] architecture, yes. [109:22] So you are your own first batch of [109:24] customers. [109:25] Uh, yes, that’s right, exactly. [109:27] How did you pull this team together, [109:28] and how many people are there now? [109:30] Our core team is small, just a few [109:32] people, but many are former students or [109:35] colleagues—people who have that kind [109:37] of "comradeship" from working together [109:39] before. [109:41] Battle-hardened veterans of the chip [109:43] industry. Oh, right. [109:44] It’s so cool, so cool, I think. [109:46] Gosh, the experience and knowledge in [109:49] your brains is kind of priceless, [109:50] honestly, for real. [109:52] Mm. [109:52] We feel like the new generation of [109:54] young people. [109:54] Mm. [109:55] Those of us. [109:57] Having not fought the battles before, [109:59] even with using AI agents now, we don't [110:01] know how to extract the knowledge and [110:03] experience currently in your heads. [110:05] Don't know how to get it. [110:07] Uh, you're also right, some experience [110:09] is, for example. [110:11] Making mistakes. Right, it's about [110:14] learning from mistakes. Chips are so [110:16] expensive, so accumulating this [110:18] experience is actually very important. [110:21] Of course, because lately AI [110:22] infrastructure has become a bottleneck, [110:24] and chips have become a bottleneck, [110:26] which used to be the case for chips. [110:28] You know, in Silicon Valley, internet [110:31] companies actually pay much higher [110:33] salaries. Many of my friends from [110:36] before actually switched careers to [110:38] become coders. And those of us who have [110:41] stayed in the semiconductor industry [110:43] grinding away are actually quite few, [110:45] yes. Mm. [110:46] It’s quite fascinating; people might [110:47] not know that in Silicon Valley, the [110:49] pay for chip engineers is much, much [110:50] lower than in the software industry. [110:52] A lot lower, yes. Why is that? [110:54] That is, this is a question of capital [110:57] operation, right? Chip company costs [111:01] come from fixed asset investment, like [111:04] hardware investment, tape-out costs, [111:06] EDA software license fees, or IP costs, [111:09] so human capital is secondary. Of [111:15] course, the top-tier companies are few, [111:17] but most people are actually engineers [111:19] using EDA tools, so their pay isn't [111:21] high. And for internet companies, a [111:27] large part of their costs—before AI, [111:30] infrastructure, or GPU setups—were [111:32] mostly invested in human resources. [111:37] Once you produce this software, you can [111:39] scale at zero marginal cost. But the [111:44] situation is changing a bit now. Now [111:48] these cloud companies have become a bit [111:50] like our traditional semiconductor [111:52] companies; their fixed capital [111:53] investment has become the main cost, [111:55] and people might actually be replaced [111:58] by AI. It's a bit like the internet [112:01] industry or hyperscalers shifting [112:03] toward how we view traditional [112:04] semiconductor companies. CAPEX has [112:09] become the largest investment, and your [112:11] revenue cost is proportional to your [112:13] CAPEX, rather than being able to scale [112:16] infinitely through the replication of [112:18] capabilities. [112:21] It really feels like, if we look at the [112:23] entire training pipeline for large [112:25] models—from pre-training to [112:27] fine-tuning and then to RL— [112:29] Yes, [112:30] Once that whole chain is built, [112:31] Mm. [112:32] The value of humans is becoming lower [112:34] and lower, meaning that now large [112:35] models are assisting humans in managing [112:37] the entire training pipeline. Right, [112:39] right. [112:39] For an individual, a fresh graduate, [112:41] Yes. [112:42] The large model guides them to perform [112:44] certain operations. [112:45] Right. I feel that you can think of [112:48] large models—using the semiconductor [112:50] industry as an analogy—as your EDA [112:52] tools. And more of our engineers, [112:55] software engineers, are becoming users [112:57] of these EDA tools. So, these big [113:01] companies are investing more in [113:03] infrastructure. Everyone might need to [113:09] accept that the top tier—a bit like [113:11] the architects in our chip industry— [113:14] can still maintain very good income or [113:16] returns, but the average engineer has [113:18] become a software engineer using AI [113:20] tools. There will definitely be some [113:25] changes in this mindset. Mm. For some [113:28] high-level roles, on one hand, these [113:30] skills are indeed scarce, and for [113:32] reasons like maintaining a leading [113:34] position, yes, [113:35] But once we enter a period of stable [113:37] development, [113:38] Right. [113:38] It's hard to imagine that companies [113:40] will still be paying such high salaries [113:42] as they are now for all sorts of roles. [113:44] Well, actually, such high salaries are [113:46] only for a very small portion of people [113:48] . People might see sensationalized [113:51] reports in the newspapers, but most... [113:53] You can think of it as an analogy to [113:55] architects in the chip industry; they [113:57] are, well, everyone knows these people [113:59] are just the core, right? [114:00] Exactly, exactly, exactly. Mm. [114:03] Oh boy. Faced with such a future [114:06] outlook, [114:08] Mm. [114:08] Do you think high school students [114:10] should still study computer science, or [114:12] still get into chip-related fields? [114:14] Uh, I think they should. It's still [114:16] needed, still needed. Everyone needs to [114:18] become an architect. Put simply, it [114:20] means you can bridge the gap from top [114:22] to bottom and use tools to help you [114:24] realize many things. So, the [114:25] requirements for people are actually [114:27] very high; you still need to learn the [114:30] underlying fundamentals. So you still [114:33] need computer science, you need to [114:35] learn business models, or business in [114:37] general, and you also have your own [114:39] tech knowledge to pick up. I mean, [114:44] everyone really should have this kind [114:46] of architect mindset, whether it's for [114:47] choosing a career or for your [114:49] professional training. Right, yeah. [114:53] Looking at it from another angle, if [114:55] your interest isn't that strong, maybe [114:57] you don't need to be so obsessed with [114:59] it. If you’re really interested, [115:00] then you should go for it. [115:01] Right. [115:02] But if you're not interested, [115:03] Yeah. [115:04] The process of becoming a top-tier [115:05] architect is quite grueling, I think. [115:08] That's true. If it’s not your passion [115:10] , I still think productivity will see a [115:13] massive boost later on. AI can enhance [115:16] productivity; you could become someone [115:18] like us in the semiconductor industry [115:20] who uses AI. You might not get top-tier [115:24] pay, but that boost in productivity can [115:26] ensure basic welfare for everyone. [115:28] Exactly, you'd at least have no worries [115:31] , living a happy life every day. Even [115:33] if the salary for an ordinary [115:35] semiconductor engineer isn't that high, [115:37] you can still maintain a decent life [115:39] without much overtime or toxic [115:41] competition. [115:42] That sounds like a pretty good thing, [115:44] doesn't it? [115:45] Well, to some extent it is, but I think [115:48] society might eventually become [115:49] polarized, and many people might not [115:52] even need to work. [115:54] I feel like I had a bit too much fun [115:56] chatting with Mr. Zhibing today. A lot [115:58] of this content wasn't prepared; Mr. [116:00] Zhibing provided such a massive amount [116:02] of information that I had to improvise [116:05] everything. I tried my best to help [116:07] everyone clear up the last 20 years, [116:09] Mr. Zhibing's history in the chip [116:11] industry, and I hope we helped everyone [116:13] sort out how many generations of [116:15] updates the chip industry, especially [116:17] here in Silicon Valley, has gone [116:19] through over the last 20-plus years. [116:21] Yes, and I'm also very grateful for [116:23] Uncle Moon's invitation. I might have [116:26] spoken a bit fast, but we could [116:28] actually talk about each of these [116:30] topics for several more hours. [116:32] How about you come on the show more [116:34] often? For this time, we’ll just [116:35] start by setting the stage, and then we [116:37] can go through each chapter in detail [116:39] later, okay? [116:40] Thank you everyone, thank you everyone, [116:42] and thank you, Mr. Zhibing.