MatX Chief Says Data Movement Logic Eats 87% of AI Chip Silicon
Dwarkesh Patel
Traditional chip designs waste vast die area on multiplexers to move data, driving AI hardware toward specialized systolic arrays that minimize routing overhead.
## Background
Dwarkesh Patel published this web/RSS podcast episode on May 22, 2026, featuring a blackboard lecture with Reiner Pope, CEO of MatX and former Google TPU/compiler engineer. The episode description notes that Pope builds up chip design fundamentals from basic logic gates all the way to why GPUs, TPUs, FPGAs, and the human brain are architected the way they are.
## TL;DR
Reiner Pope walks through hardware design from first principles, demonstrating how 16 full adders and basic AND gates compose a 4-bit multiply-accumulate (MAC) circuit for matrix multiplication. He explains why data movement and multiplexing consume up to 87.5% of traditional silicon area, prompting the shift to systolic arrays and software-managed scratchpads that trade programmability for raw matrix throughput. The discussion details physical constraints including quadratic area scaling across precision levels, pipeline register placement for clock synchronization, FPGA lookup table trade-offs, and the structural differences between fine-grained GPU streaming multiprocessors and coarse-grained TPU matrix units.
## Atmosphere & style
The session has the rigorous, intellectually demanding feel of a graduate-level engineering seminar brought to life on a physical chalkboard. Pope communicates with precise technical clarity and analytical depth, using concrete logic gate counts and spatial diagrams to unpack abstract hardware mechanics. The dynamic between Patel and Pope is deeply collaborative and fast-paced, blending curiosity with dense technical breakdown as they explore the underlying physical and mathematical constraints of compute architecture.
## Key takeaways
- `00:00` The Multiply-Accumulate (MAC) operation is the most critical primitive for AI chips because matrix multiplication relies on continuous loops of multiplication and accumulation.
- `00:00` Accumulation requires higher arithmetic precision than multiplication to prevent numerical errors from compounding across hundreds of sequential additions.
- `10:00` Lower-precision and higher-precision logic units are not fungible at the physical gate level; hardware vendors allocate fixed die areas to different precision units rather than reconfiguring gates on the fly.
- `10:00` The gate count of an integer multiplier scales quadratically with bit width, meaning halving the precision reduces the required multiplier logic area by roughly fourfold.
- `16:31` Traditional scalar or vector processor cores spend the vast majority of their physical area and energy on multiplexers and data routing rather than raw arithmetic computation.
- `26:10` Systolic arrays (tensor cores) eliminate data routing overhead by storing weights locally in spatial registers and streaming data through a fixed grid of MAC units.
- `30:00` Systolic arrays utilize a trickle-feed strategy to load weights slowly over consecutive clock cycles, keeping boundary-crossing wiring strictly to a linear scale.
- `39:11` Modern microprocessors rely on a global clock cycle to maintain deterministic synchronization across billions of parallel transistors and prevent race conditions.
- `40:00` Inserting pipeline registers breaks up long critical logic paths to double the clock frequency, but over-pipelining introduces severe silicon area inefficiencies.
- `45:00` Feedback loops dictate the maximum clock frequency of a chip because inserting a pipeline register alters the underlying mathematical behavior of the loop.
- `51:51` FPGAs use lookup tables (LUTs) and a configurable routing matrix to emulate arbitrary digital logic, offering soft reconfigurability at a 10x area and performance penalty compared to ASICs.
- `01:03:15` AI accelerators utilize software-managed scratchpads backed by High Bandwidth Memory (HBM) rather than hardware-managed caches to ensure absolute timing and latency predictability.
- `01:07:27` CPUs dedicate massive amounts of die area to branch predictors, deep cache structures, and large register files to execute deep instruction pipelines with low latency.
- `01:12:00` Slowing down a chip's clock speed reduces its overall power draw linearly but does not alter the fundamental energy required per operation, which is governed by dynamic switching power.
- `01:15:33` GPUs tile vector and matrix compute units into fine-grained Streaming Multiprocessors, whereas TPUs consolidate compute into a few coarse-grained, large-scale systolic arrays.
- `01:15:33` MatX is developing splittable systolic arrays as a hybrid approach to bridge the area efficiency of large TPU architectures with the compiler flexibility of GPUs.
## Detailed walkthrough
### `00:00` Building a Multiply-Accumulate from Logic Gates
Dwarkesh Patel opens the session by re-introducing Reiner Pope, CEO of the AI chip startup MatX (in which Dwarkesh discloses he is an angel investor). Returning for a chalkboard lecture, Reiner sets out to explain chip design from the ground up—starting with individual logic gates and building toward full production hardware architectures like GPUs, TPUs, and FPGAs. At the absolute foundational level, a chip consists of primitive logic gates (like `AND`, `OR`, `NOT`) connected by physical wires laid out as metal traces on silicon.
Because AI workloads center almost entirely on matrix multiplication, the single most crucial computational primitive on an AI chip is the **Multiply-Accumulate (MAC)** operation. In matrix multiplication (a triply nested `for` loop over indices $i, j, k$), the inner loop statement `output[i,k] += input[i,j] * weight[j,k]` executes continuously. Reiner notes that accumulation almost always requires higher precision than multiplication to prevent numerical error compound from hundreds of sequential additions. To demonstrate how a MAC circuit is physically constructed, Reiner walks through a concrete hand calculation multiplying two 4-bit numbers and accumulating the result into an 8-bit number:
$$1001_2 \times 1010_2 + 00000000_2$$
```
1001 (A = 9)
x 1010 (B = 10)
------
1001 (A * bit 0)
0000 (A * bit 1, shifted 1)
1001 (A * bit 2, shifted 2)
0000 (A * bit 3, shifted 3)
```
- **Partial Product Generation:** Generating the 16 partial products requires 16 `AND` gates, as a bit product is `1` if and only if both source bits are `1`. In general, a $P$-bit by $Q$-bit multiplication consumes $P \times Q$ `AND` gates.
- **Full Adders (3-to-2 Compressors):** The bulk of the silicon area is spent summing these terms. The core logic gate for multi-term addition is the **full adder**, which takes 3 input bits from a single column and outputs 2 bits (a sum and a carry out, representing values from 0 to 3 in binary).
- **Dadda Multiplier Strategy:** Instead of carrying values across columns like a human, the circuit repeatedly applies full adders across the column grid. Each full adder consumes 3 input bits and produces 2 output bits, eliminating exactly 1 net bit from the grid per gate.
Starting with 24 total input bits ($16 \text{ partial products} + 8 \text{ accumulation bits}$) and reducing them to an 8-bit output requires exactly $24 - 8 = 16$ full adders. For a general $P \times Q$ multiplication with $(P + Q)$-bit accumulation, the input count is $(P \cdot Q + P + Q)$, the output count is $(P + Q)$, and the total full adders required is strictly $P \times Q$. This clean $P \times Q$ hardware footprint is the second key reason MAC units are the universal building block of AI chips.
---
`10:00` Precision Scaling and Arithmetic Fungibility
Dwarkesh asks whether chip architectures can "funge" their floating-point arithmetic units—for example, dynamically repurposing hardware designed for FP8 into FP4. Reiner clarifies that at the physical gate level, lower-precision and higher-precision logic units are generally **not fungible**; a dedicated FP4 multiplier requires physically distinct silicon layout compared to an FP8 multiplier.
When hardware vendors like NVIDIA announce that a chip delivers twice the throughput at FP4 compared to FP8, it is not because the hardware is dynamically reconfiguring gates on the fly. Rather, designers allocate fixed die areas to different precision units based on customer demand and power budgets.
- **Quadratic Area Scaling:** The gate count of an integer multiplier scales quadratically ($P \times Q$) with bit width. Halving the bit precision reduces the multiplier logic area by roughly $4\times$.
- **Historical Compute Ratios:** Historically (up to NVIDIA's Hopper architecture), vendor spec sheets scaled performance linearly (doubling FLOPS when halving bit width) largely due to data-movement memory bus alignment rather than raw gate limits.
- **Modern AI Accelerators:** Starting with architectures like Blackwell (B100/B200), vendors have begun reflecting quadratic scaling benefits, yielding up to $3\times$ or $4\times$ speedups for FP4 relative to FP8.
- **Floating-Point Overhead:** Unlike raw integer multipliers, floating-point units require extra exponent alignment and normalization logic, which slightly alters the exact quadratic ratio.
---
`16:31` Muxes and the Cost of Data Movement
To illustrate why specialized AI accelerators differ so radically from traditional processors, Reiner contrasts a specialized MAC array with the architecture of a standard general-purpose CPU or classic GPU CUDA core. A conventional core relies on a **register file** connected to Arithmetic Logic Units (ALUs).
```
+---------------------------------------+
| Register File |
| [ Reg 0 ] [ Reg 1 ]... [ Reg N ] |
+---------------------------------------+
| | |
v v v
+-------+ +-------+ +-------+
| MUX 1 | | MUX 2 | | MUX 3 | (Multiplexers select inputs)
+-------+ +-------+ +-------+
| | |
+-----------+------------+
|
v
+----------------+
| ALU / MAC |
+----------------+
|
v (Writeback Route)
+---------------------------------------+
| Register File |
+---------------------------------------+
```
In a traditional scalar or vector core, every operation requires routing data from arbitrary registers in the register file down to the ALU and writing the result back. This flexible routing is implemented using **multiplexers (MUXes)**.
- **MUX Gate Cost:** An $N$-input multiplexer operating on $P$-bit data requires $N \times P$ `AND` gates to mask out unselected inputs, plus $(N - 1) \times P$ `OR` gates to combine the result.
- **Routing Overhead:** As the register file size ($N$) grows, the physical area and power spent purely on multiplexing and wire routing quickly dwarf the actual arithmetic logic gate count ($P \times Q$).
- **The "Data Movement Penalty":** In a standard CPU or legacy GPU core, the vast majority of energy and die area is consumed not by computing math, but by shuttling bits back and forth between registers and compute logic through dense MUX trees.
`20:00` Data Movement Costs and the Structure of a Multiplexer
The hidden cost of chip performance lies heavily in moving data rather than doing arithmetic. When reading values from a small register file to feed into a basic multiply-accumulate (MAC) circuit, selecting specific elements requires multiplexers (muxes). For a small register file, constructing the necessary multiplexer circuit requires three distinct input paths to feed the MAC unit. Mathematically, this data routing demands $3 \times n \times p$ AND gates compared to only $p \times q$ gates inside the logic unit performing the actual target computation.
Plugging concrete numbers into this comparison highlights the stark imbalance: if $n = 8$ and $q = 4$, the multiplexer logic demands $24 \times p$ AND gates just for data movement, while the actual multiply-adder logic requires only $4 \times p$ gates. Moving data from even a tiny register file into the execution unit takes many times more hardware area and logic gates than the computation itself.
```
+-------------------------------------------------------------+
| REGISTER FILE (n rows x p bits) |
+-------------------------------------------------------------+
| |
v v
+-----------------------------------------+
| Data Movement Logic (Multiplexers) | <-- ~87.5% of Circuit Area
| (Cost: 3 x n x p AND gates = 24p) | (Massive Overhead)
+-----------------------------------------+
|
v
+-----------------------------------------+
| Multiply-Accumulate (MAC) Unit | <-- ~12.5% of Circuit Area
| (Cost: p x q gates = 4p) | (Actual Math Goal)
+-----------------------------------------+
```
A simple 2-way, 4-bit multiplexer illustrates how selecting data relies on crude bitwise operations. Given two 4-bit numbers as inputs and a 1-hot encoded selector bit, the circuit ANDs the selector bit with every single bit of both candidate rows. One row evaluates to all zeros while the other passes through its original values. The circuit then ORs the two output rows together bit by bit to produce the final selected 4-bit result. This relies on $n \times p$ AND gates—in this 2-input case, 8 AND gates and 4 OR gates. The circuit mirrors the structural complexity of an adder, yet all that work is spent merely selecting a single row of data.
- **Gate Ratio Breakdown**: For $n=8$ register rows and a 4-bit multiplier ($q=4$), routing overhead consumes roughly 6 times as many gates ($24p$) as the logic doing the MAC operation ($4p$).
- **Multiplexer Composition**: A 2-way, 4-bit mux uses 8 AND gates and 4 OR gates to collapse two 4-bit candidate inputs down to 1 selected output row.
- **Historical Pre-Volta Architecture**: Prior to NVIDIA's Volta GPU generation, standard CUDA cores processed data using this unoptimized register-to-ALU path, where 7/8ths of the physical silicon area was wasted on data routing.
`26:10` The Architecture of Systolic Arrays and Tensor Cores
To eliminate the massive area tax of register-file data movement, hardware designers introduced tensor cores, known generically as **systolic arrays**. The architectural shift involves moving up two loop levels in the matrix multiplication algorithm and baking the entire outer loop directly into fixed-function hardware. By scaling the physical size of the execution block up, the compute-to-communication ratio improves dramatically.
Matrix multiplication inherently multiplies columns of a matrix by a vector and accumulates the products. In a simple $2 \times 2$ example, four individual multiply-accumulate units sit in a grid to calculate the column-wise dot products. Instead of continuously fetching both the input vectors and the large matrix coefficients from expensive register files every single clock cycle, the weight matrix is stored locally inside spatial registers right where the multiplication logic occurs.
```
Input Vector (Size X)
|
v
+---------------+
| [0] [1] | <-- Weights held locally in
| [2] [3] | registers inside array
+---------------+
|
v
Accumulated Output (Size X)
```
By keeping the quadratic weight matrix stationary inside local array registers and cycling streaming vectors through it, the chip achieves quadratically more compute ($X \times Y$) relative to linear data communication ($X$).
- **Dimensional Advantage**: Systolic arrays achieve $X \times Y$ compute density for only $X$ units of input/output bandwidth.
- **Local Data Reuse**: Storing weights directly inside local array registers avoids pulling the $X \times Y$ weight matrix over register-file buses on every execution cycle.
- **Historical Scaling Example**: Early Google TPUs scaled this concept up to massive $128 \times 128$ systolic arrays of fixed MAC circuits.
`30:00` Trickle-Feeding Weights and Silicon Trade-offs
Because loading full weight matrices simultaneously requires excessive die area for inter-connect wiring, systolic arrays use a "trickle-feed" strategy. Weight values are slowly daisy-chained into the grid over consecutive clock cycles. A new value enters the top row of the array, and on each subsequent clock cycle, existing values shift down by one row. This keeps the boundary-crossing wiring strictly bounded to a linear scale ($X$) rather than a quadratic scale ($X \times Y$).
"Bandwidth equals die area," so shifting weight vectors slowly over small data lanes protects precious silicon real estate. This same fundamental principle—maximizing compute relative to communication bandwidth—applies across every layer of hardware design, from bit-level floating-point precision choices up to inter-chip communication across cluster nodes during AI inference.
```
[ Top Input Lanes (Width X) ]
|
v
Row 0: [ W0,0 ] [ W0,1 ] <-- Load Top Row
| | (Shift down each cycle)
Row 1: [ W1,0 ] [ W1,1 ]
```
When designing an AI chip, engineers constantly face sizing trade-offs between the physical dimensions of the systolic array and the size of its supporting register file. If an engineering team caps data-movement silicon at 10% of total chip area and allocates 90% to the systolic array, they must carefully balance register size. A larger register file grants application flexibility and compiler freedom, but taking away silicon area directly shrinks the maximum throughput of the systolic engine.
- **Bandwidth Bound**: Weight loading relies on an $X$-wide boundary shift register running across $Y$ consecutive cycles to keep total pin count low.
- **Area Allocation Standard**: A common target balance in AI accelerator design is reserving ~90% of execution-block silicon for compute (systolic arrays) and restricting data movement/registers to ~10%.
- **Fundamental Rule**: Chip design revolves around area-sizing decisions; larger register files improve software programmability but reduce raw theoretical compute density.
`39:11` Clock Cycles and Micro-Synchronization
At a fundamental level, modern microprocessors with over 100 billion transistors rely on a global clock cycle to maintain synchronization across massive parallel structures. Unlike software systems that use expensive, non-deterministic lock primitives like mutexes in shared memory, microchips enforce synchronization deterministically at tiny time intervals—typically every single nanosecond.
`40:00` Clock Cycles and Pipeline Registers
Reiner Pope explains the concept of a clock cycle, which acts as the global heartbeat of a chip. Every nanosecond or so, the entire chip synchronizes in lockstep to transition to the next operation. In hardware, this timing is mediated by registers—storage devices that hold bits ($0$ or $1$)—placed around a "cloud of logic" (such as a systolic array or multiplier). The clock signal drives all registers simultaneously, freezing the exact input value on a wire at that instant and writing it to an output register.
```
[ Input Register ] ──► [ Cloud of Logic ] ──► [ Output Register ]
▲
Global Clock
```
To maximize chip performance, designers aim for the highest possible clock speed (for example, running at 2 GHz delivers twice as many operations per second as 1 GHz). However, the bottleneck is propagation delay: every computation inside the logic cloud must finish before the next clock edge strikes. Standard chip design margins this timing conservatively—by many standard deviations—so that signals arrive well in advance (e.g., within 75% of the clock cycle time), making setup time failures virtually impossible under normal conditions.
```
Without Pipelining (1 Long Clock Cycle):
[ Reg ] ──► [ Cloud of Logic ] ──► [ Reg ]
With Pipelining (2 Short Clock Cycles = 2x Clock Speed):
[ Reg ] ──► [ Logic A ] ──► [ Pipeline Reg ] ──► [ Logic B ] ──► [ Reg ]
```
A major task in chip design is inserting **pipeline registers** to break up long critical paths. Splitting a massive cloud of logic in half with a middle register allows the chip to run at double the clock frequency at the cost of additional storage area.
* **Global Synchronization:** Prevents race conditions where two parallel paths (e.g., functions $F$ and $G$ converging at $H$) arrive out of order due to manufacturing variations across silicon dies.
* **Clock Domain Crossings:** Rare areas in chip design where signals pass between different clock frequencies, requiring probabilistic reasoning about setup/hold time violations.
* **TSMC Process Nodes:** The choice of technology node (such as 3nm) sets baseline transistor speeds, but the actual clock speed achieved depends heavily on how effectively designers eliminate long critical paths.
---
`45:00` Feedback Loops and the Area vs. Frequency Trade-off
While inserting pipeline registers is straightforward for feed-forward logic, **feedback loops** present a far harder architectural challenge. In circuits where an output feeds back into the input—such as an accumulator circuit adding a sequence of numbers into a running sum—inserting a pipeline register changes the underlying mathematical computation. Instead of producing a single running sum, a two-stage pipeline loop produces two interleaved running sums (one for even-numbered clock cycles and one for odd-numbered clock cycles).
```
Feedback Accumulator Loop:
┌──────────────┐
▼ │
[+] ──► [ Reg ] ───────┴──► Output
▲
│ (Adding new number each cycle)
```
Because feedback loops cannot be easily split without altering logic behavior, they frequently dictate the maximum clock frequency of the entire chip. Primitive gates provided by semiconductor foundries via Process Design Kits (PDKs)—such as AND gates or full adders—have tiny propagation delays on the order of 10 picoseconds. A typical clock cycle accommodates a chain of roughly 10 to 30 sequential primitive gates.
If an architect aggressively pipelines down to single gates (a register after every AND gate), clock speeds could reach 5 GHz or more, but area efficiency collapses. A basic logic gate might occupy 1 unit of area ("gate equivalent"), whereas a register occupies around 8 units. Over-pipelining spends almost the entire chip silicon budget on synchronization rather than useful computation, driving up clock frequency while severely hurting overall throughput.
* **PDK (Process Design Kit):** Provided by foundries like TSMC to specify base logic primitives, timing constraints, and physical rules.
* **Throughput Formula:** $\text{Throughput} = \text{Work per Clock Cycle (Area Efficiency)} \times \text{Clock Frequency}$.
* **Analogy to Batch Size:** High clock frequency with minimal logic per cycle resembles running an LLM with a batch size of 1—it achieves low latency per item, but reduces the total volume of work processed per unit of time.
---
`51:51` FPGAs vs. ASICs
Field-Programmable Gate Arrays (FPGAs) and Application-Specific Integrated Circuits (ASICs) share the same underlying conceptual model: networks of logic gates (AND, OR, XOR) synchronized by registers and clocks. However, their economics and physical trade-offs differ vastly. An ASIC provides roughly an order of magnitude lower unit cost and far better energy efficiency, but requires a full "tape-out" costing upwards of $30 million for the first chip. An FPGA costs around $10,000 for a single chip off the shelf and can be reconfigured repeatedly in software.
The term "field-programmable" means the chip can be rewired after deployment in the "field" (such as inside a server rack at a high-frequency trading firm or data center). To emulate arbitrary digital logic, an FPGA built on a fixed silicon die contains three core primitives:
1. **Lookup Tables (LUTs):** Small configurable memories used to emulate arbitrary logic gates.
2. **Registers:** Dedicated flip-flops to store state and manage clock cycles.
3. **Configurable Routing Matrix:** A vast network of multiplexers (MUXs) and interconnect wires that route signals between LUTs and registers.
A standard 4-input LUT (LUT4) operates as a programmable truth table. Since a 4-bit binary input yields $2^4 = 16$ possible input combinations, the LUT stores 16 configuration bits in small static memory cells. When four logic signals enter the LUT, they act as an address to select one of the 16 stored bits, emitting the output. This allows a single LUT to implement any 4-input boolean function—whether it is a 4-way XOR, an AND gate, or complex multiplexing logic.
* **High-Frequency Trading (HFT):** Firms like Jane Street use FPGAs because they offer deterministic, low-latency execution and can be reprogrammed frequently as trading logic evolves without paying multi-million-dollar tape-out fees.
* **FPGA Overhead Cost:** Implementing a MUX-based 16-to-1 selector inside a LUT requires approximately 16 AND gates and 16 OR gates of underlying silicon area, illustrating why FPGA logic density is roughly $10\times$ less efficient than a direct ASIC implementation.
<ElicitationsGroup message="Explore related hardware and system architecture concepts:">
<Elicitation label="Compare SRAM vs. DRAM in chip caches and memory hierarchies" query="Explain the trade-offs between SRAM and DRAM in terms of speed, density, area, and how they are used in chip memory hierarchies."/>
<Elicitation label="Analyze systolic arrays for matrix multiplication" query="How do systolic arrays work in TPUs for high-throughput matrix multiplication, and how do they minimize data movement?"/>
</ElicitationsGroup>
`01:00:00` Multiplexers inside FPGAs and why custom hardware is faster
Reiner Pope and Dwarkesh Patel trace the internal composition of a field-programmable gate array (FPGA). Reiner clarifies that inside a lookup table (LUT) sits a chain of multiplexers (muxes). The FPGA uses an array of small single-bit registers and local interconnects. An eight-entry local neighborhood provides 8 input bits, which pass through four 8-to-1 muxes to select 4 bits. Those 4 selected bits then feed directly into the LUT's internal 16-to-1 mux to evaluate any arbitrary 4-input boolean function. While this configuration provides massive hardware flexibility, implementing a simple function—such as a 4-input AND gate—in an LUT requires around 32 total logic gates, compared to just 3 logic gates on an Application-Specific Integrated Circuit (ASIC). This structural bloat and wire routing overhead explain why FPGAs are roughly 10x slower and less area-efficient than direct silicon implementations.
ASIC (Direct Silicon approach):
Input A ──┐
├─► [AND] ──┐
Input B ──┘ │
├─► [AND] ──► Output
Input C ──┐ │
├─► [AND] ──┘
Input D ──┘
(Requires only 3 physical gates)
```
- A 4-input AND gate requires only 3 physical AND gates on an ASIC, but takes ~32 gates worth of multiplexer logic when emulated inside an FPGA LUT.
- FPGA manufacturers like AMD specify fixed opinionated wire topologies for long-distance routing while keeping local neighborhoods flexible.
- The fundamental cost of reconfigurability comes from explicitly storing truth table entries and multiplexing selection, rather than placing permanent polysilicon paths.
`01:03:25` Deterministic latency: CPU cache vs. TPU scratchpads
Dwarkesh notes that quantitative traders and hardware engineers prefer FPGAs over standard CPUs due to their deterministic clock cycles, ensuring packet processing arrives with exact timing. Reiner explains that non-determinism in standard CPUs is an intentional design choice rather than an inherent physical limitation. Specialized processors like Groq or Google TPUs maintain deterministic latency right in the core. In standard CPUs, non-deterministic execution times stem primarily from hardware-managed caches, which sit between the CPU die and off-chip DDR memory. Because cache hits run nearly two orders of magnitude faster than DDR fetches, and because cache hits depend on background operating system tasks, memory access times vary wildly.
To eliminate this volatility while maintaining high memory bandwidth, AI accelerators utilize software-managed memory known as a scratchpad (often backed by High Bandwidth Memory, or HBM). Unlike CPU architectures where hardware dynamically decides whether an access hits the cache or goes out to main DDR memory, a scratchpad requires explicit software instructions. The compiler specifies one distinct instruction to read or write the local scratchpad and a completely different instruction to trigger an HBM transfer, guaranteeing absolute timing predictability.
- CPU caches operate roughly 100x faster than off-chip DDR memory, meaning cache misses degrade performance by two orders of magnitude.
- Standard CPU caches introduce non-determinism through unpredictable multi-tenant state, operating system background processes, and dynamic cache replacement algorithms.
- Accelerators replace implicit hardware caches with explicit software-managed scratchpads, giving compilers complete control over memory transfers.
`01:07:27` Why CPU cores are massive compared to GPU cores
When asked whether modern multi-core chips still reflect the traditional von Neumann architecture, Reiner notes that CPUs remain faithful to it, offering around 1,000-way parallelism (roughly 100 cores each hosting a 16-way vector unit). In contrast, GPUs and AI accelerators offer massively higher parallel throughput. CPU cores consume vast amounts of die area because of complex control logic rather than raw Arithmetic Logic Units (ALUs). A huge portion of a CPU core's physical footprint is dedicated to large register files, deep cache structures, and sophisticated branch predictors.
Branch predictors are essential in high-frequency CPUs running at 1 to 2 GHz because instruction pipelines are deep. Evaluating a single conditional branch boolean and updating the program counter can take up to 5 nanoseconds (equivalent to a slow 200 MHz clock rate). To avoid stalling the entire execution pipeline during that window, the CPU's branch predictor guesses the outcome cycles in advance and speculatively executes downstream instructions. GPUs eliminate much of this die-area overhead by stripping out aggressive specualtive execution logic, tightening register files, and scheduling massive numbers of parallel threads instead.
- A modern high-end CPU die supports roughly 1,000-way total parallelism (e.g., 100 individual cores with 16-wide vector pipelines).
- Branch evaluation latency in a CPU pipeline takes approximately 5 nanoseconds, which would cap unpredicted clock speeds at 200 MHz without speculative execution.
- Removing branch predictors and branch target buffers frees up massive die area on GPUs, allowing transistor budgets to be dedicated directly to compute ALUs.
`01:12:00` Brains vs. chips: Clock rates and energy consumption
Comparing human brains to silicon accelerators highlights key architectural differences: the brain relies on fine-grained, unstructured sparsity (any neuron connecting to any other neuron) and operates at extremely slow clock rates. Reiner addresses the common assumption that running silicon chips at a millahertz or hertz frequency like the brain would unlock revolutionary energy efficiency. On silicon, lower clock frequencies do reduce power linearly, but they do not fundamentally alter the underlying physics of energy efficiency.
On a microchip, power is consumed almost entirely when digital bits toggle between 0 and 1. Storing a logical 1 involves depositing electrical charge into a microscopic parasitic capacitor; discharging that energy to ground when returning to 0 consumes energy, a process known as dynamic or switching power. Lowering the clock speed by 1,000x results in 1,000x fewer bit transitions per second, dropping total energy draw by 1,000x while sitting idle, but the net energy required *per operation* remains largely unchanged.
- The vast majority of a chip's power draw stems from dynamic (switching) power: charging and discharging parasitic capacitive nodes during bit state changes.
- Slowing down a GPU's clock speed from 1 GHz to 1 MHz yields roughly 1,000x lower total power draw, but delivers proportional reductions in throughput without dramatic gains in energy per operation.
- Human brains operate with unstructured sparsity and low-voltage biological signaling, whereas silicon chips perform dense, high-throughput matrix calculations at multi-gigahertz speeds.
`01:15:33` Architectures: A GPU is just a bunch of tiny TPUs
Reiner outlines the high-level structural differences between GPUs and TPUs. A GPU layout consists of a highly regular grid made up of dozens of small, nearly identical Streaming Multiprocessors (SMs) clustered around an L2 cache. A TPU uses a much coarser architecture: it features a few massive Matrix Multiply Units (MXUs / systolic arrays) tied directly to a central vector unit. At a conceptual level, a GPU's SM contains its own small Tensor Cores and vector units, making an SM function essentially like a miniature TPU replicated hundreds of times across the silicon die.
```
TPU vs. GPU Architecture
TPU: Coarse Granularity GPU: Fine Granularity
┌─────────────────────────────┐ ┌─────────────────────────────┐
│ [ Huge Matrix Unit (MXU) ] │ │ [SM] [SM] [SM] [SM] [SM] │
│ │ │ │ [SM] [SM] [SM] [SM] [SM] │
│ [ Central Vector Unit ] │ │ ─────── L2 Cache ────────── │
│ │ │ │ [SM] [SM] [SM] [SM] [SM] │
│ [ Huge Matrix Unit (MXU) ] │ │ [SM] [SM] [SM] [SM] [SM] │
└─────────────────────────────┘ └─────────────────────────────┘
Amortizes control overhead; High wiring bandwidth between
requires long data movement. local vector & matrix units.
```
This structural divergence creates clear trade-offs between flexibility, register overhead, and data movement. The large systolic arrays on a TPU amortize register file and control overhead far more efficiently across the matrix grid. However, transferring data between the central vector unit and the peripheral matrix units requires routing dense data through narrow interconnect bottlenecks. Conversely, GPUs tile vector logic directly alongside matrix logic across 16 or more wiring channels per SM, affording much higher internal data bandwidth at the cost of higher warp-scheduling and register control overhead. Reiner concludes by referencing his startup MatX's public work on "splittable systolic arrays"—an architectural hybrid that dynamically behaves as either a large monolithic array or multiple smaller arrays depending on the workload.
- A GPU spreads compute across many fine-grained SMs, whereas a TPU consolidates compute into a few coarse-grained, large-scale systolic arrays.
- Larger systolic arrays amortize control logic and register costs more effectively, but create long physical data paths between vector and matrix units.
- MatX is designing "splittable systolic arrays" to combine the area efficiency of large TPU-style matrix units with the fine-grained flexibility of GPU SMs.
<ElicitationsGroup message="Explore related hardware and accelerator architecture topics:">
<Elicitation label="Explain how systolic arrays process matrix multiplication" query="Explain how systolic arrays process matrix multiplication step-by-step, including data flow and register accumulation."/>
<Elicitation label="Compare TPUs, GPUs, and custom AI ASICs" query="Compare the architectural trade-offs between GPUs, TPUs, Groq LPUs, and custom AI ASICs."/>
</ElicitationsGroup>
## Notable quotes
> `10:00` Bit Width Precision vs. Gate Area Scaling
> `16:31` In a standard CPU or legacy GPU core, the vast majority of energy and die area is consumed not by computing math, but by shuttling bits back and forth between registers and compute logic through dense MUX trees. — Reiner Pope
> `30:00` Bandwidth equals die area — Reiner Pope
> `45:00` Over-pipelining spends almost the entire chip silicon budget on synchronization rather than useful computation, driving up clock frequency while severely hurting overall throughput. — Reiner Pope
> `51:51` FPGA logic density is roughly 10x less efficient than a direct ASIC implementation. — Reiner Pope
> `01:03:25` CPU caches operate roughly 100x faster than off-chip DDR memory, meaning cache misses degrade performance by two orders of magnitude. — Reiner Pope
> `01:07:27` Removing branch predictors and branch target buffers frees up massive die area on GPUs, allowing transistor budgets to be dedicated directly to compute ALUs. — Reiner Pope
> `01:12:00` On a microchip, power is consumed almost entirely when digital bits toggle between 0 and 1. — Reiner Pope
> `01:15:33` At a conceptual level, a GPU's SM contains its own small Tensor Cores and vector units, making an SM function essentially like a miniature TPU replicated hundreds of times across the silicon die. — Reiner Pope
AI accelerator efficiency is governed less by raw arithmetic logic than by the silicon area and energy required to route data. In a technical lecture, MatX CEO Reiner Pope explains how hardware architectures evolve from basic logic gates and systolic arrays to full chip layouts by optimizing the physical trade-offs of data movement, precision, and synchronization.
Arithmetic Primitives and Precision Scaling (0:00)
At the circuit level, artificial intelligence workloads are driven by matrix multiplication, making the Multiply-Accumulate (MAC) unit the fundamental computational primitive. Multiplying two numbers of bit widths P and Q and accumulating the output requires P × Q AND gates to generate partial products, alongside P × Q full adders (3-to-2 compressors) to sum them into an accumulator. Because accumulation requires higher precision to prevent compounding numerical error, the hardware footprint is tied directly to the bit width of the inputs.
Pope notes that hardware units are generally not fungible across precisions; a dedicated FP4 unit requires distinct silicon layout compared to an FP8 unit. Multiplier area scales quadratically (P × Q) with bit precision, meaning halving bit width shrinks multiplier logic area by roughly fourfold. While earlier architectures like NVIDIA's Hopper scaled spec-sheet throughput linearly due to memory bus constraints, newer designs like Blackwell exploit this quadratic scaling to offer threefold to fourfold speedups at FP4 compared to FP8.
The Silicon Tax of Data Movement (20:00)
The dominant constraint in general-purpose processor design is moving data rather than calculating operations. In conventional CPU or legacy GPU architectures, Arithmetic Logic Units (ALUs) fetch operands from register files through multiplexers (muxes). Routing an n-row register file of p-bit data into a MAC unit requires 3 × n × p AND gates, compared to only p × q gates inside the compute core itself. In a concrete configuration where n = 8 and q = 4, data routing requires 24p gates while the arithmetic unit requires only 4p gates.
As a consequence of this routing penalty, traditional compute cores prior to architectures like NVIDIA Volta expended approximately 87.5% of their physical execution block area on multiplexing and internal wiring, leaving only 12.5% for useful math. Managing this data movement overhead is the primary motivation for creating specialized AI hardware.
Systolic Arrays and Pipelining Constraints (26:10)
To bypass register-file bottlenecks, AI accelerators deploy systolic arrays (tensor cores), which hardwire the outer loops of matrix multiplication directly into silicon. By storing weight matrices locally inside spatial registers within the execution grid, the hardware achieves X × Y operations for only X units of input and output bandwidth. To keep physical wire counts manageable, weights are trickle-fed via linear shift registers across consecutive cycles rather than loaded simultaneously.
Maximizing execution throughput also requires balancing clock frequency against physical pipeline depth. Inserting pipeline registers breaks long propagation paths, allowing chips to run at higher clock frequencies (such as 2 GHz). However, feedback loops—such as accumulators that continually add new terms to a running sum—cannot be pipelined without altering mathematical behavior. Furthermore, because registers consume roughly eight times the silicon area of primitive logic gates, over-pipelining inflates the chip's synchronization budget at the expense of computational density.
Architectural Divergence Across ASICs, FPGAs, and Processors (51:51)
Specialized Application-Specific Integrated Circuits (ASICs) achieve vastly higher density and efficiency than Field-Programmable Gate Arrays (FPGAs). FPGAs achieve post-manufacturing reconfigurability by emulating logic through Lookup Tables (LUTs) and multiplexer routing matrices; implementing a 4-input AND function consumes roughly 32 logic gates inside an FPGA LUT compared to just 3 direct gates on an ASIC, rendering FPGAs roughly an order of magnitude less area-efficient.
At the system architecture tier, CPUs sacrifice die area to massive multi-level caches and complex branch predictors that speculate cycles ahead to prevent five-nanosecond pipeline stalls. In contrast, AI accelerators omit speculative branch prediction, replace dynamic CPU caches with compiler-managed scratchpad memories to ensure deterministic execution latency, and schedule massive parallel threads. Comparing accelerator layouts, Pope observes that a GPU is essentially a collection of hundreds of small, fine-grained TPU cores (Streaming Multiprocessors) clustered around an L2 cache to maximize local bandwidth, whereas a Google TPU uses large, coarse-grained systolic arrays to amortize control logic over longer data paths.
Pope illustrates that modern chip design is dictated by physical routing bottlenecks, memory synchronization, and area trade-offs rather than raw logic speed. To resolve the architectural divide between the fine-grained data routing of GPUs and the area-efficient control amortization of large TPU arrays, startup MatX is developing splittable systolic arrays that can dynamically alternate between both operational modes.
精炼阅读摘要
MatX 首席执行官 Reiner Pope 在访谈中指出，AI 芯片架构设计的核心瓶颈并非单纯的算术逻辑单元，而在物理连线与数据搬运的硬件开销；现代硬件通过脉动阵列、片上暂存器与软硬件协同设计，以牺牲传统通用性为代价来最大化计算面积效率。
算术基元与数据搬运开销 (0:00)
现代 AI 负载的核心是矩阵乘法中的乘加操作（MAC）。在门电路层面，P 位乘以 Q 位整数需消耗 P×Q 个与门生成部分积，并借助等量全加器（3-to-2 压缩器）将多输入项归约至累加精度。由于累加位宽通常大于乘法输入以防止误差累积，乘加单元在物理面积上具有 P×Q 的对称规整性。算术位宽的减半可在整数逻辑上带来约 4 倍的门面积缩减，但在 Blackwell 等最新架构之前，硬件厂商的标称算力往往因内存总线对齐限制仅呈线性翻倍，且浮点单元还需承担额外的指数对齐和规格化逻辑开销。
然而，通用计算面临的主要矛盾在于数据搬运的代价远超算术计算本身。在传统的标量或经典 GPU 核心中，从包含 n 个寄存器的寄存器堆向算术逻辑单元（ALU）路由 p 位数据时，多路复用器（MUX）需要消耗 3×n×p 个与门及大量或门。以 n=8、q=4 为例，仅用于数据寻址选择的多路复用器逻辑门数量便达到 24p，而底层乘加逻辑仅需 4p。在 NVIDIA Volta 架构之前的传统 CUDA 核心中，约 87.5%（即 7/8）的硅片面积被数据路由开销占用，真正的计算逻辑仅占少数。
脉动阵列的面积与带宽权衡 (26:10)
为消除寄存器堆路由所带来的硅片面积损耗，专用 AI 加速器引入了张量核心（即脉动阵列）。该设计直接在硬件中固化矩阵乘法外层循环，将权重矩阵直接保存在处理阵列的局部空间寄存器中，仅让向量流水穿过。这种设计以线性的边界输入输出带宽（X）换取了二次方的计算密度（X×Y）。早期 Google TPU 便将这一理念扩展到了 128×128 的庞大乘加阵列。
由于芯片面积对引脚与走线极为敏感，硬件通常采用逐周期级联下移的“滴流式”（trickle-feed）加载策略输入权重，将跨越边界的物理布线严格限制在 X 规模。在芯片微架构规划中，工程团队通常将执行单元中约 90% 的硅片面积划归脉动阵列，仅保留约 10% 供给数据搬运与寄存器。扩大寄存器堆虽然能提升编译器灵活性和通用可编程性，但必然会直接削减脉动阵列的峰值算力吞吐。
流水线时钟与反馈回路限制 (40:00)
现代拥有千亿级晶体管的芯片依赖全局时钟周期进行微同步。为规避硅片制造公差引发的竞争冒险，设计时通常在组合逻辑云之间插入寄存器，并预留保守的建立时间裕度（例如将信号传播延迟控制在时钟周期的 75% 以内）。虽然在前馈逻辑中切分组合逻辑并插入流水线寄存器可将时钟频率提升至数吉赫兹（GHz），但在累加器等输出回馈至输入的反馈回路中，插入流水线寄存器会直接改变数学语义，导致流水线拆解为奇偶交错的独立子累加。
台积电等晶圆代工厂通过工艺设计套件（PDK）提供的基本门电路传播延迟约为 10 皮秒，典型时钟周期仅能容纳 10 到 30 个串行原始逻辑门。尽管过度流水线化理论上能冲击超过 5 GHz 的超高频率，但单个寄存器占用的硅片面积（约 8 个门等效单位）远高于基础门电路（1 个单位）。过度流水线化会使芯片的大部分硅片预算浪费在同步而非运算上，虽然降低了单任务延迟，却严重压低了单位芯片面积的总体吞吐量。
FPGA与ASIC的架构取舍 (51:51)
FPGA 与 ASIC 共享门电路与时钟同步模型，但两者的实现成本截然不同。ASIC 单颗芯片流片初期投入可超 3000 万美元，但在单片单位成本与能效上具有数量级优势；现成 FPGA 单颗采购价约 1 万美元，可在现场反复重新配置。然而，FPGA 的现场可编程性依赖于查找表（LUT）中的 16×1 SRAM 静态存储与多路复用网络。实现一个简单的 4 输入与门，ASIC 仅需 3 个物理门电路，而 FPGA 查找表及其内部多路复用器等效开销达约 32 个逻辑门，造成近 10 倍的面积膨胀与延迟劣化。
在内存架构层面，传统 CPU 的不可预测延迟源于硬件自主管理的动态缓存系统：缓存命中速度比访问板外 DDR 内存快近两个数量级，且受操作系统后台任务扰动。量化高频交易以及 TPU、Groq 等 AI 芯片为了实现绝对确定的周期延迟，彻底移除了硬件缓存的隐式行为，改用软件显式控制的暂存器（Scratchpad，通常搭配 HBM）。编译器通过独立的显式指令分别调度本地暂存器读写与外部高带宽内存搬运，从而在硬件底层确立了确定性执行时序。
CPU、GPU与TPU的宏观拓扑 (67:27)
CPU 核心之所以物理面积庞大，是因为需分配大量硅片用于深层缓存、超大寄存器堆以及复杂的分支预测器。CPU 指令流水线深度极大，评估条件分支可能消耗 5 纳秒，若无投机执行机制，时钟频率将被压制在 200 MHz 以下。GPU 则彻底剔除了激进的投机执行逻辑，将面积腾出给海量计算单元，利用极高密度的并发线程调度来掩盖内存延迟。
在宏观拓扑上，GPU 由围绕 L2 缓存紧密平铺的众多细粒度流式多处理器（SM）构成，每个 SM 内部都紧邻配置了张量核心与向量逻辑；TPU 则采用粗粒度布局，将少数庞大的矩阵乘法单元（MXU）挂载在中央向量单元两侧。GPU 的微架构拥有更高的局部互连带宽（每个 SM 往往具备 16 条以上的布线通道），但控制和调度开销较高；TPU 的庞大阵列能充分摊薄控制开销，却带来了中心与外围之间长距离互连布线的数据拥堵瓶颈。针对这一结构矛盾，Reiner Pope 提及 MatX 正在开发“可拆分脉动阵列”，试图根据工作负载在宏观单体阵列与微观分离阵列之间动态切换。
芯片设计的本质是面积、带宽与控制灵活性的物理妥协。尽管生物大脑在超低频与极度稀疏连接下展现出极高能效，但基于半导体物理的芯片主要能耗来自电容充放电的动态翻转，单纯调低频率并不能降低单次操作功耗；未来的专用加速器仍将围绕确定性内存搬运和自适应脉动结构寻求演进。
Detailed notes — zh
## Background
本期内容由 Dwarkesh Patel 于 2026 年 5 月 22 日在其播客频道发布，采访了芯片初创公司 MatX 联合创始人兼 CEO Reiner Pope。节目以黑板板书推导的形式展开，深入探讨了芯片从最底层的逻辑门、乘累加原语，一步步构建出 GPU、TPU、FPGA 等现代算力架构乃至对照人脑工作机制的完整物理演进过程。
## TL;DR
节目从二进制长乘法与与门、全加器（3:2 压缩器）等门级电路出发，严密推导了乘累加单元（MAC）的物理面积与开销，揭示出经典通用架构中多路复用器（MUX）数据搬运成本远超算术单元的残酷现实。随后，Reiner 阐释了脉动阵列（Systolic Array）如何通过权重就地驻留以 $O(X)$ 的线性通信撬动 $O(X \times Y)$ 的二次方算力，并深入对比了 FPGA 查找表（LUT）、CPU 复杂分支预测器与 TPU 暂存器（Scratchpad）的设计取舍。最后，访谈从硅片顶层布局将 GPU 拆解为“密集平铺的微型 TPU”，剖析了矩阵单元与向量单元的数据搬运带宽瓶颈，并展望了兼顾吞吐与灵活性的可拆分脉动阵列架构。
## Atmosphere & style
整场对话洋溢着纯粹的硬核极客精神与扎实的工程美感，黑板板书与公式推导营造出一种如同顶尖大学微体系结构研讨课般的沉浸式学术氛围。Dwarkesh 善于从软件工程师和系统架构的角度敏锐提问，而曾深耕 Google TPU 与编译优化的 Reiner 则总能用底层晶体管计数、金属走线延迟与物理直觉予以严密拆解。两人之间的讨论节奏紧凑、思维碰撞激烈，既有层层破译芯片设计黑盒的知性快感，又处处闪烁着在硅片面积、延迟与吞吐之间进行极致权衡的工程智慧。
## Bulletin
- `12:45` 乘加器电路面积与比特位宽呈二次方缩放关系（P×Q），但传统核心中约八分之七的硅片面积被寄存器堆与多路复用器（MUX）的数据路由开销吞噬，而非算术计算本身。
- `27:31` 脉动阵列通过将权重固定在计算单元本地并将矩阵乘法外层循环硬化到硬件中，实现了算力以二次方（X×Y）增长而通信引线规模仅保持线性（O(X)）增长。
- `51:51` FPGA 通过查找表（LUT）和多路复用器网络通用模拟逻辑门，其面积开销与能耗比专用 ASIC 劣势约 10 倍，但免除了数千万美元的前期流片费用。
- `01:15:33` GPU 本质上是密集平铺的大量微型 TPU（每个 SM 包含小型 Tensor Core），相比 TPU 的超大脉动阵列，GPU 在向量与矩阵单元间具备更高的局部互连带宽，但固定硬件开销也更高。
## Key takeaways
- `00:00` 矩阵乘法核心是三层嵌套循环的乘累加（MAC），硬件上采用低精度乘法配合高精度累加，以防止沿循环维度的多次舍入误差级联。
- `04:30` 计算 P 位与 Q 位整数乘法需要 P×Q 个与门生成部分积，再通过 Dadda 乘法器结构中的 P×Q 个全加器进行 3:2 压缩归约。
- `12:45` 乘法器面积随输入位宽呈二次方（P×Q）缩放，使得低比特运算（如 FP4 相较于 FP8）具备远超线性比例的理论硅片面积与能效收益。
- `15:33` 经典处理器数据通路中，为了支持随机访问寄存器堆，仅多路复用器（MUX）的门电路开销就可达实际 MAC 逻辑单元的数倍。
- `30:00` 脉动阵列的核心架构创新在于将权重本地驻留在运算单元中，以 O(X) 的边缘引线带宽支撑 O(X×Y) 的二维算力密度。
- `32:45` 利用模型权重长期固定的特征，脉动阵列采用菊链式逐行慢速移位加载权重，用时钟周期换取珍贵的芯片互连引线与面积。
- `35:10` 芯片架构师的核心权衡是阵列与存储的尺寸配比：更大的缓存虽能提升编译调度灵活性，但会挤压脉动阵列面积并削减理论峰值 TOPS。
- `38:50` 芯片时钟周期通过流水线寄存器强制百亿晶体管实现纳秒级全局同步，以克服半导体制造公差带来的物理延迟波动。
- `46:12` 流水线切分虽可缩短关键路径以提高时钟频率，但受到反馈环路（如累加器闭环依赖）的限制，且寄存器面积（~8 GE）远高于基础门电路（~1 GE）。
- `56:45` FPGA 使用 4 输入真值表（4-LUT）模拟任意逻辑门，实现一个简单门电路需消耗数十个等效门与配置位，导致其在能效与面积上相比 ASIC 差约一个数量级。
- `01:03:25` 现代 CPU 因透明硬件缓存（Cache）引发的命中状态差异而具有非确定性延迟，而 TPU 等 AI 芯片采用软件显式调度的暂存器（Scratchpad）来换取确定性。
- `01:07:27` CPU 核心单核面积远大于 GPU 核心，主要原因在于 CPU 内部集成了庞大的分支预测器与乱序执行控制逻辑，以防止高频流水线因条件跳转停顿。
- `01:12:00` 降低芯片时钟频率并不等于获得人脑级别的超高能效，因为硅芯片主要功耗来自电容充放电的动态翻转，单纯降频并不能减少单次计算所需的能量消耗。
- `01:15:33` TPU 采用粗粒度大矩阵单元架构以摊薄控制开销，而 GPU 采用细粒度 SM 平铺架构以换取更高的向量与矩阵间瞬时数据交换带宽。
- `01:20:00` MatX 提出的可拆分脉动阵列试图在大阵列高吞吐摊薄开销与小阵列细粒度灵活性之间实现动态兼顾。
## Detailed walkthrough
### `00:00` 芯片设计的底层基石与乘累加原语
Dwarkesh Patel 再次邀请到 MatX 联合创始人兼 CEO Reiner Pope。在上一期讨论了数据中心层面的宏观运作后，这一次两人转向芯片内部的微观世界，从最基础的逻辑门一步步推导出 AI 芯片的完整架构。Dwarkesh 开场打趣地披露自己是 MatX 的早期天使投资人，“希望你们确实设计出了一颗好芯片”，并提醒听众这期黑板教学更适合在 YouTube 或 Spotify 上观看板书推导。Reiner 指出，芯片最底层的原始物理构件是极其简单的逻辑门（如与门 AND、或门 OR、非门 NOT），它们通过物理上铺设在芯片上的金属走线（metal traces）相互连接。
AI 芯片最核心的运算任务是矩阵乘法，而矩阵乘法的绝对核心原语是成对数值的乘累加（Multiply-Accumulate, 简称 MAC）。当 Dwarkesh 询问为什么 MAC 会成为计算机内部最天然的计算原语时，Reiner 解释道，如果将矩阵乘法写成代码，本质上就是一个关于三层循环的结构：对 $i$、$j$、$k$ 进行遍历，每次执行 `output[i][k] += input[i][j] * other_input[j][k]`。在矩阵乘法的每一个单步中，都在无休止地重复着乘累加操作。
Reiner 进一步强调了精度设计上的关键权衡：在 AI 芯片中，累加项的位宽几乎总是高于乘法输入项的位宽。这背后的核心原因在于，虽然低精度乘法在单次链条中只会产生一次舍入误差，但在随后的累加阶段，成百上千个数值被不断叠加在一起，每一次舍入误差都会迅速累积。因此，为了维持数值稳定性，芯片工程师会特意采用低精度乘法配合高精度累加的设计。在接下来的板书推导中，Reiner 选择了一个极具代表性且易于手算的具体案例：用一个 4-bit 整数乘以另一个 4-bit 整数，并将乘积累加到一个 8-bit 整数中。
- 矩阵乘法本质：三层嵌套循环（$i, j, k$），核心计算即 `output[i][k] += input[i][j] * other_input[j][k]`
- 精度权衡：乘法链条只有一次误差，而沿着循环维度 $j$ 的累加会产生大量舍入误差的级联，故累加器必须保留更高位宽（如本例中的 4-bit 乘法配 8-bit 累加）
### `04:30` 手算长乘法与与门（AND Gate）生成部分积
为了展示硬件电路如何计算乘累加，Reiner 带领 Dwarkesh 在黑板上用二进制长乘法一步一步手动展开计算。假设我们将 4-bit 数 $1001_2$ 与另一个 4-bit 数逐位相乘，乘数（multiplier）中的每一个二进制位都会与被乘数相乘，产生一行部分积（partial product）：
- 乘以最低位的 1 时，得到原数 $1001_2$；
- 乘以左移一位的 0 时，得到全零行 $0000_2$；
- 乘以再左移一位的 1 时，再次得到左移对齐的 $1001_2$；
- 乘以最高位的 0 时，得到全零行 $0000_2$。
在这 4 行乘积展开之后，Reiner 直接将待累加的 8-bit 数作为第 5 行写入矩阵下方，形成了一个包含 5 行数值的加法求和阵列。Reiner 提问：要得到这一步中间结果，我们在物理上需要消耗多少逻辑门？答案是 16 个与门（AND Gate）。对于单比特乘法而言，只有当被乘数的某一位是 1 且乘数的对应位也是 1 时，结果才为 1；只要其中任何一个是 0，乘积即为 0，这正是与门的真值逻辑。由此可以推广出普遍规律：计算一个 $p$-bit 整数与一个 $q$-bit 整数的乘积，硬件前端必须消耗 $p \times q$ 个与门来并行生成所有部分积。
```mermaid
flowchart TD
accTitle: 4位乘法部分积生成与累加结构
accDescr: 展示两个4位输入通过16个与门生成4行部分积，并与8位累加数组成5行加法阵列的过程
subgraph Inputs["输入操作数"]
A["4-bit 被乘数: 1001"]
B["4-bit 乘数"]
C["8-bit 待累加值"]
end
subgraph AND_Array["16 个与门 (AND Gates) 阵列"]
PP0["部分积 0 (偏移 0 位): 1 0 0 1"]
PP1["部分积 1 (偏移 1 位): 0 0 0 0"]
PP2["部分积 2 (偏移 2 位): 1 0 0 1"]
PP3["部分积 3 (偏移 3 位): 0 0 0 0"]
end
subgraph Addition_Stage["多操作数求和阵列"]
SumArray["5 行待求和位阵列<br/>(4 行部分积 + 1 行 8-bit 累加项)"]
end
A --> AND_Array
B --> AND_Array
AND_Array --> SumArray
C --> SumArray
```
### `06:45` 全加器：芯片中作为“3:2压缩器”的巨型逻辑单元
生成部分积仅仅是开端，绝大多数硬件工作量和电路面积其实消耗在后续的求和阶段。Reiner 对比了两种极端门电路的尺寸：与门是芯片上几乎最小、最简单的逻辑门；而在算术逻辑单元中，最庞大且最复杂的门电路之一是全加器（Full Adder）。在软件工程师的直觉里，一个加法器似乎能直接把两个 32-bit 数相加，但在底层物理硬件中，一个最基础的全加器仅仅负责将 3 个单比特（1-bit）输入相加。
Reiner 强调，全加器在本质上就是一个“3-to-2 压缩器”（3 to 2 Compressor）。它的输入是同一列（同等位权）的 3 个 bit，其相加的数值结果只可能是 0、1、2 或 3，恰好可以用 2 个 bit 的二进制数来完整表达：
- 输入 `0+0+0` $\rightarrow$ 输出 `00`（数值 0）
- 输入 `1+0+0` $\rightarrow$ 输出 `01`（数值 1）
- 输入 `1+0+1` $\rightarrow$ 输出 `10`（数值 2）
- 输入 `1+1+1` $\rightarrow$ 输出 `11`（数值 3）
这两个输出 bit 拥有不同的位权：一个是留在本列的本位和（Sum），另一个则是需要向左侧相邻高位列传递的进位（Carry Out）。人类在草稿纸上做竖式加法时，习惯在同一列把所有数字加总并在脑海里“记住进位”；但硬件电路无法像人脑那样悬挂记忆，必须显式地将进位 bit 输出并布线到相邻列。随后 Reiner 在黑板上演示了全加器对前面生成的 5 行待求和阵列的逐步化简：从最右侧最低位列开始，选取该列中的 bit 送入全加器，生成输出 bit 和进位 bit，将处理过的输入划掉，再依次向左侧高位列推进。通过将 3 个 bit 的输入压缩为 2 个 bit，多行复杂的二进制点阵得以逐层压缩，最终化简为硬件最终可以输出的确定数值。
| 输入状态 (3 bits) | 算术求和值 | 二进制输出 (Carry, Sum) | 功能说明 |
|:--- |:--- |:--- |:--- |
| `0, 0, 0` | 0 | `0 0` | 本位为 0，无进位 |
| `1, 0, 0` / `0, 1, 0` / `0, 0, 1` | 1 | `0 1` | 本位为 1，无进位 |
| `1, 1, 0` / `1, 0, 1` / `0, 1, 1` | 2 | `1 0` | 本位为 0，向高位进位 1 |
| `1, 1, 1` | 3 | `1 1` | 本位为 1，向高位进位 1 |
### `10:00` Dadda 乘法器与乘累加的电路规模计算
在上一节推导的基础上，Reiner Pope 进一步演示了如何将整个点阵中需要相加的数归约成最终结果。整个过程的核心逻辑是：将需要相加的比特网格看作一个整体，在每一列上不断套用全加器（Full Adder）。每个全加器每次从某一列中吃掉 3 个输入比特，同时吐出 2 个输出比特（本位和与向高位的进位）。这个过程不断重复，反复消耗网格中的比特，直到最终每一列只留下一个数，从而得到最终的和。这种利用全加器网格高效压缩位宽的结构被称为 Dadda 乘法器（Dadda multiplier），是硬件设计中兼顾面积与效率的标准全加器乘法器实现。
为了让后续对比更有量化依据，Reiner 带着 Dwarkesh Patel 精确量化了这个电路的逻辑门规模：
1. **起始比特数**：假设两个输入数位宽分别为 $P$ 和 $Q$，则部分积矩阵包含 $P \times Q$ 个比特（在 4 位示例中为 16 个），加上累加项所需的 $P + Q$ 个比特（在 4 位示例中为 8 个），初始总比特数为 $(P \times Q) + (P + Q) = 24$ 个。
2. **输出比特数**：最终输出结果的位宽为 $P + Q$ 位（即 8 位）。
3. **全加器用量**：由于每个全加器吃进 3 个比特、产出 2 个比特，每次运算净消除恰好 1 个比特。因此，要从初始比特数削减到最终比特数，所需的全加器数量恰好是：
$$(P \times Q + P + Q) - (P + Q) = P \times Q$$
对于 4 位乘法，电路里刚好需要 16 个全加器。
Dwarkesh 梳理完这一步后恍然大悟：这完美揭示了为什么 AI 芯片从一开始就选择做乘累加（MAC, Multiply-Accumulate）单元。第一个原因当然是矩阵乘法天然由乘累加构成；而硬件设计层面的第二个美妙原因，则是乘累加让电路的代数结构变得极其规整漂亮，全加器数量直接等于输入位宽的乘积 $P \times Q$。在物理芯片实现中，白板上每消除 3 个比特并写出 2 个比特的原子步骤，在硅片上就对应着 3 根输入金属导线接入一个逻辑门，再引出 2 根输出线。
```mermaid
flowchart TD
accTitle: Dadda乘累加器电路比特归约逻辑
accDescr: 展示Dadda乘累加器如何通过全加器将输入比特和累加比特逐步压缩为最终结果
subgraph Inputs["输入比特集合 (共 P×Q + P + Q 节点)"]
direction TB
PP["部分积矩阵网格<br/>P × Q 个比特 (例如: 16)"]
ACC["累加操作数<br/>P + Q 个比特 (例如: 8)"]
end
subgraph Reduction["Dadda 压缩网格 (全加器阵列)"]
FA["每个全加器 (Full Adder):<br/>消耗 3 个输入比特 → 产出 2 个输出比特<br/>每次净消除 1 个比特"]
end
subgraph Outputs["最终输出"]
OUT["最终结果位宽<br/>P + Q 个比特 (例如: 8)"]
end
PP --> FA
ACC --> FA
FA -->|"重复归约 P×Q 次"| OUT
```
---
### `12:45` 精度与芯片面积的二次方法则：FP4 vs FP8 的真实算力折算
在理清基础电路后，Dwarkesh 提出了一个常见疑惑：NVIDIA 在宣传新架构规格时，常常宣称 FP4 的算力（FLOPs）正好是 FP8 的 2 倍，这是否意味着底层硬件电路是完全通用（fungible）的？即一个 FP8 的乘法器可以在软件上直接劈成两个 FP4 乘累加来用？
Reiner 明确指正，在物理电路上它们并不完全通用。芯片架构师在设计之初就必须做出权衡取舍：分配多少晶体管面积给 FP4，分配多少给 FP8，或者按照功耗预算在不同精度之间做平衡。至于为什么官方标称算力往往刚好是 2 倍，背后有两层现实考量：
1. **数据搬运与总线打包（Packing）**：在系统架构和内存层次结构中，将两个 4-bit 浮点数打包进一个 8-bit 存储位宽极其自然，片内数据总线（bus）的位宽与存储颗粒设计由此能够严丝合缝地成倍对齐。
2. **算力与面积的二次方缩放定律（Quadratic Scaling）**：根据上述推导，乘法器的全加器电路面积与比特位宽呈**二次方关系**（$P \times Q$）。因此当位宽减半时，乘法器本身的理论面积缩小到了原来的四分之一。
如果电路开销是二次方缩放，为什么算力提升不是 4 倍而是 2 倍？Reiner 解释道：
- 历史上直到 Blackwell 架构（B100 / B200）之前，NVIDIA 确实保持着“位宽减半、算力翻倍”的宣传统径，但严格来说这个比例低估了低精度的实际收益。
- 真实的浮点运算（Floating Point）不仅有尾数乘法，还有指数对齐和处理逻辑，因此不会是完全纯粹的 4 倍整倍数。
- 但从 Blackwell 及后续芯片规格开始，NVIDIA 已经开始在产品参数上承认这种非线性优势——例如 Blackwell 宣称的 FP4 吞吐量达到了 FP8 的 3 倍（虽然理论极限接近 4 倍）。这种电路面积随位宽缩小的二次方缩放特性，正是低比特算术在神经网络中取得巨大成功的核心物理原因。
---
### `15:33` 算力背后的真正瓶颈：经典核心中的数据搬运与 MUX 开销
确立了计算单元本身的面积开销后，Reiner 开始将视角转向算力单元周边的配套电路，探讨到底是什么在吃掉芯片的面积和功耗。他带领观众回顾了 Tensor Core 出现之前的经典架构（传统 CPU 或早期 NVIDIA GPU 的 CUDA Core）：
一个标准的通用处理器数据通路（Data Path）由**寄存器堆（Register File）**和**算术逻辑单元（ALU / MAC 单元）**组成。为了让计算单元工作，寄存器堆通常包含若干条目（例如 8 个深度），每个条目可能是 4 位或标准的 32 位。在这个架构中，乘累加单元（MAC）需要从寄存器堆的任意 3 个位置随机读取操作数，完成运算后再写回任意一个寄存器。
为了支持这种能够“任意挑选操作数”的通用灵活性，硬件必须在寄存器堆和 ALU 之间插入**多路复用器（MUX, Multiplexer）**。虽然在软件层面，程序员感觉只是执行了一条指令“读取 3 号寄存器”，但在硅片底层，由于无法物理动态改道连线，必须用与门（AND）和或门（OR）构建一个庞大的 MUX 选择网络：
- 面对具有 $N$ 个条目（例如 8 行）、每条包含 $P$ 个比特的寄存器堆，选择其中一行的最笨也是最通用的物理做法是：构造一个掩码信号。
- 对每一行的每一个比特，都必须连接一个与门（AND gate），以此决定是否放通该行数据，这里总共需要 $N \times P$ 个与门。
- 随后，必须将所有行的对应比特通过或门（OR gate）逐级汇总在一起，这需要 $(N - 1) \times P$ 个或门。
由此可见，为了支持自由选择数据的灵活性，仅仅一个输入端口的 MUX 所耗费的逻辑门数量，就已经随着寄存器数量和位宽急剧膨胀。数据搬运与选择逻辑所占用的晶体管面积，往往迅速盖过了乘法器本身。
```mermaid
flowchart LR
accTitle: 经典核心中的数据通路与多路复用器开销
accDescr: 展示寄存器堆通过三路多路复用器向MAC单元提供操作数并写回的过程
subgraph RegFile["寄存器堆 (Register File)"]
direction TB
R0["Reg 0"]
R1["Reg 1"]
R2["Reg 2"]
Rdots["... (共 N 个寄存器)"]
RN["Reg N-1"]
end
subgraph MuxNetwork["输入多路复用逻辑 (MUX 网络)"]
direction TB
MuxA["MUX A<br/>(N×P AND门 + (N-1)×P OR门)"]
MuxB["MUX B<br/>(N×P AND门 + (N-1)×P OR门)"]
MuxC["MUX C<br/>(N×P AND门 + (N-1)×P OR门)"]
end
subgraph Compute["计算单元"]
MAC["乘累加单元 (MAC)<br/>P × Q 个全加器"]
end
RegFile --> MuxA
RegFile --> MuxB
RegFile --> MuxC
MuxA -->|"操作数 A"| MAC
MuxB -->|"操作数 B"| MAC
MuxC -->|"累加数 C"| MAC
MAC -.->|"写回结果"| RegFile
```
### `20:00` 数据搬运的隐藏代价：多路复用器（Mux）比计算单元昂贵得多
在硬件设计的微观视角下，软件工程师习以为常的操作往往隐藏着高昂的电路代价。Dwarkesh 提到，在软件层面上，程序员通常只觉得“我只是读了个第 3 号寄存器而已”，但在底层硬件中，仅仅是这样一个看似基础的选择操作，本身就是一个相当复杂的电路。Reiner 指出，这正是所有隐藏数据搬运成本（hidden data movement costs）的起点。
为了直观展现这一差距，Reiner 对比了数据搬运与实际计算的门电路开销：
一个乘累加（Multiply-Accumulate, MAC）操作需要接收 3 个输入（乘数 $A$、乘数 $B$ 以及累加值 $C$）。如果每个输入都要从一个包含 $n$ 个条目的寄存器堆中读取，且每个条目的位宽为 $p$ 位，那么仅仅为了从这 $n$ 个选项中选出一个结果，单个多路复用器就需要 $n \times p$ 个 AND 门。因为有 3 个输入端，所以光是把数据从寄存器堆搬到逻辑单元，就必须复制 3 组选择电路，共计消耗约 $3 \times n \times p$ 个 AND 门。
与之相对的是真正执行计算的逻辑单元。一个位宽为 $p$ 的乘加单元，其核心门电路数量大约只需 $p \times q$ 个逻辑门（其中 $q$ 是一个小常数，例如 4）。如果带入具体数字，假设寄存器堆的规模很小，只有 $n = 8$ 项，而乘加单元的 $q = 4$：
数据搬运部分仅 AND 门就需要 $3 \times 8 \times p = 24p$ 个门电路；而实际负责乘加运算的核心逻辑单元却仅仅消耗 $4p$ 个门电路。换句话说，在这个极简模型里，仅仅是将数据从寄存器搬运到计算核心的代价，就已经达到了计算单元本身开销的数倍。
- **三输入 MAC 的多路复用开销**：$3 \times n \times p$ 个门电路（$n$ 为寄存器数量，$p$ 为数据位宽）。
- **计算核心开销**：约为 $p \times q$ 个门电路（$q$ 通常仅为 4 左右）。
- **具体开销倍数**：当 $n=8$ 时，数据搬运开销（$24p$）与实际计算开销（$4p$）相差整整 6 倍，意味着电路中几乎绝大部分面积都被用在了数据路由而非计算上。
---
### `21:10` 赞助商插播：Crusoe 算力云与集群可靠性
在此处插播的赞助内容中，Dwarkesh 介绍了 SemiAnalysis 近期发布的 ClusterMAX 报告。在评测的近 100 家 GPU 云服务商中，Crusoe 是仅有的 5 家入选“黄金梯队”（Gold Tier）的云厂商之一。报告显示，即便在 GPU 标称单价相同的情况下，黄金级服务商的总体拥有成本（TCO）也能比白银级低 5% 到 15%。
这种成本优势主要来自标价之外的系统优化能力，例如故障检测与节点替换速度。Crusoe 是首批接入英伟达 NV Sentinel（用于 GPU 状态监控与自愈的系统）的云服务商之一，能够借助英伟达在各大集群积累的故障模式经验尽早捕获隐患。一旦确认硬件故障，Crusoe 可以在 10 分钟内完成健康节点替换，因为他们不运行繁琐的裸金属重新部署流程，而是直接在预先验证好的宿主机上拉起新虚拟机，免去了重新安装操作系统和配置驱动的时间损耗。
---
### `22:16` 多路复用器的门级实现：用 AND 与 OR 选择数据
为了让多路复用器的物理开销更加直观，Dwarkesh 建议在黑板上画出一个具体的 2 路复用器（2-way Mux）电路。Reiner 随即推演了一个在两个 4 位数据行之间做选择的电路结构：
假定系统有两个候选输入数值（每个数值占 4 位），控制器通过独热码（one-hot encoding）的选择器信号来指明目标（例如 `[1, 0]` 表示选第一行，`[0, 1]` 表示选第二行）。在硬件层面，这个选择过程极其繁琐：
1. **位与运算（AND）**：将第一位选择信号与第一行数据的 4 个 bit 分别进行 AND 运算（消耗 4 个 AND 门）；同时将第二位选择信号与第二行数据的 4 个 bit 分别进行 AND 运算（再消耗 4 个 AND 门）。未被选中的那一组数据经与运算后全部强制归零，而被选中的那一组则原样输出。
2. **位或运算（OR）**：将两组产生的输出按位对应输入 4 个 OR 门（消耗 4 个 OR 门）。因为未选中分支全为 0，经 OR 门汇聚后，输出端便毫无失真地保留了目标数值。
```mermaid
flowchart TD
accTitle: Two-Way Multiplexer Circuit
accDescr: Gate-level flow of a 2-way 4-bit multiplexer using one-hot selector, bitwise AND gates, and OR gates to select data.
subgraph Inputs["输入与选择信号"]
InA["数据行 A (4-bit)"]
InB["数据行 B (4-bit)"]
Sel["独热选择器 (One-hot Sel)"]
end
subgraph AND_Stage["按位与门逻辑 (4+4 ANDs)"]
AndA["AND 运算 (Sel[0] & A)"]
AndB["AND 运算 (Sel[1] & B)"]
end
subgraph OR_Stage["按位或门合并 (4 ORs)"]
OrOut["OR 运算 (A' | B')"]
end
FinalOut["最终选中输出 (4-bit)"]
InA --> AndA
Sel --> AndA
InB --> AndB
Sel --> AndB
AndA --> OrOut
AndB --> OrOut
OrOut --> FinalOut
```
Reiner 指出，这套逻辑其实非常像加法器的前置阶段——同样使用了大规模的 AND 门阵列；区别仅在于加法器需要用全加器电路去逐位进位折叠，而多路复用器则是通过 OR 门做扁平化压缩。将其推广到通用场景：如果有 $n$ 行数据，每行 $p$ 个 bit，仅 AND 阶段就需要 $n \times p$ 个与门。
Reiner 总结道，在传统的设计中，“整个电路大约八分之七（7/8）的成本全砸在寄存器堆的读写上了，只有微不足道的一小部分面积真正分给了逻辑计算单元”。这就是芯片设计者必须面对并解决的核心矛盾。
---
### `25:00` 脉动阵列与 Tensor Core 的诞生契机：打破 CUDA 核心的开销瓶颈
在 NVIDIA Volta 架构之前的世代，GPU 内部的通用 CUDA Core 本质上就是上述传统模式的写照：指令一级一级地发射，每次计算只执行一条独立的乘累加操作（MAC），每次都要跨越寄存器堆付出昂贵的多路复用通信开销。软件开发者对此毫无感知，但芯片物理面积却被大量无意义的“数据搬运税”吞噬。
这个根本性瓶颈促成了张量核心（Tensor Core）——即通用学术语境下的“脉动阵列”（Systolic Array）——的诞生。既然整个电路的大部分面积都浪费在了无法产生算力的通信上，架构师的核心思路就是：**在保持寄存器堆和通信接口尺寸不变的前提下，大幅增加本地计算单元的规模**。
- **传统 CUDA Core 的固化层级**：仅仅将最内层循环的“单次乘累加（MAC）”固化为硬件指令。
- **脉动阵列的固化层级**：向上跃迁两个循环层级，将整个矩阵乘法的多层外层嵌套循环直接“焊死”到硬件专用电路中。
通过构建更大粒度（granularity）的定功能硬件逻辑块，多次计算可以在芯片内部本地流动，而无需每次都往返穿梭于寄存器堆，从而把高昂的输入输出通信“税率”分摊降至极低水平。
---
### `27:31` 矩阵乘法的硬件映射：以二次方算力摊薄线性通信开销
Dwarkesh 提炼道，这种架构改动本质上是在矩阵乘法循环中向上走了一步，从而将天平大幅偏向了“计算”，压缩了“通信”。Reiner 证实了这一点，并指出设计中利用了两个核心效应：
1. 每次访问寄存器堆，单次往返能够完成更多的实际计算任务。
2. 矩阵乘法循环中的某些数据具有天然的“驻留性”（某些值在连续计算中保持固定不变）。
Reiner 以最基础的矩阵-向量乘法（Matrix-Vector Multiplication, $M \times v$）为例进行板书演示。若有一个 $2 \times 2$ 的矩阵与一个长度为 2 的列向量相乘，运算过程是矩阵的每一列与向量对应元素相乘后按列累加。矩阵中的每个元素位置都对应一个乘累加（MAC）操作，总共对应 4 个 MAC 单元。
```mermaid
flowchart LR
accTitle: Matrix-Vector Compute and Comm Scaling
accDescr: Illustration showing compute scaling quadratically (X * Y) while communication scales linearly (X) by caching weights.
subgraph Comm["通信开销 (线性 O(X))"]
VecIn["输入向量 (Size 2)"]
Reg["寄存器堆接口"]
end
subgraph Compute["脉动计算阵列 (算力 O(X*Y))"]
direction TB
MAC00["MAC (0,0)"]
MAC01["MAC (0,1)"]
MAC10["MAC (1,0)"]
MAC11["MAC (1,1)"]
end
VecIn --> Compute
Reg -. "避免每周期轮询".-> Compute
```
架构设计的理想目标是：
- **算力提升**：达到 $X \times Y$ 级别的二次方增长；
- **通信带宽**：仅维持在 $X$ 级别的线性增长。
此时，计算相对于通信的效能优势将随着维度 $Y$ 线性拉大。
在上述 $2 \times 2$ 的模型中，输入向量的长度仅为 2，完全落在通信预算内。然而，如果每个时钟周期都要把整个 $2 \times 2$ 矩阵重新从寄存器堆里全量搬运一遍，通信开销就会瞬间击穿预算。Reiner 点出，在 AI 计算场景下，这个矩阵（即模型权重）在相当长的周期内是完全固定不变的。与其每个时钟周期都费时费力地从外部寄存器堆拉取矩阵数据，不如让权重就地驻留（weight-stationary），从而让寄存器堆的数据搬出量严格收敛在线性规模之内。
### `30:00` 脉动阵列的核心秘诀：在逻辑单元旁本地存储权重
如果每个计算周期都要从寄存器堆（register file）引出大量布线来读取矩阵权重，芯片内部的布线拥堵将难以承受。Reiner Pope 解释说，脉动阵列（systolic array）解决这个瓶颈的“核心秘诀”在于：将权重矩阵直接在物理上本地保存在脉动阵列内部。具体而言，就是把像 0、1、2、3 这样的权重数值，直接固化在运算逻辑单元近旁的寄存器（register）中。这样一来，同一个权重矩阵就可以被大量连续流经阵列的不同输入向量反复重用。
Dwarkesh Patel 点出这一架构的精髓：矩阵乘法的数学本质是“高维逻辑”，一个点积（dot product）需要消耗大量的乘法才能凝结出一个标量结果。因此，把这个占用芯片二维空间的二次方（quadratic）存储直接嵌在计算逻辑发生的地方，就可以用极少的输入和输出数据引脚吞吐，在阵列内部“塞入海量的乘法计算”。Pope 接着在黑板上具体演示了数据在空间上的映射：输入向量（例如 3 和 7）横向送入，同时喂给同一行的多个乘法单元；点积的累加则沿着纵向列流动，从列的最顶端输入初始的 0，随着数据逐级向下累加，最终从阵列底部流出计算结果。如此一来，尺寸为 $X$ 的输入和输出向量，进出寄存器堆的数据布线规模仅为 $O(X)$，完全摆脱了 $O(X \times Y)$ 的面积爆炸噩梦。
```mermaid
flowchart TD
accTitle: 脉动阵列矩阵乘法与数据流向
accDescr: 矩阵权重保存在单元内部寄存器中，输入向量横向流动，累加值纵向流动
subgraph Inputs["输入与初始值"]
InX["横向输入向量 (X)"]
ZeroTop["顶部注入零值 (0)"]
end
subgraph SystolicBlock["脉动阵列核心单元"]
Cell["本地权重寄存器<br/>(乘累加单元 MAC)"]
end
subgraph Outputs["计算结果"]
OutY["底部输出结果向量 (X)"]
end
InX -->|"横向输入广播/流动"| Cell
ZeroTop -->|"纵向点积累加向下传递"| Cell
Cell -->|"产生点积输出"| OutY
```
---
### `32:45` 菊链式权重加载：用时间换取芯片面积
既然权重矩阵长期驻留在脉动阵列的单元内部，芯片刚启动或模型切换时，这些权重最初是如何写入阵列的？Pope 揭示了另一个绝妙的权衡设计：既然权重只在极少数时刻被更新，设计者就选择“极其缓慢地”将它们渗透式地灌入阵列。
阵列通常采用类似“菊链”（daisy-chain）的串联移位机制：在一个时钟周期内，只将一行数据写入阵列的最顶层；在接下来的 $Y$ 个时钟周期中，每一周期推入新的一行，同时阵列中现有的行全部向下平移一格。各列并行执行这一过程，耗费 $Y$ 个周期完成整个阵列权重的装填。通过这种方式，跨越脉动阵列边界所需的物理引线依然被严格限制在与 $X$ 成正比的线性量级。
Dwarkesh 梳理了这里的通信本质：“在通信层面有两个维度，一个是通信时间（communication time），另一个是通信带宽（communication bandwidth）。你的意思是，既然我们很久才加载一次，那就最大限度地压缩带宽，因为带宽直接等同于芯片晶圆面积（die area）。”Pope 完全赞同这一概括：用更窄的通信车道慢慢拉入数据，只付出多耗费几个周期的极小代价，就能省下大片珍贵的硅片布线面积。
- **通信时间 vs 通信带宽**：带宽决定了互连线的物理通道数量，即直接消耗晶圆面积（die area）；而传输时间只消耗时钟周期。
- **慢速注入（Trickle Feed）**：通过逐行向下移位寄存，无需为阵列中的每个权重单元都单独连接直通寄存器堆的专用总线。
- **边界线宽维持线性**：跨越阵列边界的连线规模严格保持在 $O(X)$，而不是与单元总数成正比的 $O(X \times Y)$。
---
### `35:10` 计算与通信之争：芯片设计中的尺寸权衡难题
Dwarkesh 联想到之前讨论的多芯片集群推理场景，指出无论宏观还是微观，系统设计的最核心目标惊人一致：尽可能提高“计算量与通信/访存带宽之比”（compute per communication）。Pope 确认了这一点，并指出这种计算与通信的平方-线性关系（甚至立方关系）在整个计算机技术栈中无处不在——从底层选择什么样精度的数值格式（例如 FP32 与低精度 INT8 之间的算术逻辑单元 ALU 面积差距），到中层的脉动阵列拓扑，再到顶层的数据中心级互联。
在脉动阵列这一层级，它已成为人类目前已知实现矩阵乘法最高效的电路架构。例如早期版本的谷歌 TPU（Tensor Processing Unit），其内部的核心脉动阵列规模直接做到了 $128 \times 128$。面对如此庞大的阵列，Dwarkesh 追问：“提高计算相对通信的比重看似显而易见，那么真正让芯片架构师夜不能寐的非直观权衡（non-obvious trade-offs）究竟是什么？”
Pope 坦言，芯片设计中绝大多数令人头疼的决策，本质上都是**尺寸决策（sizing decisions）**：
1. **阵列与缓存的面积博弈**：所有 AI 芯片都由脉动阵列和邻近提供输入输出的寄存器堆构成。设计者必须在面积预算上划定红线，例如将芯片面积的 90% 分给脉动阵列（算力），10% 分给数据搬运和寄存器堆。
2. **灵活性与峰值算力的冲突**：更大的寄存器堆能赋予软件编译器更高的调度弹性，在多变的真实应用负载中压榨出更高的有效性能；但寄存器堆变大，就必然侵占脉动阵列的面积，直接削减芯片理论上的峰值 TOPS。这一参数的微调并没有简单的最优解，始终是架构师反复推敲的难题。
---
### `37:35` 中场插播：Cursor 与论文竞赛评审工具
在讨论时钟周期前，Dwarkesh 插播了一段赞助内容。他提到自己最近发起了一场围绕 AI 重大开放课题的论文竞赛，投稿截止后，他使用 Cursor 快速搭建了两个内部评审界面。其中一个界面对参赛作品进行完全匿名化处理，隐藏作者信息，方便按赛题分组、记录笔记与评分；另一个界面则用于评估那些同时申请他研究员职位的投稿者，将申请者的论文、简历和个人网站并排展示在一套 UI 中。
Dwarkesh 提到，Cursor 的运行环境能够直接在内置浏览器中渲染这些页面、截取屏幕快照、点击各个功能模块并持续自主迭代代码。无论是阅读和可视化复杂的研究论文、编写内部评审工具，还是为黑板讲座制作复习闪卡（flashcards），Cursor 都能让 AI 随时读取屏幕上下文并进行即时协作。
---
### `38:50` 什么是时钟周期：从软件锁到百亿晶体管的纳秒同步
话题重新回到芯片的运作底层，Dwarkesh 抛出了一个基础却核心的问题：芯片的“时钟周期”（clock cycle）到底是什么？它究竟是由什么物理规律决定的？
Pope 首先指出了现代芯片令人震撼的硬件现实：芯片具备人类工程史上最极致的并行性——单一硅片上集成了多达 **1000 亿个晶体管**。而面对如此庞大、复杂的超大规模并行结构，最核心的工程挑战在于：**如何让所有并行运转的独立单元保持同步**。
Pope 对比了软件工程与芯片硬件在同步机制上的根本差异：
- **软件维度的同步代价高昂**：在多线程软件中，同步通常依赖互斥锁（Mutex）等机制。一个线程完成任务后，必须访问内存中的某个锁变量、修改其状态，并显式唤醒另一个挂起等待的线程。这种机制开销巨大且伴随显著的延迟抖动。
- **硬件维度的统一脉搏**：芯片放弃了分散复杂的动态握手逻辑，而是引入了统一的物理心跳。在大约每一纳秒（1 ns，对应约 1 GHz 频率）的时间窗口内，全芯片的所有晶体管必须严格对齐步伐，在统一的时钟节拍下同时将计算结果锁存进流水线寄存器中。硬件用近乎绝对的时间刚性，化解了百亿级晶体管并发运算可能引发的数据紊乱。
### `40:00` 时钟周期、流水线寄存器与关键路径延迟
芯片上的电路并不会杂乱无章地持续流动，而是会在每个极短的瞬间（大约每隔 1 纳秒左右）“停顿并同步一次”，这就是所谓的**时钟周期（Clock Cycle）**。通常整个芯片几乎是在统一步调下一齐跨入下一步操作。在实际电路上，这种同步是通过**寄存器（Registers）**来实现的。其基本模型是：一组存储位（0 或 1）的输入寄存器，连接着一段负责计算的“逻辑云”（例如脉动阵列单元或乘法器），经过计算后输出结果，最终写入下一级输出寄存器中。全局时钟信号统一驱动所有寄存器；当时钟信号跳变的瞬间，导线上的电平状态被精确锁存进寄存器中。
芯片架构师普遍追求极致的时钟频率（Clock Speed）。如果芯片能以 2 GHz 运行，每秒能执行的操作数就是 1 GHz 的两倍。然而，这也对逻辑路径的延迟提出了苛刻的要求：任何在逻辑云内部发生的计算，都必须在下一个时钟上升沿到来之前彻底完成。因此，缩短从输入到输出的最长延迟，构成了芯片物理设计和综合优化的核心任务。
Dwarkesh 提出疑问：既然加太多逻辑可能会错过时钟沿，加太少又会浪费周期算力，芯片设计中是否存在“赌概率”的情况——即赌某个计算刚好能按时完成？Reiner 解释道，在标准芯片设计中，工程师会预留巨大的设计裕量（Margin）。虽然物理上确实存在概率分布，但设计标准通常定在很多个标准差之外（many standard deviations out），以确保芯片在绝大多数极端工况下绝对可靠。唯有在**跨时钟域（Clock Domain Crossing, CDC）**这种由一个时钟进入另一个时钟的特殊场景下，设计者才必须真正对亚稳态概率进行精确建模与计算。在常规数据通路中，设计通常保证信号提前约 25% 的周期时长抵达，彻底杜绝违例。
```mermaid
flowchart LR
accTitle: Pipeline Register Insertion
accDescr: Demonstrates splitting a slow combinational logic block into two pipeline stages separated by a register to double the clock frequency.
subgraph S1["优化前：长组合逻辑路径"]
R0["输入寄存器"] --> CL["完整组合逻辑云 (延迟 = T)"]
CL --> R1["输出寄存器"]
end
subgraph S2["优化后：流水线打拍"]
R2["输入寄存器"] --> CL1["逻辑前段 (延迟 = T/2)"]
CL1 --> PR["流水线寄存器"]
PR --> CL2["逻辑后段 (延迟 = T/2)"]
CL2 --> R3["输出寄存器"]
end
```
- **时钟频率与性能**：更高的主频（例如 2 GHz vs 1 GHz）直接带来理论算力的翻倍，但受制于组合逻辑的最长延迟路径。
- **时钟裕量设置**：关键路径通常按提前约 25% 的周期裕量进行约束，保证因制造偏差引起的延迟不致破坏时钟同步。
- **跨时钟域例外**：当芯片内部不同频率或相位的时钟模块交互时，必须专门处理信号同步与概率问题。
---
### `43:45` 为什么芯片需要全局同步？与流水线切分的权衡
Dwarkesh 借用沙盒建造游戏《异星工厂》（Factorio）的机制提出设问：现实中为什么不能让各个部件“做完了就往下送”？传送带上有铁板就直接拿，为什么非要让整颗芯片步调一致？Reiner 解释道，制造公差（Manufacturing Variance）会导致不同物理路径的延迟不可预测。假设输入信号需要分别经过函数 $F$ 与 $G$ 的计算，最后汇总到计算节点 $H$。某些芯片批次可能 $F$ 稍快，另一些则是 $G$ 稍快；如果缺乏全局时钟节拍，跑得快的信号就会撞上上一轮甚至下一轮尚未匹配的数据，节点 $H$ 也根本无法判定何时开始计算。
这也解释了为什么采用台积电（TSMC）相同先进工艺节点（例如 3 纳米制程）制造的两颗不同芯片，运行频率却能大相径庭。芯片的工作主频并非完全由制造代工厂决定，很大程度上取决于架构师能否成功消除芯片内阻碍频率提升的最长关键路径。
在实现手段上，最直接的做法是**插入流水线寄存器（Pipeline Register Insertion）**。将一个较长的逻辑云对半切开，中间插入一组寄存器打拍。理想情况下，这能将单级延迟减半，从而使芯片频率翻倍。但这并非免费午餐，必须付出面积与功耗的代价：
- 流水线寄存器本身需要占据额外的硅片物理面积；
- 插入寄存器增加了数据延迟的周期数（Latency），虽然吞吐量（Throughput）提升了，但单个结果产出的绝对时间周期增加。
---
### `46:12` 反馈环路瓶颈与 TSMC 原语的面积开销
虽然纯前向的数据流可以通过简单插入流水线寄存器来无限切分提频，但芯片架构中存在一种无法简单拆解的棘手结构——**反馈环路（Feedback Loop）**。例如累加器逻辑（Accumulator），每个周期都要把外部输入的值加到上一周期的累加和上（$F(x) = x_{prev} + x_{in}$）。如果加法器内部延迟过长，你无法直接在其内部切一刀塞入寄存器。一旦在环路内部强行打一拍，算法逻辑就会彻底变质：它不再是所有数的连续累加，而是变成了奇数周期数字与偶数周期数字各自独立累加的双路交替求和。这种存在于芯片控制与累加逻辑中的闭环数据依赖，正是制约主频上限最难攻克的瓶颈。
关于芯片设计的底层约束，Dwarkesh 追问台积电提供的制程设计套件（PDK）是否设定了逻辑原语（Primitives）的上限。Reiner 指出，代工厂提供的基础原语尺度极小，通常是与门（AND Gate）或全加器（Full Adder）级别，其开关速度极其迅猛，大约在 10 皮秒（picoseconds）左右。根据选用的电压和标准单元库（Standard Cell Library），一个典型的时钟周期往往能串联容纳 10 到 30 个原语门的延迟。
在理论极限下，逻辑设计者甚至可以极端到每隔一个与门就插入一个寄存器，从而堆出超过 4 GHz 到 6 GHz 的恐怖主频。然而在芯片物理实现中，这会带来灾难性的面积灾难：一个标准与门占用大约 1 个门等效面积（1 Gate Equivalent），而一个触发器/寄存器却要占用约 8 个门等效面积。这意味着芯片绝大部分的硅片面积与能耗都被白白耗费在维持同步和寄存器开销上，而非核心计算本身，完全违背了能效与成本的设计初衷。
| 单元类型 | 典型逻辑延迟 | 相对硅片面积 (门等效 / GE) | 架构设计考量 |
|:--- |:--- |:--- |:--- |
| **基础逻辑原语 (如与门、加法门)** | 约 10 皮秒 | 1 GE | 核心有效计算，一个时钟周期通常可容纳 10–30 级级联 |
| **流水线寄存器 (Flip-Flop)** | 取决于建立/保持时间 | 约 8 GE | 同步与存储开销；过度切分会导致面积激增，得不偿失 |
### `50:00` 时钟频率与吞吐量的权衡
在芯片设计中，一味追求极高的时钟频率（Clock Speed）并不意味着能完成更多的工作。芯片的总吞吐量（Throughput）本质上是两个维度的乘积：每个时钟周期完成的工作量（取决于面积利用效率与并行度）乘以每秒拥有的时钟周期数。如果为了将主频推到极致，把流水线切得极细，每个时钟周期内能容纳的逻辑门数量就会受限，从而导致芯片面积效率变低、总体吞吐量下降。换言之，系统可能会获得极低的延迟，但代价是带宽和吞吐量被削弱。
这与大语言模型推理中的 Batch Size（批大小）权衡如出一辙：当 Batch Size 设得很小时，单一用户可以在极低延迟下收到下一个 Token；但从宏观来看，芯片每小时能够处理的总体 Token 数量反而远低于其原本可达到的峰值。
- **吞吐量计算**：芯片吞吐量等于“每时钟周期工作量”乘以“每秒时钟周期数”。
- **并行性代价**：将主频推向极限往往意味着单个周期内能做的事情变少，从而削弱了整体的并行处理能力。
---
### `51:51` FPGA 与 ASIC 的商业取舍与高频交易场景
在高频交易（HFT）等极端场景中，吞吐量往往不如确定性延迟（Deterministic Latency）重要，能够以完全确定的时钟周期精确控制执行是首要需求。从概念模型上看，现场可编程门阵列（FPGA）和专用集成电路（ASIC）并无本质区别：二者都由 AND、OR、XOR 等最基础的原语逻辑门组成，且都运行在由固定时钟周期和金属连线构成的同步系统中。任何能在 FPGA 上实现的设计，都可以映射到 ASIC 上，并且 ASIC 会展现出显著的物理优势——其成本通常能降低一个数量级，能效比也高出一个数量级。
两者的核心取舍在于固定流片成本（NRE）与灵活性。制造第一颗 FPGA 芯片只需要约 1 万美元（直接采购现成芯片）；而第一颗 ASIC 芯片因为需要完整的流片流程（Tape-out），研发与开模成本可能高达 3000 万美元。如果一家高频交易机构需要确定性低延迟和高并行性，但其交易策略每隔一个月甚至几周就要迭代重构，那么反复流片 ASIC 在商业上完全行不通，能够在“现场”（部署在数据中心后）反复重编程的 FPGA 便成为唯一可行的方案。
```mermaid
flowchart TD
accTitle: FPGA 与 ASIC 权衡示意图
accDescr: 展示 ASIC 与 FPGA 在初始成本、单位效率和修改周期之间的对比路径。
subgraph Choice["硬件形态选择"]
A["设计需求：极低且确定的延迟"] --> B{"逻辑是否频繁修改？"}
B -- "是：月度/周度迭代" --> C["FPGA方案"]
B -- "否：固定算法长期量产" --> D["ASIC方案"]
end
subgraph Tradeoff["经济与物理特征"]
C --> C1["首颗成本：约 1 万美元<br/>硬件开销与能耗：高出一个数量级"]
D --> D1["首颗成本：约 3000 万美元流片费<br/>性能与能效：最优，量产单价低"]
end
```
- **概念同一性**：FPGA 与 ASIC 都遵循“固定时钟周期驱动的基础门逻辑与连线网络”这一设计范式。
- **经济天平**：FPGA 初始开销极低（~$10,000），适合高频迭代；ASIC 初始流片门槛极高（~$30,000,000），但量产成本和能效领先约 10 倍。
- **术语澄清**：“Field Programmable（现场可编程）”中的“现场”，指芯片已经出厂并部署在机房、数据中心等实际使用环境中，工程师无需重新生产硅片即可远程烧录新逻辑，并非物理学意义上的电场或磁场。
---
### `54:15` FPGA 的内部构造：查找表（LUT）、寄存器与 MUX 互连网络
FPGA 本质上是在一块固定的物理硬件上“模拟”ASIC 的可编程行为。其硅片内部主要由三大部分构成的阵列（Swarm）组成：存储状态的寄存器（Registers）、充当可编程逻辑门的基本单元——查找表（LUT, Lookup Table），以及遍布芯片内部的庞大可编程互连网络。
在这个结构中，白色的物理导线是制造芯片时就已经固化在硅片上的硬件通道，而在每个 LUT 和寄存器的输入端，都配置有多路复用器（MUX）。编程人员通过写代码综合出的配置比特流（在黑板上以橙色示意），实际上是在向每一个 MUX 旁边的微型配置存储单元写入控制位。这相当于向每个 MUX 下达指令：“从左侧邻近线路取信号，还是从上方寄存器取信号”，以此在密密麻麻的固定物理连线上“虚拟勾勒”出用户所需的特定电路走线。
```mermaid
flowchart LR
accTitle: FPGA 基础单元与互连示意图
accDescr: 展示输入信号通过配置 MUX 进入 4 输入 LUT 计算后存入寄存器的信号流动过程。
subgraph Interconnect["互连选择"]
IN1["布线网络输入 A"] --> MUX["输入配置 MUX"]
IN2["布线网络输入 B"] --> MUX
CFG_MUX["MUX配置位<br/>(选择通道)"] -.-> MUX
end
subgraph LogicCell["基本可编程单元"]
MUX --> LUT["4-LUT (查找表)<br/>16位真值表"]
CFG_LUT["LUT配置位<br/>(16位真值表数据)"] -.-> LUT
LUT --> REG["流水线寄存器"]
end
REG --> OUT["输出至路由网络/下一个MUX"]
```
- **配置的本质**：编程 FPGA 并不改变物理硅原子结构，而是写入控制位来锁定成千上万个 MUX 的选通状态，从而连通特定的逻辑路径。
- **核心组件**：由寄存器负责时钟同步与状态存储，查找表（LUT）负责组合逻辑计算，MUX 负责任意两点间的路由连接。
---
### `56:45` 查找表（LUT）的本质与面积开销估算
在真实的数字电路中，LUT 承担了任意组合逻辑门的角色。以业界最为经典的 4 输入查找表（4-LUT）为例，它接收 4 位输入并输出 1 位信号。在布尔代数中，从 4 位二进制输入到 1 位输出的所有可能函数映射总共有 $2^{16} = 65,536$ 种，但每一个特定的具体函数都可以被压缩成一张拥有 16 个条目的真值表（Truth Table）。因此，在 4-LUT 内部，只需配置 16 个 SRAM 存储比特（例如 `0111001...`）。当 4 位输入信号送达时，电路直接将其视作一个 4 位的二进制行地址，在真值表中索引对应的行，并输出该行存储的单比特电平。
通过这种真值表机制，同一个物理 4-LUT 可以被随心所欲地配置为 2 输入的 AND 门、OR 门，或者 3 输入、4 输入的异或门（XOR）。选择 4 输入作为常见标准，是芯片在“计算表达力”与“连线开销”之间取得的工程平衡点：输入位太少会导致需要消耗大量 LUT 进行级联，输入位过多则会导致查找表开销呈指数级膨胀。
从晶体管门级开销的角度，可以精确解释“为什么 FPGA 比 ASIC 昂贵约一个数量级”：
- 一个 4-LUT 在本质上就是一个 16 选 1 的多路复用器（选项数 $n=16$，数据位宽 $p=1$）。
- 构建这样一个 $n \times p$ 的多路复用选择电路，大约需要消耗 $n \times p = 16$ 个 AND 门，外加 16 个 OR 门（以及前级的取反逻辑与控制连线）。
- 在 ASIC 中，如果只需要一个最简单的 2 输入 AND 门，物理上仅需消耗几个晶体管（通常为一个与非门加一个反相器）；但在 FPGA 中，为了实现这同一个 AND 门，却必须动用一个包含数十个等效门以及额外 SRAM 配置单元的完整 4-LUT。正是这种“用通用可配置结构去模拟单一固定门”的冗余机制，直接导致 FPGA 在面积利用率和能耗上比专用 ASIC 劣势约 10 倍。
### `01:00:00` FPGA 内部查找表与多路复用器的层层嵌套
在深入剖析 FPGA（现场可编程门阵列）的底层构造时，Reiner Pope 澄清了一个核心事实：FPGA 本质上就是“多路复用器套着多路复用器”（muxes all the way down）。查找表（LUT, Lookup Table）本身其实就是一个巨大的 MUX，它的作用是从 16 行真值表条目中选出 1 位输出。但在真正进入这个查找表之前，还有一个更前置的 MUX 网络，负责从庞杂的逻辑门和布线网络（"soup of gates"）中把输入信号“捞出来”。
为了让概念具体化，Reiner 举了一个单比特寄存器的例子：假设从周围临近的寄存器和 LUT 中拉出 8 个单比特输入（总计 8 位）。系统需要在这 8 位中挑出 4 位作为目标 LUT 的输入。这意味着前面实际上并列放置了 4 个小 MUX，每个 MUX 负责“8 选 1”。挑出来的 4 位信号组合在一起，作为内部那个大 MUX 的控制地址，再去 16 项的真值表里检索最终输出。因此，像 AMD（Xilinx）这类 FPGA 厂商在物理硅片上必须对通信拓扑做出预设与权衡：在局部细粒度（local grain）层面上，开发者拥有在临近小邻域内自由选择连线的灵活性；但在更长距离、更粗粒度的全局互连上，厂商已经固化了特定的布线拓扑结构。
这种架构灵活性带来了惨痛的性能与面积代偿：
- **逻辑代价悬殊**：在 ASIC 中，一个四输入与门（4-way AND）只需要 3 个基础 AND 门直接级联即可实现；但在 FPGA 中，为了通过 LUT 表达同样的逻辑，需要堆叠约 32 个门电路。
- **开销根源**：真值表本质上是用暴力穷举列出所有输入组合，而 ASIC 能够用最精简的多晶硅（polysilicon）物理排布直接把布尔逻辑刻在硅片上。用 32 个门的通用开销去模拟 3 个门的专用逻辑，直接导致了约 10 倍的速度与面积劣势。
```mermaid
flowchart LR
accTitle: FPGA LUT Logic Routing Structure
accDescr: Diagram showing how 8 neighboring single-bit inputs pass through small MUXes into a 4-input LUT MUX.
subgraph InputRouting ["输入路由阶段"]
A["周边邻近信号 (8 选 1)"] --> B1["小 MUX 1"]
A --> B2["小 MUX 2"]
A --> B3["小 MUX 3"]
A --> B4["小 MUX 4"]
end
subgraph LUT ["查找表核心 (LUT)"]
B1 & B2 & B3 & B4 -->|"4 位控制信号"| C["内部大 MUX<br/>(16 选 1 查找表)"]
C --> D["单比特输出"]
end
```
---
### `01:03:25` 确定性时钟周期与硬件缓存的隐形成本
针对高频交易等领域偏爱 FPGA 而非 CPU 的现象，Dwarkesh 提出疑问：为什么高频交易系统看重 FPGA 的确定性时钟延迟（Deterministic Latency），而传统 CPU 却无法保证数据包进出的精确时间？Reiner 解释道，确定性延迟并非专用芯片的专属专利，像 Groq 等 AI 芯片就公开宣传确定性时钟，Google TPU 内部核心同样如此，甚至理论上也能设计出具备确定性延迟的 CPU。但现代通用 CPU 为了追求极端的高速度与吞吐量，做出了引入“非确定性”的特定架构妥协。
最根本的非确定性来源正是 CPU 的硬件缓存系统（Cache）：
- **离片内存的鸿沟**：CPU 芯片外部是 DDR 内存，二者速度相差整整两个数量级（约 100 倍）。如果没有缓存，所有程序的运行速度都会直接暴跌 100 倍。
- **环境驱动的不确定性**：硬件 Cache 自动拦截内存访问，先检查数据是否命中，未命中才向漫长的 DDR 发起请求。然而，Cache 能否命中高度受制于当前的外部运行环境——系统中其他进程抢占了什么数据、最近执行了哪些代码、甚至缓存替换机制内部的伪随机逻辑等。这使得同一段代码在 CPU 上的执行周期充满了随机抖动。
---
### `01:05:00` Cache 还是 Scratchpad：硬件主导与软件主导的分野
为了彻底解决缓存带来的不可预测性与控制开销，以 TPU 为代表的现代 AI 加速芯片采取了完全不同的设计哲学——暂存器（Scratchpad）架构。二者的根本分水岭在于：内存层级访问的裁决权交给了硬件还是软件。
| 对比维度 | CPU 硬件缓存 (Cache) | TPU 暂存器 (Scratchpad) |
|:--- |:--- |:--- |
| **外部主要存储** | 片外 DDR 内存 | 片外高带宽内存 (HBM) |
| **片上暂存单元** | 硬件透明 Cache | 片上专用 Scratchpad SRAM |
| **指令集设计** | 软件只发出统一的“读/写内存”指令 | 显式拆分指令：专门的片上读写指令 vs 专门的片外 HBM 搬运指令 |
| **控制仲裁方** | **硬件决策**：硬件自动探测缓存是否命中，若未命中则挂起并拉取外存 | **软件/编译器决策**：编译器在编译期精确安排数据在何时搬入片上 |
| **延迟特征** | 动态变化、受运行环境干扰的非确定性延迟 | 严格由静态指令周期规划的确定性延迟 |
谈及经典计算机体系结构，Dwarkesh 质疑道：人们总说现代计算机遵循冯·诺依曼架构（串行处理），但在 FPGA、TPU 甚至多核 CPU 普及的今天，这种描述是否已经过时？Reiner 认为，用冯·诺依曼架构来概括现代 CPU 依然合理。即便是当今顶配的服务器级 CPU，其总并行度通常也就维持在“100 个核心 × 16 宽度的向量单元（Vector Unit）”，即约 1,000 路并行度。这与现代加速芯片动辄数万级的超高度并行有着本质区别。
---
### `01:07:27` CPU 核心为何比 GPU 核心庞大复杂得多
如果一片 CPU 晶圆（Die）上只放了区区几十到一百个核心，在线程较少时，硅片表面到底在做什么？为什么 CPU 核心不能做得像 GPU 的 CUDA 核心或 FPGA 的 LUT 那样小巧密实？Reiner 指出，CPU 核心的物理面积极其巨大且构造极其复杂。
- **逻辑计算只占边缘**：在单个 CPU 核心的硅片面积分配中，真正负责加减乘除计算的算术逻辑单元（ALU）只占极其微小的一块角落。
- **大头在存储与推测**：CPU 核心内部被海量的大容量静态缓存（L1/L2 Cache）以及深不可测的寄存器堆（Register Files）所占据。
- **GPU 没有的庞然大物——分支预测器**：虽然 GPU 也配备了寄存器堆和本地存储，但 CPU 拥有一项 GPU 完全不具备的大规模硬件模块——分支预测器（Branch Predictor）及相伴的乱序执行（Out-of-Order Execution）控制逻辑。CPU 倾注了巨大的芯片面积代价去动态预测代码的走向、激进地提前计算分支路径，只为了能在单线程遭遇停顿时尽可能榨出指令级并行度，这正是 CPU 单核面积远超 GPU 核心的核心原因。
### `01:10:00` CPU 分支预测器的作用与开销
CPU 为了追求极高的单线程执行效率，堆叠了极其庞大且复杂的预测逻辑。Reiner Pope 解释道，现代 CPU 内部布满了分支预测器（Branch Predictors），它们每时每刻都在猜测下一条分支指令何时到来以及跳转目标在哪里。GPU 之所以能在硅片面积利用率上远超 CPU，核心原因之一就是剥离了绝大部分这类预测逻辑，并将寄存器文件做得更为紧凑高效。
```mermaid
flowchart TD
accTitle: CPU分支预测工作流
accDescr: 展示分支指令的长执行耗时与流水线投机执行的关系
subgraph Execution["分支指令执行延迟 (~5ns)"]
D["识别分支指令"] --> E["计算布尔条件"]
E --> F["更新程序计数器 PC"]
F --> G["从指令内存重新读取"]
end
subgraph Pipeline["高频流水线执行 (~1-2 GHz)"]
BP["分支预测器"] -.->|"提前预测目标"| P1["预先执行指令 1"]
P1 --> P2["预先执行指令 2"]
P2 --> P3["预先执行后续指令"]
end
```
Dwarkesh 询问分支预测器是否是为了同时并行执行两个分支路径。Reiner 解释并非如此：真正的瓶颈在于处理单条指令需要耗费相当漫长的物理时间，例如大约 5 纳秒。在 5 纳秒的时间跨度内，CPU 必须完成识别分支、计算布尔条件判断真假、更新程序计数器（PC）至新跳转目标，并重新从指令内存中检索新代码。如果仅以 5 纳秒为一个时钟周期，处理器的运行频率将只有区区 200 MHz。
为了让芯片能够在 1 GHz 到 2 GHz 甚至更高的时钟频率下飞速运转，流水线绝不能停下来等待分支计算完成，而是必须立即执行后续的指令。但如果在 5 个周期后发现分支判定实际为“跳转”，之前执行的所有指令便全功尽弃，CPU 不得不撤回并跳到正确的地址重新开始。分支预测器的核心职责，就是在芯片流水线触达分支指令前数个周期，提前准确猜出分支是否发生及跳转目标，从而保证高速流水线不被清空打断。
---
### `01:12:00` 人脑结构与芯片设计的对比及能耗本质
Dwarkesh 顺着话题将专用 AI 芯片与人脑进行了宏观对比：
1. **稀疏性形式**：加速器中通常只能借助“结构化稀疏”（Structured Sparsity）节省芯片门电路面积，而人脑采用的是彻底的“非结构化稀疏”（Unstructured Sparsity），任意神经元都能以任意拓扑连接。
2. **存算一体**：人脑的内存与计算高度融合，而此前讨论的脉动阵列（Systolic Array）在某种程度上也实现了计算与局部存储的原生协同。
3. **时钟频率与能效**：人脑的工作频率极慢，而芯片时钟频率极高。通常时钟越快，为了在极短时间内完成信号稳定并确定晶体管状态，所需电压就越高，能耗随之剧增。
Reiner 对时钟频率与能效做了深入的物理机制剖析。芯片之所以追求极高频率，纯粹是为了拉高吞吐量（Throughput）以应对大批量计算——比如 GPU 往往运行在 Batch Size 为 1000 的负载下，而人脑完全没有“Batch 1000”的概念，“世上只有一个我”。
如果把一块 GPU 的频率从 1 GHz 直接降到 1 MHz，使其运行方式接近生物大脑，芯片并不会凭空获得 1000 倍的能效提升。这是由硅芯片的动态功耗（Dynamic/Switching Power）物理本质决定的：
| 能量消耗机制 | 物理行为 | 对整体功耗的贡献 |
|:--- |:--- |:--- |
| **动态翻转功耗 (Switching Power)** | 芯片内部以寄生电容形式存储电荷。逻辑值变为 `1` 时给电容充电，变为 `0` 时将电荷泄放到地线（Ground）。充放电循环直接消耗电能。 | **绝大多数功耗来源** |
| **静态漏电功耗 (Leakage Power)** | 绝缘介质物理上并非完全理想的绝缘体，晶体管关断状态下依然存在微小漏电流。 | 相对次要（随工艺缩减而变化，但在活跃翻转面前占比较低） |
当芯片时钟变慢 1000 倍时，电路在两次状态翻转之间处于漫长的空闲状态。由于电路静止闲置时几乎不消耗动态电能，翻转次数减少了 1000 倍，芯片的总能耗确实降低了 1000 倍，但**每完成一次有效计算所消耗的能量（即能效 Energy Efficiency）并没有出现大幅改善**。
---
### `01:15:33` GPU 与 TPU 架构对比：微型 TPU 的平铺与数据搬运权衡
Reiner 从芯片顶层架构模块（Top-Level Block Structure）的角度，直观剖析了 GPU 与 TPU 在设计哲学上的巨大差异：
```mermaid
flowchart TD
accTitle: TPU与GPU顶层架构对比
accDescr: 展示TPU粗粒度矩阵/向量单元与GPU细粒度SM平铺阵列的对比
subgraph TPU["TPU 粗粒度架构 (Coarse-Grained)"]
direction TB
MXU1["大型矩阵单元 (MXU / 脉动阵列)"]
VPU["居中向量单元 (Vector Unit)"]
MXU2["大型矩阵单元 (MXU / 脉动阵列)"]
MXU1 <==>|"仅 2 组较窄的数据走线界面"| VPU
VPU <==>|"仅 2 组较窄的数据走线界面"| MXU2
end
subgraph GPU["GPU 细粒度平铺架构 (Fine-Grained)"]
direction TB
subgraph TopRow["上半区 SM 群"]
SM1["SM (Tensor Core + 向量)"]
SM2["SM (Tensor Core + 向量)"]
end
L2["全局共享 L2 缓存 (居中)"]
subgraph BottomRow["下半区 SM 群"]
SM3["SM (Tensor Core + 向量)"]
SM4["SM (Tensor Core + 向量)"]
end
TopRow <--> L2
L2 <--> BottomRow
end
```
- **GPU 架构**：由大量高度一致的流式多处理器（Streaming Multiprocessor，简称 SM）规则平铺构成，中间穿插共享的 L2 缓存。每个 SM 内部都紧凑地集成了小型的 Tensor Core（本质上是微型脉动阵列）和向量计算单元。从高层次宏观视角看，**“一块 GPU 本质上就是整颗芯片上密集平铺的一大群微缩版小 TPU”**。
- **TPU 架构**：采用了极为粗粒度的计算单元划分。芯片上仅布置极少数超大尺寸的矩阵单元（MXU，即庞大的脉动阵列），配合放置在中间的大型独立向量单元（Vector Unit）。
这种设计差异带来了显著的架构权衡（Trade-offs）：
- **阵列利用率与固定开销分摊**：TPU 采用超大脉动阵列，可以最大限度分摊寄存器文件（Register File）和控制逻辑的面积开销；而 GPU 的架构则强制将一切逻辑分割为小型 SM 单元，每个 SM 都必须携带独立的寄存器堆、Warp 调度器等硬件支撑。
- **数据搬运带宽与布线边界**：TPU 粗粒度设计的硬伤在于数据搬运路径受限。大向量单元与大矩阵单元之间，仅有两条集中的物理数据接口进行数据流动；相反，GPU 的向量单元与矩阵单元在每个 SM 内随处就近互联，形成了 16 组甚至数十组并行数据布线。因此 GPU 在向量与矩阵单元之间可达成的瞬时数据交换带宽远高于 TPU。
- **局部性与跨核心代价**：GPU 计算如果能完全收敛在单个 SM 内部，其数据搬运距离和能耗都微乎其微；然而一旦出现跨 SM 的复杂通信调度，软硬件代价便骤然上升。
---
### `01:20:00` MatX 的可拆分脉动阵列与总结
面对两者的优缺点，Dwarkesh 提出猜想：像 Reiner 创立的芯片初创公司 MatX 这样的团队，或许正在尝试将两者的优势结合——既保留 GPU 风格的更小尺寸脉动阵列并由 SRAM 紧密包围以降低搬运开销，同时大刀阔斧地剔除 SM 内部为了支持通用 CUDA 编程模型而占据大量晶体管面积的沉重包袱。
Reiner 证实了这一思路并透露，MatX 此前已公开探讨过一项核心设计概念——**“可拆分脉动阵列”（Splittable Systolic Array）**。这种阵列在逻辑与物理上具备高度弹性：“在大计算吞吐需求下，它表现为一个巨大的脉动阵列以高效分摊开销；而在需要细粒度灵活性时，它又可以拆分成为多个微型脉动阵列协同运作。”
对谈在双方对未来硬件架构革新的共同期待中圆满结束。
## Notable quotes
> `00:00` 希望你们确实设计出了一颗好芯片 — Dwarkesh Patel
> `22:16` 整个电路大约八分之七（7/8）的成本全砸在寄存器堆的读写上了，只有微不足道的一小部分面积真正分给了逻辑计算单元 — Reiner Pope
> `32:45` 在通信层面有两个维度，一个是通信时间（communication time），另一个是通信带宽（communication bandwidth）。你的意思是，既然我们很久才加载一次，那就最大限度地压缩带宽，因为带宽直接等同于芯片晶圆面积（die area）。 — Dwarkesh Patel
> `01:00:00` 多路复用器套着多路复用器（muxes all the way down） — Reiner Pope
> `01:12:00` 世上只有一个我 — Reiner Pope
> `01:15:33` 一块 GPU 本质上就是整颗芯片上密集平铺的一大群微缩版小 TPU — Reiner Pope
> `01:20:00` 在大计算吞吐需求下，它表现为一个巨大的脉动阵列以高效分摊开销；而在需要细粒度灵活性时，它又可以拆分成为多个微型脉动阵列协同运作。 — Reiner Pope
## Mentioned
Reiner Pope, Dwarkesh Patel, MatX, Google, TPU, Crusoe, Cursor, Jane Street, SemiAnalysis, ClusterMAX, NVIDIA, Blackwell, Volta, TSMC, AMD, Xilinx, Groq, Factorio
## Video description
<p>New blackboard lecture with Reiner Pope: how do chips actually work - starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.</p><p><a target="_blank" href="https://reiner.org/">Reiner</a> is CEO of <a target="_blank" href="https://matx.com/">MatX</a>, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on <a target="_blank" href="https://arxiv.org/abs/2211.05102">software</a> <a target="_blank" href="https://jax-ml.github.io/scaling-book/">efficiency</a>, compilers, and TPU architecture.</p><p>Watch this one on <a target="_blank" href="https://youtu.be/oIk3R-sMX5o">YouTube</a> so you can see the chalkboard. Read the <a target="_blank" href="https://www.dwarkesh.com/p/reiner-pope-2">transcript</a>.</p><p>Sponsors</p><p>* <a target="_blank" href="https://crusoe.ai/dwarkesh">Crusoe</a> was one of only five GPU clouds that made the gold tier in SemiAnalysis' most recent ClusterMAX report. Gold-tier providers like Crusoe delivered 5-15% lower TCO than silver-tier clouds, even with identical GPU pricing. This is because optimizations like early fault detection and rapid node replacement don't necessarily show up in the sticker price, but still matter a ton in the real world. Learn more at <a target="_blank" href="https://crusoe.ai/dwarkesh">crusoe.ai/dwarkesh</a></p><p>* <a target="_blank" href="https://cursor.com/dwarkesh">Cursor</a> is where I do most of my work—from reading research papers to visualizing technical concepts to coding up internal tools for the podcast. Most recently, I used it to build two different review interfaces for my essay contest, one that anonymizes submissions for scoring and another that lets me see applicants' essays next to their resumes and websites. Whatever you're working on, you should try doing it in Cursor. Get started at <a target="_blank" href="https://cursor.com/dwarkesh">cursor.com/dwarkesh</a></p><p>* <a target="_blank" href="https://janestreet.com/dwarkesh">Jane Street</a> let me ask Ron Minsky and Dan Pontecorvo, two senior Jane Streeters, a bunch of questions about how they use AI. We discussed everything from the types of models they're training to how they think about the future of trading to why they're more bullish than ever on hiring technical talent. You can watch the full conversation and learn more about their open positions at <a target="_blank" href="https://janestreet.com/dwarkesh">janestreet.com/dwarkesh</a></p><p>Timestamps</p><p>00:00:00 – Building a multiply-accumulate from logic gates</p><p>00:16:31 – Muxes and the cost of data movement</p><p>00:26:10 – How systolic arrays work</p><p>00:39:11 – Clock cycles and pipeline registers</p><p>00:51:51 – FPGAs vs ASICs</p><p>01:03:25 – Cache vs scratchpad</p><p>01:07:27 – Why CPU cores are much bigger than GPU cores</p><p>01:12:00 – Brains vs chips</p><p>01:15:33 – A GPU is just a bunch of tiny TPUs</p> <br/><br/>Get full access to Dwarkesh Podcast at <a href="https://www.dwarkesh.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4">www.dwarkesh.com/subscribe</a>
[Source](https://api.substack.com/feed/podcast/198847047/4554a355fabc7d08073d273fc9f192d4.mp3)
[10:00] Bit Width Precision vs. Gate Area Scaling
[16:31] Reiner Pope: In a standard CPU or legacy GPU core, the vast majority of energy and die area is consumed not by computing math, but by shuttling bits back and forth between registers and compute logic through dense MUX trees.
[30:00] Reiner Pope: Bandwidth equals die area
[45:00] Reiner Pope: Over-pipelining spends almost the entire chip silicon budget on synchronization rather than useful computation, driving up clock frequency while severely hurting overall throughput.
[51:51] Reiner Pope: FPGA logic density is roughly 10x less efficient than a direct ASIC implementation.
[63:25] Reiner Pope: CPU caches operate roughly 100x faster than off-chip DDR memory, meaning cache misses degrade performance by two orders of magnitude.
[67:27] Reiner Pope: Removing branch predictors and branch target buffers frees up massive die area on GPUs, allowing transistor budgets to be dedicated directly to compute ALUs.
[72:00] Reiner Pope: On a microchip, power is consumed almost entirely when digital bits toggle between 0 and 1.
[75:33] Reiner Pope: At a conceptual level, a GPU's SM contains its own small Tensor Cores and vector units, making an SM function essentially like a miniature TPU replicated hundreds of times across the silicon die.
Original transcript
[0:00] I'm back with Rainer Pope, who is the CEO of Maddox, which is a new AI chip company. Last time we were talking about what happens inside a data center. Now I understand what happens inside an AI chip. How does a chip actually work? Full disclosure, by the way, I am an initial investor in Maddox. So hopefully you have designed a good chip. Also, if you're listening to this on an audio platform, it's much preferable to watch this Blackboard lecture on a platform where you can see what's happening. So switch over to YouTube or Spotify. So I'll start with sort of the very smallest fundamental unit of chip design, and we'll sort of build up into what an overall, like, actual production chip, what are the components of that. At the very bottom level of a chip, the primitives that we work with are logic gates, which are very simple things like and or not. And then these are connected together by wires that have to be laid out physically as metal traces on a chip. The main function that AI chips want to compute is multiplication of matrices, and really inside that is the fundamental primitive is multiply, accumulate of pairs of numbers. So we going to sort of demonstrate what that calculation looks like by hand and then sort of infer what a circuit would look like for that It turn out to be sort of easiest if I do multiplication accumulator something like a 4 number with another 4 number And then we're going to, the actual clearest primitive is actually multiply accumulate. So there's a multiply these two terms, and then we're going to add in, so product of these two terms, and then we're going to add in an 8-bit number. And can I ask a clarifying question? Why is this the natural primitive for, you know, whatever computation happens inside a computer? Yeah, so there's a few reasons for this. It's a little bit more efficient, but the reason it's natural for AI chips is that if you look what's happening during a matrix multiply, what is matrix multiply in very short? It is there a for loop over I and over J and over K of output i k plus equals to input i j times other input j k And so multiply, accumulate happens at every single step of a matrix multiply. Makes sense. And then the other observation is that the precision will almost always be higher in the accumulation step than in the multiplication step. This is maybe specific to AI chips, but you're multiplying low precision numbers, but then when you accumulate, errors accumulate quickly, and so you need more precision here. So this is why we've chosen to do a 4-bit multiplication and an 8-bit addition. Let me make sure I understood that. There's two ways to understand that. One is that the value will be larger than the inputs, and the other is that if it was a floating point number, it would be... Maybe that part is less intuitive to me, But it's maybe the same principle? It is really the same principle. I guess the separate principle is that as you are summing up this number, you are summing up a whole bunch of numbers. And so you've got a lot of rounding errors accumulating. Whereas in this case there only one multiplication in that chain And so there not a lot of rounding errors accumulating in the multiplication Why are you summing up a whole bunch of numbers It just two numbers right Just I mean this summation happens it repeated jay many times Yeah any errors accumulate I see So how would we perform this calculation by hand? I mean, as a human, we would probably separate it into two, but we can sort of do it all in one using long multiplication. So the multiplication term first, we're going to multiply this number, this 4-bit number here, by every single bit position in the other 4-bit number. So we write that out. First, 1, 0, 0, 1 multiplied by this bit position. That is this number itself. Then shifted across by 1, we're multiplying by 0. That gives us an old zeros number. Shifted across even one more to multiply by this one, we get 1, 0, 0, 1. And then finally for this last bit position we get an old zeros number again. So this sort of gives us a bunch of terms that we're going to have to add for the for the multiplication. And then while we're doing that summation of this we might as well add in the actual
[5:00] accumulated term as well. So we just copy that directly across. So this is the sum. It's a five-way sum that we're going to want to compute. So firstly, what logic gates did it take us to even get to this intermediate step? We needed to produce all 16 of these partial products. How do I produce one of these partial products? So let's take this number one, for example, here. It is one. So how do we produce this number by multiplying this number by this one over here? We can actually produce that by an AND gate. This number is one if both this bit is one and this bit is one. If either of them is zero, then the multiplication of zero times anything is zero. So to produce all of this stuff, we ended up consuming 16 AND gates. Or in the general case if I were doing a p bit multiply times a q bit multiply then this will be like p times q many ands Exactly Finally, I sum them. Actually, most of the work is going to happen in the summing. And so let me describe sort of the other logic gate that we use here. And is almost the simplest logic gate that exists on a chip. It's almost the smallest. at the other extreme typically the very largest logic gate that you'll use is something called a full adder and what this does is sort of it does like coming from software you might think that a full adder it like adds 32-bit numbers together in this case it just adds three single bit numbers together and so you can think of it as like adding zero one and one together now when i add these together the result can be zero one two or three so i can express that in binary using just two bits. So as input it has three bits and as output it has it has two bits, which in this case are one, like the number two in binary is 1, 0. So this is also known as a 3 to 2 compressor because it takes three bits of input and produces two bits of output The two inputs are an X and a by value and then some carry that came in from like Sorry, the three inputs are all bits that are in the same sort of bit position, like three bits that are in a column here. Yeah. And then the two outputs, I have sort of drawn them vertically here and horizontally here to kind of match this vertical versus horizontal layout here, which is expressing that things that are in the same column are in the same like bit position whereas things that are in the JSON columns like this is a carry out whereas this was the sum. So if the inputs in the full ladder are let's say like 101 then the output would still be 10. If it was 111 it'd be 11. It was 000. It'd be 00. It was like 010. It'd still be 01. So yeah it's just counting essentially the number of things and expressing that in binary. So this circuit actually can sort of capture what we as humans naturally do when we're doing summing along a column. So I can show that sort of, I'll show sort of one iteration of using the full adder to sum. The way I sum here is going to be a little bit unnatural for humans. Humans we would sort of sum along the column and then remember the carry but instead of remembering the carry we actually just explicitly write it out So in this we proceed from the rightmost column towards the left on the rightmost column we sum the 1 and the 1, and that produces like a 0 here and a carry of 1. So we've sort of used this full adder circuit on this pair of bits and produced a pair of bits as output. Now we can do the same thing with this column. We've got a column of one, two, three, four numbers and so maybe we'll take the first three of them, run a full adder on them and that gives us a zero and a zero as output. So the sum of these is zero, zero. So that's the full adder applied to all of these bits. As I've used up bits I'm gonna just cross them out to indicate that I've handled them. Let's just keep going a little bit more. So we'll go here. I take these three numbers, I add them, that gives me a 1 and a 0. I've dealt with these three numbers. And now I take 1, 2, and I can even take these three numbers, for example, right now, and add them, and that gives me a 1 and a 0, and I've dealt with these numbers.
[10:00] The way I should view this is that I have this whole grid of numbers that need to be added. I'm going to just keep applying full adders to all the bits that are here, constantly removing three numbers from a column and then writing out two numbers as output. Keep going with this over and over and over again until I eventually get just one single number coming out here. Something like that. This is probably the wrong sum. So this approach that I've described here, this is called a data multiplier. And this is sort of like the standard for how you do area efficient multipliers using full adders. Let's try and quantify the circuit size of this just so we have got a sense of like how big things so we can compare to them later. How many full adders did I use? I started with how many numbers? I have the 16 partial products, which is the product of all of these terms, plus the 8 terms that I adding here So I started off with 24 bits and then I produced eight bits on the output eventually And in every step I was sort of crossing off three numbers and writing two numbers out as a result. And so every single use of a full adder eliminates one of the bits here. And so how many full adders, it must be the 24 minus the eight. So there were 16 full adders in this circuit. But in general, this is true in the general case as well, there will be P times Q many full adders in this circuit. Let me make sure I understand the logic of that. So the input bits 24 is P times Q plus P plus Q. That's right. And the output bits is just P plus Q. And so P times Q plus P plus Q minus P plus Q equals P times Q. That's right. So I think this explains sort of, or at least hints that, the second reason why we chose to do a multiply-accumulate. First reason being that's actually what shows up in matrix multiplication. But second reason being it gave us this very slick P times Q very simple algebra So we sort of described like this whole procedure every single atomic step that I took here becomes a logic gate and then sort of the wires connected together Like when I had these three inputs that I salvaged to produce these two outputs, like if I think of mapping this to a physical device, there would be a wire that runs sort of connecting all three of these things together into a logic gate that produced this output. Okay, so this is the main primitive at different bit widths that is inside an AI chip. We're going to build up from here to how would you use that to run all of the other operations you might want. This might be the wrong time to ask this question, but whenever NVIDIA reports that this chip can do X many FP4 or half as many FP8, it seems to imply that those circuits are fungible uh that there's not as dedicated like fp4 versus fp8 um but the way you you're mapping it out here it seems like you would need if it has to be mapped out in the logic you would need a dedicated um fp4 multiply accumulate and then a dedicated fp8 accumulate basically can you can you funge them as drawn they actually not particularly fungible This is actually one of the main choices you have to make when designing a chip which is how much of fp4 how much of fp8 do i have and then sometimes i'll make that consideration from the point of view of like uh what do i think that is the customer requirement another way to take an angle on that is to say um what is the what is the power budget for uh equalize the power budget between fp4 and fp8 but so then when they report those numbers and they just happen to be the case that it does 2x as many FP4 as FP8. They just happen to choose, like give equivalent die areas to all the floating points. And as a result, that ended up being... Like why is the ratio exactly 2x? Yeah, exactly. Yeah. So part of it is, I mean, surely that won't be exactly, like exactly equivalent to die area. There's a data movement reason actually, and we'll maybe come back to this when we sort of look through how it goes into and out of memories. There's something really nice just from a software level of the fact that I can pack two 4-bit numbers into the same storage as an 8-bit number. And so when I store that to a memory or something like that, the sizing of the buses that I wire within the chip actually makes that work out really nicely. Actually, come to think of it, it's not just 2x. It would be...
[15:00] the amount of area it takes, it sounds like, is quadratic. It's quadratic, in fact. With the bit length. So that's why smaller precision is even more favorable than your negative thing. This is a really big reason. So, in fact, NVIDIA made a change. Historically, up until B100 or B200, every time you have the bit precision, you double the flop count. That ratio is exactly, like for the reason you said, because of this quadratic scaling, that ratio is actually slightly wrong. It should be like an even bigger, you should get an even bigger speed up than you might otherwise think. NVIDIA's product specs have sort of started acknowledging that in B3 and 100 and beyond, where the FP4 is three times faster than the FP8. Though it should be 4x. Yeah. What I've shown here is like the simplest case of integer multiply. When you're dealing with floating point, as you do in FP4 and FP8, there's this sort of other term, which is the exponent that just complicates this calculation. So what can we see already from this? It's like, I think the big observation you've made is that there this quadratic scaling with bit width which is very effective and is the single reason why low arithmetic has worked so well for neural nets But the other thing we going to do now is we going to compare the area spent on the multiplication itself with all of the circuitry that is around it. So we'll walk back in time a little bit and see how did GPUs prior to tensor cores work, which is the same way as the way that CPUs worked in fact. So which is like where do we stick this multiply accumulate unit? So generically I'll describe like a CUDA core or a CPU. You'll have some register file which stores some number of entries. Maybe it's like eight entries of like in this case I guess four bit numbers but typically like 32-bit numbers or something, which are not numbers. So this is the, like, inside the CUDA core, I'll have some register file of some depth, and then I will have my multiply and accumulate circuit, multiply and accumulate circuit. And what it going to do it going to take three arbitrary registers from this register file perform the multiply accumulate and then write back to the register file So it going to maybe write to this one but it was able to read from this one this one, and another random one. So it'll take three inputs like this. So this is the core data path of many processes. Most processes look like this. you've got some set of registers, and then you've got some set of logic units or ALUs. We want to analyze the cost of the data movement from the register file to the ALU and back. So ultimately, there's going to be some circuit that says, well, I don't always have to select this guy. I might select any of the registers at any point in time. And so a first question is, how can I build a circuit? The circuit that I'm going to look for is a MUX. So in this case, it's going to have eight inputs, one from each entry of the register file, and it's going to have one output, which is actually producing this output. And then what is the cost of this thing It like all we have to build it out of is AND and OR and so how how do we build it we we do the dumbest thing possible we like form a mask saying we okay when we want to read like the third entry um we're going to end every single entry with either one or zero based on whether that's what i want to read and then we're going to or all of them together okay just to make sure i understand the basics what the mux is doing is it just like selecting just selecting just selecting an input. Yeah, so like invisible to software is like you say I want input number three, that means there's a MUX. Yeah. And so like what is the cost of this MUX? So an N input MUX operating on P bits. Well, I'm going to, so I have N rows, that's this eight rows, and I've got like each row is P bits wide, well I have to AND every single bit. So I get n times p many AND gates. Every single input I have to say, am I going to like mask it out or not? And then I'm going to OR them all together. And so there's going to be like n minus 1 times p many OR gates. Which is saying I've got all of these different things, almost all of them are zeros, but I need to sort of get
[20:00] laps them down into, like, from my eight options down into one option. And so every step I need to or, like, one row into an existing row. Got it, yeah. It's actually kind of funny that you would sort of, you don't think at the level of hardware, you sort of just think, like, oh, I'll just select element three, and something as simple as that is a sort of, like, in and of itself, a quite complicated circuit. Yeah, I mean, this is the first step of all of the hidden data movement costs that are sharp, yeah. there um and so like the thing like we're just going to like compare like i have to pay this cost and i've got one max here and then in fact i have two more copies of that for each of the three inputs to my multiply accumulate operation and so i have this cost which is like like three times n times p and gates over here um compared to this p times q um like uh sort of gates in the actual circuit that is doing the thing I care about. And if we plug in actual numbers, like this n being 8, I get 24 times p gates over just in the data movement compared to if q is 4 4 times p gates just in the adder multiply adder And sorry where is the three coming from Three different inputs here Got it Okay So the case, like really just what I'm hinting at here is that like all of this work, which scales like as the size of the register file, and this is a very small register file, all of this work just moving the data from the register file to the logic unit is many, many times more expensive than the logic unit. In the most recent ClusterMax report, SemiAnalysis ranked almost 100 different GPU clouds. Crusoe was one of only five that made the gold tier. SemiAnalysis found that gold tier providers like Crusoe had a total cost of ownership that was 5-15% lower than silver tier ones, even when they had identical GPU pricing. This makes sense because total cost of ownership is downstream of a bunch of different things that don't necessarily show up in the sticker price, but that Crusoe has optimized. things like how well you detect faults and how quickly you replace failed nodes. For example, Crusoe was one of the first clouds to adopt Envy Sentinel, NVIDIA's own GPU monitoring and self-healing software for enhanced GPU uptime, utilization, and reliability. This sets Crusoe make use of everything that NVIDIA has learned about why chips fail across all their different fleets and deployments so that Crusoe can catch faults earlier in the process And once they identify a failure Crusoe can swap in a healthy node in less than 10 minutes Because they not running bare metal Crusoe doesn have to spend time installing an operating system or configuring drivers. They can just spin up a new VM on an already running and pre-qualified host. If you want to learn more about this or the other reasons that Crusoe made goldier, go to crusoe.ai.dwarkesh. It may be helpful to just see what a mux looks like maybe like a two-bit or a four-bit mux yeah great so we'll take some inputs we'll have maybe um uh like we'll just do a a two-way mux so we've got um two different numbers um we've got these two inputs and then we have a um so these are the inputs that are being selected between and then we have a selector which says which can either be like i want this one or or it could be i want the other one so this is a one-hot encoding um so so this is what we all start with and then we the output we want to produce like let focus on this case so so this is the actual input we got We just want to produce this guy as the result And so, like, sort of very laboriously, what we do is we and this bit with all of these. And so that produces, like, anding this bit with this row. And likewise, we and this bit with this row. That produces all zeros. so this was the there's four ands here there's four ands here and then finally we just or these two together this gives a one we all these two together this gives a one we all these two together gives a zero we all these two together and it gives a one and so this is the four ors so like this actually ends up looking a little bit like addition in fact like we we did exactly the same set of ands here. We've added all of these things together, but then instead of collapsing it by using these full ladder circuits, we just get a very simple collapsing with or gates. But I guess that doesn't look like n times p. Yeah, so this was with n equals 2 inputs. Ah, great. In the general case,
[25:00] So we will have n rows, and then we'll have p bits per row. So that gives us the n times p many and gates. So this circuit I've described here, almost all of the cost, like 7 eighths of the cost, is in the reading and writing the register file. and only a tiny fraction of the cost is in the logic unit itself. So this is the problem to solve. This essentially was the state of play prior to the Volta generation of NVIDIA GPUs. This kind of thing is what was inside the CUDA cores. And this sort of problem statement is what motivated the introduction of tensor cores, which are more generically called systolic arrays. So if we think about how are we going to solve this problem, We're spending almost all of our circuit area on something that we just really don't care about and is hidden to the software programmer, and the thing that we actually care about is not much of the area. Well make this one bigger somehow while keeping this at the same size That the goal So the evolution was we had baked this much into hardware hardware in this stage This single line is a multiply accumulate and this single thing was baked into hardware The idea of a systolic array is to go two levels of loop up and bake this entire loop out here into hardware. And so the idea being that if we have a much bigger granularity fixed function piece of logic, maybe the taxes we pay on the input and output are much smaller. It sounds like you're suggesting that if you go up one step in the matrix multiply loop, that you can tilt the balance more towards compute than communication. That's right. There's two effects that we're going to take advantage of here. One is just that we can do more stuff per every trip through a register file. And then the other thing we're going to take advantage of is, in fact, in some of this loop we can take advantage of, for example, some certain things staying fixed. So let's sort of visually we're going to look at this matrix multiplication. So this portion of the loop corresponds to a matrix vector multiplication, in fact. So we take a matrix and multiply it by a vector So how do we do this We take every column gets multiplied by the vector and then summed So we going to sum sort of along columns. So this 0 and 3 gets multiplied by the 3 and 7 and gets summed and then the 1 and 2 gets multiplied by the 3 and 7 and gets summed. So there is a multiply accumulate associated with every single one of these entries in the matrix. So we'll just draw out these four multiply accumulates. And just to make sure I understand why there's four multiply accumulates. So if each entry in the column that corresponds to the output vector is a dot product, and in this case it'll be like two multiplications and then the addition of those two multiplications, so you're accumulating. Yeah, so the addition, so really there's only one addition per dot product, but we'd like to start with zero. So what we're going to aim for is to have so we want to have quadratically more compute We do We have We got sort of x times y as much compute as we had before But we're going to want to somehow aim for having only x times as much communication. And this is sort of the intention so that we get this advantage term going as y. So we've laid down the multiplications. We're going to want to bring in a vector of size 2. And so that already is in line with our comms target. That's fine. However, we need to somehow manage the communication of this matrix, which exceeds our budget of X. And so the idea is that in an AI context, this matrix is actually going to stay fixed for a long period of time. And so instead of bringing it in from the outside, so we've got some register files sitting over here, we don't want to have the amount of stuff coming out of this register file. This is the term that we want to go as X in some sense. We don't want to bring this full matrix in from the register file every cycle because we don't have enough.
[30:00] too much in terms of wiring from the register file. And so instead we're going to store... Our key trick is that this matrix can be stored locally to the systolic array. And so where we'll store these numbers 0, 1, 2, and 3 in just a gate called a register that that physically stores these numbers, and we're going to reuse these numbers over and over again for a large number of different vectors. And so the optimization here is that the nature of matrix multiplication is you can store this square quadratic thing directly where the logic is happening, and which is higher dimension than the, or has an extra dimension compared to the inputs which you keep swapping in and out. That's right. I mean, this is the nature of what a matrix multiplication is, is that you do a lot of multiplication to get one value out. Like a dot product is the result of a lot of multiplications. And so that optimization means that you're just like, you can stuff a lot of like multiplication in before you get some value out of it That right That right So just to complete the picture here of concretely how that looks I swapped the three and the two here Three and two So just like this zero and three is going to multiply by the three and seven, and so we're going to form a dot product along columns here. So somehow we're going to feed a three and a 7 in here. These participate in sort of this feeds into this multiplication and also feeds into this multiplication. Likewise the 3 feeds into here and also into here and then we're going to sum along here with like starting at the top of a column we feed in zeros and then coming out the bottom we get results coming out. So sort of just to visually see what we've got there's a dot product that is performed along columns in a matrix and that sort of maps exactly to what is done spatially in the systolic array here. So this is one dot product summed vertically and this is a second dot product also summed vertically And then what is the data that needs to go into and out of the register file We have X amount of data that coming out here on the output And we also have sort of this data coming from the input X amount of data from the input. And so with respect to the input and output vectors, at least, we've sort of met our goal of having only X as much data going in and out of the register file. This leaves open the question of, I said that the weight matrix is stored locally in the systolic array. How did it get there in the first place? At some point, you need to boot your chip and populate this data. And so where did that come from? The trick is we just do it very slowly. So we very slowly trickle feed it in into the systolic array. the simplest strategy is that we run this daisy chain that says, like, feed a number into here, and then on the next clock cycle, it'll move down to the next entry with the systolic array. And so we can do that in every column in parallel, and that gives us the sort of, this is also going to come from here, and this is going to be another factor of approximately x units of bandwidth coming in Would you mind repeating that sentence so like we sort of like we know that we going to be bringing in numbers only rarely into the matrix Yeah. And so we just want to come up with any construction at all, such that the amount of wiring that actually feeds into, sort of, crosses this boundary of the systolic array, like this boundary right here, we just want to keep that bounded to X and not go as XY. Y. And so a particularly simple strategy is that we sort of bring in a number into the top row of the systolic array. That's what we can do in one clock cycle. And then for Y consecutive clock cycles, we're going to be bringing in the top row every time and then sort of shift all of the other rows down by one. And that keeps the wiring that needs to come from this expensive register file only down to a factor of X rather than X. I see. Okay. So there's two questions in terms of communication. There's like communication time, and then there's communication bandwidth. Yes. And you're saying, since we're only going to be loading this in once, let's maximize, let's minimize bandwidth. That's right. Because bandwidth equals die area. And let's just like load it in slowly over like smaller lanes, because we're just going to keep this value in there for a while. Exactly.
[35:00] Interesting. So it's interesting to me that when we were talking last time about inference across many chips, the big high level thing we're trying to optimize for is increase the amount of compute per memory bandwidth, that is to say per communication. And here also we're trying to increase the amount of like actual multiplies or actual additions relative to transporting information from registers to the logic. So in both cases, you're trying to maximize compute relative to communication. Yeah, this shows up sort of all the way up and down the stack. This is sort of close to the bottom, sort of like to the gates. There's sort of a version that's maybe even closer to the gates of just like even the precision of number format that you choose to use. We saw that same effect. there's like a square cube law or like squared versus linear term going on both in just purely the precision of this ALU, but then also in terms of the size of this, the matrix. Yeah. Interesting. So, so this unit is sort of the next bigger unit. We had like the multiplication circuit. And then on top of that we have a like a pretty large systolic array I drew it as two by two but in for example older TPUs they were described as 128 by 128 of this circuit shown here And this circuit ends up being, this is the most efficient known mechanism for a circuit for implementing a matrix multiply. I see. So we've talked about sort of, it seems obvious that you should try to maximize compute relative to communication. um what are non-obvious trade-offs that um actually you are you know keep you up at night about what should we do x or should we do y and it's not obvious what the answer is yeah so i mean i think most of the decisions in chip design are um sizing decisions and so uh already in what we've drawn so far, like, so AI chips all have this, this circuit in it. They have a systolic array and then somewhere near it, a register file providing inputs and outputs. The two sort of, like, even within this scope, sizing questions that you have are, how big should I make my systolic array And how big should I make the the the register file The so and then the The trade for the size of the systolic array actually these two questions are coupled is one way to think of it is to say I going to have a budget for what percentage of my chip area I want to spend on data movement. So maybe I just say that I want this to be 10% and the systolic array to be 90%. And then I can size my register file. Bigger register files are more flexible. They allow me to run sort of more, I can get more application level performance out. But then they sort of take away from this area spent on the system. Like, great. Yeah, makes sense. I recently ran an essay contest where I asked people to write about what I consider to be some of the biggest open questions about AI. The submission window closed last week, so I used Cursor to create a couple of different interfaces to help me review the entries. One interface anonymizes submissions and hides unnecessary information. It lets me group responses by question add notes, and record my scores. The other interface helps me review entrants who also want to be considered for the researcher role that I'm hiring for. The UI puts the applicant's essay right next to their resume and their personal website so that I can see everything at once. Cursor's Harness is really good at helping these models see and improve their UIs I watched it render these interfaces in the built browser take screenshots click through sections and keep iterating At this point Cursor is where I do most of my work Whether I reading and visualizing a bunch of research papers or coding up an interface to review applications, or making flashcards for my Blackboard lectures, Cursor just makes it very easy for an AI to look at whatever I'm looking at and help me understand it and work with me on it. So whatever you're working on, you should do it in Cursor. Go to cursor.com slash thorkash. Where does the clock cycle of a chip come in? What determines what that is? Yeah. And what is a clock cycle of a chip? So I guess at baseline, it's sort of worth observing that chips are incredibly, incredibly parallel, right? You've got 100 billion transistors in a chip. A key thing that you need to do whenever you have massive parallelism is you need to synchronize between the different parallel units. In software, typically, you have these very expensive synchronization methods like a mutex. So one thread will finish what it's doing. It will grab a lock somewhere stored in memory and then notify the other thread that it's done. On chips, we take a very different approach and say that every nanosecond or so, all...
[40:00] all circuitry in the chip will kind of pause for a moment and then and then synchronize every so it actually synchronizes every single nanosecond or so and so that is the clock cycle um the entire chip typically all in sort of one fell swoop um goes in lockstep to the next operation that happens and so what this looks like in circuitry is that um you will have it's typically drawn so the clock is sort of mediated by registers which are these storage devices that we've drawn elsewhere. And the way to think of it is that I have some storage, which is storing a bit, which might be zero or one, and then I have some sort of cloud of logic, which maybe is this systolic array or this multiplier or something like that. And that's going to produce some output. So my inputs, I've got a bunch of inputs feeding to this cloud of logic, and then eventually later there's going to be some output register that this writes to. There is a global clock signal which drives all of these registers. And it says at a certain instance in time when the clock strikes whatever value happens to be on this wire at that instant that what going to get stored in there And so the sort of the challenge here is like I would like to have my clock speed run as fast as possible because if I can run at two gigahertz, I can get twice as many operations done per second than if I run at one gigahertz. But what that ends up meaning is that I'm very sensitive to the delay through this cloud of logic because any computation that is going to happen in here needs to sort of finish before the next proc cycle hits. So a major point of sort of optimization on any chip then is to sort of make this delay from here as short as possible. Interesting. And is there ever because the constraint here seems to be that if you add too much logic, then you might risk missing the clock cycle. But if you don't add enough, then you're leaving potential compute on the table. Is there ever a situation where you like you take a probabilistic chance that a computation finishes and or it just like no either it going to finish by clock cycle or not Yeah. In standard chip design, you margin it such that, I mean, there is a probability, but it's like many, many standard deviations, like way standard deviations out such that for all intents and purposes, it is a reliable part. It will always meet the clock. There are some weird exceptions to that. there are clock domain crossings where you go from one clock to the other clock and then you actually do have to reason about this probability but interesting in the main path you just like you margin as such that you'll get there like 25 of the clock cycle in advance so that it's very unlikely in this in this um the clock uh where the uh clocks synchronize i guess where the registers are. This is not something you determined as a chip designer. This is sort of just like an artifact of, hey, I want whatever sequence of logic. And then the software you use to convert your Verilog into the thing you send to TSMC, that just determines like, hey, in order to make this work you got to put a register here here and here to make sure that there no one step that is too long such as it makes the whole clock cycle of the entire chip longer than it has to be Yeah so this is actually a huge part of the work of designing a chip actually is inserting them So it is done in a combination of manually and automatically. So, I mean, just to show the very sort of dumb version of what you can do here, you can take this logic and split it in half. And so, like, say, actually, instead of just one cloud of logic, I'm going to have two smaller clouds of logic, which do the same thing, but split them up by a register. Right. Feeding in like this. And this is like, like, if you split it like in the middle, you can hit twice the clock frequency. That's great. You get twice the performance at the cost of this extra register. And so at the cost of some more storage. And stepping back, why do we need to synchronize the whole chip? Like if you imagine playing Factorio or something, there's no like global clock cycle. It's just, shit is done when it's done. There's iron on the plate. You can take it if you want. Yeah, so taking that analogy, the thing that you need to be mindful of is if I've got two different pads through some logic. So I have to do a computation like F here, and then computation G here, and then they're gonna come and meet
[45:00] computation h somewhere here um and so uh there's going to be manufacturing variants here uh in some chips f will take a little longer maybe in some chips g will take a little bit longer and so if i've got some signal that's propagating through here um and the result from f and g have to sort of meet up at h um what can the the thing that can go wrong is that f can get there early and it meets like the previous value of g or the next value of j and age needs to know when to start exactly like when when has this next iteration of and so this explains why different chips made at the same process note the same like tsmc uh technology um can have different clock cycles like two chips made at three nanometer right at different clock cycles based on uh whether they were able to optimize making sure that there's no one critical path that is so long that it slows down the whole ship's clock cycle. That's right. This optimization that I showed here, this is just the sort of pipeline register insertion, it's called. We inserted in the middle of the pipeline register here This is a sort of pure trade between clock speed and an area This is the easy case There is a harder case too which is sort of drawn it as a pipeline of logic here, but in other cases you may have some some calculation which actually feeds back in on itself. So it runs some function f and then writes back to itself like this. So for example, this might be this addition, like you've got some number that you're adding into every clock cycle. And so this could be like a plus, where adding in some number every clock cycle. So this little circuit, essentially it's just going to sum all of the numbers that are presented on different clock cycles. And the challenge is, if this plus takes too long, what can I do? If I try and put a pipeline register right in the middle of it, like here in the middle of it, this will end up changing the computation that's done. Instead of forming a running sum of everything that comes here, I will actually have two different running sums I end up having a running sum of the even numbers and running some of the odd numbers so sort of this constraint where I have a loop in my logic which all chips have somewhere this is actually the thing that is the hardest thing to address and sets the clock cycle. I don't understand why it would be a problem to have that, or I'm not sure even what it would mean to have a register there. Like, it's a sort of atomic operation, right? Yeah, well, so plus is not really atomic. Like, I think... As you just demonstrated. Yeah, it took a whole lot of work to do an estimation. And so you can take the early parts of that work and then stick a register in the middle and then take the late parts of that work. Okay, yeah. And I guess it's then up to, so TSMC offers a PDK, which says, hey, here's the primitives of logic that we can grant you in the chip. And it's up to them to determine that no primitive is bigger than the clock cycle they're hoping our process node targets. but other than that is there like what further optimization can't you just say like hey here's all the primitives from tsmc and if keep adding registers in between the primitives as much as needed until you get to your desired clock cycle yeah as a logic designer like the chip architect set the clock cycle so just for for one example the primitives you get from tsmc are on the order of like and gates or full adders um they depends a lot on voltage and frequent and and and which library you choose and so on but generally they you can typically have about like 10 or 20 or 30 of these in in a clock cycle sequentially so these primitives are very very fast like 10 picoseconds or something like that And so as a logic designer, I mean, in principle, if you literally just had register and then AND gate in a loop like that, you could get an insanely fast clock speed, like more than four or five, six gigahertz, something like that. But if you take this really sort of simple circuit and you look at the area you're spending here, this is maybe like one, I mean, this is called one gate equivalent. in size, so like unit of one in area, and this thing is like unit of eight in area or something like that. And so this is just, again, almost all of your cost has been this synchronization or communication cost compared to the actual logic. And so this would be a case where you've gone too far. You've made your clock speed really, really fast at the cost of spending almost all of your area on pipeline registers. Interesting. So,
[50:00] You're hinting at is a dynamic where you can have really fast clock speed, but you're not getting that much work done. Yeah, yeah. And so you can have, like, low latency, but low bandwidth. Or throughput, rather. Yeah, it hurts your throughput, in fact, because, like, the throughput of your chip you can think of as the product of how much I can get done per clock cycle, which is based on this area efficiency thing, times how many clocks I get per second. This is actually so similar to the thing we were discussing last time about batch size, where if you have a low batch size, then any one user can receive their next token really fast. But the total number of tokens that are processed in, say, an hour will be kind of lower than it could otherwise be. Yeah, exactly. You get less parallelism out if you drive your clock speed up really high. Language models are starting to compete against the best human forecasters. I sat down with two senior Jane Streeters, Ron Minsky and Dan Pontogorvo, and asked, at some point, does AI just do what Jane Street does? There's a world that we should take seriously where, you know, we're going to build large language models or some other AI systems that are like strictly smarter than all humans on the planet and more capable at all cognitive tasks Trading in particular feels to me as like kind of AGI complete sort of like NP complete because at the end of the day trading involves figuring out what things are worth, which means making predictions about the future. Jane Street isn't betting against AI. They just signed a $6 billion compute deal. But Ron's view is that the edge keeps moving. I have never been more desperate to hire more engineers and more traders than I am today. You know, you have the usual thing of like the other hard parts that we don't yet know how to automate. Well, that ends up being where the competitive edge lies. You can find these open positions and watch the full interview at Janestreet.com slash Thorcash. Okay, so I remember talking to an FPGA engineer at Chainstreet, Clark, who actually helped me prep for the previous interview we did together. And he was explaining why they use FPGAs. And I imagine that for high frequency trading, throughput is less important than latency. And so having very specific control over the clock cycle in a deterministic way is the most important thing. Maybe it be interesting to talk about how you why you can just achieve that with an ASIC or why FPGA is the yeah why you might use an FPGA to have deterministic clock cycles for like high trading Yeah so I mean firstly like let consider the sort of the business case for an FPGA versus an ASIC. FPGAs and ASICs use largely the same, sort of, conceptual model, which is that I have a series of gates built from ANDs, ORs, XORs, those, like, very small primitives connected together with a fixed clock cycle and connected together with wires that are running in a fixed clock cycle. So anything you can express in an FPGA, you can express in an ASIC 2, and it'll be about an order of magnitude cheaper and better energy efficiency on an ASIC than an FPGA. The trade-off is that the first FPGA costs you $10,000, whereas the first ASIC you make costs you $30 million because it requires an entire tape out. So sort of the business use case for an FPGA would be that I want something that has this very deterministic latency and fast runtime and high parallelism, but I'm going to change it very frequently. I change what I do every month or something like that And so then I don want to pay the tape back cost every time Now how does an FPGA actually implement It sort of like it emulates the ASIC programming model but in a fixed piece of hardware. And so how does that work, actually? So what it has at the base is it's got the two components we just talked about. It's got these registers as storage devices. And then it's got these, these are called LUTs, lookup tables, which actually provide all of the gates. And then we're going to see even the sort of the third component, we then have like sort of a swarm of these registers and LUTs. and all of these are available and then they are connected by sort of this big set of sort of MUXs so in front of every single one of these we've got something like one of these MUXs which selects one input from everywhere else sort of selecting from all of these different things you We've got a whole bunch of different options feeding into
[55:00] into all of these things. So what this allows is essentially when I program my FPGA, I can say that I'm going to take all of these components and I'm going to sort of superimpose on top of this a particular wiring which goes through this lot and then feeds into this lot and then goes to this register and then feeds into this lot or something like that. So what I've drawn in orange is how you So FPGA means field programmable gate array. The orange is what has been programmed in the field, whereas the white is all of the wires that must exist in the FPGA in order to actually make the device in the first place. What does it mean to be programmed into a field? Programmed in the field. So the device has been deployed in a data center, it's sitting in the field, and then you can come and program it. That field isn't like an electric field. No, that field isn't like out there in the world field. And so if I look at how the field programming comes out of the first lookup table and goes into the second one, how is it? Yeah, how? Like, where are the wires that made that happen? Yeah So I got a little bit like lazy in drawing all of these Every single device here has a MUX sitting in front of it which can select from all of the nearby circuits that are available Yeah. And so the actual configuration of the FPGA amounts to it. It is the MUX control. So in this MUX here, we have the data inputs, and then we have like the control that selects. And so like there's a little storage device sitting next to every single one of these MUXs saying, this is where you're going to source your input from. Got it. And so programming it consists of like configuring every single one of these MUXs. So that makes sense. What is happening inside of the lookup table? Yeah. So the purpose of the lookup table, so it's going to also have a little bit of control feeding, telling it what to do as well. The purpose of the lookup table is to function, to be able to configurably take the role of an AND gate or gate XOR, any of those different things. So there's many ways you could consider doing that. The way it is done in sort of traditional FPGAs is to say it will support So it will be a lookup table will be It will have four bits of input, one bit of output. How many different functions are there from four bits to one bit? There are 16 different functions. And so you can actually just tabulate this as 16 different numbers. you've got a table of 0111001, 16 entries. And so what it does is this table is stored in this blue configuration bit, and then it views these four bits as binary, looks up the relevant row of the table, and emits that bit. So this is a truth table view of lookup tables, essentially. Okay, so the lookup table, if you think about an AND gate, OR gate, NOR gate, XOR gate, these are all like take as input. Those are like two input functions. Sometimes we have like more complicated, like a three input function would be a three-way XOR. Right. Or a four-way XOR. And in this case how many it just depends how big it is Typical size for LUTs is four input which is sort of just a sweet spot between um there another computer communication trade like here like if it has too few inputs then you need to use more lots yeah if it if it has too many but basically the lookup table is like a truth table it's a truth table yeah and with a truth table you can program in any gate you want that's right yeah and so it's a lookup table just thinks like a programmable gate That's right. And so, I mean, one of the things you can do here is you can see why, where the rule of thumb that an FPGA is like an order of magnitude more expensive than an ASIC comes from, is to count how many gates would be inside this lookup table. Yeah. So we can view this lookup table essentially as one of these muxes. And so it is a mux with, has to select between 16 different values. And so it is a MUX with n equals 16 options, p equals 1 bits. And so what we saw way earlier is that this circuit costs like n times p many gates. And so it costs like n times p equals 16 and gates, and also 16 ores.
[60:00] this circuit being the mux uh yeah exactly the mux the mux is the core the mux that goes into the lookup table uh so the lookup table itself you can think of as being actually a big mux that like selects from all 16 rows down to one out yeah okay that is the lookup table but the way you've drawn it here there's like a mux and then a lookup table it's muxes all the way down so i mean there's a there is a second mux that is inside here this mux is is this mux got it okay And then the other mux is just saying where it came from in this sort of mess of gates. Right. And then the second mux is, okay, now you have one value, but that value is still a four-bit value. Yeah. So I've selected four bits from the soup. Right. And then I use those four bits to select which entry in the lookup table I'm going to use. Right. Okay. Yeah. So like suppose in the first mux, there's like eight nearby, you're pulling from eight nearby registers as input. And so that's like a total of like 32 bits going in. And then out of that, four bits come out. Those four bits go into the second mux which is inside the lookup table So actually I would say in this case these registers are single bit registers So if there are eight nearby registers and lookup tables then I have eight bits total coming in nearby I select from eight down to four different values. So there's actually like four different MUXs, one associated with each of these input, like little MUX associated with each of these input bits. Each of them is selecting one out of eight. And where are those eight coming from? nearby registers and other LUTs. And each register is one bit? Yes. Yeah. And so I guess AMD or whoever makes these FEJs still has to be opinionated about what registers should connect to which registers. And then you can program in the actual gates, but they add a wire in the communication topology, right? Yeah. So you get flexibility in a local grain thing. there's a sort of nearby neighborhood where you can select from um but then more grossly like more coarsely longer distant connections they they form an opinion right yeah and the reason it's 10x slower is why so if you look at the cost of like building this lookup table it like 32 gates yeah and then it can give me the equivalent of like what one an interesting thing i can do here I can do a four AND gate And so that like I using 32 gates of lookup table to sort of implement like a four-way AND means like, what is a four-way AND? I would do like AND, AND, and then AND of AND. So to implement, like this is a circuit that I could implement in an ASIC directly using these three AND gates, but using a LUT, I can also implement it, but it's going to take these 32 gates instead of three. The overhead is really coming from the fact that the lookup table, the mux in the lookup table is there's a more concise way to describe a truth table than listing out every single possible combination of inputs, which is just to like right out the gate yeah like to like place down the polysilicon and that's right yeah yeah interesting one important point he made to me is that the reason they prefer fpgs to cpus is because they get deterministic clock cycles they know when a packet will come in and go out um why is it not a guarantee in cpus so you can actually design a cpu that has deterministic latency as well And in like the processes that are inside a lot of AI chips actually also have deterministic latency too Grok has advertised this. TPUs have that in the core as well. The challenge is getting sort of deterministic latency and high speed at the same time. And so where does the non-determinism in latency come from? non-deterministic latency comes from specific design choices in a cpu it's actually possible to remove those design choices and make a cpu that has deterministic latency those are not very attractive in the market and so people don't make those cpus anymore but actually in some sense like deterministic latency is maybe a sort of a simpler the designing starting point. And then some chip designers have added things into it to be non-deterministic. To take a concrete example of that, probably the most important example is on a CPU, just like the CPU cache itself. So in a CPU, you have the CPU, this is the CPU die itself, and then there is
[65:00] memory off on the side. This is the DDR memory off on the side. And then you have a cache system here. Inside it is the cache that sort of remembers recent accesses to DDR and stores them. And so when I'm running through my CPU instructions, every time I have an instruction that accesses memory, it first checks in cache, was the data stored in cache? And then if not, it feshers out to DDR. This is a huge optimization. The cache is like two orders of magnitude faster than DDR. If you never use the cache, basically all programs would run 100 times slower. So the presence of a cache is absolutely necessary for a CPU to run at reasonable speed. But whether or not you get a cache hit is dependent on the sort of ambient environment of the CPU, like what other programs are running, what has run recently, what is the random number generator inside the cache system doing? And so that is a big source of non-determinism in the runtime of a CPU. Yeah. So this is sort of the memory system for a CPU. The big thing that you can do differently is instead of having the hardware say I going to read memory and then decide the hardware decides whether or not it comes from cache or not, you can actually bake this decision into software. So a different design philosophy is to, and you see this in maybe, for example, TPUs. The TPU instead has, I mean, I'll draw the same diagram, but I'll call it a scratch pad. And so the main difference is, so this would be like a TPU and then HBM in this case, rather than DDR, but it's still an off-chip memory. And instead of like the software saying first access, like memory, and then the hardware decides, you've got some instructions that go here. This is like one kind of instruction and then a totally different kind of instruction that goes to HBM. Yeah. And so this style is generically known as scratchpad instead of cache. The key distinction being that you have one kind of instruction that says read or write scratchpad and a totally different instruction that says read or write HBM. So scratchpad being the cache? Yeah this thing here is the scratchpad Just to be clear So stepping way back people say computers have the quote John von Neumann architecture where there this serial processing of information And maybe just because we've been talking about parallel accelerators, but I just don't like the FPGA is super parallel. The kinds of AI accelerators, TV users are super parallel. well, even CPUs are super parallel, if you think about all the cores they have. And so is it actually, like, in what sense is modern hardware actually the von Neumann architecture? Is that actually a fair way to describe modern hardware? I think it's a fair way to describe CPUs. Like, just the amount of parallelism, like, on a CPU, the amount of parallelism you get is about 100 cores times maybe, like, 16-way vector unit. So 106, about 1,000-way parallelism on a CPU. Yeah. one question is, there is a die that is being used for the CPU, and if there's fewer threads, just as a matter of transistor voltages are switching on and off, is it just that there's literally one control flow, like a small part of the die, where voltages are switching on and off? How do you actually occupy the die area of a CPU if there only If there so few cores what am I what do you am i spending a lot of time in there yeah the cores are just much bigger and more complicated um so uh i mean like so i guess we should compare like a cpu core which takes up one 16 one 100th of the die to like i mean to a lot like a lot is just only these 16 gates so like it's clear why there are so many more LUTs in an FPGA than cores in a CPU. But then maybe why are there more CUDA cores, for example, than CPU cores, I think would be like, what's the difference between a CPU and a GPU or something like that would be a big difference. Inside the CPU, you have, so one big use of, so the sort of the top unit uses of area in a side of CPU are the cache. Very little is actually the ALUs. Like mostly it's like these register files rather than the logic units. And then both of these things have equivalents in a GPU, and so that's not a big difference. But the thing that does not have an equivalent in a GPU is the sort of this branch predictor. And so there is a whole big area in the CPU which is sort of
[70:00] Just a whole bunch of predictors that are saying, when will my next branch be and where's the branch target for that? And so stripping a lot of that out, as well as sort of making these register files tighter, in a sense, is driving a lot of where the GPU gains over the CPU. And what is the purpose of the branch predictor? To execute both branches at once? Or what does it do? So the issue is that when I've got a series of instructions, like instructions, instructions, instructions, instructions, if I have a branch like here, if this instruction is branch, the actual processing step of processing an instruction takes a really long amount of time. It takes maybe five nanoseconds or something like that. So the time to actually notice that I've got a branch and then evaluate the Boolean, whether it's true, and then update the program counter to the new target and then read from the instruction memory for that. That could take actually five nanoseconds to finish. And so in reality, this may finish somewhere down here. I don want to I want to run a clock speed that is much faster than what five nanoseconds allows like like five nanoseconds is 200 megahertz clock speed i would like to run at one or two gigahertz or something like that and so um i need to run other instructions while the branch is is being evaluated and so i like i really just want to keep running the following instructions that happen after me. But that might have been wrong. Like if the branch ended up being taken, then I need to know that instead of evaluating these instructions, I actually need to like jump to wherever the target is and run these instructions instead. And so the purpose of the branch predictor is like genuinely to predict based on like before you even get to this instruction to be like five cycles earlier to predict there was going to be a branch that's going to happen. So if I think about how the brain works versus what you're describing here, At a high level, the differences might be that while you can do structured sparsity in these accelerators and then save yourself some area that you would have otherwise had to dedicate to these gates, in the brain, there's unstructured sparsity. Any neuron can connect to any other neuron and not in ways where it would be column or whatever Then there the fact that memory and computer are co i guess you could say in a way the memory computer co on this is exactly the co-location in some sense of the that's right that's right yeah yes maybe that actually isn't a big difference and the other maybe a big a big difference is that the clock cycle on the brain is much slower than on um on computers and partly that's to preserve energy because the faster the clock cycle, the bigger the voltage needs to be in order to identify for the signal to settle and to identify what state of transistor is that. Yeah, that's right. I don't know if you have other high-level takes about how any commentary on what the brain might be doing versus how these chips work. So let's take the clock speed one first, actually. the clock speed is quite high on a chip because that I mean drives higher throughput like when we compare a like a GPU running some some workload it's running batch size 1000 or something like that whereas like the brain is not running batch size 1000 there only one of me and so you could sort of imagine saying we take a GPU and like instead of running at a gigahertz run at a megahertz or something like that And that would start to look maybe a little bit more like sort of equivalent things that you talking about in the brain. There is, in the way that silicon works, there are, like, that does not give you a 1000x advantage in energy efficiency. So what it ends up looking like is you can like you sort of just end up running this circuit once to stabilization and then it'll sit idle for a long period of time. It doesn't consume a lot of energy while it's sitting idle because most of the energy is consumed in sort of toggling bits from zero to one and back. so actually let's talk about the energy consumption of a circuit like this. The way to think of a bit being stored is you've actually deposited some charge in a capacitor somewhere sitting somewhere in the chip implicitly so it becomes charged when it is it becomes a one and then it becomes discharged when it next goes to a zero and that cycle of like charging the capacitor and then dumping that charge out to ground that is where the energy is consumed. This is called the dynamic.
[75:00] or switching power. This is most of the energy consumption of a chip. There is some other energy consumption just coming from the fact that insulators aren't perfect insulators, but we'll just discard that. Most of the energy consumption actually comes from just the charging and discharging of like toggling from zero to one and back to zero. So if you run a chip much slower and you only clock it once every thousand clock cycles or something, you will have a thousand times fewer transitions. It'll be about a thousand times less energy consumption, but not a substantial advantage at energy efficiency. Okay, so you've described how a TPU works at a high level. What is the difference at a high level between how a GPU and a TPU work? Yeah, so I mean, I think there's sort of a high-level organization principle that is different. And then there's sort of inside the cores what are different. But we'll look sort of outside the, like, at the high level. So we'll take a GPU and a TPU, and what does, like, sort of the top-level block structure look like. If you think of this as the whole chip in each case, the organization of the GPU is mostly a bunch of almost identical units which are these these are the SMs And then they've got an L2 memory in the middle, and then a bunch more of these SMs on the bottom. And so there's sort of this fairly regular grid, of course. And then if we look at a TPU in comparison, you end up with much coarser grained units of logic. And so you end up with something like some large number of maybe just a few matrix units. These are the big systolic arrays. systolic arrays, and then in the middle you've got some vector unit, and then you've got your matrix units at the bottom. So now, sort of like matrix units with a vector unit in the middle, sort of this is the whole TPU chip. You can sort of think of scaling this thing down into a really tiny unit with a smaller matrix unit smaller vector unit and that is sort of what an SM is So sort of at a very high level point of view the GPU has a lot of tiny tiny TPUs sort of tiled across the whole chip Oh interesting So like you suggesting the tensor core within a streaming SM is analogous to an MXU? Yeah, it's very, very similar. Yeah. I see. And so if you had more like more lack of structure, having a bunch of tiny TPUs makes a lot of sense whereas if you kind of just have like huge matrix multiplications you're like why don't we just why don't we avoid the cost the cost of having the individual sms with their own registers and warp schedulers and things like that why don't we just like make a huge thing and like amortize those costs across the whole thing and i mean i think this shows up in how large you can grow things we've sort of seen this theme like especially with a systolic array where largest systolic array amortizes the register file costs better yeah um this sort of design allows you to have larger systolic arrays this whereas the sort of gpu design constrains you to having small units of everything um there is a trade-off however um the there ends up being um because of this sort of coarse separation of things there you need to move a lot of data from the vector unit to the matrix units And so you need to move a lot of data through two lines of parameter here Whereas if you look at the equivalent thing here, you've got vector units everywhere, and you need to move data through this line, through this line, through this line, through this line, through this line. So the amount of data you can move between a vector unit and a matrix unit is actually much higher in a GPU than in a TPU. Because instead of having to move all the data through these just two lines, you're moving all these data through 16 lines or something of wiring instead in a GPU. Right, but also you might have to move across less area. Which is also a saving. It's an energy saving. So data ends up moving, like, if you can operate entirely within an SM, the data movement is much smaller. But then the moment you want to operate across SMs, it becomes sort of more complicated and expensive. So you don't have to comment, but one might expect that a thing Maddox might try to do is to get the GPU-like smaller structure of systolic arrays surrounded by
[80:00] SRAM, but also at the same time make it so that like the things you need in an SM to support the CUDA architecture, but take a bunch of space, you might discard. Yeah. We've talked publicly about something which we call a splittable systolic array, which is sort of in some sense you can think of as like big systolic arrays that can be small systolic arrays as well. Cool. Okay, I think that's a good note to close on. Reiner, thank you so much. Thanks, Rakesh.
