MIT Links AI Scaling Exponents to Architecture Rather Than Data

Neural Scaling Universality: Emergent Power Laws Obey Robust Exponents With MIT — Ai4

Ai4

Neural scaling power laws and their recurring one-third exponent stem from low-temperature softmax dynamics and architectural constraints rather than training data statistics.

AI-assisted summary of Neural Scaling Universality: Emergent Power Laws Obey Robust Exponents With MIT — original source. Check the original for full context.

Source published: 2026-08-18

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 18, 2026, this presentation features MIT researcher Yizhou Liu exploring the theoretical origins of neural scaling laws in large language models. According to the page description, the talk presents newly identified architectural mechanisms—geometric constraints in model width, inductive biases across depth, and intrinsic softmax dynamics—that explain why loss decays as a power law with an exponent of one-third.

## TL;DR Yizhou Liu challenges the prevailing view that neural scaling laws are driven by dataset statistics, demonstrating instead that they arise fundamentally from neural network architecture. By modeling the loss through three distinct architectural bottlenecks—feature superposition across width, layer ensembling across depth, and low-temperature Boltzmann dynamics in the softmax layer—he provides a unified theoretical explanation for observed scaling behaviors. Using an analytically solvable teacher-student framework, Liu derives the exact one-third training time exponent and validates the theoretical predictions against empirical Pythia model checkpoints.

## Atmosphere & style The talk has the crisp, intellectually satisfying tone of a foundational statistical physics lecture brought to bear on modern deep learning. Liu speaks with calm, didactic clarity, framing the black-box empirical nature of scaling laws as an intriguing scientific puzzle before resolving it with Richard Feynman-inspired toy models and Taylor expansions. Watching it feels like a methodical journey from empirical ambiguity to mathematical elegance, demystifying large language model dynamics through the lens of physical principles.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) Neural scaling laws reliably predict model performance from small-scale runs, but their underlying theoretical origins and specific exponent values have remained unexplained. - [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) Empirical scaling exponents for data and model parameters consistently cluster around 0.3 to 1/3, indicating an unexpectedly slow rate of convergence. - [04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) Contrary to the prevailing view attributing scaling laws to Zipfian data statistics, theoretical analysis reveals they are driven by model architectures and geometric inductive biases. - [04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) Width scaling follows an inverse relationship (~1/width) caused by geometric interference when superposing vast numbers of features into limited hidden dimensions. - [04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) Depth scaling arises from sequential transformer layers performing ensemble averaging, where representations align while uncorrelated noise cancels out across layers. - [09:34](https://www.youtube.com/watch?v=rVAFn58qo-0&t=574s) The language model head governed by the softmax activation functions as the primary minimal ingredient dictating power-law training dynamics. - [12:51](https://www.youtube.com/watch?v=rVAFn58qo-0&t=771s) Mapping the softmax function to a physical Boltzmann distribution establishes logit scale as inverse temperature, differentiating exponential high-temperature convergence from power-law low-temperature regimes. - [15:28](https://www.youtube.com/watch?v=rVAFn58qo-0&t=928s) In the low-temperature limit, vanishing gradients cause inverse temperature to grow as t^(1/3), analytically deriving a loss scaling exponent of exactly -1/3. - [17:08](https://www.youtube.com/watch?v=rVAFn58qo-0&t=1028s) Modern LLMs inherently operate in deep low-temperature regimes because only a tiny fraction of the ~50,000 token vocabulary is plausible in any given context. - [17:08](https://www.youtube.com/watch?v=rVAFn58qo-0&t=1028s) Empirical loss curves across Pythia model checkpoints collapse onto a single unified trajectory matching the theoretical -1/3 power-law exponent. - [17:08](https://www.youtube.com/watch?v=rVAFn58qo-0&t=1028s) Synthesizing time, width, and depth mechanisms provides a complete physical foundation for empirical scaling laws without relying on data distribution assumptions.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) The Empirical Power of Neural Scaling Laws

Yizhou Liu introduces himself as a PhD student at MIT affiliated with the Center for the Physics of Living Systems, where researchers apply statistical physics and nonlinear dynamical systems to probe fundamental questions regarding the emergence of life and intelligence. Turning to artificial intelligence, Liu raises what seems like an obvious question with a deceptively simple answer: "Why are large language models large?" Quite simply, "larger is better." Quantitatively, the test cross-entropy loss—the primary metric of model performance where lower is better—scales as a clean power law with respect to the number of tokens used during training. In a log-log plot, this relationship appears as a straight line. Once training reaches saturation for a given model, the final converged loss likewise scales as a power law with the parameter count. Combining these relationships yields a scaling law with total compute.

Highlighting a figure from OpenAI's GPT-4 technical report, Liu points out that these empirical laws reliably fit small-scale runs and can extrapolate to forecast the performance of monumental models before they are ever trained. Yet beneath this remarkable predictive utility lies a scientific vacuum: the laws are purely empirical, meaning there is no foundational theory explaining why they exist. This ignorance breeds both excitement and anxiety—the hope of discovering ways to accelerate learning versus the concern that scaling might hit an unforeseen wall. Liu frames two core questions: why does loss decrease as a power law, and what determines the specific numerical values of the power-law exponents?

- The cross-entropy loss is empirically parameterized into three components: $L(D, N) = (D_c / D)^{\alpha_D} + (N_c / N)^{\alpha_N} + L_0$. - Empirical data scaling exponent $\alpha_D$ is measured across experiments between $0.28$ and $0.37$. - Empirical parameter scaling exponent $\alpha_N$ is measured between $0.34$ and $0.35$. - $L_0$ represents an irreducible entropy floor below which cross-entropy cannot drop. - All observed exponents cluster around $\sim 0.3$ to $1/3$, which Liu notes is surprisingly small and reflects a very slow rate of convergence.

[04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) Data Statistics vs. Architectural Inductive Biases

Prior literature predominantly attributed neural scaling laws to power laws present in training data. The most prominent example is Zipf's law for word and token frequencies, where word frequency plotted against frequency rank on a log-log scale forms a line with an exponent near 1. The conventional thesis argued that models tackle features according to their prevalence, learning the most frequent and critical features first before gradually acquiring rarer ones. While each individual task or feature might converge exponentially fast, the superposition of learning rates weighted by a power-law frequency distribution aggregates into an overall power-law decay of loss.

Liu challenges this data-centric view, proposing instead that scaling power laws emerge fundamentally from architectural nonlinearities and geometric constraints rather than covariate statistics. The framework decomposes model limitations into three distinct architectural bottlenecks: training duration, hidden state width, and network depth.

```mermaid flowchart TD accTitle: Three Architectural Mechanisms of Neural Scaling Laws accDescription: Flowchart linking architectural bottlenecks to their respective scaling exponents and overall model scaling.

subgraph Architecture["Architectural Bottlenecks"] Time["Training Time (τ) Softmax & Vanishing Gradient"] Width["Model Width (d) Feature Superposition & Interference"] Depth["Model Depth (L) Ensemble Averaging across Layers"] end

subgraph Exponents["Theoretical Exponents"] TimeExp["Loss ~ τ^(-1/3) Measured: ~0.30"] WidthExp["Loss ~ d^(-1) Measured: ~0.98"] DepthExp["Loss ~ L^(-1/2) to L^(-1) Measured: ~0.20"] end

subgraph Unified["Unified Scaling Law"] TokenScale["Token Scaling: Loss ~ D^(-1/3)"] ParamScale["Optimal Parameter Scaling: Loss ~ N^(-1/3)"] end

Time --> TimeExp --> TokenScale Width --> WidthExp --> ParamScale Depth --> DepthExp --> ParamScale ```

- **Training time scaling:** Insufficient training time ($\tau$) yields a loss power-law exponent of exactly $1/3$, aligning with empirical observations hovering around $0.3$. - **Width scaling and superposition:** Real-world data contains millions of distinct features that must be squeezed into hidden representations of limited width (e.g., $10{,}000$ dimensions). Because features vastly outnumber available dimensions, they inevitably interfere geometrically. This superposition constraint produces an inverse width scaling ($1/\text{width}$), matching experimental measurements of roughly $0.98$. - **Depth scaling and layer ensembling:** Predicting the next token requires transforming context across sequential layers. Liu finds that transformer layers act similarly to ensemble averaging: individual layers drive representations in roughly the same direction while carrying uncorrelated noise that cancels across depth. By the central limit theorem, this produces an inverse depth scaling, measured in real data around $0.2$. - **Reconciliation with known laws:** Because training tokens $D$ scale proportionally to time $\tau$, the temporal exponent translates directly into $D^{-1/3}$. Balancing width and depth to optimize loss yields an overall parameter scaling of $N^{-1/3}$, offering a unified theoretical derivation for why empirical scaling exponents across the literature sit at approximately $1/3$.

[09:34](https://www.youtube.com/watch?v=rVAFn58qo-0&t=574s) LLM Architecture and the Elephant in the Room

Turning focus specifically to time scaling, Liu invokes Richard Feynman alongside the ancient parable of the blind men and the elephant. An LLM is a system of immense complexity; trying to describe it all at once is like touching an elephant's ear and concluding it is a fan, or touching its leg and declaring it a tree. To make genuine scientific progress, one must follow Feynman's advice by adopting the right approximations—isolating a minimal toy model that captures the essential core, solving it analytically, and then completing the scientific loop by verifying those insights back on production-scale language models.

Liu briefly walks through standard autoregressive LLM mechanics using the phrase "I like": 1. An input token sequence enters an embedding layer, transforming tokens into dense vectors. 2. The vectors pass through repeating transformer blocks, each consisting of multi-head self-attention (which aggregates contextual information from preceding tokens) followed by a multi-layer perceptron (MLP). The total number of stacked blocks defines the depth. 3. At the output, the representation reaches the unembedding layer (the LM head), which projects the hidden vectors back across the entire vocabulary to generate logits and next-token prediction probabilities (e.g., assigning non-zero mass to likely completions like "dog" or "cat").

Liu identifies the final stage—the language model head governed by the softmax function—as the minimal necessary ingredient responsible for power-law training dynamics.

[12:51](https://www.youtube.com/watch?v=rVAFn58qo-0&t=771s) The Teacher-Student Setup and Low-Temperature Universality

To analyze the LM head in isolation, Liu constructs a simplified single-layer model. A hidden vector $x$ is mapped through a weight matrix $W$ to form a logit vector $y = Wx$, which is then normalized into a probability vector via the softmax activation. From a statistical physics standpoint, softmax is identical to the Boltzmann (Gibbs) distribution:

$$P_i = \frac{e^{y_i}}{\sum_j e^{y_j}}$$

The network is trained using cross-entropy loss (KL divergence). Liu sets up a teacher-student framework where a fixed teacher weight matrix $W^*$ generates target label distributions, and a student weight matrix $W$ learns the mapping via standard gradient descent, mirroring LLM pretraining.

The critical physical parameter is the scale of the logits, defined by the standard deviation of weights or logits, denoted as $\beta$ (or $\beta^*$ for the teacher). In physics, $\beta$ represents inverse temperature ($1/T$): - **High temperature (small $\beta$):** Logits remain small, yielding an exceptionally flat, uniform probability distribution over the vocabulary. Under gradient descent, the loss converges rapidly and exponentially. - **Low temperature (large $\beta$):** Logits have high variance, driving the softmax function into an extreme, highly concentrated distribution where mass is focused on a handful of entries. Here, exponential convergence breaks down, and the loss curves straighten out on a log-log plot into a power law.

[15:28](https://www.youtube.com/watch?v=rVAFn58qo-0&t=928s) The Mathematical Origin of the One-Third Exponent

Tracking the trajectory of the student model reveals an alignment phenomenon: very early in gradient descent, the student matrix aligns its directional orientation with the teacher matrix. Throughout the remainder of training, the internal direction remains fixed while the overall norm grows. Consequently, training simplifies into the dynamics of a single scalar: the student's inverse temperature $\beta(t)$ growing toward the teacher's $\beta^*$.

Liu outlines the mathematical steps that unlock the $1/3$ exponent: 1. Under gradient descent, the growth rate of the inverse temperature is governed by the gradient of the loss: $\frac{d\beta}{dt} = -\frac{\partial L}{\partial \beta}$. 2. In the low-temperature limit ($\beta \gg 1$), the loss behaves like free energy and can be expanded as a Taylor series in powers of temperature ($1/\beta$). Retaining only the leading-order non-vanishing term characterizes the asymptotic regime. 3. Evaluating the gradient dynamics with this expansion reveals that the gradient vanishes as a power of $\beta$, specifically driving $\beta^2 \frac{d\beta}{dt} \sim \text{constant}$, which integrates over time to yield: $$\beta(t) \sim t^{1/3}$$ 4. Substituting $\beta(t)$ back into the leading term of the Taylor-expanded loss directly produces: $$L(t) - L_0 \sim t^{-1/3}$$

This toy model analytically yields an exact exponent of $-1/3$, not from data distributions, but from the intrinsic vanishing gradient behavior of the Boltzmann-softmax function operating in low-temperature limits.

[17:08](https://www.youtube.com/watch?v=rVAFn58qo-0&t=1028s) Empirical Validation on Pythia and Architectural Synthesis

To close the loop back to production language models, Liu demonstrates that real LLMs operate under extreme low-temperature conditions. Modern language models typically use vocabularies containing roughly $50{,}000$ distinct tokens. However, given a specific linguistic context, the number of plausible next tokens is on average around five. Squeezing probability mass so that five options out of fifty thousand dominate requires an exceptionally sharp, peaked output distribution—the exact definition of deep low-temperature physics.

Liu tests this theoretical prediction against empirical checkpoint data from the open-source Pythia model suite across various parameter scales: - Subtracting the irreducible loss $L_0$ and plotting the excess loss against training time on log-log axes collapses curves from models of vastly different sizes onto a single unified power-law trajectory. - The empirically fitted decay exponents across all Pythia checkpoints cluster tightly around $0.3$, matching the theoretical $1/3 \approx 0.333$ prediction.

Liu concludes by summarizing the unified picture: power laws in neural scaling do not stem from data statistics, but from the architectural physics of deep networks. Training time power laws with a $1/3$ exponent stem from the low-temperature softmax nonlinearity; width scaling ($1/\text{width}$) originates from geometric interference among superposed features; and depth scaling ($1/\text{depth}$) arises from ensemble noise cancellation across sequential layers. Combined, these architectural mechanisms fully account for the empirical scaling exponents observed across modern large language models.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) Why are large language models large? — Yizhou Liu

> [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) larger is better. — Yizhou Liu

> These results emphasize that the origin of neural scaling laws is in model architectures, providing insights into designing future training strategies and architectures. — Yizhou Liu

> [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) L(D, N) = (D_c / D)^{\alpha_D} + (N_c / N)^{\alpha_N} + L_0 — Yizhou Liu

> [12:51](https://www.youtube.com/watch?v=rVAFn58qo-0&t=771s) P_i = \frac{e^{y_i}}{\sum_j e^{y_j}} — Yizhou Liu

> [15:28](https://www.youtube.com/watch?v=rVAFn58qo-0&t=928s) \beta(t) \sim t^{1/3} — Yizhou Liu

> [15:28](https://www.youtube.com/watch?v=rVAFn58qo-0&t=928s) L(t) - L_0 \sim t^{-1/3} — Yizhou Liu

> [15:28](https://www.youtube.com/watch?v=rVAFn58qo-0&t=928s) \frac{d\beta}{dt} = -\frac{\partial L}{\partial \beta} — Yizhou Liu

## Mentioned Yizhou Liu, MIT, Center for the Physics of Living Systems, OpenAI, GPT-4, Richard Feynman, Pythia, Zipf's law

## Video description Yizhou Liu - Researcher, MIT

The success of today's large language models (LLMs) depends on the observation that larger models and longer training lead to better performance. However, the origin of this observation, called the neural scaling laws, that loss decreases as a power law with model size and training time, remains unclear. This talk will present the newly identified mechanisms: geometric constraints of representation lead to a part of loss inversely proportional to model width, inductive bias of residual networks leads to most layers being similar, yielding another part inversely proportional to model depth, and the intrinsic power-law behaviors in softmax and cross-entropy functions lead to a power-law loss with training time whose exponent is one-third. These results emphasize that the origin of neural scaling laws is in model architectures, providing insights into designing future training strategies and architectures.

[Source](https://www.youtube.com/watch?v=rVAFn58qo-0)

中文简报

MIT将AI扩展定律指数归因于模型架构而非数据神经扩展幂律及其反复出现的1/3指数源自低温softmax动力学和架构约束，而非训练数据统计特征。 Yizhou Liu对神经扩展定律由数据集统计特征驱动的主流观点提出了质疑，论证了它们从根本上源于神经网络架构。通过将损失建模为三种不同的架构瓶颈——宽度维度的特征叠加、深度维度的层集成，以及softmax层中的低温Boltzmann动力学——他为观察到的扩展行为提供了统一的理论解释。Liu利用解析可解的teacher-student框架推导出了精确的1/3训练时间指数，并基于Pythia模型实证检查点验证了理论预测。神经扩展定律能够根据小规模运行可靠地预测模型性能，但其深层理论起源和具体指数值一直未得到解释。 (0:04) 针对数据和模型参数的实证扩展指数持续聚集在0.3至1/3左右，表明收敛速度出人意料地缓慢。 (0:04) 与将扩展定律归因于Zipfian数据统计特征的主流观点相反，理论分析表明它们是由模型架构和几何归纳偏置驱动的。 (4:20) 宽度扩展遵循反比关系（~1/width），这是由将大量特征叠加到有限隐藏维度时产生的几何干涉引起的。 (4:20) 深度扩展源于顺序Transformer层进行的集成平均，在此过程中表征得以对齐，而不相关的噪声在层间被相互抵消。 (4:20)

Reading summary

MIT researcher Yizhou Liu argues that neural scaling power laws and their recurring exponents near 1/3 do not stem from data statistics such as Zipf's law, but instead emerge fundamentally from architectural bottlenecks and statistical physics within deep networks. Empirical Scaling Laws and the Explanatory Vacuum (0:04) Cross-entropy loss in large language models follows clean power laws with respect to training tokens and parameter counts. As observed in reports like OpenAI's GPT-4 technical documentation, empirical fits from small-scale training runs accurately extrapolate to predict monumental models before deployment. Measured exponents cluster tightly: data scaling exponents typically range from 0.28 to 0.37, while parameter scaling exponents sit between 0.34 and 0.35 above an irreducible loss floor. Liu emphasizes that despite their practical utility for forecasting compute requirements, these laws have existed in a theoretical vacuum. Existing literature offers no fundamental explanation for why loss decay follows a power law rather than exponential convergence, nor why the exponents cluster universally around the surprisingly slow rate of approximately 1/3. Architectural Constraints Versus Data Statistics (4:20) Prevailing theory attributed neural scaling to the power-law distributions found in training data, such as Zipf's law for word frequencies. Under this data-centric view, models learn frequent features rapidly and rare features slowly, superimposing distinct exponential learning rates into an aggregate power-law decay. Liu challenges this consensus, proposing that scaling behavior originates from architectural inductive biases and nonlinearities rather than data statistics. His framework identifies three architectural bottlenecks: training time, model width, and model depth. In this formulation, finite training time yields an exponent of 1/3; width constraints force millions of features to superimpose into limited hidden dimensions, producing an inverse width scaling (~0.98 measured); and network depth acts like ensemble averaging, canceling uncorrelated noise across layers to produce inverse depth scaling (~0.20 measured). Balancing width and depth across compute yields a unified parameter scaling exponent of N^(-1/3). Low-Temperature Softmax Dynamics in Model Heads (12:51) To isolate the physical driver of temporal scaling, Liu focuses on the language model head where logits are normalized via the softmax function—identical in statistical physics to the Boltzmann-Gibbs distribution. Using a teacher-student setup trained by gradient descent, Liu examines the role of logit variance, which corresponds to inverse temperature. When logit variance is low (high temperature), probabilities remain uniform and cross-entropy loss decays exponentially. However, real autoregressive language models operate under extreme low-temperature conditions: given a context, models must squeeze probability mass onto roughly five plausible next tokens out of vocabularies spanning 50,000 options. Under this extreme low-temperature regime, exponential convergence breaks down, shifting the loss decay into a straight power-law trajectory. Mathematical Derivation and Validation of the Exponent (15:28) In the student-teacher model, directional alignment between student and teacher weights occurs early in training, reducing the learning dynamics to a single scalar: the growth of the student's inverse temperature toward the teacher's value. Expanding the loss as a low-temperature Taylor series reveals that gradient descent slows as inverse temperature increases, with gradients vanishing such that the inverse temperature grows proportionally to t^(1/3). Substituting this growth back into the leading-order loss expansion yields an exact temporal scaling decay of t^(-1/3). Liu validates this derivation empirically using checkpoint data from the open-source Pythia model suite across multiple parameter sizes. When the irreducible loss floor is subtracted, excess loss curves across all model scales collapse onto a unified line on a log-log plot, exhibiting empirical decay exponents that cluster tightly around 0.30 and match the theoretical 1/3 prediction. Liu concludes that neural scaling universality is governed by physical and geometric network constraints—softmax vanishing gradients, feature superposition interference, and layer ensemble averaging—demonstrating that the slow rate of model improvement is an intrinsic property of deep transformer architectures rather than an artifact of training data.

精炼阅读摘要

麻省理工学院（MIT）研究人员刘益洲（Yizhou Liu）指出，大语言模型中测试损失随规模呈幂律衰减的神经缩放定律，并非源自训练数据的统计分布，而是根本上由模型自身的网络架构非线性、几何约束与统计物理动力学所决定的。经验缩放定律与基础理论缺失 (0:04) 在深度学习中，测试交叉熵损失随训练词元（Token）数量和模型参数量的增长在对数-对数坐标图上呈现平直的幂律下降，并可推广至对总计算量的预测。OpenAI在GPT-4技术报告中证实，这一经验法则能通过小规模试验精准外推巨型模型的表现。然而，经验公式背后的基础理论长期处于空白状态，引发了对加速学习路径以及不可预见缩放瓶颈的广泛关切。经验拟合公式将损失分解为三项：L(D, N) = (D_c / D)^α_D + (N_c / N)^α_N + L_0，其中L_0为不可约熵下限。实验测得的数据缩放指数α_D介于0.28至0.37之间，参数缩放指数α_N介于0.34至0.35之间。刘益洲强调，所有观测指数均聚拢在约0.3至1/3附近，这一极小的数值意味着模型收敛速率极其缓慢，亟待科学解释其幂律成因及具体指数的来源。三大网络架构瓶颈与统一缩放 (4:20) 以往文献多将缩放定律归结为数据本身的统计特性，例如词频服从幂律分布的齐普夫定律（Zipf's law），认为模型依词频高低逐步学习特征。刘益洲对这种以数据为中心的假说提出质疑，主张幂律实质上源于架构的归纳偏置，并将其分解为训练时长、隐藏状态宽度与网络深度三大瓶颈。具体而言，训练时长不足在理论上对应精确为-1/3的损失衰减指数，契合实测的约0.30；在宽度方面，现实中的数百万特征需压缩至有限维度（如10,000维）的隐层中，几何重叠产生的特征叠加干扰导致与宽度成反比的缩放（实测指数约0.98）；在深度方面，Transformer各层类似于集成平均，表征逐层沿统一方向推进的同时，不相关噪声因中心极限定理逐层相消，理论导出与深度成反比的缩放（实测约0.20）。当平衡宽度与深度以优化损失，结合训练词元数正比于训练时长，系统整体在数据和参数上均导出1/3的衰减指数，统一了解释文献中测得的缩放规律。极小模型构建与低温玻尔兹曼物理 (9:34) 为解析时间缩放的微观机制，刘益洲借鉴费曼物理学方法，在复杂的自回归语言模型中抽离出驱动幂律动态的最小必要单元——受Softmax函数支配的语言模型头（LM Head）。统计物理视角下，Softmax归一化概率分布在数学形式上等价于玻尔兹曼分布。在该教师-学生单层设定中，学生矩阵通过交叉熵损失与梯度下降学习教师权重，其输出Logits的标准差对应物理学中的逆温度参数β（即1/T）。在高温极限（小β）下，词表概率分布趋于平坦均匀，损失呈指数级迅速收敛；而在低温极限（大β）下，Logits方差极大，Softmax分布被压缩得极为陡峭，概率高度集中于极少数词元，促使指数收敛崩溃并转变为对数-对数坐标下的平直幂律衰减。三分之一指数推导与实证模型验证 (15:28) 在优化动力学追踪中，学生矩阵在训练初期迅速与教师矩阵完成方向对齐，此后动力学完全简化为逆温度标量β(t)的范数增长，其增长率受损失梯度调控（dβ/dt = -∂L/∂β）。在低温极限下，模型损失可按温度展开为泰勒级数，梯度随β出现幂律消失，导出关系式β^2 (dβ/dt) ~ 常数，积分后得出β(t) ~ t^(1/3)。将该结果代回损失展开式的主导项，便在数学上解析推导出了损失随时间以t^(-1/3)衰减的动力学规律。刘益洲进一步论证，真实语言模型天然处于深度低温状态：模型约有50,000个词元，但在特定语境下合理的预测选项平均仅约5个，极度尖锐的预测分布符合低温物理条件。通过对开源Pythia模型系列不同参数规模的检查点数据进行实证检验，扣除不可约损失L_0后，各模型随训练时间衰减的过剩损失曲线高度重叠，实测幂律指数紧密聚集在0.30左右，充分验证了理论推导的普适性。刘益洲的研究确立了神经缩放定律的统计物理基础，指出1/3幂律指数并非来自语料库的数据统计，而是低温Softmax非线性引发梯度消失、特征叠加几何干扰以及跨层集成噪声相消的共同结果。这一理论阐明了现代模型缓慢收敛的内在物理限制，为理解模型预训练动力学提供了可解析的理论框架。

Detailed notes — zh

## Background 该视频由 Ai4 频道于 2026 年 8 月 18 日发布在 YouTube 平台。内容为麻省理工学院（MIT）研究员 Yizhou Liu 的学术演讲，聚焦大语言模型中神经网络缩放定律（Neural Scaling Laws）的物理起源，介绍模型表示的几何约束、残差网络归纳偏置以及 Softmax 与交叉熵固有非线性如何决定模型性能的幂律指数。 ## TL;DR 演讲针对大语言模型测试损失随数据量与参数量呈幂律衰减（指数常在 0.3 附近）的经验规律，提出了全新的非线性涌现解释，打破了以往归因于数据自带幂律（如齐普夫定律）的传统假说。研究指出，表示空间的特征叠加干涉导致反宽度（1/w）缩放，残差网络的集成平均效应导致反深度（1/d）缩放。通过将 LM Head 与 Softmax 映射为统计物理中的玻尔兹曼分布，严格推导出了低温极限下逆温度动态演化导致的 1/3 时间幂律衰减。最终，该理论在 EleutherAI 的 Pythia 系列真实大模型 Checkpoints 上获得跨规模验证，实测曲线与理论预测高度契合。 ## Atmosphere & style 整场演讲洋溢着严谨优雅且极富启发性的跨学科理论物理风格，演讲者逻辑缜密、推演流畅。通过借用费曼的科学哲学与“盲人摸象”的生动比喻，将极其繁复的深度学习黑盒系统拆解为纯粹的极简数学模型。当复杂的实测数据曲线在统计物理推导下漂亮地“坍缩”对齐到理论直线上时，为观众带来了抽丝剥茧、知其所以然的强烈智性愉悦感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) 大语言模型的缩放定律（Scaling Laws）长期以来仅被视为经验性观察，缺乏底层物理与数学理论解释测试损失遵循幂律的根本原因及指数来源。 - [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) 测试损失通常可分解为训练 Token 数（D）幂律项、参数量（N）幂律项与不可约损失（L0），且经验拟合指数普遍聚集在 0.3 附近。 - [04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) 本项研究推翻了“缩放定律由数据自身的齐普夫幂律决定”的传统观点，提出缩放幂律本质上涌现自神经网络内部的非线性架构机制。 - [04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) 由于高维特征被压缩进有限隐层维度产生叠加态（Superposition），向量几何干涉导致损失反比于模型宽度（1/w）。 - [04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) 残差网络的层级结构呈现多层集成平均效应，在中心极限定理作用下相互抵消噪声，导致损失反比于模型深度（1/d）。 - [10:08](https://www.youtube.com/watch?v=rVAFn58qo-0&t=608s) 借助物理学极简模型思维，LM Head 与 Softmax 函数被确立为推导训练时间幂律的最小核心组件。 - [10:08](https://www.youtube.com/watch?v=rVAFn58qo-0&t=608s) 从物理学视角看，Softmax 函数等价于玻尔兹曼分布，而真实大语言模型预测少数有效 Token 的极度尖锐输出处于“低温”极限状态。 - [14:00](https://www.youtube.com/watch?v=rVAFn58qo-0&t=840s) 通过对低温自由能展开并结合梯度下降动力学积分，理论严格推导出逆温度随时间按 t^(1/3) 增长，损失随训练时间按 t^(-1/3) 幂律衰减。 - [17:02](https://www.youtube.com/watch?v=rVAFn58qo-0&t=1022s) 在真实的开源 Pythia 系列模型实验中，不同参数规模的损失曲线在双对数坐标系下坍缩至同一条线，其实测幂律指数高度贴近理论推导的 1/3。 - [17:02](https://www.youtube.com/watch?v=rVAFn58qo-0&t=1022s) 综合时间（t^-1/3）、宽度（1/w）与深度（1/d）三大机制，证明了缩放普遍性源于模型架构本身，为未来设计超越当前缩放极限的架构提供了理论基石。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) 神经网络缩放定律的起源与核心问题麻省理工学院（MIT）生命与非平衡系统物理中心（Center for Physics of Living and Non-Equilibrium Systems）的博士研究生 Yizhou Liu 提出了关于神经网络缩放普遍性（Neural Scaling Universality）的最新研究成果。该研究尝试利用统计物理学、非线性动力学及复杂系统的工具，去剖析大型语言模型（LLM）为何越庞大越有效。缩放定律（Scaling Laws）表明，测试损失（Test Loss，即衡量模型性能的核心指标）会随着训练 Token 数、模型参数量（N）以及总计算量（Compute）呈现幂律衰减（Power Law）。在双对数坐标系（Log-Log Scale）下，这种关系表现为一条极其平直的直线。以 GPT-4 的技术报告为例，拟合出的缩放曲线不仅能完美覆盖小模型的训练表现，还能准确预测未训练的大模型性能。然而，这一现象长期以来仅属于“经验性规律”（Empirical Observation），缺乏底层理论支撑。使用者知其然却不知其所以然，这也引发了双重影响：一方面是希望模型训练能变得更快，另一方面是担忧缩放效应是否会在某处戛然而止。为了回答这些关键疑虑，研究提出了两个根本性科学问题：**为什么测试损失会遵循幂律分布？又是什么决定了幂律指数的大小？** - **测试损失的分解公式**：通常使用的交叉熵损失（Cross-Entropy Loss）可拆解为三部分： $$L = D^{-\alpha_D} + N^{-\alpha_N} + L_0$$ - **训练 Token 数（D）的幂律关系**：当模型未充分训练时，损失随 Token 数增加而降低，实验测得的指数 $\alpha_D$ 范围通常在 $0.28 \sim 0.37$ 之间。 - **参数量（N）的幂律关系**：当模型充分训练后，受限于模型容量导致的剩余损失随参数量增大而降低，实验测得的指数 $\alpha_N$ 范围在 $0.34 \sim 0.35$ 之间。 - **不可约损失（$L_0$）**：因交叉熵损失的固有性质，损失无法降低至绝对零度，因而存在基底常数 $L_0$。 - **普遍性疑点**：不同实验测得的幂律指数均显著聚集在 $0.3$ 左右，该数值偏小，表明当前缩放改善效率整体较慢。 --- ### [04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) 传统“数据幂律假说”与非线性涌现机制在学术界早期的主流观点中，绝大多数研究者倾向于将损失的幂律归咎于“数据本身自带幂律”（Data Power Laws）。最典型的例子就是自然语言中的齐普夫定律（Zipf's Law）：将词频或 Token 频率按照降序排列，频率与排名（Rank）在双对数坐标系下呈现斜率约为 $-1$ 的直线。传统理论认为，模型在学习过程中会优先掌握高频、最重要的特征，随后再逐步学习次要特征。由于每个特征自身的收敛速度遵循由数据频率决定的幂律，将所有特征的收敛曲线累加叠加后，整体的平均损失便展现出幂律特征。然而，Yizhou Liu 的团队提出了全新的视角：**缩放幂律并不依赖于协变量或数据的幂律分布，而是直接涌现自模型内部的非线性机制（Nonlinearities）。** ```mermaid flowchart TD accTitle: 神经网络缩放定律的三大涌现机制 subgraph S1["1. 训练时间约束 (Training)"] A["训练时间不充分"] --> B["Softmax & 交叉熵非线性特征"] B --> C["时间 scaling: τ^(-1/3)"] end subgraph S2["2. 表示与宽度约束 (Representation)"] D["有限隐层维度 (Width)"] --> E["高维特征挤入低维空间 (Superposition)"] E --> F["几何干涉导致 Width scaling: 1/w"] end subgraph S3["3. 变换与深度约束 (Transformation)"] G["上下文预测下一 Token"] --> H["残差网络多层集成平均 (Ensemble)"] H --> I["中心极限定理导致 Depth scaling: 1/d"] end C & F & I --> J["解释经验幂律指数 ≈ 1/3 (0.3)"] ``` - **数据特征叠加效应**：传统观点认为单特征收敛速度极快，但因数据频率呈幂律分布，多特征叠加求和后呈现幂律。 - **训练不足（Insufficient Training）**：源于 Softmax 与交叉熵的非线性，表现为损失随时间 $\tau$ 呈现 $\tau^{-1/3}$ 缩放。 - **宽度限制（Width Constraint）**：源于叠加态（Superposition），世间数百万特征需压缩至万维隐空间，矢量间的几何干涉（Geometric Interference）导致损失按反宽度（$1/w$）缩放。 - **深度限制（Depth Constraint）**：源于残差网络的集成平均（Ensemble Averaging），多层结构通过中心极限定理相互抵消噪声，导致损失按反深度（$1/d$）缩放。 - **三大机制综合统一**：结合训练时间、宽度与深度的缩放关系，并在最佳模型形状下平衡宽度与深度，可推导出总参数量 $N$ 的幂律指数恰好约为 $1/3$，完美解释了此前文献中报道的经验测量结果。 --- ### [10:08](https://www.youtube.com/watch?v=rVAFn58qo-0&t=608s) 物理学思维模式与“盲人摸象”的极简模型演讲者借用费曼（Richard Feynman）的科学哲学思想，将复杂的大语言模型比作“盲人摸象”故事中的大象。LLM 包含嵌入层（Embedding）、自注意力层（Self-Attention）、多层感知机（MLP）以及语言模型头（LM Head），结构极其繁复。如果研究者试图全盘解构整个系统，往往会陷入摸到象耳以为是“扇子”、摸到象腿以为是“大树”的局部陷阱。真正的科学研究思路应当是“建立正确的近似”——抽取捕捉系统本质的极简最小组件（Toy Model）进行数学分析，推导理论后再回到真实 LLM 上验证，形成闭环。对于训练过程中的幂律涌现，能够复现幂律现象的最小核心组件正是模型的 **LM Head（语言模型头）与 Softmax 函数**。 ```mermaid flowchart LR accTitle: 语言模型头 (LM Head) 最小预测流程 A["隐状态 Vector (X)"] --> B["线性变换矩阵 (W)"] B --> C["未归一化对数概率 Logits (Y)"] C --> D["Softmax 非线性层"] D --> E["下一 Token 概率分布 (P)"] ``` - **语言模型预测机制**：输入上下文转换为向量后经过 Transformer 多层叠块，最后由 LM Head 将隐层向量映射为词表上的概率分布。 - **词表选择的稀疏性**：以常见 LLM 为例，词表大小（Vocabulary Size）约为 50,000，但在具体上下文下，合理的预测选择平均仅有约 5 个。 - **玻尔兹曼分布映射**：物理学视角下，Softmax 极其等价于热力学中的玻尔兹曼分布（Boltzmann Distribution）： $$P_i = \frac{e^{Y_i / T}}{\sum_j e^{Y_j / T}}$$ 其中 Logits 的标准差（$\beta$ 或指数项）扮演了逆温度（Inverse Temperature, $\beta = 1/T$）的角色。 - **高/低温状态差异**： - **高温状态（Small $\beta$）**：输出概率分布非常平坦（Flat）。 - **低温状态（Large $\beta$）**：输出概率分布极度集中（Peaked/Sharp）。 --- ### [14:00](https://www.youtube.com/watch?v=rVAFn58qo-0&t=840s) 逆温度动态学与 1/3 幂律指数的理论推导为了在简化模型中研究 Softmax 的动力学，研究引入了经典的“教师-学生模型”（Teacher-Student Setup）。固定一个线性矩阵 $W^*$ 作为“教师”，让“学生”矩阵 $W$ 通过梯度下降（Gradient Descent）去拟合教师。实验发现，在高温（低 $\beta$）状态下，损失函数呈指数级快速收敛（Exponential Convergence）；但在**低温（高 $\beta$）状态**下，双对数坐标系中的损失收敛轨迹呈现平直的直线——即涌现出标准的幂律行为。在训练大部分时间内，学生矩阵与教师矩阵保持方向对齐（Aligned Student），其主要动态表现为矩阵范数（Norm）即逆温度 $\beta$ 的增长。 - **自由能展开（Free Energy Expansion）**：在低温极限下，交叉熵损失本质上对应热力学中的自由能（Free Energy）。对温度（$\beta^{-1}$）进行泰勒级数展开（Taylor Expansion），仅保留主导阶次项（Leading Term）。 - **梯度下降动力学**：逆温度 $\beta$ 的有效变化率受制于梯度下降： $$\frac{d\beta}{dt} \propto -\frac{\partial L}{\partial \beta}$$ - **时间积分计算**：结合泰勒展开项与动力学方程进行简单的求导与时间积分后，可推导出逆温度随时间的增长关系： $$\beta(t) \propto t^{1/3}$$ - **损失缩放结论**：将 $\beta(t)$ 重新代回损失函数展开式，即可严格推导出损失随训练时间 $t$（或 Token 数 $D$）的幂律关系： $$L(t) - L_0 \propto t^{-1/3}$$ - **梯度消失机制**：Softmax 在生成极度尖锐（Peaked）的低熵分布时，非线性本身会导致梯度固有的幂律形式衰减（Vanishing Gradient），进而通过时间累积导出 $1/3$ 这个独特的数学指数。 --- ### [17:02](https://www.youtube.com/watch?v=rVAFn58qo-0&t=1022s) 实测验证与全局理论总结理论推导完成后，Yizhou Liu 团队回到真实的开源大语言模型（如 EleutherAI 推出的 Pythia 系列模型）上验证闭环。由于大语言模型在预测下一 Token 时需要从数万个词汇中精确挑选数个选项，其概率分布天然处于极度尖锐的“低温状态”。通过对不同参数量级（从数千万到数十亿参数）的 Pythia 模型 Checkpoints 进行评估，并将损失拟合为 $L = A \cdot t^{-\alpha} + L_0$ 形式，所有不同规模模型的损失收敛曲线在双对数坐标系下成功**坍缩（Collapse）到了同一条直线上**，实测拟合出的幂律指数均分布在 $0.3$ 附近（与理论值 $1/3 \approx 0.333$ 高度吻合）。 - **真实 LLM 数据特征**：词表规模 $\sim 50,000$，候选 Token 数 $\sim 5$，验证了低熵/低温假设在真实模型中的成立。 - **Pythia 开源模型验证**：跨越不同模型尺寸的梯度下降 Checkpoints 均展现出稳定在 $0.3$ 左右的衰减指数。 - **三大机制的理论集大成**： 1. **训练不足（Insufficient Training）**：Softmax 尖锐输出导致的梯度衰减，导出时间/Token 数的 $D^{-1/3}$ 缩放。 2. **表示挤压（Superposition）**：有限宽度下的特征几何干涉，导出宽度 $N_{\text{width}}^{-1}$ 缩放。 3. **变换深度（Transformation）**：残差网络集成平均与中心极限定理，导出深度 $N_{\text{depth}}^{-1}$ 缩放。 - **最终结论**：神经网络缩放定律并非源自复杂的外部数据分布，而是直接蕴含于当前模型架构（Softmax、宽深结构、残差连接）的数学与物理特性之中。这一发现不仅揭示了“为何越大越好”的本质，更为未来全新网络架构的设计与训练策略优化提供了理论基石。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) 为什么测试损失会遵循幂律分布？又是什么决定了幂律指数的大小？ — Yizhou Liu > [00:04](https://www.youtube.com/watch?v=rVAFn58qo-0&t=4s) 测试损失（Test Loss，即衡量模型性能的核心指标）会随着训练 Token 数、模型参数量（N）以及总计算量（Compute）呈现幂律衰减（Power Law）。 — Yizhou Liu > [04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) 缩放幂律并不依赖于协变量或数据的幂律分布，而是直接涌现自模型内部的非线性机制（Nonlinearities）。 — Yizhou Liu > [04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) 世间数百万特征需压缩至万维隐空间，矢量间的几何干涉（Geometric Interference）导致损失按反宽度（1/w）缩放。 — Yizhou Liu > [04:20](https://www.youtube.com/watch?v=rVAFn58qo-0&t=260s) 多层结构通过中心极限定理相互抵消噪声，导致损失按反深度（1/d）缩放。 — Yizhou Liu > [10:08](https://www.youtube.com/watch?v=rVAFn58qo-0&t=608s) 真正的科学研究思路应当是“建立正确的近似”——抽取捕捉系统本质的极简最小组件（Toy Model）进行数学分析，推导理论后再回到真实 LLM 上验证，形成闭环。 — Yizhou Liu > [14:00](https://www.youtube.com/watch?v=rVAFn58qo-0&t=840s) Softmax 在生成极度尖锐（Peaked）的低熵分布时，非线性本身会导致梯度固有的幂律形式衰减（Vanishing Gradient），进而通过时间累积导出 1/3 这个独特的数学指数。 — Yizhou Liu > [17:02](https://www.youtube.com/watch?v=rVAFn58qo-0&t=1022s) 神经网络缩放定律并非源自复杂的外部数据分布，而是直接蕴含于当前模型架构（Softmax、宽深结构、残差连接）的数学与物理特性之中。 — Yizhou Liu ## Mentioned Yizhou Liu, 麻省理工学院 (MIT), MIT生命与非平衡系统物理中心 (Center for Physics of Living and Non-Equilibrium Systems), GPT-4, 齐普夫定律 (Zipf's Law), 理查德·费曼 (Richard Feynman), EleutherAI, Pythia, Transformer, 玻尔兹曼分布 (Boltzmann Distribution), Ai4 ## Video description Yizhou Liu - Researcher, MIT The success of today's large language models (LLMs) depends on the observation that larger models and longer training lead to better performance. However, the origin of this observation, called the neural scaling laws, that loss decreases as a power law with model size and training time, remains unclear. This talk will present the newly identified mechanisms: geometric constraints of representation lead to a part of loss inversely proportional to model width, inductive bias of residual networks leads to most layers being similar, yielding another part inversely proportional to model depth, and the intrinsic power-law behaviors in softmax and cross-entropy functions lead to a power-law loss with training time whose exponent is one-third. These results emphasize that the origin of neural scaling laws is in model architectures, providing insights into designing future training strategies and architectures. [Source](https://www.youtube.com/watch?v=rVAFn58qo-0)

Source quotes

[0:04] Yizhou Liu: 为什么测试损失会遵循幂律分布？又是什么决定了幂律指数的大小？ [0:04] Yizhou Liu: 测试损失（Test Loss，即衡量模型性能的核心指标）会随着训练 Token 数、模型参数量（N）以及总计算量（Compute）呈现幂律衰减（Power Law）。 [4:20] Yizhou Liu: 缩放幂律并不依赖于协变量或数据的幂律分布，而是直接涌现自模型内部的非线性机制（Nonlinearities）。 [4:20] Yizhou Liu: 世间数百万特征需压缩至万维隐空间，矢量间的几何干涉（Geometric Interference）导致损失按反宽度（1/w）缩放。 [4:20] Yizhou Liu: 多层结构通过中心极限定理相互抵消噪声，导致损失按反深度（1/d）缩放。 [10:08] Yizhou Liu: 真正的科学研究思路应当是“建立正确的近似”——抽取捕捉系统本质的极简最小组件（Toy Model）进行数学分析，推导理论后再回到真实 LLM 上验证，形成闭环。 [14:00] Yizhou Liu: Softmax 在生成极度尖锐（Peaked）的低熵分布时，非线性本身会导致梯度固有的幂律形式衰减（Vanishing Gradient），进而通过时间累积导出 1/3 这个独特的数学指数。 [17:02] Yizhou Liu: 神经网络缩放定律并非源自复杂的外部数据分布，而是直接蕴含于当前模型架构（Softmax、宽深结构、残差连接）的数学与物理特性之中。

Source quotes

[0:04] Yizhou Liu: Why are large language models large? [0:04] Yizhou Liu: larger is better. Yizhou Liu: These results emphasize that the origin of neural scaling laws is in model architectures, providing insights into designing future training strategies and architectures. [0:04] Yizhou Liu: L(D, N) = (D_c / D)^{\alpha_D} + (N_c / N)^{\alpha_N} + L_0 [12:51] Yizhou Liu: P_i = \frac{e^{y_i}}{\sum_j e^{y_j}} [15:28] Yizhou Liu: \beta(t) \sim t^{1/3} [15:28] Yizhou Liu: L(t) - L_0 \sim t^{-1/3} [15:28] Yizhou Liu: \frac{d\beta}{dt} = -\frac{\partial L}{\partial \beta}

Original transcript

[0:04] Uh my name is Ejo and I'm a graduate uh PhD student at MIT. Our center is called uh physics [0:10] of living and non-equilibrium systems uh where we study like living systems or the I'm trying to [0:17] understand like using statistical physics uh like non-linear dynamics, dynamic systems, like try to [0:24] understand the emergence of life, the emergence of intelligence, blah blah blah. But today, [0:29] I'm going to quickly I'm going to introduce our um recent ideas or findings on neural scaling laws, [0:36] which is called neural scaling universality. So, we start with the question, big question, [0:41] like why are large language models large? Right? Large is actually part of its name. And [0:48] the answer is actually simple. It's like larger is better. More quantitatively, uh can we do next? [0:59] Okay. More quantitatively, it's like the test loss, which is uh uh measure of the performance, [1:05] right? Lower loss is loss is better. It's like scale as a power law with number of tokens [1:12] you use for training. Like in log-log scale, it's a line. And the next, let's wait. [1:21] And after you reach the saturation, like for each given model, you train to very well, and the final [1:28] loss also scale as a power law with number of parameters. Okay. And combine these two power [1:35] laws, we will find uh the loss also scales as a power law with compute, which I will define later. [1:44] And can we do next? Okay. These neural scaling laws, they're quantitatively saying like larger [1:52] is better and following power laws, they're very very uh in two sense. One sense is like this is [1:58] a plot from uh GPT-4 technical report. Like they can feed all the small models very well. [2:07] And can we do the next? And also uh the extrapolation of this fitted scaling law [2:18] can fit the larger model or can predict the larger models performance before it's even [2:23] trained. So it's powerful because it can fit all the data very well and it's predictive. [2:29] However, it's empirical. By empirical, I mean it's like there is no real understanding of it. We're [2:35] making use of it, but we don't know where it comes from. And this raise hopes and concerns. The hope [2:41] is can we make it faster? And the concern is will it stop at some uh place? And can we do the next? [2:51] So here, to understand those questions, we need to understand the science, which is like [2:59] there are two concrete questions I'm going to try to answer as why the loss is a power law [3:05] and what determines the power law exponents. And can we do the next? Okay. [3:14] And since we want to understand the exponents, I will show some quantitative things. It's like the [3:20] loss, usually we call it cross entropy loss, can be decomposed into three parts. The first part D [3:26] is like the data set size or number of training tokens. And it follows a power law. If you haven't [3:32] trained very well, you have a high loss. It also follows a power law with exponent measured like [3:37] around 0.28 to 0.37 from different experiments. And once you train to very well, as we said, [3:45] there is still a remaining term because of you have limited model size, which is N, [3:50] number of parameters, and it also follows a power law with respect to n like to the power of 0.34 to [3:57] 0.35 measured by different experiments. And at the end, we have a irreducible loss called L naught [4:04] uh simply because this cross entropy loss cannot go to zero. And we try to understand why this [4:11] exponents are all around 0.3, which is actually small and is actually slow. Uh can we do the next? [4:20] Okay. And for a little bit litera- literature review, like before us, [4:25] the majority of the people think uh we have power laws in loss because we have power laws in data. [4:32] One of the most um famous example of power laws in data is like the word frequency [4:38] or token frequency. If you uh plot the most frequent word, what's the frequency? The second [4:44] uh most frequent word, what's the frequency? And plot this frequency as a function of the rank. And [4:51] actually, you plot in log-log scale, it's a line with exponent around one. And can we do the next? [5:00] And uh with this ideas like we have power laws in data, we have different importance [5:05] for different features, different frequencies for different features. [5:09] And if you learn the most important thing first, okay, and the second important thing second. [5:16] And because they have a power law each one is like very fast convergence, but [5:20] their convergence rate follow power law. And if you sum them up, the loss, the total loss, average [5:27] loss will be a power law. And can we do the next? It's uh animation actually. And you see each [5:34] feature, each task like is this kind of convergence uh very fast convergence, and each [5:41] one but because their frequency following power law, if you sum them up, it's will be a power law. [5:50] And we can do the next. Okay, but our understanding like this is a very high-level [5:58] overview, is like this power laws actually don't depend on covariates or data power laws. [6:05] They can emerge from nonlinearities. And can we do the next? The first is about training. [6:14] Because we have limited training, the loss is a high. [6:19] And the power law is actually following training time with tau with exponent 1/3. [6:26] And I will actually focus on this in the rest of my talk. [6:34] Okay. And in the real data, [6:37] we measured the exponent around 0.3, which is actually close to 1/3. [6:45] Okay. And even you train to very well, because like you have limited model size, [6:52] uh particularly you have limited width, you cannot really represent all the features very well. [6:58] And this is what we call superposition. It's like you have millions of features in the world you're [7:03] going to represent, and each feature is basically a direction, a vector in this hidden state. [7:10] And but you only have like maybe 10,000 um like neurons or 10,000 at the dimension [7:17] hidden dimension, so this features are squeezed into a low-dimensional space, like the dimension [7:23] is much less than the number of features, and they will have to interfere with each other. And this [7:28] lead to a inverse width scaling because of geometric interference between features. [7:36] And again, it actually agree with data very well. But, I [7:43] Can we go to the next? Okay. Like, we measure this real data shows like 0.98. And at the end, [7:51] even you can train the model very well, it can represent all the features very well, there's [7:56] still a constraint of transformation because you want to predict next token. You want to transform [8:03] the previous context to what's the possible next token. And this transformation is limited [8:08] by how many layers you have or what what we call depth. And we found it's like uh Different layers, [8:17] they reduce loss by ensemble averaging, which means they they all like perform similarly [8:24] to one direction, but each one with some noise, with some error. And they try to cancel off each [8:30] other's error. And by central limit theorem, we have this one over width our over depth scaling. [8:39] Okay. Uh can we do the next slide? And in uh yeah, in experiments like [8:45] for real data, we found it's also close. It's around 0.2. And can we do the next? [8:52] And actually combining all these three scaling laws, what we can derive is uh actually the [8:59] number of training tokens is proportional to tau. So, you the 1/3 time scaling tau scaling [9:05] will transfer to a 1/3 D scaling. And by balancing the width and depth to reach the optimal loss, [9:14] we'll find at that optimal shape of the model, we'll have a 1/3 model size scale where model size [9:21] is N, which is number of total parameters. And this basically can explain the previously reported [9:27] uh new scaling laws where the measured exponents are all 1.3 if you remember. [9:34] And due to the time uh limit, I will today only focus on the time scaling uh because of what I [9:40] call low temperature universality and I will also explain universality a little bit more. [9:45] I can I go to the next? Okay. And again, we focus on this part of loss because of [9:53] insufficient training. The mechanism of power law is what I call low temperature universality [9:59] and it lead to what's the exponent? Concretely, 1/3. A very uh strange number. [10:08] And uh before that, let me briefly introduce what large language models are doing. [10:16] So, let Assuming I have this context called I like, [snorts] we first have uh this embedding [10:23] layer. I'll go to the next. Which transform each token into uh vectors. And can I go to the next? [10:32] And then we have this attention layer which grab previous tokens information [10:38] done and then we have this MLP. This is combined called a transformer layer. And can I do the next? [10:46] And we repeat transformer layers for many, many times. Number of layers is called depth. [10:51] At the end, we have a iron bedding layer or what we call LM hat. Can I go to the next? And they [10:59] will project the vectors out to be probabilities. Probabilities on the of the next token. [11:09] For example, like I like and uh the iron bedding of like is like basically a distribution of [11:17] the whole dictionary about what's the possible next token. Probably uh for example, like [11:23] I can say I like dog, I like cat. So, on dog or cat, you have some non-zero probabilities. [11:30] Okay. And uh can I go to the next? I will quote Feynman here and also use an example. It's called [11:40] our story like blind men and elephants. It's like the LM is a very complex thing which is like the [11:47] real elephant and there's too complicated like we are just like the blind men. If we touch his ear, [11:53] we say, "Oh, elephant is a fan." If we touch it like leg, we'll say, "Oh, maybe elephant [11:57] is a tree." Because of the complexity, we cannot really scientifically understand the whole system [12:04] as a whole. But, what we can do is like as Feynman suggested like we need to have a rough idea where [12:11] they're going to and uh begin by making the right kind of approximation. This is a mindset of doing [12:18] science. We need to simplify the real thing by some into some toy model capturing the essence. [12:25] And then we can try to do some analysis. But, how do we know we're making the right kind [12:29] of approximation? Actually, I need to complete the circle by from the empirical phenomenon to [12:36] like toy modeling. And once I'm toy modeling, I go back to verify this theory on LM. [12:44] And the minimum set of ingredients I found that can reproduce parallel training is basically [12:51] what we call a hat in the models. And this basically the hat is doing transfer hidden state [12:57] into probabilities. Like both the LM hat and the language model hat. And to do that, we need one [13:04] ingredient is called softmax. Let me explain the model is like we have a hidden state called X and [13:10] it goes through a linear matrix W and it will be another vector. Let's go to the next. Y which is a [13:19] vector and then we have the non-linearity softmax acting on Y. Can go to the next. [13:28] And the softmax of Y will turn into a probability. And the for physicists, uh that's our favorite. [13:35] Actually, softmax is basically the Boltzmann distribution. Uh can we go to the next? [13:41] Like e to the power of Y divided by the summation of e to the power of Y, which will normalize them [13:47] to sum to one. Okay, can we go to the next? And the last is defined as the uh KL divergence or [13:53] cross-entropy loss. You can ignore that. And two, can we go to the next? [snorts] To study this [14:00] single head, uh we actually introduce something called teacher-student setup, [14:05] where we have the teacher being a fixed sample linear matrix W star. Can I go to the next? [14:13] And then we train the student, which is the W, to learn the teacher W star [14:19] with gradient descent as what we train the LOM. Uh can we go to the next? [snorts] Also, next. Uh uh [14:27] And here I want to introduce the physic like uh intuition here as like the standard deviation of [14:34] logits, which is I call beta star or beta, is the inverse temperature in physics, which measure the [14:40] scale of the weights or the scale of the logits. And uh for small beta or high temperature, [14:48] the output distribution will be very flat. Okay. And for very large beta or low temperature, [14:56] the output distribution will be very concentrated. Okay. And can we go to the next slide? [15:03] And the basic phenomenon is that for low temperature or for high temperature, [15:08] the loss will converge like exponentially, but for low temperature, focus on those uh [15:15] blue blue lines. In log-log [snorts] scale, they tend to be a line, which means power law. [15:21] Okay. Can we to the next? Also, next. Okay, and can I go to the next? [15:28] And uh sometimes I observe this aligned students, uh which is basically the student actually uh [15:35] is proportional to the teacher in the very early stage, and they're physically increasing in norm [15:40] during most time of the training. And then, I can do some theoretical analysis [15:44] why this power law emerge. Can I go to the next? [snorts] So, uh basically, when it's aligned, [15:51] what is only trained is the norm, or it's the inverse temperature beta for uh for the student. [15:58] Can I go to the next? And it will converge to the teacher beta star. Can I go to the next? [16:06] And uh this effective dynamics of student beta is still a gradient descent dynamics. Okay, [16:13] and can I go to the next? And this is the magic. This magic is like the loss is basically some kind [16:20] of free energy. In low temperature limit, it can be expanded as a Taylor expansion with respect to [16:27] temperature, which is beta inverse, and we only keep the leading term, and combine this dynamics, [16:33] we just do a simple time integral. Can I go to the next? We [snorts] will find beta [16:39] will scale as time to the power of 1/3. This is just due to the tail expansion and time integral. [16:45] And we substitute beta into loss, we can get loss scale as time to the inverse 1/3 power law. Okay, [16:54] can I go to the next? And uh in my toy model, it simply agree with the um theory very well. [17:02] And can I go to the next? And then, we go to LLM to complete the circle. So, this plot, I just [17:08] want to show you that uh the LM LLM actually are predicting very peak distributions. And this is [17:16] very technical, but let me just give you some very concrete numbers. It's like the vocabulary size, [17:22] the dictionary, how many tokens are there for LLM is usually maybe 50,000. But uh on average, [17:31] how many tokens are possible for the next like token prediction? Our average is around five. So, [17:38] basically, you're choosing five things on average from 50,000 things. And that's the reason why [17:46] like you need to output a very peaked, a centered, a sharp distribution. And that's actually in low [17:52] temperature. And when you're trying to approximate this very sparse, low-temperature distribution, [17:57] you are falling in what uh my prediction of this 1/3 scaling law. And can can go to [18:02] the next. And then we fit the loss as a power law plus a constant. And can I go to the next? [18:09] And turns out we can perfectly collapse, or not perfectly, but well collapse onto a line [18:16] on log-log scale where from different model sizes the like exponent are all around 0.3. This is from [18:24] the Pythia model, which is an open-source model. I can evaluate the checkpoints from that. Okay. [18:29] Can we go to the next? And as a summary, it's like we have this insufficient training and we [18:37] produce a power law. And the reason is because softmax nonlinearity. The reason is you want to [18:42] output a very sharp peak distribution. And to do that, this nonlinearity intrinsically have this [18:48] property of vanishing gradient. Itself, if it it is itself a power law. And due to time integral, [18:54] you have a 1/3 uh power law. And also it's like uh we have the superposition and later inverse [19:01] width distribution, uh distribution inverse width power law. And we also have the transformation, [19:07] which lead to a inverse depth scaling. And combine them all, we can explain [19:13] like the observed data and also the previous empirical new scaling laws. Okay, thank you.