NIPS 2017 Test of Time Award "Machine learning has become alchemy.” | Ali Rahimi, Google

NIPS 2017 Test of Time Award "Machine learning has become alchemy.” | Ali Rahimi, Google — Preserve Knowledge

Preserve Knowledge

AI-assisted summary of NIPS 2017 Test of Time Award "Machine learning has become alchemy.” | Ali Rahimi, Google — original source. Check the original for full context.

Source published:

Key ideas

  • Ali Rahimi and Ben Recht received the NIPS 2017 Test of Time Award for their 2007 paper introducing random features, which used randomized projections to scale kernel machines to massive datasets. (0:07)
  • Rahimi compares modern deep learning to alchemy: practically useful and capable of tangible discoveries, but deeply reliant on empirical folklore rather than foundational, verifiable theory. (11:08)
  • Standard gradient descent can completely stall on simple architectures, such as a two-layer linear network with correlated inputs, even while second-order methods like Levenberg-Marquardt converge to machine precision in hundreds of iterations. (12:49)
  • Widely adopted techniques like batch normalization remain scientifically ungrounded, relying on the unproven assumption that mitigating internal covariate shift is what accelerates optimization. (16:11)
  • Rahimi urges the machine learning community to move beyond benchmark leaderboard chasing by investing in root-cause diagnostic experiments and developing scalable solvers beyond gradient descent. (18:29)

Detailed notes

## Background This video was published on YouTube by the channel Preserve Knowledge on March 7, 2018. It captures Ali Rahimi's acceptance address for the NIPS 2017 Test of Time Award, though no accompanying show notes or description were provided on the page.

## TL;DR Accepting the NIPS 2017 Test of Time Award for his and Ben Recht's foundational 2007 work on random features, Ali Rahimi recounts how their original empirical shortcut eventually led to rigorous theoretical foundations. Rahimi then delivers a direct critique of the modern deep learning paradigm, arguing that despite widespread societal and commercial deployment, machine learning has drifted from a principled science into a form of alchemy dependent on unverified heuristics. Highlighting gradient descent's failure on elementary two-layer models and the lack of analytical understanding behind core tools like batch normalization, he challenges the field to move beyond leaderboard chasing. He urges the community to invest in diagnostic, root-cause experiments and develop scalable alternative optimization engines to transform machine learning into a truly rigorous discipline.

## Atmosphere & style The talk opens on a lighthearted and humorous note, with Rahimi sharing affectionate, self-deprecating memories of guerrilla marketing and dodging the conference's formidable 'rigor police.' As he pivots to the state of modern deep learning, the atmosphere shifts into an earnest, bracing, and provocative wake-up call that cuts through the industry's celebratory hype. Delivered with warmth, humility, and deep respect for his peers, Rahimi's candid indictment holds the audience visibly rapt, culminating in a reflective and resounding reception from the room.

## Bulletin - [00:07](https://www.youtube.com/watch?v=x7psGHgatGM&t=7s) Ali Rahimi and Ben Recht received the NIPS 2017 Test of Time Award for their 2007 paper introducing random features, which used randomized projections to scale kernel machines to massive datasets. - [11:08](https://www.youtube.com/watch?v=x7psGHgatGM&t=668s) Rahimi compares modern deep learning to alchemy: practically useful and capable of tangible discoveries, but deeply reliant on empirical folklore rather than foundational, verifiable theory. - [12:49](https://www.youtube.com/watch?v=x7psGHgatGM&t=769s) Standard gradient descent can completely stall on simple architectures, such as a two-layer linear network with correlated inputs, even while second-order methods like Levenberg-Marquardt converge to machine precision in hundreds of iterations. - [16:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=971s) Widely adopted techniques like batch normalization remain scientifically ungrounded, relying on the unproven assumption that mitigating internal covariate shift is what accelerates optimization. - [18:29](https://www.youtube.com/watch?v=x7psGHgatGM&t=1109s) Rahimi urges the machine learning community to move beyond benchmark leaderboard chasing by investing in root-cause diagnostic experiments and developing scalable solvers beyond gradient descent.

## Key takeaways - [00:07](https://www.youtube.com/watch?v=x7psGHgatGM&t=7s) The NIPS Test of Time Award is conferred selectively to papers from a decade prior that demonstrate enduring impact on the field. - [00:07](https://www.youtube.com/watch?v=x7psGHgatGM&t=7s) Random features bypassed the quadratic scaling bottleneck of kernel machines by sampling fixed random projections and optimizing only the linear combination. - [03:06](https://www.youtube.com/watch?v=x7psGHgatGM&t=186s) The 2007 random features paper initially exhibited a gap between theory and practice, requiring far fewer projections in experiments than its uniform convergence bounds demanded. - [06:30](https://www.youtube.com/watch?v=x7psGHgatGM&t=390s) Mid-2000s NIPS enforced strict mathematical rigor through prominent theorists who scrutinized poster presentations and theoretical claims line by line. - [06:30](https://www.youtube.com/watch?v=x7psGHgatGM&t=390s) Subsequent follow-ups established that random projections act as an independent, dense basis in Hilbert space, eliminating the need to frame them as approximations of Mercer kernels. - [11:08](https://www.youtube.com/watch?v=x7psGHgatGM&t=668s) While modern machine learning has achieved remarkable commercial success and reproducibility, it has drifted into an uncritical, heuristic complacency. - [11:08](https://www.youtube.com/watch?v=x7psGHgatGM&t=668s) Like historical alchemy, deep learning produces valuable empirical inventions but lacks the principled atomic theory required to separate real mechanisms from superstition. - [12:49](https://www.youtube.com/watch?v=x7psGHgatGM&t=769s) Because machine learning models now govern high-stakes societal domains like healthcare, civic discourse, and elections, empirical trial-and-error is no longer sufficient. - [12:49](https://www.youtube.com/watch?v=x7psGHgatGM&t=769s) Gradient descent frequently fails on elementary setups, stalling on badly conditioned two-layer linear networks despite substantial non-zero gradients. - [16:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=971s) Modern deep learning stacks are highly brittle, where minor changes like internal rounding modes can cause error rates to jump catastrophically. - [16:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=971s) The standard explanation that batch normalization works by mitigating internal covariate shift lacks formal mathematical definitions, verification, or theoretical proof. - [18:29](https://www.youtube.com/watch?v=x7psGHgatGM&t=1109s) The research community must pivot away from incremental leaderboard competition toward diagnostic experiments designed to isolate root causes. - [18:29](https://www.youtube.com/watch?v=x7psGHgatGM&t=1109s) Developing scalable linear system solvers and matrix factorization engines capable of handling tens of billions of variables would free the field from total reliance on gradient descent.

## Detailed walkthrough ### [00:07](https://www.youtube.com/watch?v=x7psGHgatGM&t=7s) Presentation of the NIPS 2017 Test of Time Award

The session chair opens the morning program by explaining that a Test of Time Award is not an annual fixture at NIPS; indeed, the award itself must stand the test of time to even be conferred. For 2017, the program committee decided to look back a full decade to NIPS 2007 papers. A dedicated committee evaluated candidate works, a process the chair admits was difficult for all involved, joking that authors whose papers were passed over should not feel slighted.

```mermaid flowchart LR accTitle: Evolution of Random Features accDescr: Pipeline showing how kernel approximation evolved from 2007 to full generalization theory. subgraph S1["NIPS 2007: Kernel Approximation"] direction TB A["Exact Kernel Matrix<br>(O(N²) scaling)"] --> B["Random Fourier & Binning Features"] B --> C["Linear Solver on<br>Randomized Space"] end subgraph S2["Follow-ups & Generalization"] direction TB D["Hilbert Space Basis<br>(Beyond Kernels)"] --> E["Random Kitchen Sinks<br>& Direct Bounds"] end S1 --> S2 ```

To introduce the winning paper, the chair reads the authors' own summary describing random features as an elegant trick to accelerate supervised learning algorithms to handle big datasets spanning millions of points. Traditional learning approaches expend excessive computational effort optimizing parameters that do not strictly require optimization; instead, practitioners can draw parameters at random from a distribution and quickly optimize only the linear combination over them. Thanks to the concentration of measure phenomenon, this randomized shortcut preserves strong accuracy while remaining straightforward to implement. The chair highlights the paper's specific contributions: - Introducing random Fourier features to approximate Gaussian kernels efficiently. - Introducing random binning features to approximate separable multivariate shift-invariant kernels. - Paving the way for subsequent variants like random stumps for boosting, random support vectors for empirical kernel maps, and the widely known umbrella concept of "random kitchen sinks."

The chair then announces the award for *Random Features for Large-Scale Kernel Machines* to Ali Rahimi and Ben Recht, inviting Rahimi to the podium to deliver the address.

---

[03:06](https://www.youtube.com/watch?v=x7psGHgatGM&t=186s) The 2006 Origins: Two Emails and Guerrilla Marketing

Taking the stage alongside Ben Recht, Ali Rahimi begins with self-deprecating humor about aging in computer science, quipping that nothing makes an engineer feel quite as ancient as receiving an accolade literally called the "Test of Time." He announces that this 2007 paper formed merely the first act of an unintentional trilogy, and that as in any compelling narrative, the truly interesting turns happen in the middle and conclusion rather than the opening chapter.

Rahimi transports the audience back to NIPS 2006, an era when deep learning was first causing a stir. At the time, neural network training algorithms were notoriously intricate, and their benchmarked performance was roughly parity with linear methods such as PCA and linear support vector machines (SVMs). During workshop hallway chats, researchers noted that deep networks ought to be benchmarked against non-linear SVMs, but classical kernel machines scaled quadratically or cubically with dataset size, rendering them impossible to run on large datasets. Recht had been investigating randomized algorithms for compressed sensing, while Rahimi had been utilizing random sketches to accelerate bipartite graph matching for computer vision. Upon returning home from the conference, it took the pair just two emails to formulate a scalable kernel approximation.

```mermaid flowchart TD accTitle: Standard Kernel Machine vs Random Feature Approximation accDescr: Comparison of parameter scaling between traditional kernel machines and Rahimi-Recht random features. subgraph Classic["Traditional Kernel Machine"] C1["Data Points x_1... x_N"] --> C2["Place Kernel on Every Data Point"] C2 --> C3["Optimize N Weights (Dense QP)"] end subgraph RF["Random Features Scheme"] R1["Data Points x_1... x_N"] --> R2["Sample D Random Basis Functions"] R2 --> R3["Optimize D Weights (Linear Solver, D << N)"] end ```

The underlying mathematical observation was straightforward: standard shift-invariant kernels could be recast via Bochner's theorem as expectations over products of randomized cosine basis functions. Replacing an intractable sum of $N$ kernel functions with an empirical average of $D$ random features yielded a simple linear model with vastly fewer parameters to fit. To spread the word at their NIPS 2007 poster presentation, they engaged in what Rahimi calls algorithm guerrilla marketing, handing out flyer slips with four lines of MATLAB code: > “It’s, uh, 'Hey buddy, you want—you want to try a kernel machine? Here’s just four lines of MATLAB code.'”

However, Rahimi confesses that the paper harbored a glaring contradiction between theory and empirical behavior. Their published uniform convergence bounds demanded tens of thousands of random projections to guarantee an accurate uniform approximation of the Gaussian kernel. In actual experiments, however, they achieved exceptional validation performance using only a few hundred projections. More unsettling still, on certain benchmarks the random approximation achieved lower test error rates than the exact kernel machine it was meant to approximate.

---

[06:30](https://www.youtube.com/watch?v=x7psGHgatGM&t=390s) Evading the NIPS "Rigor Police" and Finding Pure Foundations

The mysterious gap between loose theoretical bounds and outstanding practical performance was perilous, because mid-2000s machine learning was actively hardening its standards. Rahimi reflects on how the discipline had moved past the freewheeling era that the late Sam Roweis used to term an "ideas conference," hardening into an uncompromising mathematical discipline. At poster sessions, authors were regularly grilled by roving bands of examiners that Rahimi affectionately calls the "NIPS rigor police": - Sharp theorists like Nati Srebro and Ofer Dekel. - Formidable students from Mike Jordan's laboratory. - Hardline interrogators like Shai Ben-David. - And if one was truly unlucky, Manfred Warmuth challenging assumptions line-by-line.

> “During the poster sessions you could see this roving band of what I used to call the NIPS rigor police—they would come to your poster and make sure that your arguments were airtight.”

Rather than shelving the method, Rahimi and Recht submitted the paper despite its theoretical dodge, choosing to brave the critics and promise to uncover the real explanation later. That resolution unfolded across the subsequent two papers of the trilogy. In the second paper, they stripped away the kernel framing altogether. Drawing randomly initialized smooth functions independently from an appropriate measure spans a dense basis set in an $L_2$ Hilbert space with high probability, directly paralleling how Fourier bases span functions or wide three-layer neural networks serve as universal approximators. One never needed to justify the features as approximations of Mercer kernels or eigenfunctions; randomized projections stood on their own as a complete basis for learning. In the third paper, they established formal generalization error bounds for these "random kitchen sinks" directly, bounding test error as a direct function of sample size and feature count without invoking an underlying kernel target.

Rahimi notes that he continues to deploy random features regularly in applied industry work, using them much like $k$-nearest neighbors: as a baseline diagnostic tool. When random projections effortlessly master a task, he reflects that either the technique is surprisingly capable, or the target problem was fundamentally trivial to begin with.

---

[11:08](https://www.youtube.com/watch?v=x7psGHgatGM&t=668s) From Electricity to Alchemy: Critiquing Modern Machine Learning

Turning to the present day in 2017, Rahimi underscores how dramatically the machine learning landscape has evolved over the past decade. The community largely solved its reproducibility crisis: researchers routinely publish open-source codebases, share reproducible pipelines, and test against rigorous common benchmarks. Concurrently, machine learning systems have become the bedrock of commercial reality: - Powering self-driving vehicle perception pipelines. - Transcribing voicemail messages and translating polyglot text. - Tagging consumer photo libraries and driving global advertising auctions. - Supporting multi-billion-dollar enterprise valuations.

Yet despite these triumphs, Rahimi warns that the discipline has slipped into an uncritical, euphoric complacency. He targets the ubiquitous corporate and academic boosterism claiming that deep learning represents an epochal, omnipotent utility: > “There’s a self-congratulatory feeling in the air; we say things like 'Machine learning is the new electricity.'”

> “I’d like to offer another analogy: machine learning has become alchemy.”

Rahimi stresses that comparing deep learning to alchemy is not a pure insult. Alchemy was genuinely useful; historically, alchemists made critical real-world discoveries, inventing metallurgy, industrial textile dyes, early pharmaceutical compounds, and foundational techniques for modern glassblowing. Yet because alchemists lacked an underlying principled theory of atomic elements, chemical bonding, and thermodynamics, their genuine breakthroughs were hopelessly entangled with superstition: they bloodlet patients with leeches to balance mythical humors and squandered centuries attempting to transmute base metals into gold. Rahimi argues that before physics and chemistry could catalyze the 18th-century scientific revolution, scholars were forced to systematically dismantle two millennia of accrued alchemical folklore—a cautionary tale for modern machine learning researchers relying on empirical craft over foundational understanding.

[12:49](https://www.youtube.com/watch?v=x7psGHgatGM&t=769s) Society at Stake and the Failure of Gradient Descent on Simple Models

Ali Rahimi warns that relying on alchemical heuristics might be tolerable when building consumer applications like photo-sharing apps, but machine learning now operates at a vastly higher societal level. Machine learning models govern healthcare decisions, mediate civic discourse, and influence democratic elections. Because of these consequences, Rahimi argues that society deserves infrastructure grounded in rigorous, verifiable, and thorough knowledge rather than trial-and-error guesswork, confessing: “As aggravating as the NIPS rigor police was, I miss them and I wish they’d come back.” To demonstrate where this lack of rigor causes tangible harm, he asks the audience how many have designed and trained a neural network from scratch only for it to fail, leaving them feeling as though they were personally at fault. Reassuring the crowd, he notes that this happens to him every three months and declares: “I don’t think it’s you, I don’t think it’s your fault, I think it’s gradient descent’s fault.”

```mermaid flowchart TD accTitle: Gradient Descent vs Levenberg-Marquardt on a Two-Layer Linear Net accDescr: Diagram illustrating gradient descent stalling on a badly conditioned two-layer linear network versus Levenberg-Marquardt converging to machine precision.

subgraph ProblemSetup["Test Setup: 2-Layer Linear Network"] Model["Model: Product of 2 tall matrices (Identity activations)"] Data["Labels: Badly conditioned linear function of inputs"] Model --> Loss["Loss Surface Evaluation"] Data --> Loss end

subgraph OptimizationComparison["Optimization Trajectory"] Loss --> GD["Gradient Descent & Variants"] Loss --> LM["Levenberg-Marquardt"]

GD --> Fail["Starts fast, then peters out\n(Non-zero gradient, not a saddle point or noise floor)"] LM --> Success["Reaches 0 loss in a few hundred iterations\n(Machine precision zero)"] end ```

To prove that gradient descent fails even on trivial setups, Rahimi presents the simplest deep network imaginable: a two-layer linear network with identity activation functions, where the training labels are generated by a badly conditioned linear function of the inputs (specifically, an input matrix with correlated columns). When running standard gradient descent and several of its popular variants on this model, optimization progresses rapidly at the start before abruptly petering out. Rahimi dismantles potential defenses of the algorithm: - It is not stuck in a local minimum or a saddle point, because the gradient magnitudes remain substantially non-zero. - It is not hitting a statistical noise floor or suffering from sample randomness, because running the exact expected loss produces the identical stalling curve. - In contrast, using a better descent direction like the Levenberg-Marquardt algorithm completely solves the system, dropping down to machine precision zero within a few hundred iterations.

Rahimi urges attendees to take ten minutes to reproduce this optimization experiment on their laptops, emphasizing that gradient descent is the core workhorse of modern machine learning. Addressing the counterargument that gradient descent works well on massive, complex deep networks, he remarks that the audience's raised hands show widespread personal experience to the contrary. More fundamentally, science requires testing tools on simple, interpretable setups before leaping to massive architectures: “This is how we build knowledge: we apply our tools on simple, easy-to-analyze setups, we learn, and we work our way up in complexity; we seem to have just jumped our way up.”

[16:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=971s) The Brittle Technical Stack and the Enigma of Batch Normalization

The lack of analytical clarity makes modern machine learning engineering extremely brittle. To illustrate how delicate these pipelines are, Rahimi reads an email he received just a month prior from his friend Boris: > “On Friday, someone in another team changed the default rounding mode of some TensorFlow internals from truncate toward zero to round to even. Our training broke; our error rate went from less than 25% error to almost 99% error.”

Rahimi notes that his inbox contains several messages just like this one, mirrored across public internet forums and bug trackers. This brittleness occurs because practitioners apply fragile optimization methods across loss surfaces that nobody understands, and the typical response to unexpected behavior is simply piling on more unverified layers: “Our solution is to add more mystery to an already mysterious technical stack.”

```mermaid flowchart LR accTitle: Unverified Assumptions in Batch Normalization accDescr: Flowchart mapping the widely accepted claim that batch norm reduces internal covariate shift to the unproven foundational questions raised by Ali Rahimi.

BN["Batch Normalization Inserted"] --> SpeedsUp["Accelerates Gradient Descent"] SpeedsUp -.->|"Popular explanation"| ICS["Reduces 'Internal Covariate Shift'"]

subgraph OpenQuestions["Unanswered Foundational Questions"] Q1["What is the formal definition of internal covariate shift?"] Q2["Where is the evidence that batch norm reduces it?"] Q3["Why does reducing it accelerate gradient descent?"] end

ICS -.-> OpenQuestions ```

Rahimi points to batch normalization as the quintessential example. While inserting batch normalization layers between deep network representations reliably accelerates gradient descent, the community's scientific explanation for it remains completely unsubstantiated. He acknowledges that people routinely rely on systems they do not fully understand—comparing it to flying on a passenger airplane to attend the conference without being an aerospace engineer—but points out that travelers take comfort in knowing an entire field of aeronautics rigorously investigates and tests aerodynamic principles. By contrast, the machine learning community uncritically repeats the claim that batch normalization works because it mitigates "internal covariate shift," without providing: - A rigorous mathematical definition of internal covariate shift - Empirical or theoretical evidence demonstrating that batch normalization actually reduces internal covariate shift - Any theorem or experimental proof explaining why reducing internal covariate shift would accelerate gradient descent in the first place

Rahimi summarizes the dilemma bluntly: “Batch norm has become a foundational tool in how we build deep nets, and yet as a field, we know almost nothing about it.”

[18:29](https://www.youtube.com/watch?v=x7psGHgatGM&t=1109s) From Alchemy to Electricity: A Roadmap for Rigorous Machine Learning

Recognizing that machine learning now commands immense social influence, Rahimi outlines two concrete ways the research community can fulfill its responsibilities:

1. **Shift focus from leaderboard chasing to root-cause science:** Researchers spend enormous energy testing arbitrary tweaks just to achieve an incremental performance edge on competition benchmarks. The field needs fewer leaderboard-jockeying experiments and far more diagnostic experiments designed to isolate the fundamental root causes of unexplained phenomena. Simple, controlled experiments and simple theorems are the indispensable building blocks required to comprehend larger, complex systems. 2. **Develop scalable, diverse computational engines beyond gradient descent:** Today's production systems rely almost entirely on gradient descent variants because they are currently the only solvers engineered to optimize tens of billions of parameters across commodity hardware clusters. Rahimi challenges the community to build linear system solvers and matrix factorization engines capable of scaling to tens of billions of variables on commodity systems, which would open the door to radically new optimization paradigms and model families.

Expressing deep personal affection and gratitude for the machine learning community, Rahimi concludes that his critique stems from genuine respect for the people in the room, insisting that they possess the talent needed to tackle these hard mathematical and systems challenges. He closes with an appeal for collective reform: “Ben and I are grateful for the award and the opportunity to have gotten to know many of you and to work with many of you, and we would love it if we could work together to take machine learning from alchemy and into electricity.”

[21:27](https://www.youtube.com/watch?v=x7psGHgatGM&t=1287s) Presentation of the Test of Time Award

The session chair returns to the podium amid loud applause, thanking Rahimi for the provocative and thoughtful address. The host jokes that he hopes the conference can present Rahimi with another award ten years down the line celebrating "the return of rigor" in the community. Noting that the audience is visibly stunned and deeply appreciative of the talk's core message, the host invites Rahimi's co-author, Ben Recht, onto the stage. The session chair presents Rahimi and Recht with the official certificate for the NIPS 2017 Test of Time Award, concluding the award presentation before directing the audience toward the scheduled conference break.

## Notable quotes > [11:08](https://www.youtube.com/watch?v=x7psGHgatGM&t=668s) Machine learning has become alchemy. — Ali Rahimi

> [16:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=971s) Batch norm has become a foundational tool in how we build deep nets, and yet as a field, we know almost nothing about it. — Ali Rahimi

> [12:49](https://www.youtube.com/watch?v=x7psGHgatGM&t=769s) This is how we build knowledge: we apply our tools on simple, easy-to-analyze setups, we learn, and we work our way up in complexity; we seem to have just jumped our way up. — Ali Rahimi

## Mentioned Ali Rahimi, Ben Recht, Boris, Levenberg-Marquardt algorithm, Manfred Warmuth, MATLAB, Mike Jordan, Nati Srebro, NIPS, NIPS 2007, NIPS 2017, Ofer Dekel, Random Features for Large-Scale Kernel Machines, Sam Roweis, Shai Ben-David, TensorFlow

[Source](https://www.youtube.com/watch?v=x7psGHgatGM)

中文简报

NIPS 2017 时间检验奖：“机器学习已沦为炼金术” | 谷歌 Ali Rahimi 在领取他与 Ben Recht 因 2007 年随机特征开创性工作而荣获的 NIPS 2017 时间检验奖时，Ali Rahimi 回顾了他们最初的经验性捷径最终如何促成了严谨的理论基础。随后，Rahimi 对现代深度学习范式提出了直接批评，认为尽管该技术在社会和商业领域得到了广泛应用，但机器学习已经从一门有原则的科学演变为一种依赖未经证实的经验法则的炼金术。他指出梯度下降法在基础的双层模型上可能失效，并强调批量归一化等核心工具背后缺乏解析性的理论理解，借此呼吁该领域不要局限于追逐排行榜成绩。他敦促学术界投入精力进行根因诊断性实验，并开发可扩展的替代优化引擎，以推动机器学习转变为一门真正严谨的学科。 Ali Rahimi 和 Ben Recht 凭借其在 2007 年发表的引入随机特征的论文荣获 NIPS 2017 时间检验奖，该论文利用随机投影将核方法扩展至海量数据集。 (0:07) Rahimi 将现代深度学习比作炼金术：虽然在实践中有用且能取得切实发现，但极度依赖经验传闻而非基础、可验证的理论。 (11:08) 标准梯度下降法在简单架构上可能会完全停滞，例如在输入具有相关性的双层线性网络中，而 Levenberg-Marquardt 等二阶方法却能在数百次迭代内收敛至机器精度。 (12:49) 广泛采用的技术（如批量归一化）在科学上仍缺乏充分依据，其加速优化的机制仍建立在“缓解内部协变量偏移”这一未经证实的假设之上。 (16:11) Rahimi 敦促机器学习学界停止追逐基准排行榜，转而投入资源开展根本原因诊断实验，并研发超越梯度下降法的可扩展求解器。 (18:29)

Reading summary

Accepting the NIPS 2017 Test of Time Award, Ali Rahimi argued that machine learning has slipped into an empirical craft akin to alchemy, warning that high-stakes societal infrastructure cannot safely rely on unverified heuristics and unexplained optimization tools. The Alchemical State of Modern Machine Learning (11:08) Rahimi challenged the prevailing industry boosterism that characterizes machine learning as an omnipotent utility like "the new electricity," countering that the discipline currently functions like medieval alchemy. While alchemists achieved genuine practical discoveries—such as metallurgy, industrial textile dyes, early pharmaceuticals, and glassblowing techniques—their lack of underlying atomic, chemical, and thermodynamic theories left their work entangled with folklore, leeches, and futile efforts to transmute metals into gold. Because machine learning systems now govern consequential domains including healthcare delivery, civic discourse, and democratic processes, Rahimi argued that society requires systems grounded in rigorous, verifiable knowledge rather than trial-and-error guesswork. In his view, continuing to rely on unexamined empirical craft risks building critical societal infrastructure on unstable foundations. Unverified Heuristics and Batch Normalization (16:11) Theoretical gaps between empirical performance and formal justification have persisted for a decade. Rahimi recalled that his and Ben Recht's 2007 paper on random features achieved strong practical results with hundreds of projections despite uniform convergence bounds demanding tens of thousands, ultimately requiring follow-up papers to establish proper Hilbert space foundations and generalization bounds without relying on kernel approximations. However, Rahimi warned that the modern community increasingly adopts architectural components without demanding underlying scientific explanations. Pointing to batch normalization as a prime example, he noted that while it reliably accelerates gradient descent, the standard explanation—that it mitigates "internal covariate shift"—is scientifically unsubstantiated. The field lacks a rigorous mathematical definition of internal covariate shift, empirical proof that batch normalization reduces it, or theoretical justification explaining why such a reduction would accelerate optimization. Optimization Brittleness and Gradient Descent Failures (12:49) The absence of analytical clarity renders modern deep learning pipelines exceptionally brittle. Rahimi cited an industry case where an internal TensorFlow modification from truncate-toward-zero to round-to-even rounding caused a production model's error rate to spike from under 25% to nearly 99%. When confronted with such failures, practitioners typically respond by adding further unverified layers to an already mysterious stack. To demonstrate that core optimization tools remain poorly understood, Rahimi presented an experiment on a simple two-layer linear network with correlated input features. Standard gradient descent variants stall rapidly despite non-zero gradient magnitudes that rule out saddle points or statistical noise floors. In contrast, alternative methods like the Levenberg-Marquardt algorithm converge to machine precision zero within a few hundred iterations, demonstrating that gradient descent can struggle even on elementary, interpretable setups. A Scientific Agenda for Root-Cause Diagnostics (18:29) To transition machine learning from alchemy into an established science, Rahimi outlined two priorities for researchers. First, the community must shift its focus away from leaderboard chasing and benchmark tweaking toward controlled diagnostic experiments designed to isolate the fundamental root causes of unexplained phenomena. Developing simple models and simple theorems provides the foundational knowledge necessary to understand larger, complex architectures. Second, Rahimi urged the community to build scalable computational engines beyond gradient descent. Because production systems rely on gradient descent primarily because it is engineered to scale across commodity hardware clusters, building linear system solvers and matrix factorization algorithms capable of handling tens of billions of variables would unlock entirely new optimization paradigms and model families. Rahimi emphasized that his critique reflects deep respect for the community's talent, expressing optimism that researchers can revive rigorous examination and collectively transform machine learning from an alchemical craft into a dependable, scientifically grounded discipline.

精炼阅读摘要

Ali Rahimi 在 NIPS 2017 时间检验奖演讲中直言，现代机器学习已沦为缺乏理论根基的“炼金术”；随着算法接管关键社会系统，学界必须从追逐排行榜分数转向对根因机制的严谨探究与多元求解器的研发。 随机特征的演进与理论严密性 (3:06) Ali Rahimi 与 Ben Recht 因 2007 年提出的《大规模核机器的随机特征》（Random Features for Large-Scale Kernel Machines）获得时间检验奖。面对当时非线性支持向量机（SVM）高昂的二次或三次计算复杂度，两人利用 Bochner 定理将移位不变核转化为随机余弦基函数的期望，将 intractable 的 N 项核函数求和替换为 D 个随机特征的经验均值，在大幅降低优化维度的同时保持了高准确率。然而 Rahimi 坦言，初期论文的理论界限与实验表现存在明显脱节：理论上一致收敛界要求数万个随机投影以逼近高斯核，但实际只需数百个投影便能跑出优异性能，甚至测试误差比原始精确核方法还要低。随后两人通过后续研究摆脱了核逼近的叙事框架，直接将随机平滑函数视为 L2 希尔伯特空间的一组完备基，并针对“随机厨房水槽”（Random Kitchen Sinks）推导了不依赖底层核目标的直接泛化误差界。 从电力到炼金术的技术隐患 (11:08) 针对产业界与学术界将深度学习奉为“新电力”的狂热论调，Rahimi 提出了“机器学习已成为炼金术”的对照。他指出，历史上的炼金术虽催生了冶金、染料、玻璃制造与早期药理学等实用发明，但由于缺乏原子论、化学键和热力学的底层解释，其成果长期与巫术迷信杂糅在一起，直到 18 世纪科学革命才将积攒两千年的民间经验彻底清理。如今机器学习模型广泛应用于自动驾驶、多语言翻译及商业广告，并进一步延伸至医疗决策、公众舆论和民主选举等社会关键节点。Rahimi 警告，在涉及重大利益的基础设施上继续沿用缺乏数学根基的经验试错法是极其危险的，学界必须重建过去那种对理论论证严格审视的学术风气。 梯度下降与批归一化的根基盲区 (12:49) 为了证明算法工具缺乏严谨分析，Rahimi 指出当前整个领域的核心主力——梯度下降，在极简单的问题上也会出现停滞。他在带有相关性输入的两层恒等激活线性网络中测试发现，标准梯度下降及其常见变体会在非零梯度处过早停滞，且这并非由局部极值、鞍点或样本统计噪声造成，而换用 Levenberg-Marquardt 等更优的下降方向后只需数百次迭代就能收敛到机器精度的零损失。这种脆弱性延伸到生产环境中造成了极高的系统脆弱度，例如 TensorFlow 内部仅仅将截断取整改为就近舍入，便导致某团队的测试错误率从低于 25% 飙升至近 99%。与此同时，被广泛应用以加速网络训练的批归一化（Batch Normalization），其被公认的“减少内部协变量偏移”（Internal Covariate Shift）假说，既缺乏严格的数学定义与实证支撑，也无法在理论上解释其为何能加速梯度下降。 告别刷榜：从炼金术迈向真正的科学 (18:29) 面对深度学习脆弱的黑盒现状，Rahimi 呼吁研究者停止通过盲目添加未验证层来掩盖技术栈的不确定性。他为推动机器学习走向真正的“电力时代”提出了两条具体路径：其一，将研究重心从竞赛排行榜的刷榜微调，转向旨在隔离与揭示底层因果的诊断性实验，先在可解析的简单模型上建立清晰定理，再逐步向复杂架构推进；其二，打破单一依赖梯度下降的局限，针对通用计算集群研发能够扩展至百亿级参数的大规模线性方程组求解器与矩阵分解引擎，从而为模型结构与优化范式开辟新的数学工具库。 大会主席在颁奖结尾对 Rahimi 提出的尖锐批评表示感谢，并期待未来能见证严密性在机器学习研究中的全面回归；然而，如何在百亿参数规模的高维非凸表面上彻底摆脱对经验启发的依赖，建立完备的动力学与泛化理论，仍是整个学界悬而未决的挑战。

Detailed notes — zh

## Background 该视频由 YouTube 频道 Preserve Knowledge 于 2018 年 3 月 7 日发布。内容记录了 Ali Rahimi 在 NIPS 2017 大会上凭借与 Benjamin Recht 合作的论文荣获“时间检验奖”（Test of Time Award）时的获奖演讲，页面未提供额外的简介说明。 ## TL;DR 在 NIPS 2017 颁奖典礼上，Ali Rahimi 回顾了他与 Benjamin Recht 在十年前提出利用随机特征加速核机器的研究历程，讲述了该成果如何从最初两封电子邮件的灵感演变为完备的理论体系。随后，他将演讲焦点转向整个机器学习领域，尖锐地指出当下的机器学习已演变为缺乏第一性原理指导的“炼金术”，并在诸如梯度下降失效和批归一化（Batch Normalization）机理不明等黑箱问题上越陷越深。最后，拉希米呼吁学术界告别盲目的打榜调参，重拾严谨的科学探究精神与底层数学工具研发，共同将机器学习从“炼金术时代”推向真正成熟的“电学时代”。 ## Atmosphere & style 整场演讲开篇轻松幽默，演讲者通过自嘲年龄和分享当年贴传单推销算法的轶事引得全场会心欢笑，随后语调陡然转向严肃深沉的学术省思。当他直言“机器学习已经变成了炼金术”并质问整个领域对底层机制的无知时，现场气氛充满了震撼与思想共鸣。Rahimi 的发言兼具直击痛点的犀利与对学术社区的深厚关切，犹如给狂热追逐落地与榜单的人工智能浪潮当头泼下一盆清醒剂，最终在热烈而崇敬的掌声中落幕。 ## Bulletin - [00:07](https://www.youtube.com/watch?v=x7psGHgatGM&t=7s) Ali Rahimi 与 Benjamin Recht 凭借 2007 年提出的随机特征加速核方法论文荣获 NIPS 2017 时间检验奖，该方法将非线性核学习大幅简化为随机特征上的线性优化。 - [11:08](https://www.youtube.com/watch?v=x7psGHgatGM&t=668s) Ali Rahimi 在获奖演讲中提出轰动业界的论断“机器学习已经变成了炼金术”，直指当前领域在繁荣表象下严重缺乏第一性原理理论支撑与学术严谨性。 - [14:09](https://www.youtube.com/watch?v=x7psGHgatGM&t=849s) 演讲指出深度学习技术栈极度脆弱且优化机制不明：梯度下降在简单的两层病态线性网络中即会陷入停滞，工程上底层的四舍五入微调便可能导致模型训练彻底崩溃。 - [18:29](https://www.youtube.com/watch?v=x7psGHgatGM&t=1109s) Rahimi 呼吁学术界摒弃单纯追逐测试集排名的打榜风气，转向探究根本原因的严谨科学范式，并大力研发百亿级变量规模的底层线性系统求解器与矩阵分解引擎。 ## Key takeaways - [00:07](https://www.youtube.com/watch?v=x7psGHgatGM&t=7s) NIPS 2017 时间检验奖授予 Ali Rahimi 与 Benjamin Recht 于 2007 年发表的《大规模核机器的随机特征》（Random Features for Large-Scale Kernel Machines）。 - [03:06](https://www.youtube.com/watch?v=x7psGHgatGM&t=186s) 该论文的核心思想在于利用随机傅里叶特征将位移不变核函数近似展开为随机特征的线性组合，成功解决了传统核机器无法扩展至大规模数据的计算瓶颈。 - [05:45](https://www.youtube.com/watch?v=x7psGHgatGM&t=345s) 初期算法实验中仅需数百个特征即可取得极佳效果，泛化表现甚至超越原始核模型，这一现象一度与理论推导所需的大量特征上界产生脱节。 - [08:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=491s) 作者随后发表后续研究，证明随机平滑函数基能在希尔伯特空间中稠密逼近任意函数，从几何层面将“随机厨房水槽”确立为独立合法的模型范式。 - [08:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=491s) 随机特征方法因其简单高效，与 K 近邻算法一样成为衡量任务难易程度与问题复杂度的绝佳基线工具。 - [11:08](https://www.youtube.com/watch?v=x7psGHgatGM&t=668s) 针对业界流行将机器学习比作“新时代电力”的自满言论，Ali Rahimi 提出针锋相对的警示：机器学习当前已沦为依赖经验试错的“炼金术”。 - [11:08](https://www.youtube.com/watch?v=x7psGHgatGM&t=668s) 炼金术历史上虽催生了冶金、染色与玻璃制造等实用技艺，但由于缺乏基础原理解释，也伴生着水蛭治病、点石成金等荒谬行径；现代科学的诞生正是以拆解炼金术为前提。 - [12:49](https://www.youtube.com/watch?v=x7psGHgatGM&t=769s) 当机器学习已渗透到医疗诊断、公共舆论与政治选举等关乎社会运转的核心领域时，系统必须建立在可验证、严谨的科学基石上，而非经验主义黑箱。 - [12:49](https://www.youtube.com/watch?v=x7psGHgatGM&t=769s) 研发人员训练深度神经网络频繁遭遇失败并不一定是个人能力问题，很多时候本质上是底层梯度下降优化算法的固有缺陷所致。 - [14:09](https://www.youtube.com/watch?v=x7psGHgatGM&t=849s) 即便在无非线性激活函数的极简两层线性网络中，梯度下降也会因矩阵列相关而过早停滞，而高阶优化算法（如 Levenberg-Marquardt）却能迅速收敛至零误差。 - [16:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=971s) 工程应用暴露了优化技术栈的脆弱：仅因底层框架修改了默认的数值四舍五入规则，就曾导致特定模型的错误率从 25% 飙升至近 99%。 - [16:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=971s) 批归一化（Batch Normalization）虽然实践中有效，但其原理解释“减少内部协变量偏移”缺乏严格的数学定义与实证理论支撑，属于典型的“以谜解谜”。 - [18:29](https://www.youtube.com/watch?v=x7psGHgatGM&t=1109s) 研究社区应从盲目拼接模块以微调测试集分数的“打榜驱动（Leaderboard Jockeying）”，转向设计严格对照实验追查物理原因的“根因探究（Root Cause Chasing）”。 - [18:29](https://www.youtube.com/watch?v=x7psGHgatGM&t=1109s) 不应将百亿级参数的优化完全押注在现有的梯度下降变体上，呼吁计算科学界共同开发支持超大规模变量的通用硬件线性求解器与矩阵分解工具。 - [18:29](https://www.youtube.com/watch?v=x7psGHgatGM&t=1109s) Ali Rahimi 号召全体研究者怀念并恢复以往追求极致严谨的学术精神，携手合作将机器学习真正从炼金术时代推向电学时代。 ## Detailed walkthrough ### [00:07](https://www.youtube.com/watch?v=x7psGHgatGM&t=7s) NIPS 2017 时间检验奖的揭晓与背景 大会主持人走上讲台开启颁奖环节，幽默地指出并不是每届 NIPS 都会颁发“时间检验奖”（Test of Time Award），笑称“甚至光是拿到一个时间检验奖本身就是对时间的一场严峻考验”。2017 年的程序委员会特别设立了评奖标准，决定回顾十年前（即 NIPS 2007）的论文，并组建了一个评审委员会进行评选。主持人感慨道，这项决策过程绝非易事，那些未能获奖的优秀论文作者不必太沮丧，但评审委员会最终一致认定了一篇实至名归的杰出工作。 主持人援引原作者的原话来介绍该研究的核心思想：“传统的学习算法太辛苦了，因为它们去优化了那些根本不需要优化的参数；相反，人们可以对其中一部分参数进行随机化，然后迅速对剩余参数进行优化”。得益于测度集中现象（concentration of measure），这种策略在保持极高准确率的同时，运行速度极快且极易实现。这篇论文提出了利用随机傅里叶特征采样（Random Fourier Features）来高效逼近高斯核，并提出了近似可分离多元位移不变核的随机装箱特征（Random Binning Features）。该工作随后延伸出了用于 Boosting 的随机桩（Random Stumps）以及用于经验核的随机支持向量，并在后续衍生出一系列被称为“随机厨房水槽”（Random Kitchen Sinks）的知名方法。最终主持人正式宣布获奖论文为 Ali Rahimi 与 Benjamin Recht 合作的《大规模核机器的随机特征》（*Random Features for Large-Scale Kernel Machines*），并邀请二人上台接受掌声，由 Ali Rahimi 负责发表演讲。 - 评奖依据：回顾 10 年前的 NIPS 论文（NIPS 2007） - 核心论文：《Random Features for Large-Scale Kernel Machines》（作者：Ali Rahimi 与 Benjamin Recht） - 关键概念：随机傅里叶特征（Random Fourier Features）、随机装箱特征（Random Binning Features）、随机厨房水槽（Random Kitchen Sinks） ### [03:06](https://www.youtube.com/watch?v=x7psGHgatGM&t=186s) 缘起 NIPS 2006：两封邮件催生的核方法加速方案 Ali Rahimi 登台后感谢了大家，随即打趣道：“获奖固然感觉很棒，但没有任何事情能比拿一个叫‘时间检验’的奖更让人深刻意识到自己老了，这逼得我不得不正视自己的年龄，本和我都不再年轻了。”他透露，这篇获奖论文其实是一个“三部曲”的开端，而真正精彩的内容往往发生在中间或结尾。他把现场听众带回 2006 年的 NIPS——一个“恐龙尚在地球上漫游、他和本还是精力充沛的年轻小伙”的时代。当时深度学习刚刚崭露头角，训练算法极其繁复，其效果也仅仅能与 PCA、线性支持向量机（Linear SVM）等线性方法打平。在研讨会交流时，有人指出深度学习应该与非线性方法（如非线性 SVM）进行对比，但当时的非线性核方法根本无法扩展处理那样规模的数据集。当时，Benjamin Recht 正在研究压缩感知（compressed sensing），而 Ali Rahimi 正在计算机视觉领域利用草图算法（sketches）加速二分图匹配。两人回到家后，仅仅通过两封电子邮件的沟通，就构想出了加速核机器的方法。 Ali 解释了核机器的常规机制：通常是在每个数据点上放置一个核函数并赋予相应权重，让优化器去调整这些权重以最小化训练误差。而他们的关键观察在于，当时人们使用的大多数核函数，都可以近似表示为随机函数乘积的直接求和。将该近似代入原公式后，复杂的核展开便简化为随机特征的线性组合，待求解的参数数量大幅减少，这使得当时的经典优化器能够轻松胜任。论文不仅给出了将主流核函数随机化的近似方案，还提供了理论保障，明确计算出在给定逼近精度下所需的随机特征数量。 ```mermaid flowchart TD accTitle: 加速核机器算法演进与简化流程 accDescr: 展示传统核机器与随机特征近似方法之间的对比机制 subgraph Traditional["传统核机器架构"] A["训练数据点"] --> B["在每个样本上放置核函数"] B --> C["为每个核分配待学习权重"] C --> D["优化器求解高维参数（极难扩展至大规模数据）"] end subgraph RandomFeatures["随机特征近似架构"] E["位移不变核函数"] --> F["近似为随机特征乘积的求和"] F --> G["转化为低维随机特征的线性组合"] G --> H["优化器仅需调整少量线性权重（快速且可扩展）"] end ``` - 时代背景：NIPS 2006 时深度学习刚崭露头角，仅与线性模型对比，迫切需要非线性基线 - 合作过程：二人各自研究随机化算法（压缩感知与二分图匹配草图），仅用两封电子邮件就确定了核心算法 - 论文贡献：提供常见核的随机近似方法与近似误差上界，将非线性问题降维为随机特征上的线性求解问题 ### [05:45](https://www.youtube.com/watch?v=x7psGHgatGM&t=345s) 地下传单、理论困惑与“NIPS 严谨性警察” 为了给深度学习提供可对比的非线性基线，他们原本打算直接与深度学习代码一较高下，但当时机器学习领域完全不像今天这样具备可复现性，根本找不到公开可用的代码。于是他们只能退而求其次，与当时能够复现的方法（如 Boosting 以及各种未经加速的标准核机器）进行对比。在 NIPS 的海报展示环节，他们甚至自制传单到处散发，就像搞街头“游击营销”一样推销算法：“嘿伙计，想试试核机器吗？这里只要四行 MATLAB 代码就行！” 然而，在看似完美的成果背后，却隐藏着一个让他们有些心虚的理论怪象。根据论文中严格推导出的数学界限，为了准确逼近核函数，理论上需要成千上万个随机特征；但在实际实验中，他们发现仅使用区区几百个特征就能取得绝佳效果。更令人费解的是，在部分实验里，这种近似方法在测试集上的表现，甚至比它试图逼近的原始核机器本身还要好。Ali 苦笑说，当你的近似物比被近似的目标表现更强时，事情就显得相当诡异了。 在那个年代，机器学习会议正在经历转变。Ali 提到：“那正是机器学习刚从 Samer 过去所说的‘想法交流会’转向更严谨学术会议的时期。”在海报展区，经常能看到一群被他戏称为“NIPS 严谨性警察”的学者到处巡视，抓着海报作者盘问细节以确保逻辑无懈可击。Ali 特别列举了这群令人生畏的学者名单：“我想到了 Nati Srebro、Ofer Dekel、Michael I. Jordan 的一些学生，要是你运气不好会撞上 Shai Ben-David，而如果你真的倒霉透顶，还会碰上 Manfred Warmuth。”尽管面临这种理论瑕疵被揪出的风险，他们最终还是硬着头皮提交了论文。 - 宣传策略：海报展会散发单页，附带“4 行 MATLAB 代码训练大规模核机器”的游击营销 - 理论与实验的脱节：理论界限要求数万个特征，实验中仅需数百个即见效；近似模型的泛化误差甚至反超原始核方法 - “NIPS 严谨性警察”名单：Nati Srebro、Ofer Dekel、Mike Jordan 的弟子们、Shai Ben-David、Manfred Warmuth ### [08:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=491s) 理论救赎：“随机厨房水槽”与基准思维 为了平息“严谨性警察”可能带来的审判并从根本上解释这一怪象，他们随后发表了第二篇和第三篇论文，彻底洗脱了“核逼近”的束缚。Ali 抛开复杂的核数学把式指出：该算法的本质极其质朴——无非是从数据无关的分布中独立抽取一组函数，为它们赋予权重，并调整这些权重以最小化训练误差。在第二篇论文中，他们证明了一个强有力的几何事实：正如傅里叶基构成 $L^2$ 函数空间的稠密基、三层宽神经网络可以任意逼近光滑函数一样，一组随机抽取的平滑函数也能以极大概率在希尔伯特空间中任意稠密逼近函数。这意味着随机特征本身就是一类合法且独立的学习基函数，完全不需要借由某种名核的特征函数来获得合法性。在第三篇论文中，他们终于给出了纯粹针对该算法本身的泛化误差界：直接指明达到特定测试误差所需的数据规模与随机特征数量。 这一理论跃迁让他们解开了心结。既然将各种“随机厨房水槽”（Random Kitchen Sinks）组合在一起本身就是一种合理的模型，他们便不再纠结于它是否忠实复刻了原始核机器。后来，随着学术界可复现性大幅提升，其他研究者在语音识别等任务上也成功证明了随机特征模型足以与深度学习相匹敌。Ali 坦言自己至今在实际工作中仍然频繁使用随机特征，并根据具体问题设计随机函数。当效果惊艳时，他会欣喜地感叹：“哇，随机特征太强大了，竟然攻克了这个数据集！”而心情平静时，他又会冷静地反思：“可能这个任务本来就很简单，连随机特征都能搞定。”他认为这就像对待 K 近邻（Nearest Neighbors）算法一样：它既是检验任务难易程度的试金石，也是极为扎实的诊断基线。 - 理论正名：摆脱核函数逼近的视角，直接证明随机平滑函数基在希尔伯特空间中的稠密逼近能力 - 泛化界建立：脱离核理论，直接推导出样本量、随机特征数与测试误差之间的独立泛化界 - 实用价值：与 K 近邻算法并列为工业界绝佳的 Baseline 和问题复杂度诊断工具 ### [11:08](https://www.youtube.com/watch?v=x7psGHgatGM&t=668s) 警惕自满：机器学习正在沦为“炼金术” 转眼到了 2017 年，整个机器学习领域发生了翻天覆地的巨变。开源分享代码和通用基准测试（benchmarks）让学术研究极具可复现性；工业应用也迎来了爆发式增长：自动驾驶看似近在咫尺，人工智能深入到照片打标、语音邮件转录、文档翻译、广告投放等方方面面，估值数十亿美元的独角兽企业拔地而起。 然而，在繁荣与技术飞跃的背后，Ali 觉察到了一种令人担忧的倾向。他在演讲中指出，当前领域内弥漫着一种盲目的自我陶醉与自夸，人们到处宣扬诸如“机器学习是新时代的电力”这样的宏大口号。对此，Ali 当场抛出了一个针锋相对且引发全场震动的反思性论断：“机器学习已经变成了炼金术。”（“Machine learning has become alchemy.”） Ali 随即阐明，他并非全盘否定炼金术的历史贡献。炼金术并非一无是处，它在人类历史上切实发挥过作用，炼金术士们发明了冶金术、纺织品染色法、现代玻璃制造工艺以及早期药物制剂。但与此同时，由于缺乏系统性的第一性原理解释，炼金术士也坚信水蛭能医治百病、普通金属可以嬗变为黄金。Ali 强调，在 18 世纪物理学与化学彻底颠覆人类对宇宙认知的科学革命发生之前，科学家们首先必须做的一件事，就是亲手拆解并扬弃长达两千年之久的炼金术经验积淀。 > “I'd like to offer another analogy: machine learning has become alchemy.” —— Ali Rahimi - 时代反差：技术爆发（自动驾驶、机器翻译、广告流、数十亿美金企业）与理论根基缺失并存 - 流行论调批判：反对“机器学习是新时代的电力”这种毫无批判精神的乐观论调 - 炼金术隐喻的双重性：既承认经验探索带来了冶金、染色、玻璃等实用成果，又警惕缺乏坚实理论指导下的荒谬行径（如水蛭治病、点石成金） ### [12:49](https://www.youtube.com/watch?v=x7psGHgatGM&t=769s) 告别“炼金术”：机器学习亟需严谨的科学基石 阿里·拉希米（Ali Rahimi）指出，如果大家仅仅是在构建一个照片分享系统，那么依赖“炼金术”式的经验理论或许尚可接受。然而，如今的机器学习早已越过这一阶段——它正在被用于构建管理医疗保健、调解公民对话甚至影响选举的核心系统。正因如此，拉希米表达了强烈的诉求：“我希望生活在一个系统建立在可验证、严谨、详尽知识之上，而非建立在炼金术之上的社会里。”（“I would like to live in a society whose systems are built on top of verifiable rigorous thorough knowledge and not on alchemy as aggravating as the nips Ruger police was I miss them and I wish they'd come back.”）过去 NIPS 社区那种对严谨性极致追求的“严格警察”（rigor police）固然让人头疼，但他坦言自己非常怀念他们，并真心希望那种对学术严谨的坚持能够回归。 为了证明缺乏严谨性带来的伤害，拉希米向台下抛出了一个极具共鸣的问题：有多少人曾经从零设计一个深度神经网络（包括架构在内的一切），从头开始训练，当它无法正常工作时，便感到自我怀疑，觉得是自己做错了什么？现场许多人举起了手。拉希米安慰道，这种情况大约每三个月就会发生在他自己身上一次，但他坚信这并非开发者的错，而是梯度下降（Gradient Descent）本身的问题。 - 机器学习已深入医疗健康、公共话语与政治选举等攸关社会运转的关键领域。 - 研发人员频繁遭遇网络训练失败并陷入自我否定，根源往往在于底层优化算法的缺陷而非个人能力。 --- ### [14:09](https://www.youtube.com/watch?v=x7psGHgatGM&t=849s) 玩具模型的警示：梯度下降在最简单的线性网络中受挫 拉希米用一个极度简化的两层线性网络作为反例来说明问题：该网络没有非线性激活函数（采用恒等激活函数 Identity Activation），输出标签是输入的病态条件线性函数（badly conditioned linear function）。模型本质上只是两个矩阵相乘，然而当绘制出梯度下降及其各种变体的损失曲线时，可以清晰地看到：梯度下降在刚开始时下降极快，随后便彻底停滞。这并非陷入了局部极小值（Local Minimum）或鞍点（Saddle Point），因为梯度的模长远未归零；这也不是达到了统计噪声底限（Statistical Noise Floor），因为哪怕计算损失的期望值，依然会得到完全一致的曲线，说明其停滞与随机性无关。 相比之下，若采用 Levenberg-Marquardt 等更优秀的优化方向算法，仅仅经过数百次迭代，损失就能迅速降至机器精度的零。拉希米呼吁现场观众不妨花 10 分钟在笔记本电脑上亲自测试这个案例。针对“这是一个刻意人为构造的病态矩阵”的潜在反驳，他澄清该问题仅仅是矩阵各列之间存在相关性；而针对“梯度下降在更大、更复杂的网络中表现很好”的论调，他反驳道：台下刚才举手的开发者们绝不会同意这种说法。更重要的是，人类科学知识的积累向来是从简单、易分析的系统入手，理解透彻后再逐步提升复杂度，而当下的深度学习领域却跳过了基础积累，直接跃升到了极度复杂的黑箱中。 ```mermaid flowchart TD accTitle: 梯度下降在简单线性网络中的优化失效对比 accDescr: 对比梯度下降在简单线性网络中陷入停滞与二阶算法迅速收敛至零误差的过程 subgraph ProblemSetup["简化测试场景"] A["两层线性网络 (恒等激活函数)"] --> B["矩阵乘积模型: 列相关/病态条件"] end subgraph OptimizationComparison["优化算法表现对比"] B --> C["梯度下降 (Gradient Descent)"] B --> D["Levenberg-Marquardt 算法"] C --> E["初期快速下降 -> 迅速停滞 (Peter out)<br/>- 梯度模长远非零 (非鞍点/非局部极值)<br/>- 期望损失一致 (非统计噪声)"] D --> F["数百次迭代迅速收敛<br/>- 达到机器精度零误差"] end ``` --- ### [16:11](https://www.youtube.com/watch?v=x7psGHgatGM&t=971s) 脆弱的技术栈：从 TensorFlow 四舍五入到批归一化黑箱 优化的脆弱性在现实工程中带来了巨大阵痛。拉希米念出了一个月前朋友鲍里斯（Boris）发来的一封邮件：在周五，另一个团队将 TensorFlow 内部的默认舍入模式从“向零截断”（truncate toward zero）改为了“舍入到偶数”（round to even），导致其模型的训练彻底崩溃，错误率直接从 25% 飙升至近 99%。类似的求助邮件在拉希米的收件箱里还有数封，公开互联网的错误追踪平台上也屡见不鲜。拉希米尖锐地指出，这种乱象的原因在于“我们将脆弱的优化技术应用在我们根本不理解的损失曲面上，而我们的解决方案竟是向一个本就充满神秘的技术栈中增添更多神秘感。”（“This is happening because we apply brittle optimization techniques to lost surfaces we don't understand in our solution is to add more mystery to an already mysterious technical stack.”） 批归一化（Batch Normalization, BN）便是这种“以谜解谜”的典型代表。人们将 BN 插入深层网络之间，梯度下降随即大幅加速。拉希米调侃称，自己并不排斥使用不懂的技术，就像自己坐飞机来到会场却不懂航空动力学一样，但人们之所以能安心坐飞机，是因为背后有整个成熟的航空工程学提供完备的理论支撑。反观机器学习界，关于 BN 为何起效的解释只有一句含糊的“减少了内部协变量偏移”（Internal Covariate Shift）。拉希米发出了一连串灵魂拷问： - 难道大家不想知道为什么减少内部协变量偏移能加速梯度下降吗？ - 难道大家不想看到支撑这一结论的理论定理或验证实验吗？ - 难道大家不想看到证据证明批归一化确实减少了内部协变量偏移吗？ - 难道大家甚至不想先弄清楚什么是内部协变量偏移，看一眼它的数学定义吗？ 批归一化已成为深度学习的基础构件，但整个领域对其内在机制几乎一无所知。 --- ### [18:29](https://www.youtube.com/watch?v=x7psGHgatGM&t=1109s) 走向负责任的科学：从追逐榜单到根因探究与底层算力突破 面对机器学习在社会中承担的重大责任，拉希米提出了两项具体的改进建议： | 现有研发模式 | 倡导的科学研究范式 | |:--- |:--- | | **打榜驱动（Leaderboard Jockeying）**：为了在测试集排行榜上获得微弱优势，盲目尝试各种调参技巧与拼接模块。 | **根因探究（Root Cause Chasing）**：设计严谨的对照实验，追查所观察到的异常现象的底层物理/数学原因。 | | **单一依赖梯度下降**：将数以百亿计的变量优化全盘寄托在商用硬件支持的各种梯度下降变体上。 | **拓展底层数学求解器**：研发能处理百亿级变量的线性系统求解器（Linear System Solvers）与矩阵分解引擎。 | 拉希米指出，简单的实验与简单的定理是理解更复杂系统的基石。如果我们能够在通用硬件上实现大规模的线性求解器或矩阵分解工具，必将孕育出不可思议的全新优化算法与模型范式。尽管这是艰深的数学与系统工程难题，但他坚信 NIPS 社区的智慧完全有能力攻克它们。 在演讲尾声，拉希米动情地回顾了自己在该社区结下的深厚友谊，强调自己正是出于对同仁真诚的尊重与爱戴，才站在这里呼吁大家更加严谨、摒弃炼金术。他代表共同获奖者本·雷克特（Ben Recht）感谢组委会授予他们时间检验奖（Test of Time Award），并在全场雷鸣般的掌声中发出了最后的号召：“我们非常希望能够与大家携手合作，将机器学习从炼金术时代推向电学时代。”（“Ben and I are grateful for the award and the opportunity to have gotten to know many of you and to work with many of you and we would love it if we could work together to take machine learning from alchemy and into electricity.”） --- ### [21:27](https://www.youtube.com/watch?v=x7psGHgatGM&t=1287s) 颁奖与致谢：期待严谨精神的全面回归 演讲结束后，主持人登台对拉希米表达了由衷的感谢，并半开玩笑地送上寄语：“希望十年后我们能因为严谨性在 NIPS 社区的回归，再次为你颁发一座新的奖项。”（“I hope that we'll be able to give you a new award in 10 years for a net from now for the return of the Ricker in the next community.”） 主持人表示，全场观众都被拉希米这段发人深省、切中时弊的演讲彻底震撼且深表赞同。随后，主持人邀请论文的另一位作者本·雷克特（Ben Recht）一同走上讲台，正式为两位学者颁发了 NIPS 2017 时间检验奖的官方证书。全场再次爆发出热烈的掌声，时间检验奖颁奖环节就此圆满落幕，会议进入中场休息。 ## Notable quotes > [00:07](https://www.youtube.com/watch?v=x7psGHgatGM&t=7s) 传统的学习算法太辛苦了，因为它们去优化了那些根本不需要优化的参数；相反，人们可以对其中一部分参数进行随机化，然后迅速对剩余参数进行优化 — Ali Rahimi ## Mentioned Ali Rahimi, Benjamin Recht, NIPS, Random Features for Large-Scale Kernel Machines, 随机傅里叶特征 (Random Fourier Features), 随机装箱特征 (Random Binning Features), 随机厨房水槽 (Random Kitchen Sinks), PCA, 线性支持向量机 (Linear SVM), Boosting, MATLAB, Nati Srebro, Ofer Dekel, Michael I. Jordan, Shai Ben-David, Manfred Warmuth, K近邻 (Nearest Neighbors), 梯度下降 (Gradient Descent), Levenberg-Marquardt 算法, TensorFlow, 批归一化 (Batch Normalization) [Source](https://www.youtube.com/watch?v=x7psGHgatGM)

Source quotes

[0:07] Ali Rahimi: 传统的学习算法太辛苦了，因为它们去优化了那些根本不需要优化的参数；相反，人们可以对其中一部分参数进行随机化，然后迅速对剩余参数进行优化

Source quotes

[11:08] Ali Rahimi: Machine learning has become alchemy. [16:11] Ali Rahimi: Batch norm has become a foundational tool in how we build deep nets, and yet as a field, we know almost nothing about it. [12:49] Ali Rahimi: This is how we build knowledge: we apply our tools on simple, easy-to-analyze setups, we learn, and we work our way up in complexity; we seem to have just jumped our way up.

Original transcript

[0:07] good morning I think we are ready for [0:10] the next talk if this this actually [0:13] works so not every nips has a test of [0:21] time award so it's a test of time even [0:24] to get a test a time award this year has [0:28] one and the criteria that we decided the [0:33] program decided on for for awarding it [0:37] was ten year back nips papers so we [0:42] formed a little committee and I must say [0:44] that it was not easy all the ones that [0:48] did not get their test of time award [0:50] don't feel too bad this was not an easy [0:53] decision but we did arrive we did arrive [0:56] at a paper that we definitely felt was [0:59] was service really really good and we're [1:02] happy about it so just let me introduce [1:07] it in the author's own word random [1:10] features I trick to speed up supervised [1:13] learning algorithms so they can operate [1:16] on big that as millions of data points [1:19] data says traditional learning [1:22] algorithms work too hard because they [1:25] optimized parameters that don't actually [1:28] need to be optimized instead one can [1:32] randomize over some of the parameters [1:34] and quickly optimize over the rest [1:36] thanks to the concentration of measure [1:39] of phenomenon this still provides [1:41] excellent accuracy this recipe makes for [1:44] accurate learning algorithms but one [1:46] extremely fast and are very easy to [1:49] implement so the paper introduced random [1:53] Fourier features sampling to efficiently [1:55] approximate Gaussian kernels it also [1:57] introduced random painting features to [1:59] approximate separate separate separable [2:02] multivariate shift invariant kernels [2:04] later they followed up with random [2:07] stamps for boosting and random support [2:10] vectors for empirical kernel [2:12] and from the later variations of this [2:14] paper follow a very popular term the [2:17] random kitchen sink so without any [2:20] further ado in case you shouldn't have [2:22] guessed what paper it is it has random [2:24] features for life scale kerning which [2:26] kernel machines bialya Rahimi a been [2:29] wrecked and they are here both of them [2:33] let's give them a hand [2:39] and Ali will be giving the talk thank [2:43] you would you mind oh thank you [3:02] okay [3:06] it feels great to get an award thank you [3:09] but I got to tell you nothing makes you [3:11] feel all about getting an award award [3:12] called test of times it's forcing me to [3:16] come to grips with my age Ben and I are [3:18] no longer young so we decided to name [3:21] the talk accordingly we're getting this [3:26] award for this paper this first paper up [3:28] here but this paper was the beginning of [3:31] a trilogy of sorts and like all stories [3:34] worth telling the good stuff happens in [3:37] the middle or at the end not at the [3:39] beginning so if you'll put up with my [3:41] old man ways I'd like to take you all [3:43] the way back to nips 2006 when dinosaurs [3:47] roamed the earth and Ben and I were [3:50] young spry men deep learning had just [3:56] made a had just made a splash in Mitch [3:58] 2006 the training algorithms were [4:01] complicated and the results were [4:03] competitive with linear methods like PCA [4:05] and linear SPMS at the workshop some of [4:09] us were kibitzing and somebody pointed [4:10] out that this should maybe be compared [4:12] against long nonlinear methods like [4:14] SVM's but of course at the time you [4:17] couldn't scale us v ends up to that size [4:19] David the data set Ben and I had both [4:23] been working on randomized algorithms [4:25] separately been for compressed sensing [4:27] and I for sketches to speed up bipartite [4:30] graph matching for computer vision once [4:33] we got home it took us just two emails [4:35] to come up with a way to speed up kernel [4:38] machines these two emails became that [4:41] first paper the idea was simple normally [4:46] when you train the kernel machine on a [4:48] data set you slap one kernel on top of [4:51] each data point and an associated [4:52] weights to each one of those kernels and [4:54] you let your optimizers tune those [4:56] weights to minimize the lots that are [4:58] training there here's the here's the [5:01] observation in the paper [5:02] most of the kernels people were using at [5:05] the time can be approximated as a [5:06] straight sum of the product random [5:08] functions if you plug in that [5:11] approximation in that first equation you [5:13] get a linear combination of a straight [5:16] sum which is just [5:18] linear combination but one with fewer [5:20] coefficients to sell for and that's [5:22] something the optimizes at the time [5:23] could handle the paper provided ways to [5:27] approximate popular kernels as randomly [5:31] like this and also provided guarantees [5:33] about the quality of the approximation [5:34] if you have a kernel anyone are [5:36] approximated with such and such fidelity [5:39] here's how many random features you need [5:41] and in practice the method worked very [5:45] well now we'd set out to provide a [5:49] baseline for deep learning so they could [5:51] compare against nonlinear methods but we [5:54] can find any code to compare against [5:56] this was before machine learning was [5:58] reproducible the way it is now so we [6:01] compared against what was reproducible [6:02] at the time which was boosting and non [6:04] an accelerated kernel machines of [6:06] various cons during our presentation at [6:10] the the poster session we handed out [6:13] this leaflet to showcase how easy it was [6:15] to train large-scale kernel machines [6:17] it's uh hey buddy you wanna you want to [6:21] try to crow a machine here's just four [6:23] lines of MATLAB code a little bit of a [6:25] guerilla marketing for for an algorithm [6:30] now there's something a little shady [6:33] about the story up and telling you [6:37] according to the bound I just showed you [6:39] and that bound is a fine bound in order [6:42] to approximate a kernel accurately with [6:44] these random features you need tens of [6:47] thousands of random features and yet in [6:50] our experiments we were getting away [6:52] with using just a few hundred features [6:54] and getting good results even more [6:57] strangely in some of our experiments our [7:00] approximate method was producing lower [7:04] test errors than the original kernel [7:07] machine we were trying to approximate [7:10] it's always weird when the thing you're [7:13] trying to approximate does worse than [7:15] your approximation this is relevant [7:18] because back in those days machine [7:21] learning had just finished transitioning [7:23] from what Samer always used to call an [7:25] ideas conference into something more [7:29] rigorous during the poster sessions you [7:33] could see this roving band of what I [7:35] used to call the nips rigor police they [7:38] would come to your poster and make sure [7:40] that your arguments were airtight I'm [7:41] thinking of of naughty cerebro Oh for [7:46] deckle some of Mike Jordan students if [7:50] you're unlucky shy been dahveed and if [7:54] you're really unlucky [7:56] Manfred warm booth but anyway we decided [8:01] to send the paper out as is with this da [8:04] genus in it and brave than the nips [8:07] rigor police but to do right by them we [8:11] eventually came up with an explanation [8:13] for this phenomenon and I'll share it [8:15] with you here's the algorithm without [8:18] any of the kernel flim-flam straight up [8:21] you just draw a bunch of functions [8:24] independently from your data set you [8:26] weight them and you tune those weights [8:29] so that you get a low loss on your [8:31] training data sets in the second paper [8:35] we showed this in the same way that [8:41] Fourier basis provide a dense base basis [8:45] set for an l2 ball of l2 functions or in [8:51] the same way that three layer wide [8:53] neural networks could approximate any [8:55] smooth function arbitrarily well so to [8:59] do a bunch of randomly drawn smooth [9:02] functions approximate densely I hope [9:05] function in a Hilbert space arbitrarily [9:09] well with high probability so now you [9:14] don't need to talk about kernels to [9:15] justify these random features they don't [9:17] have to be eigenfunctions of any famous [9:19] kernels they're a legitimate basis for [9:21] learning into themselves in the third [9:24] paper we finally derive generalization [9:26] bounds for the algorithm I just showed [9:28] you if you have a data set with this [9:30] many points and you want to achieve this [9:32] kind of test there on the data set [9:33] here's how many random futures you [9:38] now at the time but by this point we [9:43] were now just no longer thinking about [9:45] kernels we legitimize using Rahab taking [9:49] a bunch of random kitchen sinks and [9:50] combining them together and you don't [9:53] need to justify the approximate kernel [9:55] isn't it anyway so it didn't really [9:56] bother us if we were underperforming or [9:58] over performing Colonel machines or if [10:00] we had to use fewer parameters or more [10:02] parameters than a caramel machine we'd [10:05] set out on this journey to provide a [10:07] baseline for deep learning and we can do [10:10] it at the time but since then the field [10:12] has become much more reproducible and [10:14] various folks have provided baselines in [10:17] speech where random features are [10:20] competitive with with deep learning to [10:22] this day I myself still use random [10:26] features at work I'd like to get [10:28] creative with the random functions I use [10:31] and adapt them to the problem at hand [10:33] when they work well and I'm feeling good [10:35] I say to myself Wow random features is [10:38] so powerful they cracked this dataset or [10:40] if I'm in a more somber mood I might say [10:43] huh [10:43] this poem wasn't hard at all even random [10:46] features cracked it it's the same way I [10:48] think about nearest neighbors [10:49] when nearest neighbors does well you can [10:51] either marvel about the power of nearest [10:52] neighbors or you might conclude that [10:54] your problem wasn't hard to begin with [10:55] both of these algorithms are good solid [10:58] baselines and a way to get a diagnostic [11:00] on your problem it's not 2017 and the [11:08] field has changed we've made an [11:11] incredible progress we are now [11:14] reproducible we shared code freely and [11:17] used common tasks benchmarks we've also [11:20] made incredible technological progress [11:23] self-driving cars seem to be around the [11:25] corner [11:26] artificial intelligence tags photoes [11:28] transcribes voicemails translates [11:31] documents serves us ads billion-dollar [11:34] companies are built on top of machine [11:36] learning in many ways we're way better [11:40] off than we were 10 years ago and in [11:43] some ways we're worse off there's [11:49] self-congratulatory feeling in the air [11:52] we say things like machine learning is [11:55] the new electricity I'd like to offer [12:00] another analogy machine learning has [12:04] become alchemy now alchemy is okay [12:12] alchemy is not bad there is a place for [12:16] alchemy alchemy worked alchemists [12:21] invented metallurgy wasted died textiles [12:23] are modern glass making processes and [12:26] medications then again alchemists also [12:30] believed that could cure diseases with [12:31] leeches and transmitted base metals into [12:33] gold for the physics and chemistry of [12:38] the 1700s to usher in the sea change in [12:42] our understanding of the universe that [12:43] we now experienced scientists had to [12:46] dismantle 2,000 years worth of [12:49] alchemical theories if you're building [12:53] photo-sharing systems [12:56] alchemy is okay but we're beyond that [12:59] now we're building systems that govern [13:02] health care and mediate our civic [13:05] dialogue we influence elections I would [13:10] like to live in a society whose systems [13:12] are built on top of verifiable rigorous [13:15] thorough knowledge and not on alchemy as [13:21] aggravating as the nips Ruger police was [13:23] I miss them and I wish they'd come back [13:29] I'll give you examples of where this [13:31] hurts us [13:36] how many of you have devised the deep [13:40] net from scratch architecture and all [13:43] and trained it from the ground up and [13:45] when it didn't work you felt bad about [13:49] yourself like you did something wrong [13:51] please raise your hand this happens to [13:56] me about every three months and let me [14:00] tell you I don't think it's you I don't [14:03] think it's your fault [14:04] I think it's gradient descents fault [14:09] illustrate I'm gonna take the simplest [14:13] deep net I can think of it's a two layer [14:16] linear net so identity activation [14:19] functions and the laborer labels are [14:21] badly conditioned linear function of the [14:23] inputs on the left is my model it's a [14:26] product of two matrices that get taller [14:28] and they just multiply the input and on [14:30] the right as my loss function and this [14:32] is the progress of gradient descent and [14:34] of various variants of it it looks like [14:38] grading the Sentinels really fast at the [14:40] beginning and then it just seems to [14:41] Peter out you might say this is a local [14:44] minimum or a saddle point that's not a [14:46] local minimum it's not a cell point the [14:48] gradient magnitudes are nowhere near [14:49] zero you might say it's hitting some [14:52] statistical noise floor of the problem [14:54] that's not it either this is not a [14:56] statistical noise floor I can compute [14:57] the expectation of The Lofts run getting [15:00] consent and get the same curve this is [15:02] not due to randomness here's what a [15:04] better descent direction would do this [15:06] is Levenberg Marcotte it gets down to [15:07] zero loss it means males the solution a [15:10] few hundred iterations and it gets the [15:12] Machine precision zero if you haven't [15:16] tried optimizing this problem please [15:17] take 10 minutes on your laptop and just [15:19] try it this is our workhorse this is [15:21] what we're building models around you [15:25] might say this is a contrived problem [15:26] that badly conditions something [15:30] something just the the columns of this a [15:32] matrix are are correlated that's the [15:34] only weird thing about it this is not a [15:36] contrived problem you might say gradient [15:39] descent works just fine on more [15:41] complicated larger networks two answers [15:45] first everybody you just raise their [15:47] hands would probably disagree with [15:49] and second this is how we build [15:54] knowledge we apply our tools on simple [15:57] easy to analyze setups we learn and we [16:01] work our way up in complexity we seem to [16:06] have just jumped our way up this pain is [16:11] real here's an email that landed in my [16:14] inbox just a month ago I'll read it out [16:18] loud for you it's for my friend Boris on [16:22] Friday someone and another team changed [16:25] the default rounding mode of some [16:26] tensorflow internals from truncate [16:28] toward zero to round to even our [16:30] training broke our error rate went from [16:32] less than 25% error it's almost 99 [16:35] percent error I have several emails like [16:38] this in my inbox and you can find [16:40] similar reports on various boards [16:43] embarrassed bug reports on the public [16:45] internet this is happening because we [16:51] apply brittle optimization techniques to [16:55] lost surfaces we don't understand in our [16:59] solution is to add more mystery to an [17:03] already mysterious technical stack batch [17:09] norm is a way to speed up gradient [17:11] descent you insert batch norm between [17:13] the layers of your deep net and gradient [17:15] descent goes faster now I'm okay using [17:19] technologies I don't understand I got [17:22] here on an airplane and I don't fully [17:23] understand how our planes work but I [17:27] take comfort knowing that there is an [17:28] entire field of aeronautics dedicated to [17:32] creating that understanding here is what [17:36] we know about batch norm as a feel it [17:39] works because it reduces internal [17:42] covariance shift wouldn't you like to [17:45] know why reducing internal covariant [17:49] shift speeds up gradient descent [17:50] wouldn't you like to see a theater or an [17:53] experiment wouldn't you like to know [17:55] wouldn't you like to see evidence that [17:57] batch norm reduces internal covariant [17:59] shift [18:02] wouldn't you like to know what internal [18:04] covariant shift is wouldn't you like to [18:08] see a definition of it batch enorm has [18:14] become a foundational tool in how we [18:15] build deep Nets and yet as a field we [18:18] know almost nothing about it [18:29] machine learning has taken a new spot in [18:31] society if any of what I've been saying [18:34] resonates with you let me offer just two [18:38] ways we can assume this new position [18:40] responsibly think about in the past year [18:47] the experiments you ran where you were [18:51] jockeying for position on the leader [18:52] board and trying out different [18:53] techniques to see if they could give you [18:54] an edge in performance and now think [19:00] about in the past year the experiments [19:03] you ran where you were chasing down an [19:06] explanation for a strange phenomenon yet [19:08] observed you were trying to find the [19:10] root cause for something weird we do a [19:15] lot of the former kind of experiments we [19:19] could use a lot more of the ladder [19:24] simple experiments simple theorems are [19:27] the building blocks that help us [19:30] understand more complicated systems [19:33] here's a second second thing we could do [19:36] right now are mature computational [19:42] engines that run on commodity hardware [19:45] are all variants of gradient descent [19:46] that's what we have that can handle tens [19:49] of billions of variables imagine if we [19:53] had linear system solvers or matrix [19:56] factorization engines that could handle [19:57] tens of billions of variables and [19:59] operate on commodity hardware imagine [20:01] the amazing kinds of optimization [20:03] algorithms we could build imagine the [20:05] incredible models we could experiment [20:07] with of course it's a hard math problem [20:11] and it's a hard systems problem but this [20:14] is exactly the group that can solve [20:16] these problems [20:25] over the years many of my dearest [20:28] relationships that emerge from this [20:30] community my affection and respect for [20:33] this group are sincere and that's why [20:37] I'm up here asking us to be more [20:40] rigorous less alchemical better Ben and [20:45] I are grateful for the award and the [20:48] opportunity to have gotten to know many [20:49] of you and to work with many of you and [20:51] we would love it if we could work [20:54] together to take machine learning from [20:58] alchemy and into electricity [21:02] [Applause] [21:27] I thank you so much for this talk I hope [21:31] that we'll be able to give you a new [21:33] award in 10 years for a net from now for [21:36] the return of the Ricker in the next [21:39] community we have time for probably one [21:42] question or two since we are since since [21:57] we all just stunned by by them message [22:02] here which we all appreciated so much I [22:06] just want to invite been up here as well [22:08] so I can hand you the official [22:12] certificate for the test of time award [22:17] [Applause] [22:34] here by it is the end of the test of [22:37] time award and the program continues on [22:39] I think there's a break now so we