Pinterest's Rauser Replaces Statistical Formulas With Code Loops

O'Reilly

Software engineers can bypass opaque analytical mathematics by using basic loops and random permutations to simulate sampling distributions directly.

AI-assisted summary of John Rauser keynote: "Statistics Without the Agonizing Pain" -- Strata + Hadoop 2014 — original source. Check the original for full context.

Source published: 2014-10-17

Key ideas

John Rauser argues that software engineers already possess the tools for deep statistical fluency, as programming allows intuitive simulation instead of relying on opaque analytical math. (0:03)
Classical analytical statistics, exemplified by Welch's t-test, relies on complex formalisms like degrees of freedom formulas and lookup tables that obscure how sampling distributions actually work. (4:57)
A computational random permutation test resolves an empirical problem by iteratively shuffling labels across 50,000 runs, demonstrating that an observed difference occurred only 14 times under the null hypothesis. (8:00)
Resampling methods such as permutation tests and bootstrapping provide practitioners with a sandbox to experiment with fundamental statistical ideas through code rather than analytical equations. (10:26)

Detailed notes

## Background Published by O'Reilly on YouTube on October 17, 2014, this video captures John Rauser's keynote at the Strata + Hadoop World conference in New York City. The presentation addresses the impostor syndrome experienced by data scientists with engineering backgrounds, arguing that programming skills provide a direct, intuitive pathway to mastering core statistical concepts.

## TL;DR John Rauser argues that engineers intimidated by formal statistics already possess the tools needed for deep statistical intuition through computer programming. Using a real-world study on whether drinking beer makes humans more attractive to mosquitoes, he contrasts the opaque analytical route of Welch's t-tests and lookup tables with an intuitive computational permutation test. By shuffling labels across 50,000 iterations to build an empirical sampling distribution, Rauser demonstrates that statistical significance can be evaluated using only logic, a random number generator, and a loop. He concludes by encouraging engineers to abandon mathematical self-doubt and use code to tinker playfully with statistical problems.

## Atmosphere & style The presentation is energetic, empathetic, and irreverent toward classical statistical pedagogy. Rauser connects immediately with the audience by validating their shared frustration with traditional formulas, degrees-of-freedom jargon, and textbook memorization. Moving from comic relief about beer-drinking mosquito targets into a clean, compelling computational demonstration, the talk builds an empowering emotional arc that transforms intimidation and impostor syndrome into playful intellectual confidence.

## Bulletin - [00:03](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=3s) John Rauser argues that software engineers already possess the tools for deep statistical fluency, as programming allows intuitive simulation instead of relying on opaque analytical math. - [04:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=297s) Classical analytical statistics, exemplified by Welch's t-test, relies on complex formalisms like degrees of freedom formulas and lookup tables that obscure how sampling distributions actually work. - [08:00](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=480s) A computational random permutation test resolves an empirical problem by iteratively shuffling labels across 50,000 runs, demonstrating that an observed difference occurred only 14 times under the null hypothesis. - [10:26](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=626s) Resampling methods such as permutation tests and bootstrapping provide practitioners with a sandbox to experiment with fundamental statistical ideas through code rather than analytical equations.

## Key takeaways - [00:03](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=3s) Introductory statistics textbooks often present a disconnected grab bag of recipes rather than explaining the intuitive core ideas behind statistical tools. - [01:05](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=65s) Many software engineers experience impostor syndrome when statisticians introduce formal terminology such as power analysis, generalized linear models, or two-tailed tests. - [02:00](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=120s) Rauser's central thesis is that the ability to program a computer grants direct access to the deepest and most fundamental concepts in statistics. - [02:27](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=147s) The core purpose of statistics is to arbitrate disputes between a skeptic who attributes differences to chance and an advocate who sees a genuine effect. - [02:55](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=175s) The mosquito experiment evaluated whether drinking beer increases mosquito attractiveness compared to drinking water, finding an average difference of 4.4 mosquitoes per person across 43 subjects. - [04:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=297s) The classical analytical approach requires Welch's t-test, calculating a test statistic of t = 3.67, degrees of freedom around 39.1, and comparing against a critical value table cutoff of 2.021. - [07:00](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=420s) Traditional Stats 101 curricula cause agonizing pain because the crucial concept of a sampling distribution is obscured behind gamma functions and impenetrable definitions of degrees of freedom. - [07:35](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=455s) Even experienced professional data scientists often cannot derive analytical sampling distribution formulas from first principles. - [08:00](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=480s) A computational permutation test simulates the skeptic's hypothesis by pooling the data, randomly shuffling group labels, and recalculating the difference in group means thousands of times. - [09:30](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=570s) In 50,000 random permutation shuffles, an outcome as extreme as 4.4 mosquitoes occurred only 14 times, providing clear empirical justification to reject the skeptic's hypothesis. - [10:00](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=600s) Conducting valid statistical inference through simulation requires only native human logic, a random number generator, and a programming loop. - [10:26](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=626s) The permutation test belongs to a broader and powerful family of resampling methods that also includes bootstrapping. - [11:05](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=665s) Programming functions as a superpower for statistical learning, allowing engineers to tinker with statistical concepts in the spirit of play rather than fear and self-doubt.

## Detailed walkthrough ### [00:03](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=3s) Impostor Syndrome and the Programmer's Shortcut to Statistics

John Rauser, a data scientist at Pinterest, opens his keynote by adapting the first paragraph of an unnamed academic paper that perfectly captures his own frustration with learning statistics. When he initially tried to study the discipline, he read several textbooks that simply presented statistical recipes and procedures without providing any intuitive explanations or hints about how anyone could have invented them. Rauser explains that this talk was born out of his desire for students to encounter the deep, elegant core concepts of statistics rather than being handed a confusing grab bag of disconnected tools.

Addressing the audience at Strata + Hadoop World, Rauser observes that the crowd consists primarily of two groups: software engineers and statisticians, with engineers vastly outnumbering the statisticians. In practice, almost all the statisticians present self-identify as data scientists, while many engineers privately feel like impostors when statistical jargon enters the conversation. Rauser describes how engineers tend to nod along politely when someone starts throwing around terms like "power analysis," "generalized linear models," or "two-tailed tests," despite having no actual idea what is being discussed. Having spent about a decade teaching himself statistics as a software engineer, Rauser insists that engineers already hold the key to statistical fluency.

> “My thesis is that if you can program a computer, you have direct access to the deepest and most fundamental ideas in statistics.” — John Rauser

---

[02:27](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=147s) The Mosquito Experiment: Beer vs. Water

To test his thesis, Rauser introduces a concrete empirical problem: determining whether drinking beer makes humans more attractive to mosquitoes. While the premise elicits laughter, Rauser emphasizes that it is a serious global health question. Malaria is transmitted by mosquitoes, and epidemiological models historically assumed uniform risk of mosquito bites across individuals. However, evidence shows that humans differ dramatically in their attractiveness to mosquitoes; identifying high-risk behaviors or traits allows for targeted interventions to combat malaria more effectively.

Rauser cites a study published in *PLOS ONE* where researchers tested this dynamic using a specialized dual-choice apparatus called a Y-tube olfactometer, which allowed mosquitoes to fly either toward a volunteer or toward open air. The researchers recruited human volunteers and randomly assigned them to drink either beer or water before trapping and counting the attracted mosquitoes.

- **Sample sizes:** 25 volunteers drank beer; 18 volunteers drank water. - **Trap results:** The beer-drinking group attracted an average of 4.4 more mosquitoes per person than the water-drinking group. - **The Skeptic's view:** Drinking beer has no effect; the 4.4 difference is merely an artifact of random chance and noise. - **The Advocate's view:** A difference of 4.4 mosquitoes is substantial relative to the overall sample variation, making the skeptic's chance hypothesis highly implausible.

Rauser states that the primary purpose of statistics is to arbitrate precisely this kind of dispute between the skeptic and the advocate. To illustrate the two paths available, he promises to solve the problem twice: first through the painful analytical approach taught in standard introductory classes, and second through an intuitive computational method.

---

[04:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=297s) The Analytical Route: "Bring on the Pain"

Rauser dives into the classical analytical approach to test whether 4.4 is statistically significant, invoking the dreaded Stats 101 curriculum. Looking up standard procedures on Wikipedia, one finds the two-sample T-test. Choosing Welch's t-test for unequal variances, the user plugs in the mosquito counts and calculates a test statistic value of $t = 3.67$.

```mermaid flowchart TD accTitle: Analytical T-Test Workflow accDescr: The multi-step analytical process of running Welch's t-test using mathematical formulas and critical value lookup tables.

subgraph Computation["Test Statistic & Degrees of Freedom"] A["Observed Data (Beer vs. Water counts)"] --> B["Calculate Welch's t-statistic (t = 3.67)"] A --> C["Estimate Degrees of Freedom formula (df = 39.1)"] end

subgraph Evaluation["Inference & Rejection"] C --> D["Look up critical value in t-table (t_crit at 95% = 2.021)"] B --> E{"Compare t with t_crit"} D --> E E -->|"3.67 > 2.021"| F["Reject Null Hypothesis (p < 0.05)"] end ```

Rauser walks through the bewildering mathematical layers required to justify this test: - **Null distribution formula:** Wikipedia states that under the skeptic's assumption, $t$ follows a distribution defined by an intimidating equation containing the gamma function ($\Gamma$) and degrees of freedom ($\nu$). - **Degrees of freedom definition:** When researching the term, Wikipedia defines degrees of freedom as "the number of dimensions of the domain of a random vector"—a phrasing Rauser points out is entirely opaque to non-specialists. - **Welch–Satterthwaite equation:** Plugging the sample variances and sizes into the degree-of-freedom formula yields approximately $39.1$ degrees of freedom. - **Table lookup:** Taking that $39$ degrees of freedom to a printed critical value table, the critical cutoff for a 95% confidence level ($\alpha = 0.05$) with 40 degrees of freedom is $2.021$. - **Conclusion:** Because the observed $t$ of $3.67$ exceeds $2.021$, one triumphantly claims the result is statistically significant at the $p < 0.05$ level.

Rauser points out that despite following the cookbook steps, virtually nobody understands why the argument works. The heart of the argument relies entirely on the sampling distribution of the test statistic under the null hypothesis.

> “And the reason that Stats 101 was so incredibly painful is that the idea of a sampling distribution is really hard to understand even in the best conditions, and when it's presented in pure mathematical formalism like this, as a mathematical object all slathered up in degrees of freedom mumbo jumbo, it's just hopeless.” — John Rauser

He confesses that while working as a professional data scientist, he could not derive that analytical equation from first principles, and doubts more than a handful of people in the room could either.

---

[08:00](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=480s) The Computational Route: Shuffling and Resampling

Rauser pivots to the computational alternative: the permutation test. The central question remains whether a difference of 4.4 mosquitoes is unexpectedly large. He marks 4.4 on an empty axis, displays the original color-coded data points, and reasons from the skeptic's premise: if the skeptic is correct that beer has no effect, the "beer" and "water" labels carry no meaningful signal and can be treated as completely arbitrary.

Rauser demonstrates the algorithm step-by-step: 1. Pool all the mosquito counts together. 2. Randomly shuffle the labels ("beer" or "water") across the subjects. 3. Compute the mean of the new synthetic "beer" group and the new "water" group. 4. Calculate the difference between these two means (e.g., getting $3.3$, then $0.1$ on the next shuffle) and plot each outcome as a dot.

Repeating this shuffle iteratively—first 3, 5, 20, 50, and eventually 50,000 times—builds the sampling distribution under the skeptic's hypothesis right before the audience's eyes. In 50,000 randomized permutations, an outcome as extreme as 4.4 mosquitoes occurred only 14 times. Because an observed difference of 4.4 is extraordinarily rare under the assumption of pure chance, the skeptic's position strains credibility and is soundly rejected.

Rauser concludes that to perform this statistical inference, an engineer needs only three basic building blocks: - The ability to follow a straightforward logical argument (which humans have natively). - A random number generator. - A loop construct for iteration.

With those capabilities—standard in any modern programming language—engineers possess everything required to master fundamental statistical reasoning without the agonizing mathematical pain.

[10:26](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=626s) Resampling Methods and the Programmer's Superpower

John Rauser contrasts the intuitive clarity of computational simulation with the opaque mathematical machinery of classical analytical statistics. Referring to the simulation approach he just demonstrated, Rauser points out that anyone who writes code can understand the underlying argument at a deeply fundamental level. This stands in stark contrast to traditional analytical formulas: “the details of which you really need years of study I think to come to grips with.” He explains that this straightforward simulation technique is formally known as a random permutation test, which is merely one technique within a broader and exceptionally powerful category known as resampling methods. Rauser notes that if he had more time on stage, he would also dive into bootstrapping—another resampling method that he considers fantastically useful for working practitioners.

```mermaid flowchart TD accTitle: Resampling Methods Overview accDescr: Diagram showing how programming unlocks intuitive resampling methods like permutation tests and bootstrapping. subgraph Resampling["Resampling Methods"] Perm["Random Permutation Test"] Boot["Bootstrapping"] end Prog["Ability to Program"] --> Resampling Resampling --> Understanding["Deep, Intuitive Statistical Fluency"] ```

Lacking the time to walk through bootstrapping, Rauser pivots to his closing message, reformulating his core thesis into an empowering takeaway for software engineers and data scientists. He tells the room plainly: “The message that I want to leave you with is this: if you can program a computer, you have superpowers when it comes to learning statistics.” Writing code removes the barrier of impenetrable formal mathematics, allowing practitioners to tinker directly with core statistical principles much like taking apart gadgets or building projects as a child:

> “Because being able to program allows you to tinker with the most fundamental ideas in statistics the way that you might have tinkered with electronics when you were a kid or with mechanical things or with music or with sports.”

In his final call to arms, Rauser urges the audience to shed their impostor syndrome and embrace empirical experimentation. Rather than backing away from data science challenges due to formal mathematical intimidation, engineers should approach the discipline with confidence and curiosity: “And so I want you to go out and to attack statistical problems with a feeling of joy in the spirit of play and not from a position of fear and self-doubt.” He wraps up his keynote to warm applause from the Strata audience.

- **Statistical techniques referenced**: Random permutation test, resampling methods, and bootstrapping. - **Core takeaway**: Programming serves as an accessible sandbox for building genuine, foundational intuition in statistics without getting trapped in agonizing mathematical abstraction.

## Notable quotes > [00:03](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=3s) My thesis is that if you can program a computer, you have direct access to the deepest and most fundamental ideas in statistics. — John Rauser

> [10:26](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=626s) I want you to go out and to attack statistical problems with a feeling of joy in the spirit of play and not from a position of fear and self-doubt. — John Rauser

## Mentioned John Rauser, Pinterest, Strata + Hadoop World, O'Reilly Media, PLOS ONE, Amazon

## Video description From the 2014 Strata Conference + Hadoop World in New York City.

There are two essential skills for the data scientist: engineering and statistics. A great many data scientists are very strong engineers but feel like impostors when it comes to statistics. In this talk John will argue that the ability to program a computer gives you special access to the deepest and most fundamental ideas in statistics. John’s goal is to convince the non-statistician engineers in the audience that the road to statistical fluency is much, much shorter than they think.

About John Rauser: John has been extracting value from large datasets for over 20 years at hedge funds, small data-driven startups, Amazon, and now Pinterest. He has deep experience in machine learning, data visualization, on-line experimentation, website performance and real-time fault analysis. An empiricist at heart, “Just do the experiment!” is his favorite call to arms.

Watch more from Strata + Hadoop World 2014: http://goo.gl/UUfrR7 Find out more about the conference: http://strataconf.com/stratany2014

Don't miss an upload! Subscribe! http://goo.gl/szEauh Stay Connected to O'Reilly Media by Email - http://goo.gl/YZSWbO

Follow O'Reilly Media: http://plus.google.com/+oreillymedia https://www.facebook.com/OReilly https://twitter.com/OReillyMedia

[Source](https://www.youtube.com/watch?v=5Dnw46eC-0o)

中文简报

Pinterest的Rauser用代码循环替代统计公式软件工程师可以通过基础循环和随机置换直接模拟抽样分布，从而绕开晦涩的解析数学。 John Rauser认为，那些对形式统计学感到畏惧的工程师，其实已经通过计算机编程掌握了获得深刻统计直觉所需的工具。他以一项关于喝啤酒是否会让人们对蚊子更具吸引力的真实研究为例，将韦尔奇t检验及查表这一晦涩的解析路线，与直观的计算置换检验进行了对比。通过在50,000次迭代中打乱标签来构建经验抽样分布，Rauser证明了仅凭逻辑、随机数生成器和一个循环即可评估统计显著性。他最后鼓励工程师们放下对数学的自我怀疑，用代码以探索玩味的方式来解决统计问题。 John Rauser认为，软件工程师已经具备了深入掌握统计学的工具，因为编程允许进行直观的模拟，而无需依赖晦涩的解析数学。 (0:03) 以韦尔奇t检验为代表的经典解析统计学，依赖于自由度公式和查表等复杂的公理体系，这掩盖了抽样分布的实际运作机制。 (4:57) 计算随机置换检验通过在50,000次运行中反复打乱标签来解决经验问题，证明了在零假设下所观察到的差异仅出现了14次。 (8:00) 置换检验和自助法等重抽样方法为实践者提供了一个沙盒，使他们能够通过代码而非解析方程来对基础统计思想进行实验。 (10:26)

Reading summary

John Rauser argues that software engineers can bypass the agonizing mathematical formalism of classical statistics by using basic programming constructs and resampling simulations to master core statistical principles. Overcoming Impostor Syndrome in Statistical Pedagogy (0:03) Rauser notes that software engineers frequently experience impostor syndrome when confronted with statistical terminology such as "power analysis," "generalized linear models," or "two-tailed tests." Traditional statistics pedagogy often exacerbates this alienation by presenting disconnected formulas and recipes without providing an intuitive basis for how or why those methods were invented. Drawing on his decade-long journey of self-study as an engineer, Rauser contends that software practitioners already possess the fundamental capabilities required to understand statistics. He argues that formal mathematical presentations unnecessarily obscure foundational principles that can be accessed more directly through computational thinking. Framing the Dispute: Mosquitoes, Beer, and Chance (2:27) To ground the challenge of statistical inference, Rauser examines a PLOS ONE study investigating whether drinking beer increases human attractiveness to mosquitoes—a critical inquiry for targeting interventions against malaria. In the experiment, 25 volunteers drank beer while 18 drank water, tested via a dual-choice Y-tube olfactometer apparatus. On average, the beer-drinking group attracted 4.4 more mosquitoes per person than the water-drinking group. Rauser frames the primary task of statistics as resolving the dispute between two perspectives on this result: the skeptic, who claims the 4.4 difference is merely an artifact of random sampling noise, and the advocate, who argues the difference is too large relative to sample variation to have occurred by chance. The Analytical Route and Its Pedagogical Pitfalls (4:57) In the classical analytical framework, resolving the mosquito question involves running Welch's two-sample t-test for unequal variances. Calculating the t-statistic yields t = 3.67. Estimating the degrees of freedom using the Welch–Satterthwaite equation produces approximately 39.1. Cross-referencing this against a printed critical value table yields a cutoff of 2.021 at the 95% confidence level (alpha = 0.05), leading to a mechanical rejection of the null hypothesis. Rauser points out that despite executing these cookbook steps, practitioners rarely understand the underlying rationale. The mathematical derivation relies on complex null distributions involving gamma functions and definitions of degrees of freedom as dimensions of a random vector domain. Rauser asserts that burying the central concept—the sampling distribution under the null hypothesis—under mathematical formalism makes traditional statistics painful and nearly impossible for non-specialists to derive from first principles. Resampling and Permutation as a Computational Sandbox (8:00) As an alternative, Rauser demonstrates a computational permutation test. Grounded in the skeptic's premise that beer has no effect, the "beer" and "water" labels are treated as arbitrary. By pooling all observations, shuffling the labels, recalculating the difference between synthetic group means, and repeating this process 50,000 times, the sampling distribution under the null hypothesis is constructed visually. Across 50,000 randomized permutations, an outcome as extreme as 4.4 mosquitoes appeared only 14 times, decisively undermining the skeptic's chance hypothesis. Rauser emphasizes that this inference requires only native human logic, a random number generator, and a programmatic loop. Permutation tests and related resampling methods, such as bootstrapping, enable practitioners to tinker directly with core statistical mechanics rather than memorizing formulas. Rauser concludes that programming ability provides practitioners with an empowering shortcut to genuine statistical fluency, urging engineers to shed self-doubt and approach statistical inquiry with curiosity and experimental play.

精炼阅读摘要

Pinterest数据科学家John Rauser在Strata + Hadoop大会的主题演讲中主张，能够编写程序的人天然具备直接掌握统计学核心概念的能力；通过重采样等计算模拟手段，工程师无需深陷传统公式的泥潭，即可建立坚实的统计推断直觉。工程师的冒名顶替综合征与统计学认知困境 (0:03) Rauser观察到，在技术会议与工业界实践中，软件工程师的数量通常远多于统计学家，且许多工程师在面对统计术语（如功效分析、广义线性模型、双尾检验等）时容易产生冒名顶替综合征，倾向于不懂装懂。造成这一现象的核心原因并非工程师缺乏逻辑能力，而是标准统计学教学普遍采用“菜谱式”的流程灌输，充斥着脱离直觉的数学形式主义，却极少解释相关概念最初是如何被构想出来的。Rauser以自身耗费近十年自学统计学的经历指出，编程能力正是突破这一抽象壁垒的有效钥匙。啤酒与蚊子实验：推断统计的仲裁场景 (2:27) 为了具象化统计推断的本质，Rauser引用了一项发表在《PLOS ONE》上的真实流行病学研究。该实验利用Y型嗅觉仪测试饮用啤酒是否会增加人体对蚊子的吸引力，进而探讨针对疟疾传播的差异化干预可能。实验中25名受试者饮用啤酒，18名受试者饮用水，测试结果显示啤酒组平均每人多吸引4.4只蚊子。主张效应存在的支持者认为4.4的差值具有现实意义，而持怀疑论者则认为这纯粹是随机样本噪音造成的巧合。Rauser强调，统计学的首要目的正是为怀疑论者与支持者之间的此类分歧提供仲裁工具。传统解析方法的繁复公式与理解断层 (4:57) 按照大学入门统计学（Stats 101）的经典路线，研究人员会采用两样本Welch t检验来评估4.4的显著性。代入样本数据后，计算得出检验统计量t值为3.67。然而，要判定该结果在统计上是否成立，必须借助极为抽象的数学工具：依赖包含伽马函数的零分布公式、被称为“随机向量定义域维度”的自由度概念、通过Welch–Satterthwaite方程估算出约39.1的自由度，并最终翻阅临界值表，得出95%置信度下的临界值为2.021。Rauser指出，虽然3.67大于2.021足以拒绝原假设（p < 0.05），但整个过程极度依赖零假设下检验统计量的抽样分布，而这种被繁琐符号包裹的解析推导几乎无法给实践者带来直观理解，即便是专业数据科学家也难以从第一性原理对其进行推导。置换检验：通过代码直观重建抽样分布 (8:00) 针对解析推导的痛点，Rauser展示了基于计算模拟的随机置换检验（Permutation Test）。若怀疑论者的前提成立（即啤酒对吸引蚊子毫无作用），那么受试者身上的“啤酒”或“水”标签就仅仅是随机分配的无关符号。算法的实现过程极为清晰：将所有蚊子计数汇总，随机打乱标签分配，重新计算合成组的均值差异并记录点位。在将这一重采样过程重复50,000次后，计算机屏幕上直接显现出零假设下的抽样分布。结果显示，在50,000次随机置换中，差异达到或超过4.4只蚊子的情况仅出现了14次。由于在纯随机假设下出现该极值的概率极低，怀疑论者的观点被有力推翻。重采样思想与工程师的探索优势 (10:26) Rauser总结道，通过置换检验与Bootstrap等重采样方法进行统计推断，工程师只需具备三项基本要素：自然理解逻辑论点的能力、一个随机数生成器以及一段循环控制结构。与需要花费数年时间啃透的经典分析公式相比，现代编程语言内置的这些构件让代码编写者拥有了探索统计学的“超能力”。他指出，代码环境提供了一个可供拆解、调试和亲手操作的沙盒，使从业者能够像童年摆弄电子元件或机械构件一样理解数据机制，从而摆脱自我怀疑与恐惧，以积极玩味的心态解决统计难题。 Rauser展示了置换检验等重采样方法如何在无需繁复微积分推导的情况下实现严谨的统计推断。不过，该演讲聚焦于概念启蒙与双样本差异的直观检验，并未进一步展开探讨重采样方法在面对超大规模复杂数据集时的计算开销，亦未详述其在多元回归或更高级实验设计中的适用边界。

Detailed notes — zh

## Background 本视频由 O'Reilly 于 2014 年 10 月 17 日在 YouTube 平台发布，内容取自 2014 年纽约 Strata + Hadoop World 大会的主题演讲。视频简介指出，许多工程师在面对统计学时常有冒名顶替综合征，而曾任职于 Pinterest 的数据科学家 John Rauser 旨在通过本次演讲证明，编程能力能够为非统计学专业工程师提供直通统计学最核心思想的捷径。 ## TL;DR John Rauser 探讨了数据科学领域中软件工程师普遍面临的统计学焦虑与教学脱节问题。他以一项关于“喝啤酒是否更招蚊子”的真实科学研究为例，对比了传统统计教学中充斥着伽马函数、未知自由度与查表步骤的 Welch's t 检验，以及基于计算机模拟的随机置换检验（Permutation Test）。演讲展示了如何仅凭简单的逻辑论证、随机数生成与循环迭代，通过 50,000 次标签洗牌直观构建零假设下的经验抽样分布并推导出统计结论。Rauser 总结指出，重抽样方法使工程师能够绕开繁复的数理推导，呼吁技术人员利用编程这项“超能力”，以玩乐与探索的心态攻克统计学难题。 ## Atmosphere & style 整场演讲语调幽默风趣、富有感染力且直击行业痛点。John Rauser 以极具共情力的方式剖析工程师面对高深统计术语时的内心茫然，迅速拉近了与听众的距离。在对比枯燥的数理公式泥潭与清爽直观的代码模拟时，节奏张弛有度，现场互动与共鸣感极强。结尾处的呼吁真挚而鼓舞人心，彻底扫除了传统统计学带来的沉重与恐惧，带来一种豁然开朗的启发感。 ## Bulletin - [00:03](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=3s) 数据科学家兼工程师面临普遍的“冒名顶替综合征”，传统统计学教材机械罗列公式却缺乏直观解释，但编程能力能提供直通统计学底层本质的捷径。 - [04:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=297s) 传统两样本均值比较依赖韦尔奇 t 检验（Welch's t-test）与复杂的伽马函数、自由度推导，常让工程师陷入数学形式主义的泥潭。 - [07:58](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=478s) 通过 50,000 次随机打乱标签（置换检验）模拟零假设，实验发现仅有 14 次能达到实际观测的 4.4 只蚊子均值差，直观推翻了怀疑者的偶然假设。 - [10:26](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=626s) 随机置换检验与自助法（Bootstrapping）等重抽样方法仅依赖循环迭代与随机数生成，让工程师能够以玩乐与探索的心态摆脱对统计学的恐惧。 ## Key takeaways - [00:03](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=3s) 许多软件工程师和数据科学家在面对统计学术语时缺乏直观理解，产生严重的冒名顶替综合征。 - [00:03](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=3s) 具备计算机编程能力的人实际上拥有一条直达统计学最深层、最核心概念的捷径。 - [02:27](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=147s) 真实的科学案例（喝啤酒是否更吸引蚊子）展现了统计学解决“怀疑者”与“倡导者”分歧的核心价值。 - [04:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=297s) 传统“统计学入门”教学过分偏重公式配方，如使用韦尔奇 t 检验和晦涩的自由度公式进行推导。 - [04:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=297s) 传统统计假设检验的核心在于零假设下的抽样分布，但被繁复的数学形式主义与自由度黑话所掩盖。 - [07:58](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=478s) 置换检验（Permutation Test）将原假设具象化：如果处理无效，则受试者标签可以完全随机打乱。 - [07:58](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=478s) 通过对数据反复洗牌 50,000 次并计算均值差，计算机可以直接生成经验抽样分布，而无需解析推导。 - [07:58](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=478s) 实现计算机模拟推断仅需三项基础能力：基础逻辑思考、随机数生成以及循环迭代控制结构。 - [10:26](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=626s) 随机置换检验属于重抽样方法（Resampling Methods）家族，同家族的自助法（Bootstrapping）同样极其强大实用。 - [10:26](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=626s) 传统数理统计推导需要数年的专门理论训练，而计算模拟法能让学习者在数分钟内把握推断本质。 - [10:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=657s) 编程为探索统计学提供了“可动手摆弄”的超能力，工程师应以玩乐和探索的精神取代恐惧与自我怀疑。 ## Detailed walkthrough ### [00:03](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=3s) 告别统计学的痛苦：用编程思维直击统计学本质作为 Pinterest 的数据科学家，John Rauser 在演讲开篇引用了一篇学术论文的开头段落来表达自己的心声。他表示，当自己最初决定学习统计学时，翻阅了多本“在此出于礼貌不具名”的书籍，但一本都没读懂。绝大多数教材只是机械地罗列统计步骤然后直接套用，完全没有任何直观的解释，更不解释当初发明这些方法的人究竟是怎么想出来的。 John Rauser 直言，台下的听众大部分可能在统计学上都在“装懂”。在 Strata 大会以及整个行业中，技术人员大致分为软件工程师和统计学家两类，而工程师的人数占据绝对多数。当某些懂一点统计学的人开始高谈阔论“功效分析”（power analysis）、“广义线性模型”（generalized linear models）或者“双尾检验”（two-tailed tests）时，工程师们往往一边频频点头附和，一边内心其实茫然不知所云。John Rauser 表示自己完全理解这种冒名顶替综合征（impostor syndrome），因为他自己就是一名耗费了大约十年时间自学统计学的软件工程师，经历过面对最基础概念时的反复挣扎。因此，John Rauser 提出了整场演讲的核心论点：统计学教育不应是一堆让人摸不着头脑的公式大杂烩，如果你具备编写计算机程序的能力，你就拥有了一条直通统计学最深层、最核心概念的捷径。 > “My thesis is that if you can program a computer, you have direct access to the deepest and most fundamental ideas in statistics.” —— John Rauser ### [02:27](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=147s) 经典案例：喝啤酒会让人对蚊子更具吸引力吗？为了展示这一论点，John Rauser 引入了一项发表在知名期刊 *PLOS ONE* 上的真实科学研究：喝啤酒是否会让人更容易招蚊子。这看似是一个戏谑的话题，但背后有着严肃的流行病学意义。疟疾是通过蚊子传播的，传统的疟疾传播模型通常假设所有人被叮咬的风险是相同的。然而现有证据表明，不同人体质对蚊子的吸引力差异巨大。如果能查明哪些人处于更高风险中，公共卫生干预措施就能更具针对性，从而更有效地对抗疟疾。该实验招募了一批志愿者，并将他们随机分配去喝啤酒或喝水。研究人员使用了一种名为“Y型嗅觉仪”（Y-tube olfactometer）的装置，让蚊子在飞向人类受试者和飞向新鲜空气之间做出选择，最后捕获并清点被吸引的蚊子数量： - **喝啤酒组**：25 名受试者 - **喝水组**：18 名受试者 - **实验结果**：啤酒组平均每人吸引的蚊子数量比喝水组多出 4.4 只此时，核心的统计学问题浮现出来：这多出来的 4.4 只蚊子，是否足以证明喝啤酒确实让人更容易招蚊子？这个问题本质上是“怀疑者”（Skeptic）与“倡导者”（Advocate）之间的辩论。怀疑者认为，喝啤酒根本没有任何效果，4.4 的差异完全可能只是随机偶然现象；倡导者则反驳称，考虑到样本整体的变异程度，4.4 是一个显著的差距，怀疑者的假设极不可能成立。统计学的核心目的，正是为了解决这种分歧。 ```mermaid flowchart TD accTitle: 啤酒与蚊子吸引力实验假设对比 accDescr: 展示怀疑者与倡导者围绕4.4只蚊子平均差异所持的对立观点 A["实验数据：啤酒组比喝水组平均多吸引 4.4 只蚊子"] --> B{"统计学核心问题：差异是否由随机误差导致？"} subgraph Skeptic["怀疑者立场（原假设）"] B -->|"质疑"| C["喝啤酒对招蚊子没有任何实际影响"] C --> D["4.4 的差异纯属随机偶然"] end subgraph Advocate["倡导者立场（备择假设）"] B -->|"支持"| E["4.4 的差距相对于样本波动非常显著"] E --> F["怀疑者的偶然论在概率上极不可能成立"] end ``` ### [04:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=297s) 令人头痛的传统分析法：Welch's t 检验的数学泥潭为了形成对比，John Rauser 首先演示了大学“统计学入门”（Stats 101）课程中那种令人痛苦的传统分析方法。面对两组均值比较，人们往往会上维基百科搜索，接着找到“韦尔奇 t 检验”（Welch's t-test）。代入数据后，算出一个 t 统计量为 3.67。但接着痛苦接踵而至： 1. **分布公式与伽马函数**：根据理论，如果怀疑者是对的，t 统计量将遵循一个极其复杂的概率密度函数，里面包含形如 $\Gamma$ 的伽马函数以及表示自由度的 $\nu$。 2. **不知所云的自由度**：维基百科对自由度的解释是“随机向量定义域的维度数”（the number of dimensions of the domain of a random vector），这对大多数工程师而言宛如天书。 3. **复杂的自由度估算公式**：使用韦尔奇-萨特思韦特方程（Welch–Satterthwaite equation）代入计算，得出自由度约为 39.1。 4. **查表得出临界值**：在 40 自由度、95% 置信水平（$p = 0.05$）下查表得到临界值为 2.021。 5. **机械对比得出结论**：因为计算得到的 $t = 3.67 > 2.021$，因此可以拒绝怀疑者的假设，宣布在 $p < 0.05$ 水平下结果具有统计学显著性。 John Rauser 尖锐地指出，整套推理最核心的灵魂在于那个假设检验统计量的抽样分布（sampling distribution under the null hypothesis）。如果怀疑者拒绝承认该数学公式在现实中的有效性，整个论证大厦瞬间化为飞灰。传统教学的失败之处就在于，把本就抽象的抽样分布概念包裹在纯粹的数学形式主义和晦涩的自由度黑话中，连台下身为主流数据科学家的自己都无法从第一性原理现场推导那个公式，更遑论普通工程师。 > “And I'm willing to bet that only a few people in this audience have any idea how that argument really works, even if you are familiar with the statistical recipe that I just ran you through.” —— John Rauser ### [07:58](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=478s) 计算机模拟解法：洗牌、迭代与直观的抽样分布随后，John Rauser 展现了优雅得多的计算模拟法（置换检验 / Permutation Test）。如果怀疑者的假定是真的——即喝啤酒还是喝水对吸引蚊子毫无影响——那么每个人身上的“啤酒”或“水”标签就没有任何实际意义，完全是随机贴上去的。基于这个假设，计算机可以通过极其直观的逻辑进行模拟： 1. **打乱标签（Shuffle）**：将 43 个受试者的数据混在一起，随机打乱他们的标签，重新划分为 25 人的“假啤酒组”和 18 人的“假水组”。 2. **计算新均值差**：计算这两组的平均蚊子数差值。第一次打乱后算出的差值是 3.3，在图表上画一个点；第二次打乱后算出的差值是 0.1，再画一个点。 3. **大规模重复迭代**：将这个“打乱-计算均值-记录差值”的过程重复 20 次、50 次，直至 50,000 次。在 50,000 次随机置换后，整个图表清晰地呈现出了在“怀疑者假设（原假设）成立”的前提下，均值差可能出现的完整分布范围。结果显示，在 50,000 次随机试验中，均值差达到或超过 4.4 的情况仅仅出现了 14 次。这意味着，如果喝啤酒真的没用，产生 4.4 这种差距的概率低得异乎寻常。因此，怀疑者的立场不攻自破，实验确凿证明了喝啤酒确实让人更容易招蚊子。 John Rauser 总结道，理解并实现这种现代统计方法，根本不需要死记硬背枯燥的微积分或分布公式，只需要三样极其朴素的基础能力： - 理解简单明了的逻辑论证能力（人类与生俱来）； - 随机数生成（任何成熟编程语言的标准库均具备）； - 循环与迭代（编程语言最基础的控制结构）。工程师只需凭借这三点，就能完全掌握统计推断的核心威力。 ```mermaid flowchart TD accTitle: 计算模拟置换检验流程 accDescr: 展示通过洗牌、计算均值差和迭代生成经验抽样分布的过程 A["原始数据：25人啤酒组，18人水组 实际观测差异 = 4.4"] --> B["假定怀疑者成立：标签完全无意义"] subgraph Loop["单次模拟迭代循环"] B --> C["随机打乱受试者标签 (Shuffle)"] C --> D["重新计算两组均值差"] D --> E["在分布图上记录该均值差"] end E --> F{"迭代是否达到 50,000 次？"} F -- "否" --> C F -- "是" --> G["生成完整的经验抽样分布"] G --> H["对比观测值 4.4： 50,000 次中仅出现 14 次"] H --> I["拒绝怀疑者假设：喝啤酒显著增加吸引力"] ``` ### [10:26](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=626s) 随机置换检验与重抽样方法的核心威力在对比了繁复的传统统计推导公式与直观的模拟手段后，John Rauser 指出，台下的听众刚刚已经在一个非常深刻、本质的层面上理解了整个统计推断论证。相比之下，传统教科书里密密麻麻的数学公式与理论细节则完全相反，John Rauser 直言：“与此形成鲜明对比的是这个东西，要想真正掌握其中的细节，我认为你确实需要经过多年的专门学习。”（“and that is in contrast with this the details of which you really need years of study I think to come to grips with”）刚才所演示的那种极其简洁直观的计算手段，在统计学中被称为“随机置换检验”（Random Permutation Test）。它实际上属于一整套被称为“重抽样方法”（Resampling Methods）的庞大家族中的一种。Rauser 补充道，如果有更多的时间，他还想深入介绍该家族中另一个极其好用且强大的方法——自助法（Bootstrapping）。然而受限于演讲时长，他必须把重点收束回最初的核心论点上。 - **随机置换检验（Random Permutation Test）**：通过计算机洗牌并重置标签来模拟零假设下的样本分布，绕开了复杂的公式推导。 - **重抽样家族（Resampling Methods）**：除了置换检验外，还涵盖了极具实用价值的 Bootstrapping（自助法）。 - **学习路径差异**：基于编程与模拟的统计学思维能够直达概念本质，而传统数理推导往往需要耗费数年的数学训练才能彻底吃透。 ```mermaid flowchart LR accTitle: 重抽样方法分类及其核心逻辑 accDescr: 展示重抽样方法包含随机置换检验与自助法，并借由计算机模拟直达统计本质。 subgraph Resampling["重抽样方法 (Resampling Methods)"] A["随机置换检验 (Random Permutation Test)"] B["自助法 (Bootstrapping)"] end C["编程与计算模拟"] --> Resampling Resampling --> D["直达统计学底层核心本质"] ``` ### [10:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=657s) 编程超能力：以玩乐与探索之心拥抱统计学在演讲的尾声，John Rauser 换了一种方式重新阐述他的核心观点，并留给在场工程师们一段充满感染力的寄语。他坚定地告诉所有会写代码的人：“我想留给大家的信息是这样的：只要你会给计算机编程，在学习统计学这件事上，你就拥有了超能力。”（“The message that I want to leave you with is this: if you can program a computer, you have superpowers when it comes to learning statistics.”）这种超能力的核心在于，编程赋予了人们动手“摆弄”（tinker）统计学核心概念的能力。就像小孩子摆弄电子元件、机械结构，或者人们沉浸于玩音乐和搞体育运动一样，编写程序可以让抽象的模型变得具体可触。最后，Rauser 鼓励大家带着纯粹的乐趣和游戏玩乐的精神去攻克统计难题，彻底摆脱过去的恐惧与自我怀疑：“因此，我希望你们走出会场后，能带着一种喜悦的心情和玩乐的精神去攻克统计问题，而不是陷在恐惧与自我怀疑的泥潭之中。”（“And so I want you to go out and to attack statistical problems with a feeling of joy in the spirit of play and not from a position of fear and self-doubt.”）在全场热烈的掌声中，他的分享圆满落幕。 ## Notable quotes > [10:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=657s) The message that I want to leave you with is this: if you can program a computer, you have superpowers when it comes to learning statistics. — John Rauser > [10:57](https://www.youtube.com/watch?v=5Dnw46eC-0o&t=657s) And so I want you to go out and to attack statistical problems with a feeling of joy in the spirit of play and not from a position of fear and self-doubt. — John Rauser ## Mentioned John Rauser, Pinterest, Amazon, Strata + Hadoop World, O'Reilly, PLOS ONE, 韦尔奇 t 检验 (Welch's t-test), 韦尔奇-萨特思韦特方程 (Welch–Satterthwaite equation), 随机置换检验 (Random Permutation Test), 自助法 (Bootstrapping), 重抽样方法 (Resampling Methods) ## Video description From the 2014 Strata Conference + Hadoop World in New York City. There are two essential skills for the data scientist: engineering and statistics. A great many data scientists are very strong engineers but feel like impostors when it comes to statistics. In this talk John will argue that the ability to program a computer gives you special access to the deepest and most fundamental ideas in statistics. John’s goal is to convince the non-statistician engineers in the audience that the road to statistical fluency is much, much shorter than they think. About John Rauser: John has been extracting value from large datasets for over 20 years at hedge funds, small data-driven startups, Amazon, and now Pinterest. He has deep experience in machine learning, data visualization, on-line experimentation, website performance and real-time fault analysis. An empiricist at heart, “Just do the experiment!” is his favorite call to arms. Watch more from Strata + Hadoop World 2014: http://goo.gl/UUfrR7 Find out more about the conference: http://strataconf.com/stratany2014 Don't miss an upload! Subscribe! http://goo.gl/szEauh Stay Connected to O'Reilly Media by Email - http://goo.gl/YZSWbO Follow O'Reilly Media: http://plus.google.com/+oreillymedia https://www.facebook.com/OReilly https://twitter.com/OReillyMedia [Source](https://www.youtube.com/watch?v=5Dnw46eC-0o)

Source quotes

[10:57] John Rauser: The message that I want to leave you with is this: if you can program a computer, you have superpowers when it comes to learning statistics. [10:57] John Rauser: And so I want you to go out and to attack statistical problems with a feeling of joy in the spirit of play and not from a position of fear and self-doubt.

Source quotes

[0:03] John Rauser: My thesis is that if you can program a computer, you have direct access to the deepest and most fundamental ideas in statistics. [10:26] John Rauser: I want you to go out and to attack statistical problems with a feeling of joy in the spirit of play and not from a position of fear and self-doubt.

Original transcript

[0:03] okay so I'm John Rouser and I'm a data [0:06] scientist at Pinterest uh and the title [0:08] of this talk uh is inspired by this [0:10] paper uh this paper has perhaps the best [0:13] opening paragraph of any academic work [0:15] I've ever read uh which I will now adapt [0:18] for my purposes when I decided to learn [0:21] statistics I read several books which I [0:24] shall politely not identify I understood [0:28] none of them most most of them simply [0:30] described statistical procedures and [0:32] then applied them without any intuitive [0:35] explanation or hint of how anyone might [0:37] have invented these procedures in the [0:39] first place and this talk was born of [0:42] that frustration and my wish that future [0:44] students of Statistics will learn the [0:46] deep and elegant ideas at the heart of [0:49] Statistics rather than a confusing grab [0:51] bag of statistical [0:54] procedures I wrote this talk because I [0:56] suspect that many of the people in this [0:58] audience are faking it when it comes to [1:02] statistics so there are two kinds of [1:04] people two kinds of technical people in [1:07] this audience uh the software engineers [1:09] and the statisticians at strata and in [1:12] general the engineers are by far the [1:14] larger group actually in this room uh [1:16] maybe this picture looks more like this [1:19] right uh there are probably few [1:20] statisticians in this room that would [1:22] not all self-identify as data scientists [1:25] and I think that many of the engineers [1:28] look with some Envy on the people that [1:31] know statistics um when someone with a [1:34] little bit of stats knowledge starts [1:35] going on about power analysis or [1:38] generalized linear models or two-tailed [1:40] tests or whatever it is you know you nod [1:42] your head and you play along but you [1:44] really have no idea what they're talking [1:46] about um and I know this feeling I am a [1:51] software engineer who is self-taught in [1:54] Statistics over a period of about a [1:56] decade uh and I remember struggling with [1:59] what seemed like the most basic [2:01] questions but it doesn't have to be this [2:04] way my thesis is that if you can program [2:07] a computer you have direct access to the [2:10] deepest and most fundamental ideas in [2:13] statistics to convince you of this idea [2:16] I want to walk you through a statistical [2:18] argument uh and in order to do that we [2:20] need a problem to work on so we're going [2:22] to use statistics to figure out whether [2:24] drinking beer makes you more attractive [2:26] to [2:27] mosquitoes so the this is not a trivial [2:31] problem though it might seem one um [2:33] malaria after all is transmitted via [2:36] mosquitoes and most models for malaria [2:39] transmission have historically assumed [2:41] that all individuals are at equal risk [2:44] for uh for mosquito bites but there's [2:46] good evidence that humans vary widely in [2:48] their attractiveness to mosquitoes uh [2:51] and so if you can understand which [2:53] people are at greatest risk for mosquito [2:55] bites then you can Target your [2:57] interventions much more accurately and [2:59] you can do a better job of fighting [3:01] malaria this is such an important [3:03] problem that if you do good research on [3:05] it you can get published in plause one [3:07] an extremely reputable Journal uh here [3:10] I've redacted the keyword in the title [3:13] uh so as to leave the outcome in doubt [3:14] for a moment uh I don't have time to [3:17] describe their method in the detail that [3:19] it deserves but they they basically got [3:22] a series of volunteers and then randomly [3:25] assign them to drink either beer or [3:27] water uh and then they use this device [3:29] called the Y oldometer to let mosquitoes [3:32] choose to fly either toward the human [3:34] subject or towards open air and then [3:36] they trapped and counted the mosquitoes [3:39] here's the data they had 25 volunteers [3:42] who drank beer and 18 who drank water [3:45] and the these numbers are the numbers of [3:48] mosquitoes that were collected in the [3:50] traps for each of the volunteers uh we [3:52] can compute the average number of [3:54] mosquitoes in each group and then [3:56] subtract to find that the average person [3:58] who drank beer attracted 4.4 more [4:01] mosquitoes than the average water [4:03] Drinker uh and now we have a statistical [4:05] question right is a difference of 4.4 [4:09] sufficient evidence to claim that [4:10] drinking beer makes you more attractive [4:13] to mosquitoes we can frame this question [4:15] as a debate between a skeptic and an [4:17] advocate the skeptic says that yeah [4:19] probably it's the case that drinking [4:20] beer has no effect uh and that [4:22] difference of 4.4 that could have just [4:23] happened by random Chance The Advocate [4:26] takes the other side The Advocate says [4:28] 4.4 is a large difference [4:30] uh especially when you compare it to the [4:31] overall variation in the sample and so [4:34] so the skeptic's position is very [4:35] unlikely to be true um one of the main [4:38] goals of Statistics is to settle just [4:40] this kind of debate so what I want to do [4:42] is walk you through it twice uh first [4:44] I'll solve the problem using the painful [4:46] analytical approach that you might [4:48] remember from your statistics 101 class [4:50] uh and then I'll do it again using a [4:51] simple computational method uh that [4:53] should hopefully be much more [4:55] understandable uh okay so bring on the [4:57] pain um [5:00] if you took a stats class or you tried [5:02] to read a stats book uh you might dimly [5:04] recall something called a t test that [5:06] can be used to solve exactly this kind [5:08] of problem so you head off to Wikipedia [5:10] and you remember that the first thing [5:12] you need to do is you need to pick a [5:13] test statistic uh there are a whole [5:15] bunch of possible choices but after [5:17] reading for a few minutes you settle on [5:19] this one which is Welch's T Test uh you [5:22] plug in your numbers and you get a value [5:23] of 3.67 for your data you don't really [5:26] know what this te thing is but Wikipedia [5:29] tells you that if the skeptic is right [5:31] then uh then T is distributed according [5:34] to this formula that funny L looking [5:36] thing that's the gamma function that V [5:38] looking thing is the number of degrees [5:40] of freedom you read on Wikipedia about [5:42] degrees of freedom for a little while [5:44] and you learn that degrees of freedom is [5:46] the number of dimensions of the domain [5:48] of a random Vector you have no idea what [5:51] that [5:55] means but the page about the T Test on [5:57] Wikipedia says that you can estimate [5:59] your degree of Freedom with this [6:02] formula and so you dutifully plug in [6:04] your numbers and you get that you have [6:05] 39.1 de of Freedom how you figure out [6:08] the next step I have no idea but you [6:10] manage to figure out that what you need [6:12] to do is take that 39 uh and use it to [6:15] look up the critical value for the T [6:16] Test in a table so you do that and you [6:19] get that the critical value at the 95% [6:21] level for 40 degrees of freedom is 2.02 [6:23] one and that t statistic that you [6:25] computed five slides ago uh that thing [6:28] 3.6 is larger than the critical value [6:31] 2.21 and so sweet you say to yourself [6:34] you can now confidently reject the [6:37] Skeptics argument and say that a [6:39] difference of 4.4 additional mosquitoes [6:41] is statistically significant at the p05 [6:44] level and I'm willing to bet that only a [6:48] few people in this audience have any [6:51] idea how that argument really works even [6:55] if you are familiar with the statistical [6:58] recipe that I just ran you through [7:00] this thing is the really deep idea at [7:03] the heart of that argument if the [7:05] skeptic refuses to believe your [7:07] assertion that this is the correct [7:09] formula your entire argument falls to [7:12] the ground in ashes in the general [7:15] recipe for statistical inference this [7:16] thing is called the sampling [7:17] distribution of the test statistic under [7:19] the null hypothesis and the reason that [7:21] stats 101 was so incredibly painful is [7:25] that the idea of a sampling distribution [7:28] is really hard to understand even in the [7:31] best conditions and when it's presented [7:34] in pure mathematical formalism like this [7:36] as a mathematical object all slathered [7:39] up in degrees of freedom mumbo jumbo [7:41] it's just [7:43] hopeless um there might be a handful of [7:45] people in this audience that could sit [7:46] down and just derive this equation from [7:48] first principles right now uh I am [7:51] certainly not among those people and I [7:53] am a working data [7:55] scientist so that was stats 101 the [7:58] comput or the analytical method uh what [8:00] about this computational method that I [8:02] promised uh well remember the thing that [8:05] we're trying to figure out is whether [8:07] this 4.4 is a large or a small [8:09] difference so we'll just Mark that 4.4 [8:11] on a plot uh here's our original data [8:14] colorcoded to whether the subject drank [8:17] beer or water and now if the skeptic is [8:19] right um these these labels have [8:22] absolutely no meaning they're completely [8:23] meaningless we can just they carry no [8:25] information and so what I can do is [8:27] randomly shuffle them around and then [8:29] rearrange them and then tidy them up and [8:32] then compute some new means subtract the [8:34] means and get a new difference of 3.3 [8:37] mosquitoes uh and we'll add a little dot [8:39] to our plot at 3.3 and now we can start [8:42] that whole dance over again uh we'll [8:44] start with the original data we randomly [8:46] Shuffle we rearrange we tidy up uh [8:49] compute some means subtract the means [8:51] get a difference of 0.1 this time uh and [8:54] now we'll add that 0.1 to our plot and [8:56] we can keep on repeating this process [8:58] over and over and over again here's [9:00] three repetitions here's four here's [9:02] five here's six and look ma no hands [9:06] right what is happening here is we are [9:08] building up um a uh uh the statist or [9:12] the sampling distribution under the [9:13] skeptic's argument you are watching a [9:15] statistical process unfold here is 20 [9:18] repetitions here's 30 here's 50 here's [9:22] 50,000 repetitions so recall the [9:25] skeptic's argument that there is no [9:27] difference uh uh or there's no [9:29] difference that there's no effect uh and [9:31] that the labels were meaningless so this [9:33] data was generated under that assumption [9:36] and it shows the range of possibilities [9:38] if the skeptic is right and if the [9:40] skeptic is Right a difference of 4.4 is [9:42] fantastically rare it happened just 14 [9:44] times in 50,000 trials and so it is as [9:47] The Advocate said the skeptic's argument [9:50] strains credibility and can be safely [9:53] rejected and that of course was the [9:55] conclusion of the researchers that [9:57] gathered this data in the first place [9:59] to do the statistics that we just did [10:02] you needed three essential things the [10:05] ability to follow a straight forward [10:08] logical argument random number [10:10] generation and iteration you were born [10:13] with the first of these three things and [10:15] the second two are provided by any [10:18] decent programming language with a good [10:20] Library um with these three things you [10:24] have everything you need to understand [10:26] this argument at a very deep fun [10:29] fundamental level and that is in [10:31] contrast with this the details of which [10:33] you really need years of study I think [10:35] to come to grips with now that simple [10:38] computational method I just showed you [10:40] is called a a random permutation test uh [10:43] and it's just one of a whole class of [10:45] methods known as resampling methods the [10:48] other resampling method that I'd tell [10:50] you about if I had more time is [10:51] bootstrapping which is fantastically [10:54] useful but I don't have more time and so [10:57] I'll just restate my thesis in a [10:58] slightly different way way the message [11:00] that I want to leave you with is this if [11:03] you can program a computer you have [11:05] superpowers when it comes to learning [11:08] statistics because being able to program [11:11] allows you to Tinker with the most [11:13] fundamental ideas in statistics the way [11:16] that you might have tinkered with [11:17] electronics when you were a kid or with [11:20] mechanical things or with music or with [11:24] sports and so I want you to go out and [11:27] to attack statistical problems with a [11:29] feeling of joy in the spirit of play and [11:33] not from a position of fear and [11:36] self-doubt that's all I have thanks for [11:38] your [11:38] [Applause] [11:46] time