3Blue1Brown Strips Neural Nets to 13,000 Matrix-Vector Operations

But what is a neural network? | Deep learning chapter 1 — 3Blue1Brown

3Blue1Brown

Layered neural networks resolve complex recognition tasks not through biological mimicry, but by chaining 13,000 tunable linear algebra parameters that assemble raw pixels into hierarchical geometric features.

AI-assisted summary of But what is a neural network? | Deep learning chapter 1 — original source. Check the original for full context.

Source published:

Key ideas

  • A plain vanilla neural network maps high-dimensional inputs directly to category predictions, exemplified by a 4-layer network that transforms 784 grayscale pixels into 10 digit probabilities via roughly 13,000 tunable parameters. (10:38)
  • The computational structure relies on hierarchical abstraction, where initial layers detect localized edges, intermediate layers combine edges into components, and final layers classify complete patterns. (5:31)
  • Layer-to-layer forward propagation is compactly vectorized as σ(Wa + b), turning the entire network into a single deterministic mathematical function mapping ℝ⁷⁸⁴ to ℝ¹⁰. (13:26)
  • While historically grounded in the sigmoid activation function σ(x) = 1 / (1 + e⁻ˣ), modern deep learning predominantly uses ReLU, max(0, a), due to severe training difficulties encountered with deep sigmoidal networks. (17:03)

Detailed notes

## Background This video was published on YouTube by 3Blue1Brown on October 5, 2017, as the opening chapter of a deep learning series. According to the description, the video explores what artificial neurons are, why neural networks are structured in sequential layers, and the core mathematical framework underpinning their operation.

## TL;DR Using the classic problem of handwritten digit recognition on a 28×28 pixel grid as a motivating example, the video strips away machine learning buzzwords to explain the plain-vanilla multilayer perceptron. It defines a neuron as a container for an activation value and demonstrates how stacking layers enables hierarchical feature abstraction, from localized edges to composite shapes. The lesson details the mechanics of computing weighted sums and biases across nearly 13,000 parameters, expresses layer transitions cleanly via matrix-vector multiplication as $\sigma(Wa + b)$, and concludes with a discussion alongside guest researcher Lisha Li on why modern deep learning architectures often favor ReLU over the traditional sigmoid function.

## Atmosphere & style The presentation carries 3Blue1Brown's signature calm, elegant, and intellectually reassuring tone, grounding complex concepts in lucid geometric intuitions and smooth custom animations. Grant Sanderson speaks with measured curiosity and warmth, demystifying intimidating mathematical notation into approachable, tactile analogies like turning 13,000 dials. The brief, collaborative closing segment with guest Lisha Li adds an engaging, real-world research perspective, leaving the viewer feeling quietly empowered rather than overwhelmed by mathematical abstraction.

## Bulletin - [10:38](https://www.youtube.com/watch?v=aircAruvnKk&t=638s) A plain vanilla neural network maps high-dimensional inputs directly to category predictions, exemplified by a 4-layer network that transforms 784 grayscale pixels into 10 digit probabilities via roughly 13,000 tunable parameters. - [05:31](https://www.youtube.com/watch?v=aircAruvnKk&t=331s) The computational structure relies on hierarchical abstraction, where initial layers detect localized edges, intermediate layers combine edges into components, and final layers classify complete patterns. - [13:26](https://www.youtube.com/watch?v=aircAruvnKk&t=806s) Layer-to-layer forward propagation is compactly vectorized as σ(Wa + b), turning the entire network into a single deterministic mathematical function mapping ℝ⁷⁸⁴ to ℝ¹⁰. - [17:03](https://www.youtube.com/watch?v=aircAruvnKk&t=1023s) While historically grounded in the sigmoid activation function σ(x) = 1 / (1 + e⁻ˣ), modern deep learning predominantly uses ReLU, max(0, a), due to severe training difficulties encountered with deep sigmoidal networks.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=aircAruvnKk&t=4s) Algorithmic digit recognition on raw 28x28 pixel grids is exceptionally difficult for conventional rule-based code, prompting the use of data-driven neural networks. - [01:07](https://www.youtube.com/watch?v=aircAruvnKk&t=67s) The focus of the exploration is on the foundational 'plain vanilla' multilayer perceptron structure before examining how optimization and learning occur. - [02:42](https://www.youtube.com/watch?v=aircAruvnKk&t=162s) An artificial neuron is fundamentally a container holding an activation value bounded between 0.0 and 1.0. - [03:35](https://www.youtube.com/watch?v=aircAruvnKk&t=215s) The demonstrated architecture consists of 784 input neurons, two hidden layers of 16 neurons each, and 10 output neurons representing digits 0 through 9. - [05:31](https://www.youtube.com/watch?v=aircAruvnKk&t=331s) Layered architectures leverage hierarchical abstraction to break complex visual recognition tasks into lower-level edge and component detection problems. - [07:30](https://www.youtube.com/watch?v=aircAruvnKk&t=450s) Hierarchical decomposition applies beyond visual cognition to domains like speech recognition, which chains audio waveforms into phonemes, syllables, words, and phrases. - [08:38](https://www.youtube.com/watch?v=aircAruvnKk&t=518s) Weights determine what specific spatial pattern of pixels or previous-layer activations a neuron responds to, using positive, negative, or near-zero values. - [09:45](https://www.youtube.com/watch?v=aircAruvnKk&t=585s) The sigmoid function squashes arbitrary real-valued weighted sums into a smooth output interval between 0 and 1. - [10:38](https://www.youtube.com/watch?v=aircAruvnKk&t=638s) Biases act as activation thresholds, dictating how large the incoming weighted sum must be before a neuron becomes meaningfully active. - [11:34](https://www.youtube.com/watch?v=aircAruvnKk&t=694s) The 784-16-16-10 architecture contains almost exactly 13,000 weights and biases, which serve as the adjustable knobs tuned during the learning process. - [12:30](https://www.youtube.com/watch?v=aircAruvnKk&t=750s) Understanding the functional role of weights and biases prevents practitioners from treating neural networks purely as opaque black boxes. - [13:26](https://www.youtube.com/watch?v=aircAruvnKk&t=806s) Linear algebra condenses network computations into the matrix-vector form σ(Wa + b), drastically speeding up execution in modern numerical software. - [15:10](https://www.youtube.com/watch?v=aircAruvnKk&t=910s) The entire neural network functions as a single high-dimensional mathematical function mapping 784 real inputs to 10 output values. - [17:03](https://www.youtube.com/watch?v=aircAruvnKk&t=1023s) Modern architectures favor the Rectified Linear Unit (ReLU), defined as max(0, a), because it trains far more effectively in deep networks than traditional sigmoids.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=aircAruvnKk&t=4s) The Challenge of Recognizing Handwritten Digits

Visual recognition appears completely effortless to the human brain, but it is computationally staggering. A handwritten numeral "3", scrawled sloppily and rendered onto an extremely coarse grid of just 28 by 28 pixels, is identified by the human eye without hesitation. Even when presented with multiple variations of the digit—each possessing radically different individual pixel values and triggering entirely different photoreceptors in the eye—the visual cortex seamlessly interprets them as representing the exact same conceptual numeral, while simultaneously categorizing other configurations as distinct numbers altogether.

```mermaid flowchart LR accTitle: MNIST Digit Classification Pipeline accDescr: An input 28x28 image flows into a network of 784 inputs, two hidden layers, and ten outputs. subgraph IN["Input Stage"] IMG["28x28 Image"] --> L1["Layer 1: 784 Neurons"] end subgraph HID["Hidden Processing"] L1 --> L2["Layer 2: 16 Neurons"] L2 --> L3["Layer 3: 16 Neurons"] end subgraph OUT["Output Stage"] L3 --> L4["Layer 4: 10 Neurons<br/>(Digits 0–9)"] end ```

The difficulty becomes obvious the moment one attempts to formalize this process algorithmically. As Grant Sanderson observes: “But if I told you, hey, sit down and write for me a program that takes in a grid of 28x28 pixels like this and outputs a single number between 0 and 10, telling you what it thinks the digit is, well the task goes from comically trivial to dauntingly difficult.” Modern machine learning tackles this very gap by abandoning rigid rule-writing in favor of mathematical structures that learn directly from data.

- Input image size: 28 × 28 grayscale pixels (784 total values) - Program objective: Map the raw pixel array to an integer prediction from 0 to 9

---

[01:07](https://www.youtube.com/watch?v=aircAruvnKk&t=67s) Demystifying Neural Networks: Vanilla Architecture Over Hype

While machine learning and artificial intelligence are saturated with industry buzzwords, the core objective here is to demystify what a neural network actually represents from a purely mathematical standpoint, assuming zero prior background. Rather than tackling complex, modern architectural variations that have exploded throughout recent research, this exploration focuses on the foundational, "plain vanilla" multilayer perceptron without extra frills.

This introductory setup splits the conceptual framework into two distinct halves: first, understanding the physical and mathematical anatomy of the network structure itself, and second, understanding what it actually means when a network undergoes the process of "learning." Even in its most basic, unadorned state, a simple neural network possesses enough expressive power to classify handwritten digits, while simultaneously exhibiting clear, instructive limitations in how it handles visual data.

---

[02:42](https://www.youtube.com/watch?v=aircAruvnKk&t=162s) Deconstructing the Neuron and Network Layers

Stripping away biological romanticism, an artificial neuron is fundamentally just a container holding a real number—specifically, an "activation" value bounded between 0 and 1. The full network is organized into sequential layers of these containers:

```mermaid flowchart TD accTitle: Forward Layer Propagation accDescr: Propagation from input pixels through hidden feature detectors to final output digits. subgraph L_IN["Input Layer"] P["784 Neurons<br/>(Pixel Brightness: 0.0 to 1.0)"] end subgraph L_HID["Hidden Layers"] H1["Hidden Layer 1<br/>(16 Neurons: Edge Detectors)"] H2["Hidden Layer 2<br/>(16 Neurons: Pattern Assemblers)"] end subgraph L_OUT["Output Layer"] O["10 Neurons<br/>(Digit Confidences: 0 to 9)"] end P -->|"Weighted Influence"| H1 H1 -->|"Weighted Influence"| H2 H2 -->|"Weighted Influence"| O ```

The first layer contains 784 neurons, corresponding to every single pixel in the 28×28 input image. The activation of each input neuron mirrors the grayscale intensity of its assigned pixel, scaling from 0 for pure black up to 1 for pure white. At the opposite end stands the output layer, made up of 10 neurons corresponding to the digits 0 through 9; here, an activation near 1 signals high confidence from the model that the input matches that digit.

Between input and output sit two "hidden layers," each arbitrarily chosen to contain 16 neurons. As Sanderson points out regarding this concrete design choice: “To be honest I chose two layers based on how I want to motivate the structure in just a moment, and 16, well that was just a nice number to fit on the screen.” When an image is fed forward through the network, the activation pattern of the input layer triggers a cascade of activity through the hidden layers, culminating in an output layer where the neuron glowing with the highest activation represents the network's final classification decision.

- Input layer dimension: 784 neurons ($28 \times 28$) - Hidden layers: 2 intermediate layers containing 16 neurons each - Output layer dimension: 10 neurons (representing classifications 0 through 9) - Neuron activation limits: Real numbers restricted between 0.0 (completely off) and 1.0 (fully lit)

---

[05:31](https://www.youtube.com/watch?v=aircAruvnKk&t=331s) Why Layers? Hierarchies of Abstraction

The intuitive justification for stacking neurons into successive layers is the concept of hierarchical abstraction. Human visual cognition naturally deconstructs complex geometries into constituent subcomponents: a digit "9" consists of an upper loop connected to a vertical right-hand stroke; an "8" is composed of two vertically stacked loops; and a "4" is an assembly of three distinct straight lines.

```mermaid flowchart TD accTitle: Hierarchical Feature Composition accDescr: How raw pixels aggregate into edges, components, and finally complete digit classifications. E["Layer 2: Local Edges & Slits"] --> C["Layer 3: Subcomponents<br/>(e.g., Upper Loops, Long Lines)"] C --> D["Layer 4: Full Digits<br/>(e.g., 8, 9, 4)"] ```

In an ideal system, one would hope that neurons in the penultimate layer fire whenever generalized, high-level features appear—such as detecting an upper loop regardless of its exact pixel alignment. That detection task itself naturally breaks down into an even lower tier of sub-problems: recognizing the individual directional edges that form the boundaries of that loop. Sanderson notes that this hierarchical principle extends far beyond vision: “And even beyond image recognition, there are all sorts of intelligent things you might want to do that break down into layers of abstraction.” Speech recognition systems operate under the exact same paradigm, assembling raw sound waves into phonemes, syllables, words, and eventually high-level sentences.

---

[08:38](https://www.youtube.com/watch?v=aircAruvnKk&t=518s) Edge Detection and the Math of Weighted Sums

To translate this conceptual hierarchy into functional mathematics, every neuron in a given layer is connected to every neuron in the preceding layer, with each connection governed by an adjustable real number called a **weight**. To determine the activation of a target neuron in the second layer—such as one intended to detect an edge in a localized region—the network computes a weighted sum of all incoming activations from the first layer.

```mermaid flowchart LR accTitle: Neuron Weighted Sum and Activation accDescr: Inputs are multiplied by weights, summed together, and passed through a sigmoid function. subgraph Inputs["Previous Layer Activations"] A1["a₁"] A2["a₂"] AN["aₙ"] end subgraph Computation["Target Neuron"] SUM["Weighted Sum:<br/>∑ (wᵢ · aᵢ)"] SIG["Sigmoid Squishing:<br/>σ(x) = 1 / (1 + e⁻ˣ)"] end A1 -->|"w₁"| SUM A2 -->|"w₂"| SUM AN -->|"wₙ"| SUM SUM --> SIG SIG --> OUT["Activation (0 to 1)"] ```

Visually, these weights can be organized into a spatial grid mapping directly onto the input image's geometry: - **Positive weights (green):** Pixels in the desired edge region are assigned positive weights, meaning bright pixels in this specific area increase the sum. - **Negative weights (red):** Pixels in the immediate surrounding periphery are assigned negative weights, penalizing activations if light spills outside the target boundary. - **Zero weights:** Irrelevant background pixels across the rest of the image receive weights near zero, nullifying their impact on the computation.

Because summing these raw products can yield arbitrary real numbers far outside the unit interval, the total weighted sum must be squashed back down into a valid activation value between 0 and 1. This normalization is achieved by passing the raw sum through an activation function, historically the **sigmoid function** (also known as the logistic curve):

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

This mathematical step ensures that regardless of how large or negative the linear combination of inputs becomes, the resulting activation remains smoothly bounded within the expected $[0, 1]$ operational range.

[10:38](https://www.youtube.com/watch?v=aircAruvnKk&t=638s) Biases, Parameter Counts, and the 13,000 Knobs and Dials

When a weighted sum of inputs passes into the sigmoid squishification function, very negative inputs get mapped close to 0, positive inputs get mapped close to 1, and the curve climbs steadily around an input of 0. In this setup, a neuron's activation is essentially an indicator of how positive its weighted sum happens to be. However, you might not want a neuron to fire just because its weighted sum barely creeps above zero. You might prefer that it only turns on if the weighted sum crosses a higher threshold—say, greater than 10. To introduce a preference or bias toward inactivity, you simply add an offset such as -10 to the weighted sum before passing the total through the sigmoid squishification function. This extra offset is formally called the bias. In intuitive terms, the weights govern what specific spatial pattern of pixels or earlier activations a neuron responds to, while the bias determines how high that weighted sum must climb before the neuron becomes meaningfully active.

Scaling this logic up across the entire architecture reveals just how vast the parameter space truly is. A single neuron in the second layer connects to all 784 pixel neurons from the first layer, demanding 784 individual weights alongside its own bias. With 16 neurons in that first hidden layer, connecting layer one to layer two requires 784 times 16 weights plus 16 biases. Factoring in the connections flowing into the next hidden layer and finally out to the ten output digits, the entire network contains almost exactly 13,000 total weights and biases. When machine learning researchers talk about “learning,” they are describing the algorithmic process of nudging and turning these 13,000 individual knobs and dials until the network settles on a configuration that solves the task.

Pondering that complexity leads to a thought experiment that is equal parts entertaining and daunting: sitting down and manually setting every single one of those 13,000 numbers by hand, deliberately adjusting parameters so that the second layer captures edges and the third layer detects composite loops. While modern practitioners rely on automated optimization, Grant Sanderson emphasizes the value of understanding the underlying mechanics rather than treating the model as an opaque black box: “I personally find this satisfying rather than just treating the network as a total black box, because when the network doesn't perform the way you anticipate, if you've built up a little bit of a relationship with what those weights and biases actually mean, you have a starting place for experimenting with how to change the structure to improve.”

```mermaid flowchart LR accTitle: Single Neuron Activation Flow accDescr: Diagram showing inputs multiplied by weights, summed with a bias, and passed through a sigmoid function. subgraph Inputs["Previous Layer"] A["Activations a₀, a₁,... aₙ"] end subgraph Processing["Neuron Computation"] B["Weighted Sum: Σ(wᵢ · aᵢ)"] C["Add Bias: + b"] D["Sigmoid Squishification: σ(...)"] end subgraph Output["Current Layer"] E["Activation a'"] end A --> B B --> C C --> D D --> E ```

- Each weight indicates what pattern of pixels or previous-layer features a neuron looks for. - Each bias sets the activation threshold, determining how positive the weighted sum must be to ignite the neuron. - Layer 1 (784 neurons) to Layer 2 (16 neurons) uses 12,544 weights ($784 \times 16$) plus 16 biases. - The full demo network spans four layers ($784 \to 16 \to 16 \to 10$) totaling almost exactly 13,000 tunable parameters.

---

[13:26](https://www.youtube.com/watch?v=aircAruvnKk&t=806s) Compact Matrix Notation and the Network as a Function

Writing out weighted sums and biases across thousands of individual connections quickly becomes cumbersome, which is why neural network operations are universally expressed using linear algebra. All activations from an incoming layer are collected into a vertical column vector, while the weights linking that layer to the next are organized into a matrix where each row corresponds to all incoming connections feeding into a single destination neuron. Multiplying this weight matrix by the activation vector produces a new vector containing the raw weighted sums for every destination neuron in one fell swoop. The biases are grouped into their own vector and added directly to that product. Finally, applying the sigmoid function across the outside simply squishes each individual component of the resulting vector into a value between 0 and 1.

Beyond making theoretical notation concise, grouping operations into $\sigma(Wa + b)$ provides massive computational advantages because modern numerical libraries are heavily optimized to execute matrix-vector multiplications at high speed. From this bird's-eye perspective, the earlier idea that neurons are mere containers for static numbers gives way to a more dynamic view: each neuron is fundamentally a mathematical function that digests activations from the previous layer and outputs a value between 0 and 1. By extension, the full network is nothing more than a giant, composite mathematical function that takes 784 numbers as input and outputs 10 numbers, parameterized by 13,000 weights and biases. Grant Sanderson notes that this staggering mathematical complexity should actually be comforting: “I mean if it were any simpler, what hope would we have that it could take on the challenge of recognizing digits?”

Closing out the mathematical walkthrough, Sanderson teases that the following video will demonstrate how networks actually learn these optimal weights and biases purely from training data. He also jokes about YouTube's notification bells, quipping that viewers should subscribe less for notifications and more to train YouTube's recommendation neural networks into realizing they want to see his videos. He thanks his Patreon supporters for their patience, noting that progress on his probability series slowed down over the summer but will resume immediately after finishing this deep learning project.

- Matrix multiplication neatly packs the transformation: row $k$ of the weight matrix dotted with activation vector $a$ gives the weighted sum for neuron $k$. - The vectorized transformation is written compactly as $\sigma(Wa + b)$. - High-level view: the entire neural network is a single deterministic function mapping $\mathbb{R}^{784} \to \mathbb{R}^{10}$ through repeated matrix-vector transformations. - Viewers seeking geometric intuition for matrix transformations are pointed to Chapter 3 of the 3Blue1Brown *Essence of linear algebra* series.

---

[17:03](https://www.youtube.com/watch?v=aircAruvnKk&t=1023s) ReLU vs. Sigmoid: A Conversation with Lisha Li

To wrap up the introductory lesson, Sanderson brings in guest Lisha Li, who conducted her doctoral research on the theoretical foundations of deep learning and works at venture capital firm Amplify Partners, an early supporter that contributed funding for the video. Sanderson asks Li to clarify the practical role of the sigmoid function, remarking that while historically inspired by biological neurons switching between resting and active states, sigmoid squishing has become somewhat "old school" in contemporary practice. Li agrees, pointing out that modern architectures have largely moved away from sigmoids in favor of alternative activation functions that are far more effective to train.

Specifically, Li highlights the Rectified Linear Unit, universally known as ReLU. The ReLU activation function is defined simply as $\max(0, a)$, where $a$ represents the weighted sum plus bias. Like the sigmoid, ReLU draws loose inspiration from biology: if an incoming input fails to meet an activation threshold, the neuron remains dormant and outputs zero; once the signal crosses that threshold, the function acts as a linear identity mapping, outputting $a$ directly. Li explains that training very deep architectures using sigmoids proved exceptionally difficult, whereas switching to the simpler ReLU activation allowed deep networks to train smoothly and achieve modern breakthroughs.

- Lisha Li completed her PhD focusing on theoretical deep learning and works at Amplify Partners. - While the sigmoid function squishes all values into the $(0, 1)$ range, it frequently hinders the optimization of deep architectures. - ReLU stands for Rectified Linear Unit and follows the piecewise rule $f(a) = \max(0, a)$. - Lisha Li highlights why ReLU took over modern machine learning: “Using sigmoids didn't help training or it was very difficult to train at some point and people just tried ReLU and it happened to work very well for these incredibly deep neural networks.”

## Notable quotes > [17:03](https://www.youtube.com/watch?v=aircAruvnKk&t=1023s) Using sigmoids didn't help training or it was very difficult to train at some point and people just tried ReLU and it happened to work very well for these incredibly deep neural networks. — Lisha Li

## Mentioned 3Blue1Brown, Grant Sanderson, Lisha Li, Amplify Partners, Michael Nielsen, Chris Olah, Welch Labs, Ian Goodfellow, Yoshua Bengio, Aaron Courville, Distill, Vincent Rubinetti, Kevin Pluck, MNIST, Manim, YouTube, Patreon

## Video description What are the neurons, why are there layers, and what is the math underlying it? Help fund future projects: https://www.patreon.com/3blue1brown Written/interactive form of this series: https://www.3blue1brown.com/topics/neural-networks

Additional funding for this project was provided by Amplify Partners

For those who want to learn more, I highly recommend the book by Michael Nielsen that introduces neural networks and deep learning: https://goo.gl/Zmczdy

There are two neat things about this book. First, it's available for free, so consider joining me in making a donation to Nielsen if you get something out of it. And second, it's centered around walking through some code and data, which you can download yourself, and which covers the same example that I introduced in this video. Yay for active learning! https://github.com/mnielsen/neural-networks-and-deep-learning

I also highly recommend Chris Olah's blog: http://colah.github.io/

For more videos, Welch Labs also has some great series on machine learning: https://youtu.be/i8D90DkCLhI https://youtu.be/bxe2T-V8XRs

For those of you looking to go *even* deeper, check out the text "Deep Learning" by Goodfellow, Bengio, and Courville.

Also, the publication Distill is just utterly beautiful: https://distill.pub/

Lion photo by Kevin Pluck

Звуковая дорожка на русском языке: Влад Бурмистров.

Thanks to these viewers for their contributions to translations German: @fpgro Hebrew: Omer Tuchfeld Hungarian: Máté Kaszap Italian: @teobucci, Teo Bucci

----------------- Timeline: 0:00 - Introduction example 1:07 - Series preview 2:42 - What are neurons? 3:35 - Introducing layers 5:31 - Why layers? 8:38 - Edge detection example 11:34 - Counting weights and biases 12:30 - How learning relates 13:26 - Notation and linear algebra 15:17 - Recap 16:27 - Some final words 17:03 - ReLU vs Sigmoid

Correction 14:45 - The final index on the bias vector should be "k"

------------------ Animations largely made using manim, a scrappy open source python library. https://github.com/3b1b/manim

If you want to check it out, I feel compelled to warn you that it's not the most well-documented tool, and has many other quirks you might expect in a library someone wrote with only their own use in mind.

Music by Vincent Rubinetti. Download the music on Bandcamp: https://vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown

Stream the music on Spotify: https://open.spotify.com/album/1dVyjwS8FBqXhRunaG5W5u

If you want to contribute translated subtitles or to help review those that have already been made by others and need approval, you can click the gear icon in the video and go to subtitles/cc, then "add subtitles/cc". I really appreciate those who do this, as it helps make the lessons accessible to more people. ------------------

3blue1brown is a channel about animating math, in all senses of the word animate. And you know the drill with YouTube, if you want to stay posted on new videos, subscribe, and click the bell to receive notifications (if you're into that).

If you are new to this channel and want to see more, a good place to start is this playlist: http://3b1b.co/recommended

Various social media stuffs: Website: https://www.3blue1brown.com Twitter: https://twitter.com/3Blue1Brown Patreon: https://patreon.com/3blue1brown Facebook: https://www.facebook.com/3blue1brown Reddit: https://www.reddit.com/r/3Blue1Brown

[Source](https://www.youtube.com/watch?v=aircAruvnKk)

中文简报

3Blue1Brown将神经网络解析为1.3万次矩阵向量运算 分层神经网络并非通过模拟生物学机制，而是通过串联1.3万个可调线性代数参数，将原始像素重组为分层几何特征，从而解决复杂的识别任务。 视频以28×28像素网格上的经典手写数字识别问题为引子，摒弃了机器学习的各种流行术语，通俗解释了最基础的多层感知机。它将神经元定义为一个存放激活值的容器，并展示了如何通过堆叠网络层实现从局部边缘到复合形状的分层特征抽象。课程详细拆解了近1.3万个参数的加权和与偏置计算机制，用矩阵向量乘法将层间转换简洁地表示为$\sigma(Wa + b)$，并在结尾与特邀研究员Lisha Li探讨了为何现代深度学习架构往往更青睐ReLU而非传统的Sigmoid函数。 最基础的神经网络可将高维输入直接映射为类别预测，例如一个4层网络通过约1.3万个可调参数将784个灰度像素转化为10个数字的概率。 (10:38) 该计算结构依赖于分层抽象：最初的网络层检测局部边缘，中间层将边缘组合为局部组件，最终层对完整模式进行分类。 (5:31) 层与层之间的前向传播被紧凑地向量化为σ(Wa + b)，从而使整个网络变成一个将ℝ⁷⁸⁴映射到ℝ¹⁰的单一确定性数学函数。 (13:26) 尽管历史上以Sigmoid激活函数σ(x) = 1 / (1 + e⁻ˣ)为基础，但由于深层Sigmoid网络在训练中遇到了严重困难，现代深度学习已主要改用ReLU即max(0, a)。 (17:03)

Reading summary

Grant Sanderson explains that a standard neural network is not an opaque biological mystery, but a parameterized mathematical function designed to map high-dimensional raw inputs into discrete classifications through hierarchical abstraction. Deconstructing Recognition through Layer Hierarchies (5:31) Algorithmically identifying a handwritten digit from an image is notoriously challenging because different variations of the same digit light up entirely distinct pixels. A standard multilayer perceptron addresses this task by structuring neurons into sequential layers. In the baseline demonstration model, an input layer of 784 neurons represents the grayscale brightness (from 0.0 to 1.0) of a 28×28 pixel grid, while an output layer of 10 neurons indicates model confidence for digits 0 through 9. Between these endpoints sit two hidden layers, arbitrarily configured with 16 neurons each to illustrate feature hierarchy. The conceptual justification for stacking layers is abstraction: the human brain naturally breaks digits into subcomponents—such as an upper loop and a straight stroke forming a "9". In an idealized network, intermediate layers progressively aggregate raw pixel intensities into local edges, assemble those edges into recognizable shapes, and ultimately synthesize whole digits. Mechanics of Weights, Biases, and Thresholds (10:38) Connections between layers are mediated by numerical weights and biases. To compute a neuron's activation, the model takes a weighted sum of all incoming activations from the prior layer. Positive weights encourage activation when target pixels illuminate, negative weights penalize activations if light bleeds into peripheral regions, and weights near zero ignore irrelevant areas. To keep the resulting activation bounded between 0 and 1, the raw sum historically passed through a sigmoid squishing function. Because a neuron should not fire merely because its weighted sum nudges above zero, a bias is added as a negative or positive offset before squishing. The weight configuration determines what spatial pattern the neuron seeks, whereas the bias establishes its firing threshold. Connecting 784 inputs to 16 neurons requires 12,544 weights plus 16 biases; across the entire four-layer demonstration network, these connections total nearly 13,000 adjustable parameters that dictate system behavior. Matrix Vectorization and the Composite Network Function (13:26) Handling thousands of individual operations individually is computationally inefficient, leading neural networks to be universally formalized in linear algebra. Layer activations are organized into column vectors, while incoming connections form weight matrices where each row corresponds to a destination neuron. Adding a bias vector to the matrix-vector product and applying the activation function compresses the layer transition into the compact expression σ(Wa + b). This vectorized formulation reveals that neurons are dynamic functions rather than static numeric containers. The complete network is fundamentally a deterministic composite function that consumes an input vector from ℝ⁷⁸⁴ and outputs a vector in ℝ¹⁰. Sanderson emphasizes that building intuition for these matrix operations demystifies the system, shifting practitioners away from viewing architectures as inscrutable black boxes. The Practical Transition from Sigmoids to ReLU (17:03) While early neural networks favored the sigmoid activation function to emulate biological resting and firing states, contemporary architectures rely on alternative squishing functions. Guest researcher Lisha Li explains that while the sigmoid function cleanly bounds values within the unit interval, it severely impedes optimization in deeper networks, rendering parameter training exceedingly difficult. Modern deep learning has largely transitioned to the Rectified Linear Unit (ReLU), defined as f(a) = max(0, a). Under ReLU, a neuron remains inert with an output of zero if the signal falls below the threshold, but passes the signal through linearly once activated. Li observes that despite its simplicity, the empirical adoption of ReLU resolved key training bottlenecks and catalyzed the scaling of deep architectures. Dissecting the plain vanilla multilayer perceptron establishes that machine learning models rely on structured linear algebra rather than black-box magic, setting the stage for algorithmic methods like gradient descent to tune their 13,000 parameters directly from training data.

精炼阅读摘要

Grant Sanderson（3Blue1Brown）指出，人工神经网络并非不可捉摸的黑箱或复杂的生物拟态，其数学本质是一个由多层神经元组成的确定性复合函数，通过近13,000个可调节的权重与偏置参数，将28×28像素的手写数字图像逐层转化为对应的分类概率。 图像识别难题与层次化特征抽象 (0:04) 人类视觉系统能够无缝辨认书写风格迥异的数字，但对于传统算法而言，单纯依赖硬编码规则去解析28×28分辨率（共784个像素）的灰度图像极为困难。Sanderson强调，若要求编写显式规则程序来判定图像究竟代表0到9中的哪一个数字，任务将立刻变得极其艰巨。现代机器学习通过构建多层网络结构，以层次化特征抽象来化解该难题。 在理想设计中，网络逐层拆解视觉模式：底层负责识别局部的微小边缘，中间层将边缘整合为更高阶的几何构件（例如数字“9”上方的封闭圆圈或“4”中的折线段），输出层最终结合这些子结构判定数字类别。这一分层抽象机制不仅局限于图像分类，在语音识别等任务中同样适用，即通过将原始声波逐步组合为音素、音节、词汇直至完整语句。 神经元状态与多层感知机架构 (2:42) 剥离生物学修辞后，人工神经元在数学上仅是承载实数“激活值”（activation）的容器，其取值被约束在0.0到1.0之间。视频展示的标准“经典多层感知机”由四层结构组成：输入层包含784个神经元，分别对应输入图像中各个像素的灰度强度（0代表纯黑，1代表纯白）；输出层由10个神经元构成，各代表0至9的一个数字，激活值接近1表示模型对该类别的置信度极高。 在输入与输出之间设置了两层各包含16个神经元的“隐藏层”。Sanderson澄清，选用两层隐藏层以及每层16个神经元并非源于某种严格的理论必然性，而是出于演示直观性以及屏幕排版美观的教学考量。网络在处理图像时，前序神经元的激活值会引发后续层级的连续级联反应，最终由输出层中激活程度最高的神经元输出预测结果。 权重、偏置与万级参数体系 (10:38) 神经元之间的相互影响通过可调实数“权重”（weights）来实现。权重决定了神经元对前一层特定模式的敏感程度：例如，在边缘检测任务中，目标区域的像素会被赋予正权重，周边干扰区域赋予负权重，而无关背景像素的权重则趋近于零。然而，仅仅计算加权和无法控制神经元何时触发；系统引入了“偏置”（bias）作为偏移常数，用于设定激活门槛，明确加权总和必须达到多高数值时神经元才会显著点亮。 这种结构导致参数规模迅速膨胀：仅从784个输入神经元连接到第一隐藏层的16个神经元，就产生12,544个权重与16个偏置。算上后续隐藏层与输出层的全部连接，整套demo网络总计包含近13,000个独立的权重与偏置。Sanderson指出，机器学习中的“学习”过程，实质上就是利用算法自动调节这13,000个参数旋钮，直至找到能有效执行分类任务的最优数值组合。 线性代数表达与全局函数映射 (13:26) 为了摆脱逐一列举成千上万个标量运算的繁琐表述，全连接神经网络的运算被统一整合为线性代数操作。前一层的激活值被汇总为一个列向量，层间权重被构造成一个权重矩阵，偏置项则归纳为偏置向量。通过将矩阵乘法与偏置向量加法结合，并对计算结果逐元素应用压缩函数，前向传播被极其紧凑地表示为 σ(Wa + b)。 这种向量化表述不仅大幅精简了数学符号，还能直接调用现代高度优化的底层数值计算库进行高吞吐量的并行运算。从全局视角审视，静态容纳数值的神经元转化为动态的计算单元，而整个神经网络本质上就是一个将784维向量映射到10维向量的大型确定性数学函数。Sanderson认为，正是这种参数化复杂度，赋予了网络应对高维度视觉识别挑战所需的表达能力。 激活函数演进：Sigmoid与ReLU对比 (17:03) 传统网络普遍采用Sigmoid函数（σ(x) = 1 / (1 + e⁻ˣ)）作为激活函数，将任意实数输入平滑映射至0与1之间的区间，从而模拟生物神经元在抑制与兴奋状态之间的渐进切换。然而，受邀访谈的深度学习理论学者Lisha Li指出，尽管Sigmoid具有悠久的历史渊源，但在现代深层网络实践中它常常给梯度优化带来极大困难，早已被学术界与工业界视作传统方法。 作为替代，现代深度神经网络已广泛转向线性整流单元（ReLU）。ReLU定义为 f(a) = max(0, a)，其计算逻辑极其简单：当输入低于激活阈值时输出为零，一旦跨过阈值则直接保持线性输出。Li解释，从平滑的Sigmoid转向简单的分段线性ReLU，不仅没有削弱模型的表征能力，反而奇迹般地解决了超深层网络难以训练的工程障碍，成为推动深度学习技术爆发的关键突破之一。 该架构解析阐明了多层感知机如何通过参数化的复合数学函数实现手写数字的初步分类。但纯粹的全连接网络在处理空间不变性与海量特征时仍存在局限；后续研究的关键课题在于，如何借助反向传播与微积分自动化调校这13,000个参数，以及如何利用更深层次的现代网络架构应对复杂真实场景下的认知任务。

Detailed notes — zh

## Background 该视频由知名数学科普频道 3Blue1Brown 于 2017 年 10 月 5 日发布在 YouTube 平台上。视频简介指出，这是其深度学习系列的第一章，旨在探讨神经元的本质、网络设计为多层结构的原因以及驱动整个系统运转的底层数学原理。 ## TL;DR 视频以识别 28×28 像素手写数字为具体任务，由浅入深地拆解了经典全连接前馈神经网络的基本架构与数学本质。作者将神经元定义为保存激活值的容器，阐明了多层结构逐级抽象几何特征的直觉逻辑，并详细剖析了加权求和、偏置设定以及 Sigmoid 挤压函数的计算机制。随后，内容引入线性代数将整层计算紧凑表示为矩阵与向量形式，指出该网络本质上是一个由近 13,000 个参数构成的复合映射函数，并在片尾与嘉宾探讨了现代深度学习从 Sigmoid 转向 ReLU 激活函数的技术原因。 ## Atmosphere & style 整部内容散发着 3Blue1Brown 标志性的典雅、沉静与清晰之美，精致直观的动态数学可视化消解了高维计算的冰冷感。主讲人 Grant Sanderson 的讲解语调温和、循序渐进，善于将复杂的算法拆解为人人皆可理解的认知常识，既有对数学本质的严谨敬畏，又穿插着恰到好处的幽默打趣。片尾与领域专家的轻松对谈则进一步拓宽了工程视界，让观者在层层递进的逻辑推导中感受到一种豁然开朗的纯粹智性愉悦。 ## Bulletin - [01:07](https://www.youtube.com/watch?v=aircAruvnKk&t=67s) 本期视频以 28×28 像素手写数字识别为切入点，通过拆解包含 784 个输入神经元、两层各 16 个隐藏神经元及 10 个输出神经元的全连接前馈神经网络，建立起神经网络的基础数学架构。 - [10:38](https://www.youtube.com/watch?v=aircAruvnKk&t=638s) 一个看似简单的四层手写数字识别网络包含近 13,000 个权重与偏置参数，所谓的机器学习本质上就是让计算机通过优化算法自动寻找这批高维参数的最优配置。 - [13:26](https://www.youtube.com/watch?v=aircAruvnKk&t=806s) 现代神经网络借助线性代数将逐个神经元的线性加权与偏置紧凑表达为矩阵向量形式 $a^{(1)} = \sigma(W a^{(0)} + b)$，这不仅在概念上清晰，更能借助底层硬件优化大幅提升工程计算速度。 - [16:27](https://www.youtube.com/watch?v=aircAruvnKk&t=987s) 特邀嘉宾 Lisha Li 指出，早期模拟生物神经元抑制与兴奋状态的 Sigmoid 函数在深层网络中极难训练，现代深度学习已广泛转向计算极其简便且收敛更快的 ReLU 函数。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=aircAruvnKk&t=4s) 人类视觉系统能够瞬间将不同笔迹的 28×28 像素低分辨率手写数字归纳为抽象概念，但对传统规则编程而言，从像素灰度直接映射到分类数字是一项极其困难的任务。 - [01:07](https://www.youtube.com/watch?v=aircAruvnKk&t=67s) 本教学模型采用四层全连接前馈神经网络，输入层为 784 个像素神经元，中间设计两个各含 16 个神经元的隐藏层，输出层由对应数字 0 到 9 的 10 个神经元组成。 - [02:42](https://www.youtube.com/watch?v=aircAruvnKk&t=162s) 在严格的数学语境中，神经元本质上是存放特定实数的容器；在输入端，其存储的激活值代表该像素的灰度强度（0 代表纯黑，1 代表纯白）。 - [05:31](https://www.youtube.com/watch?v=aircAruvnKk&t=331s) 多层网络的直觉在于实现逐层分层抽象：浅层识别微小边缘与局部笔划，中层将边缘组合为圆环或直线等复合组件，顶层再根据组件组合敲定最终数字分类。 - [08:38](https://www.youtube.com/watch?v=aircAruvnKk&t=518s) 前一层神经元对后一层神经元的连接强弱由权重（Weight）决定，若将各连接权重可视化为像素网格，可通过正负权重的组合实现特定几何边缘的选择性检测。 - [08:38](https://www.youtube.com/watch?v=aircAruvnKk&t=518s) 加权求和的结果可通过 Sigmoid 函数挤压到 0 至 1 的实数区间，以保证输出数值符合下一层神经元激活值的规范定义。 - [10:38](https://www.youtube.com/watch?v=aircAruvnKk&t=638s) 偏置（Bias）相当于设置了神经元被激活的阈值，用于决定加权和需要达到多高时该神经元才会开始产生有意义的激活响应。 - [10:38](https://www.youtube.com/watch?v=aircAruvnKk&t=638s) 在示例网络中，仅输入层到第一隐藏层就包含 784×16=12,544 个权重及 16 个偏置，整个网络累计拥有近 13,000 个需要通过数据进行训练的可调节参数。 - [13:26](https://www.youtube.com/watch?v=aircAruvnKk&t=806s) 借助线性代数，整个层间的前向传播可紧凑地记为 $a^{(1)} = \sigma(W a^{(0)} + b)$，这种矩阵向量运算形式在现代计算库中得到了深度硬件加速。 - [15:17](https://www.youtube.com/watch?v=aircAruvnKk&t=917s) 从宏观视角审视，整个神经网络本质上是一个接收 784 维向量输入、吐出 10 维置信度向量输出的高维确定性数学复合函数。 - [17:03](https://www.youtube.com/watch?v=aircAruvnKk&t=1023s) 虽然 Sigmoid 激活函数在直觉上契合生物抑制与兴奋模型，但由于其在深层网络中容易导致训练困难与梯度消失，现代网络通常优先采用形式更简单的 ReLU（线性整流单元）。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=aircAruvnKk&t=4s) 视觉识别的直觉与计算机视角的困境 手写数字识别对人类大脑来说是一件微不足道的小事。哪怕是一个写得极其潦草、在低至 28×28 像素的极低分辨率下渲染出来的数字“3”，普通人的视网膜与大脑皮层也能不假思索地在瞬间辨认出它。即便把几个完全不同字迹的“3”摆在一起，虽然它们在每个像素位置上的具体明暗数值千差万别，人类眼睛中被激发的感光细胞也大不相同，但视觉皮层仍能不可思议地将这些不同的信号抽象并解析为同一个概念，同时将其他数字准确归类为截然不同的符号。 这种感知上的举重若轻掩盖了其计算本质上的极度复杂。正如 3Blue1Brown 所指出的那样：“But if I told you, hey, sit down and write for me a program that takes in a grid of 28x28 pixels like this and outputs a single number between 0 and 10, telling you what it thinks the digit is, well the task goes from comically trivial to dauntingly difficult.”（但如果我让你坐下来写一个程序，接收这样一个 28×28 的像素网格，然后输出一个 0 到 10 之间的数字来告诉你它认为这是几，这项任务瞬间就会从近乎滑稽的简单变成令人望而生畏的困难。）面对这种挑战，传统的规则编程几乎束手无策，这正是机器学习和神经网络展现力量的舞台。 - 输入分辨率：28×28 像素灰度网格 - 输出目标：识别出 0 到 9 对应的具体数字 - 人脑机制：不同感光细胞激发，经由视觉皮层归纳为统一高阶概念 --- ### [01:07](https://www.youtube.com/watch?v=aircAruvnKk&t=67s) 神经网络的探索目标与基础架构设定 当下人工智能与机器学习的概念早已铺天盖地，但本系列的核心目标并非渲染流行词，而是剥离掉那些浮夸的修辞，在不预设受众拥有专业背景的前提下，将神经网络呈现为纯粹、优美且直观的数学构造。整个讲解分为两部分：首先建立网络的基本架构，理解信息如何流动；随后探讨网络究竟是如何进行所谓的“学习”的。 为了建立扎实的理解，讲解聚焦于最基础、无额外修饰的传统全连接前馈神经网络，并以手写数字识别这一经典问题为抓手。尽管近年来各类新型架构层出不穷，但这种最质朴的形态不仅是理解前沿变体的绝对基石，而且本身就蕴含着足够丰富的数学复杂度。 ```mermaid flowchart LR accTitle: Neural Network Basic Architecture accDescr: Information flows through an input layer, two hidden layers, and an output layer. subgraph Input["输入层"] I["784 个像素神经元<br>(28×28 灰度值)"] end subgraph Hidden["隐藏层"] H1["隐藏层 1<br>(16 个神经元)"] H2["隐藏层 2<br>(16 个神经元)"] end subgraph Output["输出层"] O["10 个输出神经元<br>(数字 0 到 9)"] end I --> H1 H1 --> H2 H2 --> O ``` - 输入层：包含 784 个神经元（对应 28×28 个图像像素） - 隐藏层：共 2 层隐藏层，每层设计为 16 个神经元（便于可视化与分层递进的经验设计） - 输出层：包含 10 个神经元，各自映射 0 至 9 的候选分类 --- ### [02:42](https://www.youtube.com/watch?v=aircAruvnKk&t=162s) 什么是神经元？激活值的数学定义 虽然神经网络受生物大脑启发，但在数学与计算机科学的严格语境中，必须剥离生物学的隐喻。在当前模型中，一个“神经元”（neuron）本质上只是一个用来存放数字的容器，具体而言，存放的是一个介于 0 到 1 之间的实数。 在输入端，28×28 的图像包含 784 个像素，每个像素对应输入层的一个神经元。神经元内存储的数字表示对应像素的灰度值：纯黑像素对应数值 0，纯白像素对应数值 1，不同深浅的灰色则平滑分布在 0 和 1 之间。这个被保存在神经元里的数值被称为它的“激活值”（activation）。在视觉上，我们可以把激活值想象为灯光的亮度——数值越接近 1，该神经元亮起的光芒就越耀眼。 输出层的运作逻辑与此完全相同：10 个神经元各自持有 0 到 1 之间的激活值，代表整个系统对当前图像到底属于 0 到 9 中哪一个数字的置信度。中间隐藏层的神经元同样持有激活值，前一层的激活状态按照特定规律层层递推，决定后一层的激活状态。 - 神经元本质：承载 [0, 1] 区间内单一浮点数的数学容器 - 激活值（Activation）：代表灰度强度或分类置信度的量度 - 图像到神经元的映射：纯黑为 0.0，纯白为 1.0 --- ### [05:31](https://www.youtube.com/watch?v=aircAruvnKk&t=331s) 分层抽象的直觉：为何网络需要多层结构？ 在深究具体的数学公式之前，需要从根本上回答一个问题：为什么多层堆叠的网络结构有希望表现出某种“智能”？人类在识别手写数字时，通常不会死板地比对每个像素，而是自发地将图像拆解为更抽象的几何部件。例如，数字“9”通常由顶部的圆环与右侧的竖线组合而成；数字“8”也是顶部带有圆环，但在底部连着另一个圆环；数字“4”则由三条特定的直线段拼接而成。 基于这种人类认知模式，我们对隐藏层建立了一种极为自然的理想化预期。正如 3Blue1Brown 所构想的那样：“Now in a perfect world, we might hope that each neuron in the second to last layer corresponds with one of these subcomponents, that anytime you feed in an image with, say, a loop up top, like a 9 or an 8, there's some specific neuron whose activation is going to be close to 1.”（在一个理想的世界里，我们可能会希望倒数第二层的每个神经元都对应于这些子组件之一；每当你输入一个图像，比如带有顶部圆环的 9 或 8 时，都会有某个特定神经元的激活值接近于 1。） 这种抽象还可以进一步向下分解：圆环本身是由多条更微小的局部边缘拼接而成的；一条长直线也可以被视作一组连续小边缘构成的特定纹理。若能让第一层隐藏神经元负责识别微小的局部边缘，第二层隐藏神经元负责将这些边缘组合为圆环或笔画，最后一层就能自然地根据子组件的组合来敲定最终的数字。这种分层递进不仅适用于图像识别，也普遍存在于各种智能任务中（如语音识别中将原始声波切分为音素、音节、单词，最终提炼为高阶语意）。 ```mermaid flowchart TD accTitle: Hierarchical Abstraction in Pattern Recognition accDescr: Pixels are grouped into edges, edges form components, and components define digits. subgraph Level1["底层像素"] P["原始 784 像素网格"] end subgraph Level2["浅层几何特征"] E["微小边缘与局部笔划"] end subgraph Level3["中层组合特征"] C["复合结构 (如顶部圆环、长坚线)"] end subgraph Level4["顶层分类"] D["完整数字判定 (如数字 9)"] end P --> E E --> C C --> D ``` --- ### [08:38](https://www.youtube.com/watch?v=aircAruvnKk&t=518s) 边缘检测与权重：加权求和及 Sigmoid 挤压 为了让前一层神经元能够激活后一层表示特定特征的神经元，网络必须具备可以调节的“旋钮”或“参数”。假设第二层的某个神经元专职检测图像中某一特定区域是否存在一条边缘，那么网络应该通过何种机制来实现这种选择性识别？ 最直接的方法是给该神经元与输入层每个神经元之间的每一根连接赋予一个特定的数值，称为“权重”（weight）。随后，计算输入层所有神经元激活值的加权求和（weighted sum）。如果把所有输入权重也排布为一个与原图相同的 28×28 网格，用绿色代表正权重、红色代表负权重，其运作机制就会极其清晰： 1. 将无关区域的所有连接权重设为零，这样加权求和就完全忽略了该区域； 2. 在关注的区域内赋予正权重，这样只要该区域有亮像素，加权求和的值就会增大； 3. 如果希望精准检测边缘，还可以在目标边缘的紧邻边缘区域赋予负权重。只有当中心明亮而周围黑暗时，加权求和的结果才会达到峰值；若周围区域同样明亮，正负项互相抵消，求和值就会被压低。 ```mermaid flowchart LR accTitle: Weighted Sum and Activation Flow accDescr: Pixel activations are weighted and summed, then passed through a sigmoid function. A["前一层各激活值 (a_i)"] -->|"对应权重 (w_i)"| B["加权求和 Σ (w_i * a_i)"] B --> C["Sigmoid 激活函数 σ(z)"] C --> D["下一层神经元激活值 (0 到 1)"] ``` 计算得到的加权求和可能是一个任意大小的正数或负数。然而根据神经元的定义，下一层的激活值必须规整在 0 到 1 之间。因此，必须将这个求和结果输入到一个能够将整条实数轴“挤压”（squish）至 [0, 1] 区间的数学函数中。在经典神经网络中，最常用的变换就是 Sigmoid 函数（亦称逻辑斯蒂曲线，Logistic curve）：当输入为极大的正数时，输出无限趋近于 1；当输入为极大的负数时，输出无限趋近于 0；而在 0 附近时，输出平滑过渡。 ### [10:38](https://www.youtube.com/watch?v=aircAruvnKk&t=638s) 偏置（Bias）的作用与网络参数规模 神经元的激活值（Activation）在本质上是其对应加权和（Weighted Sum）正向程度的一种度量：当输入非常负时，通过 Sigmoid 函数压缩后会无限接近 0；输入非常正时则接近 1，且数值在输入为 0 附近平稳上升。然而在实际任务中，人们往往不希望神经元仅仅因为加权和大于 0 就被点亮。例如，若希望某个神经元仅在加权和大于 10 时才产生显著响应，就必须为它设定一种“倾向于保持静止”的倾向。为了实现这一点，可以在将加权和送入 Sigmoid 函数之前减去 10（即加上 -10）。这个额外添加的常数被称为“偏置”（Bias）。正如 Grant Sanderson 所概括：“权重告诉你第二层中的这个神经元正在捕捉何种像素模式，而偏置则告诉你加权和需要达到多高，该神经元才会开始产生有意义的激活。”（“So the weights tell you what pixel pattern this neuron in the second layer is picking up on, and the bias tells you how high the weighted sum needs to be before the neuron starts getting meaningfully active.”） ```mermaid flowchart LR accTitle: 单个神经元的计算流程 accDescr: 前一层激活值与权重相乘求和，加上偏置后经由 Sigmoid 函数输出该神经元的激活值。 subgraph Input ["上一层输入"] A["激活值向量 a"] end subgraph Calc ["线性计算与偏移"] W["加权求和: W · a"] B["加上偏置: + b"] end subgraph Activation ["非线性激活"] S["Sigmoid 函数: σ(...)"] Out["当前神经元激活值 a'"] end A --> W W --> B B --> S S --> Out ``` 每一个神经元都对应着繁重的参数量。第二层的 16 个隐藏神经元中，每一个都要与第一层的全部 784 个像素神经元相连，这意味着单是这一层之间就存在 784 × 16 个权重以及 16 个偏置。沿着网络向后推进，其他层与层之间同样布满了海量的权重与偏置连接。将整个示例网络所有层之间的参数累加起来，总共包含将近 13,000 个权重和偏置。所谓的机器学习或网络训练，正是让计算机自动寻找到这 13,000 个“旋钮与转盘”的合适设置，使整个系统能够切实解决手头的问题。 - 第一层到第二层包含 784 × 16 = 12,544 个权重，以及 16 个独立的偏置。 - 整个四层手写数字识别网络累计拥有近 13,000 个可调节参数。 - 设想纯靠人工手动去逐个调节这 13,000 个参数虽然“既有趣又有些令人毛骨悚然”，但探究每个参数的具体物理意义，能让人在网络表现失常或意外生效时，拥有清晰的排查与调整切入点，避免把网络当作不可解释的黑盒。 ### [13:26](https://www.youtube.com/watch?v=aircAruvnKk&t=806s) 线性代数表达：矩阵与向量的紧凑表示 如果为每一个神经元逐个书写加权求和与偏置的标量公式，数学表达会变得极其繁琐。在神经网络的标准文献与工程实践中，通常借助线性代数将这些计算紧密组织起来。首先，将某一层中所有神经元的激活值按列排布成一个列向量；接着，把连接这两层的所有权重整理成一个权重矩阵（Weight Matrix），矩阵中的每一行分别对应上一层所有神经元与下一层某个特定神经元之间的连接权重。如此一来，下一层神经元的所有加权求和，恰好就是该权重矩阵与激活值向量的矩阵-向量乘积（Matrix-Vector Product）。 在得到加权和向量后，再将该层的所有偏置组织为一个偏置向量并直接相加。最后，在整个括号外部套上 Sigmoid 符号 $\sigma$，表示对括号内计算出的向量中的每一个分量分别执行 Sigmoid 压缩运算。通过符号化抽象，层与层之间的复杂映射被浓缩为极其精炼的数学式：$a^{(1)} = \sigma(W a^{(0)} + b)$。这种矩阵表达不仅在概念上简洁清晰，在工程实现中更是至关重要，因为主流计算库对矩阵乘法进行了极深度的底层硬件优化，能够大幅缩减代码量并成倍提升运算速度。 - 权重矩阵的行数对应下一层神经元数量，列数对应当前层神经元数量。 - 偏置被集中表示为一个与下一层神经元数量相等的列向量。 - 外层的 Sigmoid 函数以元素级（Element-wise）方式作用于线性计算生成的向量分量。 ### [15:17](https://www.youtube.com/watch?v=aircAruvnKk&t=917s) 神经网络的本质：一个宏大的高维函数 虽然此前将神经元拟人化为“保存数值的容器”，但由于它们保存的具体数值完全取决于输入图像，因此更严谨的视角是将每个神经元视为一个独立的数学函数——它接收上一层所有神经元的输出，并计算吐出一个介于 0 和 1 之间的数值。推广到整体结构，整个神经网络本质上也是一个庞大而精密的复合函数：它接收代表图像像素的 784 个数值作为输入向量，经过多层矩阵乘法与非线性函数的交替迭代，最终吐出代表各类可能性的 10 个数值作为输出向量。 Grant Sanderson 强调，这种看似令人望而生畏的复杂度反而是合理的：“从某种角度来说，它看起来如此复杂反而让人感到安心。我的意思是，如果它再简单一点，我们还能抱有什么希望让它承担起识别手写数字的重任呢？”（“And in a way it's kind of reassuring that it looks complicated. I mean if it were any simpler, what hope would we have that it could take on the challenge of recognizing digits?”）正是这 13,000 个参数构成的庞大空间，赋予了网络拟合高度非线性特征的能力。 - 输入端：接收 784 维向量（对应 28×28 像素的灰度图像）。 - 输出端：输出 10 维向量（对应数字 0 到 9 的激活评分）。 - 内部机制：由 13,000 个参数参数化、通过多步矩阵变换级联而成的确定性数学映射。 ### [16:27](https://www.youtube.com/watch?v=aircAruvnKk&t=987s) 订阅杂谈与特别嘉宾讨论：Sigmoid 转向 ReLU 在预告下一期将深入探讨反向传播算法以及网络如何依靠数据自主学习权重偏置之后，Grant 打趣道，与其机械地喊大家点击订阅以获取推送通知，倒不如实话实说：“也许更坦白地说，我应该建议大家订阅，以便让驱动 YouTube 推荐算法的神经网络做好准备，相信你确实想看到这个频道的内容被推荐给你。”（“Maybe more honestly I should say subscribe so that the neural networks that underlie YouTube's recommendation algorithm are primed to believe that you want to see content from this channel get recommended to you.”） 随后，Grant 邀请了在深度学习理论方向取得博士学位、现就职于风投机构 Amplify Partners 的嘉宾 Lisha Li 共同探讨激活函数的演变。针对为什么现代深度学习网络不再青睐 Sigmoid 函数这一现象，两人展开了解读： | 特性 | Sigmoid 函数 | ReLU（Rectified Linear Unit，线性整流单元） | |:--- |:--- |:--- | | **数学形式** | $\sigma(x) = \frac{1}{1 + e^{-x}}$ | $\max(0, a)$ | | **生物拟态逻辑** | 将输入压缩至 0 到 1 之间，模拟神经元处于抑制或兴奋的连续状态。 | 同样受生物学启发：未达阈值时完全不激活（输出 0），超过阈值后则呈线性响应（恒等函数）。 | | **训练表现** | 在网络层数加深时极难训练，梯度极易消失。 | 计算形式极为简化，在极深的网络架构中极易收敛，显著提升了深度网络的训练效率。 | Lisha Li 指出，Sigmoid 函数在早期受生物学隐喻启发而被广泛使用，但在实际工程中，它让深度网络的训练变得异常困难。研究人员后来尝试使用极为简单的截断形式 $\max(0, a)$，即 ReLU 函数；当神经元未被激活时输出为 0，一旦跨过阈值便保持恒等输出。这种看似简化的激活方式在实践中取得了出人意料的卓越效果，从而在现代深度神经网络中几乎全面取代了 Sigmoid 函数。 ## Notable quotes > [10:38](https://www.youtube.com/watch?v=aircAruvnKk&t=638s) 权重告诉你第二层中的这个神经元正在捕捉何种像素模式，而偏置则告诉你加权和需要达到多高，该神经元才会开始产生有意义的激活。 — Grant Sanderson > [15:17](https://www.youtube.com/watch?v=aircAruvnKk&t=917s) 从某种角度来说，它看起来如此复杂反而让人感到安心。我的意思是，如果它再简单一点，我们还能抱有什么希望让它承担起识别手写数字的重任呢？ — Grant Sanderson ## Mentioned 3Blue1Brown, Grant Sanderson, Lisha Li, Amplify Partners, Michael Nielsen, Chris Olah, Welch Labs, Ian Goodfellow, Yoshua Bengio, Aaron Courville, Distill, YouTube, manim, Vincent Rubinetti, Kevin Pluck ## Video description What are the neurons, why are there layers, and what is the math underlying it? Help fund future projects: https://www.patreon.com/3blue1brown Written/interactive form of this series: https://www.3blue1brown.com/topics/neural-networks Additional funding for this project was provided by Amplify Partners For those who want to learn more, I highly recommend the book by Michael Nielsen that introduces neural networks and deep learning: https://goo.gl/Zmczdy There are two neat things about this book. First, it's available for free, so consider joining me in making a donation to Nielsen if you get something out of it. And second, it's centered around walking through some code and data, which you can download yourself, and which covers the same example that I introduced in this video. Yay for active learning! https://github.com/mnielsen/neural-networks-and-deep-learning I also highly recommend Chris Olah's blog: http://colah.github.io/ For more videos, Welch Labs also has some great series on machine learning: https://youtu.be/i8D90DkCLhI https://youtu.be/bxe2T-V8XRs For those of you looking to go *even* deeper, check out the text "Deep Learning" by Goodfellow, Bengio, and Courville. Also, the publication Distill is just utterly beautiful: https://distill.pub/ Lion photo by Kevin Pluck Звуковая дорожка на русском языке: Влад Бурмистров. Thanks to these viewers for their contributions to translations German: @fpgro Hebrew: Omer Tuchfeld Hungarian: Máté Kaszap Italian: @teobucci, Teo Bucci ----------------- Timeline: 0:00 - Introduction example 1:07 - Series preview 2:42 - What are neurons? 3:35 - Introducing layers 5:31 - Why layers? 8:38 - Edge detection example 11:34 - Counting weights and biases 12:30 - How learning relates 13:26 - Notation and linear algebra 15:17 - Recap 16:27 - Some final words 17:03 - ReLU vs Sigmoid Correction 14:45 - The final index on the bias vector should be "k" ------------------ Animations largely made using manim, a scrappy open source python library. https://github.com/3b1b/manim If you want to check it out, I feel compelled to warn you that it's not the most well-documented tool, and has many other quirks you might expect in a library someone wrote with only their own use in mind. Music by Vincent Rubinetti. Download the music on Bandcamp: https://vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown Stream the music on Spotify: https://open.spotify.com/album/1dVyjwS8FBqXhRunaG5W5u If you want to contribute translated subtitles or to help review those that have already been made by others and need approval, you can click the gear icon in the video and go to subtitles/cc, then "add subtitles/cc". I really appreciate those who do this, as it helps make the lessons accessible to more people. ------------------ 3blue1brown is a channel about animating math, in all senses of the word animate. And you know the drill with YouTube, if you want to stay posted on new videos, subscribe, and click the bell to receive notifications (if you're into that). If you are new to this channel and want to see more, a good place to start is this playlist: http://3b1b.co/recommended Various social media stuffs: Website: https://www.3blue1brown.com Twitter: https://twitter.com/3Blue1Brown Patreon: https://patreon.com/3blue1brown Facebook: https://www.facebook.com/3blue1brown Reddit: https://www.reddit.com/r/3Blue1Brown [Source](https://www.youtube.com/watch?v=aircAruvnKk)

Source quotes

[10:38] Grant Sanderson: 权重告诉你第二层中的这个神经元正在捕捉何种像素模式，而偏置则告诉你加权和需要达到多高，该神经元才会开始产生有意义的激活。 [15:17] Grant Sanderson: 从某种角度来说，它看起来如此复杂反而让人感到安心。我的意思是，如果它再简单一点，我们还能抱有什么希望让它承担起识别手写数字的重任呢？

Source quotes

[17:03] Lisha Li: Using sigmoids didn't help training or it was very difficult to train at some point and people just tried ReLU and it happened to work very well for these incredibly deep neural networks.

Original transcript

[0:04] This is a 3. [0:06] It's sloppily written and rendered at an extremely low resolution of 28x28 pixels, [0:10] but your brain has no trouble recognizing it as a 3. [0:14] And I want you to take a moment to appreciate how [0:16] crazy it is that brains can do this so effortlessly. [0:19] I mean, this, this and this are also recognizable as 3s, [0:22] even though the specific values of each pixel is very different from one [0:27] image to the next. [0:28] The particular light-sensitive cells in your eye that are firing when you [0:32] see this 3 are very different from the ones firing when you see this 3. [0:37] But something in that crazy-smart visual cortex of yours resolves these as representing [0:42] the same idea, while at the same time recognizing other images as their own distinct [0:47] ideas. [0:49] But if I told you, hey, sit down and write for me a program that takes in a grid of [0:54] 28x28 pixels like this and outputs a single number between 0 and 10, [0:59] telling you what it thinks the digit is, well the task goes from comically trivial to [1:04] dauntingly difficult. [1:07] Unless you've been living under a rock, I think I hardly need to motivate the relevance [1:10] and importance of machine learning and neural networks to the present and to the future. [1:15] But what I want to do here is show you what a neural network actually is, [1:18] assuming no background, and to help visualize what it's doing, [1:22] not as a buzzword but as a piece of math. [1:25] My hope is that you come away feeling like the structure itself is motivated, [1:28] and to feel like you know what it means when you read, [1:31] or you hear about a neural network quote-unquote learning. [1:35] This video is just going to be devoted to the structure component of that, [1:38] and the following one is going to tackle learning. [1:40] What we're going to do is put together a neural [1:43] network that can learn to recognize handwritten digits. [1:49] This is a somewhat classic example for introducing the topic, [1:52] and I'm happy to stick with the status quo here, [1:54] because at the end of the two videos I want to point you to a couple good [1:57] resources where you can learn more, and where you can download the code that [2:00] does this and play with it on your own computer. [2:05] There are many many variants of neural networks, [2:07] and in recent years there's been sort of a boom in research towards these variants, [2:12] but in these two introductory videos you and I are just going to look at the simplest [2:16] plain vanilla form with no added frills. [2:19] This is kind of a necessary prerequisite for understanding any of the more powerful [2:23] modern variants, and trust me it still has plenty of complexity for us to wrap our minds [2:28] around. [2:29] But even in this simplest form it can learn to recognize handwritten digits, [2:33] which is a pretty cool thing for a computer to be able to do. [2:37] And at the same time you'll see how it does fall [2:39] short of a couple hopes that we might have for it. [2:43] As the name suggests neural networks are inspired by the brain, but let's break that down. [2:48] What are the neurons, and in what sense are they linked together? [2:52] Right now when I say neuron all I want you to think about is a thing that holds a number, [2:58] specifically a number between 0 and 1. [3:00] It's really not more than that. [3:03] For example the network starts with a bunch of neurons corresponding to [3:08] each of the 28x28 pixels of the input image, which is 784 neurons in total. [3:14] Each one of these holds a number that represents the grayscale value of the [3:19] corresponding pixel, ranging from 0 for black pixels up to 1 for white pixels. [3:25] This number inside the neuron is called its activation, [3:28] and the image you might have in mind here is that each neuron is lit up when its [3:32] activation is a high number. [3:36] So all of these 784 neurons make up the first layer of our network. [3:46] Now jumping over to the last layer, this has 10 neurons, [3:49] each representing one of the digits. [3:52] The activation in these neurons, again some number that's between 0 and 1, [3:56] represents how much the system thinks that a given image corresponds with a given digit. [4:03] There's also a couple layers in between called the hidden layers, [4:06] which for the time being should just be a giant question mark for [4:09] how on earth this process of recognizing digits is going to be handled. [4:14] In this network I chose two hidden layers, each one with 16 neurons, [4:17] and admittedly that's kind of an arbitrary choice. [4:21] To be honest I chose two layers based on how I want to motivate the structure in [4:24] just a moment, and 16, well that was just a nice number to fit on the screen. [4:28] In practice there is a lot of room for experiment with a specific structure here. [4:33] The way the network operates, activations in one [4:35] layer determine the activations of the next layer. [4:39] And of course the heart of the network as an information processing mechanism comes down [4:43] to exactly how those activations from one layer bring about activations in the next [4:48] layer. [4:49] It's meant to be loosely analogous to how in biological networks of neurons, [4:53] some groups of neurons firing cause certain others to fire. [4:58] Now the network I'm showing here has already been trained to recognize digits, [5:01] and let me show you what I mean by that. [5:03] It means if you feed in an image, lighting up all 784 neurons of the input layer [5:08] according to the brightness of each pixel in the image, [5:11] that pattern of activations causes some very specific pattern in the next layer [5:16] which causes some pattern in the one after it, [5:18] which finally gives some pattern in the output layer. [5:22] And the brightest neuron of that output layer is the network's choice, [5:26] so to speak, for what digit this image represents. [5:32] And before jumping into the math for how one layer influences the next, [5:36] or how training works, let's just talk about why it's even reasonable [5:40] to expect a layered structure like this to behave intelligently. [5:44] What are we expecting here? [5:45] What is the best hope for what those middle layers might be doing? [5:48] Well, when you or I recognize digits, we piece together various components. [5:54] A 9 has a loop up top and a line on the right. [5:57] An 8 also has a loop up top, but it's paired with another loop down low. [6:01] A 4 basically breaks down into three specific lines, and things like that. [6:07] Now in a perfect world, we might hope that each neuron in the second [6:11] to last layer corresponds with one of these subcomponents, [6:14] that anytime you feed in an image with, say, a loop up top, [6:18] like a 9 or an 8, there's some specific neuron whose activation is [6:22] going to be close to 1. [6:24] And I don't mean this specific loop of pixels, [6:26] the hope would be that any generally loopy pattern towards the top sets off this neuron. [6:32] That way, going from the third layer to the last one just requires [6:36] learning which combination of subcomponents corresponds to which digits. [6:41] Of course, that just kicks the problem down the road, [6:43] because how would you recognize these subcomponents, [6:45] or even learn what the right subcomponents should be? [6:48] And I still haven't even talked about how one layer influences the next, [6:51] but run with me on this one for a moment. [6:53] Recognizing a loop can also break down into subproblems. [6:57] One reasonable way to do this would be to first [6:59] recognize the various little edges that make it up. [7:03] Similarly, a long line, like the kind you might see in the digits 1 or 4 or 7, [7:08] is really just a long edge, or maybe you think of it as a certain pattern of several [7:13] smaller edges. [7:15] So maybe our hope is that each neuron in the second layer of [7:18] the network corresponds with the various relevant little edges. [7:23] Maybe when an image like this one comes in, it lights up all of the [7:27] neurons associated with around 8 to 10 specific little edges, [7:31] which in turn lights up the neurons associated with the upper loop [7:35] and a long vertical line, and those light up the neuron associated with a 9. [7:40] Whether or not this is what our final network actually does is another question, [7:44] one that I'll come back to once we see how to train the network, [7:47] but this is a hope that we might have, a sort of goal with the layered structure [7:51] like this. [7:53] Moreover, you can imagine how being able to detect edges and patterns [7:56] like this would be really useful for other image recognition tasks. [8:00] And even beyond image recognition, there are all sorts of intelligent [8:04] things you might want to do that break down into layers of abstraction. [8:08] Parsing speech, for example, involves taking raw audio and picking out distinct sounds, [8:12] which combine to make certain syllables, which combine to form words, [8:16] which combine to make up phrases and more abstract thoughts, etc. [8:21] But getting back to how any of this actually works, [8:23] picture yourself right now designing how exactly the activations in one layer might [8:27] determine the activations in the next. [8:30] The goal is to have some mechanism that could conceivably combine pixels into edges, [8:35] or edges into patterns, or patterns into digits. [8:39] And to zoom in on one very specific example, let's say the hope [8:43] is for one particular neuron in the second layer to pick up [8:46] on whether or not the image has an edge in this region here. [8:51] The question at hand is what parameters should the network have? [8:55] What dials and knobs should you be able to tweak so that it's expressive [8:59] enough to potentially capture this pattern, or any other pixel pattern, [9:03] or the pattern that several edges can make a loop, and other such things? [9:08] Well, what we'll do is assign a weight to each one of the [9:11] connections between our neuron and the neurons from the first layer. [9:16] These weights are just numbers. [9:18] Then take all of those activations from the first layer [9:21] and compute their weighted sum according to these weights. [9:27] I find it helpful to think of these weights as being organized into a [9:31] little grid of their own, and I'm going to use green pixels to indicate [9:34] positive weights, and red pixels to indicate negative weights, [9:37] where the brightness of that pixel is some loose depiction of the weight's value. [9:42] Now if we made the weights associated with almost all of the pixels zero [9:46] except for some positive weights in this region that we care about, [9:50] then taking the weighted sum of all the pixel values really just amounts [9:53] to adding up the values of the pixel just in the region that we care about. [9:59] And if you really wanted to pick up on whether there's an edge here, [10:02] what you might do is have some negative weights associated with the surrounding pixels. [10:07] Then the sum is largest when those middle pixels [10:10] are bright but the surrounding pixels are darker. [10:14] When you compute a weighted sum like this, you might come out with any number, [10:18] but for this network what we want is for activations to be some value between 0 and 1. [10:24] So a common thing to do is to pump this weighted sum into some function [10:28] that squishes the real number line into the range between 0 and 1. [10:32] And a common function that does this is called the sigmoid function, [10:35] also known as a logistic curve. [10:38] Basically very negative inputs end up close to 0, positive inputs end up close to 1, [10:43] and it just steadily increases around the input 0. [10:49] So the activation of the neuron here is basically a [10:52] measure of how positive the relevant weighted sum is. [10:57] But maybe it's not that you want the neuron to [10:59] light up when the weighted sum is bigger than 0. [11:02] Maybe you only want it to be active when the sum is bigger than say 10. [11:06] That is, you want some bias for it to be inactive. [11:11] What we'll do then is just add in some other number like negative 10 to this [11:15] weighted sum before plugging it through the sigmoid squishification function. [11:20] That additional number is called the bias. [11:23] So the weights tell you what pixel pattern this neuron in the second [11:27] layer is picking up on, and the bias tells you how high the weighted [11:31] sum needs to be before the neuron starts getting meaningfully active. [11:36] And that is just one neuron. [11:38] Every other neuron in this layer is going to be connected to [11:42] all 784 pixel neurons from the first layer, and each one of [11:46] those 784 connections has its own weight associated with it. [11:51] Also, each one has some bias, some other number that you add [11:54] on to the weighted sum before squishing it with the sigmoid. [11:58] And that's a lot to think about! [11:59] With this hidden layer of 16 neurons, that's a total of 784 times 16 weights, [12:06] along with 16 biases. [12:08] And all of that is just the connections from the first layer to the second. [12:12] The connections between the other layers also have [12:14] a bunch of weights and biases associated with them. [12:18] All said and done, this network has almost exactly 13,000 total weights and biases. [12:23] 13,000 knobs and dials that can be tweaked and turned [12:27] to make this network behave in different ways. [12:31] So when we talk about learning, what that's referring to is [12:34] getting the computer to find a valid setting for all of these [12:37] many many numbers so that it'll actually solve the problem at hand. [12:42] One thought experiment that is at once fun and kind of horrifying is to imagine sitting [12:47] down and setting all of these weights and biases by hand, [12:50] purposefully tweaking the numbers so that the second layer picks up on edges, [12:54] the third layer picks up on patterns, etc. [12:56] I personally find this satisfying rather than just treating the network as a total black [13:01] box, because when the network doesn't perform the way you anticipate, [13:04] if you've built up a little bit of a relationship with what those weights and biases [13:09] actually mean, you have a starting place for experimenting with how to change the [13:13] structure to improve. [13:14] Or when the network does work but not for the reasons you might expect, [13:18] digging into what the weights and biases are doing is a good way to challenge [13:22] your assumptions and really expose the full space of possible solutions. [13:26] By the way, the actual function here is a little cumbersome to write down, [13:29] don't you think? [13:32] So let me show you a more notationally compact way that these connections are represented. [13:37] This is how you'd see it if you choose to read up more about neural networks. [13:40] Organize all of the activations from one layer into a column as a vector. [13:48] Then organize all of the weights as a matrix, where each row of that matrix corresponds [13:50] to the connections between one layer and a particular neuron in the next layer. [13:58] What that means is that taking the weighted sum of the activations in [14:02] the first layer according to these weights corresponds to one of the [14:05] terms in the matrix vector product of everything we have on the left here. [14:14] By the way, so much of machine learning just comes down to having a good [14:17] grasp of linear algebra, so for any of you who want a nice visual [14:21] understanding for matrices and what matrix vector multiplication means, [14:24] take a look at the series I did on linear algebra, especially chapter 3. [14:29] Back to our expression, instead of talking about adding the bias to each one of [14:33] these values independently, we represent it by organizing all those biases into [14:38] a vector, and adding the entire vector to the previous matrix vector product. [14:43] Then as a final step, I'll wrap a sigmoid around the outside here, [14:46] and what that's supposed to represent is that you're going to apply the [14:50] sigmoid function to each specific component of the resulting vector inside. [14:55] So once you write down this weight matrix and these vectors as their own symbols, [15:00] you can communicate the full transition of activations from one layer to the next in an [15:05] extremely tight and neat little expression, and this makes the relevant code both a lot [15:10] simpler and a lot faster, since many libraries optimize the heck out of matrix [15:14] multiplication. [15:17] Remember how earlier I said these neurons are simply things that hold numbers? [15:22] Well of course the specific numbers that they hold depends on the image you feed in, [15:27] so it's actually more accurate to think of each neuron as a function, [15:31] one that takes in the outputs of all the neurons in the previous layer and spits out a [15:36] number between 0 and 1. [15:39] Really the entire network is just a function, one that takes in [15:43] 784 numbers as an input and spits out 10 numbers as an output. [15:47] It's an absurdly complicated function, one that involves 13,000 parameters [15:51] in the forms of these weights and biases that pick up on certain patterns, [15:55] and which involves iterating many matrix vector products and the sigmoid [15:59] squishification function, but it's just a function nonetheless. [16:03] And in a way it's kind of reassuring that it looks complicated. [16:07] I mean if it were any simpler, what hope would we have [16:09] that it could take on the challenge of recognizing digits? [16:13] And how does it take on that challenge? [16:15] How does this network learn the appropriate weights and biases just by looking at data? [16:20] Well that's what I'll show in the next video, and I'll also dig a little [16:23] more into what this particular network we're seeing is really doing. [16:27] Now is the point I suppose I should say subscribe to stay notified [16:30] about when that video or any new videos come out, [16:33] but realistically most of you don't actually receive notifications from YouTube, do you? [16:38] Maybe more honestly I should say subscribe so that the neural networks [16:41] that underlie YouTube's recommendation algorithm are primed to believe [16:44] that you want to see content from this channel get recommended to you. [16:48] Anyway, stay posted for more. [16:50] Thank you very much to everyone supporting these videos on Patreon. [16:54] I've been a little slow to progress in the probability series this summer, [16:57] but I'm jumping back into it after this project, [16:59] so patrons you can look out for updates there. [17:03] To close things off here I have with me Lisha Li who did her PhD work on the [17:07] theoretical side of deep learning and who currently works at a venture capital [17:10] firm called Amplify Partners who kindly provided some of the funding for this video. [17:15] So Lisha one thing I think we should quickly bring up is this sigmoid function. [17:19] As I understand it early networks use this to squish the relevant weighted [17:23] sum into that interval between zero and one, you know kind of motivated [17:26] by this biological analogy of neurons either being inactive or active. [17:30] Exactly. But relatively few modern networks actually use sigmoid anymore. [17:34] Yeah. It's kind of old school right? [17:35] Yeah or rather ReLU seems to be much easier to train. [17:39] And ReLU, ReLU stands for rectified linear unit? [17:42] Yes it's this kind of function where you're just taking a max of zero [17:47] and a where a is given by what you were explaining in the video and [17:52] what this was sort of motivated from I think was a partially by a [17:56] biological analogy with how neurons would either be activated or not. [18:01] And so if it passes a certain threshold it would be the identity function but if it did [18:06] not then it would just not be activated so it'd be zero so it's kind of a simplification. [18:11] Using sigmoids didn't help training or it was very difficult to [18:15] train at some point and people just tried ReLU and it happened [18:20] to work very well for these incredibly deep neural networks. [18:25] All right thank you Lisha.