The spelled-out intro to neural networks and backpropagation: building micrograd

神经网络与反向传播详尽入门：构建 micrograd Karpathy 通过从第一性原理推导核心微积分概念，揭开了深度学习的神秘面纱。他从导数的数值近似以及多元链式法则的手动求值讲起，逐步构建起 `Value` 引擎，重载了 Python 算术运算符，追踪动态有向无环计算图，并利用拓扑排序实现了反向模式自动微分的自动化。在处理了梯度累加和数学运算分解等关键边缘情况后，他构建了一个包含神经元、网络层和多层感知机的面向对象神经网络框架。最后，他演示了如何通过梯度下降在二分类数据集上训练多层感知机，排查了诸如梯度清零等隐蔽错误，并直接对照 PyTorch 底层的 C++ 和 CUDA 实现验证了 micrograd 的数学准确性。 Andrej Karpathy 演示了如何使用 Micrograd 仅凭约 150 行 Python 代码从零完整实现深度神经网络的反向传播，避开复杂的张量数学，以此证明自动微分从根本上是在标量有向无环图上运行的。 (4:33) 核心自动求导引擎依赖于计算图的反向拓扑排序与局部梯度累加（+=），从而避免当变量分流至多条路径时发生静默的梯度覆盖。 (82:28) 对照 PyTorch 验证 Micrograd 确认了数值梯度的一致性，证明多维张量框架底层使用的是完全相同的链式法则微积分，仅仅是为了并行硬件执行而进行了优化。 (99:31) 深入查看 PyTorch 的生产级代码库可以发现，复杂的生产级代码库在 CPU 和 CUDA 内核中实现的局部导数公式完全相同，例如 tanh 反向传播中的 `a * (1 - b * b)`。 (141:10)

Andrej Karpathy demonstrates that modern deep learning libraries rely on backpropagation—the recursive application of the calculus chain rule over a directed acyclic graph—by implementing the minimal scalar autograd engine micrograd from scratch. The Value Class and Computational Graph Construction (19:09) Rather than relying on manual symbolic calculus expansions, which Karpathy notes become unfeasible across networks containing thousands or billions of operations, micrograd encapsulates mathematical expressions within a custom `Value` class. Each `Value` object stores a scalar float in `.data`, initializes an incoming gradient tracker `.grad` to 0.0, and records child operands and operations via internal pointers (`._prev` and `._op`). By overloading Python's arithmetic dunder methods—such as `__add__`, `__mul__`, and fallback routines like `__rmul__`—expressions dynamically construct a directed acyclic graph (DAG) during the forward pass. Local Derivatives, Closures, and Topological Ordering (69:02) Backpropagation automates the chain rule by assigning each graph node an internal `_backward` closure that computes local gradients and distributes them to parent nodes. For addition, local derivatives evaluate to 1.0, making the operator act as a pure gradient router, whereas multiplication scales the upstream gradient by the data value of the opposing operand. Karpathy highlights that operations can be defined at arbitrary abstraction levels: a complex activation like $\tanh$ can be implemented as a monolithic node using the derivative identity $1 - \tanh^2(x)$ or broken into elementary exponentiation, subtraction, and division operations, both producing identical leaf gradients. Fixing Gradient Accumulation and Orchestrating Execution (82:28) A critical bug emerges when variables are used more than once in an expression graph. Assigning gradients directly via `=` causes subsequent operations to overwrite earlier contributions, violating the multivariate chain rule. Micrograd resolves this by accumulating gradients using `+=`. To fully automate the backward traversal, the engine performs a topological sort on the DAG, ordering nodes so that every downstream consumer executes its backward closure before the upstream dependency processes incoming gradients. Constructing Multi-Layer Perceptrons and the Optimization Loop (103:55) Building atop the scalar autograd engine, Karpathy structures an artificial neural network hierarchy using `Neuron`, `Layer`, and `MLP` classes patterned after `torch.nn`. Neurons evaluate weighted sums with an added bias before applying a non-linearity. To optimize model weights on a toy classification dataset, Karpathy designs a training loop consisting of forward evaluation, loss computation via mean squared error, backpropagation via `loss.backward()`, and parameter updates via gradient descent. Karpathy highlights a classic debugging pitfall: failing to reset every parameter's gradient to zero before each backward pass causes gradients to improperly accumulate across successive iterations. Karpathy verifies micrograd's numerical accuracy by cross-checking its scalar outputs and leaf gradients directly against PyTorch tensors cast to double precision. While micrograd is restricted to scalar computations and PyTorch operates across multi-dimensional tensors, both frameworks share the same underlying architecture of forward evaluation graphs, local backward closures, and iterative parameter updates.

Andrej Karpathy 通过从零构建轻量级标量自动求导引擎 micrograd，系统剖析了现代深度学习框架的核心底层逻辑，指出神经网络本质上是由微积分链式法则驱动的有向无环图计算与反向传播优化过程。 标量引擎设计与有向无环图构建 (1:24) 在深度学习底层实现中，复杂的模型本质上是由海量基础数学运算组合而成的表达式。Karpathy 摒弃了直接处理高维张量的黑盒抽象，从最基础的标量数据结构入手，在 Python 中封装了核心类 Value。该对象维护底层浮点数值 .data、输出对该节点的局部偏导数 .grad 以及用于追踪图结构的父节点集合 _prev 和操作符标识 _op。通过对加法 __add__、乘法 __mul__ 等双下划线方法进行重载，系统在执行诸如 a * b + c 等前向运算的同时，隐式构建了有向无环计算图（DAG），从而为后续逆向遍历求导提供了完整的拓扑依赖关系。 微积分链式法则与局部闭包自动求导 (43:07) 反向传播在数学上体现为微积分链式法则的递归应用。在单个运算节点上，加法操作充当“梯度路由器”，将其接收到的下游梯度等额分流给各个输入分支；而乘法操作则遵循乘积法则，将下游梯度乘以另一操作数的数值。在 micrograd 中，Karpathy 为每个 Value 对象绑定了内部私有闭包 _backward，该函数将上游累积梯度 out.grad 与当前运算的局部导数相乘，并将计算结果直接回传给各父节点。当整个前向图延伸至最终的标量损失 L 时，系统将根节点的基础梯度设为 1.0（即 dL/dL），便能沿着反向拓扑流启动自动化求导。 拓扑排序与多元链式求导中的梯度累加 (77:32) 为了实现单次调用即完成全图求导，引擎引入了拓扑排序算法。由于一个节点必须在其所有下游消费者完成反向传播后才能最终确定自身梯度，拓扑排序为全图构建了严格的线性依赖序列，确保逆向遍历时每个闭包的调用顺序完全精确。在此过程中，Karpathy 指出了一个严重的隐性缺陷：当同一个变量在计算图中被多次使用时（例如分支汇聚或 a + a），如果局部梯度直接通过赋值符号 = 覆盖写入，先执行路径的导数信息就会被后执行路径彻底抹除。根据多元微积分的链式法则，来自不同计算路径的分支梯度必须通过 += 累加叠加，这一机制不仅修复了图分支错误，也构成了正确实现多路反向传播的前提。 运算粒度拆解与神经网络基元抽象 针对激活函数的处理，Karpathy 证明了算子抽象层级的灵活性：无论是将双曲正切函数 tanh 作为包含特定解析导数的单一复合节点，还是将其拆解为指数运算 exp、除法和减法等底层基本原子算子的长调用链，叶子节点最终反向计算出的偏导数均严格一致。在此引擎之上，系统自底向上构建了标准神经网络基元：单个人工神经元 Neuron 将输入向量与权重点积并添加偏置后通过非线性函数压缩；多神经元并行组成 Layer；多层级联则构成多层感知机 MLP。该架构通过统一的 .parameters() 方法递归收集全网络 41 个标量参数，在结构上与 PyTorch 的 nn.Module 保持对应，并通过双精度浮点数比对验证了数值一致性。 完整优化循环与工程训练陷阱 (128:38) 完整的模型训练工作流被归纳为三个顺序阶段：前向传播计算预测值并汇总均方误差（MSE）损失、反向传播触发梯度解算、以及基于一阶梯度下降更新参数值（p.data += -learning_rate * p.grad）。在调节步长超参数时，过大的学习率容易导致更新步幅越过局部最优甚至破坏训练稳定性。Karpathy 在实操中展示了一个极易忽视的高频工程错误：由于底层反向传播采用了 += 进行梯度累加，在启动新的迭代前若未显式遍历所有参数并将其 .grad 重置为 0.0，上一轮循环遗留的残余梯度就会持续叠加，进而干扰整个优化方向。 Karpathy 总结指出，微尺度标量引擎 micrograd 完整复现了 PyTorch 等工业级框架底层的反向传播与自动微分机制。尽管实际工程中现代框架依赖高维张量并行运算、动态小批量采样以及权重衰减等正则化策略来支撑工业级吞吐，但其底层的核心支柱依然是基于拓扑排序的有向图构建与微积分链式法则的递归执行。

## Background 该视频由 Andrej Karpathy 发布于 YouTube，旨在提供对神经网络反向传播及训练过程最详尽的循序渐进式讲解。视频内容仅要求观众具备基本的 Python 编程知识和高中程度的微积分概念，是其“Neural Networks: Zero to Hero”系列课程中的核心教程。 ## TL;DR 视频通过从零构建微型自动求导引擎 micrograd，深入浅出地演示了神经网络背后的数学本质。Karpathy 带领观众通过定义 Value 对象、实现运算符重载与链式法则，完成了反向传播机制的手动推导，并最终将其扩展为完整的神经元、层和多层感知机（MLP）模型。通过对比手动实现与 PyTorch 等工业级框架，观众能够清晰理解梯度下降驱动权重优化的底层逻辑，从而掌握训练现代神经网络的核心原理。 ## Atmosphere & style 视频氛围亲切且极具启发性，Karpathy 展现了一位资深专家将复杂技术“去魅”的卓越教学能力。他以一种类似结对编程的叙事节奏，耐心调试代码中的逻辑漏洞，甚至故意保留了教学中的有趣错误，使整个过程既像是一场严肃的学术研讨，又像是一次轻松的代码探险。这种手把手构建的教学方式不仅消除了对深度学习的恐惧感，更带给观众一种在底层数学中抽丝剥茧、最终获得洞见的深度成就感。 ## Bulletin - [04:33](https://www.youtube.com/watch?v=VMj-3S1tku0&t=273s) Andrej Karpathy 从零构建了标量级自动求导引擎 micrograd，仅用约 150 行 Python 代码（engine.py 与 nn.py）完整实现了反向传播与多层感知机训练闭环。 - [01:22:28](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4948s) 反向传播在计算图多路径汇聚时必须采用梯度累加（+=）而非直接赋值（=），以严格符合多元微积分链式法则并避免分支梯度覆盖 Bug。 - [01:36:20](https://www.youtube.com/watch?v=VMj-3S1tku0&t=5780s) 自动微分引擎中算子的抽象粒度可自由定制，无论是封装粗粒度 tanh 还是拆解为指数、乘法、加法等原子运算，叶子节点求得的梯度在数学上完全等价。 - [02:08:43](https://www.youtube.com/watch?v=VMj-3S1tku0&t=7723s) 在训练循环中，每次调用 loss.backward() 之前必须显式执行 zero_grad 重置参数梯度，否则累加机制会导致历史梯度无限叠加破坏训练收敛。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=VMj-3S1tku0&t=0s) 反向传播是 PyTorch 和 JAX 等现代深度学习库的数学内核，其实质是通过拓扑遍历高效评估损失函数关于权重的偏导数。 - [04:33](https://www.youtube.com/watch?v=VMj-3S1tku0&t=273s) micrograd 采用标量求导引擎而非高维张量是出于教学目的，以避免张量维度掩盖微积分与链式法则的直觉，二者在底层数学机制上毫无二致。 - [09:33](https://www.youtube.com/watch?v=VMj-3S1tku0&t=573s) 神经网络训练不能依赖符号求导或有限差分数值近似，而是通过计算图中局部导数与上游梯度的链式连乘实现解析级的高效反向传播。 - [43:07](https://www.youtube.com/watch?v=VMj-3S1tku0&t=2587s) 加法节点在反向传播中充当“梯度路由器”，其局部导数恒为 1.0，能够将上游损失梯度无损分发给所有输入子节点。 - [47:07](https://www.youtube.com/watch?v=VMj-3S1tku0&t=2827s) 乘法节点在反向传播时将上游梯度与配对操作数的当前数值进行交叉相乘并向下传递。 - [01:09:02](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4142s) 在 Value 对象中通过捕获父节点与局部导数的闭包函数（_backward），可以在前向运算时动态绑定链式求导逻辑。 - [01:17:32](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4652s) 全图自动反向传播依赖有向无环图的拓扑排序，节点必须等待其所有后继依赖节点处理完毕后方可触发自身的梯度回传。 - [01:22:28](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4948s) 计算图中同一节点若被多次引用，多元微积分链式法则要求梯度必须使用 += 累加，否则后续分支会覆盖先前的梯度导致求导错误。 - [01:31:52](https://www.youtube.com/watch?v=VMj-3S1tku0&t=5512s) 通过重载 Python 的 __rmul__、__pow__、__neg__ 等魔术方法，能将减法与除法优雅地降级为加法、乘法与幂运算的组合。 - [01:36:20](https://www.youtube.com/watch?v=VMj-3S1tku0&t=5780s) 将 tanh 拆解为指数与除法等原子操作所得的梯度与黑盒算子完全一致，证明自动微分引擎可按需在任意抽象层次定义算子。 - [02:21:10](https://www.youtube.com/watch?v=VMj-3S1tku0&t=8470s) PyTorch 底层在 C++/CUDA 内核中计算 tanh 梯度的核心数学逻辑依然是 grad * (1.0 - output * output)，与标量实现本质相同。 - [01:47:55](https://www.youtube.com/watch?v=VMj-3S1tku0&t=6475s) 构建多层感知机（MLP）仅需逐层抽象神经元（Neuron）、层（Layer）与网络模块，通过累加各层权重偏置的参数列表即可支持统一优化。 - [02:01:12](https://www.youtube.com/watch?v=VMj-3S1tku0&t=7272s) 梯度下降沿负梯度方向更新网络参数以最小化均方误差损失，学习率选取过大会引发数值震荡甚至发散，过小则收敛缓慢。 - [02:08:43](https://www.youtube.com/watch?v=VMj-3S1tku0&t=7723s) 每一次训练迭代必须严格执行“清零梯度 -> 前向传播求损失 -> 反向传播求梯度 -> 参数更新”四步闭环，遗漏梯度清零是深度学习初学者的常见 Bug。 - [02:14:03](https://www.youtube.com/watch?v=VMj-3S1tku0&t=8043s) 拥有数千亿参数的现代巨型语言模型（如 GPT）在计算图机制、链式反向传播和梯度下降更新逻辑上，与 41 个参数的微型网络完全同构。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=VMj-3S1tku0&t=0s) 课程目标与 micrograd 概览 Andrej Karpathy 开篇介绍自己训练深度神经网络已有十余年时间。在本讲座中，他希望向大家展示神经网络训练“引擎盖下”（under the hood）的真实面貌。课程将从一个完全空白的 Jupyter Notebook 开始，一步步构建出完整的代码，最终定义并训练一个神经网络，让学习者在直观层面彻底理解其底层运作机制。 Andrej 将带领大家从零构建 micrograd。这是他在大约两年前开源在 GitHub 上的一个微型库，但当时只上传了源码，没有配套讲解，学习者只能自行摸索。micrograd 本质上是一个轻量级的自动求导引擎（autograd engine，即 automatic gradient 的缩写），它的核心功能是实现反向传播（backpropagation）。反向传播算法能够极其高效地评估某个损失函数（loss function）关于神经网络权重的梯度（gradient）。一旦拥有了这些梯度，我们便可以通过迭代微调网络的权重来最小化损失函数，进而提升网络的准确率。因此，反向传播构成了当今 PyTorch 或 JAX 等现代深度学习库的数学内核。 ```mermaid flowchart LR accTitle: micrograd 自动求导与反向传播工作流 accDescr: 展示从输入变量构建计算图前向传播，再通过反向传播自动递归应用链式法则计算梯度的过程。 subgraph Inputs["输入层"] A["a: Value(-4.0)"] B["b: Value(2.0)"] end subgraph Graph["表达式图 (前向传播)"] C["中间节点 c, d, e, f"] G["输出节点 g: Value(24.7)"] end subgraph Backprop["反向传播 (g.backward)"] Grads["递归链式法则<br/>计算所有节点梯度<br/>a.grad=138, b.grad=645"] end A --> C B --> C C --> G G -.->|"g.backward()"| Grads ``` - micrograd 是一个实现自动求导与反向传播算法的核心引擎。 - 反向传播的用途：求取损失函数相对于权重的梯度，进而通过微调参数来最小化损失、提高模型精度。 - 它是包括 PyTorch 和 JAX 在内的现代深度学习库的底层数学支柱。 ### [01:24](https://www.youtube.com/watch?v=VMj-3S1tku0&t=84s) 表达式图的前向计算与反向求导机制 为了直观展示 micrograd 的工作原理，Andrej 演示了一个构建数学表达式的示例。在这个例子中，输入标量 $a = -4.0$ 与 $b = 2.0$ 被封装进 micrograd 的核心 `Value` 对象中。通过这个对象，可以对它们进行加法、乘法、常数幂次运算、偏移 1、取负、在 0 处截断（squash at zero，即 ReLU）、平方、除以常数等丰富的数学运算，逐步生成中间变量 $c$、$d$、$e$、$f$，最终输出标量 $g$。 在后台，micrograd 会自动维护整个计算图（expression graph）的数据结构。例如，它知道 $c$ 也是一个 `Value`，是由加法操作生成的，并且 $c$ 维护着指向其子节点 $a$ 和 $b$ 的指针。运行前向传播（forward pass）时，通过访问 `g.data` 属性可以得到最终的计算数值，在这个例子中结果为 $24.7$。 然而真正的关键在于反向传播：直接调用 `g.backward()`，即可从输出节点 $g$ 开始反向回溯整个计算图，递归地应用微积分中的链式法则（chain rule）。这一过程不仅能求出 $g$ 相对于内部中间节点（如 $e$、$d$、$c$）的导数，还能求出它相对于初始输入节点 $a$ 和 $b$ 的导数。查询 `a.grad` 和 `b.grad`，会发现数值分别为 $138$ 和 $645$。导数的物理意义在此十分明确：如果我们将 $a$ 向上轻微扰动（使其略微增大），$138$ 这个斜率表明 $g$ 也将随之增长；同样，$b$ 对应的增长斜率则是 $645$。它精确地刻画了微调输入时输出响应的灵敏度。 - 示例中的输入为 $a = -4.0$ 与 $b = 2.0$，封装为 `Value` 对象。 - 前向计算得到输出 $g.data = 24.7$。 - 调用 `g.backward()` 自动递归执行链式法则，求出各节点对 $g$ 的偏导数。 - 得到梯度的实际值：`a.grad = 138`，`b.grad = 645`，代表微调输入时 $g$ 变化的斜率。 ### [04:33](https://www.youtube.com/watch?v=VMj-3S1tku0&t=273s) 标量引擎与现代张量库的设计差异 对于示例中构建的繁复表达式，Andrej 调侃道，这只是他随便编造出来展示 micrograd 支持操作的无意义表达式。但事实表明，神经网络本身也只是数学表达式，甚至比这个随手捏造的表达式还要规整：它以输入数据和网络权重作为输入，经过层层计算，输出预测值或损失值。反向传播更具普适性，它并不针对神经网络，而是适用于任何数学表达式，只是恰好被应用到了神经网络的优化训练中。 Andrej 特别指出，micrograd 是一个标量值（scalar-valued）自动求导引擎，它将神经网络拆解为最底层的原子标量以及一个个基础的加号与乘号。在实际生产环境中，绝不会用这种标量引擎来训练网络，这样设计的目的完全是出于教学考量。现代深度学习框架使用高维张量（tensors），但从数学本质来看，张量只是把这些标量打包成大数组，以便利用计算机底层体系结构的并行计算能力加速处理，“所有的数学运算没有发生任何改变，做张量化纯粹是为了计算效率”。如果一开始就从张量切入，反而会掩盖微积分与链式法则的核心直觉。 他强调，micrograd 的代码其实极度精简，整个库仅包含两个核心文件： 1. `engine.py`：纯粹的自动求导引擎，对神经网络一无所知，仅用约 100 行极其平实的 Python 代码就实现了完整的反向传播； 2. `nn.py`：搭建在求导引擎之上的神经网络组件，定义神经元（Neuron）、由神经元构成的层（Layer）以及层堆叠形成的多层感知机（Multi-Layer Perceptron, MLP）。 Andrej 评价道：“nn.py 这个基于 autograd 引擎构建的神经网络库，简直就像个玩笑……所以基本上，仅仅通过 150 行代码就能产生巨大的威力。”这 150 行代码正是理解神经网络训练核心所需的全部内容，剩下的不过是工程优化。 “My claim is that micrograd is what you need to train your networks and everything else is just efficiency so you'd think that micrograd would be a very complex piece of code and that turns out to not be the case.” —— Andrej Karpathy - micrograd 与现代框架的核心区别：标量计算图 vs. 多维张量（tensors）。 - 张量化（vectorization）仅为了并行度与运行效率，底层的数学与链式法则机制完全相同。 - micrograd 的极简代码结构： - `engine.py`：约 100 行代码，实现标量表达式图与链式法则反向求导。 - `nn.py`：约 50 行代码，定义 Neuron、Layer 和 MLP。 - 总计仅约 150 行代码，即构成了深度学习训练的完整内核。 ### [08:08](https://www.youtube.com/watch?v=VMj-3S1tku0&t=488s) 标量函数的导数直觉与几何可视化 为了确保大家对导数建立扎实的直觉，Andrej 开始带领大家在 Jupyter Notebook 中从最基础的单变量标量函数切入。他首先导入了常用的标准库，接着定义了一个简单的单输入、单输出二次函数： $$f(x) = 3x^2 - 4x + 5$$ 当传入 $x = 3.0$ 时，函数计算返回的结果为 $20.0$。从数学形式很容易看出这是一条抛物线。为了观察它的几何形状，Andrej 利用 NumPy 创建了一个输入数组，范围取自 $[-5, 5)$，步长为 $0.25$（即 `np.arange(-5, 5, 0.25)`）。将这个数组直接传给函数 $f$，NumPy 会独立作用于数组中的每一个元素，得到对应的因变量数组 `ys`。随后利用 Matplotlib 进行绘制（`plt.plot(xs, ys)`），便在屏幕上清晰呈现出一条光滑的抛物线图像。这一可视化为接下来探讨微小扰动 $h$、斜率以及导数的极限定义打下了几何直觉的基础。 - 导入基本工具并定义标量测试函数：$f(x) = 3x^2 - 4x + 5$。 - 计算测试点取值：$f(3.0) = 3(3)^2 - 4(3) + 5 = 27 - 12 + 5 = 20.0$。 - 采样区间：利用 `np.arange(-5, 5, 0.25)` 构造测试点序列。 - 绘图展示：借助 Matplotlib 绘制出抛物线图像，用于直观感知输入与输出之间的响应关系。 ### [09:33](https://www.youtube.com/watch?v=VMj-3S1tku0&t=573s) 单输入简单函数的导数与微积分直觉 安德烈·卡帕西（Andrej Karpathy）首先提出了一个基础问题：对于一个简单的单变量函数 $f(x) = 3x^2 - 4x + 5$，在任意输入点 $x$ 处的导数是什么？在传统的微积分课堂上，我们通常会拿出纸和笔，应用乘积法则等各种求导公式，推导出该函数的解析导数表达式，然后代入不同的 $x$ 值来计算具体导数。然而，卡帕西指出，在神经网络中我们不能采用这种符号化的方法： > “在神经网络中，实际上没有人会去写出整个神经网络的数学表达式，那将是一个极其庞大的表达式……它会有成千上万个项，当然没有人会去手动推导它的导数，所以我们不会采用这种符号化的方法。” 因此，卡帕西带大家重新温习了导数的数学定义： $$\lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$$ 导数本质上衡量的是敏感度：如果你在某个点 $x$ 处将输入稍微增加一个极小的值 $h$，函数值会如何响应？它会上升还是下降？变化的幅度有多大？这个响应的斜率就是该点处的导数。 为了在 Python 中数值化地评估这个导数，卡帕西选择了一个极小的 $h = 0.001$。以 $x = 3.0$ 为例，此时 $f(x) = 20$。如果我们将 $x$ 稍微向正方向微调（nudge）一点，由于该点处的斜率是正的，我们预期 $f(x+h)$ 会略大于 20。通过计算变化量并除以 $h$（即 $\frac{\text{rise}}{\text{run}}$，也就是斜率），我们就能得到导数的数值近似。卡帕西解释说，虽然 $h$ 越趋近于 0 结果越精确，但在计算机中不能无限制地增加 $h$ 的零的个数： > “在某一个点上，我会得到一个错误的结果，因为我们使用的是浮点数运算，而所有这些数字在计算机内存中的表示都是有限的，到了一定程度我们就会遇到麻烦。” 在 $x = 3.0$ 处，通过这种数值方法算出的斜率约为 $14.0$。如果我们通过解析法求导， $f'(x) = 6x - 4$，代入 $x = 3$ 同样得到 $18 - 4 = 14$，两者完全吻合。随后，卡帕西又测试了 $x = -3.0$ 处的情况，此时向正方向微调 $x$ 会导致函数值下降，因此算出的斜率是负数（约为 $-22.0$）。而在该函数的顶点 $x = 2/3$（约 $0.666$）处，微调输入几乎不会引起函数值的变化，因此该点处的导数趋近于 $0$。 --- ### [14:12](https://www.youtube.com/watch?v=VMj-3S1tku0&t=852s) 多输入函数的偏导数与数值估算 为了进一步逼近神经网络的实际情况，卡帕西引入了一个稍微复杂的多元函数。该函数包含三个标量输入 $a$、$b$ 和 $c$，以及一个标量输出 $d$，其数学表达式为： $$d = a \cdot b + c$$ 在代码中，卡帕西将输入初始化为 $a = 2.0$、$b = -3.0$、$c = 10.0$，此时输出 $d = 2.0 \cdot (-3.0) + 10.0 = 4.0$。接着，他希望通过数值微调的方法，分别求出 $d$ 对 $a$、$b$ 和 $c$ 的偏导数，以此来直观感受每个输入对输出的影响力。 ```mermaid flowchart LR accTitle: d = a * b + c 的计算图 accDescr: 一个简单的计算图，展示了输入 a、b 和 c 如何通过乘法和加法操作生成输出 d。 a["a (2.0)"] --> mul["*"] b["b (-3.0)"] --> mul mul -->|"e (-6.0)"| add["+"] c["c (10.0)"] --> add add --> d["d (4.0)"] ``` 为了评估 $d$ 对 $a$ 的导数，卡帕西将 $a$ 增加了 $h = 0.0001$。因为 $b$ 是负数（$-3.0$），增加 $a$ 意味着我们在 $d$ 中加上了一个更负的数，因此预期函数值会下降。实验结果确实如此，输出从 $4.0$ 降到了约 $3.9996$，算出的斜率为 $-3.0$。这与解析导数 $\frac{\partial d}{\partial a} = b = -3.0$ 完全一致。 同理，当微调 $b$ 时，因为 $a$ 是正数（$2.0$），增加 $b$ 会使输出 $d$ 增加，算出的斜率为 $2.0$，与解析导数 $\frac{\partial d}{\partial b} = a = 2.0$ 一致。而当微调 $c$ 时，由于 $c$ 是直接加在表达式末尾的，它的变化会 1:1 地反映在输出上，因此其斜率为 $1.0$（即 $\frac{\partial d}{\partial c} = 1.0$）。通过这些例子，卡帕西帮助大家建立起了清晰的直觉：导数就是当你在输入端进行微小的调整时，输出端随之变化的速率。 --- ### [19:09](https://www.youtube.com/watch?v=VMj-3S1tku0&t=1149s) 构建 micrograd 的核心 Value 对象 在理解了导数的物理意义后，卡帕西开始着手构建神经网络库。由于神经网络本质上是极其庞大的数学表达式，我们需要设计一种能够自动追踪并维护这些表达式的数据结构。为此，他开始在 Jupyter Notebook 中构建 `Value` 类，这也是整个 `micrograd` 库的核心。 卡帕西首先写出了 `Value` 类的基础骨架。这个类非常简单，它包裹并追踪一个单一的标量数值（`data`），并实现了一个友好的字符串表示方法 `__repr__`，以便在 Python 中打印时能够清晰地看到其包裹的数据： ```python class Value: def __init__(self, data): self.data = data def __repr__(self): return f"Value(data={self.data})" ``` 通过这个定义，我们可以实例化一个对象，例如 `a = Value(2.0)`。然而，此时如果我们尝试将两个 `Value` 对象相加（如 `a + b`），Python 会抛出类型错误（TypeError），因为 Python 此时还不知道如何对两个自定义的 `Value` 对象进行加法运算。为了解决这个问题，卡帕西指出，我们需要在类中实现 Python 的特殊双下划线方法（dunder methods），比如 `__add__`，从而逐步赋予这个对象进行数学运算的能力。 ### [20:26](https://www.youtube.com/watch?v=VMj-3S1tku0&t=1226s) 实现运算符重载与 `Value` 对象的核心逻辑 在 Python 中，为了让自定义对象能够直接使用数学运算符（如 `+` 和 `*`），必须定义特殊的双下划线方法（dunder methods）。当代码中执行 `a + b` 时，Python 内部实际上调用的是 `a.__add__(b)`。在这个方法中，`self` 指向 `a`，而传入的 `other` 则是 `b`。方法执行后会返回一个新的 `Value` 实例，该实例内部包裹了 `self.data + other.data` 的计算结果。需要注意的是，此时 `self.data` 和 `other.data` 均为 Python 内置的标准浮点数，因此这里调用的加法就是原生的浮点数相加，而非 `Value` 对象之间的递归加法。对于输入 `a = Value(2.0)` 与 `b = Value(-3.0)`，相加得到的新对象包含的值即为 `-1.0`。 ```python class Value: def __init__(self, data, _children=(), _op='', label=''): self.data = data self.grad = 0.0 self._prev = set(_children) self._op = _op self.label = label def __repr__(self): return f"Value(data={self.data})" def __add__(self, other): out = Value(self.data + other.data, (self, other), '+') return out def __mul__(self, other): out = Value(self.data * other.data, (self, other), '*') return out ``` 乘法运算的实现遵循完全相同的模式，只需定义 `__mul__` 方法并在内部将底层数值相乘即可。例如创建 `c = Value(10.0)` 后，计算 `a * b` 会返回封装了 `-6.0` 的新 `Value` 对象。关于 `__repr__` 方法，如果不提供该实现，打印对象时只会显示一段晦涩难懂的内存地址；实现 `__repr__` 则能让终端清晰地输出类似 `Value(data=-6.0)` 的直观表达。有了 `__add__` 与 `__mul__` 之后，像 `a * b + c` 这样的复合表达式就能被顺利解析：它等价于 `(a.__mul__(b)).__add__(c)`，最终求值结果为 `4.0`。 - `a.__add__(b)`：加法运算重载，内部提取浮点数相加后返回封装结果的新 `Value` 对象。 - `a.__mul__(b)`：乘法运算重载，内部提取底层数值相乘并返回新对象。 - `__repr__`：提供格式化的字符串表示，避免输出晦涩的 Python 原生对象内存描述符。 --- ### [22:45](https://www.youtube.com/watch?v=VMj-3S1tku0&t=1365s) 构建计算图的骨架：父子指针与操作符追踪 仅仅能进行前向数值运算还远远不够，自动求导需要整个数学表达式的“结缔组织”（connective tissue）。系统必须清楚地记录下表达式图结构中的指针：到底是由哪些节点（输入值）通过何种运算生成了当前的节点（输出值）。为此，在 `Value` 类的构造函数中引入了新的属性参数。构造函数接收一个默认为空元组的 `_children`，并在对象内部将其转化为集合保存在私有属性 `self._prev` 中。对于像通过输入数据直接初始化的叶子节点，其 `_children` 为空元组，`_prev` 自然也是空集合；但当节点是通过加法或乘法生成时，调用方会将运算的操作数 `(self, other)` 作为子节点元组传递进去，成为新节点的来源记录。 除了记录子节点，系统还需要知道每个新值是由何种操作符生成的。因此，构造函数中加入了字符串变量 `_op`。对于原始输入的叶子节点，`_op` 默认为空字符串 `''`；而由加法生成的节点其 `_op` 被标记为 `'+'`，由乘法生成的节点则被标记为 `'*'`。这样一来，若将 `d = a * b + c` 计算出的节点打印检查，通过访问 `d._prev`，可以看到它来自封装了 `-6.0` 的中间节点和封装了 `10.0` 的节点 `c`；通过访问 `d._op`，则能准确确认生成它的是加法操作 `'+'`。 > “Looking at my code here I can't remember exactly the reason, I believe it was efficiency, but this underscore children will be a tuple for convenience but then when we actually maintain it in the class it will be just this set, yeah I believe for efficiency.” —— Andrej Karpathy - `self._prev`：使用集合维护产生当前节点的所有前驱节点（子节点引用），构造函数参数输入元组，内部转为集合存储以提高查找效率。 - `self._op`：记录产生该节点的计算操作符（如 `'+'`、`'*'`），叶子节点默认为空字符串。 - 完整表达式网络追踪：每个中间节点都具备追踪上游输入和所用操作符的能力，从而构成了完整的有向无环计算图（DAG）。 --- ### [24:54](https://www.youtube.com/watch?v=VMj-3S1tku0&t=1494s) 使用 Graphviz 可视化前向计算图 随着计算表达式逐渐变大，代码会变得复杂且难以仅靠文本阅读来理清关系，因此直观的可视化工具不可或缺。为此，Karpathy 引入了一个名为 `draw_dot` 的辅助函数，该函数依赖开源的图形渲染库 Graphviz 及其 Python 接口。`draw_dot` 首先调用辅助遍历函数 `trace`，从根节点开始递归收集计算图中所有的节点和有向边，然后遍历这些元素，在 Graphviz 图中创建对应的形状。 为了让可视化更加清晰，`draw_dot` 在图中不仅为每个 `Value` 对象创建长方形的数值节点，还额外创建了一批“虚拟操作节点”（fake operation nodes），用圆圈标记其对应的算符（如 `+` 或 `*`），并将输入变量指向这些操作节点，再由操作节点指向输出变量。Karpathy 强调，这些操作节点只是为了让图形看起来更清晰而在绘图逻辑中生成的虚拟占位符，在底层实际存储的计算图中并不存在对应的数据结构，真正的计算图节点只有那些代表数值的矩形框。 为了使图例具有更强的可读性，Karpathy 进一步修改了 `Value` 类，为其添加了可选的字符串标签 `label`。通过为各个节点设置标识符（例如将 `a * b` 的结果赋值给中间节点 `e` 并打上标签 `'e'`，将后续加法结果命名为 `d` 并打上标签 `'d'`），图形节点内部就能通过类似 `{ label | data }` 的格式将变量名与其数值一并展示出来。 ```mermaid flowchart LR accTitle: 前向计算图结构 accDescr: 展示变量 a、b、c 经过乘法和加法运算生成节点 d 的拓扑连接关系。 subgraph Inputs["输入叶子节点"] A["a | data: 2.0"] B["b | data: -3.0"] C["c | data: 10.0"] end subgraph MulStage["乘法阶段"] OpMul(("*")) E["e | data: -6.0"] end subgraph AddStage["加法阶段"] OpAdd(("+")) D["d | data: 4.0"] end A --> OpMul B --> OpMul OpMul --> E E --> OpAdd C --> OpAdd OpAdd --> D ``` - `trace` 函数：负责深度优先遍历给定的根节点，枚举出整个表达式网络中所有涉及的节点与有向边集合。 - 虚拟操作节点（Op Nodes）：在 Graphviz 绘图逻辑中额外构造的圆形运算符节点，仅用于视觉展示拓扑分支，并非底层的 `Value` 实体。 - 节点标签（`label`）：为各个中间及叶子变量赋予人类可读的字符串标识，与底层数值并在同一图例格子中展示。 --- ### [28:10](https://www.youtube.com/watch?v=VMj-3S1tku0&t=1690s) 扩展网络深度与反向传播的物理意义 为了让整个计算图的结构更接近真实的神经网络结构，Karpathy 将表达式又加深了一层。在节点 `d` 的基础上，新引入了叶子节点 `f = Value(-2.0, label='f')`，并将最终输出定义为变量 `L`，计算公式为 `L = d * f`。在给定了 `d = 4.0` 和 `f = -2.0` 之后，最终输出变量 `L` 的数值为 `-8.0`。至此，整个计算的前向传播过程（Forward Pass）全部完成：输入变量 `a, b, c, f` 汇集入一个纯由加法和乘法构成的标量运算网络中，最终在终点产生了一个单一标量输出 `L`。 随后，讲解转向了核心概念——反向传播（Backpropagation）。在前向传播完成之后，算法需要从最终的输出标量 `L` 倒着向回走，逆向计算 `L` 相对于表达式中每一个中间节点及叶子节点的导数（梯度）。输出对自身的导数 $\frac{\partial L}{\partial L}$ 显然恒等于 $1.0$。从终点出发，系统将沿着计算路径逐级倒推求解：$\frac{\partial L}{\partial f}$、$\frac{\partial L}{\partial d}$、$\frac{\partial L}{\partial c}$、$\frac{\partial L}{\partial e}$、$\frac{\partial L}{\partial b}$，直至最底层的 $\frac{\partial L}{\partial a}$。 Karpathy 明确点出了这套机制在神经网络中的核心意义：在真实的神经网络训练中，最终输出 `L` 通常代表度量预测误差的损失函数（Loss Function），而像 `a, b, c, f` 这样的叶子节点中，有一部分代表不可更改的固定输入数据（Data），而另一部分则代表神经网络的权重参数（Weights）。训练模型的核心诉求，正是计算损失函数相对于所有可学习权重的导数，获知如何调整权重能使损失下降。输入数据虽然也位于叶子节点，但我们并不关心损失对它的导数，因为数据是固定不变的；相反，权重必须根据梯度信息不断迭代更新。 为了在代码中追踪这些梯度值，Karpathy 在 `Value` 对象的构造函数中添加了名为 `grad` 的实例属性，其初始值被默认设为 `0.0`。初始值赋为 `0.0` 具有明确的直观意义：它代表在尚未进行反向求导时，假定各个节点对最终输出没有任何影响或改变量。 > “And in the neural network setting you'd be very interested in the derivative of basically this loss function L with respect to the weights of a neural network.” —— Andrej Karpathy - 计算图终点定义：$L = d \times f$，数值为 $4.0 \times (-2.0) = -8.0$。 - 反向传播本质：从标量输出节点 $L$ 沿着有向图的反方向进行遍历，求出链条上每个节点相对于 $L$ 的偏导数。 - 权重与数据的区别：叶子节点分为不可变的数据与可训练的参数，梯度计算的最终受众是参数节点，用于后续的梯度下降。 - `self.grad` 属性：在每个 `Value` 节点内维护其相对于根输出的标量导数，初始赋值为 `0.0`（代表对结果无影响）。 ### [31:16](https://www.youtube.com/watch?v=VMj-3S1tku0&t=1876s) 初始化梯度并在可视化图中展示 在构建反向传播的核心机制时，首先需要给变量赋予一个默认状态。如果一个变量的梯度是 0，说明改变这个变量并不会改变最终的损失函数（Loss Function），即它对输出结果没有任何影响。因此，所有节点的梯度 `grad` 默认均初始化为 `0.0`。 ``` data: 4.0000 | grad: 0.0000 ``` 更新可视化渲染函数后，每个节点不仅能显示自身的标量数值 `data`（格式化为 `%.4f`），还能并排展示对应的梯度值 `grad`。节点中的 `grad` 始终代表着最终输出变量（在此例中为节点 $L$）对当前变量的偏导数，即 $\frac{\partial L}{\partial v}$，例如 $\frac{\partial L}{\partial f}$、$\frac{\partial L}{\partial d}$ 等。随着图渲染完成，所有节点的梯度值初始状态均为 0，整个流程准备进入手动反向传播阶段。 - 梯度的初始语义：梯度为 0 意味着微调该输入不会引起最终输出产生任何变化。 - 梯度的定义目标：图中每个节点的 `grad` 存储的都是最终标量损失 $L$ 对该节点的导数，并非局部导数。 --- ### [32:10](https://www.youtube.com/watch?v=VMj-3S1tku0&t=1930s) 手动反向传播第一步：终点节点与乘法节点的导数推导 手动反向传播必须从整个计算图的最末端开始逐步向前推进。最末端的节点正是输出变量 $L$ 本身。第一个要解决的问题是：$L$ 对自身的导数 $\frac{\partial L}{\partial L}$ 是多少？若将 $L$ 改变一个极微小的量 $h$，$L$ 产生的变化量同样也是 $h$，两者的变化成正比例关系，因此其导数恒为 1。 为了严谨地对推导进行检验，可以使用数值梯度（Numerical Gradient）进行验证。在 Jupyter Notebook 中定义一个临时的辅助函数 `def lol():`，这样可以避免污染全局命名空间，确保所有的变量修改都局限在函数内部。通过在基准表达式 $L_1$ 之外构建一个微调版本 $L_2$（将目标变量增加 $h$），并计算差商 $\frac{L_2.\text{data} - L_1.\text{data}}{h}$，即可得到数值估计。当测试对 $L$ 增加 $h$ 时，输出的导数恰好为 1.0。将这一步作为反向传播的基准情况（Base Case），手动执行赋值并刷新绘图： ```python L.grad = 1.0 ``` 接着向前回溯至节点 $D$ 和 $F$。由于节点 $L$ 是由 $D$ 和 $F$ 相乘得到的（即 $L = D \times F$），根据导数的微积分定义： $$\frac{\partial L}{\partial D} = \lim_{h \to 0} \frac{(D + h)F - DF}{h} = \lim_{h \to 0} \frac{DF + hF - DF}{h} = \lim_{h \to 0} \frac{hF}{h} = F$$ 同理，根据对称性，$\frac{\partial L}{\partial F} = D$。已知节点数据 $D.\text{data} = 4.0$，$F.\text{data} = -2.0$，因此对应的梯度可直接赋值： ```python F.grad = 4.0 # 即 D 的值 D.grad = -2.0 # 即 F 的值 ``` 随后利用辅助函数 `lol()` 进行内联梯度检查（Inline Gradient Check）：对变量 $F$ 加上 $h$ 后计算差商，由于浮点精度极限打印出的值极其逼近 4.0；对变量 $D$ 加上 $h$，算出的数值导数精准命中 -2.0。 ```mermaid flowchart LR accTitle: 乘法节点的反向传播梯度流 accDescr: 展示最终节点 L 通过乘法运算将导数回传给输入节点 D 和 F 的过程 subgraph Inputs["输入节点"] D["D<br/>data: 4.0<br/>grad: -2.0"] F["F<br/>data: -2.0<br/>grad: 4.0"] end subgraph Operation["运算与输出"] Mul["*"] L["L<br/>data: -8.0<br/>grad: 1.0"] end D --> Mul F --> Mul Mul --> L L -.->|"dL/dF = D = 4.0"| F L -.->|"dL/dD = F = -2.0"| D ``` - 数值梯度检查机制：通过差商公式 $\frac{f(x+h) - f(x)}{h}$ 对推导出的解析梯度进行即时验证。 - 乘法求导法则：对于双输入乘法 $L = D \times F$，每个输入对应的偏导数恰好等于另一个输入的数值。 --- ### [38:08](https://www.youtube.com/watch?v=VMj-3S1tku0&t=2288s) 加法节点的局部导数与链式法则（Chain Rule）的核心推导 在解决完输出端的直连节点后，计算图进入到了最关键的一环——推导 $\frac{\partial L}{\partial C}$ 和 $\frac{\partial L}{\partial E}$。Andrej Karpathy 强调了这一节点在整个深度学习教学中的分量：“Now we're getting to the crux of backpropagation so this will be the most important node to understand because if you understand the gradient for this node you understand all of back propagation and all of training of neural nets basically.” 面临的核心问题在于：已知 $L$ 对中间变量 $D$ 的敏感度（$\frac{\partial L}{\partial D} = -2.0$），但 $C$ 并未直接连接到 $L$，$C$ 是通过影响 $D$（通过加法 $D = C + E$）进而连带影响到最终输出 $L$ 的。解决这一问题的拆解步骤分为“局部导数”与“链式合成”两部分： 1. **局部导数（Local Derivative）：** 观察加法运算 $D = C + E$。从纯局部的视角来看，这个加法节点完全不关心自身处于多么庞大的网络拓扑之中。根据极限定义： $$\frac{\partial D}{\partial C} = \lim_{h \to 0} \frac{((C + h) + E) - (C + E)}{h} = \lim_{h \to 0} \frac{h}{h} = 1.0$$ 对称地，$\frac{\partial D}{\partial E} = 1.0$。加法运算对两个输入的局部导数均为 1.0。 2. **微积分链式法则（The Chain Rule）：** 尽管 Wikipedia 上的数学表述容易让人头晕眼花，但其逻辑极其简洁：若变量 $z$ 取决于 $y$，而 $y$ 又取决于 $x$，则 $z$ 依赖于 $x$ 的瞬时变化率可以通过两段变化率相乘来计算： $$\frac{dz}{dx} = \frac{dz}{dy} \times \frac{dy}{dx}$$ 因此，对于复合路径 $C \to D \to L$，复合导数的计算方式即为全局回传梯度与局部导数的乘积： $$\frac{\partial L}{\partial C} = \frac{\partial L}{\partial D} \times \frac{\partial D}{\partial C}$$ 这就是反向传播算法的基石所在：复合函数求导本质上就是各个局部环节导数沿着路径进行的连续相乘。 - 局部导数特征：加法节点充当了一个局部的独立单元，其对各输入的局部导数恒为 1.0，完全独立于网络的后续层级。 - 链式传递直觉：已知 $z$ 相对 $y$ 的变化率以及 $y$ 相对 $x$ 的变化率，二者相乘即刻锁定了 $z$ 相对 $x$ 的变化速度。 ### [43:07](https://www.youtube.com/watch?v=VMj-3S1tku0&t=2587s) 链式法则的直观理解与加法节点的梯度路由 为了建立对微积分中链式法则（Chain Rule）的直观认知，Andrej Karpathy 举了一个生动的物理速度比喻：如果一辆汽车的速度是一辆自行车的 2 倍，而自行车的速度又是一个步行者的 4 倍，那么这辆汽车相对于步行者的速度就是 $2 \times 4 = 8$ 倍。Karpathy 强调：“And so this makes it very clear that the correct thing to do sort of is to multiply.” 这种将中间变化率（Rates of Change）相乘的做法，在直觉上直接印证了链式法则的核心思想。 将这一思想代入计算图：目标是求解损失（或最终输出）$L$ 对中间变量 $c$ 的导数，即 $\frac{dL}{dc}$。在此之前，已经推导出 $L$ 对加法节点输出 $d$ 的导数 $\frac{dL}{dd} = -2.0$。加法运算在局部极其简单，因为 $d = c + e$，其局部导数 $\frac{\partial d}{\partial c}$ 恒等于 $1.0$。根据链式法则： $$\frac{dL}{dc} = \frac{dL}{dd} \times \frac{\partial d}{\partial c} = (-2.0) \times 1.0 = -2.0$$ 同理，由于运算的对称性，加法节点的另一个输入分支 $e$ 对应的局部导数 $\frac{\partial d}{\partial e}$ 同样为 $1.0$，因此 $\frac{dL}{de} = -2.0$。 Karpathy 进而揭示了加法节点（`+` 节点）在反向传播中的本质作用：它本质上是一个梯度路由器（Gradient Router）。因为加法节点的局部梯度恒为 $1.0$，上游传来的梯度与其相乘后保持原样不变，直接无损分发给它的所有子节点（Children Nodes）。Karpathy 随后通过在代码中直接修改测试用例来验证这一结论：分别对 `c.data` 和 `e.data` 增加扰动步长 $h$，运行求导检查后得到的数值结果均精确符合理论预测的 $-2.0$。 ```mermaid flowchart LR accTitle: 加法节点的梯度路由机制 accDescr: 展示上游梯度如何通过加法节点无损分发给两个输入子节点 subgraph Inputs ["加法输入节点"] C["Node c<br/>grad = -2.0"] E["Node e<br/>grad = -2.0"] end subgraph AddOp ["加法与输出"] Plus["Op: +<br/>局部导数 = 1.0"] D["Node d<br/>grad = -2.0"] end D -->|"上游梯度 (-2.0)"| Plus Plus -->|"路由 (x 1.0)"| C Plus -->|"路由 (x 1.0)"| E ``` --- ### [47:07](https://www.youtube.com/watch?v=VMj-3S1tku0&t=2827s) 乘法节点的反向传播与局部梯度求解 完成对中间节点 $c$ 和 $e$ 的梯度推导后，反向传播继续向叶子节点递归。针对子表达式 $e = a \times b$，已知上游梯度 $\frac{dL}{de} = -2.0$，需要求出 $\frac{dL}{da}$ 和 $\frac{dL}{db}$。 Karpathy 形象地站在计算图中运算节点的视角展开推理：“So I'm a little times node inside a massive graph, and I only know that I did $a \times b$ and I produced an $e$.” 对这个局部的乘法节点（`*` 节点）而言，它根本不需要理解整个巨型图的结构，只需要关注自身的局部输入与输出。对于乘法 $e = a \cdot b$： - $e$ 对 $a$ 的局部导数即为另一个乘数的值：$\frac{\partial e}{\partial a} = b$（当前数值为 $-3.0$）。 - $e$ 对 $b$ 的局部导数即为 $a$ 的值：$\frac{\partial e}{\partial b} = a$（当前数值为 $2.0$）。 根据链式法则将上游梯度与局部梯度相乘： - `a.grad` $= \frac{dL}{de} \times \frac{\partial e}{\partial a} = (-2.0) \times (-3.0) = 6.0$ - `b.grad` $= \frac{dL}{de} \times \frac{\partial e}{\partial b} = (-2.0) \times 2.0 = -4.0$ 将这些梯度赋值到计算图可视化中刷新后，Karpathy 通过有限差分代码（对 `a.data += h` 及 `b.data += h`）进行严格校验，数值梯度输出分别为 $6.0$ 与 $-4.0$（伴随极微小的浮点舍入误差），完全吻合解析推导。 Karpathy 总结道，整个反向传播的本质非常纯粹：沿着计算图从最终输出端逐步向后递归遍历每一个节点，在每个局部节点上取出当前操作的局部导数，并将其乘以上游回传的梯度。这一全流程正是微积分链式法则在有向无环图上的递归执行过程。 --- ### [51:10](https://www.youtube.com/watch?v=VMj-3S1tku0&t=3070s) 梯度驱动的单步优化演示 为了展示反向传播所求梯度的实际效用，Karpathy 现场演示了一次极其纯粹的单步参数优化（Optimization Step）。优化目标是让计算图最终的输出标量 $L$ 增大（向正方向移动）。根据微积分定义，导数指示了函数增长最快的方向，因此只需将具有控制权的叶子节点输入向着梯度的正方向推进一步（Nudge in the direction of the gradient）： - 设定一个微小的步长 `step_size`（即学习率），对叶子节点 $a$、$b$、$c$、$f$ 的数值分别进行更新： $$\text{param.data} \mathrel{+}= \text{step\_size} \times \text{param.grad}$$ - 在该例中，更新前的基准输出 $L$ 约为 $-8.0$。 - 执行更新并重新运行前向传播（Forward Pass）后，$L$ 的值从 $-8.0$ 上升到了约 $-7.0$（变得不那么负了），成功实现了输出值的正向增长。 Karpathy 强调，这就是神经网络训练的核心机制：“And really does gradient just give us some power because we know how to influence the final outcome, and this will be extremely useful for training neural networks as well as you'll see.” --- ### [52:52](https://www.youtube.com/watch?v=VMj-3S1tku0&t=3172s) 神经元数学模型与 tanh 激活函数 在完成了基础玩具示例后，Karpathy 转向一个更贴近实际、结构更复杂的对象：对一个人工神经元（Neuron）进行手动反向传播。 在最常见的两层多层感知机（Multi-Layer Perceptron, MLP）架构中，层与层之间的神经元是全连接的（Fully Connected）。生物学意义上的神经元结构极为精细复杂，但在深度学习中，人们采用了高度抽象的数学模型： 1. **突触输入与权重乘积**：神经元接收多个输入信号 $x_i$，突触连接具有权重参数 $w_i$，二者在突触处以乘法形式交互，向胞体（Cell Body）传递的数据流为 $\sum_i w_i x_i$。 2. **偏置项（Bias）**：胞体内部还包含一个偏置项 $b$。Karpathy 将其通俗地描述为神经元“天生的触发敏感度”（innate sort of trigger happiness），偏置可以在完全独立于外部输入的情况下，让神经元整体上更容易或更难被激活。 3. **激活函数（Activation Function）**：线性组合 $\sum w_i x_i + b$ 汇聚后，必须通过一个非线性的“挤压函数”（Squashing Function），如 Sigmoid 或双曲正切函数（tanh）。 ```mermaid flowchart LR accTitle: 人工神经元计算流程 accDescr: 展示输入与权重相乘、偏置相加并经由双曲正切激活函数挤压的过程 subgraph Inputs ["突触输入"] X1["x1"] W1["w1"] X2["x2"] W2["w2"] end subgraph CellBody ["胞体汇聚与偏置"] Mul1["w1 * x1"] Mul2["w2 * x2"] Sum["求和: sum(w*x)"] Bias["偏置: b"] PreAct["净输入: sum + b"] end subgraph Activation ["激活挤压"] Tanh["tanh(z)"] Out["输出: Output"] end X1 --> Mul1 W1 --> Mul1 X2 --> Mul2 W2 --> Mul2 Mul1 --> Sum Mul2 --> Sum Sum --> PreAct Bias --> PreAct PreAct --> Tanh Tanh --> Out ``` 为了直观展现 tanh 的几何特性，Karpathy 在 Jupyter Notebook 中调用 `numpy.tanh` 绘制了一条连续曲线： - 当输入 $z = 0$ 时，输出严格为 $0$。 - 当输入向正方向增大时，输出曲线平滑上升并饱和渐近于上限 $1.0$。 - 当输入向负方向减小时，输出曲线平滑下降并饱和渐近于下限 $-1.0$。 无论输入信号的绝对值有多大，tanh 都会平滑地将整个实数空间“挤压”并映射到区间 $[-1.0, 1.0]$ 之内。 ### [55:04](https://www.youtube.com/watch?v=VMj-3S1tku0&t=3304s) 搭建神经元数学图结构与为何直接实现 tanh Andrej Karpathy 接着上一节对 tanh 激活函数（一种“挤压函数”，squashing function）的介绍，说明一个典型神经元的输出就是对权重与输入的点积再加上偏置后，整体输入到激活函数中所得到的结果。为了节省打字时间，他通过复制代码在 Notebook 中定义了一个二维神经元的结构：输入为 $x_1$ 和 $x_2$，对应的突触权重（synaptic strengths）为 $w_1$ 和 $w_2$，再加上神经元的偏置 $b$。在计算过程中，他特意拆解为多个细小步骤，分别创建中间变量并打上标签（如 $x_1 \cdot w_1$、$x_2 \cdot w_2$ 以及两者的和），最后再加上偏置 $b$ 得到神经元细胞体在未经过激活函数时的原始激活值 $n$。用 `draw_dot(n)` 渲染计算图后，可以看到输入相乘后逐级相加的清晰拓扑。 想要得到最终输出 $o$，需要执行 `o = n.tanh()`。然而，此时的 `Value` 类只支持加法和乘法，而双曲函数 $\tanh$ 需要指数运算（$\text{math.exp}$）。Karpathy 指出，虽然我们可以先实现底层的指数操作符 `exp`，再用加法、除法等基本原语拼装出 $\tanh$，但他在此处刻意选择直接在 `Value` 中将 $\tanh$ 作为单一复合操作来实现。他借此强调了自动求导的核心思想：“我们完全可以根据需要，在任意抽象层级上构建函数，无论是极其微小的原子操作如加法，还是相对复杂的复合函数，选择完全取决于我们；唯一的硬性要求，就是我们必须清楚如何对该操作求导，只要知道输入对输出的局部导数即可。” ```mermaid flowchart LR accTitle: 神经元计算图拓扑结构 accDescr: 展示二维输入、权重与偏置经过加权求和后通过 tanh 激活函数生成最终输出的过程。 subgraph Inputs["输入与权重分支"] X1["x1: 2.0"] W1["w1: -3.0"] X2["x2: 0.0"] W2["w2: 1.0"] end subgraph Mult["乘法运算"] M1["* (x1w1)"] M2["* (x2w2)"] end subgraph Accumulation["累加与偏置"] A1["+ (x1w1 + x2w2)"] B["b: 6.881373587"] A2["+ (细胞体原始激活 n)"] end subgraph Output["激活输出"] T["tanh"] O["输出 o: ~0.7071"] end X1 --> M1 W1 --> M1 X2 --> M2 W2 --> M2 M1 --> A1 M2 --> A1 A1 --> A2 B --> A2 A2 --> T T --> O ``` - 在 `Value` 类内部实现 `tanh` 方法时，利用公式 $t = \frac{e^{2x} - 1}{e^{2x} + 1}$ 计算出前向传播的数值，并返回一个新的 `Value(t, (self,), 'tanh')` 对象。 - 当把偏置调大到 $8$ 时，输入到 $\tanh$ 的值约为 $2$，输出达到 $0.96$，直观展现了输出平滑逼近 $1$ 的饱和挤压效应。 - Karpathy 随后将偏置 $b$ 精确设定为 `6.8813735870195432`，目的是让后续反向传播中的中间梯度呈现出极其整齐漂亮的数值（如 $0.5$），以便用直觉进行心算和验证。 --- ### [01:01:36](https://www.youtube.com/watch?v=VMj-3S1tku0&t=3696s) 手动反向传播：从输出节点到偏置分支的梯度推导 设置好计算图后，Karpathy 开始手动推导梯度，为每个节点填入 `.grad`。他提醒大家，在真正的神经网络中，整个网络往往包含成千上万个相连的神经元，最终由一个损失函数衡量整体预测表现；而最受关注的梯度永远是针对权重参数（如 $w_1, w_2$）的梯度，因为优化算法正是通过更新权重来减少损失。 求导从最末端节点 $o$ 出发。作为基础情形，根据定义输出对其自身的导数 $\frac{\partial o}{\partial o} = 1.0$，因此直接赋予 `o.grad = 1.0`。接下来需要沿着网络逆向跨过 `tanh` 操作。微积分中 $\tanh(x)$ 的导数形式为 $\frac{d}{dx}\tanh(x) = 1 - \tanh^2(x)$。由于当前节点的前向值已经存储了 $\tanh(n)$ 的结果（即 $o$ 的数值），局部导数直接等于 $1 - o^2$。将 $o$ 的具体数值代入公式计算后，恰好得到极为规整的数值：$1 - o^2 = 0.5$。根据链式法则，$\frac{\partial o}{\partial n} = \frac{\partial o}{\partial o} \times \frac{\partial o}{\partial n} = 1.0 \times 0.5 = 0.5$，所以细胞体原始求和节点 $n$ 的梯度为 `n.grad = 0.5`。 紧随其后的是累加偏置的加法节点。Karpathy 称赞道：“加法节点是我最喜欢的反向传播操作，因为它的处理极其简单。”加法操作的局部导数恒为 $1.0$，其实质就是一个“梯度分发器”（gradient distributor）。因此，流入 $n$ 的梯度 $0.5$ 被不折不扣、均等地分发给上一级的两个分支——偏置节点 $b$ 和中间和节点 $x_1 w_1 + x_2 w_2$。继续往前遇到另一个加法节点，同样把 $0.5$ 的梯度一分为二传递给 $x_1 w_1$ 与 $x_2 w_2$ 两个乘法输出节点。Karpathy 强调，每个节点的梯度都在物理层面上指示着方向：当梯度为正时，说明若要增大神经元的最终输出 $o$，增加这些中间项的数值将会带来正向贡献。 - 节点 $o$（最终输出）的梯度初始设置为 `1.0`。 - 借助导数公式 $\frac{do}{dn} = 1 - o^2$，计算得到输入 $n$ 处的梯度 `n.grad = 0.5`。 - 遇到加法节点时，上游梯度按 $1.0$ 的局部权重原样传递：偏置 `b.grad` 为 $0.5$，同时乘积项之和节点的梯度也为 $0.5$。 - 另一个加法节点继续分发，使得 `(x1*w1).grad` 与 `(x2*w2).grad` 均为 $0.5$。 --- ### [01:06:20](https://www.youtube.com/watch?v=VMj-3S1tku0&t=3980s) 乘法节点的反向传播与链式法则应用 在完成了所有加法分支的梯度传递后，反向传播到达了具体的输入乘法运算环节，即 $x_2 \cdot w_2$ 与 $x_1 \cdot w_1$。在乘法操作中，局部导数表现为“交叉项”法则：对于表达式 $z = x \cdot y$，$\frac{\partial z}{\partial x} = y$，而 $\frac{\partial z}{\partial y} = x$。 Karpathy 引导观众共同心算推导：在计算输入特征 $x_2$ 的梯度 `x2.grad` 时，需要将局部导数（即对应的权重数据 `w2.data`）与来自上游输出节点 $x_2 w_2$ 的梯度 `(x2*w2).grad` 相乘。应用链式法则后，其具体数值即为 `w2.data * (x2*w2).grad`。这一步骤展示了神经网络如何通过局部权重与上游回传信号的逐层连乘，将最终输出对每一个底层输入特征与突触权重的敏感度精确量化。 ### [01:06:51](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4011s) 完成人工反向传播并直观理解局部梯度 通过链式法则，计算图中最后几个叶子节点的梯度计算得以完成。对于节点乘法 $x_2 \times w_2$，局部导数交叉相乘：$w_2\text{.grad} = x_2\text{.data} \times (x_2 w_2)\text{.grad}$，而 $x_2\text{.grad} = w_2\text{.data} \times (x_2 w_2)\text{.grad}$。重新绘制计算图后发现，$w_2$ 的梯度为 0，因为其对应的输入数据 $x_2\text{.data} = 0$；而 $x_2$ 的梯度为 0.5（由于 $w_2\text{.data} = 1.0$ 且上游梯度为 0.5）。对此给出了极具直觉的解释：导数度量的是微调某个变量对最终输出产生的影响；如果输入为 0，无论如何微调其权重 $w_2$，由于被乘以 0 彻底“扼杀（squashing）”，最终输出都不会发生任何改变，因此其导数为 0 完全合情合理。 紧接着处理另一个分支 $x_1 \times w_1$。上游传递过来的梯度同样为 0.5。乘法的局部导数对 $x_1$ 而言是 $w_1\text{.data}$，对 $w_1$ 而言是 $x_1\text{.data}$。经过链式法则相乘，最终求得 $x_1\text{.grad} = -1.5$（由 $-3.0 \times 0.5$ 得到），而 $w_1\text{.grad} = 1.0$（由 $2.0 \times 0.5$ 得到）。整个神经元表达式的人工反向传播至此彻底完成。Andrej Karpathy 总结道，如果我们希望提高神经元的输出，目前调节 $w_2$ 没有任何效果，但增大 $w_1$ 则能成比例地让神经元输出增加。 - 节点 $w_2$ 的最终梯度为 0.0，因其输入端 $x_2\text{.data} = 0$ - 节点 $x_2$ 的最终梯度为 0.5（$1.0 \times 0.5$） - 节点 $x_1$ 的最终梯度为 -1.5（$-3.0 \times 0.5$） - 节点 $w_1$ 的最终梯度为 1.0（$2.0 \times 0.5$） - 调优结论：“so if this weight goes up then this neuron's output would have gone up and proportionally because the gradient is one”（因此如果该权重增加，神经元的输出也会相应增加，且比例与为 1 的梯度一致）。 --- ### [01:09:02](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4142s) 在 Value 类中实现运算的 `_backward` 闭包 纯手工计算每个节点的导数繁琐且不切实际。为了告别这种折磨，必须将反向传播的自动化逻辑内置到 `Value` 对象中。核心思路是在每个节点上存储一个内部方法 `self._backward`。当节点由某种数学运算生成时，该方法利用 Python 闭包（closure）捕获输入节点和局部链式法则逻辑，负责将当前节点（`out`）的全局梯度反向传播至其父节点（`self` 和 `other`）。 对于加法操作 `__add__`，局部导数恒为 1.0，因此闭包函数直接将输出梯度复制给参与相加的两个输入节点： ```python def _backward(): self.grad += 1.0 * out.grad other.grad += 1.0 * out.grad out._backward = _backward ``` 对于乘法操作 `__mul__`，局部导数是对方节点的数值，因此闭包将上游梯度乘上对应数值并传递： ```python def _backward(): self.grad += other.data * out.grad other.grad += self.data * out.grad out._backward = _backward ``` 对于激活函数 `tanh`，其局部导数为 $1 - t^2$（其中 $t$ 为 `tanh` 的输出数值），因此闭包为： ```python def _backward(): self.grad += (1.0 - t**2) * out.grad out._backward = _backward ``` 对于叶子节点，默认赋予一个无操作的空函数 `lambda: None`。在此过程中 Karpathy 遇到了一个典型 bug：误写成 `out._backward = _backward()`，从而执行了该函数并赋了返回值 `None`，导致后续调用时抛出 `'NoneType' object is not callable` 错误。他幽默地决定保留这一调试失误：“I thought about redoing it but I figured I should just leave the error in here because it's pretty funny”（我本想重录这段，但觉得留着这个错误挺有意思的），以此强调我们是在把函数对象本身保存下来，而不是在定义时直接调用它。 ```mermaid flowchart LR accTitle: Value 对象的 _backward 闭包机制 accDescr: 展示输出节点通过 _backward 闭包将自身梯度与局部导数相乘并传递回父节点 subgraph Inputs ["输入父节点"] S["self (数值: self.data)"] O["other (数值: other.data)"] end subgraph Operation ["前向计算与闭包绑定"] OP{"运算节点 (+ / * / tanh)"} end subgraph OutputNode ["输出节点 out"] OUT["out.data & out.grad"] BW["out._backward() 闭包"] end S --> OP O --> OP OP --> OUT OUT -.->|"触发反向传播"| BW BW -->|"chain rule: local derivative * out.grad"| S BW -->|"chain rule: local derivative * out.grad"| O ``` --- ### [01:13:56](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4436s) 手动调用 `_backward` 逐步验证梯度传递 重新修复并定义好 `Value` 类后，神经元表达式被重新构建，所有节点的梯度初始重置为 0。自动化反向传播的第一步是基准设定：必须手动将最终输出节点 `o.grad` 显式设为 1.0，因为根据微积分恒等式 $\frac{\partial o}{\partial o} = 1.0$；如果不初始化，所有乘以上游梯度的计算都将因初始值 0 而归零。 随后依次调用各个节点的 `_backward` 方法： 1. 调用 `o._backward()`：通过 `tanh` 将梯度传递给激活前的和节点 $n$，计算得出 $n\text{.grad} = 0.5$。 2. 调用 `n._backward()`：加法操作将 0.5 复制给两个分支节点，分别将偏置 $b$ 和加法节点 $x_1 w_1 + x_2 w_2$ 的梯度更新为 0.5。 3. 试图在叶子节点 $b$ 上调用 `b._backward()`：由于 $b$ 是叶子节点，其绑定的默认闭包为空函数，调用后不产生任何副作用。 4. 调用加法节点 $(x_1 w_1 + x_2 w_2)\text{._backward()}$：将 0.5 再次分流至乘积节点 $x_1 w_1$ 与 $x_2 w_2$。 5. 分别调用 $(x_1 w_1)\text{._backward()}$ 与 $(x_2 w_2)\text{._backward()}$。 重新渲染图形化界面后，计算图中各节点的数值与梯度完全还原了之前手动计算的结果（$0$、$0.5$、$-1.5$、$1.0$）。这验证了通过闭包封装局部链式法则的正确性。 --- ### [01:17:32](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4652s) 引入拓扑排序实现整图自动反向传播 虽然每个节点不再需要手动写公式计算梯度，但仍然必须人工按从右向左的顺序一个一个执行 `_backward()`。为了实现彻底的自动化，整个表达式计算图的逆向执行顺序必须由程序自动确定。 在有向无环图（DAG）中，只有当一个节点的所有后继节点（即所有依赖它的计算）都已完成了反向传播并将梯度汇入该节点后，该节点才能安全地触发自身的 `_backward()`。这种在依赖图上排列节点先后次序的经典算法即为**拓扑排序**（Topological Sort）。Karpathy 指出，拓扑排序的直观含义就是对图中的所有节点进行线性排列，使得图中所有的有向边仅从左向右单向延伸。只要构建出这种从输入到输出的正向依赖拓扑序列，然后将其完全翻转（reverse），即可得到严格自输出向输入推进的反向传播调用序列，从而实现端到端的 `o.backward()`。 ### [01:18:46](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4726s) 拓扑排序与整图反向传播的自动化实现 为了让整个计算图能够自动执行反向传播，必须按照依赖顺序逆向遍历所有节点。Karpathy 详细演示了如何通过拓扑排序（Topological Sort）来实现这一点。算法的核心是维护一个已访问节点的集合 `visited`，从输出根节点（例如单个神经元的输出 $o$）开始递归构建有向无环图（DAG）的节点序列。对于当前访问的节点，先检查其是否已访问过；若未访问，则将其标记为已访问，并递归遍历其所有子节点（即输入操作数），调用构建函数。只有当当前节点的所有子节点全部处理完成并加入列表后，该节点自身才会被追加到 `topo` 列表中。 ```mermaid flowchart LR accTitle: 拓扑排序构建反向传播调用链 accDescr: 展示拓扑排序自左向右构建的节点顺序，以及反向传播时逆序执行的过程 subgraph TopoOrder["拓扑排序正向依赖序列 (topo)"] Inputs["输入变量<br/>(x1, w1, x2, w2, b)"] --> Intermediates["乘积与求和中间节点<br/>(x1w1, x2w2, x1w1 + x2w2, n)"] Intermediates --> OutputNode["根节点<br/>o (0.7071)"] end subgraph BackwardPass["反向传播执行流 (reversed(topo))"] direction RL BackRoot["o.grad = 1.0<br/>o._backward()"] --> BackInter["中间节点调用<br/>_backward()"] BackInter --> BackInputs["叶子节点计算完毕"] end ``` 这种机制保证了一个关键的不变量（invariant）：“so basically this node that we're going to call it on like say o is only going to add itself to the topo list after all of the children have been processed and that's how this function is guaranteeing that you're only going to be in the list once all your children are in the list and that's the invariant that is being maintained”。在神经元图上执行此函数后，输出列表的最后一个元素就是数值为 $0.707$ 的根节点 $o$，其前置节点为 $n$，其余所有节点均从左到右严格依照依赖关系排列。 接下来，Karpathy 将整个反向传播过程串联起来： 1. 将所有节点的梯度重置为 $0$。 2. 将基准情况 $o.\text{grad}$ 设为 $1.0$。 3. 对生成的 `topo` 列表调用 Python 内置的 `reversed()` 进行反向遍历，依次对每个节点调用 `node._backward()`。 代码执行后，终端中打印出的各节点梯度与此前纯手动链式求导的结果完全一致。为了让 API 像 PyTorch 一样优雅，Karpathy 将整套逻辑封装到 `Value` 类的公有方法 `backward()` 中：该方法内部自动完成以 `self` 为起点的拓扑排序，将 `self.grad` 设为 $1.0$，并倒序调用所有节点的内部私有方法 `_backward()`。用户只需在最终标量输出上轻轻调用一次 `o.backward()`，无需再编写任何繁琐的拓扑遍历代码。 ### [01:22:28](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4948s) 多分支复用节点的梯度覆盖 Bug 与累加修复 当自动化反向传播封装完毕后，Karpathy 警告大家先别高兴得太早：“now we shouldn't be too happy with ourselves actually because we have a bad bug um and we have not surfaced the bug because of some specific conditions that we are we have to think about right now”。这个 Bug 在当前神经元图中被掩盖了，因为计算图中的每一个节点恰好都只被后续操作使用了一次。然而，只要一个变量在后续表达式中被引用多次，当前的求导实现就会彻底崩溃。 Karpathy 现场写下了一个最小复现用例：定义变量 $a = 3.0$，然后定义 $b = a + a$，最后调用 $b.\text{backward()}$。在前向传播中，$b$ 的数值正确地得到了 $6.0$。但在微积分的数学直觉上，$\frac{\partial b}{\partial a} = \frac{\partial (a + a)}{\partial a} = 1 + 1 = 2$。而当前的程序运行后，$a.\text{grad}$ 却变成了 $1$！ ```mermaid flowchart TD accTitle: 变量复用时的梯度反向传播 accDescr: 显示一个变量在图中多次被消费时，反向传播分支必须汇聚相加 subgraph ForwardPass["前向传播：变量复用"] A["a"] -->|"左输入分支"| B["b = a + a"] A -->|"右输入分支"| B end subgraph BackwardBug["错误机制（覆盖赋值）"] B_grad["b._backward()"] -.->|"self.grad = 1.0"| A_grad1["a.grad = 1.0"] B_grad -.->|"other.grad = 1.0 (覆盖!)"| A_grad2["a.grad = 1.0 (最终为 1)"] end subgraph BackwardFix["正确机制（多元微积分链式法则）"] B_back["b._backward()"] ==>|"+= 1.0"| A_acc["a.grad = 0 + 1.0 + 1.0 = 2.0"] end ``` 出现这一严重 Bug 的底层原因在于加法反向传播的赋值逻辑：在 `__add__` 的私有求导中，代码写的是 `self.grad =...` 和 `other.grad =...`。当执行 $a + a$ 时，`self` 和 `other` 指向的是内存中的同一个对象（即 $a$）。反向传播执行时，先将 $a.\text{grad}$ 设为 $1$，紧接着又将其重新赋值为 $1$，第二条分支的梯度直接覆盖了第一条分支。在更复杂的结构中，例如 $d = a \times b$、$e = a + b$、$f = d \times e$，变量 $a$ 和 $b$ 同时流向乘法和加法两条支路，当从输出 $f$ 反向穿过 $e$ 和 $d$ 时，后执行的反向函数会无情地洗掉先前计算出的梯度贡献。 从多元微积分的链式法则（Multivariate Chain Rule）来看，如果一个变量的输出影响了多个下游分支，那么该变量的总梯度必须是所有下游分支反向传播回来的梯度之和。因此，修复这一 Bug 的方案极其直观简洁： - 将所有操作符（`+`、`*`、`tanh` 等）内部的梯度赋值语句 `self.grad =...` 全部替换为累加语句 `self.grad +=...`。 - 因为所有节点的梯度在初始时都严格被置为 $0$，所以来自不同支路的梯度贡献自然会在各自的节点上叠加累积。 Karpathy 随手把代码修改为 `+=` 并重新运行测试：$b = a + a$ 算出的 $a.\text{grad}$ 瞬间修正为 $2.0$；复杂分支网络中的多路贡献也全部正确汇总。 ### [01:27:05](https://www.youtube.com/watch?v=VMj-3S1tku0&t=5225s) 支持标量运算：重载操作符的类型包装 解决完梯度累加的 Bug 后，Karpathy 动手清理了 Jupyter Notebook 中用于验证的临时单元格，只保留核心的 `Value` 类和图形绘制代码。此时他的目光转向了此前实现的激活函数 $\tanh$。在之前的代码里，$\tanh$ 被作为一个不可分割的黑盒原子算子整体实现。但事实上，$\tanh$ 在数学上可以展开为以自然常数 $e$ 为底的指数运算： $$\tanh(x) = \frac{e^{2x} - 1}{e^{2x} + 1}$$ Karpathy 强调，虽然直接实现 $\tanh$ 更加快捷，但将其拆解为更基础的微观算子（指数、除法、减法、加法）具有极大的教学价值：“i would like to do that now because i want to prove to you that you get all the same results and all those ingredients but also because it forces us to implement a few more expressions”。这样做不仅能向学习者直观证明黑盒函数与底层算子组合在求导结果上完全等价，更倒逼微架构补充一系列必须的基础数学运算。 然而，在尝试拆解 $\tanh$ 时，首先暴露出来的工程障碍是数据类型的兼容性。在 Python 中，如果用户尝试写形如 `a + 1` 或 `a * 2`（其中 `a` 为 `Value` 实例）的代码，解释器会当场抛出异常：`AttributeError: 'int' object has no attribute 'data'`。这是因为在 `__add__(self, other)` 和 `__mul__(self, other)` 内部，代码直接尝试访问 `other.data`，而原生 Python 的 `int` 或 `float` 显然没有这个属性。 为了提供像 PyTorch 那样符合人体工学的优雅语法体验，Karpathy 在算子入口处增加了类型安全校验与自动封装： - 在 `__add__` 中：如果传入的 `other` 并非 `Value` 的实例，则自动将其视为普通数值并封装为 `Value(other)`。 - 在 `__mul__` 中：采用完全相同的防御性逻辑，若 `other` 不是 `Value` 实例，则包装为 `Value(other)`。 修改后，`a + 1` 和 `a * 2` 均能顺畅运行并保留梯度的流通通路。然而 Karpathy 紧接着抛出了一个更为隐蔽的语言陷阱：“now here's a kind of unfortunate and not obvious part a times two works we saw that but two times a is that gonna work you'd expect it to right but actually it will not”。当操作数位置颠倒写成 `2 * a` 时，Python 默认会首先尝试调用整数对象的乘法方法 `int.__mul__(2, a)`，而标准的 Python 整数显然完全不认识用户自定义的 `Value` 类，程序再次面临报错崩溃的边缘。这一微妙细节为后续引入 Python 的反射双目运算符（如 `__rmul__`）埋下了伏笔。 ### [01:29:22](https://www.youtube.com/watch?v=VMj-3S1tku0&t=5362s) Python 反向乘法机制 `__rmul__` 的引入 在进行数值运算时，`a * 2` 会自然触发 Python 底层的 `a.__mul__(2)` 调用，这对自定义的 `Value` 对象来说完全没问题。然而，当操作数顺序调换写成 `2 * a` 时，Python 会首先尝试调用整数对象的乘法方法 `2.__mul__(a)`。由于内置的整数 `int` 完全不知道该如何与自定义的 `Value` 对象相乘，这个调用会直接报错困惑。 为了解决这个问题，Andrej Karpathy 解释了 Python 的后备降级（fallback）机制：实现 `__rmul__` 方法。当 Python 发现左侧操作数无法执行乘法时，会主动检查右侧操作数 `a` 是否知道如何乘上 `2`，进而触发 `Value` 上的 `__rmul__`。Karpathy 为此编写的代码极其优雅直观，只需调换操作数顺序即可： ```python def __rmul__(self, other): return self * other ``` “So because Python can't do 2 times a, it will check: is there an `__rmul__` in `Value`? And because there is, it will now call that.” Andrej Karpathy 解释道。借由这一重定向，`2 * a` 会交由 `__rmul__` 接管并转回已经实现好的 `a * 2`，再次运行后表达式顺利求值为 `4`。 --- ### [01:30:33](https://www.youtube.com/watch?v=VMj-3S1tku0&t=5433s) 实现指数运算 `exp` 及其反向传播 为了能够将复合的双曲正切函数（tanh）完全拆解为更基础的数学运算，系统还需要支持指数与除法。Karpathy 首先着手实现单变量的指数运算 `exp()`。该方法在结构上与 `tanh` 十分相似，都是接收一个标量输入并输出一个标量。前向传播直接取出内部数值 `self.data`，调用标准库的 `math.exp()` 进行计算，并将其包装为新的 `Value` 返回。 随后，Karpathy 留出思考空档，引导大家推导其反向传播的链式法则。指数函数有一个广为人知的微积分特性：$e^x$ 对 $x$ 的导数依然是 $e^x$。因为前向传播的结果已经保存在 `out.data` 中，本地导数无需重复计算，只需将其与输出节点的梯度 `out.grad` 相乘，即可完成梯度的反向回传： $$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial \text{out}} \cdot \frac{\partial \text{out}}{\partial x} = \text{out.grad} \cdot \text{out.data}$$ 代码中直接累加至 `self.grad += out.data * out.grad`。这一实现简洁高效，成功为 `Value` 对象赋予了指数运算与导数链式传递的能力。 --- ### [01:31:52](https://www.youtube.com/watch?v=VMj-3S1tku0&t=5512s) 泛化除法运算：基于幂函数 `__pow__` 的实现 在处理除法时，Karpathy 没有直接编写单独的除法求导逻辑，而是选择实现一种更为通用、数学上更强大的算子：常数幂运算 `__pow__`。由于除法 $a / b$ 在代数上严格等价于 $a \cdot b^{-1}$，只要系统支持 $x^k$ 形式的幂运算，除法就变成了幂运算与乘法的组合特例： $$\frac{a}{b} = a \cdot b^{-1}$$ 在编写 `__pow__(self, other)` 时，Karpathy 明确做出类型断言，要求指数 `other` 必须是标准的 Python `int` 或 `float` 数值，而不能是图中的 `Value` 节点，因为变量作为底数和常数作为指数的导数形式与两节点互求导数截然不同。对于幂函数 $y = x^n$，微积分中的幂法则（Power Rule）表明其导数为： $$\frac{d}{dx} x^n = n \cdot x^{n - 1}$$ 对应的链式法则反向传播实现如下： ```python def _backward(): self.grad += (other * (self.data ** (other - 1))) * out.grad ``` 在此基础上，除法运算 `__truediv__(self, other)` 仅用单行代码即可优雅实现：`return self * (other**-1)`。 紧接着，Karpathy 又以极简方式补齐了减法 `__sub__(self, other)`。由于减法本质上是加上一个相反数（$a - b = a + (-1 \cdot b)$），因此直接将其定义为 `self + (-other)`，而一元取负 `__neg__` 则由 `self * -1` 构成。所有高级算子都被分解并映射到加法、乘法与常数幂三个核心基石上。 ```mermaid flowchart LR accTitle: 复合运算向核心基础算子的映射关系 accDescr: 展示减法与除法如何通过幂运算与乘法间接实现 subgraph Composite["高阶运算定义"] Sub["减法 a - b"] Div["除法 a / b"] end subgraph CoreOps["核心基础算子"] Neg["取负 (-1 * b)"] Add["加法 a + (-b)"] Pow["幂运算 b^(-1)"] Mul["乘法 a * b^(-1)"] end Sub --> Neg --> Add Div --> Pow --> Mul ``` - **幂法则对应关系**：指数 `n` 对应传入的标量 `other`，底数 $x$ 取自内部属性 `self.data`。 - **运算优先级自洽**：Python 中幂运算符 `**` 优先级高于乘法 `*`，运算表达式执行顺序天然严谨。 - **正向验证正确**：输入 `Value(2.0) / Value(4.0)`，前向计算精准返回期待值 `0.5`。 --- ### [01:36:20](https://www.youtube.com/watch?v=VMj-3S1tku0&t=5780s) 拆解双曲正切 tanh 验证梯度等价性 准备好完整的算子体系后，Karpathy 回到最初构建的双输入神经元案例，着手进行核心验证：将原本黑盒封装的单一激活函数节点 `tanh`，拆解为底层的微观数学算式： $$\tanh(x) = \frac{e^{2x} - 1}{e^{2x} + 1}$$ 在代码实现中，输入经过仿射变换得到加权和 $n$ 之后，系统先计算中间节点 $e = (2 \cdot n).\text{exp}()$，随后将输出定义为 $o = (e - 1) / (e + 1)$。 “Number one we're expecting to see a much longer graph here because we've broken up tanh into a bunch of other operations. But those operations are mathematically equivalent, and so what we're expecting to see is number one the same result here so the forward pass works, and number two because of that mathematical equivalence we expect to see the same backward pass and the same gradients on these leaf nodes.” Andrej Karpathy 强调道。 运行反向传播并绘制计算图后，整张拓扑图明显拉长，原先紧凑的单一 `tanh` 节点被细粒度地展开为一连串由指数节点 `exp`、乘法节点 `*`、加法节点 `+` 以及包含 `-1` 幂次方的复合子网络。核对各叶子节点的最终导数： - 偏置与第二组输入权重：`w2.grad = 0.0`，`x2.grad = 0.5` - 第一组输入与权重：`w1.grad = 1.0`，`x1.grad = -1.5` 计算出的叶子节点梯度与此前使用整体 `tanh` 公式推导的结果完全一致。Karpathy 指出，这次试验的核心启示在于：**在自动微分引擎中，实现操作节点的粒度完全取决于开发者自身**。既可以细致入微地为每一个独立的 `+` 或 `*` 编写微观反向传播，也可以将任意复合逻辑（如整个 `tanh`）打包成一个独立粗粒度节点，只要其局部微积分导数推导正确，宏观上的链式求导结果在数学上必然等价。 ### [01:39:03](https://www.youtube.com/watch?v=VMj-3S1tku0&t=5943s) 复合运算的本质与对标工业级框架 PyTorch 在此前对 $\tanh$ 进行细粒度拆解与梯度反向传播后，Andrej Karpathy 总结道，算子分解成原子级的步骤从根本上讲是一种抽象视角：“只要你能写出局部梯度，就能链式传递梯度并继续反向传播，因此这些函数的设计完全取决于你”（“if you can write the local gradients you can chain the gradient and you can continue back propagation so the design of what those functions are is completely up to you”）。随后，他转向展示如何使用现代深度学习工业级库 PyTorch 来复现一模一样的计算图。Micrograd 的底层架构在设计之初就大致参考了 PyTorch 的 API 规范。 ```mermaid flowchart LR accTitle: Comparison between Micrograd and PyTorch accDescr: Micrograd operates on scalar Value objects while PyTorch wraps data in multi-dimensional Tensors. subgraph Micrograd["Micrograd (标量级引擎)"] direction TB MVal["Value(2.0)<br/>标量对象"] MData["属性:.data,.grad"] MVal --> MData end subgraph PyTorch["PyTorch (工业级张量引擎)"] direction TB PTen["torch.Tensor([2.0])<br/>float64 标量张量"] PReq["requires_grad=True"] PData["属性:.data,.grad<br/>取值:.item()"] PTen --> PReq --> PData end ``` Micrograd 本质上是一个标量引擎，只能处理 `2.0` 这样的单个数字；而在 PyTorch 中，所有运算均基于张量（Tensor），即由标量组成的 $n$ 维数组。为了完全对标，Karpathy 构造了只包含单一元素的标量张量。在默认情况下，PyTorch 的浮点张量采用 32 位单精度（`float32`），但由于原生 Python 的浮点数是 64 位双精度，为了确保数值计算完全一致，他在创建张量时显式将其强转为双精度（`double`，即 `float64`）。此外，出于计算效率考量，PyTorch 对网络输入叶子节点的求导追踪默认是关闭的（`requires_grad=False`），因此在定义叶子节点时必须显式声明 `requires_grad=True`。 - Micrograd 专注于单标量数据结构，而 PyTorch 原生支持多维标量数组（例如 $2 \times 3$ 维张量），能够高效执行并行矩阵运算。 - Python 原生浮点数为双精度，PyTorch 默认张量类型为 `float32`；通过 `.double()` 转换为 `float64` 可以与 Python 精度完全匹配。 - 叶子节点必须手动配置 `x.requires_grad = True`，PyTorch 默认不为叶子节点计算梯度以节省开销。 ### [01:42:01](https://www.youtube.com/watch?v=VMj-3S1tku0&t=6121s) PyTorch 前向传播与梯度验证 定义好包含叶子节点的标量张量后，Karpathy 演示了使用 PyTorch 执行加法、乘法以及 `torch.tanh` 运算的过程。PyTorch 计算得到的结果对象依然是张量，并且与 Micrograd 的设计完全对称，同样具有 `.data` 和 `.grad` 属性。调用反向传播方法 `o.backward()` 后，可以通过 `.grad` 检查各输入的偏导数。如果需要从单元素张量中解包出原生的 Python 浮点数，只需调用 `.item()` 即可。 执行代码后，PyTorch 的前向传播输出为 `0.707`，反向传播所得的叶子节点梯度分别为 `0.5`、`0.0`、`-1.5` 与 `1.0`。这与此前在 Micrograd 中手动计算以及通过拆解算子计算出的数值完全一致。Karpathy 总结道，Micrograd 所构建的一切，实际上就是 PyTorch 在所有张量均为单元素标量时的特殊情况；而 PyTorch 之所以强大，在于它能够在此数学抽象之上利用底层高效张量进行大规模并行处理。 | 节点对象 | Micrograd 前向输出 / 梯度 | PyTorch 前向输出 / 梯度 (`.item()`) | 结论 | |:--- |:--- |:--- |:--- | | **输出 $o$** | 0.7071 | 0.7071 | 前向计算完全一致 | | **输入 $x_1$ 梯度** | 0.5 | 0.5 | 局部偏导完全对标 | | **权重 $w_1$ 梯度** | 0.0 (微小浮点误差趋向 0) | 0.0 | 局部偏导完全对标 | | **输入 $x_2$ 梯度** | -1.5 | -1.5 | 局部偏导完全对标 | | **权重 $w_2$ 梯度** | 1.0 | 1.0 | 局部偏导完全对标 | ### [01:43:55](https://www.youtube.com/watch?v=VMj-3S1tku0&t=6235s) 搭建神经网络库：单个神经元实现 确立了自动求导引擎的可靠性后，Karpathy 开始在 Micrograd 之上自底向上搭建神经网络库。他指出，神经网络本质上就是一类特殊的数学表达式。为了使代码架构规范化，他仿照 PyTorch 的 `nn.Module` 设计风格，开始构建多层感知机（MLP，Multi-Layer Perceptron）的组件。第一步是实现一个可调用的单个神经元类 `Neuron`。 ```mermaid flowchart LR accTitle: Computation inside a single Neuron accDescr: Inputs are paired with weights, multiplied, summed with bias, and squashed by tanh. subgraph Inputs["输入与权重"] X["输入向量 x"] W["随机权重 w ~ [-1, 1]"] end subgraph DotProduct["点积求和"] Mul["pairwise 逐项相乘: wi * xi"] Bias["偏置 b ~ [-1, 1]"] Sum["sum(..., start=b)<br/>累加激活值 act"] end subgraph Activation["非线性激活"] Tanh["act.tanh()"] Out["输出 out"] end X & W --> Mul --> Sum Bias --> Sum --> Tanh --> Out ``` `Neuron` 的构造函数接收输入维度 `nin`（Number of Inputs），并为每个输入通道在 $[-1.0, 1.0]$ 区间内均匀随机初始化一个标量权重 `w`，同时初始化一个控制神经元触发门槛的标量偏置 `b`。在 Python 中，通过实现特殊方法 `__call__(self, x)`，可以使类实例支持类似于函数调用的语法（例如 `n(x)`）。神经元的前向计算过程即经典的加权点积运算加上偏置：$act = \sum (w_i \cdot x_i) + b$。 - 在 `__call__` 的实现中，Karpathy 运用 Python 内置函数 `zip(self.w, x)` 将权重列表和输入向量按位置配对打包成元组迭代器。 - 为了实现优雅的累加，他利用内置函数 `sum` 的第二个可选参数 `start`，将起始值直接设为偏置 `self.b`（即 `sum((wi * xi for wi, xi in zip(self.w, x)), self.b)`），免去额外的加法操作。 - 累加所得的原始激活值 `act` 经过非线性激活函数 `act.tanh()` 压缩后返回，确保神经元具备非线性表达能力。 ### [01:47:55](https://www.youtube.com/watch?v=VMj-3S1tku0&t=6475s) 神经元层（Layer）的抽象封装 在单个神经元搭建完成后，Karpathy 进一步抽象出神经网络层（`Layer`）。根据多层感知机的拓扑设计，同一层内的各个神经元之间互不相连，但每个神经元都全连接（Fully Connected）到上一层传入的全部输入。因此，一个神经元层在概念上就是一组独立对相同输入进行评估的神经元集合。 Karpathy 编写了 `Layer` 类：其构造函数接收两个参数——单个神经元的输入维度 `nin`，以及当前层包含的神经元数量 `nout`（即该层的输出维度）。该类内部维护一个由 `nout` 个独立初始化的 `Neuron(nin)` 构成的列表。当对该层实例进行调用时，它会对传入的输入向量依次执行各个神经元的计算，并以列表形式返回包含所有神经元输出的新向量。通过实例化一个接收 2 维输入、包含 3 个神经元的层（`Layer(2, 3)`），并传入示例输入，程序成功输出了由 3 个标量激活值构成的列表，完成了向多层感知机架构迈进的关键一步。 ### [01:48:53](https://www.youtube.com/watch?v=VMj-3S1tku0&t=6533s) 构建完整的多层感知机（MLP） 在完成单个神经元和单层神经元的构建后，Andrej Karpathy 将这两者组合以构建完整的**多层感知机**（MLP, Multi-Layer Perceptron）。在 MLP 中，网络各层按顺序依次首尾相接，前一层的输出直接作为后一层的输入。为了节省时间，Karpathy 快速贴出实现代码并进行拆解：初始化方法接收输入维度 `nin` 以及一个列表 `nouts`，该列表明确指定了 MLP 中每一层的神经元数量（例如 `[4, 4, 1]` 表示两个包含 4 个神经元的隐藏层和一个单输出神经元层）。代码将 `[nin] + nouts` 拼成一个完整的尺寸列表，遍历相邻的尺寸元组依次实例化各个 `Layer` 对象；在 `__call__` 方法中，输入 `x` 沿着这些层被顺序调用并传递下去。 为了匹配之前的架构图，Karpathy 实例化了一个包含 3 个输入、两个 4 神经元隐藏层和 1 个输出神经元的网络结构：`MLP(3, [4, 4, 1])`。在测试前向传播时，传入一个 3 维输入向量（如 `[2.0, 3.0, -1.0]`），网络顺利返回结果。不过 Karpathy 注意到，因为每一层都默认返回列表，即使输出层只有 1 个神经元，得到的结果也是被包裹在列表中的单元素 `[Value(...)]`。为了调用上的方便，他在 `Layer` 的实现中加入了一行判断逻辑：如果输出列表长度恰好为 1，则直接解包返回单个 `outs[0]`，否则返回完整列表。随后，他尝试调用 `draw_dot(n(x))` 绘制计算图。整个 MLP 前向传播展开后是一张庞大且错综复杂的计算图，层层推进直至最终的单个输出值。Karpathy 感叹道，这种表达式在纸笔上求导几乎不可想象，但借助 micrograd 的自动化机制，反向传播可以轻而易举地穿透整张图，计算出所有层神经元权重的导数。 ```mermaid flowchart LR accTitle: MLP Architecture Flow accDescr: A visual diagram showing a multi-layer perceptron with 3 inputs, two hidden layers of 4 neurons each, and 1 output neuron. subgraph Inputs["输入层 (3 维)"] X["x: [x0, x1, x2]"] end subgraph Layer1["Layer 1 (4 个神经元)"] L1["Layer(3, 4)"] end subgraph Layer2["Layer 2 (4 个神经元)"] L2["Layer(4, 4)"] end subgraph Layer3["输出层 (1 个神经元)"] L3["Layer(4, 1)"] end subgraph Output["最终输出"] Y["单个标量 Value (非列表)"] end X --> L1 L1 --> L2 L2 --> L3 L3 --> Y ``` ### [01:51:04](https://www.youtube.com/watch?v=VMj-3S1tku0&t=6664s) 构造小型数据集与定义均方误差损失函数（Loss） 有了可用的 MLP，Karpathy 构建了一个微型玩具数据集用于二分类任务。该数据集包含 4 个样本（每个样本是 3 维特征向量）以及对应的 4 个目标标签（`ys = [1.0, -1.0, -1.0, 1.0]`）。当网络在未经训练的初始状态下对这 4 个样本执行前向预测（`[n(x) for x in xs]`）时，输出结果分别为 `0.91`、`0.88`、`0.80` 和 `0.80`。Karpathy 逐一核对当前预测与真实标签的偏差：第一个样本预测值接近 1.0（表现良好），但中间两个样本的目标是 -1.0，当前输出却全是约 0.88 和 0.80，差距极大。 为了量化网络的预测质量并调整权重，深度学习的通用思路是“计算一个单一标量来衡量神经网络的整体表现，我们称这个数值为损失（loss）”。Karpathy 直言： > “The trick used in deep learning to achieve this is to calculate a single number that somehow measures the total performance of your neural net and we call this single number the loss.” 当前网络的损失必然很高，优化的目标就是使损失最小化。Karpathy 现场实现了**均方误差**（MSE, Mean Squared Error）损失： 1. 使用 `zip(ys, ypred)` 逐一配对真实标签与预测值。 2. 对每对标签计算差值并平方：`(yout - ygt)**2`。平方既能消除正负号的影响（类似于绝对值，确保非零误差始终为正），又能在预测与标签完全一致时让单项损失恰好为 0。 3. 将这 4 个样本的平方误差直接相加，得到总损失。 当前总损失约为 7.12。紧接着，Karpathy 执行了 `loss.backward()`，这行代码触发了全自动的梯度回溯。此时检查第一层第一个神经元的首个权重 `n.layers[0].neurons[0].w[0]`，其 `.grad` 已经填入了具体的导数值（为负数）。负梯度意味着如果微幅增加该权重，总损失就会下降。 - 输入数据集：4 个样本，每个样本为 3 个特征构成的输入向量。 - 目标标签：`ys = [1.0, -1.0, -1.0, 1.0]`。 - 初始未优化预测值：`[0.91, 0.88, 0.80, 0.80]`。 - 初始 MSE 损失值：约 `7.12`（其中第 2、3 个样本各贡献了约 3.5 左右的平方误差）。 ### [01:56:13](https://www.youtube.com/watch?v=VMj-3S1tku0&t=6973s) 庞大的计算图与网络参数（Parameters）收集 Karpathy 使用 `draw_dot(loss)` 渲染了整个损失函数的计算图。由于该图包含了全部 4 个样本的完整前向传播过程以及顶层的 MSE 损失归约，图形变得极其巨大且复杂，以至于他不禁惊呼：“哦天哪，我们现在构建出的这张图简直太夸张了。”整张庞大图谱以标量损失（7.12）作为根节点，反向传播算法沿着每一条前向路径反向渗透，将导数一路传递到最前端的输入层以及散布在各个神经元内部的权重（$w$）和偏置（$b$）。 Karpathy 指出，输入数据点（如输入的 `1.0`）同样被赋予了梯度，但这些梯度对训练并无实际价值，因为输入数据是固定的客观环境，无法人为篡改；唯独各层神经元的权重与偏置参数是可调节的。为了能够批量、统一地操作所有需要更新的变量，网络需要一个便捷接口来归拢全部参数。他在 `Neuron` 类中实现了 `parameters()` 方法，将神经元的权重列表与偏置拼接后返回：`self.w + [self.b]`。他提到，这种设计完全复刻了 PyTorch 中各个 `nn.Module` 的 `parameters()` 行为，其核心目的就是“收集神经网络的所有参数到一个数组中，以便我们能基于梯度信息同时对它们进行微调”。 ### [01:58:44](https://www.youtube.com/watch?v=VMj-3S1tku0&t=7124s) 为网络层与多层感知机实现参数收集接口 在处理完单个神经元的参数导出后，Andrej Karpathy 接着为更高层级的结构编写 `parameters` 方法。在 PyTorch 中，各种模块（Module）会返回参数张量（Tensors），而在 micrograd 中，返回的则是标量对象列表。`Layer` 本身也是一个模块，最初他写了一段直观但略显冗长的前向遍历代码：先声明一个空的 `params` 列表，用 `for neuron in self.neurons` 遍历层中的每个神经元，调用 `neuron.parameters()`，再通过 `params.extend(...)` 拼接到列表中。Karpathy 随即指出：“这代码量实在太冗长了（this is way too much code）”，并在 Python 中将其重构成嵌套列表推导式（list comprehension）：`return [p for neuron in self.neurons for p in neuron.parameters()]`。 ```mermaid flowchart TD accTitle: micrograd Parameter Aggregation Flow accDescr: Diagram showing how parameters propagate from Neuron to Layer to MLP. subgraph SG1 ["MLP 级别"] A["MLP.parameters()"] end subgraph SG2 ["Layer 级别"] B["Layer.parameters()"] end subgraph SG3 ["Neuron 级别"] C["Neuron.parameters()"] D["权重 w 与偏置 b<br/>(共 41 个标量)"] end A -->|"推导式遍历每层"| B B -->|"推导式遍历每个神经元"| C C -->|"返回 Value 列表"| D ``` 紧接着，Karpathy 对顶层的多层感知机 `MLP` 应用了完全相同的结构，写出 `return [p for layer in self.layers for p in layer.parameters()]`。由于刚给类添加了新的接口方法，而运行环境中的实例仍属于尚未定义该方法的旧类定义，Karpathy 只能重新实例化神经网络 `n`。重新初始化导致随机生成的权重发生了微小改变，损失函数的初始输出从之前的数值略微变成了 4.84。Karpathy 通过打印 `n.parameters()` 验证了这一机制，确认整个 3 层 MLP（输入 3 维，隐层各 4 维，输出 1 维）中所有权重和偏置总共包含 41 个参数标量。 - `Layer` 的参数提取实现：单行嵌套推导式 `[p for neuron in self.neurons for p in neuron.parameters()]` - `MLP` 的参数提取实现：单行嵌套推导式 `[p for layer in self.layers for p in layer.parameters()]` - 网络重新初始化导致预测值产生微小随机扰动，当前整个 MLP 的总参数量确定为 41 个 ### [02:01:12](https://www.youtube.com/watch?v=VMj-3S1tku0&t=7272s) 梯度下降的核心逻辑与符号推导 有了所有参数的统一接口后，下一步就是根据反向传播计算出的梯度来更新它们。Karpathy 选取了网络中的某个具体神经元参数进行直观剖析：该参数当前的数值 `p.data` 是 0.85，而其在当前损失上的梯度 `p.grad` 是一个负数。他解释说，如果希望沿着梯度方向调整参数，在梯度下降（gradient descent）的直觉中，“梯度向量指向的是损失增加的方向（in gradient descent we are thinking of the gradient as a vector pointing in the direction of increased loss）”。 为了最小化损失，更新方向必须与梯度方向相反。Karpathy 仔细推导了正负号的关系：如果采用 `p.data += 0.01 * p.grad`，由于该参数的梯度为负，参数本身会被减小（例如从 0.85 变到 0.84 左右）；但是偏导数为负意味着“增加该数值反而会使损失下降”，若将其减小，损失反而会上升。因此必须引入负号： ```python p.data += -0.01 * p.grad ``` 加上负号后，该参数从 0.854 微调上升到了 0.857。Karpathy 在单元格中重新运行前向传播验证了改动效果，整体损失果然如预期一般从 4.84 成功下降到了 4.36。 - 单步测试学习率（step size）：选用小步长如 0.01 - 符号逻辑：梯度为负意味着参数增大将使损失变小，因此更新量必须减去步长乘以梯度（即相反方向） - 首步更新验证：单次微调后损失从 4.84 下降到 4.36，证明符号与梯度传导完全正确 ### [02:05:08](https://www.youtube.com/watch?v=VMj-3S1tku0&t=7508s) 手动单步迭代与步长（学习率）的微妙平衡 在确认损失下降后，Karpathy 开始手动反复交替执行三步流程：前向传播计算损失、调用 `loss.backward()` 计算梯度、执行更新语句 `p.data += -0.01 * p.grad`。随着每一次手动点击运行，损失一路顺畅滑落：4.36、3.90、3.66、3.47。查看此时的预测向量 `ypred`，各个数值都在逐渐朝预定目标（1, -1, -1, 1）靠拢。 为了加速收敛，Karpathy 尝试将步长提高至 0.1。损失在极短时间内暴跌到了 0.31，并在随后降到了 0.04，此时输出预测值已经非常接近目标（如 0.99 级别的精度）。但在继续加大步幅激进点击时，训练过程瞬间出现了数值震荡（overstepping）。Karpathy 警示道，梯度仅代表局部变化率，如果步子迈得太大，就可能跳入一个损失曲面完全不同的区域，甚至导致损失爆炸。庆幸的是，在随后的一次迭代中数值奇迹般地恢复并收敛至超低状态（损失达到 $7 \times 10^{-9}$）。Karpathy 借此感慨道：“因此寻找恰到好处的步长在某种程度上是一门精妙的艺术，尤其是在使用这种朴素梯度下降算法时（finding the step size to be just right it's it's a pretty subtle art sometimes when you're using sort of vanilla gradient descent）”。 - 手动迭代路径：损失连续下降（4.36 -> 3.90 -> 3.66 -> 3.47） - 提高学习率到 0.1 后：损失先迅速下降至 0.04，随后因步长过大产生颠簸 - 最终收敛状态：损失跌至极低点 $7 \times 10^{-9}$，模型完美拟合目标数据 - 学习率权衡：过小收敛极慢，过大则会由于局部线性假设失效而越过极小点甚至发散 ### [02:08:38](https://www.youtube.com/watch?v=VMj-3S1tku0&t=7718s) 编写完整的梯度下降训练循环 在手动单步调优验证了整个流程之后，Karpathy 将散落的步骤整合为一个更规范、更具工程规范的自动化训练循环（training loop）。他清空了之前的临时测试代码，将整个网络和初始数据重新初始化，并设计了一个运行 20 个迭代步（steps）的 `for k in range(20):` 循环。 ```mermaid flowchart LR accTitle: Standard Gradient Descent Optimization Loop accDescr: The iterative cycle of forward pass, backward pass, and parameter update. A["前向传播<br/>Forward Pass"] --> B["计算损失<br/>Calculate Loss"] B --> C["反向传播<br/>loss.backward()"] C --> D["参数更新<br/>Update via -lr * p.grad"] D -->|"下一个 Step"| A ``` 在循环体内部，流程被严格固定为三个经典阶段： 1. **前向传播（Forward pass）**：遍历数据输入得到当前预测 `ypred`，进而计算均方误差损失 `loss`。 2. **反向传播（Backward pass）**：调用 `loss.backward()`，让反向传播算法沿着整个计算图自动计算出每个标量参数对最终损失的导数。 3. **参数更新（Update / Gradient Descent）**：遍历 `n.parameters()` 中的所有参数，根据反向传播计算出的梯度沿着负梯度方向更新每个标量的数值。 对于超参数的选取，Karpathy 结合刚才的实验经验指出，学习率设为 0.01 略显缓慢，而设为 0.1 又有发散风险，因此折中挑选了一个介于两者之间的值，并设置打印语句以便在控制台中清晰追踪每一个 step 的数值变化。 ### [02:09:59](https://www.youtube.com/watch?v=VMj-3S1tku0&t=7799s) 梯度未清零的经典 Bug 与修复 优化过程再次运行后，损失函数以一种更受控、更平缓的速度下降到了极低的值，预测值 `ypred` 看起来也非常接近目标。然而就在此时，Andrej Karpathy 发现了一个潜藏在代码中的严重漏洞。他自嘲道：“这真的有点尴尬，但我们代码里其实藏着一个非常严重的 Bug，而且是个很隐蔽、极其常见的 Bug，我真不敢相信自己这辈子已经是第 20 次犯这个错误了，尤其还是在镜头前录制的时候。”他原本可以把整段视频重录一遍，但他认为把这一幕留下来更有趣，也能让大家体会到平时和神经网络打交道时的真实常态。 ```mermaid flowchart TD accTitle: 训练循环中的梯度重置流程 accDescr: 展示了每次迭代中梯度清零、前向传播、反向传播与参数更新的正确执行顺序。 subgraph StepLoop["单步训练循环 (Training Step)"] A["1. zero_grad 重置梯度<br/>(p.grad = 0.0)"] --> B["2. 前向传播<br/>(计算 ypred 与 Loss)"] B --> C["3. 反向传播<br/>(loss.backward() 累加梯度)"] C --> D["4. 参数更新<br/>(p.data += -learning_rate * p.grad)"] end ``` 这个错误正是 Karpathy 很久以前发推列出的“神经网络最常见错误”排行榜里的第三名：**在调用 `backward()` 前忘记将梯度归零（zero grad）**。在 Micrograd 的设计中，所有权重参数都拥有 `data` 和 `grad` 两个属性。初始时 `grad` 为 0，第一次前向传播并调用 `backward()` 后，梯度被填入；随后虽然根据梯度更新了 `data`，但并没有清空 `grad`。当进入第二轮迭代再次调用 `backward()` 时，由于计算图中所有节点的反向传播操作都使用了 `+=`（累加），上一轮留下的梯度不仅没有被重置，反而在当前轮次中越积越大。 修复这个问题的方法非常直接，必须在反向传播前遍历神经网络的所有参数，显式地将梯度重置为零： ```python for p in n.parameters(): p.grad = 0.0 ``` Karpathy 解释称，这就等同于 PyTorch 中的 `zero_grad` 操作。之前之所以带有如此严重 Bug 的代码依然能够收敛，完全是因为当前处理的数据集和网络结构太过简单，神经网络能够极其轻易地拟合这 4 个样本；累积的梯度在效果上相当于无形中成倍放大了步长，导致优化速度快得反常。而在正式加入梯度清零操作并重新初始化网络后，损失下降的过程明显变得平稳且缓慢，需要增加迭代步数才能达到很低的损失值。Karpathy 提醒道：“搞神经网络有时非常棘手，因为代码里哪怕有一堆 Bug，网络可能依然看起来在正常工作；但如果面对的是复杂问题，这种 Bug 就会导致根本无法把损失优化下去。” - 错误根源：反向传播中的梯度计算是累加模式（`+=`），若不显式清理，上一轮的梯度就会无限叠加进下一轮。 - 掩盖现象：在极简单任务下，未清零的梯度被反复累加，相当于产生了一个巨大的有效学习率步长，给出了“快速收敛”的假象。 - 调试教训：在复杂问题上，此类梯度累积错误会导致训练直接崩盘或陷入糟糕局部解。 --- ### [02:14:03](https://www.youtube.com/watch?v=VMj-3S1tku0&t=8043s) 神经网络与梯度下降的核心原理总结 排除了 Bug 之后，Karpathy 对整个教学过程进行了体系化回顾。所谓的神经网络，本质上就是一串相对简单的数学表达式。以多层感知机（MLP）为例，它将输入数据以及网络自身的权重、偏置等参数结合起来，通过前向传播计算出输出，紧接着接入一个损失函数（Loss Function）。损失函数负责衡量网络预测的好坏程度：当预测值与目标值高度吻合时，损失就会很低；反之亦然。整个训练的核心思想，就是人工构建并操控这个损失函数，使得“当损失降到最低时，网络所做的事情正好是我们希望它完成的目标任务”。 定义好前向过程和损失后，算法通过反向传播（Backpropagation）计算出损失关于每一个参数的局部导数，进而得知如何微调参数以在局部降低损失；通过将这一过程不断循环迭代——即所谓的梯度下降（Gradient Descent），网络就能沿着梯度的反方向逐步逼近最优解。Karpathy 感慨道，虽然当前构建的只是一个仅有 41 个参数的微型网络，但现代那些具有数百亿乃至上万亿参数的巨型模型（如 GPT），本质上也是建立在完全相同的基石之上。 对于更大规模的模型而言，虽然网络结构更加精密、参数规模膨胀到了庞大无匹的“模拟神经组织”级别，但其内核并未改变： 1. **统一的表征与计算**：前向传播依然由数值（Value）与计算图构成，反向传播依然是沿着图追踪链式法则积累梯度。 2. **训练模式相同**：依然遵循前向计算、损失求导、梯度更新的迭代闭环。 3. **细分技术差异**：工程实践中会采用更高级的优化算法（而非极度简化的随机梯度下降），在自然语言预测等任务中会使用交叉熵损失（Cross-Entropy Loss）而非均方误差（MSE）。 Karpathy 强调，正是这种简单的数学机制经过海量参数与大规模数据的催化，才展现出了令人震撼的涌现能力（Emergent Properties）：“你在整个互联网文本上训练这个模型，它就会展现出非常惊人的涌现特性；但从根本原理来说，它的底层运转机制与我们今天手写的代码是完全一致的。” --- ### [02:16:46](https://www.youtube.com/watch?v=VMj-3S1tku0&t=8206s) Micrograd 源码深度解析：Engine 与 NN 模块 Karpathy 随后带领大家逐行检视了 Micrograd GitHub 仓库里的真实源码，以验证视频开头所说的承诺——“学到最后，你将彻底看懂 Micrograd 的每一个细节”。代码库的结构非常紧凑：根目录下的 `__init__.py` 是空文件，核心逻辑主要收敛在 `engine.py` 和 `nn.py` 两个核心模块中。 | 模块 / 文件 | 核心类与内容 | 职责与技术细节 | |:--- |:--- |:--- | | `engine.py` | `Value` 类 | 包含 `data`、`grad`、`_backward`、`_prev`（子节点集合）以及 `_op` 操作符。<br/>实现了基础加法、乘法与标量幂运算；所有衍生运算（如减法、除法）均由基础算子组合而成。 | | `engine.py` | 激活函数 | 源码当前内置了 ReLU 激活函数及其导数。<br/>与视频教学中推导的双曲正切函数（`tanh`）相比，ReLU 和 `tanh`、`sigmoid` 都可以作为非线性变换，视频中之所以选用 `tanh`，是因为其平滑的数学形式能更好地锻炼局部梯度的链式法则推导。 | | `nn.py` | `Module` 父类 | 借鉴了 PyTorch 的 `nn.Module` 接口规范，抽离出了公共方法，包括统一的 `zero_grad()` 梯度清零以及 `parameters()` 参数收集。 | | `nn.py` | `Neuron`、`Layer`、`MLP` | 分别继承自 `Module`，逐层抽象神经元输入加权求和、多神经元并行层、以及多层全连接感知机结构。 | | `test/` 单元测试 | 梯度与前向校验 | 在微型表达式上同时运行 Micrograd 和 PyTorch 代码，对前向输出值与反向计算得到的每一个梯度做逐一严格对比，断言结果完全相同。 | | `demo.ipynb` | 分类实战演练 | 演示了一个比教程中 4 个点更复杂的二维平面二分类任务（红蓝点月牙/同心圆分类），使用规模稍大的 MLP 模型进行端到端训练。 | Karpathy 解释了设计上的微小差异：Micrograd 源码中采用了基类 `Module` 来抽离 `zero_grad` 等通用逻辑，这完全是为了在 API 层面与 PyTorch 的 `torch.nn.Module` 保持高度统一。单元测试部分则通过对照 PyTorch 验证了代码的数学准确性。至此，从底层标量自动求导引擎到高层神经网络 API，整个框架的每一个模块都做到了完全透明、清晰可见。 ### [02:19:24](https://www.youtube.com/watch?v=VMj-3S1tku0&t=8364s) 批处理、高级损失函数与完整训练流程 Andrej Karpathy 接着梳理了 micrograd 官方仓库中展示的更完整训练示例。在之前的简易演示中，数据集仅包含 4 个样本，因此可以直接对全量数据执行前向传播。但在真实的工业级场景中，数据集规模动辄达数百万条，无法一次性塞入内存或单次计算图中。实际做法是随机抽取数据子集（即批次 batch），仅针对该批次进行前向传播、反向传播与参数更新，避免了在完整训练集上反复做昂贵的全量计算。 ```mermaid flowchart LR accTitle: Batch Training Loop Flowchart accDescr: Micrograd full training loop showing data batching, forward loss calculation with regularization, backward step, and learning rate decay update. subgraph Data["1. 数据采样"] B["抽取随机批次 (Batch)"] end subgraph Compute["2. 前向与损失"] FP["前向传播 (Forward)"] L["Max-Margin 损失 + L2 正则化"] B --> FP --> L end subgraph Optimize["3. 反向传播与优化"] ZG["梯度清零 (zero_grad)"] BP["反向传播 (backward)"] UPD["参数更新 (学习率衰减)"] L --> ZG --> BP --> UPD end ``` 在该进阶示例中，损失函数的设计也更为复杂：它采用了常用于二分类支持向量机（SVM）的 max-margin 损失，而视频此前演示中使用的是最容易理解的均方误差（Mean Squared Error, MSE）。此外，二元交叉熵损失（Binary Cross Entropy, BCE）也是常用的二分类目标，但在简单示例中它们的表现差异并不显著。该实现还引入了用于控制机器学习过拟合并提升泛化能力的 L2 正则化项（L2 Regularization）。整个训练循环由熟悉的标准步骤组成——前向计算、梯度清零、反向传播、梯度更新。值得注意的是，更新步骤中引入了学习率衰减（Learning Rate Decay）：初始阶段采用较高的学习率快速收敛，随着迭代推进逐步缩小步长，以帮助网络稳定捕捉决策边界的微调细节。最终，可视化生成的决策边界清晰展示了模型如何根据输入坐标成功将红蓝数据点完全分隔开。 - 进阶示例支持批处理（batching），避免对海量数据集进行全量前向计算。 - 引入 max-margin 损失替代之前的 MSE，并加入了 L2 正则化来抑制模型过拟合。 - 学习率随迭代次数增加而衰减（Learning Rate Decay），在训练初期保持高步长、后期细化微调。 --- ### [02:21:10](https://www.youtube.com/watch?v=VMj-3S1tku0&t=8470s) 深入 PyTorch 生产级底层实现：寻找 tanh 的反向传播代码 为了让观众感受“工业级实战”（real stuff）与教学玩具 micrograd 之间的差异，Karpathy 尝试在 PyTorch 的开源代码库中寻找 `tanh` 的反向传播实现。在 micrograd 中，`tanh` 的导数公式清晰透明：$\frac{d}{dx}\tanh(x) = (1 - t^2) \times \text{out.grad}$（其中 $t$ 为前向传播的输出值，后半部分来自链式法则）。Karpathy 希望在 PyTorch 庞大的 C++/CUDA 底层源码里找到完全相同的逻辑。 然而在工业级项目中定位核心代码极其困难。正如 Andrej Karpathy 所说：“And that's because these libraries unfortunately they grow in size and entropy and if you just search for tanh you get apparently 2800 results and 406 files so I don't know what these files are doing honestly.”这类工业级代码库由于历史演化和复杂度熵增，其设计初衷是便于调用而非便于拆解检视。在翻找约 15 分钟无果后，他借助一份试图修改 `tanh` 反向传播逻辑的提交讨论，顺藤摸瓜找到了 CPU 和 GPU（CUDA）对应的核函数（Kernel）。 在 CPU 内核中，代码之所以异常臃肿，是因为必须覆盖复数类型（complex types）、低精度类型（如 Bfloat16）等各种边界分支；剥除这些包装后，终于在名为 `BinaryOpsKernel` 的代码深处找到了核心逻辑：`a * (1 - b * b)`（其中 `b` 为 `tanh` 输出，`a` 为传入的梯度 `grad_output`）。类似地，在 CUDA 的 GPU 核函数里也找到了纯粹执行该代数运算的单行逻辑。这证实了工业级框架底层数学本质与 micrograd 完全相同，只是裹上了大量适配硬件与数据类型的繁复工程实现。 - 在 PyTorch 源码库搜索 `tanh` 会得到多达 2,800 条结果、涉及 406 个文件。 - 底层实现按硬件平台划分为 CPU Kernel 和 GPU (CUDA) Kernel。 - 底层 C++/CUDA 代码为了适配 `bfloat16`、复数计算及向量化优化而极度冗长，但核心计算本质依然是 `grad * (1.0 - output * output)`。 --- ### [02:23:43](https://www.youtube.com/watch?v=VMj-3S1tku0&t=8623s) 自定义 PyTorch 算子：如同乐高积木般的计算图拓展 Karpathy 展示了 PyTorch 官方文档中关于如何为系统注册自定义算子的示例。以实现三阶勒让德多项式（Legendre Polynomial 3）为例，开发者只需创建一个继承自 `torch.autograd.Function` 的类，并分别定义两个静态方法：`forward`（前向传播计算结果并保存必要上下文）与 `backward`（接收上游梯度并应用局部导数计算下游梯度）。 正如 Andrej Karpathy 所总结：“As long as you can do the forward pass of this little function piece that you want to add and as long as you know the local derivative the local gradients which are implemented in the backward PyTorch will be able to backpropagate through your function and then you can use this as a Lego block in a larger Lego castle of all the different Lego blocks that PyTorch already has.” 只要向框架提供局部的正向变换与局部导数逻辑，PyTorch 的自动微分引擎就能无缝接管并沿计算图完成反向传递。 - 继承 `torch.autograd.Function` 并实现 `forward` 与 `backward`，即可让自定义算子无缝接入 PyTorch 自动求导机制。 - 只要提供局部导数（Local Gradient），框架就能依据链式法则在由各色算子构建的“乐高城堡”中全局传导。 --- ### [02:24:39](https://www.youtube.com/watch?v=VMj-3S1tku0&t=8679s) 总结与花絮 在全课结尾，Karpathy 总结了整个 micrograd 构建旅程，希望观众通过从零手写标量引擎、反向传播与多层感知机，获得了对神经网络运作机理的本质洞察。他表示会在视频简介中提供相关链接与讨论社区，方便大家深入交流，并可能制作答疑视频汇总高频问题。在视频最后的趣味花絮（Outtakes）中，记录了录制过程中的多次吃螺蛳与口误瞬间：在推导梯度时突然卡壳自问“Wait what is the problem?”；把 micrograd 不小心念成了“micro-crab”（微型螃蟹）；以及在测试乘法算子重载边界时不慎报错爆出的“oops”。录制在轻松幽默的气氛中落下帷幕。 ## Notable quotes > [04:33](https://www.youtube.com/watch?v=VMj-3S1tku0&t=273s) My claim is that micrograd is what you need to train your networks and everything else is just efficiency so you'd think that micrograd would be a very complex piece of code and that turns out to not be the case. — Andrej Karpathy > [01:51:04](https://www.youtube.com/watch?v=VMj-3S1tku0&t=6664s) The trick used in deep learning to achieve this is to calculate a single number that somehow measures the total performance of your neural net and we call this single number the loss. — Andrej Karpathy ## Mentioned Andrej Karpathy, micrograd, PyTorch, JAX, Python, Jupyter Notebook, NumPy, Matplotlib, Graphviz, GitHub, CUDA, GPT, ReLU, SVM, Bfloat16 ## Video description This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school. Links: - micrograd on github: https://github.com/karpathy/micrograd - jupyter notebooks I built in this video: https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/micrograd - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - "discussion forum": nvm, use youtube comments below for now :) - (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube comments Exercises: you should now be able to complete the following google collab, good luck!: https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing Chapters: 00:00:00 intro 00:00:25 micrograd overview 00:08:08 derivative of a simple function with one input 00:14:12 derivative of a function with multiple inputs 00:19:09 starting the core Value object of micrograd and its visualization 00:32:10 manual backpropagation example #1: simple expression 00:51:10 preview of a single optimization step 00:52:52 manual backpropagation example #2: a neuron 01:09:02 implementing the backward function for each operation 01:17:32 implementing the backward function for a whole expression graph 01:22:28 fixing a backprop bug when one node is used multiple times 01:27:05 breaking up a tanh, exercising with more operations 01:39:31 doing the same thing but in PyTorch: comparison 01:43:55 building out a neural net library (multi-layer perceptron) in micrograd 01:51:04 creating a tiny dataset, writing the loss function 01:57:56 collecting all of the parameters of the neural net 02:01:12 doing gradient descent optimization manually, training the network 02:14:03 summary of what we learned, how to go towards modern neural nets 02:16:46 walkthrough of the full code of micrograd on github 02:21:10 real stuff: diving into PyTorch, finding their backward pass for tanh 02:24:39 conclusion 02:25:20 outtakes :) [Source](https://www.youtube.com/watch?v=VMj-3S1tku0)

[4:33] Andrej Karpathy: My claim is that micrograd is what you need to train your networks and everything else is just efficiency so you'd think that micrograd would be a very complex piece of code and that turns out to not be the case. [111:04] Andrej Karpathy: The trick used in deep learning to achieve this is to calculate a single number that somehow measures the total performance of your neural net and we call this single number the loss.

[4:33] Andrej Karpathy: my claim is that micrograd is what you need to train your networks and everything else is just efficiency

[0:00] hello my name is andre [0:01] and i've been training deep neural [0:02] networks for a bit more than a decade [0:04] and in this lecture i'd like to show you [0:06] what neural network training looks like [0:08] under the hood so in particular we are [0:10] going to start with a blank jupiter [0:12] notebook and by the end of this lecture [0:14] we will define and train in neural net [0:16] and you'll get to see everything that [0:18] goes on under the hood and exactly [0:20] sort of how that works on an intuitive [0:21] level [0:22] now specifically what i would like to do [0:24] is i would like to take you through [0:26] building of micrograd now micrograd is [0:29] this library that i released on github [0:30] about two years ago but at the time i [0:32] only uploaded the source code and you'd [0:34] have to go in by yourself and really [0:37] figure out how it works [0:39] so in this lecture i will take you [0:40] through it step by step and kind of [0:42] comment on all the pieces of it so what [0:44] is micrograd and why is it interesting [0:47] good [0:48] um [0:49] micrograd is basically an autograd [0:51] engine autograd is short for automatic [0:53] gradient and really what it does is it [0:55] implements backpropagation now [0:57] backpropagation is this algorithm that [0:59] allows you to efficiently evaluate the [1:01] gradient of [1:03] some kind of a loss function with [1:05] respect to the weights of a neural [1:07] network and what that allows us to do [1:09] then is we can iteratively tune the [1:11] weights of that neural network to [1:12] minimize the loss function and therefore [1:14] improve the accuracy of the network so [1:16] back propagation would be at the [1:18] mathematical core of any modern deep [1:20] neural network library like say pytorch [1:22] or jaxx [1:24] so the functionality of microgrant is i [1:25] think best illustrated by an example so [1:27] if we just scroll down here [1:29] you'll see that micrograph basically [1:31] allows you to build out mathematical [1:32] expressions [1:34] and um here what we are doing is we have [1:36] an expression that we're building out [1:37] where you have two inputs a and b [1:40] and you'll see that a and b are negative [1:43] four and two but we are wrapping those [1:46] values into this value object that we [1:48] are going to build out as part of [1:49] micrograd [1:51] so this value object will wrap the [1:53] numbers themselves [1:54] and then we are going to build out a [1:56] mathematical expression here where a and [1:58] b are transformed into c d and [2:01] eventually e f and g [2:03] and i'm showing some of the functions [2:05] some of the functionality of micrograph [2:07] and the operations that it supports so [2:08] you can add two value objects you can [2:11] multiply them you can raise them to a [2:13] constant power you can offset by one [2:15] negate squash at zero [2:18] square divide by constant divide by it [2:21] etc [2:22] and so we're building out an expression [2:24] graph with with these two inputs a and b [2:27] and we're creating an output value of g [2:30] and micrograd will in the background [2:32] build out this entire mathematical [2:34] expression so it will for example know [2:36] that c is also a value [2:38] c was a result of an addition operation [2:41] and the [2:42] child nodes of c are a and b because the [2:46] and will maintain pointers to a and b [2:48] value objects so we'll basically know [2:50] exactly how all of this is laid out [2:53] and then not only can we do what we call [2:55] the forward pass where we actually look [2:57] at the value of g of course that's [2:58] pretty straightforward we will access [3:00] that using the dot data attribute and so [3:03] the output of the forward pass the value [3:06] of g is 24.7 it turns out but the big [3:09] deal is that we can also take this g [3:11] value object and we can call that [3:13] backward [3:14] and this will basically uh initialize [3:16] back propagation at the node g [3:19] and what backpropagation is going to do [3:21] is it's going to start at g and it's [3:23] going to go backwards through that [3:25] expression graph and it's going to [3:26] recursively apply the chain rule from [3:28] calculus [3:30] and what that allows us to do then is [3:32] we're going to evaluate basically the [3:34] derivative of g with respect to all the [3:36] internal nodes [3:38] like e d and c but also with respect to [3:40] the inputs a and b [3:43] and then we can actually query this [3:45] derivative of g with respect to a for [3:47] example that's a dot grad in this case [3:50] it happens to be 138 and the derivative [3:52] of g with respect to b [3:54] which also happens to be here 645 [3:57] and this derivative we'll see soon is [3:59] very important information because it's [4:01] telling us how a and b are affecting g [4:04] through this mathematical expression so [4:06] in particular [4:08] a dot grad is 138 so if we slightly [4:11] nudge a and make it slightly larger [4:14] 138 is telling us that g will grow and [4:18] the slope of that growth is going to be [4:19] 138 [4:20] and the slope of growth of b is going to [4:22] be 645. so that's going to tell us about [4:25] how g will respond if a and b get [4:27] tweaked a tiny amount in a positive [4:29] direction [4:31] okay [4:33] now you might be confused about what [4:34] this expression is that we built out [4:36] here and this expression by the way is [4:38] completely meaningless i just made it up [4:40] i'm just flexing about the kinds of [4:42] operations that are supported by [4:43] micrograd [4:44] what we actually really care about are [4:46] neural networks but it turns out that [4:48] neural networks are just mathematical [4:49] expressions just like this one but [4:51] actually slightly bit less crazy even [4:54] neural networks are just a mathematical [4:56] expression they take the input data as [4:59] an input and they take the weights of a [5:00] neural network as an input and it's a [5:02] mathematical expression and the output [5:04] are your predictions of your neural net [5:06] or the loss function we'll see this in a [5:08] bit but basically neural networks just [5:10] happen to be a certain class of [5:12] mathematical expressions [5:13] but back propagation is actually [5:15] significantly more general it doesn't [5:17] actually care about neural networks at [5:18] all it only tells us about arbitrary [5:20] mathematical expressions and then we [5:22] happen to use that machinery for [5:24] training of neural networks now one more [5:26] note i would like to make at this stage [5:28] is that as you see here micrograd is a [5:30] scalar valued auto grant engine so it's [5:32] working on the you know level of [5:34] individual scalars like negative four [5:36] and two and we're taking neural nets and [5:37] we're breaking them down all the way to [5:39] these atoms of individual scalars and [5:41] all the little pluses and times and it's [5:43] just excessive and so obviously you [5:45] would never be doing any of this in [5:47] production it's really just put down for [5:48] pedagogical reasons because it allows us [5:50] to not have to deal with these [5:52] n-dimensional tensors that you would use [5:54] in modern deep neural network library so [5:56] this is really done so that you [5:58] understand and refactor out back [6:00] propagation and chain rule and [6:02] understanding of neurologic training [6:04] and then if you actually want to train [6:06] bigger networks you have to be using [6:08] these tensors but none of the math [6:09] changes this is done purely for [6:11] efficiency we are basically taking scale [6:13] value [6:14] all the scale values we're packaging [6:16] them up into tensors which are just [6:17] arrays of these scalars and then because [6:20] we have these large arrays we're making [6:22] operations on those large arrays that [6:24] allows us to take advantage of the [6:26] parallelism in a computer and all those [6:28] operations can be done in parallel and [6:30] then the whole thing runs faster but [6:32] really none of the math changes and [6:33] that's done purely for efficiency so i [6:35] don't think that it's pedagogically [6:36] useful to be dealing with tensors from [6:38] scratch uh and i think and that's why i [6:40] fundamentally wrote micrograd because [6:42] you can understand how things work uh at [6:44] the fundamental level and then you can [6:46] speed it up later okay so here's the fun [6:48] part my claim is that micrograd is what [6:51] you need to train your networks and [6:52] everything else is just efficiency so [6:54] you'd think that micrograd would be a [6:56] very complex piece of code and that [6:58] turns out to not be the case [7:01] so if we just go to micrograd [7:03] and you'll see that there's only two [7:05] files here in micrograd this is the [7:07] actual engine it doesn't know anything [7:09] about neural nuts and this is the entire [7:10] neural nets library [7:12] on top of micrograd so engine and nn.pi [7:17] so the actual backpropagation autograd [7:19] engine [7:21] that gives you the power of neural [7:22] networks is literally [7:26] 100 lines of code of like very simple [7:28] python [7:30] which we'll understand by the end of [7:31] this lecture [7:32] and then nn.pi [7:33] this neural network library built on top [7:35] of the autograd engine [7:37] um is like a joke it's like [7:40] we have to define what is a neuron and [7:42] then we have to define what is the layer [7:44] of neurons and then we define what is a [7:46] multi-layer perceptron which is just a [7:47] sequence of layers of neurons and so [7:50] it's just a total joke [7:52] so basically [7:53] there's a lot of power that comes from [7:55] only 150 lines of code [7:57] and that's all you need to understand to [7:59] understand neural network training and [8:00] everything else is just efficiency and [8:02] of course there's a lot to efficiency [8:05] but fundamentally that's all that's [8:07] happening okay so now let's dive right [8:09] in and implement micrograph step by step [8:11] the first thing i'd like to do is i'd [8:12] like to make sure that you have a very [8:13] good understanding intuitively of what a [8:16] derivative is and exactly what [8:18] information it gives you so let's start [8:20] with some basic imports that i copy [8:22] paste in every jupiter notebook always [8:25] and let's define a function a scalar [8:27] valued function [8:28] f of x [8:30] as follows [8:31] so i just make this up randomly i just [8:33] want to scale a valid function that [8:34] takes a single scalar x and returns a [8:36] single scalar y [8:38] and we can call this function of course [8:40] so we can pass in say 3.0 and get 20 [8:42] back [8:43] now we can also plot this function to [8:45] get a sense of its shape you can tell [8:47] from the mathematical expression that [8:48] this is probably a parabola it's a [8:50] quadratic [8:51] and so if we just uh create a set of um [8:56] um [8:57] scale values that we can feed in using [8:59] for example a range from negative five [9:01] to five in steps of 0.25 [9:03] so this is so axis is just from negative [9:06] 5 to 5 not including 5 in steps of 0.25 [9:11] and we can actually call this function [9:12] on this numpy array as well so we get a [9:14] set of y's if we call f on axis [9:17] and these y's are basically [9:20] also applying a function on every one of [9:23] these elements independently [9:25] and we can plot this using matplotlib so [9:28] plt.plot x's and y's and we get a nice [9:31] parabola so previously here we fed in [9:33] 3.0 somewhere here and we received 20 [9:36] back which is here the y coordinate so [9:39] now i'd like to think through [9:40] what is the derivative [9:42] of this function at any single input [9:44] point x [9:45] right so what is the derivative at [9:47] different points x of this function now [9:49] if you remember back to your calculus [9:51] class you've probably derived [9:52] derivatives so we take this mathematical [9:54] expression 3x squared minus 4x plus 5 [9:57] and you would write out on a piece of [9:58] paper and you would you know apply the [9:59] product rule and all the other rules and [10:01] derive the mathematical expression of [10:03] the great derivative of the original [10:05] function and then you could plug in [10:06] different texts and see what the [10:08] derivative is [10:09] we're not going to actually do that [10:11] because no one in neural networks [10:13] actually writes out the expression for [10:15] the neural net it would be a massive [10:16] expression um it would be you know [10:18] thousands tens of thousands of terms no [10:20] one actually derives the derivative of [10:22] course and so we're not going to take [10:24] this kind of like a symbolic approach [10:26] instead what i'd like to do is i'd like [10:27] to look at the definition of derivative [10:29] and just make sure that we really [10:30] understand what derivative is measuring [10:32] what it's telling you about the function [10:34] and so if we just look up derivative [10:42] we see that [10:43] okay so this is not a very good [10:44] definition of derivative this is a [10:46] definition of what it means to be [10:47] differentiable [10:48] but if you remember from your calculus [10:50] it is the limit as h goes to zero of f [10:52] of x plus h minus f of x over h so [10:55] basically what it's saying is if you [10:58] slightly bump up you're at some point x [11:00] that you're interested in or a and if [11:02] you slightly bump up [11:04] you know you slightly increase it by [11:06] small number h [11:08] how does the function respond with what [11:09] sensitivity does it respond what is the [11:11] slope at that point does the function go [11:13] up or does it go down and by how much [11:16] and that's the slope of that function [11:18] the [11:18] the slope of that response at that point [11:21] and so we can basically evaluate [11:23] the derivative here numerically by [11:26] taking a very small h of course the [11:28] definition would ask us to take h to [11:30] zero we're just going to pick a very [11:31] small h 0.001 [11:34] and let's say we're interested in point [11:35] 3.0 so we can look at f of x of course [11:37] as 20 [11:38] and now f of x plus h [11:40] so if we slightly nudge x in a positive [11:42] direction how is the function going to [11:44] respond [11:45] and just looking at this do you expect [11:47] do you expect f of x plus h to be [11:49] slightly greater than 20 or do you [11:51] expect to be slightly lower than 20 [11:54] and since this 3 is here and this is 20 [11:57] if we slightly go positively the [11:59] function will respond positively so [12:01] you'd expect this to be slightly greater [12:03] than 20. and now by how much it's [12:05] telling you the [12:06] sort of the [12:07] the strength of that slope right the the [12:09] size of the slope so f of x plus h minus [12:12] f of x this is how much the function [12:14] responded [12:16] in the positive direction and we have to [12:17] normalize by the [12:19] run so we have the rise over run to get [12:22] the slope so this of course is just a [12:24] numerical approximation of the slope [12:26] because we have to make age very very [12:28] small to converge to the exact amount [12:32] now if i'm doing too many zeros [12:35] at some point [12:36] i'm gonna get an incorrect answer [12:38] because we're using floating point [12:39] arithmetic and the representations of [12:41] all these numbers in computer memory is [12:43] finite and at some point we get into [12:45] trouble [12:46] so we can converse towards the right [12:47] answer with this approach [12:50] but basically um at 3 the slope is 14. [12:54] and you can see that by taking 3x [12:56] squared minus 4x plus 5 and [12:58] differentiating it in our head [13:00] so 3x squared would be [13:02] 6 x minus 4 [13:04] and then we plug in x equals 3 so that's [13:07] 18 minus 4 is 14. so this is correct [13:10] so that's [13:12] at 3. now how about the slope at say [13:15] negative 3 [13:17] would you expect would you expect for [13:19] the slope [13:20] now telling the exact value is really [13:22] hard but what is the sign of that slope [13:24] so at negative three [13:26] if we slightly go in the positive [13:28] direction at x the function would [13:30] actually go down and so that tells you [13:32] that the slope would be negative so [13:33] we'll get a slight number below [13:36] below 20. and so if we take the slope we [13:39] expect something negative [13:40] negative 22. okay [13:43] and at some point here of course the [13:45] slope would be zero now for this [13:47] specific function i looked it up [13:48] previously and it's at point two over [13:51] three [13:52] so at roughly two over three [13:54] uh that's somewhere here [13:55] um [13:57] this derivative be zero [13:59] so basically at that precise point [14:03] yeah [14:04] at that precise point if we nudge in a [14:06] positive direction the function doesn't [14:07] respond this stays the same almost and [14:09] so that's why the slope is zero okay now [14:11] let's look at a bit more complex case [14:14] so we're going to start you know [14:15] complexifying a bit so now we have a [14:18] function [14:19] here [14:20] with output variable d [14:22] that is a function of three scalar [14:24] inputs a b and c [14:26] so a b and c are some specific values [14:28] three inputs into our expression graph [14:30] and a single output d [14:32] and so if we just print d we get four [14:36] and now what i have to do is i'd like to [14:38] again look at the derivatives of d with [14:40] respect to a b and c [14:42] and uh think through uh again just the [14:44] intuition of what this derivative is [14:46] telling us [14:47] so in order to evaluate this derivative [14:49] we're going to get a bit hacky here [14:52] we're going to again have a very small [14:53] value of h [14:55] and then we're going to fix the inputs [14:57] at some [14:58] values that we're interested in [15:00] so these are the this is the point abc [15:02] at which we're going to be evaluating [15:04] the the [15:05] derivative of d with respect to all a b [15:07] and c at that point [15:09] so there are the inputs and now we have [15:11] d1 is that expression [15:13] and then we're going to for example look [15:15] at the derivative of d with respect to a [15:17] so we'll take a and we'll bump it by h [15:19] and then we'll get d2 to be the exact [15:22] same function [15:23] and now we're going to print um [15:26] you know f1 [15:28] d1 is d1 [15:31] d2 is d2 [15:32] and print slope [15:35] so the derivative or slope [15:37] here will be um [15:39] of course [15:41] d2 [15:42] minus d1 divide h [15:44] so d2 minus d1 is how much the function [15:47] increased [15:48] uh when we bumped [15:50] the uh [15:51] the specific input that we're interested [15:53] in by a tiny amount [15:55] and [15:56] this is then normalized by h [15:59] to get the slope [16:02] so [16:03] um [16:05] yeah [16:06] so this so if i just run this we're [16:08] going to print [16:10] d1 [16:12] which we know is four [16:15] now d2 will be bumped a will be bumped [16:18] by h [16:20] so let's just think through [16:22] a little bit uh what d2 will be uh [16:26] printed out here [16:27] in particular [16:29] d1 will be four [16:31] will d2 be a number slightly greater [16:33] than four or slightly lower than four [16:35] and that's going to tell us the sl the [16:37] the sign of the derivative [16:40] so [16:42] we're bumping a by h [16:45] b as minus three c is ten [16:48] so you can just intuitively think [16:49] through this derivative and what it's [16:51] doing a will be slightly more positive [16:54] and but b is a negative number [16:57] so if a is slightly more positive [17:00] because b is negative three [17:03] we're actually going to be adding less [17:06] to d [17:08] so you'd actually expect that the value [17:10] of the function will go down [17:13] so let's just see this [17:16] yeah and so we went from 4 [17:18] to 3.9996 [17:20] and that tells you that the slope will [17:22] be negative [17:23] and then [17:24] uh will be a negative number [17:26] because we went down [17:27] and then [17:29] the exact number of slope will be [17:31] exact amount of slope is negative 3. [17:33] and you can also convince yourself that [17:35] negative 3 is the right answer [17:36] mathematically and analytically because [17:39] if you have a times b plus c and you are [17:41] you know you have calculus then [17:43] differentiating a times b plus c with [17:46] respect to a gives you just b [17:48] and indeed the value of b is negative 3 [17:50] which is the derivative that we have so [17:52] you can tell that that's correct [17:54] so now if we do this with b [17:57] so if we bump b by a little bit in a [17:59] positive direction we'd get different [18:02] slopes so what is the influence of b on [18:04] the output d [18:06] so if we bump b by a tiny amount in a [18:08] positive direction then because a is [18:10] positive [18:11] we'll be adding more to d [18:13] right [18:14] so um and now what is the what is the [18:17] sensitivity what is the slope of that [18:18] addition [18:19] and it might not surprise you that this [18:21] should be [18:22] 2 [18:24] and y is a 2 because d of d [18:27] by db differentiating with respect to b [18:30] would be would give us a [18:31] and the value of a is two so that's also [18:34] working well [18:35] and then if c gets bumped a tiny amount [18:37] in h [18:38] by h [18:39] then of course a times b is unaffected [18:41] and now c becomes slightly bit higher [18:44] what does that do to the function it [18:45] makes it slightly bit higher because [18:47] we're simply adding c [18:48] and it makes it slightly bit higher by [18:50] the exact same amount that we added to c [18:53] and so that tells you that the slope is [18:55] one [18:56] that will be the [18:59] the rate at which [19:01] d will increase as we scale [19:04] c [19:05] okay so we now have some intuitive sense [19:06] of what this derivative is telling you [19:08] about the function and we'd like to move [19:10] to neural networks now as i mentioned [19:11] neural networks will be pretty massive [19:13] expressions mathematical expressions so [19:15] we need some data structures that [19:16] maintain these expressions and that's [19:17] what we're going to start to build out [19:19] now [19:20] so we're going to [19:22] build out this value object that i [19:24] showed you in the readme page of [19:26] micrograd [19:27] so let me copy paste a skeleton of the [19:30] first very simple value object [19:33] so class value takes a single [19:36] scalar value that it wraps and keeps [19:38] track of [19:39] and that's it so [19:41] we can for example do value of 2.0 and [19:43] then we can [19:45] get we can look at its content and [19:48] python will internally [19:50] use the wrapper function [19:52] to uh return [19:54] uh this string oops [19:56] like that [19:58] so this is a value object with data [20:00] equals two that we're creating here [20:03] now we'd like to do is like we'd like to [20:04] be able to [20:07] have not just like two values [20:10] but we'd like to do a bluffy right we'd [20:12] like to add them [20:13] so currently you would get an error [20:15] because python doesn't know how to add [20:17] two value objects so we have to tell it [20:21] so here's [20:22] addition [20:26] so you have to basically use these [20:27] special double underscore methods in [20:29] python to define these operators for [20:31] these objects so if we call um [20:35] the uh if we use this plus operator [20:39] python will internally call a dot add of [20:43] b [20:43] that's what will happen internally and [20:45] so b will be the other and [20:48] self will be a [20:50] and so we see that what we're going to [20:52] return is a new value object and it's [20:54] just it's going to be wrapping [20:56] the plus of [20:58] their data [20:59] but remember now because data is the [21:02] actual like numbered python number so [21:04] this operator here is just the typical [21:06] floating point plus addition now it's [21:09] not an addition of value objects [21:11] and will return a new value so now a [21:14] plus b should work and it should print [21:16] value of [21:17] negative one [21:18] because that's two plus minus three [21:20] there we go [21:21] okay let's now implement multiply [21:24] just so we can recreate this expression [21:25] here [21:26] so multiply i think it won't surprise [21:28] you will be fairly similar [21:31] so instead of add we're going to be [21:33] using mul [21:34] and then here of course we want to do [21:36] times [21:36] and so now we can create a c value [21:38] object which will be 10.0 and now we [21:41] should be able to do a times b well [21:44] let's just do a times b first [21:46] um [21:47] [Music] [21:48] that's value of negative six now [21:50] and by the way i skipped over this a [21:52] little bit suppose that i didn't have [21:53] the wrapper function here [21:55] then it's just that you'll get some kind [21:57] of an ugly expression so what wrapper is [21:59] doing is it's providing us a way to [22:02] print out like a nicer looking [22:03] expression in python [22:05] uh so we don't just have something [22:07] cryptic we actually are you know it's [22:09] value of [22:10] negative six so this gives us a times [22:14] and then this we should now be able to [22:16] add c to it because we've defined and [22:18] told the python how to do mul and add [22:20] and so this will call this will [22:22] basically be equivalent to a dot [22:24] small [22:26] of b [22:27] and then this new value object will be [22:29] dot add [22:31] of c [22:32] and so let's see if that worked [22:34] yep so that worked well that gave us [22:36] four which is what we expect from before [22:39] and i believe we can just call them [22:40] manually as well there we go so [22:44] yeah [22:45] okay so now what we are missing is the [22:46] connective tissue of this expression as [22:49] i mentioned we want to keep these [22:50] expression graphs so we need to know and [22:52] keep pointers about what values produce [22:54] what other values [22:56] so here for example we are going to [22:58] introduce a new variable which we'll [23:00] call children and by default it will be [23:02] an empty tuple [23:03] and then we're actually going to keep a [23:04] slightly different variable in the class [23:06] which we'll call underscore prev which [23:08] will be the set of children [23:11] this is how i done i did it in the [23:13] original micrograd looking at my code [23:15] here i can't remember exactly the reason [23:17] i believe it was efficiency but this [23:19] underscore children will be a tuple for [23:20] convenience but then when we actually [23:22] maintain it in the class it will be just [23:23] this set yeah i believe for efficiency [23:27] um [23:28] so now [23:29] when we are creating a value like this [23:31] with a constructor children will be [23:33] empty and prep will be the empty set but [23:36] when we're creating a value through [23:37] addition or multiplication we're going [23:39] to feed in the children of this value [23:42] which in this case is self and other [23:46] so those are the children [23:48] here [23:50] so now we can do d dot prev [23:52] and we'll see that the children of the [23:55] we now know are this value of negative 6 [23:58] and value of 10 and this of course is [24:00] the value resulting from a times b and [24:03] the c value which is 10. [24:06] now the last piece of information we [24:08] don't know so we know that the children [24:10] of every single value but we don't know [24:12] what operation created this value [24:14] so we need one more element here let's [24:16] call it underscore pop [24:19] and by default this is the empty set for [24:21] leaves [24:22] and then we'll just maintain it here [24:25] and now the operation will be just a [24:27] simple string and in the case of [24:29] addition it's plus in the case of [24:31] multiplication is times [24:33] so now we [24:35] not just have d dot pref we also have a [24:37] d dot up [24:38] and we know that d was produced by an [24:40] addition of those two values and so now [24:42] we have the full [24:44] mathematical expression uh and we're [24:46] building out this data structure and we [24:47] know exactly how each value came to be [24:49] by word expression and from what other [24:51] values [24:54] now because these expressions are about [24:56] to get quite a bit larger we'd like a [24:58] way to nicely visualize these [25:00] expressions that we're building out so [25:02] for that i'm going to copy paste a bunch [25:03] of slightly scary code that's going to [25:06] visualize this these expression graphs [25:08] for us [25:09] so here's the code and i'll explain it [25:11] in a bit but first let me just show you [25:13] what this code does [25:14] basically what it does is it creates a [25:16] new function drawdot that we can call on [25:19] some root node [25:20] and then it's going to visualize it so [25:22] if we call drawdot on d [25:24] which is this final value here that is a [25:27] times b plus c [25:29] it creates something like this so this [25:31] is d [25:32] and you see that this is a times b [25:34] creating an integrated value plus c [25:36] gives us this output node d [25:40] so that's dried out of d [25:42] and i'm not going to go through this in [25:44] complete detail you can take a look at [25:46] graphless and its api uh graphis is a [25:48] open source graph visualization software [25:51] and what we're doing here is we're [25:52] building out this graph and graphis [25:54] api and [25:56] you can basically see that trace is this [25:58] helper function that enumerates all of [26:00] the nodes and edges in the graph [26:02] so that just builds a set of all the [26:04] nodes and edges and then we iterate for [26:06] all the nodes and we create special node [26:08] objects [26:08] for them in [26:11] using dot node [26:13] and then we also create edges using dot [26:15] dot edge [26:16] and the only thing that's like slightly [26:18] tricky here is you'll notice that i [26:20] basically add these fake nodes which are [26:22] these operation nodes so for example [26:24] this node here is just like a plus node [26:27] and [26:28] i create these [26:31] special op nodes here [26:34] and i connect them accordingly so these [26:37] nodes of course are not actual [26:39] nodes in the original graph [26:41] they're not actually a value object the [26:43] only value objects here are the things [26:46] in squares those are actual value [26:48] objects or representations thereof and [26:50] these op nodes are just created in this [26:52] drawdot routine so that it looks nice [26:55] let's also add labels to these graphs [26:57] just so we know what variables are where [26:59] so let's create a special underscore [27:01] label [27:02] um [27:03] or let's just do label [27:05] equals empty by default and save it in [27:08] each node [27:11] and then here we're going to do label as [27:13] a [27:15] label is the [27:17] label a c [27:22] and then [27:24] let's create a special um [27:27] e equals a times b [27:30] and e dot label will be e [27:34] it's kind of naughty [27:35] and e will be e plus c [27:38] and a d dot label will be [27:40] d [27:42] okay so nothing really changes i just [27:44] added this new e function [27:46] a new e variable [27:48] and then here when we are [27:50] printing this [27:51] i'm going to print the label here so [27:54] this will be a percent s [27:56] bar [27:56] and this will be end.label [28:01] and so now [28:03] we have the label on the left here so it [28:05] says a b creating e and then e plus c [28:07] creates d [28:08] just like we have it here [28:10] and finally let's make this expression [28:12] just one layer deeper [28:14] so d will not be the final output node [28:17] instead after d we are going to create a [28:20] new value object [28:21] called f we're going to start running [28:23] out of variables soon f will be negative [28:25] 2.0 [28:27] and its label will of course just be f [28:30] and then l capital l will be the output [28:34] of our graph [28:35] and l will be p times f [28:38] okay [28:38] so l will be negative eight is the [28:40] output [28:42] so [28:44] now we don't just draw a d we draw l [28:50] okay [28:52] and somehow the label of [28:54] l was undefined oops all that label has [28:56] to be explicitly sort of given to it [28:59] there we go so l is the output [29:01] so let's quickly recap what we've done [29:03] so far [29:04] we are able to build out mathematical [29:05] expressions using only plus and times so [29:08] far [29:09] they are scalar valued along the way [29:11] and we can do this forward pass [29:14] and build out a mathematical expression [29:16] so we have multiple inputs here a b c [29:18] and f [29:19] going into a mathematical expression [29:21] that produces a single output l [29:24] and this here is visualizing the forward [29:26] pass so the output of the forward pass [29:28] is negative eight that's the value [29:31] now what we'd like to do next is we'd [29:33] like to run back propagation [29:35] and in back propagation we are going to [29:37] start here at the end and we're going to [29:39] reverse [29:40] and calculate the gradient along along [29:43] all these intermediate values [29:45] and really what we're computing for [29:46] every single value here [29:48] um we're going to compute the derivative [29:50] of that node with respect to l [29:55] so [29:56] the derivative of l with respect to l is [29:58] just uh one [30:00] and then we're going to derive what is [30:01] the derivative of l with respect to f [30:03] with respect to d with respect to c with [30:06] respect to e [30:07] with respect to b and with respect to a [30:10] and in the neural network setting you'd [30:12] be very interested in the derivative of [30:13] basically this loss function l [30:16] with respect to the weights of a neural [30:18] network [30:19] and here of course we have just these [30:20] variables a b c and f [30:22] but some of these will eventually [30:23] represent the weights of a neural net [30:25] and so we'll need to know how those [30:27] weights are impacting [30:29] the loss function so we'll be interested [30:31] basically in the derivative of the [30:32] output with respect to some of its leaf [30:34] nodes and those leaf nodes will be the [30:36] weights of the neural net [30:38] and the other leaf nodes of course will [30:39] be the data itself but usually we will [30:41] not want or use the derivative of the [30:44] loss function with respect to data [30:45] because the data is fixed but the [30:47] weights will be iterated on [30:50] using the gradient information so next [30:52] we are going to create a variable inside [30:54] the value class that maintains the [30:57] derivative of l with respect to that [30:59] value [31:00] and we will call this variable grad [31:03] so there's a data and there's a [31:05] self.grad [31:07] and initially it will be zero and [31:09] remember that zero is basically means no [31:12] effect so at initialization we're [31:14] assuming that every value does not [31:16] impact does not affect the out the [31:18] output [31:19] right because if the gradient is zero [31:21] that means that changing this variable [31:23] is not changing the loss function [31:25] so by default we assume that the [31:27] gradient is zero [31:28] and then [31:31] now that we have grad and it's 0.0 [31:36] we are going to be able to visualize it [31:38] here after data so here grad is 0.4 f [31:42] and this will be in that graph [31:45] and now we are going to be showing both [31:47] the data and the grad [31:50] initialized at zero [31:53] and we are just about getting ready to [31:55] calculate the back propagation [31:57] and of course this grad again as i [31:58] mentioned is representing [32:00] the derivative of the output in this [32:02] case l with respect to this value so [32:05] with respect to so this is the [32:06] derivative of l with respect to f with [32:08] respect to d and so on so let's now fill [32:11] in those gradients and actually do back [32:12] propagation manually so let's start [32:14] filling in these gradients and start all [32:16] the way at the end as i mentioned here [32:18] first we are interested to fill in this [32:20] gradient here so what is the derivative [32:22] of l with respect to l [32:25] in other words if i change l by a tiny [32:27] amount of h [32:29] how much does [32:30] l change [32:32] it changes by h so it's proportional and [32:35] therefore derivative will be one [32:37] we can of course measure these or [32:39] estimate these numerical gradients [32:40] numerically just like we've seen before [32:43] so if i take this expression [32:45] and i create a def lol function here [32:49] and put this here now the reason i'm [32:51] creating a gating function hello here is [32:53] because i don't want to pollute or mess [32:55] up the global scope here this is just [32:57] kind of like a little staging area and [32:58] as you know in python all of these will [33:00] be local variables to this function so [33:02] i'm not changing any of the global scope [33:04] here [33:05] so here l1 will be l [33:10] and then copy pasting this expression [33:13] we're going to add a small amount h [33:17] in for example a [33:20] right and this would be measuring the [33:22] derivative of l with respect to a [33:25] so here this will be l2 [33:28] and then we want to print this [33:29] derivative so print [33:31] l2 minus l1 which is how much l changed [33:35] and then normalize it by h so this is [33:37] the rise over run [33:39] and we have to be careful because l is a [33:41] value node so we actually want its data [33:45] um [33:46] so that these are floats dividing by h [33:48] and this should print the derivative of [33:50] l with respect to a because a is the one [33:53] that we bumped a little bit by h [33:55] so what is the [33:57] derivative of l with respect to a [33:59] it's six [34:01] okay and obviously [34:03] if we change l by h [34:06] then that would be [34:09] here effectively [34:12] this looks really awkward but changing l [34:14] by h [34:16] you see the derivative here is 1. um [34:20] that's kind of like the base case of [34:23] what we are doing here [34:24] so basically we cannot come up here and [34:26] we can manually set l.grad to one this [34:29] is our manual back propagation [34:31] l dot grad is one and let's redraw [34:35] and we'll see that we filled in grad as [34:37] 1 for l [34:39] we're now going to continue the back [34:40] propagation so let's here look at the [34:42] derivatives of l with respect to d and f [34:45] let's do a d first [34:47] so what we are interested in if i create [34:49] a markdown on here is we'd like to know [34:51] basically we have that l is d times f [34:54] and we'd like to know what is uh d [34:57] l by d d [35:00] what is that [35:01] and if you know your calculus uh l is d [35:03] times f so what is d l by d d it would [35:06] be f [35:08] and if you don't believe me we can also [35:10] just derive it because the proof would [35:11] be fairly straightforward uh we go to [35:14] the [35:15] definition of the derivative which is f [35:18] of x plus h minus f of x divide h [35:22] as a limit limit of h goes to zero of [35:24] this kind of expression so when we have [35:26] l is d times f [35:28] then increasing d by h [35:31] would give us the output of b plus h [35:33] times f [35:35] that's basically f of x plus h right [35:38] minus d times f [35:42] and then divide h and symbolically [35:44] expanding out here we would have [35:46] basically d times f plus h times f minus [35:50] t times f divide h [35:52] and then you see how the df minus df [35:54] cancels so you're left with h times f [35:57] divide h [35:58] which is f [35:59] so in the limit as h goes to zero of [36:03] you know [36:04] derivative [36:06] definition we just get f in the case of [36:09] d times f [36:12] so [36:13] symmetrically [36:14] dl by d [36:15] f will just be d [36:18] so what we have is that f dot grad [36:21] we see now is just the value of d [36:24] which is 4. [36:28] and we see that [36:30] d dot grad [36:31] is just uh the value of f [36:36] and so the value of f is negative two [36:41] so we'll set those manually [36:45] let me erase this markdown node and then [36:47] let's redraw what we have [36:50] okay [36:51] and let's just make sure that these were [36:53] correct so we seem to think that dl by [36:56] dd is negative two so let's double check [36:59] um let me erase this plus h from before [37:02] and now we want the derivative with [37:03] respect to f [37:05] so let's just come here when i create f [37:06] and let's do a plus h here and this [37:08] should print the derivative of l with [37:10] respect to f so we expect to see four [37:14] yeah and this is four up to floating [37:16] point [37:17] funkiness [37:18] and then dl by dd [37:21] should be f which is negative two [37:25] grad is negative two [37:26] so if we again come here and we change d [37:31] d dot data plus equals h right here [37:35] so we expect so we've added a little h [37:37] and then we see how l changed and we [37:40] expect to print [37:42] uh negative two [37:44] there we go [37:47] so we've numerically verified what we're [37:49] doing here is what kind of like an [37:50] inline gradient check gradient check is [37:53] when we [37:54] are deriving this like back propagation [37:56] and getting the derivative with respect [37:57] to all the intermediate results and then [38:00] numerical gradient is just you know [38:03] estimating it using small step size [38:06] now we're getting to the crux of [38:08] backpropagation so this will be the most [38:10] important node to understand because if [38:12] you understand the gradient for this [38:14] node you understand all of back [38:16] propagation and all of training of [38:17] neural nets basically [38:19] so we need to derive dl by bc [38:23] in other words the derivative of l with [38:24] respect to c [38:26] because we've computed all these other [38:27] gradients already [38:29] now we're coming here and we're [38:30] continuing the back propagation manually [38:33] so we want dl by dc and then we'll also [38:36] derive dl by de [38:38] now here's the problem [38:40] how do we derive dl [38:41] by dc [38:44] we actually know the derivative l with [38:46] respect to d so we know how l assessed [38:48] it to d [38:50] but how is l sensitive to c so if we [38:53] wiggle c how does that impact l [38:55] through d [38:58] so we know dl by dc [39:01] and we also here know how c impacts d [39:04] and so just very intuitively if you know [39:06] the impact that c is having on d and the [39:09] impact that d is having on l [39:11] then you should be able to somehow put [39:12] that information together to figure out [39:14] how c impacts l [39:16] and indeed this is what we can actually [39:18] do so in particular we know just [39:20] concentrating on d first let's look at [39:22] how what is the derivative basically of [39:24] d with respect to c so in other words [39:27] what is dd by dc [39:31] so here we know that d is c times c plus [39:34] e [39:35] that's what we know and now we're [39:37] interested in dd by dc [39:39] if you just know your calculus again and [39:41] you remember that differentiating c plus [39:43] e with respect to c you know that that [39:45] gives you [39:46] 1.0 [39:47] and we can also go back to the basics [39:49] and derive this because again we can go [39:51] to our f of x plus h minus f of x [39:54] divide by h [39:56] that's the definition of a derivative as [39:58] h goes to zero [40:00] and so here [40:01] focusing on c and its effect on d [40:04] we can basically do the f of x plus h [40:06] will be [40:07] c is incremented by h plus e [40:10] that's the first evaluation of our [40:12] function minus [40:14] c plus e [40:16] and then divide h [40:18] and so what is this [40:19] uh just expanding this out this will be [40:21] c plus h plus e minus c minus e [40:25] divide h and then you see here how c [40:27] minus c cancels e minus e cancels we're [40:30] left with h over h which is 1.0 [40:33] and so [40:35] by symmetry also d d by d [40:38] e [40:39] will be 1.0 as well [40:42] so basically the derivative of a sum [40:44] expression is very simple and and this [40:46] is the local derivative so i call this [40:49] the local derivative because we have the [40:51] final output value all the way at the [40:52] end of this graph and we're now like a [40:54] small node here [40:55] and this is a little plus node [40:58] and it the little plus node doesn't know [41:00] anything about the rest of the graph [41:02] that it's embedded in all it knows is [41:04] that it did a plus it took a c and an e [41:07] added them and created d [41:09] and this plus note also knows the local [41:11] influence of c on d or rather rather the [41:14] derivative of d with respect to c and it [41:16] also [41:17] knows the derivative of d with respect [41:18] to e but that's not what we want that's [41:21] just a local derivative what we actually [41:23] want is d l by d c and l could l is here [41:27] just one step away but in a general case [41:30] this little plus note is could be [41:32] embedded in like a massive graph [41:34] so [41:35] again we know how l impacts d and now we [41:38] know how c and e impact d how do we put [41:41] that information together to write dl by [41:43] dc and the answer of course is the chain [41:46] rule in calculus [41:47] and so um [41:50] i pulled up a chain rule here from [41:51] kapedia [41:52] and [41:53] i'm going to go through this very [41:54] briefly so chain rule [41:57] wikipedia sometimes can be very [41:58] confusing and calculus can [42:00] can be very confusing like this is the [42:02] way i [42:03] learned [42:05] chain rule and it was very confusing [42:06] like what is happening it's just [42:08] complicated so i like this expression [42:10] much better [42:12] if a variable z depends on a variable y [42:15] which itself depends on the variable x [42:18] then z depends on x as well obviously [42:20] through the intermediate variable y [42:22] in this case the chain rule is expressed [42:24] as [42:25] if you want dz by dx [42:28] then you take the dz by dy and you [42:30] multiply it by d y by dx [42:33] so the chain rule fundamentally is [42:34] telling you [42:36] how [42:37] we chain these [42:39] uh derivatives together [42:41] correctly so to differentiate through a [42:44] function composition [42:46] we have to apply a multiplication [42:48] of [42:49] those derivatives [42:51] so that's really what chain rule is [42:53] telling us [42:54] and there's a nice little intuitive [42:56] explanation here which i also think is [42:58] kind of cute the chain rule says that [42:59] knowing the instantaneous rate of change [43:01] of z with respect to y and y relative to [43:03] x allows one to calculate the [43:04] instantaneous rate of change of z [43:06] relative to x [43:07] as a product of those two rates of [43:09] change [43:10] simply the product of those two [43:12] so here's a good one [43:14] if a car travels twice as fast as [43:16] bicycle and the bicycle is four times as [43:18] fast as walking man [43:19] then the car travels two times four [43:22] eight times as fast as demand [43:25] and so this makes it very clear that the [43:27] correct thing to do sort of [43:29] is to multiply [43:30] so [43:31] cars twice as fast as bicycle and [43:33] bicycle is four times as fast as man [43:36] so the car will be eight times as fast [43:38] as the man and so we can take these [43:42] intermediate rates of change if you will [43:44] and multiply them together [43:46] and that justifies the [43:48] chain rule intuitively so have a look at [43:50] chain rule about here really what it [43:52] means for us is there's a very simple [43:54] recipe for deriving what we want which [43:56] is dl by dc [43:59] and what we have so far [44:01] is we know [44:03] want [44:05] and we know [44:07] what is the [44:08] impact of d on l so we know d l by [44:12] d d the derivative of l with respect to [44:14] d d we know that that's negative two [44:17] and now because of this local [44:19] reasoning that we've done here we know [44:21] dd by d [44:23] c [44:24] so how does c impact d and in [44:27] particular this is a plus node so the [44:29] local derivative is simply 1.0 it's very [44:32] simple [44:33] and so [44:34] the chain rule tells us that dl by dc [44:37] going through this intermediate variable [44:40] will just be simply d l by [44:44] d [44:44] times [44:49] dd by dc [44:51] that's chain rule [44:53] so this is identical to what's happening [44:55] here [44:56] except [44:58] z is rl [44:59] y is our d and x is rc [45:03] so we literally just have to multiply [45:05] these [45:06] and because [45:10] these local derivatives like dd by dc [45:12] are just one [45:14] we basically just copy over dl by dd [45:17] because this is just times one [45:19] so what does it do so because dl by dd [45:22] is negative two what is dl by dc [45:25] well it's the local gradient 1.0 times [45:29] dl by dd which is negative two [45:31] so literally what a plus node does you [45:33] can look at it that way is it literally [45:35] just routes the gradient [45:37] because the plus nodes local derivatives [45:39] are just one and so in the chain rule [45:41] one times [45:43] dl by dd [45:45] is um [45:47] is uh is just dl by dd and so that [45:50] derivative just gets routed to both c [45:53] and to e in this case [45:55] so basically um we have that that grad [45:59] or let's start with c since that's the [46:01] one we looked at [46:02] is [46:03] negative two times one [46:06] negative two [46:08] and in the same way by symmetry e that [46:11] grad will be negative two that's the [46:13] claim so we can set those [46:16] we can redraw [46:19] and you see how we just assign negative [46:20] to negative two so this backpropagating [46:23] signal which is carrying the information [46:25] of like what is the derivative of l with [46:26] respect to all the intermediate nodes [46:28] we can imagine it almost like flowing [46:30] backwards through the graph and a plus [46:32] node will simply distribute the [46:34] derivative to all the leaf nodes sorry [46:36] to all the children nodes of it [46:39] so this is the claim and now let's [46:40] verify it so let me remove the plus h [46:43] here from before [46:45] and now instead what we're going to do [46:46] is we're going to increment c so c dot [46:48] data will be credited by h [46:50] and when i run this we expect to see [46:52] negative 2 [46:54] negative 2. and then of course for e [46:58] so e dot data plus equals h and we [47:01] expect to see negative 2. [47:03] simple [47:07] so those are the derivatives of these [47:09] internal nodes [47:11] and now we're going to recurse our way [47:13] backwards again [47:15] and we're again going to apply the chain [47:17] rule so here we go our second [47:19] application of chain rule and we will [47:20] apply it all the way through the graph [47:22] we just happen to only have one more [47:24] node remaining [47:25] we have that d l [47:27] by d e [47:28] as we have just calculated is negative [47:30] two so we know that [47:32] so we know the derivative of l with [47:33] respect to e [47:36] and now we want dl [47:39] by [47:40] da [47:41] right [47:42] and the chain rule is telling us that [47:44] that's just dl by de [47:48] negative 2 [47:50] times the local gradient so what is the [47:52] local gradient basically d e [47:55] by d a [47:56] we have to look at that [48:00] so i'm a little times node [48:02] inside a massive graph [48:04] and i only know that i did a times b and [48:06] i produced an e [48:09] so now what is d e by d a and d e by d b [48:12] that's the only thing that i sort of [48:14] know about that's my local gradient [48:17] so [48:17] because we have that e's a times b we're [48:20] asking what is d e by d a [48:24] and of course we just did that here we [48:26] had a [48:27] times so i'm not going to rederive it [48:30] but if you want to differentiate this [48:32] with respect to a you'll just get b [48:34] right the value of b [48:36] which in this case is negative 3.0 [48:41] so [48:41] basically we have that dl by da [48:45] well let me just do it right here we [48:47] have that a dot grad and we are applying [48:49] chain rule here [48:50] is d l by d e which we see here is [48:54] negative two [48:56] times [48:57] what is d e by d a [48:59] it's the value of b which is negative 3. [49:04] that's it [49:07] and then we have b grad is again dl by [49:10] de [49:11] which is negative 2 [49:13] just the same way [49:14] times [49:15] what is d e by d [49:18] um db [49:19] is the value of a which is 2.2.0 [49:23] as the value of a [49:25] so these are our claimed derivatives [49:28] let's [49:30] redraw [49:32] and we see here that [49:33] a dot grad turns out to be 6 because [49:36] that is negative 2 times negative 3 [49:38] and b dot grad is negative 4 [49:41] times sorry is negative 2 times 2 which [49:43] is negative 4. [49:45] so those are our claims let's delete [49:47] this and let's verify them [49:50] we have [49:52] a here a dot data plus equals h [49:57] so the claim is that [49:59] a dot grad is six [50:01] let's verify [50:03] six [50:04] and we have beta data [50:07] plus equals h [50:08] so nudging b by h [50:11] and looking at what happens [50:13] we claim it's negative four [50:15] and indeed it's negative four plus minus [50:17] again float oddness [50:20] um [50:21] and uh [50:23] that's it this [50:24] that was the manual [50:26] back propagation [50:28] uh all the way from here to all the leaf [50:30] nodes and we've done it piece by piece [50:33] and really all we've done is as you saw [50:35] we iterated through all the nodes one by [50:37] one and locally applied the chain rule [50:39] we always know what is the derivative of [50:41] l with respect to this little output and [50:44] then we look at how this output was [50:45] produced this output was produced [50:47] through some operation and we have the [50:49] pointers to the children nodes of this [50:51] operation [50:52] and so in this little operation we know [50:54] what the local derivatives are and we [50:56] just multiply them onto the derivative [50:58] always [50:59] so we just go through and recursively [51:01] multiply on the local derivatives and [51:04] that's what back propagation is is just [51:05] a recursive application of chain rule [51:08] backwards through the computation graph [51:10] let's see this power in action just very [51:12] briefly what we're going to do is we're [51:14] going to [51:15] nudge our inputs to try to make l go up [51:19] so in particular what we're doing is we [51:21] want a.data we're going to change it [51:24] and if we want l to go up that means we [51:26] just have to go in the direction of the [51:27] gradient so [51:29] a [51:30] should increase in the direction of [51:32] gradient by like some small step amount [51:34] this is the step size [51:36] and we don't just want this for ba but [51:38] also for b [51:41] also for c [51:44] also for f [51:46] those are [51:47] leaf nodes which we usually have control [51:49] over [51:50] and if we nudge in direction of the [51:52] gradient we expect a positive influence [51:54] on l [51:55] so we expect l to go up [51:58] positively [51:59] so it should become less negative it [52:01] should go up to say negative you know [52:03] six or something like that [52:05] uh it's hard to tell exactly and we'd [52:08] have to rewrite the forward pass so let [52:09] me just um [52:12] do that here [52:13] um [52:16] this would be the forward pass f would [52:18] be unchanged this is effectively the [52:20] forward pass and now if we print l.data [52:24] we expect because we nudged all the [52:27] values all the inputs in the rational [52:28] gradient we expected a less negative l [52:30] we expect it to go up [52:32] so maybe it's negative six or so let's [52:34] see what happens [52:36] okay negative seven [52:38] and uh this is basically one step of an [52:41] optimization that we'll end up running [52:43] and really does gradient just give us [52:46] some power because we know how to [52:47] influence the final outcome and this [52:49] will be extremely useful for training [52:50] knowledge as well as you'll see [52:52] so now i would like to do one more uh [52:55] example of manual backpropagation using [52:58] a bit more complex and uh useful example [53:02] we are going to back propagate through a [53:04] neuron [53:05] so [53:07] we want to eventually build up neural [53:08] networks and in the simplest case these [53:10] are multilateral perceptrons as they're [53:12] called so this is a two layer neural net [53:15] and it's got these hidden layers made up [53:17] of neurons and these neurons are fully [53:18] connected to each other [53:20] now biologically neurons are very [53:21] complicated devices but we have very [53:23] simple mathematical models of them [53:26] and so this is a very simple [53:27] mathematical model of a neuron you have [53:29] some inputs axis [53:31] and then you have these synapses that [53:33] have weights on them so [53:36] the w's are weights [53:39] and then [53:40] the synapse interacts with the input to [53:42] this neuron multiplicatively so what [53:44] flows to the cell body [53:47] of this neuron is w times x [53:49] but there's multiple inputs so there's [53:51] many w times x's flowing into the cell [53:53] body [53:54] the cell body then has also like some [53:56] bias [53:57] so this is kind of like the [53:59] inert innate sort of trigger happiness [54:02] of this neuron so this bias can make it [54:04] a bit more trigger happy or a bit less [54:06] trigger happy regardless of the input [54:08] but basically we're taking all the w [54:10] times x [54:11] of all the inputs adding the bias and [54:13] then we take it through an activation [54:15] function [54:16] and this activation function is usually [54:18] some kind of a squashing function [54:20] like a sigmoid or 10h or something like [54:22] that so as an example [54:24] we're going to use the 10h in this [54:26] example [54:28] numpy has a [54:29] np.10h [54:31] so [54:32] we can call it on a range [54:34] and we can plot it [54:36] this is the 10h function and you see [54:38] that the inputs as they come in [54:41] get squashed on the y coordinate here so [54:44] um [54:45] right at zero we're going to get exactly [54:47] zero and then as you go more positive in [54:49] the input [54:50] then you'll see that the function will [54:52] only go up to one and then plateau out [54:55] and so if you pass in very positive [54:57] inputs we're gonna cap it smoothly at [55:00] one and on the negative side we're gonna [55:02] cap it smoothly to negative one [55:04] so that's 10h [55:06] and that's the squashing function or an [55:08] activation function and what comes out [55:10] of this neuron is just the activation [55:12] function applied to the dot product of [55:14] the weights and the [55:16] inputs [55:18] so let's [55:19] write one out [55:21] um [55:22] i'm going to copy paste because [55:27] i don't want to type too much [55:28] but okay so here we have the inputs [55:31] x1 x2 so this is a two-dimensional [55:33] neuron so two inputs are going to come [55:34] in [55:35] these are thought out as the weights of [55:37] this neuron [55:38] weights w1 w2 and these weights again [55:41] are the synaptic strengths for each [55:43] input [55:45] and this is the bias of the neuron [55:47] b [55:49] and now we want to do is according to [55:51] this model we need to multiply x1 times [55:54] w1 [55:55] and x2 times w2 [55:57] and then we need to add bias on top of [56:00] it [56:01] and it gets a little messy here but all [56:03] we are trying to do is x1 w1 plus x2 w2 [56:06] plus b [56:07] and these are multiply here [56:09] except i'm doing it in small steps so [56:12] that we actually have pointers to all [56:13] these intermediate nodes so we have x1 [56:15] w1 variable x times x2 w2 variable and [56:19] i'm also labeling them [56:21] so n is now [56:23] the cell body raw [56:25] raw [56:26] activation without [56:28] the activation function for now [56:30] and this should be enough to basically [56:32] plot it so draw dot of n [56:37] gives us x1 times w1 x2 times w2 [56:41] being added [56:43] then the bias gets added on top of this [56:45] and this n [56:47] is this sum [56:49] so we're now going to take it through an [56:50] activation function [56:52] and let's say we use the 10h [56:54] so that we produce the output [56:56] so what we'd like to do here is we'd [56:58] like to do the output and i'll call it o [57:01] is um [57:03] n dot 10h [57:05] okay but we haven't yet written the 10h [57:08] now the reason that we need to implement [57:09] another 10h function here is that [57:12] tanh is a [57:14] hyperbolic function and we've only so [57:16] far implemented a plus and the times and [57:18] you can't make a 10h out of just pluses [57:20] and times [57:22] you also need exponentiation so 10h is [57:25] this kind of a formula here [57:27] you can use either one of these and you [57:28] see that there's exponentiation involved [57:30] which we have not implemented yet for [57:32] our low value node here so we're not [57:34] going to be able to produce 10h yet and [57:36] we have to go back up and implement [57:37] something like it [57:39] now one option here [57:42] is we could actually implement um [57:44] exponentiation [57:46] right and we could return the x of a [57:49] value instead of a 10h of a value [57:52] because if we had x then we have [57:54] everything else that we need so um [57:56] because we know how to add and we know [57:58] how to [58:00] um [58:01] we know how to add and we know how to [58:02] multiply so we'd be able to create 10h [58:04] if we knew how to x [58:06] but for the purposes of this example i [58:08] specifically wanted to [58:10] show you [58:11] that we don't necessarily need to have [58:13] the most atomic pieces [58:15] in [58:16] um [58:16] in this value object we can actually [58:19] like create functions at arbitrary [58:23] points of abstraction they can be [58:24] complicated functions but they can be [58:26] also very very simple functions like a [58:27] plus and it's totally up to us the only [58:30] thing that matters is that we know how [58:31] to differentiate through any one [58:33] function so we take some inputs and we [58:35] make an output the only thing that [58:37] matters it can be arbitrarily complex [58:38] function as long as you know how to [58:41] create the local derivative if you know [58:43] the local derivative of how the inputs [58:44] impact the output then that's all you [58:46] need so we're going to cluster up [58:49] all of this expression and we're not [58:51] going to break it down to its atomic [58:52] pieces we're just going to directly [58:54] implement tanh [58:55] so let's do that [58:57] depth nh [58:59] and then out will be a value [59:02] of [59:03] and we need this expression here so [59:05] um [59:08] let me actually [59:10] copy paste [59:14] let's grab n which is a cell.theta [59:17] and then this [59:18] i believe is the tan h [59:21] math.x of [59:24] two [59:25] no n [59:27] n minus one over [59:28] two n plus one [59:30] maybe i can call this x [59:33] just so that it matches exactly [59:35] okay and now [59:37] this will be t [59:40] and uh children of this node there's [59:42] just one child [59:44] and i'm wrapping it in a tuple so this [59:46] is a tuple of one object just self [59:48] and here the name of this operation will [59:50] be 10h [59:52] and we're going to return that [59:56] okay [59:58] so now valley should be implementing 10h [60:02] and now we can scroll all the way down [60:03] here [60:04] and we can actually do n.10 h and that's [60:06] going to return the tanhd [60:09] output of n [60:11] and now we should be able to draw it out [60:12] of o not of n [60:14] so let's see how that worked [60:18] there we go [60:19] n went through 10 h [60:21] to produce this output [60:24] so now tan h is a [60:26] sort of [60:27] our little micro grad supported node [60:30] here as an operation [60:33] and as long as we know the derivative of [60:35] 10h [60:36] then we'll be able to back propagate [60:37] through it now let's see this 10h in [60:39] action currently it's not squashing too [60:41] much because the input to it is pretty [60:43] low so if the bias was increased to say [60:46] eight [60:49] then we'll see that what's flowing into [60:51] the 10h now is [60:53] two [60:54] and 10h is squashing it to 0.96 so we're [60:57] already hitting the tail of this 10h and [60:59] it will sort of smoothly go up to 1 and [61:01] then plateau out over there [61:03] okay so now i'm going to do something [61:04] slightly strange i'm going to change [61:06] this bias from 8 to this number [61:09] 6.88 etc [61:11] and i'm going to do this for specific [61:13] reasons because we're about to start [61:15] back propagation [61:16] and i want to make sure that our numbers [61:19] come out nice they're not like very [61:21] crazy numbers they're nice numbers that [61:22] we can sort of understand in our head [61:24] let me also add a pose label [61:26] o is short for output here [61:30] so that's zero [61:31] okay so [61:32] 0.88 flows into 10 h comes out 0.7 so on [61:36] so now we're going to do back [61:37] propagation and we're going to fill in [61:38] all the gradients [61:40] so what is the derivative o with respect [61:43] to [61:44] all the [61:45] inputs here and of course in the typical [61:47] neural network setting what we really [61:48] care about the most is the derivative of [61:51] these neurons on the weights [61:53] specifically the w2 and w1 because those [61:56] are the weights that we're going to be [61:57] changing part of the optimization [61:59] and the other thing that we have to [62:00] remember is here we have only a single [62:02] neuron but in the neural natives [62:03] typically have many neurons and they're [62:04] connected [62:07] so this is only like a one small neuron [62:09] a piece of a much bigger puzzle and [62:10] eventually there's a loss function that [62:12] sort of measures the accuracy of the [62:13] neural net and we're back propagating [62:15] with respect to that accuracy and trying [62:16] to increase it [62:19] so let's start off by propagation here [62:21] in the end [62:22] what is the derivative of o with respect [62:24] to o the base case sort of we know [62:26] always is that the gradient is just 1.0 [62:30] so let me fill it in [62:32] and then let me [62:35] split out [62:37] the drawing function [62:40] here [62:43] and then here cell [62:47] clear this output here okay [62:50] so now when we draw o we'll see that oh [62:52] that grad is one [62:53] so now we're going to back propagate [62:55] through the tan h [62:56] so to back propagate through 10h we need [62:58] to know the local derivative of 10h [63:01] so if we have that [63:03] o is 10 h of [63:07] n [63:08] then what is d o by d n [63:12] now what you could do is you could come [63:13] here and you could take this expression [63:15] and you could [63:16] do your calculus derivative taking [63:19] um and that would work but we can also [63:21] just scroll down wikipedia here [63:23] into a section that hopefully tells us [63:26] that derivative uh [63:28] d by dx of 10 h of x is [63:31] any of these i like this one 1 minus 10 [63:33] h square of x [63:35] so this is 1 minus 10 h [63:37] of x squared [63:39] so basically what this is saying is that [63:41] d o by d n [63:43] is [63:44] 1 minus 10 h [63:47] of n [63:48] squared [63:51] and we already have 10 h of n that's [63:52] just o [63:54] so it's one minus o squared [63:56] so o is the output here so the output is [63:59] this number [64:02] data [64:04] is this number [64:06] and then [64:08] what this is saying is that do by dn is [64:10] 1 minus [64:11] this squared so [64:13] one minus of that data squared [64:16] is 0.5 conveniently [64:18] so the local derivative of this 10 h [64:21] operation here is 0.5 [64:24] and [64:25] so that would be d o by d n [64:27] so [64:28] we can fill in that in that grad [64:33] is 0.5 we'll just fill in [64:42] so this is exactly 0.5 one half [64:45] so now we're going to continue the back [64:47] propagation [64:49] this is 0.5 and this is a plus node [64:52] so how is backprop going to what is that [64:55] going to do here [64:56] and if you remember our previous example [64:58] a plus is just a distributor of gradient [65:01] so this gradient will simply flow to [65:03] both of these equally and that's because [65:05] the local derivative of this operation [65:07] is one for every one of its nodes so 1 [65:10] times 0.5 is 0.5 [65:12] so therefore we know that [65:14] this node here which we called this [65:18] its grad is just 0.5 [65:21] and we know that b dot grad is also 0.5 [65:24] so let's set those and let's draw [65:28] so 0.5 [65:30] continuing we have another plus [65:32] 0.5 again we'll just distribute it so [65:34] 0.5 will flow to both of these [65:37] so we can set [65:39] theirs [65:43] x2w2 as well that grad is 0.5 [65:47] and let's redraw pluses are my favorite [65:50] uh operations to back propagate through [65:51] because [65:53] it's very simple [65:55] so now it's flowing into these [65:56] expressions is 0.5 and so really again [65:58] keep in mind what the derivative is [65:59] telling us at every point in time along [66:01] here this is saying that [66:04] if we want the output of this neuron to [66:06] increase [66:08] then [66:08] the influence on these expressions is [66:10] positive on the output both of them are [66:13] positive [66:16] contribution to the output [66:20] so now back propagating to x2 and w2 [66:23] first [66:24] this is a times node so we know that the [66:26] local derivative is you know the other [66:28] term [66:28] so if we want to calculate x2.grad [66:32] then [66:33] can you think through what it's going to [66:34] be [66:40] so x2.grad will be [66:42] w2.data [66:44] times this x2w2 [66:48] by grad right [66:51] and [66:52] w2.grad will be [66:55] x2 that data times x2w2.grad [67:01] right so that's the local piece of chain [67:03] rule [67:07] let's set them and let's redraw [67:09] so here we see that the gradient on our [67:11] weight 2 is 0 because x2 data was 0 [67:15] right but x2 will have the gradient 0.5 [67:18] because data here was 1. [67:20] and so what's interesting here right is [67:22] because the input x2 was 0 then because [67:25] of the way the times works [67:28] of course this gradient will be zero and [67:30] think about intuitively why that is [67:33] derivative always tells us the influence [67:35] of [67:36] this on the final output if i wiggle w2 [67:39] how is the output changing [67:41] it's not changing because we're [67:42] multiplying by zero [67:44] so because it's not changing there's no [67:46] derivative and zero is the correct [67:47] answer [67:48] because we're [67:49] squashing it at zero [67:52] and let's do it here point five should [67:54] come here and flow through this times [67:57] and so we'll have that x1.grad is [68:01] can you think through a little bit what [68:03] what [68:04] this should be [68:07] the local derivative of times [68:09] with respect to x1 is going to be w1 [68:12] so w1 is data times [68:15] x1 w1 dot grad [68:18] and w1.grad will be x1.data times [68:23] x1 w2 w1 with graph [68:27] let's see what those came out to be [68:29] so this is 0.5 so this would be negative [68:31] 1.5 and this would be 1. [68:34] and we've back propagated through this [68:36] expression these are the actual final [68:38] derivatives so if we want this neuron's [68:40] output to increase [68:43] we know that what's necessary is that [68:47] w2 we have no gradient w2 doesn't [68:49] actually matter to this neuron right now [68:51] but this neuron this weight should uh go [68:54] up [68:55] so if this weight goes up then this [68:57] neuron's output would have gone up and [68:59] proportionally because the gradient is [69:01] one okay so doing the back propagation [69:03] manually is obviously ridiculous so we [69:05] are now going to put an end to this [69:06] suffering and we're going to see how we [69:08] can implement uh the backward pass a bit [69:11] more automatically we're not going to be [69:12] doing all of it manually out here [69:14] it's now pretty obvious to us by example [69:17] how these pluses and times are back [69:18] property ingredients so let's go up to [69:20] the value [69:22] object and we're going to start [69:24] codifying what we've seen [69:27] in the examples below [69:29] so we're going to do this by storing a [69:31] special cell dot backward [69:34] and underscore backward and this will be [69:37] a function which is going to do that [69:39] little piece of chain rule at each [69:41] little node that compute that took [69:43] inputs and produced output uh we're [69:45] going to store [69:46] how we are going to chain the the [69:49] outputs gradient into the inputs [69:51] gradients [69:52] so by default [69:54] this will be a function [69:55] that uh doesn't do anything [69:58] so um [69:59] and you can also see that here in the [70:01] value in micrograb [70:03] so [70:04] with this backward function by default [70:06] doesn't do anything [70:08] this is an empty function [70:10] and that would be sort of the case for [70:11] example for a leaf node for leaf node [70:13] there's nothing to do [70:15] but now if when we're creating these out [70:18] values these out values are an addition [70:21] of self and other [70:24] and so we will want to sell set [70:27] outs backward to be [70:29] the function that propagates the [70:31] gradient [70:34] so [70:35] let's define what should happen [70:40] and we're going to store it in a closure [70:42] let's define what should happen when we [70:44] call [70:45] outs grad [70:47] for in addition [70:50] our job is to take [70:52] outs grad and propagate it into self's [70:55] grad and other grad so basically we want [70:57] to sell self.grad to something [71:00] and we want to set others.grad to [71:02] something [71:04] okay [71:05] and the way we saw below how chain rule [71:08] works we want to take the local [71:10] derivative times [71:11] the [71:12] sort of global derivative i should call [71:14] it which is the derivative of the final [71:16] output of the expression with respect to [71:18] out's data [71:21] with respect to out [71:22] so [71:24] the local derivative of self in an [71:27] addition is 1.0 [71:29] so it's just 1.0 times [71:31] outs grad [71:34] that's the chain rule [71:35] and others.grad will be 1.0 times [71:38] outgrad [71:39] and what you basically what you're [71:40] seeing here is that outscrad [71:42] will simply be copied onto selfs grad [71:45] and others grad as we saw happens for an [71:48] addition operation [71:49] so we're going to later call this [71:51] function to propagate the gradient [71:53] having done an addition [71:55] let's now do multiplication we're going [71:57] to also define that backward [72:02] and we're going to set its backward to [72:04] be backward [72:07] and we want to chain outgrad into [72:11] self.grad [72:14] and others.grad [72:17] and this will be a little piece of chain [72:18] rule for multiplication [72:20] so we'll have [72:21] so what should this be [72:23] can you think through [72:28] so what is the local derivative [72:30] here the local derivative was [72:32] others.data [72:35] and then [72:36] oops others.data and the times of that [72:39] grad that's channel [72:42] and here we have self.data times of that [72:44] grad [72:45] that's what we've been doing [72:49] and finally here for 10 h [72:51] left backward [72:54] and then we want to set out backwards to [72:57] be just backward [73:00] and here we need to [73:02] back propagate we have out that grad and [73:04] we want to chain it into self.grad [73:09] and salt.grad will be [73:11] the local derivative of this operation [73:13] that we've done here which is 10h [73:16] and so we saw that the local the [73:17] gradient is 1 minus the tan h of x [73:20] squared which here is t [73:23] that's the local derivative because [73:25] that's t is the output of this 10 h so 1 [73:27] minus t squared is the local derivative [73:30] and then gradient um [73:32] has to be multiplied because of the [73:33] chain rule [73:34] so outgrad is chained through the local [73:36] gradient into salt.grad [73:39] and that should be basically it so we're [73:41] going to redefine our value node [73:44] we're going to swing all the way down [73:46] here [73:48] and we're going to [73:49] redefine [73:51] our expression [73:52] make sure that all the grads are zero [73:55] okay [73:56] but now we don't have to do this [73:57] manually anymore [73:59] we are going to basically be calling the [74:01] dot backward in the right order [74:04] so [74:05] first we want to call os [74:07] dot backwards [74:14] so o was the outcome of 10h [74:17] right so calling all that those who's [74:20] backward [74:22] will be [74:23] this function this is what it will do [74:26] now we have to be careful because [74:29] there's a times out.grad [74:31] and out.grad remember is initialized to [74:34] zero [74:38] so here we see grad zero so as a base [74:41] case we need to set both.grad to 1.0 [74:46] to initialize this with 1 [74:53] and then once this is 1 we can call oda [74:56] backward [74:57] and what that should do is it should [74:58] propagate this grad through 10h [75:02] so the local derivative times [75:04] the global derivative which is [75:05] initialized at one so [75:08] this should [75:11] um [75:15] a dope [75:17] so i thought about redoing it but i [75:19] figured i should just leave the error in [75:20] here because it's pretty funny why is [75:22] anti-object not callable [75:24] uh it's because [75:27] i screwed up we're trying to save these [75:29] functions so this is correct [75:31] this here [75:33] we don't want to call the function [75:34] because that returns none these [75:36] functions return none we just want to [75:38] store the function [75:39] so let me redefine the value object [75:42] and then we're going to come back in [75:43] redefine the expression draw a dot [75:46] everything is great o dot grad is one [75:50] o dot grad is one and now [75:53] now this should work of course [75:55] okay so all that backward should [75:58] this grant should now be 0.5 if we [76:00] redraw and if everything went correctly [76:03] 0.5 yay [76:05] okay so now we need to call ns.grad [76:10] and it's not awkward sorry [76:13] ends backward [76:14] so that seems to have worked [76:17] so instead backward routed the gradient [76:21] to both of these so this is looking [76:22] great [76:24] now we could of course called uh called [76:26] b grad [76:27] beat up backwards sorry [76:30] what's gonna happen [76:32] well b doesn't have it backward b is [76:34] backward [76:35] because b is a leaf node [76:37] b's backward is by initialization the [76:40] empty function [76:41] so nothing would happen but we can call [76:44] call it on it [76:45] but when we call [76:48] this one [76:50] it's backward [76:53] then we expect this 0.5 to get further [76:56] routed [76:57] right so there we go 0.5.5 [77:00] and then finally [77:02] we want to call [77:05] it here on x2 w2 [77:10] and on x1 w1 [77:16] do both of those [77:17] and there we go [77:19] so we get 0 0.5 negative 1.5 and 1 [77:23] exactly as we did before but now [77:26] we've done it through [77:28] calling that backward um [77:30] sort of manually [77:32] so we have the lamp one last piece to [77:34] get rid of which is us calling [77:36] underscore backward manually so let's [77:38] think through what we are actually doing [77:40] um [77:41] we've laid out a mathematical expression [77:43] and now we're trying to go backwards [77:44] through that expression [77:46] um so going backwards through the [77:48] expression just means that we never want [77:50] to call a dot backward for any node [77:54] before [77:55] we've done a sort of um everything after [77:58] it [77:59] so we have to do everything after it [78:01] before we're ever going to call that [78:02] backward on any one node we have to get [78:04] all of its full dependencies everything [78:06] that it depends on has to [78:08] propagate to it before we can continue [78:10] back propagation so this ordering of [78:14] graphs can be achieved using something [78:16] called topological sort [78:17] so topological sort [78:20] is basically a laying out of a graph [78:23] such that all the edges go only from [78:24] left to right basically [78:26] so here we have a graph it's a directory [78:29] a cyclic graph a dag [78:31] and this is two different topological [78:34] orders of it i believe where basically [78:36] you'll see that it's laying out of the [78:37] notes such that all the edges go only [78:39] one way from left to right [78:41] and implementing topological sort you [78:44] can look in wikipedia and so on i'm not [78:46] going to go through it in detail [78:48] but basically this is what builds a [78:51] topological graph [78:54] we maintain a set of visited nodes and [78:56] then we are [78:59] going through starting at some root node [79:02] which for us is o that's where we want [79:03] to start the topological sort [79:05] and starting at o we go through all of [79:08] its children and we need to lay them out [79:10] from left to right [79:12] and basically this starts at o [79:14] if it's not visited then it marks it as [79:17] visited and then it iterates through all [79:19] of its children [79:20] and calls build topological on them [79:24] and then uh after it's gone through all [79:26] the children it adds itself [79:28] so basically [79:29] this node that we're going to call it on [79:31] like say o is only going to add itself [79:34] to the topo list after all of the [79:37] children have been processed and that's [79:39] how this function is guaranteeing [79:41] that you're only going to be in the list [79:43] once all your children are in the list [79:45] and that's the invariant that is being [79:46] maintained so if we built upon o and [79:49] then inspect this list [79:52] we're going to see that it ordered our [79:54] value objects [79:56] and the last one [79:58] is the value of 0.707 which is the [80:00] output [80:01] so this is o and then this is n [80:04] and then all the other nodes get laid [80:07] out before it [80:09] so that builds the topological graph and [80:12] really what we're doing now is we're [80:13] just calling dot underscore backward on [80:16] all of the nodes in a topological order [80:19] so if we just reset the gradients [80:22] they're all zero [80:23] what did we do [80:24] we started by [80:27] setting o dot grad [80:29] to b1 [80:31] that's the base case [80:33] then we built the topological order [80:38] and then we went for node [80:41] in [80:42] reversed [80:44] of topo [80:46] now [80:47] in in the reverse order because this [80:49] list goes from [80:50] you know we need to go through it in [80:52] reversed order [80:53] so starting at o [80:56] note that backward [80:58] and this should be [81:01] it [81:03] there we go [81:05] those are the correct derivatives [81:07] finally we are going to hide this [81:08] functionality [81:10] so i'm going to [81:11] copy this and we're going to hide it [81:13] inside the valley class because we don't [81:15] want to have all that code lying around [81:18] so instead of an underscore backward [81:19] we're now going to define an actual [81:21] backward so that's backward without the [81:23] underscore [81:26] and that's going to do all the stuff [81:27] that we just arrived [81:29] so let me just clean this up a little [81:30] bit so [81:32] we're first going to [81:37] build a topological graph [81:38] starting at self [81:41] so build topo of self [81:44] will populate the topological order into [81:46] the topo list which is a local variable [81:49] then we set self.grad to be one [81:52] and then for each node in the reversed [81:55] list so starting at us and going to all [81:57] the children [82:00] underscore backward [82:02] and [82:03] that should be it so [82:06] save [82:08] come down here [82:09] redefine [82:09] [Music] [82:11] okay all the grands are zero [82:13] and now what we can do is oh that [82:15] backward without the underscore [82:17] and [82:21] there we go [82:22] and that's uh that's back propagation [82:26] place for one neuron [82:28] now we shouldn't be too happy with [82:29] ourselves actually because we have a bad [82:32] bug um and we have not surfaced the bug [82:35] because of some specific conditions that [82:36] we are we have to think about right now [82:39] so here's the simplest case that shows [82:42] the bug [82:43] say i create a single node a [82:48] and then i create a b that is a plus a [82:51] and then i called backward [82:54] so what's going to happen is a is 3 [82:57] and then a b is a plus a so there's two [83:00] arrows on top of each other here [83:03] then we can see that b is of course the [83:05] forward pass works [83:06] b is just [83:08] a plus a which is six [83:10] but the gradient here is not actually [83:11] correct [83:12] that we calculate it automatically [83:15] and that's because [83:17] um [83:19] of course uh [83:20] just doing calculus in your head the [83:22] derivative of b with respect to a [83:24] should be uh two [83:27] one plus one [83:28] it's not one [83:30] intuitively what's happening here right [83:32] so b is the result of a plus a and then [83:34] we call backward on it [83:36] so let's go up and see what that does [83:42] um [83:43] b is a result of addition [83:45] so out as [83:46] b and then when we called backward what [83:49] happened is [83:50] self.grad was set [83:53] to one [83:54] and then other that grad was set to one [83:57] but because we're doing a plus a [83:59] self and other are actually the exact [84:02] same object [84:03] so we are overriding the gradient we are [84:06] setting it to one and then we are [84:07] setting it again to one and that's why [84:10] it stays [84:11] at one [84:13] so that's a problem [84:14] there's another way to see this in a [84:16] little bit more complicated expression [84:21] so here we have [84:23] a and b [84:25] and then uh d will be the multiplication [84:28] of the two and e will be the addition of [84:30] the two [84:32] and [84:33] then we multiply e times d to get f and [84:35] then we called fda backward [84:37] and these gradients if you check will be [84:39] incorrect [84:40] so fundamentally what's happening here [84:42] again is [84:45] basically we're going to see an issue [84:46] anytime we use a variable more than once [84:49] until now in these expressions above [84:51] every variable is used exactly once so [84:53] we didn't see the issue [84:54] but here if a variable is used more than [84:56] once what's going to happen during [84:57] backward pass we're backpropagating from [85:00] f to e to d so far so good but now [85:03] equals it backward and it deposits its [85:05] gradients to a and b but then we come [85:08] back to d [85:09] and call backward and it overwrites [85:11] those gradients at a and b [85:14] so that's obviously a problem [85:17] and the solution here if you look at [85:19] the multivariate case of the chain rule [85:22] and its generalization there [85:23] the solution there is basically that we [85:26] have to accumulate these gradients these [85:28] gradients add [85:30] and so instead of setting those [85:32] gradients [85:34] we can simply do plus equals we need to [85:37] accumulate those gradients [85:39] plus equals plus equals [85:41] plus equals [85:44] plus equals [85:46] and this will be okay remember because [85:48] we are initializing them at zero so they [85:50] start at zero [85:51] and then any [85:53] contribution [85:54] that flows backwards [85:57] will simply add [85:58] so now if we redefine [86:01] this one [86:03] because the plus equals this now works [86:06] because a.grad started at zero and we [86:08] called beta backward we deposit one and [86:11] then we deposit one again and now this [86:13] is two which is correct [86:14] and here this will also work and we'll [86:16] get correct gradients [86:18] because when we call eta backward we [86:20] will deposit the gradients from this [86:21] branch and then we get to back into [86:23] detail backward it will deposit its own [86:26] gradients and then those gradients [86:28] simply add on top of each other and so [86:30] we just accumulate those gradients and [86:31] that fixes the issue okay now before we [86:34] move on let me actually do a bit of [86:35] cleanup here and delete some of these [86:38] some of this intermediate work so [86:41] we're not gonna need any of this now [86:42] that we've derived all of it [86:44] um [86:45] we are going to keep this because i want [86:48] to come back to it [86:49] delete the 10h [86:51] delete our morning example [86:53] delete the step [86:55] delete this keep the code that draws [86:59] and then delete this example [87:02] and leave behind only the definition of [87:03] value [87:05] and now let's come back to this [87:06] non-linearity here that we implemented [87:08] the tanh now i told you that we could [87:10] have broken down 10h into its explicit [87:13] atoms in terms of other expressions if [87:16] we had the x function so if you remember [87:18] tan h is defined like this and we chose [87:20] to develop tan h as a single function [87:22] and we can do that because we know its [87:24] derivative and we can back propagate [87:26] through it [87:26] but we can also break down tan h into [87:29] and express it as a function of x and i [87:31] would like to do that now because i want [87:33] to prove to you that you get all the [87:34] same results and all those ingredients [87:36] but also because it forces us to [87:38] implement a few more expressions it [87:40] forces us to do exponentiation addition [87:42] subtraction division and things like [87:44] that and i think it's a good exercise to [87:46] go through a few more of these [87:48] okay so let's scroll up [87:50] to the definition of value [87:52] and here one thing that we currently [87:53] can't do is we can do like a value of [87:56] say 2.0 [87:58] but we can't do you know here for [88:00] example we want to add constant one and [88:02] we can't do something like this [88:05] and we can't do it because it says [88:06] object has no attribute data that's [88:08] because a plus one comes right here to [88:11] add [88:12] and then other is the integer one and [88:14] then here python is trying to access [88:16] one.data and that's not a thing and [88:18] that's because basically one is not a [88:20] value object and we only have addition [88:22] for value objects so as a matter of [88:24] convenience so that we can create [88:26] expressions like this and make them make [88:28] sense [88:29] we can simply do something like this [88:32] basically [88:33] we let other alone if other is an [88:35] instance of value but if it's not an [88:37] instance of value we're going to assume [88:39] that it's a number like an integer float [88:40] and we're going to simply wrap it in in [88:43] value and then other will just become [88:45] value of other and then other will have [88:46] a data attribute and this should work so [88:49] if i just say this predefined value then [88:51] this should work [88:53] there we go okay now let's do the exact [88:55] same thing for multiply because we can't [88:57] do something like this [88:58] again [88:59] for the exact same reason so we just [89:01] have to go to mole and if other is [89:04] not a value then let's wrap it in value [89:07] let's redefine value and now this works [89:10] now here's a kind of unfortunate and not [89:12] obvious part a times two works we saw [89:15] that but two times a is that gonna work [89:19] you'd expect it to right but actually it [89:21] will not [89:22] and the reason it won't is because [89:24] python doesn't know [89:26] like when when you do a times two [89:28] basically um so a times two python will [89:31] go and it will basically do something [89:32] like a dot mul [89:34] of two that's basically what it will [89:36] call but to it 2 times a is the same as [89:39] 2 dot mol of a [89:41] and it doesn't 2 can't multiply [89:44] value and so it's really confused about [89:46] that [89:47] so instead what happens is in python the [89:49] way this works is you are free to define [89:51] something called the r mold [89:54] and our mole [89:55] is kind of like a fallback so if python [89:58] can't do 2 times a it will check if um [90:02] if by any chance a knows how to multiply [90:05] two and that will be called into our [90:07] mole [90:08] so because python can't do two times a [90:11] it will check is there an our mole in [90:12] value and because there is it will now [90:15] call that [90:16] and what we'll do here is we will swap [90:18] the order of the operands so basically [90:21] two times a will redirect to armel and [90:23] our mole will basically call a times two [90:26] and that's how that will work [90:28] so [90:29] redefining now with armor two times a [90:31] becomes four okay now looking at the [90:33] other elements that we still need we [90:34] need to know how to exponentiate and how [90:36] to divide so let's first the explanation [90:38] to the exponentiation part we're going [90:40] to introduce [90:41] a single [90:42] function x here [90:45] and x is going to mirror 10h in the [90:47] sense that it's a simple single function [90:49] that transforms a single scalar value [90:51] and outputs a single scalar value [90:53] so we pop out the python number we use [90:56] math.x to exponentiate it create a new [90:58] value object [90:59] everything that we've seen before the [91:00] tricky part of course is how do you [91:02] propagate through e to the x [91:04] and [91:05] so here you can potentially pause the [91:07] video and think about what should go [91:09] here [91:13] okay so basically we need to know what [91:15] is the local derivative of e to the x so [91:18] d by d x of e to the x is famously just [91:21] e to the x and we've already just [91:23] calculated e to the x and it's inside [91:25] out that data so we can do up that data [91:27] times [91:28] and [91:29] out that grad that's the chain rule [91:32] so we're just chaining on to the current [91:33] running grad [91:35] and this is what the expression looks [91:36] like it looks a little confusing but [91:38] this is what it is and that's the [91:40] exponentiation [91:41] so redefining we should now be able to [91:43] call a.x [91:45] and [91:46] hopefully the backward pass works as [91:47] well okay and the last thing we'd like [91:49] to do of course is we'd like to be able [91:50] to divide [91:52] now [91:53] i actually will implement something [91:54] slightly more powerful than division [91:56] because division is just a special case [91:57] of something a bit more powerful [91:59] so in particular just by rearranging [92:02] if we have some kind of a b equals [92:04] value of 4.0 here we'd like to basically [92:07] be able to do a divide b and we'd like [92:09] this to be able to give us 0.5 [92:11] now division actually can be reshuffled [92:14] as follows if we have a divide b that's [92:17] actually the same as a multiplying one [92:18] over b [92:19] and that's the same as a multiplying b [92:21] to the power of negative one [92:24] and so what i'd like to do instead is i [92:25] basically like to implement the [92:27] operation of x to the k for some [92:29] constant uh k so it's an integer or a [92:32] float um and we would like to be able to [92:35] differentiate this and then as a special [92:36] case uh negative one will be division [92:40] and so i'm doing that just because uh [92:42] it's more general and um yeah you might [92:45] as well do it that way so basically what [92:46] i'm saying is we can redefine [92:49] uh division [92:51] which we will put here somewhere [92:54] yeah we can put it here somewhere what [92:56] i'm saying is that we can redefine [92:58] division so self-divide other [93:00] can actually be rewritten as self times [93:03] other to the power of negative one [93:05] and now [93:07] a value raised to the power of negative [93:09] one we have now defined that [93:11] so [93:12] here's [93:13] so we need to implement the pow function [93:15] where am i going to put the power [93:17] function maybe here somewhere [93:20] this is the skeleton for it [93:22] so this function will be called when we [93:24] try to raise a value to some power and [93:26] other will be that power [93:28] now i'd like to make sure that other is [93:30] only an int or a float usually other is [93:33] some kind of a different value object [93:35] but here other will be forced to be an [93:37] end or a float otherwise the math [93:40] won't work for [93:42] for or try to achieve in the specific [93:43] case that would be a different [93:45] derivative expression if we wanted other [93:47] to be a value [93:49] so here we create the output value which [93:51] is just uh you know this data raised to [93:53] the power of other and other here could [93:55] be for example negative one that's what [93:56] we are hoping to achieve [93:59] and then uh this is the backwards stub [94:01] and this is the fun part which is what [94:03] is the uh chain rule expression here for [94:07] back for um [94:09] back propagating through the power [94:11] function where the power is to the power [94:13] of some kind of a constant [94:15] so this is the exercise and maybe pause [94:17] the video here and see if you can figure [94:18] it out yourself as to what we should put [94:20] here [94:26] okay so [94:29] you can actually go here and look at [94:30] derivative rules as an example and we [94:32] see lots of derivatives that you can [94:34] hopefully know from calculus in [94:36] particular what we're looking for is the [94:37] power rule [94:39] because that's telling us that if we're [94:40] trying to take d by dx of x to the n [94:42] which is what we're doing here [94:44] then that is just n times x to the n [94:46] minus 1 [94:48] right [94:49] okay [94:50] so [94:51] that's telling us about the local [94:53] derivative of this power operation [94:55] so all we want here [94:58] basically n is now other [95:00] and self.data is x [95:03] and so this now becomes [95:06] other which is n times [95:08] self.data [95:10] which is now a python in torah float [95:13] it's not a valley object we're accessing [95:14] the data attribute [95:16] raised [95:17] to the power of other minus one or n [95:19] minus one [95:21] i can put brackets around this but this [95:22] doesn't matter because [95:25] power takes precedence over multiply and [95:27] python so that would have been okay [95:29] and that's the local derivative only but [95:31] now we have to chain it and we change [95:33] just simply by multiplying by output [95:34] grad that's chain rule [95:36] and this should technically work [95:40] and we're going to find out soon but now [95:42] if we [95:43] do this this should now work [95:46] and we get 0.5 so the forward pass works [95:49] but does the backward pass work and i [95:51] realize that we actually also have to [95:52] know how to subtract so [95:54] right now a minus b will not work [95:57] to make it work we need one more [96:00] piece of code here [96:01] and [96:02] basically this is the [96:05] subtraction and the way we're going to [96:06] implement subtraction is we're going to [96:08] implement it by addition of a negation [96:10] and then to implement negation we're [96:12] gonna multiply by negative one so just [96:14] again using the stuff we've already [96:15] built and just um expressing it in terms [96:17] of what we have and a minus b is now [96:20] working okay so now let's scroll again [96:22] to this expression here for this neuron [96:25] and let's just [96:26] compute the backward pass here once [96:28] we've defined o [96:30] and let's draw it [96:32] so here's the gradients for all these [96:33] leaf nodes for this two-dimensional [96:35] neuron that has a 10h that we've seen [96:37] before so now what i'd like to do is i'd [96:39] like to break up this 10h [96:41] into this expression here [96:44] so let me copy paste this [96:46] here [96:47] and now instead of we'll preserve the [96:49] label [96:50] and we will change how we define o [96:53] so in particular we're going to [96:55] implement this formula here [96:56] so we need e to the 2x [96:58] minus 1 over e to the x plus 1. so e to [97:01] the 2x we need to take 2 times n and we [97:04] need to exponentiate it that's e to the [97:07] two x and then because we're using it [97:08] twice let's create an intermediate [97:10] variable e [97:12] and then define o as [97:14] e plus one over [97:16] e minus one over e plus one [97:19] e minus one over e plus one [97:22] and that should be it and then we should [97:24] be able to draw that of o [97:26] so now before i run this what do we [97:29] expect to see [97:30] number one we're expecting to see a much [97:32] longer [97:33] graph here because we've broken up 10h [97:35] into a bunch of other operations [97:37] but those operations are mathematically [97:39] equivalent and so what we're expecting [97:41] to see is number one the same [97:43] result here so the forward pass works [97:45] and number two because of that [97:47] mathematical equivalence we expect to [97:49] see the same backward pass and the same [97:51] gradients on these leaf nodes so these [97:53] gradients should be identical [97:55] so let's run this [97:58] so number one let's verify that instead [98:00] of a single 10h node we have now x and [98:03] we have plus we have times negative one [98:06] uh this is the division [98:08] and we end up with the same forward pass [98:10] here [98:11] and then the gradients we have to be [98:13] careful because they're in slightly [98:14] different order potentially the [98:16] gradients for w2x2 should be 0 and 0.5 [98:19] w2 and x2 are 0 and 0.5 [98:22] and w1 x1 are 1 and negative 1.5 [98:25] 1 and negative 1.5 [98:27] so that means that both our forward [98:28] passes and backward passes were correct [98:31] because this turned out to be equivalent [98:33] to [98:34] 10h before [98:35] and so the reason i wanted to go through [98:37] this exercise is number one we got to [98:39] practice a few more operations and uh [98:41] writing more backwards passes and number [98:43] two i wanted to illustrate the point [98:45] that [98:46] the um [98:47] the level at which you implement your [98:49] operations is totally up to you you can [98:51] implement backward passes for tiny [98:53] expressions like a single individual [98:54] plus or a single times [98:56] or you can implement them for say [98:58] 10h [99:00] which is a kind of a potentially you can [99:01] see it as a composite operation because [99:03] it's made up of all these more atomic [99:05] operations but really all of this is [99:07] kind of like a fake concept all that [99:08] matters is we have some kind of inputs [99:10] and some kind of an output and this [99:11] output is a function of the inputs in [99:13] some way and as long as you can do [99:14] forward pass and the backward pass of [99:16] that little operation it doesn't matter [99:19] what that operation is [99:21] and how composite it is [99:23] if you can write the local gradients you [99:24] can chain the gradient and you can [99:26] continue back propagation so the design [99:28] of what those functions are is [99:30] completely up to you [99:31] so now i would like to show you how you [99:33] can do the exact same thing by using a [99:35] modern deep neural network library like [99:37] for example pytorch which i've roughly [99:40] modeled micrograd [99:41] by [99:42] and so [99:43] pytorch is something you would use in [99:44] production and i'll show you how you can [99:46] do the exact same thing but in pytorch [99:48] api so i'm just going to copy paste it [99:50] in and walk you through it a little bit [99:52] this is what it looks like [99:54] so we're going to import pi torch and [99:56] then we need to define these [99:59] value objects like we have here [100:01] now micrograd is a scalar valued [100:04] engine so we only have scalar values [100:07] like 2.0 but in pi torch everything is [100:10] based around tensors and like i [100:11] mentioned tensors are just n-dimensional [100:13] arrays of scalars [100:15] so that's why things get a little bit [100:17] more complicated here i just need a [100:19] scalar value to tensor a tensor with [100:21] just a single element [100:23] but by default when you work with [100:25] pytorch you would use um [100:28] more complicated tensors like this so if [100:30] i import pytorch [100:33] then i can create tensors like this and [100:36] this tensor for example is a two by [100:38] three array [100:39] of scalar [100:41] scalars [100:42] in a single compact representation so we [100:45] can check its shape we see that it's a [100:46] two by three array [100:48] and so on [100:49] so this is usually what you would work [100:50] with um in the actual libraries so here [100:54] i'm creating [100:55] a tensor that has only a single element [100:58] 2.0 [101:00] and then i'm casting it to be double [101:03] because python is by default using [101:05] double precision for its floating point [101:07] numbers so i'd like everything to be [101:08] identical by default the data type of [101:12] these tensors will be float32 so it's [101:14] only using a single precision float so [101:16] i'm casting it to double [101:18] so that we have float64 just like in [101:21] python [101:22] so i'm casting to double and then we get [101:24] something similar to value of two the [101:28] next thing i have to do is because these [101:29] are leaf nodes by default pytorch [101:31] assumes that they do not require [101:32] gradients so i need to explicitly say [101:35] that all of these nodes require [101:36] gradients [101:37] okay so this is going to construct [101:39] scalar valued one element tensors [101:43] make sure that fighters knows that they [101:44] require gradients now by default these [101:47] are set to false by the way because of [101:48] efficiency reasons because usually you [101:50] would not want gradients for leaf nodes [101:53] like the inputs to the network and this [101:55] is just trying to be efficient in the [101:57] most common cases [101:59] so once we've defined all of our values [102:01] in python we can perform arithmetic just [102:03] like we can here in microgradlend so [102:06] this will just work and then there's a [102:07] torch.10h also [102:09] and when we get back is a tensor again [102:12] and we can [102:13] just like in micrograd it's got a data [102:15] attribute and it's got grant attributes [102:18] so these tensor objects just like in [102:19] micrograd have a dot data and a dot grad [102:22] and [102:23] the only difference here is that we need [102:25] to call it that item because otherwise [102:28] um pi torch [102:30] that item basically takes [102:32] a single tensor of one element and it [102:34] just returns that element stripping out [102:36] the tensor [102:37] so let me just run this and hopefully we [102:39] are going to get this is going to print [102:41] the forward pass [102:42] which is 0.707 [102:44] and this will be the gradients which [102:46] hopefully are [102:48] 0.5 0 negative 1.5 and 1. [102:51] so if we just run this [102:53] there we go [102:54] 0.7 so the forward pass agrees and then [102:57] point five zero negative one point five [102:59] and one [103:00] so pi torch agrees with us [103:02] and just to show you here basically o [103:05] here's a tensor with a single element [103:08] and it's a double [103:09] and we can call that item on it to just [103:12] get the single number out [103:14] so that's what item does and o is a [103:16] tensor object like i mentioned and it's [103:18] got a backward function just like we've [103:20] implemented [103:22] and then all of these also have a dot [103:23] graph so like x2 for example in the grad [103:26] and it's a tensor and we can pop out the [103:28] individual number with that actin [103:31] so basically [103:32] torches torch can do what we did in [103:35] micrograph is a special case when your [103:37] tensors are all single element tensors [103:40] but the big deal with pytorch is that [103:42] everything is significantly more [103:43] efficient because we are working with [103:45] these tensor objects and we can do lots [103:47] of operations in parallel on all of [103:49] these tensors [103:51] but otherwise what we've built very much [103:53] agrees with the api of pytorch [103:55] okay so now that we have some machinery [103:57] to build out pretty complicated [103:58] mathematical expressions we can also [104:00] start building out neural nets and as i [104:02] mentioned neural nets are just a [104:03] specific class of mathematical [104:05] expressions [104:07] so we're going to start building out a [104:08] neural net piece by piece and eventually [104:09] we'll build out a two-layer multi-layer [104:12] layer perceptron as it's called and i'll [104:14] show you exactly what that means [104:15] let's start with a single individual [104:17] neuron we've implemented one here but [104:19] here i'm going to implement one that [104:21] also subscribes to the pytorch api in [104:24] how it designs its neural network [104:26] modules [104:27] so just like we saw that we can like [104:28] match the api of pytorch [104:31] on the auto grad side we're going to try [104:33] to do that on the neural network modules [104:35] so here's class neuron [104:38] and just for the sake of efficiency i'm [104:40] going to copy paste some sections that [104:42] are relatively straightforward [104:45] so the constructor will take [104:47] number of inputs to this neuron which is [104:49] how many inputs come to a neuron so this [104:52] one for example has three inputs [104:55] and then it's going to create a weight [104:57] there is some random number between [104:58] negative one and one for every one of [105:00] those inputs [105:01] and a bias that controls the overall [105:03] trigger happiness of this neuron [105:06] and then we're going to implement a def [105:08] underscore underscore call [105:11] of self and x some input x [105:14] and really what we don't do here is w [105:15] times x plus b [105:17] where w times x here is a dot product [105:19] specifically [105:21] now if you haven't seen [105:22] call [105:24] let me just return 0.0 here for now the [105:26] way this works now is we can have an x [105:28] which is say like 2.0 3.0 then we can [105:31] initialize a neuron that is [105:32] two-dimensional [105:33] because these are two numbers and then [105:35] we can feed those two numbers into that [105:37] neuron to get an output [105:39] and so when you use this notation n of x [105:42] python will use call [105:45] so currently call just return 0.0 [105:50] now we'd like to actually do the forward [105:52] pass of this neuron instead [105:54] so we're going to do here first is we [105:57] need to basically multiply all of the [105:58] elements of w with all of the elements [106:01] of x pairwise we need to multiply them [106:04] so the first thing we're going to do is [106:05] we're going to zip up [106:07] celta w and x [106:09] and in python zip takes two iterators [106:12] and it creates a new iterator that [106:14] iterates over the tuples of the [106:16] corresponding entries [106:17] so for example just to show you we can [106:20] print this list [106:22] and still return 0.0 here [106:30] sorry [106:34] so we see that these w's are paired up [106:36] with the x's w with x [106:41] and now what we want to do is [106:47] for w i x i in [106:50] we want to multiply w times [106:52] w wi times x i [106:54] and then we want to sum all of that [106:56] together [106:57] to come up with an activation [106:59] and add also subnet b on top [107:02] so that's the raw activation and then of [107:04] course we need to pass that through a [107:05] non-linearity so what we're going to be [107:07] returning is act.10h [107:09] and here's out [107:12] so [107:13] now we see that we are getting some [107:14] outputs and we get a different output [107:16] from a neuron each time because we are [107:17] initializing different weights and by [107:19] and biases [107:21] and then to be a bit more efficient here [107:22] actually sum by the way takes a second [107:25] optional parameter which is the start [107:28] and by default the start is zero so [107:31] these elements of this sum will be added [107:34] on top of zero to begin with but [107:35] actually we can just start with cell dot [107:37] b [107:38] and then we just have an expression like [107:39] this [107:45] and then the generator expression here [107:47] must be parenthesized in python [107:49] there we go [107:53] yep so now we can forward a single [107:55] neuron next up we're going to define a [107:57] layer of neurons so here we have a [107:59] schematic for a mlb [108:02] so we see that these mlps each layer [108:05] this is one layer has actually a number [108:07] of neurons and they're not connected to [108:08] each other but all of them are fully [108:09] connected to the input [108:11] so what is a layer of neurons it's just [108:13] it's just a set of neurons evaluated [108:15] independently [108:16] so [108:17] in the interest of time i'm going to do [108:20] something fairly straightforward here [108:23] it's um [108:25] literally a layer is just a list of [108:27] neurons [108:28] and then how many neurons do we have we [108:30] take that as an input argument here how [108:32] many neurons do you want in your layer [108:34] number of outputs in this layer [108:36] and so we just initialize completely [108:38] independent neurons with this given [108:40] dimensionality and when we call on it we [108:43] just independently [108:44] evaluate them so now instead of a neuron [108:47] we can make a layer of neurons they are [108:49] two-dimensional neurons and let's have [108:51] three of them [108:52] and now we see that we have three [108:53] independent evaluations of three [108:55] different neurons [108:57] right [108:58] okay finally let's complete this picture [109:00] and define an entire multi-layer [109:02] perceptron or mlp [109:04] and as we can see here in an mlp these [109:06] layers just feed into each other [109:07] sequentially [109:09] so let's come here and i'm just going to [109:11] copy the code here in interest of time [109:14] so an mlp is very similar [109:16] we're taking the number of inputs [109:18] as before but now instead of taking a [109:20] single n out which is number of neurons [109:22] in a single layer we're going to take a [109:24] list of an outs and this list defines [109:26] the sizes of all the layers that we want [109:28] in our mlp [109:30] so here we just put them all together [109:31] and then iterate over consecutive pairs [109:34] of these sizes and create layer objects [109:36] for them [109:37] and then in the call function we are [109:39] just calling them sequentially so that's [109:41] an mlp really [109:42] and let's actually re-implement this [109:44] picture so we want three input neurons [109:46] and then two layers of four and an [109:48] output unit [109:49] so [109:50] we want [109:52] a three-dimensional input say this is an [109:54] example input we want three inputs into [109:57] two layers of four and one output [110:00] and this of course is an mlp [110:03] and there we go that's a forward pass of [110:05] an mlp [110:06] to make this a little bit nicer you see [110:08] how we have just a single element but [110:09] it's wrapped in a list because layer [110:11] always returns lists [110:13] circle for convenience [110:15] return outs at zero if len out is [110:18] exactly a single element [110:20] else return fullest [110:22] and this will allow us to just get a [110:23] single value out at the last layer that [110:25] only has a single neuron [110:28] and finally we should be able to draw [110:29] dot of n of x [110:31] and [110:32] as you might imagine [110:34] these expressions are now getting [110:36] relatively involved [110:38] so this is an entire mlp that we're [110:40] defining now [110:45] all the way until a single output [110:48] okay [110:49] and so obviously you would never [110:50] differentiate on pen and paper these [110:52] expressions but with micrograd we will [110:55] be able to back propagate all the way [110:56] through this [110:58] and back propagate [110:59] into [111:00] these weights of all these neurons so [111:02] let's see how that works okay so let's [111:04] create ourselves a very simple [111:06] example data set here [111:08] so this data set has four examples [111:11] and so we have four possible [111:13] inputs into the neural net [111:15] and we have four desired targets so we'd [111:17] like the neural net to assign [111:21] or output 1.0 when it's fed this example [111:24] negative one when it's fed these [111:25] examples and one when it's fed this [111:26] example so it's a very simple binary [111:28] classifier neural net basically that we [111:30] would like here [111:32] now let's think what the neural net [111:33] currently thinks about these four [111:34] examples we can just get their [111:36] predictions [111:37] um basically we can just call n of x for [111:40] x in axis [111:42] and then we can [111:43] print [111:45] so these are the outputs of the neural [111:46] net on those four examples [111:48] so [111:50] the first one is 0.91 but we'd like it [111:52] to be one so we should push this one [111:55] higher this one we want to be higher [111:58] this one says 0.88 and we want this to [112:00] be negative one [112:02] this is 0.8 we want it to be negative [112:04] one [112:05] and this one is 0.8 we want it to be one [112:08] so how do we make the neural net and how [112:10] do we tune the weights [112:12] to [112:12] better predict the desired targets [112:16] and the trick used in deep learning to [112:18] achieve this is to [112:20] calculate a single number that somehow [112:22] measures the total performance of your [112:24] neural net and we call this single [112:25] number the loss [112:28] so the loss [112:29] first [112:31] is is a single number that we're going [112:32] to define that basically measures how [112:34] well the neural net is performing right [112:36] now we have the intuitive sense that [112:37] it's not performing very well because [112:38] we're not very much close to this [112:40] so the loss will be high and we'll want [112:43] to minimize the loss [112:44] so in particular in this case what we're [112:46] going to do is we're going to implement [112:47] the mean squared error loss [112:49] so this is doing is we're going to [112:51] basically iterate um [112:54] for y ground truth [112:56] and y output in zip of um [112:59] wise and white red so we're going to [113:01] pair up the [113:03] ground truths with the predictions [113:06] and this zip iterates over tuples of [113:07] them [113:08] and for each [113:11] y ground truth and y output we're going [113:13] to subtract them [113:16] and square them [113:18] so let's first see what these losses are [113:20] these are individual loss components [113:22] and so basically for each [113:25] one of the four [113:26] we are taking the prediction and the [113:28] ground truth we are subtracting them and [113:30] squaring them [113:32] so because [113:33] this one is so close to its target 0.91 [113:36] is almost one [113:38] subtracting them gives a very small [113:40] number [113:41] so here we would get like a negative [113:43] point one and then squaring it [113:45] just makes sure [113:47] that regardless of whether we are more [113:49] negative or more positive we always get [113:51] a positive [113:52] number instead of squaring we should we [113:55] could also take for example the absolute [113:56] value we need to discard the sign [113:59] and so you see that the expression is [114:00] ranged so that you only get zero exactly [114:03] when y out is equal to y ground truth [114:06] when those two are equal so your [114:07] prediction is exactly the target you are [114:09] going to get zero [114:10] and if your prediction is not the target [114:12] you are going to get some other number [114:15] so here for example we are way off and [114:17] so that's why the loss is quite high [114:19] and the more off we are the greater the [114:22] loss will be [114:24] so we don't want high loss we want low [114:26] loss [114:27] and so the final loss here will be just [114:30] the sum [114:32] of all of these [114:33] numbers [114:34] so you see that this should be zero [114:36] roughly plus zero roughly [114:38] but plus [114:39] seven [114:40] so loss should be about seven [114:43] here [114:44] and now we want to minimize the loss we [114:47] want the loss to be low [114:49] because if loss is low [114:51] then every one of the predictions is [114:54] equal to its target [114:56] so the loss the lowest it can be is zero [114:58] and the greater it is the worse off the [115:01] neural net is predicting [115:04] so now of course if we do lost that [115:05] backward [115:07] something magical happened when i hit [115:09] enter [115:10] and the magical thing of course that [115:12] happened is that we can look at [115:14] end.layers.neuron and that layers at say [115:16] like the the first layer [115:18] that neurons at zero [115:22] because remember that mlp has the layers [115:24] which is a list [115:26] and each layer has a neurons which is a [115:28] list and that gives us an individual [115:29] neuron [115:30] and then it's got some weights [115:32] and so we can for example look at the [115:34] weights at zero [115:38] um [115:40] oops it's not called weights it's called [115:42] w [115:44] and that's a value but now this value [115:46] also has a groud because of the backward [115:48] pass [115:50] and so we see that because this gradient [115:52] here on this particular weight of this [115:54] particular neuron of this particular [115:56] layer is negative [115:57] we see that its influence on the loss is [116:00] also negative so slightly increasing [116:02] this particular weight of this neuron of [116:04] this layer would make the loss go down [116:08] and we actually have this information [116:10] for every single one of our neurons and [116:12] all their parameters actually it's worth [116:13] looking at also the draw dot loss by the [116:16] way [116:17] so previously we looked at the draw dot [116:19] of a single neural neuron forward pass [116:21] and that was already a large expression [116:23] but what is this expression we actually [116:25] forwarded [116:27] every one of those four examples and [116:29] then we have the loss on top of them [116:30] with the mean squared error [116:32] and so this is a really massive graph [116:36] because this graph that we've built up [116:38] now [116:39] oh my gosh this graph that we've built [116:41] up now [116:42] which is kind of excessive it's [116:44] excessive because it has four forward [116:46] passes of a neural net for every one of [116:48] the examples and then it has the loss on [116:50] top [116:51] and it ends with the value of the loss [116:53] which was 7.12 [116:55] and this loss will now back propagate [116:56] through all the four forward passes all [116:58] the way through just every single [117:00] intermediate value of the neural net [117:03] all the way back to of course the [117:05] parameters of the weights which are the [117:06] input [117:07] so these weight parameters here are [117:10] inputs to this neural net [117:12] and [117:13] these numbers here these scalars are [117:15] inputs to the neural net [117:16] so if we went around here [117:18] we'll probably find [117:20] some of these examples this 1.0 [117:22] potentially maybe this 1.0 or you know [117:25] some of the others and you'll see that [117:26] they all have gradients as well [117:28] the thing is these gradients on the [117:30] input data are not that useful to us [117:33] and that's because the input data seems [117:36] to be not changeable it's it's a given [117:38] to the problem and so it's a fixed input [117:40] we're not going to be changing it or [117:42] messing with it even though we do have [117:43] gradients for it [117:46] but some of these gradients here [117:49] will be for the neural network [117:50] parameters the ws and the bs and those [117:53] we of course we want to change [117:55] okay so now we're going to want some [117:58] convenience code to gather up all of the [117:59] parameters of the neural net so that we [118:01] can operate on all of them [118:03] simultaneously and every one of them we [118:05] will nudge a tiny amount [118:08] based on the gradient information [118:10] so let's collect the parameters of the [118:11] neural net all in one array [118:14] so let's create a parameters of self [118:17] that just [118:18] returns celta w which is a list [118:22] concatenated with [118:24] a list of self.b [118:27] so this will just return a list [118:29] list plus list just you know gives you a [118:31] list [118:32] so that's parameters of neuron and i'm [118:35] calling it this way because also pi [118:36] torch has a parameters on every single [118:38] and in module [118:40] and uh it does exactly what we're doing [118:42] here it just returns the [118:44] parameter tensors for us as the [118:46] parameter scalars [118:48] now layer is also a module so it will [118:50] have parameters [118:52] itself [118:54] and basically what we want to do here is [118:56] something like this like [119:00] params is here and then for [119:03] neuron in salt out neurons [119:07] we want to get neuron.parameters [119:10] and we want to params.extend [119:14] right so these are the parameters of [119:16] this neuron and then we want to put them [119:17] on top of params so params dot extend [119:21] of peace [119:22] and then we want to return brands [119:25] so this is way too much code so actually [119:28] there's a way to simplify this which is [119:31] return [119:33] p [119:35] for neuron in self [119:38] neurons [119:39] for [119:41] p in neuron dot parameters [119:45] so it's a single list comprehension in [119:47] python you can sort of nest them like [119:49] this and you can um [119:51] then create [119:52] uh the desired [119:54] array so this is these are identical [119:57] we can take this out [120:00] and then let's do the same here [120:04] def parameters [120:06] self [120:07] and return [120:09] a parameter for layer in self dot layers [120:13] for [120:15] p in layer dot parameters [120:20] and that should be good [120:23] now let me pop out this so [120:26] we don't re-initialize our network [120:28] because we need to re-initialize [120:31] our [120:35] okay so unfortunately we will have to [120:37] probably re-initialize the network [120:38] because we just add functionality [120:41] because this class of course we i want [120:43] to get all the and that parameters but [120:45] that's not going to work because this is [120:47] the old class [120:49] okay [120:50] so unfortunately we do have to [120:52] reinitialize the network which will [120:53] change some of the numbers [120:55] but let me do that so that we pick up [120:57] the new api we can now do in the [120:58] parameters [121:00] and these are all the weights and biases [121:02] inside the entire neural net [121:05] so in total this mlp has 41 parameters [121:11] and [121:12] now we'll be able to change them [121:15] if we recalculate the loss here we see [121:18] that unfortunately we have slightly [121:19] different [121:22] predictions and slightly different laws [121:26] but that's okay [121:28] okay so we see that this neurons [121:31] gradient is slightly negative we can [121:33] also look at its data right now [121:36] which is 0.85 so this is the current [121:38] value of this neuron and this is its [121:40] gradient on the loss [121:43] so what we want to do now is we want to [121:45] iterate for every p in [121:47] n dot parameters so for all the 41 [121:49] parameters in this neural net [121:51] we actually want to change p data [121:55] slightly [121:56] according to the gradient information [121:59] okay so [122:00] dot dot to do here [122:02] but this will be basically a tiny update [122:05] in this gradient descent scheme in [122:08] gradient descent we are thinking of the [122:10] gradient as a vector pointing in the [122:13] direction [122:14] of [122:15] increased [122:16] loss [122:19] and so [122:20] in gradient descent we are modifying [122:22] p data [122:24] by a small step size in the direction of [122:26] the gradient so the step size as an [122:28] example could be like a very small [122:29] number like 0.01 is the step size times [122:32] p dot grad [122:35] right [122:36] but we have to think through some of the [122:37] signs here [122:38] so uh [122:40] in particular working with this specific [122:43] example here [122:44] we see that if we just left it like this [122:47] then this neuron's value [122:49] would be currently increased by a tiny [122:51] amount of the gradient [122:53] the grain is negative so this value of [122:56] this neuron would go slightly down it [122:58] would become like 0.8 you know four or [123:00] something like that [123:02] but if this neuron's value goes lower [123:06] that would actually [123:08] increase the loss [123:10] that's because [123:12] the derivative of this neuron is [123:14] negative so increasing [123:16] this makes the loss go down so [123:19] increasing it is what we want to do [123:21] instead of decreasing it so basically [123:23] what we're missing here is we're [123:24] actually missing a negative sign [123:26] and again this other interpretation [123:29] and that's because we want to minimize [123:30] the loss we don't want to maximize the [123:31] loss we want to decrease it [123:33] and the other interpretation as i [123:34] mentioned is you can think of the [123:36] gradient vector [123:37] so basically just the vector of all the [123:39] gradients [123:40] as pointing in the direction of [123:42] increasing [123:44] the loss but then we want to decrease it [123:46] so we actually want to go in the [123:47] opposite direction [123:49] and so you can convince yourself that [123:50] this sort of plug does the right thing [123:51] here with the negative because we want [123:53] to minimize the loss [123:55] so if we nudge all the parameters by [123:57] tiny amount [124:00] then we'll see that [124:02] this data will have changed a little bit [124:04] so now this neuron [124:06] is a tiny amount greater [124:08] value so 0.854 went to 0.857 [124:13] and that's a good thing because slightly [124:16] increasing this neuron [124:18] uh [124:18] data makes the loss go down according to [124:21] the gradient and so the correct thing [124:23] has happened sign wise [124:26] and so now what we would expect of [124:27] course is that [124:29] because we've changed all these [124:30] parameters we expect that the loss [124:32] should have gone down a bit [124:35] so we want to re-evaluate the loss let [124:37] me basically [124:39] this is just a data definition that [124:41] hasn't changed but the forward pass here [124:44] of the network we can recalculate [124:49] and actually let me do it outside here [124:51] so that we can compare the two loss [124:52] values [124:54] so here if i recalculate the loss [124:57] we'd expect the new loss now to be [124:59] slightly lower than this number so [125:01] hopefully what we're getting now is a [125:03] tiny bit lower than 4.84 [125:06] 4.36 [125:08] okay and remember the way we've arranged [125:10] this is that low loss means that our [125:12] predictions are matching the targets so [125:15] our predictions now are probably [125:16] slightly closer to the [125:18] targets and now all we have to do is we [125:22] have to iterate this process [125:24] so again um we've done the forward pass [125:26] and this is the loss [125:28] now we can lost that backward [125:30] let me take these out and we can do a [125:32] step size [125:34] and now we should have a slightly lower [125:35] loss 4.36 goes to 3.9 [125:39] and okay so [125:41] we've done the forward pass here's the [125:43] backward pass [125:44] nudge [125:45] and now the loss is 3.66 [125:50] 3.47 [125:52] and you get the idea we just continue [125:54] doing this and this is uh gradient [125:56] descent we're just iteratively doing [125:58] forward pass backward pass update [126:01] forward pass backward pass update and [126:02] the neural net is improving its [126:04] predictions [126:05] so here if we look at why pred now [126:09] like red [126:12] we see that um [126:14] this value should be getting closer to [126:16] one [126:16] so this value should be getting more [126:17] positive these should be getting more [126:19] negative and this one should be also [126:20] getting more positive so if we just [126:22] iterate this [126:23] a few more times [126:26] actually we may be able to afford go to [126:28] go a bit faster let's try a slightly [126:30] higher learning rate [126:34] oops okay there we go so now we're at [126:35] 0.31 [126:39] if you go too fast by the way if you try [126:41] to make it too big of a step you may [126:43] actually overstep [126:47] it's overconfidence because again [126:48] remember we don't actually know exactly [126:50] about the loss function the loss [126:51] function has all kinds of structure and [126:53] we only know about the very local [126:55] dependence of all these parameters on [126:57] the loss but if we step too far [126:59] we may step into you know a part of the [127:01] loss that is completely different [127:03] and that can destabilize training and [127:04] make your loss actually blow up even [127:08] so the loss is now 0.04 so actually the [127:11] predictions should be really quite close [127:13] let's take a look [127:15] so you see how this is almost one [127:17] almost negative one almost one we can [127:19] continue going [127:21] uh so [127:22] yep backward [127:24] update [127:25] oops there we go so we went way too fast [127:28] and um [127:29] we actually overstepped [127:31] so we got two uh too eager where are we [127:34] now oops [127:36] okay [127:37] seven e negative nine so this is very [127:39] very low loss [127:41] and the predictions [127:43] are basically perfect [127:45] so somehow we [127:47] basically we were doing way too big [127:48] updates and we briefly exploded but then [127:50] somehow we ended up getting into a [127:51] really good spot so usually this [127:54] learning rate and the tuning of it is a [127:56] subtle art you want to set your learning [127:58] rate if it's too low you're going to [128:00] take way too long to converge but if [128:02] it's too high the whole thing gets [128:03] unstable and you might actually even [128:05] explode the loss [128:07] depending on your loss function [128:08] so finding the step size to be just [128:10] right it's it's a pretty subtle art [128:12] sometimes when you're using sort of [128:14] vanilla gradient descent [128:15] but we happen to get into a good spot we [128:17] can look at [128:19] n-dot parameters [128:22] so this is the setting of weights and [128:25] biases [128:26] that makes our network [128:29] predict [128:30] the desired targets [128:31] very very close [128:33] and [128:35] basically we've successfully trained [128:37] neural net [128:38] okay let's make this a tiny bit more [128:40] respectable and implement an actual [128:41] training loop and what that looks like [128:43] so this is the data definition that [128:45] stays this is the forward pass [128:47] um so [128:49] for uh k in range you know we're going [128:52] to [128:53] take a bunch of steps [128:57] first you do the forward pass [129:00] we validate the loss [129:03] let's re-initialize the neural net from [129:05] scratch [129:06] and here's the data [129:08] and we first do before pass then we do [129:11] the backward pass [129:19] and then we do an update that's gradient [129:21] descent [129:26] and then we should be able to iterate [129:27] this and we should be able to print the [129:29] current step [129:30] the current loss um let's just print the [129:33] sort of [129:34] number of the loss [129:36] and [129:38] that should be it [129:40] and then the learning rate 0.01 is a [129:42] little too small 0.1 we saw is like a [129:44] little bit dangerously too high let's go [129:46] somewhere in between [129:47] and we'll optimize this for [129:50] not 10 steps but let's go for say 20 [129:52] steps [129:54] let me erase all of this junk [129:59] and uh let's run the optimization [130:03] and you see how we've actually converged [130:05] slower in a more controlled manner and [130:08] got to a loss that is very low [130:11] so [130:12] i expect white bread to be quite good [130:15] there we go [130:19] um [130:22] and [130:23] that's it [130:24] okay so this is kind of embarrassing but [130:25] we actually have a really terrible bug [130:28] in here and it's a subtle bug and it's a [130:31] very common bug and i can't believe i've [130:33] done it for the 20th time in my life [130:36] especially on camera and i could have [130:38] reshot the whole thing but i think it's [130:39] pretty funny and you know you get to [130:41] appreciate a bit what um working with [130:44] neural nets maybe [130:45] is like sometimes [130:47] we are guilty of [130:50] come bug i've actually tweeted [130:52] the most common neural net mistakes a [130:54] long time ago now [130:56] uh and [130:57] i'm not really [130:59] gonna explain any of these except for we [131:01] are guilty of number three you forgot to [131:03] zero grad [131:04] before that backward what is that [131:09] basically what's happening and it's a [131:10] subtle bug and i'm not sure if you saw [131:12] it [131:12] is that [131:14] all of these [131:15] weights here have a dot data and a dot [131:17] grad [131:19] and that grad starts at zero [131:22] and then we do backward and we fill in [131:24] the gradients [131:25] and then we do an update on the data but [131:27] we don't flush the grad [131:29] it stays there [131:31] so when we do the second [131:33] forward pass and we do backward again [131:35] remember that all the backward [131:36] operations do a plus equals on the grad [131:39] and so these gradients just [131:41] add up and they never get reset to zero [131:44] so basically we didn't zero grad so [131:47] here's how we zero grad before [131:50] backward [131:51] we need to iterate over all the [131:52] parameters [131:54] and we need to make sure that p dot grad [131:56] is set to zero [131:58] we need to reset it to zero just like it [132:00] is in the constructor [132:02] so remember all the way here for all [132:04] these value nodes grad is reset to zero [132:07] and then all these backward passes do a [132:09] plus equals from that grad [132:11] but we need to make sure that [132:13] we reset these graphs to zero so that [132:15] when we do backward [132:17] all of them start at zero and the actual [132:18] backward pass accumulates um [132:21] the loss derivatives into the grads [132:25] so this is zero grad in pytorch [132:28] and uh [132:30] we will slightly get we'll get a [132:31] slightly different optimization let's [132:33] reset the neural net [132:34] the data is the same this is now i think [132:37] correct [132:38] and we get a much more [132:40] you know we get a much more [132:42] slower descent [132:44] we still end up with pretty good results [132:46] and we can continue this a bit more [132:48] to get down lower [132:50] and lower [132:51] and lower [132:54] yeah [132:56] so the only reason that the previous [132:57] thing worked it's extremely buggy um the [132:59] only reason that worked is that [133:03] this is a very very simple problem [133:05] and it's very easy for this neural net [133:07] to fit this data [133:09] and so the grads ended up accumulating [133:12] and it effectively gave us a massive [133:13] step size and it made us converge [133:16] extremely fast [133:19] but basically now we have to do more [133:20] steps to get to very low values of loss [133:24] and get wipe red to be really good we [133:26] can try to [133:27] step a bit greater [133:34] yeah we're gonna get closer and closer [133:36] to one minus one and one [133:38] so [133:39] working with neural nets is sometimes [133:41] tricky because [133:43] uh [133:44] you may have lots of bugs in the code [133:47] and uh your network might actually work [133:49] just like ours worked [133:51] but chances are is that if we had a more [133:53] complex problem then actually this bug [133:55] would have made us not optimize the loss [133:57] very well and we were only able to get [133:59] away with it because [134:01] the problem is very simple [134:03] so let's now bring everything together [134:04] and summarize what we learned [134:06] what are neural nets neural nets are [134:09] these mathematical expressions [134:11] fairly simple mathematical expressions [134:13] in the case of multi-layer perceptron [134:15] that take [134:16] input as the data and they take input [134:19] the weights and the parameters of the [134:20] neural net mathematical expression for [134:22] the forward pass followed by a loss [134:24] function and the loss function tries to [134:26] measure the accuracy of the predictions [134:29] and usually the loss will be low when [134:31] your predictions are matching your [134:32] targets or where the network is [134:34] basically behaving well so we we [134:37] manipulate the loss function so that [134:38] when the loss is low the network is [134:40] doing what you want it to do on your [134:42] problem [134:44] and then we backward the loss [134:46] use backpropagation to get the gradient [134:48] and then we know how to tune all the [134:50] parameters to decrease the loss locally [134:52] but then we have to iterate that process [134:54] many times in what's called the gradient [134:55] descent [134:56] so we simply follow the gradient [134:58] information and that minimizes the loss [135:01] and the loss is arranged so that when [135:02] the loss is minimized the network is [135:04] doing what you want it to do [135:06] and yeah so we just have a blob of [135:09] neural stuff and we can make it do [135:11] arbitrary things and that's what gives [135:13] neural nets their power um [135:15] it's you know this is a very tiny [135:16] network with 41 parameters [135:19] but you can build significantly more [135:20] complicated neural nets with billions [135:24] at this point almost trillions of [135:25] parameters and it's a massive blob of [135:28] neural tissue simulated neural tissue [135:31] roughly speaking [135:32] and you can make it do extremely complex [135:34] problems and these neurons then have all [135:37] kinds of very fascinating emergent [135:39] properties [135:40] in [135:41] when you try to make them do [135:43] significantly hard problems as in the [135:45] case of gpt for example [135:47] we have massive amounts of text from the [135:49] internet and we're trying to get a [135:51] neural net to predict to take like a few [135:53] words and try to predict the next word [135:55] in a sequence that's the learning [135:56] problem [135:57] and it turns out that when you train [135:58] this on all of internet the neural net [136:00] actually has like really remarkable [136:02] emergent properties but that neural net [136:04] would have hundreds of billions of [136:05] parameters [136:07] but it works on fundamentally the exact [136:09] same principles [136:10] the neural net of course will be a bit [136:12] more complex but otherwise the [136:15] value in the gradient is there [136:17] and would be identical and the gradient [136:19] descent would be there and would be [136:21] basically identical but people usually [136:23] use slightly different updates this is a [136:25] very simple stochastic gradient descent [136:27] update [136:28] um [136:29] and the loss function would not be mean [136:30] squared error they would be using [136:32] something called the cross-entropy loss [136:34] for predicting the next token so there's [136:36] a few more details but fundamentally the [136:37] neural network setup and neural network [136:39] training is identical and pervasive and [136:42] now you understand intuitively [136:44] how that works under the hood in the [136:46] beginning of this video i told you that [136:47] by the end of it you would understand [136:48] everything in micrograd and then we'd [136:50] slowly build it up let me briefly prove [136:52] that to you [136:54] so i'm going to step through all the [136:55] code that is in micrograd as of today [136:57] actually potentially some of the code [136:59] will change by the time you watch this [137:00] video because i intend to continue [137:01] developing micrograd [137:03] but let's look at what we have so far at [137:05] least init.pi is empty when you go to [137:07] engine.pi that has the value [137:10] everything here you should mostly [137:11] recognize so we have the data.grad [137:13] attributes we have the backward function [137:15] uh we have the previous set of children [137:17] and the operation that produced this [137:19] value [137:20] we have addition multiplication and [137:22] raising to a scalar power [137:25] we have the relu non-linearity which is [137:27] slightly different type of nonlinearity [137:28] than 10h that we used in this video [137:30] both of them are non-linearities and [137:32] notably 10h is not actually present in [137:34] micrograd as of right now but i intend [137:37] to add it later [137:38] with the backward which is identical and [137:40] then all of these other operations which [137:42] are built up on top of operations here [137:45] so values should be very recognizable [137:47] except for the non-linearity used in [137:48] this video [137:50] um there's no massive difference between [137:52] relu and 10h and sigmoid and these other [137:54] non-linearities they're all roughly [137:55] equivalent and can be used in mlps so i [137:58] use 10h because it's a bit smoother and [138:00] because it's a little bit more [138:01] complicated than relu and therefore it's [138:03] stressed a little bit more the [138:05] local gradients and working with those [138:07] derivatives which i thought would be [138:09] useful [138:10] and then that pi is the neural networks [138:12] library as i mentioned so you should [138:14] recognize identical implementation of [138:16] neuron layer and mlp [138:18] notably or not so much [138:20] we have a class module here there is a [138:22] parent class of all these modules i did [138:24] that because there's an nn.module class [138:27] in pytorch and so this exactly matches [138:29] that api and end.module and pytorch has [138:31] also a zero grad which i've refactored [138:33] out here [138:36] so that's the end of micrograd really [138:38] then there's a test [138:40] which you'll see [138:41] basically creates [138:42] two chunks of code one in micrograd and [138:45] one in pi torch and we'll make sure that [138:47] the forward and the backward pass agree [138:49] identically [138:50] for a slightly less complicated [138:51] expression a slightly more complicated [138:53] expression everything [138:55] agrees so we agree with pytorch on all [138:57] of these operations [138:58] and finally there's a demo.ipymb here [139:01] and it's a bit more complicated binary [139:03] classification demo than the one i [139:04] covered in this lecture so we only had a [139:07] tiny data set of four examples um here [139:09] we have a bit more complicated example [139:11] with lots of blue points and lots of red [139:13] points and we're trying to again build a [139:15] binary classifier to distinguish uh two [139:18] dimensional points as red or blue [139:20] it's a bit more complicated mlp here [139:22] with it's a bigger mlp [139:24] the loss is a bit more complicated [139:26] because [139:27] it supports batches [139:29] so because our dataset was so tiny we [139:31] always did a forward pass on the entire [139:32] data set of four examples but when your [139:35] data set is like a million examples what [139:37] we usually do in practice is we chair we [139:39] basically pick out some random subset we [139:41] call that a batch and then we only [139:43] process the batch forward backward and [139:45] update so we don't have to forward the [139:47] entire training set [139:49] so this supports batching because [139:51] there's a lot more examples here [139:53] we do a forward pass the loss is [139:55] slightly more different this is a max [139:57] margin loss that i implement here [140:00] the one that we used was the mean [140:01] squared error loss because it's the [140:03] simplest one [140:04] there's also the binary cross entropy [140:06] loss all of them can be used for binary [140:08] classification and don't make too much [140:10] of a difference in the simple examples [140:11] that we looked at so far [140:13] there's something called l2 [140:14] regularization used here this has to do [140:17] with generalization of the neural net [140:19] and controls the overfitting in machine [140:21] learning setting but i did not cover [140:23] these concepts and concepts in this [140:24] video potentially later [140:26] and the training loop you should [140:27] recognize so forward backward with zero [140:31] grad [140:32] and update and so on you'll notice that [140:35] in the update here the learning rate is [140:36] scaled as a function of number of [140:38] iterations and it [140:40] shrinks [140:41] and this is something called learning [140:43] rate decay so in the beginning you have [140:44] a high learning rate and as the network [140:47] sort of stabilizes near the end you [140:49] bring down the learning rate to get some [140:50] of the fine details in the end [140:53] and in the end we see the decision [140:54] surface of the neural net and we see [140:56] that it learns to separate out the red [140:58] and the blue area based on the data [141:00] points [141:01] so that's the slightly more complicated [141:03] example and then we'll demo that hyper [141:05] ymb that you're free to go over [141:07] but yeah as of today that is micrograd i [141:10] also wanted to show you a little bit of [141:11] real stuff so that you get to see how [141:13] this is actually implemented in [141:14] production grade library like by torch [141:16] uh so in particular i wanted to show i [141:18] wanted to find and show you the backward [141:20] pass for 10h in pytorch so here in [141:23] micrograd we see that the backward [141:25] password 10h is one minus t square [141:28] where t is the output of the tanh of x [141:33] times of that grad which is the chain [141:34] rule so we're looking for something that [141:36] looks like this [141:38] now [141:39] i went to pytorch um which has an open [141:42] source github codebase and uh i looked [141:45] through a lot of its code [141:47] and honestly i i i spent about 15 [141:49] minutes and i couldn't find 10h [141:51] and that's because these libraries [141:53] unfortunately they grow in size and [141:55] entropy and if you just search for 10h [141:57] you get apparently 2 800 results and 400 [142:01] and 406 files so i don't know what these [142:04] files are doing honestly [142:07] and why there are so many mentions of [142:09] 10h but unfortunately these libraries [142:11] are quite complex they're meant to be [142:12] used not really inspected um [142:15] eventually i did stumble on someone [142:18] who tries to change the 10 h backward [142:21] code for some reason [142:22] and someone here pointed to the cpu [142:24] kernel and the kuda kernel for 10 inch [142:26] backward [142:27] so this so basically depends on if [142:29] you're using pi torch on a cpu device or [142:31] on a gpu which these are different [142:33] devices and i haven't covered this but [142:35] this is the 10 h backwards kernel [142:37] for uh cpu [142:40] and the reason it's so large is that [142:43] number one this is like if you're using [142:45] a complex type which we haven't even [142:46] talked about if you're using a specific [142:48] data type of b-float 16 which we haven't [142:50] talked about [142:52] and then if you're not then this is the [142:54] kernel and deep here we see something [142:57] that resembles our backward pass so they [143:00] have a times one minus [143:02] b square uh so this b [143:05] b here must be the output of the 10h and [143:07] this is the health.grad so here we found [143:10] it [143:11] uh deep inside [143:14] pi torch from this location for some [143:15] reason inside binaryops kernel when 10h [143:18] is not actually a binary op [143:21] and then this is the gpu kernel [143:25] we're not complex [143:26] we're [143:27] here and here we go with one line of [143:29] code [143:30] so we did find it but basically [143:33] unfortunately these codepieces are very [143:34] large and [143:36] micrograd is very very simple but if you [143:38] actually want to use real stuff uh [143:40] finding the code for it you'll actually [143:41] find that difficult [143:43] i also wanted to show you a little [143:45] example here where pytorch is showing [143:47] you how can you can register a new type [143:49] of function that you want to add to [143:51] pytorch as a lego building block [143:53] so here if you want to for example add a [143:55] gender polynomial 3 [143:59] here's how you could do it you will [144:00] register it as a class that [144:03] subclasses storage.org that function [144:06] and then you have to tell pytorch how to [144:07] forward your new function [144:10] and how to backward through it [144:12] so as long as you can do the forward [144:14] pass of this little function piece that [144:15] you want to add and as long as you know [144:17] the the local derivative the local [144:19] gradients which are implemented in the [144:20] backward pi torch will be able to back [144:22] propagate through your function and then [144:24] you can use this as a lego block in a [144:26] larger lego castle of all the different [144:28] lego blocks that pytorch already has [144:31] and so that's the only thing you have to [144:32] tell pytorch and everything would just [144:33] work and you can register new types of [144:35] functions [144:36] in this way following this example [144:38] and that is everything that i wanted to [144:40] cover in this lecture [144:41] so i hope you enjoyed building out [144:42] micrograd with me i hope you find it [144:44] interesting insightful [144:46] and [144:47] yeah i will post a lot of the links [144:50] that are related to this video in the [144:51] video description below i will also [144:53] probably post a link to a discussion [144:55] forum [144:56] or discussion group where you can ask [144:58] questions related to this video and then [145:00] i can answer or someone else can answer [145:02] your questions and i may also do a [145:04] follow-up video that answers some of the [145:06] most common questions [145:08] but for now that's it i hope you enjoyed [145:10] it if you did then please like and [145:11] subscribe so that youtube knows to [145:13] feature this video to more people [145:15] and that's it for now i'll see you later [145:22] now here's the problem [145:24] we know [145:25] dl by [145:28] wait what is the problem [145:31] and that's everything i wanted to cover [145:33] in this lecture [145:34] so i hope [145:35] you enjoyed us building up microcraft [145:38] micro crab [145:42] okay now let's do the exact same thing [145:43] for multiply because we can't do [145:44] something like a times two [145:47] oops [145:50] i know what happened there