Geoffrey Hinton talk "What is wrong with convolutional neural nets ?"

杰弗里·辛顿演讲：卷积神经网络有什么问题？ 在这场学术报告中，杰弗里·辛顿（Geoffrey Hinton）指出标准卷积神经网络存在根本性缺陷，因为其池化层在盲目追求不变性的过程中丢弃了空间关系与坐标系。相反，他主张将视觉识别视为逆向图形学过程，即神经网络利用“胶囊”（capsules）将像素强度映射为显式的多维姿态向量。通过一致性动态路由等机制，高层胶囊可检测线性姿态预测的紧密聚类，从而排除偶然巧合并表征层级化的“部分-整体”关系。辛顿展示了基于期望最大化聚类与因子分析器的实验原型，证明这种逆渲染方法在 MNIST 等任务上仅用极少量的标注数据即可实现很高的统计效率。 杰弗里·辛顿提出用“胶囊”作为结构性解决方案，以解决标准卷积神经网络因池化而丢弃精确空间关系的缺陷。 (2:14) 该研究引入了“一致性路由”，胶囊可根据单亲约束和高维姿态一致性，动态将信息分配给父级对象。 (34:14) 通过融入先验的几何图形学模型，胶囊架构仅用25个标注样本就在 MNIST 上取得了具有竞争力的分类性能。 (62:15) 胶囊网络将视觉视为“逆向图形学”问题，利用线性变换矩阵来捕捉视角变化的潜在流形。 (27:34)

Geoffrey Hinton argues that standard convolutional neural networks are fundamentally flawed because max pooling discards precise spatial relationships and fails to model the coordinate frames intrinsic to human perception. He proposes capsule networks, which treat vision as inverse graphics and preserve geometric structure through high-dimensional coincidence filtering and dynamic routing. The Flaws of Viewpoint Invariance and Max Pooling (10:24) Hinton contends that convolutional neural networks (ConvNets) solve the wrong computational problem by enforcing viewpoint invariance in neural activations rather than in learned synaptic weights. When viewpoints change, visual features displace across pixels—a phenomenon Hinton terms 'dimension hopping.' Standard architectures address this by using local max pooling to discard precise spatial locations, conflating feature detection with spatial routing. Hinton characterizes pooling as a biologically implausible mechanism that ignores how the human visual system relies on rectangular coordinate frames to recognize shapes. To demonstrate the cognitive rigidity of reference frames, Hinton cites Irving Rock's perception studies and his own sliced tetrahedron experiment. When a regular tetrahedron is cut in half, observers routinely fail to reassemble the two identical pieces because each piece induces an intrinsic elongated coordinate frame that clashes with the canonical triangular-base pyramid frame. Similarly, human observers quickly recognize rotated forms such as a tilted letter 'R' within roughly 250 milliseconds, yet require continuous mental rotation to resolve handedness (chirality). Hinton attributes this delay to the visual system's inability to solve high-order parity problems in a single feedforward pass, proving that spatial relationships are anchored by distributed coordinate systems rather than viewpoint-invariant labels. Capsules and Vision as Inverse Graphics (27:34) Hinton proposes replacing standard neurons with 'capsules'—groups of neurons modeled after biological cortical mini-columns. A capsule outputs both a scalar probability representing the presence of an entity and a multidimensional pose vector (typically 20 to 50 dimensions) specifying properties such as position, orientation, scale, and deformation. Instead of relying on brute-force data augmentation to learn viewpoint transformations, Hinton insists vision must operate as inverse graphics: taking estimated part poses and multiplying them by viewpoint-invariant transformation matrices to predict the pose of a whole object. Because coincidences in high-dimensional spaces are statistically impossible by chance—such as two independent estimates matching across six dimensions within 10% variation having roughly one-in-a-million random odds—higher-level capsules evaluate evidence by detecting tight clusters among incoming predictions. Hinton compares this mechanism to modernized Hough transforms and RANSAC algorithms. However, while early Hough transforms relied on low-dimensional features voting into accumulator bins, high-dimensional capsule features cast single, precise point votes that inherently filter out background outliers. Dynamic Routing by Agreement and Mixture Modeling (34:14) To parse visual scenes without pooling, Hinton outlines dynamic routing by agreement governed by a single-parent constraint: in an opaque world, any detected sub-part belongs to at most one parent object. Ambiguous low-level parts initially distribute their pose predictions across prospective parent candidates. Higher-level capsules evaluate incoming candidate votes using an Expectation-Maximization (EM) clustering loop, fitting a mixture of a Gaussian and a uniform baseline over approximately four iterations. When predictions cluster tightly, the likelihood ratio score increases, and top-down feedback routes the lower-level capsule's output exclusively to the agreeing parent. In working proof-of-concept implementations on image patches, primary capsules convert raw pixel intensities into linear pose coordinates alongside logistic presence units. Because primary capsule types share transformation weights across image patches, Hinton enforces translation equivariance by adding the patch's spatial coordinate offsets directly to the first two dimensions of the predicted pose vector. While non-linear networks are necessary to extract initial poses from raw pixels, Hinton emphasizes that mappings between part poses and whole poses must remain strictly linear. Manifold Disentanglement and Sample Efficiency (59:32) Hinton demonstrates that mapping pixels into explicit pose coordinates linearizes the underlying manifold of shape variations. Drawing on doctoral work by Tijmen Tieleman, an unsupervised de-rendering framework extracts entity poses and intensities, reconstructing digits through an innate graphics engine using learned feathered stroke templates. By concatenating capsule pose coordinates across the image and applying factor analysis, the network disentangles intrinsic shape deformations from global affine transformations along interpretable stylistic axes like slant and loopiness. This coordinate-based representation yields substantial improvements in statistical sample efficiency. On the MNIST dataset, standard backpropagation achieves roughly 1.6% test error using 60,000 labeled images, while Stéphane Mallat and Joan Bruna's scattering transforms reach 1.7% error with 2,000 labeled examples. In contrast, fitting a mixture of 25 factor analyzers to unsupervised capsule pose representations and querying the true class of just the single most confident example per component (25 labels in total) achieves an average test error of 1.75%. While capsule networks achieve high sample efficiency and structural interpretability, Hinton identifies computational overhead as a major bottleneck: an unoptimized Expectation-Maximization inner loop required two full days to train on a laptop, compared to 10 to 30 minutes for a standard ConvNet. Fully scaling the paradigm will require deeper capsule hierarchies, validation on complex real-world images, and faster algorithms to search for high-dimensional vector agreement.

Geoffrey Hinton 指出，现代卷积神经网络（CNN）依靠最大池化强求神经活动的不变性，在认知心理学和几何逻辑上存在根本缺陷。他主张机器视觉应被严格视为计算机图形学的逆过程，通过“胶囊”（Capsules）显式表征实体的坐标参考系，利用高维一致性的动态路由，实现突触权重中的真正知识不变性与神经活动的等变性。 池化机制缺陷与人类坐标系感知 (10:24) Hinton 批评当前 CNN 试图通过逐层池化（Pooling）消除视角差异是“在解决错误的问题”。最大池化虽然在工程上有效，但它丢弃了特征的精确位置，将特征提取与空间路由混为一谈。视角的变换会导致视觉特征跨越完全不同的像素网格（即“维度跳变”），而简单依赖局部最大池化无法解决这种动态格式错位。更为致命的是，CNN 缺乏人类形状感知中至关重要的矩形坐标系。 Hinton 通过将规则四面体对半切开的拼装难题指出，由于切开后暴露的截面与整体四面体的常规基底参考系脱节，即便高智商学者也极难在脑海中将其还原。当人类观察旋转的非洲地图或倾斜 45 度的正方形（菱形）时，认知表征也会因所设参考系的变化而截然不同。CNN 缺乏这种内在的坐标参考系机制，因而无法像人类一样依据视角灵活重组相同的像素阵列。 逆向图形学与高维一致性检测 (27:34) 为弥补现有神经网络缺乏“实体”概念的结构性缺陷，Hinton 提出将大脑皮层微柱的功能抽象为“胶囊”（Capsule）。每个胶囊输出两类核心信息：实体存在的概率，以及描述其大小、方向、颜色和形变等多维属性的广义姿态向量。Hinton 强调，计算机视觉应当完全遵循“逆向图形学”原则。在计算机图形学中，渲染引擎使用变换矩阵由整体姿态计算局部姿态；在视觉识别中，系统则应通过部件姿态乘以逆变换矩阵来预测整体姿态。 部件与整体之间的相对空间关系在任何视角下均保持恒定，因此完全可以固化在不变的突触权重中，而视角变化所带来的影响则由神经激活值的连续变化（即等变性）来承载。高层胶囊接收多个低层部件在 20 至 50 维空间中的姿态预测并搜寻紧密聚类。由于高维空间中的偶然巧合在统计上概率极低（例如在 6 维空间中独立匹配的随机概率仅约百万分之一），这种机制能在极高背景噪声中迅速锁定实体并完全剔除离群伪预测。 一致性动态路由与 EM 聚类实现 (34:14) 针对视角改变引发的特征流动，Hinton 提出了基于一致性的动态路由（Dynamic Routing by Agreement）以替代盲目的最大池化。在严格的“单一父代约束”下，一个未定归属的低层部件（例如一个既可能是眼睛也可能是车轮的圆形）会向多个候选的高层胶囊分配权重预测；高层胶囊通过自上而下的反馈竞争争取能够相互印证的输入，经过数轮迭代即可自适应构建出整幅图像的解析树，且不会破坏底层的感知表征。 在具体实现中，Hinton 演示了从图像块中提取姿态与激活概率的“初级胶囊”（Primary Capsules），并将其与类型相关的毕加索权重（Picasso weights）和线性坐标变换相结合。高层数字胶囊在 6 维姿态空间中运行期望最大化（EM）聚类算法，将输入预测拟合为高斯分布与背景均匀分布的混合模型。通过计算对数似然比打分，模型能够敏锐捕捉紧密聚类并忽略离群噪声，最终在 MNIST 手写数字分类任务中验证了该机制的分类能力。 无监督去渲染与极低标注学习 (62:15) 为克服深度学习对海量标注数据的依赖，Hinton 团队进一步探索了无监督的“去渲染”（De-rendering）架构。该系统内嵌一个硬编码的图形引擎进行重构，迫使胶囊网络在无需类别标签的情况下，自主发现并学习出具有羽化边缘的笔画模板。由于属于同一物体的各部件在视角变化下协同移动，提取出的姿态参数之间具有极高的互信息，并在姿态空间中构成清晰的线性流形。 通过对拼接后的姿态参数拟合包含 25 个组件的因子分析器混合模型（Mixture of Factor Analyzers），网络自然解耦出了倾斜度、笔画环状度等内在几何形变维度。在 MNIST 实验中，标准反向传播依靠 60,000 个全标注样本达到约 1.6% 错误率，而基于散射变换的先进基线也需要 2,000 个标注样本达到约 1.7% 错误率；相比之下，该胶囊系统通过无监督学习后，仅向每个因子分析器最具信心的单一原型各询问一次类别（总共仅使用 25 个标签），便达到了 1.75% 的测试错误率，展现出极高的统计样本效率。 Hinton 坦承，尽管该原型验证了高维一致性路由与线性流形建模的可行性，但当前依赖 MATLAB 运行的 EM 内部循环带来了极大的计算瓶颈，其训练耗时比传统 CNN 长出近两个数量级。此外，如何将胶囊架构拓展至复杂多目标的真实自然图像、实现更深层次的端到端层级，以及是否能推广至语音等非视觉时空信号，仍是有待进一步探索和优化的开放问题。

[10:24] Geoffrey Hinton: The worst property of convolutional nets is they fail to use an underlying linear manifold which is going to make it very easy to deal with the effects of viewpoint. [10:24] Geoffrey Hinton: We do not want the neural activities to be invariant to viewpoint at least not till the very top. [12:57] Geoffrey Hinton: When people do shape perception, they do it by imposing rectangular coordinate frames on things. [34:14] Geoffrey Hinton: I want a routing principle where you route information to the capsule that knows how to deal with it. [24:07] Geoffrey Hinton: Many neuroscientists think if you have big receptive fields, that means you've got low accuracy. That's exactly the opposite of the truth.

[0:00] uh he among his many titles he was the [0:04] founding director of the Gatsby unit and [0:06] University College London. He is [0:08] university professor at the University [0:10] of Toronto and he was also a recently [0:13] appointed distinguished researcher at [0:15] Google. Um of course Jeff is uh one of [0:19] the founders and the main name in the [0:22] fields of neur artificial neural [0:24] networks as well as in the field of [0:26] machine learning. uh his coming at MIT [0:29] is also a bit of an event. For those of [0:31] you who had had the talk at Seale [0:33] yesterday, there was 500 plus people and [0:35] basically people couldn't get in. Uh [0:38] today we starting at 4 because I'm sure [0:40] it going to be similar effect. Uh [0:43] there's also um a panel discussion that [0:47] Tommy is organizing at six uh with Jeff [0:51] in this same room about uh path to [0:53] intelligence. [0:55] Uh so I won't uh I won't say much uh [1:00] much more because we all came to hear [1:02] him just to say that the the ransom of [1:04] success might be what you see on his uh [1:08] web page which says information for [1:10] prospective student. I will not be [1:12] taking any new graduate student, [1:14] visiting student, summer student or [1:16] visitors. So please do not apply to work [1:20] with me. Jess, [1:27] thank you so much. If however you'd like [1:29] a job at Google, that's something [1:33] else. [1:34] Um, so there's a lot of things wrong [1:37] with the neural nets we're using. Um, [1:40] they've been quite successful at speech [1:42] recognition and object recognition, [1:43] particularly recently. But there's a lot [1:46] of things that are very unlike the brain [1:48] and that I believe are making them work [1:50] not as well as they could. So one thing [1:54] wrong is any complex engineered system [1:57] should have various levels of structure. [1:58] Neural nets have very few levels of [2:00] structure. There's neurons, there's [2:03] layers of neurons which aren't at all [2:04] like layers in cortex and there's whole [2:06] neural nets and that's it for most of [2:08] these neural nets. [2:11] Um, one thing that's missing in these [2:14] neural nets is there's no explicit [2:15] explicit notion of an entity. So for [2:18] those innatists in the audience, I am [2:20] willing to admit that it might be worth [2:22] building into a neural net something to [2:24] do with the idea that there are [2:26] entities. Um, and I want to build that [2:28] into the architecture. That's what this [2:29] talk's going to be about. So what I want [2:31] to do is take the neurons in what we [2:33] call a layer and group them into subsets [2:36] and have activities of neurons in those [2:38] subsets represent different properties [2:40] of the same entity. I want the neural [2:42] network to decide what the entities are [2:43] and how they interact with each other. [2:46] But I want the sort of built-in property [2:48] that there's going to be [2:50] entities. That's all I'm going to [2:51] concede to the innatists. Um I also want [2:55] to push the idea that a mini column is [2:57] the place where you represent an [2:59] entity. one entity per mini [3:02] column. Um, and I'm going to call this [3:04] thing in the artificial net a capsule. [3:06] And the idea is a capsule is going to [3:08] have two kinds of instantiation [3:10] parameters. Is can have one that says [3:12] whether its entity is present. I'll talk [3:14] about images mainly. So in the current [3:16] input image um and then it's going to [3:19] have other things that describe [3:21] properties of this entity. If the entity [3:23] is not present, you can say all you like [3:24] about the properties. It doesn't matter. [3:26] But if the entity is present, then you [3:28] want to know its properties. And the [3:29] properties are going to be things like [3:30] its orientation, its size, its velocity, [3:32] its color, all sorts of things like [3:34] that, its [3:35] deformation. Um, and what a capsule [3:38] outputs that goes to higher level [3:40] capsules because I want this in a [3:41] hierarchy is the probability that the [3:44] entity is present and the generalized [3:46] pose of the entity which envision is [3:48] going to be an object or part of an [3:49] object. Um, and that includes all sorts [3:53] of [3:56] parameters. And what capsules do that [3:59] does not go on in normal neural nets, [4:01] this is the sort of basic computation [4:03] they're meant to do [4:06] um is they take predictions from lower [4:11] level capsules about what their [4:13] generalized pose should be. So about a [4:16] multi-dimensional [4:18] vector. And they look for predictions [4:22] that agree tightly. and they don't care [4:24] if there's lots of predictions that are [4:26] outliers. What they're concerned with is [4:28] there are a small subset of predictions [4:30] that agree well. If you did computer [4:33] vision many many years ago before it got [4:35] silly, um this is things like ransack [4:37] and huff transforms. Um so a [4:41] capsule it has a highdimensional pose [4:44] space maybe 20 dimensions or 50 [4:46] dimensions. So this is a picture of that [4:48] space. Um, so this is a 20dimensional [4:51] space, let's say. And there's [4:53] predictions coming from capsules below. [4:55] And these are the predictions. These are [4:57] the vectors that are predicted for the [4:59] the pose of this capsule. And what we [5:02] want is something that'll find this [5:03] cluster. And what it'll output is it'll [5:07] output a probability that says, hey, I [5:11] really am present because look, that [5:12] didn't happen by chance. I'm really [5:14] there. I've got lots of evidence that I [5:16] exist. And I can output the center of [5:18] gravity of that. and I'm going to ignore [5:19] all these. And neural networks at [5:22] present are not good at doing that. They [5:24] may somehow be able to fake it, but [5:26] they're not built to do that kind of [5:27] thing. Now, the point about [5:29] highdimensional coincidences is they [5:31] don't happen by chance. Even if I have a [5:33] six-dimensional thing and I have two [5:35] sixdimensional things that agree on [5:37] every dimension to within 10% relative [5:40] to the normal variation, then there's a [5:42] chance of one in a million of that. Um, [5:44] so if you see a few things that agree [5:47] sharply, it's not a co, this coincidence [5:50] can't just be a coincidence, if you see [5:52] what I mean. Um, there's something [5:54] really must have caused it. Um, a model [5:57] of this is if you think about filtering [5:59] intelligence information, if you see all [6:01] sorts of conversations that say New York [6:03] or say September, um, that doesn't mean [6:07] much. But if you see a whole bunch of [6:09] conversations, maybe just four of them [6:10] that say New York September the 11th and [6:13] you see four of those, you should get [6:15] extremely suspicious because now it's a [6:16] highdimensional coincidence. And even if [6:18] there's lots of other things that say [6:20] Chicago and blah blah blah, the fact [6:22] that it's high dimensional is a [6:23] coincidence should make you think [6:24] there's something real going [6:25] on. Okay? And I'm going to argue [6:29] that I'm going to take a Marian [6:31] perspective and say in order to [6:34] understand the brain, we need to figure [6:35] out what it's computing. [6:37] And I'm going to give you a reason why [6:39] the brain needs to do this computation. [6:41] And then I'm going to suggest this is [6:42] what many columns are doing. It's wild [6:44] speculation, but at least it's marian [6:46] wild speculation because it's based on [6:49] it comes from the computation that needs [6:51] to be [6:53] done. Okay, so here's how neural nets [6:56] currently do object recognition and do [6:58] it very well. Um, compared with methods [7:01] that are even worse. [7:03] Um this is largely due to Yanaka who [7:06] since 1987 has been getting back prop to [7:09] do this and has developed the technology [7:10] a lot. Um he calls them convenets. [7:15] Um and they use multiple layers of [7:17] learned feature detectors. So that's [7:19] good. And the feature detectors are [7:22] local. The locality gets bigger as they [7:24] get bigger domains as they go up. [7:26] They're replicated across space because [7:28] you believe that if a feature is worth [7:30] having here, it's worth having there. [7:33] Um, that's [7:36] good. As you go up, the spatial domains [7:39] of these feature detectors get bigger. [7:41] That's good. And the feature extraction [7:44] layers are interled with max pooling [7:46] layers or average pooling layers or some [7:48] kind of pooling layer. And what the [7:50] pooling layers do is they say, um, I'm a [7:53] pooling neuron. I'm going to look at the [7:55] neurons in a layer, look at nearby ones, [7:57] and I'm going to attend to the typically [8:00] I'm going to attend to the most active [8:02] one. Um, it may be probabilistic, but [8:05] let's say I just attend to the most [8:06] active one. So, I'm going to report the [8:08] activity level of the most active one [8:10] and forget where it was. That's going to [8:12] give you a small amount of translational [8:14] invariance and it's going to give you [8:16] less active neurons. So, you're going to [8:19] be able to have more feature types at [8:21] the next layer. [8:23] Um, this is what I don't believe in. [8:26] This is an integral part of the things [8:27] that work really well. Um, the fact that [8:30] it works so well is extremely [8:32] unfortunate because it's going to make [8:33] it harder to get rid of. Um, but that's [8:37] just the way it [8:39] is. So, as I was saying, the pooling um [8:42] gives you some translational invariance [8:44] because it throws away where the most [8:45] active feature is. Um, actually it [8:48] doesn't have to lose positional [8:50] information if you have lots of pools [8:51] that overlap, but if you do that, you [8:53] get less advantage from reducing the [8:55] number of units. [9:01] Um, I'll leave it at that for that. [9:07] Now, if you take these convolutional [9:09] nets, I kind of showed yesterday that [9:12] with one of these convolutional nets, [9:14] you can take the activities in the last [9:17] hidden layer. If it's been trained to [9:19] recognize lots of different categories [9:21] of object, you can now train it. So, [9:23] this is a repeat of what I said [9:24] yesterday, just briefly. Um, you can [9:27] train it with a recurrent neural net to [9:30] actually give you a caption. And I'm [9:32] going to skip a bit here. Well, I'll [9:34] just show you what it does. So, you show [9:36] it this and the recurrent and you ask [9:39] it, "What do you see?" And the recurrent [9:40] neural in effect and the recurrent [9:42] neural net says two pizzas sitting on [9:44] top of a stove top oven. Um, if you run [9:47] it again, it's stochastic. It'll say um [9:51] a pizza on top of a pan on top of an [9:53] oven. Um whether or not it really [9:57] understands these relations or whether [9:58] that's just in the language model, we [9:59] don't yet know. um because it's learned [10:02] to model language as well. Um but that's [10:05] just to show you components can do [10:06] impressive [10:09] things. Um there's another caption. And [10:13] now here's why I don't believe in the [10:14] pooling. I believe in the convolution. I [10:16] just don't believe in the pooling. Um [10:19] it's a really bad fit to the psychology [10:20] of shape perception. There's things [10:22] about the psychology of shape perception [10:24] that argue very strongly that we're not [10:26] using convolutional neural nets. [10:28] um it's solving the wrong problem. We do [10:31] not want the neural activities to be [10:33] invariant to viewpoint at least not till [10:36] the very top. What we want is that the [10:39] knowledge is invariant to viewpoint. The [10:41] same knowledge can get applied to [10:43] something with a new viewpoint. That [10:45] doesn't mean the activities have to be [10:46] invariant to viewpoint. Um I'll go over [10:49] each of these in more detail later. Um [10:52] the worst property of convolutional nets [10:55] is they fail to use an underlying linear [10:58] manifold which is going to make it very [11:00] easy to deal with the effects of [11:01] viewpoint. It's the linear manifold [11:03] that's used in computer graphics. Um [11:06] here's a piece of natural reasoning. Um [11:09] there's two things that have no problem [11:11] with viewpoint. Um one is us and other [11:15] brains and the other is computer [11:18] graphics and therefore they work the [11:20] same way. [11:21] Okay. [11:22] [Music] [11:23] Um, and the last thing is pooling is a [11:27] very bad way or very primitive way to [11:29] try and do rooting. So in vision, what [11:33] happens is as you change the viewpoint, [11:35] the same thing shows up on different [11:37] pixels. That's not usual in machine [11:39] learning. If you change illumination, [11:41] for example, that doesn't happen. But [11:42] when you change viewpoint, you've moved [11:44] information from one set of pixels to [11:46] another set of pixels. So that's what I [11:48] call dimension hopping. And for machine [11:50] learning, you better unpick that, [11:52] otherwise you're not going to be able to [11:52] make any sense of things. Imagine two [11:55] hospitals and one codes patients by age, [11:58] weight, and financial status because [12:00] it's America. And the second codes [12:02] patients by weight, financial status, [12:04] and age. If you just took those records [12:07] and applied machine learning without [12:08] sorting that out, you wouldn't expect it [12:10] to work very well. Um, but in vision, [12:13] that's exactly what Viewpoint's doing. [12:15] And we better sort it out. And that's a [12:17] rooting problem. We need to take the [12:18] information in the pixels and root it [12:20] correctly to the neurons that know how [12:22] to deal with that kind of information. [12:24] And people haven't really most neuronet [12:25] people haven't really faced up to the [12:27] problem that we're having to solve a [12:28] rooting problem. Um people doing parts [12:31] based models are faced up to that but [12:32] not the convolutional net [12:36] people. So I'm going to go over these [12:38] four arguments. Now the talk is mainly [12:40] going to be about um trying to convince [12:42] you that convolutional nets are no good [12:43] even though they work very well. um or [12:46] rather the max pooling aspect of [12:47] convolutional nets is no good. [12:50] Um and I'm going to start by saying it's [12:53] a bad fit to the psychology of shape [12:55] perception. [12:57] So when people do shape perception, they [13:01] do it by imposing rectangular coordinate [13:03] frames on [13:04] things. And if you take the same object [13:07] and you impose a different rectangular [13:08] coordinate frame, you don't even realize [13:10] it's the same object. That's how much [13:12] effect it has. I can show you an object [13:14] and show you exactly the same object [13:15] again with exactly the same retinal [13:17] image and if you impose a different [13:18] coordinate frame, you won't realize [13:20] you've even seen it before. [13:23] Um, it's a huge effect and convolution [13:26] just can't explain that. They can't [13:27] explain how the same pixels can be [13:29] processed completely differently [13:31] depending on the coordinate frame [13:32] because they've got no notion of [13:33] imposing a coordinate frame. Now, this [13:35] isn't necessarily quite true. It may be [13:37] that neural nets are so powerful that [13:39] with these multiple layers, they kind of [13:40] fake imposing coordinate frames. And we [13:42] don't know whether that's true or not, [13:43] but let's suppose it isn't. Um, so I'm [13:46] going to try and give you a [13:47] demonstration that illustrates the power [13:48] of coordinate frames. [13:52] Um, here's something you wouldn't [13:54] believe. I take a simple object like a [13:58] tetrahedrin. A tetrahedrin is a pyramid [14:00] with a triangular [14:02] base. And I slice it with a [14:05] plane. That's a flat thing. I slice it [14:07] with a plane. Um, so I get two pieces [14:11] and then I take an intelligent person [14:14] and I give them the two pieces and I [14:16] say, "Okay, make the tetrahedrin." And I [14:18] make sure they know what a tetrahedrin [14:19] is. And they can't do it. Now, that [14:22] clearly needs Now, you don't believe [14:23] that presumably it's just two pieces. [14:25] Surely you can put them together to make [14:26] a tetrahedrin. Um, well, I've been doing [14:29] an experiment today on MIT professors. [14:32] Um, I got one sample 30 years ago. That [14:35] was a professor called Carl Huitt who [14:37] was very smart. I gave him the two [14:39] pieces and he looked at them for a long [14:43] time. He didn't manipulate them. Looked [14:44] at them for a long time. After 10 [14:45] minutes, he wrote down a proof it was [14:48] impossible. So, his time to solve the [14:50] puzzle is [14:52] infinite. Okay. Today I've been doing [14:54] experiments on MIT professors and the [14:56] number of minutes they take to solve the [14:58] problem is about the number of years [15:02] they've been at [15:03] MIT roughly speaking. It's definitely [15:06] very the length of time is very [15:08] positively correlated with how long [15:09] they've been at MIT. Um so now I'm going [15:12] to show you this puzzle um because it's [15:15] extraordinary. It can be so hard because [15:16] it's completely trivial and there's a [15:18] very obvious way to solve it that people [15:20] don't figure out. They figure out after [15:22] a few minutes, but okay, here's the two [15:26] pieces, right? I can do my sort of [15:28] conjuring trick now. Um, these two [15:31] pieces seem, they're the same. And [15:33] here's what people do. They say, "Okay, [15:35] tetrahedrin. Yeah, like, well, that's [15:36] not a tetrahedrin. Um, wait a minute, [15:40] that's not a tetrahedrin." And then they [15:42] try sticking the ends [15:44] together. And then they do [15:46] this. And they really do this, and [15:48] people have witnessed them doing this. [15:50] They do that and say, "Well, that's not [15:51] a tetrahedrin." Well, what about that? [15:54] That's not a [15:55] tetrahedrin. And after a few minutes, if [15:58] they're young, [16:00] um, they go, [16:02] "Oh, right. That's a [16:05] tetrahedrin." So, now, why is this [16:08] puzzle almost impossible? Why is it so [16:10] hard? Because it's a two-piece jigsaw [16:13] puzzle, and they're MIT professors. And [16:19] um oh incidentally I tried this on a [16:21] Google vice president just to reassure [16:23] the MIT professors. I gave the two [16:25] pieces to the Google vice president and [16:26] I said it's a really hard task. Can you [16:27] make a tetrahedrin? And he said [16:31] yes. Honestly [16:36] um so they didn't pay me to say that [16:39] it's true. Uh so why how could this be [16:42] so hard? Because look you know it's all [16:44] made of triangles of faces. And look, [16:46] here's a couple of squares. So, you got [16:47] to put the squares together because you [16:49] got to get rid of those squares somehow. [16:51] So, you have to put the squares [16:52] together. And everybody does that. And [16:53] when that doesn't work, they do that. [16:56] And that doesn't work either. It doesn't [17:00] work. And why don't they do [17:02] that? Well, the answer is when I show [17:05] you this piece, there's a natural Oops. [17:07] There's a natural coordinate frame. And [17:09] you can see the natural rectangular [17:10] coordinate frame for this piece. It's [17:12] got a long axis and it's got a axis [17:14] across an axis up and down. And that [17:17] coordinate frame does not line up at all [17:20] with the natural coordinate frame you [17:21] use for [17:22] tetrahedrin. Unless you're over 15 and [17:25] went to an American public school. If [17:27] you're over 50, went to an American [17:28] public school, you would have got milk [17:29] in tetrahedral cartons that were [17:32] stacked. And they were stacked like [17:34] Whoops. They were stacked like that. And [17:36] you have a model that says it's a line [17:38] there, a line here. join up all those [17:41] lines and a tetrahedrin looks like that. [17:43] And if you also have that model, which [17:45] I'll call a quadraedron because it's so [17:47] different, then it's obvious that if you [17:49] slice it here, you get a long thin [17:50] rectangle that way. If you slice it [17:52] there, you get a long thin rectangle [17:53] that way. And some mathematician said if [17:55] you slice it halfway, it better be in [17:56] between. And so you better get a square [17:58] somewhere. Um, if you have that model, [18:01] I'll call it the quadraed model, this [18:03] puzzle is completely obvious. And a [18:04] crystalographer will do that instantly. [18:06] Um but a normal person um just can't do [18:10] this puzzle. And it's because the the [18:12] frame of reference they use for tetra [18:14] region doesn't line up with the frame of [18:16] reference you naturally impose on one of [18:17] these pieces. There's other subsidiary [18:20] things like the pieces have a different [18:21] relationship to the whole and you don't [18:24] expect that because they're symmetrical. [18:25] You expect antisymmetric relationship to [18:26] the whole. [18:28] Anyway, that's it. That's the end of my [18:30] evidence that convolutional nets are not [18:32] psychologically correct. [18:36] Um yeah, that's sufficient to convince [18:41] me show the puzzle. Um Irving Rock [18:44] pointed out all this stuff a long time [18:46] ago. Um if you show people a map [18:50] um like this and ask them what country [18:53] it is, [18:55] um most people say, "Well, it's a little [18:58] bit like Australia, but it's not [18:59] Australia." [19:01] Um but if you tell people what country [19:04] do you think Sarah Palin might think [19:06] this is? Um people who are politically [19:09] savvy are they say this is Africa. [19:12] Um Sarah Palin actually thought it was a [19:15] country. [19:16] Um and if you see the top [19:19] here then it's immediately obvious it's [19:22] Africa. But if you impose the top wrong [19:23] you don't recognize it. And the very [19:26] familiar example is a tilted square. If [19:29] you see it as a tilted square and you [19:30] see it as a diamond, it's two different [19:32] internal objects. If you see it as a [19:34] diamond, you're totally unaware of [19:36] whether this is a right angle. I can [19:37] make that be 85 degrees or 95 degrees or [19:41] 90°. None none of them looks better than [19:43] the other. If you see it as a tilted [19:45] square, you're you're accurate to within [19:47] about one degree on whether that's a [19:48] right angle. So, lots of evidence for [19:51] rectangular frames and convolutional [19:53] don't use [19:55] them. One more bit of evidence. Okay, so [19:59] that tells me that people's visual [20:02] systems are recognizing things by [20:03] imposing rectangular coordinate frames [20:05] and probably a hierarchy of them. Um, [20:09] just like computer graphics does. [20:10] Computer graphics has to say what the [20:11] relation is between a part and a whole. [20:13] And so it has to impose a frame on the [20:15] whole, impose a frame on the part, and [20:18] then tell you the matrix that will map [20:21] points in the hole relative to the frame [20:23] of the hole to points relative to the [20:24] frame of the part. [20:26] Um, so it's all linear. That's the [20:28] linear manifold. But you can only [20:30] express it by imposing this coordinate [20:34] frame. Luckily, there's a chance in the [20:38] discussion later when Tommy can point [20:39] out how what I say is all [20:42] Um, I can read his mind. [20:45] And now, let me give you some more [20:48] evidence. And this is evidence not only [20:49] do we impose coordinate frames, but we [20:52] represent the imposed coordinate frames [20:55] by a bunch of separate neural [20:57] activities. Not by having one neuron [21:00] that says I'm imposing this frame. I [21:03] want to show you it's in a bunch of [21:04] different neural activities. [21:06] So suppose I show you um this what you [21:11] know within about 250 milliseconds or [21:14] less is that it's a capital letter R and [21:18] the top is here. What you don't know is [21:21] whether it's a correct R or mirror image [21:24] R. And to figure out that what you do is [21:27] you go chunk chunk chunk chunk until [21:29] it's upright. And then when it's upright [21:31] you say R. It faces the wrong way. So, [21:33] it's a mirror image [21:35] art. And why is that? A more impressive [21:38] demonstration of that is if you go with [21:41] a woman to a fancy shoe shop and you [21:44] show her a shoe, she can tell you within [21:47] about 250 milliseconds um how much it [21:50] costs, who made it, which other stores [21:52] it's available in. This is a bit sexist, [21:54] but there you go. [21:56] Um she can't tell you whether it's a [21:59] right-hand shoe or a lefthand shoe. Now [22:02] that's extraordinary. I mean she knows [22:03] all these properties of this shoe but [22:05] she doesn't know whether it's right hand [22:06] or left hand. Um she has to do this kind [22:08] of chunk chunk chunk to get in canonical [22:11] orientation. [22:12] Um so how could that be? So my argument [22:16] is that you have you know the [22:18] relationship between this and the [22:20] viewer. Okay. So you have the pose of [22:22] this in viewer center [22:24] coordinates and you need to know whether [22:27] the matrix that relates the intrinsic [22:30] frame of this to the viewer frame is a [22:34] left hand one or right hand one. And if [22:37] you have the matrix represented by a [22:38] bunch of separate numbers like they do [22:40] in computer graphics to know whether [22:42] it's a left hand one or a right hand one [22:44] you need to know the sign of the [22:45] determinant of the matrix. In other [22:47] words, you need to solve a high order [22:48] parity problem. Reversing any one [22:51] changes the answer. If you reverse the [22:54] reverse the value of any one number, the [22:56] answer has [22:57] changed. [22:59] So neural nets aren't good at high order [23:01] parity problems. So that's why we can't [23:02] do handedness. And the fact that we [23:04] can't do-handedness, I take as strong [23:06] evidence that our representation of the [23:09] imposed coordinate frame of the [23:11] relationship between that and the viewer [23:13] is spread over many numbers just like it [23:16] is in computer graphics. [23:21] Okay. Um, and so the way we solve it is [23:25] we do a continuous transformation that [23:26] preserves handedness until we've got it [23:28] down to just checking one direction and [23:30] seeing which way it goes and then we [23:33] solve it. And that's what that's how we [23:34] do it. And so all those mental rotation [23:36] things, they're not to recognize [23:38] objects. They're to deal with this [23:39] problem that you don't know the [23:42] handedness. Okay. So the conclusion for [23:45] argument one is people use rectangular [23:47] coordinate frames embedded in objects [23:48] and embedded in parts of objects and [23:51] they represent those coordinate frames [23:54] that is they represent the pose of the [23:55] object um the relation between its [23:58] embedded coordinate frame and the viewer [24:00] they represent that spread over a bunch [24:03] of numbers not just in one neuron. [24:07] Argument two, [24:08] equivariance. What convolutional nets [24:10] are trying to do is make the [24:13] representation not change with [24:14] viewpoint. And of course, that's what [24:16] you want the label to [24:18] do. But when you look at a face, for [24:21] example, it's not that you say face and [24:23] I don't know where it is or what its [24:24] orientation is or what its size is. You [24:26] look at a face and you know exactly what [24:28] its orientation is and exactly where it [24:30] is. So you haven't lost that information [24:31] like convolutional nets try and do. [24:33] You've got it all there very precisely. [24:35] Now, many neuroscientists think if you [24:37] have big receptive fields, that means [24:38] you've got low [24:40] accuracy. That's exactly the opposite of [24:42] the truth. If you want to have very high [24:45] accuracy for where something is and you [24:47] have a limited number of neurons, you [24:49] should make them have very big receptive [24:51] fields that overlap a whole lot. You'll [24:53] get much higher accuracy that way [24:55] because you'll divide space into many [24:56] tiny regions. And the number of regions [24:58] you make is proportional to the surface [25:00] area of these receptive fields. And to [25:02] get a lot of surface area, you make the [25:03] fields bigger. So the signature of [25:06] wanting really high accuracy for these [25:08] instantiation parameters is you should [25:09] have big fields. Of course you lose [25:12] resolution. So you can only afford to do [25:14] that for things where there aren't many [25:16] of them. You can't do it for edges [25:18] because there's too many of them around [25:20] because and you don't want to lose the [25:22] resolution. But as you go up to high [25:23] level things like faces um in some [25:27] reasonable area there's only a few [25:28] faces. So you can have big fields. Okay. [25:31] So what we want is we want a [25:34] representation where as you change [25:35] viewpoint the neural activities change [25:38] and they change just like the viewpoint [25:40] does. So convolutional nets without max [25:43] pooling are like [25:44] that. They say they got viewpoint [25:46] invariance but not at all. Here's a two [25:49] and here's the neurons that get active. [25:51] You translate the two and the neurons [25:53] that get active translate. It's the same [25:56] pattern but it's different neurons. This [26:00] is equivariance right change this and [26:03] this changes and it changes the same [26:07] way. Now I want to distinguish two types [26:10] of equariance using pseudo neuroscience [26:14] terminology in this if we translated by [26:17] a whole number of [26:19] pixels then these will move [26:22] across and this is what I call place [26:25] equivariance. As you change the pixels [26:28] this is on the neurons that represent it [26:31] change call that place [26:34] equivariance. There's a different kind [26:36] of equariance which is ratecoded [26:38] equivariance which says as I move the [26:41] object around the same neurons are [26:43] encoding it but the activities of the [26:46] neurons is changing. Okay. So it's a [26:49] rate code rather than a place code. And [26:51] what's happening in the visual system is [26:53] my belief is low level we have very [26:56] small [26:57] domains. So tiny changes change the rate [27:02] and then if you change by more than a [27:03] tiny amount you change to another neuron [27:06] or another bunch of neurons another [27:07] capsule and so that's now place coding. [27:10] It's sort of like cell phones with [27:12] little cell domains. You move around a [27:14] little bit and it stay in the same [27:15] domain and then you jump to another [27:18] domain and you move around a little bit [27:20] in there. As you go up these domains get [27:23] bigger. So when you're at a high level [27:24] you can move a long way without changing [27:26] which neurons are coding it but the [27:28] activities of the neurons changes to [27:30] tell you where it [27:31] is. [27:34] Okay. Now argument three. I think this [27:38] is the most powerful argument. Um if you [27:40] ask how current neural networks deal [27:43] with invariance [27:45] um what they do is they just train on a [27:48] lot of different [27:49] viewpoints. [27:51] Um and that's quite sensible. It just [27:54] requires a lot of training data and [27:55] going through a lot of training data [27:56] takes a lot of time. They don't have a [27:59] builtin bias to generalize in just the [28:02] right way across viewpoint. So I guess [28:04] this is another piece of anatism. [28:07] I'm getting soft in my old age and [28:09] allowing these tiny little you allow [28:11] those anatists to get even a foothold on [28:14] your model and pretty soon they'll be [28:15] infecting it. [28:17] Um, okay. [28:21] So, a much better approach would be to [28:24] say there's a linear [28:26] manifold which is what computer graphics [28:28] uses. If we get from pixels to the [28:31] coordinate [28:33] representation of pieces of objects or [28:36] objects that is it's this kind of thing [28:38] and here's its pose in a bunch of [28:41] numbers then everything's linear in [28:44] that and now you can do massive [28:47] extrapolation. You can train on little [28:49] faces like this here and then I can show [28:50] you a huge upside down face and you'll [28:52] correctly recognize it because on the [28:55] linear manifold you can extrapolate. Um, [28:58] that's what neural nets can't do. And if [28:59] they could do that, we could train them [29:01] on a whole lot less [29:04] data. [29:09] Okay. So, the idea is a long time ago [29:12] and for many years, people have been [29:13] saying um, you can think of vision as [29:16] inverse graphics, but they didn't say [29:19] it. They didn't mean it [29:21] literally. And I want to do it [29:23] literally. I want inside my system that [29:26] does computer vision, I want it to be [29:27] doing graphics backwards. When graphics [29:30] takes a hole and takes the pose of the [29:32] hole and multiplies it by a matrix to [29:34] get the pose of the part, I want my [29:36] computer vision system to be taking a [29:38] part and taking the pose of the part and [29:40] multiplying it by the inverse matrix to [29:42] get the pose of the whole. Um, that's [29:45] what I call inverse graphics. So, when I [29:48] say it's literally doing inverse [29:49] graphics, I mean literally [29:51] literally. Okay. [29:54] Um, and if you do that, then you can [29:57] represent the relationship between a [29:59] whole and a [30:00] part as just a matrix of weights. And [30:04] that matrix of weights is completely [30:06] viewpoint [30:07] invariant. So, however much I change the [30:10] pose, it's the same matrix of weights [30:12] that takes the pose of the hole and [30:13] gives me the pose of the part or goes [30:15] the other way, the pose of the part and [30:16] gives me the pose of the hole. So, I've [30:18] got complete independence of viewpoint [30:20] in the weights. That's where we want the [30:22] invariance. We don't want it in the [30:23] neural activities. We want it in the [30:24] weights. And we can get perfect [30:26] invariance in the weights this [30:28] way. Now, this is messed up by the fact [30:31] that if you have multiple instances of [30:33] the same entity, you've got to have [30:34] small domains because you have to code [30:36] multiple ones. And each of these [30:37] capsules can only deal with one thing at [30:39] a time. Because it's using simultaneity [30:42] to do the binding. It's saying within my [30:43] capsule, I got a bunch of neurons. The [30:46] activities of those neurons represent [30:47] different properties of the same thing. [30:49] And there better only be one thing. And [30:52] so when there's when things are this far [30:54] apart, you need fields this size. When [30:56] things are this far apart, you can use [30:57] fields this size. Um you can make the [31:00] fields overlap a bit and fudge it a bit, [31:02] but if you violate that, you're going to [31:04] your perceptions are going to go all [31:05] wrong, and that's called crowding. Um if [31:09] you put things too close together, [31:11] you're very bad at seeing them. You're [31:12] better at seeing smaller things further [31:14] apart. Um but I'm not going to push [31:17] that. [31:19] So given this view of the world, here's [31:22] how you ought to do shape recognition. [31:23] And this is computer vision from 1980s. [31:27] Um, you identify some familiar part like [31:31] a [31:32] nose. And you have [31:35] a logistic unit that says what's the [31:38] probability that it's there. So this [31:39] varies between one and zero. So it's [31:42] true this is invariant but just this [31:44] little bit. [31:47] Um, this guy in here has the pose [31:50] parameters of the nose. So, if we just [31:52] think geometrically, this will be say [31:54] six numbers in here that tell you the [31:55] pose of the nose relative to the [31:57] viewer. You then have a matrix that [32:00] operates on this pose and gives you a [32:03] prediction, this thing here, for the [32:05] pose of the [32:07] face. Here you've got an identified [32:10] mouth. You have the pose parameters. [32:14] you operate on them with this completely [32:16] viewpoint invariant matrix and you get [32:18] another prediction for the pose of the [32:20] face. If those predictions are roughly [32:22] the same, that's very good evidence that [32:24] there's a face. Those predictions will [32:26] only be roughly the same if the nose and [32:28] the mouth are related in the right way [32:29] to make a face. So really by checking [32:32] this identity, you're checking these are [32:34] related correctly to make a face. And if [32:36] they are, you say, "Hey, there's a face [32:38] here." And you send out the average of [32:40] what they predict, the average of this [32:41] and this. and you make this be one and [32:45] you carry on. So this is a way of doing [32:47] coincidence filtering to recognize [32:49] larger parts from smaller [32:50] parts. And this is very robust. There [32:53] can be other capsules that are making [32:54] different predictions for the face and [32:56] you will just ignore those if you have a [32:59] way of finding the little cluster that [33:00] agrees among a lot of [33:05] noise. And oldfashioned visual people [33:07] will say why isn't this just a huff [33:09] transform? That is a huff transform. [33:10] That's what huff transforms are all [33:12] about. [33:13] Um and it is just a huff transform um [33:16] but it's a modern huff transform in the [33:18] following sense. In the old days when [33:20] they did huff transforms they would have [33:21] fairly lowdimensional features trying to [33:23] predict the pose highdimensional poses. [33:26] And if the dimensionality of the feature [33:28] is lower than the dimensionality of the [33:29] object you have to predict a [33:32] subspace and so you have to have bins [33:34] and you have to put lots of votes in [33:35] lots of bins. [33:38] If you just want to make one vote, so [33:40] you don't need all these bins, then you [33:43] better have it that the features have as [33:45] many dimensions as the highdimensional [33:47] things. And to get features like that [33:50] you can reliably extract from pixels, [33:52] you need machine learning. You need good [33:53] machine learning. And the reason they [33:55] couldn't make huff transfers work before [33:56] was because they couldn't get the [33:57] low-level features to have enough [33:59] dimensions and be reliable so that they [34:02] could make point predictions. [34:05] Okay, that's my claim about why this is [34:07] different from old fashioned huff [34:08] transforms, but it is a huff [34:11] transform. And the last argument I want [34:14] to make is to do with this routting. So [34:17] the idea is what viewpoint does is it [34:20] changes where things show up in the [34:22] image. And so a nose is here now on [34:25] these pixels. And let's suppose let's [34:28] suppose it was a face. And let's suppose [34:29] you just have one great big face capsule [34:32] at the top. So you need to route these [34:34] pixels to the face capsule and in a [34:36] different image you need to root these [34:37] pixels to the face [34:39] capsule. And I want to show you there's [34:41] a nice way to do rooting. So the first [34:44] order of doing rooting is you make an [34:45] eye movement. You put what you're [34:47] interested in in the middle of your [34:48] retina. And that's a very powerful [34:50] rooting algorithm. But even if I I do [34:54] aistic so I force you to fixate in one [34:56] point you can still do rooting. Um, and [35:02] here's how you might do [35:03] it. Um, well, if you take comnets, I [35:06] already mentioned this. The way they do [35:07] rooting, they have a pooling unit and it [35:09] just attends to the most active guy and [35:11] the same all these levels up. So, [35:13] that'll route things but just based on [35:16] how noisy, how loud they are. [35:18] [Music] [35:19] Um, I want a rooting principle where you [35:22] root information to the capsule that [35:25] knows how to deal with it. [35:27] So the idea is that um you assume that [35:33] we're going to assume the world's [35:34] opaque. We're going to assume that um it [35:38] can be modeled by a pars tree. So we're [35:40] going to assume each part that we [35:41] discover has one parent. That's the [35:43] single parent constraint or possibly no [35:45] parents, but it doesn't have multiple [35:47] parents. So we want to find what it's a [35:49] part of. And there's just going to be [35:51] one of those. And so when you discover a [35:54] part, suppose I discover a circle. A [35:57] circle and suppose I have a limited [35:59] world that contains only cars and faces. [36:01] A circle might be a left eye of a face. [36:03] It might be a right eye of a face. It [36:05] might be the front wheel of a car or the [36:06] back wheel of a car. And I don't know [36:08] from the circle which it is. So what I'm [36:10] going to do is take the pose and I send [36:12] it to all those places. But I'm going to [36:15] send a kind of weight with that which [36:17] says my bet is a quarter that it's this [36:19] and a quarter that is that and a quarter [36:20] that is that and a quarter that that. So [36:22] you send it to these high level capsules [36:24] lots of weak bets. And then what the [36:27] high level capsule needs to do is look [36:29] at all these incoming weak bets and find [36:31] a bunch that agree. And when it finds a [36:33] bunch that agree [36:38] um then [36:41] um so the low-level capsule is sending [36:45] its pose to several higher level [36:47] capsules [36:49] um and initially it waits by the prior. [36:52] So there's going to be a prior which a [36:54] circle might be part of a face, it might [36:56] be part of a car is probably not part of [36:58] a fridge. So you don't send it to the [37:00] fridge or you send it with very low [37:01] weight to the fridge and with higher [37:03] weights to a face in a car. [37:05] [Music] [37:06] Um then in these [37:09] capsules you find the clusters. This is [37:12] a magic computation. I'm not telling you [37:14] how you do this computation but let's [37:15] suppose we can do this magic [37:17] computation. Like I say this is a marian [37:19] approach of what computation do you need [37:21] to do? And that's what we're going to [37:23] suppose a capsule does. And we reason [37:25] about this computation by the nature [37:26] from the nature of the task. And then [37:28] it's somebody else's problem of how it [37:30] does it. Well, it's my problem and I [37:32] can't make it work very well, but here [37:34] we go. Um, I'll show you how I do it [37:35] currently later. Um, so you sent it here [37:40] and it was this blue prediction and it [37:42] agreed with the cluster. You sent it [37:44] here and it was this prediction, it [37:46] didn't agree with the cluster. So now [37:47] what you want to do is you want to send [37:49] top- down feedback or you want to have [37:51] lateral [37:52] interactions that say, let's suppose it [37:54] was top down feedback. The top down [37:55] feedback from this capsule says, "Send [37:58] me more. I can account for this stuff. [38:00] Please send me your output." And this [38:02] guy says, "I can't account for this [38:04] stuff. Please don't send me your [38:05] output." And a few iterations of that [38:08] and you'll be sending all of the output [38:11] of this guy to here and sending none of [38:13] it to there. And now you will have a pse [38:15] tree. You will have established that [38:17] this belongs to that. There's obviously [38:20] competition. Um but yeah, I'm trying to [38:24] get the basic ideas over [38:25] here. So, [38:28] um that's my proposal. Either by lateral [38:30] interaction between these two guys [38:33] saying that their the weights on them [38:35] must add up to one and this guy fits in [38:37] nicely, so he'd like more weight and [38:38] this guy doesn't fit in nicely, so he'd [38:40] like less weight, so give this guy more [38:42] weight, and this guy less weight. Or you [38:44] could do that by top down. But notice [38:46] the top down isn't going to be like [38:47] loopy belief propagation. is not going [38:49] to say I'm going to use this to revise [38:51] my opinion about what's here. It's just [38:54] going to use this for rooting. It's [38:55] going to say okay this capsule can [38:58] account for me very nicely. I want the [39:01] information from here rooted to this [39:02] capsule. So it's a rooting by agreement [39:05] algorithm and it's using consistency to [39:09] do the rooting as opposed to just saying [39:12] um I'm going to ignore everybody but the [39:14] loudest guy that I can see which is what [39:16] Max Pooling is doing. [39:21] So now I'm going to show you two [39:23] computer programs. [39:25] Um unfortunately [39:28] um one of them was written by me. Um so [39:31] it's not a very good program. It runs [39:33] very slowly. Um but the idea is that if [39:37] we could make this basic module that can [39:40] check highdimensional agreement and [39:42] throw it [39:43] outliers, then we could have a deep [39:46] system. We've got a front-end problem [39:48] which is how do you get to the first [39:51] things that have poses? That's what I [39:53] call drendering the image. It's getting [39:54] from pixel intensities which are to do [39:56] with light to the poses which you do [39:58] with geometry. And then the higher [40:00] levels are all going to work the same [40:01] way. They're going to put pieces [40:02] together into larger and larger pieces. [40:05] Um I'm going to have a not very deep [40:07] system to begin with. Um, but what we [40:10] want to do is be able to use the power [40:12] of stochcastic gradient descent and lots [40:15] of data to learn a parts based hierarchy [40:18] with no hand engineering other than what [40:20] I said already, other than putting in [40:22] this idea that there's going to be [40:23] coincidence [40:24] filtering and rerooting based on [40:27] agreement. So here's my proof of [40:30] concept. I'm going to take emnest [40:33] digits and I'm going to have little [40:35] patches. [40:37] Um, it's going to be convolutional in [40:39] the following sense that in the first [40:42] stage, I'm going to take a patch. It's [40:44] going to have some weights connecting it [40:46] to some hidden [40:48] units. And that's the black patch. [40:52] There's going to be a blue patch. It's [40:54] going to have exactly the same weights [40:57] connecting it to 300 blue [41:00] units. Okay? So, the way I process a [41:02] patch is completely the same wherever [41:05] the patch is. that's [41:07] convolutional. These are going to be [41:09] nonlinear units. And what I'm going to [41:12] ask these units to do is to give me [41:16] um the poses of I actually used seven, [41:20] but I've shown three here. The poses of [41:23] three different types of capsule. So [41:26] capsules that detect different kinds of [41:28] entity. This might be vertical edges, [41:29] horizontal edges, corners, or something [41:31] like that. I'm not going to specify what [41:33] these entities should be. I'm going to [41:35] learn it all by gradient descent from [41:37] the right answers. But the way the first [41:40] stage of the system is going to work is [41:42] it's going to convert some pixel [41:45] intensities into some vectors of pose [41:48] parameters. Pose instantiation pose [41:51] parameters and these are going to be [41:53] different pose parameters. They're going [41:55] to be pose parameters of different kinds [41:56] of [41:57] things. In addition, it's going to say [41:59] whether it thinks this thing is there or [42:00] not. So that's going to be a logistic [42:02] unit and everything's going to be [42:04] learned, but it's all going to be shared [42:06] across the image by all these [42:10] patches. And these are going to be [42:12] linear [42:16] units. [42:18] Now those things are going to be what I [42:20] call the primary capsules. They're the [42:22] first level of capsules. They're the [42:23] first level of which you have explicit [42:25] pose coordinates. And I'm going to have [42:26] a very flat system to begin with. It's [42:28] just got two levels. It goes from that [42:30] first level to a second level. I've made [42:33] deeper systems, but this is the easiest [42:35] one to understand. [42:38] Um, in the second level, it has a bunch [42:40] of capsules. There should be the same [42:43] number of guys here as there are there. [42:45] I was just lazy with the PowerPoint. [42:47] Okay, sorry about that. I should have [42:48] put more things in here. [42:52] Um, so you take a type A primary capsule [42:55] that's looking at the black patch. [42:58] You've extracted some numbers from it. [43:00] You've learned to extract these numbers. [43:01] We'll say how in a minute and you apply [43:04] a coordinate transform to that vector of [43:07] pose parameters to make a prediction for [43:10] the pose of the zero if there's a zero [43:14] there. Um, also from whether or not this [43:18] is [43:19] active, you make a prediction for [43:21] whether the zero is present. That's what [43:23] I call a Picasso weight. So a picasso [43:26] weight says let's look at the type of [43:28] the capsule and predict whether this [43:30] thing is present solely based on the [43:32] type and ignoring all geometric [43:34] constraints. So if you see an eye there [43:36] might be a face and even if the eyes in [43:38] completely the wrong place if I put [43:39] enough eyes and noses around you'll see [43:40] a face um or you'll think there's sort [43:43] of a face there even if the geometry is [43:44] all wrong just because eyes go into [43:46] faces. So this is the reason it says [43:49] because it's an eye there might be a [43:50] face. [43:52] Um this is doing the real work. It's [43:55] saying I take the pose of this feature [43:57] I've identified this highdimensional [43:59] feature and I predict the pose of the [44:02] zero. And you're going to do the same [44:04] from all of these different primary [44:09] capsules. And you also have a weight on [44:11] this [44:12] prediction. Um so this is a weighted [44:19] bet. And now [44:22] um for the same patch, for the same [44:24] black patch in the original image, I'll [44:27] have a type B capsule that is detecting [44:29] a different kind of feature and it's [44:31] making predictions. It's predicting the [44:32] pose of the zero and the pose of the [44:34] one. [44:38] Um [44:41] and as well as having the predictions [44:43] that come from the black patch for the [44:45] different types of feature, I have [44:47] predictions that come from all the other [44:48] patches from the different types of [44:50] feature. So type A feature and type B [44:53] feature from the red patch making [44:54] predictions. So I get a whole lot of [44:56] predictions here. [44:58] And in particular, this weight matrix [45:01] that relates this pose of a type A [45:03] feature in the black patch to this [45:07] prediction is going to be the same as [45:10] the coordinate transform for the red [45:13] patch to make a [45:15] prediction. And that's not quite right. [45:17] So we have to fix that up because the [45:19] red patch is in a different place. And [45:22] so it ought to predict a different pose. [45:24] And so we have to fix up the [45:26] translational [45:28] bit. And we're going to do that by [45:31] saying when I make my prediction from [45:34] this primary capsule about the pose of [45:37] this [45:38] digit, I'm going to take the first two [45:41] coordinates. I'm going to say their [45:43] position and I'm going to add on the [45:46] location of this patch in the image to [45:48] the first two coordinates. I'm just [45:50] going to add it to whatever this [45:51] computes. And so that's the offset you [45:54] get in the position due to the offset of [45:56] the patch. Everything else about the [45:58] patch is the same. But as you offset the [46:01] patch, it should offset where you say [46:03] the thing [46:04] is. And that's implemented by this. It [46:06] also means these first two coordinates [46:07] are going to be interpretable. So that's [46:10] the whole system. And now what we want [46:12] to do is say if you get a whole bunch of [46:13] agreements here, it's a zero. If you get [46:16] a whole bunch of agreements here, it's a [46:17] one. [46:19] And so we want to back [46:21] propagate. Um we know what the class is. [46:23] Let's suppose it's a two. We want lots [46:25] of agreement here and not much agreement [46:27] there. So we need some measure of [46:29] agreement and we need to be able to back [46:30] propagate it. And we need to say if [46:33] you're not getting much agreement here, [46:34] please get more. And if you're getting [46:35] too much agreement here, please get [46:36] less. And then we can learn all of these [46:38] weights. We can learn the weights that [46:40] turn pixels into the poses of primary [46:42] capsules. We can learn these coordinate [46:44] transforms. Um that's fixed for now. So [46:48] we don't learn that um and we learn [46:50] various biases and things. So [46:56] first [47:00] yes. [47:04] No because if you want to get from pixel [47:07] intensities to the poses of features [47:09] that's highly [47:11] nonlinear. This is getting the [47:12] coordinates of a feature from pixel [47:15] intensities. [47:16] But if you want to get from the pose of [47:19] a feature to the pose of a larger thing [47:21] containing it, that's completely linear. [47:23] So you really really don't want any [47:25] nonlinearity in [47:28] there. It's a very good [47:33] question. Um so in this little system I [47:36] implemented on each of the high level [47:38] capsules there's seven feature types. [47:41] There's 181 patches in the image and it [47:43] gets that many predictions. it gets a [47:45] whole lot many of which are very weak. [47:48] So it's organized so that blank patches [47:50] make very weak predictions. [47:53] Um and coming with each prediction [47:56] there's a bet about so for any one [47:59] primary capsule it makes predictions and [48:01] the sum of the bets for this prediction [48:03] add up to one and [48:06] um this is treated as a fractional [48:09] observation for what comes next. Um the [48:13] higher level capsule then is looking for [48:15] agreement and so we need some way with [48:18] all these bets with these over a [48:19] thousand bets coming we need some way to [48:21] find a subset that agree nicely. [48:24] Um and so what we're going to do is [48:26] we're going to try and compute um for [48:28] this high level capsule whether it's got [48:30] good [48:33] agreement. And the way we're going to do [48:35] it is in this pose space for the high [48:38] level [48:39] capsule, we're going to fit a mixture of [48:42] a Gaussian and a uniform to the [48:44] predictions. Let's suppose it's a 60 [48:47] space and we're going to ask how good a [48:50] model is that of my predictions compared [48:53] with how good a model do I get by just [48:54] fitting a [48:55] uniform. And if when I fit a mixture of [48:58] a gausian in a uniform, I get a much [48:59] better prediction, a much better log [49:02] prop for these predictions. So we treat [49:04] it as a kind of unsupervised learning [49:05] problem. You've got all these [49:06] predictions. I want to build a model of [49:08] them. And I have two alternative models. [49:10] One is just uniform. The other it's [49:12] uniform and there's a Gaussian somewhere [49:14] and I'm going to allow the mean of that [49:16] Gaussian to float around. [49:19] So as this neural network is running, [49:21] the mean of its Gaussian is floating [49:22] around and the variance of exion is [49:24] floating around and the mixing [49:25] proportion of the Gaussian is floating [49:26] around. What isn't floating around? What [49:28] isn't floating around is these [49:29] coordinate [49:30] transforms and the weights on the bets [49:32] to begin with at least. [49:35] Um [49:37] so we're going to get a score for how [49:40] good a cluster this is. So let's suppose [49:43] for this cluster [49:45] um these red dots have very high [49:49] posterior probability under the [49:50] Gaussian. These ones are sort of can't [49:53] decide whether under the Gaussian or the [49:54] uniform and these ones are under the [49:56] uniform. And so we compute the sum of [50:00] all for all these data points each [50:01] weighted by the fraction of an [50:03] observation it is um of the log [50:06] probability of seeing that under a [50:09] mixture of a gas and a uniform. We also [50:11] compute the log probability of seeing [50:13] that just given a [50:15] uniform. And our score is the difference [50:17] in these log props. It's important to [50:20] use something like that because what you [50:21] want is that if I put in a stray vote [50:23] here, it doesn't affect the [50:25] score and it won't affect the score [50:28] much. I mean, depending on how you it [50:30] won't affect it at all. Um because [50:33] that'll be coded under the uniform in [50:34] both [50:35] cases. But if you find a cluster, this [50:38] score will be big. Again, if it's a [50:39] tight cluster, this score will be much [50:41] better. So now, let's look at what it [50:43] does. So that's the score. You then take [50:46] this score and you put it into a softmax [50:50] um to try and make a decision about what [50:52] class it is. And so so you treat this as [50:55] the loit that goes into softmax. You [50:57] scale it first because that's a sort of [50:58] hack. Um so there's a hack scale here. [51:03] Um if everything was probabilistically [51:05] correct, you should need a scale of one, [51:07] but you don't. [51:08] [Music] [51:10] um and you look at the decision you [51:12] make. If it's the wrong decision, you [51:14] say up the score for the right guy, down [51:16] the score for the wrong guy, and you [51:18] back propagate derivatives of that score [51:20] through the whole system. So there's an [51:22] inner loop that involves doing em and it [51:25] only takes about four [51:27] iterations. Um so you found the cluster. [51:30] Now I don't believe that's how the brain [51:31] does it. That's just how this model does [51:33] it. I believe the brain has to have some [51:34] other way of solving this computation. [51:36] Um, but for now that's the best I've [51:39] got. I do that computation and now I [51:42] just learn the whole [51:44] system. [51:45] And after I've learned it, if you showed [51:49] a digit like this and you look at the [51:52] scores that you get and the clusters [51:54] that you get for the various highle [51:57] capsules, I'm not showing you all of [51:59] them. There's other ones over here. [52:01] Um, I'm just showing you the first two [52:04] coordinates because I know those are [52:06] going to be to do with [52:07] position. Um, and so the zero, it gets [52:11] lots of very weak bets. Those are coming [52:13] from patches that don't have anything in [52:15] them. The size of a circle here is the [52:19] posterior for that thing being accounted [52:21] for by the Gaussian. This is at two [52:23] standard deviations, I think. Yeah. [52:27] Um, and so the votes that you get for a [52:30] zero look like that. The votes that you [52:32] get for a five look like this. They're [52:34] much more tightly clustered. And so your [52:37] Gaussian is a sharper Gaussian, which [52:39] means it can give higher probability to [52:41] things. So the this is the log of the [52:45] variance. And when the log of the [52:46] variance is negative, that means it's [52:48] sharp. This is a sharp calcium. That's a [52:51] much less sharp calcium. [52:53] Um, this is the score you get. This is [52:56] the difference between the probability [52:57] under the mixture and the probability [52:59] under the uniform. And this gets a high [53:01] score and the other guys get low scores. [53:03] So it says it's a five. Um, one more [53:07] example. So [53:11] um, I'm showing you more of the digits [53:13] here. This one he gets a high score [53:15] because it's a six and it's a tight [53:17] cluster. Um, this cluster is relatively [53:20] tight. So this has a log variance of [53:22] minus one. This has a log variance of [53:24] minus a half. So this is fairly tight. [53:26] It's a bit zero like um uh but it knows [53:31] it's a six. Now a system like this does [53:34] about as well as a convolutional neural [53:36] network on emnest. The difference is [53:39] that if you take a Mac air and you train [53:41] a convolutional neural network, you can [53:43] train it in sort of 10 minutes to half [53:45] an hour. Um and if you take my Mac Air [53:48] and you train this, it takes two days. [53:50] And that's because this inner loop of [53:52] doing all this em to get the scores and [53:54] stuff is very slow compared with just [53:56] matrix operations through the neural [53:58] net. Um but if we could find a fast way [54:00] of doing that that stuff then this would [54:02] be fine. But it actually works. It [54:04] manages to recognize digits by finding [54:06] the agreement in the pose [54:08] predictions. That's one demo. There's [54:11] another demo that's a bit more [54:13] convincing. [54:14] Um so there's all sorts of things that [54:17] need to be done to make this better. We [54:18] need multiple simultaneous digits. I can [54:20] get that working. That's fine. In this [54:22] example I showed you, we didn't [54:24] redistribute the votes. We just did [54:25] bottomup votes and we didn't rewe them [54:27] based on rooting by agreement. So I need [54:29] to currently someone is adding rooting [54:31] by agreement to this and it will work [54:32] better. Um we didn't have deeper [54:36] hierarchies. We didn't have real images. [54:38] They just emnested. There's lots to be [54:39] done. Um, one thing we have done is [54:43] tried to get the primary capsules not by [54:46] back propagating the errors you make in [54:48] digit classification but by doing [54:50] unsupervised learning. You like to do [54:52] unsupervised learning to get from pixels [54:55] to entities that have poses. So that [54:59] first stage if you can do it with [55:00] unsupervised learning it's going to be [55:02] much less you're going to need much less [55:05] labelled data for example. [55:09] So, [55:11] [Music] [55:14] um, we want to drender the image and get [55:17] it to entities that have poses, [55:19] um, [55:20] unsupervised. And here's we have several [55:23] ways of doing that. And here is so [55:26] here's what we'd really like to do. And [55:28] I've to simplify the PowerPoint, I've [55:30] just assumed we're getting [55:31] two-dimensional pose, which is just [55:32] position. But really we were going to do [55:34] this for full pose which is a full fine. [55:37] Um so you have an image and you would [55:41] like to go through some nonlinear units [55:44] and get out in these different capsules. [55:47] You'd like to get um the position of an [55:50] entity and in this case the intensity of [55:53] the entity. So I've changed from a [55:55] probability to an intensity which is a [55:56] slight fudge which I need for what I do [55:57] later. [55:59] Um the work I'm describing now is work [56:02] that was done in Tamman Tieleman's [56:04] thesis [56:05] um which recently came out. [56:09] Uh so that's what we'd like to achieve. [56:12] We'd like these to learn to be entities [56:14] and they decide what kind of entity they [56:16] should be. We'd like them to give us [56:17] their poses and how how much they're [56:21] present. [56:26] And the way we're going to do it is by [56:28] having a simple graphics [56:30] model. Oh dear. You see, I told you if [56:33] you let the inists in, then they'll take [56:35] over. So, we're going to have an innate [56:37] graphics model now, but that's just for [56:40] this system. [56:41] Um, and we're going to try and [56:45] learn to reconstruct the image by first [56:49] extracting the capsules and then [56:51] reconstructing the image from the [56:52] capsules but using built-in graphics. [56:56] So, it works like [56:58] this. You saw this bit already. This is [57:02] going to learn all these learn and all [57:03] these learn. It's going to learn to [57:05] extract some kind of entity and the pose [57:06] of the entity and the intensity of the [57:09] entity. [57:11] Um, and it knows which entity it is [57:13] because it's these [57:15] neurons. And then these instantiation [57:17] parameters are going to be fed to the [57:19] graphic system. And the graphic system [57:21] from these parameters is going to have [57:23] to reconstruct the image. And the [57:25] graphic system is going to learn a [57:27] little template for each entity. But [57:29] what the graphic system is going to be [57:31] able to do unlike a normal neural net is [57:34] it's going to be able to translate this [57:35] template according to this X and Y. and [57:38] it's able to scale the template, scale [57:40] the intensities here according to the [57:42] eye and then it's just going to add it [57:43] to the [57:45] image. So you've wired in some graphics [57:48] and you're learning to invert graphics [57:52] here, but it's going to also learn what [57:54] these entities should be. You just what [57:57] you've wired in when you wired in the [57:58] graphics is just the ability to take [58:00] some entity that you've learned and [58:01] translate it and more generally do a [58:05] full aine. Okay. So [58:09] now if you ask what templates does it [58:12] learn in modeling digits, these are the [58:15] templates it learns. And you can see [58:17] they're little pieces of stroke with [58:18] feathered ends. So you can add them [58:19] together and they make a nice [58:21] digit. [58:22] Um this is now the case when it [58:27] can yes this thing can apply full aines [58:31] to these templates. And so let's see [58:34] what how it now decomposes [58:38] digits. So if you show it this digit, [58:41] this is its [58:42] reconstruction. And for each of its 10 [58:45] capsules, [58:47] um this is the contribution from the [58:49] capsule. So if you look at this capsule, [58:52] it's making different [58:53] contributions. Um but you'll see it's [58:56] sort of found the corresponding pieces. [58:58] For this six, that part of the loop is [59:00] coded here. For this two, it's coding [59:03] this piece of the curve here. Um, for [59:06] this eight, it's coding this bottom [59:09] piece of the curve here. And so these [59:12] contributions when you add them up do a [59:13] jolly good job of reconstructing the [59:14] digits. It works very well for [59:17] reconstruction. Okay. So, so far what [59:20] we've done is we've learned bottom up to [59:25] what the pieces should be and we've [59:27] learned how to look at an image and [59:29] extract the poses of those pieces. [59:32] So now having done that [59:34] unsupervised, let's now try and do some [59:36] supervised learning on top of this. And [59:39] remember after you've got to poses, [59:41] everything's linear. So life's going to [59:42] be [59:43] easy. So here's the idea. If we just had [59:47] two postparameters, you take each of the [59:49] capsules and you take its pose [59:51] parameters you extracted from the image [59:53] and you just concatenate them all. You [59:55] get a big [59:56] vector. And of course, if it's a [59:59] particular shape, there's lots of mutual [60:02] information between all these guys. [60:03] They're all related in the right way. [60:05] They can all change as you change your [60:06] view on the shape, but they have [60:08] relationships between them. So, if you [60:10] do factor analysis on this, factor [60:13] analysis will find underlying [60:15] factors. So, six for the aine and four [60:17] for the defamation. underlying [60:20] factors that model this very [60:23] nicely because as you change your [60:25] viewpoint all of this stuff is changing [60:28] and by changing the aphine represented [60:30] by the factors you can model that [60:33] perfectly. So the factor loadings these [60:36] three factor loadings here will actually [60:38] be modeling the relationship between the [60:40] hole and the part and by fitting a [60:42] factor [60:43] analyzer if I just had one digit I could [60:47] do that. Now, I've actually got a [60:48] mixture of digits, so I'm going to fit a [60:50] mixture of factor analyzers. It would be [60:52] nice if I could just fit 10 of them, but [60:54] that doesn't work so well. So, we fitted [60:55] a mixture of 25 factor [60:58] analyzers. [61:01] Um, actually, if you just fit a mixture [61:03] of 10 factor analyzers, you can fit [61:05] factor anal analyzers, a mixture of [61:07] factor analyzers, you can fit it to the [61:09] raw pixels. If you fit to the raw [61:11] pixels, these are the means of the [61:14] factor analyzers. [61:16] If you fit the factor analyzers to the [61:19] capsules and then take what they tell [61:21] the capsules to do and reconstruct from [61:23] what they tell the capsules to do, these [61:26] are the means of the factor analyzers. [61:28] This is all [61:29] unsupervised, so you can tell it's done [61:31] a pretty good job. Fours and nines is a [61:33] bit confused about, but it's really [61:36] nailed the 10-way. There's 10 classes [61:38] here. [61:40] Um and in fact if you look at this [61:42] factor analyzer and you look at the 10 [61:45] factors and you look at what changing a [61:48] factor does if you take a factor take [61:50] the mean value and subtract two standard [61:53] deviations you'll get that and if you [61:56] add two standard deviations you'll get [61:57] that. So this is a factor that's dealing [61:59] with itallicness. This is a factor for [62:01] loopiness and so on. Now in this data we [62:04] didn't actually do much transformation. [62:06] So most of the factors are right [62:09] deformation if you use 25 [62:12] factors. Oh, forget that [62:15] one. If you want to recognize D with [62:18] very few labeled examples. Now what you [62:20] can do is you can do standard back prop. [62:23] Standard back prop on MN list with [62:25] 60,000 examples gets about 1.6% error. [62:28] With all sorts of tricks, you can get [62:29] that down to about 1%. [62:32] Um, if you do a very clever thing called [62:34] scattering transforms developed by Bruno [62:37] Mallet, they were very pleased because [62:39] they managed to divide that by a factor [62:41] of about 30. [62:44] Um, actually with 30,000 you get 1.7. So [62:48] say 15. Um, with 2,000 label examples [62:51] they can do just as well. So they're [62:53] getting much more statistical [62:54] efficiency. They can get this this error [62:56] rate with 2,000 examples. If you do [62:59] unsupervised learning of the kind I just [63:01] showed you and you have 25 clusters that [63:05] is you do a mixture of 25 factor [63:07] analyzers on these concatenated [63:10] instantiation parameters of the primary [63:12] capsules after you've done that you just [63:15] take each factor analyzer and you [63:18] say for the example that I'm most [63:20] confident should go to that factor [63:22] analyzer just tell me what its name is [63:25] tell me what its class is and so you [63:27] have to ask 25 questions [63:29] And if you ask 25 questions, [63:32] um, you can get 1.75% errors on [63:36] average. So now that's much closer to [63:39] what humans can [63:41] do. You show them a lot of digits and [63:45] they ask you a few questions and then [63:46] they know how to classify these [63:49] digits. And so this is an example where [63:53] because we grabbed the linear manifold, [63:56] we made the unsupervised learning work, [63:57] right? So it found the natural classes [64:00] and having done that you can learn with [64:02] very few labels. [64:09] Um yeah I just said all [64:12] that and that's the end. Okay I'm done. [64:36] unsupervised then [64:39] okay so the question was how does that [64:42] compare there's people out there how [64:43] does that compare with doing um [64:45] unsupervised followed by supervised like [64:47] unsupervised pre-training presumably [64:49] followed by supervised is much much [64:52] better so if you do standard [64:54] unsupervised followed by supervised If [64:56] you say stack up autoenccoders, um [64:59] you'll do about the same as the Bruner [65:01] and Mallet. That's you won't do much [65:04] better than that. Maybe you can get down [65:05] to a thousand, but this is like almost [65:08] two orders of magnitude better than that [65:10] because you grab the linear manifold. [65:17] Yeah, maybe another version of that [65:19] question is how would it compare with [65:20] doing some kind of generic model like [65:23] mixture factor analyzers? [65:27] I remember back in the day. Yeah. So I [65:28] showed [65:29] you like maybe 10 digit [65:36] it wasn't nearly as good as this if you [65:38] so you can look at the purity of the [65:39] mixture components. No it's it's clearly [65:41] much better but you could compare it to [65:44] that. [65:45] Yes. Um I imagine Well that just won't [65:49] get you down to 1.7% errors. [65:53] But if you were able to label would have [65:55] been a pretty good model, right? I think [65:58] to get to 1.7% errors, you probably you [66:02] probably need to fix something like a [66:03] thousand components. I don't I I mean I [66:05] have to check up. I doubt you'd be able [66:07] to do it with 100 components in there. [66:08] I've tried those models and tried lots [66:10] of components and with 100 components, [66:12] I'd be very surprised if you could um [66:15] get down to 1.7%. [66:22] object seems like very young children [66:25] something like proto objects that aren't [66:27] fully fleshed out detailed shape [66:30] representation but they're maybe kind of [66:31] fuzzy plots that [66:33] obey I was wondering if you thought that [66:36] that kind of representation would be [66:38] useful for learning this mapping from [66:39] pixel [66:43] space that [66:47] are you trying to get at the idea that [66:48] actually their low resolution is helpful [66:50] by suppressing details and they just see [66:53] the sort of essence of it at low [66:54] resolution and they can learn more [66:55] easily. Yeah, it's not just low [66:57] resolution. [67:00] Um, possibly. I don't want to comment. I [67:03] haven't thought about that issue and I'd [67:06] want to think about it before I've said [67:07] anything. I I'm not disagreeing. It [67:10] might be helpful, but I don't know. [67:14] So, you you [67:17] critical aspect of this concept is that [67:20] the manifold is some kind of [67:23] hardwired into the system. So why humans [67:26] are so [67:31] bad? Hang on. When you say why are [67:34] humans so bad? Oh, the question was a [67:37] critical aspect of this is that the um [67:40] relational thing is hardwired. The fact [67:42] that it's a linear transformation. So, [67:43] why are humans so bad at doing this [67:45] rotational transformation of objects? [67:48] Um, so there's two things you might mean [67:51] by why are they so bad? [67:54] Um, humans are actually pretty good at [67:56] recognizing objects in funny [67:58] orientations. You're not good at doing [68:00] very fine discriminations for that. But [68:01] if I show you an upside down face, you [68:03] know right away it's an upside down [68:04] face. You don't know whose it is, but [68:06] you know right away it's an upside down [68:07] face. So, you do that kind of instantly. [68:10] um there's there's probably a 10 [68:12] millisecond penalty for that, but the [68:15] mental rotation is going to be like over [68:17] a 100 milliseconds to get it upright. [68:19] So, it's a different order of magnitude. [68:21] So the things where you really have to [68:24] do the slow mental rotation are to make [68:26] handiness judgments or to do things like [68:29] will this grand piano fit through this [68:31] studio door where you have to do [68:33] physical [68:34] um but recognition you're actually [68:36] pretty good at dealing with different [68:38] orientations. That capital letter R I [68:39] showed you you instantly knew it was a [68:41] capital letter [68:42] R. And when I say instantly within about [68:45] 250 millconds and then it took you 100 [68:47] millconds to rotate. If it was a 3D [68:49] thing, you'd rotate it even slower. [68:51] Dimensional. That was two dimensional. [68:53] In 3D, you do mental rotation too, but [68:56] it's quite a lot slower. [69:00] Yeah. So, you mentioned that the [69:02] existing algorithm can run in 10 minutes [69:04] and yours require two days for the [69:06] learning. Yes, for the learning. Um, [69:08] what kind of heristics or [69:10] computational approximations do you make [69:12] to bring that time down? [69:15] Um, oh, I'm sure you can bring it down [69:17] by a whole lot. Um, you could start off [69:21] by not programming it in mat [69:24] lab or even better, you could start off [69:26] by not having me programming it. That [69:29] would give you about an order of [69:30] magnitude. Um, but the real essence of [69:33] it is this thing that what's different [69:35] from neural normal neural nets here is [69:38] that the core of it is something that [69:39] looks for agreement between activity [69:41] vectors. It's not looking for agreement [69:43] between a weight vector and activity [69:45] vector. That's what a filter does. What [69:47] we want is agreement between activity [69:49] vectors which is going after coariance [69:51] structure. And that computation to find [69:54] these highdimensional agreements among a [69:56] whole bunch of random [69:58] stuff. There must be ways to make that [70:01] efficient. And I have some ideas about [70:02] how to make it efficient. And I'll talk [70:03] about those if they work. Um, but that's [70:06] what's slowing it up. [70:13] The capsules here are [70:16] very kind of you have ideas about what [70:20] the features of capsules might be for [70:22] other problems. Some so let's take [70:27] speech. I have a long-suffering graduate [70:30] student who I got to do a thesis. I'm [70:33] trying to apply these ideas to speech. [70:36] um he was a very good student and these [70:40] ideas are quite difficult to get working [70:41] on speech. But for [70:43] example, you can apply you can say [70:46] there's changes in frequency and changes [70:48] in onset time and changes in amplitude [70:52] and you can try and get entities that [70:54] tell you this complicated acoustic event [70:56] is happening and here's when it's [70:58] happening and here's what frequency it's [71:00] happening at and here's the amplitude [71:02] it's happening at. and he managed in the [71:05] end to get things that worked as well as [71:07] standard neural networks using capsules [71:10] ideas in speech but they're more natural [71:12] for [71:12] vision. I'll leave it at [71:17] that question. So let me remind you [71:19] there's a panel in this room at 6 on the [71:22] path to intelligence and in between you [71:24] have all right to reception outside. [71:25] Thank you.