Stanford AA274A Principles of Robotic Autonomy | Autumn 2019 | Modern Computer Vision

斯坦福 AA274A 机器人自主原理 | 2019年秋季 | 现代计算机视觉 本讲座追溯了机器人计算机视觉在2012年这一关键转折点前后的演进历程。讲座回顾了经典的手工设计范式，包括结合SVM分类器的HOG描述符、K-means与概率图模型分割，以及诸如ORB-SLAM和DSO等几何视觉SLAM系统，并强调了它们繁重的人工工程负担。随后，课程探讨了AlexNet与卷积神经网络的突破如何通过多层级学习特征取代人工启发式方法来革新感知能力，展示了其在现代自动驾驶汽车的目标检测、轨迹预测和多传感器融合中的端到端应用，同时也强调了可解释性方面依然存在的挑战。 AlexNet在2012年ImageNet竞赛中的突破标志着机器人感知领域发生了根本性的范式转变，将分类错误率从手工特征约26%的瓶颈期大幅降至16%，并最终借助深度ResNet降至5%的人类基准以下。 (47:04) 经典的机器人视觉流程（如HOG+SVM检测器、ORB-SLAM和DSO）依赖于人工设计的归纳偏置，并舍弃了高达90%的原始图像数据，以在配置一般的CPU上维持30至60+ Hz的实时性能。 (40:15) 现代端到端卷积架构在很大程度上取代了复杂的概率图模型和聚类启发式方法，使自主系统能够同时跨相机和激光雷达执行多任务检测、跟踪、深度估计以及传感器融合。 (65:23) 尽管在标准化图像识别数据集上达到了超越人类的基准准确率，但当代的深度视觉系统仍面临关于黑盒可解释性、形式化验证以及在分布外输入上出现过度自信失效等关键未决挑战。 (72:11)

In a guest lecture for Stanford's AA274A, course assistant Boris details how the 2012 deep learning breakthrough permanently transformed robotic perception from manual feature engineering into end-to-end learned representations, while classical geometry continues to anchor spatial localization. The Pre-2012 Handcrafted Vision Paradigm (8:02) Before 2012, robotic computer vision depended heavily on manual feature engineering and strong inductive biases. Researchers designed specialized algorithms such as SIFT, SURF, and ORB to detect corners, edges, and invariant keypoints, coupling them with classifiers like Support Vector Machines (SVMs) or k-nearest neighbors. For object detection, standard pipelines used sliding windows combined with Histogram of Oriented Gradients (HOG) descriptors to extract localized pixel intensity gradients. Similarly, semantic scene segmentation relied either on unsupervised clustering—such as mapping pixels into a 5D feature space of spatial coordinates and color channels via k-means—or on probabilistic graphical models that optimized noisy boundary classifications against neighboring grid pixels. Geometric Foundations and Engineering in SLAM (29:59) Despite advances in classification, spatial reasoning remains grounded in classical multi-view geometry. Simultaneous Localization and Mapping (SLAM) systems rely on structure-from-motion, epipolar geometry, and loop-closure optimizations to track poses and map environments in real time. Feature-based pipelines like ORB-SLAM track repeatable keypoints across frames to achieve high localization accuracy, though filtering out around 90% of image pixels leaves the resulting point-cloud map sparse. Conversely, direct methods such as Direct Sparse Odometry (DSO) track photometric errors across image gradients and edges to build denser maps, though they introduce greater sensitivity to perceptual ambiguity. The 2012 Inflection and Learned CNN Hierarchies (41:50) The modern perception paradigm began at the 2012 ImageNet Large Scale Visual Recognition Challenge when SuperVision (AlexNet) achieved an error rate of 16%, dropping nearly 50% relative to the ~26% error rates of competing handcrafted ensembles. Rather than relying on human-designed filters, convolutional neural networks (CNNs) optimize filter weights end-to-end through backpropagation and gradient descent. This architecture forms an automatic hierarchy of visual abstraction: early layers learn low-level edge detectors and color-opponent gradients, intermediate layers capture corners, circles, and textures, and deep layers assemble complex semantic structures such as wheels or traffic signs. Consolidation of Robotic Tasks and Multimodal Fusion (65:23) The deep learning shift restructured perception pipelines by replacing disparate stages with unified networks. In modern object detection, a single network jointly optimizes classification scores alongside bounding-box coordinate regressions, while autonomous vehicle pipelines extend this to tracking and trajectory forecasting. For semantic segmentation, end-to-end pixel-wise labeling has largely superseded complex graphical energy minimization. In SLAM front ends, convolutional depth estimation has demonstrated roughly 66% pixel-wise accuracy in indoor benchmarks compared to approximately 12% for traditional feature matching. Furthermore, CNN architectures streamline multimodal sensor fusion—such as combining Waymo's mid- and short-range lidars, five surround cameras, and pose sensors—by integrating them into unified channel vectors. While deeper architectures have achieved superhuman classification on fine-grained benchmarks like ImageNet, neural perception networks function largely as black boxes that lack calibrated uncertainty when presented with nonsense or out-of-distribution inputs. The lecturer notes that model interpretability and formal verification remain critical open research challenges for autonomous robotics.

斯坦福大学AA274A课程指出，机器人视觉在2012年经历了从手工特征工程向深度学习的根本范式转变，卷积神经网络通过端到端优化重塑了目标检测、语义分割与SLAM等核心感知任务。 2012年前经典视觉与手工特征工程 (8:02) 在2012年之前，计算机视觉高度依赖研究人员注入先验归纳偏置与手工特征工程。经典的算法流程聚焦于边缘、角点和斑点检测，并设计了如SIFT、SURF与ORB等局部特征描述子。在目标检测任务中，标准方法通常采用滑动窗口配合方向梯度直方图（HOG）提取局部梯度特征，再交由线性支持向量机（SVM）进行分类与阈值筛选。这类经典方法极其依赖精巧的人工启发式设计，尽管工程复杂度高，但其算力开销相对可控，在当时的双核移动处理器或低配置硬件上便能以数十毫秒的延迟实现30至60赫兹的实时运行。 经典语义分割与SLAM的几何工程 (17:38) 面对复杂的场景理解与建图需求，经典方法探索了聚类与图优化路径。语义分割早期采用K-Means在融合空间坐标与颜色通道的5D特征空间（X, Y, R, G, B）中聚类，后续发展为以概率图模型（PGM）为主导，利用能量最小化优化滑动窗口分类器输出的高噪声边界。在同时定位与建图（SLAM）领域，ORB-SLAM等基于特征点的算法通过保留约10%的高稳定性特征点计算相机位姿变换，实现了极强的位姿追踪鲁棒性，但生成的地图极度稀疏；直接稀疏测距法（DSO）等直接法则利用图像灰度梯度生成稠密结构，但易受感知歧义干扰，且两者均需依赖回环检测消除长周期累积漂移。这类几何SLAM的实现高度依赖底层C/C++工程调优，代码结构极为庞大。 ImageNet突破与卷积神经网络演进 (47:04) 2012年ImageNet挑战赛（ILSVRC）成为视觉计算的分水岭。此前由牛津大学、阿姆斯特丹大学等传统强队构建的手工特征工程集成方案，其分类错误率长期徘徊在26%的瓶颈期；而多伦多大学团队提交的AlexNet（SuperVision）凭借8层卷积神经网络将错误率骤降至16%，带来接近50%的相对性能提升。此后网络架构迅速加深，从19层、22层演进至152层的ResNet，错误率跌破5%，超越了约5%的人类基准识别误差水平。 端到端特征层级与多任务现代感知架构 (50:37) 卷积神经网络摒弃了人工指定滤波器系数的做法，通过反向传播让网络从数据中自发学习分层特征：浅层网络学习定向边缘算子与互补色彩反差，中层组合为角点、圆环与网格纹理，深层网络则表征出车轮、交通标志等复杂语义构件。在现代机器人感知中，单一卷积网络通过联合构建分类损失与边界框回归损失，实现了目标检测、多目标跟踪及未来轨迹预测的端到端统一处理；在语义分割中，全卷积架构直接输出像素级稠密标签，大幅取代了离散概率图模型；在SLAM前端，深度卷积深度估计将室内环境下的像素级深度准确率由传统特征匹配的约12%提升至66%，后端则继续沿用刚体位姿变换等经典几何优化算法。 异构传感器融合优势与模型解释性挑战 (69:50) 现代深度学习极大地简化了自主移动平台的异构传感器融合工程。例如在配置有中近程激光雷达、5路环视相机、GPS、IMU及轮速计的Waymo自动驾驶系统中，传统手工对齐极其繁琐，而现代深度架构可直接将激光雷达数据或位姿状态转换为额外的数据通道或向量拼入神经网络，配合损失函数实现联合端到端特征提取。这一范式在维持高推断效率的同时显著拔高了感知上限，但其核心代价在于黑箱特性引发的模型可解释性缺失与非分布内异常输入的静默失效，这在严苛的安全关键型场景中仍是尚未彻底解决的研究难点。 现代计算机视觉与多模态学习已成为机器人自主系统的核心基石，但在安全关键型应用中，深度神经网络的黑箱特性、缺乏不确定性自知机制以及形式化验证困难，依然是学术界与工业界亟待攻克的开放性课题。

[0:05] Prof. Marco Pavone: I would say that as a roboticist a good fraction of your time is spent debugging connectivity issues.

[0:05] Let me start with the video discussion [0:07] about the feedback that we got from you. [0:09] Again, for us this feedback is very [0:11] important. This is a a fairly complex [0:14] class to teach because of the complexity [0:17] of the mathematics, the complexity of [0:19] the software. Uh, it's not just a [0:21] script, it's an entire infrastructure [0:23] for robotic software engineering that [0:25] we're teaching, and of course the [0:26] hardware. [0:27] So, every time we try to improve the [0:29] class. [0:30] So, we identified a few items that I [0:32] think are important to discuss. The [0:35] first item is about the sections. [0:39] So, the way that the question was [0:42] phrased maybe was a little bit [0:45] um [0:47] confusing. [0:48] In the sense that [0:50] the role and the goal of the section is [0:52] not to [0:54] allow you to understand better the [0:56] material that is explained in lectures. [0:59] Instead, they're meant to be a [1:01] complement to the lectures, and in [1:02] particular to learn those skills in [1:04] robotic software engineering that are [1:07] very important, but are almost [1:09] impossible to teach in lecture. You only [1:11] learn those through hands-on activities. [1:14] In previous years, we were delegating [1:16] all of these to the [1:19] precepts. [1:20] But, we felt that that was not optimal [1:23] as well. So, this year we found as a [1:25] compromise to have a sections whereby we [1:28] have a hands-on activities with a fairly [1:30] light grading under the guidance of SCA. [1:35] So, the way that the question was [1:36] phrased was possibly a little bit [1:37] confusing, but again, I want to [1:39] emphasize the goal of the sections is to [1:42] complement the lectures by focusing on [1:44] the software engineering skills that [1:45] would be very difficult to teach, [1:47] you know, through slides. [1:50] So, some of you mentioned that you had [1:53] some hiccups during the sections, and [1:55] particularly in terms of uh connectivity [1:58] and the software installation. [2:00] I would say that as a roboticist a good [2:03] fraction of your time is spent debugging [2:05] connectivity issues. Would you agree? I [2:07] mean, it's inevitable. So, if you want [2:09] to be a robotics, get used to it. [2:11] Uh but having said that, uh these are [2:13] problems that we are fixing. So, [2:15] hopefully for the next sections, that [2:17] shouldn't be uh a problem. Also, in [2:19] terms of the software installation, we [2:21] have identified a couple of ways that uh [2:24] we could uh address them. Not this year, [2:26] unfortunately, it's too late, but next [2:27] year. So, at least uh your feedback is [2:30] going to be useful for the next [2:31] generation of uh students. [2:34] Now, another aspect was about uh not [2:37] having received feedback before uh [2:41] about your homework before the uh [2:44] midterm. That actually was a little bit [2:46] unfortunate. Unfortunately, grading this [2:48] homework is uh complicated, and we are [2:50] setting up this year for the first time [2:53] the auto-grading procedure, so that is [2:55] uh takes a little bit [2:56] uh quite a bit of time. But anyways, so [2:59] I believe that by the end of today or [3:01] anyway by Sunday, we're going to release [3:02] the grades for the midterm. [3:04] And in general, people did pretty well. [3:06] I was I was discussing today about the [3:09] shape of the grade distribution. I was [3:11] very happy to see that was a Gaussian [3:13] with a fairly narrow variance. [3:16] That is great because since this class [3:18] has people from several different [3:19] backgrounds, electrical engineering, [3:21] computer engineering, aerospace [3:22] engineering, and both undergrads and [3:24] grads, uh I was afraid of seeing like a [3:27] multimodal distribution. That would have [3:29] been bad. So, actually, it's a unimodal [3:31] distribution. That's great. So, good [3:33] job, Boris. Good job. Good job, myself. [3:36] >> [snorts] [3:36] >> Uh [3:37] all right. [3:38] But having said that, uh I know we [3:40] apologize for it is that setting up the [3:43] auto-grading procedure is lengthy. So, [3:44] unfortunately, for this year, there is a [3:46] there is a mismatch between uh uh [3:49] when you receive the grading for your [3:51] homework and when you do actually do the [3:52] midterm. [3:54] Some other questions were about why we [3:57] don't post [3:58] solutions. That has two ramifications. [4:01] One, of course, having access to the [4:02] solutions [4:04] would help you to prepare better for the [4:06] midterms. [4:07] And two, since the philosophy of this [4:09] class is to incrementally build the [4:10] autonomy stack, which means that the [4:12] what you do in homework two [4:14] relied on what you did in homework one, [4:16] what you are going to do in three and [4:18] four, and for the final project is going [4:20] to rely on everything else. Of course, [4:22] having a bulletproof code that the CAs [4:24] developed would be very helpful. [4:27] The reason why we don't do it is that [4:28] the homework that you are taking, as you [4:30] might have appreciated, took hundreds of [4:33] hours to develop. [4:34] So, if people start putting the code and [4:37] the solution out in the open, it would [4:39] be disastrous. It would mean [4:41] several additional months of full-time [4:42] work in order to redo brand new [4:45] homework. [4:46] So, as I mentioned at the beginning of [4:48] the class, the strategy is that [4:50] after the homework is due [4:52] feel free to come to office hours or [4:55] even discuss it in section [4:57] with your solution, and the CAs will [4:59] help you in fine-tuning your solution to [5:01] make sure that you have a code base that [5:02] is working. [5:07] Unfortunately, there is no perfect [5:08] solution in the world that what we feel [5:10] that at this time is the best solution. [5:13] Um [5:14] what else are we talking about? Oh, [5:16] homework. So, some people complained [5:19] that the homework are too long. [5:21] Uh [5:22] a little bit homework one, homework two [5:23] was perceived as of the right length. [5:26] Homework three is perceived as a too [5:28] long, especially given the time [5:31] allocated to it. And we agree with you. [5:33] Actually, it's probably too long for the [5:35] time allocated to it. So, we discuss it [5:38] today every every week we have [5:40] a meeting of me with all the CAs. [5:43] Um [snorts] [5:44] so, we're going to extend So deadline [5:45] for that homework to November the 5th. [5:48] And we agreed that that given the length [5:51] of the homework having a [5:53] one additional week is more compatible [5:55] with it. [5:56] So today [5:59] we will be posting of course an [6:00] announcement on Canvas and then a post [6:03] on Piazza to say that uh [6:05] the deadline is extended to November the [6:06] 5th. It's fine. Uh it also lines quite [6:09] well with the schedule of the [6:12] uh lectures. [6:16] So [6:19] any other questions or feedback? [6:22] Your feedback is here. Actually, I'm [6:24] taking uh [6:26] I'm taking really taking into account. [6:27] So [6:29] there were some feedback from the SCPD [6:31] students of not having enough of this [6:33] hours. Actually, we increased the number [6:35] of this hours available to [6:37] uh to [6:38] SCPD students. [6:40] Um but something else anything else? [6:47] Good? All right. [6:48] So today the lecture will be on model [6:51] robotic perception. You can view this [6:53] this lecture will be given by Boris, uh [6:55] our lead CA. [6:56] And you can view this lecture as in a [6:59] way the a wrap-up of everything we've [7:02] done so far in terms of robotic [7:04] perception that is more the classical [7:06] view of robotic perception. [7:09] Uh [7:10] what are its shortcomings? And what [7:12] happens in a very at a very specific [7:14] time, 2012, that really changed the way [7:17] of how robotic perception is done and [7:21] then uh in turn how other functions in [7:24] the autonomous stack are engineered. [7:27] And it change has been the advent of uh [7:30] uh more machine learning techniques and [7:31] neural network techniques to [7:34] uh robot perception. [7:35] We're not going to go into uh too many [7:38] details. This is more like a sort of [7:40] overview with some in-depth [7:43] insights, [7:44] uh but of course the new wave of [7:46] perception will then be covered in [7:49] 274B, which is the class that I teach in [7:51] the winter. [7:53] All right. So, did I miss anything about [7:55] the feedback? [7:56] >> No. [7:57] >> All right, cool. [8:02] >> So, while Marco [8:03] uh thinks [8:04] and dreams [8:06] Um yes, so today's lecture is going to [8:08] be on modern robotic perception. [8:10] Specifically, [8:12] we are going to be talking about [8:15] modern advancements in computer vision [8:17] and how those have been applied [8:20] in um [8:22] robotic use cases. [8:29] All right, cool. [8:32] So, hello again everyone. I'm Boris, as [8:35] Marco said. [8:36] Uh so, today's goal is to gain a [8:38] high-level understanding of some of the [8:40] more modern methods used in computer [8:42] vision, which have then been applied to [8:43] robotics. [8:45] And so, the readings are are where I [8:47] source this material is kind of from a [8:50] few sources here and there, mainly from [8:52] the courses here at Stanford CS231A and [8:55] CS231N, [8:57] uh as well as some various computer [8:59] vision conference papers, which kind of [9:01] just to illustrate a few examples. I [9:02] mean, we're not expecting you to go and [9:03] read the the latest and greatest, [9:05] although, you know, by all means, that [9:07] would be a something very interesting as [9:09] well. [9:11] So, [9:12] to to kick things off, let's let's [9:13] discuss a little bit about where we are [9:15] currently as a course in the perception [9:19] stack or at least in the computer vision [9:20] timeline. [9:21] So, starting from the pinhole camera and [9:25] and kind of making our way up to, you [9:26] know, feature detectors, descriptors, [9:28] and and etc., we've covered, you know, [9:30] from time periods in the 1500s to kind [9:33] of the this early 2000s period of [9:35] computer vision research. [9:37] And specifically what what I'm [9:39] interested in talking about today and [9:40] and giving you guys an insight on is [9:42] what has happened from this early 2000s [9:45] period to about 2012 and from 2012 and [9:47] onward. And we're going to talk about [9:48] why that's a special time [9:50] in recent computer vision history. [9:53] So [9:55] to kick things off just as a high-level [9:56] overview, a lot of early 2000s computer [9:59] vision was really dominated by [10:03] it you know, enforcing structure and [10:05] inductive biases into the methods and [10:07] and approaches that were used in [10:09] computer vision. And you know, feature [10:12] engineering really reigned supreme. And [10:13] you guys have gotten a taste of this by [10:15] seeing stuff, you know, [10:17] topics such as edge detection, corner [10:19] detection, blob detection. Okay, maybe [10:20] you haven't seen that in class yet or [10:22] you you won't. [10:23] Key point detectors and descriptors, you [10:25] know, methods like SIFT, SURF, ORB, etc. [10:28] And and answering questions like how can [10:29] you best describe a point in a in an [10:32] image. And so these methods here, they [10:35] are really [10:37] what they are is really applying human [10:39] ingenuity and engineering [10:42] skill into extracting as much [10:44] information as possible out of raw [10:45] pixels. [10:48] And so now what we want to do [10:50] is you know, these are very interesting [10:52] methods, but we want to shift our focus [10:55] from one of [10:57] focusing on image processing and how we [10:59] can extract information from images to [11:01] what problems can we solve with this [11:03] information. And and more specifically [11:06] for this course, what are we even caring [11:08] about with regards to computer vision [11:10] for robotics? [11:11] And so we want to shift our [11:12] shift our thinking from a method [11:14] specific focus to a problem specific [11:16] focus. And I'm going to describe today a [11:18] few different important problems [11:21] that you know, [11:22] come up in robotics and some approaches [11:24] from computer vision that have been used [11:26] to solve them. [11:28] So one of the first main classes of [11:30] problems is object detection and [11:32] classification. [11:33] And, you know, at at at a broad level, [11:36] this is the problem of finding if [11:38] there's an object of interest in an [11:40] image, and also classifying what that [11:43] object is. [11:44] And so, the general solution idea [11:47] is or at least pre-2012, the general [11:49] solution idea [11:51] is to describe a region of an image, [11:54] classify that uh [11:57] region, [11:58] and then shift your focus to some other [12:01] or the next part of the image and rinse [12:03] and repeat. [12:04] And so, this this kind of iterative [12:05] process [12:06] is how you might, you know, be able to [12:08] handle images of different sizes or [12:10] different uh scales. [12:14] And to dive a little bit deeper into [12:15] some of the methods that have been [12:17] employed in this space, [12:19] so towards describing a region of an [12:21] image, we've had methods like HOG or [12:24] histogram of oriented gradients, SIFT, [12:26] which is a a key point um [12:29] description method. [12:30] Uh patch similarity, code words, these [12:32] are some other methods that were used. [12:35] Then, in order to classify that [12:36] description, usually some methods from [12:39] applied probability and statistics or [12:41] machine learning have been used, such as [12:43] naive Bayes, nearest neighbor, SVMs, et [12:44] cetera. Doesn't matter. You you don't [12:46] have to know these terms. Just these are [12:47] some of the examples of the different [12:49] methodologies used for this specific [12:50] subproblem. [12:52] And then, you know, to shift our focus [12:53] to the next window, okay, [12:55] there's not really too much nothing [12:56] special here. We've more just kind of [12:58] looked at sliding window approaches [12:59] where you know, you might be looking at [13:00] one part of the image, do some [13:01] processing, and then look at the next [13:03] part, and keep going. [13:05] And [13:06] to to give you guys an example of what [13:07] this might look like uh pictorially, [13:10] we're going to dive into one of the [13:13] what I would say would was most popular [13:15] approaches uh along this [13:18] uh scheme to solve object detection and [13:21] classification, and that's specifically [13:23] applying this idea of histogram of [13:25] oriented gradients with a [13:27] uh classifier called a uh support vector [13:30] machine. Again, it doesn't matter that [13:32] you know these terms. It's just that [13:34] this is kind of the exemplary method [13:36] which really [13:37] um gives us an idea of what was going on [13:39] at the time. [13:40] So, what what would this method do? [13:42] Well, you would start kind of as we [13:44] discussed before. You have these three [13:45] key steps that you're doing and then [13:47] looping over. So, the first is you pick [13:49] some region in image that you're [13:50] interested in describing. [13:53] You then [13:54] extract some feature descriptor of that [13:56] and specifically this is what something [13:59] like Histogram of Oriented Gradients [14:00] might give you. [14:02] Um it is, as the name implies, literally [14:05] a histogram of the pixel gradients in [14:08] that space. [14:10] Gradients like edges, for example. [14:13] And then from these features, cuz these [14:15] these really are just a set of numbers, [14:16] so you can kind of make a really big [14:17] vector out of this. [14:19] And then what you might do from this is [14:20] you might take this vector and then try [14:22] and classify it. And when I say [14:23] classify, I mean you might have, you [14:26] know, a big set of vectors which are [14:28] extracted from images of people's faces [14:30] and you might say, "Okay, all of these [14:31] vectors are faces and then all of these [14:33] other vectors are not faces." And so, [14:36] that's what That's exactly actually what [14:37] this image here is trying to show is [14:39] that, you know, an SVM is a kind of [14:41] linear classifier which will look at [14:43] whatever vectors you have and draw a [14:45] line between them in order to segment [14:47] out two different classes. [14:50] And so, what you might do is or what you [14:52] would do is you would [14:54] run these points through your [14:55] classifier, obtain some score, [14:58] whether that might be, you know, how [14:59] likely it is to be a a face or not, [15:02] and then based on some threshold, you [15:04] would consider this a detection or not. [15:07] So, in this case, you know, our score [15:08] is, I don't know, 0.3 or something, [15:10] which is not very indicative of a face, [15:12] so we're not going to add this to our [15:13] list of detections. [15:15] All right, so what would do what would [15:16] we do next? Well, we slide our region of [15:19] interest over, [15:21] perform the exact same description, [15:24] run it through our linear classifier, [15:26] again, same lower score, [15:28] and then you basically really just [15:30] continue this up until maybe you [15:32] actually get to some region or object of [15:35] interest, [15:36] run it through the detector, [15:38] and now [15:41] we might have some higher score because [15:42] for the you know for the kinds of [15:44] vectors for the kinds of classification [15:45] problem that this model was trained on, [15:48] it is able to detect that this is a face [15:49] and it is able to give us a high score [15:51] or a high confidence that this is a [15:52] face. [15:53] And so now when we have this, okay, what [15:55] we're going to do is we're going to well [15:57] add this coordinate as well as maybe the [15:58] size of the filter to our let's say [16:01] bounding box list and then keep that in [16:03] mind for when we're going to report you [16:05] know our detection result after. [16:08] So, [16:10] that's you know one [16:12] general class of problems and that was [16:13] kind of one general solution method for [16:15] that. Another important class of [16:17] problems, especially in a robotic [16:18] setting, is something called scene [16:21] understanding. [16:22] And scene understanding is well [16:24] generally kind of understanding what's [16:25] going on in you know the environment [16:27] around you or the environment around a [16:29] robot. And one interesting problem that [16:31] we're going to be looking at today is [16:32] something called semantic segmentation. [16:35] And what that is is the problem of [16:37] developing an understanding of what's [16:38] going on in this scene around you by [16:41] classifying every observed point. [16:44] And so here's an example of what a [16:46] semantic segmentation might look like. [16:48] So if this is like a camera observed [16:50] image of some I guess this is like a [16:52] simulated environment, [16:54] then here you can see all of the pixels [16:57] are colored based on what class they [16:58] belong to. And so you might have classes [17:00] like roads, vehicles, sidewalks, [17:03] pedestrians, stop you know traffic [17:05] signs, etc. [17:07] And these are important for a robot to [17:09] understand because well obviously they [17:10] also encode regions of importance in [17:12] images and you know you definitely want [17:14] to segment and something like a traffic [17:15] light compared to something like a tree. [17:18] And and you know, where you might direct [17:19] your focus to. [17:21] And [17:22] I would say pre-2016, [17:24] there were these two general classes of [17:27] methods for solving such such a problem. [17:31] One was clustering based, and then the [17:33] other was graph based. And we're going [17:35] to actually look at some examples of [17:36] both of these today. [17:38] So, clustering based segmentation [17:43] and actually specifically within [17:44] clustering base, we're going to talk [17:45] about K-means. [17:48] And just before I dive any deeper, what [17:51] K-means clustering is [17:53] is [17:54] you know, at a very high level [17:56] you are It is a technique for segmenting [18:00] unlabeled data into groups or into [18:02] potentially semantically meaningful [18:04] groups. And so, this picture here really [18:06] describes what the goal of clustering [18:08] is. You know, if I give you just [18:10] a spattering of points in who knows what [18:12] high-dimensional space, the idea is can [18:15] we recover maybe what groups they might [18:17] belong to or maybe what individual [18:18] classes they might actually encode. And [18:21] so, for example, here at the bottom [18:23] well, this is I guess like like the [18:24] actual ground truth. This is like a This [18:26] would be a perfect clustering. So, this [18:28] is kind of showing us that there's [18:29] actually three separate distinct [18:31] clusters that form [18:33] uh this image here. [18:35] And specifically [18:37] what K-means is for us is is a method of [18:39] grouping together like or similar [18:41] features in their feature space. [18:45] And the reason it's called K-means and [18:46] and why there's that K there is because [18:48] inside the algorithm that you would run [18:50] in order to obtain such a a clustering, [18:53] you actually have to choose how many [18:54] clusters you want to segment out a [18:57] priori. So, it's a little bit of a [18:59] There's a little bit of a [19:00] hyperparameter tuning game that you have [19:01] to play, you know, how many clusters K [19:03] is the best and and kind of check how [19:06] your performance is [19:07] uh with regards to different Ks. [19:09] Now, I I dive into the actual algorithm [19:11] for this method. It's actually quite [19:13] simple and I really implore you to to [19:16] well, Google or search online what the [19:19] algorithm looks like. There's some very [19:20] nice animations of how the algorithm [19:22] actually segments out points, but that's [19:24] not the the important part here today. [19:28] The important part here today is to [19:29] understand how we might use such a [19:31] clustering method to actually segment [19:34] out similar points in an image. [19:37] And the idea, the key idea here is [19:39] actually to map each pixel in an image [19:42] into a five-dimensional feature space. [19:45] Specifically, [19:46] X, Y, and then RGB. [19:50] And And then what you do is once these [19:52] pixels are mapped to features, this [19:54] feature space, you can then try and [19:56] cluster them out using K-means or or [19:58] whatever. [20:00] And why why would we do this [20:03] uh feature space? Well, [20:05] the the intuitive part of this is that [20:08] in these five dimensions, X and Y encode [20:11] an element of spatial locality. And what [20:13] this means is that, you know, in an [20:14] image we intuitively believe that if one [20:17] part of an image is close to another [20:18] part of the image, [20:20] then they're probably going to be [20:21] related to the same object in 3D. [20:25] Uh I mean, unless you have some very [20:26] sharp gradient there, which you could. [20:28] Maybe there's an edge or something like [20:29] that. But if you were, you know, if I [20:30] were to take an image of the room right [20:32] now and I pick one pixel on that, you [20:34] know, one red chair there and then look [20:37] 10 pixels over and it's, you know, the [20:39] same red color, it's probably going to [20:40] be the same chair, for example. [20:43] And then why you have the RGB [20:46] is that this is kind of encoding our [20:49] intuition that similar things will have [20:52] similar colors. [20:54] And that's kind of just encoding like a [20:55] like a sense of visual locality rather [20:58] than just spatial locality. [21:00] And so if we apply this method, so here [21:02] we have just like a normal image, I [21:05] don't know, let's say taken from a plane [21:06] or a drone of some beach scene. [21:09] And we maybe want to cluster out what [21:11] the different parts of this image are. [21:13] If we run this, you know, feature we we [21:16] map each pixel to this five-dimensional [21:17] feature space, and we run K-means [21:19] clustering on this feature space, [21:22] if we use something like K equals three, [21:24] so we're looking to segment out three [21:25] different classes, we're going to get an [21:27] image like this. [21:29] And this is actually quite interesting [21:31] because immediately you can see [21:33] that this quite simple clustering idea [21:35] has segmented out the island or or [21:38] almost entirely the island [21:40] from shallow water, from deep water. [21:44] And so that's kind of interesting [21:45] because this simple five-dimensional [21:46] idea has already given us such a, you [21:48] know, such a strong um [21:50] performance in uh uh performance in the [21:53] output. [21:54] And the cool thing is because K-means [21:56] has this K parameter that you can tune, [21:58] we can actually see what happens if we [22:00] add maybe two more potential classes [22:03] that this method can extract. So here, [22:06] now not only are we extracting, let's [22:07] say, the island, but now we might have [22:10] actually a notion of the flora and fauna [22:12] that are on this island that maybe is [22:13] differing it from just grass. [22:15] And here we might have a sense of maybe [22:18] like a moderately shallow, [22:21] very shallow, and deep or or something [22:23] like that classes there. [22:25] And and [22:27] we can even continue doing this. [22:30] So now if we if I allow seven classes, [22:32] then now we're getting kind of almost [22:33] pretty close to the original image in [22:35] terms of, you know, every pixel already [22:37] in the image original image has like its [22:39] own color that we might just choose to [22:41] be a separate class itself. And And [22:42] maybe, you know, we're we're kind of [22:44] getting [22:45] pretty close to this [22:46] um [22:47] segmentation of classes. [22:49] >> If the space tree with like K equals the [22:52] number of pixels, wouldn't you just put [22:53] every pixel in some class? [22:55] >> Yep. [22:55] >> Cuz it's just like averaging the pixels. [22:57] >> Absolutely. So the the question was uh [23:00] Um, if I just chose K equals the number [23:02] of pixels in the image, would I not just [23:03] recover the original image? Exactly. The [23:06] problem is from a computational [23:07] perspective, that's not really ideal. [23:10] And also, you're not really getting much [23:11] semantic meaning out of there. [23:12] Um, but the other thing I will point out [23:14] is actually [23:15] this question is is kind of interesting [23:16] if if you think about how you if you [23:18] treat this K as a kind of like a sliding [23:21] parameter that you can change, [23:23] you can actually think of this solution [23:25] method as yes, it is giving you this [23:27] interesting discrete um segmentation of [23:29] same image. [23:30] But you can also think of it as actually [23:33] giving us a [23:35] a [23:36] um [23:38] like a compression of the original [23:40] image, right? So, if I have K equals [23:42] three, [23:44] think about it. That's actually going to [23:45] be a very efficient representation of an [23:47] image. I literally only have to store [23:50] three numbers for the for the classes, [23:52] and then like a big list of, you know, [23:54] pixel locations. Or actually, not even [23:56] that. I'll just store a It's It'll be a [23:58] big matrix, and then I just put in 0 1 2 [24:00] for the classes. Here, 0 1 2 3, okay, [24:02] maybe I need [24:03] 0 1 2 3 4, maybe I need more bits. Here, [24:05] I need even more bits. So, you can [24:06] actually think of this as also like an [24:08] image compression algorithm. And, you [24:09] know, hint hint, it was also used for [24:11] such a purpose. [24:13] >> All right. So, the beginning we want to [24:15] try to get them out of K, so we get more [24:17] information for our [24:20] clusters. [24:21] >> So, you [24:23] it seems like that, but actually what [24:24] you want to do is you want to pick this [24:26] Goldilocks K value that doesn't destroy [24:29] too much information, because if I just [24:31] pick K equals one, I'm just going to get [24:33] a big green image. And that's, well, [24:35] pointless, cuz everything's now the same [24:36] class. [24:37] But if I pick maybe K equals three, then [24:38] I get this kind of a coarse uh [24:41] segmentation. [24:42] And then maybe with this, I get, I don't [24:43] know, too much of a fine segmentation. [24:45] Maybe I can't now discern different [24:46] classes. So, you want to pick this K [24:48] value to be this kind of [24:50] right at the elbow point of, you know, [24:52] coarse and and fine segmentation. [24:56] And you know, here's another example of [24:58] uh of when I you have K equals six on [25:00] another scene. [25:04] So, another method that was applied to [25:06] this problem of semantic segmentation, [25:08] and I and I will say this is the more [25:10] advanced and the more in vogue in in [25:13] later years method [25:15] is semantic segmentation via graph-based [25:17] methods. [25:18] And what is the the the key idea of [25:21] graph-based methods is that images [25:23] inherently have this grid-like [25:25] structure, which we're not really taking [25:28] advantage of when we're just doing [25:30] clustering-based segmentation. I mean, [25:32] yes, we have this X and Y and we map [25:34] them to this feature space, but here, [25:36] when you have a graph-based method, you [25:37] are really explicitly encoding this [25:40] structure because what you end up doing [25:42] is you put a probabilistic graphical [25:44] model. I don't know if any of you have [25:46] seen that, but actually, has anyone [25:48] taken Michael Cocanougher's [25:50] decision-making under uncertainty course [25:51] here at Stanford? [25:53] Excellent. Okay, so some of you guys [25:55] might have seen a Bayes net or something [25:56] like that. [25:58] A probabilistic graphical model in a [25:59] general form [26:00] is well, this is the generalization of a [26:02] Bayesian network. So, for those of you [26:04] who know, great. If you don't, it's [26:06] completely fine. [26:07] What this is is basically you're [26:10] developing a probabilistic model of an [26:13] image, [26:15] whereby each pixel and its neighbors [26:18] have a special [26:20] um [26:22] conditional independence relationship. [26:25] Maybe is if that's the best way to put [26:27] it, or a conditional relationship in [26:28] terms of a probability [26:30] um [26:31] like a like a random variable sense. [26:33] And so, one way that you can encode and [26:36] and the reason I have this image here [26:38] uh depicting two different ways of [26:40] encoding this um [26:42] the the kind of connectedness you might [26:44] have in an image. So, on the left here, [26:45] this is like the most common, well, you [26:47] know, an image is a grid and and every [26:49] pixel is kind of connected to its [26:50] neighbors, and well, this makes sense [26:52] because you know, the pixels are side by [26:54] side. [26:55] But something that you might also do is [26:56] you might say well, [26:58] a pixel is actually influenced by many [27:00] more than just its own neighbors. And [27:02] you might come up with some elaborate [27:03] schemes for connecting, you know, [27:05] different pixels with their neighbors [27:07] and and [27:09] and um [27:10] however you might want to. [27:13] And I want to say here that although, [27:15] you know, we can view images as grids, [27:17] the literature in graph-based [27:19] segmentation actually goes much deeper [27:20] than that. Um there was really [27:23] for for [27:25] quite some years, there was [27:27] a very deep literature in how you can [27:30] extract the the best kind of structure [27:33] from an image or the most intuitive or [27:35] the most, you know, trading off [27:37] computational complexity because the the [27:39] more connections you have, the more [27:40] computation you have to do, the slower [27:41] it's going to run with accuracy, you [27:43] know, segmentation accuracy. And so the [27:45] here is some pictures of of some of the [27:47] different methods that were used, like [27:49] some of the different graph structures [27:50] that people built, but just know that [27:52] that the core idea is that you treat one [27:54] pixel as like an observed random [27:56] variable and then what you're trying to [27:58] do is, you know, obtain a probabilistic [28:00] sense of what class might have generated [28:03] this pixel's appearance and location, [28:06] for example. [28:08] And [28:10] so this method, usually what it was [28:13] usually how this was used was you would [28:16] have some first pass classification [28:19] method. So this could be via our, you [28:21] know, sliding window detector, but our [28:23] sliding window detector might give us [28:24] kind of a noisy signal. I mean, [28:25] especially around the edges. So here, [28:28] you know, this is like an image, so you [28:29] can see what the image actually looks [28:31] like. It's like a cow or something in a [28:32] field. And here you can see that our [28:34] whatever classifier that we used that we [28:35] initially passed over was quite noisy [28:37] around the edges. And I mean, this makes [28:39] sense. If your classifier sees half of [28:40] an animal, it's going to be I guess 50% [28:43] um sure that there's an animal at that [28:45] uh [28:46] And so usually what what methods would [28:48] do or what graphics methods would do is [28:51] you would start with some classification [28:54] initial classification, [28:56] perform some approximate global [28:59] optimization in order to find these [29:01] probabilistic, you know, the the values [29:03] for these probabilistic variables for [29:04] these random variables, [29:06] and then you would obtain your final [29:08] output. [29:09] And, you know, I will say that so, I [29:11] actually took this figure from a paper. [29:13] And so, you can see that, you know, this [29:14] initial prediction, uh it's pretty [29:16] noisy. I mean, this is pretty I I [29:18] wouldn't be too happy with this if this [29:20] is my final prediction. But, after [29:22] passing through a probabilistic [29:23] graphical model of a specific kind, [29:26] which was very popular at the time, [29:28] you get a very very very accurate [29:31] prediction. And so, these models, these [29:34] probabilistic graphical um based methods [29:36] for semantic uh segmentation were very [29:39] popular. And I'll And I'll say even now [29:41] there's kind of remnants of their use. [29:43] Um but, they were really the the [29:44] dominant force in this field, I'll say [29:46] pre-2016 and before. [29:50] And another problem that we're going to [29:52] talk about today, and one that you guys [29:53] will be more familiar with um coming up [29:56] soon in in 274A, is simultaneous [29:59] localization and mapping. [30:01] And what this is is well, it's the [30:03] problem of knowing where you are in a [30:07] scene while you yourself are mapping [30:10] that scene. So, it's kind of you're [30:11] observing the world and then placing [30:12] yourself within it at the same time. [30:14] Hence the name. [30:16] And this is really a key component of [30:18] robotic motion planning um [30:20] pipelines. [30:22] And traditionally, this problem was [30:26] solved really just by focusing on the [30:29] geometry of what's going on. I mean, and [30:31] you guys have seen this, right? Even [30:32] when we when last Friday when when I was [30:35] here talking about epipolar geometry and [30:38] if you have uh stereo cameras and how [30:40] you might reconstruct 3D information, [30:42] and then we talked a little bit about [30:44] structure from motion. These kinds of [30:46] methods really are the the the eminent [30:50] methods that you'll see in SLAM. [30:53] And even even nowadays, honestly, this [30:55] kind of you you really you cannot ignore [30:57] the geometries of scenes and and the how [30:59] the 3D world interplays with the 2D [31:00] world. [31:02] And and actually, I will say that, you [31:04] know, although we we have definitely [31:05] described some methods in this course, [31:08] the most state-of-the-art method like [31:10] state-of-the-art methods in SLAM [31:12] are really quite complex large-scale [31:15] engineering efforts. [31:17] And and when I mean this, I mean that, [31:18] you know, getting a paper with a [31:20] state-of-the-art result nowadays, it [31:23] takes something like, you know, a small [31:24] team of dedicated software engineers who [31:27] are working in highly optimized C code [31:29] or C++ code who then also have a team [31:32] of, I don't know, two or so or three [31:34] researchers which are nitpicking at [31:36] different parts of the SLAM pipeline. [31:38] And to to show you what this looks like, [31:40] so this was a method that I actually [31:42] worked with like not developed, but like [31:44] just used as a user back in, I don't [31:47] know, 2015 or or whatever when it was [31:48] still new. And this is, you know, [31:51] this is the paper they this is the [31:52] figure they put in the paper, right? As [31:54] this is something that people should be [31:55] understanding when they look at, "Oh, [31:57] clearly this is what's going on in your [31:59] algorithm, right?" I mean, it's it's [32:00] insane. You have a huge amount of [32:03] different moving components, [32:05] each of which internally have their own [32:07] specialized research results which have, [32:09] I don't know, 20 papers behind them, [32:11] which have a dedicated research [32:13] scientist or a software engineer who [32:15] specifically spent months programming [32:17] them. So, you know, such a such systems [32:20] are really really complex, but I will [32:22] say they're really performant. And [32:24] actually, what we're going to do today [32:25] is we're going to look at some of the [32:28] uh as what like what some results [32:30] actually look like from this uh from [32:33] these methods. [32:37] And so, we're going to look at one of [32:39] these is called Orb-Slam [32:43] from Zaragoza. [32:46] And let's see if I can get some [32:51] So, here on the left, you can see the [32:53] camera image marked with what with some [32:56] features that that were first segmented [32:58] out. And you can see how the drone is [33:00] like it's kind of moving through [33:02] um kind of noisily and you know, with a [33:04] lot of of rotation. [33:06] But, you can see on the on the right [33:08] what the output is like the localization [33:09] output. So, this method Orb-Slam is very [33:12] good for localization. Not so good for [33:13] mapping, and we'll talk about why, but [33:15] it's very good for localization. And so, [33:17] you can see [33:19] I mean, so this is showing like [33:21] different robotic applications and you [33:23] know, different data sets and how it [33:24] might very nicely reconstruct, you know, [33:27] where the robot was in previous time [33:29] steps. [33:31] But, I want to see there's a really nice [33:35] All right, so this [33:37] is [33:39] This is you know, someone's walking [33:40] around their office environment and [33:41] they're they're kind of mapping where [33:42] they walked around and where they moved [33:43] the camera. [33:45] And I think there's one very interesting [33:47] part where they like [33:51] Yes, yes, yes. [33:54] Oh, okay, that's it. But, basically, [33:56] there was also Yeah, so now you can also [33:58] see what the map output looks and and [33:59] how it kind of accurate that is. [34:01] So, [34:02] to give you guys a a little bit of [34:03] background of why this method is good [34:06] for um [34:08] you know, this method is good for [34:10] localization but not mapping. What's [34:11] going on internally just to demystify [34:13] things a little bit. What's going on [34:14] internally [34:16] is [34:17] you have the first stage of this [34:19] pipeline is detecting feature detecting [34:23] key points in an image. And that's these [34:25] like green dots that you saw here. [34:28] Oh, yeah, so So like the green dots you [34:30] see in the image here. So what this [34:32] method first does [34:34] is it finds these key points in an image [34:37] which have some properties. So [34:39] specifically these key points have [34:40] properties of like uh this point is [34:43] believed to be very stable. What is [34:45] stable mean? Stable means that it's [34:47] probably part of a static object which [34:50] has probably, you know, which has colors [34:52] that are very good for um detecting in [34:55] terms of different light um you know [34:58] aperture settings, shadows, whatever, [35:00] rotations. [35:02] And so you first find these interesting [35:04] key points. [35:05] And then [35:07] more or less you perform the same um [35:11] you know, structure from motion step [35:14] that we saw last Friday where you try [35:16] and now, you know, find the rotation and [35:17] translation matrix between two [35:19] consecutive images. [35:21] Uh and the reason you would use these [35:24] the reason why you do this first this [35:25] this like feature point detection step [35:27] is that it makes your correspondence [35:30] matching problem a little bit easier [35:31] because these points you've kind of [35:34] chosen these points such that they are [35:36] very nice for your correspondence [35:39] problem later. I.e. they don't really [35:40] change much given different lighting [35:42] conditions, even given different [35:43] rotations, they're still easy to detect. [35:46] And so this is why a method like this [35:47] would be really good at localization. [35:49] Why it's not good at mapping, and when I [35:51] say mapping I mean like it doesn't [35:53] really give you a dense point cloud at [35:55] the end [35:56] is because you're only doing your slam [35:59] on these key points. [36:00] And unfortunately, I mean, there's only, [36:03] let's say, even if if this [36:06] um frame had, I don't know, 2,000 [36:08] pixels, there's only maybe [36:10] 200 key points. And so of these 2,000 [36:13] pixels you're only using 10%. [36:15] And so this is why your output, your map [36:17] output, is not very dense. [36:20] So that's one one method. [36:22] And another one [36:25] which is very nice to look at [36:29] is DSO. Stands for I think it's direct [36:33] sparse odometry, yeah. [36:35] So this was actually the the direct [36:37] successor to that pipeline image I [36:39] showed you. [36:41] And so here you can see someone is kind [36:43] of taking a a [36:45] a camera and and walking it through, I [36:47] guess, some outdoor scene with hedge [36:50] which has like stairs and stuff. And you [36:52] can see how, you know, super accurate [36:54] and super stable like the map is. Like, [36:56] you know, if you if you rotate the [36:57] camera, then you don't have walls that [36:59] are now changing their shape just [37:00] because, you know, maybe your key points [37:02] look a little bit different. [37:07] And a lot of the engineering that goes [37:09] into these methods [37:10] like the reason why I say that there's a [37:12] significant software engineering effort [37:14] is because yes, I mean this this [37:16] structure from motion part of the [37:18] problem, it it's difficult. It requires [37:20] a lot of software engineering. But where [37:22] a lot of the the real key engineerings, [37:24] and I will say a lot of the like [37:25] contributions that these papers make, is [37:27] how you maintain this global map [37:29] information. Because remember, it you [37:32] learn a lot of information if you have [37:35] So actually, yeah, this this is [37:36] important, actually. What a what a good [37:38] ending time. They actually say that what [37:41] they've done is they've gone through, I [37:42] guess, a subway station and they've gone [37:44] through different parts of the subway [37:45] station and then they came back to where [37:47] they originally were. [37:48] Right? So they came back to the same, I [37:50] guess, bicycle parking. And look, [37:53] right? Look at this accumulated error. [37:55] These should be the same ending points. [37:57] The end point is the start of the same, [37:58] you know, the end point is the same [38:00] start point. But but here they have some [38:02] accumulated drift. And this is a [38:05] something that has been plaguing SLAM [38:07] forever and it always will just because [38:09] of the nature of the problem. But [38:10] there's something that you can do, [38:11] right? If you have a global map that you [38:13] can update all the time, something that [38:15] you can do, which a lot of these methods [38:16] do and I'm pretty sure they're going to [38:17] like show this next, is something called [38:19] loop closure. [38:20] Where what happens is you walk through [38:21] this subway station, whatever. [38:23] And when you detect that you came back [38:24] to the same spot, you can actually [38:27] perform kind of like a post hoc [38:29] correction of oh, I've seen this before. [38:31] So, let me correct all of my [38:33] trajectories and let me correct all of [38:35] the map that I've seen to that point [38:37] such that it is now [38:40] um [38:41] like self-similar. Like such that it is [38:43] now correctly handling this or correctly [38:47] removing this tracking error. [38:51] So, let's see if they show the exact [38:52] same [39:00] Okay, and then they're going to show [39:04] lots more [39:06] fun [snorts] stuff. [39:10] And the reason why So, the reason why [39:12] this method is called dense whatever it [39:14] was [39:15] is So, the map that they produce [39:19] is actually denser than the previous one [39:21] that we saw because this method and kind [39:25] of this similar family of dense-based [39:27] methods [39:28] when they have like a 2,000 pixel image [39:32] they actually use most of those pixels. [39:34] Rather than doing first key points, [39:36] detecting key points and then doing, you [39:38] know, your your procedure after on that. [39:40] What they do is that they detect edges [39:43] and other much more dense features. So, [39:46] yes, this can be more noisy, of course, [39:48] because you might have now one edge [39:49] which looks very similar to another, but [39:51] it's not the same between images. And [39:53] so, of course, you're going to [39:54] accumulate some error in this manner. [39:56] But, [39:57] what this gives you is a much denser [39:59] map. So, you're going to get a much [40:01] better map out of it even if you might [40:03] suffer a little bit in your [40:04] localization. So, that's kind of the [40:05] game you play with these two different [40:07] methods. [40:09] All right. So, [40:11] let's continue. [40:15] All right. So, I kept bringing up this [40:17] this this date, this 2012 [40:19] date in computer vision. [40:21] And [40:22] some of you might remember, yeah. [40:25] >> Is this [40:25] the previous How long does it take to [40:27] process an image like this? [40:29] >> Real time. [40:30] A huge amount of this software [40:32] engineering effort is actually also to [40:34] make it run in real time. So, you can [40:36] like they do this in milliseconds, [40:37] right? Not maybe single digit [40:39] milliseconds, maybe double digit [40:40] milliseconds, but [40:42] these methods run at 30 plus hertz on [40:47] like a cell phone [40:49] like a cell phone CPU. [40:52] Let alone, you know, a laptop CPU. Yeah, [40:54] no, these these methods like [40:57] I'm telling you, software engineering is [40:59] 80 to 90% of these methods, right? [41:02] Because you have all this math that's [41:03] going on. And then you have these [41:05] ridiculous, you know, compute times of [41:07] like [41:08] I don't know. I've definitely seen [41:09] methods which is like 60 plus hertz. So, [41:12] it's like just imagine how many [41:13] milliseconds That's all the compute [41:14] budget you have. And it's like on an i5 [41:17] or like on a cell phone camera. In 2016. [41:20] We're not even talking like the fancy, [41:22] you know, Snapdragon quad I don't know, [41:24] 8500 or whatever they have now. We're [41:26] talking like [41:27] less smart phones, okay? [41:30] We're talking like this is engineering [41:32] to the max, okay? This this this is like [41:34] Yeah, I'm not kidding when I say this is [41:36] a really significant amount of [41:37] engineering effort. [41:40] And a really cool research, right? I [41:41] used to do this in like 2015. [41:43] It was good stuff. [41:45] Fun memories [41:46] of like 2000 line C code repositories. [41:50] Um so, okay. So, in 2012 [41:53] before most of you were even born, there [41:55] was a [41:57] Some you might remember there was kind [41:59] of this this funny scare of ah, the [42:00] world might end and all that. It didn't. [42:03] But, for computer vision [42:06] for a lot of the computer vision [42:07] research, it kind of might have well. [42:10] And there was a really large paradigm [42:12] shift in 2012 that really spun the [42:15] entire field on its head. [42:17] And you know, part of this you know, [42:20] this lecture wouldn't be what it is if I [42:21] didn't tell you what happened here and [42:23] give a little bit of insight into what's [42:25] going on. [42:26] So, before we understand the paradigm [42:27] shift, we have to get a little bit of [42:29] background into a parallel line of [42:31] research that happened at this time. [42:33] And this parallel line of research, as [42:35] you might have guessed, was machine [42:37] learning. I mean, machine learning was [42:38] already used in computer vision at this [42:40] point. I mean, you guys already saw this [42:42] support vector machine line [42:43] classification thing. This is like a [42:45] core machine learning method which was [42:47] which was you know, in development and [42:49] has been used for decades before 2012. [42:52] Um and and the core idea of machine [42:54] learning, and this is I must say [42:56] pretty much a pretty crazy [42:57] oversimplification. So, don't don't put [42:59] too much bearing into this. This is [43:01] again, just to give you guys a [43:02] high-level idea of why these methods [43:04] might play together well. [43:06] So, in machine learning, usually we want [43:09] to predict some quantity Y from some [43:12] input X, and we want to we basically [43:14] want to find a function parameterized by [43:17] parameters theta which performs this [43:19] mapping for us. [43:21] And [43:23] so, usually the way that we do this [43:26] is well, we define some function F, some [43:28] structure, [43:29] and then we formulate a loss function L [43:32] which basically denotes or [43:35] it kind of tells us how bad our [43:37] predictor is. [43:39] And then [43:40] the the main [43:42] thing that we put our effort towards is [43:43] minimizing this loss function L with [43:46] respect to the parameters stated that [43:48] define F. So, F might have some fixed [43:50] structure, [43:51] but it it's going to have some [43:53] changeable or tunable parameters [43:54] internally. And I mean, you guys have [43:56] seen this in uh uh you know, who here [43:58] knows what linear regression is? [44:01] Fantastic. So, well, as you all should [44:03] from the prerequisites. Um but exactly, [44:06] linear regression is is a is a perfect [44:08] example of this, right? Y equals mx plus [44:10] b, m and b are your tunable parameters. [44:12] So, in machine learning you just have [44:14] more complicated functions rather than [44:16] mx plus b. [44:18] And and I've put here this kind of [44:20] this this equation, seemingly simple, [44:23] but this is kind of the the the basis [44:24] for a huge amount of machine learning uh [44:26] effort, and that is the equation for [44:28] gradient descent. [44:29] Doesn't matter, but the idea is that [44:31] basically [44:33] in order to optimize these parameters [44:34] beta, [44:35] what we will do is we will first find [44:38] how does our loss function change with [44:40] these parameters? I mean, if some of you [44:42] might remember from multivariable [44:43] calculus that the gradient tells us the [44:47] direction of steepest change [44:50] uh that a function has with respect to [44:52] some specific [44:54] um [44:54] variable. And so, for us, this variable [44:57] is beta. [44:58] And what we do [45:00] is [45:01] in order to optimize this beta, we [45:03] basically find these directions of of [45:05] largest change. [45:07] And [45:08] since we want to decrease our loss [45:10] function, [45:11] we take steps in the negative direction. [45:14] So, gradient gives you information about [45:16] how you might climb up the function in [45:19] the best way. We absolutely don't want [45:21] that. We want the the exact opposite. We [45:22] want to climb down this function. And [45:25] so, this is kind of giving you an [45:25] intuition for why gradient descent why [45:28] the formula looks like this. [45:32] And so, as I said before, machine [45:33] learning has existed for a long time. [45:36] But the really to to to be a bit more [45:40] uh correct about some terminology here, [45:42] it's existed for a long time because [45:45] machine learning itself takes a lot of [45:47] inspiration and a lot of tools from you [45:50] know, different fields such as applied [45:52] probability, statistics, linear algebra, [45:55] multivariable calculus. [45:57] And machine learning itself, why it's [46:00] kind of now it's own little subfield is [46:03] because it's typically concerned with [46:04] modeling continuous data. So, for [46:06] example, regression tasks [46:07] um [46:08] or performing something like a discrete [46:10] segmentation. So, like a you know, [46:12] forming like a like a solving like a [46:14] classification task like the one that we [46:16] saw earlier. [46:17] And I will say up until this point, [46:20] images were definitely considered [46:21] objects of interest. I mean, [46:23] they're just matrices. They're just, you [46:25] know, 2D or 3D tensors depending on if [46:27] you have color or not. [46:28] Um but the problem that you know, a lot [46:31] of machine learning practitioners and a [46:33] lot of computer vision practitioners [46:35] were facing at the time [46:37] is that existing methods in machine [46:39] learning weren't really taking advantage [46:40] of the structure of an image. [46:43] I mean, we were talking a lot about how, [46:45] you know, the graph-based and and all [46:47] these different like segmentation [46:48] methods, like they thought of ah, okay, [46:50] images have spatial locality and they [46:52] have visual locality and all this. But a [46:54] lot of the machine learning methods at [46:56] the time weren't able to encode spatial [46:58] locality or visual similarity. [47:02] And so, [47:04] what happened in 2012, so just to set [47:06] the scene a little bit. So, prior to So, [47:10] this is a So, ILSVRC stands for the [47:13] ImageNet Localization and No, no, [47:16] ImageNet Large Scale Visual Recognition [47:18] Challenge. Something that's developed at [47:19] Stanford in the [47:21] uh in the vision lab. [47:22] And [47:24] it's a it's a problem it's basically [47:26] what what this challenge is is you have [47:29] available to you, I don't know, a [47:31] million images or millions of training [47:34] images which allow you to develop some [47:37] you know, some algorithm for detecting [47:40] and classifying objects within those [47:42] images. [47:43] And so, what this graphic is showing [47:46] here [47:47] is [47:48] a lot of different methods that were [47:49] submitted to this challenge. And the [47:51] reason I say submitted in all this is [47:53] because this challenge had like a like a [47:57] website with a submission server. So, [47:58] you know, to be fair and and everything [48:00] what they would do is they had like a [48:01] held out data set which no one knew. [48:03] Like you couldn't see this. And you [48:05] would submit your algorithm to them and [48:08] then they would run your algorithm on [48:09] this data and then they would report the [48:11] error that your algorithm achieves. [48:14] And so we want lower error. And so what [48:16] we're looking here, so this is error. [48:19] And what what we're looking here is some [48:20] different methods that were submitted to [48:22] this uh visual recognition challenge. [48:26] And we can see that they, you know, at [48:28] best they come close to 25%. It's [48:30] actually 20 Yeah, 26% [48:33] error rate. [48:34] And what these are, you know, so this is [48:36] So, Oxford had a [48:38] a method and they submitted it. [48:40] University of [48:41] Amsterdam had some method and, you know, [48:43] some other industry groups and [48:44] consortiums developed computer vision [48:46] methods and uh [48:48] and submitted them. [48:49] But, you know, [48:51] at at a high level these methods are [48:53] really representing kind of the best at [48:56] that time [48:57] of human ingenuity and feature [49:00] engineering in computer vision. [49:03] And so, you know, this is representing [49:04] hundreds of hours and years and years of [49:05] research which kind of lead up to [49:08] achieving these error rates of, you [49:10] know, what is this? 26% [49:13] And so, [49:15] another I'm hiding one result here. [49:18] And this single bar that's going to come [49:21] up here changed really all of computer [49:24] vision when this result was released. [49:27] We went from 26% error [49:30] to 16% error. [49:33] This is ridiculous. This is a almost a [49:37] 50% jump [49:39] in state-of-the-art performance. That's [49:41] insane. You This never happens. [49:44] Okay? This Look at these methods, right? [49:47] Look at this. [49:49] The difference between these two methods [49:51] is like 1% and these people spent I [49:53] don't know hundreds of hours, right? [49:55] This mysterious team with a funny name [49:59] came up with who knows what [50:01] and absolutely demolished them. [50:04] It's ridiculous. [50:05] And I'm going to give you a little bit [50:06] of insight. This team, this supervision [50:09] team [50:10] is Canadian. [50:11] >> [laughter] [50:12] >> From my university. [50:15] Um [50:17] I Yeah, I guess they found it pretty [50:19] funny calling it supervision, but [50:20] whatever, who cares. So, [50:23] what was this method? What was this [50:26] ridiculous paradigm shift? [50:28] You might have heard of this before and [50:30] that's okay. It still doesn't take a lot [50:31] of the bang out of this presentation. [50:34] But the method was convolutional neural [50:37] networks. So, let's let's go back to our [50:41] um the structure that we developed for [50:43] machine learning methods. So, you [50:45] formulate some function L which which [50:47] denotes how bad your F is [50:50] and then you minimize it with respect to [50:51] the parameters that [50:52] govern your F. [50:55] So, what a convolutional neural network [50:57] is is a specific structure of this [51:00] predictor function F. It's a specific [51:03] arrangement of your parameters theta [51:06] that develop this interesting function [51:08] and it turns out that this method really [51:11] reigns supreme for visual data. [51:15] And [51:17] roughly what they are [51:19] is it's a hierarchical set of successive [51:23] convolutions with learnable filters. So, [51:25] you guys have seen or are seeing right [51:27] now in homework three and you've seen in [51:29] lecture before [51:30] that you know, if you have a linear [51:31] filter, you can well convolve it across [51:33] an image and you obtain some output [51:35] image which looks different or it's been [51:38] filtered in some manner. [51:40] And what a convolutional neural network [51:42] does is really it just applies [51:45] lots of these filters one after the [51:47] other. [51:48] And that's I mean that's literally [51:50] what's going on in this in this image [51:52] here. You have some image which goes [51:54] through some linear filtering to develop [51:56] some other value. [51:58] You then do it again. [51:59] You do it again. You do it again. [52:01] And [52:03] the the crux of the whole methodology is [52:07] that these filters are trainable [52:09] parameters. [52:10] And so rather than, you know, someone [52:13] developing an edge detector and like [52:15] actually putting some values inside the [52:17] filter and saying, "Okay, I these values [52:18] have some properties that I know that [52:20] they will detect edges or or whatever." [52:22] Here you just say [52:24] "I'm going to initialize these filters [52:25] with some random values, who knows what, [52:28] and I'm going to let this minimizing [52:30] this function L [52:32] give me [52:34] some interesting filters and and learn [52:36] some interesting filters." [52:37] And actually it turns out that that's [52:39] actually exactly what happens. So, in [52:41] this convolutional neural network [52:43] so, the the way that I'm going to [52:45] describe, you know, this part of the [52:47] network is maybe low-level because it's [52:49] the closest to the image, which is kind [52:51] of the the input raw data. Then we have [52:53] maybe the middle of the network and then [52:55] like the the upper area of the network. [52:59] Upper because it's kind of the farthest [53:01] abstracted away from the image. [53:03] And the kinds of filters that are [53:06] learned are really interesting. [53:08] And [53:10] so, at the very lowest level, so these [53:11] are filters that are closest to the [53:14] image, [53:15] you actually learn edge detectors. [53:18] And specifically, because you're taking [53:21] in color images, you learn edge [53:23] detectors kind of in a grayscale sense. [53:26] So, here you can see white and black. [53:28] So, that's you know, literally white you [53:29] know, what this is denoting is white is [53:31] like a one, like a plus one value, and [53:34] then black is zero or minus one. So, [53:36] you're you're literally subtracting two [53:38] neighboring pixels in some orientation, [53:39] so So, it's an edge detector of that [53:41] type. [53:42] And what you also see, which is really [53:43] interesting, I think, [53:45] is you see color edge detectors. So, [53:47] yes, we have these black and white, you [53:49] know, which might be good for detecting [53:50] grace just general edges, [53:53] but here we have a blue and an orange, [53:56] you know, [53:57] filter, which is kind of putting them [53:59] side by side. And the reason for this is [54:01] because, as you guys are well aware in [54:03] homeworks, filters can have channels, [54:04] too. [54:05] And so here, different parts of the [54:08] channels are subtracting at different [54:10] times. [54:11] So, that's kind of interesting. [54:13] Moving up the convolutional neural [54:15] network, we have some middle-level [54:17] features. And this is [54:19] quite interesting, too, because if you [54:20] think about it, these look like [54:23] combinations [54:24] of edges, right? It seems like I mean, [54:27] especially look at this, right? It looks [54:28] like you're trying to detect corners, [54:30] which is the intersection of two edges. [54:32] So, it kind of seems like [54:34] this middle-level this middle layers of [54:37] the network, the filters that they learn [54:39] are encoding like a little bit of a [54:42] like a more um [54:45] a more abstract feature or more abstract [54:48] set of filters. So, for example, looking [54:50] for edges, looking for circles, looking [54:54] for [54:55] maybe grid patterns or something like [54:57] that. [54:58] Maybe general slopes. And then, of [55:00] course, you have the exact same thing, [55:02] but with colors. [55:03] Right? So, across different colors. [55:05] And then, perhaps most interestingly, at [55:08] the highest level of abstraction, you [55:09] know, one of the last layers before you [55:10] step into the into the classification [55:13] part of the network, [55:15] you know, have some really high-level [55:16] features. [55:17] This is, you know, detecting honeycombs, [55:19] I guess, in an image. [55:22] This is maybe detecting parts of a [55:23] wheel. I mean, [55:25] you know, what what's going on here is [55:26] you [55:27] maybe [55:28] I think what they're specifically [55:30] looking at here is, you know, they were [55:31] maybe training on [55:33] many different images of, I don't know, [55:36] cars or something. And so, you might see [55:37] these interesting filters come out, [55:39] which are detecting, you know, parts of [55:42] circular things or I mean, that almost [55:44] looks like a stop sign, right? This [55:47] filter almost looks like it's detecting [55:48] something which has a red bulb on top [55:50] and then like a silver, you know, stick [55:52] on the bottom. And and many other [55:55] interesting filters there. [55:57] So, [56:00] >> [snorts] [56:01] >> I actually have there's a really nice [56:02] demo which I want to show you guys and [56:03] we're going to look at that right now [56:05] of how these different layers, these [56:08] different filters play together in order [56:11] to actually perform classifications. [56:14] So, [56:15] let's pull this up. [56:19] So, first things first, [56:21] I want you to see the dot CA up there. [56:24] Canada. All right. So, this is [56:27] visualizing a convolutional neural [56:29] network. What it's doing is it's showing [56:32] us just a little bit of a different view [56:34] of it. And then the coolest part is that [56:36] it's actually 3D. So, if I I can kind of [56:39] take a peek at some of the different [56:40] layers and some of the different aspects [56:41] that are going on here. So, at the very [56:43] bottom is our image. [56:45] And you'll see how this works when I [56:47] actually draw like an image there. But [56:48] at the very bottom is our image. [56:50] This first layer, so the way that a [56:54] convolutional neural network usually has [56:56] its different layers is it stacks them [56:59] together as different like [57:02] it's called activation maps. Basically, [57:04] it's the result of performing [57:06] convolution or correlation with [57:07] different filters. So, for example, one [57:09] filter's output is going to be [57:10] visualized here. Another filter's output [57:12] is going to be visualized here. [57:14] So, it appears here that we have six [57:16] filters [57:17] for this first uh [57:19] for this first layer of the [57:20] convolutional network. [57:22] And then we're going to have [57:25] uh [57:26] I don't know, 24. [57:27] Let's guess. Filters for the second [57:28] part. And then that's it. So, it seems [57:30] like we Here we This convolutional [57:32] network only has two convolutional [57:34] layers. This first one here and then the [57:36] second one here. [57:38] This is a pooling operation. Don't worry [57:40] about it, but just know that this is not [57:42] a convolution operation. [57:44] All right. So, let's actually look at [57:46] what this looks like. Let's get some [57:47] pretty colors on here. So, I'm going to [57:49] draw a number here [57:51] and my favorite number is [57:53] I think seven. [57:55] So, let's draw that. [57:58] All right. So, now what it's showing us [58:01] is [58:02] so, it's showing my original image down [58:04] here. [58:05] And then it's actually showing us the [58:07] result of the filtering in this first [58:10] layer here. So, you can see [58:12] that we have the same seven that I drew, [58:15] but take a look at this, right? It seems [58:17] that this first filter might have [58:19] learned edge detection, horizontal edge [58:21] detection, because you can see it's [58:23] maximally activated at these horizontal [58:25] edges. [58:26] This other filter might have learned [58:27] some diagonal edge thing. [58:29] This filter might have learned just a [58:30] general, I don't know, maybe [58:33] maybe just it's doing subtraction of [58:35] different uh cuz you know, this is kind [58:36] of like a zero one. So, maybe it's [58:38] learned how to just subtract nearby [58:40] pixels and then and so on. [58:43] And then we have at higher levels [58:47] mhm some I mean, you can see that [58:50] there's still a seven here kind of, but [58:54] it becomes a it becomes a bit harder to [58:56] interpret at higher levels. [58:58] And this is this is actually pretty [58:59] endemic to convolutional neural networks [59:01] in general. And this is also why a lot [59:03] of times you might hear people call [59:04] these methods black box methods because [59:08] I don't really understand what's going [59:09] on here. I I don't really know, other [59:10] than maybe this first layer. It's very [59:12] hard to say why or like how this seven [59:16] percolates through and becomes actually [59:18] a correct prediction up above. So, if [59:20] you see up there, it actually does [59:21] output that the prediction is a seven. [59:24] Now, the the best thing about this [59:26] visualization here [59:28] is actually that you can see what parts [59:31] of the network are using what parts of [59:33] the original image. And And let me show [59:34] you what that looks like. [59:35] So, here's that my, you know, image down [59:37] at the bottom. And if I put my mouse [59:39] over this part, this first result, [59:43] what I can see is [59:45] what the [59:47] filter [59:49] what parts of the image was the filter [59:52] at when it generated this output value. [59:55] Right? So, this gives you the the [59:56] intuition that really what's going on in [59:58] a convolutional [59:59] neural network is [60:01] you are Oh, that's kind of cool. [60:03] Uh you are basically scanning some [60:06] filter across an image [60:08] and then just putting all of its outputs [60:10] in another image. Right? So, this gives [60:12] you that idea right here. [60:15] And then you you rinse and repeat. Well, [60:16] this is that pooling thing, but don't [60:17] worry about it. Then you rinse and [60:19] repeat, but now it's maybe looking at [60:20] different channels because you just have [60:22] a It's another image. It just has more [60:23] channels. [60:25] Blah blah blah blah blah, and then it [60:26] goes up to the top, and then you can [60:27] see, you know, when it comes to the [60:30] fully connected part. So, this is [60:32] performing some classification. [60:35] It's basically just doing WX + B with [60:37] some fanciness on top of it. [60:39] But, you can see at the end all of this [60:42] percolates up and it gives you a correct [60:43] response. [60:44] Now, the coolest thing about this is [60:45] well, you can keep playing with it. So, [60:46] if I do the more European seven, uh-oh, [60:48] what's this? [60:50] So, it is actually still The first guess [60:52] is actually still seven. So, it's still [60:53] correct, but not by much. The second [60:55] guess looks like a three. And maybe you [60:58] can see how it looks like a three, [60:59] right? But, the you know, [61:01] let's let's just keep messing with it. [61:03] I don't know. What number is this? [61:06] It's a six. [61:07] Or an eight. Okay, what if I keep going? [61:10] Oh, yeah, it's an eight. I believe it. [61:13] What's that? Another eight. [61:15] What's this? [61:17] What's this? [61:19] Right? So, the cool thing is with this [61:22] that's a one, obviously. So, the the [61:25] cool thing with this is that [61:27] you All you really have is just some [61:28] fixed size input, and then you can run [61:30] whatever you want through it. So, that's [61:31] kind of interesting. [61:32] The the slight problem is if I did Well, [61:35] which I did. If I would just put some [61:36] garbage here, [61:38] I don't really know that it's garbage at [61:39] my end. [61:41] I like that the highest layer. I mean, [61:43] you might use this to I don't know, sort [61:45] mail by maybe people's [61:47] zip codes and just say, "No, this is [61:49] actually one of the first use cases for [61:50] convolutional neural networks." Was was [61:52] classifying mail like [61:55] doing optical character recognition on [61:58] handwritten mail like USPS post [62:01] letters. That was actually where a lot [62:03] of this original work was done. That was [62:04] one of the original data sets. [62:06] And so, what the problem is is if you're [62:08] running, you know, your your envelopes [62:11] through this system and someone, maybe [62:14] their pen broke or something, you're [62:16] going to be like, "Oh, yes, clearly we [62:17] should send it to somewhere in New [62:19] York." Or whatever a one corresponds to [62:22] in the US at the you know, if it's at [62:23] the first position. [62:25] All right. So, [62:27] enough with this visualization. [62:31] But, [62:32] you guys should [62:34] play with this more. [62:36] So, [62:37] after this this ridiculous result, this [62:39] demolishing old methods, [62:41] now it became this kind of equivalent [62:44] space race, but race to the lowest [62:45] error. So, now it was really off to the [62:48] off to the races. So, [62:50] 2012, we had this [62:53] University of Toronto method called [62:55] AlexNet after the first author, Alex [62:57] Krizhevsky. [62:58] Um and it received a 16% error rate, and [63:01] it had about four [63:02] had about eight convolutional layers. [63:05] Then, the next year after, woof, already [63:09] a 5% decrease. [63:11] Same eight layers, but maybe some [63:13] fancier structure internally or maybe a [63:14] better way of handling data. [63:17] Oh, another 4% decrease. 19 layers, [63:20] uh-oh, something's going on here. [63:22] 22 layers, a little bit of decrease. [63:25] Oh, boy. [63:26] 152 [63:28] and [63:29] well, the cool thing is this is actually [63:30] superhuman performance. [63:32] So, this is better at classifying images [63:33] than we are. [63:36] And and someone actually measured, so [63:38] one of the creators of this data set, [63:40] this ImageNet data set, Olga [63:41] Russakovsky, one of the works that she [63:44] performed in, I don't know which year, [63:47] was actually classifying how good is a [63:48] human at this kind of task. And it's a [63:50] turns out we make about a 5% error rate. [63:52] But, that's also that could also just be [63:54] because, you know, [63:55] I don't know what a gibbon looks like if [63:57] you just give me a random image of one. [64:02] ImageNet [64:03] is the name. It's It's the exact same [64:05] data set that all of these [64:07] ImageNet large scale visual recognition [64:09] challenge. So, that ImageNet first part, [64:11] all of these methods were trained on [64:13] this [64:14] uh ImageNet and they were kind of [64:15] evaluating this held out test set of [64:17] ImageNet. [64:19] So, [64:22] what we see is that now methods have [64:24] kind of, [64:25] you know, lots of methods are now diving [64:27] into maybe making these networks larger, [64:29] maybe having a bit more feature [64:30] abstraction. [64:32] And so, okay, let's revisit those same [64:34] three problems that we saw initially. [64:36] >> Yeah. And those algorithms, would they [64:38] work [64:39] in real time or are they slow? [64:41] >> No, slow. It's exactly for the same [64:43] reason that you are probably thinking. [64:45] If you have 152 layers of these things, [64:47] you have to perform these you know, [64:48] computations 152 times. So, no, they're [64:50] not fast, unfortunately. [64:52] >> I mean, [64:53] like [64:54] in prediction phase, they're slow. [64:55] >> Yes. [64:56] >> Not just in training. [64:56] >> Oh, in training? Well, training took So, [64:59] training this [65:02] took 3 weeks [65:04] on a GPU. This is the first time that [65:05] this kind of method was trained on GPU. [65:07] You laugh, but this was research back [65:09] then. 3 weeks of time. [65:11] Um [65:13] Nowadays, I think I can train on [65:16] No, I probably still takes time, but not [65:17] weeks, hours. Um so, let's let's revisit [65:21] those problems that we were at before. [65:23] And actually, I [65:24] coming closer to time. So, [65:27] now a lot of methods post-2012 for this [65:29] problem of object detection and [65:30] classification, what a lot of methods [65:32] will do [65:33] is they will focus on having a [65:34] convolutional neural network output both [65:37] regions of interest and classifications [65:40] of those regions of interest. And so, [65:42] you can see here some examples of what [65:44] that might look like for like an output [65:45] sense. But, the way that you do this is [65:48] your loss function, rather than just [65:50] having a classification like score on [65:52] it, you have a classification term and a [65:54] regression term. Regression specifically [65:57] for the bounding box location in the [65:59] image. [66:02] And actually, now what a lot of method I [66:05] mean, it's actually quite interesting. [66:07] You Now, a lot of methods, they're [66:08] really shoving a lot of [66:10] problems into just one CNN. So, here [66:13] Uber has been using CNNs to detect [66:16] objects, track them, predict their [66:19] motion with a single network. [66:22] Okay, so that's three problems being [66:23] solved in one. [66:25] And you can see what some of the outputs [66:26] look like. And And if you if you guys [66:28] ever have the chance to like you know, [66:29] someone from Uber comes and gives a talk [66:31] and and you they're going to talk about [66:32] this kind of stuff. The videos they [66:33] make, oh, they're beautiful. I mean, [66:35] these are just single images, but uh [66:37] from a paper, but the the videos always [66:39] look really cool. [66:41] Semantic segmentation, you know, all of [66:43] that crazy graph math and all of the [66:45] optimization and probabilistic stuff, [66:47] throw it away. Who needs it? All now [66:50] that methods like some method uh [66:52] this was one of the first ones to do it, [66:53] but now a lot of similar work has been [66:55] done in the spaces you take some input [66:57] image, run it through some convolutional [66:59] stuff, and then have your output [67:03] the loss function, have your output be [67:06] to generate some image, which then tries [67:08] to look like the actual correct image. [67:11] Then so your loss might be, you know, a [67:12] pixel-wise, [67:13] I predicted that this pixel is part of [67:15] a, I don't know, the bunny or I guess [67:18] it's a cat. [67:21] No, it's a dog and a cat. Yeah, my bad. [67:23] 5%, right? So, [67:26] you might predict some pixel here as [67:27] being part of the, I don't know, the cat [67:29] and and then you get a loss signal like, [67:30] "No, no, no, it was actually part of the [67:31] dog." [67:32] And so this is a another example of a [67:34] research work that has one convolutional [67:35] network output just the result for you. [67:39] And I will say, [67:42] probabilistic graphical models that [67:43] graph-based segmentation work, it [67:45] actually does it is still used as like a [67:48] post-correction step, [67:50] but it's really there's a lot less [67:52] research effort into going towards [67:53] solely using them. [67:55] Um they still live on because of methods [67:57] like deep generative models which maybe [67:59] you'll hear about in some other course. [68:01] Um and but I will say some methods like [68:03] K-means or or mean shift which I [68:05] mentioned before, in computer vision, [68:07] almost never are they used for this this [68:08] kind of task. However, [68:10] both of them are very useful standalone [68:12] methods. That that problem of taking a, [68:15] you know, data set of points and trying [68:18] to cluster them, that is a fundamental [68:20] problem that many different fields, you [68:22] know, have and and use these methods to [68:24] solve. So they're definitely still used [68:26] in in those other um problems. [68:29] And in SLAM, right? These ridiculously [68:31] complex methods, now I will say a lot of [68:35] that engineering is still kept in [68:37] because, you know, this this global map [68:39] update and all of these different, you [68:40] know, keeping track of our your rotation [68:42] matrix and translations and all that, [68:44] you just have to do this. [68:45] But what a lot of methods will do [68:48] is they will use a convolutional neural [68:50] network in order to gain better [68:52] performance [68:54] in predicting maybe um [68:57] you know, the structure the 3D structure [68:59] of the world from an image. Rather than [69:01] having maybe something like structure [69:02] from motion, you might have structure [69:04] from motion augmented by a CNN. And you [69:06] can see here, you know, when when [69:08] measured just so if you look at some [69:09] office environment [69:11] and they were measuring the accuracy of [69:13] a depth prediction. So this is what like [69:14] perfect would look like. [69:16] A CNN, [69:18] not bad. [69:20] You know, 66% accuracy, I guess, on a [69:22] pixel-wise sense. [69:23] You know, on one of these older methods, [69:25] which was feature point-based, [69:28] 12%. I mean, remember these videos we [69:30] were watching, they're they're amazing, [69:31] right? They they real time and they look [69:33] really cool. And yet we have now these [69:35] methods which are like with with CNNs [69:37] like three to four times better in [69:39] accuracy. [69:40] And so [69:43] and again, the cool thing is when you [69:44] use CNNs, because of their architecture [69:46] and especially if you keep the number of [69:48] layers pretty small, [69:50] you can still have that real-time [69:51] performance. So you kind of you don't [69:53] trade that much to get this performance. [69:56] And that's why a lot of these and that's [69:58] why like [69:59] really a lot of work was changed by this [70:03] introduction of CNNs is because you now [70:05] have this sudden boost in performance [70:07] without too much of a loss in [70:10] in runtime, [70:12] in computational efficiency, maybe in [70:14] interpretability, yes. [70:17] And that's something that is being [70:18] addressed right now. [70:19] But at least in this first wave of kind [70:21] of results-oriented [70:22] works, this wasn't that much of a [70:24] concern. [70:26] Now, another thing that's really [70:27] important and something that we want to [70:28] kind of step back from just focusing on [70:30] problems and their solutions, [70:32] something that's really interesting [70:33] about robotics [70:34] is the heterogeneity of data that you [70:37] see in robotics. [70:39] So a lot of modern robotic platforms [70:41] have a really wide array of sensors and [70:44] specifically a wide array of sensor data [70:47] modalities that come from them. [70:49] And so for example, here, a Waymo [70:51] vehicle has something like one mid-range [70:54] lidar, which maybe gives them, I don't [70:55] know, tens of meters of of of [70:57] estimation. For short-range, very [71:00] high-accuracy lidars, for something like [71:01] maybe [71:02] parking, or maybe if you're kind of [71:04] coming close to another car, you might [71:05] want that information. [71:07] Five different cameras in order to see, [71:08] you know, to get a visual [71:10] uh view of, you know, maybe 360 or 270° [71:13] or something around the car. [71:15] And then they synchronize this data, and [71:16] then they have also, you know, some [71:18] other sensors in order to obtain [71:19] something like vehicle pose. So, that [71:20] could be GPS, that could be IMU, wheel [71:22] encoders, whatever. I'm I'm not even [71:24] including probably half of the sensors [71:26] there. [71:28] And the cool thing is is that [71:30] before 2012, you know, before this [71:32] advent of uh deep learning in computer [71:34] vision, [71:36] you would it would be quite hard, it [71:37] would be quite a task to integrate all [71:39] these different sensor modalities in [71:42] your classification, or in your, you [71:44] know, methodology that you might be [71:46] using to solve some specific problem. [71:49] But now, it's actually as easy I I say [71:52] easy in quotes. It's still There's still [71:53] effort to include this information, but [71:55] it's as easy as adding something like [71:57] extra vectors or extra data channels to [71:59] your image, [72:01] and then maybe altering your loss [72:03] function as well, and then having your [72:04] CNN just take that information into [72:06] account now, and generate better [72:08] performance better predictions now. [72:11] So, in summary, [72:14] so we we've covered a lot of topics [72:15] here, but in summary, the the key [72:17] messages that I want to to leave you [72:19] guys with [72:20] is before 2012, [72:24] a lot of methods in computer vision were [72:27] very engineering heavy. [72:29] And and published methods were a result [72:32] of like clever feature selection, [72:34] clever detection, interesting tricks for [72:36] feature description in certain [72:38] scenarios. [72:39] Whereas nowadays, you might have a [72:41] single convolutional neural network that [72:43] takes care of all of this for you, [72:46] and it optimizes for the best feature [72:48] and the best descriptors or whatever all [72:50] at the same time. Because of this, you [72:52] know, this [72:54] updating of your filters based on data. [72:57] And really, even more in general, this [72:59] new wave of deep learning has really [73:02] revolutionized not only computer vision, [73:04] but other, [73:05] you know, huge fields of research like [73:07] natural language processing, for [73:08] example. [73:10] And the best methods from these [73:14] approaches are now actually trickling [73:15] down to other related fields, such as [73:18] robotics. And And this is why we have [73:20] Now, you know, nowadays we have an [73:22] access to like a wide variety a wide [73:24] array of methods that have been seen to [73:26] work very well in computer vision, and [73:28] now we're actually seeing that they work [73:29] well in robotics and in other visual [73:30] areas where we might use this [73:32] information. Now, [73:34] yes, everything looks like it's all [73:35] working and everything, but I but I have [73:36] to say, there's still a huge amount of [73:38] research to be done. [73:39] There's still a lot unknown in this [73:41] space, especially in the areas of [73:44] interpretability. [73:45] Um, [73:46] but having said that, I hope you guys [73:48] now have a little bit of a sense of [73:49] what's going on nowadays and and what [73:51] happens maybe after the computer vision [73:53] side of the the course that we just [73:54] covered. [73:55] Uh, so with that, thanks for your time [73:57] and I'll be here if you guys have any [73:58] questions.