Pavone Models Camera Calibration to Resolve Robot Depth Ambiguity

Pavone建模相机标定以解决机器人深度模糊问题 单个相机传感器需要数学标定和多视角三角测量，因为透视投影会使三维深度沿无法观测的光线发生坍缩。 Marco Pavone教授详细解析了将三维世界坐标映射到二维相机像素所需的数学流程，深入探讨了透视投影、传感器离散化以及齐次坐标。他建立了结合内参和外参刚体变换的完整相机投影模型。本次课程介绍了利用已知目标对应点进行线性相机标定的方法，通过奇异值分解（SVD）求解由此产生的约束优化问题，并利用RQ分解提取参数。最后，Pavone探讨了镜头畸变等现实世界中的挑战，指出了单目图像固有的深度模糊问题，并介绍了立体视觉三角测量。 相机投影被建模为一个端到端的三维到二维流程，通过外参（旋转与平移）和内参（焦距缩放与光学中心偏移）将世界坐标映射为二维像素坐标。 (0:05) 将笛卡尔坐标提升至齐次投影空间，使得非线性的深度Z透视除法能够被隔离到最后一个归一化步骤中进行。 (12:12) 线性相机标定至少需要6个三维到二维的点对应关系（从而为11个自由参数提供12个方程），并在向量化投影矩阵的单位范数约束下通过SVD求解。 (58:41) 完整的3x4投影矩阵M可通过RQ分解拆解为上三角内参矩阵K和正交旋转矩阵R，而单目深度模糊则需要通过立体三角测量或运动恢复结构（SfM）来实现三维重建。 (65:04)

Prof. Marco Pavone details the mathematical formulation of camera projection models and the calibration procedures required to invert two-dimensional digital images into three-dimensional robotic spatial perception. Unified Projective Geometry and Camera Formulation (38:57) Pinhole camera projections map physical 3D points $(X, Y, Z)$ to 2D image coordinates via perspective scaling by focal length $f$. Accounting for corner-indexed sensor boundaries $(x_0, y_0)$ and non-square pixel densities ($k_x, k_y$) yields the lumped intrinsic parameters $\alpha = k_x f$, $\beta = k_y f$, $u_0 = k_x x_0$, and $v_0 = k_y y_0$. Because dividing by depth $Z$ makes the perspective projection nonlinear in Cartesian coordinates, the formulation embeds coordinates into higher-dimensional homogeneous spaces, isolating perspective division to a final scalar normalization step. Rigid-body extrinsics transform global world coordinates $P_W$ into camera-frame coordinates $P_C$ using an orthogonal rotation matrix $R \in \mathrm{SO}(3)$ and a translation vector $T \in \mathbb{R}^3$. Combining the $3 \times 3$ upper-triangular intrinsic calibration matrix $K$ with the $3 \times 4$ extrinsic matrix $[R \mid T]$ yields the complete projection equation $p_h = K [R \mid T] P_h = M P_h$, mapping 4-element homogeneous world vectors directly into 3-element homogeneous pixel coordinates. Calibration Linear Systems and Constrained Optimization (58:41) Because manufacturing variations prevent intrinsic parameters from being derived reliably from factory datasheets, cameras require experimental calibration. By observing an engineered target (such as a 3D or planar checkerboard) with known spatial fiducials $P_{w,i}^h$, practitioners record matching pixel observations $(u_i, v_i)$. Expanding the projection equation $M P_{w,i}^h$ into row vectors yields two linear constraints per point: $u_i (m_3 P_{w,i}^h) - m_1 P_{w,i}^h = 0$ and $v_i (m_3 P_{w,i}^h) - m_2 P_{w,i}^h = 0$. While the matrix $M$ contains 12 scalar entries, projective scale invariance leaves 11 independent degrees of freedom, mirroring the sum of physical camera parameters (5 intrinsics including skew, 3 rotation angles, and 3 translation components). A minimum of 6 point correspondences is mathematically necessary ($2n > 11$). To account for measurement noise, the overdetermined system $\tilde{P} m \approx 0$ is framed as a constrained least-squares minimization $\min_{m} \|\tilde{P} m\|$ subject to $\|m\| = 1$. Solving via Lagrange multipliers reveals that $m$ is the eigenvector corresponding to the smallest eigenvalue of $\tilde{P}^T \tilde{P}$, calculated in practice using the right singular vector associated with the smallest singular value via Singular Value Decomposition (SVD). Parameter Recovery via RQ Factorization (65:04) After estimating the composite projection matrix $M$, its parameters are decomposed by partitioning $M = [A \mid b] = [K R \mid K T]$, where $A$ is a $3 \times 3$ block and $b$ is a $3 \times 1$ translation column. Disentangling the upper-triangular intrinsic matrix $K$ from the orthogonal rotation matrix $R$ is achieved through RQ matrix decomposition ($A = R_{\text{tri}} Q_{\text{orth}}$). Pavone notes that software naming conventions can cause confusion: algorithmic output $R_{\text{tri}}$ corresponds to intrinsic matrix $K$, while $Q_{\text{orth}}$ corresponds to camera rotation $R$. Extrinsic translation is subsequently recovered through $T = K^{-1} b$. Pavone emphasizes that real lenses deviate from the idealized pinhole model by introducing radial and tangential optical distortions, such as barrel distortion in wide-angle lenses and pincushion distortion. Because small manufacturing variations alter optical alignments, calibration must be executed per individual physical sensor unit rather than once per sensor production model. Resolving Single-View Ambiguity through Multiview Geometry (71:17) Monocular vision faces an inherent limitation: an infinite line of points along a projection ray passing through the optical center maps to the exact same image pixel, making absolute depth unobservable from a single vantage point without high-level semantic priors. While depth from focus can recover depth via the thin lens equation ($1/f = 1/Z + 1/z_v$) by identifying peak feature sharpness, iteratively adjusting focal lengths is mechanically cumbersome for dynamic robotics. Practical perception systems instead deploy stereo vision or structure from motion (SfM). Stereo systems use two calibrated cameras separated by a known baseline to triangulate scene points. Because sensor noise and pixel quantization prevent back-projected optical rays from intersecting perfectly in physical space, 3D point estimation is framed as an optimization problem minimizing total reprojection error across both image planes. However, this process remains fundamentally constrained by the correspondence problem—the requirement to reliably match distinct features across disparate viewpoints. The lecture formalizes the linear algebra required to extract calibrated camera matrices from known fiducial targets, but leaves practical non-linear lens distortion modeling and advanced stereo correspondence algorithms to subsequent advanced lectures.

Marco Pavone 教授系统阐释了机器人三维空间感知的核心相机模型，指出必须通过齐次坐标与参数标定建立物理世界到像素平面的线性化投影映射，并强调单目视觉存在固有的深度二义性，需依赖双目立体视觉三角测量等方法实现三维重建。 投影流水线与齐次坐标线性化 (0:05) 机器人进行空间几何重构的前提是明确三维物理点到二维像素的映射机制。该过程分为三个阶段：将世界坐标系下的三维点通过刚体变换转换至相机坐标系，利用针孔模型透视投影映射至连续图像平面，最后通过平移与像素缩放因子离散化为传感器网格上的像素坐标 (u, v)。由于相机传感器像素可能为非正方形，需分别引入水平与垂直方向的比例因子 kx 与 ky，连同焦距 f 和主点偏移 (x0, y0)，整合为四个内参参数：α = kx f、β = ky f、u0 = kx x0 以及 v0 = ky y0。由于透视投影涉及除以场景深度 Z，标准笛卡尔坐标下的映射呈现非线性，无法直接写为矩阵乘法形式。计算机视觉通过引入齐次坐标，将二维或三维点提升到高维等价类空间，使透视除法与刚体变换均能表示为紧凑的线性矩阵乘法，仅在最终反齐次化时处理深度除法。完整的相机投影模型由此表述为 ph = K [R | T] Ph，其中 K 为 3×3 的内参标定矩阵，外参 [R | T] 包含 3×3 正交旋转矩阵与 3×1 平移向量，最终组合成 3×4 的全投影矩阵 M。 外参刚体变换与视觉里程计应用 (27:22) 在相机坐标系与世界坐标系原点重合的理想情况下，通过两坐标系基向量的点积运算可直接构建方向余弦矩阵，即行列式为 1 且各列相互正交的旋转矩阵 R。当原点发生偏移时，任意空间点需叠加相机原点至世界原点的平移向量 T，构成标准刚体变换 PC = R PW + T。该几何关系不仅是正向几何投影的基础，也是移动机器人自主导航的核心工具。在野外机器人与深空探测任务中，自主系统往往反向利用该方程：通过连续图像序列中的特征匹配反推相机在前后时刻的 R 与 T，从而在没有外部定位基础设施的情况下推算自身位姿与运动轨迹。这一技术即为视觉里程计，已被实际应用于 NASA 的火星车地面自主导航中。 相机标定约束与矩阵 SVD 分解 由于镜头制造公差与装配误差，相机出厂标称参数精度不足以支撑精确的三维重构，因此必须在投入使用前进行传感器标定。标定通常基于已知几何尺寸的平面或立体棋盘格靶标，利用精确已知的世界坐标点与图像像素点建立匹配对应。每个点对应关系可分解为两组线性约束：ui (m3 Pw,i^h) - m1 Pw,i^h = 0 与 vi (m3 Pw,i^h) - m2 Pw,i^h = 0，其中 m1、m2、m3 为投影矩阵 M 的各行向量。对于包含 11 个独立自由度的 3×4 矩阵 M（12 个参数扣除齐次尺度因数），理论上至少需要 6 组对应点（提供 12 个方程），实际工程中则采集大量超定数据以抑制噪声。为避免平凡解并消除任意尺度因数 λ 的影响，标定算法施加参数范数约束 ||m|| = 1，将问题转化为带约束的最小二乘优化 min ||P_tilde m||。利用拉格朗日乘子法求解，最优解对应于 P_tilde^T P_tilde 的最小特征值所关联的特征向量，在数值计算中通过对系数矩阵 P_tilde 进行奇异值分解（SVD）提取右奇异向量得出。随后，借助 RQ 分解将估计出的矩阵块 A 拆分为上三角内参矩阵 K 与正交旋转矩阵 R，并求解平移向量 T = K^(-1) b。 单目深度二义性与双目立体三角化 (68:32) 即便内参和外参完全已知，单幅二维图像像素也无法唯一确定三维世界点。从投影中心穿过图像像素的反向投影射线上的所有空间点，均会落在图像平面的同一位置，导致沿视线方向存在固有的深度二义性。人类在单目图像中判断尺度依赖于熟悉物体与语义先验，但在几何层面纯粹的尺度信息已经丢失。为突破这一物理瓶颈，工程中主要采用双目立体视觉或运动恢复结构（SfM）等方案。双目系统利用已知基线距和相对位姿的两台同步相机同时观测目标，理论上通过两条反向投影射线的空间几何交点即可通过三角测量恢复深度。然而在真实工况下，由于传感器噪声、像素量化误差以及标定残差，两条射线往往无法在空间精确相交。工程实践通常将三维重构建模为最小化重投影误差的非线性优化问题，即寻找一个最优的三维点 Q，使其在左右相机平面上的投影与实际观测像素之间的偏差最小。立体视觉落地的关键技术瓶颈在于立体匹配，必须在两幅图像间准确找到极具辨识度的同名特征点。 相机针孔模型为空间定位提供了严谨的线性代数框架，但在实际工程应用中，真实透镜引入的径向畸变与切向畸变会打破理想线性假设，必须对单台传感器进行独立校正；此外，立体视觉中双目特征对应的高精度匹配与极线约束计算，构成了从理论投影迈向高鲁棒性三维场景重建的核心挑战。

[73:32] Prof. Marco Pavone: The depth from focus idea is fairly simple but very cumbersome. So most of the times this is not used.

[0:05] All right. So today we're going to [0:07] continue our discussion about how to [0:10] establish a relationship between points [0:13] in the real world and their projection [0:16] projection onto the image plane. [0:19] The reason why we want to establish this [0:21] uh uh relation is that um once we get [0:26] the images of the environment, we want [0:30] to use this relationship to go in the [0:32] opposite direction. Meaning once we have [0:34] images, how do we reconstruct the real [0:36] world leveraging these relationships? [0:40] So the idea as discussed last time is to [0:43] break down the uh projection of a point [0:47] in the real world into the point on the [0:51] image plane into three steps. First we [0:55] want to understand how to map a point P [1:00] expressed with respect to the camera [1:02] reference frame onto the image plane. [1:06] Second, we want to understand how to [1:08] then transform that projected point into [1:12] coordinates with respect to the uh [1:15] camera system in terms of pixel [1:17] coordinates. And finally, we want to [1:19] find the relation between the point P [1:21] when expressed with respect to the [1:23] camera reference frame and the same [1:24] point where when expressed with respect [1:27] to a global reference frame. Yeah, [1:30] >> I haven't started yet. I have a [1:31] question. [1:32] >> I have a more general question. Why are [1:33] we why are we spending um so much time [1:36] learning about the ways to like invert [1:39] these camera modern cameras already like [1:41] invert the the picture for us like [1:43] display correctly as like a pixel so [1:45] that we can interact with [1:46] >> well so you get the pixel absolutely but [1:49] then once you have those images you want [1:50] to reconstruct the uh environment and [1:54] that procedure we require the [1:55] relationship that we're going to cover [1:57] first of all second of all uh in order [2:00] for this procedure to be carried out. [2:03] You need some as we will see some [2:05] intrinsic parameters for the uh camera [2:08] in terms of a focal length, pixel [2:10] resolution and so on. And you have specs [2:12] but those specs usually are not due to a [2:17] number of reason manufacturing reasons [2:18] and so on are not precise enough to [2:21] carry out this process. Which basically [2:24] means that every roboticist before [2:26] starting using a camera for uh [2:28] environment construction has to do a [2:31] calibration process for a camera or for [2:34] any sensor. And trust me calibration is [2:36] one of the most annoying and tedious [2:38] processes they could do and I can share [2:41] you with you offline a number of [2:43] stories. For example, when I was [2:44] calibrating cameras when I was working [2:46] for NAZA, [2:48] brutal process. So I would like to give [2:52] you some tips about how to do this so [2:55] that if in the future you you will have [2:57] the task of reconstructing environment [2:59] or calibrating a camera, you don't get a [3:02] shell shock as it did when I went to [3:04] NAZA. Pretty much cool. All right. So I [3:08] hope I motivated that and this will [3:10] become clear later on today. So we [3:13] already know how to do uh the first step [3:15] how to map a point in the 3D world whose [3:20] coordinate are expressed with respect to [3:22] the camera reference frame into a point [3:26] on the image plane. [3:29] And the mapping exploits the projection [3:31] equation that we derived last time [3:34] whereby the xcoordinate in the uh image [3:39] reference frame is equal to f the focal [3:43] length the distance between the pinhole [3:45] and the camera center. The xcoordinate [3:50] with respect to the camera reference [3:52] frame and the zcoordinate with respect [3:53] to the camera reference frame. So this [3:55] is what we already discussed last time. [3:57] Similar equation holds for the [3:59] y-coordinate. [4:01] We don't have a zcoordinate for small p [4:03] because a zcoordinate would always be [4:05] equal to the focal length. The image [4:08] plane is at fixed distance from the [4:10] camera reference frame. [4:12] Now [4:14] um now that we have the coordinates for [4:17] the point in the image plane, we want to [4:20] relate those coordinates to the [4:21] coordinates that a camera really uses, [4:25] which basically are the pixel [4:27] coordinates. In order to do so, we have [4:29] to account for two facts. Usually the [4:32] coordinate system that the camera um [4:36] uses is not centered at the center of [4:38] the image plane but is actually centered [4:41] at one of the uh corners [4:44] at the idea that basically you read [4:46] pixel in a sort of row and a column uh [4:49] format. So just to make things clear by [4:53] looking at image plane from this side [4:56] let's assume that this is the reference [4:59] system that we have used for the image [5:00] plane the actual reference system that a [5:03] camera would use is situated at a corner [5:06] say bottom right could also be up left [5:08] and so on but usually it's at the corner [5:10] which means that uh the origin is here [5:13] and then the coordinate changes as you [5:15] go farther and farther from the corner I [5:17] mean it's just a convention but this is [5:19] what it is. So how do we go from a [5:22] reference system that is centered at the [5:24] center of the image plane to a camera [5:26] system that is centered at one of the [5:28] corners? Well, let x0 y 0 be the [5:33] coordinate of the origin of the camera [5:35] reference system or the image plane [5:37] reference system with respect to the [5:39] camera reference system. Then the [5:41] mapping between uh a point expressed [5:47] with respect to the image reference [5:49] frame and a point expressed in terms of [5:52] the camera reference frame is simply [5:55] related by a translation and the [5:57] translation is given by the coordinates [5:59] of the center of the uh reference frame [6:02] in the image plane. Yes, [6:04] >> sorry a little bit of a basic question [6:05] but why is the coordinate plane of the [6:07] >> camera centered on the bottom or center [6:09] of camera? Yeah. Can I Why is it [6:12] centered in the bottom versus the top? [6:14] Because iron pixels usually centered on [6:17] the top right corner. [6:18] >> Oh yeah. The important thing is that [6:19] usually it is at the corner. [6:21] >> Okay. [6:21] >> Here, just as an example, I'm placing it [6:24] here, but you're right often times is at [6:26] the top. [6:26] >> The important thing is that you need to [6:28] apply a translation in order to relate [6:31] coordinates with respect to the [6:32] reference system we've been using so far [6:35] uh in terms of the reference system that [6:36] the camera really uh uses. For example, [6:39] if you have a point that according to [6:43] our previous reference system is uh [6:45] here, then the coordinates in the camera [6:48] reference system will be x0 and y0 [6:51] because this will be zero and this will [6:52] be zero. Does this make sense? [6:54] >> Yes. [6:55] >> Any questions here? So, we've just [6:58] applied the translation in terms of the [7:00] fn system. [7:03] >> Yeah. [7:04] >> What would be the names of the frame so [7:06] I can just have it more clearly? So the [7:08] new [7:08] >> I will call this image plane reference [7:11] frame and this is the um [7:16] true camera reference frame. I mean it [7:18] gets a little bit confusing because this [7:19] is then is the camera reference frame [7:21] and then we have a word frame. In [7:23] reality, the three that really matters, [7:25] well, the two that really matters are [7:27] the word frame and the camera frame [7:31] because of course you read pixels in [7:34] terms of the camera frame and you [7:39] reconstruct points in the real world [7:41] with respect to a u um external [7:44] reference frame. So these are just [7:47] intermediate reference frames that [7:49] simplify the derivation of these [7:51] relations. [7:53] Does it make sense? [7:58] Okay. So the last step is that [8:01] so far we have been considering uh [8:05] continuous values for the coordinates [8:07] but in reality a camera reasons in terms [8:11] of pixel coordinates. So discrete [8:13] coordinates for uh the points in the [8:15] image [8:18] which means that basically we have to [8:20] discretise [8:21] um the uh values that we have computed [8:25] so far in order to translate them into [8:27] pixel coordinates. In particular the [8:29] idea is to consider two numbers x kx and [8:33] k y that represent the number of pixel [8:36] per unit of distance in an image. [8:41] And so a pixel that has c say that we [8:44] have three pixel per cm. I mean of [8:47] course you have many more but say that [8:48] we have three pixel per cm [8:52] and say that uh x tilda is equal to one [8:56] 1 cm then the pixel coordinate is equal [8:59] to three. [9:03] So we basically multiply by the number [9:06] of pixels per unit of distance and then [9:08] we apply rounding [9:11] make sense. [9:13] So for example if the x coordinate is 2 [9:17] cm then the pixel coordinate will be [9:20] six. [9:22] And of course um there is in a way a [9:26] loss of information because um if we [9:29] have three pixels per centimeter then [9:33] all those pixels that are within 33 cm [9:36] are mapped to the same pixel value which [9:40] is standard discretization error. [9:45] So with this understanding then a point [9:49] in the 3D world expressed with respect [9:52] to the camera reference system this [9:55] reference system here [9:57] is then mapped into pixel coordinate u [10:00] and v. U will be the horizontal pixel [10:03] coordinate. V will be the vertical pixel [10:05] coordinate as kx [10:08] times the projection plus kx x0 where x0 [10:13] and y0 again are the coordinates of the [10:17] reference system in the image plane with [10:19] respect to the reference system that the [10:20] camera uses. [10:22] >> Yeah. [10:23] >> Is it common that kx and k y are the [10:25] same or they all values? often times [10:27] there are different values and often [10:28] times you have to uh estimate that from [10:32] data through a process referred to as [10:33] calibration that we're going to discuss [10:35] later. [10:37] So that's why uh here we refer to kx and [10:41] ky differently as opposed to just say k. [10:45] All right. So let's just uh call this [10:50] product alpha, this product u0, this [10:54] product v 0 and this product beta. So we [10:57] can accurately express this [10:59] transformation according to the [11:01] equations here. In general, alpha, beta, [11:05] u 0 and v 0 are not quantities that you [11:08] know or that you know in a way that is [11:10] accurate enough which basically will [11:11] lead us to the explanation of the [11:13] calibration procedure that allows you to [11:15] infer those quantities from data. [11:19] Okay. So one thing here to note is that [11:23] a projective transformation as already [11:26] out emphasized the last time is a [11:28] nonlinear transformation. nonlinear [11:31] because [11:34] the zcoordinate is divided. So we cannot [11:38] really express this transformation uh as [11:41] the product of a matrix times a vector [11:43] which from a mathematical standpoint is [11:45] it is an inconvenience. [11:48] So uh um what well known trick in [11:52] computer vision is to then consider uh a [11:56] new coordinate system referred to as a [11:58] homogeneous coordinate system whereby [12:01] one is able to transform the nonlinear [12:04] projection transformation into linear [12:06] ones. The nonlinearity of course does [12:10] not magically disappear. it disappears [12:12] in that the nonlinearity is then taken [12:15] into account when you transform back [12:17] from a homogeneous coordinates to [12:19] nonhomogeneous coordinates. In [12:21] particular, what is a homogeneous [12:23] coordinate system? So let's assume that [12:26] you're given a point xy [12:31] in R2. [12:34] Then we associate [12:36] each point XY in R2 [12:40] to a point in R3 [12:44] written as X Y1 times lambda. So it's [12:48] not a one to one mapping is a one to [12:50] many mapping. So to each point in R2 we [12:54] associate the points in R3 that are [12:57] equal to XY and one. So we we add one [13:00] dimension and they um are parameterized [13:06] by a scalar lambda. [13:11] So this seems very awkward and the [13:12] reason why we do this will become clear [13:14] in a second. So for now we're just [13:16] establishing a correspondence between [13:19] points in R2 to a class of points in R3. [13:22] a class because we have a family of [13:25] points in our three parameterized by [13:27] this coefficient lambda. [13:31] So now once you have so this will be [13:35] your original point in non-homogeneous [13:37] coordinates. This is the uh point [13:41] represented in the uh homogeneous [13:44] coordinates. [13:50] Okay. So we know how to go from [13:51] nonomogeneous to homogeneous [13:53] coordinates. How do we go back from um [13:57] homogeneous coordinates to [13:58] nonhomogeneous one? [14:01] Well, so as you can see from here, if [14:04] you're given the vector in nomogeneous [14:06] coordinates, how do you reconstruct the [14:07] ver the So if you're given a vector in [14:09] homogeneous coordinates, how do you [14:11] reconstruct the vector in homogeneous [14:13] coordinates? [14:16] >> By the value in the [14:17] >> exact so basically this will be lambda x [14:21] lambda y lambda. So if I give you this [14:25] point in order to find X, what you do is [14:30] you divide the first component by the [14:32] third one [14:34] and the second component by the third [14:37] one. Lambda Y over lambda. [14:42] Which basically means that if you're [14:43] given in general a point x y w in r3 [14:52] then it's a nonhomogeneous [14:54] representation will be x over w and y / [14:58] w so that we have a correspondence [15:01] between points in a nonomogeneous [15:04] representation and points in a [15:05] homogeneous representation. [15:08] The important thing to note is that here [15:10] we have an equivalence equivalent [15:12] equivalency class of points that [15:14] represent a sigma point in R2. [15:22] The same thing can be done in R3 for [15:24] what matters can be done for any vector [15:26] in Rn can be transformed into [15:28] homogeneous coordinates in Rn + one. [15:32] So this just a mathematical construction [15:34] right now. I mean there are very many [15:36] good reasons why one would consider uh [15:40] such a such uh new coordinates. The [15:43] reason why we consider them is by [15:46] considering the uh following. [15:50] Let's consider the homogeneous [15:52] representation of uh the point in the 3D [15:57] world as represented with respect to the [16:00] um camera reference frame. So PC [16:07] so we have PC [16:11] which is given by XC YC ZC [16:17] so it's homogeneous representation [16:20] that I refer to as a superscript [16:23] um H [16:27] will be XC [16:30] YC [16:32] ZC C lamb sorry 1 [16:38] time lambda. [16:42] So now let's introduce the matrix [16:49] um alpha [16:52] 0 [16:54] is 0 is the matrix that we have over [16:57] there and we have a zero. Then we have a [17:00] zero beta v 0 and then we have zero and [17:05] then we have 0 0 1 0 0 1 0. [17:13] So this matrix as we will see later on [17:16] is referred to as the camera matrix K. [17:19] And let's multiply this by the [17:21] homogeneous representation of [17:25] the 3D point times lambda. [17:30] So this [17:36] when we carry out [17:39] the product, [17:42] what we get is [17:45] alpha xc [17:49] plus 0 plus u 0 uh um sc. [17:55] Then we have uh beta [17:59] yc [18:00] plus v 0 s and then we have z [18:07] um zc [18:09] times lambda. [18:13] Okay, [18:15] which is what we have there. We don't [18:17] have the lambda and I'll tell you in a [18:19] second why that is the case. [18:22] All right. So then [18:27] let's divide [18:31] this vector by the third component. [18:35] So what we get is and we eliminate the [18:39] third component. So what we get is the [18:41] vector alpha xc over zc plus u 0 and [18:49] uh beta yc over zc plus v 0. [18:58] So you should recognize that this is [19:00] really the projected point P in the um [19:05] pixel reference frame [19:08] which means that this was the point P [19:14] in the um pixel reference frame [19:18] expressed in homogeneous coordinates. [19:22] So what is the point of all of this is [19:24] that when we express both the 3D point [19:30] in nonomogeneous coordinates and its [19:32] projection in non sorry when we express [19:35] both the 3D point in homogeneous [19:37] coordinates and its projection in [19:39] homogeneous coordinates the projection [19:41] transformation is just a linear [19:43] transformation in matrix times a vector [19:46] which as we will see simplifies a number [19:48] of operations including the process of [19:50] camera calibration. [19:53] Where did that nonlinearity go? Well, [19:56] the nonlinearity went into the process [20:00] of transforming back homogeneous [20:03] coordinates into normogenous ones. So [20:05] the idea is to lift everything to [20:07] homogeneous coordinates do all our [20:09] computation over there treating now that [20:13] everything is linear and once we're done [20:14] with that we go back to the true [20:17] coordinates and nonomogeneous ones and [20:20] at that time account for the [20:21] nonlinearity so it's a clever way of [20:23] exploiting linear transformations in the [20:26] nonlinear setting [20:28] does that make sense [20:30] so the reason that here I didn't put [20:32] lambda is that uh Since homogeneous [20:36] coordinates are an equivalency class, we [20:39] can just pick pick one example. So just [20:41] fix lambda equal one instead of having [20:43] to train this lambda. As you see, if I [20:46] pick it picked lambda one here, I will [20:48] have obtained the same result. So just [20:51] for mathematical convenience. [snorts] [20:53] Now this matrix here, [20:58] where is it? Oops. [21:03] This matrix here, [21:12] this 3x3 matrix is referred to as the [21:15] camera matrix as you and as you can see [21:18] it contains some intrinsic parameters of [21:22] a camera. So parameters that [21:24] characterize the manufacturing [21:28] um characteristics of a camera. Um in [21:31] particular [21:33] um uh alpha and beta for example are [21:35] related to the uh focal length and the [21:38] pixel scaling and u 0 v 0 is related to [21:41] the position of the central reference [21:44] system in the image plane. [21:47] These are parameters that in general [21:50] need to be estimated from data through a [21:52] calibration procedure that we're going [21:53] to discuss next. [21:56] So then we can write the uh projection [21:59] transformation by leveraging homogeneous [22:02] coordinates in a linear fashion whereby [22:05] we have a matrix that accounts for the [22:08] camera matrix and um [22:13] column of zeros time the homogeneous [22:16] representation of the point in the 3D [22:19] world. [22:21] Yeah, [22:23] >> maybe off the topic, but what does [22:25] homogeneous mean in this context? [22:28] >> Homogeneous, so that um homogeneous [22:32] coordinates have a much richer theory, I [22:35] would say a much broader application in [22:38] terms of projective geometry. For the [22:40] sake of this class, we just use it as a [22:43] convenient mathematical trick in order [22:45] to express the projection transformation [22:47] that is nonlinear into a projection [22:51] transformation that is a linear in a [22:53] lifted space where we have augmented the [22:56] coordinates with one additional [22:58] dimension. So view it for for the sake [23:01] of this class as a mathematical trick. [23:03] In reality there are many more [23:04] application uh for this concept. [23:09] Cool. [23:11] In reality, this and I'm not going to go [23:15] too much into the details, but this um [23:20] uh number here sometimes is not zero and [23:23] this refer to gamma and represents the [23:26] so-called skew parameter. Basically it [23:28] accounts for the fact that sometimes the [23:31] axis of the uh uh reference frame that [23:36] the camera uses are not exactly [23:38] perpendicular. [23:40] Usually uh this is a quite small number [23:44] for good cameras. So for the sake of [23:46] this class we will treat that gamma as [23:48] zero. But just for your knowledge in [23:52] general one should also consider this [23:54] skew parameter in the calibration [23:56] process. [24:01] All right. So now we can move to the [24:04] topic of today that is to complete this [24:08] projection procedure by also accounting [24:11] for the transformation from uh the [24:14] coordinates expressed in a word frame [24:17] with respect to the coordinates in 3D as [24:20] expressed in the camera reference frame. [24:24] Any [24:26] [clears throat] [24:29] questions in the meantime? [24:36] I'm going a little bit lower than usual [24:38] because sometimes this is with this [24:41] concept of homogeneous coordinates is a [24:43] little bit um confusing at first. [24:52] >> Yep. So uh the camera matrix assumes a [24:56] single focal length but if you're [24:58] refocusing a camera that focal length [24:59] changes right so you need to recomputee [25:02] the entire matrix every time you're [25:03] doing that [25:04] >> in principle if you change focal length [25:06] in a way that cannot be characterized in [25:09] other ways. Yeah. [25:12] So today we're going to uh [25:16] continue the procedure of establishing a [25:18] relationship between a point in 3D and [25:20] its projection in image plane. But most [25:22] importantly, we're going to talk about [25:24] how to estimate from data the numbers [25:28] that appear in the camera matrix and [25:30] we'll see whether it's useful. So as [25:33] before the setup is that we have a point [25:36] in 3D [25:38] and we want to understand how it is [25:40] mapped into uh pixel coordinates. [25:44] Once again to make sure that you don't [25:46] get confused I refer to this reference [25:47] system as the word reference system. [25:50] this reference system as the camera [25:52] reference frame. This reference the [25:54] reference system as the reference system [25:56] in the image plane and the reference [25:59] system in terms of pixel as the pixel [26:01] reference frame. [26:03] At the end of the day, what we're [26:04] interested is to go from the word [26:05] reference frame to the pixel reference [26:07] frame. We introduced two more reference [26:10] frames just to is the uh derivation. [26:15] So the last step that we need to account [26:17] for is how do we transform coordinates [26:20] from the word reference frame to the [26:23] camera reference frame. Once we do that [26:25] we pl we put all this transformation [26:28] together and we have the end to end [26:30] transformation from PW to pixels. [26:35] All right. So to some of you this might [26:38] be well known but to some others may not [26:41] be well known. So we're going to derive [26:45] the relation [26:47] that relates uh the coordinates of a [26:50] point in two different reference [26:52] systems. [26:59] So let's assume first for simplicity [27:02] that so here in this plot [27:06] um this is the camera reference system [27:08] and this is the word reference system. [27:10] So let's assume first for simplicity [27:12] that the origin of the camera reference [27:16] system and of the water reference system [27:18] coincide. [27:20] So in other words this vector t is equal [27:22] to zero. So we're in the situation [27:25] whereby this will be the [27:30] camera reference system [27:33] and then let's see if I can mimic the [27:36] drawing there. [27:42] This will be IW. [27:45] This will be JW and this will be KW. And [27:48] just to be clear, I put underscore to [27:51] denote that these are vectors. So these [27:55] are the unit vectors along each of the [27:57] axis. I is the unit vector in the along [27:59] the X- axis. J is the unit vector along [28:02] the Y axis and so on. [28:04] So we're given a point Q. So this is the [28:07] origin. [28:08] say Q [28:12] and we express this point Q with respect [28:15] to two different uh different coordinate [28:18] system the word reference system and the [28:20] camera reference system. We want to [28:22] understand how these coordinates are [28:23] related. So if I give you the um point [28:29] expressed in terms of the word reference [28:31] frame, how do I map those coordinates [28:33] into the coordinates as expressed in [28:35] terms of the camera reference frame? [28:40] Okay, that's what we want to do. [28:43] All right, so [28:46] let's translate mathematically what we [28:48] want to do. So Q, which is also a [28:51] vector. So let me put it here. [28:54] So I said for simplicity we're first [28:56] considering the case where the origin of [28:58] the board reference system coincide with [29:00] the origin of the camera reference [29:01] system. We'll then consider the case [29:02] where they are separated. So we can [29:05] write Q as Q CX [29:10] I plus Q C Y [29:15] plus Q C Z [29:18] uh K. [29:22] C means that this is the coordinate with [29:25] respect to the camera reference frame. X [29:27] means that this is the coordinate with [29:29] respect to the X-axis [29:32] where I represent the unit vector of [29:35] along the X-axis for the camera [29:37] reference frame. [29:39] Then I can also represent a Q with [29:42] respect to a different coordinate [29:43] system. So the word reference system. So [29:45] in that case I will have a Q C W I W [29:52] plus Q [29:55] or uh sorry Q W X I W plus Q WY [30:02] JW plus a Q W Z KW. [30:10] Okay. So, so far I've done nothing but [30:15] writen this point into different [30:17] reference systems. Now, Q is the same [30:19] point [30:21] which means that [30:25] this vector has to be equal to this [30:27] vector. So this means that this is also [30:30] equal to Q CX I plus Q C Y J plus Q C Z [30:42] K. [30:46] So if I give you [30:49] the um [30:52] coordinates of the point Q with respect [30:54] to the world reference frame and I tell [30:57] you in in the sense that I will make [30:59] more explicit in a second what is the [31:01] relation between I and IW J and JW and K [31:06] and KW. So basically how the camera [31:09] frame is rotated with respect to the war [31:11] reference frame. How will we find the [31:13] relation between the coordinates and [31:15] camera reference frame and the [31:16] coordinates and the world reference [31:18] frame? [31:25] So the deal or the trick is to multiply. [31:29] So let's assume that we want to make [31:33] find a relation between the QCX and the [31:36] coordinates in the world reference [31:37] frame. So what we do is we multiply. So [31:43] this is an inequality. So this is equal [31:46] to this. We multiply both sides by i. So [31:51] if I multiply this vector by i, multiply [31:55] them in a dot product. What is the [31:57] result? [32:01] >> Yes. Why? [32:02] >> Because all the other components go away [32:04] because they're [32:06] >> right. So y * k is zero because y is [32:10] orthogonal to k. We selected the [32:12] reference system in orthogonal basis. y [32:15] * j is 0 and y * i is equal to 1 because [32:18] i is the unit uh vector in the x-axis [32:21] direction. So we [snorts] have Q CX [32:25] equal to Q W X I W * I plus a Q W Y uh J [32:38] W * I plus plus Q W Z [32:44] K W [32:46] * I [32:49] okay and then I can Do the same for the [32:53] other coordinates. I will multiply this [32:55] time by j. So Q uh C Y is equal to Q WX [33:03] I W * J plus Q oh uh yes plus Q W Y J Y [33:15] * J plus plus Q WZ KWY * J and then the [33:23] same will be for QCZ [33:25] this time multiply by K. [33:31] Okay. [33:33] So I can then [33:35] represent these equations more compactly [33:39] by [33:41] writing a Q CX [33:45] Q C Y Q C Z [33:49] as some matrix that I denote as R times [33:53] Q C [33:57] qwx [33:59] Q WY QZ [34:03] So what is this matrix R? Well, we just [34:06] read the rows from here. And that's [34:10] exactly what you have [snorts] over [34:12] there. [34:17] This just pattern matching this with [34:19] that. [34:21] So for example, the first component will [34:24] be so first element of the first row is [34:26] IW * I. second element of the first row [34:30] is JW * I and so on. [34:36] So this is to say that if we know all of [34:38] these dot products if you give me any [34:41] point [34:43] if you give me the coordinates of any [34:45] point in the world reference frame by [34:48] doing this multiplication I can [34:49] immediately find the coordinates in the [34:51] camera reference frame. In particular, [34:54] knowing these dot products implies that [34:56] we know what are the relative angles [34:59] between the um basis vectors for the two [35:04] coordinates reference systems which [35:05] basically means that we know what is the [35:08] orientation of one reference system with [35:09] respect to the other one. For this [35:11] reason, this matrix R is referred to as [35:15] a rotation matrix which has the property [35:17] of being an orthogonal matrix. But the [35:19] way we construct it the vector are the [35:22] dotproduct between the vector are uh in [35:26] between the columns are zero and then [35:28] there is the the fact that the [35:30] determinant is equal to one. Cool. All [35:33] right. So is this clear? [35:38] You may already know this but it doesn't [35:40] hurt to derive it. No. So what happens [35:43] if [snorts] now the two origins do not [35:46] overlap? [35:47] Now we can represent the point PW [35:53] the point P in general with respect to [35:55] the camera reference frame as the [35:58] summation of [36:01] so P in the coordinate in the reference [36:04] frame in the camera reference frame is [36:06] equal to T [36:09] plus Q. [36:13] So t [36:16] I just put an underscore there to [36:18] emphasize that this is a vector. So t is [36:21] the vector that points from the origin [36:24] of the camera reference frame to the [36:25] origin of the word reference frame [36:29] and q is the position of the point p [36:33] with respect to the word reference [36:35] frame. [36:37] Now [36:39] assume that we know [36:41] T. [36:45] How can we express? So Q is the vector [36:49] that expresses the point P [36:53] with respect to the word reference [36:54] frame. But we can think of translating [36:57] this vector Q to the camera reference [36:59] frame. And then we recover the previous [37:01] problem of mapping the vector Q as [37:04] expressed in the word reference frame [37:06] into the camera reference frame. Which [37:09] basically means that the Q as before can [37:12] be written as a rotation matrix [37:16] times P W. [37:19] And this expresses the full [37:21] transformation between coordinates in [37:23] the world reference frame to coordinates [37:25] in the um camera reference frame. [37:28] basically the same as before but we had [37:30] to account for the fact that the origins [37:32] of the two reference frame are different [37:35] and that difference is captured by a [37:37] translation vector. [37:42] >> Yeah. [37:44] >> Translate first right before going be [37:47] the same. [37:47] >> Yeah. Sure. [37:50] Yeah. [37:51] Okay. So all of this to say that uh [37:55] given assume that we know t and r. So [37:58] assume that we know what is the relative [38:00] position between the two origins and we [38:03] know how one reference frame is rotated [38:05] with respect to the other reference [38:06] frame. Then if you give me the [38:08] coordinates in one reference frame, I [38:09] can transform those coordinates in [38:11] another in the other reference frame by [38:12] using this radation which is a [38:14] translation plus rotation. [38:21] In reality, [38:25] often times you use the equation in [38:28] reverse, meaning that [38:32] um you try to estimate from a sequence [38:34] of images what R and T are, which allows [38:37] you to understand how your camera is [38:39] moving with respect to the word frame. [38:41] And that is a project that is a process [38:43] that for example allow you to understand [38:45] from images how the robot is moving [38:47] through a procedure I also refer to as [38:49] visual autoometry. whether they use Mars [38:51] rovers as well. We'll see that in more [38:53] detail in a second. Now [38:57] the idea in order now to put everything [38:59] together is to express the uh coordinate [39:02] transformation also in the uh framework [39:05] of homogeneous coordinates and in [39:08] particular by considering homogeneous [39:10] coordinates. [39:14] Let's represent a PC in terms of its [39:17] homogeneous representation and then [39:20] let's represent a PW in terms of its [39:22] homogeneous representation. [39:25] Then the equation [39:27] the Korean transformation can be [39:29] equivalent written as R T 01 [39:35] where 03 * 1 is a vector with three [39:40] elements of zeros. [39:43] So if you write down the multiplication [39:45] you will see immediately that what you [39:46] get is exactly what we had [39:50] >> is one. Yeah sure of course [39:53] is [39:55] this is a a row vector. [39:59] So this will be basically [40:02] 0 0 0. So as pointed out should be one [40:06] by three one row and three columns. [40:08] Yeah. X. [40:14] So the result will be PC is equal to RPW [40:17] plus T. The last component will be one. [40:21] So once you divide by the last [40:22] component, you obtain exactly what we [40:24] have. [40:26] All right. So now we're in a position to [40:28] put everything together. Finally, we [40:29] found a transformation from points in [40:32] expressed with respect to board [40:33] reference frame and points expressed in [40:36] pixel coordinates. And the full [40:38] expression is expression that goes from [40:42] the point as expressed in the camera [40:43] reference system then pixel coordinates [40:45] through the camera matrix. But now we [40:47] also have the relation between the point [40:49] in the camera reference system and the [40:51] point in the world reference system [40:53] through the rotation and translation. We [40:55] put everything together and that's the [40:56] end result. [41:00] Any question about this? This is really [41:02] by chaining all the transformation that [41:03] we found so far. [41:08] So in particular we can multiply [41:13] this matrix by this matrix. So this will [41:15] become R + T and this will be uh zero [41:20] and the end result is this equation. [41:22] Basically PH again PH means the [41:25] projected projected point in pixel [41:28] coordinates as expressed in homogeneous [41:31] pixel coordinates. PH [41:36] is equal to [41:38] uh K [41:45] R T [41:47] capital P H [41:51] okay work [41:56] is this clear to everyone just let me [41:58] know because otherwise the calibration [42:00] process becomes impossible to [42:05] Is lowerase P the camera frame [42:08] >> is what [42:10] >> lowerase P is it [42:12] >> frame? Yeah. [42:12] >> These are the coordinates in pixels. [42:15] >> Okay. [42:15] >> So with respect to what's [42:18] >> full transformation [42:18] >> full transformation. [42:20] >> Yeah. And basically the translation with [42:23] respect to the corner and the [42:25] pixelization is taken care of by the [42:28] camera matrix as we discussed before. So [42:31] this is really the full transformation. [42:32] You give me coordinates like three mters [42:36] for X, four meters for Y resper frame. [42:39] This is and and here you get pixel 3 and [42:43] pixel four. [42:49] All right. So this matrix referred to as [42:52] the camera matrix contains the so-cal [42:54] intrinsic parameters of the cameras. So [42:57] there's interesting properties of the [42:58] cameras in terms of focal length, [43:00] displacement of the um original [43:04] reference system in the image plane with [43:05] respect to the pixel coordinate [43:07] reference frame and so on. R is a [43:10] rotation. It expresses the rotation [43:12] between the world reference frame and [43:15] the camera reference frame and T is a [43:17] translation. Basically if you think that [43:21] when you start the operation of your [43:22] robot the original word reference frame [43:25] coincides with the camera reference [43:26] frame this is basically the translation [43:28] that your robot has been undergoing. [43:33] So in principle if [43:36] you knew this transformation and as we [43:39] will discuss later you knew uh points in [43:45] projected points in maybe different [43:47] images for example through stereo vision [43:48] or other procedure then you can [43:50] reconstruct where that point was in the [43:53] Warframe. [43:55] That's how for example this equation is [43:57] used in order to reconstruct the 3D [44:00] environment by only looking at the 2D [44:02] images. By the way, this is also what [44:04] your brain does. Um the problem though [44:07] is in general you don't know K, you [44:10] don't know R, you don't know T. [44:15] So this leads to the so-cal uh [44:18] calibration procedure. The goal of the [44:20] calibration procedure is to determine [44:23] that projection transformation and in [44:25] particular to pinpoint the intrinsic [44:28] values of your cameras and later on you [44:30] can use for scene reconstruction. [44:33] So the idea of camera calibration is [44:36] relatively straightforward. Making it [44:38] work is uh extremely well not extremely [44:42] but it's a challenging it requires some [44:44] experience. And the first problem in the [44:46] pet three will um give you an [44:50] opportunity to um test the calibration [44:54] procedure on some sample images to see [44:57] how uh all of this works. So the idea of [45:00] the calibration process is as follows. [45:03] We're given um a pattern. So for [45:06] example, if you go in robotics labs, you [45:09] often times often times you might have [45:11] seen checkboard type of patterns. those [45:14] are used to calibrate cameras [45:18] and you will be working with that for [45:20] your uh homework number three. All [45:22] right. So that you know it's [45:25] in a way it's like vaccination. The [45:27] first time you do it is a little bit [45:29] unpleasant but once you do it once or [45:30] twice then you're pretty much set for [45:32] life but the first time is unpleasant. [45:35] Anyway, so you're given this uh [45:37] checkboard or pattern. So here is a [45:38] threedimensional one could be planer one [45:40] and so on. you have a world reference [45:44] system that is placed in a nonposition [45:46] with respect to this pattern. For [45:48] example, here the origin will be this [45:50] corner and then the axis will be aligned [45:53] with the axis of the pattern. So assume [45:55] you know that [45:57] you have uh your camera that is pointing [46:01] at the pattern [46:03] and then you assume that you know pairs [46:08] of [46:10] points on the pattern as expressed with [46:13] respect to the wall reference system and [46:15] the projected point in pixel [46:16] coordinates. So you're given these [46:19] pairs. The reason why these pairs are [46:21] given is that you have a full control of [46:23] where these points are. So you know [46:25] exactly how each point in the world [46:28] reference frame corresponds to a [46:31] projected point [46:33] because of this distinctiveness of you [46:36] know this structure right [46:40] so basically I give you a sequence of [46:43] 345 in reference frame corresponds to [46:46] pixel 567. [46:48] The reason why you can establish this [46:49] relationship is because it's clear from [46:52] the way that I constructed a pattern to [46:55] which point in the camera in the pixel [46:58] in the in pixels corresponds a point in [47:01] the work in the word. Does that make [47:04] sense? [47:06] So I give you now it's different. Now I [47:07] give you these pairs and I want to infer [47:10] from data the transformation by [47:12] exploiting the fact that we know the [47:13] structure of the transformation as we [47:15] derived it before. [47:18] So in particular assume that we have n [47:21] points in the word uh express to the [47:25] world reference frame and we have n [47:28] corresponding [47:31] projections of those points in pixel [47:33] coordinates. [47:35] Given that we want to compute the [47:38] unknown parameters KRT by applying the [47:41] structure of a perspective projection. [47:48] So the idea to do that is to divide a [47:50] procedure in two steps. The first step [47:54] is here when I brought the perspective [47:58] projection I separated it into the its [48:01] structure. So the camera matrix R and T. [48:03] So the first step is just to call this [48:05] matrix M. [48:09] So this is a matrix with [48:12] um four rows [48:15] and three columns. [48:22] So it should be also here. Yeah. [48:28] Actually this is a 3x4. Sorry 3x4. [48:34] Okay. So [48:36] um no actually it's 4x3. What am I [48:39] saying? Yeah. So the idea is in the [48:42] first step to first estimate these big [48:45] matrix M and then break it down into K [48:48] RT. Yeah. [48:51] >> Because the world is going to be [48:53] homogeneous and that's four times. [48:55] >> Oh yeah. [48:59] Oops. Sorry. Yep. Any [49:03] questions here? [49:05] >> Yeah. [clears throat] If you're [49:06] converting from a homogeneous frame to [49:08] another homogeneous frame, shouldn't you [49:09] be a square matrix? [49:11] >> No, because this is um remember that in [49:15] homogeneous plane [49:18] >> yeah is a two while in the water frame [49:20] is three. So the homogeneous for the [49:22] word is four homogeneous for the image [49:24] plane is three. So that's why is a three [49:26] four. [49:28] Good. So these are good questions. [49:31] So uh oops. All right. So the idea is to [49:42] to partition M into uh rows. So let's [49:47] write M into row number one, row number [49:53] two, row number three. Each one of them [49:56] is a one by four vector. [50:02] So if I give you n of these relations, [50:08] what do this relation entail in terms of [50:11] constraints for the unknown quantities [50:14] which are these vectors m1, m2, m3? So [50:18] basically the rows of the matrix. Well, [50:21] let's just carry out the uh [50:23] multiplications. So [50:27] ah [50:35] so if I multiply [50:37] m_sub_1 [50:39] m2 [50:41] m3 [50:42] by uh pwh [50:52] w uh ih [50:55] What I get is [50:58] m1 [51:00] pwh [51:02] m2.pw [51:04] i h m3pw [51:09] ih. [51:11] So then these will be the homogeneous [51:15] coordinates for the um projected point [51:20] in pixels. [51:23] So then the pixels with respect to the x [51:26] direction referred to as u is equal to [51:30] m_sub_1 [51:32] * pw [51:35] i h over m3 * pwh [51:41] and v is equal m1 pwh [51:46] sorry m2 [51:49] pwh [51:51] over m3 3 P W I H. So I went from [51:57] homogeneous coordinates to [51:59] non-homogeneous coordinates. Again, U is [52:02] the pixel coordinate in the X direction. [52:04] V is the pixel coordinate in the Y [52:06] direction. That's standard nomure which [52:09] is what you have here. [52:12] Then we can rewrite this equivalently as [52:21] so this will be ui vi so ui * [52:28] um [52:30] m3 [52:32] * pw i h [snorts] minus m_sub_1 pwh [52:40] equals Here [52:43] remember that we assume that we know u n [52:48] similarly v i * m3 pwh [52:54] - m_sub_2 pwh [52:57] equals z. Remember that we assume that [52:59] we know ui pwi [53:03] viwi. [53:06] The unknowns are these vectors here [53:09] which represent the perspective [53:11] transformation. [53:13] So basically by having multiple pairs [53:15] I'm able to build a number of equations [53:20] that constraint constrain what those m [53:24] vectors can be. [53:29] And note that this is a set of linear [53:31] equations in the values for m for the [53:35] components of the m1 m2 m3 vectors [53:40] which means that they can represent [53:46] >> pixel [53:49] >> are. So remember that we had a vector [53:51] pih [53:54] the projected point u and v will be the [53:57] xy component of a pih p i is a [54:01] threedimensional because it's in a [54:03] homogeneous coordinate so basically we [54:04] have appended so basically p ih is ui vi [54:09] 1 times lambda [54:15] all right so I can I have a set of [54:18] linear equation equations. In [54:20] particular, for each pair, I have a two [54:24] linear equations, which means that for n [54:26] pairs, I have two n linear equations. [54:30] Now, let's go back to [54:34] the projection matrix. How many [54:38] degrees of freedom we have? [54:44] So, how many things do we need to [54:46] estimate in general? [54:48] So [54:54] if we go yeah here [54:57] so K in general has four parameters [55:01] alpha beta U 0 with zero sometimes [55:04] skewness so five [55:06] R is a 3x3 matrix but it's an orthogonal [55:09] matrix which means that you have six six [55:12] constraints due to the orthogonality [55:14] which means that even though you have [55:15] nine entries you have three degrees [55:17] freedom which correspond to the fact [55:19] that you can reorient [55:22] coordinate system with respect to three [55:25] basis angles angles and t is three [55:28] dimensional as well. So you have in [55:31] total either 6 + 4 or 6 + 5 so 11 [55:35] degrees of freedom. [55:39] But here [55:47] m is 3x4. So there are 12 elements. [55:53] So we have 11 degrees of freedom but [55:54] we're estimating something with 12 [55:56] unknowns. [55:58] So I'm trying to build intuition then we [56:00] formalize that. Why is there is this [56:02] discrepancy? The discrepancy is due to [56:04] the fact that since we are operating in [56:06] homogeneous coordinates, all these [56:08] equations are defined up to a scaling [56:11] factor lambda. So we can always rescale [56:14] this matrix by the scaling factor and [56:17] have one of the entries be equal to one. [56:19] So that's why we have indeed the 11 [56:21] degrees of freedom we want to estimate. [56:22] Yeah. [56:24] >> If you go one side forward [56:27] >> here, [56:28] >> but we do that and [56:29] >> Okay, I haven't done that yet. Okay, [56:32] >> let me do it. [56:35] >> Um, yeah. So, [56:40] so this is a set of linear equations. We [56:42] stuck them into uh matx format pw oh [56:49] well yes pwm equal zero whereby I define [56:55] m as a stacked column vector whereby I [56:59] take all those rows and I put them [57:02] together. [57:05] Remember that mi was one of the rows of [57:08] the matrix. M1 transpose becomes a [57:10] column. So the dimension of this is 12 * [57:14] 1. [57:17] Pilda is just a matrix of coefficients [57:20] that encode the um equation that I had [57:24] before. Like for example, [57:27] what do you think is the first row of [57:29] pila? [57:39] >> Yeah. Could you repeat? [57:45] >> What do you say? [57:45] >> The first you put UI time. [57:48] >> Yeah. But we have to write it into into [57:52] blocks, right? So we will have minus P [57:57] W1H. [58:02] Then we don't multiply anything by M2. [58:06] So we have a zero. [58:08] and then we multiply by uh m3. So we [58:12] will have a ui [58:15] pw1 [58:16] h technically transpose [58:19] since now I have transpose those uh [58:23] vectors and then for and then I will [58:25] have for v1 would be uh minus p v1 h [58:32] transpose [58:36] uh sorry it will be zero [58:41] 0 minus P V1 H transpose because this [58:46] multiplies [58:48] uh M_sub_2 [58:50] plus [58:51] so this U1 this V1 P W 1 H transpose and [58:59] then the pattern repeats. [59:04] All right. So then what we want to solve [59:07] is this system of linear equations. In [59:10] reality in general you never have [59:13] equality because there is some always [59:14] some measuring error. So in general you [59:17] want to build a number of equation that [59:19] is uh large enough to have a system that [59:24] uh allows you for robust calibration. In [59:26] particular we said that with n [59:28] measurements we have a 2 n constraints. [59:32] So in order to have at least the minimum [59:34] number of constraints to pinpoint the [59:36] value of m you want to have a 2 n [59:39] greater than 11 the parameters that you [59:42] want to uh determine in general you want [59:44] to have more so that your calibration [59:46] procedure is um [59:50] is robust. This means that at least you [59:52] want to have six correspondences in [59:54] reality more. So this [59:57] okay so then how do we solve the [59:59] problem? [60:01] As I said, this matrix M is defined up [60:05] to a scale factor due to the fact that [60:07] we work with homogeneous coordinates. [60:11] In particular, as you can see from here, [60:15] if M is a solution to that system of [60:19] linear equations, say M bar, then lambda [60:23] and bar is also a solution. So, we can [60:26] just fix the scale in order to deal with [60:29] this ambiguity. And of course m bar [60:32] equals zero is always a solution but [60:34] will be the trivial solution not [60:35] something we're interested in because it [60:37] would not represent a physical camera. [60:39] So we are interested in nonzero [60:41] solutions for which we fix the scale. [60:43] For example we can fix the scale at m [60:46] over bar equal one. So basically we take [60:49] the m vector such that the norm is equal [60:53] to one. [60:55] All right, [60:57] we said that in general we want to have [60:59] a number of equations that well a number [61:03] of equations that is much larger than 11 [61:07] and in general due to measurement error [61:09] p is not exactly equal to zero. So the [61:12] problem that in the end we want to solve [61:14] with a calibration procedure is to find [61:18] the minimum 12 dimension 12 dimensional [61:21] vector M such that the norm of a pt M is [61:27] minimized. We want to make we can't [61:29] really make it zero of times due to [61:31] measurement error but at least we want [61:32] to make it as small as possible [61:35] such that [61:39] the norm of m [61:43] is equal to 1. [61:45] That's the problem we want to solve. [61:51] Yeah. Then what can we mean by [61:52] assumption that m is equal to one like [61:55] >> so if m is a solution lambda m is also a [61:58] solution [62:00] so let's just pick the scale once and [62:02] for all and then we solve with respect [62:04] to that scale I mean m if you want you [62:06] can make m equal to four it's just to [62:09] fix the scale or m equal to five any [62:11] number we can make m equal to lambda [62:14] but [62:17] one just leads to cleaner mathematics [62:21] So how do we solve this problem? [62:25] Well, so for the graduate students among [62:27] you, if you're undergrad, this is like [62:29] bonus. If you're graduate students, this [62:31] is not bonus. You should know it. So [62:34] this is a constraint uh optimization [62:36] problem. So as we saw for the lecture on [62:40] indirect methods, the way this problem [62:42] are solved is by forming the lranjen. So [62:45] the lrion L is equal to M transpose P [62:49] tilda transpose P M I've just [62:54] expanded the norm plus [62:58] lranch multiplier lambda* [63:01] 1 - mrpose n. [63:05] So the minimum is a critical point of [63:07] the lrang. So the derivative of the [63:08] lrang with respect to m is equal to 2 pm [63:14] plus actually minus [63:18] uh minus 2m and this has to be equal to [63:20] zero. It has to be a critical point of [63:22] the lrang which means [63:25] uh lambda m which means that a necessary [63:30] condition for a vector m to be minimizer [63:32] is that the pn is equal to lambda m. [63:36] Can you see? [63:39] Which means that M has to be an igen [63:41] vector of so we call this P. M has to be [63:44] an vector of PTA. So what we do? We [63:47] compute the values of P tilda and then [63:50] the corresponding vectors. Among those [63:52] there is the minimizer. But which one is [63:54] the minimizer? [63:56] Well, let's take uh let's um lambda be [64:03] an vector value pair. Then I plug that [64:07] into the cost function. [64:11] The cost function is M transpose P tilda [64:15] P M. [64:18] But since you know that M and lambda is [64:20] an vector value pair, which means P M is [64:23] equal to lambda M. This is equal to M [64:28] transpose [64:30] lambda squar M. But we have the [64:34] constraint that the norm of M is equal [64:36] to one. So this is actually equal to [64:38] lambda square. So among all the vectors, [64:42] which one do we pick? [64:44] >> Small one. [64:45] >> Yeah, we pick the one that corresponds [64:47] to the smallest values. So basically [64:50] calibration procedure entails forming [64:52] this matrix Pilda computing the values [64:54] and vectors for example with the [64:56] singular value the composition picking [64:58] the vectors with minimum value. [65:01] All right. So we found how to compute M. [65:04] Yes. [65:04] >> And that's also the singular value [65:07] decomposition ensuring that M is right. [65:10] >> Yeah. But I mean you can [65:12] >> actually recover [clears throat] [65:13] the matrix correctly. [65:16] >> You really like to go one step ahead [65:18] >> because [65:19] >> that's what we're doing here. [65:20] >> Yes. [65:21] >> Yeah. But you're hinting the right [65:22] direction. So now we have M which is a [65:26] 12 um sorry is a 3x4 matrix [65:33] but we want now to compute the [65:36] individual components K R and KT. [65:43] How do we do that? Well, we can [65:46] partition M [65:49] because at the end of the day, that's [65:50] what we want to compute. K, R, and T. [65:53] So, we can we can partition [65:59] M [66:02] as [66:04] a 3x3 matrix and a 3x one vector. [66:12] So we know this and we know this. From [66:15] this a we want to compute KR. So we want [66:19] to be able to separate the intrinsic [66:21] from the rotation. How do we do that? [66:23] Well, we use a fact in linear algebra [66:26] that is referred to as a QR and RQ [66:29] decomposition which is a little bit [66:31] unfortunate name in this context. [66:32] Basically the idea is that every [66:36] um n byn matrix can be uh decomposed [66:41] into um [snorts] an orthogonal matrix q [66:45] and an upper triangular matrix r. So [66:49] this can be decomposed [66:51] as r * q where r is upper triangular. [66:59] This is a fact from linear algebra and [67:02] this is orthogonal. [67:06] So basically there is a line I mean [67:09] there's a function in numpy that you can [67:11] call in order to do this RQ de [67:13] composition. The reason why this is [67:14] unfortunate is that R is not the [67:17] rotational matrix is the upper [67:18] triangular matrix. So the R in our [67:21] context will be this Q and this will be [67:24] K. unfortunately is called the RQ uh de [67:29] composition but just want to show that [67:31] you are not confused by the terms. [67:35] So once you have computed the rotation [67:38] matrix which will be the Q there and the [67:40] upper triangular matrix which correspond [67:42] to the cal the camera matrix which is [67:44] the R there how do you compute A which [67:48] will be the translation vector. [67:51] Well, you know from here that a is equal [67:55] to kt. [67:57] So to compute the t, what do you do now [67:59] that you know k [68:02] inverse? [68:02] >> Yeah, we don't take inverse. [68:06] So that's how you go from pairs of [68:08] correspondences to the intrinsic and [68:10] extrinsic of the camera [68:14] and you explore you'll be exploring this [68:16] further in your pet. [68:22] Fortunately, it looks relatively simple, [68:23] but there are some nuances that you'll [68:25] see. [68:32] All right. [68:33] So there is an additional complication [68:35] that we won't see in this class but just [68:37] for you to know is that here we have [68:39] assumed basically a linear model meaning [68:42] that we don't have a magnification due [68:45] to uh the fact that we're using a lens [68:48] and not a pinhole camera in our image. [68:50] But in reality you might have [68:53] magnification as you move away from the [68:55] center of your image. If we have a [68:57] magnification that is larger at the [68:59] center and goes farther as you go away [69:01] like for a fisheye camera then we talk [69:04] about barrel distortion or conversely we [69:06] talk about pin caching distortion. So in [69:09] general cameras also have a distortion [69:12] factor that you have to account for [69:14] which makes the calibration process even [69:16] more complicated because now you also [69:17] have to infer the distortion factor from [69:20] the calibration procedure. [69:22] Usually fortunately apart from you know [69:24] fisheye cameras and specific cameras [69:26] this distortion factor is quite small uh [69:29] and we and this is so this estimation is [69:32] beyond the scope of this class but just [69:33] for you to know that in full generality [69:35] we should also account for an additional [69:37] parameter. Yeah [69:38] >> if you're working with cameras and you [69:39] buy a particular model of camera or [69:41] image sensor um do you have to [69:44] recalibrate every single particular [69:46] sensor or just when you have a new model [69:47] of sensor? [69:48] >> Oh yeah they calibrate every single [69:50] camera. [69:50] >> Every single one. [69:51] >> Oh yeah pretty much. Yeah, even small [69:53] manufacturing uh uh differences might I [69:57] mean in general. [70:01] All right. So now we have done all of [70:04] that. So we have a good um [70:09] understanding of how points in in the [70:12] world are projected onto a camera. How [70:16] do we use that information? But the very [70:18] first way to use that information is [70:20] trying to recover structure. Meaning now [70:22] you only have images. You want to build [70:24] a 3D representation of the word. So the [70:28] question is if a given image and now you [70:31] have estimated your uh perspective [70:35] projection uh matrix and so on. Can you [70:38] recover the point P capital P that gave [70:42] rise to a given point P in your image [70:44] plane? If you do that and now you're [70:46] able to reconstruct the environment. The [70:49] answer is unfortunately no which is one [70:52] of the big challenges in computer [70:54] vision. And the problem is due to the [70:56] fact that you have an inherent death [70:59] ambiguity in terms of where the point P [71:02] could be through our perspective [71:05] equation. What we basically found is [71:08] that any point along the ray that [71:11] connects the projected point to the [71:14] pinhole could be a possible point in the [71:17] 3D world that gave rise to this point P. [71:20] Meaning that a point here or a point [71:22] here or a point here or a point here [71:24] will give rise exactly to the same [71:26] projected point in the image plane. [71:29] Which means that the only thing that you [71:31] can infer is that the point in 3D that [71:35] might have generated this point small P [71:37] is somewhere along this line, but you [71:38] don't know exactly where. In other [71:42] words, if you go to Italy and you take [71:43] this famous picture, it seems that [71:45] everyone goes to pizza, take this image [71:47] for whatever reason. [71:50] In principle, you can really really say [71:52] if the pizza tower is bigger or smaller [71:55] than this person. as a human you can but [71:58] because you're using a higher level [72:00] understanding of the camera. So [72:01] basically you are relating you're saying [72:03] well this will also be people. So since [72:05] this are small it means that this is in [72:07] the background and this is in the [72:08] foreground. But without that level of [72:11] semantic understanding just by looking [72:13] at this image you cannot really recover [72:14] the scale of the scene which means that [72:18] you do something more. And basically [72:20] there are four main ways of recovering [72:22] structure meaning death in the scene uh [72:26] from planner from 2D images. The first [72:30] the first first one is through the [72:31] recognition of landmarks or semantic [72:34] understanding what I was explaining [72:35] before and it's something that goes a [72:37] little bit beyond the scope of this [72:39] class. The second one is that from focus [72:42] as we'll see in a second. The idea is to [72:43] take images until when the image is [72:46] perfectly in focus and by using a thin [72:49] lens equation extrapolate what the uh [72:53] depth of the um uh image really is. [72:56] We'll see this in a second. Stereo [72:58] vision stereo vision that relies on [73:00] using two cameras with a known [73:02] displacement and rotation between the [73:04] two and non camera matrices in order to [73:07] reconstruct scenes through a procedure [73:10] referred to as triangulation. In [73:12] particular, the idea is to take two [73:14] distant images at the same time and fuse [73:17] the information in order to reconstruct [73:19] that. That's what you do as a human or [73:22] structure for motion. So now you take [73:25] multiple images at different times and [73:27] then you try to reconstruct the scene by [73:30] using that. [73:32] All right. So let's start with death [73:34] from focus. The death from focus idea is [73:37] fairly simple but very cumbersome. So [73:40] most of the times this is not used. The [73:42] idea is that uh you take an image [73:48] but you change the focus in such a way [73:51] that the image becomes uh as sharp as [73:55] possible which means that the image is [73:56] in focus which means that basically you [73:59] have a pinpointed ZV value for your for [74:02] your thin lens. Now from last time you [74:05] know the thin lens equation. So if you [74:07] know the focal length and if you know [74:09] the depth in terms of the image plane at [74:13] which the image is in focus then by [74:16] using the thin lens equation you can [74:17] then infer capital Z which is the [74:19] distance between the camera and the [74:22] point in the 3D world but that means [74:24] that for each point you have to do this [74:26] business of putting a point in focus so [74:28] it's a bit cumbersome [74:30] so the standard way or one of the two [74:32] standard ways the other standard one [74:33] will be structure for motion in order to [74:35] recover structure is to use a stereo [74:38] vision. Studio vision as I said entails [74:40] and that's what your brain does to take [74:43] [snorts] an image with two cameras, two [74:46] eyes in your case and exploit the [74:48] disparity or the differences between [74:51] analogous points in the two images in [74:53] order to reconstruct the scene by using [74:55] a process known as a triangulation. [74:58] The triangulation process is fairly [75:01] intuitive. So this is the image plane. [75:03] This is the pinhole for the first camera [75:05] image plane and pinhole for the second [75:06] camera. It is a pro point P that has [75:10] been projected in the first camera and [75:12] in the second camera. P and P prime are [75:14] the projections. [75:16] Now if you only have one image, the only [75:18] thing that you can say is that the point [75:21] capital P lies somewhere along this ray. [75:25] And then if you only have this image, [75:27] the only thing that you can say is that [75:29] the point capital P lies at somewhere [75:32] around this ray. But now if you have two [75:35] images, [75:37] then since you know that the point [75:38] capital P has to lie along this ray and [75:41] the point capital P has to lie along [75:43] this ray, then the intersection is [75:45] really the position of the point in a 3D [75:47] world. And that's how by fusing [75:50] information by using this procedure [75:52] referred to as triangulation you can [75:54] pinpoint where the point is in the 3D [75:56] world. Of course in order to do this you [75:59] need to know what is the uh position of [76:02] this image plane with respect to this [76:04] image plane. So basically to know the [76:06] position of this camera reference system [76:08] with respect to this camera reference [76:09] system and also know the intrinsic [76:11] parameters for both the first camera and [76:12] the second camera. [76:15] So there is also a procedure to [76:16] calibrate the steer rig as opposed to [76:18] mono camera. Now last thing is that in [76:22] reality due to measurement noise and [76:25] other inaccuracies [76:30] a point capital P may map into two [76:34] points P and P prime such that when you [76:36] extend the rays outward they do not [76:39] intersect at a point capital P because [76:41] of measurement error. [76:44] In principle they should so if our model [76:47] was perfect they should intersect a [76:49] capital P and that's what we would do [76:51] for triangulation but in general they [76:53] may not. So in reality in order to [76:56] triangulate what you do is to there are [76:59] few different ways to do it but a common [77:01] one is to consider the estimation of P [77:06] by considering the so-called [77:07] reprojection error. So basically we try [77:09] to find the point in the 3D board. So [77:12] the point Q such that the its projection [77:17] in the first image and the second image [77:19] gives rise to the lowest possible [77:21] reprojrojection error. Meaning the [77:23] distance between the projection that we [77:25] have observed and the projection that [77:27] our estimate of P that we call Q will [77:31] give. Ideally we would like for this [77:33] distance this distance to be zero which [77:35] would be the case if the two original [77:37] rays were to intersect. In reality we [77:39] can't. We just have to find the best [77:40] estimate. [77:43] Does it make sense? [77:45] Cool. One of the challenges to your [77:47] vision is how do you know that the point [77:50] P here corresponds to this point P prime [77:53] here. That's why it's very important to [77:55] find feature that is very distinctive so [77:57] you can make these correspondences. [78:00] There are a number of structural [78:02] properties of the stereo matching [78:03] process that one can leverage to do that [78:05] effectively. But it's something that we [78:07] will discuss on uh Friday for the those [78:11] of you that are enrolled in the gradu [78:13] version of this class. We will be [78:14] talking about stereo vision and then the [78:17] other procedure to reconstruct structure [78:19] which is uh structure for motion. Right. [78:23] So yeah, two minutes late. Sorry but uh [78:26] see you tomorrow.