Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 19: Model-Based RL

斯坦福 AA203 最优与基于学习的控制 | 2026春 | 第19讲：基于模型的强化学习 Gammelli 博士深入探讨了基于模型的强化学习的核心挑战，重点阐释了不可约减的偶然随机性与数据稀疏性导致的认知不确定性之间的关键区别。在对比精确解析的高斯过程与高可扩展性的深度神经网络集成后，他详细介绍了 PETS（基于轨迹采样的概率集成）算法，该算法在 MPC 循环中将集成前向模拟与交叉熵方法相结合，展现出卓越的样本效率。课程最后梳理了完整的层级化自主技术栈——涵盖动态规划、开环规划、MPC、PID 控制以及 Hamilton-Jacobi 可达性安全包络——从而解释了为何生产级系统在底层执行层面刻意避免使用端到端神经控制。 仅依据输出熵评估动力学模型会将不可约减的偶然噪声与认知参数不确定性混为一谈，未能在规划器进入未观测状态空间时发出警报。 (41:28) 高斯过程为动力学建模提供了严格解析的闭式后验更新，但与深度集成相比，其立方级 O(N^3) 的协方差求逆计算成本限制了其扩展性。 (55:35) PETS 算法在 MPC 框架中融合了概率神经网络集成、轨迹采样与交叉熵方法，以少几个数量级的样本量达到了匹敌 PPO 和 SAC 等无模型基线的表现。 (70:16) 工业级自动驾驶架构维持着严格的分层隔离——将轨迹优化、MPC 和 PID 控制与 Hamilton-Jacobi 安全包络进行分层整合——刻意在底层控制接口避免使用端到端学习。 (79:02)

In the final lecture of Stanford AA203, Dr. Daniele Gammelli synthesizes policy optimization and model-based reinforcement learning, arguing that managing uncertainty is essential when coupling learned models with numerical optimizers across modern autonomous control stacks. Surrogate Objectives and Stabilized Policy Optimization (1:05) Modern machine learning treats learning as continuous numerical optimization, but nominal on-policy gradient methods traditionally discard data after a single step. To close this gap, algorithms introduce surrogate objectives that match true policy gradients when differentiated via automatic differentiation frameworks. Weighting these objectives by importance sampling ratios allows parameter updates on offline rollout batches, yet this shift introduces instability if policy parameter drift diverges too far from the data-generating policy. From TRPO Constraints to First-Order PPO (7:49) Trust Region Policy Optimization (TRPO) limits policy drift by constraining the Kullback–Leibler (KL) divergence in action distribution space rather than raw parameter space. However, enforcing this bound demands second-order optimization overhead via conjugate gradient descent to invert the Fisher Information Matrix, which scales poorly with deep neural networks. Proximal Policy Optimization (PPO) replaces explicit constraints with a clipped surrogate objective that limits the probability ratio to an interval around one and applies a pessimistic minimum against the advantage function. This structure saturates destructive updates while preserving compatibility with first-order stochastic gradient descent. Model-Based RL and the Optimizer's Curse (26:47) Model-based reinforcement learning (MBRL) promises high sample efficiency by fitting transition dynamics directly from interaction data. A naive implementation—fitting transition dynamics offline with regression and passing the learned model to a trajectory planner—breaks down under covariate distribution shift once the planner steers the system into unfamiliar states. While model predictive control (MPC) and iterative data aggregation mitigate this drift, high-capacity function approximators risk overfitting sparse transition data. Numerical optimizers aggressively exploit the resulting hallucinated errors in unvisited regions, selecting trajectories that appear optimal under the inaccurate model but fail on physical hardware. Uncertainty Estimation and Probabilistic Trajectory Sampling (41:28) Preventing optimizer exploitation requires explicit quantification of epistemic uncertainty—the model's lack of knowledge—rather than relying on output entropy, which captures only irreducible aleatoric noise. While Gaussian processes provide analytical posteriors over functions, their exact implementation scales cubically with dataset size. Instead, deep bootstrap ensembles approximate the parameter posterior by training multiple neural networks from different random initializations. The PETS algorithm leverages this approach by sampling trajectories across neural network ensembles, scoring candidate plans via the Cross-Entropy Method, and executing receding-horizon MPC to achieve sample efficiency gains over model-free baselines. Hierarchical Decomposition in Modern Autonomy (75:46) Rather than treating control paradigms as mutually exclusive alternatives, real-world autonomous systems organize them into a functional hierarchy. Dynamic programming handles strategic high-level decisions, open-loop optimal control and model-based methods generate kinematic trajectories, MPC tracks reference paths, and PID loops execute actuator commands. Gammelli notes that while end-to-end learning has advanced from perception into mid-level trajectory planning, safety-critical lowest-tier tracking and actuation still rely on deterministic controllers and formal verification tools like Hamilton-Jacobi reachability. While model-free algorithms like PPO and Soft Actor-Critic remain the robust off-the-shelf standard for continuous control, model-based reinforcement learning provides substantial sample-efficiency advantages at the expense of higher algorithmic complexity.

Daniele Gammelli: in actually the majority of the cases we are ending up off the cliff. [65:45] Daniele Gammelli: simulating in our head what is happening. [65:45] Daniele Gammelli: This is obviously just a choice... the takeaway here should not be, this is the best way to go for when you're talking about model-based reinforcement learning, because ultimately, it depends on the problem at hand. [73:25] What's your favorite method? [79:02] Daniele Gammelli: bullish companies or approaches that claim end-to-end approaches, typically they don't mess with the lower level components.

[0:05] Well, welcome to the last lecture for this quarter. [0:09] Today, we will be completing our roadmap. [0:12] Specifically, we'll be starting by wrapping up [0:15] our discussion on model-free reinforcement learning first, [0:18] where we're looking at policy optimization methods. [0:20] And then we'll discuss model-based RL. [0:25] So, if you recall, basically, what we were saying last time [0:29] is that we started looking at policy optimization approaches [0:32] for model-free reinforcement learning, whereby [0:35] the main idea was to basically build and compute these policy [0:38] gradients, which are basically the gradients of the ground [0:42] truth reinforcement learning objective that we're [0:44] interested in optimizing, which enabled us to basically derive [0:47] the number of algorithms ranging from reinforce [0:49] to actor-critic algorithms. [0:52] And then we basically concluded with having to basically discuss [1:00] some more recent reinforcement learning algorithms that relate [1:03] with policy optimization. [1:05] And if we take a step back and think about how state-of-the-art [1:12] algorithms in machine learning more broadly typically work, [1:15] they ultimately boil down learning to numerical [1:19] optimization problem. [1:21] So let's think back, for example, [1:22] at our classical regression problem, which [1:26] is described as a supervised learning problem, [1:29] there we typically formulate it as a training error minimization [1:33] problem, which we can optimize sequentially [1:36] taking a data set of observation until we [1:38] converge to some local optima. [1:43] And this is fundamentally different from what [1:45] we've been doing so far in reinforcement learning. [1:47] Because if we think back at value-based methods, [1:49] kind of Q-learning being a prototypical example, [1:51] this is-- well, first of all, fundamentally different [1:54] because we're not really optimizing an objective, [1:56] but we're doing fixed-point iteration, which [1:59] can, in principle, include all of the transitions [2:02] from an offline data set. [2:04] But from the perspective of trying [2:06] to mirror this same formalism, this [2:09] is not really optimizing the ground truth reinforcement [2:12] learning optimization problem. [2:15] On the other hand, we saw how policy optimization methods. [2:18] On one hand, do optimize for the ground truth reinforcement [2:21] learning objective, but we only do basically one step [2:24] of gradient descent at a time. [2:26] So we saw how, at least nominally, these [2:29] are on-policy algorithms, where we have some interactions [2:32] with the environment, we collect data, take one gradient step [2:35] and repeat. [2:35] So this is really, in a way, fundamentally [2:37] different from just positing, this as a numerical optimization [2:40] problem and crunching the numbers until we [2:43] converge, if that makes sense. [2:45] So we'll discuss two specific algorithms, namely trust region [2:50] policy optimization, TRPO for short, and proximal policy [2:53] optimization, or PPO for short, which originally kind of-- [2:57] were born by trying to replicate this [3:00] formalism of positing an optimization problem [3:03] and crunching it at convergence for the purpose of reinforcement [3:07] learning. [3:09] Where the goal? [3:10] Therefore, will be to define a surrogate problem [3:14] and have it such that we can take continuous updates [3:17] to our policy before potentially grabbing new experience [3:21] from our environment. [3:25] And so we saw how in policy optimization, what [3:29] we want to do kind of a face value [3:31] is to compute the policy gradient, which, last Monday, we [3:35] defined as being the expected value of the [? grad ?] [3:39] replies multiplied by some quantity, which is [3:41] a metric of the reward to go. [3:43] We saw how an attractive quantity to have here [3:47] is the advantage A. Now, the first idea [3:51] is that what we would like to have [3:53] is define this surrogate optimization problem, whereby [3:57] if we were to take gradients of this objective function, which [4:00] we define, the gradients would themselves [4:02] be the policy gradient, which are interested in following. [4:06] And so a simple idea, which is also how you might have seen [4:10] by playing with your [? p-sets, ?] is that when you [4:13] want to introduce a concretely implement, these kind of things, [4:17] and solve them through automatic differentiation, [4:20] since we're interested, at the end of the day, [4:21] in following this object over here, [4:23] kind of a pragmatic way of doing so, is to define a surrogate [4:27] objective, which is a quantity that we don't necessarily care [4:31] too much about in practice. [4:33] But if we were to take the gradient of this objective, [4:35] we would actually get the policy gradient. [4:38] So in practice, this is how we would normally [4:41] implement this through auto differentiation methods. [4:44] We will define the surrogate objective, [4:46] and autodiff will basically give us [4:48] the gradient of this object, which is the policy gradient, [4:50] and we can follow that. [4:54] An alternative representation of this surrogate objective, [5:00] which we might want to use in the exact same way, [5:02] is what I'm referring here as the loss IS where [5:08] stands for importance sampling, which is essentially [5:12] an equivalent objective to the one shown here above, whereby [5:17] rather than having this defined as the log pi's multiplied [5:21] by the advantage, we have this probability density ratio which [5:25] comes from an old version of our policy pi, [5:28] described by some old parameters theta [5:31] old and potentially new version of that same policy described [5:35] by some current parameters theta. [5:38] So first of all, to sanity check, [5:40] that the gradient that we get from these two [5:43] is equivalent just by applying the chain rule, [5:46] we will see that the-- [5:47] well, the grad of the logarithm of a function [5:50] is actually equal to the gradient of this ratio. [5:54] So the gradient of the function divided by the function [5:58] evaluated at theta old. [6:01] And so long story short is that these [6:04] are two separate surrogate functions [6:07] we could use to nominally get the policy gradient out [6:11] of it in the same way. [6:12] However, the second one, that is the importance sampling one, [6:16] has this nice interpretation whereby this objective is [6:20] actually well-defined even if we are using a collecting [6:25] data according to a previous version of the policy, [6:28] in this case, f theta pi theta old. [6:32] So you're probably starting to imagine [6:34] how this form-- this objective is still valid in the moment [6:38] where I'm updating my policy multiple steps sequentially [6:43] in a row. [6:44] So I'm kind of diverging from my original theta old policy. [6:48] And this objective is still valid in the sense [6:49] that the importance sampling is kind [6:51] of reweighting my probability that I [6:54] give to the action with respect to my previous version [6:57] of the policy. [6:58] So I'm aware that I'm going through this quite fast. [7:01] But the main ideas that we want to reach the state where [7:08] we are defining a surrogate objective, which [7:11] still follows the main ideas of policy gradient. [7:14] Here we're seeing two different alternatives, [7:16] which are equivalent in that sense, [7:18] because these are two loss functions which [7:20] if we differentiate, we'll still get the policy gradient. [7:24] But this importance sampling version [7:25] has this nice interpretation which [7:27] still holds when we are actually considering [7:30] data collected according to a previous version [7:33] of our same policy. [7:35] Does this make sense? [7:42] And so, of course, this is the objective [7:49] that is used within the trust region policy optimization [7:52] algorithm with a caveat. [7:54] And the caveat, which we didn't discuss here, [7:56] is that-- so here we are mentioning [7:58] this idea of taking multiple steps [8:01] and therefore starting to diverge from our original theta [8:04] old. [8:05] But of course, the more we go further away, [8:08] so the more we move from our original theta old, [8:11] the more our estimate, for example, of the advantage [8:14] will be less accurate, and the more potentially [8:18] this ratio will stagnate learning to some extent. [8:21] And so what we want to do is we want to potentially [8:25] take multiple steps sequentially, as we were saying, [8:29] but we want to do this while remaining within a neighborhood [8:35] of the original policy within which [8:37] we are kind of confident that what we're doing [8:39] is still meaningful. [8:41] And so this is the idea of introducing what is often [8:44] referred to as a trust region. [8:46] This is a pretty common technique [8:48] in optimization, whereby given that same objective that we were [8:53] discussing before, we're now introducing [8:54] this constraint, which is basically expressed [8:58] through the KL divergence, that is a measure of divergence [9:01] between distributions. [9:03] Such that, we want that the distribution over action [9:05] induced by our old policy. [9:07] See, the old has to be closer with delta close [9:14] to the new policy, to the new distribution over action [9:18] induced by the new policy by theta, if that makes sense. [9:23] So why is this interesting? [9:25] Is because, of course, there could [9:26] be many possible ways of defining closeness for two [9:31] policies. [9:32] Maybe one more intuitive way would [9:35] be to just measure distance in parameter space. [9:38] For example, penalizing, like the norm [9:41] of the distance between the two theta vectors. [9:44] But this is pretty attractive because this is directly [9:48] evaluating distance in policy distribution space, which is [9:52] ultimately what we care about. [9:54] So we are measuring distance with respect [9:56] to the distribution over actions induced by the two policies. [9:59] So maybe the parameters vary quite a lot. [10:02] But ultimately what we care about [10:04] is that the distribution induced by the policies remains similar. [10:08] Yeah. [10:08] Is it that there's also expectation [INAUDIBLE]? [10:22] Yeah. [10:23] So this depends also if you're averaging [10:28] over different trajectories. [10:29] So remember that this will be doing an empirical evaluation [10:33] of this. [10:33] So we'll be sampling multiple trajectories over time [10:36] and doing empirical means instead of expectations, [10:39] but this is why. [10:39] Yeah. [10:44] So this was incredibly successful at the time, meaning, [10:51] it was able to be a pretty reliable reinforcement learning [10:55] algorithm, especially for continuous control problems. [10:58] However, it came with its own set of limitations [11:01] because, of course, now we are changing [11:04] from having to solve an unconstrained problem [11:07] to actually having a constraint in our formulation, which [11:11] complicates things. [11:12] Specifically, in order to solve these problems, [11:15] people usually use conjugate gradient methods, which [11:18] can be tricky to implement. [11:21] And empirically, this algorithm usually [11:27] seemed to perform pretty weakly when [11:29] we're using maybe deep neural networks, [11:32] such as convolutional neural networks or large transformer [11:34] networks, which was indeed one of the main practical [11:40] limitations of TRPO. [11:43] So this algorithm was still extremely impactful [11:47] because, in a way, it motivated successive algorithms which [11:52] try to simplify and remove these limitations, which is [11:56] exactly what PPO actually is. [11:57] So PPO in a way is a derivative of TRPO, [12:01] which is trying fundamentally to answer the question of, [12:04] can we replicate the exact same objective [12:06] that we were trying to achieve with TRPO without having [12:10] to resort to this second order optimization, [12:13] which is basically entailing this more complex optimization [12:16] problem that we have to solve in the constrained problem [12:19] that we introduced in the previous slide. [12:22] Is this setting clear on the transition from TRPO to PPO? [12:29] And so, if you look around for different implementations [12:36] of PPO, most likely you will stumble [12:38] upon one of these two versions. [12:41] So the first one is, well, we have this constraint problem. [12:44] We are saying that this is introducing [12:45] some challenges, at least from an implementation perspective. [12:49] Let's basically move the constraint [12:51] to the objective function with a penalty [12:54] through Lagrange multipliers, which then leads [12:57] to an unconstrained problem. [12:59] And let's solve this through stochastic gradient descent [13:02] tools, as we have always done. [13:05] So in a way, this is completely valid. [13:07] Although, I would say that the most popular version is [13:11] this second version here, where we're basically [13:16] trying to empirically replicate the behavior of the constraint [13:21] through an empirically derived equation in a way. [13:27] So we are basically introducing this specific objective here, [13:32] where we are trying to replicate the behavior of this constraint, [13:36] which we introduced before. [13:38] How are we doing so? [13:39] We're doing so by, first of all, defining the ratio just [13:43] for compactness as a small r of theta. [13:46] So this is the ratio between the two policy distributions [13:49] which we're using before. [13:50] Of course, if we evaluate r at the old version of the parameter [13:56] theta, we will get r equals to 1. [13:59] And the objective is comprised as the minimization [14:02] between two terms. [14:04] The minimization here is used as such [14:06] that we are basically having that this overall objective is [14:11] a lower bound of the ground truth [14:13] objective of the ratio times the advantage. [14:16] So we're basically picking the minimum [14:18] between the ratio times the advantage, [14:20] and this clip term over here. [14:23] So what does the clip function do? [14:27] We're basically taking the ratio, small r, [14:32] assume this is the axis where we are considering small r. [14:36] And we are basically saying that we [14:37] are constraining it to be between 1 minus epsilon [14:40] and 1 plus epsilon. [14:43] So if r falls within these two bounds, we are keeping the-- [14:50] well, the value of r. [14:52] Otherwise, we are clipping it to r theta would be here, [14:56] we would be using 1 minus epsilon. [14:58] If r theta would be greater than 1 plus epsilon, [15:01] we would be using 1 plus epsilon. [15:02] So we're basically keeping r theta [15:04] in between these values, and then the extremes [15:07] when we're going beyond that value. [15:11] Of course, this gets then multiplied by the advantage. [15:14] So you're starting to see how this [15:16] is, at least intuitively, an empirical way in which you can [15:20] somehow approximate this idea whereby we want to stay closer [15:25] to the original policy, because we're basically these values [15:30] are constraining the ratio r theta to be only as kind [15:36] of not incredibly different from the original policy, if that [15:39] makes sense. [15:44] Yeah. [15:45] What is better? [15:46] [INAUDIBLE] [15:58] So the point of the minimum here is that-- [16:02] so the full picture here is that we [16:06] are interested in the gradients that we get from this objective. [16:10] And specifically, we want to saturate the gradients [16:14] whenever we are beyond these clip values, [16:18] but only if these are in a negative direction. [16:23] Meaning, if we are moving very far away from the policy, [16:26] but we are moving in a direction that [16:29] is negative for performance, we want to clip it. [16:31] On the other hand, if taking a bigger step [16:35] would allow us to bring us closer to the previous policy, [16:43] we are still enabling gradients to flow. [16:45] So this minimum-- [16:47] I advise you to maybe even just plot this [16:50] and trying to see case by case scenarios. [16:52] But the overall intuition, I think, there are two. [16:54] One, this allows us to have a lower bound [16:57] on the actual objective. [16:58] And two, if you look at the behavior [17:00] of the resulting gradients that you get from this objective, [17:03] you're getting something that saturates whenever [17:05] it is actually good to saturate, and it [17:07] doesn't when you want to recover, essentially. [17:09] Yeah. [17:13] But yeah. [17:14] So long story short, PPO is derived from the same intuitions [17:18] that we were discussing before from TRPO. [17:21] And this is definitely the most popular version [17:23] you will find out there. [17:25] And I would arguably say that now PPO is probably [17:27] one of the most, if not the most popular reinforcement learning [17:30] algorithm out there. [17:37] So before moving to today's lecture, [17:40] first of all, any questions so far? [17:46] So I want to spend just a couple of minutes really driving home [17:50] the main messages which we discussed [17:52] so far for model-free RL. [17:55] And so remember how we basically started all of our discussions [17:58] a few weeks back by talking about different ways [18:01] to estimate value functions, where we were originally [18:04] motivated by the appealing properties [18:06] of dynamic programming, where these were essentially [18:10] allowing us to decompose the problem over time [18:18] by enabling us to have an exact representation of the value [18:22] for each state at the end of our problem [18:25] by just looking at the one step propagation. [18:29] And we were able to do this because we are actually [18:31] able to compute exactly those expectations [18:34] through the use of the knowledge of the system dynamics. [18:37] And so saw how these methods were in principle exact [18:40] because we were not losing any information through this one [18:44] step expectation, but it required [18:46] us to have a knowledge of the underlying MDP. [18:49] And so this motivated to introduce [18:51] two fundamental model-free learning paradigms, Monte Carlo [18:54] learning, which basically used sampling from our environment. [19:00] In other words, interactions with our environment [19:02] to approximate that expectation and specifically define [19:06] the update to our value function as taking a step [19:09] in the direction of the observed returns. [19:11] So we would sample in our environments, [19:13] reach a terminal state, observe a return, [19:15] and update our estimate of the value [19:17] function towards that return. [19:20] On one hand, this is great because it's unbiased. [19:22] This is an actual unbiased estimator [19:24] of the ground truth return. [19:26] But it is high variance because there [19:28] is a lot of stochastic sampling happening over time, [19:31] and we need to reach a terminal state in order [19:35] to actually sum backwards all of the rewards [19:37] that we have obtained. [19:39] On the other hand, temporal difference learning [19:41] tries to combine ideas from both of these paradigms, the sampling [19:45] of Monte Carlo because we're still [19:47] interacting with the environment, [19:48] and the bootstrapping of dynamic programming [19:51] because we are updating our guess towards the one step ahead [19:55] estimate of our future return. [19:59] Which is great, because this lowers the variance [20:01] because we're sampling only once, instead of having [20:04] to account for all the stochasticity [20:06] of the possible futures and it can learn online, [20:09] because we can do this update with just a single step. [20:13] But it does introduce bias because we are updating [20:16] the [INAUDIBLE] I guess. [20:18] And so there are some trade-offs. [20:21] Of course, we mentioned how this is true both for state value [20:24] functions V, but also for Q functions, [20:27] and this holds equally. [20:31] The other main idea is that we saw how all of these ideas [20:35] basically held true for the tabular representations, [20:39] whereby value functions are just lists or matrices, [20:42] and each entry in those lists of matrices [20:45] is our estimate for the value function. [20:49] And it holds similarly for value function approximation. [20:53] Whereby rather than having lists or matrices, [20:56] we have a parametric function described [20:58] by some parameters theta, which we can basically [21:01] tune in order to better approximate these values. [21:04] How do we tune them? [21:06] Exactly how we were defining before these backups [21:09] for the tabular case. [21:11] Now the same targets, that is the return and the TD target [21:15] represent elements in a update for in parameter space. [21:23] With all of these machinery, we then [21:24] looked at value-based methods and policy optimization methods [21:27] as two broad families of model-free reinforcement [21:31] learning approaches. [21:33] In value-based methods, we used generalized policy iteration [21:36] as the fundamental scheme in order to obtain optimal policy. [21:41] In generalized policy iteration, loosely speaking, [21:44] alternates between two processes, a policy evaluation [21:50] step, whereby giving a policy, we estimate [21:52] the value for the policy. [21:54] And the policy improvement step, which on the other hand, [21:56] tries to be greedy with respect to the newly updated value [22:00] function. [22:02] And we saw on-policy and off-policy algorithms, [22:04] namely SARSA and Q-learning as specific examples, [22:07] and how these were instantiations [22:08] of that same framework. [22:12] Of course, this applies also when [22:13] you have deep neural nets in the picture, [22:16] and we looked at deep neural networks as a specific example. [22:21] For policy optimization, which we just concluded today, [22:24] we, on the other hand, said that in policy optimization [22:28] we care about learning an explicit parametric policy theta [22:31] pi theta with some parameters theta. [22:34] And we want to optimize these parameters to directly optimize [22:39] the reinforcement learning objective, as we said today. [22:42] And this at a very high level entails well, [22:46] estimating the gradient if we want [22:47] to solve them through gradient descent techniques. [22:50] And so all of our discussions of policy gradient [22:52] fall into step one here. [22:54] And then we want to do approximate gradient [22:56] descent using this gradient. [22:59] We saw how policy gradient naturally [23:02] describes this idea of trial and error [23:03] learning by essentially being a weighted version [23:07] of the maximum likelihood gradient. [23:10] If you recall here, we were describing this idea [23:13] where the policy gradient tries to basically push up [23:16] the probability assigned to actions which have high reward. [23:22] And so this encodes this idea of trial and error learning. [23:29] But we saw how policy gradient is fundamentally doomed [23:32] by this problem of high variance of the gradient itself. [23:35] And so we discussed a number of different strategies [23:37] to reduce the variance. [23:38] Namely, we looked at baselines and we [23:40] introduced critics in actor-critic methods [23:43] and so forth. [23:47] So if you wish this is like a 3-minute run through everything [23:50] we've been doing over the last few lectures. [23:51] But I think it's good to make sure that we are [23:54] driving the main ideas home. [23:56] But any questions on this? [24:06] So remember that basically we were introducing this skeleton, [24:13] and all of the methods we discussed so far, essentially [24:16] are different instantiation of this key RL skeleton. [24:20] So we always have this generate samples step where we [24:24] interact with an environment. [24:26] We typically could estimate a return or a value function [24:31] of some sort, so we could be having a value function [24:33] approximator, for example. [24:35] We'll see today how we could fit the model as well. [24:38] And then we could use whatever we do in the second step [24:41] to improve the policy, either through a greedy policy [24:45] improvement, as we're doing in Q-learning, [24:46] or through gradient descent, as we're doing [24:49] in policy optimization methods. [24:52] And at the beginning of our reinforcement learning lectures, [24:56] we briefly mentioned a number of trade-offs [24:58] between algorithms and ways to navigate these algorithms. [25:02] But I hope that maybe before these trade-offs were, [25:06] I guess, a little bit vague. [25:07] But now if we actually look at those same slides again, [25:10] I hope that most of these trade-offs start to make sense. [25:13] So we're discussing trade-offs in terms of sample efficiency [25:16] and stability. [25:17] And so I think here on-policy and off-policy algorithms are [25:20] exactly hinting at these trade-offs over here, [25:23] that is, how much is it expensive [25:26] for us to interact with the environment? [25:29] We looked at continuous or discrete action spaces, [25:33] which could have implications in choosing value-based methods as [25:38] opposed to policy optimization methods. [25:40] We looked at episodic or infinite horizon [25:42] cases, where Monte Carlo requires an episodic setting, [25:46] because we need a terminal state to back propagate those values. [25:52] And loosely speaking, I think we started [25:54] to explore this spectrum of sample efficiency. [25:59] So we were looking at, for example, [26:01] ranging from the less efficient algorithms, [26:04] all the way here we see on-policy, policy gradient [26:07] algorithms. [26:08] We have actor-critic algorithms, which [26:10] try to leverage value function approximation to become [26:14] more sample efficient. [26:15] We looked at off-policy Q-learning [26:17] as example, where we could use offline data or data gathered [26:20] from other policies, which makes it more sample efficient. [26:23] And today we'll be introducing some key ideas [26:27] in the realm of model-based RL. [26:29] But long story short, is that we introduced these trade-offs, [26:34] I guess, pretty vaguely in the first lecture. [26:37] But hopefully you're appreciating [26:39] how many of these trade-offs now are more concrete. [26:47] And so let's move to model-based approaches. [26:51] Specifically today, we'll be starting from-- well, [26:55] introducing a basic recipe for model-based reinforcement [27:00] learning. [27:02] And we'll discuss the key limitations and challenges that [27:06] come from this basic recipe. [27:09] And then we'll start to see basically different ways [27:11] of addressing those challenges and limitations, namely [27:15] through the lens of uncertainty quantification. [27:17] And so we'll have a small parenthesis on uncertainty [27:20] quantification in general. [27:22] And then we'll conclude with some examples, as well as [27:25] some more broader scope remarks for the entire class. [27:31] The caveat here is that, of course, [27:33] model-based RL is goes clearly beyond anything [27:38] that we'll be discussing today. [27:40] But for today, I hope that at the very least, [27:42] we'll be conveying the main ideas and challenges that [27:45] come through model-based reinforcement learning. [27:51] So let's start from introducing this general recipe. [27:54] So at the very kind of fundamental level, [27:58] if we knew the dynamics, that is a function that basically [28:02] predicts the next state, given the state and an action [28:04] that we take in that state, we could use tools [28:08] from optimal control. [28:09] So basically pick a method from the first half [28:12] of this class we could use. [28:13] We have a model, we could plan through it, [28:15] we could use a direct method, indirect method. [28:18] Basically, our favorite choice. [28:20] And so a very simple idea would be, [28:23] well, we want this object because it [28:25] could allow us to use any of the tools we discussed before. [28:29] Let's use interactions with the environment [28:32] to collect data and examples of these transitions and let's [28:36] learn an approximate model of this ground truth dynamics. [28:43] And so we could apply a strategy that [28:46] looks like the one shown here on the slide, whereby [28:49] we might have a base policy pi 0, in this case. [28:53] This could be random. [28:54] It could be an exploration policy [28:56] that is given to us from some domain knowledge or anything [28:59] that we choose. [29:00] And we could collect and we would [29:02] run this in the environment and use [29:04] this to collect a data set of transitions, which are basically [29:08] comprised of these triplets over here. [29:11] So we would be running our policy in the environment. [29:17] In doing so, we would be seeing examples of these one step [29:21] transitions, which we can then collect to basically [29:26] be examples of mapping from state and control [29:31] to a next state. [29:34] Once we have this data set, we can then fit a dynamical model [29:38] to this data through our well, favorite regression [29:41] or supervised learning technique. [29:44] For example, minimizing some form of squared error [29:46] or maximizing some log likelihood of our data. [29:49] And then we could use this model to basically plan. [29:54] So the question here is, would this work? [29:57] Well, in some cases, yes. [29:59] So especially if dynamics are relatively simple, for example, [30:05] we could have some linear time invariant dynamics. [30:07] This tends to work pretty well. [30:10] And this also relates to some of the discussions we were doing [30:14] on system identification, whereby [30:17] if you have a fairly well-understood model, [30:20] if the underlying dynamics are within the function class [30:24] that we're trying to fit and maybe we only [30:26] have to tune a limited number of parameters, [30:30] then these tools actually work quite well, [30:33] and we might hope to generalize good enough to the real system. [30:41] Of course, this doesn't hold in probably the majority of cases, [30:45] specifically when we're dealing with complex, potentially [30:48] nonlinear dynamics, and especially [30:51] in the combination of complex dynamics, [30:53] and high capacity approximate models, [30:56] for example, neural networks. [30:59] And this is because extrapolating from observed data [31:02] to, for example, out-of-distribution examples [31:05] from our environment is difficult and can be misleading. [31:09] So for basically the rest of this class, [31:11] we'll be looking at ways to basically or approaches [31:16] to try and have this exact same formalism work in the cases [31:20] where we might be having complex dynamics to learn. [31:30] So why is this an issue? [31:33] So let us look at a fictitious and a pictorial example. [31:38] So say we are in this very simple world, [31:40] the agent is the black dot here on the slide, [31:42] and we have an obstacle which we wish to avoid. [31:45] The goal here is to go as further north as possible. [31:49] So in this case, say that our exploration policy pi [31:52] 0 is something that explores a certain amount, certain subset [31:56] of the state space, specifically describing the state [32:01] distribution under policy pi 0 P sub pi [32:04] 0 of x, which is basically the subset of the state [32:06] space induced by that policy. [32:10] Of course, if we were to learn now [32:12] a dynamical model on this transitions, which we collected [32:15] through this experience, we might-- and then [32:19] we use this for planning, we might [32:21] be ending doing something extremely meaningful [32:25] within the realm or the states that we have observed. [32:28] But the moment where we change and we [32:30] go outside of the state distribution we have covered, [32:34] this is obviously not the case. [32:36] And so if you remember also our discussions [32:38] on imitation learning, this is another example [32:42] of covariate shift and distribution shift, [32:46] whereby basically, when we use the learned dynamical model [32:50] as an inner component of our planning algorithm, [32:53] we're then observing a different state distribution. [32:57] And so here generalization and extrapolation [33:00] across these settings is actually what matters. [33:07] So what could be first fairly heuristic and empirical way [33:13] to improve the basic recipe we were describing before? [33:18] So again, let's consider our three steps where [33:20] we take our policy, run it in the environment, [33:23] collect some data, use this data to fit a model, [33:27] and then say that we want to use this to plan. [33:29] Of course, what we could do, first of all, [33:31] is to plan in a receding horizon fashion. [33:35] So we know that the plan might be slightly off or at least [33:41] might be accurate closer to our initial time, [33:44] but then we might want to reevaluate where we actually are [33:48] in the environment and replan. [33:50] And so in this case, using a model predictive control [33:53] approach might be beneficial. [33:55] So this is a simple improvement here. [33:59] But also what we could do is we could basically [34:03] use the transitions that we are observing [34:06] while deploying this policy to basically add them to the data [34:09] set and refit our model. [34:12] So in a way, we are gradually closing [34:14] the gap between the state distribution induced [34:17] by the original policy and the state distribution induced [34:21] by this planning policy, if that makes sense. [34:30] So these are indeed two simple steps [34:33] which can in practice make a substantial improvement. [34:39] However, I would say the main focus [34:42] for today is on uncertainty quantification. [34:44] And we'll see-- we'll see why shortly. [34:49] Specifically, the reason why these techniques fail [34:57] is that in theory, what we would like to have is-- well, [35:01] we have a very complex dynamical model or system [35:03] we're interacting with, and we would [35:05] like to have a high capacity model that [35:08] is able to represent these complex dynamics [35:11] in the high data regime. [35:12] So we want to have high capacity models. [35:14] However, we also want this high capacity model [35:17] to not overfit to the observed data [35:20] if we only have a limited set of that data. [35:23] The reason why we don't want this overfitting problem [35:27] is because if we then use an optimizer or a planning [35:33] algorithm, for example, as we were doing [35:35] in our previous recipe to optimize over our learned model, [35:39] the optimization will be prone to exploiting errors [35:45] in the positive direction, if that makes sense. [35:49] So visually, say that we have only a limited number [35:55] of data points represented by the red dots here on the slide, [36:01] here we have two equivalent functions that fit the data. [36:06] But the blue one here is potentially [36:08] representing a higher capacity model that [36:11] is overfitting to the data. [36:12] So it's fitting perfectly our observed data, [36:14] but then it is doing-- it is behaving quite wildly [36:20] outside of the observed data. [36:23] Now, say that you are an optimizer that [36:25] wants to maximize the performance from the blue model, [36:29] you might be tempted to choose behavior that basically leads [36:33] to exploiting the positive reward [36:36] predictions under our model. [36:37] And so I think this is a key problem here [36:41] and a key tension when you want to use [36:43] learned models on one hand that might not extrapolate [36:46] or generalize well with an optimization that [36:50] is in a way acting as a black box optimizer [36:53] and could exploit errors in the positive direction. [37:01] So for today, what we will be looking [37:04] at is a possible solution to this problem, which is basically [37:08] considering uncertainty and loosely speaking, how certain [37:13] our model is about its prediction as a mechanism [37:16] to basically push against these possible flaws. [37:25] Does that make sense? [37:26] Any questions on this? [37:35] So first of all, we're talking about uncertainty estimation [37:39] and what do we mean by uncertainty on the predictions [37:43] of our model. [37:44] And specifically, what we mean by uncertainty [37:47] is we mean an expression of a distribution [37:51] over possible outcomes. [37:53] So for example, instead of predicting a single point [37:58] estimate of what is the most likely next best [38:02] state from given a state and a control, [38:05] we want our system models to predict [38:09] a full distribution and associated likelihood [38:12] of what is most likely to happen in the future, [38:15] if that makes sense. [38:16] So visually, what we want from our dynamical model [38:18] is something that gives us a range of possible scenarios [38:22] that we could be optimizing for with the associated probability. [38:28] We want this because this essentially [38:30] allows us to reason in terms of expectations under our model. [38:37] So why is this-- [38:38] why can this really a form planning? [38:41] So consider again this simple example, [38:44] where we are the black dot on the slide. [38:48] We want to reach a goal state represented by the green star, [38:51] and we have this cliff-like environment where the lines here [38:56] represent the edge of the cliff. [38:58] So if we go beyond this, episode terminates essentially. [39:02] Now, say that we have two candidate open loop [39:06] trajectories. [39:07] So we basically assume we have access [39:10] to U1 to UT prime and U1 double prime to UT double prime. [39:26] And say we want to use something along the lines [39:28] of the model-based algorithms we're using before, we would [39:33] basically be simulating in our head [39:36] these two possible action trajectories, open loop [39:39] trajectories, where one would lead [39:41] us to very close to the goal and one would lead us [39:45] a little bit more further away. [39:47] In this case, the circles represent [39:49] our estimate of how certain we are about where we would end up. [39:54] In this case, both. [39:56] In both cases, we are fairly certain about where [39:58] we are ending up, and so it is clear [40:00] that the trajectory associated with the yellow circle [40:04] is better because it allows us to be closer to the goal. [40:09] However, despite having in a second case assumed that now we [40:14] have broader uncertainty bands around our terminal point, [40:18] despite the terminal the best-- the expected value [40:22] being the same in both cases, now we [40:25] have a greater uncertainty over those. [40:27] And so the expected reward under this higher variance prediction [40:32] would be lower for the yellow circle. [40:34] Because under this new uncertainty bound, [40:37] we are actually considering the case [40:39] that in actually the majority of the cases [40:41] we are ending up off the cliff, if that makes sense. [40:49] So the reason why we care about uncertainty estimation [40:51] is because it assuming we are able to do this. [40:54] So we're able to learn a model that [40:56] gives us an estimate of how certain [40:58] we are about our prediction. [41:01] It gives us a mechanism to express performance [41:04] as expected reward under our model. [41:22] Any questions here? [41:25] Cool. [41:28] So the next question is, how can we model uncertainty? [41:32] We say we want to do this. [41:33] What are different ways of doing so? [41:36] So the first idea is we could use the output [41:39] entropy from our given model. [41:43] So again, pictorially, what we are trying to do [41:47] is we are trying to learn a parametric function, which [41:50] maps from states and controls to probabilities over next state. [41:56] In order to do this, we might be fitting, either-- [42:00] depending on whether the state space is discrete or continuous, [42:03] we might be fitting either a categorical distribution [42:06] over next states. [42:07] So we're assigning a probability to each [42:09] of our possible next states. [42:11] Or we might be fitting a continuous distribution, say [42:15] a Gaussian, which basically expresses [42:18] the mean value of where we expect to be [42:20] and our standard deviation and the variance of our uncertainty. [42:25] In this case, of course, we could [42:28] use the entropy that basically comes from the definition [42:31] of the distribution itself. [42:33] For example, we might be using the standard deviation [42:36] of the fitted Gaussian. [42:41] However, model entropy is different from what [42:47] is often referred in literature as epistemic uncertainty. [42:51] So you might have heard about this distinction before, which [42:54] is basically the distinction between aleatoric uncertainty, [42:57] that is the noise that is intrinsically related [43:01] with the process that we are observing. [43:04] For example, here on the slide, we see this parabolic process. [43:09] We have some observations of this process, [43:12] and there is some inherent noise that might [43:15] be observed from this process. [43:18] The other type of uncertainty is epistemic uncertainty, [43:21] whereby we are actually uncertain about the model [43:25] itself. [43:26] So here we were certain about, for example, [43:29] the specific parameters of this parabola. [43:32] It's just there was some inherent noise that [43:33] is impossible to model. [43:36] On the other hand, again, using this kind of fictitious example [43:41] over here, we might have two possible alternative models, [43:46] the green one and the blue one, which [43:48] with associated different parameters, which [43:50] fit equally the model, the observed data, but are just [43:53] two different models. [43:56] So here we will have two different values [43:58] of the parameters theta that describe these two [44:00] different curves. [44:01] And so this is an uncertainty between on the model itself. [44:05] So which of these two realizations of my parameters [44:08] is actually the one that is better representing my data. [44:19] So using the entropy gets us some way along this path. [44:24] But it clearly does not account for what we actually care about, [44:29] which is the potentially mutually [44:33] coexistent and alternative explanations of the data [44:36] according to different models. [44:46] So how do we model for this-- [44:49] how do we estimate for this model uncertainty? [44:52] I would say probably the most common approach [44:56] for doing so is leveraging Bayesian statistics [44:59] and Bayesian approaches, whereby typically what we are [45:05] interested in doing with Bayesian [45:07] approaches is that we want to basically express this model [45:12] uncertainty as a distribution over parameter values [45:16] theta given the observed data. [45:20] So until now, when we were discussing fitting a model [45:24] to our data, we're often just saying [45:26] that we would be converging to a specific value [45:31] of our parameters theta. [45:33] On the other hand, if we are uncertain about the model [45:36] itself, we might be interested in modeling [45:39] maybe a full distribution of how certain [45:41] we are about what the real value of the parameters theta is. [45:47] And so this is what we mean by the distribution over parameters [45:51] given the data set. [45:54] That is we are given some observation [45:55] and we want to ideally model the full distribution [45:58] of plausible hypothesis. [46:05] Why is this useful? [46:06] Is because once we have this full distribution, [46:10] we could actually predict through our learned dynamical [46:14] model by basically averaging out all [46:18] the possible future predictions that [46:20] would happen if we were to basically average out [46:22] over this full distribution. [46:25] And so this is referred to as the predictive posterior [46:29] distribution. [46:34] Formally, this is expressed through this integral over here [46:37] where we're basically saying that we're [46:39] computing the prediction of our next state averaged [46:42] over all possible values of my parameters theta. [46:48] I think as often the case, to visualize [46:52] these integrals is often useful to see [46:55] this in the discrete setting. [46:56] So say we have only two values of our parameters theta. [47:00] This integral, this predictive posterior distribution [47:03] would essentially entail running my predictions [47:06] for both of my values of theta, and then [47:09] weighting the prediction based on the probability [47:11] of each individual theta. [47:14] This is the continuous expansion of that same idea. [47:21] How does this look in practice? [47:23] Well, of course, we won't have time to cover this here. [47:26] But Bayesian statistics basically provides [47:29] and the Bayes' theorem provides basically a mechanism [47:32] to retrieve the posterior distribution over parameters [47:36] from data. [47:38] But let's look at what this means in practice. [47:42] For example, say we want to solve a regression problem where [47:46] we have some-- [47:48] well, we have an x. [47:50] So we want to predict the value of y given x. [47:54] We want to fit here just a line to this data. [47:57] So we have two parameters that we want to fit. [48:01] So the intercept and the slope. [48:03] Say that we have only one observation from our ground [48:07] truth process, which is the dashed line over here. [48:11] If we fit our linear model to this data, of course, [48:15] there could be many possible-- well, [48:17] in principle, infinite lines that [48:19] pass through the point, which in practice would mean [48:22] we would have a very broad uncertainty over possible values [48:26] of our parameters represented by this plot over here. [48:29] So here we're plotting the two parameters, [48:31] say the intercept and the slope. [48:33] And so we could see that while the ground truth parameter is [48:37] represented here by the red cross, [48:40] our posterior distribution is still pretty broad. [48:44] What does this mean in practice? [48:45] Is that if we were to sample, n, realizations of my posterior [48:50] distribution and plot the line, you [48:52] would get these different red curves. [48:55] If we would then average out according [48:57] to the predictive posterior distribution, [48:59] we would get these confidence bounds [49:01] that we have on the right. [49:03] Are the plots clear? [49:11] Yeah. [49:12] And so what we're showing here is basically progression [49:15] as we collect more data. [49:17] So here we were looking at the case [49:19] where we had only one data point. [49:21] Here we have more data points. [49:23] And so our posterior distribution over parameters [49:25] is more narrow and closer to the ground truth. [49:28] And you see how the overall uncertainty [49:30] bounds that come from the posterior predictive [49:32] are also more concentrated, until here we [49:36] are observing enough data to have a pretty reliable estimate [49:40] of the ground truth model. [49:51] And so given that this is clear, let's briefly [49:56] mention a couple of, I would say, [49:58] the most popular approaches for modeling dynamical [50:02] systems in a way that we are accounting for this uncertainty, [50:05] and we'll be looking at two. [50:07] The first one is Gaussian processes, [50:09] and the second one is ensembles. [50:14] So a Gaussian Process loosely speaking [50:17] is-- basically represents a distribution over functions. [50:21] So you can imagine that if we are sampling [50:26] from a prior distribution or functions, that [50:29] is before observing any data, we will be getting [50:32] pretty different behavior. [50:34] On the other hand, Gaussian processes basically [50:38] formalize analytically our mechanism [50:41] to go from a prior distribution over functions [50:44] to a posterior distribution over function [50:46] in light of observed data, which would basically be still [50:50] a distribution over function that [50:52] is consistent with the data at hand. [50:59] When I first approach also Gaussian processes, [51:02] I would say one of the most confusing aspects [51:05] to me was this idea of distribution over functions [51:07] and what actually meant. [51:09] In practice, Gaussian processes model data [51:13] as multivariate Gaussians. [51:14] So this is why they're referred to as Gaussian processes. [51:18] And basically encode this idea that n-dimensional Gaussian, [51:25] for example a two-dimensional Gaussian [51:30] can be mapped-- so a sample from n-dimensional Gaussian [51:34] can be mapped to basically an approximation of a function. [51:38] So say that we are sampling from this two-dimensional Gaussian, [51:41] say this point over here, which could be 5 and 4. [51:48] Now, we could take this two-dimensional basically a [51:52] vector, which we just sampled, and if we plot it still [51:57] in two dimensions, what we could get [51:59] is that we could get a first value of 5, a second value of 4. [52:06] And by connecting this, we're basically [52:07] getting a discretization of a function now described only [52:13] in this small interval. [52:15] Now, of course, we can only plot things in two dimensions. [52:17] So maybe this might not yet look like a function to you. [52:20] But now imagine that we're sampling an n-dimensional vector [52:23] from a multivariate Gaussian. [52:26] Here we would get just a vector with more numbers. [52:29] And if we plot them in the same way, [52:31] we would basically get the expression-- [52:35] the approximation of a function. [52:43] So this is usually how these are implemented in practice [52:47] and why people refer to these as distribution over functions. [52:51] In practice, you can imagine that Gaussian processes [52:53] essentially provide a mechanism to update [52:57] from a prior distribution over functions to a posterior [53:01] distributions once we have observed data. [53:08] Again, we're sticking to intuition, [53:10] but I hope this is making sense. [53:12] Yeah. [53:20] So here I was trying to provide an intuitive [53:24] understanding of what people refer to as [53:27] distribution over functions. [53:29] So a Gaussian process is essentially [53:32] representing multivariate Gaussian distribution. [53:36] So for example here, I was representing it [53:39] in two-dimensional. [53:40] So if we have this two-dimensional Gaussian, [53:43] a sample from this two-dimensional Gaussian [53:45] could just be this point over here. [53:48] Here I was imagining 4 or 5 as coordinates. [53:52] If we would be wanted to plot this point as a function, [53:59] essentially, this would essentially [54:02] be something like this, where we are basically [54:05] mapping this point and these coordinates of this point, [54:08] but just unfolding it into a function, if that makes sense. [54:12] Now, the intuition is that what happens [54:15] if I want to build a full continuous function as we were [54:19] doing before, you could imagine that it's basically [54:22] a more finely discretized version of this [54:25] with an n-dimensional vector. [54:27] So rather than having two-dimensional Gaussians, [54:30] I would have an n-dimensional Gaussian, say 100, [54:34] then we would be able to basically represent [54:37] points on a very fine grid. [54:40] And by basically plotting these points, [54:42] you would get a function out. [54:44] Yeah. [54:46] Yeah. [54:47] So is it kind of like [INAUDIBLE] [54:50] the values of the function? [54:52] Or is it like-- [54:55] does it have to be a sequence of numbers [54:57] that the function would output? [54:59] Yeah. [55:00] So maybe to rephrase what you're saying. [55:01] Here we are implicitly assuming an order of the dimensions. [55:05] Yeah. [55:07] But from the perspective of learning [55:08] this doesn't really change anything [55:10] because we are still fitting the Gaussian that we are learning [55:15] to a high-dimensional object. [55:17] So for us, data will be living in this high-dimensional space. [55:21] And we're just fitting a n-dimensional Gaussian [55:24] to that data. [55:26] This is just the way we would then plot it. [55:28] Yeah. [55:35] So why are Gaussian processes popular? [55:38] First of all, Gaussian processes are extremely data efficient, [55:43] and they provide an analytical expression [55:47] for the exact posterior, that is the exact posterior [55:51] over functions described by your data. [55:54] Meaning, given some data points and some assumptions [55:58] on Gaussianity, this distribution [56:03] is actually the exact representation [56:05] of the posterior distribution over functions [56:07] that explain this data. [56:09] This is pretty cool, because if we wanted to achieve something [56:14] similar with neural networks, this would be very hard, [56:16] especially with this amount of data. [56:19] Because this is an extremely low data regime, [56:22] and so getting these extremely well-behaved confidence bounds [56:26] is very challenging to have. [56:28] On the other hand, Gaussian processes achieve this [56:33] by having an exact representation of the posterior. [56:38] On the other hand, I mentioned analytical expression [56:43] of the posterior. [56:44] This analytical expression, at least in the nominal case, [56:48] entails having to do some matrix inversions. [56:51] And so depending which as a function of n, where [56:54] n is the number of data sets of data points, [56:57] and so scaling Gaussian processes [56:59] to high-dimensional data is actually complex. [57:04] And so there are extensions of Gaussian processes [57:06] that try to do so. [57:07] But at least in their face value, [57:11] this is definitely a weakness. [57:16] A second approach for estimating uncertainty [57:20] is referred to as bootstrap ensembles. [57:23] And the idea is that-- [57:26] so we have this general mechanism [57:28] for learning dynamical models through supervised learning. [57:33] That is, we just fit a black box model to our data, [57:37] and we want to have an encode an idea of uncertainty [57:43] in this estimation problem. [57:45] So what we can do is that the idea of ensembles [57:49] is let's train multiple models and see if they agree and where. [57:54] With the underlying idea that these different models will-- [57:58] so different models trained independently [58:00] to fit the same data will likely agree [58:03] when they are certain about their predictions, [58:05] and they will likely disagree whenever there is uncertainty. [58:08] So say you have n versions and copies of the models that have [58:12] been trained independently. [58:15] If they will be probably agreeing [58:17] close to the regime of observed data, [58:19] and they will potentially disagree when we are [58:21] going far away from our data. [58:25] And so ensembles basically instantiate this idea [58:30] through an algorithm, which is basically saying nothing [58:35] else that we're saying here. [58:36] In ensembles, you would just train independently and copies [58:38] of your model, and then average the prediction [58:42] according to these n models. [58:47] So specifically, what you would be doing is [58:49] we say we have five different neural network [58:53] models, which we train. [58:56] We could then get five different predictions for our next state. [59:01] And so what we can then do is maybe [59:03] our prediction would be the average prediction [59:05] across these models. [59:06] This could be one idea. [59:15] And maybe another way to visualize this is [59:20] say that we have-- so here the x-axis represents the parameter [59:23] space theta, and the y-axis represents our-- basically [59:29] this is the posterior distribution [59:31] over parameters given data. [59:34] So say that the ground truth posterior distribution [59:36] is represented by the shaded multimodal distribution here. [59:42] And for now disregard VI, which stands [59:46] for variational inference, and this multi-SWAG algorithm. [59:50] The ensembles would essentially mean, in this case, [59:54] fitting for different models, the blue ones, independently. [60:00] Through optimization, these four models [60:03] will converge to for different values of the parameters theta. [60:07] And so these parameters theta might [60:10] converge to different local minima, [60:12] depending on how the optimization goes. [60:14] And so these are essentially an empirical approximation [60:17] of the ground truth posterior distribution. [60:20] Because they will naturally-- because of the stochasticity [60:24] of stochastic gradient descent and the optimization process, [60:28] they will converge to different local minima. [60:31] And so these are a way of empirically exploring [60:35] the landscape and approximating the ground truth posterior [60:38] distribution. [60:42] Yeah. [60:44] Are you saying the differences between the different ones [60:49] because the goal-- [60:54] here you have a multimodal distribution over theta, [60:57] your goal is to figure out what all the modes are by having-- [61:02] these each one of these you said neural networks. [61:06] Each one of these represent a different mode that you [61:10] hopefully get [INAUDIBLE]? [61:12] Yeah, indeed. [61:13] So the idea here is that this learning problem, very likely [61:17] has a highly multimodal landscape of solutions. [61:23] The approaches we were discussing before, [61:25] like Gaussian processes, try to derive analytical expressions [61:30] for this exact distributions. [61:32] Ensembles here, on the other hand, [61:35] disregard any of this formal approach [61:38] and just decide to instantiate n copies of these models, [61:43] maybe initialize them differently. [61:47] The stochasticity of gradient descent [61:48] still will lead them to converge to different final solutions. [61:54] And by doing so, the hope is that they [61:57] will cover meaningful different modes of the ground truth [62:01] posterior distribution. [62:04] So I think your original question [62:05] is, how do you differentiate between these copies? [62:10] I would say, if you're talking about neural networks [62:12] specifically, typically these models are initialized randomly. [62:16] So the initial values of the parameters are random. [62:19] And also we're optimizing them to stochastic gradient descent. [62:23] So even the directions that they're following [62:25] will be random to some extent. [62:27] So there is some natural stochasticity that comes. [62:29] [INAUDIBLE]? [62:37] Yes. [62:39] And then also, why Dirac distributions? [62:42] This might just be because I forgot [INAUDIBLE]. [62:44] Yeah, no, indeed. [62:45] So this is something that I did not discuss. [62:48] So the idea is that formally-- so I [62:51] think, hopefully in practice, it makes sense what we're doing. [62:53] So we're initializing for models, [62:55] and then we have for different converge parameters theta, [62:59] and then we could average over them. [63:01] What are we doing in practice is that we are approximating [63:05] the ground truth posterior distribution by taking [63:09] the average of what is referred to as [63:13] a Dirac distribution, which is essentially saying that-- [63:17] say that I have one observation of my theta. [63:22] The Dirac distribution is something that is basically [63:24] a spiky distribution that gives all the probability mass [63:27] exactly to that point. [63:30] And so this is, if you wish more formally, [63:35] how this process of averaging out predictions [63:37] from different models would translate [63:39] to the predictive posterior that we were describing before [63:44] through the lens of these Dirac distributions. [63:53] So remember that in the predictive posterior, what [63:55] we are saying is that we want to average out [63:57] over all the different models that we're training [64:00] or specifically average out over the posterior [64:03] distribution of our parameters. [64:07] So do you remember the discussion [64:08] we were doing on predictive posterior and what that means? [64:12] How do we compute those confidence bounds? [64:14] Is that we were basically-- [64:18] for each aim in the discrete case, [64:20] for each value of our parameters, [64:22] we were basically predicting with that value [64:25] of the parameters, and then weighting that probability-- [64:29] that prediction with the probability of how certain [64:33] we were of that specific value of the parameters. [64:37] In this case, we are doing so not in the continuous case, [64:41] but through these Dirac approximations. [64:46] Does that make sense? [64:57] But so long story short, what we've been doing so far [65:02] is that we introduced this basic recipe [65:05] for model-based reinforcement learning, [65:08] namely some version of run a policy in the environment, [65:14] collect transitions, fit the model to those transitions, [65:18] and then use that model for planning. [65:20] We motivated how uncertainty, or at least estimating uncertainty [65:24] on those predictions is likely very important. [65:27] And now let's look at some specific examples [65:31] and how we could make this work in practice. [65:34] So how can we make use of this additional knowledge [65:42] that we just gained? [65:45] Specifically, one mechanism to do so could be the following. [65:48] So let's assume that we have a candidate action sequence again, [65:52] some given to us by some planning algorithm. [65:56] And now we have approximate model [65:59] of the posterior distribution of the parameters theta, which [66:02] we learned through, for example, one of the methods [66:05] we discussed today. [66:09] So we could sample a specific value of our parameters theta [66:14] from this posterior distribution. [66:16] And we could now propagate the learned dynamics [66:19] through this specific instantiation [66:22] of the dynamical model. [66:28] We could then use this propagated trajectory [66:31] to compute a predicted reward. [66:34] So we're basically simulating in our head what is happening. [66:37] We are given a candidate action sequence. [66:39] We are sampling one of the models [66:42] from our posterior distribution over models. [66:44] And we're using this to propagate forward [66:46] what is happening, and scoring the quality of what [66:48] is happening. [66:52] And then we can just repeat these three steps [66:56] to basically compute the average reward [66:59] according to our learned posterior distribution. [67:06] So specifically, we would be-- so what this algorithm is doing [67:11] is directly instantiating this idea that we were describing [67:15] pictorially over here. [67:17] So in the plot there, we are saying that uncertainty gives us [67:20] this mechanism through which we can compute the expected reward [67:24] under our model. [67:26] And so in this way, we could basically [67:30] approximate the expected reward under all of our predicted [67:35] samples. [67:39] Does that make sense? [67:40] Maybe, again, more visually what we're doing [67:41] is that we are given a candidate action sequence. [67:49] We sample maybe one version or one parameters [67:53] from our posterior distribution and we propagate the model [67:57] forward. [67:58] Here we would be reaching a specific sum [68:02] of rewards over time according to this specific model. [68:07] We could then sample a different version of that model, [68:11] and we could have a different propagated trajectory, [68:13] with a different reward, right? [68:18] If we repeat this multiple times, [68:21] we would then have n of these propagated futures. [68:25] And so what we could do is that if we average them out, [68:28] we would then be computing an approximation [68:31] of the expected reward under the uncertainty of our model. [68:46] So of course, one thing that I really [68:50] want to make sure to convey is that this is obviously [68:55] just a choice. [68:55] So we're not takeaway way here, I think should not be-- [69:00] this is the best way to go for when you're [69:02] talking about model-based reinforcement learning, [69:04] because ultimately, it depends on the problem at hand. [69:07] So maybe there is even no need at all [69:09] to look at uncertainty quantification, [69:11] because just having your linear time invariant model [69:15] is well-estimated enough and so there [69:18] is no need to do any of this. [69:20] But in the general case where system dynamic approximation is [69:25] challenging and you want to use high capacity [69:29] models for approximating our dynamics, [69:32] uncertainty plays a big role. [69:34] And so this could be one instantiation [69:36] of this idea of let's use whatever favorite way [69:40] to estimate a posterior distribution over parameters [69:44] of our model, and then use this to meaningfully represent [69:48] expectation under the uncertainty. [69:55] And so just to repeat myself, this is only a choice. [69:59] But this is one instantiation of this more general idea. [70:02] That is, whenever we want to plan through a learned model, [70:05] we might want to be-- [70:07] we might be interested in evaluating the expected reward [70:11] under the learned model. [70:16] So a particularly impactful paper, now a few years old, [70:22] but that uses exactly these ideas, introduces algorithm [70:26] called PETS. [70:28] Whereby the main idea is that they [70:30] were using as a model, basically an ensemble of neural networks [70:34] to approximate the posterior over the model. [70:38] They were then using a sample, basically different models [70:44] from this ensemble, and use them to generate predictions [70:47] of different features, exactly as we were [70:49] saying in the previous slide. [70:53] And then they would be using this [70:57] in our model predictive control fashion. [70:59] So they would basically have a cross-entropy method [71:03] to generate this candidate plans. [71:09] Through the exact same algorithm, [71:12] which we just described, they will basically [71:14] take these candidate plans and score them [71:16] according to samples from these ensemble of neural networks, [71:20] and then pick the best action based on that, [71:23] and implement this in a receding horizon fashion. [71:26] So you predict the full plan, you [71:29] actuate only the first step in the plan, [71:31] and you repeat this process. [71:34] And so this is-- just a quick note, [71:36] but it's something that I think it's [71:37] a-- this is a nice paper if you want to read [71:39] about these models in practice. [71:46] Does it make sense how this is one instantiation [71:49] of what we're saying? [71:55] So in this paper specifically, and I'm using this [71:58] as an example to highlight the classical benefits [72:01] of model-based RL, is that here they [72:05] would be measuring performance of-- well, their method in blue, [72:09] compared to a bunch of different baselines. [72:12] But for example, they're comparing [72:14] against PPO or soft actor-critic, which is also [72:18] a version of the actor-critic methods we discussed [72:20] in class, across a number of different tasks. [72:24] And I would say the most interesting result here [72:27] is that these model-based methods, namely the blue curves, [72:31] generally are much more sample efficient. [72:34] So if you compare the performance [72:37] on the y-axis in general, with respect [72:39] to the number of time steps through which they [72:42] had to interact with the environment, [72:44] you could see that maybe the performance at convergence [72:47] is fairly similar. [72:49] But the blue curves typically tend to converge to this-- [72:54] converge performance much faster. [72:56] And this is because through learning the model itself, [73:01] we can basically circumvent a lot of the interactions [73:04] with the environment. [73:09] So generally, the benefits in model-based RL [73:12] often relate with sample efficiency. [73:25] So any questions before wrapping up the class? [73:36] Yeah. [73:36] What's your favorite method? [73:38] [LAUGHTER] [73:40] So I like to be fairly agnostic and pragmatic, depending [73:48] on the problem at hand. [73:50] I would say that-- [73:53] [LAUGHS] you're saying this is the diplomatic answer. [73:57] So I would say that the model-free algorithms are [74:00] definitely more mature as a technology, [74:03] and so I would say algorithms such as PPO [74:05] and soft actor-critic are typically the way [74:07] to go for continuous control. [74:10] Model-based RL is still very active line of research. [74:13] So if I have to pick one, I would go [74:16] for a soft actor-critic or PPO. [74:18] Yeah. [74:18] Well, those are two, but those two. [74:21] Yeah. [74:23] So I think this is a nicely way into-- basically [74:26] what I wanted to discuss here. [74:28] Because throughout this class, we [74:30] have been introducing n different ways of behaving when [74:35] looking at optimal control. [74:37] So ranging from open loop methods, [74:39] it could be indirect and direct, closed-loop, dynamic [74:42] programming, reinforcement learning, imitation learning. [74:45] And so the question that you could ask is, [74:47] I mean this is great, but you're just confusing me. [74:49] Now I have to pick among hundreds of these algorithms, [74:52] how do I navigate this? [74:54] And the point here is that these methods are not [74:59] in competition with each other, but they're actually-- well, [75:03] different methods will be tailored [75:04] to different applications. [75:06] And actually most of the time, it's these methods [75:08] are combined with each other. [75:10] And this is because the decision-making pipeline [75:16] is typically organized as through [75:18] a hierarchical structure. [75:21] Specifically, if we were to decompose [75:23] this hierarchical structure, generally, we [75:26] would start with having a model of the world or a representation [75:30] of what is interesting for us. [75:32] For example, take the case of autonomous driving. [75:37] We might have some maybe semantic maps [75:39] of what we are observing in our environment. [75:42] And given whatever representation of the world [75:45] we are interested in, we would then [75:46] have the autonomy stack is typically [75:50] decomposed through different modules that [75:52] interact with each other. [75:54] So the first kind of decision that you have to make [75:57] is typically some higher level goal for our system. [76:00] So we are perceiving the world, and now we [76:01] have to choose what we want to perform. [76:05] Specifically, these high-level decisions [76:07] might be continuing the autonomous driving example, [76:10] something along the lines of picking if I [76:13] want to change a lane or not. [76:14] So these very high-level actions and decisions [76:18] that we want to make. [76:19] At this level of the hierarchy, this [76:22] is typically where methods that belong [76:24] to closed-loop dynamic programming like algorithms lie. [76:28] And this is because for these higher [76:30] level decision-making aspects, we typically [76:33] want to encode stochasticity and want to make sure [76:36] that we are dealing with this stochasticity [76:38] when picking decisions. [76:41] And the output of this is typically in a form of a goal, [76:45] for example, lane change. [76:48] Now we have this goal, but we have to make it actionable. [76:51] So we want to make this such that we [76:53] have an actual trajectory which we can enact in the world. [76:59] And so this is something that usually is dealt [77:01] with open-loop techniques. [77:03] So we would have this higher level goal, [77:05] like a navigation goal. [77:06] And then we want to translate this navigation goal [77:09] into a finer grained trajectory, which we want our car to follow. [77:14] This trajectory obviously encodes some notion [77:17] of the dynamics of the vehicle, despite this [77:20] might be maybe some lower fidelity models of the dynamics. [77:24] But still we want to encode some constraints [77:27] in terms of safety, some constraints in terms [77:30] of dynamics. [77:31] And so this is where open-loop techniques typically fall. [77:37] Once we have a trajectory, we want [77:39] to make sure this is physically tractable and implemented [77:46] on the hardware. [77:47] So this is where usually ideas from model predictive control, [77:54] specifically in its tracking instantiations commonplace. [77:58] So we would have a reference trajectory [78:00] that is given from the open-loop algorithm. [78:03] And now we want to make sure, first of all, [78:05] that we're tracking this over time, [78:07] but also potentially including a finer grained dynamical models [78:11] of the system, maybe some safety-oriented constraints, [78:18] which we were avoiding in the open-loop stage, and so forth. [78:24] So we start from representation of the world. [78:27] We use a higher level decision-making [78:30] to have a high-level goal for our system. [78:33] We compute a trajectory that tries to achieve that goal, [78:36] we then want to implement this trajectory in practice, [78:38] so we want to make sure we are tracking it. [78:41] And obviously last but not least, we [78:43] feed these to the actuators. [78:46] So at the end of the day, we have [78:48] to make this work on the real-world hardware. [78:51] And so we want to make sure that this is actually tracked by what [78:55] is happening in reality. [78:56] And so this is usually where things like PID controls [79:00] come in place. [79:02] Of course throughout all of this, [79:05] we might have some safety concerns [79:08] and we could be dealing with safety critical applications. [79:12] For example, in the case of driving, [79:15] we might have some set of states which we do not [79:19] want to visit because of some safety concepts which [79:22] we care about. [79:23] And so the safety concepts which may be computed through things [79:28] such Hamilton-Jacobi reachability [79:30] are typically mechanism through which [79:32] we can inform both or either or either of open-loop [79:37] and the tracking module. [79:41] So in a way, I hope this is starting to make a little bit [79:45] more concrete. [79:45] What I mean when I say that these techniques, which [79:48] we introduced in this class, are definitely not [79:50] exclusive, but actually typically [79:52] combine in autonomy stack. [79:56] And the details of these algorithms [79:58] will obviously depend on the application. [80:00] So we hope that with this class, we gave you, [80:03] first of all, a broad understanding of all [80:06] of these concepts, and also focused on the higher level [80:13] understanding of how these modules could be combined, [80:16] and the pros and cons of these ideas. [80:20] All of this is obviously in-- [80:24] we should, I think, concretize this in a moment in time [80:28] where learning-based and end-to-end approaches [80:30] are definitely becoming more and more popular. [80:33] And so I would say that historically, learning [80:36] has started from the higher levels of this abstraction. [80:40] So perception is the first situation [80:44] where learning basically became the de facto standard. [80:48] And I would say that right now, end-to-end approaches [80:52] are more or less starting to carve their way down [80:57] along this hierarchy. [80:58] I would say right now they are basically somewhere over here. [81:01] So in between the open-loop and the closed-loop trajectory [81:04] tracking stage. [81:06] I would say even the most bullish companies or approaches [81:10] that claim end-to-end approaches, [81:13] typically they don't mess with the lower level components. [81:17] And this is because here it's really [81:19] where you want to make sure you have [81:21] fine-grained control, your encoding hard constraint [81:24] guarantees, and making sure that everything works concretely [81:28] in practice. [81:32] So if there are no further questions, [81:34] I want to really thank you very much for sticking [81:36] with us throughout this quarter, and wishing you best [81:40] of luck for the exam. [81:41] [APPLAUSE]