Physical Intelligence’s π0.7 Beats Specialist Robots Without Tuning

Physical Intelligence通用模型开箱即用，表现超越专用机器人 基于多尺度上下文提示调节的统一基础模型现已超越针对特定任务定制的策略，无需再为单个现实任务微调机器人。 Chelsea Finn概述了推动机器人技术从脆弱的实验室演示走向稳健现实应用所需的技术转变，并强调物理系统需要近乎完美的自主可靠性，因为其实际错误会直接改变物理环境。为克服物理试错带来的硬件瓶颈，Physical Intelligence将针对性的人工遥操作干预与摊销价值函数，以及将压缩视频潜变量与长期文本摘要相结合的多尺度记忆架构融为一体。其通用基础模型π₀.₇开箱即可使用，能够在复杂的日常家务任务中实现持续数小时的不间断运行，达到甚至超越了特定任务的专用模型。最终，Finn表明，将多样化的多任务数据集与结构化元数据提示条件相结合，可以释放真正的组合泛化能力和零样本跨本体技能迁移能力。 Physical Intelligence推出了统一机器人基础模型π₀.₇，该模型开箱即可使用，无需针对每个任务进行强化学习或监督微调，即可媲美甚至超越经过特定任务微调的专用模型。 (30:20) 机器人强化学习无法复制LLM重算力推演的方法（如PPO或GRPO），因为为一项耗时1分钟的任务生成1,000,000次物理推演将造成相当于700个机器人日的持续硬件磨损。 (8:05) 为避免高频控制中的token上下文爆炸（4个摄像头在50 Hz下运行10秒就会产生500,000个token），研究人员开发了一种多尺度记忆，将压缩的10秒潜变量视频与10至15分钟的文本空间摘要相结合。 (17:36) 在结构化元数据提示的调节下，基础模型能够对带噪声且质量参差不齐的演示数据进行规模化训练而不出现性能下降，并通过强化学习后训练实现了2倍的吞吐量提升。 (27:01)

Chelsea Finn argues that general-purpose robotics is transitioning from task-specific fine-tuning into an out-of-the-box foundational era, though achieving true physical autonomy requires overcoming hardware-bound sample inefficiency, context memory limits, and near-zero error tolerances that digital software never faced. Zero-Error Demands and the Hardware Bottleneck (1:23) Unlike digital machine learning deployments where human users absorb flawed outputs in ad ranking or code generation, physical robots execute force directly on their surroundings. To provide tangible value rather than an operational nuisance, physical AI must hit error rates orders of magnitude lower than software models. While autonomous driving demonstrates this reliability is solvable—with Waymo crossing 250,000 weekly autonomous rides—applying standard reinforcement learning (RL) directly to physical hardware hits a severe wall-clock barrier. While language models scale by generating millions of trajectory rollouts across data-center GPUs, robotics binds real motors, gears, and workspaces. Finn calculates that running a modest 1,000,000 one-minute trial rollouts on physical hardware amounts to 700 continuous robot days of execution and wear. Because brute-force exploration is economically and operationally prohibitive, robotic RL cannot blindly copy LLM algorithms and must achieve drastic sample efficiency. Sample-Efficient RL via Interventions and Value Functions (9:54) To prevent hardware time from being wasted on dead-end trajectories—such as a gripper seizing two flush cardboard boxes at once—Physical Intelligence introduces human teleoperation interventions to demonstrate recovery maneuvers before returning autonomous control. This mechanism ensures physical rollouts produce high-signal interaction data rather than irrecoverable failures. To eliminate the need for running dozens of trial rollouts per prompt (common in LLM post-training algorithms like PPO and GRPO), the team amortizes credit assignment using a general-purpose visual value function trained on robot video logs. This model inspects autonomous video frames to predict progress and time-to-success across disparate tasks without task-specific retraining. In practice, this closed loop yielded an approximate 2x boost in task throughput compared to pre-training and supervised fine-tuning, sustained a 13-hour continuous run making lattes with over 90% success, and automated production box folding at the Dandelion Chocolate factory. Hierarchical Memory for Extended Non-Repetitive Workflows (17:36) Robotic foundation models have historically been reactive and memoryless because processing high-frequency video context is computationally prohibitive. Streaming 10 seconds of 4-camera video at a standard 50 Hz control rate generates roughly 500,000 tokens, and even 1 Hz subsampling yields 10,000 tokens for just 10 seconds. Reactive models suffice for motor primitives, but multi-step sequencing depends on tracking completed sub-goals. Physical Intelligence addressed this bottleneck by introducing a multi-timescale memory hierarchy: an efficiently computed short-term visual memory covering the previous 10 seconds paired with a compressed text summary logging the preceding 10 to 15 minutes of events. This dual context allows robots to autonomously execute non-repetitive, 10- to 15-minute chore checklists—such as wiping counters, drying them with a paper towel, stowing mustard in a refrigerator, putting away clean dishes, and washing dirty dishes—without resetting state. Compositional Generalization and Out-of-the-Box Generalists (29:08) Paralleling AI's historical progression from ImageNet and BERT to GPT, Finn argues that robotics must transition from per-task fine-tuning to unified out-of-the-box generalists capable of compositional generalization. Training their π0.7 model across heterogeneous data—mixing teleoperation, RL failures, human videos, and web data—combined with metadata tags and high-level subtask planners produced a generalist that matched or outperformed downstream specialist models that had received extensive task-specific RL or supervised fine-tuning. Furthermore, the model demonstrated zero-shot compositional transfer across distinct hardware morphologies. An industrial dual-arm robot with unfamiliar linkages, arm lengths, and joint kinematics successfully folded clothing on its first attempt, despite all prior garment demonstrations having been collected exclusively on an entirely different platform. Hardware Constraints, Control Paradigms, and Emergence (39:44) Finn stresses that physical AI will not replicate the viral distribution curve of software LLMs (which reached one million users within five days) because hardware deployment requires localized testing, manufacturing, and physical safety rollouts. At the control layer, Physical Intelligence predicts target joint angles and 3D end-effector poses executed by classical PD controllers; Finn notes that while raw torque control enables variable compliance, position control does not currently act as an operational bottleneck. Beyond intentional architecture design, Finn highlights emergent capabilities discovered during deployment. During a bimanual pinwheel assembly task where 100% of training demonstrations used the right hand to insert a pin into paper, an accidental perturbation knocked the parts across the table. The robot autonomously executed the insertion with its left hand, demonstrating unprompted bilateral hand transfer. Finn concludes that while foundation policies are operating in production environments at startups like Ultra and Weave, physical robots remain slower than human workers and face an embodied data bottleneck that passive internet video cannot resolve. While autonomous self-play and foundation pre-training offer clear headroom, hardware distribution and real-world safety validation mean physical AI adoption will remain a deliberate, localized process.

[0:06] Chelsea Finn: any robot to do any task in the real world [0:06] Chelsea Finn: cool videos of robots doing lots of different things [5:11] Chelsea Finn: unfortunately, this rarely works reliably on the very first try. [24:47] Chelsea Finn: you actually aren't getting a general purpose model. [31:16] Chelsea Finn: The first time we saw the robot do this we were floored, because there was no training data for this task [on this robot]... The robot here is quite different from the other robot not just in size but also in the lengths of the linkages of the robot, in the configuration of the joints of the robot, and so forth. [39:44] Chelsea Finn: very much on the horizon in the next few years [42:24] Chelsea Finn: no one knows what the best route is to make these models more and more powerful. [45:25] Chelsea Finn: train must match test [45:25] Chelsea Finn: If I watch Roger Federer play tennis, it doesn't mean I can play tennis as well as him, unfortunately. [57:35] Chelsea Finn: Hey, I'm interested in working in your lab. [57:35] Chelsea Finn: getting your feet wet, trying stuff out, and learning from that experience.

[0:06] Everyone, [0:08] today I'm going to be talking about the [0:10] state-of-the-art of physical [0:12] intelligence. [0:14] And in particular, two years ago, I [0:17] founded a company called physical [0:19] intelligence. And uh we're really [0:22] interested in how we can basically uh [0:26] develop any robot allow any robot to do [0:28] any task in the real world. Uh and I was [0:32] actually spoke at this event a year ago [0:35] uh last year and at the event last year [0:37] I shared some of our progress in uh at [0:40] the company at physical intelligence [0:42] where we could do things really [0:45] complicated tasks like uh folding [0:47] unloading and folding laundry. Uh, and [0:49] we I also talked about how for the first [0:51] time we showed how robots can do useful [0:55] tasks in environments in rooms they've [0:58] never been in before. [1:00] Now, since then, since one year ago, we [1:03] have gotten robots to do a lot of other [1:06] really cool things. So, for example, [1:08] we've gotten robots to be able to wash a [1:11] greasy pan in the top right or peel a [1:14] carrot uh in the video below that or [1:17] make a grilled cheese sandwich uh in the [1:19] video below that or slice a zucchini uh [1:21] and so forth. But what I'd really like [1:24] to focus on today isn't cool videos of [1:28] robots doing lots of different things, [1:31] but how what it actually takes to get [1:33] robots to be useful in the real world. [1:36] And specifically how can we develop [1:38] generalpurpose robots that are useful in [1:40] the real world. Now there are two [1:43] aspects of this. The first is general [1:46] purpose. How we can develop general [1:47] purpose models and the second is [1:49] actually bringing those models to the [1:50] real world so that they can actually [1:53] have an impact and be useful to people. [1:56] And in the first part I'll talk about [1:58] being useful in the real world. [2:00] So to actually bring a technology to the [2:02] real world, I think we need to figure [2:04] out it's helpful to actually look at [2:06] what people have done in the past uh to [2:08] bring AI into the real world. And if we [2:10] look at a timeline of major production [2:12] launches that are leveraging technology [2:14] like machine learning, we um can see a [2:17] timeline like this. So I think the the [2:19] really the first early examples of [2:21] machine learning being used for real in [2:23] the real world were for things like [2:24] product recommendations and ad ranking. [2:28] uh and then five years later we started [2:30] to see uh not just machine learning uh [2:33] being used but deep learning being used [2:35] for the same sorts of applications. Uh [2:38] this was a really exciting advance [2:40] because deep learning is an algorithm [2:42] that actually isn't like you can kind of [2:44] really apply it out of the box um to [2:46] scenarios that involve really complex uh [2:49] like inputs and outputs and so forth and [2:51] it makes it easier to translate to other [2:53] applications. [2:54] But from there, I think that even more [2:56] exciting um kind of uh moment in time [2:59] that we saw in terms of machine learning [3:00] and AI in production was in 2022 uh with [3:03] the launch of Chhat GBT. Uh and this was [3:06] the first time where we saw a general [3:08] purpose model truly being used by many [3:11] different people in the real world. Uh [3:13] within five days, Chhat GBT had reached [3:15] a million users. And then of course more [3:18] recently we've seen things uh like cloud [3:20] code uh also be incredibly useful uh [3:22] hopefully to many of us uh in the real [3:24] world and other coding agents. Now if we [3:27] look at how AI has been used in the real [3:30] world uh and and kind of look at this I [3:32] think there's a few different takeaways [3:33] we could make. Uh the first is that [3:36] generalist models are increasingly being [3:38] used for real world problems. And so [3:40] we're actually seeing uh general purpose [3:42] like generalist AI models that can do [3:44] many many different things actually be [3:46] used in the real world and we see that [3:47] transition from the left to the right. [3:50] But I also think that there's a more [3:51] nuanced observation that we can make uh [3:54] from looking at these applications [3:57] and in particular if we look at all of [3:58] these different applications that are [4:00] used uh where machine learning has like [4:02] actually been useful in the real world [4:04] and actually uh been profitable and so [4:06] forth. And in all of these applications, [4:10] the customer is making a decision based [4:14] off of the recommendation of the AI [4:16] model more or less. Uh, and this means [4:19] that if the customer is ultimately like [4:21] kind of making the decision, this means [4:23] that if the system makes a mistake, um, [4:25] that's okay because usually the person [4:27] can kind of recognize that or or decide [4:29] what to do even despite that mistake. [4:31] Uh, and so even when these sorts of [4:33] systems aren't perfect, they're still [4:35] incredibly useful to different people. [4:37] and there's less pressure on them to be [4:39] completely perfect. [4:41] And I think that actually physical AI [4:43] and robotics is pretty different from [4:45] this where if we think about physical AI [4:47] that are actually operating in the [4:49] physical world, they have to be directly [4:51] making decisions that affect the [4:52] physical world. Uh and this means that [4:55] they're going to be far more useful when [4:56] they're operating fully autonomously. [4:59] And as a result, this requires us to [5:02] develop physical AI systems that make [5:04] far fewer mistakes than the machine [5:06] learning systems that have been deployed [5:08] thus far. Now, one really exciting thing [5:11] to kind of highlight um that has [5:13] happened recently is uh a year ago Whimo [5:16] passed the uh quarter of a million [5:18] weekly autonomous rides suggesting that [5:20] it is really possible to develop a [5:22] machine learning based system that can [5:24] uh operate in a trustworthy and [5:26] autonomous way uh directly in the [5:28] physical world. Um and I think that [5:29] brings a lot of hope and optimism for [5:31] actually um doing the same with the rest [5:34] of AI in the physical world. [5:37] So um if we want to develop general [5:39] purpose robots in the real world, I [5:41] think we need to think about how we're [5:43] going to make them autonomous for long [5:45] periods of time uh so that they're [5:47] actually useful uh rather than having [5:49] them uh rather than something where a [5:51] human is basing decisions on the [5:53] predictions of the model. [5:55] So um to think about long-term autonomy, [5:57] I want to ground this in a specific [5:58] example and say that we wanted a robot [6:00] to make espresso. If we wanted it to be [6:03] actually be useful for us, we needed to [6:04] make espresso reliably uh so that we [6:07] don't have to babysit the robot uh very [6:09] frequently uh in order uh for it to help [6:11] serve drinks. Now even on its own this [6:14] task is really difficult. So actually [6:16] operating the porter filter requires [6:18] very precise and forceful control to [6:20] insert it um appropriately. [6:23] It also needs to smoothly handle cups [6:26] with liquid in it and not spill those [6:28] cups. And it also needs to have an [6:30] accurate sense of timing, which often [6:32] isn't actually an issue in other areas [6:34] of machine learning. [6:36] And not only do we want to do this [6:37] pretty challenging task, we want to do [6:38] it with over 90% reliability. [6:42] So how can we do this? So the first step [6:45] in machine learning is always to collect [6:48] some data set, train a model, and [6:50] evaluate how good your model is. [6:53] And unfortunately, this rarely works [6:55] reliably on the very first try. [6:59] Now, in practice, then it's a bit better [7:00] to then iterate on the model that you've [7:02] developed where you will try to um [7:06] collect more data or improve the quality [7:09] of the labels in your data set or make [7:10] the labels more detailed, collect more [7:12] data of the edge cases, the scenarios [7:14] where it's not working well, adjust the [7:16] balancing of the data set and so forth. [7:18] Um, and while this generally improves [7:20] the reliability of the model, uh, people [7:22] eventually get tired and it's hard to [7:24] get really, really high reliability with [7:26] a person that's manually tuning this. [7:28] And so what would be even better is if [7:30] the AI system itself can iterate on the [7:33] scenario in which you want it to have [7:35] higher reliability where it on its own [7:37] automatically seeks out places where it [7:39] needs more data, where it needs more [7:40] supervision. uh and if we can do this [7:43] for many many more iterations because [7:45] it's automatic rather than a person [7:46] doing it then this might be the way to [7:48] get really really high like 99 plus [7:51] percent reliability from physical AI [7:54] systems and so this is the approach that [7:56] we'll take and this looks a lot like a [7:58] reinforcement learning algorithm that's [8:00] trying to attempt the task learn from [8:01] its failures and get better and better [8:03] on its own [8:05] so then how do we develop a scalable [8:08] reinforcement learning recipe for [8:09] robotics [8:11] well In language models, we have [8:12] algorithms like PO and GRPO and these [8:15] have scaled to large language models and [8:17] have enabled really complex reasoning. [8:19] Um, but there's a bit of a challenge in [8:21] applying this to robotics, which is that [8:24] these these algorithms have been trained [8:26] with uh millions of attempts or [8:28] sometimes even tens of millions of [8:29] attempts uh by scaling up the compute [8:32] because each attempt is simply running [8:34] the language model uh in a data center [8:36] um just by using compute. [8:40] And if we were to translate this very [8:42] very approximately to robotics um say we [8:44] had maybe not millions or tens of [8:46] millions but just one million [8:47] trajectories of a one minute robot task. [8:49] This is even shorter than the espresso [8:51] task that I talked about. This would [8:53] correspond to 700 robot days um to get [8:56] high reliability for that task. Now [8:59] maybe this isn't completely out of the [9:00] question but this would be quite [9:02] challenging uh to do and that's because [9:06] the calculus is a little bit different. [9:07] We're not just running compute to [9:09] optimize for a use case. We're actually [9:12] running the robot in the real world and [9:14] using the hardware and attempting the [9:15] task in the real world. [9:18] And so we'd like to have an algorithm [9:20] that can iterate much more efficiently. [9:22] And there are actually ways that we can [9:24] make these algorithms a lot more [9:25] efficient. So um there's a couple [9:28] inefficiencies like large inefficiencies [9:30] in these reinforcement learning [9:31] algorithms for language models. [9:34] The first is that they spend a lot of [9:36] time on deadend trajectories. And maybe [9:38] this is okay if you're just spending [9:39] compute on it. But this would cost a lot [9:42] in the physical world. So we can look at [9:44] a concrete example. Say that we want a [9:46] robot to construct cardboard boxes and [9:48] stack them uh on the right. And in this [9:50] trajectory right here, the robot [9:52] accidentally grabbed two boxes that are [9:54] flush against each other. And if we let [9:56] it continue, it would just continue to [9:58] try to fold that box rather than [10:00] separate out the two boxes. and trying [10:02] to fold two boxes together isn't useful [10:04] data that will teach the model how to [10:06] get better at the task. And so that [10:08] would be kind of wasting a lot of time [10:10] on the robot attempting to go down the [10:12] wrong path for solving the problem. And [10:15] so instead of spending a lot of time [10:17] trying to do that task, what we'll do is [10:19] we'll actually have a human intervene [10:21] and show the robot what to do and how to [10:23] recover from that situation. And so what [10:26] you can see here is a human is [10:27] teleoperating and intervening uh with [10:29] the robot and showing it um that to [10:31] recover from this situation, it needs to [10:34] essentially try to separate out the two [10:36] boxes. Um it then kind of puts its [10:38] gripper in, sees it if the then if the [10:40] robot could autonomously recover, it [10:41] doesn't autonomously recover. And so the [10:43] person then intervenes again um to help [10:46] uh help it get back on the right track [10:48] so that we're efficiently using the data [10:50] on the robot. [10:54] Okay, so this is the first thing that we [10:55] can do which is we can show the robot [10:57] how to recover early or how to recover [11:00] so that we're not spending time on dead [11:01] end trajectories or at the very least [11:02] just terminate the episode early. [11:05] Now the second thing that we can do is [11:09] uh PBO and GRPO and these kinds of [11:11] algorithms they make many attempts at a [11:14] single prompt. Uh and depending on the [11:16] algorithm they're they're essentially [11:18] trying to estimate for these different [11:19] responses what is a good response and [11:22] what's a bad response. Uh and so even [11:24] for an individual prompt they're [11:25] actually going to kind of roll out like [11:27] 10 or 50 times uh for that individual [11:30] prompt. And they're doing this because [11:32] they're trying to estimate the value of [11:33] these different um attempts uh to then [11:36] kind of upweight or kind of increase the [11:38] likelihood of good things and decrease [11:39] the likelihood of bad things. But we can [11:42] actually amortize this cost uh rather [11:44] than trying to collect a lot of attempts [11:45] for a single prompt. We can amortize [11:47] this across different prompts and learn [11:48] a much more general value estimate of [11:50] what's good and what's bad and use this [11:52] to improve with our autonomous [11:54] experience. [11:56] And so what this looks like is um we can [11:58] train a general purpose value function [12:00] on lots of uh kind of videos of the [12:03] robot experience. And this can learn [12:05] things like if it accidentally unfolds a [12:07] shirt when it's trying to fold that [12:09] that's bad and that's making negative [12:11] progress. It's shown in red. Or if it's [12:13] making kind of forward progress, it [12:15] recognizes that as well. Um and the same [12:18] value function can also estimate what's [12:20] good and bad for a completely different [12:21] scenario. uh in this case for retrieving [12:24] an item from a fridge. And so this sort [12:27] of kind of general purpose value model [12:29] that's kind of predicting basically the [12:31] time to success can um basically [12:35] significantly reduce the amount of [12:36] attempts needed to learn how to improve [12:39] from experience. [12:41] Um so with these two um kind of [12:44] improvements to a reinforcement learning [12:45] system, we kind of have a a general [12:48] improvement algorithm that trains a [12:49] foundation model on diverse data, then [12:52] collects experience from that with a [12:54] human intervening as necessary to help [12:56] prevent dead end trajectories and then [12:59] train a general purpose estimate of [13:00] what's good and bad the value function [13:02] and then use that to improve the model. [13:06] And with this sort of improvement um [13:08] we're able to uh kind of fine-tune a [13:10] foundation model to higher degrees of [13:12] performance. [13:14] So um in the task of making a latte in [13:17] this case we'll be making a latte in [13:18] collaboration with a person where the [13:20] robot is in charge of making the [13:21] espresso uh and the person is in charge [13:24] of steaming the milk. Um this is what [13:26] the uh what the kind of model looks [13:29] like. The model is directly controlling [13:30] the joints of the robot using uh the [13:33] images from the robot's cameras as [13:35] input. [13:37] And we can see that the model um is able [13:39] to do the pretty challenging task of [13:41] inserting the PA filter um uh waiting [13:44] the appropriate amount of time for the [13:46] espresso to dispense, pouring the [13:48] steamed milk into the cup. [13:51] And then the last part of this task is [13:53] actually the most challenging uh where [13:54] it needs to take a very full latte uh [13:58] cup of latte and transfer that over to [14:02] the coaster. [14:04] And so here's actually the observation [14:05] that the robot sees directly. [14:09] And you can see that the policy is super [14:11] delicate and able to balance the cup [14:13] appropriately and smoothly so that the [14:16] latte doesn't spill. [14:20] So this is kind of gives you a sense of [14:21] the difficulty of this kind of task. Uh [14:24] and [14:26] kind of going back to this reliability [14:27] question, we took this policy and we ran [14:30] it not just once, but we ran it for 13 [14:33] hours straight. Uh and we basically [14:36] wanted to evaluate is this policy not [14:38] only good at making a latte once, but [14:40] can it do so reliably to the extent that [14:42] it would be needed to be useful in the [14:43] real world? Um and so here's a time [14:46] lapse um of that process. Uh and indeed [14:49] we found that the robot was reliable [14:51] enough to be useful for long stretches [14:54] of time without making mistakes uh [14:56] frequently. [14:58] Now the same algorithm isn't specific [15:00] for making lattes of course and so um we [15:03] also applied this to other applications [15:04] as well. Um, Dandelion Chocolate Factory [15:06] is a few blocks from our office. And so [15:08] we took a workflow that they typically [15:11] have a person do, which is to construct [15:12] these cardboard boxes, label them, and [15:14] stack them. And we trained our robot to [15:17] basically do exactly their real [15:18] workflow, uh, and trained it with the [15:21] reinforcement learning algorithm that I [15:22] talked about to get a policy that is far [15:25] more reliable at constructing, labeling, [15:28] and stacking these boxes. [15:31] And then we also applied this algorithm [15:33] to fold and close as well. And we wanted [15:36] in this case to not just test how well [15:37] the model could do one task in one [15:39] environment, but to do it in many [15:40] environments. And so these are clothing [15:42] items that the robot has never seen [15:43] before in a home that's never seen [15:45] before. Um, and it's able to do so and [15:48] act autonomously for an extended period [15:50] of time. [15:53] Now, um, videos aren't uh don't always [15:56] kind of show everything. And so we also [15:57] quantitatively measured the reliability [15:59] of these models. And we care both about [16:01] the reliability as well as the speed [16:03] like how how many boxes can it build per [16:06] hour. And so we're going to measure [16:07] throughput which kind of couples both uh [16:10] success rate and speed. And we find that [16:13] over the phases of training from [16:14] pre-training to like an SFT- like stage [16:16] to an RL post training like stage we see [16:18] a drastic increase in um in success rate [16:22] indeed in throughput uh and specifically [16:25] um around a 2x throughput just from the [16:27] RL stage itself showing how we can get [16:30] much greater reliability from [16:31] reinforcement learning. And for the [16:33] espresso task, if you look specifically [16:34] at the success rate, we achieved a 90% [16:37] or over 90% success rate um on making [16:39] espresso. [16:42] So the takeaways for this part is that [16:43] we we can develop a scalable recipe for [16:45] high reliability of complex robotic [16:48] manipulation tasks. Uh and we saw in [16:50] this case a 2x higher throughput from [16:52] using experience and interventions. But [16:54] most importantly, we saw how we can [16:56] achieve long-term autonomy in real [16:59] workflows that people um that people [17:02] actually care about in the real world. [17:04] And this is what it's going to take, I [17:05] think, for robots to be useful in the [17:07] real world. Now, there's also a lot more [17:10] work and a lot more opportunities. Uh [17:13] the we actually only ran a few [17:14] iterations of improvement of this [17:16] algorithm and with more iterations, we [17:18] should be able to see even greater [17:19] improvement or even greater reliability. [17:22] Uh, and even with this improvement, the [17:23] robot still makes mistakes. It's also [17:26] still slower than people. And so, uh, [17:27] there's many like a ton of room for [17:29] improvement for developing even more [17:31] powerful recipes. So, we've seen [17:33] long-term autonomy, uh, for these [17:35] different workflows. But there's [17:37] actually one more ingredient that I'd [17:39] like to talk about for enabling robots [17:41] to be autonomous and useful for long [17:44] periods of time. And that ingredient is [17:47] memory. So, you might be surprised to [17:50] hear that most state-of-the-art [17:52] foundation models for robotics have no [17:54] memory or no context. They're just [17:57] operating on the current sensor [18:00] observations, the current camera [18:01] readings, uh, and predicting actions [18:04] based off of that. And you actually can, [18:08] um, do short motor skills, you can do [18:11] repetitive tasks without memory. The [18:13] videos that I showed before didn't have [18:14] any context either. But if you want to [18:17] do a long task that involves multiple [18:19] different steps in sequence, then memory [18:21] is critical for tracking progress of the [18:24] steps that you've completed so far. [18:27] So if it's critical for doing these [18:29] kinds of long horizon tasks, then why [18:30] don't these models have any context or [18:32] memory? Uh there's a couple reasons for [18:34] this that are technical and and I'll [18:36] talk through one of them um which is [18:37] that if you naively approach memory uh [18:40] and try to feed in context uh like pass [18:43] video to a robot foundation model. Um [18:47] say that you would just pass in 10 [18:48] seconds of video. Uh maybe this 10 [18:50] seconds of video is sampled at 50 hertz [18:52] which is a a common control frequency in [18:53] robotics and you feed in all four camera [18:56] streams on the robot and you use around [18:58] kind of 256 tokens per image. Uh this [19:00] corresponds to passing in half a million [19:02] tokens into your model which is a lot of [19:05] tokens. Uh and trying to do that in real [19:07] time um into your model right now uh is [19:10] quite challenging. Even if you subsample [19:12] to one frame per second, you're still [19:15] going to be passing in 10,000 tokens [19:16] into your model which at least right now [19:18] is prohibitively expensive for these [19:20] models and that's still only 10 seconds [19:22] of memory. [19:24] So um I don't have time to go into the [19:26] technical details of uh exactly what we [19:29] did here but um we also developed a [19:31] solution for this context problem and [19:34] specifically uh we developed a system [19:36] that has memory at multiple time scales. [19:38] The first is a short-term video memory [19:40] that has about 10 seconds of video [19:42] memory um but is done so and computed [19:45] much more efficiently than naively [19:46] passing it into the model. And then for [19:49] longer memory, for memory that spans [19:51] multiple minutes or multiple hours, we [19:53] don't necessarily need video of exactly [19:56] what happened in that past uh in that [19:58] past history. And so instead, we [20:00] represent memory for those [snorts] [20:02] parts in text where we summarize what [20:04] happened uh in text space and then [20:06] incorporate that much more compressed [20:08] textual summary of what happened over [20:10] the past 1015 minutes uh into the model [20:14] uh as well. And with this sort of kind [20:17] of memory at multiple different time [20:19] scales, we're able to enable robots to [20:24] do tasks that can operate for 10 or 15 [20:26] minutes at a time completely [20:28] autonomously. And what's different from [20:30] the previous slide or the kind of what I [20:32] showed previously is that this task [20:33] isn't repetitive. Um, so this is going [20:35] to be a 10 to 15 minute task that's [20:37] involved cleaning a kitchen. And the [20:39] robot isn't just repeatedly making [20:40] espresso over and over again. Um, what [20:43] it involves is wiping the counter with a [20:45] sponge. Uh, kind of then drying the [20:48] counter with a paper towel, throwing [20:49] away the paper towel. Next, it's going [20:51] to put away the mustard uh into the [20:53] fridge. Then, it will put the dishes [20:55] away uh into the uh into the cabinet, [20:58] wash some of the dirty dishes in the [20:59] sink, and so on. [21:01] And so, um, by incorporating memory, [21:04] it's able to do a task that requires [21:06] keeping track of all of these different [21:08] steps, um, that is done to, uh, clean [21:10] the kitchen and successfully operate [21:12] for, uh, 10 to 15 minutes, uh, [21:15] completely autonomously. [21:18] Great. Um, so those were a couple [21:20] ingredients for long-term autonomy. Now, [21:22] I'd like to build on that and actually [21:24] take those ingredients and put it into a [21:26] general purpose model uh that can do [21:29] everything that I showed before, but [21:31] also can do that in a single model um [21:33] and can do some other things as well. [21:37] And to think about developing such a [21:39] general purpose model, I think it's [21:40] really helpful to contextualize [21:43] like where robotics is at uh within the [21:46] timeline of other developments in [21:49] generalist AI. [21:51] And if we think about how kind of [21:53] generalist AI systems have evolved over [21:56] the past like 15 years, I think the [21:59] first major milestone was in 2012 when [22:01] we saw that a deep learning system [22:03] trained from scratch. This was the first [22:05] time that it topped an external [22:07] benchmark and all of the previous [22:09] methods for that benchmark were [22:11] specifically designed for that [22:13] application. So all of the previous [22:14] methods um specifically this was the [22:16] imageet benchmark were designed [22:18] specifically for image classification. [22:20] um whereas this was the first time that [22:22] a deep learning based system actually [22:24] outperformed those more specialist [22:25] systems. And this is a kind of a much [22:27] more general algorithm um that wasn't [22:29] specifically designed for image [22:30] recognition. [22:32] And then just a couple years later, we [22:34] found that we weren't just training [22:36] algorithms from scratch, but we were [22:38] able to get models like pre-trained [22:40] models that are useful for fine-tuning [22:43] to downstream tasks. And so became the [22:45] norm to take a model that was [22:46] pre-trained on imageet and then [22:48] fine-tune it on a downstream task. we [22:49] actually saw better performance from [22:51] using that pre-trained model like BERT [22:53] or like an imageet pre-trained model. [22:56] And then from there the the I think the [22:58] next big phase and the next big [23:00] transition in generalist AI models [23:02] wasn't um using pre-trained models but [23:04] kind of moving from a pre-training [23:05] fine-tuning regime to a regime where [23:08] we're just using generalist models out [23:10] of the box. Um this was like with models [23:12] like the start of like GPT2 and of [23:14] course all like almost all the models [23:15] that we interact with today worked just [23:17] out of the box without um without [23:19] fine-tuning or at least most of the [23:22] consumer models. There's actually other [23:24] other models that that still use a lot [23:25] of fine-tuning. [23:27] And then I think one other um kind of [23:30] milestone that I want to highlight was a [23:31] milestone in 2021 where we think I saw [23:34] the first signs of compositional [23:36] generalization in these models. um and [23:38] kind of one specific instance of that [23:40] was with Dolly and I'll talk a little [23:41] bit more about that uh in a later slide. [23:44] So this is how kind of generalist AI has [23:46] advanced um over the past 15 years. Uh [23:49] and then meanwhile if we kind of think [23:51] about physical AI [23:53] even just three years ago uh in 2023 it [23:57] was extremely common for people working [23:59] on robotics to collect a bespoke data [24:02] set from scratch or an individual [24:04] project and train from scratch on that [24:06] data set. And this is analogous to say [24:10] collecting imageet from scratch and [24:13] training on imageet uh or training on [24:15] the data set that you just collected [24:16] from scratch on that data set. And so if [24:19] you want to develop a general purpose [24:21] model, if you have to collect the the [24:23] data set from scratch for every single [24:25] project, you're probably not going to [24:26] make a lot of progress. [24:28] Um and so um until just a few years ago, [24:31] I think we were kind of pretty far on [24:32] the left of um of this timeline. Uh and [24:36] until recently, I think that more [24:38] recently, I think that we've been kind [24:39] of in the 2014 phase where we have some [24:40] good pre-trained models. Um but we [24:42] haven't really been truly in the regime [24:44] on the right. [24:47] So how do we get to that regime on the [24:48] right? Specifically, how do we develop a [24:50] single general purpose model that works [24:52] out of the box and also shows [24:54] compositional generalization? [24:58] Um, so this has two goals. The first is [24:59] out of the box models. This is analogous [25:01] to going from like BERT to GPT. [25:04] And right now the best robot [25:07] performance, if you want to get like the [25:09] your model to perform um the best that [25:11] it can on a given task, it always [25:12] requires fine-tuning. So some of the [25:14] videos that I showed at the beginning, [25:16] they were fine-tuned models um to do [25:18] things like unlocking a lock. Um other [25:20] work that we've done on measuring human [25:21] to robot transfer also needed [25:23] fine-tuning to get the best performance. [25:25] Uh and of course all of the videos that [25:27] I showed with RL post training, we're [25:29] also fine-tuning on an individual task [25:31] uh to get the best performance on [25:33] something like making espresso. [25:35] But if you have to fine-tune a model, [25:37] you actually aren't getting a general [25:39] purpose model um for the things that you [25:41] want it to do because you have to [25:43] fine-tune it for each individual thing. [25:45] And so our first goal is to move towards [25:47] a single general purpose model that can [25:49] actually do all of the things that you [25:50] want it to do. Um then the second goal [25:53] that I mentioned is compositional [25:54] generalization. Uh and this is inspired [25:56] by uh the Dolly result from 2021. And I [26:01] think that this was a really important [26:03] and exciting milestone because of the [26:06] compositional generalization that it [26:07] achieved. Uh and specifically when you [26:10] have compositional generalization, when [26:12] you can basically bridge the concept of [26:14] an avocado and a chair and show that you [26:16] kind of can combine those two, it means [26:18] that the model has at least some kind of [26:19] conceptual understanding of what an [26:21] avocado is and what a chair is to the [26:23] point that it can combine them into [26:25] something that exhibits both concepts at [26:27] the same time. [26:29] And then second, it means that you have [26:31] some degree of data efficiency where [26:33] your data doesn't need to cover all of [26:34] the possible combinations of concepts [26:37] represented in your data. You don't need [26:39] pictures of avocado chairs in your data [26:40] set in order to generate something like [26:42] this. Um or you don't need combinations [26:45] of other things that you might ask the [26:46] model to do at uh when it's deployed. [26:50] Now, even back in 2021, it wasn't [26:52] perfect, but these signs of [26:53] compositional generalization were really [26:55] exciting for uh demonstrating these two [26:58] attributes of the model. [27:01] So, we have these two goals um that we'd [27:03] like to do. We like an out-of-the-box [27:04] model and compositional generalization. [27:07] Now the tried and tested recipe for [27:09] developing this kind of model is to [27:13] first take a sufficiently large and [27:14] diverse data set and to second um train [27:18] a model with sufficient capacity. [27:20] And so what we're going to do is we're [27:22] going to do that. We're going to try to [27:23] use all of the data that we have [27:25] available. This includes really diverse [27:26] robot demonstration data including [27:28] really lowquality demonstration data. [27:31] Uh, it's also going to include policy [27:34] rollout data. Basically, attempts from [27:36] the robot of doing the task. Basically, [27:38] all of the training data that was used [27:40] for reinforcement learning for the [27:41] previous tasks will be included in the [27:43] training recipe. We're also going to [27:45] include videos of humans. Uh, and we're [27:47] also going to include data from the web. [27:49] Basically, all of the data that we have. [27:52] And then to train a model with [27:54] sufficient capacity, of course we'll [27:56] train a model that's large enough, but [27:58] to fit data that's so heterogeneous, we [28:01] also find it um particularly important [28:05] to prompt the model with all of the [28:07] context that it needs in order to [28:09] predict actions. And we found that this [28:12] idea was really the key unlock to using [28:14] this kind of data and is data of this [28:16] kind of degree of heterogeneity. [28:20] And specifically what this looks like is [28:22] we're going to train uh a foundation [28:24] model that takes as input uh the memory [28:27] like that that I mentioned before an [28:29] instruction of what to do, but it's also [28:31] going to take as input a subtask [28:33] instruction um of kind of what the next [28:35] immediate thing it should do is. It'll [28:37] also take as input metadata that kind of [28:38] indicates the quality of the data, the [28:40] length of the episodes and so forth. And [28:42] this metadata gives it a lot more [28:43] information about how it should predict [28:46] the next action. [28:47] And then optionally we'll also uh train [28:50] the model with a sub goal image as a [28:52] prompt to the model that's essentially [28:54] saying like a few seconds from now you [28:57] should try to reach something that looks [28:58] like this image. [29:01] Um with this detailed prompting we find [29:02] that the model can really make use of [29:04] much more heterogeneous data and I'll [29:06] show some compar comparisons later that [29:08] really um kind of show how important it [29:10] is. And now to actually deploy this [29:12] model we need to then provide things [29:14] like this subtask construction and sub [29:15] goal images. And so with that we can [29:17] train um a high level policy that [29:20] predicts the subtask instruction like [29:21] what to do next. What is the next like [29:23] subtask for the task of uh cleaning the [29:26] kitchen. Um and we'll additionally train [29:28] a world model to generate images for [29:30] what the robot should do next um as sub [29:33] goal image conditioning. [29:36] And so with this um we'll train a single [29:38] model on with those attributes on all of [29:41] the diverse data that we had available. [29:43] And here are some examples of what that [29:45] single model can do. So all of these [29:47] videos are from a single model. Uh [29:49] specifically a model that we called the [29:50] PIO7 model. Uh and on left you can see [29:54] it doing things like uh folding uh a [29:56] collared shirt. On the top right it's [29:58] doing a really precise uh assembly step [30:01] where it needs to insert a screw and [30:04] drill that screw into a robot arm. [30:07] And on the bottom right, the robot is uh [30:10] replacing a trash bag in a trash can. [30:15] And so we had two goals at the start of [30:17] this. The first was to move towards an [30:19] out-of-the-box model. And even those [30:21] videos showed that out of the box, the [30:22] model is able to do quite a bit. Um, but [30:25] really the key question here is how does [30:27] this pre-trained model compare to the [30:30] specialists that were trained [30:32] specifically for coffee making, [30:34] specifically for box building that I [30:36] talked about previously. [30:38] And if we measure the throughput and the [30:40] success rate of this single PIO7 model [30:42] versus the fine-tuned PIO6 model, we see [30:45] that the across the board the single PIO [30:49] like pre-trained PIO7 model matches or [30:52] outperforms the fine-tuned specialists [30:54] that were developed with reinforcement [30:56] learning post-training for those [30:58] downstream tasks. And so we see that [31:00] it's able to match the performance of [31:02] specialists. It also holds for SFT [31:04] specialists, not just RL post-trained [31:06] models as well. um suggesting that we do [31:09] indeed have a single model that can do a [31:11] lot of different tasks with a really [31:12] high degree of performance out of the [31:14] box. [31:16] Okay, so that was the first goal of out [31:18] of the box models. And the second goal [31:19] is compositional generalization. [31:22] Um and there's a few different ways to [31:24] measure this. Uh and there's many [31:25] different ways you might try to combine [31:27] concepts in robotics. Um the first test [31:30] that we wanted to do was to see if a [31:32] robot could um interact with an [31:34] appliance that's quite rare like an air [31:36] fryer. Um and uh this is kind of an [31:39] example. We basically wanted to see if [31:40] it could um open an air fryer, put an [31:43] air a sweet potato in the air fryer and [31:45] close the air fryer. Uh and we picked [31:47] this because we thought that the data [31:48] set didn't have any air fryers in it. We [31:50] didn't intentionally collect any [31:51] training data with air fryers. Um, [31:54] after we did some analysis on the data [31:56] set, we actually found that our data set [31:58] was so diverse that it did actually have [31:59] three episodes with air fryers in it. We [32:01] expect that they likely weren't having [32:02] an impact um on and that even if we [32:05] didn't include those exact three [32:06] episodes, it is likely would still work. [32:08] Um but what we found generally is that [32:10] the um the robot was able to interact [32:13] with appliance that was basically hardly [32:14] represented at all in the training data [32:16] set um and combine the skill of um [32:21] interacting with it like opening it, [32:22] closing it and so forth um with this [32:25] object that it hasn't seen before. [32:28] Um and then after instructing it uh like [32:30] Lucy did, we can train a high level [32:32] policy to do this task fully [32:33] autonomously. um and you can see um the [32:36] robot doing that in this video. [32:39] So that's the first form of [32:40] compositional generalization. Um the [32:42] second compositional generalization test [32:44] that we wanted to look at is we wanted [32:46] to see if we could compositionally [32:47] generalize between tasks and robot [32:49] platforms. And so we wanted to take a um [32:52] a robot platform called a barm robot. [32:55] It's actually a very large um kind of [32:57] industrial robot platform. And we wanted [32:59] to see if it could fold close. um [33:00] despite the fact that we didn't collect [33:02] any folding data on uh this robot [33:05] platform. So specifically we had data of [33:08] uh folding clothes like folding a shirt [33:10] on the robot platform that's pictured [33:12] here on the left. And then we wanted to [33:14] see like out of the box without [33:16] collecting any folding data on this very [33:19] different robot platform on the right uh [33:21] could the robot successfully do the [33:23] task. [33:24] And what we see in this video is uh the [33:27] we indeed did see that it kind of [33:30] compositionally generalized in this [33:31] manner. The first time we saw the robot [33:33] do this we were like like floored uh [33:35] because we like there was no training [33:37] data um for this uh for this task. And [33:40] the robot here is quite different from [33:42] the other robot not just in size but [33:44] also in the lengths of the linkages of [33:46] the robot in the configuration of the [33:48] joints of the robot and so forth. [33:52] Okay. we can. And the this is a 1x speed [33:55] video, so it's not the the fastest [33:56] thing. And obviously, if you haven't [33:57] seen any training data on something, you [33:59] might not uh if it's literally the [34:00] robot's first time folding a shirt. Um [34:02] it might take uh a few attempts, [34:05] but eventually it will get to the folded [34:08] shirt. You can also see the generated [34:09] subgoal images on the top left. Um so [34:12] those those are basically um the models [34:15] trying to generate images that will make [34:16] progress on the folding task. And then [34:18] those are passes input to the model. Uh, [34:21] and we see uh the the folded shirt here. [34:23] I think it's going to make a couple [34:25] small corrections at the end uh to try [34:26] to make it a little bit smoother. [34:30] Cool. So, the takeaway here is that um [34:31] both in terms of like language object [34:34] interactions and in terms of task robot [34:36] interactions, we see strong signs of [34:38] compositional generalization in this [34:40] model. [34:42] Okay. Um and then quantitatively we also [34:44] see that uh as we kind of get to these [34:48] more advanced models like the PIO7 model [34:50] the per performance of folding uh towels [34:53] and folding shirts on this platform that [34:55] hasn't seen before increases [34:57] dramatically uh and it even approaches [34:59] the performance of human teleop uh [35:01] despite the fact that we didn't have any [35:03] robot specific training data uh for [35:06] folding clothes. [35:09] And then for the last experiment that we [35:10] did here, um I think this is maybe [35:12] perhaps the most interesting experiment [35:14] is we wanted to test, how important are [35:17] the two ingredients that I mentioned. [35:18] How important is diverse data and how [35:19] important is this sort of like capacity [35:22] or detailed prompting for the kinds of [35:25] results that I showed? And so if we [35:27] remove the diver most diverse data um [35:30] from the model training uh shown in like [35:33] the grayish color, we find that the [35:35] performance on held out tasks decreases [35:38] dramatically. Whereas if we um just take [35:40] out a random 20% of the data that's less [35:43] diverse than the most diverse subset, [35:45] the performance um only decreases a [35:47] little bit. And so this suggests that [35:48] actually having really diverse data [35:51] plays an important role in enabling it [35:52] to generalize to new tasks. Um and then [35:55] we tried to also ablate the fact that we [35:57] are prompting the model with metadata. [36:00] And for this experiment we looked at um [36:03] with and without prompting with [36:04] metadata. Um so with prompting is shown [36:06] in yellow and without prompting is shown [36:08] in the gray color. And um with prompting [36:12] it helps uh kind of significantly. Um [36:14] but the most interesting thing is if you [36:15] look at um when you add so this plot is [36:18] showing as you add more and more data [36:19] and specifically as you add more and [36:21] more lowquality data what is the [36:22] performance and without metadata [36:24] prompting when you add lower quality [36:26] data from 80% data to 100% data the [36:29] performance actually decreases which is [36:31] perhaps not too surprising because [36:32] you're adding lowquality data to your [36:34] data mixture whereas with the metadata [36:36] prompting the performance actually [36:39] increases when you add that lowquality [36:41] data um sug suggesting that's actually [36:43] able to make get get a lot more juice [36:45] out of even lowquality data um when you [36:47] include this kind of prompting. [36:51] Cool. Um so the takeways here are that [36:53] um we found that we're able to train a [36:55] single model to control the robots that [36:57] matches or exceeds the performance of [36:59] specialized post-trained models. Kind of [37:01] like going from a BERT-like pre-trained [37:03] model to a model that really works out [37:05] of the box like GPT. We also saw strong [37:08] signs of compositional generalization in [37:11] a dolly like way. Um, for example, in [37:14] compositionally generalizing skills [37:15] applied to appliances and skills applied [37:18] to new robots in ways that weren't seen [37:20] in the training data. [37:22] Um, cool. And then all the like videos [37:24] and experiments that I showed were just [37:25] evaluating the model out of the box [37:27] without any post- trainining. Um, and [37:29] the paper and the technical report [37:31] online have a lot more experiments and a [37:33] lot more details. [37:35] Okay, so we've talked about long-term [37:38] autonomy. We then showed how we can kind [37:40] of develop that in a single general [37:42] purpose model. Um, where are we at now? [37:47] Um, the first thing that I'll mention is [37:50] if we go back to the timeline of [37:51] generalist AI, I think that we now kind [37:53] of firmly have physical intelligence in [37:56] the right side of this timeline. we're [37:57] kind of firmly more in like a GBT and [38:00] Dolly like era uh for robotics and [38:03] physical intelligence which is really [38:04] exciting and I think that um and we've [38:07] kind of went there in in in just a few [38:09] years. Uh and lastly uh we also have [38:13] these models that are actually deployed [38:15] um in real world circumstances. So the [38:16] two videos on the top are actually two [38:18] YC companies Ultra and Weave uh that [38:21] have taken find uh PI models and [38:23] post-trained them to do uh in deployment [38:25] to do tasks like folding laundry and um [38:29] and packaging in a warehouse. Uh the [38:32] kind of video on the bottom uh bottom [38:34] left is uh the video that I showed [38:36] previously and this kind of model works [38:39] for a really diverse set of robot [38:41] embodiment. Um the ones on on kind of [38:44] the top and the left are kind of a more [38:46] standard by manual platform. Um but it [38:48] also can be adapted to things like [38:51] drones, uh quadcopters, surgical robots, [38:53] and on the bottom right um for things [38:55] like tractors. And so this is really [38:57] kind of truly showing how physical [38:59] intelligence can make an impact uh not [39:01] just in demos and research and so forth, [39:04] but actually in real world deployment. [39:06] And I think that we'll um as we go we'll [39:08] start to see robots that are more and [39:10] more robots actually re deployed for [39:12] real in the physical world with all the [39:14] advances that we've been seeing over the [39:16] past few years. Awesome. Um so the last [39:19] thing that I'll mention shamelessly is [39:20] that we are hiring at physical [39:22] intelligence. So if you're excited about [39:23] some of the stuff that I talked about um [39:25] we encourage you to u to take a look at [39:27] some of the open roles and apply and um [39:29] yeah definitely have time for for [39:31] questions and and happy to u happy to [39:34] get all your thoughts. Thanks. [39:44] Okay, so the first question is how far [39:47] away are we from a chat GBT moment for [39:49] robotics and what will that look like? [39:51] So I'll start with the f the second part [39:53] actually which is that I I'm not sure it [39:55] will really look like the chat GBT [39:58] moment that we saw in language models [39:59] which is that with something like chat [40:01] GBT we saw like what like um it passed [40:03] like what a million new users in five [40:04] days uh I think that the distribution [40:07] channel for physical models is going to [40:09] be slower uh unfortunately because you [40:11] actually need a physical robot there and [40:13] I think that we've seen for something [40:14] like Whimo um it like the rollout um [40:17] it's actually been like incredible to [40:18] see the rollout that they've had but it [40:20] still takes time um to actually deploy [40:23] things on physical devices. Um so I [40:26] don't know if we'll have a single moment [40:27] that has the distribution that chatgpt [40:30] had. Um at the same time in terms of the [40:32] capabilities of these models I think [40:34] that we are really starting to get to [40:36] the point where these models are [40:37] actually useful in the real world and I [40:39] think that getting to the kind of the [40:42] capabilities of chat GBT I think is um [40:45] yeah very much on the horizon in the [40:47] next few years. [40:49] Cool. Well, the second question is when [40:51] should a small team switch from scaling [40:54] per site models to a generalist policy [40:56] and what does that transition actually [40:58] look like? What signals tell you it's [41:01] time? So, this is a good question. I [41:04] think that the I mean at the very least [41:06] I actually think that just starting with [41:07] a generalist policy and then fine-tuning [41:09] it even like right off the bat uh can be [41:12] really effective. And we've um and like [41:16] fortunately a lot of generalist policies [41:18] like they're actually really powerful [41:19] generalist policies that are open [41:20] source. Um and so the PI zero and PIO5 [41:23] models are are open source uh for [41:25] example. And we've seen a lot of people [41:27] um get a lot of use out of those models [41:30] already. Uh, and we're also working with [41:32] a lot of partners like the tractor [41:33] company, like Ultra, like weave um to [41:35] actually develop to take our most recent [41:37] models and um and get even more juice [41:41] out of them, even more powerful models [41:42] for their own applications. Um, so even [41:44] right off the bat, I think that you can [41:46] use them. Um, the only scenario in which [41:48] I actually wouldn't use them is if if [41:49] you're really in a really constrained [41:51] environment. I' I've talked to um some [41:53] folks working on surgical robots that [41:54] are like in an operating room in the [41:56] basement with no internet connection and [41:58] a really bad GPU. Uh, and so sometimes [42:00] like it's just really hard to use a [42:02] larger model. Um, but you still can do [42:04] local inference with um on a workstation [42:07] with these models. And so, um, I think [42:09] that yeah, right away I just taking like [42:11] PIO5 or or or your favorite model and [42:13] and fine-tuning it is, uh, is the way to [42:16] go. And I think that we'll see lots of [42:17] these small companies. And I think [42:18] there's like, yeah, so much work to do [42:19] in terms of actually getting these [42:20] robots to work with this technology in [42:22] the real world. [42:24] Great. The next question is given how [42:26] fast robotics is moving in industry what [42:28] are the real advantages and drawbacks of [42:30] doing a PhD today especially for someone [42:32] who wants to go into industry [42:33] afterwards. [42:36] So I was not planning to do a PhD. I was [42:40] always planning to go straight to [42:41] industry. My parents uh are engineers uh [42:44] and worked in industry and I kind of [42:47] thought that the way to have impact was [42:48] to go to a company and so forth. and the [42:52] and my dad even told me that he wouldn't [42:54] hire someone with a PhD. Uh and so I was [42:57] like maybe I shouldn't get a PhD if I uh [42:59] wouldn't be able to get a job. Uh but [43:02] well there's also he's in a different [43:04] field as well um in civil engineering. [43:06] Uh and so [43:09] the but at the same time I I think that [43:11] like a PhD is an incredible opportunity [43:13] and I I I love my PhD and obviously it [43:17] depends a lot on the adviser. It depends [43:18] a lot on um like what you would be doing [43:20] and so forth. But I think the PhD is an [43:22] incredible opportunity to first learn a [43:25] lot about how to handle uncertainty, how [43:27] to pick good problems to work on. And I [43:29] think that um basically like in in [43:31] research you don't no one kind of even [43:33] gives you the problem to work on. You [43:35] have to pick the problem. uh and you [43:36] don't know for the problem that you [43:37] picked if it is achievable to make [43:39] progress on that problem in a six-month [43:41] time span, two-year time span, 10 year [43:43] time span. Um and so you learn about how [43:45] to deal with that uncertainty. And then [43:48] the so that's really useful. Uh and then [43:51] it also um it also is an opportunity to [43:55] like do amazing research uh and do [43:57] whatever in many cases do like have a [43:59] lot of freedom to work on the research [44:00] that you find most exciting. And so I [44:02] think that today it's still uh yeah an [44:04] amazing opportunity to do do work to [44:07] learn about uncertainty. I think that [44:08] learning about uncertainty is really [44:10] useful in the startup environments in [44:12] being at the frontier of AI because we [44:14] don't know now like no one knows what [44:16] the best route is to make these models [44:18] more and more powerful. Um at the same [44:21] time there's also a lot of incredible [44:22] opportunities in industry. Uh I think [44:24] that the uh first in terms of what goes [44:27] into developing like everything that I [44:30] showed it's not just the research there [44:32] is a whole like software stack that [44:34] needs to run on the robot needs to run [44:35] reliably there's obviously everything on [44:37] the hardware side uh and then there's [44:39] also even the machine learning [44:40] infrastructure the data infrastructure [44:42] um and all that and so um even with like [44:46] and you don't need a PhD necessarily to [44:48] do a lot of that engineering work um and [44:50] then on the research side there's often [44:52] opportunities to get involved [44:54] um as well and a lot of research is [44:56] engineering these days as well. Uh and [44:58] so I think it kind of depends on um yeah [45:02] it's a very personal decision uh and [45:04] what you want uh I I even today I think [45:09] I probably like kind of retrospectively [45:11] um would want to do a PhD just to learn [45:13] about how to handle uncertainty to do [45:15] research because I really love being at [45:17] the frontier and thinking about [45:18] challenging problems. Um but there's [45:20] also a lot of uh really amazing [45:21] opportunities in in both paths. [45:25] Okay. Um the next question is large [45:27] language models learn from the internet [45:29] but robots don't really have an internet [45:31] scale data set a physical experience. [45:33] What's the robotics equivalent and how [45:35] do we get it? [45:37] So in robotics, well, so maybe in [45:39] language models to start off, uh the [45:41] data on the web is language data and not [45:45] all of it's high quality. Uh but some of [45:47] it is like really informative and [45:49] useful. Uh and it is data that is in [45:52] kind of the that that kind of reflects a [45:55] lot of what the like what you want a [45:57] model to do. Like you want it to be able [45:59] to predict uh text uh and be able to [46:02] complete text uh answer questions and so [46:04] forth. And there's a lot of questions [46:05] being answered on the internet and a lot [46:06] of text that's being completed on the [46:07] internet. Uh and so in general with [46:09] machine learning, you want train to [46:10] match test. And so you want the thing [46:13] that you're going to be training your [46:14] model on to be reflective of the thing [46:15] that you want you're going to be asking [46:16] it to do later on. Uh and so I think the [46:18] equivalent uh in robotics is data of [46:22] robots operating in real world [46:24] circumstances. Uh and the [46:27] way that we approach it at physical [46:29] intelligence um is to collect data like [46:32] collect robot experience. uh of robots [46:34] doing all sorts of tasks. Uh you can [46:36] collect this with teleyoperation as a [46:37] kind of to get initial data of robots [46:40] doing useful things. Uh but in the long [46:42] run I think it will also contain a lot [46:43] of autonomous experience of robots [46:45] deployed attempting things. Uh and just [46:47] like how we see in language models how [46:49] now a lot of time is spent actually [46:51] generating data, generating synthetic [46:53] data by actually running the model and [46:55] having it think through things. I think [46:57] a lot of the data in the future in [46:58] robotics is going to be the robot [47:00] attempting to do lots of tasks in lots [47:02] of real world circumstances. And so, um, [47:06] yeah, I think that that's kind of what [47:07] it looks like. I also think that there [47:09] are other possible sources of [47:11] information that's really useful for [47:12] model training, like videos of people [47:14] doing things, uh, like YouTube, uh, like [47:16] web data and captioned images, uh, that [47:19] tell you like this is a kitchen that has [47:22] a fridge on the right of the sink and so [47:24] forth. Uh and all of that data I think [47:26] can be really useful for developing a [47:28] kind of frontier multimodal model that [47:31] can control robots to do things, reason [47:33] through how to do a long task, uh and [47:34] also control the robot to do those [47:37] tasks. Um I think that there's no [47:39] substitute for the robot experience [47:41] itself. Uh you can't just like if you [47:44] watch a human do something like if I [47:46] watch Roger Federer play tennis doesn't [47:47] mean I can't I can play tennis as well [47:49] as him unfortunately. Uh and likewise um [47:51] robots can't like watch a person doing [47:53] something and then figure out how to do [47:55] it themselves directly. They really need [47:56] their experience on their own platform [47:59] um to learn effectively. And I think [48:00] that uh we will need large data sets. I [48:03] think that doesn't mean the human video [48:04] isn't useful. It's useful to watch Roger [48:06] Federer play tennis. But the um but the [48:09] actual experience on robot platforms [48:10] will be a critical component of [48:12] developing an analogous data set for [48:14] robotics. [48:16] The next question is, is it possible [48:18] that general purpose robotics models get [48:19] democratized via open source the way [48:21] that large language models did or will [48:23] the cost of embodied data and hardware [48:26] keep the best models concentrated in a [48:28] few well-resourced labs? [48:30] So, I think this is a good question. Um, [48:34] so I do think the cost of embodied data [48:36] and hardware will like could very much [48:39] make this look different. uh because I [48:40] think that it's harder to get data even [48:43] to like distill a model for example um [48:48] just like readily on the internet. Uh I [48:50] also think that we've seen uh kind of [48:53] pretty large data sets get open source [48:55] as well uh and pretty powerful models [48:57] get open sourced. The I think it's [48:59] really hard to say exactly what will [49:02] happen. Um [49:04] and so yeah I I don't know. Uh the one [49:09] thing that I will say is that I like [49:12] with language models even aside from [49:14] like doc de docuation like Kimmy and and [49:17] like really getting models that like [49:18] perform at the state-of-the-art even [49:20] then companies that are focusing a lot [49:22] on closed source models are also doing a [49:23] lot of open sourcing and so um there are [49:26] like like Gemma for example and um the [49:29] GBT open source and so forth. I think [49:30] these these companies like to support [49:32] open source because it actually helps [49:33] build the ecosystem around the things [49:35] that they're building and so I imagine [49:37] there being I guess I'm optimistic of [49:39] that there will be a strong open source [49:41] community regardless but I don't know if [49:43] it will exactly play out exactly the way [49:45] that language models played out. [49:48] Okay, the next question is does the [49:49] model output raw motor commands directly [49:51] or does it output a target hand position [49:54] and let a controller solve for the joint [49:55] angles and what makes that the right [49:58] level to learn at? So the mo all the [50:00] models that I showed were outputting um [50:03] target joint positions. So like what is [50:05] the angle of this joint? What is the [50:06] angle of this joint and so forth that [50:08] you want to hit? And then there's a [50:10] controller like a PD controller um that [50:13] is trying to then hit that target [50:14] position for those joints. Um the model [50:17] actually is also trained to predict [50:19] target uh gripper like like yeah gripper [50:21] positions like where in 3D space should [50:24] my gripper be? The um and you could also [50:26] use that as well and back out the joint [50:28] positions. Um you could also like [50:29] another thing you could do is you could [50:30] go directly to like motor torqus or to [50:33] uh like voltages or efforts. Uh there [50:37] are pros and cons of different options. [50:39] Uh we have found controlling joints uh [50:42] and controlling um in like the the 3D [50:45] space of the gripper uh to be to both [50:48] work well. Um [50:51] and so there yeah there's pros and cons. [50:53] I think that one thing that would be [50:54] nice about going directly to the like [50:56] the voltages is that you could also get [50:59] a more stiff uh kind of output or a less [51:02] stiff output. Uh whereas the controller [51:05] if you have a fixed controller then you [51:07] you're not letting your model control [51:09] that um that aspect and so yeah there's [51:12] different pros and cons. Um [51:14] the [51:16] yeah I think that the it see like what [51:19] we're working seems to work. It doesn't [51:20] seem to be a bottleneck. Um and and I [51:22] often like to focus on the things that [51:24] seem to be bottlenecks uh versus um [51:27] versus things that don't seem to be [51:28] bottlenecks. [51:30] Okay. Um next question is do robots need [51:33] something like imagination uh the [51:35] ability to picture what should happen [51:37] next before they can become truly [51:38] useful? So the pio7 model that I showed [51:41] has something like this where it can [51:45] kind of imagine what a future image [51:47] should look like and then try to [51:48] accomplish that. uh the [51:52] we found that that leads to improvement [51:53] and we saw in the shirt folding example [51:55] we saw like a quantitative bump from [51:57] using that sort of imagination compared [51:59] to not using it. At the same time I [52:01] think that the model actually performed [52:02] surprisingly well without that as well. [52:04] And we were actually hoping we were [52:05] thinking about writing an entire paper [52:07] an entire technical report just about [52:08] that capability in that model. Um, but [52:10] the the model without that was so good [52:12] that we felt like we needed to actually [52:14] have that play a bigger part of the [52:16] story because it it that seemed like [52:17] that was really delivering uh in terms [52:19] of actually getting really strong [52:20] results. And so I mean seems like one [52:24] design choice. I think that the uh [52:28] I I don't I think it's hard to say if [52:29] it's going to be a critical component or [52:31] not. I um the I guess the good news with [52:35] with these kinds of models is that the [52:37] if you develop a good data set, you can [52:38] kind of run experiments and you can [52:40] continue to test things uh with the data [52:41] set that you have uh quite effectively. [52:45] Um I also think that being able to [52:46] predict the future seems like a very [52:47] relevant objective compared to [52:49] predicting future actions and the um so [52:53] that should I would imagine kind of help [52:55] in terms of learning uh from as all the [52:58] data that you have available to you. So, [53:01] um, yeah, hard to say if it'll [53:02] necessarily be a critical component or [53:04] not. It seems like empirically so far it [53:06] seems to help. Um, although perhaps not [53:08] as much as you might expect and even [53:09] without that imagination, the robot can [53:11] do pretty incredible things. [53:14] Okay. Um, next is right now it seems [53:17] that robots are doing amazing tasks, but [53:18] in a very slow manner. What is needed to [53:20] improve the speed? Uh, I'm really [53:22] excited about improving the speed. The [53:25] and we we did see speed improvements [53:27] from reinforcement learning. We also [53:29] have another release uh called the RL [53:31] token that where we showed actually even [53:33] faster speed and actually faster speed [53:35] than human teleop. Um I think one of the [53:38] bottlenecks is that when you telly [53:40] operate robots to do things which is the [53:42] easiest way to teach a robot to do [53:43] something people are kind of slow at [53:45] telly operating the robot. Um we have a [53:48] couple projects in the pipeline that I [53:50] think have really promising results in [53:51] terms of getting fast policies. Uh and [53:53] so I think more to come there. Uh, and [53:57] yeah, I think it's either you need to [53:59] figure out how to make the data faster [54:01] or you need to figure out how to be [54:02] faster than the data. We've seen the [54:04] evidence of being able to be a little [54:05] bit faster than the data. And um, in [54:08] terms of the next steps, it's either [54:09] either to kind of get go even further [54:11] than that or or make the data faster. [54:16] Cool. What's the most surprising task [54:18] you've seen a robot complete recently? [54:19] What do you want to see it do next? Um [54:23] so the most surprising thing was not [54:24] really a task but uh when we were [54:27] working on PIO7 I was I personally [54:29] trained one of the policies for um what [54:31] like some of the initial tests for [54:33] assembling this uh pin wheel and when I [54:36] was working on train that training it to [54:38] uh to construct the pin wheel. One thing [54:41] that really surprised me was um in all [54:44] the data we kind of carefully control [54:46] controlled the strategy for how to [54:47] assemble the pin wheel where you um [54:50] basically will take the the kind of [54:51] pre-cut piece of paper and take a little [54:53] pin and insert the pin into a hole in [54:55] the paper. And in all of the data we [54:57] picked up the pin with the right hand [54:59] and picked up the paper with the left [55:01] hand and inserted it and the robot kind [55:03] of started doing that and then it [55:04] actually made a mistake and the paper [55:06] ended up on the right side and the pin [55:08] ended up on the left side. And what the [55:10] robot did is it picked up the paper and [55:12] it picked up the pin with its left [55:13] gripper and it put the pin with it left [55:15] gripper and inserted it into the paper [55:17] with its right. And it had never seen [55:19] data of inserting the pin with its left [55:21] gripper. And it kind of showed that even [55:23] that wasn't in the post- training data [55:25] at all like it was only like and it [55:27] wasn't even in pre-training either. Um [55:28] the robot essentially had learned this [55:31] sort of equivariance between his left [55:33] hand and his right hand so that it could [55:34] actually transfer uh behaviors from one [55:37] hand to another. uh despite the fact [55:39] that that was never in the data. Uh and [55:40] so that was like a really cool moment. I [55:42] feel like I don't know if other people [55:44] were as excited about it as I was uh [55:45] when I shared it with some people. Um [55:47] but it kind of shows this kind of [55:49] emergent capability in these models that [55:51] I hadn't seen before. Um [55:54] and then yeah, in terms of what I'd love [55:57] to see, I don't know. Um [56:00] I I love seeing robots do anything. And [56:04] the [56:05] I guess I think that there's still a lot [56:07] of a long way to push in terms of [56:08] reliability for robots being able to do [56:10] tasks for really long periods of time. I [56:12] don't necessarily think that much about [56:13] individual tasks, but more so about [56:15] capabilities and how to get the next [56:17] capability from these models. The um [56:22] yeah, and so I yeah, anything um I guess [56:24] the yeah, a robot doing anything always [56:28] gets me excited even if it's something [56:30] that hasn't been done before. I guess [56:31] one thing that we've been doing recently [56:32] is um having robots uh use knives to [56:35] slice vegetables. I think that there's a [56:37] lot that you can do there once you [56:38] actually can use knives safely, which is [56:40] one thing that we've done recently. [56:43] Okay. And then the last question uh is [56:47] how can someone break into robotics from [56:49] a software engineering background? [56:52] Great. Um so I think that the I guess [56:56] first there's a lot of software [56:57] engineering in robotics and so uh [57:00] there's I think you could try joining a [57:02] a robotics company as a software [57:04] engineer. Uh and then another thing that [57:06] I would mention and I've actually seen [57:08] someone take this path uh so someone um [57:10] who actually now works at physical [57:12] intelligence her name is Jenny. she uh [57:14] worked in trading, algorithmic trading [57:16] for a while. Um then she worked at [57:17] Harvey and was doing legal stuff and um [57:20] she was really excited about robots and [57:22] so she bought a cheap robot uh and [57:24] basically in her bedroom um like played [57:27] around with it uh and tried like [57:29] fine-tuning an open source model and [57:31] trying to get it to do something. And [57:32] then she shared what she had done and um [57:35] she sent me a cold email and was like, [57:37] "Hey, can I like I'm interested in [57:39] working in your lab." And it seemed like [57:40] her profile was promising and that she [57:42] actually like actually got out there and [57:43] like tried it and done it and that she [57:45] really was really excited about that. Uh [57:47] and so the and now now she works at [57:49] physical intelligence the but yeah I [57:52] think just like getting your uh getting [57:53] your feet wet trying stuff out uh and [57:56] learning from that experience uh and [57:58] then yeah using that experience to uh [58:00] kind of yeah share with people have it [58:02] on your resume and so forth I think is a [58:04] great way to do stuff and fortunately [58:07] there's a lot of open source stuff out [58:08] there uh that can allow you to get [58:10] started on those kinds of things. [58:12] Great. Um, that was the last uh [58:14] question. Uh, thanks everyone for [58:16] listening.