Sergey Levine: Humanoid Robotics Results, Chinese Labs & Future Timelines

Sergey Levine：人形机器人成果、中国实验室与未来时间表 Sergey Levine解释称，现代机器人技术仍处于寻找合适的可扩展算法杠杆的探索阶段，并将当前的概念验证模型与后期可预测的LLM扩展曲线进行了对比。他认为，真正的进展取决于泛化能力、语义错误恢复，以及将被动的多模态数据建立在多样化的真实世界感知运动经验之上，而非炫技式的杂技表演或狭隘的工业重复操作。在审视行业动态时，Levine强调了对具有竞争力的本土硬件制造业的迫切需求，将机器人精简的基础软件栈与自动驾驶汽车笨重的单体架构进行了对比，并探讨了为何注入广泛的先验知识优于从零开始的“白板”学习。 机器人技术目前处于类似于GPT-3问世前早期语言模型的探索阶段，重点在于确定可扩展的杠杆和数据配比，而非可预测的参数扩展。 (0:40) 语言模型允许用户通过迭代来优化不完美的输出，但机器人技术不同，它要求完全自主的可靠性，这意味着如果无法跨越从95%到100%成功率的差距，其整个价值主张就会瓦解。 (37:10) 对Physical Intelligence的π0.5等先进模型的实证评估展示出语义层面上“孩童般”的失败模式，例如临时把物品放进烤箱收纳，或是把碎盘子遮盖起来以标记任务完成。 (4:20) 与传统的预训练层级相反，Sergey Levine认为，机器人的感知运动遥测数据必须作为首要的基础层，模型在此基础上才能有效地将被动的人类视频数据与现实建立关联并加以吸收。 (27:13)

Robotics has not yet reached the predictable scaling curves seen in large language models, operating instead in an exploratory phase to discover the foundational architectural and data levers. Sergey Levine argues that achieving general-purpose autonomy requires grounding models in diverse physical experience, deploying thin end-to-end software stacks, and overcoming an unforgiving reliability threshold within single-digit years. The Pre-Scaling Frontier and Emergent Failures (0:40) Levine frames artificial intelligence development in two phases: first uncovering scalable architectures and data mixtures, and second executing industrial scale-up. While natural language processing transitioned from LSTMs to predictably scalable transformers between GPT-2 and GPT-4, robotics remains in the first phase. Current demonstrations validate underlying mechanisms rather than brute-force compute scaling, making precise timeline forecasting difficult. Despite this early stage, modern robotic policies exhibit emergent behaviors that were virtually non-existent before 2024. During a late 2024 evaluation, a robot unexpectedly disentangled two concurrently grabbed shirts, returned one to a hamper, and folded the target garment. Similarly, failures in models like Physical Intelligence's π0.5 now mirror coherent, childlike misunderstandings rather than random noise—such as stashing silverware in an oven when a drawer resisted opening, or parking over a dropped plate to conceal it while checking off task completion on an internal scratchpad. Embodied Physical Grounding Inverts Traditional Training (27:13) In natural language processing, standard methodology pretrains on vast internet text before fine-tuning on domain-specific data. Levine contends robotics must invert this order: direct sensorimotor experience must serve as the primary foundation, after which models can meaningfully interpret passive video and simulation. Drawing on research by Suraj Nair and Simar Kareer, Levine notes that scaling multi-robot telemetry enables internal feature representations to cluster by task identity rather than mechanical embodiment, allowing human video demonstrations to align directly with robotic executions. Cross-embodiment architectures have also proved simpler than anticipated. Rather than employing complex morphologically factored heads, unified policies can output high-dimensional vectors with zero-padded channels for robots with fewer degrees of freedom. For fine manipulation across disparate kinematics, systems leverage visual milestone dreaming—such as a Universal Robots UR5 synthesizing target images to fold t-shirts without prior hardware-specific demonstrations. This marks an evolution away from Levine's early 'Arm Farm' experiments at Google, demonstrating that tabula rasa learning fails without external prior knowledge and semantic scaffolding. The Fallacy of Homogeneous Data Flywheels (13:15) Levine refutes the premise that robotic data is a fungible commodity capable of creating a winner-take-all data flywheel through repetitive deployments. Collecting one million identical welds per month from a factory arm merely optimizes that specific weld on that specific chassis, yielding near-zero general capability. A functional data flywheel functions as an educational curriculum requiring extreme behavioral and environmental heterogeneity, meaning attempts to scale narrow industrial verticals will fail to produce foundation models. Addressing advancements in Chinese robotics, Levine urges Western institutions to view China as an example of an integrated hardware and software ecosystem. Algorithmic advances cannot compensate for neglected domestic manufacturing; researchers require access to low-cost, reliable physical hardware. Levine highlights Tony Zhao's ALOHA study—which paired roughly $7,000 hobbyist Trossen Robotics arms with Action Chunking with Transformers (ACT) to accomplish delicate tasks like swapping remote batteries—as proof that accessible hardware combined with straightforward imitation learning can outperform expensive, bespoke systems. Deployment Horizons and the Reliability Premortem (34:36) Levine projects meaningful real-world deployments on single-digit year horizons, with semi-structured commercial applications rolling out within a year and unstructured home environments following shortly after. Contrasting robotics with autonomous vehicles, Levine highlights that Waymo spent over a decade navigating deeply layered, hand-crafted software stacks under extreme safety constraints. Humanoid robotics operates on an exceptionally thin runtime foundation model stack where physical manipulation errors—such as dropping fragile objects—present far lower safety risks than motor vehicle collisions. In a premortem analysis, Levine identifies the primary failure risk not as hardware or software architecture, but as the unforgiving bar for real-world robustness. Unlike large language models where users tolerate flaws by interactively refining prompts, robotics loses its economic value if humans must constantly intervene. Pushing reliability from a 50% or 95% ceiling toward enterprise-grade completion cannot depend on current models alone, requiring reinforcement learning over autonomous real-world experience and semantic common sense for unprompted error recovery. Sergey Levine dismisses both utopian and dystopian forecasts of robots as 1:1 'mechanical people' displacing human workers, projecting instead that physical actuation will decentralize into ubiquitous everyday tools, mirroring the trajectory of microprocessors and coding assistants. The unresolved technical question remains whether robotic policies can autonomously self-improve and correct their own errors when deployed into unfamiliar environments without plateauing.

[18:20] Sergey Levine: capping out at like 50% [27:13] Sergey Levine: add salt [31:49] Sergey Levine: move left arm thirty-seven centimeters to the left [35:57] Sergey Levine: that's a lot less bad than having a car hit somebody [37:10] Sergey Levine: it doesn't make it obvious how far or how close it is to reaching that practically relevant level of robustness. [39:55] Sergey Levine: If you have a particular problem you want to solve, it is better to train a more general model that can use data from a breadth of problems and if you do it right, it'll actually be better at the specialized problem you want to solve than a narrow specialist. [45:19] Sergey Levine: a little bit of physical actuation in everything [46:44] Sergey Levine: partly a shameless plug [46:44] Sergey Levine: candidly like 99.9% of the work on this was done by Tony [46:44] Sergey Levine: The details matter, but the details don't have to be complicated. [54:36] Sergey Levine: Okay, now I can pick up anything, but like so what? [54:36] Sergey Levine: at least not very eloquently [54:36] Sergey Levine: take prior knowledge more seriously.

[0:00] To me, that's kind of mind-blowing [0:01] because like the base model wasn't [0:02] trained on any human data at all. [0:04] >> This is Sergey Lavine, one of the [0:06] world's leading robotics researchers, [0:08] and I asked him all about the current [0:09] state of humanoid robotics. What are the [0:12] most astonishing emergent capabilities [0:14] you've seen so far? [0:15] >> I don't think anybody watching that eval [0:17] thought that the robot was going to do [0:19] this. [0:20] >> I have some questions here on China. [0:22] There's no avoiding it sometimes. [0:25] If humanoid robotics did not succeed in [0:27] singledigit years, what do you think [0:29] would be the most likely reason why [0:31] humanoid robotics failed? [0:35] Here's the full episode. [0:40] LLMs have caused unprecedented impact [0:43] and investment in that area. Physical AI [0:46] and humanoid robots could potentially be [0:49] even bigger. And so I wanted to ask you [0:52] today about where are we today with [0:55] humanoid robotics and how do you foresee [0:58] this technology actually being deployed [1:00] into the world. I guess with machine [1:03] learning uh what we've learned over the [1:05] last few years over the last decade [1:07] rather is that um it works when you do [1:10] it at scale and this is like very [1:11] obvious now but it wasn't always [1:13] obvious. Uh but there's a caveat which [1:15] is you have to like scale the right [1:17] thing and uh you know initially when [1:20] people started working on um models for [1:22] language for example the dominant uh [1:25] design was uh LSTMs um some people [1:28] remember what those are. They were like [1:30] kind of okay they were a lot better than [1:32] what came before that but they didn't [1:33] really scale as well. And then the the [1:35] big thing with transformers was not that [1:37] transformers were somehow like uh [1:39] particularly um you know mathematically [1:41] elegant or anything like that. It's just [1:42] that they scaled better. So they were [1:43] easier to train a very large amounts of [1:45] data with lots of parameters. So the [1:47] technology proceeds in phases. First you [1:48] figure out what you can scale basically [1:50] what is the scalable technology and then [1:52] you pour on a lot more of kind of an [1:54] industrial scale effort adding lots of [1:56] data adding to model size and that's [1:58] when kind of the magic happens. So when [2:01] we're doing kind of more fundamental [2:02] technology development the key is to [2:04] understand like what are those scalable [2:05] levers. So figure out the design, figure [2:08] out like roughly the mixtures and and [2:10] that by itself does something pretty [2:12] cool, but not like uh but that's that's [2:15] not the thing that actually changes the [2:16] world. It's like when you start pulling [2:17] that lever that things actually change. [2:19] So with LLMs um when uh like the first [2:23] GPT models came out uh with GPT2 like it [2:27] did some stuff but it was sort of like [2:28] um like a parlor shook, right? So you [2:30] you could like get it to synthesize a [2:32] story about like unicorns in Peru or [2:35] something and it was like coherent [2:36] English but it wasn't like a thing that [2:37] would solve lots of real world problems. [2:39] But the folks that worked on this kind [2:41] of recognized that like hey there's [2:43] something kind of magical that happens [2:44] because uh as you add uh more data and [2:47] you make the model bigger like you know [2:48] this stuff gets more coherent and more [2:49] effective. So they they they could see [2:51] that if we do a lot more of that then [2:53] it'll become a lot more powerful. So uh [2:55] to come back to your question what I [2:58] would say about robotics is that it's [3:00] not in the in the like GPT4 to GPT5 [3:03] stage where it's like an industrial [3:05] scale effort to kind of make the model [3:07] bigger and and u get more capability out [3:10] it. It's in that stage where we're [3:11] establishing the fundamental [3:12] technologies and because of that what uh [3:16] one should expect to see right now is [3:18] not necessarily that like you know each [3:21] month the model gets bigger and more [3:23] powerful by some predictable uh kind of [3:25] scaling curve. It's that the I guess the [3:28] scaling properties themselves are [3:30] evolving as we develop the right [3:31] technologies. So, uh, to to bring this [3:33] back to something like closer to [3:35] reality, um, I'm very happy with like [3:37] the demos that we're doing here at [3:39] physical intelligence, and I think that [3:40] a lot of the results that other people [3:42] are coming out with are really cool, but [3:43] these are to put them in context, we [3:46] should not expect these to be the things [3:47] that are actually illustrating the power [3:48] of scale. We should expect them to be [3:50] developing the fundamental technologies [3:52] that will be scaled up after that. So, [3:55] so where I think we're at now is that [3:57] we're actually kind of getting all those [3:59] puzzle pieces in place. And I think it's [4:00] actually very close. I think a lot of [4:01] the puzzle pieces are falling in place. [4:03] Uh but it's not like uh what makes it so [4:06] hard to prognosticate about where the [4:08] technology is going to go is that it's [4:10] not yet at that predictable scaling [4:11] stage. It's at the stage where we're [4:12] figuring out the puzzle pieces, which I [4:15] think is really exciting, but it means [4:16] that it's also very very hard to foresee [4:18] like you know sort of what the [4:19] coefficients on that will be. [4:20] >> What are the most astonishing emerging [4:23] capabilities you've seen so far? that [4:25] that is the thing that is the most fun [4:27] and and certainly we've seen a a lot [4:30] more of that happening as we progress. [4:32] Um like you know in the very beginning [4:35] it was like it was kind of little things [4:36] but they were kind of magical because [4:39] you know in robotics basically prior to [4:41] like 2024 the stuff like never happened. [4:43] Uh so the little things that happen and [4:45] this was maybe like at this point about [4:47] two years back we would see things like [4:49] okay we train our policy for folding [4:51] laundry and it takes out like individual [4:53] shirts out of the hamper and tries to [4:55] fold them and then um um one very vivid [4:59] memory I have uh in late 2024 is we we [5:02] were watching one of the evals and it [5:03] takes out like two shirts at the same [5:04] time and I'm I'm watching this I'm like [5:06] okay like like it's done for like [5:09] there's no way it can possibly do this [5:10] and then it puts the two shirts on the [5:13] table disentangles and puts one of them [5:14] back and then starts folding the other [5:16] one. It's like wow like that. Okay, like [5:18] in retrospect you can do some like [5:20] detective work and figure out where it [5:21] got that from some piece of training [5:22] data, but that was like one of those [5:24] moments where I don't think anybody [5:26] watching that eval thought that like the [5:28] robot was going to do this. It's like a [5:30] little thing. It's like exhibiting the [5:31] common sense you expect people to have. [5:33] Um but actually the thing that I find [5:34] more interesting recently is um uh some [5:37] of the mistakes because one of the [5:39] things that's uh that was pretty [5:41] remarkable about LLMs is that once they [5:44] got good enough even the mistakes kind [5:46] of made sense in the sense that they [5:47] weren't like crazy mistakes where just [5:48] outputs like ZZZ all the time but they [5:51] were mistakes that sort of semantically [5:52] are sensible. Uh we had uh an evaluation [5:55] uh last year for pio5 where the robot [5:58] was uh uh cleaning up a kitchen and it's [6:00] told like put away all the all the [6:02] utensils. Like there's like some some [6:03] spoons, spatulas, etc. And uh it tries [6:07] to open the drawer where it thinks the [6:08] silverware goes and it can't get the [6:09] drawer open. So it slides over and opens [6:11] the oven which is right next to it and [6:13] then starts putting this stuff in the [6:14] oven. [laughter] It's like, you know, [6:16] you can sort of imagine that if you ask [6:18] like a child to clean stuff up and put [6:19] it away, like they might they might [6:20] decide to do that because, okay, it's [6:21] like a container and you can put stuff [6:22] there and nobody sees it. Um, another [6:25] experiment we had is uh uh washing all [6:27] the plates. So, it would like pick up [6:29] the plates, wash them with a sponge, and [6:30] put them on the drying rack. Uh, and uh [6:32] this was an experiment on memory because [6:34] it has to keep track of everything that [6:36] it's doing. It has like a scratch pad [6:37] kind of memory where it's writing down [6:38] like, hey, you know, I had three plates. [6:40] I cleaned the gray one. I cleaned the [6:41] green one. And then it drops one of them [6:43] on the floor and drives the base over. [6:45] so you can't see. I was like, "Okay, [6:46] I've cleaned the the gray plate. It's [6:48] it's [laughter] done." [6:50] Um, so I mean, obviously like these are [6:52] not the things that we want to see, but [6:54] it's kind of interesting that some of [6:55] the mistakes, they're almost like what [6:57] you would associate with like a child [6:59] trying to do the task. So now now now it [7:01] just needs to grow up. [7:03] >> You mentioned uh the advancements. [7:05] They're kind of happening all over the [7:08] industry. And I I know there's a lot of [7:10] people building humanoid robots. There's [7:13] Figure, there's there's Tesla and [7:14] there's many other competitors and you [7:17] have the expertise of what is hard and [7:19] what isn't. When you look at the [7:21] competitors, has there been any [7:24] advancement or achievement where you [7:26] think, "Oh, that's really admirable and [7:28] that's impressive." actually think that [7:31] one of the most inspiring things to me [7:33] in the industry is to see the um the [7:36] kind of takeoff that autonomous driving [7:37] systems have had because you know like [7:39] one of the criticisms that is sometimes [7:42] leveled against robotics researchers is [7:44] uh you know it's it's like uh you know [7:46] same as like nuclear fusion like it's [7:47] it's the technology of the future but [7:49] it's always in the future but that's [7:50] what people said about autonomous [7:51] driving too and now like you know we're [7:53] in San Francisco you can go outside you [7:54] can take a Whimo and it will actually [7:56] like take you to your destination and [7:57] there's no driver sitting there So I [8:00] like you know without getting too much [8:02] into the technical details I think [8:03] what's really inspiring about that is [8:04] just like this case and point that yes [8:07] you can actually have one of these [8:09] technologies of the future and it [8:10] actually does land and I think it's not [8:12] an accident that it's landing now in the [8:14] mid 2020s because a lot of the puzzle [8:17] pieces for largecale ML are getting to [8:20] the level where we can put them together [8:23] with like actual physical systems and [8:25] there's a lot of differences between [8:26] driving and robotic manipulation of [8:27] course but I think the illustration that [8:30] like yeah we can actually land uh [8:32] learning based technologies in the real [8:34] physical world I think that's really [8:35] inspiring [8:36] >> if openthropic [8:38] started investing you know more heavily [8:40] into robotics how do you think that [8:43] would impact the industry do you think [8:45] competitors would be you know worried [8:47] about that robotics is um is an area [8:52] where [8:53] maybe to be fair like the ecosystem [8:55] hasn't been as healthy as it has in [8:57] other areas of machine learning And what [8:58] I mean by that is that um like computer [9:01] vision and NLP are things that sort of [9:03] lend themselves naturally to a machine [9:07] learning based ecosystem because there's [9:09] freely available data. Uh people kind of [9:11] have a a general acceptance that they're [9:13] going to be using learning. You know, [9:15] there aren't concerns as severe concerns [9:17] about like safety, at least physical [9:19] safety, right? You know, people [9:21] rightfully are concerned about AI [9:22] safety, but it's not the same as like a [9:24] physical device uh causing some physical [9:26] harm. um and uh because of that I think [9:29] it's a bit easier to spin up a very [9:32] serious large scale ML effort in those [9:33] areas. Robotics is not like that. [9:35] Robotics traditionally is not a [9:38] discipline that um you know really [9:41] embraces sharing of data for example. So [9:44] I think that the more activity there is [9:46] around learning in robotics, the more I [9:49] think it'll shift people's thinking [9:50] towards this kind of future where we [9:52] accept that robots will be controlled by [9:54] learned models, not by handdesigned [9:56] controllers, that there will be data [9:58] that data will need to be shared because [10:00] there's no way that somebody can uh [10:02] build out a true foundation model in a [10:03] single vertical uh and basically kind of [10:06] like shift the entire thinking around [10:07] robotics to look more like how we think [10:09] about vision and NLP as opposed to [10:10] traditional factory automation. So I [10:13] think in that sense like you know much [10:15] as I'm like proud of the work that we're [10:16] doing in physical intelligence I think [10:18] it'll take more than one company to kind [10:20] of like shift every everyone's thinking [10:21] in that direction. As a bystander I see [10:24] on Twitter these really impressive [10:26] demonstrations of Chinese robotics. It [10:28] it almost feels like they're ahead in [10:31] some sense but I don't have that uh deep [10:35] domain expertise. And so I was curious [10:37] your thoughts on, you know, what do you [10:39] think of China's robotics and, you know, [10:43] are are they further along? [10:45] >> I think that one thing that um [10:49] is very useful and constructive for for [10:52] us to do uh those of us that uh work on [10:54] these things in the United States and in [10:56] Europe, uh is to [10:59] ask what is the lesson to learn? And to [11:02] me one lesson is that it's important to [11:06] have a healthy ecosystem [11:08] and ecosystem means that there should be [11:11] um you know obviously good researchers, [11:14] good engineers working on these things. [11:16] there should be like healthy open source [11:18] but it also means that the different [11:22] industries that contribute to robotics [11:25] need to individually be very healthy and [11:28] those industries are not just the um you [11:30] know computer science ML and and model [11:32] building stuff it's also uh supply [11:35] chains manufacturing [11:38] hardware R&D like these are all very [11:40] important things and [11:42] aspects of those things are things that [11:45] uh the United States does quite Other [11:47] aspects of them are things where the [11:49] United States has sort of uh you know [11:50] let things go a little bit. Uh and I [11:52] think that what we should do is we [11:54] should look at what's going on in the [11:55] world look at the uh some of the uh [11:59] excellent results uh that uh Chinese [12:01] labs are doing that labs in other [12:03] countries are doing and we should take [12:04] away that lesson that we should strive [12:05] to build a healthier ecosystem and that [12:07] means investing in all the different [12:09] facets uh that contribute to this. So [12:12] you know I'm not like I'm not much of a [12:13] business person. I'm not much of a [12:15] investment person. So I don't I can't [12:17] claim to know how to do this, but I [12:18] think that it's important to sort of [12:20] fully embrace that this is like a [12:22] holistic thing and not something where [12:23] we can do just like one piece of it and [12:24] like outsource everything else basically [12:27] >> of all that the pieces of the ecosystem [12:30] are there parts of it as a robotics [12:33] researcher in the US if it was better [12:36] that would have the biggest impact on [12:37] the advancement of robotics? Yeah, I I [12:39] think certainly um availability of [12:42] reliable lowcost hardware is a big deal [12:45] and right now I mean certainly for uh [12:48] hardware used for robotics research a [12:50] lot of that does come from China and [12:52] it's it's good it's uh relatively [12:54] inexpensive it's of high quality and uh [12:56] meets the uh standards that people [12:59] generally need. Uh but it would be [13:01] awfully nice to be able to uh source all [13:03] that domestically as well. And I I I [13:05] don't think there's anything impossible [13:06] about that. I think it's just a matter [13:08] of sort of uh embracing the fact that [13:10] that entire uh ecosystem needs to be [13:13] supported rather than just one piece of [13:14] it. [13:15] >> You can imagine that the lab that is [13:18] first to get to scale will be the first [13:22] to break out there. There may be this [13:24] exponential growth effect where once you [13:27] deploy deployment helps you grow faster [13:29] and growing faster helps you deploy and [13:31] you know this this flywheel and so do [13:35] you uh believe that that will happen in [13:37] this industry where one lab whether in [13:40] US or China will hit some breakout point [13:44] and they'll kind of jump everyone else. [13:47] >> There is a lot of um uh truth to that. I [13:50] think that there's an important detail [13:51] to keep in mind. Um the detail is that [13:55] um you kind of have to like scale the [13:56] right thing. Um so I think that [13:59] basically the statement that like the [14:02] way that I would phrase this is [14:05] having an effective um positive feedback [14:08] loop where more deployed robots [14:11] translates to more model capability. [14:13] That's kind of the key. Uh and like that [14:16] that makes total sense. the the trick is [14:19] that there are lots of ways that um that [14:22] it can be done wrong. So, I'll tell you [14:24] like a few obvious ones. Like one [14:25] obvious one is um let's say that uh I um [14:29] you know I'm a car company and I have a [14:32] robotic arm that is welding cars and [14:34] it's like there on the uh on the [14:36] assembly line and it welds cars every [14:38] day and it gets like you know a million [14:39] welds every month and so on. [14:42] If I just use that as my data flywheel, [14:45] I'm unlikely to get something more [14:47] capable than a robot that welds cars. [14:49] So, since the robot is already there and [14:51] it's already welding cars, like [14:52] presumably the marginal improvement for [14:53] that is not all that valuable. So, [14:55] that's an example of how you kind of [14:57] have to like scale the right thing [14:58] because data is not like um it's not [15:01] quite as funible like it's not like uh [15:03] electricity or oil. You can't just like [15:04] buy more of it. It has to be [15:05] heterogeneous. So I I think that [15:08] basically it's right but you have to [15:09] scale the right thing with the right [15:10] technology and the right kind of source [15:12] of diverse learning like data is more [15:15] like an education program for your robot [15:16] than it is a funible commodity. [15:19] >> If you think about that data flywheel [15:22] and I guess putting out the the proper [15:24] platform that would generate data that [15:26] matters that would then improve that [15:28] platform. When do you first see this [15:32] kind of happening in the world? I think [15:34] on the technology side like things are [15:36] advancing very rapidly towards that and [15:38] I think that maybe um a particular [15:40] balancing act to strike that would [15:42] significantly determine that timeline is [15:44] kind of how much structure somebody's [15:47] willing to admit. So, you know, on the [15:49] one on the one extreme, you could [15:51] imagine like jumping straight to fully [15:52] unstructured deployment domains like [15:54] home robots and that could be really [15:56] exciting because then you have a lot of [15:57] diversity like right off the bat, but [15:59] the bar is a lot higher to uh be [16:02] effective enough in that domain to be [16:03] safe enough like you know safety is much [16:05] bigger issue there because it's around [16:07] um uh people in their home. On the other [16:09] extreme, you could imagine much more [16:11] structured tasks. Maybe not quite the [16:13] welding robot, but sort of like, you [16:14] know, maybe the robot down the hall that [16:16] does something more unstructured, right? [16:18] And that could be a much easier domain [16:20] in the sense that there's much less uh [16:21] safety concerns because it might be [16:23] around trained uh humans. Uh the task [16:26] might be more uh predictable and so on, [16:28] but there is uh um the marginal value of [16:32] each bit of data you get in that domain [16:34] is lower because there's less variety. [16:35] Um, so it's like you could take off [16:37] earlier but with a smaller slope or [16:39] later but with a larger slope. And you [16:40] kind of have to like calibrate that. But [16:42] my sense is that whichever end of that [16:44] extreme we're talking about, it's in the [16:47] uh singledigit years rather than the [16:48] double digit years at this point. So it [16:51] could be that the more structured ones [16:52] might be happening like now or next [16:54] year. The less structured ones might be [16:56] a few more years out, but probably not [16:58] like a decade. [16:59] >> A lot of people talk about timelines and [17:01] it's kind of more uh nebulous. And I'm [17:04] I'm wondering if if you were to put down [17:06] a I guess somewhat of a road map, but [17:09] not so concrete, but just milestones [17:12] towards that north star of [17:15] >> robot in my home doing. [17:16] >> Yeah. Yeah, that's a really good [17:18] question. So um maybe to to preface this [17:21] answer, I think that there's one thing [17:23] that is very important to say about [17:25] robotics that is very easy to miss um [17:28] from kind of like I guess like the hype [17:30] cycle and the demos that people put out, [17:32] which is that the the hard thing in [17:34] robotics was always um generalization. [17:38] But when somebody shows a demonstration [17:41] of their system, you know, the [17:43] demonstration alone usually doesn't make [17:45] it clear what level of generalization is [17:47] being shown. the uh highly acrobatic [17:50] robot demos for example are really [17:51] exciting to look at. But typically if [17:53] it's uh something that is like a little [17:55] bit more staged like sometimes it's [17:57] literally on stage if it's like a show [17:58] like you know that that is obviously [18:01] like rehearsed and that's okay because [18:02] it's like literally just a show but it's [18:05] not the same as doing a task every time [18:08] reliably in any home. And the [18:12] generalization piece often doesn't look [18:15] that impressive when viewed in isolation [18:17] because generalization is sort of a [18:18] property of many trials, not of one [18:20] trial. So you might see the robot doing [18:22] something fairly mundane and [18:23] unimpressive. But what's exciting about [18:25] it is that it's doing with an object [18:27] that it's never seen before in an [18:29] environment that has never been tested [18:30] in before. And that's actually harder [18:32] than like doing a acrobatic backflip [18:34] that it's practiced like a millions of [18:37] times. So um with that said uh my answer [18:41] to your question is that the road map is [18:44] all about both achieving better [18:46] generalization and uh kind of that [18:48] second order effect of having a [18:50] mechanism to get more generalization as [18:52] you generalize. So one of the steps on [18:54] that road map is to have a very concrete [18:58] demonstration of a robotic system that [19:00] gets better with autonomous experience [19:03] that is collected in in a setting that [19:06] it wasn't originally trained for. So I I [19:09] do whatever I do on on on the back end, [19:11] you know, in the lab in and whatever. I [19:13] get my model, I get my uh adaptation [19:15] algorithm, I put it in a new setting. [19:18] Maybe it's a home, maybe it's a factor, [19:20] whatever it is, something where it's [19:21] doing something real. uh and it does [19:23] okay, but then over time it gets better [19:25] and better and it gets better and better [19:27] to the point where it reaches sort of [19:30] practically relevant levels of [19:31] robustness without sort of capping out [19:33] at like 50%. Like that I think would be [19:35] a major milestone. Uh because now that [19:37] that says okay if this is truly an [19:39] automated process, it's improving, it's [19:40] getting better even as it's collecting [19:42] useful experience. Now I can take it and [19:44] I can put it in lots of different [19:45] domains, collect useful experience, do [19:47] something that people actually want and [19:49] it'll improve the model. So that I think [19:51] is a really major step uh on that road. [19:54] I think another really major step is to [19:57] demonstrate a very concrete and [20:01] practically useful way to transfer [20:03] knowledge to transfer common sense to [20:05] achieve robustness. So that's like the u [20:08] kind of the the other scenario where you [20:10] don't get to practice. If you're driving [20:11] your car on the road and you see a fire [20:13] truck and you see like a bunch of [20:14] traffic cones, even if you've never been [20:16] in that situation before, like your [20:17] common sense tells you like, "Hey, I [20:19] should like slow down." like what what [20:20] maybe I don't know how to react [20:21] optimally, but I shouldn't just like [20:22] barrel on through the cones and like you [20:24] know uh upset the firefighters and so [20:26] on. Like so that's common sense. And if [20:28] you can apply that common sense to [20:30] effectively recover from unexpected [20:32] situations like when the robot put the [20:34] spatula in the oven like you know it [20:36] should probably open it up, take it out [20:37] like it knows that this is not the thing [20:39] you do semantically. That I think is [20:41] like another important step because that [20:42] tells us that we can use common sense to [20:44] fix mistakes. I could imagine um a more [20:48] narrowly scoped robot. Let's say it's a [20:51] humanoid robot. That's one step in [20:54] assembly line in manufacturing. In my [20:57] understanding, you're less interested in [20:59] that because that's not really a step [21:01] towards that general intelligence. What [21:03] I would say, it's not that I'm less [21:05] interested in it. It's that um I think [21:08] that the real world has these leaky [21:10] abstractions that make that kind of [21:12] stuff a lot more complex than it seems. [21:15] Um let me try to explain this with an [21:17] analogy. So in the in the '9s um when [21:21] people were like really started working [21:24] uh kind of full steam on autonomous [21:25] driving, there was this idea that we [21:27] could avoid a lot of the hard problems [21:29] by kind of instrumenting the environment [21:30] a little bit. like we'll have like [21:32] magnetic sensors along the highway and [21:33] so on. And cars will have like a little [21:36] uh you know uh sensor on them and a [21:38] little um transmitter so they can tell [21:40] where each of the cars are. Kind of like [21:42] the way you do it with aircraft [21:43] basically. Uh and then people thought [21:45] well we don't really need like fancy AI. [21:47] We'll just like have these sensors and [21:48] and it'll just work. And that just [21:50] basically didn't go anywhere because the [21:53] real world has so many messy exceptions [21:55] and special cases that even if 99% of [21:57] the time like the magnetic sensors and [21:58] all that stuff just allow the car to [21:59] drive, the 1% when like someone steps in [22:02] the middle of the road or there's like a [22:03] piece of trash or whatever, it just [22:04] messes everything up. So the thing that [22:07] actually worked was when said like, [22:09] "Hey, we're going to not try to avoid [22:11] the hard problem. we're going to [22:12] actually try to deploy our cars not in [22:14] like middle of nowhere but in San [22:16] Francisco like very messy and let's just [22:18] let's just deal with it head-on and that [22:20] allowed the that community to make [22:22] progress and I think robotic [22:24] manipulation is going to be the same way [22:25] that that um past the fully structured [22:29] world of the factory if you want to go [22:30] even a little bit outside of that even [22:32] if 99% of the time it's all like pretty [22:34] straightforward that 1% when something [22:36] weird happens means that you really need [22:38] to the full scope of the problem [22:39] basically to be addressed. This kind of [22:41] reminds me I remember Figure had this [22:43] demo where they live streamed the robot [22:46] kind of sorting packages when you [22:48] watched that demo. Is that a did is does [22:51] it demonstrate generalization in your [22:53] opinion? [22:54] >> Yeah. Yeah, I think it does. And by the [22:57] way, this is something that I find very [22:58] encouraging is that like I mentioned [23:00] that it's hard to show generalization in [23:01] a video. And it's clear that lots of [23:03] people are thinking about that and that [23:04] are thinking about how do you like how [23:07] do you present something that that [23:09] somebody can watch and take in kind of [23:11] at a glance what generalization is. I [23:14] think you can see a lot of creative [23:15] steps towards that. Live demos, these [23:17] kind of like really long time lapses. I [23:19] think it's a great idea and I think [23:20] that's a like a a really nice way to [23:22] move towards elevating the importance of [23:25] generalization in people's kind of [23:27] conscious consciousness. Um you know we [23:30] um when we were working on the um the PI [23:33] star 6 project the RL project that we [23:35] did uh late last year we um wanted to do [23:39] some longer uh horizon experiments. In [23:42] some case it's obvious like we had our [23:44] robot assembling boxes at at Dandelion [23:45] Chocolate Factory. So they're like, you [23:47] know, it's an actual chocolate factory, [23:48] so they need the boxes. So we run it for [23:50] several days. Uh but we had this coffee [23:52] task which was uh the robot was using an [23:54] espresso machine to make espresso. So [23:56] what we did is we ran it for 13 hours [23:58] making espresso drinks. Uh and we are, [24:00] you know, we tried to be like very uh I [24:02] guess um environmentally conscious about [24:04] this. So we didn't want to like throw [24:05] out the coffee. So after 13 hours, [24:06] everyone in the office was like a little [24:08] wiry cuz someone has to like drink the [24:11] coffee. But it like it ran for 13 hours [24:13] and it was pretty cool. But it it [24:14] screwed up a few times. Like it'll um [24:16] spill all the the coffee grounds and [24:18] then it needs to go and get like a cloth [24:19] and wipe it down. But like it does it [24:21] and it's you know nothing exploded 13 [24:23] hours went by. Probably the most [24:25] negative consequence was like loss of [24:27] sleep from too much caffeination. [24:29] >> But when it spills the coffee grounds [24:31] and cleans it up is that's it did that [24:33] by itself. [24:34] >> Well, so the way the way that that [24:36] experiment was done is that there is a [24:38] high level prompting. So like roughly [24:40] the prompt is updated maybe every like [24:42] uh 5 minutes or so like in between [24:44] semantically coherent tasks. So you tell [24:46] it like make espresso, clean up the [24:48] machine etc. So those steps the actual [24:50] like clean up the the machine was [24:52] commanded by a person. Um in principle [24:54] we could automate that. In fact one of [24:55] the things we're spending a lot of [24:57] effort now is improving our high level [24:58] policy that does those commands. But for [25:00] that experiment yeah like every 5 [25:01] minutes somebody basically updates what [25:02] is being asked to do. Like the way we [25:04] intended it was like the commands would [25:06] be like if you go to to an actual coffee [25:07] shop, you say like, "Oh, I want a latte. [25:09] I want an espresso." Like that was [25:10] supposed to be the prompt. Except then [25:12] you also have to tell it, "I want you to [25:13] clean it up before you do the next one." [25:15] [laughter] [25:15] >> That makes sense. Open AAI, Enthropic, [25:19] Cursor, and Versel all use this product [25:22] to make their lives better. And the [25:24] problem it solves is when you're [25:25] building SAS or an AI product and you [25:28] want to sell to other companies, there's [25:30] all these requirements you need to meet. [25:32] There's SSO, there's SKIM, there's [25:35] arbback, there's audit logs. These are [25:37] all things that take time to integrate [25:39] but aren't the main focus of your app. [25:41] Work OS is an API layer that lets you [25:43] meet all of these requirements in just a [25:45] few lines of code. So, let's say you [25:47] have a new SAS product and you want to [25:49] sell to other companies. Work OS will [25:51] solve all of these critical feature gaps [25:53] for you. You can check them out at [25:56] workos.com to learn more and get [25:58] started. and I appreciate them for [26:00] supporting my work and sponsoring this [26:02] podcast. It sounds like on the way to [26:05] generalizing [26:07] uh data is a very important part of that [26:10] and I was reading there's different [26:11] types of data there's simulated data you [26:14] could uh collect like physical [26:16] interactive data and I wanted to hear [26:20] your take on what's the best data to get [26:22] what's the worst and what are the pros [26:24] and cons. Uh this is by the way like a [26:26] question that is uh I guess quite like [26:30] there's a lot of discussion in the [26:31] robotics community about this question [26:32] and some people have like very opposite [26:35] opinions on it. My own take on this is [26:38] that um a lot of different data sources [26:42] are easier for the model to internalize [26:44] if it can ground them in a thorough [26:47] physical understanding of the world. So [26:49] uh let me try to explain what I mean [26:50] with like a few examples. Um, if you [26:54] want to learn to fly an airplane, you [26:56] will probably use a a simulator, at [26:58] least during part of your training. Uh, [27:01] but the simulator makes a lot of sense [27:03] to you because when you start using the [27:05] simulator, you have a lot of world [27:07] knowledge that you can use to ground [27:09] what's going on. Like you know that when [27:11] you uh are using the flight simulator to [27:13] learn how to fly the airplane, you're [27:14] not just like playing a video game. [27:15] You're trying to acquire knowledge that [27:17] will then that you will then use with a [27:18] real airplane. and you understand that [27:20] there's sort of an abstraction there. [27:22] Same thing if you um uh if you're [27:25] playing like a really cartoony like you [27:27] know Atari game or something right like [27:29] you know that all the symbols on the [27:30] screen you can sort of connect them to [27:32] things that you've experienced in your [27:34] life and you can make an analogy there. [27:35] So, a lot of that, like even though it [27:38] kind of seems like these simulated [27:40] environments reflect aspects of the real [27:42] world, to us, they make a lot of sense [27:44] because we kind of bring to bear a lot [27:46] of our own prior experience and we fill [27:48] in the blanks that that the that the [27:50] that the simulation has. Um, and also if [27:53] you if you want to use uh data uh you [27:56] yourself as a person of somebody else [27:58] doing something, if you you watch [28:00] someone uh let's say cooking a meal, [28:01] right? Um, even though you don't [28:04] experience every movement they're [28:05] experiencing, you have a lot of that [28:07] knowledge that you bring to bear and [28:08] you're like, "Okay, I see they're [28:09] picking up the salt shaker. Like, I've [28:10] put salt on things before, so I kind of [28:11] roughly know what's going on there." And [28:13] I can file it away at this level [28:14] abstraction of like add salt without [28:16] having to like figure out all their [28:18] muscle movements. So my point with this [28:20] is that once you have that understanding [28:22] of how you do things physically with [28:23] your own body and how you experience the [28:25] physical world, now all these other [28:27] sources of knowledge can be connected up [28:28] to it because that foundation you get [28:31] from your experience helps you ground [28:33] everything. So where I'm going with this [28:35] is that if we have a robotic foundation [28:37] model that is trained on lots of real [28:40] embodied data that provides that [28:42] grounding, it might actually be much [28:43] better able to absorb other sources of [28:45] knowledge. And this is actually like a [28:47] little bit upside down relative to how [28:49] some people think about it because it's [28:50] very tempting looking at the success of [28:52] like internet data for LLMs to say well [28:55] maybe we should do the opposite. Maybe [28:56] we should like start with like YouTube [28:58] videos and then put robot data on top of [29:00] that. But I think it's actually the [29:01] other way around. And I actually I even [29:03] have like a little bit of evidence for [29:04] this. So my um my colleague uh sir Saraj [29:08] together with um uh Simar from Georgia [29:11] Tech they had a project together uh a [29:13] while back where they took our robot [29:16] foundation model and they added human [29:18] video data but they they didn't start [29:21] with human video data. They actually [29:22] started with a model train on robot data [29:24] and then added video data on top of it. [29:26] And what they did is they looked at the [29:27] representations inside the model. Uh [29:29] basically how does the model represent [29:31] human experience versus robot [29:32] experience? And they found that if you [29:34] use a small model with a small amount of [29:36] robot data, predictably the human [29:38] experience and the robot experience are [29:40] fully separated. Meaning that the [29:41] feature representations are different. [29:43] But if you train on lots of robot data [29:44] from lots of different robots, then the [29:47] features are grouped much more by what [29:49] task is being done rather than by [29:50] whether it's a human or a robot. And [29:52] like when we looked at the feature [29:54] plots, it was just like mindboggling [29:55] because we literally when you when you [29:57] crank up the amount of robot data to [29:58] 100%, they just line up perfectly. like [30:01] you you see the you know you do this TC [30:03] embedding you see the shapes of the [30:04] embeddings and it's just all task [30:07] identity and like minimal sensitivity to [30:09] embodiment and to me that's kind of [30:11] mind-blowing because like this the base [30:13] model wasn't trained on any human data [30:14] at all but once you start adding human [30:16] data it represents it exactly the same [30:18] way and I think that's really exciting [30:20] and I think that that to me is like one [30:22] one of the strongest indicators that if [30:24] you have that good foundation of robot [30:25] experience you can put everything else [30:27] on top of it and it's actually better at [30:28] absorbing that [30:29] >> is it important that that base model has [30:33] data that was collected using that [30:35] specific set of motors, specific set of [30:37] joints. So far, we've uh obviously put a [30:42] lot of effort into cross embodyment [30:43] models that can handle many different [30:44] robot types, but generally you do need [30:47] data of the robot you're going to be [30:48] deploying on to get good performance. Uh [30:51] so kind of the metric of uh of [30:53] generalization there is not can you zero [30:55] shot a new robot but it's mostly can you [30:57] get away with less experience from the [30:59] new robot and transfer skills from other [31:01] robots. Okay so that's the current state [31:03] of things. Now there is a little bit of [31:05] a kind of surprisingly positive read on [31:07] that which is even though you need data [31:09] from these robots um the amount of [31:12] special stuff that the model is doing is [31:15] uh kind of minimal. Like when we started [31:17] doing all of this, I had like a big long [31:19] list of all the cool research I wanted [31:21] to do to better accommodate different [31:22] morphologies like can you like [31:24] factoriize the model's representation in [31:26] some way so that there's like a you know [31:28] a six degree of freedom arm head and a [31:30] seven degree head and a gripper head [31:32] etc. We didn't do any of that like we [31:33] the model just outputs like a big vector [31:35] of numbers. Uh if the robot has less [31:37] degrees of freedom than the number it [31:39] outputs it just zero pads it. There's [31:40] just like nothing fancy. Uh and that's [31:42] it. and then it just train on all the [31:44] robots and outputs the correct actions [31:47] based on what is seeing through the [31:48] camera essentially. [31:49] But now to to your point about whether [31:51] you can uh handle new robots. Um so so [31:56] far the thing that we focused on and I [31:58] think this is showing some promise is to [32:01] be able to transfer skills between [32:03] robots and this is actually where [32:07] the um particular choices in how the [32:10] model works uh seem to matter. Um for [32:13] example uh you can have a model that [32:17] does some intermediate thinking and that [32:20] thinking can be done in different [32:21] modalities. So you can think in text and [32:23] thinking in text is really good for [32:24] transferring um highle behavioral [32:27] structure. So that's basically how you [32:29] understand that hey if I want to like [32:31] clean the kitchen and put away like the [32:33] silverware first open the drawer like [32:34] that's kind of a semantic inference and [32:36] you can transfer that uh very well [32:38] because obviously that's like largely [32:39] agnostic to any embodiment or anything [32:40] like that. But even lower level things [32:42] can be transferred if you use the right [32:44] representation. So one experiment we did [32:46] is we had a a thinking stage that is [32:49] expressed in images where you basically [32:51] like dream up an image of the next [32:54] milestone in the task. And with that, we [32:56] could actually get um a robot, the UR5 [33:00] robot to fold a t-shirt, even though we [33:02] didn't have any t-shirt folding data on [33:03] the UR5 because while getting the the [33:05] the arm motions correct is very hard [33:07] because the robot basically requires [33:09] very different joint angles to do the [33:10] task, uh cooking up an image of what it [33:12] looks like for it to fold a shirt is not [33:14] that hard because like you've seen the [33:15] robot arm in all sorts of different [33:16] poses. You've seen the shirt in all [33:18] different stages of being folded and [33:19] unfolded. You know roughly where it [33:20] should hold it. So getting a good [33:22] generative model to cook up that image [33:24] is pretty straightforward. And once you [33:25] have the image then from that backing [33:28] out the correct actions is easy too [33:30] because you can just look at the [33:31] synthesized arm angle and just like back [33:33] out what the angle should be. So it it's [33:36] it's not changing the problem but it's [33:37] just introducing this intermediate step [33:39] that makes it easier to solve. Just like [33:41] if you're solving a math problem if you [33:42] figure out like the right intermediate [33:44] step kind of the answer is obvious from [33:46] that intermediate step. And I think [33:48] that's really exciting because now now [33:49] that shows that this level of [33:51] generalization across robots and I'm [33:53] sure other generalization too can be [33:54] facilitated with thinking just like an [33:56] LLMs but with a twist. You have to think [33:58] in the right modality. [34:00] >> Interesting. So it it outputs a I guess [34:03] that image is what its video sensor is [34:07] seeing and it's like the next step. [34:09] >> Yeah. You can almost think of it like [34:10] image editing. You can do the same thing [34:12] with video. You can do it with video [34:13] prediction. But the key is to like [34:15] imagine what it would look like to [34:18] progress on this task. Like that seems [34:20] like a very human thing to do, right? [34:21] You know, some things you can you you [34:23] plan semantically and some things you [34:25] plan spatially. Like if you're doing [34:27] rock climbing, you're probably not [34:28] thinking like, hey, left arm to rock 37 [34:31] cm to the left. You're probably more [34:32] like imagining your arm reaching for the [34:34] rock. Earlier in the conversation, you [34:36] mentioned that Whimo was very inspiring [34:40] and um you know their kind of path to [34:43] productionization is proof that you can [34:45] do real world generalized robotics. And [34:49] if I recall correctly when I was, you [34:52] know, a lot younger, it was kind of this [34:54] early promise of this is going to happen [34:58] and then in reality it took a lot [35:01] longer. And so I guess my question is in [35:04] the case of humanoid robotics, [35:07] what would make you say singledigit [35:10] years it's coming versus a long tail and [35:14] you know policy challenges as well. I [35:16] could imagine. [35:17] >> I think one big difference between how [35:21] robotic foundation models address the [35:22] problem and how more traditional [35:24] engineered systems address the problem [35:26] is that the stack is really thin. So [35:28] it's not easy to train a foundation [35:30] model. You need to uh obviously you need [35:32] to get the right data. There's a lot of [35:33] work that goes into curating, labeling, [35:35] all that other kind of stuff. But the [35:36] actual software that runs on the robot [35:40] is very very simple. Uh so uh you know [35:43] you might have some kind of thinking or [35:45] reasoning stage. You might have uh you [35:47] might have the model produce actions. It [35:49] needs to be fast enough. But the like [35:52] just if you think about in terms of raw [35:54] lines of code, it's much much lower than [35:57] a more traditional AV stack. Uh and you [36:01] know, partly that's because modern [36:05] autonomous vehicles, the work on that [36:07] started a lot earlier with very [36:08] different technologies and evolved over [36:10] time. Partly it's also because the [36:12] problem is more safety critical. Like [36:13] yes, you don't want uh a robotic [36:15] manipulator to like drop a fragile [36:17] object, but at the end of the day, [36:19] that's a lot less bad than having a car [36:22] hit somebody. So that is not to say that [36:24] the safety challenges with robots are uh [36:28] are not real. They're very real and it's [36:29] very important to tackle them. In fact, [36:30] it's probably one of the harder ends of [36:32] the problem. But they are not as much of [36:35] a hard stop to practical deployments [36:39] because you can come up with tasks and [36:41] environments and domains and also [36:43] physical hardware where those problems [36:44] are a lot less uh severe. So uh I think [36:48] that that that combination radically [36:50] simpler software stack plus uh less uh [36:55] drastic software challenges I think [36:56] actually make it a lot easier and [36:58] because you know to your earlier point [37:00] uh that there is this kind of flywheel [37:01] effect that there's a positive feedback [37:03] loop that uh you know starting to get [37:05] things out in the world even under some [37:06] constraints will actually facilitate [37:08] getting them out more and more. I think [37:10] a lot of people are familiar with this [37:12] idea of a post-mortem, [37:14] you know, looking back on why something [37:17] failed. But in this case, I'm curious, [37:19] what would you say to a a premortem? And [37:23] in the sense of if humanoid robotics did [37:26] not succeed in single-digit years, what [37:28] do you think would be the most likely [37:30] reason why humanoid robotics failed? [37:34] Ultimately, for these things to to be [37:37] truly useful, they do need to reach a [37:39] level of reliability and robustness and [37:41] generalization that is higher than what [37:43] we typically expect from LLMs for [37:46] example uh or uh you know generative AI [37:48] for like images and video because [37:50] typically like these tools they are very [37:52] much human interactive tools like you [37:54] you get an LLM to do something and it [37:56] doesn't do quite what you want so you [37:58] sort of revise your prompt and you you [37:59] basically like you iterate with it and [38:01] and that's why like even the earlier um [38:03] LM tools like the first version of Chad [38:05] GPT, even though they were much more [38:07] primitive than what we have now, they [38:08] were still already useful because [38:10] somebody could just like keep hammering [38:11] out until it it basically solves their [38:14] problem. You know, just like if if [38:15] you're using a search engine, like you [38:16] type something into the search engine, [38:17] you don't get quite what you want. You [38:18] revise your query and then you get what [38:19] you want. Whereas with a robot, like the [38:22] the full value of it is unlocked when [38:24] it's actually doing the thing [38:25] autonomously. So it's it's having to [38:28] have somebody like constantly iterate [38:29] for every single task is almost like [38:31] antithetical to the benefit that you're [38:32] getting. So I think a lot of the risk [38:35] has to do with how easy is it to get [38:37] that level of reliability and [38:39] robustness. And that's again where some [38:41] of the demos might be like a little bit [38:42] misleading because if someone shows a [38:44] demo of the robot doing something cool I [38:46] mean you know obviously if if everything [38:48] is presented in a forthright way that's [38:50] that could still be very good indicator [38:52] of progress but it doesn't make it [38:54] obvious how far or how close it is to [38:57] reaching that practically relevant level [38:59] of robustness. So, I'm personally a big [39:02] believer in uh using uh techniques like [39:06] reinforcement learning that can actually [39:08] benefit from autonomous experience to [39:10] kind of fine-tune that last few [39:11] percentage points to make it go from [39:13] like 95 to actually 100%. But that's [39:16] really important and it's not yet a [39:17] solved problem. [39:18] >> If it did take longer than expected, [39:21] it's because the bar is higher. because [39:23] the bar is higher and in particular like [39:25] those last few uh the kind of the last [39:27] inch so to speak is something that [39:30] requires not just really good models but [39:32] also new innovations in technology. I [39:35] mean I don't think that you know I'm not [39:38] the kind of person that would say like [39:39] oh we should like throw out everything [39:40] that we know about foundation models and [39:41] start over. I don't think it's that at [39:42] all. I think that roughly the puzzle [39:44] pieces that we have are actually very [39:45] good puzzle pieces. But still we should [39:47] acknowledge that right now the methods [39:50] and the models need more work to cross [39:53] that level of robustness. [39:55] >> In LLMs it feels like everyone is doing [39:58] kind of the same thing but different [40:00] flavors in the robotics industry. Is it [40:04] is everyone doing kind of the same [40:05] thing? Are there any hottake [40:07] architectures that are different [40:09] direction? I actually think that there's [40:10] a lot more heterogeneity than it might [40:12] seem. Um, one big dividing line that I [40:15] think is maybe not as obvious uh from [40:18] just kind of looking at the results is [40:20] the distinction between kind of fully [40:23] embracing the foundation model ethos so [40:25] to speak versus uh focusing on spec [40:29] specific like kind of vertical areas. [40:32] And I think this is like kind of hard to [40:35] tease out sometimes because obviously [40:36] like you know everyone's going to say [40:37] like oh I'm doing the thing that LM did [40:39] like because like that's the cool thing [40:41] but you know the foundational model [40:42] ethos fundamentally is something like [40:45] this that if you have a particular [40:47] problem you want to solve it is better [40:49] to train a more general model that can [40:52] use data from a breadth of problems and [40:54] if you do it right it'll actually be [40:56] better at the specialized problem you [40:57] want to solve than a narrow specialist. [41:00] So you know to again to come back to the [41:01] LM analogy if you want to do machine [41:03] translation don't build a machine [41:05] translation system build a language [41:06] model that understands all language [41:08] tasks and then throw it at machine [41:09] translation [41:10] and in robotics I think that is like [41:12] actually very deeply uncomfortable to [41:14] people because if someone is actually [41:15] like working on an application like [41:17] they're doing like warehouse automation [41:18] it is very awkward to then to to like [41:21] think like oh if I want to do warehouse [41:23] automation let me like collect data of [41:25] like putting away silverware in [41:26] kitchens. It just it just sounds [41:28] bizarre. But that is the foundation [41:31] model lesson that if you have enough [41:32] breadth, if you collect data from a wide [41:34] range of different tasks, then you will [41:36] acquire those generalizable skills and [41:39] if your model is is is built correctly, [41:41] it will repurpose those skills for [41:42] whatever situation it encounters. So I [41:45] think it is actually true that if even [41:46] if you want to build a warehousing [41:48] robot, you're better off collecting a [41:50] breath of data and will be better at [41:51] handling all the weird edge cases you [41:53] might encounter even in that warehouse [41:54] domain. But this is not something that's [41:56] easy for people to accept because it's [41:57] just so like antithetical to the uh [42:00] principle of building kind of a a [42:03] traditional vertically integrated [42:04] robotic system. [42:05] >> I noticed this this uh new interesting [42:09] phenomenon with these AI companies which [42:12] is if they are wildly successful it [42:16] creates this um I guess worry or new set [42:19] of things. So for instance when [42:22] enthropic became had a very powerful [42:24] model then the government comes in and [42:27] there's these you know worries about [42:30] safety and and risk and all that and I'm [42:33] I'm curious how you think about that [42:35] like if if physical intelligence this [42:38] year had a phenomenal incredibly capable [42:42] generalized model how do you think about [42:45] those kinds of topics that might come up [42:48] >> working on AI safety is not a new uh my [42:50] my colleague at UC Berkeley Stuart [42:52] Russell was talking about this stuff [42:53] like over a decade ago and lots of [42:55] people spent a lot of time working on [42:57] it. It's just that the trouble is when [42:59] the technology moves so fast the [43:02] important problems are not just a [43:03] function of like you know the core [43:05] principles it's also a function of how [43:07] society reacts to it what kind of tools [43:09] are adopted and so on um and I think [43:12] that's very very hard to anticipate so [43:15] um I don't have like a very satisfying [43:17] answer here in terms of how we are [43:21] approaching it our philosophy around all [43:23] this stuff is [43:26] basically one of empirical [43:28] experimentation like let's get stuff out [43:30] there. Let's see what happens in the [43:31] real world. Let's see what goes right [43:32] and what goes wrong so that we have as [43:34] much of a preview for you know what the [43:37] technology can do, what are its [43:39] weaknesses, what are its strengths uh [43:42] and and so on. But you know at the end [43:44] of the day you kind of have to just like [43:45] keep your eyes open, see what happens [43:47] and adjust as you go. It's it's very [43:49] hard to anticipate. And you know, I [43:51] think your question though is very [43:52] spoton even though I don't have a great [43:54] answer for you because like yeah, if [43:57] we're having like this much um concern [44:00] and and uh issues with AI systems that [44:04] are basically limited to using [44:06] computers, we're presumably going to [44:08] have strictly more concerns and issues [44:10] with AI systems that can do everything [44:12] in the physical world that we can do, [44:14] right? So the issues are real. It's [44:16] just, you know, you kind of have to like [44:17] see what happens and then adjust. And [44:19] that's kind of in the the scary path. [44:21] But in the in the happy path, if [44:24] everything goes well and we have [44:26] incredibly capable models and uh robots [44:31] in 10 years, is the north star that [44:34] that's the end of human labor. I believe [44:38] it's a mistake to think of robots as [44:41] mechanical people, right? like um you [44:44] know computers at some level are kind of [44:45] like mechanical brains but when personal [44:48] computers like really took off in the [44:50] '9s early 2000s etc. It's not like the [44:54] first thing that happened is that you [44:55] know people replaced their brains with [44:57] computers rather what we saw is actually [44:58] a proliferation of very different kinds [45:00] of computers. We saw kind of like uh [45:02] ubiquitous computing. So you would have [45:04] a computer on your desk but you might [45:05] also have one in your pocket. You might [45:06] have one in your refrigerator and in [45:08] your car like because computing became [45:10] so accessible, you could have a little [45:12] bit of computing in everything. And you [45:14] know, I don't think that that's what [45:15] like the the people that first started [45:19] thinking about this stuff in the 40s and [45:20] 50s would have imagined. They would have [45:21] imagined like, you know, roomsiz [45:22] computers that whose job it is to like [45:25] control the the the you know, the policy [45:28] of an entire country or something rather [45:30] than like a little bit of computer in [45:31] everybody's refrigerator. So I think [45:33] it's it's you know by analogy of that we [45:34] might imagine there might be like a [45:36] little bit of physical actuation in [45:37] everything and it might just be like [45:40] lots of everyday things that you have to [45:42] do yourself now you get like a little [45:44] bit of help with it. I think the other [45:47] example that's worth thinking about is [45:50] um modern coding agents, right? So I I [45:52] think that you know this is something [45:53] where of course the jury is still out as [45:55] to what the endgame of coding agents is, [45:57] but certainly uh from the experience of [46:01] uh software engineers today. Like it [46:03] kind of seems like probably fair to say [46:05] that most would consider coding agents [46:07] to be more empowering them rather than [46:10] like uh you know uh somehow causing them [46:13] to have a panic. I mean, obviously some [46:15] people might might have a panic, but in [46:16] general, at least from the software [46:19] engineers that that I've talked to and [46:20] from my own experience, it's more [46:22] empowering to have to kind of be able to [46:23] amplify how much work you can do uh with [46:27] AI tools. So, I think from that and and [46:29] that maybe is like a pretty direct [46:30] analogy because that is straight up an [46:32] example of an actual real job where AI [46:35] has entered into it and has actually [46:37] provided more leverage to people doing [46:38] that job. So I think that's another [46:40] example that we can look to but you know [46:41] the truth is that I think it remains to [46:43] be seen. [46:44] >> So in in LLM's there's a few seinal [46:47] papers that if you read those papers you [46:50] kind of get a sense of the lineage of [46:51] the breakthroughs that mattered and [46:54] understanding where we are today in the [46:56] robotics industry. Are there a set of [47:00] top papers that you really think kind of [47:03] show the breakthroughs that people [47:05] should know about if they're curious [47:07] about the state-of-the-art in terms of [47:09] uh humanoid robotics? One thing I would [47:12] point out um uh and this is like partly [47:15] a shameless plug because I am a [47:16] co-author on that paper though candidly [47:18] like uh like 99.9% of the work on this [47:21] was done by by um Tony who was lead [47:23] author is the uh the original ACT paper [47:26] the Aloha paper. It's kind it's an [47:28] interesting example because in some ways [47:31] the ideas weren't really that new but [47:33] they were illustrated in a really nice [47:34] way and the idea was that hey um if you [47:38] set up the right kind of lowcost robot [47:40] setup in in his case it was based on um [47:44] these robot arms from Trusson Robotics [47:46] that they're like $7,000 hobbyist arms. [47:48] He set them up in a banual setup with a [47:51] leader follower to the operation device. [47:53] And he showed that actually if you do it [47:55] right without really any particularly [47:56] fancy tricks, you could easily collect [47:59] the operation data of extremely dextrous [48:01] tasks that people had previously thought [48:02] would require like very sophisticated [48:04] hardware and all sorts of like really [48:05] expensive stuff and then set up like a [48:08] fairly [48:09] straightforward transformer-based model [48:11] and it could actually do a lot of those [48:12] tasks. And it's kind of like an [48:14] interesting thing because usually in [48:16] academic research we put a big premium [48:18] on like you know do you have some like [48:19] sophisticated new mathematical thing or [48:21] some sophisticated like uh technical [48:23] insight and in that paper which I think [48:26] at this point has been hugely [48:27] influential. The insight is really just [48:29] like yeah just put together the right [48:31] pieces and have a little bit more faith [48:35] in what a simple robot could do so to [48:37] speak equipped with a good intent [48:38] learning system. and he showed like [48:40] things like um uh replacing batteries in [48:43] a remote control. Uh he even had he even [48:45] got like a little like a mannequin foot [48:48] and he showed that you could put a shoe [48:49] on it like for like an assistive task [48:51] sort of like you know some some people [48:52] need help getting their shoes on so [48:53] that's good. Um but like what what [48:56] people find found I think so interesting [48:58] about that paper is just how far you [49:00] could get with like relatively simple [49:02] building blocks. And at this point the [49:04] the a like he he open sourced the code [49:06] for it and the act code has been used by [49:08] like lots of people sort of like if [49:09] someone wants a very basic starter kit [49:13] for like robotic learning that's usually [49:14] what they grab and I think that it's [49:18] worth for somebody who wants to get into [49:20] the field to go through that paper and [49:22] really understand what's going on there [49:24] because even though in some ways it's [49:26] not that sophisticated [49:28] I think it provides like a bit of [49:29] calibration on what matters right like [49:31] you know the details matter But the [49:33] details don't have to be complicated. [49:35] >> Before all this uh large model robotics [49:39] kind of uh wave prior to that um Boston [49:43] Dynamics had these really impressive [49:46] demonstrations and tons of um mind [49:50] share. I guess I wasn't even in the [49:52] field by saying wow they're really doing [49:55] incredible robotics. And then in the [49:58] last I don't know how many years I don't [50:00] really hear about them much anymore. Um [50:04] is there some shift in the industry that [50:08] made that so or you know is that [50:10] something you could explain? [50:12] >> So the way I would explain it is this [50:13] that um there are you know robotics [50:18] at at some level is about building [50:19] complex systems. So um even though it [50:23] kind of it's very tempting to say like [50:25] oh there's different areas of AI there's [50:27] like LLMs and vision and robotics one of [50:30] those is not like the others because for [50:31] robots you actually need like all the [50:32] parts everything from like how you you [50:36] know uh how you wire up the robot what [50:38] the power source is what does the [50:39] actuator look like all the way to how [50:42] does it do like high level planning to [50:43] determine like what task to do next and [50:47] even though it's kind even though we [50:49] could look at these things and say like [50:50] oh all of these different videos and [50:52] different companies and different demos. [50:53] They're all robotics. They're really [50:55] kind of a different parts of the stack. [50:57] Um, [50:58] a lot of what the classic Boston [51:01] Dynamics results show is um, very [51:04] sophisticated hardware, very carefully [51:06] designed hardware with a uh, traditional [51:10] control approach with very smart [51:12] controls engineers setting everything [51:14] up, but with comparatively less emphasis [51:17] on the kind of decision-m aspect and I [51:20] think that there, you know, at a [51:21] particular point in time that actually [51:22] made a lot of sense because we can't [51:24] build the physical body like it doesn't [51:26] matter what kind of decision making [51:27] system is running on it. Um [51:30] but uh to our earlier discussion about [51:32] generalization, [51:34] you know, at this point we're at a stage [51:36] in the development of these things that [51:39] even though we can do more on hardware, [51:42] in many ways it's good enough. And the [51:44] big challenge is how to have the [51:45] decision-making loop that actually works [51:47] and that reacts intelligently to [51:50] everything in the environment. And the [51:52] way and the place where I would draw the [51:53] the dividing line between those is [51:56] it's like decision-m loop doesn't mean [51:58] symbolic decisions. It could mean [52:00] low-level decisions. The question is do [52:02] you need to take the rest of the [52:03] environment into account or are you just [52:05] dealing with the robot? So if you want [52:06] to do a backflip on flat ground, you [52:08] most have to deal with the robot. But if [52:10] you want to pick up a coffee cup off of [52:12] a table, even though that's maybe in [52:14] some ways simpler than doing a backflip, [52:16] you really have to understand what's [52:17] going on in the rest of the world rather [52:18] than just your own body. And that [52:21] dividing line, I think the way the [52:23] technology has panned out, I think it's [52:25] fair to say that that is the dividing [52:26] line between AI and controls. Like [52:29] controls is when you have to control the [52:32] robot body. AI is when you have to take [52:34] into account what goes on outside of the [52:36] robot. [52:37] and and and I think that that's why you [52:39] see this divide because I think a lot of [52:41] the demos where you mostly needed to [52:44] deal with the robot itself and not the [52:45] rest of the world really good controls [52:48] could allow you to could admit a very [52:49] good solution there a and you know [52:51] another thing I would say here is like [52:53] okay if there's a lot of controls work [52:55] that goes into doing some particular [52:58] skill well there is actually something [53:00] to learn from that because if you can [53:03] handdesign a controller that performs a [53:05] sophisticated behavior very likely you [53:07] can also learn that controller. So like [53:09] just that proof of existence that the [53:11] thing is possible and not only possible [53:13] but also simple enough that a person [53:15] could build it because remember people [53:16] are you know at the end of the day even [53:18] even with code these days the kind of [53:20] complexity that people can handle is not [53:22] as high as the kind of complexity the AI [53:23] can handle. So if a person can [53:24] handdesign something uh to do a backflip [53:27] or do some acrobatics that's a really [53:29] great proof of existence that there [53:31] exists some relatively parsimmonious [53:32] control law for doing that skill and [53:34] parsimonious does mean generalizable. So [53:36] if it's simple enough for a person to [53:38] design, probably there's something [53:39] fairly general in there and if you can [53:41] learn it uh and automate it without [53:43] having to have the human controls [53:44] engineers in the loop, that's that's [53:46] good news. [53:48] >> And then last question for you is you if [53:50] you could go back to when you just [53:52] entered the industry and give yourself [53:53] some advice knowing everything you know [53:55] now, what would you say? [53:57] One thing that I've learned uh over the [53:59] last uh few years um which I think is a [54:03] little different than kind of my [54:04] original mindset is I think that [54:06] addressing robotics effectively requires [54:09] using um very broad prior knowledge. And [54:14] I think that there there's this idea [54:15] that a lot of people in robotic learning [54:18] have which I think I I shared initially [54:19] that you know since people learn things [54:22] kind of from scratch maybe robots should [54:24] learn things from scratch too. Um so uh [54:27] like for example uh in um some of our [54:30] early work on large scale robotic [54:31] learning at Google we uh we had this uh [54:34] what we call the the arm farm project [54:36] like we set up a bunch of robot arms in [54:38] a in a conference room actually because [54:41] we didn't have a proper lab but it was a [54:43] conference room and we had them all like [54:44] grasping objects and the idea was well [54:46] if they grasp like millions of objects [54:48] they'll learn very general like grasping [54:50] strategies and it basically worked like [54:52] they could learn to grasp objects but it [54:54] was very hard to like take get further [54:56] from that to the next level. So, okay, [54:57] now I can pick up anything, but like so [54:58] what? Uh can like it it didn't serve as [55:01] a very good stepping stone for more [55:02] complex skills. And I think part of that [55:04] was that we were approaching this like [55:06] very blank slate like let's start from [55:08] zero and see if knowing nothing in [55:10] advance the robot could start picking up [55:11] behaviors. But I think that it's much [55:14] much more practical to get all this to [55:16] work if you can combine robot experience [55:18] with knowledge that you can pull in from [55:20] other sources. Like for example, I was [55:21] very skeptical initially about the [55:23] utility of language and I think [55:26] scientifically this is defensible which [55:28] is that like hey um you know like [55:31] animals can do some pretty impressive [55:32] things like monkeys can do really cool [55:34] stuff but monkeys as far as I know can't [55:36] speak at least not very eloquently um so [55:39] maybe our robots should also be able to [55:40] do stuff and they don't necessarily need [55:42] to understand language but I think the [55:44] subtlety there is what's important is [55:46] not language it's it's [55:49] prior knowledge that you can put in as a [55:51] scaffold on your learning process. And [55:54] you can pull in that knowledge in all [55:55] sorts of ways. Like, you know, humans [55:56] don't necessarily pull that in entirely [55:58] through language. Humans all and monkeys [56:00] certainly don't. Uh they do it from [56:02] observation, from observing other [56:03] people, other creatures and so on. So, [56:05] there's lots of sources of prior [56:06] knowledge. But the point is that you got [56:07] to get that prior knowledge in there. [56:08] Otherwise, you're actually faced with a [56:09] harder problem than what humans and [56:11] animals have to solve. Because like, you [56:13] know, if a person had to figure out how [56:15] to like assemble IKEA furniture, but [56:17] they've never actually encountered any [56:19] article of furniture in their entire [56:20] life, like [laughter] [56:22] okay, that would be like pretty [56:23] difficult because they don't even know [56:24] what like what the point of this is or [56:26] what the the endgame looks like. Uh so [56:28] yeah, prior knowledge is important. And [56:30] while I'm still a big fan of learning [56:32] things through experience, I think that, [56:34] you know, my advice to myself would have [56:35] been take prior knowledge more [56:37] seriously. [56:38] >> Awesome. Well, thank you so much for [56:39] your time, Sergey. I really appreciate [56:40] it. [56:41] >> Yeah, thank you for your questions. Hey, [56:42] thank you for watching this podcast. If [56:44] you liked it and you want to see the [56:45] show grow, please support with a comment [56:47] or a like. Also, if you have any [56:50] recommendations for people you want me [56:52] to bring on, please drop a comment. [56:54] Guests like Barbara Liskov, Mike [56:56] Stonereaker, Mark Brooker, these were [56:59] all people that I brought on because [57:01] someone left a comment. On another note, [57:03] aside from the podcast, I'm working on [57:05] building the ergonomic keyboard that I [57:07] wish existed. Here's a glance at the [57:09] prototype. It's a split keyboard, so [57:11] there's two sides. This is in the case, [57:14] but yeah, we launched on Kickstarter and [57:16] we hit our goal within eight hours of [57:18] launching. I really appreciate it if you [57:20] were one of the people who grabbed one [57:21] of the early units. Um, we're now [57:23] working on the long journey of building [57:25] the tooling now. And so, if you still [57:27] want to pick one up, I've left the late [57:29] pledges open on Kickstarter, so you can [57:31] grab one there. I'll put a link in the [57:33] description. Thank you again for [57:35] watching the podcast and I'll see you in [57:37] the next