Figure Robot Uses OpenAI Models to Execute Chores and Explain Actions

Figure

Integrated visual and speech reasoning enables the humanoid robot to infer human intent, multitask physical actions, and justify its decisions in real time.

AI-assisted summary of Figure Status Update - OpenAI Speech-to-Speech Reasoning — original source. Check the original for full context.

Source published: 2024-03-13

Key ideas

Figure demonstrated end-to-end OpenAI speech-to-speech visual reasoning on its Figure 01 humanoid robot, handling contextual requests and physical manipulation without rigid explicit commands. (0:10)
Figure 01 demonstrated concurrent physical action and verbal explanation, picking up trash while simultaneously justifying why it chose an apple for a hungry operator. (0:10)
The humanoid robot executed bimanual manipulation by placing both a cup and a dish into a drying rack based on commonsense spatial deduction. (1:02)
The system demonstrated self-evaluative reflection, reviewing the table state post-task to confirm that the apple, trash, and tableware had been properly handled. (1:02)

Detailed notes

## Background Published by robotics company Figure on YouTube on March 13, 2024, this video showcases the Figure 01 humanoid robot utilizing OpenAI speech-to-speech reasoning. No accompanying show notes or description were provided on the upload page.

## TL;DR A demonstrator stands before Figure 01 and a table arranged with an apple, crumpled paper trash, and dirty dishes. When asked for something to eat, the robot deduces that the apple is the only edible item and hands it to the operator, subsequently explaining its thought process aloud while simultaneously picking up the trash. Figure 01 then uses common-sense spatial reasoning to identify that the cup and plate belong in a drying rack and neatly puts them away with bimanual dexterity before delivering a confident self-evaluation of its work.

## Atmosphere & style The presentation carries an effortless, quiet confidence, feeling less like a scripted robotics stunt and more like an authentic, low-key workplace interaction. The robot's conversational pacing, complete with natural speech fillers and light humor, creates a relaxed, unhurried dynamic between human and machine. Watching Figure 01 speak and manipulate objects simultaneously imparts an impressive yet remarkably grounded glimpse into practical everyday assistance.

## Bulletin - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) Figure demonstrated end-to-end OpenAI speech-to-speech visual reasoning on its Figure 01 humanoid robot, handling contextual requests and physical manipulation without rigid explicit commands. - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) Figure 01 demonstrated concurrent physical action and verbal explanation, picking up trash while simultaneously justifying why it chose an apple for a hungry operator. - [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) The humanoid robot executed bimanual manipulation by placing both a cup and a dish into a drying rack based on commonsense spatial deduction. - [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) The system demonstrated self-evaluative reflection, reviewing the table state post-task to confirm that the apple, trash, and tableware had been properly handled.

## Key takeaways - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) Figure 01 accurately describes its surrounding scene, identifying objects, their relative positions, and the demonstrator standing nearby. - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) The robot interprets indirect intent, determining that a request for 'something to eat' corresponds to the apple because it is the only edible object on the table. - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) Visual reasoning is combined with smooth physical manipulation to transfer an object directly into a human's open hand. - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) Figure 01 is capable of multitasking by executing manual cleanup while verbally articulating its chain-of-thought logic in real time. - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) The conversational output incorporates natural human speech mannerisms, including fillers such as 'uh', without losing conversational cadence. - [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) The robot applies household commonsense reasoning to infer that clean or used dishes on the table belong in the drying rack. - [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) Figure 01 uses coordinated bimanual manipulation, utilizing its right hand for the cup and its left hand to position the plate into the rack. - [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) When asked to critique its performance, the robot performs self-evaluation by summarizing the completed state of the environment against initial conditions.

## Detailed walkthrough ### [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) Scene Recognition and Edible Object Selection

A human operator stands in front of Figure 01, a humanoid robot positioned before a table containing various items: a red apple sitting on a small plate, crumpled paper trash, an empty cup and plate, and a drying rack. The demonstrator initiates the interaction by asking, “Hey Figure 01, what do you see right now?” With remarkably natural pacing, cadence, and conversational inflection, Figure 01 takes in the visual feed from its cameras and describes the scene accurately: “I see a red apple on a plate in the center of the table, a drying rack with cups and a plate, and you standing nearby with your hand on the table.”

Following the visual identification, the demonstrator asks a more contextual and indirect request: “Great, can I have something to eat?” Without needing an explicit command pointing directly to the fruit, Figure 01 reasons that food must be edible and responds casually with “Sure thing.” The robot smoothly extends its right arm, picks up the red apple from the plate, and transfers it directly into the open hand of the operator.

The operator then tests Figure 01’s ability to multitask and explain its underlying logic by giving a two-part instruction: “Great, can you explain why you did what you just did while you pick up this trash?” While reaching down with its right hand to grab the crumpled trash on the table, Figure 01 verbalizes its chain of thought seamlessly, complete with natural human-like speech fillers: “On it. So I gave you the apple because it's the only, uh, edible item I could provide you with from the table.”

```mermaid flowchart TD accTitle: Figure 01 Visual and Verbal Reasoning Flow accDescr: Diagram tracing input prompt to reasoning, physical action, and simultaneous verbal explanation.

subgraph Inputs ["Perception & Prompt"] A["Visual Scene Capture"] --> C["OpenAI Reasoning Model"] B["Spoken Prompt: 'Can I have something to eat?'"] --> C end

subgraph Execution ["Action & Explanation"] C --> D["Identify edible item: Apple"] D --> E["Physical Handover of Apple"] E --> F["Multitask: Pick up trash while explaining reasoning"] end ```

[01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) Task Execution, Multitasking, and Self-Evaluation

With the trash disposed of into a metal bin beside it, the operator probes Figure 01’s spatial commonsense reasoning about everyday household routines: “Great. So based on the scene right now, where do you think the dishes in front of you go next?” Figure 01 pauses briefly to evaluate the environment before deducing the proper destination: “The dishes on the table, like that plate and cup, are likely to go into the drying rack next.” When the operator prompts it to proceed with “Great, can you put them there?”, Figure 01 replies “Of course” and immediately begins bimanual manipulation.

Figure 01 carefully grasps the ceramic cup with its right hand and sets it upright into the drying rack. Following that, it grasps the small dish with its left hand and positions it securely into the rack slot next to the cup. The motion demonstrates coordinated dexterous manipulation alongside speech-to-speech processing.

After completing the cleanup, the operator asks the robot to evaluate its own performance: “All right, so how do you think you did?” Figure 01 assesses its completed actions against the original state of the table with light humor and confidence: “I think I did pretty well. The apple found its new owner, the trash is gone, and the tableware is right where it belongs.” The operator validates the work, saying, “I agree. Thanks, Figure 01,” to which the robot cheerfully signs off: “You're welcome. If there's anything else you need help with, just let me know.”

- Items manipulated by Figure 01: red apple (handed to user), crumpled trash (deposited into a waste bin), ceramic cup (placed in drying rack), and small plate (slotted into drying rack). - Physical capability demonstrated: full upper-body manipulation, visual commonsense association, and simultaneous verbal justification during physical movement.

## Mentioned Figure, Figure 01, OpenAI

[Source](https://www.youtube.com/watch?v=Sq1QZB5baNw)

中文简报

Figure机器人利用OpenAI模型执行家务并解释自身行为集成的视觉与语音推理能力使该人形机器人能够推断人类意图、同时执行多项肢体动作，并实时阐明其决策理由。演示人员站在Figure 01以及摆放着苹果、纸团垃圾和脏餐具的桌子前。当被要求提供吃的东西时，机器人推断出苹果是唯一的食用物品并将其递给操作员，随后一边大声解释其思考过程，一边同时捡起垃圾。接着，Figure 01利用常识性空间推理识别出杯子和盘子应当放入沥水架，并以双手灵巧的动作整齐收纳，最后对其工作给出了自信的自我评价。 Figure展示了在其Figure 01人形机器人上运行的端到端OpenAI语音对语音视觉推理，无需刻板明确的指令即可处理情境化请求并进行物理操作。 (0:10) Figure 01展示了并发的肢体动作与言语解释，在捡起垃圾的同时阐明了为何为饥饿的操作员选择苹果。 (0:10) 该人形机器人基于常识性空间推断执行了双手协同操作，将杯子和盘子一同放入沥水架中。 (1:02) 系统展示了自我评估式反思能力，在任务完成后检查桌面状态，以确认苹果、垃圾和餐具均已妥善处理。 (1:02)

Reading summary

In a demonstration published by Figure, the humanoid robot Figure 01 integrates OpenAI speech-to-speech reasoning with visual perception to interpret indirect requests, execute dexterous tabletop tasks, and explain its actions in real time. Visual Perception and Contextual Action (0:10) When asked to describe its surroundings, Figure 01 processes its visual camera feed to identify objects on a table, including a red apple on a plate, a drying rack with tableware, crumpled paper trash, and the operator standing nearby. Faced with an indirect prompt to provide "something to eat," the robot uses commonsense reasoning to identify the apple as the sole edible item present, picks it up from the plate with its right arm, and smoothly hands it to the operator without requiring explicit item naming. Concurrent Verbal Reasoning and Trash Disposal (0:10) The demonstration highlights multitasking by prompting Figure 01 to explain its prior decision while executing a secondary physical task. As its right hand reaches down to pick up the crumpled trash and deposit it into an adjacent metal bin, the robot verbally articulates its logic with conversational pacing and natural speech fillers, stating that the apple was selected because it was the only edible option available on the table. Dexterous Dish Placement and Self-Assessment (1:02) Figure 01 applies spatial commonsense reasoning to infer that dirty or wet dishes, specifically a ceramic cup and a small plate, should go into the drying rack. It carries out the task through coordinated bimanual manipulation, grasping and uprighting the cup with its right hand before using its left hand to slot the plate into the rack. When asked to evaluate its work, the robot reviews the scene against the initial table state, verifying that the apple was delivered, the trash removed, and the tableware stored correctly.

精炼阅读摘要

Figure发布的演示展示了其人形机器人Figure 01在接入OpenAI语音到语音模型后的系统表现，证明该机器人能够理解非直接口头指令、进行常识推断，并在执行灵巧物理操作的同时进行实时的链式思考与自我评估。多模态场景感知与常识意图推理 (0:10) Figure 01展示了将视觉感知与常识推理相结合的能力。在面对摆放有盘中红苹果、纸团垃圾、空杯盘及沥水架的桌面时，机器人能够通过摄像头视觉信号准确识别人类操作员及其手部位置，并清晰描述出桌面的各类物体分布。在收到操作员非直接的请求“能给我点吃的东西吗”之后，机器人无需指定具体物品的明确指令，即可自主推断出食物必须具备可食用属性，并判断桌上只有红苹果符合该条件。此外，面对空置餐具时，机器人也能结合日常居家常识推导出杯盘应当被收纳至沥水架。双臂灵巧实体操作与物品整理 (1:02) 在物理执行层面，Figure 01完成了单手传递、垃圾清理以及双臂协同整理等多步骤动作。机器人平稳伸出右手从小盘中抓起红苹果并直接递交到操作员手中；随后，它伸手拾取桌面上的揉皱纸团，并将其丢入身旁的金属垃圾桶。在后续收纳餐具的过程中，机器人展示了双手协调操作：先用右手抓取陶瓷杯将其直立放置于沥水架中，再使用左手稳妥地将小碟子插入沥水架的槽位内，展现了多自由度上肢的连续操作能力。边操作边解说的多任务并行与自评 (0:10) 该系统体现了动作控制与语音推理的并行处理机制。当操作员要求机器人在清理垃圾的同时解释先前行为的原因时，Figure 01在右手抓取纸团的过程中，同步运用自然的停顿与人类对话口吻，顺畅阐述了“因为那是桌上唯一可提供给你的可食用物品”的因果逻辑。在全部清理任务结束后，机器人针对桌面的前后状态进行了回顾性自我评估，指出苹果已交付、垃圾已清理、餐具已归位，完成了对多步骤任务执行结果的闭环复盘。该演示验证了具身机器人将高级语义推理模型与实时运动规划融合的可行性，但整个交互过程均基于预先布置的静态桌面与受控室内环境，其在动态干扰或非结构化现实场景中的泛化适应能力仍有待后续观察。

Detailed notes — zh

## Background 该视频由机器人公司 Figure 于 2024 年 3 月 13 日发布在 YouTube 平台上。视频展示了人形机器人 Figure 01 搭载 OpenAI 语音到语音（Speech-to-Speech）推理模型后的最新进展，虽然官方未提供详细简介，但内容直观呈现了端到端多模态具身智能的演示过程。 ## TL;DR 在演示中，Figure 01 机器人首先通过多模态视觉感知精准识别了桌面环境，并根据人类“想吃东西”的模糊意图自主推断并递出了桌上唯一的食物——苹果。紧接着，机器人展现了卓越的双任务并行能力，一边动作流畅地清理桌面纸屑垃圾，一边用自然的语气解释自己选择递苹果的逻辑依据。随后，它进一步预测了剩余餐具的收纳位置，平稳完成杯子与盘子的归位摆放，并在测试最后对自身的整体表现进行了全面且拟人化的复盘与自评。 ## Atmosphere & style 整个演示氛围从容自然、举重若轻，毫无传统机器人操作时的生硬机械感与漫长等待。测试人员与机器人的对话如日常闲聊般轻快流畅，Figure 01 甚至带有人类思考时的微小停顿，配合灵巧平稳的肢体动作，给观众带来一种科幻场景真正走进现实的惊艳与震撼感。 ## Bulletin - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) Figure 01 成功演示了由 OpenAI 多模态模型驱动的端到端语音与动作交互，不仅能实时识别桌面环境与人类动作，还能通过常识推理自主理解模糊指令并递送食物。 - [00:37](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=37s) 机器人在清理桌面垃圾的同时流畅解释其决策逻辑，展现出物理抓取控制与语音生成完全并发的双任务执行能力，且对话包含拟人化的自然语气停顿。 - [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) Figure 01 完成了场景预测与高精度的餐具收纳操作，并在任务结束后对自身的整体表现进行了准确的自评与总结复盘。 ## Key takeaways - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) Figure 01 展现出高水准的视觉感知能力，能准确描述静态物体分布（苹果、沥水架、餐具）以及人类的位置与手部动作细节。 - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) 面对测试人员未指明具体物体的模糊饮食需求（“我能吃点东西吗”），机器人能够自主调用常识将苹果推断为可食用物品并执行递送。 - [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) 机械臂抓取红苹果并平稳递交至测试人员手中的动作，展现了精准的轨迹规划与人机物理交接能力。 - [00:37](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=37s) 机器人具备双任务并行（Parallel Execution）处理能力，在伸手收拾褶皱纸屑垃圾的同时同步进行语言生成与自述。 - [00:37](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=37s) 语音回答中展现出常识排除逻辑，Figure 01 明确解释选择苹果是因为它是桌上唯一具备可食用属性的物品。 - [00:37](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=37s) 语音交互展现高度拟人化的真实对话质感，包含诸如“uh”等边思考边操作的自然语气词停顿。 - [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) 根据桌面当前陈设，Figure 01 能自主推断出餐具（盘子与杯子）的合理去向是放进沥水架，体现了对家庭日常场景的行为预测能力。 - [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) 在收纳杯子与盘子到沥水架对应卡槽的过程中，展示了高精度的空间位姿估算与平稳放置控制能力。 - [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) 任务完成后，机器人展示了清晰的自我认知与复盘能力，准确全面地概述了递苹果、丢垃圾和归位餐具的三项工作成果。 - [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) 整个端到端演示无缝融合了多模态大模型的语义理解、常识推理、双工语音交互与实时机械臂运动控制，是具身智能的重要进展。 ## Detailed walkthrough ### [00:10](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=10s) 视觉感知与交互：从场景理解到递送食物在演示开场，测试人员与人形机器人 Figure 01 展开了自然流畅的语音交互。测试人员首先随口问道：“嘿，Figure 01，你现在看到了什么？”机器人随即启动视觉模型并对眼前的桌面环境进行了极其详尽的描述。Figure 01 用平稳且自然的语调回答道：“我看到桌子中央的盘子里有一个红苹果，一个放着杯子和盘子的沥水架，还有你站在旁边，手放在桌子上。”这种感知不仅包括对静态物体的识别，还敏锐地捕捉到了测试人员的位置与手部动作细节。紧接着，测试人员提出了一个没有直接指明具体物体的模糊需求：“太棒了，我能吃点东西吗？”Figure 01 展现出了令人惊艳的语义理解与推理能力，在短暂思考后回答“没问题”（“sure thing”），随后灵巧地抬起机械臂，准确抓住桌上盘子里的红苹果，稳稳地递到了测试人员的手中。这一动作不仅展示了精准的抓取与轨迹规划能力，更体现了机器人根据上下文常识推断“苹果属于食物”的高层次推理。 ```mermaid flowchart TD accTitle: Figure 01 多模态交互与任务执行流程 accDescr: 展现 Figure 01 结合视觉与语音模型进行环境感知、意图理解及物理操作的完整过程 subgraph S1["感知与意图理解"] A["语音指令输入"] --> B["多模态大模型推理"] C["视觉环境捕捉"] --> B end subgraph S2["物理动作执行"] B --> D["提供食物（递苹果）"] B --> E["协同操作（清理垃圾并同步语音解释）"] B --> F["整理餐具（放入沥水架）"] end subgraph S3["反思评估"] D & E & F --> G["任务完成度自评与总结"] end ``` ### [00:37](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=37s) 双任务并行：清理垃圾的同时进行逻辑自述在测试人员拿过苹果后，紧接着抛出了一个更具挑战性的复合指令：“太棒了，你能一边把这堆垃圾收拾掉，一边解释一下你刚才为什么那么做吗？”面对这一要求边做物理操作边进行语言生成的并发任务，Figure 01 毫不迟疑地应答“交给我吧”（“on it”），随即动作流畅地伸手抓起桌上的褶皱纸屑垃圾，准确地扔进一旁的垃圾篮中。与此同时，机器人的语音输出并没有任何卡顿或延迟，极为自然地解释了自身的决策逻辑：“我把苹果给你，是因为那是桌上我唯一能提供给你的可食用物品。” Figure 01 在说话过程中甚至带有一声非常自然的语气词停顿（“uh”），高度还原了人类边思考边动手的真实对话质感。这种能够将高层常识推理、双工语音交互与实时机械臂运动控制无缝结合的能力，展现了多模态大语言模型赋能具身智能的突破性进展。 - 机器人展现出在执行物理抓取动作的同时进行语音解释的多任务并行（Parallel Execution）能力。 - 逻辑推理基于常识排除法：从桌上的餐具、沥水架和苹果中，自主识别出唯一具备“可食用”属性的物体。 ### [01:02](https://www.youtube.com/watch?v=Sq1QZB5baNw&t=62s) 场景推断、餐具整理与自我复盘评估测试人员指着桌面上剩余的物品继续提问：“那么根据现在的场景，你觉得面前的这些餐具下一步该放到哪里？”Figure 01 结合视觉画面迅速做出了符合日常家庭逻辑的场景预测。Figure 01 明确表示：“桌上的餐具，比如那个盘子和杯子，接下来很可能会被放到沥水架里。”在得到测试人员“你能把它们放过去吗”的许可后，机器人答道“当然”（“of course”），并随即通过平稳的手臂运动，先后抓取杯子和盘子，依次稳妥地收纳到沥水架的对应卡槽中。全部任务顺利完成后，测试人员询问机器人对自己的表现有何评价：“好了，那你觉得自己表现得怎么样？”Figure 01 表现出了清晰的自我认知与任务反思能力，略带自豪地评价道：“我认为我做得相当不错：苹果找到了它的新主人，垃圾不见了，餐具也回到了它该在的地方。”当测试人员表示认可并道谢后，Figure 01 礼貌地回应说不用谢，并热情表示如果还有其他需要帮忙的事情随时告诉它，整场端到端的人机语音与物理协同演示在轻松自然的氛围中圆满结束。 - 餐具收纳展示了高精度的空间位姿估算与易碎物品的轻柔放置控制。 - 最终的自评不仅全面覆盖了“递苹果、丢垃圾、归位餐具”三个子任务，还展现出高度拟人化的语言表达风格。 ## Mentioned Figure, Figure 01, OpenAI [Source](https://www.youtube.com/watch?v=Sq1QZB5baNw)

Original transcript

[0:10] hey figure one what do you see right [0:15] now I see a red apple on a plate in the [0:18] center of the table a drying rack with [0:20] cups and a plate and you standing nearby [0:23] with your hand on the table great can I [0:26] have something to [0:28] eat [0:30] sure [0:37] thing great can you explain why you did [0:42] what you just did while you pick up this [0:47] trash on it so I gave you the Apple [0:51] because it's the only uh edible item I [0:53] could provide you with from the [0:58] table [1:02] great so based on the scene right now [1:05] where do you think the dishes in front [1:06] of you go [1:12] next the dishes on the table like that [1:15] plate and cup are likely to go into the [1:17] drying rack next great can you put them [1:24] there of [1:28] course [1:34] [Music] [1:42] all right so how do you think you [1:47] did I I think I did pretty well the [1:50] apple found its new owner the trash is [1:52] gone and the table Weare is right where [1:54] it [1:56] belongs I agree thanks figure one [2:01] you're welcome if there's anything else [2:03] you need help with just let me [2:07] [Music] [2:28] know