AI Coding Models Upend Robotics as Founders See Human Parity in Two Years

Robot-Use Agents: Why General-Purpose Models May Win in Robotics — Y Combinator

Y Combinator

General-purpose foundation models bypass scarce physical training data by translating robotic control into software code, setting up human-level manipulation within 24 months.

AI-assisted summary of Robot-Use Agents: Why General-Purpose Models May Win in Robotics — original source. Check the original for full context.

Source published:

Key ideas

  • Frontier researchers and startups like Waddle Labs and RoboCurve are demonstrating that general-purpose coding agents can control diverse robot embodiments out of the box with little to no domain-specific retraining. (0:00)
  • By shifting robot control from direct low-level token generation to code as a Chain of Thought, physical manipulation bypasses the severe teleoperation data bottleneck by piggybacking on web-scale code and tool-use datasets. (1:52)
  • Spatial intelligence breakthroughs in frontier models stem largely from pre-training on computer-use traces and CAD software, allowing GUI interactions to supply the geometric priors needed for physical world manipulation. (23:01)
  • Industry consensus across frontier labs projects general-purpose manipulation models capable of human-teenager parity within roughly two years, relying on offline 'sleep' consolidation loops to distill slow in-context reasoning into fast local policies. (26:15)

Detailed notes

## Background Published on September 26, 2026, by Y Combinator on their podcast Decoded via RSS and web feeds, this episode examines why general-purpose foundation models are emerging as the leading approach to robotics. The episode features the founders of Waddle Labs and RoboCurve discussing the research behind 'robot-use agents'—a paradigm popularized by MIT professor Philip Isola where coding and vision-language models control diverse robotic embodiments with minimal robot-specific training.

## TL;DR The discussion explores how modern coding agents bypass the physical data bottleneck in robotics by framing real-world manipulation as program induction and structured tool use. Drawing comparisons to RT-2 and the Chain-of-Thought breakthrough, the founders explain how code representations, GUI pre-training, and Philip Isola's Platonic Representation Hypothesis enable foundation models to generalize spatial reasoning zero-shot to hardware. They detail how software harnesses compile in-context experiences into deterministic skills, and outline a two-year horizon for general-purpose manipulation powered by an offline, biological 'sleep' consolidation loop that refactors slow foundation model reasoning into fast, low-latency policies.

## Atmosphere & style The conversation carries the electric, forward-leaning momentum of an insider whiteboard session among frontier AI practitioners who are watching theoretical bets pan out in real hardware. The hosts, François, and the founders share an easy, rapid-fire technical rapport, moving fluidly from abstract philosophy—like Rich Sutton's Bitter Lesson and Platonic representations—to tangible engineering bottlenecks like inference latency and inverse kinematics solvers. The mood is intellectually validating and quietly urgent, defined by a shared realization that general intelligence is breaking out of digital software stacks and directly into the physical world far faster than society expects.

## Bulletin - `00:00` Frontier researchers and startups like Waddle Labs and RoboCurve are demonstrating that general-purpose coding agents can control diverse robot embodiments out of the box with little to no domain-specific retraining. - `01:52` By shifting robot control from direct low-level token generation to code as a Chain of Thought, physical manipulation bypasses the severe teleoperation data bottleneck by piggybacking on web-scale code and tool-use datasets. - `23:01` Spatial intelligence breakthroughs in frontier models stem largely from pre-training on computer-use traces and CAD software, allowing GUI interactions to supply the geometric priors needed for physical world manipulation. - `26:15` Industry consensus across frontier labs projects general-purpose manipulation models capable of human-teenager parity within roughly two years, relying on offline 'sleep' consolidation loops to distill slow in-context reasoning into fast local policies.

## Key takeaways - `00:00` MIT professor Philip Isola's viral essay on 'robot-use agents' argues that general foundation models can orchestrate complex physical hardware via code without fine-tuning on dedicated robotics datasets. - `01:52` Early VLA architectures like RT-2 fine-tuned language backbones to output discrete Cartesian coordinates directly, forcing end-to-end mapping without intermediate reasoning compute. - `01:52` Using code as an explicit Chain of Thought offers lower Kolmogorov complexity, enabling models to generate compact, expressive programs that synthesize trajectories dynamically instead of outputting raw joint positions. - `04:47` Robotics has long been bottlenecked by scarce, expensive teleoperated demonstrations; translating control into code and API calls brings physical manipulation directly into the models' dense web-scale training distribution. - `07:22` The conceptual lineage of robot-use agents traces back to Minecraft agents like Voyager, which iteratively synthesized modular Python functions and saved them to an expanding skill library. - `10:00` Coding agents naturally possess procedural sequencing abilities, enabling single-shot assembly of parameterized functions into complex physical routines without robotics-specific fine-tuning. - `10:00` Pure in-context learning via prompt appending hits steep empirical walls, including non-monotonic performance, context saturation after 20 to 40 examples, and practical reasoning degradation before nominal context limits. - `15:00` Harness architectures convert ephemeral in-context execution traces into compiled, deterministic code routines, freeing up inference compute by keeping VLMs active only at high-variance decision points. - `15:00` Per François Chollet's thesis on intelligence, general sample efficiency requires program induction—generating dynamic programs mapping inputs to outputs—rather than informationally inefficient pure transduction. - `20:00` The Platonic Representation Hypothesis suggests that training across distinct modalities (text, code, CAD, egocentric video) converges toward a shared statistical world model that naturally supports physical manipulation. - `23:01` GUI interactions and CAD tool use instill spatial reasoning, depth, and 3D coordinate awareness; framing robot APIs as computer-use tools dramatically boosts out-of-the-box physical competence. - `26:15` Deploying foundation models in tight real-time robot control loops is latency-prohibitive, necessitating an offline 'sleep' phase inspired by DreamCoder and DAgger to distill in-context experiences into compact, high-speed policies.

## Detailed walkthrough ### `00:00` The Emergence of Robot-Use Agents

Over the past few years, one of the most unexpected developments in artificial intelligence has been the sheer degree to which coding agents generalize beyond traditional software engineering. Frontier researchers are now proving that this generalizability extends directly into the physical world to control robots. The phenomenon recently prompted MIT professor Philip Isola to publish a viral essay arguing that AI has entered the era of “robot-use agents”—a paradigm where general-purpose foundation models make diverse physical embodiments radically more capable with little to no domain-specific retraining. To explore the mechanics behind this shift, the hosts, joined by François, convene the founders behind two startups at the center of the movement: Han Mei and Vincent from Waddle Labs, and Jay from RoboCurve.

The founders’ work directly inspired Isola’s essay following several high-profile demonstrations that went viral on Twitter. The demos highlighted general-purpose large language models orchestrating complex manipulation tasks—such as unscrewing bottle caps, coordinating multi-robot handoffs, and uncapping a pen. Waddle Labs approaches the frontier by developing high-leverage evaluation harnesses that allow LLMs to control hardware effectively, gathering data from those runs, and feeding it back into training. RoboCurve focuses on the evaluation side of physical AI, benchmarking models across a wide variety of form factors, including anthropomorphic hands, parallel grippers, industrial arms, humanoids, and quadrupeds across both modern LLM agents and classic Vision-Language-Action (VLA) architectures.

- **Waddle Labs' Core Thesis:** Construct an effective software harness for LLM-based physical control, collect execution traces, and use that real-world interaction data to train more proficient models. - **RoboCurve's Evaluation Mandate:** Provide model- and embodiment-agnostic benchmarking covering LLMs, VLAs, and World Action Models (WAMs) across diverse hardware configurations. - **Viral Demonstrations:** Multi-robot collaboration, uncapping fine pens, and unscrewing bottle caps without task-specific policy learning.

---

`01:52` RT-2, Kolmogorov Complexity, and the Chain-of-Thought Shift

The conversation turns to the lineage of foundational research connecting language models to real-world control, beginning with Google DeepMind’s landmark RT-2 (Robotics Transformer 2) paper. RT-2 introduced the VLA framework by adapting a language model pre-trained on web text and images to output physical movements. Rather than outputting English text, RT-2 was fine-tuned to emit discrete tokens representing end-effector poses—Cartesian coordinates that an inverse kinematics solver translates into low-level joint commands. As Jay explains:

> “In some cases, in some sense, it's very similar to what we see now with large language models, just that instead of fine tuning, the models are good enough to just do it out of the box.”

```mermaid flowchart TD accTitle: Evolution of Robot Policy Architectures accDescr: Diagram contrasting RT-2 direct action token generation with the modern Code Chain-of-Thought agent architecture.

subgraph RT2["RT-2: Direct Token Generation"] A["Image + Language Goal"] --> B["VLA Model"] B -->|"Direct Low-Level Output"| C["End-Effector Pose Tokens"] C --> D["Joint Commands"] end

subgraph Agentic["Modern Agent: Code Chain-of-Thought"] E["Perception / Scene State"] --> F["General-Purpose LLM"] F -->|"Reasoning / Planning"| G["Scratchpad / CoT Code"] G -->|"Iterative Execution"| H["Executable Python Tools"] H --> I["Robot Hardware Control"] end ```

The hosts draw a direct historical parallel between the jump from RT-2 to robot-use agents and the pivotal “Chain-of-Thought” (CoT) breakthrough in mathematical reasoning benchmarks like GSM8K. In early mathematical evaluation setups, a model prompted with a simple arithmetic word problem—such as Susie having eight dollars and spending five—was forced to jump straight from input to output, producing four delimiter hash marks, the final answer, and an immediate end-of-sequence (`EOS`) token with zero intermediate compute. Once researchers unshackled models by giving them a scratchpad to reason step-by-step (`8 - 5 = 3`), performance surged. Similarly, RT-2 suffered from an architectural straightjacket: it was forced to map raw perception straight to motor actions without allocating inference-time compute to plan or break down complex tasks. By contrast, leveraging code as an explicit chain of thought provides a representation with vastly lower Kolmogorov complexity, enabling the agent to write compact, expressive programs that synthesize complex physical trajectories on the fly. Jay underscores this philosophical alignment with Rich Sutton's timeless doctrine:

> “If you give the agents or you give the AI model more autonomy and if you unshackle it, a bit more and give it more resources. It can actually do a lot of the things that we fine-tune it to do.”

---

`04:47` Swallowing the Bitter Lesson: Escaping the Robotics Data Bottleneck

Vincent expands on the implications of Sutton's Bitter Lesson, pointing out that VLAs are already built on top of language model backbones, meaning the real differentiator is not neural architecture but data distribution. For years, robotics labs have slammed into the severe data bottleneck inherent to collecting teleoperated real-world demonstrations, resulting in painfully sluggish progress across the field. Modern robot-use agents bypass this wall entirely by translating the physical control problem into a modality where data is already abundant, expressive, and well-understood by frontier models: code and structured tool use.

> “Like, how do you take something that is traditionally out of distribution, like robotics, and make it something that's in distribution?”

The core power of RT-2 was never its physical robotics data; it was the fact that the underlying backbone had already absorbed internet-scale web images and text. In the years since RT-2, state-of-the-art general-purpose models have developed profound in-context learning capabilities, tool use, and environment exploration. By leveraging these models directly through coding interfaces, researchers can sidestep fine-tuning brittle, task-specific policies. Vincent captures the team's governing design principle:

> “And so to me, it's like, okay, why not just build a really good foundational LLM and then use that to control robots rather than training a model that's specifically for robotics and that's more dependent on just robot data.”

- **The VLA Data Trap:** Physical robot interaction data remains expensive, scarce, and difficult to standardize across embodiments. - **Modality Transfer:** Transforming physical control into a programming and tool-calling problem brings physical tasks into the well-optimized training distribution of frontier models. - **The Modern LLM Tool Stack:** Frontier models can actively inspect robot APIs, write parameterized trajectory code, interpret runtime errors, and self-correct using in-context feedback.

---

`07:22` Code as Policies, Voyager, and the Unintended Liftoff of Coding Agents

To trace how the field reached this inflection point, the group revisits the origins of "Code as Policies"—a line of work that emerged in late 2022 alongside the release of ChatGPT, when automated code generation was largely restricted to comment autocompletion. Jay points to systems like Voyager in Minecraft as the true conceptual ancestor of current physical agents. In Voyager, an LLM started with simple Python built-in primitives (`for`, `while`, `sort`), used them to interact with the game environment, and iteratively wrote new modular functions—such as saving a custom tool as `Francois.py`—which it added to an expanding skill library to invoke later when facing harder challenges.

This architecture demonstrated that code could serve as a durable medium for compressing experience, abstract reasoning, and tool synthesis. However, when top AI lab leaders began aggressively pouring capital and compute into coding agents around 2024, their explicit goal was not physical robotics, but automating the software and machine learning engineering loops to reach recursive self-improvement. Jay points out the unintended consequence of that massive push:

> “I don't think they were so insightful to know that, oh, if I did this, then we will have LLMs that will be good enough to write policies for robotics.”

Earlier implementations, such as the original Code as Policies work from Google DeepMind, were severely constrained compared to modern agents. They relied on human engineers providing static, hand-crafted API wrappers—such as predefined functions for picking up an object, elevating the end effector, or translating to fixed coordinate poses. The model merely acted as a logic layer gluing together rigid primitives, a far cry from modern autonomous agents capable of generating expressive control code, dynamic feedback loops, and multi-robot orchestration from scratch.

`10:00` Coding Agents and the In-Context Learning Hierarchy

The discussion opens with how coding agents operate when exposed to physical hardware: developers provide a list of capabilities directly as Python functions. Because coding agents have been exposed to vast amounts of software engineering data, they can synthesize code orchestrating these functions into complex robotic behaviors in a single shot, without requiring supplementary robot datasets. They inherently possess an algorithmic sense of procedural sequencing—such as understanding what to execute first and second to place a block into a bowl. This one-shot capability and capacity for in-context exploration have served as the core catalyst driving teams like Waddle Labs to apply large language models (LLMs) to physical robotics.

```mermaid flowchart TD accTitle: In-Context Learning vs Weight Updates Hierarchy accDescr: Spectrum of learning methods from context-based updates to full weight fine-tuning.

subgraph ICL["In-Context & Retrieval Layer (Low Data Regime)"] A["Pure ICL Appending<br>(State, Action, Reward tuples)"] --> B["Self-Refine & Multi-Turn Reflection"] B --> C["Active Memory & RAG Retrieval<br>(Textbook lookup / Skill library)"] end

subgraph Weights["Parametric Updates Layer (High Data Regime)"] D["Low-Rank Adaptation<br>(LoRA Rank 1 - 100)"] --> E["Full SFT & RL<br>(End-to-end policy training)"] end

C -.->|"Distill / Compile"| D ```

Francois outlines an experimental framework analyzing sample efficiency: how best to feed a state-action-reward tuple back into an agent's policy. The simplest approach—appending feedback directly into the prompt context—yields cheap, immediate adaptation on validation tasks (such as held-out GSM8K benchmarks) without spending FLOPs on stochastic gradient descent (SGD). However, pure in-context learning (ICL) quickly hits harsh empirical limits:

1. **Non-monotonic improvement:** Performance fluctuates wildly, worsening and improving erratically across iterations. 2. **Rapid saturation:** Performance typically plateaus after 20 to 40 examples; appending further instances yields no accuracy gains. 3. **Effective context degradation:** If a model is trained on a 100,000-token context, its practical reasoning window is closer to 50,000 tokens. Beyond that threshold, attention degrades and performance falls off.

To scale beyond raw prompt stuffing, agents can load relevant examples dynamically into active memory via retrieval-augmented generation (RAG)—analogous to taking an open-book exam with a textbook rather than relying on raw memorization. Across the broader learning spectrum, this sits below parametric interventions like low-rank adaptation (LoRA at ranks 1, 10, or 100) and full supervised fine-tuning (SFT) or reinforcement learning (RL). While high-data autonomous vehicle operations like Tesla would never rely on ICL to steer cars, low-data robotics regimes benefit immensely from compressing ICL experiences directly into reusable tool calls.

- Pure ICL updates offer zero-FLOP parameter adaptation but are constrained by the model's post-training multi-turn self-reflection abilities. - Context saturation causes models to drop performance well before exhausting their nominal context window limit. - “And if you're Tesla and you have all the data, infinite data, and you're learning doing self-driving car by ICL, what are you doing?”

---

`15:00` Harnesses, Distillation, and Latency Scaling

The founders of Waddle Labs frame their robot execution harness as a mechanism for domain specialization. When a robot enters an unfamiliar operational setting—such as an automated wet lab handling biological assays—it acquires initial procedures in-context. Rather than forcing subsequent agents to re-derive these behaviors, the harness packages validated experiences into discrete, executable code routines. This program synthesis acts as a concrete distillation step, transforming ephemeral multi-turn memory into permanent, structured tools.

This architecture closely mirrors meta-learning paradigms: a larger, highly capable model programs and configures a smaller, faster model wrapped inside a specialized harness. Because theoretical research indicates that transformer forward passes during in-context learning can mathematically approximate gradient descent steps, the operational boundary between symbolic execution and neural weight space becomes fluid.

```mermaid flowchart LR accTitle: Harness Distillation Workflow accDescr: Big model compiles in-context experience into fast routines managed by a harness.

subgraph Meta["Meta-Level Planning"] BM["Large Frontier Model<br>(High latency / Complex reasoning)"] end

subgraph Exec["Runtime Execution"] Harness["Waddle Harness Engine"] Skill["Compiled Deterministic Skill<br>(Grasp, Approach, Transport)"] VLM["In-the-loop VLM<br>(Object checks & Failure detection)"] end

BM -->|"Programs & Specializes"| Harness Harness --> Skill Skill -->|"Exception / Branching"| VLM VLM -->|"State Recovery"| Skill ```

Analyzing a live demonstration where an agent controls robotic arms to move a block into a bowl highlights the mechanical differences between raw model streaming and harness-mediated control:

- **Direct Model Control vs. Harness:** In the un-harnessed setup, the model processes multi-camera streams and continuously outputs tool calls specifying end-effector poses. While flexible, this approach incurs substantial latency. - **Compiled Skills:** Routine mechanical motions (approaching a receptacle, grasping, linear movements) are fully deterministic once discovered and should run as compiled code outside the model's primary loop. - **Targeted VLM Insertion:** Vision-language models (VLMs) remain inside the policy graph strictly at points of high variance—such as identifying ambiguous objects or evaluating whether a grasp failed—providing adaptive fallback routines without stalling the entire loop. - **Inference Latency Velocity:** While current video control exhibits noticeable pauses, latency for reasoning-class models is dropping at approximately 2x per month, putting real-time closed-loop control on track to become viable by the end of the year.

Reflecting on the foundations of machine intelligence, the speakers connect this execution pattern to François Chollet's foundational work *On the Measure of Intelligence*. Chollet differentiated between pure transduction and program induction:

> “The biggest insight I think I've had a change in worldview on how to perform machine learning and how to get the rest of the distance on AGI was like a lot of conversations I've had with François Chollet.”

Directly mapping inputs to outputs ($X \to Y$) via pure transduction is informationally inefficient, demanding immense volumes of paired training data or massive hand-crafted inductive biases. Achieving generalizable sample efficiency instead demands program induction: using an expressive generator to synthesize programs that map inputs to outputs dynamically.

`20:00` Neurosymbolic Code Generation and the Platonic Representation Hypothesis

The conversation explores how code generation acts as an intuitive bridge between perception and action. The panel compares prompt-driven policy writing to a classic whiteboard coding interview: given inputs $X$ and desired outputs $Y$, the model's job is simply to synthesize a function $f$ that maps $X \to Y$. Historically, program synthesis struggled because researchers had to manually handcraft restrictive inductive biases to find that mapping. With modern frontier models, the paradigm has shifted toward neurosymbolic execution—neural networks that directly emit symbolic code.

```mermaid flowchart TD accTitle: The Platonic Representation Convergence accDescr: Diverse data modalities converging into a unified physical world model that feeds robotic execution. subgraph DataInputs ["Multimodal Data Streams"] A["Language & Code"] B["Computer-Use & GUIs"] C["CAD & 3D Assets"] D["Egocentric Video"] end subgraph FoundationModel ["Frontier Foundation Model"] E["Shared Latent World Model<br/>(Platonic Representation)"] end subgraph ActionOutput ["Robot-Use Execution"] F["Symbolic Code & Policies"] G["Physical Actuation"] end A --> E B --> E C --> E D --> E E --> F F --> G ```

This dynamic leads directly into MIT professor Philip Isola’s "Platonic Representation Hypothesis." Drawing an analogy to Plato’s allegory of the cave—where observers only see shadows cast by underlying reality—the hypothesis posits that disparate neural models trained across different objectives and data modalities inevitably converge on the same shared statistical structure of the physical world. For robotics, this has radical implications: a sufficiently powerful language and multimodal model will implicitly learn the same latent world model required for embodied manipulation. This serves as an ultimate validation of Rich Sutton's "Bitter Lesson," proving that massive scale on general data outpaces hand-engineered, domain-specific robotics architectures.

As one founder points out regarding this convergence: “And if that's the case, then if you have a really strong language model, you also have a really strong robotics model using if we believe in the platonic representation hypothesis.”

- Program synthesis traditionally failed due to the difficulty of engineering the right inductive biases to constrain search spaces. - Neurosymbolic systems resolve this by allowing neural weights to emit symbolic code representations directly. - The Platonic Representation Hypothesis suggests that training on web-scale text, computer interfaces, and vision converges onto a universal, unified model of physical reality. - The Bitter Lesson asserts that general compute and large models will consistently beat domain-tailored robotics pipelines.

---

`23:01` Computer-Use Data, GUI Pre-training, and Spatial Intelligence

The speakers turn to the mystery of spatial intelligence in newly released frontier models like OpenAI's Astra. Recent benchmarks show sudden, step-function jumps in spatial reasoning, 3D awareness, and CAD software control (such as driving Blender) that models from just six months prior could not perform. The group attributes this breakthrough to pre-training on massive amounts of "computer-use" data. When an AI agent learns to navigate an operating system—dragging a mouse cursor to orbit a 3D part in CAD or navigating complex menus—it implicitly learns spatial relationships such as depth, top-down alignment, and geometric perspective.

The panel reflects on the historical irony of this feedback loop. In the 1970s and 1980s, computer scientists at Xerox PARC invented the Graphical User Interface (GUI) by borrowing metaphors from the physical world: desktops, files, folders, and later, viewport orbiters in tools like SolidWorks and Autodesk. Decades later, because gathering physical robot interaction data at internet scale is prohibitively difficult, AI researchers train models on those virtual desktop simulations—and the models learn enough physics and geometry to control machines back in the real world. Research from Princeton and Cloud-based robotics labs demonstrates that when robot control harnesses are deliberately framed as computer-use tools (like letting an agent steer a robot by dragging an onscreen cursor), foundation models adapt to physical manipulation tasks with far higher zero-shot competence.

Reflecting on the recursive evolution of computing interfaces, one speaker observes: “I think it's so funny that if you go to 1980s, Xerox PARC or whatever, we literally made the graphical user interface to be more like the physical world so that we could interface with it.”

- Frontier models obtain rich spatial priors not from physical teleoperation alone, but from computer-use traces and CAD manipulation. - Interaction with GUIs instills foundational concepts of spatial reasoning, coordinate planes, and top-down versus lateral orientation. - Structuring robot APIs to mimic familiar desktop GUIs or cursor-drag tools allows out-of-the-box coding and agent models to operate hardware far more effectively. - Combining coding, computer use, and egocentric video into a single general-purpose model produces a more adaptable robot agent than fine-tuning on narrow robot datasets.

---

`26:15` The Two-Year Horizon and the "Sleep" Consolidation Architecture

Looking forward, the founders highlight an emerging consensus inside frontier research labs and robotics foundation companies: general-purpose robots are likely only two years away. By "general-purpose," they envision a system that can be given arbitrary natural language commands and achieve physical manipulation parity with a competent human teenager using their bare hands—a true "ChatGPT moment" for embodied AI that society remains largely unprepared to meet.

However, real-world deployment faces severe runtime bottlenecks, primarily inference latency. Running a massive model like Astra in the tight closed-loop control of a robotic arm at every millisecond step is far too slow and economically prohibitive. To solve this, the founders argue for an architecture mirroring biological sleep and memory consolidation. During daytime operation, the robot records interactive state-action-reward $(s, a, r)$ trajectories in-context. Later, in an offline "sleep" phase—analogous to how the human brain's hippocampus uses sharp-wave ripples to replay and transfer short-term memories into the neocortex—the system consolidates those slow, deliberative reasoning traces into lightweight, high-speed local policies using frameworks like DAgger (Dataset Aggregation) and program refactoring inspired by MIT’s DreamCoder.

Highlighting the rapid timeline and societal underestimation of embodied agents, Jay emphasizes: “And when we say general purpose robots, we mean something like if you give any natural language instruction, it can do what a competent teenager could do with their bare hands.”

```mermaid flowchart LR accTitle: Two-Stage Deployment and Sleep Consolidation Loop accDescr: Fast execution loop backed by slow foundation model reasoning and an offline sleep consolidation phase. subgraph WakePhase ["Active Execution (Wake Phase)"] A["Task Prompt"] --> B["Astra / Frontier Model<br/>(In-Context Reasoning)"] B --> C["High-Latency Action / Skill"] C --> D["State-Action-Reward<br/>Traces (s, a, r)"] end subgraph SleepPhase ["Memory Consolidation (Sleep Phase)"] D --> E["Hippocampal-Style Replay<br/>& Skill Refactoring"] E --> F["DAgger Distillation"] F --> G["Compact, Low-Latency<br/>Weight Policy"] end G -.->|"Deploy Fast Model"| C ```

- Frontier AI labs anticipate that general-purpose manipulation models capable of open-vocabulary physical tasks will arrive within roughly 24 months. - Direct closed-loop foundation model inference is too slow and costly for sustained high-throughput factory or home manipulation. - Biological consolidation provides the blueprint: fast in-context collection during waking tasks, followed by an offline sleep cycle that compresses experiences into low-latency policy weights. - Drawing on DreamCoder's program-library synthesis and DAgger distillation, startups like Waddle Labs are focusing on the orchestration layer: pruning, refactoring, and maintaining a dynamically growing codebase of modular physical skills.

## Notable quotes > `26:15` when we say general purpose robots, we mean something like if you give any natural language instruction, it can do what a competent teenager could do with their bare hands. — Jay

> `04:47` why not just build a really good foundational LLM and then use that to control robots rather than training a model that's specifically for robotics and that's more dependent on just robot data — Vincent

## Mentioned Philip Isola, Waddle Labs, RoboCurve, Han Mei, Vincent, Jay, Google DeepMind, RT-2, Rich Sutton, Voyager, Tesla, François Chollet, On the Measure of Intelligence, Astra, OpenAI, Xerox PARC, Blender, SolidWorks, Autodesk, DreamCoder, DAgger

## Video description <p>One of the biggest surprises in AI over the last few years has been how well coding agents generalize beyond software. In a recent essay, MIT professor Philip Isola argued that we may be entering the era of robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training.In this episode of Decoded, we're joined by the founders of Waddle Labs and RoboCurve, two of the startups whose work helped drive this realization. They're working at the frontier of using general-purpose models to control robots, and their recent demos helped inspire the growing conversation around robot-use agents. </p><p>Together, we dig into the research behind that idea, from code-as-policies and vision-language-action models to the harnesses and evals needed to make these systems work in the real world.</p>

[Source](https://anchor.fm/s/8c1524bc/podcast/play/126339684/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2026-8-26%2F432708848-44100-2-ba9049a9c4a07.mp3)

中文简报

AI代码模型颠覆机器人领域，创始人预计两年内达人类同等水平 通用基础模型通过将机器人控制转化为软件代码，绕过了稀缺的物理训练数据，有望在24个月内实现人类水平的操作能力。 本次讨论探讨了现代代码智能体如何通过将现实世界的操作构建为程序归纳和结构化工具调用，从而绕过机器人领域的物理数据瓶颈。创始人们对比了RT-2与思维链（Chain-of-Thought）的突破，阐释了代码表征、GUI预训练以及Philip Isola的柏拉图表征假说（Platonic Representation Hypothesis）如何使基础模型能够零样本泛化至硬件的空间推理中。他们详细说明了软件框架如何将上下文体验编译为确定性技能，并概述了一个为期两年的通用操作实现前景，该前景由一种类似生物“睡眠”巩固循环的离线机制提供支持，可将缓慢的基础模型推理重构为快速、低延迟的策略。 前沿研究人员以及Waddle Labs和RoboCurve等初创公司正在证明，通用代码智能体几乎无需针对特定领域重新训练，即可开箱即用控制多种不同的机器人形态。 (0:00) 通过将机器人控制从直接的底层Token生成转变为作为思维链的代码，物理操作借助网络规模的代码和工具使用数据集，绕过了严重的遥操作数据瓶颈。 (1:52) 前沿模型在空间智能方面的突破主要源于对计算机使用轨迹和CAD软件的预训练，这使得GUI交互能够提供物理世界操作所需的几何先验。 (23:01) 各前沿实验室的行业共识预计，大致在两年内将出现达到人类青少年水平的通用操作模型，该模型依赖离线“睡眠”巩固循环将缓慢的上下文推理蒸馏为快速的本地策略。 (26:15)

Reading summary

MIT professor Philip Isola and founders from Waddle Labs and RoboCurve argue that robotics is entering an era of 'robot-use agents,' where general-purpose frontier models control physical hardware through code generation and tool use rather than domain-specific policy training. By translating physical manipulation into programming and computer-use distributions, these systems bypass the acute data scarcity that has long constrained traditional robot learning. From Action Tokens to Code Chain-of-Thought (1:52) Robotics research previously prioritized Vision-Language-Action (VLA) architectures like Google DeepMind’s RT-2, which adapted pre-trained models to output discrete end-effector pose tokens mapped to joint commands via inverse kinematics. Jay of RoboCurve notes that RT-2 forced models into an architectural straightjacket: mapping raw perception straight to motor actions without allocating inference-time compute for planning. The panel likens this to early math evaluations on GSM8K, where models failed when forced to produce an answer without a step-by-step scratchpad. Modern agents instead adopt executable code as an explicit chain of thought. Tracing back to systems like Voyager in Minecraft, code serves as a compact representation with low Kolmogorov complexity that enables models to iteratively write, save, and invoke modular functions. Drawing on François Chollet's distinction between pure transduction and program induction, the speakers argue that generating dynamic programs to orchestrate hardware primitives is fundamentally more sample-efficient than training neural networks to memorize rigid, direct input-to-output mappings. Escaping Data Bottlenecks via Computer-Use Pre-Training (4:47) Progress in robotics has historically stalled against the data bottleneck of collecting real-world teleoperated demonstrations. Vincent of Waddle Labs argues that modern robot-use agents bypass this barrier by translating physical control into modalities where training data is already abundant on the internet: code and structured tool use. Recent step-function improvements in spatial awareness—such as OpenAI's Astra manipulating 3D objects in CAD software like Blender—stem primarily from pre-training on operating system and graphical user interface (GUI) interactions rather than physical robot logs. The panel connects this mechanism to Philip Isola’s Platonic Representation Hypothesis, which posits that neural models trained on diverse modalities (text, code, GUI navigation, and egocentric video) converge toward a shared statistical representation of physical reality. Structuring robot control to emulate computer interfaces—such as directing an arm by dragging an on-screen cursor—allows foundation models to leverage these latent spatial priors out of the box, aligning with Rich Sutton’s 'Bitter Lesson' that general compute and large-scale data consistently outperform hand-tailored robotics pipelines. In-Context Learning Boundaries and Harness Distillation (10:00) While general-purpose coding agents can sequence hardware capabilities exposed as Python functions zero-shot, relying purely on in-context learning (ICL) quickly encounters empirical limits. François outlines that appending state-action-reward interaction tuples directly into prompts results in erratic, non-monotonic progress, saturates after 20 to 40 examples, and causes attention degradation well before context limits are reached (such as effective reasoning degrading beyond 50,000 tokens in a 100,000-token model). To bridge this gap, Waddle Labs uses execution harnesses to specialize models without continuous, high-latency frontier model queries. Once an agent discovers a functional procedure in-context, the harness compiles the routine into deterministic code for repetitive physical actions like reaching and grasping. High-latency vision-language models (VLMs) are retained inside the execution graph only at high-variance decision points, such as inspecting whether an object slipped from a gripper, while inference latency across reasoning models drops at roughly 2x per month. Biological Sleep Consolidation and Deployment Timelines (26:15) The founders highlight an internal consensus among frontier AI labs that general-purpose manipulation models—defined as systems capable of matching the manual dexterity of a competent teenager via natural language instructions—could arrive within roughly 24 months. However, running multi-billion-parameter foundation models in tight, closed-loop physical control at millisecond intervals remains economically and computationally unviable for sustained real-world deployment. To resolve this execution bottleneck, the panel advocates an architecture modeled on biological memory consolidation. During daytime 'wake' phases, a robot executes tasks using slower, deliberative in-context reasoning while logging state-action-reward trajectories. In an offline 'sleep' phase, the system replays these experiences using DAgger distillation and DreamCoder-style program refactoring, compressing ephemeral multi-turn reasoning into high-speed, lightweight local policy weights. While robot-use agents have demonstrated zero-shot capabilities in multi-robot handoffs and precise manipulation like uncapping pens, the architecture's ultimate viability hinges on whether offline distillation can match the latency requirements of real-world physical deployment before the anticipated two-year horizon arrives.

精炼阅读摘要

麻省理工学院教授 Philip Isola 以及 Waddle Labs 和 RoboCurve 的创始人提出，机器人领域正进入“机器人使用智能体”（Robot-Use Agents）时代。通用大语言模型无需特定硬件重训，即可通过生成代码与调用工具直接驱动物理实体，在泛化效率上可能全面超越传统针对机器人微调的视觉-语言-动作（VLA）模型。 代码思维链与程序归纳的控制演进 (1:52) 在早期技术路径中，Google DeepMind 的 RT-2 等视觉-语言-动作（VLA）架构将预训练模型直接微调为输出末端执行器的离散位姿 Token，但这种从感知到动作的直接映射缺乏推理与规划计算，类似于数学推理模型在没有思维链（CoT）时直接输出答案。RoboCurve 创始人 Jay 指出，现代智能体将代码作为显式思维链，以具有更低柯氏复杂度的紧凑程序即时合成复杂物理轨迹。这种模式契合了 François Chollet 提出的“程序归纳”理论：相比于依赖海量配对数据直接将输入映射至输出的纯转导，由模型生成动态控制代码具有更高的样本效率。 早期类似 DeepMind 的“Code as Policies”依赖人工编写的静态 API 封装，模型仅充当胶水逻辑；而现代系统受 Minecraft 的 Voyager 智能体启发，能从基础 Python 原语出发，自主编写并沉淀如自定义工具等模块化技能。Jay 指出，前沿实验室起初投入代码智能体是为了软件工程自动化与递归自我改进，却意外赋予了模型生成底层机器人控制策略的能力。 突破数据墙：柏拉图表征与计算机使用 (4:47) 传统机器人研究长期受困于昂贵且难以标准化的遥操作真实世界数据瓶颈。Waddle Labs 联合创始人 Vincent 认为，VLA 模型的核心能力并非源于机器人专属数据，而是其底座吸收的互联网级图文知识；通过将物理控制转化为代码与结构化工具调用，研究者直接将物理任务映射进了前沿模型高度成熟的训练分布中。这一演进印证了 Rich Sutton 的“苦涩的教训”（Bitter Lesson），即通用计算与大规模数据驱动终将战胜手工设计的特定领域架构。 此外，Philip Isola 的“柏拉图表征假说”推论，跨模态训练最终会收敛于对物理世界相同的统计潜在表征。近期如 OpenAI Astra 等前沿模型在空间智能与 CAD（如 Blender）控制上的跳跃式突破，正是源于海量的“计算机使用”（Computer-Use）数据训练。Xerox PARC 在 20 世纪 80 年代借鉴现实物理世界设计了图形用户界面（GUI），如今模型通过在虚拟桌面中学习拖拽光标与三维视口旋转，反向掌握了深度与空间几何先验。研究表明，将机器人接口抽象为类似 GUI 拖拽的操作工具，能显著提升模型的零样本控制表现。 软件线束设计与上下文学习极限 (10:00) 在硬件控制实践中，开发者直接以 Python 函数形式向模型公开物理能力。然而实验分析显示，纯粹的上下文学习（ICL）存在显著局限：在无梯度更新的情况下，反馈拼接通常在 20 至 40 个样本后性能就会迅速饱和，且表现呈现非单调波动；在标称 100,000 Token 的上下文中，模型实际有效推理窗口仅约 50,000 Token，超出后注意力机制会出现退化。 为解决此类瓶颈，Waddle Labs 构建了软件线束（Harness）来实现领域特化与程序蒸馏。智能体在陌生的湿实验环境初次探索时通过 ICL 获取流程，线束则将验证后的经验编译为固定的确定性动作程序，脱离大模型主循环运行。视觉语言模型（VLM）仅被挂载在物体识别或抓取失败校验等高方差决策节点上，从而规避了全流程模型流式输出造成的巨大延迟。随着推理延迟以每月约 2 倍的速度下降，这种线束介导的闭环控制正逐步具备实时执行能力。 两年期通用愿景与仿生睡眠整合架构 (26:15) 前沿研究团队与创业公司正在形成共识，预计通用机器人将在约 24 个月内迎来“ChatGPT 时刻”，届时系统将能根据自然语言指令达到与人类青少年徒手操作相当的物理灵巧水平。然而，在毫秒级闭环控制中直接调用超大前沿模型的推理成本与延迟目前在经济和工程上均不可行。 为克服该阻碍，创始人主张借鉴生物大脑海马体向大脑皮层转移记忆的“睡眠巩固”机制：机器人在日间执行任务时于上下文中记录状态-动作-奖励轨迹；在离线“睡眠”阶段，系统通过类比 DreamCoder 的程序重构与 DAgger（数据聚合）算法，将缓慢深思的推理轨迹蒸馏整合为紧凑、低延迟的本地策略权重，从而使初创公司能够专注于维护和修剪不断增长的物理技能代码库。 尽管通用大模型通过代码与工具介导展现出跨越实体控制瓶颈的巨大潜力，但实时高频闭环执行的高延迟与硬件成本仍是现实阻碍。当前关于通用机器人将在两年内达到青少年灵巧度的预测，仍高度依赖离线程序蒸馏架构的工程落地，以及前沿模型在物理常识与空间因果推断上的持续泛化检验。

Detailed notes — zh

## Background 本期节目由 Y Combinator 于 2026 年 9 月 26 日在 RSS / Web 播客平台发布。节目聚焦 MIT 教授 Philip Isola 关于“机器人使用智能体（Robot-Use Agents）”时代到来的核心观点，邀请了 Waddle Labs 与 RoboCurve 的创始团队，共同探讨通用模型如何打破传统机器人专有数据瓶颈，利用代码与评测框架控制多样化硬件并学习物理世界新任务。 ## TL;DR 对话深入探讨了机器人控制从传统 VLA 预测末端位姿向通用大模型“代码即策略（Code as Policies）”范式的剧烈演进。嘉宾们指出，原本为自动化软件工程训练的代码与电脑使用（Computer-Use）数据，意外成为赋予模型极强空间智能与复杂动作规划能力的关键底座。面对上下文学习的容量上限与高推理延迟，未来的工程核心在于通过 Harness 框架将慢速思考编译、离线提炼为低延迟的确定性技能库。行业前沿普遍预期，类似于 ChatGPT 时刻的通用机器人将在未来两年内快速到来。 ## Atmosphere & style 整场对话洋溢着前沿技术突破带来的兴奋感与求知欲，充满了极客式的思维碰撞与底层范式拆解。主持人和嘉宾的交流极为硬核且节奏紧凑，从一线的机械臂实机演示直接跃升至柯尔莫哥洛夫复杂度、柏拉图表征假说以及人机交互拟物化闭环等哲学与数学高度。听众能清晰感受到技术创业者面对“通用机器人时代仅剩两年倒计时”时的迫切感与笃定，既有对传统机器人学被通用模型降维打击的清醒洞察，又饱含对构建物理智能未来的宏大野心。 ## Bulletin - `00:00` 机器人控制范式正经历从端到端直接预测位姿（如 RT-2）向代码思维链（Code CoT）与 Robot-Use Agent 的演进，通用大模型无需针对机器人微调即可直接生成控制策略。 - `04:44` 代码被视为解决机器人真实物理数据匮乏的最优通用模态，将分布外（OOD）的物理动作映射为大模型极其擅长的分布内代码调用，全面契合“苦涩的教训”。 - `22:20` 前沿多模态大模型的空间智能爆发，主要得益于海量计算机使用（Computer-Use）数据和 CAD 三维设计轨迹对三维几何常识的反哺，形成了 GUI 拟物化设计反向教会 AI 物理规律的闭环。 - `25:00` 行业领军者预期通用机器人将在两年内到来，具备相当于十几岁青少年的双手通用操作能力；工程落地的核心瓶颈在于推理延迟，正通过类似生物睡眠与 DreamCoder 的技能蒸馏架构进行解决。 ## Key takeaways - `00:00` MIT 教授 Philip Isola 指出行业正迈入“机器人使用 Agent（robot-use agents）”时代，通用模型无需专用微调即可跨形态操控硬件并自行编写策略。 - `02:35` 传统 VLA（如 RT-2）强制模型接收输入后立即输出动作步进，缺乏动态测试期算力（test-time compute），而代码思维链让模型具备了生成前推演的能力。 - `02:35` 基于柯尔莫哥洛夫复杂度视角，紧凑的代码比端到端拟合底层连续轨迹具有更低的信息复杂度与更高的抽象效率。 - `04:44` 代码是验证过的高杠杆模态，利用吸收了全网代码与计算机使用数据的通用 LLM 编写策略，比昂贵真机遥操作数据驱动的专用基础模型更具扩展性。 - `06:56` 各大实验室押注编码模型原意是实现自动化机器学习工程师，却意外使得 Coding Agent 溢出成为驱动物理硬件与图形生成的最优底座。 - `10:00` 通用 LLM 具备凭借代码先验“一发即中（One-shot）”完成积木抓取等物理任务的能力，但在小样本上下文学习中存在非单调波动、20-40 样本即容量饱和以及长上下文退化等局限。 - `15:00` Waddle Labs 提出的 Harness 架构充当领域特异化外壳，将即时探索的上下文经验编译固化为确定性代码程序，形成宏观大模型规划下层轻量模型执行的元学习架构。 - `17:00` 真实机械臂演示表明，高频重复性动作应编译为确定性代码流水线，而关键断点（如失败判定、自适应重试）嵌入 VLM 提供弹性反馈，从而平衡延迟与灵活性。 - `19:00` 主流机器学习正从低效的输入输出直接映射（$X \to Y$）转向“程序归纳（Program Induction）”，让高维生成器自主归纳并合成程序符号来驾驭连续物理世界。 - `20:00` “柏拉图表征假说”若成立，意味着极强语言模型的世界表征与机器人模型高度收敛，规模化的通用大模型将凭借统一的世界表征直接超越专用机器人小模型。 - `22:20` 用户在 3D 软件和操作系统中拖动画笔与旋转视角的 GUI 交互数据，本质上教会了多模态模型末端执行器在笛卡尔坐标系中的变换与遮挡规律。 - `25:00` 为克服昂贵大模型的推理延迟，新一代系统借鉴生物海马体向大脑皮层转录机制与 MIT DreamCoder 思路，在系统“睡眠阶段”将在线运行轨迹重构蒸馏为轻量级端侧策略。 ## Detailed walkthrough ### `00:00` 机器人使用 Agent 的兴起与 VLA 范式演进 过去几年 AI 领域最令人震惊的现象之一，是编码 Agent（coding agents）跨越软件领域的超强泛化能力。前沿研究者们正证明这种能力同样可以用来控制物理机器人。麻省理工学院（MIT）教授 Philip Isola 在一篇引发广泛关注的随笔中指出，我们或许正在步入“机器人使用 Agent（robot-use agents）”的时代：通用模型无需针对特定机器人进行大量训练，就能操控不同形态的硬件、自行编写策略（policies）并掌握物理世界的新任务。Waddle Labs 联合创始人 Han Mei 与 Vincent 专注于开发控制机器人的 LLM，其核心路径是构建高效的测试与运行框架（harness）、收集数据并训练出更强大的模型；RoboCurve 联合创始人 Jay 则将公司定位为物理 AI（Physical AI）的评测平台，覆盖从机械手、夹爪、机械臂到双足人形、四足机器人等各类形态，评测对象横跨通用 LLM 以及经典视觉-语言-动作模型（VLA）与世界动作模型（World Action Models）。 ```mermaid flowchart LR accTitle: 从 RT-2 到推理代码策略的演进 accDescr: 对比传统 VLA 模型的直接末端输出与现代基于代码思维链的模型推理路径。 subgraph VLA["传统 VLA 路径 (RT-2)"] A1["多模态输入<br/>(图像 + 文本)"] --> A2["微调 LLM 骨干"] A2 --> A3["直接输出末端位姿<br/>(固定坐标，无 Test-time Compute)"] end subgraph CAP["代码思维链 / Robot-Use 路径"] B1["任务目标 + 环境反馈"] --> B2["通用 LLM 推理<br/>(Code CoT / 思考时间)"] B2 --> B3["生成紧凑代码 / 策略函数"] B3 --> B4["调用机器人 API<br/>执行复杂长程动作"] end ``` 针对两家团队近期在 Twitter 上走红并被 Philip Isola 引用的演示视频——包括大模型控制机械臂拧开瓶盖、多机器人协作通讯以及拔掉笔帽等高难度动作——讨论回溯到了语言模型迁移至机器人控制的早期源头。以 Google 的 RT-2（Robotics Transformer 2）论文为例，该工作通过在网络图文数据上预训练模型，再微调输出机器人的末端执行器位姿（end effector pose）坐标，进而转换为关节控制指令。Jay 总结道，现在的通用大模型已经强大到无需针对此类动作做专门的微调，即可开箱即用（out of the box）。 - Waddle Labs 的核心工程闭环：构建 Harness 基础设施 $\rightarrow$ 采集控制交互数据 $\rightarrow$ 迭代训练更强的通用控制模型。 - RoboCurve 的评测覆盖面：兼容 VLA、世界动作模型与前沿 LLM，横跨 Gripper、多指手、人形及四足机器人。 - RT-2 的机制本质：利用海量 Web 数据预训练的骨干网络，微调使其直接预测离散化的末端执行器位姿坐标。 --- ### `02:35` 思维链（CoT）革命：从固定动作到代码思维链 机器人控制领域的范式转变，与自然语言处理中“思维链（Chain of Thought, CoT）”的爆发高度相似。在早期的数学推理基准 GSM8K 中，模型面对“Susie 有 8 美元，花掉 5 美元，还剩多少？”的问题时，被强制要求直接吐出格式标记与最终答案（如“#### 3”），完全没有推理缓冲空间；直到后来允许模型生成推理路径（“8 - 5 = 3”），推演能力才真正解锁。早期的 RT-2 等 VLA 模型就像最初的 GSM8K，接收输入后必须立即输出动作步进，无法在遇到复杂任务时动态分配更多测试期算力（test-time compute）。 现代 Agent 引入了“代码思维链（Code Chain of Thought）”机制：即使最终不直接把代码暴露给底层执行器，也可以让模型在内部调用 Astra 等工具进行充分思考推演，再下发动作。从柯尔莫哥洛夫复杂度（Kolmogorov complexity）的角度来看，采用高度紧凑的代码表达策略，比强行端到端拟合底层轨迹具有更低的信息复杂度与更高的抽象效率。给予模型更多自主权并打破底层束缚，正如强化学习先驱 Rich Sutton 的“苦涩的教训（The Bitter Lesson）”所揭示的规律：依靠增加算力和通用方法，模型能自然超越人工微调的瓶颈。 > “If you give the agents or you give the AI model more autonomy and if you unshackle it, a bit more and give it more resources. It can actually do a lot of the things that we fine-tune it to do.” —— Jay --- ### `04:44` 数据瓶颈与“苦涩教训”：代码作为通用模态的威力 VLA 模型虽然在架构上也是基于语言模型构建、理论上具备推理与写代码的能力，但整个领域受制于“机器人专属数据匮乏（robotics data bottleneck）”这一瓶颈，多年来进展缓慢。Waddle Labs 的团队强调，核心问题不在于模型架构设计，而在于哪种模态的数据最具杠杆效应。代码正是被网络数据充分验证过的“天选模态（blessed modality）”——其数据极其充沛，通用大模型对代码的逻辑结构有着极深的理解。 将机器人控制这种传统上属于严重“分布外（out of distribution, OOD）”的任务，通过代码映射转化为大模型极度擅长的“分布内（in distribution）”问题，是践行“苦涩教训”的最优解。相比专门训练依赖昂贵真机遥操作（teleoperation）数据的机器人专用基础模型，直接复用吸收了全网代码与计算机使用数据（computer use data）的通用大模型更为高效。模型在上下文学习（in-context learning）、工具调用（tool use）以及自主探索外部环境接口方面的爆发式成长，让模型可以直接写出复杂的控制策略。 > “We want to be bitter lesson pill, right? We want to use all kinds of data. pour in computer use data into our robot models. You want to pour in coding data into our robot models. But when you do that, you just end up getting what we think of as these general purpose LLM agents. And so to me, it's like, okay, why not just build a really good foundational LLM and then use that to control robots rather than training a model that's specifically for robotics and that's more dependent on just robot data.” —— Vincent - VLA 数据瓶颈：机器人物理采集成本极高，导致纯机器人轨迹数据集规模与网络级图文、代码数据存在数量级差距。 - 模态转换思路：将物理控制逻辑翻译为代码调用，将物理空间推理转化为代码执行与工具调用。 - 能力飞跃的核心驱动：上下文学习、Tool-use 强化学习与复杂环境感知，而非底层模型架构的重构。 --- ### `06:56` Code as Policies 与 Voyager：从工具组装到策略生成 讨论随后回溯了“代码即策略（Code as Policies, CaP）”的演进历史。尽管大众记忆可能停留在早期用注释自动补全 Python 代码块的时代，但探索 Agent 组合代码工具的代表性里程碑是 Minecraft 中的 Voyager。Voyager 展示了 Coding Agent 的核心运作逻辑：环境仅提供基础的 Python 内置语法（如 `if`、`for`、`while`、`sort` 等基础工具），模型通过代码将这些基础工具组装成高阶模块（例如创建一个新的脚本文件作为工具），进而沉淀经验并在后续任务中动态调用。 在 2024 年前后，各大实验室全力押注编码模型——从 Anthropic 的 Dario Amodei 到 OpenAI 的 Sam Altman，行业共识是倾注一切算力与资源攻克代码，以期实现自动化机器学习工程师（ML Engineer）并迎来能力腾飞（liftoff）。然而，当时的实验室领导者或许并未完全预料到这一举措的连锁反应：当编码模型足够强大时，它不仅能写软件，甚至能通过编写原生策略去降维取代专门的机器人动作规划，或通过编写 JavaScript 去颠覆传统图形艺术创作。早在 2022 年底 ChatGPT 问世前后，Google DeepMind 提出的 Code as Policies 论文便初现端倪，它通过提供封装好的高层 API（如抓取物体、提升、移动至指定位姿），让语言模型直接生成调用代码来驱动物理硬件。 > “I don't think they were so insightful to know that, oh, if I did this. then we will have LLMs that will be good enough to write policies for robotics. And I can displace all of robotics with actual coding agents. And I can displace artists with coding agents that are writing JavaScript to make beautiful art. I'm not sure that they had that insight, to be honest.” —— Jay - Voyager 的机制：在沙盒游戏中将基础代码操作封装为新函数，通过持久化工具库（Skill Library）实现经验压缩与长程推理。 - 战略溢出效应：行业原本为了自动化软件工程而训练的 Coding Agent，意外成为了控制物理硬件的最优推理底座。 - 早期 CaP 范式：DeepMind 等团队定义基础机器人动作原语（primitives），利用大语言模型依据文本提示词拼装 Python 函数以驱动机械臂。 ### `10:00` 代码智能体的一发即中与上下文学习（ICL）的局限性 编码智能体（Coding Agent）展现出一种令人极为惊艳的跨领域迁移能力：开发者只需将一系列底层控制逻辑以纯粹的 Python 函数形式提供给模型，它便能自主编写调用这些函数的代码，从而驱动物理机器人完成极其复杂的任务。这一过程甚至完全不需要预先收集机器人专有领域的真实数据，智能体依靠在大规模代码语料上沉淀的先验常识，即可“一发即中”（One-shot）地推断出将积木放入碗中应该“先做什么、后做什么”。这种凭借上下文探索（In-context exploration）直接打通物理动作链条的能力，构成了利用通用大语言模型（LLM）控制机器人的基石。 ```mermaid flowchart TD accTitle: 从上下文学习到全参数微调的学习层次 accDescr: 展示了从低成本快速上手的上下文学习与RAG，到参数高效微调LoRA，再到全量SFT与RL的数据与计算阶梯。 subgraph ICL_Layer ["上下文空间 (Context Space)"] A["上下文学习 (ICL)<br/>零训练成本 / 样本饱和快"] B["动态知识检索 (RAG)<br/>带参考书开卷考试模式"] end subgraph Param_Layer ["权重参数空间 (Weight Space)"] C["低秩自适应 (LoRA)<br/>Rank 1/2 到 Rank 10/100"] D["全量 SFT 与强化学习 (RL)<br/>海量数据驱动 / 自动驾驶专用"] end A -->|"知识外置与激活"| B B -->|"技能沉淀与压缩"| C C -->|"参数全量更新"| D ``` 针对学习机制究竟该沉淀在“权重参数”还是停留在“上下文上下文”中，Francois 提出了一套基于“单样本智能效率”（Intelligence per sample）的评估框架。以在保留任务（如 GSM8K）上进行 ICL 实验为例，虽然 ICL 仅靠在上下文中追加“状态-动作-奖励”（State-Action-Reward）元组就能让模型在验证集上极其廉价且快速地提升性能，完全无需消耗 SGD 的浮点运算（FLOPs），但该方式存在三大无法回避的瓶颈： - **非单调式提升（Non-monotonic improvement）**：模型的表现呈现剧烈的上下波动，“它一会儿变差，一会儿变好，反复拉扯得相当剧烈”。 - **容量迅速饱和（Rapid saturation）**：通常塞入 20 到 40 个示例后，模型的表现就会彻底封顶，继续追加示例不再带来任何增益，这直接受限于模型在后训练（Post-training）中所学到的多轮反思（Self-reflection）与上下文利用能力。 - **有效上下文窗口缩水（Context window exhaustion）**：如果一个模型标称训练上下文长度为 100k，其实际有效注意力窗口往往在 50k 左右；一旦超出这一界限，模型注意力无法完全覆盖所有内容，指标反而会快速退化。 “如果你是特斯拉，手里拥有无限的真实驾驶数据，你却试图用上下文学习（ICL）来做自动驾驶，那你在搞什么？开玩笑吗？显然我们绝不会那么干。”Francois 如此总结道。对于拥有无限数据的自动驾驶或特定人形机器人场景，端到端的全量监督微调（SFT）与强化学习（RL）是必然路径；但在极度缺乏物理数据的小样本区间（Low data regimes），ICL 具有无可替代的威力，而下一步的研究前沿正是“如何将 ICL 的经验压缩并固化为工具（Tool use）”。 --- ### `15:00` Harness 架构、技能提炼与元学习的交汇 Waddle Labs 团队深入探讨了其核心架构“Harness”（测试框架/外壳）的运作哲学。Harness 本质上充当了一种领域特异化（Domain specificity）的机制。举例而言，当一台机械臂被部署到生物湿实验室（Wet lab）执行液体采样或容器抓取时，它起初完全可以通过上下文学习逐步摸索规律；但若要将这些即时经验沉淀给未来的智能体使用，最优雅的途径是将探索出的技能编译成具体的确定性代码程序与结构化记忆： “将这些技能编写成代码、将这些记忆形式化地记录下来，本身就是一种沉淀形式。这就像是将过去的经验提炼蒸馏出来，供未来的智能体直接使用。” 这种机制在深层逻辑上呼应了机器学习史上的元学习（Meta-learning）思想：由具备强推理能力的宏观模型负责规划与生成代码，进而“编程”驱动轻量、快速的下层执行模型。下层模型寄宿在 Harness 内高效运转，即便参数极小也能极速处理具体步骤，而上层的通用大模型则专心处理领域特异化的编排与反思。 ```mermaid flowchart LR accTitle: Harness 中的分层控制架构 accDescr: 展现了宏观通用模型通过Harness将动态经验编译为确定性技能，并指挥下层敏捷模型执行控制的流程。 subgraph Meta_Level ["元规划层 (Meta Level)"] M1["通用前沿大模型<br/>(高推理能力 / 负责提炼与纠错)"] end subgraph Harness_Core ["Waddle Harness 枢纽"] H1["编译出的技能库<br/>(Deterministic Programs)"] H2["环境感知与动态插值<br/>(VLM 决策断点)"] end subgraph Exec_Level ["物理执行层 (Execution Level)"] E1["轻量快速执行器 / 机械臂<br/>(低延迟 / 工具调用控制)"] end M1 -->|"编写/编译技能"| H1 M1 -->|"失败反思介入"| H2 H1 -->|"确定性轨迹驱动"| E1 H2 -->|"工具调用修正"| E1 ``` 从理论层面来看，学术界已有研究证明上下文学习在某些数学表征下能等价逼近梯度下降算法，这使得“权重参数空间（Weight space）”与“符号代码空间（Symbolic space）”之间的绝对界限变得愈发模糊，给通过代码外挂实现机器人自适应提供了坚实的理论依托。 --- ### `17:00` 真实机械臂实机演示：直接工具调用 vs. Harness 技能编译 在随后的实机演示视频中，前沿多模态大模型 Astra 正通过外部多路摄像头捕获的视觉流，闭环控制机械臂拾取桌面上的木块并精准投放入碗中。演示中，Astra 并非直接从头生成底层控制脚本，而是通过多轮连续的“工具调用（Tool call）”向机械臂输出末端执行器（End-effector pose）的空间坐标指令。 面对“直接让 Astra 控臂”与“通过 Waddle 的 Harness API 控制”之间的本质差异，团队给出了明确的工程剖析：如果机械臂执行的是高频、机械式的重复任务，让昂贵且存在延迟的通用大模型完全保持在控制闭环中（Human/LLM in the loop）显然不是最优解。更好的方式是将成功摸索出的动作路径“编译为一个固化技能（Compiled skill）”，后续直接调用代码函数运行，不仅执行效率跃升，面对边界异常（Edge cases）时也更加稳健。 演示现场呈现出极其明显的机械停顿与戏剧性的动作延后，这正是受制于 Astra 模型的云端推理延迟。但团队对未来的工程演进持非常乐观的态度： - 目前同级别顶尖前沿大模型（如 Fable 级别的 LLM）的推理延迟正在以“每月降低约 2 倍”的惊人斜率迭代。 - 若此趋势在年内持续，到年底前通用多模态模型完全有希望直接打通机器人的“实时闭环控制（Real-time control）”。 在整个控制流程的拆解中，团队强调了“确定性与变异性并存”的架构：像机械臂向物体逼近、平移等物理轨迹，在经过数次尝试后完全可以被编译为 100% 确定性的代码流水线；而系统的泛化能力则来自于嵌入在控制图谱（Policy graph）关键断点上的视觉语言模型（VLM）——例如在物体识别定位、判定抓取是否失败、以及抓取滑落后触发的自适应重试逻辑等环节，交由 VLM 提供弹性反馈，从而兼顾确定性程序的效率与通用模型的灵活性。 --- ### `19:00` 范式革新：从外延映射（Transduction）走向程序归纳（Program Induction） 在讨论的收尾阶段，核心开发者分享了自己在通用人工智能（AGI）与机器学习底层世界观上的重大转变。这一转变深受 Francois Chollet 在 2018 至 2020 年间发表的经典论文《On the Measure of Intelligence》（论智能的度量）的启发，核心聚焦于“外延转导（Transduction）”与“程序归纳（Program Induction）”的本质路线之争。 “在很长一段时间里，主流机器学习追求的都是从输入 $X$ 到输出 $Y$ 的直接映射（$X \to Y$），但这种做法在深层逻辑上是行不通的——它不仅极其缓慢，而且信息效率极其低下。” 传统的拟合范式要求耗费难以估量的高清标注对 $(X, Y)$ 才能建立映射；当物理世界的数据极其稀缺时，模型就必须依赖海量人工注入的归纳偏置（Inductive bias）。真正的智能并不等同于庞大死板的查找表或端到端黑盒拟合，它迫切需要的是一个高维的“生成器（Generator）”——即不再直接输出动作本身，而是具备自主归纳、合成程序逻辑的能力，利用程序符号来驾驭连续物理空间的多样性。这一认知从底层夯实了“以代码作为策略代理驱动物理机器人”的技术正当性。 ### `20:00` 神经符号系统、代码合成与“柏拉图表征假说” 对话延续了对模型生成代码本质的讨论。嘉宾们指出，给定一组输入 $X$ 和输出 $Y$，模型做的事情就是吐出将 $X$ 映射到 $Y$ 的函数 $f$——这正是代码的定义，也就像在白板面试中给出问题和测试用例后写出对应逻辑一样。传统的程序合成（program synthesis）研究过去之所以步履维艰，是因为它将难题从“寻找映射”转移到了“寻找能够缩小搜索空间并完成映射的正确归纳偏置（inductive biases）”，而人工提炼这种归纳偏置极其困难。现在的突破在于，诸如 Astra 这样的通用模型正在承担这项工作，它们由神经网络生成离散的代码符号，本质上构成了一种现代的“神经符号（neurosymbolic）系统”。 主持人顺势将话题引向 MIT 教授 Philip Isola 的论文《柏拉图表征假说》（The Platonic Representation Hypothesis）。该假说借用柏拉图洞穴寓言中“影子反映现实”的隐喻，提出大量证据表明，随着多模态与语言模型在海量异构数据上以不同策略进行重叠训练，它们内部学到的表征距离正逐渐收敛，趋向于对现实世界某种统一且一致的投射。如果该假说成立，意味着语言模型不仅会持续自我进化，还能将其世界表征直接泛化到机器人控制等物理世界任务中。 嘉宾进一步强调了这一理论在机器人学与语言模型之争中的决定性意义：如果极强语言模型的世界表征与极强机器人模型的世界表征高度重合，那么拥有超强语言模型就等同于拥有了超强的机器人控制基础。嘉宾断言：“如果这是真的，那么这就是苦涩的教训（The Bitter Lesson）最纯粹的体现。”因为此时你只需要一个通用的极强模型，无论其微观架构如何，仅凭规模与通用性就能击败那些专用于特定领域的较弱小模型。 - 程序合成的传统困境：定义合适的归纳偏置往往比拟合数据本身还要艰巨。 - 神经符号机制的重构：神经网络（神经元）负责概率推理，输出具象的符号与可执行程序（代码）。 - 柏拉图表征假说的核心逻辑：不同模态模型规模不断扩大后，会在高维空间中收敛至相同的几何世界表征。 - 对机器人学的推论：无需从零为机器人开发专有的大脑，强大的通用大语言模型天生具备物理控制所需的深层表征。 --- ### `22:20` 空间智能之谜：CAD、电脑使用数据与图形界面的奇妙反哺 主持人提出疑问：为什么像 Astra 这样的最新模型在空间智能（Spatial Intelligence）上突然展现出跨越式的提升？在网络演示中，它已经能娴熟地操作 Blender 制作三维模型，并在各类基准测试中展现出比半年前模型明显上了一个台阶的表现。那些半年前的模型同样受益于算力扩张，同样擅长刷代码和数学题，却唯独没有解开物理空间推理的关卡。 嘉宾们分析认为，Astra 的视觉与空间感知能力来自于预训练数据分布的巨变：它极有可能被喂入了前所未有的大规模计算机使用（Computer-Use）数据和 CAD 三维设计数据。初听起来，“在电脑屏幕上点击光标”似乎与物理世界毫不相干，但嘉宾解释道，当模型观察人类在 Blender 或 CAD 软件中拖动画笔、旋转观察视角（orbit）、调整 3D 物体时，它在被迫学习关于物理空间的基本几何法则——俯视、仰视、左、右、深度遮挡以及视角变换。这种高密度交互数据传授了操纵实体所需的一切核心空间常识。 对话中出现了一段关于人机交互史的绝妙讽刺与哲学闭环：20 世纪 80 年代，施乐帕克研究中心（Xerox PARC）为了让人类更容易使用电脑，刻意模仿物理世界发明了图形用户界面（GUI），引入了桌面、文件和文件夹的概念；随后像 SolidWorks 和 Autodesk 这样的软件也通过模拟物理世界的旋转与建模来适配人类直觉。几十年后，人类在机器人落地物理世界屡屡受挫时，反过来用这些模拟物理世界的 GUI 操作数据去训练 AI，结果 AI 终于借此学会了理解现实物理世界。 ```mermaid flowchart TD accTitle: GUI与机器人空间智能的数据演进闭环 accDescr: 展示从物理世界到图形界面，再通过GUI交互数据反哺机器人控制的演化循环。 subgraph Phase1 ["历史拟物化设计 (1980s)"] A["物理世界实体概念<br>(桌面、文件、三维旋转)"] --> B["数字化图形界面 GUI<br>(Xerox PARC / CAD / Blender)"] end subgraph Phase2 ["大规模人机交互训练 (现代)"] B --> C["沉淀海量交互轨迹<br>(拖拽光标、视角旋转、空间排布)"] C --> D["输入多模态通用模型<br>(Astra 预训练数据)"] end subgraph Phase3 ["空间能力具身迁移"] D --> E["模型内化空间智能<br>(空间遮挡、上下左右、透视推理)"] E --> F["部署至真实机器人控制<br>(通过工具 Harness 操作物理世界)"] end ``` - 空间智能的训练来源：大规模 CAD 模型工程数据，结合用户在 3D 软件中操作视角的交互轨迹。 - GUI 数据对机器人的启发：拖动鼠标改变三维物体的朝向，本质上就是在学习末端执行器（end-effector）在笛卡尔坐标系中的变换规律。 - 具身交互的软件化包装：普林斯顿大学与相关研究证明，若将机器人的控制接口封装得类似于电脑操作工具（例如让智能体像拖动屏幕光标一样指定机器人移动目标），大模型的物理操作表现会显著提升。 --- ### `25:00` 通用机器人的两年倒计时与“睡眠压缩”架构 面向未来，嘉宾与主持人探讨了机器人领域何时会迎来属于自己的质变时刻。RoboCurve 创始人 Jay 透露，在顶级前沿实验室（Frontier Labs）与机器人基模创业公司之间，已经形成了一个高度共识：“在未来两年内甚至更短的时间里，我们就会拥有通用机器人。”这一节点类似于机器人学界的“ChatGPT 时刻”，意味着只要给出一句自然语言指令，机器人就能像一个合格的十几岁少年用双手完成未曾见过的任务，并在完全陌生的环境中展现出极强的零样本泛化能力；然而，整个主流社会对此显然毫无心理准备。 针对 Waddle Labs 正在构建的技术框架，创始人指出当前最大的拦路虎之一在于模型推理延迟。如果在控制闭环的每一步都调用昂贵且庞大的 Astra 模型进行思考，机器人的行动将极其迟缓，在工业和商业场景下完全不具备经济可行性。因此，未来的工程核心在于：如何把 Astra 这种慢速的大上下文语境学习（in-context learning），离线蒸馏、固化为极其高效的低延迟特定技能策略（policy），从而能够以高吞吐量在端侧高频运行。 为了形象地说明这一架构，Waddle Labs 团队打了个生物学比方：自然界中几乎所有智慧生命都需要睡觉，人类在白天的经历被迅速写入海马体，而在深度睡眠期间，大脑通过尖波涟漪（sharp-wave ripples）将短期记忆压缩并转录至大脑皮层，更新长期权重大脑。新一代机器人框架也应当借鉴这一机制以及经典强化学习中的 DAgger 范式与类似 MIT DreamCoder 的思路：白天（在线运行期）在上下文里记录新的状态-动作-奖励轨迹（$(s, a, r)$）；夜晚（离线阶段）系统进入“睡眠”，反思、修剪并重构日常轨迹与技能库，将其蒸馏并更新到更轻量级模型的权重文件中。 - 通用机器人的性能对标：“如果给它任何自然语言指令，它能做到一个能干的青少年用双手能做的事情。” - 核心瓶颈：推理延迟过高，必须解决将高维反思模型沉淀为端侧快速执行策略的工程路径。 - 架构类比：参考 DreamCoder 的程序重构机制，将松散膨胀的执行日志在“睡眠状态”下提炼为模块化的紧凑技能库（growing library of skills）。 - 结语展望：主持人总结道，通用模型对机器人领域的渗透速度将远远超出公众预期，未来数年内人类将见证物理智能领域颠覆性的范式革命。 ## Notable quotes > `25:00` 在未来两年内甚至更短的时间里，我们就会拥有通用机器人。 — Jay > `19:00` 在很长一段时间里，主流机器学习追求的都是从输入 $X$ 到输出 $Y$ 的直接映射（$X \to Y$），但这种做法在深层逻辑上是行不通的——它不仅极其缓慢，而且信息效率极其低下。 — 核心开发者 > `15:00` 将这些技能编写成代码、将这些记忆形式化地记录下来，本身就是一种沉淀形式。这就像是将过去的经验提炼蒸馏出来，供未来的智能体直接使用。 — Waddle Labs 团队 ## Mentioned Philip Isola, Han Mei, Vincent, Jay, Francois, Francois Chollet, Rich Sutton, Dario Amodei, Sam Altman, Waddle Labs, RoboCurve, Google, Google DeepMind, MIT, OpenAI, Anthropic, Tesla, Xerox PARC, Autodesk, SolidWorks, RT-2, Astra, Voyager, Code as Policies, DreamCoder, Blender, Minecraft, GSM8K ## Video description <p>One of the biggest surprises in AI over the last few years has been how well coding agents generalize beyond software. In a recent essay, MIT professor Philip Isola argued that we may be entering the era of robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training.In this episode of Decoded, we're joined by the founders of Waddle Labs and RoboCurve, two of the startups whose work helped drive this realization. They're working at the frontier of using general-purpose models to control robots, and their recent demos helped inspire the growing conversation around robot-use agents. </p><p>Together, we dig into the research behind that idea, from code-as-policies and vision-language-action models to the harnesses and evals needed to make these systems work in the real world.</p> [Source](https://anchor.fm/s/8c1524bc/podcast/play/126339684/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2026-8-26%2F432708848-44100-2-ba9049a9c4a07.mp3)

Source quotes

[25:00] Jay: 在未来两年内甚至更短的时间里，我们就会拥有通用机器人。 [19:00] 核心开发者: 在很长一段时间里，主流机器学习追求的都是从输入 $X$ 到输出 $Y$ 的直接映射（$X \to Y$），但这种做法在深层逻辑上是行不通的——它不仅极其缓慢，而且信息效率极其低下。 [15:00] Waddle Labs 团队: 将这些技能编写成代码、将这些记忆形式化地记录下来，本身就是一种沉淀形式。这就像是将过去的经验提炼蒸馏出来，供未来的智能体直接使用。

Source quotes

[26:15] Jay: when we say general purpose robots, we mean something like if you give any natural language instruction, it can do what a competent teenager could do with their bare hands. [4:47] Vincent: why not just build a really good foundational LLM and then use that to control robots rather than training a model that's specifically for robotics and that's more dependent on just robot data

Original transcript

[0:00] One of the big surprises the last few years has been the generalizability of coding agents across different domains. And now frontier researchers are showing that this includes controlling robots. This led MIT professor Philip Isola to suggest in a recent viral essay that we may be entering the era of robot use agents, where general purpose models could make different robots more capable. So today, Francois and I invited the founders of Waddle Labs and Robocurve, two groups of startups that are working at the very frontier of making robots more capable with LLMs. Thank you. Maybe you guys want to briefly introduce yourselves and just say a little bit about what each of your companies focuses on. I'm Han Mei. I'm from Model Labs together with Vincent. We work on building LLMs that control robots, and we do this by doing two things: building a harness that allows the LLMs to do this very effectively, collecting data, and then using that data to train better LLMs. Hi, I'm Jay, co-founder of RoboCurve. We are an evals company for physical AI. We measure everything, any robot, any model, including LLMs and also classical approaches such as VLA's, Vision Language Action Models and World Action Models. evaluate all kinds of environments like hands, grippers, arms, um humanoids quadruplets all kinds of stuff so in the last few weeks videos from both of you guys went quite viral on twitter and i think inspired i think both of them are actually quoted in that philip isola essay talking about robot use agents um the videos showed things like loms being able to unscrew caps being able to communicate between multiple robots being able to do tasks like uh uncapping a pen for example and so i thought maybe what would be fun is for francois and i to dig in with you guys on some of the research that led to this moment to begin and then maybe we can show some demonstrations of what this can actually do. To start, why don't we talk about some of the early research around transfer between language models and robot control? So I want to talk about some of the early work on pre-training language models for decision making, and then also the RT2 paper, which established the VLA. So maybe, Jay, do you want to tell us a little bit about some of this early work involving being able to use language models to do any kind of robot control? the RT2 paper where they use a pre-trained language model on web text and images and use that to control robots and the interesting thing about that is that it's a fine-tuned version of a language model and instead of outputting English for example they just output um what we call an end effector pose, which is the coordinates that you can then translate into join commands that can control the robots. In some cases, in some sense, it's very similar to what we see now with large language models, just that instead of fine tuning, the models are good enough to just do it out of the box. And this is actually quite similar to the mapping that I have in my head. is kind of like the COT moment. And so like when we did GSM 8K and you said, Susie had $8, she spent five, how much does she have now? Um, it had to output, Like four hash marks and then the answer. and then EOS. There was no ability for it to chain of thought. And so then we allowed it to actually have like, okay, let me go eight minus five is three. And so like, hash, hash, hash, hash, three. And so you allowed it to do this kind of thinking before it actually gave an action or an answer. And similarly now, like VLA is where basically like in RT2 is basically like it has to output an action. There's no like I can't allocate more compute for a more complex task. And then now I have this code chain of thought thing that I can do. And I can say, OK, even even if you were using the example where you're not actually outputting the code, you're just giving it to Astra and let it think, think, think, think, think. And then output an action. Similarly now, like and then with code, it's a. more comagraph complexity or lower comagraph complexity code length to just use code and really compact code to just like, here's code, just go. I totally agree with you on that. I feel like at the end of the day, it's a lot about the better lesson, right? If you give the agents or you give the AI model more autonomy and if you unshackle it, a bit more and give it more resources. It can actually do a lot of the things that we fine-tune it to do. I'm just going to hop in here as well. I think the really interesting thing about the bitter lesson here is that... like VLA's by design architecturally, they're built on top of language models as well, right? They can potentially reason, they can write code. So perhaps the, This lesson here isn't necessarily what architecture built necessarily, but it's what data is most useful, right? Like people have been... sort of nagging at this VLA data bottleneck for years now, and we're seeing very, very slow progress. And so the lesson here is maybe we have a modality of data that we know works very well. And this idea of transferring across different modalities [5:00] which Ham and I are super excited about. Like, how do you take something that is traditionally out of distribution, like robotics, and make it something that's in distribution? Like, maybe the way of harnessing the beta lesson is saying, let's pick a data for which... We know this modality is pretty blessed. There's a lot of data. These models understand it well. And then use this as a way of unlocking a lot of other domains as well. Yeah, and to add to that, the reason why RT2 is such a good model is because it is using a language model that taps into the modality of all the data that the language model is trained on. So all those web images and web text actually improves the VLA compared to just training a specific robotics foundation model DLA approaches. from pre-training, I guess, what's the limitation in the, I guess, action-taking, fine-tuning approach that, you can now get around if you can directly write code. Like you're referring to like a difference in data there. What exactly is that difference in data you're referring to? There's a few things that models have got much better at since RD2. Like one, I mean, they're much better tool use, for example. So, you know, one kind of data is now these models can, well, I mean, they write code much better. So now they can write complex policies as code. They're also much better using tools to explore the kinds of environments they're in, like what arms they have access to. And a lot of this comes from in-context learning. Like the key difference between like these VLAs and like GPT-6 or LLM isn't necessarily the architecture, but kind of the approach you take towards training. We want to be bitter less than pill, right? We want to use all kinds of data. pour in computer use data into our robot models. You want to pour in coding data into our robot models. But when you do that, you just end up getting what we think of as these general purpose LLM agents. And so to me, it's like, okay, why not just build a really good foundational LLM and then use that to control robots rather than training a model that's specifically for robotics and that's more dependent on just robot data. On that note, maybe one of the things that I think, if we could rewind the clock a couple of years, may have been a good sign that we were training is research on CODA's policies, right? Do you guys want to talk a little bit about when the research community refers to CODA's policies, what exactly that means? And I think... It's worth putting this in the context of when this paper came out, which was coding agents just starting to work. I think this was still in the era in which people were like putting comments and auto filling, you know, Python code blocks, which, you know, now we think of as ancient history. But this was like two years ago. So how does code as policies work and how did that inspire some of what you guys are now seeing as possible? I probably rewind back to Voyager where like Voyager was like the one of the first. What is what does a coding agent mean? creation. That's called code. Okay, you have Python built-ins. Let's just say I have only Python built-ins. Those are tools. And I have sort, and I have like if, and I have for, and I have while, and I have all these things. And I need to like use those tools to assemble a new tool called a new... Francois.py, right? And now that's a new tool, and I get to use that. That was Voyager. And yeah, I'm not going to say Voyager was the first one to do this. Voyager was the most popular one to do this for Minecraft. And then they created tools that they could invoke on the fly to help them play the game better and compressed thinking and experience into a new tool that I can later invoke. And then I think because everyone in 2024 had this insight, it was like, okay, if we just pour... You know, Dari and Sam both all... realized if I pour all of my heart and soul and resources and compute and intelligence into getting better at coding, we will have liftoff. Then I can automate the ML engineer. Then I can have liftoff and I'll have all the things. I'll solve all the things. And so then that's what everyone did. And I don't think that they had the insight. I don't think they were so insightful to know that, oh, if I did this. then we will have LMs that will be good enough to write policies for robotics. And I can displace all of robotics with actual coding agents. And I can displace artists with coding agents that are writing JavaScript to make beautiful art. I'm not sure that they had that insight, to be honest. Could we talk a little bit about what some of these early code as policy methods were even able to do? So this is well ahead of us having coding agents that are widely available and using RL methods to actually train those coding agents to be really good at tool calling. What did some of those early ones, maybe even before Voyager, like Coda's policies, which I think the paper came out in 2022 at the end, right around when ChatGPT came out. What was the capability of those compared to what you can see now? Yeah, those CODIS policy papers, especially ones for Google DeepMind team were incredible because what they did is they created these kind of functions like pick up an object, lift up or like move to certain posts. Like they provided this [10:00] list of functions. to a coding agent in the form of literally Python functions. And then the coding agent will be able to write code involving these functions to then control the robot to do very complex hacks. And what was very surprising, what excelled was that coding agents can do this very one shot. They did not need additional robot data in order to work with this code because they were already trained on so much coding data. They already have a sense of what to do first, what to do second, in order to move a block into a bowl, for example. And I think it's this one-shot ability, this in-context exploration ability, that really motivated a lot of later work to continue exploring, including us, to continue exploring how we can apply LLMs to robotics. I know, Francois, you have this framework we've talked about of the various ways that learning can happen. There's going to be learning that's embedded into the weights versus the in-context learning and so on. about where all the research over the last few years has fit into that framework especially the direction it seems to be going i mean i've been working on this experiment and actually i haven't really crystallized it into a paper yet but basically it's like um what is the most efficient way from a intelligence first sample perspective to input a learning a let's say let's in this context a state action result or state action reward tuple back into the policy there's icl learning where like i just like append it this is how most people are are uh using llms they're just like oh no don't do it like this do it like this and it's like okay and then it's still in the context and then i can remember i can iterate and that really only works if you train the llm or at least post train the llm to be able to learn and improve and that self-refines a reflection like it literature from way back that like actually allowed multi-turn um into the training set if you conducts learned but even then there's a limit to how good icl can go so i've done this experiment where like you take an llm that we've trained on and I've held out a task, let's just say GSMAK for simplicity. And then I ICL it and I want to measure on the valset how much it improves on a per sample basis. And it improves greatly very cheaply. It doesn't cost much. I don't have to, there's no SGD, so I don't have any flops. And so very quickly I can adapt and improve on my valset, example, example, example, One is non-monotonic improvement. which is wild. So it gets worse, it gets better, it gets worse, it gets better, like pretty aggressively. Number two is that it caps out very quickly. And so after like 20, 30, maybe 40 examples, it is basically saturated and more examples back into the context does not improve. You're like, you're constrained by the model's ability to intelligently use all of its context. Exactly. From the post-training, how many multi-turns did it actually get to be able to improve and do self-reflection? And then the most important one for Shirley can't context length of the model that it was trained on if it was trained on 100 000 really that means you have a context window of like 50 000 once you exceed 50 000 you don't improve anymore you actually just get worse because the model can't attend over everything um there's a really good example you said which i didn't really think about is like well if i can rag or i can i can load in my act into active memory and this is very prime agent continual harness kind of like thinking where i can pull stuff in um on the fly example similar examples then it's like me like of an exam. or I have an exam with a textbook. I'm gonna do better if I have the textbook to look up as reference. And so that is definitely gonna work. And it's still, and it also scales. I would imagine much better. And then the last two paradigms would be Laura... with rank one or rank two or whatever, Laura with rank two. 10 or rank 100 and then all the way to full SFT RL. And if you're Tesla and you have all the data, infinite data, and you're learning doing self-driving car by ICL, what are you doing? Like, are you kidding? It's like, clearly we wouldn't do that, right? And if you're a figure or something like that. But it's amazing how good in low data regimes, you can do with icl and then even cooler we can compress the icl into tool use into tools or like compression into learnings and that's that's the hierarchy that we haven't really figured out that you guys are probably on the forefront of yeah do you guys want to elaborate on that a little bit because i imagine that's a central part of how you guys think about building waddle there's a few things that are quite interesting here i think to unpack yeah the first thing is we've been sort of thinking about this idea of a harness almost as a form of um like domain specificity like when you for example deploy a robot in a new environment maybe it's in a wet lab and it just needs to do a lot of tests you're picking up for example you can learn a lot of this in context but then one way of consolidating this context for future agents is sort of packaging each of these skills that you learn into specific programs for example like writing out these skills and write out these [15:00] memories is a form of consolidating. It's like a form of distillation from past experience for your future agents to use. So this is quite interesting. The other thing that's quite interesting is it almost seems like there's some relationship to the broader meta-learning literature. throughout in machine learning history. Like you have this sort of bigger model that then programs a smaller model to do things. And one of the very interesting things is that, yes, these smaller models are gonna learn in context. They're sort of wrapped inside the harness and they're doing the task. You can make them maybe smaller, you can make them run faster, as long as the bigger model, can sort of do this domain specialization of your harness quite well so this is something that uh we're pretty interested in right now. And last point as well, I think the really interesting theoretical question is how far in-context learning can get you. And I think it's really anyone's guess as to that. There's papers that show that these models sort of can approximate gradient descent during in-context learning as well. So this line between weight space and symbolic space, I think is a little bit blurry. YC's next batch is now taking applications. Got a startup in you? filling out the app will level up your idea. Okay, back to the video. So what exactly are we watching here? We're watching Astra controlling the arms to pick a block off the table and put it into the bowl. So here Astra is using just these camera inputs, and presumably it's aware of the robot that it's controlling, and it's going to write code that controls the robot at its individual joints. Yes, so it does look at all the cameras. It has all the camera feeds. Instead of code, it's more like a tool call. it sends commands to the robot to control and put the block into the bowl. So in this situation, we're using Astra directly. But if we were using, say, Waddle's harness, what's the kind of difference between directly using Astra to control this versus doing this through Waddle's API? Sometimes directly, like having Astra directly command the post a robot should go to is not the optimal tool to use. If it's a repetitive task, you don't want Astra in the loop. or if it's a task, you've done a similar task before, you should be able to call a skill that has compiled and use that skill to do the task faster and handle edge cases better. So here we can see it seems to be Approaching in on this guy? Oh, very dramatic. Yeah, you can notice that the latency is a bit slow, because we are bottlenecked by the latency of Asherah. If you look at the trends, the latency of these models are improving very rapidly. So one thing that we saw is that for Fable class LLMs, their latency is improving by around 2x. per month, which is very, very fast. If the trends continue, we could get real-time control by end of the year. So now that we saw this task go from end to end, why don't we walk through what that actually entailed? So there's a coding agent in the loop here running this. Yes. What are the steps that the coding agent would have taken? And let's kind of contrast the direct coding agent version versus the waddle harness in the loop version. Yes. So what just from the cameras and it outputted the end effect of pose for the robot to go to. And it does it over multiple turns to complete the task. This is less of code as policies, but more of tool calls. The unscreened part is pretty repetitive, so you can actually automate a lot of that using code, as well as actually approaching the bottle, picking it up, like a lot of these things. or plea. deterministic once you've done a task quite a few times. The interesting part is where you build in the variation into the coders policy graph. And so we have a few points of variation. This can come from, for example, when you detect an object, maybe you use a VLM as part of your tool call. Or when something fails, for example, how do you check that it's failed? How do you then potentially do something else depending on the failure? Like there's kind of more... flexible responses, we tend to put a VLM Um, inside a loop to ensure that, you know, while the coding graph is sort of deterministic, there's points of variation that allow it to generalize. The biggest insight I think I've had a change in worldview on how to perform machine learning and how to get the rest of the distance on AGI was like a lot of conversations I've had with Francis Cholet. And in 2020, maybe even 2018, when he did On the Measure of Intelligence, he talked a lot about transduction versus is just wrong versus program induction. maps from x's to y's. And why is that wrong? It's just slow and it's information inefficient. And so to go from x's to y's, I need a lot of pairs of x's to y's. And if I have a very small amount, then you need a lot of inductive bias. And what you need to act is a good generator to map from x's to y's. So now my theta takes in 3NP [20:00] of X's and Y's, and he emits the function that maps from x to y's yeah and that's what code is and so that's like if you give me a coding interview even whiteboard you say okay here's a coding problem here's some examples okay cool write the function and then we are emitting an f that will map from x to y and and this is quite interesting right because i think i mean this was a lot of the traditional um program synthesis literature and i feel like a lot of reason why those methods didn't work as well was because that inductive bias as you you're shifting the difficulty of the problem from finding that mapping to finding the right inductive biases to map it, to the site. smaller code to then do the mapping but finding that set of inductive biases is super hard but maybe that's what astra is buying us like even in cocci there's a giant shift in the literature from you know to using Astra to write code to map. Maybe that's a cool way to think about it. Which is neurosymbolic. That is neurosymbolic. We have neurons and then they're emitting symbols, which are codes. I think this is actually a pretty good segue to a kind of final topic, which is around, inspired by this paper also from Philip Isola that he titles the Platonic Representation Hypothesis, where Platonic, because as a reference to Plato's cave and the idea that you're seeing shadows of various realities. And I think here the... Point he's making is that there's extensive evidence to show that language models and various representations of different data is actually learn these distance mappings between similar things under different training policies that are overlapping. And so like the idea there would be that as you. train these systems on increasingly large amounts of data, they converge to a kind of consistent mapping of the world. And maybe this implies that we would expect language models to get better and better at things over time that would make them useful for new tasks like robot control. But maybe, I imagine this representation hypothesis is actually pretty central to all of your guys' worldview or your view of your companies. And so maybe why don't you guys tell me a little about how you think about this and also maybe we can use that to make some guesses as to why we think models like Astra seem to be so much better at robot control than previous models and where, you know, make some predictions for where we might be going here. Yeah, I think one way to make this concrete for the robotics models versus language models debate is that the very, very strong language models will have very similar representations of the world with very strong robotics models. And if that's the case, then if you have a really strong language model, you also have a really strong robotics model using if we believe in the platonic representation hypothesis. And if that's true, then Bitter Lesson is the best manifestation of the Bitter Lesson, right? Because you just need one really strong model regardless of architecture and they would be outperforming any specific models that is slightly weaker. The interesting thing that's maybe not entirely intuitive, and maybe I'd be curious to hear you guys all think about, is why specifically it seems like these newest models, specifically Astra, seems to be so much better at spatial intelligence. You know, we've all seen the demos online of, you know, controlling Blender and making great 3D images. Now it seems like, you know, you showed Jay in a benchmark that is like a meaningful step function improvement on certain tasks. models, what has likely changed about OpenAI's approach there that made them so much better compared to models even six months earlier? Like, you know, that still we're benefiting from their lesson and we're still extremely good at code and we're still solving math problems, but haven't quite cracked the spatial intelligence needed here. I think what Astra does incredibly well is its, like, vision capabilities. It was probably pre-trained on way more computer-used data than ever before. It started pre-trained on so much, like, CAD data. And it's like, all of these kinds similar, like, understanding of, like, the physical world as, like, a lot of robot data might. So the computer use data is an interesting point because it's not totally intuitive, I think, why computer use data is useful for understanding the physical world and that, you know, it tends to be like a computer where you're clicking around. Like, say more about why you think that is a big unlock because I totally buy they trained way more computer use data than ever before. Right. I think immediately if you were just training a model for computer use, that would not be able to apply for robotics. But if you feed a computer use data into a big model like Astra, And by computer use data, I mean, like you drag a cursor around on a screen to orbit some cat object in order to design in Blender. This tells you like how to reason about spaces, at least tells you about like top down, left, right, all these concepts that you need to control a robot. Right. And I think that's why this data helps so much for making these LLMs so much better at robot use. I think like from the other direction, like a lot of people in the robotics community were trying more and more different kinds of data as well. teleoperation to a broader range of this data, because it's not just robot data that can teach a model how to use a robot. [25:00] So if we take this to the extreme, it's like, why not feed every kind of data, coding, computer use, egocentric into the same model? I think that's how we get to the most capable robot use agent. I think it's so funny that if you go to 1980s, Xerox PARC or whatever, we literally made the graphical user interface to be more like the physical world so that we could interface with it. And what we ended up doing is building an environment that was actually helpful for robotics to learn how to use the physical world. We have file systems. We have, like, files. We have folders. We have, you know, the GUIs for, like, SolidWorks and, like, Autodesk to, like, spin around stuff to make it similar to the physical world. And then, like... we couldn't get robots to work in the physical world. So we just trained on that. And now it works. Right, exactly. There were incredible papers, I think, from Princeton and also CloudPaste Robotics touches on this. It's like, if you design a right harness, where you make the tools that can face with a robot look like computer use tools, like you have an agent drag a cursor around to control where the robot goes, that improves how well the LLM is able to perform on these physical tasks. Where do you guys see, you know, given like reasonable guesses as to where the base models are going to continue and your guys' own investments in either evaluating these models or building harnesses around them, what do you think is going to be capabilities that we maybe now see as challenging to do, but that are going to be increasingly possible or even trivial very few months from now? Six months ago, even the demonstrations that I've seen you guys post on your Twitter, I think would have been kind of mind-blowing to imagine coming from LLM. There's some consensus within the Frontier Labs and also in the Robotics Foundation models companies that we will have general-purpose robots within the next two years. or even earlier. And this is something that society is probably unaware of or even unprepared for. And when we say general purpose robots, we mean something like if you give any natural language instruction, it can do what a competent teenager could do with their bare hands. It's kind of like the chat GPT moment, but for robotics in terms of capabilities where it can generalize to unseen tasks and unseen environments. For you guys, for the Waddle Labs guys, what does that mean for what you guys are building? I think... We're going to want to take steps to get there in about two years' time. I think there's so many challenges that are very visible. For example, I love people online talk about latency. This is a big problem. If you just have Astra being in the loop thinking at every step, this is really slow. It's not going to be economically useful. So how do you consolidate a first pass by Astra, this contextual learning, into a faster skill or policy that you can then run repeatedly at incredibly high throughput? things to humans will get more and more like this where like the the the optimal thing to do may be to to put a new s ar state action reward back into context it's very quick but then there needs to be some like of go to sleep for a while like almost everything that is intelligent sleeps like tell me an intelligent system that doesn't sleep right in some way and then during sleep compression happens the weird thing that happens from your hippocampus and it shortwave ripples to both lobes there's weird, you know, paths from memories that were compressed throughout the day to train the weights. And similarly, maybe what the right thing to do is similar to Dagger, a data segregation framework in our classic RL, where you're going, you're collecting a bunch of data, and then you're somewhat reflecting on it, and then you're using it to update your weight file. And then you have distillation in from those experiences back into an updated weight file that maybe isn't Astro, but maybe is your own models. It sounds very dream coder-esque, but a build. To take Dreamcoder, for example, right, it had this library of skills, and then during the sleep phase, it basically refactored everything into more compact representation and so on. I mean, we're seeing sort of similar things just happen not as rigidly as before in the space of programs, but now it's maybe like, maybe you want to refactor traces, maybe you want to refactor skills. A lot of these robotic coders policy papers, right, they have this growing library of skills, and it's anyone's guess really how you print them, how you organize them, and things like as Waddle goes forward, like how you manage our growing context of skills, of deployment data, all of that is... going to be quite interesting. I think with that, I think this was an excellent discussion. Thank you so much, Vincent, Hanming, and Jay for being here. I'm very excited for all the incredible robotics advancements I think we're going to see over the next few years. And I think you're totally right, Jay, that I don't think broader society is totally aware of how much is coming. And it's going to be a very incredible few years to come. So thank you so much.