Frontier AI Models Upend Robotics as Startups Bypass Hardware Training

Y Combinator

General-purpose foundation models will achieve human-level robotic manipulation within two years because internet-scale coding and computer-use data build physical world models far faster than costly teleoperation.

AI-assisted summary of Robot-Use Agents: Why General-Purpose Models May Win in Robotics — original source. Check the original for full context.

Source published: 2026-09-26

Key ideas

General-purpose coding agents are bypassing the physical teleoperation data bottleneck by controlling hardware via code generation and tool calling, enabling zero-shot physical manipulation without bespoke robotic fine-tuning. (4:07)
Frontier LLM inference latency is halving roughly every month, bringing real-time, closed-loop visual control of physical actuators within practical engineering reach. (17:31)
Recent step-function improvements in robotic 3D spatial reasoning stem from computer-use and CAD pre-training mixtures, where models absorb coordinate frames and spatial transformations from software like Blender and SolidWorks. (23:37)
Frontier robotics researchers project a 'ChatGPT moment' for physical AI within two years, defining general-purpose autonomy as executing arbitrary natural-language manipulation tasks achievable by a competent teenager. (25:58)
To bypass the economic and latency limits of running large foundation models per motor tick, emerging architectures utilize offline 'sleep' phases inspired by DreamCoder and DAgger to distill interaction traces into fast local policies. (27:06)

Detailed notes

## Background Published by Y Combinator on YouTube on September 26, 2026, this episode of *Decoded* examines the rise of "robot-use agents"—a concept highlighted in a recent essay by MIT professor Phillip Isola on how general-purpose coding models can control diverse robots and learn physical tasks with minimal robot-specific training. The hosts are joined by the founders of Waddle Labs and RoboCurve, two startups whose viral demos helped spark this movement, to unpack the evolution from vision-language-action models and code-as-policies to the real-world harnesses and evaluation benchmarks powering physical AI.

## TL;DR General-purpose coding LLMs are rapidly emerging as powerful robot controllers because outputting parameterized code and structured tool calls provides a low-complexity reasoning scratchpad that bypasses the severe data bottlenecks of traditional Vision-Language-Action (VLA) models. The founders of Waddle Labs and RoboCurve explain how modern agentic harnesses overcome the limits of in-context learning and high inference latency by allowing a frontier model like Project Astra to explore tasks via live camera feeds, then distilling those successful trajectories into fast, reusable software skills. They trace recent step-function leaps in 3D spatial intelligence to pre-training on computer-use and CAD workflows, which aligns with the Platonic Representation Hypothesis that scaled multimodal models naturally converge on a shared internal model of physical reality. Ultimately, the panel predicts a "ChatGPT moment" for general-purpose robotics within two years, unlocked by wake-sleep consolidation cycles that compress slow, deliberative cloud reasoning into high-throughput local motor weights.

## Atmosphere & style The conversation feels like an energetic, high-bandwidth roundtable among deeply technical insiders, blending rigorous machine learning theory with pragmatic startup engineering. The hosts and founders move fluidly between citing foundational academic frameworks—from Rich Sutton's Bitter Lesson and François Chollet's program induction to hippocampal memory consolidation—and dissecting the practical realities of live dual-arm robot demos. As the discussion progresses from early milestones to the sudden emergence of spatial reasoning in frontier models, the mood builds into a palpable sense of momentum and urgency, leaving the listener with the distinct impression that a massive technological inflection point is unfolding just out of public view.

## Bulletin - [04:07](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=247s) General-purpose coding agents are bypassing the physical teleoperation data bottleneck by controlling hardware via code generation and tool calling, enabling zero-shot physical manipulation without bespoke robotic fine-tuning. - [17:31](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1051s) Frontier LLM inference latency is halving roughly every month, bringing real-time, closed-loop visual control of physical actuators within practical engineering reach. - [23:37](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1417s) Recent step-function improvements in robotic 3D spatial reasoning stem from computer-use and CAD pre-training mixtures, where models absorb coordinate frames and spatial transformations from software like Blender and SolidWorks. - [25:58](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1558s) Frontier robotics researchers project a 'ChatGPT moment' for physical AI within two years, defining general-purpose autonomy as executing arbitrary natural-language manipulation tasks achievable by a competent teenager. - [27:06](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1626s) To bypass the economic and latency limits of running large foundation models per motor tick, emerging architectures utilize offline 'sleep' phases inspired by DreamCoder and DAgger to distill interaction traces into fast local policies.

## Key takeaways - [01:50](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=110s) Classical vision-language-action models like RT-2 directly output motor coordinates without intermediate inference, denying the model test-time compute scratchpads analogous to Chain-of-Thought reasoning. - [01:50](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=110s) Formulating robot trajectories as parameterized code represents a dramatically lower Kolmogorov complexity representation of motion compared to verbose joint-by-joint telemetry streams. - [04:07](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=247s) The Bitter Lesson applies to physical AI: scaling general models over abundant web, code, and computer interaction data systematically outpaces collecting scarce real-world robot demonstrations. - [07:40](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=460s) Massive capital expenditure dedicated to building software engineering agents unintentionally supplied the precise programmatic logic and tool-synthesis primitives required to orchestrate physical machines. - [10:41](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=641s) Pure in-context learning plateaus after 20 to 40 demonstrations and degrades beyond long context thresholds, necessitating structured retrieval or symbolic distillation for lifelong robot learning. - [14:22](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=862s) Robot execution harnesses act as a domain-specialization layer where large reasoning models synthesize, debug, and compile modular programmatic tools for smaller, faster models to run locally. - [16:14](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=974s) Direct multimodal model tool-calling provides superior adaptability in unfamiliar scenes, but compiled skill libraries are essential for low-latency, deterministic physical routines. - [19:13](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1153s) The transition from transduction to program induction represents neurosymbolic AI in practice: neural networks supply intuitive world priors that emit explicit symbolic code to execute tasks. - [20:56](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1256s) The Platonic Representation Hypothesis posits that scaling disparate multimodal models forces convergence toward a shared physical world model, making scaled frontier LLMs natural robotic foundation models. - [23:37](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1417s) In a historical reversal, graphical user interfaces originally designed to emulate real-world physical metaphors now serve as digital training curriculums teaching models how to interact with the real world. - [25:58](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1558s) Consensus across frontier labs sets the horizon for human-level teenage manipulation and out-of-distribution physical generalization within two years. - [27:06](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1626s) Robots require a biological-style sleep consolidation cycle using DAgger-style aggregation to compress daytime runtime interaction traces into lightweight local parametric weights. - [28:36](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1716s) Adapting the DreamCoder paradigm to neural robotics requires continuously refactoring, pruning, and managing swelling libraries of generated code-as-policies tools.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=0s) Intro & The Rise of Robot-Use Agents

Over the past few years, one of the most unexpected developments in artificial intelligence has been the sheer extent to which coding agents generalize far beyond pure software engineering. Frontier researchers are now demonstrating that these general-purpose models can be leveraged directly to control physical robots. This shift was recently crystallized by MIT Professor Phillip Isola in a viral essay proposing that robotics is entering the era of "robot-use agents"—paradigms where general-purpose models operate across disparate robot hardware, formulate control policies, and master novel physical tasks with negligible robot-specific fine-tuning. To unpack this breakthrough, the hosts bring together the founders of Waddle Labs and RoboCurve, two startups operating at the bleeding edge of controlling physical hardware with LLMs.

The founders explain their respective wedges in the physical AI landscape: - **Jaime and Vincent (Waddle Labs):** Focused on building LLMs that control robots directly. Their core technical loop involves engineering an operational harness that empowers models to act effectively, collecting rich interaction data from those deployments, and utilizing that data to train even more capable foundation models. - **Jay (RoboCurve):** Focused on evaluation and benchmarking for physical AI. RoboCurve measures performance across virtually any hardware and model paradigm—spanning standard LLM agents, vision-language-action models (VLAs), and world-action models—across varied form factors including hands, parallel-jaw grippers, robotic arms, humanoids, and quadrupeds.

Recently, demonstrations from both Waddle Labs and RoboCurve garnered significant viral attention on social media and were cited directly in Isola's essay. These demos showcased LLMs solving contact-rich and coordinated physical challenges that historically required bespoke robotics engineering—such as unscrewing bottle caps, coordinating multi-robot communications, and uncapping pens.

---

[01:50](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=110s) The Evolution from RT-2 to Code Chain-of-Thought

To understand how modern models control hardware, the discussion tracks back to the foundation laid by Google's RT-2 (Robotic Transformer 2) paper, which introduced the vision-language-action (VLA) approach. RT-2 adapted web-scale vision-language models pre-trained on diverse internet text and images, fine-tuning them to output physical control commands directly. Instead of generating English tokens, the model was conditioned to predict an end-effector pose—cartesian spatial coordinates that downstream inverse kinematics controllers convert into raw robot joint commands. While RT-2 proved that internet-scale pre-training dramatically outperforms models trained purely from scratch on robot data, it locked the model into an immediate, token-level action output.

One of the hosts contrasts this early architecture with the historical inflection point in natural language reasoning: the emergence of Chain of Thought (CoT). In benchmark problems like GSM8K (e.g., "Susie had $8. She spent five. How much does she have now?"), early architectures were forced to emit an immediate answer token (e.g., delimiter hash marks and the final digit `3`) followed by an end-of-sequence token (`EOS`), completely devoid of intermediate inference steps. Once models were allowed to output scratchpad reasoning (`8 - 5 = 3`) before committing to the final answer, performance exploded. Classical VLAs and RT-2 suffer from that exact same restriction: they are compelled to map raw perception straight into an immediate action vector with no computational leeway to deliberate over complex physics or allocate test-time compute. By shifting the output space from raw coordinate tokens to expressive code, models gain an execution scratchpad. Expressing movement as compact, parameterized code also represents a drastically lower Kolmogorov complexity representation of motion compared to verbose joint-by-joint telemetry trajectories.

```mermaid flowchart TD accTitle: Evolution of Robot Control Paradigms accDescr: Comparison between early VLA end-effector prediction and modern Code Chain-of-Thought execution.

subgraph ClassicalVLA ["Classical VLA (e.g., RT-2)"] A["Image & Text Prompt"] --> B["Pre-Trained VLM"] B -->|"Direct Fine-Tuned Output"| C["End-Effector Pose Coordinates"] C --> D["Joint Motor Commands"] end

subgraph CodeAsPolicy ["Modern Robot-Use Agent"] E["Visual & Scene Context"] --> F["General-Purpose LLM"] F -->|"CoT Scratchpad & Reasoning"| G["Parametric Code / Python Script"] G -->|"Harness Execution"| H["Hardware Control & Tools"] end ```

---

[04:07](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=247s) The Bitter Lesson for Robotics

The conversation moves to Rich Sutton's foundational thesis, "The Bitter Lesson"—the principle that general methods leveraging search and compute relentlessly outpace human-crafted domain heuristics. While VLAs are fundamentally built upon underlying language model architectures and theoretically possess the latent capability to reason, they run headlong into a massive robotics data bottleneck. Collecting high-quality teleoperated real-world demonstration data has progressed at an agonizingly slow pace for years. The insight behind robot-use agents is to bypass this physical data scarcity altogether by leveraging a modality where data is already abundant and "blessed": code, web text, and computer interaction.

Instead of hand-designing robotic foundation models that require trillions of expensive physical tokens, teams are recognizing that frontier LLMs already absorb spatial, logical, and tool-use priors from diverse modalities. Pre-training on massive codebases and general web data teaches models how to dynamically inspect unfamiliar environments, leverage available tooling, and construct robust control policies on the fly through in-context learning. Rather than spending astronomical capital building a narrow foundation model tied strictly to proprietary robot embodiments, developers are "bitter lesson pilled": they rely on general-purpose frontier models and design the abstraction layer so the robot appears to the model like any other software environment.

---

[07:40](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=460s) From Voyager to On-the-Fly Tool Creation

Tracing the genealogy of coding agents controlling environments, the discussion looks back at projects like *Voyager* in Minecraft as the precursor to modern Code as Policies. A capable coding agent cannot survive merely on a fixed, static set of predefined functions; it requires proficient tool use and, crucially, autonomous, on-the-fly tool *creation*. In a standard execution environment, a language model begins with elementary Python built-ins—basic logic constructs like loops, conditionals, and primitives. By interacting with the world, the model chains those primitives together into reusable modular scripts (such as creating an autonomous function, humorously dubbed `francois.py`), which compresses complex problem-solving into a repeatable asset stored in its library for later invocation.

This capability drove major AI labs to channel enormous compute, research focus, and capital throughout 2024 into turning LLMs into world-class software engineers. The driving ambition was to automate the machine learning engineer itself to unlock a self-improving recursive research flywheel. Yet, as the speakers emphasize, this intense focus on software coding had an unintended, powerful side effect: in building models that excel at structuring programmatic logic, synthesizing tools, and navigating unknown codebases, the industry inadvertently created models that possess the exact primitives required to orchestrate physical machines.

[08:56](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=536s) From Early Coding Agents to Code as Policies

The conversation begins with a reflection on how unanticipated it was that scaling language models on broad coding datasets would suddenly yield agents capable of writing low-level robot policies or generating complex JavaScript artwork. Looking back at early milestones such as Google DeepMind's "Code as Policies" paper—released in late 2022 alongside the public launch of ChatGPT—the breakthrough centered on exposing basic robot primitives directly to a coding model as Python functions, such as picking up an object, lifting, or moving to a specific 6D pose.

Rather than training bespoke models on large corpuses of robot teleoperation data, the language model acted as an executive planner. It synthesized multi-step Python scripts that chained these primitives together to complete complex manipulation tasks, such as moving a colored block into a target bowl. What astonished researchers was the model's out-of-the-box, one-shot capability: the agent had absorbed enough structural reasoning, temporal order, and commonsense physical logic from software code alone to orchestrate physical movements without needing robot-specific fine-tuning. This zero-shot in-context exploration provided the foundation that spurred the current wave of robot-use agent architectures.

```mermaid flowchart LR accTitle: Code as Policies Architecture accDescr: High-level overview showing how an LLM transforms primitives into physical robot actions. subgraph LanguageModel["General-Purpose Model"] LLM["Coding Agent"] end subgraph ExecutionHarness["Harness & Primitives"] API["Python Primitives<br/>(pick, lift, move_to_pose)"] Code["Generated Policy Script"] end subgraph Hardware["Real World"] Robot["Robot Arm / End-Effector"] end

[10:41](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=641s) The Limits of In-Context Learning vs. Weight Updates

The discussion shifts to a fundamental question: what is the most sample-efficient mechanism to incorporate a learning signal—a state-action-reward tuple—back into a policy? The group analyzes the spectrum of adaptation methods, contrasting purely prompt-driven in-context learning (ICL) with weight-space updates:

``` Fast / Zero Flops Slow / Data-Intensive In-Context Learning (ICL) ───► RAG / Active Memory ───► LoRA (r=1 to 100) ───► Full SFT / RL ```

While appending correction steps into an ongoing prompt context allows an agent to iterate rapidly without stochastic gradient descent (SGD) or extra compute flops, experiments show clear ceilings to this approach: - **Non-monotonic learning:** As more multi-turn examples are added to the prompt, performance often fluctuates erratically, dropping and rebounding rather than improving steadily. - **Rapid saturation:** When tested on holdout benchmark tasks like GSM8K, performance gains plateau after roughly 20 to 40 demonstrations; beyond that, feeding additional examples yields negligible improvement. - **Context length degradation:** A model trained on a 100k context window effectively begins losing attention and degrading past 50k tokens, rendering long unstructured histories counterproductive. - **The data-scale dichotomy:** In massive-scale industrial domains like autonomous driving (e.g., Tesla) or full-body humanoid control (e.g., Figure), relying purely on prompt-level ICL would be completely ill-suited compared to end-to-end reinforcement learning and supervised fine-tuning. However, in low-data regimes, ICL remains uniquely potent if raw experience can be consolidated into reusable tools.

An effective intermediate mechanism is active retrieval (RAG) within an agentic harness. Much like a student consulting a reference textbook during an exam rather than relying strictly on memorization, an agent that dynamically retrieves relevant past interaction traces into working memory scales significantly better than one using a static context window.

[14:22](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=862s) Harnesses as Distillation and Domain Specialization

To bridge the gap between volatile context windows and rigid weight updates, the harness functions as a domain-specialization and meta-learning layer. When a robot is introduced to a new environment—such as a wet lab tasked with manipulating test tubes—it initially relies on in-context exploration. Once a task is executed successfully, the harness packages and distills those learned state-action trajectories into modular, parameterized programs or tools.

Writing out these persistent routines and memories serves as symbolic distillation: past operational experience is compiled into reusable software primitives that future agents can invoke instantly. This creates a two-tier hierarchy echoing classical meta-learning literature: - A larger, highly capable reasoning model designs, debugs, and specializes the operational harness. - A smaller, faster model runs bounded inside that specialized harness to execute high-frequency physical tasks.

Because theoretical research increasingly shows that transformer forward passes during in-context learning can mathematically approximate gradient descent steps, the boundary between updates in "weight space" and programs in "symbolic space" becomes blurred. Distilling context into callable code acts as a tangible form of weightless policy consolidation.

[16:14](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=974s) Live System Demo: Direct Tool Calling vs. Compiled Skills

The panel reviews a live video demonstration of Project Astra directly controlling dual robot arms to grasp a physical block from a table and deposit it into a bowl. In this setup, Astra continuously ingests multiple real-time camera streams of the workstation. Rather than outputting raw Python source code for each motor tick, Astra issues structured tool calls that transmit operational pose targets down to the robot's inverse kinematics and joint controllers.

While direct end-to-end multi-modal model control is feasible, it exposes a critical trade-off between flexible reasoning and operational latency:

In the demo, the physical execution appears cautious and visibly paced because every incremental action step awaits a forward inference pass from Astra. The ideal production architecture therefore does not keep a heavyweight model in the low-level loop indefinitely; once Astra determines the geometric trajectory, the harness compiles that behavior into a high-frequency skill to execute the movement smoothly and rapidly.

[17:31](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1051s) Latency Scaling and the Anatomy of Robot Tool Calling

The discussion opens with an observation about the dramatic reduction in latency for foundational AI systems. Across frontier class LLMs, inference latency has been cutting in half roughly every month—a blistering rate of progress. If this trajectory holds, developers expect full real-time closed-loop control to become viable by the end of the year.

To illustrate what this looks like in practice, the speakers break down how a system like Astra completes a physical manipulation task from start to finish. Rather than synthesizing an entire monolithic script upfront via classic "code as policies," the agent interacts across successive conversational turns. Astra consumes multi-camera image streams from the robot workspace and outputs target end-effector poses for the arm via structured tool calls. The mechanics of the workflow split into two regimes: deterministic execution and reactive fallback. Baseline trajectories—such as homing in on an object, reaching, and basic pick-and-place routines—become highly repeatable once executed a few times and can be automated straight through code. The real engineering leverage lies in handling variation within the policy graph.

```mermaid flowchart TD accTitle: Robot Tool Calling and Policy Graph Architecture accDescr: Diagram illustrating how camera inputs feed into the agent to execute deterministic motions while using VLM calls for variations and recovery.

subgraph Perception["Perception & Supervision"] CAM["Multi-Camera Images"] --> VLM["VLM Reasoning / Tool Call"] end

subgraph PolicyExecution["Policy Graph Execution"] VLM --> DET["Deterministic Motion (Reaching, Gripping)"] DET --> CHECK{"Failure Detected?"} CHECK -- "No" --> NEXT["Next Sequential Step"] CHECK -- "Yes" --> VLM end

subgraph PhysicalWorld["Robot Hardware"] DET --> POSE["End-Effector Pose Target"] POSE --> ROBOT["Robot Arm Execution"] ROBOT -.-> CAM end ```

By embedding a vision-language model (VLM) directly into the harness, the system handles points of variance dynamically. The VLM acts as an evaluator and decision engine: verifying whether an object has been acquired, recognizing unexpected slippage or misalignments, and triggering alternative branches in the policy graph when an initial attempt fails.

- Frontier LLM latency is halving at approximately 2x per month, bringing real-time robotic control within reach. - Astra operates via iterative tool calls, outputting 6-DoF end-effector poses directly based on live camera feeds. - Deterministic steps (approaching, grasping) are encoded into standard execution paths, while edge-case recovery and semantic object verification are routed through the VLM.

---

[19:13](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1153s) Program Induction vs Transduction: The Neurosymbolic Shift

Reflecting on what makes this paradigm viable, the conversation shifts to fundamental machine learning theory, citing François Chollet's influential essay *On the Measure of Intelligence*. Historically, machine learning has relied heavily on transduction: attempting to train a static parameter set $\theta$ that directly maps inputs $x$ to outputs $y$. The group notes why transduction fundamentally struggles with general physical dexterity—it is information-inefficient and requires massive datasets of paired trajectories for every conceivable scenario. Without an immense inductive bias, pure transduction simply fails to generalize.

Program induction flips this dynamic on its head. Instead of learning the mapping directly, the system uses an intelligent generator to produce an explicit program $f$ that performs the transformation:

$$\theta(\{x_i, y_i\}_{i=1}^n) \to f \quad \text{such that} \quad f(x) = y$$

This is identical to how a software engineer approaches a technical interview on a whiteboard: given three or four example input-output pairs, the candidate does not memorize the raw values, but writes a clean function $f(x)$ that resolves the underlying logic. While classical program synthesis struggled for decades because hand-crafting the necessary inductive biases into domain-specific languages was nearly impossible, modern models like Astra bypass this hurdle entirely. Astra acts as a pretrained universal prior that writes symbolic code directly. The speakers emphasize that this is the true realization of neurosymbolic AI: neural networks providing the deep intuitive priors, which then emit symbolic code that reliably executes the downstream task.

---

[20:56](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1256s) The Platonic Representation Hypothesis and Robotics

The conversation connects program induction to the "Platonic Representation Hypothesis," a framework articulated by MIT professor Phillip Isola. Named after Plato's Allegory of the Cave, the hypothesis posits that neural networks trained on radically different tasks, modalities, and objectives converge toward a shared, consistent latent model of the physical reality casting those sensory shadows. As models scale up on vast, multimodal corpora, their internal geometry aligns into an overlapping statistical representation of the world.

This convergence has profound ramifications for physical autonomy. If the Platonic Representation Hypothesis holds, an exceptionally capable general-purpose language model will naturally possess an internal world model that mirrors that of an ideal robotics-specific policy. Consequently, a sufficiently advanced frontier LLM is inherently an advanced robotic foundation model. The founders argue this serves as the ultimate manifestation of Rich Sutton's Bitter Lesson: rather than designing bespoke architectures or collecting narrow, expensive teleoperation datasets for every single robotic form factor, a single general-purpose foundation model operating at massive scale outperforms fragmented, domain-specific models.

- The Platonic Representation Hypothesis asserts that scaling distinct modalities forces convergence toward a shared representation of physical reality. - Specialized robotic models risk being eclipsed by scaled multimodal agents that share this unified spatial model. - The Bitter Lesson implies that general-purpose scale systematically outcompetes bespoke robotic priors.

---

[23:37](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1417s) Computer-Use Data, CAD, and Emergent Spatial Intelligence

The discussion digs into why frontier models—and Astra in particular—have experienced sudden, step-function improvements in spatial intelligence and 3D reasoning over previous iterations. Six months prior, state-of-the-art models excelled at competitive mathematics and coding benchmarks, yet consistently failed at basic spatial reasoning. The critical breakthrough appears to be the aggressive inclusion of computer-use and Computer-Aided Design (CAD) workflows in pre-training and post-training mixtures.

Operating a computer desktop requires an agent to manipulate cursors, interact with spatial viewports, and navigate 2D and 3D canvases. When a model trains on workflows where an operator manipulates SolidWorks, navigates Autodesk suites, or orbits a 3D mesh inside Blender, it absorbs core physical and geometric primitives: perspective transforms, top-down versus profile projections, spatial boundaries, and coordinate systems. While teleoperation and egocentric video capture real human actions, they represent only a slice of visual understanding. Training on computer interaction turns interactive software interfaces into a massive spatial curriculum for physical robotics.

The speakers highlight the delicious historical irony of this dynamic: in the 1970s and 1980s, computer scientists at Xerox PARC deliberately invented the Graphical User Interface (GUI) to mimic physical metaphors—desktops, files, folders, and manipulable spatial objects—so humans could intuitively navigate digital computing. Decades later, because gathering physical robot data remains slow and constrained, developers feed those very same human-oriented virtual environments back into frontier models, effectively using digital interfaces to teach AI how to reach back out into the physical world. Emerging literature from Princeton and projects around clip-based robotics validate that when robot control harnesses are engineered to resemble computer-use interfaces, general models adapt to physical actuators with surprising ease.

[25:58](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1558s) The Timeline for General-Purpose Robotics

The discussion shifts toward the near future of general-purpose models in physical manipulation, prompted by the observation that even having an agent drag a cursor on screen to direct where a robot arm moves significantly boosts the LLM's performance on real-world manipulation tasks. Looking at the speed of progress—where Twitter demos that would have seemed impossible just six months earlier are already routine—the panel considers what capabilities will become trivial over the coming months as foundational models, harness engineering, and evaluation benchmarks evolve in tandem.

There is a growing consensus across frontier AI research labs and robotics foundation companies that society is on the cusp of true general-purpose robots within the next two years, if not sooner. This shift is described as an impending "ChatGPT moment" for robotics—a massive qualitative leap that society at large is currently neither anticipating nor prepared for. Operationally, the panel defines a general-purpose robot not by narrow factory metrics, but by human-scale autonomy: the ability to receive arbitrary natural language instructions and carry out whatever a competent teenager could achieve using their hands, generalizing seamlessly across entirely unseen environments and novel physical tasks.

- Consensus timeline: General-purpose robots within two years or earlier across frontier labs. - Definition of general-purpose: Capable of performing any manipulation task a competent teenager could do with their hands given natural language instructions, generalizing to out-of-distribution tasks and novel environments.

[27:06](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1626s) Latency Bottlenecks and the "Sleep" Phase for Skill Consolidation

For the founders at Waddle Labs, bridging the gap to that two-year horizon requires solving urgent, visible engineering bottlenecks. The most pressing hurdle discussed by the robotics community online is system latency. Relying on an upstream foundation model like Project Astra to think through every single step in an active control loop is far too sluggish and economically unviable for real-world deployments. The key technical challenge is learning how to distill or consolidate that slow, deliberative in-context reasoning pass into a lean, lightning-fast execution policy or motor skill that can run locally at massive throughput.

This dynamic closely mirrors human biological cognition. Putting new state-action-reward $(s, a, r)$ tuples back into working context is fast, but real intelligence requires an offline "sleep" phase where memory consolidation occurs. Biological systems rely on hippocampal sharp-wave ripples projecting out to neocortical lobes, compressing the day's experiences to permanently update synaptic weights. The panel suggests that modern robotics will adopt a similar loop inspired by classic imitation and reinforcement learning algorithms like DAgger (Dataset Aggregation): the agent spends its waking cycle collecting interaction trajectories, engages in reflective synthesis, and distills those raw multi-step experiences into an updated local weight file that runs independently of expensive cloud models.

```mermaid flowchart TD accTitle: Wake-Sleep Skill Consolidation Cycle accDescr: Diagram illustrating how daytime runtime actions are collected and compressed into fast local weights during an offline sleep phase.

subgraph WakeCycle ["Wake Phase: High-Latency Reasoning & Execution"] LLM["Foundation Model / In-Context Reasoner"] -->|"High-latency deliberation"| Actions["Robot Task Execution"] Actions -->|"Record transitions"| Buffer["Experience Buffer (s, a, r traces)"] end

subgraph SleepCycle ["Sleep Phase: Offline Consolidation & Distillation"] Buffer -->|"Hippocampal-style replay"| Reflection["Refactoring & Data Aggregation (DAgger)"] Reflection -->|"Weight distillation"| FastPolicy["Pruned, High-Throughput Motor Policy"] end

FastPolicy -.->|"Deploys back to"| WakeCycle ```

- Latency bottleneck: An upstream reasoning model like Astra in the feedback loop at every frame is too slow and economically prohibitive for production tasks. - Sleep analogy: "Tell me an intelligent system that doesn't sleep in some way." Offline compression turns transient experiential traces into permanent parametric weights. - Algorithmic bridge: DAgger-style aggregation frameworks allow the robot to gather trajectory data during operation, reflect and distill the data offline, and output a compact local model.

[28:36](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1716s) The DreamCoder Paradigm and the Future of Skill Libraries

The panel connects this architecture to the principles of Kevin Ellis et al.'s *DreamCoder* system, which alternated between wakeful problem-solving and an offline sleep phase that refactored discovered programs into reusable, highly compressed subroutines. While DreamCoder applied this concept strictly to symbolic domain-specific languages and program synthesis, the modern robotics equivalent operates across flexible neural spaces: refactoring raw interaction traces, pruning obsolete behaviors, and synthesizing modular skills.

Current "code-as-policies" robotics architectures naturally generate an ever-expanding ladder of tool abstractions and procedural subroutines. As robots encounter increasingly varied real-world scenarios, managing this continuously swelling library—deciding which execution traces to keep, how to prune redundancies, and how to structure long-term deployment data—becomes the central operational hurdle. Closing out the conversation, the host emphasizes that society is largely unprepared for the velocity of this shift, thanking Vincent, Hanmin, and Jay for providing a clear view into how general-purpose foundation models are rapidly redefining physical robotics.

- *DreamCoder* parallel: Translating symbolic sleep-phase refactoring into modern neural trace refactoring and skill abstraction. - Library growth: Code-as-policies setups generate growing ladders of skills that require systematic pruning, organization, and context-window management over prolonged deployments.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=0s) robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training.

> [04:07](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=247s) The Bitter Lesson for Robotics

> [07:40](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=460s) From Voyager to On-the-Fly Tool Creation

> [08:56](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=536s) Code as Policies

> [10:41](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=641s) The Limits of In-Context Learning vs. Weight Updates

> [14:22](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=862s) Harnesses as Distillation and Domain Specialization

> [19:13](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1153s) Program induction flips this dynamic on its head. Instead of learning the mapping directly, the system uses an intelligent generator to produce an explicit program f that performs the transformation

> [20:56](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1256s) The Platonic Representation Hypothesis and Robotics

> [20:56](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1256s) The Bitter Lesson implies that general-purpose scale systematically outcompetes bespoke robotic priors.

> [27:06](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1626s) Tell me an intelligent system that doesn't sleep in some way.

> [28:36](https://www.youtube.com/watch?v=Jv5B5CEaPJI&t=1716s) The DreamCoder Paradigm and the Future of Skill Libraries

## Mentioned Waddle Labs, RoboCurve, Philip Isola, Jaime, Vincent, Jay, Google DeepMind, RT-2, GSM8K, Rich Sutton, Voyager, Tesla, Figure, Project Astra, François Chollet, On the Measure of Intelligence, Plato, SolidWorks, Autodesk, Blender, Xerox PARC, Princeton University, ChatGPT, DAgger, Kevin Ellis, DreamCoder, Hanmin

## Video description One of the biggest surprises in AI over the last few years has been how well coding agents generalize beyond software. In a recent essay, MIT professor Philip Isola argued that we may be entering the era of robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training.

In this episode of Decoded, we're joined by the founders of Waddle Labs and RoboCurve, two of the startups whose work helped drive this realization. They're working at the frontier of using general-purpose models to control robots, and their recent demos helped inspire the growing conversation around robot-use agents.

Together, we dig into the research behind that idea, from code-as-policies and vision-language-action models to the harnesses and evals needed to make these systems work in the real world.

Chapters:

00:00 — Intro + The Rise Of Robot Use Agents 04:07 — The Bitter Lesson for Robotics 07:40 — From Coding Agents to Robot Policies 10:41 — In-Context Learning vs Model Training 14:22 — Building a Harness for Robot Control 17:50 — Turning Robot Actions Into Reusable Skills 20:55 — How AI Models Learn the Physical World 26:05 — How Close Are We To General Purpose Robots?

https://www.waddlelabs.ai/ https://robocurve.org/

Apply to Y Combinator: https://www.ycombinator.com/apply Work at a startup: https://www.ycombinator.com/jobs

[Source](https://www.youtube.com/watch?v=Jv5B5CEaPJI)

中文简报

前沿AI模型颠覆机器人领域，初创企业绕开硬件训练通用基础模型将在两年内实现人类水平的机器人操作，因为互联网规模的代码和计算机使用数据构建物理世界模型的速度远快于昂贵的遥操作。通用代码大语言模型（LLM）正迅速崛起为强大的机器人控制器，因为输出参数化代码和结构化工具调用提供了一种低复杂度的推理暂存空间，从而绕开了传统视觉-语言-动作（VLA）模型的严重数据瓶颈。Waddle Labs和RoboCurve的创始人解释了现代智能体框架如何克服上下文学习与高推理延迟的局限：该框架允许像Project Astra这样的前沿模型通过实时摄像头画面探索任务，随后将这些成功轨迹蒸馏为快速、可复用的软件技能。他们将近期3D空间智能的阶跃式提升归因于在计算机操作和CAD工作流上的预训练，这与柏拉图表征假说（Platonic Representation Hypothesis）相契合，即规模化的多模态模型会自然收敛于对物理现实的共享内部模型。最终，讨论小组预测通用机器人将在两年内迎来“ChatGPT时刻”，这一突破将由“清醒-睡眠”巩固周期推动，把云端缓慢而审慎的推理压缩为高吞吐量的本地电机控制权重。通用代码智能体正通过代码生成和工具调用来控制硬件，从而绕过物理遥操作的数据瓶颈，在无需定制机器人微调的情况下实现零样本物理操作。 (4:07) 前沿大语言模型的推理延迟大约每月减半，使物理执行器的实时闭环视觉控制在工程实践中变得可行。 (17:31) 机器人3D空间推理能力的近期阶跃式提升，源于计算机使用和CAD数据的混合预训练，模型从中吸收了来自Blender和SolidWorks等软件的坐标系与空间变换能力。 (23:37) 前沿机器人研究人员预计物理AI将在两年内迎来“ChatGPT时刻”，并将通用自主性定义为能够执行胜任的青少年所能完成的任意自然语言操作任务。 (25:58) 为克服在每个电机运行节拍中调用大型基础模型带来的经济与延迟限制，新兴架构借鉴DreamCoder和DAgger，利用离线“睡眠”阶段将交互轨迹蒸馏为快速的本地策略。 (27:06)

Reading summary

Frontier AI researchers and startup founders argue that general-purpose coding and multimodal models are bypassing the physical robotics data bottleneck by controlling hardware via code generation and tool calling. Rather than training bespoke models on scarce teleoperation data, this 'robot-use' approach leverages web-scale reasoning, computer-use priors, and offline skill distillation. Code as Policies and Program Induction (1:50) Early vision-language-action (VLA) architectures like Google's RT-2 directly fine-tuned models to predict raw cartesian end-effector coordinates. Panelists contrast this with the shift toward Chain-of-Thought reasoning, arguing that forcing immediate action-vector outputs denies models test-time deliberation. By instead outputting parameterized code or structured tool calls, models gain an execution scratchpad with drastically lower Kolmogorov complexity than raw joint telemetry trajectories. This architecture reflects a transition from transduction to program induction, drawing on François Chollet's intelligence framework. Rather than memorizing direct mappings between visual inputs and motor outputs, the model induces an explicit executable program that handles the task. Modern multimodal models serve as universal neurosymbolic priors that synthesize modular code to orchestrate physical movements across varying hardware setups. The Bitter Lesson and Emergent Spatial Priors (4:07) Collecting high-quality real-world teleoperation data has long created a bottleneck for robotics foundation models. Participants view the rise of robot-use agents as validation of Rich Sutton's 'Bitter Lesson': general methods fueled by web-scale compute consistently outperform narrow, domain-crafted solutions. Under MIT Professor Phillip Isola's Platonic Representation Hypothesis, neural networks trained across diverse modalities converge toward a shared statistical model of reality, meaning scaled multimodal models inherently develop competent internal physical world models. Speakers attribute sudden advances in spatial reasoning to the inclusion of computer-use and Computer-Aided Design (CAD) workflows in training data. Interacting with 2D and 3D canvases across platforms like SolidWorks, Autodesk, and Blender exposes models to coordinate frames, perspective shifts, and spatial boundaries. Digital graphical interfaces originally built as metaphors for human physical desktops now supply the geometric curriculum that enables AI to act back in physical space. Harness Engineering and the Limits of In-Context Learning (10:41) While pure prompt-driven in-context learning (ICL) allows models to adapt without gradient steps, the panel notes clear technical ceilings: learning can be non-monotonic, gains on benchmarks saturate after roughly 20 to 40 demonstrations, and long histories cause attention degradation. Although high-volume industrial domains like autonomous driving or humanoid locomotion require end-to-end training, low-data regimes benefit from intermediary agentic harnesses using active retrieval (RAG). In demonstrations using systems like Project Astra, models issue structured tool calls based on multi-camera streams to output end-effector poses. To prevent slow inference latency from crippling real-time control, engineering harnesses decouple flexible reasoning from routine motion. Deterministic paths run through fast programmatic primitives, while vision-language models step in selectively to verify object acquisition, resolve failures, or navigate branching logic in the policy graph. Wake-Sleep Distillation and General-Purpose Timelines (25:58) Operating upstream foundation models inside a continuous closed loop remains computationally expensive and slow for production tasks. Drawing parallels to biological sleep consolidation and Kevin Ellis et al.'s DreamCoder system, founders at Waddle Labs describe a two-stage architecture: agents use high-latency foundation models during active deployment to gather operational traces, then enter an offline consolidation phase. Algorithms like DAgger aggregate and distill these experiences into compact, high-frequency local weight files or reusable modular tools. The panel notes an emerging consensus across frontier labs that general-purpose manipulation models—defined as systems capable of completing natural language manipulation tasks that a competent teenager could achieve—could arrive within two years. However, managing continuously expanding skill libraries, pruning redundant routines, and handling offline weight consolidation remain central operational challenges. While panelists project human-level dexterity could emerge within two years, the approach's production viability hinges on resolving latency bottlenecks, managing bloated skill libraries, and proving that offline programmatic distillation can reliably substitute for real-time foundational reasoning.

Source quotes

[0:00] robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training. [4:07] The Bitter Lesson for Robotics [7:40] From Voyager to On-the-Fly Tool Creation [8:56] Code as Policies [10:41] The Limits of In-Context Learning vs. Weight Updates [14:22] Harnesses as Distillation and Domain Specialization [19:13] Program induction flips this dynamic on its head. Instead of learning the mapping directly, the system uses an intelligent generator to produce an explicit program f that performs the transformation [20:56] The Platonic Representation Hypothesis and Robotics [20:56] The Bitter Lesson implies that general-purpose scale systematically outcompetes bespoke robotic priors. [27:06] Tell me an intelligent system that doesn't sleep in some way. [28:36] The DreamCoder Paradigm and the Future of Skill Libraries

Original transcript

[0:00] One of the big surprises the last few [0:01] years has been the [0:03] coding agents across different domains. [0:05] And now frontier researchers are showing [0:06] that this includes controlling robots. [0:08] This led MIT Professor Philip Isola to [0:11] suggest in a recent viral essay that we [0:13] may be entering the era of robot use [0:15] agents where general purpose models [0:17] could make different robots more [0:18] capable. So today Francois and I invited [0:20] the founders of Waddle Labs and [0:22] RoboCurve, two groups of startups that [0:24] are working at the very frontier of [0:25] making robots more capable with LLMs. [0:35] Maybe you guys want to briefly introduce [0:37] yourselves and just say a little bit [0:38] about um what each of your companies [0:39] focuses on. [0:40] >> I'm Jaime. I'm from Waddle Labs together [0:42] with Vincent. We work on uh building [0:45] LLMs that control robots and we do this [0:47] by doing two things: building a harness [0:49] that allows the LLMs to do this very [0:50] effectively, collecting data, and then [0:52] using that data to train better LLMs. [0:54] >> Hi, I'm Jay, uh co-founder of RoboCurve. [0:57] We are an eval company for physical AI. [1:01] We measure everything, any robot, any [1:03] model, uh including LLMs and also [1:05] classical approaches such as uh VLAs, [1:08] vision language action models, and world [1:10] action models. And we [1:12] evaluate all kinds of environments like [1:14] hands, um grippers, arms, um humanoids, [1:19] quadrupeds, all kinds of stuff. [1:20] >> So in the last few weeks videos from [1:22] both of you guys went quite viral on [1:24] Twitter and I think inspire I think both [1:25] of them are actually quoted in that [1:27] Philip Isola essay talking about robot [1:29] use agents. Um the videos showed things [1:31] like LLMs being able to unscrew caps, [1:33] being able to communicate between [1:34] multiple robots, being able to do tasks [1:37] like uh uncapping a pen for example. And [1:40] so I thought maybe what would be fun is [1:41] for Francois and I to dig in with you [1:43] guys on some of the research that led to [1:45] this moment to begin and then maybe we [1:47] can also show some demonstrations of [1:48] what this can actually do. To start, why [1:50] don't we talk about some of the early [1:51] research around transfer between [1:54] language models and robot control. So I [1:57] want to talk about um some of the early [1:59] work on pre-training language models for [2:00] decision-making and then also the RT2 [2:03] paper, which established the VLA. So, uh [2:05] maybe Jay, do you want to tell us a [2:07] little bit about some of this early work [2:08] involving being able to use language [2:10] models to do any kind of robot control? [2:11] >> One of the earliest successful [2:14] approaches of using AI on robots is the [2:17] RT2 paper, where they use a pre-trained [2:21] language model on web uh text and images [2:25] and use that to control uh robots. And [2:27] the interesting thing about that is that [2:29] it's a fine-tuned version of uh language [2:33] model and instead of outputting English, [2:36] for example, they just output um [2:39] what we call an end effector pose, which [2:42] is the coordinates that you can then [2:43] translate into joint commands that can [2:46] control the robots. Uh in some cases, [2:49] it's um in some sense, it's very similar [2:51] to what we see now with large language [2:53] models, just that instead of [2:55] fine-tuning, [2:56] the models are good enough to just do it [2:58] off the box. [2:58] >> This is actually quite similar to like [3:00] the mapping that I have in my head is [3:03] kind of like a the COT moment. And so [3:05] like when we did GSM8K and you said, [3:07] "Susie had had $8. She spent five. How [3:09] much does she have now?" Um it had to [3:11] output like four hash marks and then the [3:14] answer. [3:16] And then EOS. [3:17] There was no ability for it to chain of [3:19] thought. And so then we allowed it to [3:21] actually have like, "Okay, let me go 8 - [3:24] 5 is 3." Yeah, [3:26] and so like hash hash hash hash 3. And [3:28] so you allowed it to do this kind of [3:29] thinking before it actually gave an [3:31] action or an answer. Um and similarly [3:34] now, like VLA is we're basically like an [3:37] RT2 is basically like it has to output [3:39] an action. There's no like I can't [3:40] allocate more compute for a more complex [3:42] task. And then now I have this code [3:45] chain of thought thing that I can do. [3:46] And I can say, "Okay, even even if you [3:49] were using the example where you're not [3:50] actually outputting the code, you're [3:51] just giving it to Astra and let it think [3:53] think think think think and then output [3:54] an action. Similarly now like and then [3:57] with code, it's a [3:58] more Kolmogorov complexity or lower [4:01] Kolmogorov complexity code code length [4:03] to just use code and really compact code [4:05] to just like here's the code. Just go. [4:07] >> I totally agree with you on that. I feel [4:09] like at the end of the day, it's a lot [4:11] about the bit of less, right? If you [4:13] give the [4:14] agent or if you give the AI model more [4:16] autonomy and you if you unshackle it a [4:19] bit more and give it more resources, it [4:21] can actually do a lot of things that we [4:24] fine-tuned it to do. [4:25] >> I'm just going to hop in here as well. I [4:27] think the the really interesting thing [4:28] about the bit of lesson here is that um [4:31] like VLAs by design architecturally [4:33] they're built on top of language models [4:35] as well, right? They they can [4:37] potentially reason, they can write code. [4:39] So perhaps the bit of lesson here isn't [4:41] necessarily what architecture you build [4:43] necessarily, but it's what data is most [4:45] useful, right? Like people have been [4:47] sort of nagging at this [4:49] VLA data bottleneck for years now and [4:51] we're seeing very very slow progress. [4:53] And so the lesson here is maybe we we [4:55] have a modality of data that we know [4:56] works very well and this idea of [4:58] transferring across different [4:59] modalities, which Hamming and I are [5:01] super excited about. Like how do you [5:02] take something that's traditionally out [5:04] of distribution like robotics and make [5:05] it something that's in distribution? [5:07] Like maybe the way of harnessing the bit [5:09] of lesson is saying let's pick a data [5:11] for which we know this modality is [5:12] pretty blessed. There's a lot of data. [5:14] These language well these models [5:16] understand it well and then use this as [5:18] a way of uh unlocking a lot of other [5:21] domains as well. [5:22] >> Yeah, and to add to that, the reason why [5:24] RT2 [5:25] is such a good model is because it is [5:28] using a language model that taps into [5:30] the modality of of all the data that a [5:33] language model is trained on. So all [5:35] those um [5:36] web images and web tags actually [5:38] improves the VLA compared to just [5:41] training a specific robotics foundation [5:43] model without any pre-training. [5:45] >> So so you're benefiting in the case of [5:47] RT2, these early VLA approaches, [5:49] from the from pre-training. Um I guess [5:52] what's the limitation in the I guess [5:54] action-taking fine-tuning approach that [5:57] you can now get around if you can [5:59] directly write code. Like you're you're [6:00] referring to like a difference in data [6:02] there. What exactly is that difference [6:04] in data you're referring to? [6:05] >> There's a few things that models have [6:06] got much better at since RT2. Um [6:09] like one, I mean [6:11] they're much better tool use, for [6:12] example. So, you know, one one kind of [6:14] data is now these models can uh well, I [6:16] mean they they write code much better. [6:18] So, now they can uh write complex [6:19] policies as code. They're also much [6:21] better using tools to explore um the [6:23] kinds of environments they're in, like [6:24] what arms they have access to. And a lot [6:26] of this comes from in-context learning. [6:28] >> Like the key difference between like [6:30] these VLAs and or like GPT-6 or LLM [6:32] isn't necessarily the architecture, but [6:34] kind of the approach you take towards [6:36] training. We want to be bitter lesson [6:38] pill, right? Like we want to benefit [6:40] from all kinds of data. We want to pour [6:42] in computer computer-used data into our [6:44] robot models. You want to pour in coding [6:46] data into our robot models. But when you [6:47] do that, you just end up getting what's [6:49] what we think of as like these [6:51] general-purpose LLM agents. And then so, [6:53] to me it's like, okay, why not just [6:55] build a really good foundational LLM and [6:57] I use that to control robots, rather [6:59] than training like a model that's [7:01] specifically for robotics and that's [7:02] more dependent on just robot data. [7:05] >> On that note, maybe one of the things [7:06] that I think um [7:08] if if we could wind the clock a couple [7:09] years, may have been a good sign that we [7:11] were trending in a good direction to [7:12] this, is research on code as policies, [7:14] right? Do you guys want to talk a little [7:15] bit about when the research community [7:17] refers to code as policies, what exactly [7:19] that means? And I think it it's worth [7:21] putting this in the context of when this [7:22] paper came out, um which was coding [7:25] agents just starting to work. I think [7:26] this was still in the era in which [7:28] people were like putting comments and [7:29] auto-filling, you know, Python code [7:31] blocks, which you know, now we think of [7:33] as agent history, but this was like two [7:34] years ago. Um so, yeah, how how does [7:36] code as policies work and how did that [7:38] inspire some of what you guys are now [7:39] seeing as possible? [7:40] >> I'd probably rewind back to Voyager, [7:43] where like Voyager was like the one of [7:44] the first What is What is a coding agent [7:46] mean? Coding agent requires good tool [7:49] use and on-the-fly tool creation. That's [7:52] called code. Okay, you have Python [7:54] built-ins. Let's just say I have only [7:56] Python built-ins. Those are tools and I [7:59] have sort and I have like if and I have [8:01] for and I have while and I have all [8:03] these things and I need to like use [8:04] those tools to assemble a new tool [8:06] called a a new [8:07] francois.py, right? And like now that's [8:10] a new tool and I get to use that. That [8:11] was Voyager and like yeah, I don't know [8:13] I'm not going to say Voyager was the [8:14] first one to do this. Voyager was the [8:15] most popular one to do this for [8:16] Minecraft. And then they created tools [8:18] that they can invoke on the fly [8:20] to help them play the game better and [8:22] compressed thinking and experience into [8:25] a new tool that they can later invoke. [8:27] And then I think because everyone in 20 [8:30] 24 had this insight was like okay, if we [8:33] just pour if you know, Darian and Sam [8:35] both all realized if I pour all of my [8:38] heart and soul and resources and compute [8:41] and intelligence into getting better at [8:43] coding, we will have lift off. Then I [8:45] can automate the ML engineer, then I can [8:48] have lift off and I'll have all the [8:49] things. I'll I'll solve all the things. [8:51] And so then that's what everyone did and [8:52] I don't think that they had the insight [8:54] I don't think they were so insightful to [8:56] know that oh, if I did this, then we [8:59] would have LLMs that would good enough [9:01] to write policies for robotics and I can [9:05] displace all of robotics with actual [9:07] coding agents and I can displace artists [9:10] with like, you know, a coding agents [9:12] that are writing JavaScript to make [9:14] beautiful like art. Like I'm not sure [9:16] that they had that insight to be honest. [9:18] >> Can we talk a little bit about what some [9:19] of these early coders policy methods [9:21] were even able to do? So even you know, [9:22] this is well ahead of us having coding [9:24] agents that are widely available and [9:25] people intuitively understanding how [9:27] they work. It's well ahead of people [9:28] using RL methods to actually train those [9:30] coding agents to be really good at tool [9:31] calling. What did some of those early [9:33] ones maybe even before Voyager like [9:34] coders policies [9:36] um, is which I think the paper came out [9:37] in 2022 at the end, right around when [9:39] ChatGPT came out. What was the [9:41] capability of those compared to what you [9:42] can see now? [9:43] >> Yeah, those, uh, Codas policy papers, [9:46] especially ones from Google DeepMind [9:47] team, were incredible because what they [9:49] did is, uh, they created these kind of, [9:53] um, functions like pick up an object, [9:55] lift up, or like move to certain pose. [9:58] Uh, like they provided this list of [10:00] functions to a coding agent, uh, in the [10:02] form of like literally Python functions. [10:05] Uh, and then the coding agent would be [10:06] able to write code, uh, involving these [10:08] functions to then control the robot to [10:10] do very complex tasks. And what was very [10:12] surprising, or what what this excelled [10:14] was that coding agents can do this very [10:16] one-shot. They did not need additional [10:18] robot data to in order to work with this [10:20] code because they're already trained on [10:21] so much coding data. They already have a [10:23] sense of like how to what what to do [10:25] first, what to do second in order to [10:26] like move a block into a bowl, for [10:28] example. And I think it's this one-shot [10:31] ability, this in-context exploration [10:33] ability, that really motivated a lot of [10:35] later work to continue exploring, [10:37] including us, to continue exploring how [10:38] we can apply LLMs to robotics. [10:41] >> I know Francois, you have this, um, [10:43] framework we've talked about of, you [10:44] know, the various ways that learning can [10:46] happen. There's going to be learning [10:47] that's embedded into the weights versus [10:49] be in context learning and so on. Do you [10:50] want to quickly talk about that and then [10:51] maybe we can [10:52] think about where all the research of [10:54] the last few years has fit into that [10:55] framework, especially the direction it [10:57] seems to be going. [10:57] >> I mean, I've been working on this [10:59] experiment and actually I haven't really [11:01] crystallized it into a paper yet, but [11:02] basically it's like, um, what is the [11:05] most efficient way from a, uh, [11:07] intelligence per sample perspective to [11:10] input a learning, a, let's say let's in [11:13] this context a state action result or [11:15] state action reward tuple back into the [11:18] policy. There's ICL, there is, [11:22] >> in-context learning what we're [11:23] describing. [11:23] >> learning where like I just like append [11:25] it. This is how most people are are are, [11:28] uh, using LLMs. They're they're like, [11:30] "Oh, no, don't do it like this, do it [11:31] like this." And I'm was "Okay." And then [11:32] it's it's still in the context and then [11:34] I can remember and I can iterate. And [11:36] that really only works if you've trained [11:38] the LLM or at least post-trained the LLM [11:40] to be able to learn and improve. And [11:42] that self-refines a reflection like in [11:45] literature from way back that like [11:47] actually allowed multi-turn into the [11:50] training set. If you don't have that, it [11:51] doesn't learn doesn't in context learn. [11:53] But even then there's a limit to how [11:55] good ICL can go. So, I've done this [11:57] experiment where like you take an LLM [12:00] that we've trained on and I've held out [12:02] a task, let's just say GSMA K for for [12:06] simplicity. And and then I ICL it and I [12:09] want to measure on the val set how much [12:11] it improves on a per sample basis. And [12:13] it improves greatly very cheaply. It [12:15] doesn't cost much. I don't have to [12:16] there's no SGD, so I don't have any [12:17] flops. And so, very quickly I can adapt [12:19] and improve on my val set. Example [12:21] example example. [12:23] One is non-monotonic improvement, [12:26] which is wild. So, it gets worse, it [12:27] gets better, it gets worse, it gets [12:28] better like pretty gra- aggressively. [12:31] Number two is that it caps out very [12:33] quickly. And so, after like 20, 30, [12:35] maybe 40 examples, it is basically [12:37] saturated and more examples back into [12:39] the context does not improve. [12:41] >> So, you're like you're constrained by [12:42] the model's ability to intelligently use [12:44] all of its context. [12:45] >> Exactly, from the post-training. How [12:46] many multi-turns did it actually get to [12:49] be able to improve and do [12:50] self-reflection? And then the most [12:51] important one if for surely can't do is [12:54] after it hits the context window context [12:56] length of the model that it was trained [12:58] on. If it was trained on 100,000, really [13:00] that means you have a context window of [13:01] like 50,000. Once you exceed 50,000, you [13:04] don't improve anymore. You actually just [13:05] get worse cuz the model can't attend [13:06] over everything. Um there's a really [13:08] good example you said which I didn't [13:09] really think about is like, "Well, if I [13:11] can rag or I can I can load in my act [13:15] into active memory." And this is very [13:16] prime agent continual harness kind of [13:19] like thinking where I can pull stuff in [13:21] um on the fly example similar examples, [13:23] then it's like me like, "Hmm, I I a I [13:25] have an exam [13:27] or I have an exam with a textbook. [13:29] I'm going to do better if I have the [13:30] textbook to look up as reference. And so [13:32] that is um [13:34] you know, definitely going to work and [13:35] it's still and it also um scales I would [13:38] imagine much better. And then the the [13:40] last two paradigms would be Laura with [13:42] rank one or rank two or whatever. Laura [13:45] with rank [13:47] 10 or rank 100 and then all the way to [13:50] full SFT RL. And if you're Tesla and you [13:53] have all the data, infinite data [13:56] and you're learning doing self-driving [13:58] car by ICL, what are you doing? Like are [14:00] you kidding? It's like clearly we [14:01] wouldn't do that, right? Um and if [14:03] you're figure or something like that. [14:04] But um [14:05] it's amazing how good in the in low data [14:08] regimes uh you can do with ICL and then [14:11] even cooler we can compress the ICL into [14:15] tool use into tools or like compress it [14:17] into learnings. And that's that's the [14:19] hierarchy that we haven't really figured [14:20] out that you guys are probably on the [14:22] forefront of. [14:22] >> Yeah, do you guys want to elaborate on [14:24] that a little bit cuz I imagine that's a [14:25] central part of how you guys think about [14:26] building models. [14:27] >> There's a few things that are quite [14:28] interesting here I think to unpack. [14:29] Yeah, the the first thing is we've been [14:31] sort of thinking about this idea of a [14:33] hardness almost as a form of um [14:37] like domain um specificity. Like when [14:40] when you for example deploy a robot in a [14:42] new environment, maybe [14:43] it's in a wet lab and it just needs to [14:45] do a lot of um test tube picking up for [14:47] example. You can learn a lot of this in [14:49] context, but then one way of [14:50] consolidating this context for future [14:52] agents is sort of packaging each of [14:54] these skills that you've learned into [14:55] specific um programs for example. Like [14:58] writing out these skills and writing out [14:59] these memories is a form of [15:01] consolidating. It's it's like a form of [15:03] distillation um from past experience for [15:05] your future agents to use. So this is [15:07] quite interesting. The other thing [15:08] that's quite interesting is [15:10] um it almost seems like there is [15:13] some relationship to the broader [15:15] meta-learning literature [15:16] uh throughout in machine learning [15:18] history. Like you have this sort of [15:19] bigger model that then programs a [15:21] smaller model to do things. And one of [15:23] the very interesting things is that yes, [15:25] these smaller models are going to learn [15:26] in context. They're sort of wrapped [15:27] inside the harness and they're doing the [15:28] task. You can make them maybe smaller, [15:30] you can make them run faster, as long as [15:32] the bigger model can sort of do this [15:34] domain specialization of your harness [15:36] quite well. So so this is something that [15:38] we're we're pretty interested in right [15:40] now. And last point as well, I think the [15:43] the the really interesting theoretical [15:44] question is how far in-context learning [15:46] can get you. And I think it's really [15:47] anyone's guess as to that. Like there's [15:49] papers that show that, you know, these [15:51] models sort of can approximate gradient [15:53] descent during in-context learning as [15:54] well. So it's [15:56] this this line between weight space and [15:58] symbolic space, I think it's is a little [15:59] bit blurry. [16:00] >> YC's next batch is now taking [16:02] applications. Got a startup in you? [16:05] Apply at ycombinator.com/apply. [16:08] It's never too early, and filling out [16:10] the app will level up your idea. Okay, [16:13] back to the video. [16:14] >> So what exactly are we watching here? [16:16] >> Uh we're watching Ashra controlling the [16:18] arms to pick a block off the table and [16:21] put it into the bowl. [16:22] >> So here Ashra is using just these camera [16:25] inputs, and presumably it's aware of the [16:26] robot that it's controlling, and it's [16:28] going to write code that controls the [16:29] robot at its individual joints. [16:31] >> Yeah, yes, so it it does look at all the [16:33] cameras. It has all the camera feeds. [16:36] Um instead of code, it's more like a [16:38] tool call. So it sends commands to the [16:41] robot to to control and put the block [16:45] into the bowl. [16:46] >> So now in this situation we're using [16:47] Ashra directly, but if we were using say [16:49] Watched Harness, what's the kind of [16:51] difference between directly using Ashra [16:53] to control this versus doing this [16:55] through Watched API? [16:56] >> Sometimes directly, like having Ashra [16:58] directly command the pose the robot [17:00] should go to, is not direct the optimal [17:02] tool to use. Um if it's a repetitive [17:04] task, you don't want Ashra in the loop. [17:06] Maybe you want to write code that can [17:07] run repeatedly very very fast. Or if [17:10] it's a task you've done a similar task [17:11] before, you should be able to call a [17:12] skill that has compiled and use that [17:15] skill to do the task faster and handle [17:17] edge cases better. So, here we can see [17:18] it seems to be [17:20] approaching in on this guy. Oh, very [17:22] dramatic. [17:22] >> Yeah, you can look uh you can notice [17:24] that the latency is a bit slow because [17:26] we are bottlenecked by the latency of [17:28] Astra. Uh but if you look at the trends, [17:31] the latency of these models are [17:33] improving very rapidly. [17:34] >> Right. [17:35] >> So, one thing that we saw is that for [17:36] stable class LLMs, [17:39] their latency is improving by around 2x [17:42] per month. [17:43] >> Which is very, very fast. [17:44] >> Very fast. [17:44] >> Yeah. [17:45] >> Yeah. If the trends continue, we could [17:46] get real-time control by [17:49] uh end of the year. [17:50] >> So, now that we saw this task go from [17:51] end to end, why don't we walk through [17:53] what that actually entailed? So, there's [17:55] a coding agent in the loop here running [17:56] this. [17:57] >> Yes. [17:57] >> Um what are the steps that the coding [17:59] agent would have taken? And let's kind [18:00] of contrast the [18:02] uh direct uh coding agent version versus [18:04] the water harness in the looper. [18:06] >> Yes. So, what just happened was that um [18:08] in a series of turns, Astra received the [18:12] uh images from the cameras and it [18:15] outputted the end effector pose for the [18:17] robot to go to. And it does it over [18:20] multiple turns to complete the task. [18:23] This is less of code as policies but [18:25] more of um tool calls. [18:27] >> The on-screen part is pretty repetitive. [18:29] So, you could actually automate a lot of [18:30] that using code uh as as well as [18:32] actually approaching the bottle, picking [18:34] it up. Like a lot of these things [18:36] um [18:36] are pretty [18:38] uh deterministic once you've done the [18:40] task kind of a few times. Um [18:42] the interesting part is where you're [18:43] building the variation into the code's [18:45] policy graph. Um and so, [18:47] we have a few points of variation. This [18:48] can come from, for example, when you [18:50] detect an object. Maybe you use a VLM as [18:52] part of the tool call. Uh or when [18:54] something fails, for example, how do you [18:56] check that it's failed? How do you then [18:58] um potentially do something else [18:59] depending on the failure? Like these [19:01] kind of more uh flexible responses, we [19:04] tend to put a VLM um [19:07] inside the loop to ensure that, you [19:08] know, while while while the coding graph [19:09] is sort of [19:10] deterministic, there's points of [19:11] variation that allows to [19:13] >> The biggest insight I think I've had a [19:14] change in world view on how to perform [19:18] machine learning and how to get the rest [19:19] of the distance on AGI was like a lot of [19:21] conversations I've had with Francois [19:22] Chollet. And in 2020, maybe even 2018 [19:26] when he did on the measure of [19:28] intelligence, he talked a lot about [19:30] transduction versus is just wrong versus [19:32] program induction. And transduction just [19:34] means I'm learning a function theta that [19:36] maps from x's to y's. And why is that [19:39] wrong? It's just slow and it's [19:40] information [19:42] inefficient. And so to go from x's to [19:44] y's, I need a lot of pairs of x's and [19:46] y's. [19:46] >> Mhm. [19:47] >> And if I have a very small amount, then [19:49] you need a lot of inductive bias and you [19:50] what you need to act is a good [19:53] generator to map from x's to y's. So now [19:55] my theta takes in [19:58] three n n pairs of x's and y's and emits [20:02] the function that maps from x to y. [20:04] >> Yeah, yeah. [20:04] >> And that's what code is. And so that's [20:06] it like if you give me a coding [20:08] interview with a whiteboard, you say, [20:09] "Okay, here's a [20:10] coding problem, here's some examples. [20:11] Okay, cool, write the function." And [20:13] then that we are emitting an f that will [20:15] map from x to y. [20:17] >> And and this is quite interesting, [20:18] right? Because I think I mean this was [20:20] a lot of the traditional program [20:22] synthesis literature. And I feel like a [20:24] lot of reason why those methods didn't [20:25] work as well was because that inductive [20:27] bias as you said, right? It's [20:29] you're shifting the difficulty of the [20:30] problem from finding that mapping to [20:32] find the right inductive biases to map [20:33] it to this like smaller code to then do [20:36] the mapping. But finding that set of [20:38] inductive biases is super hard. But [20:40] maybe that's what Astra is buying us. [20:41] Like even in CogSci, there's a giant [20:43] shift in the literature from, you know, [20:45] specifically neurosymbolic methods to [20:46] just using Astra to write code to map. [20:49] And so [20:50] maybe that's a cool way to think about [20:51] it. [20:51] >> neurosymbolic. That is neurosymbolic. We [20:53] have neurons and then they're emitting [20:54] symbols which are code. [20:56] >> I think this is actually a pretty good [20:57] segue to the kind of final topic which [20:59] is around [21:00] inspired by this paper also from Filippo [21:03] Ola that he titles the Platonic [21:05] Representation Hypothesis, right? Where [21:08] Platonic because uh is a reference to [21:09] Plato's Cave and you know, the idea that [21:11] you know, there you're seeing shadows of [21:13] uh various realities. And I think here [21:15] the [21:16] um point he's making is that um there's [21:19] extensive evidence to show that um [21:22] language models and various [21:23] representations of different data is [21:25] actually learn these uh distance [21:27] mappings between uh similar things under [21:30] different training policies that are um [21:33] overlapping. And so, like the idea there [21:34] would be that as you [21:37] train these systems on increasingly [21:39] large amounts of data, they converge to [21:42] um a kind of consistent mapping of the [21:44] world. And maybe this implies that we [21:46] would expect language models to get [21:48] better and better at things over time. [21:50] That would make them useful for new [21:51] tasks like robot control. But maybe I'm [21:52] I I imagine this representation [21:54] hypothesis is actually pretty central to [21:56] all of your guys's worldview or your [21:58] view of your companies. And so, maybe [21:59] one of you guys tell me a little bit [22:01] about how you think about this. And [22:02] also, maybe we can use that to make some [22:05] guesses as to why we think models like [22:07] Astra seem to be so much better at robot [22:09] control than previous models. And where, [22:11] you know, make some predictions for [22:12] where we might be going here. [22:13] >> Yeah. I think one way to make this [22:14] concrete for the [22:17] uh robotics models versus language [22:18] models debate is that the very, very [22:21] strong language models will have very [22:23] similar representations of the world [22:25] with very strong robotics models. And if [22:28] that's the case, then if you have a [22:29] really strong uh language model, you [22:32] also have a really strong robotics [22:34] model. [22:34] >> Right. [22:34] >> Uh using if we believe in the Platonic [22:37] uh representation hypothesis. And uh if [22:40] that's the true, that then beta lesson [22:42] is that is the best manifestation of the [22:45] beta lesson, right? Because you just [22:46] need one really strong model regardless [22:49] of architecture, and they would be [22:52] outperforming any specific models uh [22:54] that is slightly weaker. [22:56] >> The interesting thing that that's maybe [22:57] not entirely intuitive and maybe I'd be [22:59] curious to hear you guys all think about [23:00] is why specifically it seems like these [23:02] newest models, specifically Astra, seems [23:04] to be so much better at spatial [23:05] intelligence. You know, we've all seen [23:06] the demos online of it, you know, [23:07] controlling Blender and making great 3D [23:09] images. Now, it seems like uh you know, [23:12] U Show J in a benchmark that is like a [23:14] meaningful step function improvement on [23:16] um certain tasks. Um where do you guys [23:18] think the the pre-training and end [23:21] post-training I guess of those models, [23:23] what is likely changed about OpenAI's [23:25] approach there that made them so much [23:26] better compared to models even 6 months [23:28] earlier? Like, you know, that's still [23:30] benefiting from the bitter lesson and [23:31] we're still extremely good at code and [23:32] we're still solving math problems, but [23:34] haven't quite cracked the spatial [23:35] intelligence needed here. [23:37] >> I think what Astra does incredibly well [23:39] is it's like vision capabilities. It was [23:42] probably pre-trained on way more [23:43] computer used data than ever before. [23:45] It's probably pre-trained on so much [23:46] like CAD data. And it's like all of [23:49] these kinds of data probably teach the [23:51] model like similar [23:53] uh like understanding of like physical [23:55] world as like a lot of robot data might. [23:58] >> So, the computer used data is an [23:59] interesting point because it's not [24:00] totally intuitive, I think, why computer [24:02] used data is useful for understanding [24:03] the physical world and that it you know, [24:05] tends to be like a computer where you're [24:06] clicking around. Like, say more about [24:08] why you think that is a big unlock [24:09] because I totally buy they trained on [24:11] way more computer used data than ever [24:12] before. [24:13] >> Right. Um I think immediately if you [24:15] were just training a model for computer [24:16] use, that would not be able to apply to [24:17] robotics. But, if you feed a computer [24:19] used data into a big model like Astra, [24:22] um and by computer used data I mean like [24:23] you drag a cursor around on a screen to [24:25] orbit some CAD object in order to design [24:27] in Blender, this tells you um like how [24:30] to reason about spaces. It at least [24:32] tells you about like top down, left [24:34] right. All of these concepts that you [24:35] need to control a robot, right? Um and I [24:39] think that's why this data helps so much [24:41] for making these LLMs so much better at [24:43] robot use. I think like from the other [24:45] direction, like a lot of people in the [24:46] robotics community were trying more and [24:48] more different kinds of data as well. [24:50] Like, egocentric videos. Kind of went [24:51] from just teleoperation to to like, you [24:54] know, a broader range of this data um [24:56] because it's not just robot data that [24:58] can teach a model how to use a robot. Um [25:00] so if we take this to the extreme, it's [25:02] like why not feed every kind of data, uh [25:04] coding, computer use, egocentric into [25:06] the same model. I think that's how we [25:08] get to the most capable like robot use [25:10] agent. [25:11] >> Yeah, it's so funny that like, you know, [25:12] if you go to 1980s Xerox Park or [25:15] whatever, like we literally made the [25:17] graphical user interface to be more like [25:19] the physical world so that we could [25:20] interface with it. And like what we [25:23] ended up doing is building an [25:24] environment that was actually helpful [25:25] for robotics to learn how to use the [25:27] physical world. [25:29] Right? We we have file systems, we have [25:31] like files, we have folders, we have, [25:33] you know, the GUIs for like SolidWorks [25:35] and like Autodesk to like spin around [25:37] stuff to make it simulate similar to the [25:39] physical world. And then like we [25:42] couldn't get robots to work in the [25:43] physical world, so we just trained it on [25:44] that and then now it works in the [25:46] physical [25:46] >> Right, exactly. There were incredible [25:47] papers, uh I think from Princeton and [25:49] also clip clap base robotics touches on [25:51] this. It's like we just if you design a [25:53] right harness, where you make the tools [25:55] that can face with the robot look like [25:56] computer use tools, like you have a [25:58] agent dragging cursor around to control [25:59] where the robot goes, that improves how [26:01] well the LLM uh is able to perform on [26:04] these physical tasks. [26:05] >> Where do you guys see, you know, given [26:08] like any reasonable guesses as to where [26:10] the base models are going to continue [26:11] improving and your guys' own investments [26:14] in either evaluating these models or [26:16] building harnesses around them? [26:18] What do you think is going to be [26:19] capabilities that we maybe now see as [26:21] challenging to do, but that are going to [26:23] be increasingly possible or even trivial [26:25] for a few months from now? Like, you [26:26] know, 6 months ago even the [26:27] demonstrations that I've seen you guys [26:28] post on your Twitter I think would have [26:29] been kind of mind-blowing to imagine [26:31] coming from an LLM. [26:32] >> There's some consensus within the [26:33] frontier labs and also in the robotics [26:35] foundation [26:36] uh robotics companies that we will have [26:39] general-purpose robots within the next 2 [26:41] years or even earlier. And this is [26:43] something that society is probably [26:45] unaware of or even unprepared for. And [26:48] when we say general-purpose robots, we [26:50] mean something like if you give any [26:51] natural language instruction, it can do [26:54] what a competent teenager could do with [26:56] their hands. It's kind of like the chat [26:58] GPT moment but for robotics in terms of [27:01] capabilities where it can generalize to [27:03] unseen tasks and unseen environments. [27:06] >> For you guys, for the Wild Labs guys, [27:08] what does that mean for what you guys [27:09] are building? I think [27:11] >> we want to take steps to get there in [27:13] about 2 years time. I think there are so [27:14] many challenges that are very visible. [27:16] For example, what what people online [27:18] talk about latency, this is a big [27:19] problem. I I think if you just have [27:21] Astra being in the loop thinking at [27:23] every step, this is really slow. It's [27:24] not going to be economically like [27:26] useful. So, how do you you know, [27:29] consolidate kind of a first pass by [27:31] Astra like this this this contextual [27:33] learning into like a faster skill or [27:36] policy that you can then run repeatedly [27:38] at incredibly high throughput. [27:40] >> I think it's that the mappings to humans [27:43] will get more and more like this where [27:44] like the the the optimal thing to do may [27:48] be to to put a new S AR state action [27:52] reward back into context is very quick, [27:54] but then there needs to be some like of [27:56] go to sleep for a while like almost [27:58] everything that is intelligent sleeps. [28:00] Like tell me an intelligent system that [28:02] doesn't sleep, right? In some way. And [28:04] then [28:05] during sleep compression happens. The [28:06] weird thing that happens from your [28:07] hippocampus and it short wave ripples to [28:10] both lobes and like there's weird you [28:12] know, paths from memories that were [28:13] compressed throughout the day to train [28:15] the weights. And similarly, maybe what [28:17] the right thing to do is similar to [28:18] dagger [28:19] data segregation framework in our [28:21] classic RL where you're you're going, [28:23] you're collecting a bunch of data, and [28:25] then you're somewhat reflecting on it, [28:26] and then you're using it to update your [28:27] weight file, and then you have [28:29] distillation in from those experiences [28:31] back into an updated weight file that [28:33] maybe isn't Astra, but maybe it's your [28:35] own models. [28:36] >> It sounds very Dream Coder [28:37] >> Mhm. [28:37] >> ask. [28:38] >> Mhm. [28:38] >> But a lot of those [28:40] >> specific tools I think that people used [28:41] to build. [28:42] >> To take Dream Coder for example, right? [28:43] It had this library of skills, and then [28:45] during the sleep phase, it basically [28:46] refactored everything into more compact [28:48] representation and so on. I mean, we're [28:50] seeing sort of similar things just [28:52] happen not as rigidly as before in the [28:54] space of programs, but now it's maybe [28:57] like maybe you want to refactor traces. [28:59] Maybe you want to [29:01] refactor skills. A lot of these robotic [29:02] code as policy papers, right? They have [29:04] this growing ladder of skills and it's [29:05] anyone's guess really how you prune [29:07] them, how you organize them and things [29:08] like that. So, yeah, I think as as world [29:11] goes forward, like how you manage a [29:13] growing contact of skills um of [29:15] deployment data, all of that is going to [29:18] be quite interesting. [29:19] >> I think with that, I think this was an [29:21] excellent discussion. Thank you so much [29:22] Vincent, Hanmin, and Jay for being here. [29:24] I'm very excited for all the incredible [29:26] robotics advancements I think [29:28] we're going to see over the next few [29:29] years and I think you're totally right [29:30] Jay that I don't think broader society [29:33] is totally aware of how much is coming [29:35] and it's going to be a very incredible [29:36] few years to come. So, thank you so [29:38] much.