Google DeepMind Deploys Gemini Robotics AI for Direct Motor Control

Power the future of robotics with Gemini — Google for Developers

Google for Developers

Deploying multimodal foundation models with physical common sense eliminates custom computer-vision pipelines by allowing robots to autonomously translate ambiguous human instructions into collision-free motor trajectories.

AI-assisted summary of Power the future of robotics with Gemini — original source. Check the original for full context.

Source published:

Key ideas

  • Google DeepMind introduced Gemini Robotics Embodied Reasoning (ER 1.6), a model fine-tuned on robotics data that provides open-vocabulary perception and spatial reasoning for physical agents. (1:33)
  • Gemini Vision-Language-Action (VLA) models enable direct mapping from streaming camera frames and language prompts to low-level motor joint values without requiring intermediate symbolic code. (11:30)
  • The Gemini Live API enables bidirectional, low-latency conversational audio and video streaming, using function calling to turn natural speech into robotic actuation. (13:30)
  • To ensure physical hardware safety, DeepMind applies a multi-layer 'Swiss cheese' model benchmarked against the Asimov framework, incorporating hospital injury data from the NEISS database and industrial ISO standards. (15:57)

Detailed notes

## Background Published by Google for Developers on YouTube on May 21, 2026, as part of Google I/O 2026, this presentation features Paul Ruiz exploring how Google's multimodal AI bridges digital models and the physical world. The session details how Gemini models leverage advanced spatial awareness for complex robotic perception, how the Live API powers low-latency human-robot interaction, and how developers can utilize simulations, safety benchmarks, and action models.

## TL;DR Paul Ruiz outlines Google DeepMind's strategy for shifting artificial intelligence from on-screen chatbots to embodied physical agents capable of navigating messy, real-world environments. He breaks down the perceive-plan-actuate pipeline, highlighting Gemini Robotics Embodied Reasoning (ER 1.6) for open-vocabulary spatial grounding, physical common sense, temporal video verification, and agentic vision. The presentation further examines how Vision-Language-Action (VLA) models translate high-level task plans directly into low-level motor actuation across diverse robot embodiments. Finally, Ruiz demonstrates interactive natural dialogue via the Gemini Live API, browser-based physical simulation using MuJoCo in Google AI Studio, and multi-layered safety validation informed by the Asimov benchmarks.

## Atmosphere & style Grounded, pragmatic, and maker-centric, the talk carries the earnest enthusiasm of an engineer who loves tinkering with tangible machinery as much as cutting-edge code. Ruiz maintains a steady, instructional cadence, eschewing excessive marketing hyperbole to candidly acknowledge where traditional factory control theory still suffices and where real-world unpredictability demands multimodal intelligence. The pacing moves smoothly from architectural diagrams to concrete developer workflows, giving the session the engaging, reliable feel of a practical engineering workshop.

## Bulletin - [01:33](https://www.youtube.com/watch?v=kCXQce1dtNE&t=93s) Google DeepMind introduced Gemini Robotics Embodied Reasoning (ER 1.6), a model fine-tuned on robotics data that provides open-vocabulary perception and spatial reasoning for physical agents. - [11:30](https://www.youtube.com/watch?v=kCXQce1dtNE&t=690s) Gemini Vision-Language-Action (VLA) models enable direct mapping from streaming camera frames and language prompts to low-level motor joint values without requiring intermediate symbolic code. - [13:30](https://www.youtube.com/watch?v=kCXQce1dtNE&t=810s) The Gemini Live API enables bidirectional, low-latency conversational audio and video streaming, using function calling to turn natural speech into robotic actuation. - [15:57](https://www.youtube.com/watch?v=kCXQce1dtNE&t=957s) To ensure physical hardware safety, DeepMind applies a multi-layer 'Swiss cheese' model benchmarked against the Asimov framework, incorporating hospital injury data from the NEISS database and industrial ISO standards.

## Key takeaways - [00:01](https://www.youtube.com/watch?v=kCXQce1dtNE&t=1s) Transitioning AI from digital software to physical robotics requires embodied reasoning capable of interpreting unstructured, unpredictable environments rather than clean digital data. - [01:33](https://www.youtube.com/watch?v=kCXQce1dtNE&t=93s) Robotics systems follow a perceive, plan, and actuate loop, where Gemini Robotics ER 1.6 acts as the spatial reasoning engine for the perception and planning layers. - [01:33](https://www.youtube.com/watch?v=kCXQce1dtNE&t=93s) ER 1.6's open-vocabulary object detection eliminates the need to build, label, and train custom closed-vocabulary vision models for every unique operational setting. - [04:12](https://www.youtube.com/watch?v=kCXQce1dtNE&t=252s) Embodied reasoning incorporates physical common sense regarding mass, material fragility, and spatial context to prevent mechanical damage and resolve underspecified instructions. - [06:06](https://www.youtube.com/watch?v=kCXQce1dtNE&t=366s) Built on the Gemini 3 Flash backbone, ER 1.6 natively processes multi-image and continuous video streams to serve as a temporal supervisor for task verification and success detection. - [07:23](https://www.youtube.com/watch?v=kCXQce1dtNE&t=443s) Agentic vision enables the model to generate and execute Python code to crop, rotate, and refine degraded or skewed images before making perceptual decisions. - [09:52](https://www.youtube.com/watch?v=kCXQce1dtNE&t=592s) Task orchestration decomposes broad natural language commands into structured function calls, intermediate waypoints, and collision-free trajectories for actuation. - [11:30](https://www.youtube.com/watch?v=kCXQce1dtNE&t=690s) Vision-Language-Action (VLA) models directly predict continuous actuator and motor outputs from visual streams, complementing ER 1.6 in a dual-system high-level and low-level architecture. - [11:30](https://www.youtube.com/watch?v=kCXQce1dtNE&t=690s) Gemini VLA models are designed to be embodiment-agnostic, providing a general-purpose control architecture adaptable across humanoids, quadrupeds, and bipedal rigs. - [13:30](https://www.youtube.com/watch?v=kCXQce1dtNE&t=810s) The Gemini Live API combines low-latency bidirectional voice dialogue with simultaneous video streams, bridging natural human communication and mechanical action via function calling. - [15:06](https://www.youtube.com/watch?v=kCXQce1dtNE&t=906s) Google AI Studio bundles web-based MuJoCo physics simulation templates, enabling rapid, hardware-free prototyping and fail-safe testing of robotic grasping and trajectory plans directly in the browser. - [15:57](https://www.youtube.com/watch?v=kCXQce1dtNE&t=957s) Because physical AI hallucinations can cause tangible damage or injury, DeepMind enforces a multi-layered safety defense spanning semantic intent comprehension, physical force limits, and operational geofencing. - [15:57](https://www.youtube.com/watch?v=kCXQce1dtNE&t=957s) The Asimov safety benchmarks validate autonomous agent behavior against real-world hospital injury data from the NEISS database alongside standardized industrial ISO manufacturing thresholds.

## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=kCXQce1dtNE&t=1s) Bridging Digital AI and the Physical World

Paul Ruiz introduces himself as the developer relations lead for robotics at Google DeepMind. Setting a personal tone, he describes himself at heart as a maker, someone who has always gravitated toward creating tangible things in the physical realm—ranging from low-tech crafts like woodworking and gardening to hands-on tinkering with Internet of Things devices and intricate machinery. This background drives his excitement for DeepMind’s robotics initiative: moving artificial intelligence beyond screen-bound digital utility into physical reality.

While traditional AI models have demonstrated extraordinary power in text editing, code generation, and processing massive datasets, Ruiz explains that stepping into the physical world presents a fundamentally different challenge. In a real-world setting, a robot cannot rely on clean digital inputs; it must deduce that an unexpected discoloration or reflection on a floor might be a liquid spill, or map out collision-free navigational trajectories through an unpredictable, active factory. DeepMind’s goal with Gemini is to transition from static chatbots to embodied physical agents equipped with genuine spatial awareness, allowing them to navigate messy, unstructured human environments rather than sterile, scripted layouts.

As Paul Ruiz emphasizes when outlining DeepMind's broader ambition: “We aren't just teaching machines to see the world, but we're enabling them to perceive it with advanced spatial awareness and understanding.”

[01:33](https://www.youtube.com/watch?v=kCXQce1dtNE&t=93s) Perceive, Plan, Actuate and Gemini Robotics ER 1.6

In robotics architecture, system operations typically follow a three-stage pipeline: perceive, plan, and actuate. Google’s vision-language models (VLMs) directly power the perception and planning layers. DeepMind's specialized model for this domain is Gemini Robotics Embodied Reasoning (ER 1.6). Rather than functioning as a generic large language model, ER 1.6 is fine-tuned specifically on robotics data to serve as the spatial reasoning logic unit for physical machines. In practical deployments, the model handles tasks such as 2D pointing, object localization, grasping point identification, and detecting object states (such as verifying whether a container is closed or sitting on a desk).

```mermaid flowchart TD accTitle: Robotics Perception and Planning Loop with Gemini ER 1.6 accDescr: Diagram illustrating the perceive, plan, actuate flow powered by Gemini Robotics ER 1.6.

subgraph Perception["1. Perception (Gemini ER 1.6)"] A["Sensory Input (Images/Video)"] --> B["Open Vocabulary Grounding"] B --> C["Spatial & State Reasoning"] end

subgraph Planning["2. Planning (Task Orchestration)"] C --> D["High-Level Goal Decomposition"] D --> E["Function Tool Calling"] end

subgraph Actuation["3. Actuation (Hardware Execution)"] E --> F["Low-Level Motor Actions"] end ```

Developers can interact with ER 1.6 directly through Google's Python SDK by initializing the client, specifying the Gemini Robotics ER model ID, passing a prompt, and tuning runtime configurations such as temperature or thinking level. Ruiz contrasts this vision-language approach with traditional computer vision detectors like YOLO, which rely on closed-vocabulary datasets like COCO or ImageNet. With YOLO, an object cannot be detected if it was not explicitly pre-labeled in the training dataset; if a robot needs a specific woodworking tool to cut a dovetail joint, a closed-vocabulary system fails. Gemini Robotics ER 1.6 operates via open-vocabulary object detection, mapping visual representations and natural language into a shared semantic space. Consequently, developers can issue abstract, context-dependent queries—such as asking for the screwdriver needed for a specialty screw, the tool that appears most heavily used, or a machine component that is overheating—without retraining or labeling custom vision pipelines.

Highlighting the engineering payoff of this architectural shift, Paul Ruiz states: “For developers, this is a massive win, as it eliminates the need to build, label, and maintain custom vision models for every unique environment.”

[04:12](https://www.youtube.com/watch?v=kCXQce1dtNE&t=252s) Physical Common Sense and Semantic Reasoning

DeepMind explicitly labels this model "embodied reasoning" rather than a standard vision model because it understands scene physics rather than simply parsing pixel patterns. Ruiz illustrates the limitation of traditional robotics through a common domestic chore: clearing a dining table. A conventional scripted robot tasked with clearing a table might attempt to hoist a heavy serving platter beyond its motor payload limits, inadvertently stripping its mechanical gears or shattering dinnerware. Gemini Robotics ER 1.6 incorporates what Ruiz terms "physical common sense." It evaluates structural mass, material fragility, and spatial relationships, recognizing that it must first transfer food into smaller, manageable containers before lifting, request human assistance, or avoid attempting to lift furniture bolted directly to the floor.

Ruiz notes that this foundational physical comprehension solves an ongoing engineering bottleneck: “It's this kind of practical intuition that has historically been a nightmare to script manually.”

In addition to physical intuition, ER 1.6 brings semantic reasoning to bear on vague or underspecified human requests. When a human offhandedly instructs a robot, "Hey, put this away," traditional systems stall because both "this" and "away" lack concrete definitions. ER 1.6 resolves this ambiguity using ambient visual context and internal thinking steps. If the model sees an operator holding a screwdriver alongside an open toolbox with an empty slot, it reasons that "away" refers to that specific tray. This contextual inference frees developers from authoring fragile, brittle conditional if-else statements for every conceivable edge case, shifting their focus toward establishing high-level operational objectives.

[06:06](https://www.youtube.com/watch?v=kCXQce1dtNE&t=366s) Long-Horizon Temporal Reasoning and Video Understanding

Traditional model workflows evaluate static snapshots—sending a single image to an API endpoint and awaiting a single prediction. However, robotics is fundamentally a temporal discipline where present decisions depend heavily on actions taken seconds prior. Built upon the Gemini 3 Flash backbone, Gemini Robotics ER 1.6 natively ingests multi-image and continuous video streams, enabling long-horizon temporal reasoning. Because visual frames are tokenized chronologically, the model tracks motion dynamics and state changes over time instead of merely assessing static endpoints.

This capability transforms how roboticists implement task verification and success detection. Previously, verifying whether an action succeeded required engineering complex, ad-hoc heuristic checks. With ER 1.6, a developer can supply a 30-second clip of a manipulation attempt and ask whether a mechanical gripper firmly secured an object or let it slip out of grasp. By comparing motion deltas across sequential frames, the model assesses whether physical changes aligned with the intended goal.

Regarding this operational supervisory role, Paul Ruiz notes: “Now, you can simply use the model as a temporal supervisor that watches the robot's work and confirms that the physical state of the world has actually changed in the way that you intended.”

[07:23](https://www.youtube.com/watch?v=kCXQce1dtNE&t=443s) Agentic Vision and Task Orchestration

Even advanced visual reasoning can stumble when encountering poor-quality inputs caused by extreme scale, odd camera angles, rotational skew, or cluttered factory environments. To overcome these sensory obstacles, Gemini Robotics ER 1.6 introduces support for agentic vision. By toggling the SDK's code execution tool, the model is empowered to write and execute programmatic routines that actively transform and refine incoming imagery before making its final deduction.

Ruiz demonstrates this using a circuit board manufacturing line where a robot must inspect and record the serial ID printed on a specific component, such as an ESMT chip: - Standard models frequently struggle to decipher tiny, crowded, or inverted text on densely populated boards. - With agentic vision enabled, ER 1.6 generates Python code to identify the bounding area of the target chip and crops the image to isolate it. - Recognizing that the cropped text remains upside down, the model executes a secondary programmatic transformation to rotate the crop right-side up, ensuring clean character recognition.

Beyond reading micro-text on electronics, this agentic tooling allows roving facility robots to inspect analog dial gauges, detect environmental anomalies, check for floor spills or snow accumulation, and confirm fire doors are latched.

Once perception is established, the system transitions into planning via task orchestration. Developers provide the model with a functional toolbox of available robotic actions—such as opening a gripper, moving to spatial coordinates, detecting objects, or returning to home position. When given a high-level command like "put the blue block in the orange bowl," Gemini ER breaks down the overarching objective into an ordered series of sub-operations, coordinating the entire end-to-end plan.

Explaining how intermediate image transformations work under the hood, Paul Ruiz states: “By enabling the code execution tool with the SDK, the ER model can now generate code for intermediate steps, allowing the models to manipulate those images itself to get a better understanding of content.”

[09:52](https://www.youtube.com/watch?v=kCXQce1dtNE&t=592s) Micro-Planning and the Transition to Actuation

Paul Ruiz explains how the Gemini Robotics ER (Embodied Reasoning) model orchestrates multi-step physical routines. Rather than merely detecting items in isolation, the model evaluates visual scenes—identifying objects like a blue block and an orange bowl—and determines the precise sequence of functional calls required to complete a task: opening the gripper, translating toward the target block, closing the gripper, moving to the designated drop-off zone, opening the gripper to release, and returning to a neutral pose. Beyond this high-level task choreography, the ER model conducts micro-planning by generating intermediate waypoints and collision-free trajectories, calculating exact paths around obstacles rather than simple point-to-point lines.

This planning culminates in actuation—the physical phase where the gripper meets the object. In robotics, actuation involves translating computed plans into motor joint values and torque commands that drive physical hardware. Ruiz acknowledges that traditional industrial manufacturing has little need for generative AI in fixed, repetitive setups:

> “Now, I know the tendency is to say, 'Look at this amazing AI feature that should change everything.' But, in traditional robotics, if you have a machine in a factory repeating the exact same motion 10,000 times a day, standard control theory works just fine.”

The necessity for multimodal AI arises when robots step outside controlled factory floors into unstructured, unpredictable real-world environments. An autonomous sidewalk delivery robot, for instance, cannot rely on static programming when it runs into scenarios absent from its baseline training data, such as a discarded couch obstructing an entire walkway or an unyielding flock of territorial Canadian geese.

- **High-level orchestration:** Decomposes natural language commands into structured function-call sequences (e.g., open gripper, move, close gripper, transport, release, reset). - **Micro-planning:** Generates multi-point waypoint trajectories that dynamically route around physical obstacles. - **Actuation definition:** Transmitting precise numerical values to motors, joints, and actuators to execute physical work.

---

[11:30](https://www.youtube.com/watch?v=kCXQce1dtNE&t=690s) Vision-Language-Action (VLA) Models and Embodiment Agnosticism

To bridge semantic understanding and physical movement in open environments, Google introduced Vision-Language-Action (VLA) models, spearheaded by their flagship Gemini robotics model (available through a trusted tester program). VLAs bypass intermediate symbolic code by directly mapping raw visual camera frames and natural language prompts into low-level motor values. When given a high-level instruction such as “Clean up the desk,” the VLA continuously streams camera inputs to compute and output the exact joint and actuator trajectories needed to manipulate items in real time. For complex, multi-stage assignments, developers can chain models: the Gemini Robotics ER model ingests video to segment a broad goal into discrete visual milestones, which the VLA model then physically executes step by step.

```mermaid flowchart TD accTitle: Dual-System Architecture for Gemini Robotics accDescr: Diagram showing how Gemini Robotics ER provides high-level reasoning and feeds sub-tasks to VLA models for physical actuation.

subgraph HighLevel["High-Level Planning (Gemini Robotics ER)"] A["Visual Feed & Language Prompt"] --> B["Scene Understanding & Task Decomposition"] B --> C["Micro-Planning & Intermediate Waypoints"] end

subgraph LowLevel["Low-Level Execution (Gemini VLA)"] C --> D["Streaming Camera Frames"] D --> E["VLA Joint & Motor Prediction"] E --> F["Hardware Actuation (Motors / Grippers)"] end

F -.->|"Visual Runtime Feedback"| D ```

Because these VLAs are trained on the foundational Gemini backbone, they possess expansive world knowledge and zero-shot reasoning capabilities. Ruiz highlights an experiment where researchers gave their robot a miniature basketball and hoop setup and instructed it to perform a "slam dunk." Despite never receiving training for basketball mechanics, the model recognized the objects' affordances, grasped the ball, and placed it through the rim. DeepMind is leveraging this foundation to achieve fine motor dexterity—such as inserting cords into sockets or performing delicate benchtop assembly—while maintaining an embodiment-agnostic philosophy that spans humanoids, quadrupeds, and bipedal rigs:

> “These models are designed to be embodiment-agnostic, providing a powerful, general-purpose brain that can be adapted to whatever hardware you're working with.”

---

[13:30](https://www.youtube.com/watch?v=kCXQce1dtNE&t=810s) Natural Human-Robot Interaction with the Gemini Live API

Completing the perceive, plan, and actuate loop solves mechanical execution, but it leaves open the challenge of natural interaction between humans and machines. Pre-programmed voice trees struggle in human environments because everyday language is filled with phrasing variations; a robot that only recognizes rigid scripts will freeze or behave awkwardly when greeted or thanked in unexpected ways.

Ruiz highlights the Gemini Live API as the solution that shifts robots from rigid appliances into intuitive interactive partners. The API delivers low-latency, bidirectional audio dialogue, allowing operators to issue casual, context-dependent requests such as, “That screw looks a little loose. Can you tighten it?” Furthermore, the Live API simultaneously ingests a live visual stream from the robot's onboard cameras, allowing users to converse while the model monitors physical progress and updates operational plans on the fly.

Ruiz identifies function calling as the critical link between conversation and physical motion:

> “This is the bridge that allows a natural conversation to turn into precise physical action, opening the door to incredibly fluid human robot interactions.”

- **Bidirectional low-latency audio:** Enables real-time conversational steering without unnatural pauses or rigid phrase matching. - **Multimodal streaming:** Combines audio feeds with continuous camera video to adjust robotic tasks mid-execution. - **Function calling integration:** Translates conversational conclusions and visual observations directly into developer-defined programmatic triggers for hardware control.

---

[15:06](https://www.youtube.com/watch?v=kCXQce1dtNE&t=906s) Rapid Prototyping with AI Studio and Browser-Based MuJoCo Simulation

Testing and refining robotic control code directly on physical hardware can be slow and risky. Ruiz highlights Google AI Studio as an essential developer sandbox, allowing builders to test prompts, refine system instructions, and examine how Gemini models interpret camera imagery from custom hardware without having to repeatedly flash firmware or run new scripts on physical devices.

Within AI Studio's Build section, DeepMind provides ready-to-run web templates that embed the open-source MuJoCo physics engine directly into browser environments. Ruiz demonstrates a virtual robotic arm executing autonomous pick-and-place sequences inside the browser: the arm relies on the Gemini Robotics ER model to visually locate blocks in the 3D simulation and compute grasp targets entirely within software.

> “Like you can see here, where we're running a virtual robotic arm that uses the Gemini Robotics ER model to detect block locations, then performs a pick and place task based on that information without the risk of testing on real hardware, really lending itself to a fail-fast, fail-safe strategy.”

- **Hardware-free iteration:** Test prompting logic, spatial awareness, and perception pipelines without reloading hardware drivers. - **In-browser MuJoCo engine:** Validates physical interactions, grasping dynamics, and kinematics safely inside web-based simulation templates. - **Fail-fast development:** Detects logical failures and trajectory mistakes in virtual environments before deploying code to physical joints and motors.

---

[15:57](https://www.youtube.com/watch?v=kCXQce1dtNE&t=957s) Safety Grounding: The Asimov Benchmarks and Multi-Layer Defense

In pure software applications, an artificial intelligence hallucination results in mild inconveniences like a nonsensical cooking recipe or a syntax error in code. In physical robotics, however, models govern heavy hardware, sharp manipulators, and high-torque motors moving in close proximity to human beings. Ruiz draws a comparison to workshop equipment like a table saw, noting that adding an autonomous agent requires engineering safety from the start:

> “When you add an autonomous brain to a machine, that respect for safety has to be built in from the ground up.”

To address this, Google DeepMind employs the classic "Swiss cheese" safety model. Because no individual algorithmic or mechanical guardrail is completely foolproof, developers stack overlapping layers across three key tiers: semantic comprehension (evaluating the intent and context of instructions), physical constraints (force limits, speed caps, and obstacle avoidance), and operational boundaries (emergency stops and workspace geofencing).

To empirically validate these defensive layers, DeepMind established the Asimov safety benchmarks. These tests move beyond theoretical safety by incorporating real-world accident data from the National Electronic Injury Surveillance System (NEISS) database, which aggregates hospital injury records to provide the model with grounding in physical common sense. Furthermore, the benchmarks incorporate established industrial ISO standards to ensure autonomous agents respect the identical operational and protective thresholds enforced on modern factory floors. Ruiz concludes by inviting developers to explore Google DeepMind's documentation, experiment with GitHub cookbooks, and apply for the Gemini robotics trusted tester program.

- **Swiss cheese defense model:** Layering semantic, physical, and operational barriers so vulnerabilities in one safety layer are caught by the next. - **NEISS database grounding:** Hospital injury surveillance data used to teach models common-sense awareness of physical human hazards. - **Industrial ISO compliance:** Benchmarks engineered to reflect standardized manufacturing and collaborative robot safety requirements. - **Developer onboarding:** Accessible documentation, code samples, and trusted tester signups available via the Gemini robotics portal.

## Notable quotes > [00:01](https://www.youtube.com/watch?v=kCXQce1dtNE&t=1s) We aren't just teaching machines to see the world, but we're enabling them to perceive it with advanced spatial awareness and understanding. — Paul Ruiz

> [09:52](https://www.youtube.com/watch?v=kCXQce1dtNE&t=592s) In traditional robotics, if you have a machine in a factory repeating the exact same motion 10,000 times a day, standard control theory works just fine. — Paul Ruiz

> [15:57](https://www.youtube.com/watch?v=kCXQce1dtNE&t=957s) When you add an autonomous brain to a machine, that respect for safety has to be built in from the ground up. — Paul Ruiz

## Mentioned Paul Ruiz, Google, Google DeepMind, Google for Developers, Google I/O 2026, Gemini, Gemini 3 Flash, Gemini Robotics Embodied Reasoning (ER 1.6), Gemini Live API, Google AI Studio, MuJoCo, Python SDK, YOLO, COCO, ImageNet, Asimov safety benchmarks, National Electronic Injury Surveillance System (NEISS), ISO

## Video description Discover how Google’s multimodal AI bridges the cloud and physical worlds. Explore how the latest Gemini models utilize advanced spatial awareness to help robots perceive complex environments. Learn about the Live API's capabilities for natural, low-latency human-robot interaction, and explore the essential ecosystem for simulation, safety, and action models.

Resources: Gemini Robotics code samples → https://goo.gle/3PnFTOX Gemini Robotics overview docs → https://goo.gle/4tmD52H Gemini Live API overview → https://goo.gle/4tjVLjz MuJoCo website → https://goo.gle/42ghv4U AI Studio Bundled Robotics → https://goo.gle/4tfdR68

Speakers: Paul Ruiz

Watch the AI sessions from Google I/O 2026 → https://goo.gle/AI-at-IO26

Subscribe to Google for Developers → https://goo.gle/developers

#GoogleIO

Event: Google I/O 2026

Products Mentioned: AI/Machine Learning

[Source](https://www.youtube.com/watch?v=kCXQce1dtNE)

Reading summary

Google DeepMind is adapting its multimodal Gemini models into embodied spatial-reasoning engines and vision-language-action controllers to help robots navigate messy, unstructured human environments. Developer relations lead Paul Ruiz argues that grounding artificial intelligence in physical common sense and temporal video understanding replaces brittle, hand-scripted computer vision with flexible perception and planning. Open-Vocabulary Grounding and Embodied Physical Reasoning (1:33) Traditional robotics vision relies on closed-vocabulary detectors like YOLO, which cannot detect objects outside their pre-labeled datasets. DeepMind's Gemini Robotics Embodied Reasoning (ER 1.6) fine-tunes vision-language models on robotics data, mapping visual inputs and natural language into a shared semantic space. This allows developers to query context-dependent states without training custom pipelines—such as locating worn tools, identifying overheating components, or determining whether a container is closed. Temporal Reasoning and Agentic Image Transformation (6:06) Because physical tasks evolve over time, ER 1.6 uses continuous multi-image and video streams to monitor state changes. Rather than assessing static snapshots, the model tracks sequential motion frames to act as a temporal supervisor, verifying whether actions succeeded—such as detecting if a gripper securely lifted an item or let it drop. Agentic Vision and Dual-System Actuation (7:23) When sensory inputs are degraded by poor lighting, small scale, or awkward angles, ER 1.6 uses code execution to crop, rotate, and enhance imagery dynamically before reading serial numbers or inspecting gauges. For physical execution, Ruiz explains that standard control theory suffices for fixed, repetitive factory routines. In unstructured spaces, however, systems combine high-level ER planning with Vision-Language-Action (VLA) models. Conversational Interaction and Multi-Layer Safety Benchmarks (13:30) To move beyond rigid voice trees, the Gemini Live API pairs bidirectional, low-latency audio with simultaneous video feeds, using function calling to convert casual conversational requests into hardware commands. For prototyping, Google AI Studio embeds the MuJoCo physics engine into browser templates, allowing builders to validate pick-and-place trajectories safely before deploying code to physical joints. Because physical AI failures risk severe real-world injury and damage, Ruiz notes that autonomous deployment relies on defense-in-depth safety layers, combining semantic filtering, physical force limits, and operational stops evaluated against hospital injury databases and industrial ISO standards.

Source quotes

[0:01] Paul Ruiz: We aren't just teaching machines to see the world, but we're enabling them to perceive it with advanced spatial awareness and understanding. [9:52] Paul Ruiz: In traditional robotics, if you have a machine in a factory repeating the exact same motion 10,000 times a day, standard control theory works just fine. [15:57] Paul Ruiz: When you add an autonomous brain to a machine, that respect for safety has to be built in from the ground up.

Original transcript

[0:01] [music] [0:06] >> Hey all, I'm Paul Reese and I'm the [0:07] developer relations lead for robotics at [0:10] Google DeepMind. [0:11] For those of you that don't know me, I [0:13] would generally describe myself as a [0:14] maker. Whether it's something low-tech [0:17] like gardening and woodworking or more [0:19] advanced projects involving the Internet [0:21] of Things and complex machines, I've [0:23] always preferred doing things in the [0:24] real world, which is why I'm very happy [0:27] to be able to tell you about all the [0:29] work we're doing in robotics here at [0:31] DeepMind. [0:32] All right, so typically AI has existed [0:34] within the digital realm. It has been [0:36] amazing for things like editing text, [0:38] analyzing massive data sets, or writing [0:41] code. But the physical world is a whole [0:43] other beast. [0:45] For a robot, understanding that a [0:47] discoloration and reflection on the [0:49] floor is potentially a spilled liquid or [0:52] figuring out the exact trajectory needed [0:54] for mapping a busy factory has [0:56] historically been a massive challenge. [0:59] DeepMind has been working to bridge that [1:01] gap by using Gemini models to move from [1:03] static chatbots to physical agents. [1:06] We aren't just teaching machines to see [1:08] the world, but we're enabling them to [1:10] perceive it with advanced spatial [1:12] awareness and understanding. [1:14] We want robots that can navigate [1:16] unstructured and messy environments [1:18] rather than only predictable and [1:20] organized scenes while being able to [1:22] interact naturally with humans and [1:24] perform complex tasks. [1:26] Okay. [1:27] That's all good, but anyone can say that [1:29] they're doing great things. So, let's [1:31] actually look at this in more detail. [1:33] In robotics, we usually follow a [1:35] perceive, plan, actuate flow. [1:38] Google's vision language models are [1:40] perfect for those first two stages, [1:42] including the latest version of Gemini [1:43] robotics embody reasoning or ER 1.6. [1:48] Unlike a general LLM, the ER model is [1:51] fine-tuned on robotics data, [1:53] specifically for spatial reasoning and [1:55] understanding to essentially be the [1:57] logic unit of the robot system. [2:00] To give you an idea of what this [2:01] actually looks like in practice, let's [2:03] look at some of the capabilities. [2:05] First, there's 2D pointing and object [2:08] localization. [2:09] The model is exceptional at identifying [2:11] points on objects, specific parts on [2:14] objects where you may want to grasp [2:16] them, and general labeling, or giving [2:18] the model a list of objects to look for [2:20] by their state, such as on desk or [2:23] closed, and having it identify those [2:25] objects based on their unique states. [2:28] What's great about this is that it's [2:29] easy to get started with our Python SDK. [2:32] You can initialize the client, pass in [2:34] the model ID for the latest version of [2:36] Gemini Robotics ER, your prompt [2:38] requesting the objects that you're [2:40] looking for, be it specific items or [2:42] more abstract requests like the [2:44] screwdriver I need for the specialty [2:46] screw, and any additional Gemini [2:48] configuration items, like temperature or [2:50] thinking level, to get a response that [2:53] you can use in your robotics project. [2:55] All right, so let's look under the hood [2:57] for a second. If you've worked with [2:58] computer vision before, you've probably [3:01] used a model like YOLO that was trained [3:03] on fixed data sets like Coco or [3:05] ImageNet. [3:06] Those are fantastic for what they are, [3:08] but they are closed vocabulary, meaning [3:10] they can only identify a predefined list [3:13] of objects. [3:14] If you need your robot to find a [3:16] particular tool for cutting a dovetail [3:17] joint when woodworking, [3:19] but that wasn't in your training data [3:21] set, you're pretty much out of luck. [3:24] With Gemini Robotics ER, we've moved to [3:26] a vision language model or VLM [3:28] architecture. When you use the generate [3:31] content method in the SDK, you aren't [3:33] just running a classification script. [3:35] The model uses semantic grounding to [3:37] locate whatever you describe in natural [3:39] language. [3:40] Because language and vision are mapped [3:42] in the same space, the model doesn't [3:43] need a specific label for every object [3:46] in your warehouse. [3:47] This is called open vocabulary object [3:49] detection. [3:50] You can ask for the tool that looks like [3:52] it's been used the most, or the [3:54] component that is currently overheating. [3:57] The model reasons through the visual [3:59] data to ground those abstract concepts [4:01] into precise coordinates. [4:03] For developers, this is a massive win, [4:05] as it eliminates the need to build, [4:08] label, and maintain custom vision models [4:10] for every unique environment. Now, the [4:13] reason we call it embodied reasoning [4:15] instead of just a vision model is [4:17] because it understands the physics of [4:19] the scene, not just the pixels. In [4:21] traditional robotics, if you told a [4:23] robot to clear the table, it might try [4:26] to pick up a dish that's far too heavy [4:28] for it, potentially stripping a gear or [4:31] dropping the plate, because it doesn't [4:33] understand weight or structural [4:35] integrity. [4:36] Gemini Robotics ER has what we call [4:38] physical common sense. When it looks at [4:41] a scene, it isn't just trying to [4:42] identify the plate and the food, but [4:45] it's reasoning about the relationship [4:46] between them. [4:47] It knows that to move a dish, it must [4:49] first move food into smaller containers [4:51] that could be lifted, or request human [4:54] intervention. [4:55] It also understands that a glass bottle [4:57] is fragile, while a plastic one isn't. [5:01] This reasoning layer allows the robot to [5:02] handle unseen scenarios, like knowing it [5:05] shouldn't try to lift a table that is [5:07] obviously bolted to the floor. [5:09] It's this kind of practical intuition [5:12] that has historically been a nightmare [5:14] to script manually. [5:16] Beyond physical common sense, there's [5:18] semantic reasoning. [5:20] One of the hardest things for an [5:21] AI-backed robot to navigate isn't [5:23] necessarily a messy environment, but [5:25] rather a vague human prompt. [5:28] If we tell a robot, "Hey, put this [5:30] away." [5:31] it's suddenly trying to figure out what [5:33] is this, and where is away. [5:35] The Gemini Robotics ER 1.6 model excels [5:39] at navigating this ambiguity. [5:41] It uses the visual context of the [5:42] environment to make an educated guess. [5:45] If it sees someone holding a [5:46] screwdriver, and there's an open toolbox [5:48] nearby, it reasons that a way likely [5:51] means that specific tray through the use [5:53] of the model's thinking capabilities. [5:56] For developers, this means we're [5:57] spending less time writing brittle if [5:59] else statements for every possible edge [6:01] case, and more time defining the actual [6:04] high-level goals for your robots. [6:06] Now, typically when we use these models, [6:08] we think in terms of snapshots, like [6:10] sending one image to generate content [6:12] and getting one answer. [6:14] But robotics is inherently a temporal [6:16] problem. So, we need to think about what [6:19] happened 10 seconds ago in order to [6:21] solve what we need to do now. [6:23] Because Gemini Robotics ER 1.6 is built [6:26] on the Gemini 3 Flash backbone, it [6:28] natively supports video and multi-image [6:31] inputs. [6:32] This allows for what we call long [6:34] horizon temporal reasoning. [6:36] Instead of asking, is the door open? [6:39] You can pass a sequence of frames from [6:41] the robot's journey through a facility. [6:43] And because the model tokenizes these [6:45] frames chronologically, it can reason [6:47] about state changes. [6:49] For example, you can prompt the model [6:51] with something like, "Look at the last [6:53] 30 seconds of video. Did the gripper [6:55] successfully secure the object, or did [6:57] it slip?" [6:58] The model isn't just looking at the [7:00] final frame. It's looking at the motion [7:02] and delta between frames. [7:04] This is a game-changer for success [7:06] detection. [7:07] In the past, you'd have to write complex [7:09] heuristic code to check if a task was [7:12] completed. [7:13] Now, you can simply use the model as a [7:15] temporal supervisor that watches the [7:17] robot's work and confirms that the [7:19] physical state of the world has actually [7:21] changed in the way that you intended. [7:23] Reasoning gets us 90% of the way there, [7:26] but sometimes the visual input is pretty [7:27] difficult to process. [7:29] One issue we see fairly regularly with [7:31] standard Gemini is that images sent to [7:33] the model may not be that great for [7:36] determining what's in it due to size, [7:38] rotation, clutter, or really a variety [7:41] of other factors. [7:43] We've addressed this by adding support [7:44] for a Gentic vision to the Gemini [7:46] Robotics ER 1.6 model. [7:49] By enabling the code execution tool with [7:51] the SDK, the ER model can now generate [7:54] code for intermediate steps, allowing [7:57] the models to manipulate those images [7:58] itself to get a better understanding of [8:01] content. [8:02] Let's take a look at an example. [8:04] Say you have a production line that [8:05] makes circuit boards, and one of the [8:07] steps involves taking the unique ID off [8:09] of a certain chip to record it, such as [8:12] the ESMT chip in this image. [8:15] With traditional LLMs, this text might [8:18] be difficult to read, or it could get [8:20] mixed up with other text on the board. [8:23] By enabling the code execution tool, [8:25] Gemini writes the code to take the [8:26] image, locate the chip that you're [8:28] looking for, and then crops the image [8:30] down to just the necessary area. [8:33] In this particular example, the chip and [8:35] text are still upside down. So, the [8:38] Gemini Robotics ER model does one more [8:40] step with its generated code to rotate [8:43] the intermediate image into a more [8:45] readable orientation, making it far more [8:48] likely to correctly get the numbers off [8:50] of that chip. [8:51] The usefulness of this feature extends [8:53] far beyond just reading text, as Gemini [8:55] Robotics ER has been optimized for the [8:57] common task of reading analog gauges and [9:00] understanding common environments, like [9:02] factories and plants, to detect [9:04] anomalies. [9:05] If you have a robot running around a [9:07] facility, you can have it checking [9:08] pressure gauges, making sure doors are [9:11] properly closed, looking for spills or [9:13] snow on the ground, or performing any [9:15] number of other tasks with the same [9:17] simple tooling. [9:19] Once the robot perceives what's around [9:21] it, we can move into the planning stage [9:23] through task orchestration. [9:25] Think of this like giving a robot a [9:26] toolbox of functions. In your code, you [9:29] can provide the model with a list of [9:30] specific operations that your robot can [9:33] perform, like open gripper, move to [9:35] coordinate, detect objects, return to [9:38] home, or really anything else. [9:40] When you give the robot a high-level [9:42] task, like put the blue block in the [9:44] orange bowl, the Gemini ER model will [9:47] break that task into a series of smaller [9:49] task to plan out the entire operation. [9:52] It looks at the scene, identifies the [9:54] blue block and the orange bowl, and then [9:56] determines what order the functions [9:58] should be called to open the gripper, [10:00] move towards the block, close the [10:01] gripper, move it over to the drop-off [10:03] zone, open the gripper, and then return [10:06] to a neutral pose. [10:07] Along with high-level orchestration, the [10:09] Gemini Robotics ER model can handle [10:11] micro planning, like coming up with [10:13] trajectories that a robot should follow. [10:16] Instead of just a start and end point, [10:18] the Gemini Robotics ER model can [10:20] generate a series of intermediate [10:22] waypoints, while also highlighting ways [10:24] to avoid obstacles. So, the final part [10:27] of the perceive, plan, actuate loop, as [10:29] you can probably guess, is actuate. This [10:32] is where the gripper meets the object, [10:34] so to speak. [10:35] In robotics, actuation is just a fancy [10:38] way of saying that we're moving the [10:39] hardware, usually by sending specific [10:42] values to actuators, or motors and [10:44] joints, that make the robot go. [10:47] Now, I know the tendency is to say, [10:49] "Look at this amazing AI feature that [10:50] should change everything." But, in [10:53] traditional robotics, if you have a [10:54] machine in a factory repeating the exact [10:57] same motion 10,000 times a day, standard [11:00] control theory works just fine. [11:03] But, there are so many scenarios that [11:04] people are exploring now that aren't in [11:06] a controlled environment as they branch [11:08] out into the real world. [11:10] This leads to environments that are [11:12] messy, full of objects the robot has [11:14] never seen before, and situations that [11:16] weren't in the original training data. [11:18] For example, a robot rolling down the [11:20] street trying to deliver food could [11:22] encounter everything from a junky free [11:24] couch blocking the entire sidewalk to a [11:27] flock of unyielding grumpy Canadian [11:29] geese. [11:30] To help solve this, we've introduced our [11:32] vision, language, action, or VLA models, [11:36] including the flagship Gemini robotics [11:38] model, which is currently available [11:40] through our trusted tester program. [11:42] These models map camera pixels and [11:44] natural language instructions directly [11:46] to blocks of motor values. [11:49] You give it a prompt like, "Clean up the [11:50] desk." and the VLA streams camera frames [11:53] to determine exactly how the actuators [11:56] should move to get the job done. [11:59] This can even be used in conjunction [12:00] with the Gemini robotics ER model to [12:03] take a video of a larger task and break [12:05] it into individual parts for actuation, [12:08] letting the robot solve a series of [12:10] smaller problems to complete the big [12:12] one. [12:13] What makes this special is that we've [12:15] built these VLAs on top of the [12:17] state-of-the-art Gemini backbone. [12:20] Because the model has a massive general [12:22] understanding of the world, the robot is [12:24] able to understand context and details [12:26] about the objects. [12:28] For example, when we gave our robot a [12:30] small basketball and a net game and [12:32] asked it to do a slam dunk, the model [12:35] was able to take the ball and place it [12:37] through the hoop despite not being [12:39] previously trained for this. [12:41] Beyond simple pick-and-place operations, [12:43] we're also pushing for true dexterity. [12:46] We want our machines to do more than [12:48] just move around, but we're training our [12:50] models for complex tasks like plugging [12:53] in cords or performing fine detail [12:55] precision operations on a workbench. [12:58] Finally, we've taken a generalist [13:00] approach to embodiments. [13:02] Whether you are working with a humanoid, [13:04] a quadruped, or a mounted bipedal setup, [13:07] our goal is to meet you where you are. [13:10] These models are designed to be [13:11] embodiment-agnostic, [13:13] providing a powerful, general-purpose [13:16] brain that can be adapted to whatever [13:18] hardware you're working with. [13:20] By combining the high-level reasoning of [13:22] the ER model with the reactive, [13:24] real-time control of the VLA, we're [13:26] closing the loop on truly autonomous [13:28] physical agents. [13:30] All right, so at this point we've gone [13:32] over the tools for the full perceive, [13:34] plan, and actuate loop. But, that's [13:36] really just scratching the surface. [13:39] The next big question is how do you [13:40] actually talk to the robot? And how does [13:43] the robot keep you in the loop? [13:45] For example, if you have a humanoid [13:47] robot, people will likely tell it things [13:49] like hello and thank you. But, there's a [13:51] lot of variants in how they might say [13:53] these. [13:54] If your robot isn't programmed for each [13:56] variation, it could get kind of awkward [13:59] as it stands there and just feels [14:01] unnatural. But, with additional tooling, [14:04] the work for smooth human robot [14:05] interactions could be handled for you. [14:09] To accomplish this, I'm a big fan of the [14:11] Gemini Live API. [14:13] This is what turns a robot from a [14:14] pre-programmed machine into an [14:16] interactive partner. [14:18] It enables low latency bidirectional [14:20] conversation between you and the Gemini [14:22] model, [14:23] letting you give instructions through [14:25] natural language, like saying, "That [14:27] screw looks a little loose. Can you [14:28] tighten it?" [14:30] On top of that, this isn't just about [14:32] audio. The Live API allows you to stream [14:35] camera frames directly to the model, [14:37] providing constant visual context. [14:40] This helps you create feedback loops [14:41] where you can use this stream to even [14:43] alter plans during runtime. [14:46] Where the real magic happens though is [14:48] through function calling. Based on what [14:50] the model hears and sees, it can decide [14:52] to trigger a specific developer-defined [14:54] function. [14:55] This is the bridge that allows a natural [14:57] conversation to turn into precise [15:00] physical action, opening the door to [15:02] incredibly fluid human robot [15:04] interactions. [15:06] And while you're planning out your [15:07] builds, having a playground to test your [15:09] logic is definitely a nice to have. And [15:12] that's where AI Studio comes in. [15:14] This lets you rapidly prototype prompts [15:16] and test how the model perceives images [15:19] from your specific hardware without [15:21] having to constantly re-flash or reload [15:23] scripts on the robot. [15:25] Along with this, one of my favorite [15:27] areas is the build section. [15:30] We've actually created a few web app [15:31] templates to show off what's possible, [15:34] like integrating the MuJoCo simulation [15:37] engine directly into the browser. Like [15:39] you can see here, where we're running a [15:41] virtual robotic arm that uses the Gemini [15:43] Robotics ER model to detect block [15:46] locations, then performs a pick and [15:48] place task based on that information [15:51] without the risk of testing on real [15:53] hardware, really lending itself to a [15:55] fail-fast, fail-safe strategy. [15:57] And to wrap up this whirlwind tour of [15:59] Google DeepMind's robotics offerings, [16:01] let's talk about safety. [16:03] In the digital world, if an AI [16:05] hallucinates, it might give you a weird [16:07] recipe or a buggy snippet of code. Not [16:10] ideal, but you know, not catastrophic. [16:13] When this happens in robotics, we're [16:14] talking about real-world hardware, [16:17] sometimes with significant weight, [16:19] potentially moving around in the same [16:21] space as people. [16:22] As we all probably know, tools like a [16:25] simple table saw deserve respect because [16:27] of the physical risk involved. [16:29] When you add an autonomous brain to a [16:31] machine, that respect for safety has to [16:33] be built in from the ground up. [16:36] This is why our team is engaged in [16:38] ongoing safety research to develop a [16:40] holistic approach to robot protection. [16:43] We like to think of it as the Swiss [16:45] cheese model of defense. No single layer [16:47] is a perfect barrier, but by stacking [16:49] multiple layers of safeguards, spanning [16:52] the semantic, physical, and operational [16:54] aspects of safety, we can effectively [16:57] mitigate risk. [16:58] To evaluate these layers, we've [17:00] introduced the Asimov safety benchmarks. [17:03] As discussed in our research, these [17:05] aren't just theoretical. They are [17:07] grounded in reality using the NEISS [17:11] or National Electronic Injury [17:13] Surveillance System database. [17:15] This contains real-world injury reports [17:18] from hospitals to teach the model about [17:20] physical common sense. [17:22] Second, we ground our work in [17:24] established industrial ISO standards to [17:27] ensure our benchmarks reflect the same [17:29] operational safety constraints used in [17:31] factories today. [17:33] So, that's a quick look into what we're [17:34] working on within the robotic space at [17:36] Google DeepMind. At the end of the day, [17:38] we're building tools to be your partners [17:40] in creation. So, I'm looking forward to [17:42] seeing what you all build. You can find [17:44] more information in our developer docs, [17:46] cookbooks on GitHub, and sign up for the [17:48] trusted tester program on the Gemini [17:50] robotics webpage. Thanks for listening [17:52] in, and I'm looking forward to seeing [17:54] what you build. [17:54] >> [music] [18:02] [music]