AI Engineer
Replacing traditional speech-to-text pipelines with native audio processing eliminates latency bottlenecks, allowing real-time voice agents to manage multilingual interruptions and external tool calls directly.
AI-assisted summary of Can Your Agent Hear You Now? Building Live Voice Agents with Gemini — Thor Schaeff — original source. Check the original for full context.
Source published:
Key ideas
- Google DeepMind's new Nano Banana Light model enables real-time image generation with a latency of approximately two to three seconds. (5:15)
- The Gemini Interactions API facilitates stateful chaining of multimodal generations, allowing models to maintain visual and stylistic consistency across consecutive calls. (9:02)
- Gemini 3.5 Live Translate provides near real-time, bidirectional translation with automatic language detection, supporting 90 languages. (7:07)
- The Gemini Live API utilizes a native audio model, bypassing traditional speech-to-text and text-to-speech pipelines to reduce latency and improve conversational fluidity. (14:36)
Detailed notes
## Background This talk was published by the AI Engineer channel on YouTube on October 11, 2026, and features Thor Schaeff, a developer experience engineer at Google DeepMind. The session provides an overview of building live voice agents, showcasing how to leverage Gemini’s real-time capabilities to create interactions that see, hear, and speak in various languages.
## TL;DR Thor Schaeff demonstrates the latest capabilities of the Gemini API, focusing on real-time voice and multimodal integration. He showcases practical applications like a narrative-driven story canvas using the Interactions API, real-time multilingual translation, and a reactive AI radio DJ powered by Lyria 3. The presentation explains the shift from traditional speech-to-text pipelines to native audio models, highlighting features like native interruption support, Google Search grounding, and stateful context management.
## Atmosphere & style The session feels like an energetic, high-stakes tech demonstration performed in front of a live, tech-savvy audience at the AI Engineer World's Fair. Thor balances technical expertise with a lighthearted, slightly improvisational stage presence, using humor and live demo hiccups to maintain a dynamic and engaging tempo. The atmosphere is optimistic and forward-looking, characterized by a sense of rapid innovation where even the speaker’s own slides are humorously rendered obsolete by the speed of new model releases.
## Bulletin - [05:15](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=315s) Google DeepMind's new Nano Banana Light model enables real-time image generation with a latency of approximately two to three seconds. - [09:02](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=542s) The Gemini Interactions API facilitates stateful chaining of multimodal generations, allowing models to maintain visual and stylistic consistency across consecutive calls. - [07:07](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=427s) Gemini 3.5 Live Translate provides near real-time, bidirectional translation with automatic language detection, supporting 90 languages. - [14:36](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=876s) The Gemini Live API utilizes a native audio model, bypassing traditional speech-to-text and text-to-speech pipelines to reduce latency and improve conversational fluidity.
## Key takeaways - [01:12](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=72s) Gemini TTS offers highly expressive, steerable vocal delivery with low-latency streaming capabilities suitable for production-grade audio pipelines. - [03:45](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=225s) Omni Flash is a new model released by Google DeepMind designed specifically for video generation and editing workflows. - [03:45](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=225s) The Nano Banana story canvas demo shows how voice agents can issue tool calls to dynamically generate accompanying imagery in real time. - [05:42](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=342s) The `previous_interaction_id` parameter is critical for developers to maintain context, character mood, and aesthetic fidelity across sequential image generations. - [06:47](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=407s) The Interactions API acts as a unified stateful gateway that connects foundation models, generative media, and agent runtimes. - [07:07](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=427s) Gemini 3.5 Live Translate handles autonomous language detection, allowing users to switch languages mid-conversation without manual toggling. - [10:21](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=621s) Gemini 3.1 Flash Live serves as the core conversational engine for real-time voice agents, supporting complex tool execution. - [10:56](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=656s) Lyria 3 enables voice agents to generate custom musical tracks on demand, demonstrating the potential for complex agent-to-tool coordination. - [15:07](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=907s) Native audio models allow for interruption (barge-in) and more natural, responsive conversations compared to cascading pipeline architectures. - [14:36](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=876s) The Live API operates over WebSockets, maintaining persistent state throughout the duration of the conversational session. - [16:06](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=966s) For enterprise-grade scaling of real-time media, developers are encouraged to utilize partnership integrations like LiveKit and Pipecat. - [16:25](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=985s) Google AI Studio is the recommended primary interface for developers to test and configure the Live API with features like Google Search grounding.
## Detailed walkthrough ### [00:12](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=12s) Guess the Model: Gemini TTS
Thor Schaeff opens the session with self-deprecating humor about scheduling confusion, welcoming the small group of attendees who “wandered in” and apologizing to anyone who accidentally showed up expecting a quiet co-working lounge. Setting up a talk focused on voice and agentic experiences using the Gemini API, he kicks off by playing an expressive, theater-announcer audio clip:
> “Welcome one and all to the grandest stage on Earth. You're standing right at the heart of the largest AI engineer World's Fair in history. Let's build the future together, shall we?”
He immediately challenges the room to guess the underlying model. After disqualifying a colleague who already knows the answer and dropping a hint that “it starts with G,” audience members guess Gemini and Gemini Live. Thor clarifies that while it is indeed Gemini, it is not Live—it is Gemini TTS (Text-to-Speech). He explains that the model excels at expressive, steerable vocal delivery through inline audio controls, and its latency has dropped substantially enough to make it a prime candidate for developers building high-quality audio pipelines.
- **Model identified**: Gemini TTS (expressive speech with inline audio steerability and low-latency streaming). - **Speaker**: Thor Schaeff, Developer Experience team at Google DeepMind (focusing on the Gemini API and Google AI Studio), joined by colleague Philip, who works on managed agents.
---
[03:15](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=195s) What Shipped Today: Omni Flash and Nano Banana Light
Thor highlights the rapidly expanding capabilities of Gemini’s multimodal foundation and media generation stack. He quips that his slides are already obsolete because Google DeepMind shipped two new models just that morning: Omni Flash and Nano Banana Light. While Omni Flash addresses video generation and editing, Nano Banana Light focuses directly on slumping the latency of image generation and image modification.
To illustrate how these modalities snap together, Thor introduces a prototype he built: the Gemini Nano Banana story canvas. The goal is to conduct an improv-style storytelling session driven conversationally by Gemini Live, which subsequently issues tool calls to Nano Banana Light to dynamically generate accompanying illustrations in real time.
- **Omni Flash**: New model for video generation and video editing. - **Nano Banana Light**: High-speed image generation and editing model engineered to cut rendering latency down to roughly two to three seconds.
---
[03:45](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=225s) Demo: An Illustrated Story Canvas
Stepping into an impromptu theater mode, Thor invites the audience to throw out a narrative seed. Setting the scene in 1980s Taipei, Gemini Live begins narrating with atmospheric flavor:
> “The neon lights of Taipei's night market pulsed with an almost magical energy reflecting off the wet pavement and the chrome of passing scooters. Rain had just passed, leaving a humid mist that hung in the air. Amidst the throng of shoppers and vendors, a young woman clutched a small, strangely glowing object under her jacket.”
Thor steers the narrative further, nudging the agent to bring in the iconic 1980s Taiwanese invention: bubble tea. The voice agent seamlessly picks up the cue, describing a vendor vigorously shaking a frothy metal cup and serving a customer who marvels at the chewy tapioca pearls at the bottom. As the voice agent narrates, it fires tool calls in the background to render canvas art corresponding to the scene.
```mermaid flowchart TD accTitle: Story Canvas Flow with Interactions API accDescr: Gemini Live triggers tool calls into the stateful Interactions API to generate consistent imagery via Nano Banana Light.
subgraph UserInteraction ["User Voice Input"] VoiceIn["Audience / User Prompt"] end
subgraph GeminiLiveSession ["Real-Time Agent"] GLive["Gemini Live (Voice Agent)"] ToolCall["Tool Call: generate_image"] end
subgraph MediaBackend ["Generative Media"] InterAPI["Stateful Interactions API"] NanoB["Nano Banana Light"] end
VoiceIn --> GLive GLive -->|"streams speech"| VoiceOut["Spoken Story"] GLive -->|"triggers tool"| ToolCall ToolCall -->|"passes previous_interaction_id"| InterAPI InterAPI --> NanoB NanoB -->|"renders scene (2-3s)"| CanvasUI["Consistent Story Canvas"] ```
---
[05:42](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=342s) The Interactions API: Stateful Context Chaining
As the canvas images materialize within two to three seconds each, Thor dives into the technical mechanism powering character and style coherence: the Gemini Interactions API. Rather than treating each creative generation as an isolated, stateless request, the Interactions API keeps persistent context across calls.
The first generation executes without a reference ID, but every subsequent call forwards the `previous_interaction_id`. Thor explains:
> “The interactions API is exciting because it is a stateful API which means we can chain our interactions very seamlessly.”
Because the model retains stylistic awareness and character features from the prior turn, it does not invent an entirely new visual language for each story beat. Thor emphasizes that the Interactions API functions as a unified gateway across all Google DeepMind models and managed agents, giving builders a coherent foundation for chained multimodal workflows.
- **Key parameter**: `previous_interaction_id` to maintain character, mood, and aesthetic fidelity across sequential generations. - **Unified role**: Serves as the single stateful interface spanning generative media, foundation models, and agent runtimes.
---
[07:07](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=427s) Demo: Near Real-Time Translation with Gemini 3.5 Live Translate
Turning to real-time audio, Thor pivots to Gemini 3.5 Live Translate. He pulls up a live translation session on screen and displays a QR code, instructing audience members to scan it with their smartphones, plug in their headphones, and pick their native or preferred language.
As Thor speaks English into his microphone, the platform processes the incoming stream, derives contextual meaning on the fly, and broadcasts synthetic audio simultaneously in attendees' chosen languages. To prove the model's bidirectionality and autonomous language detection, Thor pauses, switches into German mid-stride, and watches the engine automatically detect the change in source tongue without manual toggling. Attendees confirm through hand signals that their German and Chinese audio streams are arriving crisp and clear.
> “Gemini Live Translate is a really incredible model. It's near real-time translation. So as I'm speaking, as soon as the model is recognizing kind of the context of what I'm speaking about, it will then start translating in real time to your desired language.”
- **Model**: Gemini 3.5 Live Translate, exposed directly via the Gemini API. - **Capabilities**: Zero-configuration automatic language detection, streaming translation, context-aware sentence translation, and synchronized multilingual broadcast to connected client devices.
---
[10:21](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=621s) Gemini 3.1 Flash Live & The Lyria 3 Radio DJ Demo
Thor transitions to conversational voice agents, confirming that the underlying engine driving real-time interaction in the official Gemini consumer app is Gemini 3.1 Flash Live. To show how a voice agent coordinates external tools, Thor displays a retro-themed "live jukebox" project built in Google AI Studio.
The concept recreates classic call-in radio shows where listeners phoned the host to request custom tracks. Here, Gemini 3.1 Flash Live acts as an energetic, British-accented radio DJ. When a request is finalized, the agent calls an external API tool powered by Google DeepMind's Lyria 3 music generation model to generate custom tracks on demand.
When Thor connects to the live jukebox, the DJ jumps in:
> “Well, hello there. You're live on the live jukebox right now. What absolute banger are we cooking up for you today, mate?”
Thor requests a high-energy reggaeton track celebrating the AI Engineer World's Fair in San Francisco. The DJ matches his banter, playfully asking whether to include specific tech terminology or lean into festival energy before confirming:
> “Prepare for a huge musical surprise generating that track right now. Mixing that reggaeton rhythm with the tech buzz for the World's Fair. It's going to blow your headphones off.”
Lyria 3 immediately compiles the request and starts outputting an upbeat reggaeton groove complete with dynamic synth stabs, pulsing bass, and generated vocals singing about neural networks firing up and tech pioneers mingling on the conference floor.
[12:36](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=756s) Building and Architecting with the Live API
Thor Schaeff transitions from the live demonstrations to explaining the underlying mechanics of the Gemini Live API. He emphasizes that while the specific example of a radio DJ might be niche, the core components—multilingualism, real-time capability, and tool calling—are essential building blocks for any modern conversational agent. He notes that the same infrastructure used for the music demo can be applied to business-critical tasks like querying a CRM or creating backend support tickets.
The Live API operates as a stateful WebSocket session. Unlike traditional voice pipelines that rely on a disjointed flow of speech-to-text, an LLM, and text-to-speech, the Gemini Live model is native audio. This architecture bypasses the need for intermediate text conversion, allowing for significantly lower latency and more natural interaction. "So if you've worked with voice agents before, this is a little bit different from kind of your cascading voice agent pipeline because this is a native audio model," Schaeff explains. "So the model itself does not go through text, but rather the intelligence is baked into the audio model."
* **Core Capabilities:** * **Native Audio:** The model processes and generates audio directly, supporting fluid transitions. * **Multilingual Support:** Currently supports 90 languages. * **Interruption (Barge-in):** Users can interrupt the model while it is speaking. * **Tool Use:** Allows the model to execute functions, such as searching Google or triggering external APIs like Lyria 3. * **Ecosystem Partners:** Schaeff recommends using tools like LiveKit and Pipecat to handle the complexities of scaling real-time media.
```mermaid flowchart LR A["User Input (Audio/Video/Text)"] -->|WebSocket| B["Gemini Live Model"] B -->|Native Audio Generation| C["Audio Output"] B -->|Transcript| D["Text Output"] B -->|Function Call| E["Tools/Google Search"] E -->|Data| B style B fill:#f9f,stroke:#333 ```
[16:25](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=985s) Live Demonstrations and Conclusion
Schaeff guides the audience through the practical starting point for developers: Google AI Studio. He humorously acknowledges the investment Google has made in their domain names, encouraging attendees to utilize the AI Studio platform to experiment with the Gemini 3.1 Flash Live model. Within this interface, developers can toggle features such as Google Search grounding, which allows the model to retrieve real-time information for dynamic queries, such as current weather or historical facts.
In a final demonstration, Schaeff tests the model's vision and multilingual capabilities in real-time. He probes the model to identify his clothing and sweatshirt branding, then switches to Mandarin to engage the agent in a conversation about its "dietary habits" as a digital entity. He concludes the session by inviting attendees to follow up during a deeper-dive workshop scheduled for the following day.
* **Key Resources Mentioned:** * **Google AI Studio:** `ai.studio` (or `ai.dev`) for testing the Live API. * **Configuration:** Users can select specific model variants (e.g., Gemini 3.1 Flash Live) and choose from different voice personas (e.g., "Puck"). * **Grounding:** Enabling Google Search allows the model to function with external, up-to-date knowledge.
Regarding the technical advantage of the native model, Schaeff notes: "So this is what allows us to to switch very fluidly between different languages." He wraps up the talk by thanking the audience and offering an invitation for further collaboration: "If you want to learn more about building with Gemini and the Gemini Live API, join me tomorrow morning in room 2009."
## Notable quotes > [05:45](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=345s) The interactions API is exciting because it is a stateful API which means we can chain our interactions very seamlessly. — Thor Schaeff
> [07:07](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=427s) Gemini Live Translate is a really incredible model. It's near real-time translation. So as I'm speaking, as soon as the model is recognizing kind of the context of what I'm speaking about, it will then start translating in real time to your desired language. — Thor Schaeff
## Mentioned Thor Schaeff, Google DeepMind, Gemini, Gemini Live, Gemini TTS, Omni Flash, Nano Banana Light, Interactions API, Gemini 3.5 Live Translate, Lyria 3, Gemini 3.1 Flash Live, LiveKit, Pipecat, Google AI Studio, AI Engineer World's Fair
## Video description An agent that hears you, sees you, and answers in your language, in real time. Here's how to build one with Gemini.
Thor Schaeff, who works on developer experience at Google DeepMind, runs a set of live demos of Gemini's real-time and media models. He builds an illustrated story with Gemini Live and a fast Nano Banana image model, using the stateful Interactions API to keep characters and style consistent between images. He has the audience scan a QR code to hear the talk in their own language with Gemini 3.5 Live Translate, then calls up an AI radio DJ that uses Lyria 3 to generate a reggaeton track on request. He explains how the Live API works: a stateful WebSocket session that takes in text, audio and video and streams back audio plus a transcript, using a native audio model rather than a speech-to-text and text-to-speech pipeline. It supports 90 languages, interruptions, tool calls and Google Search grounding. He ends with a voice and vision demo in Google AI Studio.
In this talk: • Chaining image generations with the stateful Interactions API • Near real-time speech translation with Gemini 3.5 Live Translate • A voice agent that calls tools, like Lyria 3 for music • How the Live API works: native audio, barge-in, tools and 90 languages
SPEAKER Thor Schaeff, Developer Experience, Google DeepMind LinkedIn: https://www.linkedin.com/in/thorwebdev/ X: https://x.com/thorwebdev GitHub: https://github.com/thorwebdev
LINKS Gemini Live API docs: https://ai.google.dev/gemini-api/docs/live Interactions API docs: https://ai.google.dev/gemini-api/docs/interactions-overview Gemini 3.5 Live Translate: https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-live-3-5-translate/ Lyria 3 for developers: https://blog.google/innovation-and-ai/technology/developers-tools/lyria-3-developers/ Google AI Studio: https://aistudio.google.com
CHAPTERS 0:00 Intro 0:56 Guess the model: Gemini TTS 2:01 Meet Thor 3:15 What shipped today 3:45 Demo: an illustrated story canvas 5:42 The Interactions API 7:07 Demo: Gemini Live Translate 10:21 Gemini 3.1 Flash Live 10:56 Demo: an AI radio DJ with Lyria 3 14:13 How the Live API works 15:07 Native audio, not a pipeline 16:06 Partners: LiveKit and Pipecat 16:31 Try it in AI Studio 17:55 Demo: voice, vision and Mandarin 19:42 Wrap-up
Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI.
AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/
#Gemini #VoiceAI #AIEngineer
[Source](https://www.youtube.com/watch?v=xfPVBzDYjH4)