OpenAI
Decoupling conversational audio streaming from background computation allows voice models to execute complex multi-step tasks and absorb interruptions without stalling speech output.
AI-assisted summary of Building with GPT-Live and Image Generation | DevDay 2026 — original source. Check the original for full context.
Source published:
Key ideas
- OpenAI introduced GPT-Live, a full duplex, turnless voice model that eliminates traditional Voice Activity Detection (VAD) by continuously streaming bi-directional audio to handle simultaneous speaking, listening, and real-time interruptions. (0:19)
- The platform introduces a split delegation architecture that decouples instantaneous voice interaction from asynchronous backend reasoning, code execution, and visual rendering. (9:27)
- Backend systems coordinate with the voice model via three distinct event streams: Silent Context Updates for unspoken background state, Commentary Streams for vocalized announcements, and Instructions Streams for dynamic behavioral steering. (11:11)
- Developers can architect background execution using either Client Delegation to preserve existing custom agent frameworks or Responses Delegation for fully managed OpenAI-hosted tool invocation. (13:50)
Detailed notes
## Background Published on YouTube by OpenAI on November 11, 2026, this DevDay presentation explores how to build multimodal applications using the GPT-Live model and real-time image generation. According to the show description, the session covers how voice agents can brainstorm, play games, generate visuals, and coordinate background workflows while designing for delegation, latency, interruptions, and shifting user requests.
## TL;DR OpenAI engineers Alex and Erika demonstrate GPT-Live, a full-duplex voice architecture that unifies conversational audio into a continuous stream rather than relying on conventional turn-taking and voice activity detectors. Guided by their live AI co-host Marin, the presenters highlight steerable agent personalities and demonstrate how to delegate heavy tasks—like real-time Pictionary vector sketching—to background execution harnesses without interrupting spoken dialogue. They outline the architectural divide between Client Delegation and Responses Delegation, explaining how backends communicate through silent context, speakable commentary, and dynamic instruction streams. The session concludes with live resilience tests against heavy background noise and a 12-image batch generation, illustrating how decoupling speech, interface rendering, and backend logic maintains fluid conversational flow.
## Atmosphere & style The presentation carries a crisp, energetic, and playful tone, unfolding as an entirely hands-free stage dialogue between human engineers and their responsive AI co-host, Marin. Live demonstrations—ranging from an improv storytelling duel with a dry, pessimistic British persona named Colin to a lively round of audience-guessed Pictionary—give the technical walkthrough the spontaneous charm of collaborative live theater. The effortless, uninterrupted back-and-forth between the hosts and the AI creates an atmosphere of polished ease, making complex background delegation feel natural and immediate.
## Bulletin - [00:19](https://www.youtube.com/watch?v=jDxKzlGLCas&t=19s) OpenAI introduced GPT-Live, a full duplex, turnless voice model that eliminates traditional Voice Activity Detection (VAD) by continuously streaming bi-directional audio to handle simultaneous speaking, listening, and real-time interruptions. - [09:27](https://www.youtube.com/watch?v=jDxKzlGLCas&t=567s) The platform introduces a split delegation architecture that decouples instantaneous voice interaction from asynchronous backend reasoning, code execution, and visual rendering. - [11:11](https://www.youtube.com/watch?v=jDxKzlGLCas&t=671s) Backend systems coordinate with the voice model via three distinct event streams: Silent Context Updates for unspoken background state, Commentary Streams for vocalized announcements, and Instructions Streams for dynamic behavioral steering. - [13:50](https://www.youtube.com/watch?v=jDxKzlGLCas&t=830s) Developers can architect background execution using either Client Delegation to preserve existing custom agent frameworks or Responses Delegation for fully managed OpenAI-hosted tool invocation.
## Key takeaways - [00:19](https://www.youtube.com/watch?v=jDxKzlGLCas&t=19s) GPT-Live operates in full duplex mode, allowing simultaneous listening and speaking without waiting for silence-based turn boundaries. - [00:19](https://www.youtube.com/watch?v=jDxKzlGLCas&t=19s) Presentations and UI elements can be driven entirely hands-free via natural voice steering rather than physical controls or hardcoded pipelines. - [04:01](https://www.youtube.com/watch?v=jDxKzlGLCas&t=241s) Traditional VAD, ASR, LLM, and TTS chained architectures are replaced by a unified model that natively decides when to take turns, back-channel, or yield. - [04:01](https://www.youtube.com/watch?v=jDxKzlGLCas&t=241s) Independent secondary voice sessions with distinct behavioral prompts and personas can be spawned dynamically to serve as rehearsal partners or evaluation agents. - [04:01](https://www.youtube.com/watch?v=jDxKzlGLCas&t=241s) Model personality traits—including assertiveness, patience, humor, and interruption tolerance—are fully steerable using standard system prompts. - [07:35](https://www.youtube.com/watch?v=jDxKzlGLCas&t=455s) Delegation separates the low-latency conversational audio layer from compute-heavy backend operations like image generation and code execution. - [09:27](https://www.youtube.com/watch?v=jDxKzlGLCas&t=567s) A three-tier separation of concerns divides tasks between frontend visual rendering, real-time voice streaming, and backend execution harnesses. - [11:11](https://www.youtube.com/watch?v=jDxKzlGLCas&t=671s) Pre-sending answers and hint states to the voice layer reduces latency by avoiding synchronous round-trip backend calls during live interactions. - [11:11](https://www.youtube.com/watch?v=jDxKzlGLCas&t=671s) The Silent Context Stream feeds updates into the model's memory so it can answer direct user questions without cluttering continuous spoken output. - [11:11](https://www.youtube.com/watch?v=jDxKzlGLCas&t=671s) Over-indexing on speakable updates can degrade user experience; voice agents require deliberate filtering between silent awareness and vocal commentary. - [13:50](https://www.youtube.com/watch?v=jDxKzlGLCas&t=830s) Client Delegation returns execution events to custom application harnesses, enabling developers to integrate proprietary tools, legacy pipelines, and third-party models. - [13:50](https://www.youtube.com/watch?v=jDxKzlGLCas&t=830s) Responses Delegation offers an out-of-the-box solution where the OpenAI Responses API autonomously manages context and tool calls like web search or DALL-E. - [17:17](https://www.youtube.com/watch?v=jDxKzlGLCas&t=1037s) Turnless voice architecture maintains robustness in high-noise environments where traditional threshold-based VAD systems continuously fail. - [17:17](https://www.youtube.com/watch?v=jDxKzlGLCas&t=1037s) Voice interfaces flatten complex UI hierarchies, allowing users to configure deeply nested features and nuanced stylistic settings through direct verbal requests. - [20:37](https://www.youtube.com/watch?v=jDxKzlGLCas&t=1237s) A single continuous GPT-Live session can orchestrate brainstorming, interactive games, audio filtering, volume changes, and batch image generation without disconnecting.
## Detailed walkthrough ### [00:19](https://www.youtube.com/watch?v=jDxKzlGLCas&t=19s) Introducing GPT-Live and Full Duplex Voice
Alex, an engineer on the GPT-Live team, and Erika, who collaborates with startups creating voice applications, open the session to demonstrate how to build with the new GPT-Live model. Unlike typical conference presentations anchored to a podium computer, the entire presentation is hands-free and controlled completely through natural speech—including advancing visual slides and technical diagrams. To demonstrate this firsthand, they introduce an interactive voice agent named Marin as their official co-presenter. Marin introduces herself with high energy: “I'm Marin, a voice agent and your joyous co-host for today.”
Marin immediately highlights what sets GPT-Live apart from traditional voice bots: it operates in full duplex mode. It can listen to incoming audio and speak out loud simultaneously, maintaining an uninterrupted conversational flow while triggering and monitoring background operations. Alex and Erika map out the presentation agenda around three core technical concepts demonstrated across live demos:
- **Model Personality:** Exploring full duplex audio, natural conversational texture, and prompt-driven voice traits. - **Delegation:** Coordinating background workflows, tools, and secondary reasoning models without stalling speech output. - **Turnless Architecture:** Managing continuous audio streaming, user interruptions, and state synchronization without relying on standard conversational turns.
[01:37](https://www.youtube.com/watch?v=jDxKzlGLCas&t=97s) Eliminating Turn-Taking: The "Fortunately, Unfortunately" Demo with Colin
Traditional voice systems rely strictly on rigid back-and-forth turns, where a system must wait for silence before processing input and crafting a reply. Alex explains that GPT-Live replaces this model by constantly streaming audio in and out: “GPT Live is a full duplex voice model, meaning unlike other voice models which are architected around user and assistant turns, GPT Live is constantly streaming audio in and out.” To exhibit how this works when multiple speakers interact, Alex asks Marin to bring another AI persona onstage. Marin immediately spins up Colin, who greets the audience with dry wit: “And I'm here because apparently someone thought a grumpy British voice was missing.”
Alex kicks off a quick round of the improv game *Fortunately, Unfortunately*, alternating between Marin's sunny temperament and Colin's pessimism. Using the starter prompt of being at DevDay with a demo running smoothly, the agents trade sentences rapidly without awkward dead air. Marin enthusiastically chimes in that the crowd feels like a warm, supportive heartbeat, while Colin counters that the audience's laughter might just be mistaken for a taped laugh track by someone looking for a real applause sign. Marin neatly brings the story to a close by imagining someone pressing the button, confetti falling, and everyone taking a bow as if the chaos was entirely planned. Colin bids the crowd a dry goodbye and drops off the stage, leaving Marin to reflect cheerfully on the quick back-and-forth: “That was a fun little chaos sandwich.”
[04:01](https://www.youtube.com/watch?v=jDxKzlGLCas&t=241s) Architecture of Turnless Conversation and Steerable Personalities
Alex and Erika pull up an architectural diagram to explain why the interaction felt so remarkably fluid compared to standard conversational bots. In conventional voice pipelines, developers chain together an external Voice Activity Detector (VAD) with Automatic Speech Recognition (ASR), a Large Language Model (LLM), and Text-to-Speech (TTS). That approach forces the software to guess when a speaker has concluded their sentence using silence thresholds or arbitrary timeouts. In contrast, GPT-Live unifies audio processing within a single system. Alex emphasizes the architectural leap: “The model handles turn-taking itself without needing a separate voice activity detector model to decide that someone is done speaking.”
This internal decision-making allows the model to produce subtle vocalizations, insert brief back-channel affirmations (like "mm-hmm" or quiet laughter), yield instantly when an interlocutor speaks over it, or decide when to keep going. Colin was not a hardcoded script or a plugin, but an entirely independent GPT-Live session spun up on the fly with custom behavioral instructions and vocal qualities. Erika notes that running concurrent, lightweight voice sessions is a practical technique for running rigorous end-to-end evaluations: “A second session can be a rehearsal partner.” Developers can configure simulated personas with varying accents, interruption frequencies, and adversarial goals to stress-test their voice products.
Every nuance of GPT-Live’s conversational manner is completely steerable via standard system prompting. Developers can configure how assertive or patient the agent should be, whether it laughs, how quickly it yields when spoken over, or whether it remains totally quiet until addressed. Marin summarizes this granular control over her own personality: “Whether I'm bubbly or measured, whether I'm quick with a joke or quieter, that texture is promptable too.”
[07:35](https://www.youtube.com/watch?v=jDxKzlGLCas&t=455s) Delegation in Practice: The Real-Time Pictionary Drawing Game
The presentation shifts to delegation—how a voice agent stays engaged in chat while delegating computationally intensive operations to separate models in the background. Alex notes that interactive games serve as a great stress test for voice systems because processing latencies, awkward pauses, and robotic behaviors become glaringly obvious right away. To put this to the test, Alex has Marin spin up an interactive drawing game where the system sketches an object on the monitor while attendees shout guesses.
Marin launches the drawing canvas and begins rendering vector strokes in real time while bantering with the room. Attendees shout out ideas from their seats, with Alex and Erika amplifying guesses into their handheld microphones: - **Round 1:** The model sketches strokes on the digital canvas, and the audience quickly identifies a person flying a kite. - **Round 2:** A long-necked animal sketch appears; the crowd calls out "Giraffe!", which Marin confirms with high energy. - **Round 3:** A more intricate structure appears. An attendee guesses "Pizza Hut," prompting Marin to laugh and praise the creative attempt before another audience member correctly guesses a "merry-go-round" / carousel.
Throughout each drawing round, Marin never paused her audio stream or went silent while the code ran; she responded to shouts, offered encouragement, confirmed winners, and managed the flow seamlessly.
[09:27](https://www.youtube.com/watch?v=jDxKzlGLCas&t=567s) The Split Architecture: UI, Voice Interface, and Backend Execution
To break down the mechanics behind the drawing game, Alex asks Marin to bring up the live runtime dashboard and freeze it in place. The dashboard reveals two parallel execution tracks: continuous audio streaming at the front end, alongside distinct delegation threads triggered in the background. Alex summarizes the architectural necessity of this separation: “In this full duplex architecture, GPT Live can keep streaming audio to the user, but for more complex requests, it can be helpful to get assistance from a more powerful model or agent harness.”
```mermaid flowchart TD accTitle: GPT-Live Split Delegation Architecture accDescr: Diagram showing how audio streaming, UI updates, and backend code execution are separated across components.
subgraph ClientLayer["Frontend Experience"] UI["Visual UI / Canvas<br/>(Displays Live Drawing)"] User["User / Audience Voice"] end
subgraph VoiceLayer["Real-Time Audio Layer"] GPTLive["GPT-Live Voice Model<br/>(Full Duplex Streaming & Banter)"] end
subgraph BackendLayer["Background Agent Harness"] BackendAgent["Reasoning Model & Code Execution<br/>(Generates Code / Vector Strokes)"] end
User <-->|"Bi-directional Audio"| GPTLive GPTLive -->|"Delegates Task"| BackendAgent BackendAgent -->|"Sends Canvas Updates"| UI BackendAgent -->|"Returns Completion / State"| GPTLive ```
The system cleanly separates concerns into three distinct tiers: 1. **The User Interface:** Displays the dynamic visual output, rendering the code and canvas strokes sent from the backend so the audience can see the doodle unfold in real time. 2. **The Voice Interface (GPT-Live):** Manages the live acoustic channel, interprets conversational intent, maintains back-channel banter, and handles audience interruptions without stuttering. 3. **The Backend Execution Agent:** Operates asynchronously in a dedicated agent harness utilizing code execution to generate and evaluate the drawings, feeding status updates back into the voice model when tasks complete or states change.
By decoupling the conversational voice interface from the heavy background compute, developers can maintain instantaneous, natural latency on speech while safely coordinating complex, multi-step actions behind the scenes.
[11:11](https://www.youtube.com/watch?v=jDxKzlGLCas&t=671s) Background Updates and the Three Information Streams
During interactive sessions like the drawing game, latency is managed by decoupling backend processing from conversational responses. The correct answer, along with hints and timing triggers for when to reveal them, is pre-sent to the voice interface. This architecture allows the voice model (Marin) to evaluate and respond to player guesses instantaneously without needing a synchronous round-trip delegation on every attempt. Meanwhile, state updates—such as recording a successful guess—execute quietly in the background without halting the image streaming. As Astra streams image rendering to the frontend, the backend continuously feeds updates into GPT-Live. When asked by the presenters to display the architecture, Marin pulls up the event diagram on screen.
```mermaid flowchart TD accTitle: GPT-Live Backend Event Streams accDescr: Three distinct event streams feed data from the backend application into the GPT-Live voice interface.
subgraph Backend["Application Backend"] SC["Silent Context Stream"] CC["Commentary Stream (Speakable)"] IS["Instructions Stream"] end
subgraph Live["GPT-Live Voice Model"] Thinking["Model Thinking / Memory"] Speech["Audio Out (Spoken)"] Steering["System Instructions & Guardrails"] end
SC -->|"Updates state without talking"| Thinking CC -->|"Provides text to announce"| Speech IS -->|"Dynamically steers behavior"| Steering ```
The speakers explain that backends can communicate with GPT-Live through three primary streams, each serving a distinct product requirement:
1. **Thinking Stream (Silent Context Updates):** Injects background state directly into the model's ongoing context window. If a user asks for a hint or round status, the model already possesses the necessary facts to answer without initiating a fresh delegation call. 2. **Commentary Stream (Speakable Content):** Supplies text explicitly designed for the model to vocalize. For example, if a competitive leaderboard compares scores against rival teams, the backend pushes score announcements through this stream for Marin to read aloud. 3. **Instructions Stream:** Enables dynamic mid-session steering, acting as live guardrails or updating system prompts (e.g., instructing the voice model to become more concise, change tone, or steer clear of specific subjects).
Determining when and how frequently to dispatch these events is a critical product design choice. Drawing from internal experience, one speaker notes: “When we were first internally dogfooding voice in Codex, one of the initial complaints was that updates were too verbose and too frequent.” If a developer kicks off a 20-minute Codex coding task, hearing tool invocations broadcast every 30 seconds is exhausting; however, if the user directly asks what is happening, the model must already hold sufficient silent context to give an immediate, accurate summary.
---
[13:50](https://www.youtube.com/watch?v=jDxKzlGLCas&t=830s) Client Delegation vs. Responses Delegation
At the presenters' request, Marin transitions the display to the delegation diagram. The API offers two foundational architectural patterns for offloading heavy tasks from the conversation: **Client Delegation** and **Responses Delegation**.
```mermaid flowchart LR accTitle: Comparison of Client vs Responses Delegation accDescr: Two ways GPT-Live coordinates tasks: routing events through custom harnesses or directly via OpenAI Responses API.
subgraph ClientMode["Client Delegation"] direction TB V1["GPT-Live Voice"] -->|"Delegation Event (ID, Timestamp)"| App["Your Custom App / Harness"] App -->|"Execute Task"| Model["Any Backend Model or Logic"] App -->|"Return: Silent, Speakable, or Instruction"| V1 end
subgraph ResponsesMode["Responses Delegation"] direction TB V2["GPT-Live Voice"] -->|"Context & Tool Call"| RAPI["OpenAI Responses API"] RAPI -->|"Automated Tool Execution"| Tools["Hosted Tools (Search, DALL-E)"] RAPI -->|"Direct Output Integration"| V2 end ```
Client delegation is highlighted as the method powering both the live DevDay presentation and Codex Voice. In this pattern, when GPT-Live determines it needs outside help, the client application receives an explicit delegation event containing an ID and a timestamp. The client’s existing agent harness takes the conversation transcripts and app state, renders a backend request, executes arbitrary business logic or queries any model provider, and chooses whether to return results as silent context, speakable text, or updated system instructions. This model is ideal for developers who already have mature agent frameworks, complex evaluation suites, or legacy speech-to-text / LLM / text-to-speech cascaded pipelines where GPT-Live simply replaces the edge audio interface. Praising its ease of adoption, a presenter emphasizes: “This backend flexibility that we're talking about right now is the most underrated part of GPT Live.”
Conversely, Responses Delegation targets rapid development and out-of-the-box management. The developer configures a model running directly on OpenAI’s Responses API alongside tool definitions and system instructions. GPT-Live automatically manages conversational context, emits structured outputs (function names and arguments), and natively coordinates hosted OpenAI tools like web search and image generation.
To ensure the audience absorbs the architectural contrast, the speakers prompt Marin for a recap. Marin delivers the breakdown concisely: “GPT Live delegates work to a backend while staying in the conversation. You can use your own harness with client delegation or let responses handle it and send updates as silent context, speakable text, or instructions.”
---
[17:17](https://www.youtube.com/watch?v=jDxKzlGLCas&t=1037s) Turnless Voice Architecture in Noisy Environments
Moving into the final live demonstration, the speakers explore how GPT-Live's "turnless" architecture handles intense, continuous background noise. Traditional voice agents rely on Voice Activity Detection (VAD) or threshold-based silence detection; when background music plays or ambient noise spikes, standard VAD-based systems register constant speech and break down entirely.
To demonstrate resilience, the presenters have Marin open the Live dashboard and play royalty-free background audio. They instruct her to push the audio volume up to 50% and then dynamically steer the style by requesting: "Marin, can you update the vibe to more of like a wax cylinder kind of recording? Yeah, like 1888. Bring us back." Marin acknowledges the instruction over the loud playback and successfully introduces a crackly, vintage acoustic filter, remarking: “It feels cozy and crackly. Like we've stepped into an old-timey radio hour.”
Throughout the experiment, the presenters speak over the noisy vintage music, interrupting, asking questions, changing instructions on the fly, and ordering Marin to lower the volume back down to 15%. Marin tracks every command without missing a beat, isolating the speakers' speech patterns and executing tool delegations simultaneously without stalling.
- Voice inputs naturally flatten complex application control hierarchies, enabling users to request deep, niche features (like dialing in an 1888 wax cylinder filter) without hunting through nested settings menus. - Voice provides rich, information-dense context because users are consistently willing to speak out loud for two minutes rather than type comprehensive paragraphs. - By abandoning silence-based turn boundaries, the model ingests continuous noisy audio and perpetually makes real-time interaction decisions—deciding whether to speak, listen silently, or trigger a background tool. - This design supports real-world deployment contexts where users cannot find quiet rooms, such as loud airport terminals, crowded conference venues like Fort Mason, busy factory floors, or speakerphones in moving cars.
---
[20:37](https://www.youtube.com/watch?v=jDxKzlGLCas&t=1237s) Grand Finale: 12-Up Grid and Developer Resources
To conclude the session, the presenters put Marin and OpenAI's underlying image systems through one final stress test, requesting a generation that combines high speed with multi-image output: generating 12 distinct pictures of the DevDay mascot critters traveling the world. Marin acknowledges the multi-tool command immediately: “Starting a 12-up goodbye grid.”
The presenters reflect on the continuity of the entire demonstration, noting that every single feature shown—brainstorming, interactive guessing games, real-time image rendering, audio playback, volume manipulations, and guest interactions—took place inside one single, uninterrupted GPT-Live session. Across all these use cases, the software maintained a strict separation of concerns: - What the backend infrastructure processes and coordinates - What the visual UI renders and streams to the screen - What the voice model needs to know silently versus announce aloud
Marin is asked to pull up the final developer onboarding screen, prompting attendees to connect Codex to the OpenAI Developer plugin and explore the new GPT-Live Starter App repository to begin building their own voice-enabled agent workflows.
[22:03](https://www.youtube.com/watch?v=jDxKzlGLCas&t=1323s) Closing Remarks and Wrap-Up
To wrap up the demonstration, the presenter asks the model if there is anything it would like to say to bring the session to an end. The voice model responds with genuine warmth: “Just a big warm thank you. I loved hanging out with you.” Reacting to the charming goodbye, the host chuckles, remarking, “Oh, so sweet,” before thanking the model for participating throughout the live walkthrough.
Turning back to the audience, the presenter brings the presentation to an official close, thanking attendees and expressing high hopes for the rest of their conference experience: “I hope you've had an amazing DevDay.” Looking forward to developer adoption of real-time voice, delegation, and multimodal capabilities, the host concludes by stating, “And we can't wait to see what you build with GPT Live. Thank you very much.”
## Notable quotes > [01:37](https://www.youtube.com/watch?v=jDxKzlGLCas&t=97s) GPT Live is a full duplex voice model, meaning unlike other voice models which are architected around user and assistant turns, GPT Live is constantly streaming audio in and out. — Alex
> [09:27](https://www.youtube.com/watch?v=jDxKzlGLCas&t=567s) In this full duplex architecture, GPT Live can keep streaming audio to the user, but for more complex requests, it can be helpful to get assistance from a more powerful model or agent harness. — Alex
## Mentioned Alex, Erika, Marin, Colin, GPT-Live, OpenAI, DevDay 2026, Codex, OpenAI Responses API, DALL-E, Fort Mason
## Video description See GPT-Live brainstorm, play games, and create images while coordinating work in the background. Learn to add voice to agents and design for delegation, latency, interruptions, and changing requests.
[Source](https://www.youtube.com/watch?v=jDxKzlGLCas)