OpenAI Upgrades Audio Models as Sierra Tests AI Customer Agents

OpenAI 升级音频模型，Sierra 测试 AI 客服智能体 直接语音到语音架构相比级联技术栈将响应延迟降低高达 200%，使实时语音智能体在企业运营中切实可行。 在本次 Build Hour 网络研讨会中，OpenAI 的 Teri Yu 和 Erika Kettleson 介绍了一套全新的实时语音模型，重点推介了具备 128K 上下文窗口、前置说明（preambles）以及高级并行工具调用功能的 GPT-Realtime-2。通过实时代码演示，他们展示了语音驱动的电商购物和无需手动操作的产品分析仪表板。此外，来自 Sierra 的特邀工程师也加入会议，分享了面向企业客户体验智能体的生产级架构，重点介绍了自定义 VAD 调优、安全护栏和评估策略，最后以互动问答环节结束。 OpenAI 推出全新实时音频模型矩阵，包括支持动态声音克隆与多语言互译的 GPT-Realtime-Translate、流式转录延迟低至 200 毫秒的 GPT-Realtime-Whisper，以及旗舰语音智能模型 GPT-Realtime-2。 (2:06) GPT-Realtime-2 上下文窗口扩充 4 倍至 128K tokens（可容纳约 1 小时连续音频），原生支持前置短语（preambles）、可控表现力与并行工具调用，覆盖 Voice-to-Action、System-to-Voice 及 Voice-to-Voice 三大集成架构。 (4:36) 现场演示展示了该模型在电商导购中自主串联调用多项外部工具，以及在企业产品分析场景下以静默模式操控 UI、仅在被要求总结时输出语音分析的交互能力。 (12:32) 企业客户 Sierra 演示了生产级语音 Agent 的构建经验，通过结合专有 Agent Harness、自定义 VAD 与合规护栏，GPT-Realtime-2 相比传统 STT-LLM-TTS 级联架构将 P50 延迟降低约 30%，P90 延迟降低高达 200%。 (21:24)

OpenAI introduced its real-time audio suite led by GPT-Realtime-2, arguing that direct speech-to-speech architectures and expanded context enable reliable, low-latency enterprise voice agents when paired with strict orchestration harnesses. Real-Time Suite Capabilities and Architecture Upgrades (2:06) OpenAI expanded its audio portfolio with three dedicated models: GPT-Realtime-Translate, supporting over 70 input and 13 output languages with dynamic voice cloning and tone preservation; GPT-Realtime-Whisper, offering configurable transcription latency down to 200 milliseconds across 80 input languages; and the flagship GPT-Realtime-2. The flagship model incorporates a 128K-token context window capable of handling roughly an hour of continuous speech history, parallel tool calling, fine-grained expressiveness controls such as whispering, and preambles that output conversational filler while executing asynchronous function calls. Multimodal Enterprise Workflows and Voice-to-Action (6:58) Demonstrations highlighted how voice-to-action interfaces manipulate complex software systems without relying entirely on continuous verbal exchanges. In an e-commerce scenario, an agent evaluated account history, searched product catalogs, inspected negative reviews, queried external weather forecasts, and populated shopping carts through 15 to 20 background tools while updating the visual UI in real time. In an enterprise analytics workflow, an agent operated silently to isolate conversion drops across Mobile Safari users, generating visual filters and speaking only when explicitly prompted for an executive engineering summary. Enterprise Guardrails and Voice Failure Modes (19:04) Engineers from Sierra emphasized that deploying voice agents in enterprise customer service requires closing strict reliability gaps, noting that an error rate as low as 0.1% creates severe operational risk. Unlike controlled text benchmarks, real-world voice models must handle background noise, conversational interruptions, and passive backchannel cues like 'mhm' without losing task state or corrupting alphanumeric details during tool execution. Direct speech-to-speech processing reduced Sierra's response times by approximately 30% at P50 and up to 200% at P90 relative to cascaded text-to-speech and speech-to-text stacks. Turn Management and Supervisory Context Architectures (30:09) Production deployments rely on targeted API controls and dual orchestration patterns. Erika Kettleson highlighted turn-by-turn Voice Activity Detection (VAD) disabling, which lets developers block barge-in during mandatory legal opt-outs or disclosures while re-enabling it for conversational turns. To maintain context across long-running or chained sessions, Sierra deploys asynchronous supervisor models to inject background prompts or context using methods like conversation.item.create without disrupting live audio streams. While GPT-Realtime-2 resolves major reasoning and latency bottlenecks internally, production reliability remains dependent on surrounding application logic, custom endpoint tuning, and prompt disambiguation to prevent conflicting model behaviors.

OpenAI 推出以 GPT-Realtime-2 为核心的端到端实时语音模型矩阵，将语音交互推进至具备高阶推理与并行工具调用能力的行动智能体阶段；与此同时，企业级实践表明，在严苛的生产环境中保障系统可靠性，仍高度依赖外围定制化工程框架以解决极低容错率、环境噪声和状态漂移等现实挑战。 模型矩阵与端到端架构升级 (4:36) OpenAI 推出了包含 GPT-Realtime-2、GPT-Realtime-Translate 和 GPT-Realtime-Whisper 的实时音频产品组合。其中，GPT-Realtime-Translate 支持超过 70 种输入语言与 13 种输出语言，并具备动态音色克隆与情感音调保留能力；GPT-Realtime-Whisper 支持 80 种输入语言，流式转录延迟可调节至最低 200 毫秒。作为旗舰智能模型，GPT-Realtime-2 具备 GPT-5 级别的推理能力，其上下文窗口扩展 4 倍至 128K token，可支持约 1 小时的连续音频会话，显著降低了长音频会话因截断丢失上下文的风险。 该模型摒弃了传统语音识别（STT）、大语言模型（LLM）和语音合成（TTS）的分层级联管道，采用直接的语音到语音架构。Sierra 的基准测试显示，在同等环境下，这一架构相比级联管道在 P50 响应时间上缩短约 30%，在 P90 响应时间上提升达 200%。为解决大模型推理延迟带来的交互停顿感，GPT-Realtime-2 引入了前置语机制（Preamble），使模型在触发后台异步函数调用时能自然输出过渡性填充词，并支持耳语模式与情感语调的精细化指令控制。 多模态交互范式与工具执行 (6:58) 针对语音交互形态，会议梳理了三种核心模式：语音到动作（Voice-to-Action）、系统到语音（System-to-Voice）以及双向语音对话（Voice-to-Voice）。在 Supply Co 户外电商演示中，智能体展现了 Voice-to-Action 的协同能力：它在用户未手动筛选界面的情况下，自主并行调用 15 至 20 个后端工具，结合用户的历史装备记录、450 美元预算以及 3 至 4 人的规格约束，检索出 419.85 美元的帐篷并实时同步渲染前端页面。智能体还能进一步调取西雅图外部天气预报评估结构防风风险、抓取低星评价进行摘要分析，并基于存储的鞋码（美码 10 号）匹配防水登山靴，将总计 644.70 美元的订单实时写入购物车。 在 Metric Loop 产品分析看板的演示中，交互展现了受控的多模态模式。智能体在排查欧洲移动端转化率骤降问题时保持静默执行模式，完全通过后台工具调用过滤数据与调整可视化组件，精准定位出 Safari 移动端在商品详情页尺码选择器上的软件回归漏洞，并在仅收到明确口述指令后才生成精炼的工程总结。这种设计消除了传统深层菜单点击的操作摩擦，同时避免了不必要的冗长语音干扰。 企业级落地的容错门槛与失效模式 (19:04) 客户体验平台 Sierra 的工程师 Ken Murphy 和 Soham 指出，在面向财富 100 强企业的实际服务中，语音交互面临极高业务风险。AI 智能体并非单纯生成文本，而是在直接做出 API 触发与数据库写入决策；在企业服务场景下，即使是 0.1% 的幻觉或工具执行错误率，都可能引发严重的合规风险与商业损失，导致未经约束的通用模型无法直接部署上线。 Soham 进一步剖析了真实语音环境中特有的级联失效模式：其一为拼写恢复脱节，当用户口头更正复杂姓名或拼写时，模型即使口头确认，内部工具调用参数却常继续沿用错误的旧值；其二是背景反馈声（Backchannel）误读，人类自然的“嗯”、“啊”应答常被模型误判为正式交接对话轮次；其三是在遭遇打断时，正在进行的思维推理链极易中断并丢失状态；其四是高风险操作的不可逆性，例如机票预订业务中因语音误解导致退票，将带来严重的现实后果。为此，Sierra 必须在底层模型之外构建包含业务规则、数据脱敏及 PCI 支付安全合规在内的外层架构 Harness。 生产环境调度：VAD、会话与异步集成 (30:09) 为应对真实场景中车载噪音、背景杂音与孩童干扰，企业在生产部署时往往需要结合定制化语音活动检测（VAD）。OpenAI 解决方案工程师 Erika Kettleson 强调了一项关键控制实践：按对话轮次选择性关闭 VAD。例如在播报强制性法律免责声明或支付协议时临时禁用打断检测，确保合规文本 100% 完整传达，而在互动问答轮次中重新开启低延迟打断功能。 针对超过 1 小时的长音频会话，技术团队建议采用分段串联（Session Chaining）策略，并在因网络掉线或超时重建连接时，利用 128K token 的上下文容量将历史状态重新注入新会话。在架构协作方面，系统支持双轨模式：一方面可利用后台独立的大模型（如 GPT-5）担任异步监督者，防止语音模型逻辑偏航；另一方面，开发人员可调用 conversation.item.create 接口静默注入后台耗时工具的执行结果或外部上下文，使智能体在不中断连续前台语音交流的前提下无缝获取最新数据。此外，将复杂系统提示词回传给大模型检测逻辑冲突并进行重构，是提升指令遵循一致性的实用工程技巧。 本次技术分享展现了端到端语音大模型在延迟优化、多模态同步与复杂推理层面的实质进展，确立了语音作为高意图操作界面的技术可行性；然而，真实工业落地依然面临复杂声学环境过滤、长效状态保持以及极端场景下极低容错率的严苛限制，这要求开发者在底层通用推理能力与外围确定性工程 Harness 之间建立严密的架构协同。

## Background 该视频由 OpenAI 于 2026 年 6 月 3 日发布在其官方 YouTube 频道，属于“Build Hour”技术研讨直播系列。视频简介指出，本次活动聚焦于 GPT-Realtime-2、GPT-Realtime-Translate 与 GPT-Realtime-Whisper 三款全新实时语音模型，旨在演示如何构建具备低延迟、跨工具推理及原生多模态操作能力的生产级语音智能体，并邀请了企业客户 Sierra 深入分享一线落地实践。 ## TL;DR 本期活动全面剖析了 OpenAI 全新实时语音模型矩阵的核心特性，重点展现了旗舰模型 GPT-Realtime-2 所具备的 128K 超大上下文、原生思维链推理与并行工具调用能力。主讲人通过电商语音导购与静默操作数据分析看板两个现场演示，具象化呈现了从“拟人对话”迈向“语音驱动应用操作（Voice-to-Action）”的范式转变。企业客户 Sierra 深入分享了严苛客服场景下应对背景噪音、误判抢话与业务漂移的 Agent Harness 架构体系。最后，团队围绕打断判定（Barge-in）、长通话状态注水恢复及异步主管纠偏等高频工程痛点进行了详尽解答。 ## Atmosphere & style 整场研讨会散发着硅谷技术前沿特有的极客兴奋感与务实作风，讲者一边调侃“向 Demo 之神祈祷”，一边行云流水地完成多工具实时联动，展现出极强的技术自信。从 OpenAI 团队畅想人机语音交互终将取代打字的宏大愿景，到企业一线工程师冷静拆解真实生产环境中“0.1% 错误即不可上线”的残酷挑战，现场氛围在浪漫的创新期许与严密的工程敬畏之间形成了张力十足的平衡。快节奏的代码演示与高密度的实战问答让整场直播信息量拉满，使人切实感受到端到端语音智能体正迎来跨越工业应用门槛的历史性拐点。 ## Bulletin - [02:06](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=126s) OpenAI 推出全新音频模型矩阵，包括原生端到端语音大模型 GPT-Realtime-2、支持 70+ 语言流式互译的 GPT-Realtime-Translate 以及延迟低至 200ms 的 GPT-Realtime-Whisper。 - [04:36](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=276s) GPT-Realtime-2 将上下文扩容 4 倍至 128K Token，原生支持并行工具调用与 Preambles 拟人化垫话机制，可容纳近 1 小时连续多轮对话而不丢失上下文。 - [21:54](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1314s) 企业级客服企业 Sierra 分享落地实践，采用端到端模型后中位数延迟降低 30%，P90 尾部极端延迟实现 200% 的速度飞跃，并配合定制 VAD 与 Agent Harness 运行底座解决复杂杂音与合规控制问题。 - [37:09](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=2229s) 通过 conversation.item.create 接口，开发者可在后台异步工具执行完成后静默向历史会话注入数据，无需触发即刻语音响应打断当前对话。 ## Key takeaways - [02:06](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=126s) OpenAI 推出 GPT-Realtime-2、GPT-Realtime-Translate 与 GPT-Realtime-Whisper 三款核心音频模型，推进极低延迟实时语音与动作交互。 - [02:06](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=126s) GPT-Realtime-Whisper 实现 200ms 级转写延迟与 80 种语言输入，能更早触发下游函数调用；GPT-Realtime-Translate 支持多发言人音色与语气动态保留。 - [04:36](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=276s) 实时语音开发范式被归纳为三种：语音到动作（Voice to Action）、系统到语音（Systems to Voice）和语音到语音（Voice to Voice）。 - [04:36](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=276s) GPT-Realtime-2 上下文窗口跃升至 128K Token，支持原生并行工具调用（Parallel Tool Calling）并内置 Preambles 垫话机制掩盖后台推理延迟。 - [06:58](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=418s) 电商搜索 Agent 演示展现了端到端模型挂载 15 至 20 个工具的能力，不仅能分析低星差评并结合实时天气提供决策，还能直接驱动前端 UI 联动交互。 - [12:32](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=752s) Metric Loop 看板演示了静默式 Voice-to-Action 模式，模型理解何时需要纯静默操作 UI 筛选数据，何时按需进行口头归因汇报。 - [18:36](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1116s) 企业级应用的核心挑战在于将错误率降至极低：0.1% 的业务幻觉或越权操作即可导致 Agent 无法上线，这需要严苛的评估与约束架构。 - [21:54](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1314s) 原生 Voice-to-Voice 模型相较传统级联方案（ASR+LLM+TTS），在 Sierra 生产呼叫中将 P50 延迟降低 30%，P90 极端延迟优化达 200%。 - [24:34](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1474s) 真实生产呼叫中复合故障模式呈指数级爆发，包含逐字拼读错误反复固守、高风险业务理解漂移，以及对用户无意识反馈声音（Backchanneling）的机械抢话。 - [30:09](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1809s) OpenAI 透露可通过单轮精准控制机制（Turn-by-turn VAD Disabling）在播报免责声明或关键条款时强制禁用用户打断，确保合规内容不可打断。 - [32:15](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1935s) 口述语速约为打字速度的 4 倍，极大降低了用户表达长上下文意识流意图的认知成本，使复杂咨询与随性倾诉更具效率。 - [35:50](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=2150s) 针对超长通话与意外掉线场景，最佳实践不是维持无限 WebSocket 连接，而是通过状态注水（Rehydration）将历史快照写入新会话并利用 128K 上下文无缝接力。 - [37:09](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=2229s) Sierra 生产环境采用异步主管架构（Supervisor Pattern）在会话中动态审计纠偏，结合 GPT-Realtime-2 的内置思考与前沿文本模型分级路由处理复杂决策。 - [40:01](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=2401s) GPT-Realtime-2 指令遵循能力极强，若系统提示词出现逻辑矛盾会导致模型撕裂，建议直接调用大模型自审反思提示词内部的隐性冲突。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=0s) 欢迎与开场介绍 OpenAI 初创生态营销负责人 Sarah Urbonas 热情欢迎大家来到新一期的 Build Hour，并介绍了两位在 OpenAI 内部无需过多介绍的重量级嘉宾——多模态 API 产品经理 Teri Yu 以及解决方案工程团队的 Erika Kettleson（负责帮助平台顶尖的数字化原生客户实现 API 规模化落地）。本期活动的核心焦点是刚刚发布的 GPT-Realtime-2 模型。Sarah 表示，Build Hour 旨在向开发者和创业团队提供最佳实践、工具以及 AI 专业知识，帮助无论是初创企业创始人还是大公司工程师的受众，迅速将最新模型落地到生产环境中，加速业务扩展。 本期一小时的议程非常紧凑：首先回顾最新发布的实时语音模型族及其技术定位；随后通过两个现场实时构建的 Demo 展示核心能力——一个是语音驱动的电商搜索 Agent，另一个是产品分析看板（Erika 调侃称现场实时演示总得“向语音 Demo 之神祈祷一切顺利”）；最后邀请来自客户企业 Sierra 的团队分享基于 GPT-Realtime-2 构建生产级语音客服 Agent 的一线实战心得。 - **活动定位**：通过现场代码与 Demo，指导开发者将 OpenAI 语音模型快速部署至生产环境。 - **互动机制**：观众可在屏幕下方提交 Q&A，由后台专家实时解答，并在活动结尾安排专门的三方联席答疑环节。 - **开源与资源**：代码仓库通过 GitHub（github.com/openai/build-hours）同步开放。 --- ### [02:06](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=126s) 实时语音模型矩阵与实时翻译能力演示 Teri 指出，OpenAI 在前一周四集中发布了三款全新的音频模型。为了让大家身临其境地感受其突破，团队决定直接从多语言交互 Demo 开场，并让聊天室观众发弹幕说明自己所在的国家或地区。鉴于聊天室里有大量讲西班牙语的观众，Erika 迅速将演示语言切换为西班牙语，并邀请大清早还没吃早饭的 Teri 尝试说话。Teri 随口聊起了自己饥肠辘辘时最想吃的早餐——阿萨伊碗（Açaí bowl）以及燕麦奶拿铁，屏幕上随即实时显现出左侧的原始英文转录和右侧流畅地道的西班牙语流式翻译。 这次互动直接展示了前两款全新的底层模型：首先是 **GPT-Realtime-Translate**，它能彻底消除跨语言交流的隔阂，支持超过 70 种输入语言和 13 种输出语言的极低延迟流式互译，非常适合视频会议、跨国直播与全球化客户服务场景；其次是暗中提供强力支撑的 **GPT-Realtime-Whisper** 实时语音识别模型，具备可低至 200 毫秒的超低延迟与 80 种语言的输入支持，为下游任务带来了更早的函数调用（Function Calling）与更好的指令遵循能力，可赋能实时字幕、会议纪要生成与环境式 Agent（Ambient Agent）上下文捕获。此外，系统在翻译时还结合了动态声音克隆与音色匹配（Tone matching），甚至能分辨并保留不同发言人的说话风格。 ```mermaid flowchart LR accTitle: 实时语音处理流水线 accDescr: 展示用户音频如何流经 GPT-Realtime-Whisper 与 GPT-Realtime-Translate，最终合成目标语言语音的极低延迟链路 subgraph AudioIn["音频采集"] A["多语言原始语音输入"] end subgraph CoreModels["实时模型处理链路"] B["GPT-Realtime-Whisper<br/>(延迟低至 200ms / 80+ 语言)"] C["GPT-Realtime-Translate<br/>(70+ 输入 / 13+ 输出)"] end subgraph AudioOut["动态交付"] D["音色与情绪匹配输出<br/>(动态声音克隆)"] end A --> B B --> C C --> D ``` - **GPT-Realtime-Translate**：支持 70+ 种输入语言、13 种输出语言，面向实时通话与跨境客服。 - **GPT-Realtime-Whisper**：流式转录，延迟可调低至 200ms，显著提早触发 Tool Calling。 - **音色动态匹配**：多发言人场景下可保留说话者的原始语调与情绪表达。 --- ### [04:36](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=276s) 语音 AI 构建范式与 GPT-Realtime-2 新特性 作为本次发布的核心旗舰，**GPT-Realtime-2** 是 OpenAI 目前最智能的语音模型，正式将“GPT-5 级别”的推理能力引入了实时语音交互。该模型在指令遵循、对话中工具调用（Tool Calling）以及多语言性能上均实现了巨大跃迁，而这三点正是语音应用进入生产级工作流的决定性基石。Teri 将基于实时语音的开发场景梳理为三种主流范式： 1. **语音到动作（Voice to Action）**：全免提的语音驱动型应用程序； 2. **系统到语音（Systems to Voice）**：如同能够将各类后台数据和日程组织成自然对话的“语音幕僚长（Chief of Staff）”； 3. **语音到语音（Voice to Voice）**：高自然度的端到端交互，如 T-Mobile 等跨国巨头部署的新一代客户服务电话。 GPT-Realtime-2 带来了一系列针对生产痛点打造的核心升级。首先是“前置垫话/开场白（Preambles）”功能，Agent 在执行复杂耗时的推理或检索时，会像真人一样自然地说出“让我帮您查询一下（Let me check on that）”，避免尴尬的静音；其次，上下文窗口扩大了 4 倍，一举跃升至 **128K**，足可容纳近一个小时的持续对话而无需强行截断，大幅增强了多轮任务的上下文连贯性；再者，支持并行工具调用（Parallel Tool Calls），彻底告别了过往串行“瀑布流（Waterfall）”式的低效等待。此外，模型在医疗、AI 等垂直领域的术语理解有了质的提升，且支持可控的表现力调节——开发者可以在提示词中直接要求它“低语窃窃私语”、“表现得格外兴奋”或“带着嫉妒的语气”。 - **上下文窗口扩展**：从上一代扩容 4 倍至 128K 上下文（约对应一小时连续交互）。 - **工具调用升级**：原生支持并行函数调用，消除串行工具执行瓶颈。 - **拟人化反应**：内置 Preambles 机制掩盖后台推理延迟；表现力（如窃窃私语、兴奋等）可通过 Prompt 精准控制。 --- ### [06:58](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=418s) 现场演示：语音驱动的电商搜索 Agent Erika 带来了第一个重磅现场 Demo：一个带有语音搜索 Agent 的户外装备电商网站（Supply Co）。这里的 Realtime API 不单是陪伴聊天的助手，而是能够“通过工具调用真正接管并操作前端 UI”的智能体。Erika 设定了一个具体的生活场景：她正在筹划一次去太平洋西北地区（Pacific Northwest）的徒步旅行，此前已经采购了部分装备，现在需要助手帮忙清点并继续采买。 Erika 开启免提对话：“嘿，能提醒我上次停在哪儿了吗？我这次旅行还需要买什么？”Agent 迅速调用用户数据工具，给出了清晰回应：“我来调出你的购物计划，看看还缺什么。你还需要一顶帐篷和徒步鞋；你的日用背包、越野袜和保温水壶已经买好了。你想先挑哪样？” 随后，Erika 要求筛选“450 美元以内、适合 3 到 4 人”的帐篷。Agent 实时触发前端 UI 过滤，同时口头总结两款入选装备：一款是评分较高、售价 419.85 美元且次日免运费的帐篷；另一款是售价 357 美元的快速搭建款，但库存不足。Erika 紧接着要求查看高价帐篷的一星和二星差评。Agent 在后台扫视低分评论后，条理分明地提炼出要点：差评主要集中在首次搭建比想象中耗时，且抗强风暴雨性能较弱；紧接着 Agent 主动给出了极具环境感知的建议——“如果是典型的太平洋西北地区周末，这些缺点在中小雨天气下不算硬伤，但如果你预期会遭遇恶劣天气，我建议换一款更防风暴的帐篷。” ```mermaid flowchart TD accTitle: 语音电商 Agent 多轮任务流 accDescr: 展示用户通过语音下达查询需求，Agent 调用个人购物清单、商品库、差评总结与天气 API 并联动操作前端 UI 的过程 UserQuery["用户语音指令<br/>('我还缺什么？找450美元以内的帐篷')"] --> AgentBrain["GPT-Realtime-2 推理引擎"] subgraph ToolInvocations["并行与跨域工具调用"] T1["检查历史订单 / 购物清单"] T2["商品搜索与价格过滤 API"] T3["差评聚类与内容扫描 API"] T4["西雅图天气预报 API"] end subgraph UIandAudio["执行与多模态反馈"] UI["前端页面联动操作<br/>(高亮商品 / 展开详情页 / 加车)"] VoiceOut["自然语音回复<br/>(前置垫话 + 购买决策建议)"] end AgentBrain --> ToolInvocations ToolInvocations --> UIandAudio ``` 听到防风抗雨的考量，Erika 顺势提出新要求：“我有一阵子没看天气了，能帮我查查下下个周末西雅图地区的天气预报吗？”Agent 立即调用外部天气查询工具并评估风暴风险：“风暴风险中等，会下一些雨并伴有微风，但不是典型暴风雨。那顶帐篷应付普通的下雨周末完全足够，但我建议搭配地布（Footprint）和坚固的地钉。”得到满意的解答后，Erika 指示将帐篷加入购物车并转而寻找防水徒步鞋；Agent 一边发出“我把帐篷加进购物车，然后查询你的尺码并寻找防水靴”的 Preambles 垫话，一边高效完成了加车操作，并直接调出美码 10 号（US 10）的男女推荐款，无缝切换进入下一个购买决策流程。 - **多轮深层推理**：Agent 不仅按价格和人数筛选商品，还能自主分析低星评论并结合旅行目的地的实际气候提出购买建议。 - **多工具跨域联动**：单一语音会话中连续打通了购物车系统、商品搜索、用户画像（鞋码尺寸）及外部第三方天气预测。 - **UI 实体操控**：伴随语音对话，前端网页实时同步完成商品高亮、筛选条件设置、详情页跳转以及商品添加进购物车等全套界面操作。 ### [10:29](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=629s) 结账与反思：从单纯对话迈向多工具视觉交互 在 Supply Co 电商网站的购物演示尾声，Erika 敲定了自己两周内的徒步装备，要求语音助手把美码 10 号（US 10）的登山靴加入购物车并打开购物车查看总价。语音助手不仅迅速完成了界面跳转，还清晰地播报出账单细节：“你的购物车里有帐篷和靴子，总金额是 644.70 美元”，并诚实地提示无法在此直接完成最终扣款。随后 Erika 随口询问是否有推荐的选配件，助手立即从容地建议添加防雨罩（rain layer）和帐篷地布防风地钉套件（tent footprint stay kit），以应对潮湿多风的天气。当 Erika 表示今天先不买防雨罩、感谢协助时，助手以极其自然的语调道别：“不客气，Erika。祝你旅途愉快，等你准备好买防雨罩时我随时都在。” Erika 随后停下来分享了这项演示背后令人惊艳的技术跃迁。她强调，虽然模型的声音质感极佳、非常自然且极具可控性（steerable），但真正颠覆体验的核心在于**模型展现出的极高智能**。以往的语音系统往往局限于“语音输入、单次动作输出”的机械交互，而这次后台为模型挂载了多达 15 到 20 个工具——这在以往的 Realtime 模型中是完全不可想象且极不推荐的做法。GPT-Realtime-2 能够自主审视当前页面的 UI 状态，在海量工具间精准选择，甚至能够按需连续触发多个工具（包含查询外部天气预报），让用户无需离开购物页面就能完成所有决策闭环。 - **多工具调度能力**：单次会话接入 15–20 个工具，支持复杂的多步骤连续调用（parallel & sequential tool calling）。 - **界面与数据联动**：模型在与用户自然对话的同时，实时更新前端视觉界面与购物车状态，实现真正的“边说边操作”。 - **外部知识无缝融合**：将外部天气工具与商品库存、尺码、配件建议融合，大幅减少用户在不同页面和应用间的跳转成本。 --- ### [12:32](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=752s) Voice-to-Action：产品分析仪表盘与静默工具执行 ```mermaid flowchart TD accTitle: 语音驱动的产品数据分析流 accDescr: 展示产品经理通过语音指令驱动 Metric Loop 仪表盘排查异常、触发工具并获得结构化口头汇报的流程 subgraph VoiceToAction ["语音驱动动作 (Voice-to-Action)"] PM["产品经理口头指令<br/>'过滤欧洲数据 / 对比前后7天'"] --> Router["GPT-Realtime-2<br/>指令理解与工具路由"] Router --> SilentOps["静默更新 UI 与筛选器<br/>(不产生冗余语音回复)"] end subgraph Investigation ["深度排查与分析"] SilentOps --> MultiFilter["组合下钻<br/>语音搜索 + 新用户 + 鞋类"] MultiFilter --> RootCause["调用根因分析工具<br/>对比 Mobile Safari 与 Chrome"] end subgraph OralSummary ["按需口头汇报"] RootCause --> AudioReport["两句话口头归因总结<br/>'PDP 尺码校验缺陷阻碍加购'"] end ``` Erika 随后切换视角，从普通消费者变成了 Supply Co 的产品经理（PM），展示了一个全新的工作流概念：“Voice-to-Action”（语音转行动）。在传统认知中，语音 AI 总是遵循“我说一句、它回一句”的来回对话模式；但对专业工作流而言，用户往往希望直接驱动软件操作，并不需要助手在每一次筛选操作时都插话确认。Erika 打开了名为 Metric Loop 的复杂产品分析仪表盘，面对繁多密杂的图表，她直接开启语音会话发出指令：“帮我按欧洲地区筛选，我最近听到不少那边的反馈……再看一下最近 7 天，并与前 7 天做个环比对比。” 系统迅速且完全静默地调整了看板，高亮显示了转化率的大幅下滑，并指明 Safari 浏览器存在明显异动。Erika 没有手动去点击界面上的复杂复选框，而是继续让助手建议并追加过滤条件：“按语音搜索、首次购买用户和鞋类商品进行筛选”。在确认欧洲区新客鞋类购买流程出现断崖式下跌后，她立即指示模型启动根因调查，重点对比移动端 Safari 与 Chrome。调查完成后，Erika 明确下达了发声指令：“Metric Loop，用两句话大声向我说明这次根因调查的结果，以便我向工程团队解释。”模型随即给出专业精准的口头复盘：“调查显示这是针对移动端 Safari 的特定功能回退，商品详情页（PDP）的尺码选择器校验无法正常更新，导致欧洲首次购买鞋类的用户在选好尺码后卡住无法加入购物车。而 Chrome 的数据接近基线，说明这是 Safari 上的 PDP 发版行为缺陷，而非整体流量质量或搜索层面的问题。” Erika 总结指出，这种极度可靠的指令遵循（instruction following）能力是模型智能的重大突破。模型能够感知何时应当保持安静纯粹执行 UI 动作，何时需要用语音总结；用户随时可以点开推理链路查看底层思考日志和对话转录，甚至通过打字介入协同。结合模型强大的代码编写、图表渲染与复杂虚拟数据推理能力，它就像一个随时待命的驻场数据分析师（analyst in the loop），在完全不剥夺开发者对数据和 UI 控制权的前提下，实现了实时动态探索。 --- ### [17:24](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1044s) 语音交互的多元未来与场景展望 会话转回到产品负责人 Teri Yu。Teri 热情地分享了语音交互正在解锁的广阔应用蓝海。随着 GPT-Realtime-2 等模型在延迟与工具调用上的跨越式进步，语音技术不再仅仅是外挂的语音助理，而是演变为贯穿各类终端与应用程序的核心交互层（voice layer）。她列举了多种充满想象力的落地场景： - **智能硬件与设备集成**：无缝融入各种物联网硬件与智能终端，实现脱离屏幕的自然直达操作。 - **编程助手与开发体验**：程序员能够通过流畅的语音指令重构代码、排查错误，解放双手以加速开发流程。 - **移动 App 语音浮窗（Voice Box）**：在任何现有移动应用内快速植入语音模块，直接执行应用内深层操作。 - **互动娱乐与游戏控制**：在视频游戏中通过简单的语音口令控制角色的跳跃、移动和动作，创造全新的体感与游戏机制。 - **专业辅导与身心训练**：涵盖即时反馈的公众演讲教练、自动整理思路的会议记录员、私人财务顾问以及具备 Agent 能力的音视频会议协同助手。 Teri 动情地展望道，人机交互的范式正在发生代际转移——“在未来，孩子们与 AI 说话的频次，很可能远远超过他们打字交互的频次”。此外，她还特别提到本次演示幻灯片中的概念图全部由新一代图像模型生成，开发者在构思产品原型和创意概念验证时，也可以充分借助这一工具进行视觉化探索。 --- ### [18:36](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1116s) 客户焦点：Sierra 如何为企业级客户打造生产级语音 Agent 随后环节进入了重磅的客户分享，来自知名 AI 客服初创企业 Sierra 的语音工程团队成员 Ken Murphy 与 Soham 连线接入。Ken 介绍道，Sierra 的核心使命是帮助企业利用 AI 打造更卓越、更具人情味的客户体验。在实际业务中，Sierra 的 AI Agent 绝非只是机械回答常见问题的问答机器人，而是能够端到端彻底解决真实售后与业务问题的自主智能体。其客户群覆盖了包括《财富》100 强企业在内的大量顶尖品牌，这意味着系统必须在极高的吞吐并发下保持极高稳定性，无缝对接企业内部极其复杂的异构系统，并分毫不差地遵守复杂的业务规范。 Ken 强调了企业级落地中极其严苛的工程挑战：Agent 不仅是在生成自然的回复文本，更是在决定何时该采取行动、该调用何种工具、从庞大的知识库中调取哪项政策，以及特定动作是否越权或合规。在直接面向消费者的场景中，极其微小的错误率都会迅速放大为巨大的商业与合规风险，“哪怕只有 0.1% 的概率幻觉出一条错误政策或执行了错误操作，这个 Agent 都是绝对不可上线的（not shippable）”。因此，工程上的真正壁垒不仅在于让声音听起来有多自然，更在于如何构建、评测、约束和运营这套值得企业托付品牌声誉的系统。 随后，团队现场演示了一段名为 Demo Lots 的房产中介 AI 坐席“Jade”的外呼测试音频： > **Jade**：“嗨 Ken，我是 Demo Lots 的 Jade。我们带来了一个好消息，但首先我必须声明：您可以随时通过口头表达退出本次通话，且为了服务质量，本通电话可能会被录音。上次我们在 3 月份交流时，您当时一直在关注诺伊谷（Noe Valley），请问目前的意向区域有变化吗？” > **Ken**：“嗯……是的，我依然在看旧金山，但现在改看日落区（Sunset）了。” > **Jade**：“留在旧金山并转向日落区对很多买家来说都非常合理。请问您是在寻找独栋别墅（single-family home）还是公寓（condo）？” > **Ken**：“我想找独栋住宅。” > **Jade**：“明白，日落区的独栋确实很合适。请问您的预算或最高购买价格是多少？” > **Ken**：“大概 35 万美元左右。” > **Jade**：“您的持牌按揭顾问 Sarah 随时可以提供协助，您尽可放心交给我们。” 音频演示结束后，Ken 深入剖析了他们对 GPT-Realtime-2 感到极其兴奋的技术动因。语音交互是一种“极其不容忍缺陷（unforgiving）”的媒介，在日常交谈中，即使是半秒钟的迟滞都会让人感觉突兀或系统崩溃。GPT-Realtime-2 直面并解决了当前生产环境中最棘手的问题：极低延迟、轮流发言机制（turn-taking）、深度推理以及高质量发音。端到端的语音到语音（voice-to-voice）架构彻底规避了以往将语音切分为外部语音转文字（STT）和文字转语音（TTS）的传统分步流水线，为高拟真对话带来了根本性的质变。 ### [21:54](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1314s) Sierra Agent Harness 生产级架构与端到端延迟飞跃 Sierra 团队指出，原生端到端（Voice-to-Voice）实时模型能够直接剔除传统级联链路（ASR $\to$ LLM $\to$ TTS）中的多个环节，使交互显著更加敏捷、平滑且具人性化。然而，GPT-Realtime-2 仅构成底层基石，要在真实商业生产环境中稳定、安全地服务全球超大型企业，核心在于构建围绕模型运转的 Agent Harness（智能体运行底座与控制架构）。 ```mermaid flowchart TD accTitle: Sierra Agent Harness Architecture accDescr: 展现从嘈杂音频输入、经由定制VAD、Agent Harness生产治理到GPT-Realtime-2实时模型的端到端执行流程。 subgraph InputStage["输入与边界检测"] A["嘈杂真实音频\n(背景噪音/方言口音/插话抢话)"] --> B["Sierra 自定义微调 VAD\n(精细化轮替控制)"] end subgraph HarnessStage["Sierra Agent Harness 生产控制层"] B --> C["工作流与客户业务策略编排"] C --> D["企业品牌调性与提示词 Guardrails"] C --> E["敏感信息脱敏 Redaction & PCI 合规支付"] C --> F["全链路调用追踪 Tracing & 确定性工具集"] end subgraph CoreModel["端到端语音大模型"] D --> G["GPT-Realtime-2\n(端到端语音转语音推理)"] E --> G F --> G end ``` 通过 Agent Harness，Sierra 针对不同客户定制独立工作流，严格管控模型可用工具、品牌语言规范、安全防护栏（Guardrails）以及业务规则对齐（Grounding）。此外，面对背景噪音干扰、方言口音、中途打断及语向突变等真实挑战，Sierra 专门训练了定制化微调的 VAD（Voice Activity Detection，语音活动检测）模型接管轮替判断（Turn-taking）。Harness 同时内嵌链路追踪（Tracing）、敏感数据脱敏（Redaction）以及符合 PCI 标准的安全支付流程，将原始模型打造成可控、可观测的工业级系统。 - **延迟性能基准测试**：与传统级联堆栈相比，GPT-Realtime-2 在 Sierra 真实呼叫测试中，中位数延迟（P50）降低了约 30%，尾部极端延迟（P90）更是实现了高达 200% 的速度飞跃。 - **声音质感评测**：内部测评显示，GPT-Realtime-2 的原生语音表现力已完全能够匹敌市面上顶尖的独立专用语音合成（TTS）方案。 - **端到端仿真评估体系**：传统级联架构下转写、Agent 逻辑与合成往往各自独立评测；而在 Voice-to-Voice 系统中，评测必须覆盖“倾听—推理—行动—表达”的全链路闭环。Sierra 依靠回放真实业务通话的定制模拟系统，核心指标直指“任务是否真正完成”，而非“听起来好不好听”。正如讲者强调：“这才是生产级语音的硬门槛——不是做出一个声音像人的 Agent，而是做出一个值得托付具体工作的 Agent。” --- ### [24:34](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1474s) 真实语音的残酷性与生产级 Failure Modes 剖析 Sierra 语音研究团队的 Soham 登台分享，他呼应了前面主讲人的观点：“Erika 常说语音是她最爱的交互模态，而 Ken 则指出语音环境极其苛刻、容错率极低（unforgiving）——有趣的是，这两句话同时完全成立。” 现实客服语音极其微妙复杂，绝非理想实验室样本。理想文本交互边界分明、语意明确无损，而真实生产通话“从无法挑选输入音频”：用户可能正在高速公路路肩求助，可能在车内伴随着吵闹的孩子，或者脾气焦躁、频繁强行插话打断。Agent 既要在极其恶劣的声学环境下保持对话，又必须准确推动业务流运转。 当对话生成与工具调用（Tool Calls）相交织时，复合故障模式（Compound Failure Modes）会呈指数级爆发： | 故障类别 | 典型表现模式 | 生产环境具体危害与连锁反应 | |:--- |:--- |:--- | | **字符拼读错误** (Spelling Errors) | 用户逐字拼写姓名、邮箱或证件编码时漏记或错认单字母 | Agent 缺乏优雅容错恢复机制；随后调用下游工具 API 时反复固守最初的拼写错误，陷入自身制造的逻辑泥潭无法自拔 | | **高风险业务理解漂移** (High-Stakes Logical Failures) | 航班取消、改签等场景中发生实体指代错误 | 用户原本需要取消特定行程，Agent 误触另一班次并实际执行；在航空客服等高风险场景下直接导致旅客滞留异地 | | **反馈回音误判** (Backchanneling Mishandling) | 无法区分用户无意识的伴随反馈（如 “mhm”、“ah-ha”、“yes”）与实质指令 | 大多数通用语音模型被训练为“听到声音就必须作答”，缺乏人类根据语境选择性忽略（Selective ignoring）的默契，造成机械式抢话 | --- ### [28:42](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1722s) Thinking Models 在多轮高抗扰语音中的突破性进展 传统级联模型虽有延迟包袱，但过往工程师能通过针对性规则针对复杂边界情况进行深度定制。然而，端到端 Voice-to-Voice 模型如今展现出惊人的上下文容纳能力与复杂推理跃升。Soham 指出，Dialpad 等业界评测榜单已完全被“具备深度思考能力的推理模型（Thinking Models）”所统治。 但在语音交互中实现“边想边聊”绝非易事： - 系统绝不能像文本终端那样直接陷入死寂的缓冲等待。 - Agent 必须学会使用拟人化过渡用语（如 "Okay, give me a moment." 或 "Let me think about that."）。 - 在模型思考推演或调用工具的关键区间，系统必须无缝支持被用户随时打断，且在打断后精准维持状态机（State Machine）完整性，绝不能丢弃会话进展。 随着 GPT-Realtime-2 在复杂逻辑推理与跨语种能力上的重大飞跃，业界长期期待的端到端原生语音正式迈入主流工业落地阶段。 --- ### [30:09](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1809s) 问答环节：打断判定（Barge-in）与 VAD 生产落地策略 在演讲结束后的 Q&A 环节中，线上开发者最集中的疑问聚焦在打断判定（Barge-in Detection）以及“到底应该直接信任平台内置的端测轮替，还是自建应用层检测逻辑”。 - **Sierra 的领域特定选择**：GPT-Realtime-2 开箱即用的 VAD（涵盖语义断句与服务端检测）在通用场景表现出色。Sierra 之所以额外叠加定制微调 VAD，是因为其客群深度集中在客服呼叫中心，普遍充斥高分贝家庭电视杂音、孩童吵闹等非目标人声，专用模型针对此类脏数据进行了极端抗噪训练。 - **环境适配与 Padding 调优**：OpenAI 团队建议，不同终端调用设备与输入声学环境下，音频缓冲填充（Padding）参数存在显著差异，开发者必须在真实硬件路径上执行端到端 Tuning。 - **单轮次精准禁用 VAD（Turn-by-turn VAD Disabling）**：OpenAI 工程师透露了一个鲜为人知的高级实践——开发者可以在特定轮次（Turn-by-turn）通过 API 主动关闭 VAD 打断功能。例如在朗读关键免责声明（Legal Disclaimers）或高风险确认信息时，直接强制禁用中断机制；一旦关键语句播报完成再恢复监听。这种做法彻底跳出了单纯依赖 Prompt 指令约束（Instruction following）的不确定性，在底层协议层实现了不可打断的执行保障。 ### [32:15](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1935s) 实用型 AI 助手与重塑业务流的本质差异 现场问答环节抛出了一个极具分量的问题：“究竟是什么将一个普通好用的 AI 助手，与真正能够改善现实商业工作流的生产级系统区分开来？” 主持人调侃这简直是给 Sierra 团队量身定制的“送分题”。OpenAI 团队首先分析了语音与文本模态的权衡：文本虽然拥有极高的信息密度，但语音擅长处理“快速捕获与快速意图”（fast capture, fast intent）。用户在开车、移动中，或者脑海中只有一个模糊宽泛的想法时，语音交互远比打字便捷，模型能够将碎片化、意识流的口述重构为逻辑清晰的提示词。此外，年轻人以及巴西、印度等具有“语音优先（voice-first）”习惯的国家和地区，表现出显著的模态偏好。更核心的优势在于效率差异——人们说话的速度大约是打字速度的四倍（"you can speak like four times faster than you can type"），这使得用户在咨询购房、选车等复杂决策时，更愿意通过这种具有亲密感与随性风格的方式倾诉大量背景信息。 Sierra 团队随后从生产实践角度补充了更残酷的技术现实：真实的客服场景充满了打断、背景杂音与各种口音，即便在基准测试中表现良好，也无法直接代表生产流量的复杂性。生产级 Agent 的关键不在于单轮的机智应答——GPT-Realtime 虽然擅长在单轮对话内迅速响应并决定行动，但核心难点在于“整通通话的编排（orchestrate the entire call）”。Sierra 需要针对不同客户的具体业务（例如退货与退款政策）定义多步骤的工作流，并在恰当的业务节点将最精准的上下文暴露给模型。 对话间还穿插了一段有趣的幕后花絮：OpenAI 员工在办公室走动时，越来越频繁地听到同事对着电脑说话。导播制作团队在准备这场直播时，甚至直接戴着耳机通过 GPT-Realtime 实时模型与 Codex 编程助手语音对话，大家都在切实践行“吃自家狗粮（drinking our own champagne）”。 - **语音与打字速度对比**：口述语速约为键盘打字速度的 4 倍，极大降低了用户表达长上下文意识流意图的认知阻力。 - **地域与人口学特征**：印度、巴西等国家以及年轻一代展现出强烈的语音优先倾向。 - **单轮与全局编排的分界**：基础实时模型擅长单轮快问快答与工具触发，而商业级 Agent 依赖对多阶段复杂业务逻辑（如企业退换货流程）的端到端调度与上下文精准供给。 --- ### [35:50](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=2150s) 超长通话与异常掉线的上下文恢复架构 针对观众关于“当通话录音或实时会话时长超过一小时该如何处理”的提问，团队探讨了长音频会话的状态管理方案。如果单次会话时间过长，或者更常见的现实情况是用户意外掉线、挂断后再次拨回，关键在于保存当前会话的状态机，以便随时在新的会话中进行“状态注水（rehydrate the state）”。 团队指出，应对超过一小时的长会话，最佳实践并不是强行将底层 WebSocket 连接无休止地维持下去，而是及时切断并平滑开启一个新会话，同时利用 GPT-Realtime-2 扩展至 128,000（128K）Token 的超大上下文窗口，将上一个小时累积的核心上下文、历史总结和状态快照一次性注入新会话中。这不仅解决了长时会话的内存瓶颈，也为网络抖动和用户二次进线提供了高可用的容灾基础。 ```mermaid flowchart TD accTitle: 长时与中断通话状态恢复流程 accDescr: 展示通话中断或超过一小时如何通过状态注水迁移到新会话并利用128K上下文继续处理。 subgraph Phase1["第一阶段：初始长会话"] A["进行中通话 (>1 小时 / 意外掉线)"] --> B["实时状态抓取与摘要存储"] end subgraph Phase2["第二阶段：会话平滑接力"] B --> C["启动全新 Realtime 2 实例"] C --> D["状态注水 (Rehydrate 128K 上下文)"] D --> E["无缝继续业务对话"] end ``` - **长时会话管理策略**：通过分段级联（chained sessions）或新会话平滑接力，避免单一会话无限延长带来的系统不稳定。 - **上下文容量提升**：借助 128K Token 的上下文窗口，系统能够在会话重启或重连时携带极高密度的历史状态与背景知识。 --- ### [37:09](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=2229s) 顾问模式决策：何时向 GPT-5 或前沿推理模型升级？ 现场有观众询问何时应该通过“顾问模式（advisory pattern）”将任务从实时模型升级（escalate）至 GPT-5。OpenAI 团队回顾道，在上一代 Realtime 模型时代，几乎任何涉及复杂推理的任务都会被建议分流给更强大的前沿模型，因为早期的音频模型在深层推理智力上存在客观短板；而是否升级的核心判定标准永远取决于开发者的评测基准（evals）。 Sierra 团队从研究与生产两个维度拆解了方案演进。在研究阶段，团队最初考虑过“挂起（hold）”通话逻辑——让语音助手让用户稍等片刻，后台调用大型纯文本模型深思熟虑后返回结果；但伴随着 GPT-Realtime-2 这类具备内在思考能力（thinking models）的模型推出，实时模型已经能够在语音交互中自然地对用户说出“请给我一分钟”，自主在模型内部完成推导再给出回复。而在 Sierra 的工业生产环境里，主要运行着两套互补架构： 1. **异步主管机制（Supervisor Pattern）**：会话中内嵌一个异步主管 Agent，它在对话进行过程中实时审计上下文流，一旦发现实时模型有偏离轨道的趋势，便会动态向上下文注入补充指引以纠偏。 2. **分级路由机制（Complexity Routing）**：系统同时支持 GPT-Realtime-2 与 GPT-5 等前沿文本大模型。对于追求极致低延迟响应的场景优先采用 GPT-Realtime-2；而对于极其复杂的业务决策，则回退到传统文本推理模型组合。 OpenAI 工程师在此补充了一个极其实用的 API 特性：开发者可以利用 `conversation.item.create` 接口在任何时刻向模型注入背景上下文，且该操作**完全不需要触发模型的即刻语音响应**。这意味着后台耗时较长的异步工具调用可以安静运行，模型在前台继续和用户畅聊，待后台计算完毕后再将结果静默插入对话历史，供实时模型在下一轮自然调用，整体体验丝滑流畅。 - **`conversation.item.create` 的非阻塞优势**：支持在后台异步工具执行完成后静默插入数据，不强行打断当前正在进行的语音对话。 - **动态干预机制**：Sierra 生产环境中的 Supervisor 监控层无需直接接管通话，而是通过上下文动态注入（Context Injection）实现无感纠偏。 --- ### [40:01](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=2401s) 多工具调用下的上下文连贯与指令遵循优化 在直播回答的最后一个问题中，团队针对“实时 Agent 在与多个工具频繁交互时，如何保证上下文与决策的一致性”展开了探讨。主持人提到了官方博客中关于 Perplexity 的生产级案例研究：早期在 Realtime 1.5 上，团队需要设计非常繁琐的上下文截断（truncation）与精细管理来维持稳定；但在 GPT-Realtime-2 时代，模型的底层架构发生质的飞跃。 GPT-Realtime-2 本身就是一个具备思维链（Chain of Thought）的推理模型。它拥有前置引导（preambles）等机制，能够像主流前沿模型一样清晰展现其推理过程。在并行触发多个外部工具调用时，模型不会丢失上下文，因为对话状态在轮次之间（turn over turn）是默认自动维持并传递的，开发者不必再为每一轮繁琐地手工重建状态机。 针对复杂 Prompt 的调优，OpenAI 团队给出了一个非常关键的实践提醒：GPT-Realtime-2 的指令遵循（instruction following）能力极强，以至于当系统提示词中出现哪怕微小的情感冲突或逻辑矛盾时，它会极力试图同时执行相互对立的要求，从而引发非预期行为。团队推荐的实用排查手段是“让大模型自己审查 Prompt”——直接将写好的提示词交给前沿模型分析，询问“这段提示词是否存在任何相互矛盾的建议，或者是否有更优的表述方式”，利用模型的元分析能力来提炼消除歧义、更精简高效的提示词版本。 - **思维链与原生推理**：GPT-Realtime-2 引入内在推理与前置引导（preambles），并行工具调用时的上下文稳定性与主流前沿文本模型看齐。 - **状态维护开箱即用**：API 默认在各轮次之间自动维护上下文流，大幅减轻了开发者手动做上下文工程的负担。 - **Prompt 冲突自检法**：面对强指令遵循模型，可利用模型自身的反思能力剔除提示词内部的隐性矛盾，避免智能体陷入逻辑撕裂。 ### [42:04](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=2524s) 总结致谢、开发资源整理与下期活动预告 活动在尾声阶段由主持人向特邀嘉宾——来自 Sierra 的 Ken 表达了由衷的感谢，并称赞了观众在互动区提出的诸多高质量技术问题，相关问题均已由团队在聊天框中实时解答。为了让开发者能够无缝衔接会后的动手实操，团队在活动结束后会向参会者统一下发全套参考资料。这些资源包括 Teri 在分享中提及的官方博客文章（深入拆解了各项实际用例与落地最佳实践）、语音智能体的开发者官方文档、支持即时交互测试的 Realtime Playground，以及持续更新的官方 Build Hour 代码仓库，方便大家直接获取本次演示的代码片段并在本地进行调试探索。 随后，主持人正式揭晓了下一期 Build Hour 的具体安排：活动将于 5 月 28 日举行，核心主题是 OpenAI 近期全新推出的 Agents SDK。与会者可以通过活动页面下方的链接预约后续的直播场次，或随时在 YouTube 官方频道回顾包含本期在内的所有录播视频。在告别之前，团队特别提醒参会者留意会后弹出的调研问卷，并鼓励大家积极建言献策。团队强调，Build Hour 系列的主题完全取决于开发者真实关心的方向，"我们非常努力地将这些环节建立在大家最感兴趣、对大家的实际构建最具影响力的内容之上"，因此非常期待听取社区心声以决定下一次深入拆解的技术领域。 - **官方配套资源**： - Realtime 语音技术专题博客（包含核心用例与工程化最佳实践） - 语音智能体开发文档（Voice Agents Docs）与实时语音控制台（Playground） - GitHub 演示源码仓库（OpenAI Build Hours Repo，提供开箱即用的代码片段） - **下期活动排期**： - **举办日期**：5 月 28 日 - **核心主题**：深入讲解最新发布的 Agents SDK 架构与开发 - **观看与报名渠道**：提供在线研讨会预约链接，且全程录像将归档于 YouTube 官方播放列表 - **社区反馈机制**：会后通过问卷收集开发者感兴趣的痛点与高价值场景，直接作为后续技术分享选题的决策依据。 ## Notable quotes > [00:00](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=0s) 向语音 Demo 之神祈祷一切顺利 — Erika Kettleson > [18:36](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1116s) 哪怕只有 0.1% 的概率幻觉出一条错误政策或执行了错误操作，这个 Agent 都是绝对不可上线的 — Ken Murphy > [21:54](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1314s) 这才是生产级语音的硬门槛——不是做出一个声音像人的 Agent，而是做出一个值得托付具体工作的 Agent。 — Sierra 团队讲者 > [24:34](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1474s) Erika 常说语音是她最爱的交互模态，而 Ken 则指出语音环境极其苛刻、容错率极低——有趣的是，这两句话同时完全成立。 — Soham > [24:34](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1474s) 从无法挑选输入音频 — Soham > [17:24](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1044s) 在未来，孩子们与 AI 说话的频次，很可能远远超过他们打字交互的频次 — Teri Yu > [32:15](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1935s) you can speak like four times faster than you can type — OpenAI 团队讲者 > [32:15](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1935s) 整通通话的编排 — Ken Murphy > [32:15](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=1935s) 吃自家狗粮 — OpenAI 团队讲者 > [40:01](https://www.youtube.com/watch?v=qGS9Ghnq1RU&t=2401s) 这段提示词是否存在任何相互矛盾的建议，或者是否有更优的表述方式 — OpenAI 团队讲者 ## Mentioned OpenAI, Sarah Urbonas, Teri Yu, Erika Kettleson, GPT-Realtime-2, GPT-Realtime-Translate, GPT-Realtime-Whisper, Sierra, Ken Murphy, Soham, T-Mobile, Perplexity, Supply Co, Metric Loop, Demo Lots, Jade, Codex, Agents SDK, Dialpad ## Video description Build with the next wave of realtime voice AI. In this Build Hour, you’ll learn how to use GPT-Realtime-2, GPT-Realtime-Translate, and GPT-Realtime-Whisper to build low-latency voice agents that can translate live speech, reason across tools, operate apps, and support more natural voice-to-voice and voice-to-action experiences. In this session, Teri Yu (Product) and Erika Kettleson (Solutions Engineering) will cover: • Building with new realtime audio models for translation, streaming speech-to-text, and intelligent voice agents • Using GPT-Realtime-2 capabilities like preambles, 128K context, parallel tool calling, domain understanding, context over turns, and controllable expressiveness • Creating voice-powered workflows for shopping and product analytics dashboards • Customer Spotlight on how Sierra (https://sierra.ai/) is designing production customer experience agents with guardrails, VAD tuning, tracing, redaction, evals, and customer-specific harnesses. 👉 Realtime Voice Blog: https://openai.com/index/advancing-voice-intelligence-with-new-models-in-the-api/ 👉 Voice Agents Docs: https://developers.openai.com/api/docs/guides/voice-agents 👉 Playground: https://platform.openai.com/audio/realtime 👉 Follow along with the code repo: http://github.com/openai/build-hours 👉 Sign up for upcoming live Build Hours: https://webinar.openai.com/buildhours 00:00 Welcome and intro 02:06 Realtime voice models overview 02:26 GPT-Realtime-Translate and GPT-Realtime-Whisper demo 04:36 GPT-Realtime-2: three ways to build with voice AI 05:14 What’s new in GPT-Realtime-2 06:58 Demo: Voice-powered search agent 12:32 Demo: Product analytics dashboard 17:24 What can you build with voice AI? 18:36 Customer spotlight: Sierra 29:56 Q&A 42:05 Resources & Upcoming Build Hours [Source](https://www.youtube.com/watch?v=qGS9Ghnq1RU)

[0:00] Erika Kettleson: 向语音 Demo 之神祈祷一切顺利 [18:36] Ken Murphy: 哪怕只有 0.1% 的概率幻觉出一条错误政策或执行了错误操作，这个 Agent 都是绝对不可上线的 [21:54] Sierra 团队讲者: 这才是生产级语音的硬门槛——不是做出一个声音像人的 Agent，而是做出一个值得托付具体工作的 Agent。 [24:34] Soham: Erika 常说语音是她最爱的交互模态，而 Ken 则指出语音环境极其苛刻、容错率极低——有趣的是，这两句话同时完全成立。 [24:34] Soham: 从无法挑选输入音频 [17:24] Teri Yu: 在未来，孩子们与 AI 说话的频次，很可能远远超过他们打字交互的频次 [32:15] OpenAI 团队讲者: you can speak like four times faster than you can type [32:15] Ken Murphy: 整通通话的编排 [32:15] OpenAI 团队讲者: 吃自家狗粮 [40:01] OpenAI 团队讲者: 这段提示词是否存在任何相互矛盾的建议，或者是否有更优的表述方式

[2:26] Teri Yu: Conversations feel like language barriers disappear [6:58] GPT-Realtime-2 Agent: The low-star reviews mainly say setup can be slower than expected the first time, and it's not a great pick for heavy storms or strong wind and downpour. [12:32] GPT-Realtime-2 Agent: The investigation shows a mobile Safari specific regression where the product detail page size selector validation doesn't update correctly. So first-time Europe footwear shoppers get stuck after choosing a size and can't add to cart. Chrome is near baseline, which points to the PDP release behavior on Safari rather than a broad traffic quality or search issue. [20:31] Jade (Sierra Demo Agent): Hey Ken, this is Jade from Demo Lots. We've got great news, but first I got to say, you may verbally opt out at any time. This call may be recorded for quality. [4:36] Let me check on that... [24:34] mhm [24:34] uh-huh [27:34] ah-ha [27:34] yeah [27:34] Let me look that up for you...

[0:00] Hi everyone, welcome back for another [0:02] build hour. We're so excited to have you [0:05] here today. My name is Sarah Urbonas. I [0:07] lead startup marketing here at OpenAI. I [0:09] am joined by two absolute legends who [0:12] internally need no introduction, but [0:15] Terry, Erica, can you introduce [0:16] yourselves and share what you've been up [0:17] to this past week? [0:19] >> Absolutely. So, I'm Terry. I am [0:22] a multimodal API PM here. [0:24] >> And I'm a solutions engineer on our [0:26] technical success org helping our [0:28] largest digital native customers scale [0:30] on our API platform. [0:32] >> And so today, everything is about GPT [0:34] real-time two, which we just released [0:36] last week. So, the goal of build hours [0:39] is really to empower you with the best [0:41] practices, tools, and AI expertise to [0:44] scale your company using OpenAI APIs and [0:46] models. Whether you're a founder or an [0:48] engineer at a larger company, we hope [0:50] what you learn today can actually take [0:52] these products and put them into [0:53] production, help you scale faster. You [0:55] can sign up for all of our build hours [0:57] at the link below. Um, and also find [1:00] them on YouTube. So, there's lots of [1:02] great content that we have, and we're [1:04] going to jump right into it because we [1:05] have a packed hour. So, today we're [1:07] going to go through kind of just an [1:09] overview of what we released last week, [1:11] um, share a little bit more about what [1:13] you can specifically build with and [1:15] where you should focus on, as well as [1:17] going through a couple of specific [1:18] demos. So, we're first going to go [1:20] through voice-powered search agents, [1:22] then product analytics dashboard. As [1:25] always with build hours, we're going to [1:26] build these live, so pray to the voice [1:29] demo gods that everything works out. And [1:32] then, we are really excited to be joined [1:34] by the team at Sierra. They're going to [1:36] share how they're building with [1:37] real-time two, and also some tips from [1:39] the pros with building with voice agents [1:41] in general. As always, we will save some [1:43] time for Q&A at the end. There's a Q&A [1:46] button that you'll see on your screen. [1:48] As we're going through the session [1:49] today, feel free to ask questions there. [1:51] We have folks in the studio who will [1:52] answer them live, and then at the end, [1:54] we'll always save some time to answer [1:56] questions both with Terry, Erica, and [1:59] the Sierra team. [2:01] So, with that, let's kick it off. [2:03] >> Amazing. [2:05] >> Awesome. So, we had a big audio drop [2:07] last Thursday with three new models. [2:10] Uh so, we'll go through them today. [2:12] Here's just a quick very uh quick view [2:14] of everything going on. Um but, we [2:16] wanted to actually show you [2:18] uh and immerse you in the experience. [2:20] So, we're going to go through some [2:22] demos. [2:23] Um so, then for the next slide, we're [2:26] going to make sure to [2:29] uh we want to know where everyone's [2:30] from. And so, if you could just drop a [2:32] flag for where you're calling from or [2:35] where you're dialing in from, uh we want [2:37] to show you potential translation model [2:40] in this [2:41] um potential language. So, [2:44] yeah. [2:45] >> Looks like we have We always have a lot [2:47] of Spanish speakers in the room, so [2:48] let's choose that for translation. I'm [2:50] going to pop over to our translation [2:52] demo. [2:53] And Terry, I'd love you to give this a [2:56] spin. [2:57] >> Cool. So, when I wake up this early, I'm [3:00] I'm really [3:05] Uh so, because I'm hungry, I'm going to [3:07] talk about some breakfast items. [3:13] >> Ooh, great idea, Terry. [3:15] >> An acai bowl. [3:24] What's better than breakfast is [3:25] potentially a new OpenAI [3:28] >> Okay, so I'm going to stop with the [3:30] translation [3:32] >> [laughter] [3:34] [clears throat] [3:34] >> Okay, so I'm estoy emocionado. [3:37] Let's see. [3:39] >> So, you can see here on the left is our [3:41] transcript, and then the output [3:43] translation transcript. [3:45] >> Absolutely. So, number one, you're [3:47] already seeing in action, which is the [3:49] real-time translate model. So, it [3:51] conversations that feel like language [3:53] barriers disappear. It supports over 70 [3:56] input and 13 output languages with low [3:58] latency streaming translation. [4:00] And it's really great for video calls, [4:02] live streams, customer service. [4:04] And then also you're seeing number two, [4:06] which is our GBT real-time whisper [4:08] model. [4:09] That's secretly powering this experience [4:10] as well, where you can see streaming [4:12] capability with tunable latency as low [4:15] as 200 milliseconds with 80 input [4:17] languages. And so, that means you can [4:20] have earlier function calling, better [4:21] instruction following, and live products [4:23] will feel faster, more responsive for [4:26] captions, meeting notes, ambient agentic [4:28] context. And then we have our third [4:30] model that we released as well that is [4:34] GBT real-time two. [4:37] So, you'll see here it is our most [4:39] intelligent voice model. [4:41] It brings GBT 5 class reasoning into [4:44] voice. [4:45] And it's very strong at prompt [4:47] adherence, talk tool calling, [4:48] multilingual performance, which are all [4:51] critical to voice production workflows. [4:53] And so, you now have these three [4:55] different ways that you can build. So, [4:57] you have voice to action like hands-free [4:59] voice driven apps. You have your systems [5:01] to voice, which is like a voice chief of [5:03] staff, but it's organized into spoken [5:05] voice. And then you have voice to voice [5:08] like customer service calls. Companies [5:10] like T-Mobile use it all around the [5:11] world. [5:13] >> And someone commented in the chat in [5:15] Spanish and said, "Wow, that's good." [5:17] So, I'm assuming that it translated. [5:19] I'll say you both and oat milk latte was [5:21] >> I was I was worried about that. [5:23] >> [laughter] [5:24] >> It actually hopefully everybody could [5:25] hear. [5:26] We have dynamic voice cloning as part of [5:28] this. So, we can have multiple speakers [5:31] and you can actually tell the different [5:32] people are speaking, which I think is [5:34] really cool. [5:35] >> tone matching is really great for sure. [5:38] Cool. And then for real-time two, [5:40] there's a bunch of new capabilities. So, [5:42] beyond just like the emotion matching [5:45] that you get from voice to voice, [5:46] there's also things like preambles where [5:48] you can say, "Let me check on that." [5:51] Like a real human would before [5:52] reasoning. [5:54] There's also increased context windows. [5:56] This was a major request that we had. It [5:58] increased 4x to 128k now. [6:02] And so that roughly translates to [6:04] >> Like almost an hour which is also [6:06] coincidentally our session. [6:08] >> [laughter] [6:09] >> So it really increases the instruction [6:11] following the intelligence because we [6:13] don't have to truncate. [6:15] >> Yes, exactly. And then there's also [6:17] parallel tool calls. So you no longer [6:19] have to like waterfall the tool calls. [6:21] There's also better domain vocabulary [6:23] and understanding like healthcare or AI [6:25] terms. And then there's context over [6:27] turns which means better agentic [6:29] behavior. And then controllable [6:30] expressiveness. So you can try telling [6:32] it to whisper, be excited, jealous, you [6:35] try different things. We're excited to [6:36] see what you come up with. And on [6:39] benchmarks you'll see major jumps in [6:41] both intelligence, instruction following [6:43] and tool calling. [6:44] And so yeah, I'm very excited about the [6:46] next wave of AGI that'll be more global, [6:48] hands-free, [6:50] more accessible to everyone around the [6:51] world. But now we're going to have Erica [6:54] actually show us what the it's like [6:56] under the hood. [6:57] >> I'm really excited to show off our first [6:59] demo today which is an e-commerce site [7:02] with a voice powered search agent. [7:05] In this demo the real-time API is [7:07] powering search that's not just chatting [7:10] but actually operating the UI through [7:12] through tool use. And I think this [7:14] really highlights the power of our [7:15] reasoning models and parallel tool [7:17] calling. So let me pop over. [7:21] So I have been planning this trip to the [7:23] Pacific Northwest. I'm going hiking. I [7:26] have already bought a few things from my [7:28] favorite trail supply company. And I [7:31] need to keep shopping. So I want to use [7:33] the Supply Co shopping assistant to help [7:36] me make some more purchases. [7:41] Hey, can you remind me where did we [7:42] leave off? What do I still need to buy [7:44] for this trip? [7:47] >> I'll pull up your shopping plan and see [7:49] what's still missing. You still need a [7:52] tent and hiking shoes. Your daypack, [7:54] trail socks, and insulated bottle are [7:56] already covered. Which one do you want [7:58] to look for first? [8:00] >> Um let's start with tents, and I want to [8:03] see tents under $450 that would fit [8:06] three to four people. [8:09] >> I'll search tents and narrow to options [8:11] under your budget. [8:14] Two good options are highlighted. One is [8:17] a higher-rated three-to-four-person tent [8:19] at $419.85 [8:22] with free shipping tomorrow. [8:24] The other is a quicker open style at [8:26] $357, [8:28] but it's not marked as full stock. [8:30] Do you want to open [8:31] >> Okay. Actually, can you tell me about [8:33] any one- or two-star reviews on the more [8:35] expensive tent? [8:38] >> I'll scan the low-rated reviews for that [8:40] tent. [8:42] The low-star reviews mainly say setup [8:44] can be slower than expected the first [8:45] time, and it's not a great pick for [8:47] heavy storms or strong wind and [8:49] downpour. [8:50] For a typical Pacific Northwest weekend, [8:53] those complaints aren't dealbreakers if [8:55] you expect light to moderate rain, but [8:57] I'd switch to a more storm-ready tent if [8:59] you expect rough weather. [9:01] >> You know, I actually haven't checked the [9:03] weather in a while. Would you mind [9:05] looking up the forecast for the Seattle [9:08] area the weekend after next for me? [9:12] >> I'll check the Seattle forecast for that [9:14] weekend and weigh the storm risk. Storm [9:17] risk looks moderate. Some rain and a [9:19] breezy spell, but not a clear storm. [9:22] That tent is probably fine for a normal [9:24] rainy weekend, but I'd use a footprint [9:26] and solid stakes. [9:27] >> Okay, that's good advice. Um okay, add [9:30] that to my cart, and then let's move on [9:32] and look at some waterproof hiking [9:35] boots. [9:42] >> I'll add the tent to your cart, then [9:43] I'll check your safe size and find [9:45] waterproof boots. [9:54] I added the tent to your cart. For [9:57] waterproof boots in US 10 with free [9:59] shipping, I highlighted a higher rated [10:01] pair and a cheaper pair. [10:03] Do you want to open the higher rated one [10:04] or the lower priced one? [10:06] >> Um [10:08] let's look at the cheaper one first. Can [10:11] you show me the product page for that? [10:16] >> I'll open the cheaper boots product page [10:18] so you can review the details. This pair [10:20] is $224.85, [10:23] rated 4.5 stars, and has free shipping. [10:26] US 10 [10:27] >> Okay, well, I see a break-in takes a few [10:29] hikes. I've got to get busy in the next [10:31] 2 weeks, but otherwise I kind of want to [10:32] save a little bit of money here. So, can [10:34] you add these ones to my cart and then [10:36] pull up my cart? I want to see my total. [10:41] >> I'll add these goods in US 10, then I'll [10:43] open your cart so you can see the total. [10:47] Your cart has the tent and the boots, [10:49] and the total is $644.70. [10:53] I can't check you out here. [10:55] >> Is there anything else? Like, tell me a [10:56] little bit about these optional add-ons. [11:01] >> Nice optional add-ons would be a rain [11:03] layer and a tent footprint stay kit, [11:05] since they help in wet breezy [11:06] conditions. [11:08] >> Yeah, maybe I'll come back for a rain [11:09] layer, but I think this is good for now. [11:12] Um thanks so much for helping me shop. [11:18] >> You're very welcome, Erica. Have an [11:20] awesome trip, and I'll be here when [11:21] you're ready for that rain layer. [11:24] >> Amazing. Okay, so the thing I think is [11:27] really exciting here is our model sounds [11:30] really good. It sounds really natural, [11:32] the voice is really steerable, but the [11:35] most exciting thing is this [11:36] intelligence. It's operating the UI [11:38] through tools. It can choose between I [11:41] think I'm passing it 15 or 20 tools [11:43] here, which is something that was way [11:45] beyond what we could expect or [11:48] um you know, would recommend for our [11:49] previous real-time models. So it can [11:51] choose between those tools, it can [11:52] inspect the current page and decide what [11:54] actions are needed. You could see it was [11:56] calling multiple tools in a row when [11:58] that was what was needed, including [12:00] external tools like weather, so that I [12:02] didn't have to leave the Supply Co site [12:04] in order to understand like what [12:07] purchase I should make. Um so I think [12:09] this just leads to a lot more natural [12:11] than our kind of speech in single action [12:14] out previous workflows. This is like an [12:16] actual shopping assistant that can [12:18] reason across all these tools and update [12:20] the visual experience as it talks. [12:24] Um so I think the next thing I want to [12:26] show you is really the other side of [12:30] this. So I was just a shopper on an [12:32] e-commerce website, Supply Co, but now I [12:36] want to demo as if I'm the product [12:38] manager of Supply Co, and I think this [12:41] is really interesting because I think [12:44] anybody who's used ChatGPT or other [12:47] maybe voice-to-voice solutions is kind [12:49] of like understands this. Um I talk, the [12:52] model talks back kind of experience, but [12:54] really we're thinking about voice to [12:56] action as this new kind of workflow [12:58] because the model's so good at tool [13:00] calling. And not everybody wants the [13:02] model to constantly be talking back. Um [13:05] so I'm going to swatch swap over to my [13:10] product analytics dashboard, right? So [13:12] I'm a product manager at Supply Co. [13:15] And actually [13:16] um there's a lot going on this [13:17] dashboard, but so I I kind of want help [13:20] with this. If only I had an AI assistant [13:22] that could help me with this. Um the [13:24] thing I've I've hearing is about Europe. [13:26] So let me open up my session and start [13:28] talking. [13:31] Um let's filter by Europe. That's what [13:36] I'm hearing issues about right now. [13:43] Cool. And let's look at the last 7 days [13:45] and compare that to the 7 days prior. [13:53] Okay, cool. Yeah, I can see a big drop [13:56] there. [13:57] Um looks like Safari is a problem. And [14:02] oh, I can see thanks for highlighting [14:03] this. [14:05] Um [14:06] can you point out any other filters that [14:10] you think [14:12] um [14:13] would be relevant here to this [14:14] investigation? [14:23] Okay, cool. Okay, so filter by voice [14:25] search, first-time shoppers, and [14:27] footwear. We'll start there. [14:35] Cool. Okay, yeah, I can see this is like [14:37] a huge activation drop-off. [14:40] Um [14:41] Okay, what's next? Actually, can you [14:44] kick off the root cause investigation [14:46] here? [14:50] And I think I just want to compare [14:52] mobile Safari to Chrome cuz that seems [14:54] like where a lot of these issues are [14:56] taking place. [15:03] Perfect. [15:05] Yeah, Safari is really the problem here. [15:08] Okay, let me expand this back open. [15:13] Metric Loop, can you give me a [15:16] two-sentence overview out loud um [15:19] explaining this root cause investigation [15:22] so I can explain it to my engineering [15:24] team. [15:28] >> The investigation shows a mobile Safari [15:30] specific regression where the product [15:32] detail page size selector validation [15:34] doesn't update correctly. So first-time [15:37] Europe footwear shoppers get stuck after [15:39] choosing a size and can't add to cart. [15:41] Chrome is near baseline, which points to [15:43] the PDP release behavior on Safari [15:46] rather than a broad traffic quality or [15:48] search issue. [15:51] >> Okay, amazing. This is super helpful. [15:53] I'm going to pass this on to my [15:55] engineering team in a ticket. Super [15:58] helpful. Thank you so much. Um okay, [16:01] cool. So [16:03] why do I think this is cool? The model's [16:06] good enough at instruction following [16:07] that it only talked when I asked it to. [16:10] Like I didn't need it to confirm with me [16:12] every time it did a filter. I just [16:14] wanted it to take that action so I [16:15] didn't have to go through and click [16:17] myself. And I can go in, click in and [16:20] see its reasoning at any time and see [16:21] the transcript so that I can interact it [16:23] with chat as well. Um this to me is a [16:26] huge step forward in intelligence, in [16:29] instruction following, and it means that [16:32] the model can behave like an analyst in [16:33] the loop. So like the e-commerce site, [16:36] I'm passing in a ton of tools. I also [16:39] asked the model to reason over a huge [16:41] amount of mock data, which I think is [16:44] really exciting because we have that um [16:46] reasoning, it can actually write code, [16:49] it can create dashboards, it can make [16:51] artifacts. Um our old models could [16:53] handle simple voice commands and call [16:55] some tools, but I hope this demo shows a [16:58] more durable pattern. Right, this is a [16:59] real-time agent that can actually route [17:02] across tools, um maintain investigation [17:04] state, and turn this live analytics [17:06] workflow into something conversational [17:10] without actually losing developer [17:11] control over the data or the UI. [17:15] And with that, I want to pass it back to [17:17] Terry to talk a little bit about some of [17:20] the use cases that we're seeing for [17:23] voice. [17:23] >> Yeah, so we're really excited for you to [17:26] build with voice interfaces across [17:29] agents, across apps. Here are just some [17:31] of the use cases that were [17:33] we'd love to show you. So, there's a lot [17:35] of smart device integrations that you [17:37] could potentially think about, different [17:39] coding assistants, and voice experiences [17:41] that make things fast, easy. There's [17:44] mobile apps you can add a voice box to [17:46] for any kind of voice experience. [17:48] There's the video game up and down and [17:51] jump experience that you could control [17:53] with voice. You have coaches, like [17:55] public speaking coaches, note-taking, [17:58] finance voice US, agentic voice video [18:00] calls. All this can be done with a voice [18:03] layer now. So, [18:04] really exciting time that we're in where [18:07] you know, in the future kids will learn [18:09] will speak to AI more than they will [18:12] even necessarily know how to type to AI. [18:14] So, we're ready to build for that wave. [18:16] And [18:17] the other just slight nod here is that [18:19] all these were created to with GBT image [18:22] too. So, if you're in a place where you [18:24] just want to ideate and you need more [18:27] brainstorming, GBT image 2 can do some [18:29] of this for you or some mocking as well [18:31] for different concepts. So, [18:34] Yeah. Amazing. [18:37] >> And I think next we're switching to our [18:39] customer spotlight. So, I think we're [18:41] really excited to have the Sierra team [18:42] here. [18:54] I know we have Ken and Soham. Are they [18:57] on the line? [18:59] Can we hear them? [19:00] >> Yes. Can you hear us? [19:01] >> Oh, sorry. [19:02] >> Hey guys. [19:03] >> Hi. [19:03] >> Hi. There we go. [19:04] >> Hello. Hey. Hey Soham, Ken Murphy. I'm [19:06] an engineer on the voice team at Sierra. [19:09] Uh so, Sierra helps businesses build [19:11] better, more human customer experiences [19:13] with AI. [19:15] So, in practice, we build AI agents that [19:17] can resolve real customer service issues [19:19] end-to-end, not just answer questions. [19:22] So, many of our customers are large [19:24] enterprises, including many Fortune 100 [19:26] companies, which means these agents need [19:27] to work at high scale, [19:29] and integrate with complex systems, and [19:31] follow nuanced business policies. So, [19:34] you know, on the slide, here are a [19:35] couple examples of the incredible brands [19:37] that we work with and the scale that [19:39] we're operating at. [19:41] So, what makes this technically hard is [19:43] that the agent is not just generating a [19:45] response, it's deciding when to act, [19:48] what tools to call, what information to [19:49] use from a large knowledge database, and [19:52] whether a given action is allowed. [19:54] So, in a customer-facing environment, [19:56] small error rates quickly become real [19:58] business risk. So, an agent that [20:00] hallucinates a policy or takes the wrong [20:03] action even 0.1% of the time is just not [20:06] shippable. [20:07] So, the challenge is not just, can we [20:09] make the agent sound natural, it's [20:11] whether we can build, evaluate, [20:13] constrain, and operate agents that [20:15] businesses trust to represent them [20:17] directly with their customers. [20:20] So, now we're going to look into what we [20:22] built to operate agents at the scale and [20:24] how we're using the real-time API. [20:31] Oh, hello. [20:33] >> Hey Ken, this is Jade from Demo Lots. [20:36] We've got great news, but first I got to [20:39] say, you may verbally opt out at any [20:41] time. This call may be recorded for [20:43] quality. [20:44] Last time we chatted back in March, you [20:46] were eyeing Noe Valley. Has that shifted [20:48] at all? [20:48] >> Uh yeah, still [20:50] uh still looking in San Francisco, but [20:52] now in the Sunset. [20:56] >> Staying in San Francisco and shifting to [20:58] the Sunset makes sense for a lot of [21:00] buyers. [21:01] Are you looking for a single-family [21:02] home, condo, or [21:03] >> I'm looking for a single-family home. [21:06] >> Got it. Single-family homes in the [21:08] Sunset can be a nice fit. [21:10] What's your budget or max purchase [21:12] price? [21:12] >> About 350k. [21:17] >> Your licensed mortgage banker Sarah is [21:19] here to help. You're in great hands. [21:24] >> Well, so so that's an example of of one [21:27] of the demo agents that we've been [21:28] testing. Um so in general, we're very [21:30] excited about real-time too. You know, [21:32] directly goes after some of the biggest [21:34] production voice problems that we face. [21:36] So you know, latency, turn-taking, [21:38] reasoning, and speech quality. Like [21:40] voice is pretty unforgiving. You know, a [21:43] pause of even half a second can feel [21:45] awkward or broken. [21:47] And that's why the voice-to-voice [21:48] architecture is so compelling. So we [21:49] don't have to have this separate uh [21:52] speech-to-text transcription or [21:54] text-to-speech synthesis. Instead, [21:56] real-time can just cut out parts of the [21:58] traditional cascaded stack and make the [21:59] interaction feel faster, smoother, and [22:01] more human. [22:03] Uh but while real-time 2 gives us a [22:04] stronger foundation, like at Sierra, the [22:06] the model is just one piece. Around it, [22:08] we still run our agent harness, [22:10] which handles all of the additional [22:12] infrastructure needed to run agents [22:13] reliably and securely in production. So [22:16] with that harness, we define the [22:18] workflows needed for each individual [22:20] customer. Uh that includes the tools the [22:22] agent can use, language and branding it [22:24] should follow, guardrails it needs, and [22:26] the grounding required to keep it [22:28] aligned with the customer-specific [22:30] policies. [22:31] Uh we also use our own custom-tuned vad [22:33] models to decide when a user has stopped [22:35] speaking, which gives us a lot more [22:36] control over turn-taking. Uh we've found [22:38] that this performs better for our use [22:40] cases, where real-world audio is often [22:43] messy and includes a lot of background [22:44] noise, uh accents, interruptions, people [22:47] changing course mid-sentence. [22:49] And then this uh harness also handles [22:52] tracing, like redaction of sensitive [22:54] information, PCI-compliant payment [22:56] flows, uh and a range of other [22:58] production features that help ensure [23:00] that agents stay accurate, safe, and on [23:03] brand. And it's this production layer [23:05] that takes us from a strong base model, [23:07] like real-time 2, to a system that is [23:10] controllable, observable, and safe [23:12] enough for some of the world's largest [23:13] companies to to trust with their [23:14] customers. [23:16] So, um you know, impact so far is that [23:21] you know, mainly what we're looking for [23:22] is faster relevant audio, more natural [23:25] conversations, stronger tool execution, [23:27] and higher end-to-end task success. [23:30] Uh in some of our initial testing, we [23:32] have definitely seen latency gains. So, [23:34] we've seen calls that are roughly 30% [23:36] faster at P50 and up to 200% faster at [23:40] P90 compared to our cascaded system. So, [23:43] it does really help with naturalness [23:45] overall. Uh voice quality has been [23:47] strong. It's been competitive with some [23:48] of the dedicated synthesis providers [23:51] that we use, you know, based on our [23:53] internal evaluations. [23:55] Uh and you know, [23:56] speed and voice quality are not enough. [23:58] Like in a traditional stack, uh we have [24:01] to evaluate transcription, the agent [24:02] harness, and synthesis separately. With [24:04] voice-to-voice, we have to evaluate the [24:06] full call, you know, listening, [24:07] reasoning, acting, speaking. [24:09] Uh so, do to do that, we also use [24:11] simulations that replay realistic [24:13] customer calls customized to each [24:14] customer's workflow, which lets us [24:16] measure whether the agent actually [24:17] completes the task, not whether it [24:19] sounds good. So, that's kind of the the [24:20] real bar for production voice, not just [24:23] an agent that sounds human, but one that [24:24] can be trusted to do the work. So, going [24:26] to hand it off to Soham, uh who will [24:28] talk about how Voisi and how Sierra [24:31] evaluates voice models end to end. [24:34] >> Hello. Uh yeah, I'm Soham, and I [24:37] represent the voice research team at [24:39] Sierra. [24:40] Um I know Erica said that voice is her [24:42] favorite modality, and Ken said that [24:44] voice is really unforgiving. So, [24:47] it's really fun that both of them are [24:48] true at the same time. [24:50] Um okay. So, [24:53] I guess the first thing about voice [24:55] agents is that [24:57] they don't get to pick their audio. [24:59] Um, [25:00] customer service calls are [25:02] sophisticated, they're nuanced, and [25:04] everything can said, and [25:07] calls very rarely look like the ideal [25:10] calls we want them to. Ideal calls are [25:13] like often text like where there's like [25:15] clear turn boundaries, and it's easy to [25:18] disambiguate what's being said, [25:20] information is like conveyed crisply, [25:23] and it's not lossy. And production calls [25:26] almost never look like this, right? Like [25:29] there's always [25:30] interruptions, there's accents, [25:33] and the question we really ask ourselves [25:36] is [25:37] can an agent, can a voice agent handle a [25:40] call from a customer who's like on the [25:42] side of a highway or in a car with kids, [25:46] or can it handle a customer who is [25:48] really impatient and aggressive [25:50] interrupter, and [25:53] all doing all of this while trying to [25:55] like make progress on the task that [25:57] they're trying to accomplish at the same [25:58] time, and this is hard. [26:01] Um, and this brings us to like common [26:04] failure modes that we see with agents [26:07] like this, and [26:09] um, if we can move to the next slide, [26:12] that would be great. [26:15] Yes. Um, so [26:19] there are different forms of failure [26:21] modes. I think we have found that some [26:23] agents can be really good at the [26:24] conversation part of things, whereas [26:27] some agents are more are better with [26:29] just getting things done and like tool [26:30] calls, and when you bring both of them [26:32] together, the error modes really [26:34] compound, and we see a lot of [26:36] performance degradation in these [26:38] situations. [26:40] Um, some really common forms of error [26:43] are like with spelling out names and [26:45] numbers. [26:46] This can be quite tricky in that if you [26:49] get like one letter wrong, [26:51] are you able to recover from this [26:52] gracefully? If you do, do you keep that [26:55] in your memory or like [26:57] like when you make a tool call later [26:59] with my name, [27:00] do you get the spelling right or do you [27:02] use the incorrect spelling from the [27:03] beginning? And we see that agents [27:05] sometimes get mired in their own [27:07] mistakes and are unable to like move on [27:09] from there. [27:10] Um [27:12] and at the same time, there's different [27:14] forms of errors where like they're more [27:15] logical in the sense of [27:17] um oh, you wanted to cancel a flight, [27:19] but the agent misunderstood which flight [27:21] this was and [27:23] this like airline is quite high stakes, [27:26] right? So, like you can be really stuck [27:28] [laughter] in like dire situations where [27:30] you're in a city or country you don't [27:32] want to be in and you're stranded [27:34] because your voice agent canceled the [27:35] wrong flight and we don't want this. [27:38] Um there's also [27:40] parts of conversation that are hard to [27:43] work with. Um [27:45] like I think we don't realize that, but [27:47] humans often [27:48] backchannel with like mhm's and ah-ha's [27:51] and yes, that's right. [27:53] And a lot of our voice models are [27:55] trained to respond to everything that we [27:56] say [27:57] and humans are much better with being [27:59] selective with this. [28:01] It can be hard to [28:03] be like, okay, that's that's a signal I [28:05] should ignore versus that's a signal I [28:07] should respond to. [28:09] Um yeah, and we have found cascaded [28:13] models to really shine in this situation [28:16] because we can really overfit to these [28:19] conditions and customers are complex and [28:22] we want the brands to like we want our [28:26] agents to represent the brands. [28:28] And [28:29] it's been really fun to see that [28:31] like voice-to-voice models are now able [28:33] to absorb that complexity [28:35] and really improve their reasoning and [28:38] their communication [28:40] and that's been really exciting. So, [28:42] yeah, this brings us to thinking models, [28:44] which is what we're talking about today. [28:49] So, thinking models have been a real [28:52] step change in [28:55] voice-to-voice models. Our leaderboard [28:57] at Dialpad has been dominated by [28:59] thinking models, and it's [29:03] not that easy to think in voice [29:06] conversations. You can't just have a [29:08] buffering loop while you're thinking. [29:10] Like you have to say, "Okay, like give [29:11] me a moment." or "Let me think about [29:13] that." and you have to be okay with [29:16] being interrupted in these situations [29:18] and maintaining state. And not like [29:21] Yeah, we should be able to carry forward [29:22] even if you're interrupted during these [29:24] situations. And [29:26] all of these issues sort of expect like [29:30] expect load multilingually, and it's [29:32] been really fun to see that um OpenAI [29:35] has made like some really big strides [29:37] here in real time too significantly [29:39] better. Um there's a high ceiling, but [29:43] we are really getting there. And for [29:44] someone who's been like waiting for [29:45] voice-to-voice models for a while, [29:48] >> [snorts] [29:48] >> it is very exciting to see [29:49] voice-to-voice in the mainstream. [29:53] Yeah. [29:54] That is me. Thank you. [29:57] >> Sierra, thank you for being such an [29:58] amazing team to work with. You're one of [30:00] my favorite like companies ever. So, uh [30:03] yeah, it's been great to just see you [30:05] guys continue. [30:07] Thanks. [30:09] >> Great. And we're going to move into Q&A. [30:10] If you just exit or escape and refresh [30:13] the deck, I dropped in some questions. [30:16] We got a lot of great questions um [30:18] through the chat. There were a lot of [30:20] questions about interruptions and how [30:23] those are handled. Um so, wanted to kick [30:26] it off. Sierra, you shared a little bit [30:28] about how you kind of layer your own [30:31] barge and detection on top of real time [30:33] too. There were a few questions if [30:35] people should rely on built-in turn [30:38] detection versus adding app side [30:41] interruption logic and kind of what the [30:43] best practice is there. [30:47] >> Yeah, I think it depends on your [30:48] specific use case. So, I mean, out of [30:51] the box real-time too does come with [30:53] their own bad models, like you were [30:55] talking about both semantic and server. [30:57] And we we're finding they were working [30:59] pretty well. Uh we just have our own [31:02] customized bad model that is very [31:03] specific to uh customer service call [31:06] audio. So, like having very heavy [31:08] background noise. Yeah. [31:10] Kids in the background, a TV in the [31:12] background, that kind of thing. [31:16] >> Any recommendations from you both? [31:17] >> Yeah, um there are different like [31:19] padding recommendations um that we have [31:22] um [31:23] but also just it really depends on the [31:25] situation of like where your device what [31:27] kind of device you're calling in, what [31:29] kind of like environment you're in. Um [31:31] it does require often like tuning and [31:33] optimization um just to see what is [31:36] actually working um for you. Um but is [31:39] there anything uh that you want to share [31:41] beyond the semantic [31:43] versus [31:43] >> I think like not even just on that [31:46] piece, but I one thing that I think [31:48] people don't necessarily know is that [31:49] you can disable it on a turn-by-turn [31:51] basis. So, if you have something you [31:53] need the model to say like a disclaimer, [31:56] you can actually disable uh you know, [31:58] VAD on that turn. So, the model will [32:00] definitely say that, it won't be [32:01] interrupted, and then you can enable it [32:03] going forward. And so, I think that's [32:04] kind of a cool uh thing cuz it's not [32:06] just relying on like instruction [32:08] following, it's like actually preventing [32:09] the user from interrupting. [32:10] >> Yeah. [32:11] >> Which I think is kind of a cool feature. [32:13] Yeah. [32:13] >> That is cool. [32:15] >> The next question is also just like such [32:18] a tee up for the Sierra team. And a [32:20] question that we get all the time as [32:21] builders are starting to explore uh [32:24] real-time models and bring more voice [32:26] modalities into their products. What [32:28] actually separates a useful AI assistant [32:31] from an AI system that can actually [32:33] improve real-world business workflows? I [32:36] think reading between the lines of this [32:37] like for real time specifically, [32:40] um as the models are starting to become [32:42] more capable, um especially in like [32:44] those real world interactions. [32:46] >> Yeah, maybe I'll go first and then I can [32:48] pass it to the Sierra team. [32:50] Uh so, we're seeing a lot of um great [32:53] use cases in these like fast capture [32:55] fast intent um where it is easier to [32:58] drive things with voice and more [33:00] convenient than it necessarily is with [33:02] text. Um so, there's trade-offs when [33:04] you're building with either uh with text [33:06] obviously there's um more information [33:09] density um but with voice you can use it [33:12] when there you're driving, you can use [33:14] it on the go on mobile, you can use it [33:16] if you just have a generalized idea of [33:18] what you want but the model can [33:20] interpret it into something that's like [33:22] more um of a like coherent prompt. Um [33:26] so, different use cases we're seeing for [33:28] sure. Um I think in particular like [33:30] there's [33:31] uh like also [33:33] a younger uh demographics can like use [33:36] voice more than text in some cases. So, [33:38] um [33:39] definitely a lot of different use cases [33:41] but Sierra, I'll pass it to you. [33:44] >> Yeah, I think on the customer service [33:46] front um with our voice we try to like [33:48] bake in many things that we see in [33:51] production traffic with like [33:52] interruptions and noise and accents um [33:55] but [33:56] like it is a benchmark so it is it's not [33:58] production traffic and there are many [34:00] more complexities that go into that and [34:02] I think Ken is like much more familiar [34:03] with this. [34:04] >> Yeah, for production agents at least [34:06] with Sierra a lot of it goes into how do [34:09] you orchestrate the entire call. So, I [34:12] think real time is great at responding [34:14] within a turn and figuring out what to [34:16] do [34:17] but a lot of what goes into the Sierra [34:20] side is how do you define the different [34:21] workflows that an agent like can need to [34:25] work through. So, like we we work [34:26] directly with individual customers to [34:28] figure out you know, what does your [34:29] refund policy look like and then how do [34:31] we expose the right context to uh time [34:34] to at the right time. [34:39] >> I think fundamentally [34:40] the biggest difference between voice and [34:42] text is that you can speak like four [34:44] times faster than you can type. So, that [34:47] alone I think unlocks a lot of use cases [34:50] and to your point a lot of people feel [34:52] comfortable like stream of consciousness [34:54] speaking about maybe like a house that [34:56] they want to buy or a car. So, you can [34:58] think about it as a more intimate kind [35:01] of casual way for somebody to even [35:03] interact with an application and provide [35:05] more context than they might if they had [35:06] to actually type it out. [35:08] >> And since we are like doing a kind of [35:10] global live stream here, like there are [35:13] countries that are very voice first. [35:17] So, Brazil, India can be some of these [35:19] countries that might prefer to interact [35:22] with your app via voice. [35:24] So, just different considerations. [35:27] >> It's cool walking around our offices too [35:29] and hearing people talking to their [35:31] computers increasingly and even as we [35:33] were preparing for this, our production [35:36] team was sharing that they're using [35:38] Codex through the real time model so [35:41] that they're actually like talking to [35:42] Codex through their headset which is [35:44] just so cool. It's neat to see how we're [35:46] like drinking our own champagne with [35:47] these real time models internally. [35:50] All right, [35:51] we got some questions on like what if [35:53] audio sessions are more than one hour of [35:55] calls? What approaches should we try? [35:59] >> Yeah, I'm just going to answer. [36:01] I'm just impressed that you're able to [36:03] get the question out [36:04] and answer so quickly. [36:06] Um [36:07] Yeah, so I think there's different ways [36:11] that you could set it up. I mean, if you [36:12] could have multiple sessions [36:15] running at once, you could also stage [36:17] them such that you have different [36:18] sessions [36:20] like chained after one another. [36:23] Um [36:23] >> I think like the biggest thing and this [36:25] isn't not even just like oh my session's [36:28] more than an hour, but you might have [36:29] somebody call back or maybe you have [36:31] like a call drop for example, user hangs [36:33] up accidentally. [36:34] Actually saving that state of what's [36:36] been happening so you can rehydrate the [36:38] next session is probably the most [36:40] important thing. So if you are going [36:42] over an hour, like start a new session [36:43] and then hydrate that with all the [36:45] context from the previous hour and [36:47] because we have that extend extended [36:50] window now to 128,000 tokens, you can [36:53] provide way more context in that [36:56] session. So that's probably what I would [36:58] recommend is just like you're going to [37:00] have to start a new session and make [37:02] sure that you've saved all that state [37:04] from the first hour. [37:09] >> And when do you escalate to GPT-55 [37:12] via an advisory pattern? [37:16] >> Based on your evals, I think this year [37:18] our team could probably talk about that. [37:20] In our old real-time models, we kind of [37:21] recommended that for basically anything [37:23] with reasoning because they just [37:24] fundamentally didn't have that [37:25] intelligence. So let the real-time model [37:28] handle this kind of maybe small easier [37:29] to handle set of tools and then escalate [37:33] out to a frontier model for anything [37:35] else, but it's always going to be based [37:37] on your evals. So you maybe maybe I'll [37:39] pass it to the Sierra team to talk about [37:41] how you think about evaluating the model [37:44] for handling certain kinds of [37:46] complexity. [37:49] >> Yeah, I think [37:50] um [37:52] like I'm assuming by advisory pattern [37:54] you mean like something like okay, in [37:55] this case maybe we more solid reasoning [37:58] or something and [38:00] I think for a while we thought about [38:04] and so I'm on the research side, maybe [38:05] Ken you can talk about what we do in [38:06] production, but on the like on the [38:08] research side for a while we thought [38:09] about okay, maybe let's sort of [38:12] put this conversation on hold for a [38:13] second and think about this using a text [38:15] model and get back to us, but I think [38:17] with real-time too and thinking models, [38:19] it's been real cool to see this the [38:21] model does it itself. The model says [38:22] that, you know, give me a minute. It [38:24] thinks about it and then it comes back [38:26] to you with an answer. [38:28] Um [38:29] and [38:31] what do we do? [38:32] >> Yeah, we we have two approaches on the [38:35] production side. Uh one, we have [38:37] supervisors within an agent that [38:41] asynchronously is reviewing the [38:42] conversation as it goes and deciding, [38:44] you know, like do we need to inject [38:47] additional information into the context [38:49] to kind of get it back on the rails. [38:51] Um the second approach honestly is that [38:54] like we support both real-time two and [38:58] just having everything go through [38:59] GPT-54. [39:01] Um [39:02] so depending on the complexity of the [39:04] agent, we'll [39:06] sometimes use real-time two if it's [39:07] something that we want to be really [39:09] fast. And for more complex agents, we'll [39:12] we'll stick with traditional uh [39:13] text-based models. [39:17] >> Yeah, love that. And I think you can [39:20] always inject context at any time [39:22] without even triggering a model response [39:23] with conversation item create. So, to [39:25] that point, you can have a background [39:28] uh you know, tool call running and the [39:30] model can keep talking. It's totally [39:32] fine with these kind of asynchronous [39:33] tool calls happening. So, then when that [39:35] background longer running process [39:36] happens, you can then just inject, you [39:39] know, whatever tool call results you had [39:40] back in so that the real-time model has [39:42] that and actually can create a response. [39:46] >> Yeah, I think in general, we do a lot of [39:47] orchestration and context management. [39:52] >> Yeah. [39:53] Especially important when you were [39:55] dealing with a smaller context window, [39:56] but even here this window is smaller [39:58] than frontier models. So, definitely [39:59] super important. [40:01] That's a great segue into our last [40:04] question that we'll answer live, which [40:05] is how do you real-time agents maintain [40:07] that context and decision consistency [40:10] while interacting with multiple tools? [40:14] >> Um I think one of the resources that we [40:16] have, [40:18] uh, on our blog is about perplexity [40:20] using, uh, context management to kind of [40:23] get to this production state. [40:25] Um, [40:26] yeah, we definitely recommend, like [40:28] reading that, but there's a lot of, [40:29] like, truncation optimization on the and [40:32] when they had the real-time 1.5 set up. [40:35] Now with real-time 2 and the larger [40:37] context window, there's a lot more [40:39] possible, for sure. [40:41] >> Yeah, absolutely. I think just I want to [40:43] call out, like, this is a reasoning [40:44] model. So, if you're familiar with [40:45] reasoning models from, you know, just [40:47] like our normal main line models, like, [40:49] this model has chain of thought. This [40:51] model, operates a little differently in [40:53] that it kind of has like this idea of [40:55] preambles, but you it's going to provide [40:58] you, kind of, it's reasoning the same [41:00] way our frontier models are. So, it's [41:03] not losing context as it's making those [41:04] parallel tool calls, it's just calling [41:06] tools like our any of our other frontier [41:07] models. So, I think it's actually [41:09] bringing it up to par with what you your [41:11] experience would be of our existing [41:13] models as opposed to like some other [41:15] kind of context or decision consistency. [41:17] I The only call it, I guess, I would [41:19] have is that you don't have to maintain [41:20] the state turn over turn. You can [41:22] dynamically, you know, context engineer [41:25] it, but we will, by default, pass all of [41:28] that turn to turn, so that, you know, [41:30] we're kind of automatically maintaining [41:31] that state and context. [41:33] >> Yeah, absolutely. And yeah, the thing I [41:35] want to stress with this model is how [41:36] good it is at the instruction following [41:38] piece. So, if, uh, you have any [41:40] conflicting instructions in your prompt, [41:43] um, it will, uh, [41:44] you know, try to do both. And so, one [41:47] thing we found really helpful is just [41:49] to, um, basically ask models themselves, [41:52] like, what is there any, [41:54] kind of, conflicting advice or, [41:57] um, better ways to optimize this prompt? [41:58] And it's pretty good at, like, giving [42:00] you that, uh, more optimized version of [42:02] it. [42:04] >> Great. So, um, Ken, thank you so much [42:06] for joining us. Um, we got some really [42:09] great questions and we've been answering [42:10] them in the chat. We're going to send [42:12] out some resources after this, the blog [42:14] that Terry mentioned where you can [42:15] actually see some of those use cases, [42:16] best practices, docs, playground, and as [42:19] always the Build Hour repo where we will [42:21] share um some code snippets with you to [42:23] start playing around with. [42:24] So, our next Build Hour is going to be [42:27] on May 28th. It's going to be on the [42:29] Agents SDK which we also recently [42:32] launched. And as always, you can find [42:34] all of our upcoming Build Hours at the [42:36] link below as well as the recorded [42:38] sessions on YouTube. So, thank you for [42:41] joining us today. You will get a survey [42:44] after this just asking for any feedback [42:46] on topics that you want to learn more [42:48] about. We really try to base these [42:49] sessions on what's interesting, what [42:51] would be most impactful for you to build [42:53] with. So, let us know what you want us [42:55] to cover next. Thanks for tuning in.