LiveKit Challenges AI Leaderboards With Full-Stack Voice Benchmark

LiveKit推出全栈语音基准挑战AI排行榜 公开模型基准难以准确评估实时语音智能体，因其孤立的延迟指标掩盖了破坏自然对话的复合错误及话轮转换延迟。 Jesse Hall 认为，如果开发者仅针对单一组件的速度进行优化，而非兼顾拟人化的对话动态，实时语音智能体就会失效。构建高效的语音系统需要管理严格的延迟预算（目标约 600 毫秒，上限 1.5 秒），衡量首段音频时间（TTFA）和首句时间（TTFS）等感知延迟，并将工具执行卸载到异步线程中。此外，稳健的话轮转换需要音频感知检测来防止误打断，而整体系统性能必须通过端到端用户模拟（如 LiveKit 开源的 Infinite Benchmark）进行验证。 实时语音 Agent 的核心目标是拟人化的对话行为，延迟应被视为运行预算而非最终目的，因为人类用户在交谈中天然会打断、停顿和抢话。 (0:04) 传统的控制台指标容易产生误导：TTS 首字节输出后可能伴随长达 750 毫秒的静音，开发者必须通过 TTFA（首段音频时间）和 TTFS（首句时间）来衡量感知延迟，而非仅看 TTFB 或 TTFT。 (6:00) 通过异步工具调用将耗时 2 至 40 秒的后端查询与主会话解耦，Agent 能够在后台执行任务的同时通过垫话维持对话流畅度。 (7:36) 结合原始声学特征与文本转录的音频感知轮次检测，可在同等延迟预算下将打断错误率从 9.9% 降低至 4.5%，建议将端到端延迟控制在 600 毫秒到 1.5 秒内。 (11:30) 孤立的模型排行榜无法反映系统级表现，LiveKit 推出了开源的“Infinite Benchmark”，通过数百个模拟真实噪音与打断的用户场景来评估完整 Agent 的任务达成率。 (14:53)

LiveKit argues that conversational voice AI requires treating latency as a finite budget toward humanlike behavior rather than pursuing speed for its own sake. Building functional voice agents requires evaluating cascaded pipelines end-to-end and managing perceived audio delays instead of relying on isolated component benchmarks. Managing the Conversational Latency Budget (11:30) Unlike background agents that perform asynchronous tasks over minutes or hours, real-time voice agents fail when they cannot handle everyday conversational dynamics such as pauses and interruptions. LiveKit frames response latency as a budget to spend on natural behavior. An end-to-end latency of roughly 600 milliseconds delivers a responsive, humanlike exchange, whereas 1,500 milliseconds represents the functional ceiling before pauses become awkward. To meet this budget without cutting callers off, systems cannot rely solely on text transcripts for turn detection. LiveKit developed an audio-aware turn detector that evaluates acoustic properties like pitch, rhythm, and intonation alongside provisional text. According to LiveKit, this approach reduced premature user interruptions from 9.9% in alternative models to 4.5%. By running provisional speech-to-text inference and large language model generation concurrently with audio turn detection, the pipeline streams audio the moment a complete sentence is formed. The Pitfalls of Isolated Component Metrics (4:00) Selecting models from isolated public leaderboards introduces substantial production risks because benchmarks test individual components in quiet laboratory conditions. In cascaded pipelines—composed of speech-to-text (STT), large language models (LLM), and text-to-speech (TTS)—errors compound downstream. For instance, if an STT engine transcribes 'Tuesday the 3rd' as 'Tuesday the 30th,' the LLM will confidently confirm the incorrect date regardless of its underlying reasoning capability. Furthermore, standard latency metrics often obscure actual user experience. TTS vendors typically report Time to First Byte (TTFB), yet LiveKit's testing showed some popular engines output up to 750 milliseconds of silence after the initial byte before audible sound plays. Developers must instead track Time to First Audio (TTFA). Similarly, because TTS engines require complete sentence structures to generate natural intonation, LLMs must be evaluated on Time to First Sentence (TTFS) rather than Time to First Token (TTFT). Asynchronous Tool Calling for Continuous Dialogue (7:36) Production voice agents must perform external tool actions, such as querying databases or checking reservation systems, which frequently take between 2 and 40 seconds. In a synchronous pipeline, these latencies cause unnatural pauses that break conversational flow. LiveKit recommends decoupling these operations by executing tool calls on background threads while keeping the voice channel open. By returning control of the microphone to the agent immediately, the system can provide conversational filler or ask follow-up questions while data fetches complete. In a demonstration of a hotel reservation agent ('The Gilded Broom Hotel'), the agent gathered guest details, accepted mid-sentence user interruptions, and processed database writes in the background without introducing pauses into the dialogue or stalling the caller. Evaluating End-to-End Stacks with Simulated Users (14:53) Because isolated model benchmarks fail to reflect interactive speech, LiveKit created the 'Infinite Benchmark,' an open-source testing framework that evaluates complete voice agent applications against simulated user personas. These simulations introduce real-world speech anomalies, including background noise, mumbling, mid-sentence topic changes, and sudden interruptions. The benchmark measures end-to-end task completion—such as successfully recording an accurate booking in a database—while penalizing inefficient dialogue paths and redundant clarifying questions. The open-source harness allows engineering teams to substitute underlying STT, LLM, and TTS providers byte-for-byte to evaluate how complete architectures perform under identical, repeatable conversational conditions. While emerging direct speech-to-speech models offer an alternative to multi-stage cascaded pipelines, LiveKit considers them not yet universally ready for enterprise production. Delivering reliable voice agents currently depends on setting strict latency limits, decoupling tool latency through asynchronous execution, and evaluating whole systems against simulated human behavior.

LiveKit在Ai4大会上提出，构建实时语音AI的核心目标并非单纯追求低延迟，而是实现自然类人的交互行为；延迟应当被视作系统资源的预算，用于权衡模型智能、表达力与多任务处理能力。 串联语音流水线与公开排行榜的局限 (2:19) 目前主流语音智能体主要采用由语音转文字（STT）、大语言模型（LLM）和文字转语音（TTS）构成的级联流水线。尽管端到端语音直连模型正在演进，但在企业级生产环境尚未完全成熟。开发者往往依赖公开排行榜选择单项评分最高的模型，但排行榜只在安静实验室中评测孤立组件，无法反映真实场景。在实际通话中，背景杂音如交通噪音或车载免提极其普遍，一旦前端STT将“3号”误识为“30号”，下游LLM便会基于错误信息得出确认答复，引发严重的复合错误传递。 表面延迟指标与真实用户感知延迟的脱节 (6:00) 传统监控面板统计的测量延迟无法准确反映人类实际体验。TTS供应商常用首字包时间（TTFB）衡量速度，但在接收到首字节后，部分主流TTS服务可能产生长达750毫秒的静音无声期，因此实际指标应为首音频时间（TTFA）。同样，LLM的首Token时间（TTFT）也不能代表语音生成就绪，TTS需要完整的语法结构才能合成自然语音，核心考核应是首句完成时间（TTFS）。一个首Token输出极快但首句拼装缓慢的模型，其端到端听觉延迟反而更差。 异步工具调用与毫秒级轮换控制机制 (7:36) 真实业务中的数据库检索或接口调用往往耗时2至40秒，单靠模型推演速度无法掩盖这一停顿。人类前台会使用垫话（如“正在帮您查询，系统稍慢请稍候”）来维持交流，语音智能体同样需采用异步工具调用，将耗时任务派发至后台线程，把音频控制权即刻交还给主对话流。在话轮转换方面，LiveKit通过结合音频底层特征与文字转录的音频感知检测器，在相同延迟预算下将抢话切断率从竞品的9.9%降至4.5%。整个延迟预算建议以600毫秒作为高度拟真的黄金标准，1.5秒则是对话不显尴尬的极限。 无限基准仿真测试与全栈性能验证 (14:53) 针对静态榜单无法预测系统级表现的问题，LiveKit推出了开源评估工具“无限基准（Infinite Benchmark）”。该方案通过自动化模拟数百位真实用户，刻意注入含糊发音、中途改口和打断等人类对话特征，全面评测任务最终达成率，并对产生冗余追问的低效路径予以扣分。开发者可以在保持对话场景完全一致的情况下，byte-for-byte无缝替换底层模型以评估整体架构。测试显示，运行在LiveKit推理栈上的Gemma 4 31B比其他平台上的同款模型速度提升两倍，成功容纳于实时预算窗口之内。 构建高质量语音智能体需要开发者转变思路：先确立严格的延迟预算（推荐600毫秒至1.5秒之间），以TTFA和TTFS测量真实感知延迟，将耗时操作解耦为异步执行，并利用全流程模拟基准而非单模榜单持续验证业务闭环。

## Background 该视频由频道 Ai4 于 2026 年 8 月 20 日发布在 YouTube 平台上，由 LiveKit 的 Staff Developer Advocate（员工开发者布道师）Jesse Hall 主讲。视频简介指出，演讲聚焦于如何在实时对话中选拔与构建 AI 技术栈，揭示公开排行榜与仪表盘延迟的认知误区，并提出以拟人化行为为导向的评测体系、决策框架及开源基准测试。 ## TL;DR 演讲指出实时语音智能体的核心目标并非单纯追求极速，而是以“延迟作为预算”来实现高度拟人化的自然交互体验。公开排行榜的单模评测和供应商仪表盘上的账面数据（如 TTFB、TTFT）无法反映级联错误与用户真实感知的首音延迟（TTFA/TTFS），容易产生度量盲区。通过采用主线程解耦的异步工具调用、结合声学特征的重叠式话轮检测（Turn Detection）以及压低误打断率，系统能够化解长时间死寂与抢话冲突。此外，LiveKit 提出了“无限基准测试”，通过端到端对抗模拟与真实业务结果对模型矩阵进行严苛评估，为开发者提供了一套生产级落地清单。 ## Atmosphere & style 演讲整体氛围务实犀利、幽默风趣且极具工程实战感，讲者通过现场举手表决和调侃峰会 Wi-Fi 等轻松互动迅速调动现场情绪。叙事节奏从对传统语音客服笨拙死板体验的生动吐槽，层层推进至深水区架构逻辑的精密拆解，展现出扎实的系统级思考。配合极速订房通话演示与硬核实测数据的对比，整场分享透露出一种拨开行业跑分迷雾、回归真实人机体验本质的理性与自信。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=4s) 实时语音智能体与后台智能体存在本质区别：后台智能体用户可以等待推送，而实时语音中用户会自然插话或打断，单纯追求低延迟无法解决交互崩溃问题，核心目标必须是打造高度拟人化的体验。 - [01:38](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=98s) 构建语音智能体主流采用 STT、LLM 和 TTS 级联流水线架构；尽管端到端语音大模型正在兴起，但目前在通用生产环境中仍不够成熟，系统整合面临复杂的工程挑战。 - [03:22](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=202s) 公开基准排行榜仅衡量静室环境下的单体组件能力，无法反映真实嘈杂环境下的表现，上游 STT 的细微误识会引发下游 LLM 和 TTS 的灾难性级联错误。 - [06:00](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=360s) 供应商仪表盘指标存在度量盲区：TTS 的 TTFB 可能包含数百毫秒静音，真正关键的是首音频时间（TTFA）；LLM 的 TTFT 无法指导语音合成，必须关注生成首个完整句子的时间（TTFS）。 - [07:36](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=456s) 耗时较长的后台工具调用必须与对话交互主线程完全解耦并异步执行，智能体在等待结果时应像人类前台一样提供即时口头反馈，避免出现长时间的沉默死寂。 - [11:36](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=696s) 传统基于文本转录的话轮检测反应迟钝且易误判停顿，LiveKit 采用直接分析底层原始声学信号与波形特征的 Turn Detector，将打断率从 9.9% 压低至 4.5%。 - [12:10](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=730s) 通过重叠式流式流水线优化延迟预算：在用户说话时基于临时转写并发启动 LLM 推理，大模型生成首句后 TTS 立即流式回传，最大化压缩首个可听字节时间。 - [14:10](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=850s) 延迟窗口存在物理上限（1.5 秒显迟钝，600 毫秒接近自然对话），LiveKit 提出“无限基准测试”，强调脱离具体业务场景选模型是伪命题，必须针对完整架构进行定制评估。 - [15:15](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=915s) 无限基准测试采用模拟真实用户对抗施压的方式，重点考核数据库写入正确性等真实业务结果，并对冗余提问与多余步骤设置效率惩罚机制，支持逐字节横向置换底层模型对比性能。 - [17:38](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=1058s) 语音 Agent 不仅仅是带有扬声器的聊天机器人，而是拟人化的实时基础设施；在满足端到端延迟预算的前提下，应将剩余时间预算全部倾斜于提升回复质量与拟人真实度。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=4s) 后台智能体与实时智能体的鸿沟：从“追求低延迟”到“追求拟人感” 我们平时构建的大多数 AI 智能体（Agent）其实都在后台默默工作。我们交代给它们一项任务，转身离开，等过段时间回来看时，它们已经在后台写完了代码、做完了调研。在这种场景下，根本没有人在意智能体是多花了 10 秒钟、10 分钟甚至 10 个小时，大家只需要等手机弹出一条“任务已完成”的推送通知即可。然而在过去的一年里，几乎每个人都开始接触到另一类完全不同的智能体——实时语音智能体（Realtime Agent）。无论是在药房打电话续处方药、致电航空公司客服，还是联系汽车经销商，我们总会在某个时刻做出最符合人类本能的举动：直接打断对方或插话。而就在你插话的那一瞬间，整个交互瞬间崩塌。智能体要么继续自顾自地往下念，要么直接死机，糟糕的体验让人忍不住对着话筒抓狂大喊：“转人工！快给我人工客服！让我离这个破 AI 机器人远点！” 如果向业内同行请教如何解决这一痛点，大家给出的第一反应往往是：“提速、降低延迟、换一个响应更快的模型。”诚然，速度和低延迟确实重要，但这完全不是根本答案。“延迟只是一笔预算，而我们真正的目标截然不同：打造一个尽可能拟人化（Humanlike）的智能体。”所谓拟人，意味着它能够在说话的同时倾听对方的声音；能够准确分辨你是已经说完了完整的句子，还是中间停顿正在思考；当你中途突然改变主意时，它能够心领神会地自然接话，而不是彻底宕机。单纯追求速度根本无法带我们抵达这个目标，真正的关键在于为具体场景挑选合适的模型，并对流水线（Pipeline）进行合理的系统级架构设计。然而现实中的最大难题在于：现有的评测体系根本无法告诉你究竟哪款模型最适合你的特定业务场景。 ### [01:38](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=98s) 拟人化目标与级联管线的现实挑战 在深入探讨技术细节前，Jesse Hall 面向现场观众做了一个举手表决：在场真正亲手构建过语音智能体并成功推向生产环境面向真实用户的开发者，仅仅只有寥寥数人。相比之下，当问到谁用过 ChatGPT 的高级语音模式（Advanced Voice Mode）进行实时对话时，绝大多数人都举起了手。OpenAI 当初正是选择了 LiveKit 来驱动这项标杆级的功能。LiveKit 是一个开源的实时框架与基础设施平台，致力于让智能体实现极致自然、高度拟人的语音交互。智能体不仅能听、能说，还能通过视觉进行感知，甚至 LiveKit 内部还设有机器人事业部来探索具身智能（Physical AI）。但无论形态如何延伸，语音始终是人机交互中最自然、最直观的切入点，同时也是集中爆发工程挑战的“深水区”。 构建一个最基础的语音智能体，底层至少需要将三个独立模型串联成一个级联管线（Cascaded Pipeline）：首先由语音转文字（STT / Speech-to-Text）模型捕捉声音，再交给大语言模型（LLM）理解并生成回复文本，最后通过文字转语音（TTS / Text-to-Speech）模型合成语音输出。尽管业内也涌现出了一批端到端语音大模型（Speech-to-Speech），这类新技术在特定垂直场景下表现惊艳，但目前在通用生产环境中还远未达到全面成熟的标准。这意味着开发者不仅要在成熟的级联管线与新兴的端到端模型之间反复权衡，还要面对市场上成百上千种模型以及数十家云服务提供商的排列组合，整体系统架构的复杂度呈指数级上升。 ```mermaid flowchart LR accTitle: 典型语音智能体级联流水线架构 subgraph Input["用户输入端"] A["用户音频输入"] end subgraph CascadePipeline["三阶段级联处理管线"] B["STT (语音转文本)"] C["LLM (逻辑决策与文本生成)"] D["TTS (文本转语音合成)"] end subgraph Output["音频输出端"] E["实时音频播放"] end A --> B B --> C C --> D D --> E ``` ### [03:22](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=202s) 排行榜陷阱与级联误差：单模评测为何无法代表真实体验 绝大多数开发者在技术选型时，往往习惯直接参考各大公开基准排行榜（Leaderboard），直接挑选榜单排名第一的模型部署进项目。然而在实时语音系统中，这种做法必定会踩坑。排行榜衡量的是隔离状态下的单体组件能力，但开发者最终交付给用户的却是一整套协同运转的动态系统。当某个单点组件在真实环境里出现微小偏差时，整套管线便会引发灾难性的级联错误（Cascading Errors）。 - **静室评测与真实噪杂环境脱节**：公开基准测试大多在安静无噪的实验室录音中跑分；但真实用户往往是在嘈杂的早晚高峰车流中开着免提通话，后座还伴随着小孩子的吵闹声，这种恶劣的真实信噪比是单模排行榜完全无法覆盖的。 - **级联错误向下击穿整个系统**：假设用户在电话里说“想预约周二 3 号（Tuesday the 3rd）”，但 STT 模型在噪音下误识别成了“周二 30 号（Tuesday the 30th）”。此时后面的 LLM 无论有多聪明、逻辑推理能力有多强，它都会无比自信地按照 30 号去办理预约，而后续环节的 TTS 模型也只能将错就错地读出错误日期。整条管线中的下游模型对上游发生的错误浑然不知。 - **提问维度的根本错位**：许多团队启动语音项目时，都在纠结“我到底该选哪款 TTS 声音最好听？”或者“哪款 LLM 推理最强？”。但正确的提问方式绝不该是单选某一个组件，而是“我到底应该交付哪一整套系统技术栈（Stack）？” ### [06:00](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=360s) 仪表盘延迟 vs 感知延迟：TTFB 与 TTFT 的度量盲区 即便避开了排行榜的模型选择盲区，团队在建立度量指标时也极易落入“测量错误数据”的窠臼。最典型的误区就是将供应商仪表盘上的账面指标直接等同于人类耳朵的实际感受，混淆了“测量延迟”（Measured Latency）与“感知延迟”（Perceived Latency）。 在 TTS 层面，主流语音合成供应商通常会在监控面板上重点标榜其“首字节时间”（Time to First Byte, TTFB）。TTFB 在图表上看起来非常亮眼，但其根本漏洞在于：“字节只是数据，字节不是声音（A byte is not sound）。”LiveKit 团队在实测中发现，某些市面上非常流行的 TTS 供应商在推送出第一个音频数据字节后，紧接着会输送长达四分之三秒（750 毫秒）的静音数据帧，之后扬声器才会发出真实声响。对于守在电话旁的人类用户而言，这 750 毫秒完全是令人抓狂的沉默死寂。因此，真正具有工程指导意义的衡量指标必须是“首音频时间”（Time to First Audio, TTFA），即用户耳朵真实听见声音的那一刻。 ```mermaid flowchart TD accTitle: 文本指标与感知指标的度量对比 subgraph DashboardMetrics["供应商面板指标 (容易误导)"] M1["TTFB: 传输首个二进制字节 (可能夹带 750ms 静音)"] M2["TTFT: 吐出第一个 Token (无法触发语音合成)"] end subgraph PerceivedMetrics["人类感知真实指标 (真正关键)"] P1["TTFA: 扬声器真正发出声音的瞬间"] P2["TTFS: 模型吐出首个完整句子的瞬间"] end M1 -.->|"实际转化为"| P1 M2 -.->|"实际转化为"| P2 ``` 大语言模型侧也存在一模一样的度量盲区。业界往往狂热追求 LLM 的“首 Token 时间”（Time to First Token, TTFT），但 TTFT 本质上只是一个纯文本维度的监控项。对于级联管线而言，下游的 TTS 模型根本不可能根据单个 Token 就开始起唇发音，它必须拿到包含语法语义结构的完整语句。因此，语音团队真正需要衡量的是“首句时间”（Time to First Sentence, TTFS）。在实际评测中经常会出现反直觉的现象：流式吐出第一个 Token 最快的模型，由于输出长句节奏不均或分词策略差异，往往并不是最先输出完整第一句话的模型。 ### [07:36](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=456s) 异步工具调用：打破主线程阻塞的拟人化技巧 在实际业务场景中，智能体不可能只陪用户聊天，它必须挂载工具函数（Tool Calling / Function Calling）去执行具体操作——查询后台数据库、调取订单详情、查询空余客房排期等。这些网络 I/O 与后台业务逻辑的处理耗时往往极其不稳定，少则 2 秒，长则 10 秒甚至 40 秒以上。面对数十秒的业务阻塞，前端哪怕换用世界上速度最快的 AI 模型，也不可能凭空掩盖 40 秒的死寂。 此时最有效的破局方式依然是师法人类：想象一下现实中专业的人类前台接待员是如何处理类似状况的。当系统卡顿时，人类前台绝不会面无表情地沉默 40 秒，而是会一边在键盘上敲打，一边抬起头对客户说：“请稍等我一秒钟，我正在为您调取系统记录……今天系统有点慢，请再给我一小会儿，马上就好。”人类天然懂得通过即时反馈和状态同步来为后台任务争取时间。在 LiveKit 智能体中，只需通过简单的代码逻辑即可实现这一行为：彻底打破传统单线程阻塞模式，将对话交互主线程与后台工具调用线程完全解耦，以异步（Asynchronous）方式运行。智能体发出工具请求的同时将控制权交还给语音循环，继续向用户说话，而无需在死寂中挂起等待。 ```mermaid flowchart TD accTitle: 异步工具调用与对话解耦流程 subgraph ThreadMain["对话交互主线程 (持续沟通)"] T1["用户提出需求: 预订下周房间"] --> T2["智能体立即确认并索取补充信息"] T2 --> T3["询问床型偏好 / 确认姓名电话"] T3 --> T4["自然推进预订流程"] end subgraph ThreadAsync["后台异步工作线程 (不阻塞主线程)"] A1["发起异步工具调用: 查询房态"] A2["后台执行数据检索与落盘"] A3["返回查询结果 / 若失败则触发纠错分支"] end T1 -.->|"非阻塞触发"| A1 A1 --> A2 --> A3 ``` - **现场音频实录演示**：Jesse Hall 播放了一段预录的“金扫帚酒店（Gilded Broom Hotel）”订房通话演示（之所以不用现场实测，他幽默地调侃道：“大家都知道各大技术峰会的现场 Wi-Fi 是个什么德行”）。 - **极速丝滑的交互体验**：录音展示了极高的来回交互流转度。当智能体正在介绍房间规格并询问“您想要单人床还是……”时，演示者直接抢话打断说“大床房（King）”，智能体毫不停滞地自然顺应并记录下来，接着迅速询问预订姓名与邮箱。 - **后台工具静默运行**：在智能体快速向用户核对“Jesse Hall”、收集邮箱与“555-1234”电话号码的同时，后台一直在持续异步调用工具存储信息，它不需要等工具执行完毕才抛出下一个问题。一旦后台工具中途调用失败，智能体也会优雅地回溯一句“请允许我快速核对一下信息”，从而在不制造冷场的前提下修正错误。 ### [11:36](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=696s) 轮流发言检测（Turn Detection）：超越文本的音频感知 公开排行榜无法解决的最后一个深层机制，是对话中的轮流发言控制（Turn-taking）。人类在日常交流中，凭借对方说话的语气语调、语速节奏以及上下文暗示，往往只需要几百毫秒就能本能地完成发言权的交接与协商，精准知道何时该保持倾听、何时轮到自己开口。 传统的语音机器人往往只盯着由 STT 输出的单薄文本转录（Transcript）来进行断句，依赖固定时长的静音阈值。这种纯靠文本的管线不仅反应极其迟钝，而且极易误判人类在组织语言时的中间停顿。为此，LiveKit 专门构建了一套在对话过程中持续倾听底层原始声学信号的发言轮换检测器（Turn Detector）。它不仅综合参考实时文本转录，更是直接深度分析音频波形本身的声学特征，从而在毫秒级预算内做出兼具敏锐度与拟人感的对答决断。 ### [12:10](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=730s) 话轮转换与重叠式流式流水线 在实时的语音交互中，系统必须直接倾听音频本身，捕捉用户的节奏、音调和语调起伏，而不仅仅是等待文字转写。根据 LiveKit 公布的实测数据，在相同的延迟预算下，市面上表现最好的竞品方案打断用户的概率仍高达 9.9%，而 LiveKit 的系统将这一打断率压低至 4.5%。这一测试框架已经完全开源，开发者可以在自己的实际业务场景中验证它对降低打断率的效果。话轮转换（Turn-taking）是一个极其关键的衡量维度，但它从来不是单个模型能单独决定的，而是整个系统级工程协作的结果。 所谓的延迟预算，本质上是指“用户说完最后一个音节”到“Agent 返回第一声问候音频”之间的绝对时间窗口。为了抢占这一时间差，流水线各环节必须高度重叠运作：话轮结束检测（End-of-Turn Detection）直接根据实时音频信号触发，甚至先于最终转写文本生成；语言模型（LLM）则在临时转写（Provisional Transcript）的基础上提前并发运行。人类在日常表达中往往会用不同方式把一句话重复三遍，而大模型足够聪明，在用户还在说话时就已经推断出意图并开始生成回复——只要用户没有在句中突然改变主意，模型就已经走在用户前面了。一旦大模型生成出第一个完整句子，文本转语音（TTS）引擎便立即开始回传流式音频，而不是等待所有 Token 全部生成完毕。系统追求的核心指标是“首个可听字节时间”（Time to First Audible Byte），而非传统意义上的首字生成时间。 ```mermaid flowchart TD accTitle: 实时语音 Agent 重叠式流式处理流水线 subgraph InputStage["输入感知层"] A["实时用户音频流"] --> B["音频声学特征检测<br/>(话轮结束预判)"] A --> C["临时文本转写<br/>(Provisional STT)"] end subgraph LLMStage["思考决策层"] B --> D["并发启动 LLM 推理"] C --> D D --> E["首个完整句子生成"] end subgraph OutputStage["输出合成层"] E --> F["TTS 流式音频合成"] F --> G["首个可听字节回传<br/>(首音播放)"] end ``` - **打断率对比**：在同等延迟预算基准下，最佳替代方案打断通话者的概率为 9.9%，LiveKit 优化架构降至 4.5%。 - **流水线重叠机制**：基于音频声学特征的话轮判定先于最终 STT 落地，LLM 依据临时转写提前生成，TTS 命中首句即开启流式播放。 --- ### [14:10](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=850s) 延迟预算与“无限基准测试”（Infinite Benchmark） 延迟窗口具有固定的物理上限。如果交互延迟达到 1.5 秒的上限，对话勉强能够维持，但已经开始让人感到尴尬与停顿；而当延迟压缩至 600 毫秒左右时，交互才会真正展现出接近人类对话的自然感。因此，开发实时 Agent 时核心问题绝不是单纯询问“系统能跑多快”，而是“在这个固定的延迟预算内，Agent 能够做到多么像人”。正如演讲所强调的理念：“延迟是预算，像人才是目标（Latency is a budget, but humanlike is the goal）”。 脱离具体的系统去孤立地询问“应该选哪个大模型”，实际上是一个无法回答的伪命题。为了解决这一痛点，LiveKit 构建了一套被称为“无限基准测试”（Infinite Benchmark）的评测体系。之所以称之为“无限”，是因为业界不存在通用的单一排行榜，有多少种 Agent 架构和业务场景，就应该有多少种对应的基准测试。 --- ### [15:15](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=915s) 基于真实结果的对抗式评估与模型评测 在这套基准测试中，评估对象是一个包含真实后端功能与工具调用的完整 Agent（例如开源的前台接待员 Agent）。测试系统会模拟数百个虚拟用户对其发起极限施压——这些模拟用户会像真实人类一样中途插话、喃喃自语（Mumble）、在说到一半时突然变卦，竭尽全力去“击垮”Agent。评估的核心标准绝非主观感受（Vibes），而是真实业务结果（Outcomes）。 ```mermaid flowchart LR accTitle: 无限基准测试的对抗评估与模型置换流程 subgraph Simulation["真实对抗模拟"] U["数百个模拟用户<br/>(打断/含糊/变卦)"] --> AG["完整业务 Agent<br/>(集成真实工具与后端)"] end subgraph Evaluation["基于业务结果的评分"] AG --> M1["业务正确性<br/>(预订入库/无幻觉)"] AG --> M2["效率惩罚机制<br/>(冗余提问/多余步骤)"] end subgraph Swap["模型矩阵替换"] M1 & M2 --> SW["逐字节输入相同场景<br/>横向置换不同底层模型"] end ``` 评估体系设立了严格的拟人化底线与效率惩罚机制： - **核心业务完成度**：检验预订信息是否准确写入数据库、通话是否被异常截断、是否存在模型幻觉或调用未授权工具等行为。 - **效率惩罚机制**：仅完成任务不能直接打勾，如果 Agent 提出了不必要的冗余问题，或者原本 2 步就能搞定的流程硬是走了 10 步，系统将直接扣分，以防止模型通过投机取巧“刷分”。 - **横向置换对比**：在固定整个 Agent 业务架构与模拟对抗场景的前提下，系统会逐字节（Byte-for-byte）输入完全相同的测试数据，仅替换底层模型，从而清晰绘制出不同模型在特定业务场景下的真实表现图谱。 --- ### [17:38](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=1058s) 语音 Agent 架构哲学与落地行动清单 LiveKit 本身并不自研 STT、LLM 或 TTS 基础大模型，而是专注于构建全生态接入的基础设施。这种中立角色使其能够客观地审视所有模型的赛跑状态并挑选优胜者。例如在 LiveKit Inference 上运行的 Gemma 4 31B 模型，并非直接照搬开源开箱版本，而是针对低延迟、高拟人度和快速响应进行了专门调优与推理加速，其实际推理速度达到了在其他通用推理平台上运行同款模型的两倍。 演讲总结指出：“语音 Agent 绝不仅仅是一个挂着扬声器的聊天机器人（A voice agent is not just a chatbot with a speaker attached to it），它是一个能够像人一样进行感知和响应的拟人化实时基础设施。”当开发思维发生这一转变时，技术选型便会清晰明确。 **生产级实时语音 Agent 落地清单：** 1. **优先设定延迟预算**：首先界定能够接受的延迟上限，测量终端用户感知到的端到端延迟（首个可听字节时间）。 2. **预算倾斜于质量**：在满足延迟预算的前提下，把剩余的时间预算全部投入到提升回复质量与拟人真实度上，无需无意义地卷极致速度。 3. **耗时调用全异步化**：任何耗时超过 1 秒的后端工具调用都必须转为异步执行，确保 Agent 能够继续与用户交流而不被阻塞挂起。 4. **全链路端到端评测**：测试整个对话流的完整系统行为，而非孤立测试各个单点组件。 5. **在生产环境中持续评测**：利用开源的基准测试工具集，在实际生产环境中保持迭代，构建出面对打断能自然应对、具备真正人类质感的语音系统。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=4s) 转人工！快给我人工客服！让我离这个破 AI 机器人远点！ — Jesse Hall > [00:04](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=4s) 提速、降低延迟、换一个响应更快的模型。 — Jesse Hall > [00:04](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=4s) 延迟只是一笔预算，而我们真正的目标截然不同：打造一个尽可能拟人化（Humanlike）的智能体。 — Jesse Hall > [06:00](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=360s) 字节只是数据，字节不是声音（A byte is not sound）。 — Jesse Hall > [03:22](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=202s) 我到底该选哪款 TTS 声音最好听？ — Jesse Hall > [03:22](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=202s) 我到底应该交付哪一整套系统技术栈（Stack）？ — Jesse Hall > [07:36](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=456s) 请稍等我一秒钟，我正在为您调取系统记录……今天系统有点慢，请再给我一小会儿，马上就好。 — Jesse Hall > [14:10](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=850s) 延迟是预算，像人才是目标（Latency is a budget, but humanlike is the goal） — Jesse Hall > [17:38](https://www.youtube.com/watch?v=8ENc4qoVlYA&t=1058s) 语音 Agent 绝不仅仅是一个挂着扬声器的聊天机器人（A voice agent is not just a chatbot with a speaker attached to it），它是一个能够像人一样进行感知和响应的拟人化实时基础设施。 — Jesse Hall ## Mentioned Jesse Hall, LiveKit, OpenAI, ChatGPT, Advanced Voice Mode, Gemma 4 31B, LiveKit Inference, Infinite Benchmark, STT (Speech-to-Text), LLM (Large Language Model), TTS (Text-to-Speech), TTFB (Time to First Byte), TTFT (Time to First Token), TTFA (Time to First Audio), TTFS (Time to First Sentence) ## Video description Jesse Hall - Staff Developer Advocate, LiveKit With hundreds of models to choose from, how do you pick a stack that holds up in a live conversation? Most agents work in the background, but the moment one has to hear, see, and speak with a human in real time, everything about how you build and evaluate it changes. I'll show why public leaderboards fail for realtime agents, and why the latency on your dashboard isn't the latency your users feel. Then I'll define the outcomes you want as human-equivalent behaviors, and work backwards to your evals, your models, and a production iteration loop. You'll leave with a concrete decision framework and an open benchmark you can run on your own stack. [Source](https://www.youtube.com/watch?v=8ENc4qoVlYA)

[0:04] Jesse Hall: 转人工！快给我人工客服！让我离这个破 AI 机器人远点！ [0:04] Jesse Hall: 提速、降低延迟、换一个响应更快的模型。 [0:04] Jesse Hall: 延迟只是一笔预算，而我们真正的目标截然不同：打造一个尽可能拟人化（Humanlike）的智能体。 [6:00] Jesse Hall: 字节只是数据，字节不是声音（A byte is not sound）。 [3:22] Jesse Hall: 我到底该选哪款 TTS 声音最好听？ [3:22] Jesse Hall: 我到底应该交付哪一整套系统技术栈（Stack）？ [7:36] Jesse Hall: 请稍等我一秒钟，我正在为您调取系统记录……今天系统有点慢，请再给我一小会儿，马上就好。 [14:10] Jesse Hall: 延迟是预算，像人才是目标（Latency is a budget, but humanlike is the goal） [17:38] Jesse Hall: 语音 Agent 绝不仅仅是一个挂着扬声器的聊天机器人（A voice agent is not just a chatbot with a speaker attached to it），它是一个能够像人一样进行感知和响应的拟人化实时基础设施。

[0:04] Jesse Hall: When interacting with voice bots at a pharmacy, an airline, or a car dealership, users inevitably do the most human thing possible: they talk over the agent. [0:04] Jesse Hall: Latency is merely a budget, whereas the actual goal is to create an agent that behaves as naturally and humanlike as possible. [0:04] Jesse Hall: A humanlike agent listens while speaking, senses whether a user has finished a thought or is merely pausing, and gracefully adapts when a user changes their mind mid-conversation. [4:00] Jesse Hall: Instead of asking isolated questions like "Which TTS should I use?", developers must evaluate system-level questions like "Which stack should I ship?" [6:00] Jesse Hall: Testing reveals that some popular TTS providers output up to three-quarters of a second (750 ms) of pure silence after the first byte before delivering actual audio. [6:00] Jesse Hall: A TTS engine cannot begin generating natural speech from a single token; it requires a complete sentence structure. [7:36] Jesse Hall: By handing the microphone back to the agent while backend functions run, the bot continues talking to the caller instead of blocking the conversation. [11:30] Jesse Hall: Human turn-taking relies on subtle vocal cues, pitch, rhythm, and intonation, with turn transitions negotiated within a few hundred milliseconds. [11:30] Jesse Hall: Under identical latency budgets, alternative turn-detection approaches cut off callers 9.9% of the time, whereas LiveKit's turn detector reduces interruption errors to 4.5%. [14:53] Jesse Hall: Rather than evaluating individual components or relying on subjective impressions ("vibes"), it evaluates complete agents against simulated user scenarios. [18:36] Jesse Hall: A voice agent is not merely a chatbot with a speaker attached; it is real-time infrastructure designed to sense and respond like a person.

[0:04] Well, most of the agents that uh that we  build, they do work in the background, [0:10] right? We we set them, we give them a task and  and we we walk away. We check back. They are [0:16] running they're they're compiling code. They're  uh running research. They're doing things for [0:21] us in the background. And nobody really cares if  it takes a background agent an extra 10 seconds, [0:28] an extra 10 minutes, maybe even 10 hours. We don't  care. We're just going to check back on it. Maybe [0:33] we get a ping on our phone saying it's done,  right? But at some point in the last year or so, [0:38] you have probably interacted with another  type of agent, a real time agent. Maybe it was [0:45] uh a pharmacy phone call where you uh were re  refilling your prescription. Maybe it was an [0:50] airline. I called one the other day or I talked to  one the other day when I called a car dealership [0:55] and at some point in that conversation you did the  most human thing possible. You talked over it and [1:03] everything fell apart. Maybe it interrupted you.  Um but it just was not a good experience and you [1:09] just started shouting operator human like just  get me away from this AI bot. Right? Uh and so [1:15] if you ask other people in the industry how to  fix this, right? the first thing that's going [1:20] to uh that they're going to say is speed, low  latency, pick a faster model. Um you know those [1:25] those things are important. Yes, but that is  not the right answer. You see uh latency it's a [1:31] budget but the goal is something else entirely.  The goal is to create an AI agent that is as [1:38] humanlike as possible. And so uh we want an agent  that can listen while it talks. uh one that knows [1:47] the difference between when you've finished your  thought or you're still thinking, right? Uh maybe [1:52] you change your mind mid conversation. We want it  to just roll with it, right? So speed alone does [1:58] not get us there. What gets us there is picking  the right models and the way that we configure our [2:03] pipeline. But this is the problem. No one can tell  you how to pick the right model for your use case, [2:10] for your agent. And and so this is the problem  that we need to fix. So a quick show of hands [2:19] here. Who here has uh actually built a voice agent  and shipped it to production to real users? Okay, [2:29] just just a couple of you. So a couple of you know  the challenges that you might face in shipping one [2:34] of these to production. Um for the rest of us,  you've probably again experienced an agent. Uh, [2:40] who here has used Chat GBT's advanced voice mode  to have a conversation with ChatgBT? Okay, very [2:48] good. Most of us have done that. So, OpenAI chose  LiveKit to power that feature. LiveKit is the [2:55] open-source real-time framework and infrastructure  for the most natural humanlike sounding voice [3:01] agents possible. Uh, they can hear, they can see,  and they can speak with a person, right? So it's [3:09] not just voice uh it also handles uh video data  and even physical AI. We have a robotics division. [3:16] Um but voice is the natural onboarding experience  and it is where a lot of challenges come into [3:22] play. So that is what we're going to focus on in  this talk. So a voice agent is at minimum three [3:29] models put together working together. Right? So  we have speech to text and LLM and then text to [3:36] speech. And I do have to acknowledge yes there are  real-time models some speechtoech models those are [3:43] a new technology uh they are they're they're great  at certain use cases but they are not production [3:49] ready for every use case. And so the fact that  we have this cascaded pipeline to choose from [3:55] and then we also have this real real time models  to choose from that even makes it more complex, [4:00] right? And so here is what we actually  have to choose from. Hundreds of models, [4:04] dozens of providers. Uh a show of hands here  again. Who here has picked a model based on what [4:10] they saw at the top of a leaderboard? Right? Most  of us just pick that top model and we go with it, [4:15] right? Um but this does not work with real time  agents. Leaderboards they score the components [4:23] the individual components but we are shipping  a pipeline. We're shipping a system right? So [4:28] leaderboards don't score that today. So let's  say that your uh your caller wants to book an [4:33] appointment for Tuesday the 3. Uh but your  speechtoext model heard Tuesday the 30th. [4:42] From that moment on, doesn't matter how smart your  language model is, it is now confidently booking [4:47] the wrong day and every error goes goes downhill.  The no none of the other models know about this [4:53] error, right? And so those leaderboards, those  scores, most of them came from a quiet room [5:00] um in in in isolation, right? Uh but your users,  they're on a speaker phone talking in in traffic, [5:07] maybe with kids in the back seat, right? the  leaderboards can't score your environment, [5:13] your users, right? So there um how do we then a  avoid this leaderboard trap? Um the the score it [5:23] measures the model, but again we're shipping a  conversation. Uh so uh everyone building a voice [5:29] agent today, they they all start out by asking the  same question. Which text to speech should I use? [5:34] They focus on what the what the voice sounds like,  right? Which one should I choose? or which LLM or [5:39] which which TTS should I choose? Right? But that's  not the right answer. The the real or the that's [5:44] not the right question. The real question is which  stack should I ship? So these are two completely [5:51] different questions and the leaderboards again  they don't have an answer to this question, right? [5:56] But there is an answer and we'll talk a little bit  more about that in a minute. Um and so let's say [6:00] you fall into this trap. You pick the model based  on what you saw on a leaderboard and uh you start [6:05] measuring. This is the next issue. You're probably  measuring the wrong thing. So, your TTS provider [6:13] um they report time to first bite. Uh and that  looks great on a dashboard, but bytes a bite is [6:21] not a audio. A bite is not sound, right? So, uh  we've actually measured some popular providers [6:27] that are shipping up to 3/4 of a second of silence  after that first bite before you ever hear audio. [6:36] And and so time to first audio is what your caller  actually hears. So that's what we want to measure. [6:42] Time to first audio, not time to first bite. And  and so this is the distinction that this entire [6:47] talk kind of hangs on is measured latency versus  perceived latency. What does the dashboard say [6:54] versus what your ears actually hear. And so the  same thing goes with large language models. So [7:00] time to first token, that's a text metric. Um,  but a TTS can't start speaking on a token. It [7:09] needs a sentence. So, time to first sentence  is the thing that we should be measuring for [7:14] uh for language models because um a a language  model that streams a token the fastest, the first [7:21] token the fastest is not always the model that  finishes the first sentence the fastest. So, [7:27] we have to measure those differently. Um so now  even if you we measure everything perfectly um [7:36] there's another part that's ignored here and this  is that a real agent has to do things. It has to [7:41] call tools. It has functions. It has to go out and  get context contact a data database or something. [7:46] It looks up an order. It checks for availability.  It books rooms, right? And so uh these tools take [7:52] time and and this is not something that we can  really speed up within the agent itself. It's it's [7:58] kind of dependent on these other things, right?  So 2 seconds, 10 seconds, 40 seconds or more. A [8:03] model choice is not going to hide 40 seconds or  more of latency. Uh and so think about what a [8:09] human receptionist would actually do here. A human  would uh would say, "Yeah, just give me give me a [8:16] second. I'm pulling the system up, looking looking  up the order. Um it's running a little slow. Give [8:21] me another second. Almost there." Right? So a  human is actually listening and giving updates [8:27] and it's it's keeping it's buying time basically  uh for for the the person on the other end. And so [8:33] that is the humanlike behavior. And so why can't  an agent do that? Why can't an agent just uh do [8:41] things asynchronously while other things are going  on? Well, it totally can. Here's what that looks [8:46] like. Uh, and this is the only code that you'll  see in these slides, but this is um a a basic [8:52] tool in a LiveKit agent. And it is uh there's  just one line here. And this one line is what [9:01] uh what the first update hands the mic back to  the agent. So basically, we're we're we're making [9:08] asynchronous tool call. So we're just taking the  conversation and the tool call and we're stop [9:13] we're we're not sharing a thread anymore, right?  So, we put those on two separate threads. Uh, [9:18] but don't take my word for it. Let's listen to  a real conversation. Thank you for calling the [9:23] Gilded Broom Hotel. Talk to me and they'd like  to book a room. [snorts] What dates are you [9:29] looking at? Uh, next week, uh, Wednesday through  Friday. Next Wednesday, July 30th to Friday, [9:37] August 1. That's right. How many guests will be  there? Two. I have we do that eating double V. [9:51] Would you prefer or push? Yeah,  please. May I have your name [9:57] for the booking? Jeffy [snorts] Paul. Is  that Jeffy Paul? Yes. What's your email? [10:11] Correct. That's right. And  your phone number 5 by51 23 4 [10:22] is that 5551 23 or by fix that. That's right. I'll  need a credit card to fold the roof. What's the [10:32] card number? We'll stop. Okay. So, obviously  I didn't want to try that uh live here on, [10:38] you know, conference Wi-Fi. No, no offense  to conferences, but they're all the same. [10:42] But having that recorded, you could see how quick  that back and forth was, how humanlike it was. I [10:47] even interrupted it. It was asking me which bed  that I wanted. I knew I wanted a king-size and I [10:51] just said king. And the agent didn't care. It just  it accepted that and it moved on. Notice also that [10:56] it was record it was actually it's it's actually  calling tools. It's recording all of the things [11:01] that it's asking me, but it's not waiting for that  tool to finish before it goes moves on to the next [11:06] question. It calls the tool, moves on to the next  question, it keeps talking. If for some reason [11:11] that tool would have failed, it would have gone  back to it and said, "Let me verify something real [11:15] quick." It would have gone back to it. Right? So  that is the human-like behavior that we're looking [11:19] for. And so this is not just one model being good.  This is the entire system being good. This is the [11:25] infrastructure, the framework, uh the models,  they're all working together. And this is the [11:30] thing that a leaderboard can't show you today.  So there's one more thing that leaderboards get [11:36] wrong. um they they they don't know uh how to  measure when the agent should speak, when it [11:43] shouldn't speak. Um so humans negotiate turns um  ba basically on a few hundred milliseconds based [11:49] on tone and the context of the conversation.  Like we just know when it's our turn to talk, [11:54] right? So um a pipeline that's running on just the  transcript alone uh is going to be very slow. So, [12:01] we built a turn detector that listens uh while  the conversation is going on and it listens to [12:07] the actual audio. It doesn't just look at  the transcript. It does use the transcript, [12:10] but it listens to the audio. So, it actually hears  the the rhythm, the pitch, the intonation. Uh and [12:17] and we published the numbers on this. So, at um  the same latency budget, the best alternative cuts [12:24] callers off 9.9% of the time still. uh ours  is 4.5%. And so uh this eval the the harness [12:33] is is open source. You can actually test it on  your own uh use case to see if it cuts your uh [12:39] interruptions down as well. Um so turn taking is  a very important thing to measure. That is again [12:45] not model specific, it's system specific. So let's  go a little bit a little bit deeper here. So we're [12:52] we're now going to talk about that budget. So  the gap between the user's last syllable and the [12:57] agent's first audio replying back to the user. So  end of turd detection fires from the audio before [13:04] the transcript ever lands. Uh the language model  is already running on a provisional transcript. [13:11] Um and if the final transcript matches then we've  already bought some time back, right? So the model [13:17] humans do this weird thing where we say the same  thing three times in a row in different ways. the [13:22] LLM, it already knows what what I'm about to say.  It says, "Just stop talking. I already know. I'm [13:28] already producing a reply, right?" And I just did  the same thing just now, if you didn't notice. So, [13:33] LLMs are smart. They start producing that the  the reply quickly. As long as we don't change [13:40] our mind mid-sentence or something like that,  it's already ahead of us, right? And then the TTS [13:45] um uh on that first complete sentence starts  sending uh starts sending audio back, right? And [13:52] it's continuously streaming that audio as the LLM  continues uh continues to finish as well. So we're [13:58] not waiting for all of the last tokens to come in  before we move on. We move on as soon as possible [14:03] at each stage. And so again, time to first audible  bite is what we're looking for, not not the first [14:10] bite. Uh so every model that we choose it shows  up in this. They all work together. Again, [14:14] this is a system. And so this window, it has a  fixed size. This is that budget, right? So if we [14:21] stay in roughly 1 and a half seconds max, then  the conversation still holds. It's starting to [14:27] feel a little awkward at 1 and a half seconds.  Um but if we can get down to 600 milliseconds, [14:33] that's when it really starts to sound humanlike,  right? So the question uh at the beginning uh [14:39] uh how fast uh can I go like that that wasn't  the right answer like it's how humanlike can my [14:46] agent be within this budget. So again latency is  a budget but humanlike that's the goal. So if we [14:53] add all this up that question at the beginning  um which uh which model should I pick that was [14:58] never actually answerable. And so because it's  not answerable we built something that we wish [15:04] existed. We built the uh I don't know exactly  what to call it yet. We haven't really named it, [15:10] but like the infinite benchmark. And I'll explain  why why I say infinite in just a minute. So, [15:16] we take the the a complete working agent, that  receptionist that you just heard. We take that [15:21] agent and um it has again real backend uh real  functions. It's it's actually doing things and [15:29] that is actually open source and you can look  at the repo as well. But we take that agent and [15:33] we run hundreds of simulated users against that  agent. The the simulations interrupt, they mumble, [15:41] they change their mind mid-sentence, they do  things that humans would actually do. And we [15:45] try to break the agent. And so then we we don't  score it on vibes. We we score it on outcomes. [15:52] And so uh we define these the way that a human  would act would have actually done the job. Did [15:58] the right booking get placed? uh did it end up in  the in the database correctly, right? Did a caller [16:04] get cut off? Were there hallucinations? Did it try  to call tools that it didn't that it didn't have? [16:08] You know, we're looking for humanlike behaviors.  And so that's the bar that we measure against. And [16:14] then completion alone is not a check mark because  we know that agents can can um you know try to get [16:20] around completion. They can try to fake fake their  way, right? So we penalize that as well. So I if [16:27] um if a if it asks questions that didn't need to  be asked, that's pen penalized. If it takes 10 [16:32] steps when it should have taken just two steps,  that's penalized as well. So then at that point, [16:38] we've we've measured it. We've got our benchmark  now or we're we're starting to get a picture of [16:43] our benchmark. We take that same stack, the  same uh same scenarios and but we change out [16:49] the models. We put different models in  there and then every model has the same [16:54] uh same input bite for bite and so we just  keep swapping out models and that is building [16:59] our benchmark for this particular agent. And so  this is why infinite it's the infinite benchmark [17:07] because there are as many benchmarks as there are  agents. So uh the benchmark is actually yours. [17:12] You can run your agent against this and build the  benchmark for your use case. And so that's how we [17:19] catch surprises. The whole entire the entire  thing is uh is open source. The methodology, [17:24] the email sets sets and even that particular agent  is open source. You can see how it's done. Um but [17:30] you can point the same thing uh at your agent  again and build that benchmark for for yourself. [17:38] Um and so here's one uh quick important fact about  LiveKit is that we don't make ST LLM TTS models. [17:46] Um, but all of the models that you want to use,  they all plug into our ecosystem. So, we don't [17:51] have a horse in the race, but we we watch all of  the horses run a lot. And so, we can confidently [17:56] pick winners. And so, what you're looking at here  is Gemma 4 31B running on LiveKit inference. Uh, [18:03] and this isn't just an offtheshelf Gemma 4  31B. This is one that is tuned for latency, [18:09] tuned for being as human-like and quick as  possible. And it's also smart. Um and so it's [18:15] also two times faster than the next model which is  actually itself the same model just running on a [18:20] different different inference provider. So this is  available today in lifkit uh inference and again [18:26] the goal here is to have the most human-like  agent possible while sticking to that latency [18:32] budget. So, if you remember one thing from this uh  talk, make it this one, a voice agent is not just [18:38] a chatbot with a speaker attached to it, right?  It's a human-like infrastructure. It's a real-time [18:43] system that senses and responds the way that  a person would. And so, when we start thinking [18:49] about that way, all of our decisions start to  fall in line. So, here's a checklist. Make sure [18:54] that your lat you you you set your latency budget  first. Measure perceived latency. spend whatever [19:00] budget is left on uh quality, on realism, and  don't worry about speed as long as you're within [19:05] that budget. Anything over a second, any calls  over a second should go asynchronous so that the [19:11] agent can continue the conversation and doesn't  get blocked. And then test the conversation, [19:17] not the components, right? And then keep testing  in production. So, here's the benchmark. You can [19:23] scan the QR code to see our benchmark uh on our  agent and then you can again run that on yours. [19:28] Um, and then sometime soon hopefully you'll call a  voice agent and you'll speak over it and it won't [19:36] uh inter it won't interrupt you. It'll just handle  it like a human would. It'll roll with it. Um, [19:40] but here's the thing that I need you to  do. I need you to go build that. Thank you.