Modulate Deploys Audio AI to Stop Voice Bots Fueling Customer Churn

Voice Agent Supervision: Finding the Hidden Cost of Conversational Blind Spots With Modulate — Ai4

Ai4

Because spoken words convey as little as 3% of conversational meaning, text-reliant voice bots mistake customer sarcasm for satisfaction, driving hidden churn and exposing help desks to deepfake fraud.

AI-assisted summary of Voice Agent Supervision: Finding the Hidden Cost of Conversational Blind Spots With Modulate — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 19, 2026, this presentation features Carter Huffman, CTO and co-founder of Modulate. The session addresses how enterprises deploying voice AI silently lose return on investment to conversational blind spots like missed tone and emphasis, presenting an audio-native framework to supervise voice agents and catch live interaction friction.

## TL;DR Carter Huffman argues that conventional voice AI pipelines discard 80% to 97% of conversational meaning at the transcription boundary, causing agents to misread sarcasm, escalate customer frustration, and fall into repetitive dialogue loops. Drawing from his background designing resource-efficient onboard systems at NASA JPL, Huffman introduces Modulate's 'Velma' architecture—an ensemble of lightweight acoustic models that monitors tone, cadence, and emotion in real time at roughly 1/100th the operational cost of monolithic models. Beyond improving customer support workflows and reducing churn, this audio-native observability defends against deepfake-driven social engineering attacks on enterprise help desks and powers broad trust and safety moderation across gaming and social platforms.

## Atmosphere & style The talk carries the crisp, pragmatic cadence of an experienced systems engineer translating complex acoustic modeling into direct business and security outcomes. Huffman pairs analytical technical explanations with relatable, slightly comedic depictions of broken bot dialogues and stark, high-stakes scenarios of hospital cyberattacks. The resulting atmosphere is clear, grounded, and persuasive, conveying both the operational urgency and practical mechanics of supervising voice AI beyond literal transcripts.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=E2thTe__EKo&t=4s) Over 80% of enterprises are adopting voice AI with rapid payback periods, but frequent model upgrades introduce subtle conversational regressions that standard post-call surveys fail to detect. - [00:04](https://www.youtube.com/watch?v=E2thTe__EKo&t=4s) Applying NASA JPL design principles of high accuracy under constrained computational budgets enables effective, low-overhead voice agent supervision. - [03:16](https://www.youtube.com/watch?v=E2thTe__EKo&t=196s) Standard Speech-to-Text and LLM pipelines create severe blind spots by interpreting literal words rather than tone, frequently misclassifying sarcastic customer frustration as successful resolution. - [05:10](https://www.youtube.com/watch?v=E2thTe__EKo&t=310s) Linguistic research indicates that literal words convey only 3% to 20% of meaning, while 80% to 97% of conversational intent and emotion is carried through acoustic nuance like pitch, pacing, and cadence. - [05:10](https://www.youtube.com/watch?v=E2thTe__EKo&t=310s) Stripping acoustic context at the transcript boundary is a major driver of voice bot failure, with over 59% of consumers reporting anger and frustration when dealing with automated voice systems. - [06:39](https://www.youtube.com/watch?v=E2thTe__EKo&t=399s) Modulate's Velma architecture uses an ensemble of lightweight, dedicated acoustic models to achieve over 99% emotional accuracy at roughly 1/100th the operational cost of generalized multimodal foundation models. - [06:39](https://www.youtube.com/watch?v=E2thTe__EKo&t=399s) Audio-native observability feeds dual operational workflows: triggering real-time mid-call interventions like dynamic re-prompting and escalation, while powering post-call aggregate dashboards. - [10:46](https://www.youtube.com/watch?v=E2thTe__EKo&t=646s) Effective emotional supervision maps conversational trajectories relative to each speaker's personal baseline rather than using static, universal sentiment classifications. - [12:08](https://www.youtube.com/watch?v=E2thTe__EKo&t=728s) Real-time acoustic analysis detects high-pressure social engineering and deepfake voice attacks against IT help desks, preventing single-point breaches that lead to ransomware and major regulatory fines. - [13:37](https://www.youtube.com/watch?v=E2thTe__EKo&t=817s) Audio comprehension eliminates repetitive politeness loops by enabling voice agents to recognize wrap-up cadence and conclusive tones when callers attempt to end an interaction. - [13:37](https://www.youtube.com/watch?v=E2thTe__EKo&t=817s) The same audio-native understanding used in enterprise voice agents scales to platform trust and safety, identifying toxic harassment, extremist radicalization, and child grooming across social and gaming ecosystems.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=E2thTe__EKo&t=4s) NASA JPL Roots and the Rapid Velocity of Voice AI

Carter Huffman, CTO and co-founder of Modulate, traces his core engineering philosophy back to his work at the NASA Jet Propulsion Laboratory over a decade ago. At JPL, he built extremely resource-efficient yet highly accurate models designed to execute live onboard spacecraft during active operational missions. That foundational requirement—delivering uncompromising precision under tight computational and resource constraints—remains the guiding design principle behind Modulate's suite of audio-native conversational understanding ensembles. Today, voice AI is evolving at an unprecedented pace, with leading AI providers releasing new models and version updates on a weekly cadence. These upgrades promise continuous improvements across dozens of distinct axes, including lower latency, increased accuracy, richer emotional range, better key-term recognition, and improved cost-efficiency.

While adopting cutting-edge models delivers massive commercial upside, it introduces hidden regressions. More than 80% of enterprises are integrating voice AI into their platforms, driven by compelling business cases where return on investment and payback periods are measured in months rather than years. However, upgrading from one model checkpoint to the next makes it difficult to evaluate what subtle conversational capabilities are being lost. Standard post-interaction metrics like CSAT scores and end-of-call surveys fail to capture conversational friction as it happens. Detecting when a voice AI update creates a negative user experience requires continuous, dynamic tracking of how callers respond to agents across every live interaction.

- **Enterprise Adoption Rate**: Over 80% of enterprises are actively deploying voice AI solutions across their platforms. - **Payback Velocity**: Capital payback periods for voice AI rollouts are typically realized within months rather than years. - **The Upgrade Dilemma**: Multi-dimensional model updates (cost, latency, emotiveness, accuracy) often introduce regressions that traditional text-based evaluations overlook.

---

[03:16](https://www.youtube.com/watch?v=E2thTe__EKo&t=196s) The Sarcasm Blind Spot: Transcript vs. Audio-Native Reality

When organizations supervise voice AI systems, they routinely rely on pipelines that transcribe the audio and run downstream text-based large language models over the resulting script. This approach creates severe blind spots, particularly when customer tone diverges from literal syntax. In a typical customer service scenario, a user calls in frustrated because their billing statement increased unexpectedly. When the voice agent cheerfully responds, "I am happy to help. Let me explain to you what's going on," the customer replies with heavy sarcasm: "Oh, perfect. Now I understand it. Incredible."

```mermaid flowchart TD accTitle: Transcript Evaluation versus Audio Native Reality subgraph Traditional["Traditional Text Pipeline"] T1["Customer Audio"] --> T2["Raw Text: 'Oh, perfect. Now I understand it.'"] T2 --> T3["Text LLM Classifier"] T3 --> T4["Classified: Positive / Resolved ✅"] end subgraph AudioNative["Audio-Native Understanding"] A1["Customer Audio"] --> A2["Acoustic Analysis: Anger + Sarcastic Inflection"] A2 --> A3["Velma Ensemble"] A3 --> A4["Classified: Severe Friction / Churn Risk ❌"] end ```

Because the voice agent and text-based evaluation tools only parse literal words, the agent responds enthusiastically: "Oh, I'm happy I was able to help. Great!" A standard text-based sentiment monitor records this exchange as a completely resolved, positive interaction. In reality, the customer wanted to register an angry complaint rather than listen to an explanation, and left the call alienated. Without native acoustic analysis, organizations misinterpret severe customer friction as successful resolution, quietly accumulating hidden costs and customer churn.

---

[05:10](https://www.youtube.com/watch?v=E2thTe__EKo&t=310s) The Architectural Bottleneck: Words Account for Only 3% to 20% of Meaning

The failure of voice agents to recognize nuance is an architectural limitation rather than a temporary model quality issue. In the standard voice AI pipeline, incoming caller audio is converted to a text transcript, fed into an LLM for reasoning, and finally synthesized back into audio via text-to-speech. However, linguistic research demonstrates that the literal words spoken in human conversation convey only 3% to 20% of the total meaning. The remaining 80% to 97% of conversational intent, mood, and context is carried entirely through acoustic channels: pitch, emphasis, cadence, pacing, and tone.

```mermaid flowchart LR accTitle: Standard Voice AI Pipeline vs Audio Information Loss subgraph TraditionalStack["Conventional Voice Pipeline"] InAudio["Caller Audio"] --> ASR["Speech-to-Text"] ASR -->|"Words Only (3% - 20% of Meaning)"| LLM["LLM Core"] LLM --> TTS["Text-to-Speech"] TTS --> OutAudio["Agent Voice"] end subgraph LostData["Lost Acoustic Data (80% - 97%)"] L1["Emotional Valence"] L2["Sarcasm & Emphasis"] L3["Cadence & Latency Frustration"] end InAudio -.-> LostData ```

By discarding acoustic nuance at the speech-to-text boundary, agents operate without the necessary context to respond appropriately. Upgrading to larger text-based foundation models cannot resolve this bottleneck because the fundamental input remains stripped of context. Consequently, user frustration with automated systems continues to mount. By mid-2026, over 59% of consumers reported experiencing anger and frustration when interacting with AI voice agents, a metric that continues to climb despite overall advancements in model fluency.

- **Conversational Information Density**: Literal spoken words represent only 3% to 20% of total communicative meaning; up to 97% is transmitted through acoustic signals. - **Consumer Sentiment**: More than 59% of consumers in 2026 report direct frustration and anger when dealing with AI voice agents.

---

[06:39](https://www.youtube.com/watch?v=E2thTe__EKo&t=399s) Audio-Native Observability and the Velma Architecture

Rather than replacing existing conversational infrastructure, organizations can address this blind spot by deploying an audio-native observability and analytics layer alongside their core stack. Modulate implements this through "Velma," an ensemble listening architecture that runs coordinated, lightweight acoustic models in parallel. Instead of relying on a single monolithic foundation model, Velma extracts dozens of distinct audio features simultaneously, including vocal emotion, conversational cadence, emphasis, assistive technology usage, and synthetic voice or deepfake artifacts.

```mermaid flowchart TD accTitle: Audio-Native Observability and Intervention Architecture subgraph Telephony["Telephony & Contact Center"] VoIP["VoIP / CCaaS Platform"] AgentStream["Agent Audio Stream"] UserStream["Caller Audio Stream"] VoIP --> AgentStream VoIP --> UserStream end subgraph Analysis["Modulate Velma Layer"] AgentStream --> Velma["Velma Ensemble Listening Engine"] UserStream --> Velma Velma --> Signals["Extracted Behavioral Signals (Tone, Latency, Emotion)"] end subgraph Action["Operational Response"] Signals --> RealTime["Real-Time Intervention (Re-prompting / Escalation)"] Signals --> PostCall["Post-Call Analytics & Statistical Dashboards"] end ```

Velma ingests dual audio streams—capturing both caller and agent channels directly from VoIP or CCaaS platforms—to map how conversational dynamics evolve turn-by-turn. Specialized acoustic ensembles achieve over 99% classification accuracy on vocal emotions and delivery styles without the hallucinations common in large generative systems. Operating lightweight, dedicated models also slashes computational overhead, delivering full conversational supervision at roughly 1/100th of the cost of running generalized foundation models for post-call audio analysis. Organizations can feed these extracted signals back into service desks to trigger live interventions (such as dynamic re-prompting or human escalation) or aggregate them into post-call dashboards to identify systemic macro issues.

- **Acoustic Accuracy**: Modulate's targeted audio-native models achieve over 99% accuracy in identifying specific emotional states and spoken nuances. - **Inference Efficiency**: The ensemble architecture provides comprehensive acoustic analysis at 1/100th (or less) of the operational cost of massive multimodal foundation models. - **Dual Integration Paths**: Real-time signal output enables mid-call intervention and automated human handoffs, while post-call processing populates aggregate observability dashboards.

---

[10:46](https://www.youtube.com/watch?v=E2thTe__EKo&t=646s) Rich Behavioral Signals and Longitudinal Emotion Tracking

Comprehensive audio supervision extends beyond binary sentiment detection to map complex emotional trajectories throughout a call. Because vocal baselines vary widely across individuals, effective supervision tracks relative emotional shifts against each speaker's personal baseline. Carter illustrates this dynamic using his own communication style: when he places a call, he typically speaks with high natural energy and enthusiasm; if his delivery shifts to flat or neutral phrasing, it indicates growing dissatisfaction, even though a neutral tone might appear completely acceptable for a different speaker.

Beyond basic frustration or enthusiasm tracking, audio-native analysis identifies sophisticated interpersonal behaviors and conversational strategies. This includes detecting when a caller is actively attempting to manipulate, emotionally coerce, or socially engineer an agent (human or AI) into granting unauthorized exceptions or taking prohibited actions. Capturing these behavioral nuances equips organizations with the conversational visibility needed to protect agent workflows and optimize operational ROI.

[12:08](https://www.youtube.com/watch?v=E2thTe__EKo&t=728s) Audio-Native Observability and Help Desk Fraud Detection

Complex adversarial behaviors—such as prompt injection attacks against voice agents or subtle social engineering tactics—are nearly impossible to uncover through raw text transcripts alone, but they become clearly visible when analyzed through an audio-native observability layer. Modulate deploys its speech-understanding model, Velma, across high-stakes environments like hospital IT help desks to monitor and dissect incoming audio streams in real time. In healthcare settings, malicious callers increasingly deploy voice deepfakes and artificial urgency to bypass security protocols, claiming things like, "Oh, you know me. You recognize my voice," and insisting, "A patient's going to die if you don't give me access to my computer system, if you don't reset my password."

Because help desk personnel are inherently driven by a desire to assist people in emergencies, they are exceptionally susceptible to these high-pressure appeals. Even if support staff follow standard protocol 99% of the time by requesting photographic identification, a 1% success rate is all an attacker needs. A single fraudulent password reset or multi-factor authentication (MFA) device modification can trigger an catastrophic chain reaction: unauthorized system penetration, enterprise ransomware deployment, multi-million dollar regulatory fines, stolen records dumped on the dark web, and tangible threats to patient safety. By evaluating the acoustic nuances and authenticity of the voice in real time, audio-native monitoring catches these attacks before access is granted.

```mermaid flowchart TD accTitle: Help Desk Deepfake Attack Escalation vs Audio Supervision subgraph Attack["Social Engineering Attack"] A["Deepfake Caller Imposes False Urgency"] --> B["Help Desk Agent Appeals"] end subgraph Breakdown["Vulnerability & Impact (1% Breach)"] B --> C["MFA Swap / Password Reset"] C --> D["Ransomware & Dark Web Breach"] D --> E["$6M Regulatory Fine & Patient Risk"] end subgraph Defense["Audio-Native Observability"] A -.-> F["Velma Audio Model"] F -.-> G["Real-Time Deepfake & Threat Flagging"] end ```

- **Help Desk Vulnerability:** Attackers exploit human empathy and medical urgency to bypass standard ID verification barriers. - **Breach Consequences:** A single compromise can result in full ransomware deployment, dark web data leakage, and upwards of a $6 million fine for the affected hospital system.

[13:37](https://www.youtube.com/watch?v=E2thTe__EKo&t=817s) Broken Agent Loops and Trust & Safety Across Voice Ecosystems

Audio-native supervision is equally critical for preventing customer-facing AI agents from "going off the rails" during routine interactions. A common failure mode in voice deployments is the politeness loop: when a user attempts to conclude a call with, "No, I'm good. Appreciate it," an oblivious agent might endlessly repeat, "Great. Is there any more information I can give to you?" While blunt callers will simply hang up, polite users get trapped in an awkward, circular dialogue that severely degrades customer satisfaction. Because the AI model lacks acoustic comprehension, it fails to recognize the conclusive tone in the user's voice and remains stuck in its script.

The same foundational audio models used to supervise automated customer support are also deployed across broad trust and safety applications. This includes monitoring whether human agents are being harassed on calls, as well as moderating real-time social platforms to identify toxic harassment, the spread of extremist ideology, and predatory child grooming. Carter Huffman concludes by emphasizing that the future of voice AI depends on audio-native conversational understanding that analyzes "not just what was said, but how it was said," inviting attendees to continue the discussion at Modulate's conference booth.

- **Polite Loop Friction:** Inability to hear conversational cadence and sign-off tone causes voice bots to repeatedly prompt callers who have already finished their business. - **Cross-Domain Trust and Safety:** Velma's audio-native voice understanding powers moderation in gaming and social platforms to flag hate speech, extremist radicalization, and child grooming. - **Conference Location:** Modulate hosted attendees for follow-up discussions and demonstrations at Booth 940.

## Notable quotes > [03:16](https://www.youtube.com/watch?v=E2thTe__EKo&t=196s) I am happy to help. Let me explain to you what's going on

> [03:16](https://www.youtube.com/watch?v=E2thTe__EKo&t=196s) Oh, perfect. Now I understand it. Incredible.

> [03:16](https://www.youtube.com/watch?v=E2thTe__EKo&t=196s) Oh, I'm happy I was able to help. Great!

> [12:08](https://www.youtube.com/watch?v=E2thTe__EKo&t=728s) Oh, you know me. You recognize my voice

> [12:08](https://www.youtube.com/watch?v=E2thTe__EKo&t=728s) A patient's going to die if you don't give me access to my computer system, if you don't reset my password.

> [13:37](https://www.youtube.com/watch?v=E2thTe__EKo&t=817s) No, I'm good. Appreciate it

> [13:37](https://www.youtube.com/watch?v=E2thTe__EKo&t=817s) Great. Is there any more information I can give to you?

> [13:37](https://www.youtube.com/watch?v=E2thTe__EKo&t=817s) not just what was said, but how it was said — Carter Huffman

## Mentioned Carter Huffman, Modulate, NASA Jet Propulsion Laboratory, Velma, Ai4

## Video description Carter Huffman - CTO, Modulate

What if the biggest ROI drain on your voice agent deployment isn't a bug in the system — it's a tone the agent never learned to hear?

Voice agents are scaling fast, but most enterprises have no way to measure where they're quietly losing money to conversational blind spots: misread emotion, missed emphasis, and frustration signals buried in audio that never surface in a transcript until they show up as escalations and churn. This talk will explain how to supervise voice agents through conversational understanding, treating tone and emphasis as first-class signals so teams can catch and correct friction mid-conversation. It will also walk through how to translate agent-level errors into the financial metrics that make or break the ROI case for voice AI. If you're accountable for justifying the cost of voice agents at your company, this talk will give you a concrete framework for finding and fixing the value you're leaving on the table.

[Source](https://www.youtube.com/watch?v=E2thTe__EKo)

中文简报

Modulate部署音频AI以遏制语音机器人加剧客户流失 由于口头表达仅传达少至3%的对话含义，依赖文本的语音机器人易将客户的反讽误判为满意，从而引发隐性流失并使服务台面临deepfake欺诈风险。 Carter Huffman指出，传统语音AI管线在转写边界处丢弃了80%至97%的对话含义，导致智能体误读反讽、加剧客户沮丧情绪并陷入重复对话循环。借鉴在NASA JPL设计高效利用资源的机载系统的经验，Huffman推出了Modulate的“Velma”架构——该轻量级声学模型集成能以单体模型约1/100的运营成本实时监测语气、节奏和情绪。除了改善客户支持工作流程和减少流失外，这种原生音频可观测性还能抵御针对企业服务台的deepfake社会工程攻击，并为游戏及社交平台提供广泛的信任与安全审核支持。 超过80%的企业正在加速部署语音AI，且投资回收期通常缩短至数月，但频繁的模型版本迭代往往引入细微的对话退化，传统事后调查难以捕捉此类摩擦。 (0:04) 传统语音管道在语音转文字（STT）过程中会丢失关键声学信息；语言学研究表明文字仅占对话含义的3%至20%，而高达97%的真实情绪与意图藏于音调与语速中，导致模型常将讽刺性不满误判为圆满解决。 (5:10) Modulate 推出基于轻量级声学模型集成的 Velma 架构，以大模型约百分之一的推理成本实现超过99%的情绪识别准确率，兼顾实时通话干预与通话后全局质检。 (6:39) 音频原生可观测性不仅能依据说话人的个人基线追踪情绪轨迹，还能实时识破IT服务台面临的语音深度伪造与社会工程学攻击，并避免语音助手陷入重复确认的死循环。 (12:08)

Reading summary

Carter Huffman, co-founder and CTO of Modulate, argues that conventional voice AI pipelines create critical blind spots by discarding acoustic signals, causing enterprises to misinterpret customer dissatisfaction and overlook severe social engineering threats. He contends that deploying lightweight, audio-native observability models alongside existing systems captures true conversational intent and mitigates fraud at a fraction of the cost of monolithic foundation models. The Acoustic Bottleneck in Text Pipelines (3:16) Standard enterprise voice architectures typically transcribe caller audio into text before feeding the transcript to large language models for reasoning. According to Modulate, this pipeline discards critical communication context: literal spoken words convey only 3% to 20% of total meaning, while the remaining 80% to 97% is transmitted through acoustic channels such as pitch, emphasis, pacing, and tone. Traditional post-call metrics like CSAT surveys and text-based evaluators fail to capture live friction, leaving organizations unaware when multi-dimensional model updates cause subtle conversational regressions. This structural blind spot frequently misinterprets negative interactions. For example, when an upset customer responds to an explanation with heavy sarcasm—such as "Oh, perfect. Now I understand it. Incredible"—text pipelines classify the literal phrase as a positive resolution. In reality, the customer leaves alienated, quietly driving churn. Despite rapid weekly model iterations and enterprise adoption exceeding 80%, consumer friction persists: by mid-2026, over 59% of consumers reported experiencing direct anger and frustration when interacting with AI voice agents. Lightweight Parallel Ensembles for Voice Observability (6:39) To bridge this gap without rebuilding core infrastructure, Modulate proposes adding an audio-native observability layer. Rooted in Huffman’s experience building resource-efficient systems at NASA’s Jet Propulsion Laboratory, Modulate developed "Velma," an ensemble listening engine that runs specialized acoustic models in parallel. Rather than relying on a single generative foundation model, the system analyzes dual audio streams directly from VoIP or contact center platforms to track emotional shifts, latency, and deepfake artifacts. Modulate claims these targeted acoustic ensembles achieve over 99% classification accuracy for vocal emotions and delivery styles without the hallucinations common in generative systems. Operating lightweight, dedicated models also reduces computational overhead to roughly 1/100th the cost of using generalized foundation models for post-call audio analysis. Organizations can direct these real-time signals into live service desks to prompt dynamic re-prompting and human escalation, or aggregate them into post-call dashboards to monitor macro trends. Mitigating Deepfake Fraud and Help Desk Manipulation (12:08) Audio-native supervision also addresses adversarial threats like prompt injection and social engineering, which are difficult to spot in raw text. In high-stakes environments like hospital IT help desks, attackers exploit medical urgency and deploy synthetic voice deepfakes to pressure agents into unauthorized password resets or multi-factor authentication (MFA) device reconfigurations. Because support staff are naturally inclined to assist in medical emergencies—such as claims that a patient will die without immediate system access—attackers require only a 1% success rate to bypass standard identification protocols. A single breach can lead to enterprise ransomware deployment, patient safety risks, dark web data exposure, and regulatory fines exceeding $6 million. Modulate asserts that real-time acoustic analysis detects artificial urgency, synthetic voice markers, and emotional manipulation before access is granted. Preventing Dialogue Loops and Moderating Harassment (13:37) Beyond security, acoustic comprehension prevents customer-facing voice bots from failing during routine interactions, such as getting trapped in "politeness loops." When a caller attempts to end an exchange with "No, I'm good. Appreciate it," an agent lacking acoustic awareness may miss the finality in the user's tone and repeatedly ask how else it can assist. While blunt callers hang up, polite users get caught in frustrating, circular dialogues. Effective supervision tracks shifts relative to an individual's personal vocal baseline rather than static thresholds, catching subtle declines into annoyance. Modulate deploys these same audio-native models across broader trust and safety applications. In contact centers, the system monitors whether human representatives are being subjected to caller harassment. Across gaming and social platforms, the technology analyzes live audio streams to identify toxic abuse, extremist radicalization, and predatory child grooming. Huffman concludes that the future of voice AI depends on conversational models capable of analyzing not just what was spoken, but how it was spoken. However, the source relies entirely on Modulate's internal performance metrics and benchmarks, leaving unaddressed how easily these acoustic layers integrate across legacy, low-bandwidth telephony networks or heterogeneous multi-vendor environments.

精炼阅读摘要

Modulate联合创始人兼首席技术官Carter Huffman指出，当前语音AI普遍依赖将音频转写为文本再交由大语言模型处理的架构，这种方式丢失了人类交流中80%到97%的声音信息；只有引入原生音频监控层，企业才能消除由讽刺语气、循环死锁和深度伪造欺诈引发的隐性商业成本与安全漏洞。 转写架构的语意缺失与升级陷阱 (0:04) 超过80%的企业正在业务平台中集成语音AI，投资回报与回收期通常在数月内即可实现。然而，主流系统采用“语音转文本（ASR）—大语言模型（LLM）推理—文本转语音（TTS）”的传统流程，形成了严重的技术瓶颈。语言学研究表明，口语对话中字面词汇仅传达3%至20%的含义，其余高达80%至97%的情感、语调、重音与节奏信息完全通过声学信号传递。转写过程剥离了这些关键线索，导致即使升级至更大的文本模型也无法弥补语境缺失。调查显示，截至2026年年中，已有超过59%的消费者在与语音AI交互时感到沮丧与愤怒。 技术厂商每周高频更新模型版本，虽然在延迟、准确率和情绪广度上有所改善，但也带来了难以察觉的性能回退。由于传统的转写分类器只分析字面文字，当遭遇讽刺表达时便会出现致命盲区。例如，客户在账单异常上升后愤怒并讽刺地说“太棒了，我现在明白了”，文本分类器会将其判定为“满意解决”，智能体随后做出热情的误判回应。这种将严重冲突记录为积极交互的现象，使得传统CSAT评分和事后调查难以捕捉真实的体验摩擦，导致企业在无形中流失客户。 Velma声学集成架构与行为追踪 (6:39) 为了在不推倒现有技术栈的前提下弥补这一缺陷，Modulate提出了基于原生音频的可观测性分析层Velma。Huffman将其实施哲学归结为早年在NASA喷气推进实验室（JPL）为航天任务开发轻量高精度模型的经历。Velma采用轻量化声学模型组成的集成系统，而非单一庞大的多模态基础模型，能够以大约通用大模型百分之一的运行成本，并行提取包括情绪、语调、重音、语速、辅助技术使用痕迹及深度伪造特征在内的数十种声学属性，在特定情绪和表达风格分类上的准确率达到99%以上。 在具体监控机制上，系统直接从VoIP或CCaaS平台接入客户与坐席的双向音频流，避免了生成式系统的幻觉问题。该方案不仅支持事后数据看板分析，还能进行毫秒级实时信号输出，用于动态提示或人工坐席升级。在行为理解方面，Velma建立了个体声学基线并进行纵向情绪追踪。例如，习惯高能量发音的个体如果转为平淡语调，即代表出现隐性不满。此外，系统还能识别来电者是否在施加情绪操纵或社会工程学手段，从而更全面地评估对话质量。 防御深度伪造与防范流程死锁 (12:08) 在医院IT服务台等高风险场景中，攻击者正利用语音深度伪造（Deepfake）和虚假紧迫感实施社会工程学渗透，例如声称“病人快死了，必须立刻重置密码”，以此利用服务人员的救助同情心绕过身份核验。Huffman警告，哪怕客服人员99%的情况下遵循核验规范，仅凭1%的疏漏即可导致多因子认证（MFA）或密码被篡改。这可能引发全系统勒索软件感染、暗网数据泄露，以及高达600万美元以上的监管罚款。原生音频层能在通话过程中实时甄别声学伪造特征与虚假情绪，在权限被越权授予前拦截威胁。 在常规客户交互中，原生音频监督同样用于打破系统死锁。由于缺乏对对话节奏和收尾语调的感知能力，许多语音智能体容易陷入“礼貌死循环”：当客户表达“不用了，挺好的，谢谢”试图结束对话时，缺乏语调判断的AI会机械地重复询问“太好了，还有什么可以帮您”，导致礼貌的客户被困在尴尬的重复交互中。此外，相同的原生音频理解技术已被拓展至更广泛的信任与安全领域，包括监控人工坐席是否遭受语言骚扰，以及在社交和游戏平台中识别极端仇恨言论与针对儿童的诱拐行为。 Modulate强调，语音AI的未来取决于从单纯分析“说了什么”转向理解“怎么说的”原生声学能力。企业若仅依赖文本转写评估对话，将持续承受客户流失与安全攻击等隐性代价；如何在保持低推理成本的同时将声学信号无缝融合至现有客服与业务决策体系，是语音智能体规模化落地的核心课题。

Detailed notes — zh

## Background 该内容由 YouTube 频道 Ai4 于 2026 年 8 月 19 日发布，主讲人为 Modulate 首席技术官 Carter Huffman。页面简介指出，企业在快速扩展语音 AI 代理时往往因无法识别语调、重音和挫败感等对话盲区而导致投资回报率（ROI）隐性受损，本场演讲旨在阐述如何通过原生音频对话理解对语音代理进行实时监督，并提供量化与挽回业务价值的具体分析框架。 ## TL;DR Carter Huffman 指出，文字仅占人类语音交流实际含义的 3% 至 20%，传统的“转录文本+大模型”质检链路会漏掉反讽、语气与情绪等关键副语言信号，造成严重的用户体验断层与客户流失。Modulate 推出了基于高效专用小模型集成的原生音频可观测层 Velma，能以仅为基础大模型 1/100 的运行成本实时捕捉情绪演变、语调、重音及伪造音频特征。该架构不仅支持在通话中进行实时干预（如动态注入情绪提示词或升级转人工），还能用于识别针对医疗等关键领域的 Deepfake 欺诈攻击、阻断智能体死循环并保障企业信任与安全。 ## Atmosphere & style 整场演讲洋溢着严谨、务实的硬核工程技术风格，Huffman 结合自身在 NASA 喷气推进实验室的能效算法背景，条理清晰地剖析了现有语音 AI 架构的结构性缺陷。演讲通过讽刺语境误判、医疗服务台勒索攻击等具体生动的案例推进，叙事节奏沉稳紧凑，既揭示了盲目升级模型背后的隐形危机，又展现出用轻量精准架构破解行业痛点的技术自信与启发性。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=E2thTe__EKo&t=4s) Modulate 首席技术官 Carter Huffman 将在 NASA JPL 积累的高效、精准算法工程哲学引入语音 AI，打造低资源消耗的原生音频对话理解模型。 - [01:53](https://www.youtube.com/watch?v=E2thTe__EKo&t=113s) 超过 80% 的企业已部署语音 AI 并实现数月内的快速回本，但频繁升级模型容易引入企业无法察觉的细微退化。 - [01:53](https://www.youtube.com/watch?v=E2thTe__EKo&t=113s) 传统的 CSAT 评分或通话后问卷等宏观指标，无法捕捉通话进行中细微的体验滑坡与交互摩擦。 - [03:21](https://www.youtube.com/watch?v=E2thTe__EKo&t=201s) 传统基于文本转写（STT）和大语言模型（LLM）的质检存在致命盲区，无法识别讽刺语气，常将引发反感的糟糕互动误判为成功通话。 - [05:10](https://www.youtube.com/watch?v=E2thTe__EKo&t=310s) 人类交流中字面文本仅承载 3% 至 20% 的信息量，现有语音 AI 架构丢失了 80% 到 90% 的语气、重音与副语言上下文，导致用户挫败感持续攀升。 - [06:38](https://www.youtube.com/watch?v=E2thTe__EKo&t=398s) 在现有技术栈中引入原生音频可观测层（Audio-Native Observability Layer），能实时追踪数十种细粒度声学信号与行为特征，且专用小型模型集成准确率超过 99%。 - [08:46](https://www.youtube.com/watch?v=E2thTe__EKo&t=526s) Modulate 的 Velma 引擎通过双向音频流实时分析，支持在通话中进行 Prompt 动态注入或人工转接干预，运行成本仅为大型基础模型的百分之一。 - [10:46](https://www.youtube.com/watch?v=E2thTe__EKo&t=646s) 系统通过构建多维情绪图谱感知用户基线偏好偏差，并能精准识别来电者针对 AI 坐席的情感操控或胁迫行为。 - [12:08](https://www.youtube.com/watch?v=E2thTe__EKo&t=728s) 在医疗 IT 服务台等高危场景中，原生音频模型可实时拦截利用 Deepfake 与心理施压发起的社工攻击，防止勒索软件入侵与巨额罚款。 - [13:37](https://www.youtube.com/watch?v=E2thTe__EKo&t=817s) 原生音频理解还能避免 AI 智能体陷入无休止的客套死循环，并在信任与安全（Trust & Safety）领域广泛用于识别网络霸凌与有害言论。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=E2thTe__EKo&t=4s) 语音 AI 的演进速度与效率哲学 Modulate 首席技术官兼联合创始人 Carter Huffman 开门见山地介绍了自己的技术背景与公司的核心理念。在创立专注于语音 AI 模型的 Modulate 之前，Huffman 十多年前曾在 NASA 喷气推进实验室（Jet Propulsion Laboratory, JPL）工作，当时他的核心任务是开发高资源利用效率且极其精准的算法模型，以便在航天器执行实际太空任务期间直接进行实时机载运算。这种“在极端资源受限下追求极致精准与高效”的工程哲学，如今完全贯穿在 Modulate 构建的原生音频（Audio-Native）高准确度对话理解模型体系之中。 当前语音 AI 领域的发展速度十分惊人，顶尖 AI 厂商几乎每周都在发布全新模型、版本迭代以及全新功能。这些模型在诸多维度上不断进化——成本更低、准确率更高、专有名词识别更准、声音表现更富有情感、更自然且延迟更低。然而，随着企业频繁更换和升级模型，虽然整体上看似是正向提升，但企业很难察觉在版本迁移过程中究竟丢失了什么细微能力。 ### [01:53](https://www.youtube.com/watch?v=E2thTe__EKo&t=113s) 盲目升级的隐患与传统指标的失效 在业务层面，拥抱前沿语音 AI 技术的收益极为显著。目前超过 80% 的企业已在自身平台中以某种形式集成了语音 AI，其投资回报率（ROI）极高，投资回报周期是以“月”而非“年”来计算。然而，当频繁迭代的模型在某些特定维度出现退化或失效时，企业往往毫无知觉。 企业通常依赖客户满意度评分（CSAT）或通话结束后的问卷调查来评估体验，但 Huffman 直言，这些宏观指标根本无法捕捉通话过程中的微观恶化。真正的解决之道在于深度的“对话理解”（Conversational Understanding）——实时跟踪用户对 AI 代理（Agent）的反应动态，细致监测每一通对话在说些什么、情绪如何随时间演变，以及如何量化这些微妙的体验滑坡。 - **企业普及率**：超过 80% 的企业正在积极部署语音 AI 解决方案。 - **回报周期**：语音 AI 项目的投资回收期已缩短至数月内。 - **监测误区**：仅依赖通话后评分（CSAT/问卷）无法揭示通话进行中的交互摩擦。 ### [03:21](https://www.youtube.com/watch?v=E2thTe__EKo&t=201s) 文本转写的盲区：讽刺语气背后的隐藏成本 目前即便许多企业对语音 AI 进行了监控，也仅仅是将通话录音转换为文本（Transcript），再将文字灌入大语言模型（LLM）进行质检。Huffman 举了一个极具代表性的账单纠纷案例来拆解这种做法的致命漏洞：一位客户因为账单莫名上涨而愤怒致电，AI 代理依照设定礼貌地回应："I am happy to help. Let me explain to you what's going on."（我很乐意帮忙，让我为您解释一下情况）。解释完毕后，怒火未消的客户用极度讽刺的语调说道："Oh, perfect. Now I understand it. Incredible."（噢，太完美了，我现在全懂了，真棒）。 由于大模型普遍具有乐于助人（Happy-to-help）的讨好型倾向，并且只读取字面文本，AI 代理便天真地回应："oh, I'm happy I was able to help. Great!"（很高兴我能帮上忙，太好了！）。从传统的文本分析质检来看，这通电话被判定为一次完美的正向互动；但在现实中，这却是一次极其糟糕、引发客户极大反感的失败体验。这种无法被文本捕捉的体验断层，正是企业在不知不觉中流失客户、耗费成本的隐形黑洞。 ```mermaid flowchart TD accTitle: 对比文本分析与原生音频分析对用户情绪的识别差异 subgraph Traditional["传统文本分析链路（丢失情绪上下文）"] A1["客户讽刺发言: 'Oh, perfect...'"] --> B1["语音转文字 (STT)"] B1 --> C1["大语言模型仅读字面含义"] C1 --> D1["误判为：满意度极高的成功通话"] end subgraph AudioNative["Modulate 原生音频分析链路"] A2["客户真实音频流（愤怒与讽刺声调）"] --> B2["原生音频集成监听模型 (Velma)"] B2 --> C2["提取语调、重音与情绪分布"] C2 --> D2["精准识别：严重不满与讽刺，触发干预"] end ``` ### [05:10](https://www.youtube.com/watch?v=E2thTe__EKo&t=310s) 架构缺陷：文字仅占人类沟通含义的 3% 到 20% Huffman 强调，这种理解偏差并非模型质量高低的问题，而是现有语音 AI 架构的根本性技术缺陷。典型的语音 AI 技术栈通常遵循固定流水线：从原始音频输入，转化为字面文本（STT），输入 LLM 进行推理，最后交由语音合成模块（TTS）朗读。 然而，在人类日常的语音交流中，字面词汇所传递的实际信息量仅占全部含义的 3% 至 20%。这意味着现有的语音代理在处理用户请求时，丢失了 80% 到 90% 的关键上下文数据。无论如何升级换代底层 LLM，都无法解决这种架构性缺失。这一缺陷带来了严重的现实后果：进入 2026 年仅半年时间，已有超过 59% 的消费者表示在与 AI 语音代理交互时体验到了挫败感与愤怒，且这一比例仍在持续上升。 - **语义占比失衡**：对话中的纯文字内容仅承载 3%～20% 的真实含义，其余 80%～90% 依赖语气、语调、停顿与重音等副语言信号。 - **用户负面情绪攀升**：2026 年已有超过 59% 的消费者对 AI 语音代理产生过愤怒与挫败感，模型能力的提升并未自然缓解这一矛盾。 ### [06:38](https://www.youtube.com/watch?v=E2thTe__EKo&t=398s) 原生音频可观测层：捕捉被遗漏的 80% 信号 解决该问题的有效方案并非重构整个基础架构，而是在现有技术管线中加入一层**原生音频可观测与分析层（Audio-Native Observability Layer）**，用以全面监听并捕获那被忽略的 80% 至 90% 的副语言特征。 当 Modulate 的技术介入语音 AI 通话分析时，系统会实时追踪数十种细粒度声学信号：声音中的情绪、语调、语速节奏（Cadence）、强调重音（Emphasis，即用户究竟在哪个词上加重了语气），乃至更为复杂的特征——例如用户是否使用了辅助技术或合成语音（用于识别 Deepfake 等潜在威胁）。基于完整的多维音频特征，系统可以精准检测出数百种复杂的交互行为，例如用户是否因为 AI 的误解而逐渐丧失沟通热情，或是由于系统延迟过高而变得不耐烦。 与容易产生幻觉的大型生成式模型不同，Modulate 采用小型专用模型集成（Ensembles of Small Models）方案，其原生音频模型在识别用户情绪或判断特定短语语调时的准确率可超过 99%，为系统提供了高度确定性的可信数据支撑。 ### [08:46](https://www.youtube.com/watch?v=E2thTe__EKo&t=526s) Velma 模型架构与 1/100 成本的实时干预 在具体的系统集成中，Modulate 依托名为 **Velma** 的集成监听模型。Velma 内部并行运行多个高效且精准的小型模型，并将各个维度的分析结果融合成对整个对话的完整表征。 在呼叫中心或企业级部署中，系统直接从 VoIP 或 CCaaS（云呼叫中心平台）接入坐席与客户的双向音频流，同步输入 Velma 引擎进行双向互动追踪与动态演变分析。一旦从音频中捕获到挫败感、高延迟、低音质等负面信号，系统可将结构化事件即时推送回服务台（Service Desk）。企业既可以在通话进行中进行**实时干预**（例如为 AI 提示词注入最新情绪上下文，或直接升级转接人工坐席），也可以在**通话后（Post-call）**通过可观测大盘进行宏观统计分析，精准定位未知的系统性隐患。 在成本控制方面，由于坚持轻量高效的专用模型路线，运行整套完整音频分析套件的成本仅为调用大型基础模型尝试做同类分析的 **1/100 甚至更低**。 ```mermaid flowchart LR accTitle: Velma 原生音频可观测集成架构 subgraph Ingestion["音频接入"] V["VoIP / CCaaS 系统"] V --> A_Stream["坐席/AI 音频流"] V --> C_Stream["客户音频流"] end subgraph Processing["Modulate Velma 引擎"] A_Stream & C_Stream --> Ensemble["高效小型模型集成"] Ensemble --> MultiSignal["提取情绪/语速/重音/伪造特征"] end subgraph Action["业务动作与分析"] MultiSignal --> Realtime["实时干预 (Prompt注入 / 人工转接)"] MultiSignal --> PostCall["通话后大盘 (聚合统计 / 根因定位)"] end ``` ### [10:46](https://www.youtube.com/watch?v=E2thTe__EKo&t=646s) 构建丰富对话表征与复杂行为挖掘 通过聚合全维度的声学信号，系统实现的能力远超简单的“通话是否失败”或“用户是否沮丧”等二元判断，而是构建出极度丰富的对话表征体系。 该体系能够输出完整的**情绪分布图谱**，追踪用户在整通电话中的心路历程，并结合用户的基线偏好进行动态校准。例如，某位用户在通话开始时极具活力、热情高涨，随后语调逐渐转为平淡中立——尽管对普通人而言“中立平淡”听起来毫无问题，但相对其初始基线，这一转变已经明确预示着交互出现了摩擦。此外，系统还能识别更高阶的人机博弈行为，例如判断来电者是否正在尝试对 AI 代理进行**情感操控（Manipulation）**、通过情绪施压或胁迫诱导 AI 做出违规让步。 - **多维动态建模**：不仅记录单点情绪，更能绘制整通通话的情绪起伏分布。 - **基线偏差感知**：通过用户个人情绪基线（如高热情降至中立）捕捉隐性不满。 - **复杂博弈识别**：支持检测用户针对 AI 坐席的情感操控、道德绑架或胁迫行为。 ### [12:08](https://www.youtube.com/watch?v=E2thTe__EKo&t=728s) 原生音频可观测层与医院反欺诈安全防线 仅凭纯文本转录文本（raw transcript），企业几乎无法察觉诸如提示词注入（Prompt Injection）或恶意操纵等隐蔽行为，但借助原生音频的可观测层（Audio-Native Observability Layer），这些行为将无所遁形。Modulate 的语音理解模型 Velma 被广泛应用于医院等大型机构的 IT 服务台（Help Desk）场景中，用以实时分析和识别通话风险。最典型的威胁之一便是攻击者利用 Deepfake 伪造声音进行社会工程学诈骗。 攻击者常利用人们的信任与助人心理，制造极度紧迫的恐慌情绪，例如声称：“你认识我的声音，如果你不马上帮我重置密码、开通系统权限，病人就会有生命危险！”服务台的客服人员本职就是协助他人，面对这种生死攸关的恳求极度脆弱。哪怕客服人员在 99% 的情况下都能坚持原则要求对方提供证件照片，只要有 1% 的疏忽被攻击者得逞，就会导致密码被重置或多因素认证（MFA）设备被篡改。这种单一漏洞往往会迅速演变为灾难性的大规模数据泄露和勒索软件攻击，给医院带来高达 600 万美元的巨额罚款，使敏感数据流向暗网，甚至真正危及患者的生命安全。如果能在原生音频层面深度理解对话并实时发出预警，就能在关键时刻拦截此类攻击。 ```mermaid flowchart TD accTitle: 医疗服务台 Deepfake 攻击与音频可观测防御流程 subgraph Attack["攻击阶段"] A["攻击者使用 Deepfake 伪造医生/员工声音"] --> B["制造紧急事态：'不重置密码病人会死'"] end subgraph Defense["客服与防御判断"] B --> C{"Help Desk 客服人员"} C -- "99% 严格合规" --> D["要求提供身份证明"] C -- "1% 心理防线被击穿" --> E["协助重置密码 / 修改 MFA"] E --> F["勒索软件部署 / 数据泄露 / 600万美元罚款"] B -. "Velma 原生音频模型实时监听".-> G["识别合成语音与异常语调风险"] G -. "实时告警阻断".-> C end ``` ### [13:37](https://www.youtube.com/watch?v=E2thTe__EKo&t=817s) 智能体失控循环与广泛的信任与安全应用 除了应对安全欺诈，原生音频可观测层在保障 AI 智能体自身稳定性方面同样发挥着关键作用。很多企业部署的语音 AI 往往会出现“脱轨”现象，陷入滑稽却极具破坏性的死循环。例如用户在通话尾声礼貌地说“太感谢了”，AI 回应“不客气，还有什么可以帮您的吗？”当用户回答“不用了，挺好的”时，AI 无法理解语调中的完结意味，反而继续追问“太好了，还有什么信息需要我提供吗？”如果遇到脾气急躁的用户可能会直接挂断，但讲礼貌的用户则会被困在无休止的客套循环中，产生极差的服务体验。 这种基于音频理解的技术同样被深入应用在信任与安全（Trust & Safety）领域。无论是在客服场景中检测针对坐席的言语骚扰，还是在大型在线社交空间中识别针对他人的网络霸凌、极端主义思想传播，乃至诱导或侵害儿童（Grooming）的恶劣行径，底层使用的都是完全相同的原生音频语音理解模型。 - **音频原生对话理解的核心价值**：语音 AI 的未来绝不仅仅是识别“说了什么”（What was said），而是透彻理解“怎么说的”（How it was said）——捕捉语气、语调与情绪层面的第一信号。 - **展位互动**：演讲最后，Carter Huffman 邀请现场对该技术感兴趣的参会者前往 940 号展位进行深入交流探讨。 ## Notable quotes > [03:21](https://www.youtube.com/watch?v=E2thTe__EKo&t=201s) I am happy to help. Let me explain to you what's going on. — AI 语音代理 > [03:21](https://www.youtube.com/watch?v=E2thTe__EKo&t=201s) Oh, perfect. Now I understand it. Incredible. — 愤怒的客户 > [03:21](https://www.youtube.com/watch?v=E2thTe__EKo&t=201s) oh, I'm happy I was able to help. Great! — AI 语音代理 > [05:10](https://www.youtube.com/watch?v=E2thTe__EKo&t=310s) 在人类日常的语音交流中，字面词汇所传递的实际信息量仅占全部含义的 3% 至 20%。 — Carter Huffman > [05:10](https://www.youtube.com/watch?v=E2thTe__EKo&t=310s) 现有的语音代理在处理用户请求时，丢失了 80% 到 90% 的关键上下文数据。 — Carter Huffman > [08:46](https://www.youtube.com/watch?v=E2thTe__EKo&t=526s) 运行整套完整音频分析套件的成本仅为调用大型基础模型尝试做同类分析的 1/100 甚至更低。 — Carter Huffman > [12:08](https://www.youtube.com/watch?v=E2thTe__EKo&t=728s) 你认识我的声音，如果你不马上帮我重置密码、开通系统权限，病人就会有生命危险！ — Deepfake 攻击者 > [13:37](https://www.youtube.com/watch?v=E2thTe__EKo&t=817s) 语音 AI 的未来绝不仅仅是识别“说了什么”，而是透彻理解“怎么说的”——捕捉语气、语调与情绪层面的第一信号。 — Carter Huffman ## Mentioned Carter Huffman, Modulate, NASA 喷气推进实验室 (Jet Propulsion Laboratory, JPL), Velma, VoIP, CCaaS, Deepfake, MFA (多因素认证), Ai4 ## Video description Carter Huffman - CTO, Modulate What if the biggest ROI drain on your voice agent deployment isn't a bug in the system — it's a tone the agent never learned to hear? Voice agents are scaling fast, but most enterprises have no way to measure where they're quietly losing money to conversational blind spots: misread emotion, missed emphasis, and frustration signals buried in audio that never surface in a transcript until they show up as escalations and churn. This talk will explain how to supervise voice agents through conversational understanding, treating tone and emphasis as first-class signals so teams can catch and correct friction mid-conversation. It will also walk through how to translate agent-level errors into the financial metrics that make or break the ROI case for voice AI. If you're accountable for justifying the cost of voice agents at your company, this talk will give you a concrete framework for finding and fixing the value you're leaving on the table. [Source](https://www.youtube.com/watch?v=E2thTe__EKo)

Source quotes

[3:21] AI 语音代理: I am happy to help. Let me explain to you what's going on. [3:21] 愤怒的客户: Oh, perfect. Now I understand it. Incredible. [3:21] AI 语音代理: oh, I'm happy I was able to help. Great! [5:10] Carter Huffman: 在人类日常的语音交流中，字面词汇所传递的实际信息量仅占全部含义的 3% 至 20%。 [5:10] Carter Huffman: 现有的语音代理在处理用户请求时，丢失了 80% 到 90% 的关键上下文数据。 [8:46] Carter Huffman: 运行整套完整音频分析套件的成本仅为调用大型基础模型尝试做同类分析的 1/100 甚至更低。 [12:08] Deepfake 攻击者: 你认识我的声音，如果你不马上帮我重置密码、开通系统权限，病人就会有生命危险！ [13:37] Carter Huffman: 语音 AI 的未来绝不仅仅是识别“说了什么”，而是透彻理解“怎么说的”——捕捉语气、语调与情绪层面的第一信号。

Source quotes

[3:16] I am happy to help. Let me explain to you what's going on [3:16] Oh, perfect. Now I understand it. Incredible. [3:16] Oh, I'm happy I was able to help. Great! [12:08] Oh, you know me. You recognize my voice [12:08] A patient's going to die if you don't give me access to my computer system, if you don't reset my password. [13:37] No, I'm good. Appreciate it [13:37] Great. Is there any more information I can give to you? [13:37] Carter Huffman: not just what was said, but how it was said

Original transcript

[0:04] So, let's go ahead and jump right in. Hey  everybody, I'm Carter Huffman. My company [0:08] Modulate is a voice AI models company. I'm the  CTO and co-founder. I got my start actually [0:16] at the NASA Jet Propulsion Lab over a decade ago.  I was working on extremely resource efficient, [0:23] but still very accurate models to run live on  board spacecraft in the middle of operational [0:29] missions. And that philosophy of extremely  efficient but accurate models has pervaded [0:35] everything that Modulate builds into our entire  ensembles of audio native, efficient, and accurate [0:44] voice conversation understanding capabilities.  I'm super excited to talk to you today about [0:50] voice AI agents and how to supervise and analyze  them, especially finding hidden costs and blind [0:57] spots in the conversations that humans are  having with AI agents. So, let's jump right in. [1:03] Voice AI moves incredibly fast. On a week-by-week  basis, leading AI companies are publishing new [1:12] models, new model versions, new capabilities,  and these models are improving along dozens of [1:19] different axes. So, you've got models that are  more cost-effective. You've got models that are [1:25] more accurate. You've got models that are better  at key terms or more emotive, sound more natural, [1:31] lower latency. There are so many things that  change about voice AI models every single day, [1:38] every single month. And as you're upgrading and as  you're changing to these new models, in general, [1:45] it's an overall win, but it's hard to tell what  you're losing moving from one version to another. [1:53] The benefits of being on the cutting edge  and being right there front and center [1:57] and adopting new voice AI models are pretty  extreme. Over 80% of enterprises are adopting [2:05] voice AI as part of their platform in one way  or another. The return on investment is huge [2:10] and the payback period for adopting voice AI  is measured in months, not years. So, being on [2:17] the cutting edge is really, really important  and impactful to your business. But again, [2:21] as you adopt these new models, when they start to  to to fail or change in some of these dimensions, [2:29] how are you going to know and what's the impact of  that? So, how do you know when a voice AI update [2:38] creates a negative experience for your  users? And I'm not talking CSAT scores. [2:45] I'm not talking about end-of-call surveys. It's  pieces in the conversation that have changed [2:52] that are making a negative experience for  the users on the other end of the line. [2:57] The answer is understanding the conversation. You  have to understand the dynamics of how the users [3:04] responding to the agents and you have to track  that over time. Every single conversation, [3:09] every single interaction, how's it going? What are  they saying? How's it changing? And how can you [3:16] measure it? So, let's jump into a little bit of  an example. So, even companies that are monitoring [3:23] voice AI conversations and interactions, often  times they'll just look at the transcript of the [3:29] call and run that through some kind of language  modeling or processing to see what's going on. So, [3:33] here's an example. Right? You've got a customer,  they're calling in and they're talking about their [3:39] bill has changed, right? Nobody loves  that. Everybody's had this experience. [3:43] And the agent's like, "I am happy to help. Let me  explain to you what's going on." Customer contact. [3:51] No and perfect. Great. And the agent and all  LLMs and generative AI are like this. It's like, [3:58] oh, it's just very happy to help, right? It's  happy to be there. Awesome, great, right? Well, [4:04] if you dig beyond the transcript into an audio  native understanding, uh you get a very different [4:09] picture. So, the customer calls in and they're  angry right from the start. Of course they are. [4:13] Their bill's changed, it's increased. Like, what's  going on? They're angry about it. The agent, [4:18] of course, is just reading the transcript. It's  responding and trying to be helpful. It's trying [4:23] to be useful. It'll explain the situation. And  that's not at all what the customer wanted here, [4:28] right? They're calling to complain. They're angry.  And so, their response is sarcastic. Oh, perfect. [4:34] Now I understand it. Incredible, right? And the  AI agent, again, just reading the transcript, [4:40] oh, I'm happy I was able to help. Great, right?  Is that a good interaction with the agent? Like, [4:46] no, absolutely not. And so, you've gone  from what a typical analysis would say [4:51] is a positive interaction, that's a win,  into a hidden cost, something that's a very [4:57] negative experience for the users, uh but  if you don't have that level of analysis, [5:03] you don't know anything about it. And again,  this can change as models change just like that. [5:10] It's not just a problem of are the models high  enough quality. This kind of misunderstanding and [5:16] this kind of problem in the voice agent ecosystem  is a fundamental technology and architectural [5:22] problem. So, the typical AI voice agent stack  goes directly from the audio and the voice [5:31] into a transcript, the words that people are  saying, and then that gets fed into an LLM, [5:36] and then eventually goes over to a synthesis  piece that responds. So far so good, right? Well, [5:42] the words that you're saying in a conversation  account for anywhere between 3 and 20% of all the [5:50] actual meaning generated in any voice conversation  between people. And so, the transcript that these [5:57] these agents are working off of has almost none of  the actual data it needs to respond appropriately [6:04] in a in any given situation. So, it's a whole  architectural problem. You upgrade to the latest [6:09] model, that's not going to solve it. And this is  having real impact. Over 59% of consumers report [6:18] frustration and anger involving AI voice agents  in 2026. We're only halfway through the year, [6:26] and that number is growing, not shrinking. Even  though the models are getting better, people [6:31] are getting more and more frustrated with these  systems. It's a big problem. So, what's the fix? [6:39] You add to this pipeline. You don't have  to change the underlying architecture, [6:43] but you add an observability and analytics  layer that is audio native. So, I said earlier, [6:51] transcripts, the words you're saying, are only  3 to, you know, 15-20% of what's being said, [6:58] observe the other 80 to 90% of it. So, for  example, when our technology is deployed to [7:06] to do analysis and observation of these  voice AI agent conversations, we look [7:11] at dozens of different signals. The emotion in  the voice, the tone, the cadence, the emphasis, [7:18] which words are you really focused on, as well  as more exotic stuff like are you using assistive [7:24] technology or synthetic voice? Maybe that's a deep  fake, etc., etc. All of these different components [7:31] of the audio completely lost in a transcript,  but important to understanding the meaning. [7:36] When you have that full picture, when you have  that rich conversation understanding, you can [7:42] detect an extraordinary number of things that  you care about. Hundreds of different behaviors [7:46] or events that wouldn't be available to you from  just those raw words. So, for example, is the user [7:54] losing enthusiasm for in a discussion based on  an agent misunderstanding something? Are they [8:00] angry because the latency is too high and it's  getting annoying? Etc. Etc. Etc. Etc. And in an [8:09] audio native observability layer, the way we do it  with ensembles of small models, you can actually [8:16] get extremely high accuracy and confidence in  these observations and these understandings. [8:22] Some of the audio native models can be over 99%  accurate on what emotion a user is expressing [8:29] or how they're saying a certain phrase. And  so, you can get a very confident representation [8:34] and a confident picture, whereas large generative  models tend to hallucinate. And so, sometimes it's [8:40] very difficult to trust what's coming out of  those models and understand what's going on. [8:46] So, here's how you can hook it in. This is an  example with uh with with our audio native stack. [8:51] So, we have a model called Velma. It's an ensemble  listening model. So, it actually runs a number of [8:56] different individual small models, each of which  are very efficient and accurate, and then composes [9:03] them together in a full representation  or understanding of the conversation. [9:07] So, in a Voice AI stack in a call center, for  example, you'll have your VoIPer or C-Cast system, [9:13] and it's got the audio stream for the agent and  for the caller or customer. We ingest both of [9:20] those streams together into our ensemble listening  model, and we analyze all of them. So, you can see [9:26] the back and forth between the agent and the user,  the conversation, and how it's evolving over time. [9:33] Once we've extracted those, we can send out a  variety of different signals and events. Again, [9:39] frustration, problems, bad user experience, bad  latency, bad call quality, bad signals, all of [9:46] these different things that you can infer from  the audio in the call over back into your system, [9:53] into your service desk, etc. And you can take  action on that. You can either do it in real time, [9:58] come in and intervene as the call is going wrong  and bring it back, either by prompting the agent [10:04] in a different way with new context, escalating  to a human, etc. Or you can do it post-call on [10:11] the recording and get an observability dashboard  and a set of analytics so you can pinpoint, okay, [10:18] what's going wrong in our calls  in an aggregate statistical level [10:22] that you didn't know about beforehand. Running  all of these models sounds like it could be pretty [10:26] expensive. Fortunately, again, if you focus in  on resource-efficient, cost-effective models, [10:34] you can actually do this entire suite of analysis  that I mentioned for 1/100 of the cost or better [10:39] than running a large foundation model  to try to do the same kind of analysis. [10:46] So, building up all of this picture  and running all of those signals, [10:50] you can do more than just say, "Did this call go  wrong?" Or, "Is this person getting frustrated?" [10:56] These signals together compose into an extremely  rich conversation representation that builds [11:04] understanding for users who are looking and  analyzing statistically across your calls, [11:11] and for the AI agents themselves that they  can use and interact with. So, for example, by [11:16] running audio-native analysis across a call, you  can get an entire suite of emotion distributions. [11:22] What was this user's journey as they experienced  this call? How you know, what was their prior if [11:28] they were coming in? For example, when I call in  I'm like really energetic, I'm enthusiastic, and [11:33] then if I go neutral, something's been something's  gone wrong. Even if to somebody else listening to [11:38] that exact same statement, it sounds okay, that's  totally natural. That's that's totally fine. [11:44] And you can extract a variety of different signals  beyond just like the user's frustrated. So, [11:50] for example, uh some of the the behaviors that  we can extract from a voice conversation include [11:56] like, is the user trying to manipulate the agent,  right? agent or an AI agent, whatever. So, are you [12:03] calling in? Are you trying to appeal emotionally?  Are you trying to coerce them into taking an [12:08] action? Are you trying to prompt inject your voice  agent? Etc. etc. All of those behaviors, very hard [12:15] to find from just a raw transcript, but available  to you with an audio native observability layer. [12:22] Some of the use cases of doing that, um  they're pretty broad. Uh so, on the one hand, [12:27] uh there are there are uh a lot of customers  like uh hospitals that we work with [12:32] that use our model Velma to analyze and understand  these calls in their help desk. And they'll find [12:42] is somebody calling in and using a deepfake.  Okay, that could be an assistive technology or [12:47] that could be someone trying to commit fraud cuz  they're like, "Oh, you know me. You recognize my [12:50] voice." They're creating a sense of urgency. A  patient's going to die if you don't give me access [12:56] to my computer system, if you don't reset my  password, right? The agent on the other end of the [13:01] line, like they're trying to help people. That's  what they That's what they That's what they do. [13:05] And so, they're very, very vulnerable to those  kinds of appeals. And even if 99% of the time they [13:10] say, "Sorry, you need to send me a picture of your  ID." 1% of the time the attacker gets to them, [13:16] and now they have a password reset, now  they have a multi-factor device change, [13:20] that leads to a data breach, ransomware  deployment, $6 million fine for the hospital, [13:26] data leaked on the dark web. People can die,  right? Those are the sorts of things that you can [13:31] do if you're really understanding a conversation  and can flag these sorts of things in real time. [13:37] All the way on the other end, folks use our models  for is your AI agent going off the rails? Like one [13:43] fun one, a lot of agents, maybe you've experienced  this when you called into a call center, [13:48] you know, you'll say, "Okay, like  thanks so much." And they'll say, [13:50] "Sure thing. Are there Is there anything else  I can help you with?" And you're like, "No, [13:54] I'm good. Appreciate it." And it's like, "Great.  Is there any more information I can give to you?" [13:59] And it'll just get in a loop of like asking you  a question. And if you're rude, you just hang up. [14:03] But if you're polite, you can stay on the  line over and over again and have a bad [14:07] experience because the AI agent doesn't know  any better. It's getting caught in a bad loop. [14:12] Same thing all the way over into trust and  safety. Is somebody harassing an agent or even [14:17] in like social spaces are people harassing  each other, promoting extremist ideology, [14:22] even like attempting to groom children? We'll  detect that in social conversations and spaces [14:27] using exactly the same audio native voice  understanding models that are used to [14:35] power the observability for voice AI agents. This  stuff is extremely broad and extremely powerful. [14:41] And ultimately, the future of voice AI is  audio native conversation understanding [14:47] where your models can see not just  what was said, but how it was said. [14:53] If this sounds cool to you, come visit us at  our booth, 940. Uh, super excited to chat with [15:00] you all. Thanks so much for listening. Really  appreciate it. Hope you have a lovely afternoon.