Beyond “Good Enough”: How deployments accelerate research at the Audio AI Frontier by ElevenLabs

Beyond “Good Enough”: How deployments accelerate research at the Audio AI Frontier by ElevenLabs — @Scale

@Scale

AI-assisted summary of Beyond “Good Enough”: How deployments accelerate research at the Audio AI Frontier by ElevenLabs — original source. Check the original for full context.

Source published:

Key ideas

  • Word Error Rate benchmarks are saturating as frontier transcription models reach ~2.7% WER, outperforming legacy evaluation datasets that themselves contain human transcription errors. (0:05)
  • ElevenLabs ties research directly to production through a three-pillar feedback loop combining fundamental research, end-to-end product development, and forward deployed engineering. (2:13)
  • Production deployments with Meta on Reels dubbing revealed an inherent tradeoff between exact vocal identity and target-language intelligibility across distant language pairs. (11:38)
  • To solve cross-lingual accent friction, ElevenLabs introduced a configurable 'cloning strength' slider (0.0 to 1.0) allowing platforms to balance theatrical dubbing clarity against raw acoustic identity. (13:18)

Detailed notes

## Background Published by @Scale on YouTube on November 3, 2026, this presentation features Shaheen Lavie-Rouse from ElevenLabs exploring what lies beyond traditional audio AI benchmarks. According to the page description, the talk examines how ElevenLabs scales a learning loop across research, product, and forward deployed engineering to tackle emotional intelligence and taste, highlighting real-world lessons from dubbing Reels and the creation of a cloning strength control.

## TL;DR As frontier speech models push Word Error Rate to practical saturation limits, the central challenge in audio AI has shifted from objective defect reduction to subjective human qualities like taste and emotional intelligence. ElevenLabs navigates this shift through a continuous feedback loop connecting fundamental research, end-to-end product development, and forward deployed engineering with enterprise partners. Practical deployments revealed critical bottlenecks at the 'seams' of multi-model systems, prompting advances in full-duplex conversational agents and interruption handling. In cross-lingual dubbing deployments with Meta Reels, ElevenLabs discovered that high-fidelity voice cloning can preserve unintelligible source accents, leading to the development of a tunable 'cloning strength' slider that balances personal vocal identity with native listener comfort.

## Atmosphere & style The presentation carries an articulate, forward-looking, and insider technical tone, seamlessly blending rigorous systems engineering with a musician's attuned ear for aesthetic nuance. Interactive audio demonstrations—ranging from synthetic voices that interrupt human speakers mid-sentence to evocative spoken-word performances—give the talk a tactile and dynamic feel. Rather than dry benchmark reporting, it feels like an engaging, real-time dispatch from the bleeding edge of conversational AI, inviting listeners to rethink how humans will communicate with computing interfaces.

## Bulletin - [00:05](https://www.youtube.com/watch?v=ghLHxHO42oY&t=5s) Word Error Rate benchmarks are saturating as frontier transcription models reach ~2.7% WER, outperforming legacy evaluation datasets that themselves contain human transcription errors. - [02:13](https://www.youtube.com/watch?v=ghLHxHO42oY&t=133s) ElevenLabs ties research directly to production through a three-pillar feedback loop combining fundamental research, end-to-end product development, and forward deployed engineering. - [11:38](https://www.youtube.com/watch?v=ghLHxHO42oY&t=698s) Production deployments with Meta on Reels dubbing revealed an inherent tradeoff between exact vocal identity and target-language intelligibility across distant language pairs. - [13:18](https://www.youtube.com/watch?v=ghLHxHO42oY&t=798s) To solve cross-lingual accent friction, ElevenLabs introduced a configurable 'cloning strength' slider (0.0 to 1.0) allowing platforms to balance theatrical dubbing clarity against raw acoustic identity.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=ghLHxHO42oY&t=5s) Speech-to-text baselines shifted drastically between 2025 and 2026, with Scribe v2 lowering WER to 2.7% and moving beyond the limits of standard historical datasets. - [02:13](https://www.youtube.com/watch?v=ghLHxHO42oY&t=133s) Keyboards remain ubiquitous largely because speech transcription was historically too inaccurate to serve as a reliable primary computing interface. - [02:13](https://www.youtube.com/watch?v=ghLHxHO42oY&t=133s) A continuous loop linking researchers, product teams, and forward deployed engineers grounds model development in production realities rather than isolated benchmarks. - [04:03](https://www.youtube.com/watch?v=ghLHxHO42oY&t=243s) Major accuracy disparities remain between English and Tier-2, Asian, or longtail languages like Swahili and Catalan. - [04:03](https://www.youtube.com/watch?v=ghLHxHO42oY&t=243s) Standard audio benchmarks fail to evaluate messy enterprise acoustic conditions, such as noisy airport terminals or domain-specific terminology. - [04:03](https://www.youtube.com/watch?v=ghLHxHO42oY&t=243s) Speaker diarization remains an unsolved structural bottleneck in speech-to-text workflows. - [06:24](https://www.youtube.com/watch?v=ghLHxHO42oY&t=384s) Audio AI evaluation has evolved from basic defect counts to Elo-based preference arenas measuring human pacing, natural breath, and dramatic intonation. - [08:38](https://www.youtube.com/watch?v=ghLHxHO42oY&t=518s) Voice agent failures frequently happen at the seams connecting discrete models rather than within individual STT or TTS components. - [08:38](https://www.youtube.com/watch?v=ghLHxHO42oY&t=518s) Deep turn models enable full-duplex conversations where agents can listen while speaking, interpret intent, and handle natural mid-sentence interruptions. - [13:18](https://www.youtube.com/watch?v=ghLHxHO42oY&t=798s) High-fidelity voice cloning across linguistically distant languages like Japanese to English inadvertently carries over heavy non-native accents and prosody. - [13:18](https://www.youtube.com/watch?v=ghLHxHO42oY&t=798s) The adjustable cloning strength slider ranges from 0.0 (accentless theatrical dub) to 1.0 (exact acoustic clone including room noise and source accent). - [16:01](https://www.youtube.com/watch?v=ghLHxHO42oY&t=961s) The new frontier in audio AI centers on subjective parameters such as taste, emotional conviction, and fine-grained stylistic nuance.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=ghLHxHO42oY&t=5s) The Saturation of Objective Benchmarks in Audio AI

Looking at the landscape of AI in 2026, the velocity of progress across every discipline—especially audio—has become blistering. When ElevenLabs entered the speech-to-text arena in 2025 by shipping their first transcription model, hitting the number-one spot on public leaderboards with a 7.7% Word Error Rate (WER) was a landmark achievement. Yet just twelve months later, with the release of Scribe v2, the entire competitive baseline had shifted dramatically. Every single frontier model on the leaderboard now substantially outperforms what had been the world-record benchmark just one year prior.

This leap in accuracy is not merely incremental; it represents a qualitative phase shift in utility. As Shaheen Lavie-Rouse explains, “For speech to text, the difference between 90% accurate and 99% accurate is the difference between a model you can use to replace your keyboard and a model that like misses every so often the words that you're trying to say and is not so useful.” However, this relentless march has pushed classic benchmarks to their practical limits. Evaluating models on traditional test sets has hit a wall because human transcription and historical evaluation corpora are themselves riddled with errors. As Lavie-Rouse notes, “We are reaching a point where the frontier models are actually more accurate than the data sets that are being used to evaluate them because human transcription and these historical data sets are not always correct.” Benchmarking organizations like Artificial Analysis have even had to begin extensive data-cleanup initiatives just to repair test-set quality. With Scribe v2 hitting a 2.7% WER and Scribe v3 already on the horizon, the central dilemma for the field shifts from raw error reduction to discovering what attributes researchers should actually optimize for next.

- In 2025, ElevenLabs captured the #1 spot on the speech-to-text leaderboard with a 7.7% Word Error Rate. - By 2026, Scribe v2 pushed WER down toward 2.7%, while the industry baseline surpassed the previous year’s top score. - Benchmarks are saturating because ground-truth human annotations frequently contain more mistakes than the frontier models evaluating against them.

---

[02:13](https://www.youtube.com/watch?v=ghLHxHO42oY&t=133s) The Three-Pillar Learning Loop: Research, Product, and Deployments

ElevenLabs operates across the full audio spectrum—spanning synthetic speech, transcription, voice agents, dubbing, sound effects (SFX), and generative music. As developer and enterprise demand surges, the company sees the realization of a longstanding industry vision: computing interfaces governed by spoken conversation rather than manual typing. Human beings are biologically wired to communicate via voice; the prevalence of physical and on-screen keyboards has historically been a workaround for poor machine listening. Lavie-Rouse reminds the audience, “The only reason we have keyboards on our phones and on our laptops is transcription has just never been that good at capturing what we're saying and what we're trying to do.”

To move beyond generic leaderboard metrics, ElevenLabs has structured its organization into a unified feedback loop connecting fundamental research, end-to-end product development, and forward deployed engineering (FDE). Instead of training isolated models in a vacuum, researchers work in tandem with product teams who map out complex end-to-end audio workflows and engineers who embed directly with enterprise partners deploying AI to millions of end consumers. When models encounter the messy realities of public deployment, real friction points are funneled back into basic research, accelerating the model development cycle.

```mermaid flowchart TD accTitle: ElevenLabs Three-Pillar Learning Loop accDescr: Diagram illustrating how Research, Product, and Forward Deployed Engineering form a continuous feedback loop.

subgraph Pillars ["Core Operational Pillars"] R["Research<br/>(Frontier Audio Models)"] P["Product<br/>(End-to-End Workflows)"] FDE["Forward Deployed Engineering<br/>(Public Enterprise Deployments)"] end

R -->|"powers capabilities"| P P -->|"ships customer workflows"| FDE FDE -->|"reveals edge failures"| P P -->|"defines next attributes"| R FDE -.->|"direct frontier insights"| R ```

- Audio disciplines covered include Text-to-Speech (TTS), Speech-to-Text (STT), voice agents, dubbing, generative music, and SFX. - Forward deployed engineering teams work side-by-side with high-volume production clients facing the general public. - The learning loop systematically shifts research optimization away from generic benchmarks toward production-grade reliability.

---

[04:03](https://www.youtube.com/watch?v=ghLHxHO42oY&t=243s) Uncovering the Real Frontier: Language Parity and Imperfect Environments

While English speech-to-text benchmarks appear effectively conquered on paper, conventional benchmarks obscure massive performance gaps across the broader world. Once models venture beyond English into Tier-2 languages—such as French and Spanish—accuracy metrics begin to slip, and the deficit widens considerably across Asian languages and longtail tongues like Swahili or Catalan. The true product frontier is ensuring that voice can completely replace keyboards across all global languages, not merely dominant Western ones. Furthermore, structural audio challenges such as speaker diarization (accurately parsing who spoke when) remain far from solved.

Benchmarks can also paint an artificially flattering picture because they typically run on clean, studio-grade audio. In enterprise applications, audio arrives corrupted by packet loss, screaming airport terminals, car engine hum, and low-grade smartphone microphones. Furthermore, generic datasets fail to test specialized domain vernacular, such as niche television titles, regional restaurant dish names, or dense financial and data-analytics terminology. Emphasizing the deceptive nature of generic tests, Lavie-Rouse points out, “Equally importantly though is to figure out where benchmarks can kind of lie or not be representative of what you might see in practice where these benchmark are generic.” Fortunately, the same AI acceleration driving research makes it easier for enterprises to spin up custom evaluation suites that reflect their own acoustic environments and domain lexicons.

- Benchmark accuracy drops significantly in Tier-2 languages (French, Spanish), Asian languages, and longtail languages (Swahili, Catalan). - Speaker diarization remains an active research bottleneck with significant room for architectural improvement. - Standard benchmark test sets overlook acoustic noise (airports, phone calls) and high-context industry jargon.

---

[06:24](https://www.youtube.com/watch?v=ghLHxHO42oY&t=384s) Moving Beyond Word Error Rate: Taste, Elo Ratings, and Emotional Intelligence

The most demanding frontier in audio AI is navigating attributes that cannot be quantified through simple defect counts or error rates. In expressive mediums like music generation or emotional voice acting, success is dictated by aesthetics, conviction, and emotional intelligence. Text-to-speech underwent a comparable historical evolution: a decade ago, models were judged almost entirely on whether they could read a script without mispronouncing, skipping, or hallucinating words. Once basic intelligibility became a solved problem, the evaluation paradigm shifted toward Elo-based preference arenas, pitting models head-to-head on how convincing, well-paced, and human they sound.

Shaheen Lavie-Rouse, who brings a musical background to his work at ElevenLabs, underscores that human ears are becoming increasingly attuned to the finer nuances of synthetic voice. During the talk, he plays a historical progression of TTS generations reading the identical prompt: *“Am I good enough? I'm saying the right words. I'm pacing the right way. Is this good enough?”* The early generations demonstrate flat, robotic delivery, while successive versions master breath control, dramatic pauses, dynamic intonation, and even fluent accent changes (*“I can even sound French”*). As Lavie-Rouse notes when discussing subjective audio capabilities:

> “The most interesting part though where I'm going to focus the rest of my talk is what you can't even measure with an objective metric like word error rate and that's applicable to broader set of our audio models like music right how do you evaluate that here it has to do with taste has to do with emotional intelligence it has to do with audio that's actually able to generate the types of conviction the types of emotions that the users are trying to you're trying to inspire with your users.”

- TTS evaluation has migrated from mechanical Word Error Rate to crowd-sourced Elo preference rankings. - Elo ratings capture subtle human parameters: pacing, micro-intonation, emotional delivery, and naturalness. - Model development must now optimize for listener conviction and perceived emotional resonance rather than just syntactic correctness.

---

[08:38](https://www.youtube.com/watch?v=ghLHxHO42oY&t=518s) Solving the Seams: Voice Agents, Interruption Handling, and Deep Turn Models

When developers started using standalone transcription and voice synthesis models, they immediately attempted to stitch them together into autonomous voice agents and dubbing pipelines. However, ElevenLabs discovered that agents often collapse not because of weak underlying TTS or STT, but because of failures occurring at the "seams" between systems. Lavie-Rouse stresses, “Because a lot of these things to be the best in the world, it's not enough just to have the best speech to text, text to speech, to have the best voice agent.” Conversational agents must orchestrate complex, real-time mechanics: managing turn-taking, executing mid-dialogue tool calls, querying knowledge bases, and handling human interruptions.

Basic voice agent setups attempt to handle turn-taking with crude silence timeouts, which results in unnatural, walkie-talkie style conversations. To address this, ElevenLabs trains specialized deep turn models designed to learn the subtle acoustic, linguistic, and prosodic cues of natural human conversation. Lavie-Rouse demonstrates this capability with a recorded dialogue between a human and an agent:

- **Human:** "Ready for your talk? Isn't it like tomorrow?" - **Agent:** "Yeah. Uh, I was wondering if you can help me. Should I show a demo conversation with a voice agent for this talk?" - **Human:** "Oh, absolutely. Nothing proves how far—" - **Agent:** *(Cutting in dynamically)* "Wait, wait. Specifically, I want to see like turn taking, interruption handling. Is that kind of something that you think needs a live demo?" - **Human:** "Oh, see, you just did it. That's exactly what modern voice agents do. They listen while they speak, detect when you want the floor, and stop mid-sentence to let you jump in, just like you did with me. It's the perfect thing to show live because hearing that natural flow is way more convincing than just describing it."

By operating in full duplex—listening while speaking, continuously judging intent, and yielding the conversational floor mid-syllable—agents eliminate awkward latencies and simulate natural human interaction.

[10:54](https://www.youtube.com/watch?v=ghLHxHO42oY&t=654s) Crossing Seams in Chained Audio Models

Following an interactive demo audio clip where a synthesized voice rallies the crowd with, “Wait, are you recording this right now? Oh, no. Okay. Hi, everyone at Scale. You are going to crush this talk tomorrow,” Shaheen Lavie-Rouse shifts the focus from conversational agent turn-taking to the broader structural hurdles in audio pipelines. Whenever complex systems string multiple discrete models together—whether powering interactive voice agents, automated dubbing pipelines, or creative studio workflows—friction and disconnections inevitably emerge at the boundaries between those components.

ElevenLabs is deliberately directing its research toward bridging these awkward transitions rather than merely optimizing each component in isolation. As Shaheen Lavie-Rouse explains: “Every use case we've found where we chain together models or we're actually trying to now serve users who are trying to achieve the end goals they have with audio whether that's agents dubbing creative has these kinds of seams.” By recognizing where compounding errors and unnatural shifts occur across these seams, researchers can design models that natively span across steps, producing smoother, more continuous end-to-end audio experiences.

[11:38](https://www.youtube.com/watch?v=ghLHxHO42oY&t=698s) Deployments at Scale: Partnering with Meta on Reels Dubbing

To determine where models break down in practice, ElevenLabs deploys forward deployed engineers (FDEs) directly alongside engineering teams integrating voice models into production consumer platforms. Working side-by-side with product builders provides immediate, real-world feedback when systems are exposed to massive, diverse user bases. This direct observation of how audio models behave under specific domain constraints, ambient environments, and user intents creates an operational feedback loop that supplies ElevenLabs’ core researchers and product teams with their next generation of high-priority research problems.

Shaheen illustrates this dynamic through ElevenLabs' collaboration with Meta across Instagram Reels and Facebook Reels. In conventional media production, dubbing has always involved a fundamental compromise: the translated speech is performed by a completely different human voice actor whose vocal identity diverges from the on-screen talent. As Shaheen Lavie-Rouse highlights: “When you hire a human voice actor to dub, it's always going to be a different person who's voicing the dub compared to the original speaker.” ElevenLabs approached the Reels deployment with an ambitious vision for automated AI dubbing: clone the creator's exact voice from the source language into the target language, faithfully maintaining the speaker's original cadence, emotional inflections, and pitch range so the resulting translation sounds uniquely like them.

[13:18](https://www.youtube.com/watch?v=ghLHxHO42oY&t=798s) The Cloning Strength Slider: Balancing Voice Identity and Native Accent

Putting automated dubbing into widespread production revealed an unexpected friction point: hyper-realistic voice cloning can sometimes be too realistic. While cross-lingual voice cloning produces a convincing, uncanny resemblance between source and target speech when languages share close phonetic roots, major issues arise when translating across linguistically distant languages. For example, when cloning a native Japanese speaker into English, a pure voice clone will preserve not just vocal timbre, but also phoneme habits, source prosody, and a heavy non-native accent, yielding English speech that sounds unnatural or difficult to understand.

```mermaid flowchart LR accTitle: Cloning Strength Spectrum accDescr: Diagram illustrating the spectrum of cloning strength in ElevenLabs dubbing technology from zero to full strength. subgraph Spectrum ["Cloning Strength Spectrum"] Zero["Cloning Strength: 0.0"] --> Mid["Tuned Default: Mid-Point"] Mid --> Full["Cloning Strength: 1.0"] end Zero --- ZeroDesc["Pure synthetic native voice<br/>Zero source accent retention<br/>Clean, theatrical dub feel"] Mid --- MidDesc["Balanced identity retention<br/>Natural target-language flow<br/>Optimal user taste setting"] Full --- FullDesc["Exact acoustic clone<br/>Retains source accent & room noise<br/>Raw, unpolished realism"] ```

To solve this dilemma, ElevenLabs engineered a configurable "cloning strength" parameter. At a setting of zero, the engine outputs a purely synthetic, accentless native speaker in the target language who sounds like a polished, theatrical dubbing actor over the video. At full strength (1.0), the engine strictly preserves the source speaker's raw acoustic properties, warts and all—including poor recording acoustics, idiosyncratic speech mannerisms, and heavy native accents. As Shaheen Lavie-Rouse notes regarding this control: “And so this toggle lets us fix these kinds of issues, but also it's become a dial for taste.” By dialing the global default toward a balanced middle setting and exposing the slider to enterprise dubbing partners, platforms can balance personal vocal identity against native listening comfort.

- **Accent Retention Bottleneck:** High-fidelity voice transfer across distant language pairs (such as Japanese to English) naturally carries over source language phonetics, undermining target intelligibility. - **Cloning Strength at 0.0:** Generates an idealized, accent-free native voice in the target language, prioritizing clarity and traditional theatrical dubbing aesthetics. - **Cloning Strength at 1.0:** Produces an exact acoustic facsimile of the original speaker, reproducing background noise, room acoustic imperfections, and regional accent traits. - **Production Default Shift:** ElevenLabs migrated its system baseline away from maximum fidelity to a balanced middle value, enabling enterprise partners to adjust the dial to suit different genres and audiences.

[16:01](https://www.youtube.com/watch?v=ghLHxHO42oY&t=961s) The Next Frontier: Taste, Emotional Intelligence, and 11/11

In his closing reflections, Shaheen argues that traditional quantitative benchmarks like Word Error Rate (WER) have largely saturated, leaving them insufficient for measuring state-of-the-art audio AI. The true frontier of voice technology now lies in subjective human qualities—specifically taste, stylistic nuance, and emotional conviction. As Shaheen Lavie-Rouse explains: “Like I was saying, we're really seeing some of these objective error-based metrics saturate and finding the new frontier in taste based and emotional intelligence.” To mark this progression, ElevenLabs scheduled its next wave of core model releases for November 11th (11/11).

Drawing on his personal background as a musician, Shaheen shares a final audio sample demonstrating how fine-grained emotional intent and atmosphere can be carried through AI translation. The rendered demo captures an intimate, moody performance: “The lamps low and the night leans in. The room is soft with a blue kind of light.” Shaheen concludes by expressing excitement for the pace of generative audio progress moving into the coming year, urging developers and creators across the industry to identify their own high-impact audio use cases as the frontier expands beyond mere technical correctness.

## Notable quotes > [02:13](https://www.youtube.com/watch?v=ghLHxHO42oY&t=133s) The only reason we have keyboards on our phones and on our laptops is transcription has just never been that good at capturing what we're saying and what we're trying to do. — Shaheen Lavie-Rouse

> [00:05](https://www.youtube.com/watch?v=ghLHxHO42oY&t=5s) We are reaching a point where the frontier models are actually more accurate than the data sets that are being used to evaluate them because human transcription and these historical data sets are not always correct. — Shaheen Lavie-Rouse

> [00:05](https://www.youtube.com/watch?v=ghLHxHO42oY&t=5s) For speech to text, the difference between 90% accurate and 99% accurate is the difference between a model you can use to replace your keyboard and a model that like misses every so often the words that you're trying to say and is not so useful. — Shaheen Lavie-Rouse

## Mentioned Shaheen Lavie-Rouse, ElevenLabs, Artificial Analysis, Scribe v2, Scribe v3, Meta, Instagram Reels, Facebook Reels, Scale

## Video description Presenter: Shaheen Lavie-Rouse from ElevenLabs

As audio AI models saturate traditional benchmarks like word error rate, what comes next?

ElevenLabs is scaling a learning loop across research, product, and forward deployed engineering to find the new frontier: emotional intelligence and taste — audio that carries the speaker’s intent and conviction. Drawing on ElevenLabs’ work dubbing Reels, this talk shows how deployments revealed a trade-off between preserving a speaker’s own voice and sounding like a native speaker in the target language, and the new cloning strength control built to manage it.

[Source](https://www.youtube.com/watch?v=ghLHxHO42oY)

中文简报

超越“足够好”：ElevenLabs如何通过实际落地加速音频AI前沿研究 随着前沿语音模型将词错误率推向实际饱和极限，音频AI的核心挑战已从客观上的减少缺陷，转向品味和情商等主观人类特质。ElevenLabs通过将基础研究、端到端产品开发以及面向企业合作伙伴的前向部署工程相结合的持续反馈循环，来应对这一转变。实际落地揭示了多模型系统“接缝”处的关键瓶颈，进而推动了全双工对话智能体和打断处理方面的技术进展。在与Meta Reels开展的跨语言配音落地项目中，ElevenLabs发现高保真声音克隆可能会保留难以听懂的源口音，由此研发出可调节的“克隆强度”滑块，在个人声音特征与母语听众舒适度之间取得平衡。 随着前沿转录模型的词错误率达到约2.7%，词错误率基准测试正趋于饱和，其表现已超越自身就包含人工转录错误的传统评估数据集。 (0:05) ElevenLabs通过结合基础研究、端到端产品开发和前向部署工程的三支柱反馈闭环，将研究与生产直接挂钩。 (2:13) 在Meta Reels配音的生产落地中，发现在差异较大的语言对之间，精准的声音特征与目标语言的清晰度之间存在固有权衡。 (11:38) 为解决跨语言口音摩擦，ElevenLabs推出了可配置的“克隆强度”滑块（0.0至1.0），允许平台在影视级配音清晰度与原始声学特征之间取得平衡。 (13:18)

Reading summary

Shaheen Lavie-Rouse of ElevenLabs argues that traditional objective benchmarks like Word Error Rate have saturated, shifting the audio AI frontier toward subjective qualities—including taste, emotional intelligence, and seamless real-world pipeline integration discovered through production deployments. Benchmark Saturation and Evaluation Realities (0:05) In 2025, ElevenLabs led public speech-to-text benchmarks with a 7.7% Word Error Rate (WER). By 2026, its Scribe v2 reduced WER to 2.7%, while the broader industry baseline surpassed the prior year's world-record performance. Lavie-Rouse observes that objective error-reduction benchmarks have reached a practical limit because frontier models have become more accurate than historical evaluation datasets, which frequently contain human transcription errors. Consequently, benchmarking organizations such as Artificial Analysis have had to initiate test-set data cleanup efforts. Furthermore, standard benchmarks rely on studio-quality English audio, obscuring performance degradation in messy acoustic environments like noisy airports, low-grade microphones, or phone calls with packet loss. Accuracy also drops sharply in Tier-2 languages like French and Spanish, Asian languages, and longtail tongues like Swahili and Catalan. In addition, structural capabilities such as speaker diarization and domain-specific terminology remain unresolved technical bottlenecks. The Three-Pillar Deployment Feedback Loop (2:13) To counter the limitations of static leaderboards, ElevenLabs organizes its operations around a continuous learning loop connecting fundamental research, product teams designing end-to-end audio workflows, and Forward Deployed Engineering (FDE). Instead of training isolated models, FDE teams embed directly with enterprise partners deploying audio models to millions of end consumers. Messy edge failures and domain constraints encountered in public deployments are funneled directly back to research teams to steer future architectural priorities. Solving Seamless Execution in Chained Pipelines (8:38) Autonomous voice agents and multi-model pipelines frequently break down not from deficits in standalone transcription or text-to-speech models, but at the 'seams' connecting them. Primitive agents rely on crude silence timeouts, resulting in unnatural, walkie-talkie exchanges. To resolve this friction, ElevenLabs trains deep turn models capable of interpreting acoustic, prosodic, and linguistic cues for real-time turn-taking and interruption handling. Operating in full duplex enables agents to listen while speaking, judge intent, and yield conversational turns mid-syllable when interrupted. Balancing Vocal Identity Against Native Accents (13:18) Deploying cross-lingual dubbing with enterprise partners like Meta across Instagram and Facebook Reels surfaced trade-offs between vocal identity and listener comprehension. While conventional dubbing relies on distinct human voice actors, AI dubbing can clone the original creator's timbre, cadence, and pitch across languages. However, full-fidelity cloning across distant language pairs—such as Japanese to English—preserves source phonetics and heavy accents, impeding clarity for target audiences. To manage this trade-off, ElevenLabs introduced a configurable cloning strength parameter. At a setting of 0.0, the model outputs an idealized, accent-free synthetic native speaker resembling a theatrical dub. At 1.0, it produces an exact acoustic facsimile retaining background room noise and source accents. ElevenLabs shifted its default setting to a balanced middle value, treating the slider as an adjustable dial for enterprise client taste. As quantitative metrics saturate, ElevenLabs contends that the next competitive frontier centers on subjective conviction, Elo preference arenas, and emotional resonance rather than purely mechanical accuracy. Unresolved challenges persist, however, in achieving true language parity across non-Western tongues, handling complex multi-speaker diarization, and standardizing aesthetic evaluation across diverse artistic domains.

精炼阅读摘要

ElevenLabs的Shaheen Lavie-Rouse指出，随着传统客观基准（如词错率）在语音AI领域的逐步饱和，音频模型的研究重心正从单纯的错误率消除转向真实部署中的可靠性、品味与情感表现力。 客观基准饱和与评测困境 (0:05) 在语音转文字领域，ElevenLabs于2025年以7.7%的词错率位居公开排行榜首位，而至2026年，其Scribe v2模型已将词错率降至2.7%，行业前沿模型普遍超越了前一年的纪录。Lavie-Rouse表示，90%到99%的准确率飞跃代表了工具实用性的质变，意味着语音技术已具备取代物理键盘的潜力。 然而，这种进展导致传统评测数据集触碰天花板。由于历史测试集与人工转录本身存在错误，前沿模型的准确度在某些情况下甚至已超过评测基准本身，以至于像Artificial Analysis等评测机构必须启动数据清洗工作。该领域的关键挑战随之演变为寻找词错率之外的新优化维度。 三位一体闭环打破实验室孤岛 (2:13) 为应对基准测试的局限，ElevenLabs构建了连接基础研究、端到端产品与前向部署工程（FDE）的协同机制。研究团队不再孤立训练模型，而是将工程师直接派驻到面向海量终端用户的企业合作伙伴现场，捕捉模型在现实环境中暴露出的摩擦点。 这种闭环不仅涵盖文本到语音和语音到文字，还横跨语音代理、视频配音、音效及生成式音乐等全音频形态。通过将高并发实际场景中的边缘失效案例直接反馈给研究部门，研发路线得以脱离空泛的公开跑分，优先解决生产环境中的稳定性难题。 真实场景与多语言落差 (4:03) 公开基准通常基于录音室级别的高质量英语音频，掩盖了实际部署中的巨大鸿沟。当模型应用于法语、西班牙语等二级语言时，识别表现出现下滑；而在亚洲语言以及斯瓦希里语、加泰罗尼亚语等长尾语种中，准确率落差更为明显。此外，区分不同说话人的说话人日志分离技术仍是尚未完全解决的架构瓶颈。 在企业级实际应用中，音频往往混杂着丢包损坏、机场噪音频带、汽车引擎轰鸣或低端手机麦克风的失真。通用测试集同样无法覆盖专业垂直词汇，如特定电视片名、区域菜品名称或密集的金融与数据分析术语。面对这些脱节，企业开始构建贴近自身声学环境与行业语料的定制化评测套件。 跨越模型缝隙与全双工交互 (8:38) 在构建自主语音代理与复杂音频管线时，系统失灵往往并非源于底层的转录或合成模型太弱，而是产生在拼接多个离散模型之间的“缝隙”。传统的简单静音超时判定会导致对讲机式的生硬停顿，无法模拟自然的交流节奏。 为此，ElevenLabs研发了深度轮流模型，通过学习人类对话中的声学、语言与韵律线索来实现全双工交互。系统能够在发声的同时持续倾听，在几毫秒内判断用户意图并在说话途中主动让出话语权。消除组件间的割裂感与级联错误，使多模型串联的管线能够输出平滑连贯的交互体验。 规模化落地与配音克隆强度控制 (11:38) 在与Meta合作针对Instagram与Facebook Reels推出自动化配音时，传统配音中配音演员与原始说话者声音完全不同的妥协被打破。新技术旨在跨语言完整复刻创作者原有的声音特质、节奏、情感起伏和音高范围。然而规模化部署揭示了新问题：跨语系（如日语到英语）的纯粹高保真克隆会将源语言的发音习惯、韵律及浓重口音一并带入，导致译文晦涩难懂。 为权衡声音辨识度与目标语言的自然听感，团队设计了0.0至1.0的“克隆强度”滑动参数。设置为0时，系统输出完全标准的目标语言合成音，呈现传统译制片的质感；设置为1.0时，则完全保留源音频的声学细节与原始口音。ElevenLabs将默认参数调整至折中的平衡点，并开放给企业根据内容类型自由调节，使技术参数转变为控制审美与品味的旋钮。 主观表现力、审美趣味与情商正成为音频AI的新竞争前沿。随着纯错误率指标见顶，基于Elo评分的人类偏好竞技场逐渐取代机械测量，模型开始转向捕捉呼吸节奏、戏剧性停顿与情感信服力。然而，如何在保持高保真个人声音身份的同时消弭非母语口音障碍，仍需要在不同场景与文化偏好中持续权衡。

Source quotes

[2:13] Shaheen Lavie-Rouse: The only reason we have keyboards on our phones and on our laptops is transcription has just never been that good at capturing what we're saying and what we're trying to do. [0:05] Shaheen Lavie-Rouse: We are reaching a point where the frontier models are actually more accurate than the data sets that are being used to evaluate them because human transcription and these historical data sets are not always correct. [0:05] Shaheen Lavie-Rouse: For speech to text, the difference between 90% accurate and 99% accurate is the difference between a model you can use to replace your keyboard and a model that like misses every so often the words that you're trying to say and is not so useful.

Original transcript

[0:05] It's 2026 and the pace of AI is [0:07] accelerating in every domain including [0:10] audio AI. In this talk, I'm going to [0:13] talk about what we're seeing at 11 Labs [0:15] at the frontier and how we're working on [0:17] defining the next set of characteristics [0:20] and attributes we want in our models to [0:23] be successful, emotionally intelligent, [0:25] and bring the field to the next level. [0:28] To give a taste of what I'm talking [0:30] about, we can look at speech to text [0:32] benchmarks. Just like 2025 last year, we [0:37] launched our first speech text model. We [0:40] were so proud to hit number one on the [0:42] leaderboard at a 7.7% word error rate. [0:47] But fast forward one year, we launched [0:49] Scribe V2. But if you now look at all [0:52] the rest of the models, you'll see that [0:54] literally every model on the leaderboard [0:57] is ahead of what was number one the year [1:00] before substantially. [1:02] For speech to text, the difference [1:04] between 90% accurate and 99% accurate is [1:09] the difference between a model you can [1:11] use to replace your keyboard and a model [1:13] that like misses every so often the [1:15] words that you're trying to say and is [1:17] not so useful. [1:20] This benchmark though is becoming less [1:22] and less useful itself because it's hard [1:25] to conceive of but actually the [1:27] benchmark test is not 100% accurate. We [1:30] are reaching a point where the frontier [1:32] models are actually more accurate than [1:35] the data sets that are being used to [1:36] evaluate them because human [1:38] transcription and these historical data [1:40] sets are not always correct. And so [1:43] we're even seeing artificial analysis [1:45] who have put out this benchmark start a [1:48] cleanup to kind of improve and improve [1:51] the data quality of the benchmark [1:52] itself. But even more broadly, what [1:56] should we work on next once we hit 2.7% [1:59] word error rate and we're about to go [2:01] into scribe v3 with even better results? [2:04] What should we be optimizing for? So [2:07] that's mainly going to be the topic of [2:08] our talk, how we're identifying that [2:11] next set of attributes. [2:13] Just to quickly introduce 11 Labs and [2:16] this domain of audio AI. If you're not [2:18] familiar, we work on audio AI. That [2:21] means synthetic speech, transcription, [2:24] voice agents, dubbing, music, SFX. We're [2:28] seeing kind of a surge in demand from [2:31] enterprises, developers building on our [2:34] platforms, new voice experiences. And we [2:37] have industry luminaries actually been [2:38] telling us actually from before our lab [2:40] was even founded about the future being [2:43] controlled by voice where humans are [2:46] used to communicating with each other [2:48] with voice. The only reason we have [2:50] keyboards on our phones and on our [2:52] laptops is transcription has just never [2:54] been that good at capturing what we're [2:56] saying and what we're trying to do. [2:58] That's going to change as you see those [3:00] leaderboards improve and the models do [3:03] better. [3:05] At 11 Labs, we work across research and [3:09] we've expanded from our origins and [3:11] research to building products that build [3:14] end-to-end workflows for audio and for [3:17] deployed engineering that work side by [3:19] side with the companies that are [3:21] implementing these at the frontier [3:24] facing the general public. And we've [3:26] been establishing this feedback loop [3:28] where we're able to go from just [3:30] optimizing for being number one on some [3:33] benchmark that's more generic and less [3:36] applicable to a particular user to [3:38] building models that are right for these [3:40] end-to-end workflows. And finally, where [3:43] the rubber hits the road are working [3:45] well in practice for the general public. [3:47] And so we're going to cover this in [3:49] sequence working upwards from from [3:51] research to where is the sit frontier [3:54] what you can do to find your frontier [3:56] but also how we're learning across these [3:58] various areas to make our learning loop [4:01] go faster. [4:03] So 11 Labs started as a again as an [4:06] audio AAI lab almost four years ago and [4:10] started out by launching our text to [4:13] speech and then expanding from there. [4:16] Like I showed we're seeing that with the [4:18] benchmark for speech to text for example [4:21] these measurements of these objective [4:23] metrics defect rates error rates is [4:26] starting to kind of sachiate and [4:29] saturate and hit it the end of its [4:31] useful life. But I still didn't want to [4:33] talk about where you can find the [4:35] frontier for you and where it's standing [4:38] now that that word array benchmark is [4:40] doing so well across the industry. [4:43] First up, we did see a benchmark for [4:45] English. So the moment you even go to [4:48] the second tier of languages, French, [4:50] Spanish, and more, those accuracy rates [4:54] are still not unfortunately not at that [4:57] same level. And we're working on again [5:00] hitting those same marks in Asian [5:02] languages, but even longtail languages [5:05] like Swahili or Catalan and making sure [5:08] that we're getting the same level of [5:10] accuracy. So it's not just you can use [5:12] your voice as your keyboard. You can do [5:14] that no matter what language you're [5:15] speaking in. There's other places too [5:17] where benchmarks still are working on [5:19] better accuracy like diorization where [5:21] again we're not hitting near perfect [5:24] metrics and there's definitely room to [5:26] improve our next models coming out soon. [5:30] Equally importantly though is to figure [5:31] out where benchmarks can kind of lie or [5:33] not be representative of what you might [5:35] see in practice where these benchmark [5:37] are generic. They're running off of high [5:40] quality, clean audio. And your use case, [5:43] your product might have a specific [5:45] jargon like TV shows or restaurant [5:48] dishes or data analysis terms that maybe [5:52] these models aren't capturing as well as [5:53] just general speech. You might have [5:55] users that aren't calling with the best [5:57] audio. They might be in a noisy [5:59] environment. They might be in an [6:00] airport. And again, that was also going [6:03] to impact. So there's definitely these [6:04] kind of edge cases where the benchmarks [6:06] are just not representative. So they're [6:08] kind of lying in a sense and you can [6:10] find your own frontier. And AI just like [6:14] it's accelerating the pace of research [6:15] is actually making it easier and easier [6:17] for you to stand up a very thorough [6:19] evaluation of all of these models to [6:21] find out which one works best for you. [6:24] The most interesting part though where [6:25] I'm going to focus the rest of my talk [6:27] is what you can't even measure with an [6:29] objective metric like word error rate [6:31] and that's applicable to broader set of [6:33] our audio models like music right how do [6:36] you evaluate that here it has to do with [6:38] taste has to do with emotional [6:40] intelligence it has to do with audio [6:42] that's actually able to generate the [6:43] types of conviction the types of [6:45] emotions that the users are trying to [6:49] you're trying to inspire with your [6:50] users. So to give an example, we've [6:53] actually seen text to speech go through [6:54] a similar evolution where maybe 10 years [6:57] ago it was just word error rate. Is it [7:01] just saying the text you gave it? Is it [7:03] speaking the script or is it making up [7:05] like words or skipping or garbling some [7:07] of them? But as we've seen word error [7:10] rate get solved more or less, we've [7:13] moved on to ELO scores to be [7:16] benchmarking TTS where you can now hear [7:18] side by side the same TTS engine, the [7:21] same voice delivering the same exact [7:25] message. This is going to capture a [7:27] preference rating that's capturing all [7:30] the nuances of voice and user [7:32] preferences and tastes. the pacing, the [7:35] intonation, how natural it is, the [7:38] emotional delivery. And we've seen the [7:41] benchmarks evolve like that. And we'll [7:44] play now a couple samples of our own TTS [7:47] generations over the years and see how [7:51] that capability has deolved. [7:53] >> Am I good enough? I'm saying the right [7:56] words. I'm pacing the right way. Is this [7:58] good enough? Am I good enough? I'm [8:01] saying the right words. I'm pacing the [8:03] right way. Is this good enough? [8:07] Am I good enough? I'm saying the right [8:10] words. I'm pacing the right way. Is this [8:13] good enough? I can even sound French. [8:18] >> So, as you can see, I grew up playing [8:20] music. That's something that drew me to [8:22] 11 Labs. your ear is going to get more [8:25] and more challenged to kind of if you [8:27] want to play back and listen and really [8:29] identify those nuances, decide what [8:31] would you vote for and you can go and [8:33] actually do these votes. Now for the [8:35] leaderboards [8:38] the next place where we're working on [8:41] learning more where the frontier lies is [8:44] through our product work. We started [8:46] building models that do specific things [8:49] speech to text, text to speech. What we [8:52] found people build with them is voice [8:55] agents, dubbing creative. [8:58] And so we've been building products and [9:00] platforms to empower builders who are [9:02] building these endto-end experiences and [9:05] catching all these different places that [9:08] fail at the seams. Because a lot of [9:10] these things to be the best in the [9:12] world, it's not enough just to have the [9:14] best speech to text, text to speech, to [9:16] have the best voice agent. There's more [9:18] seams like turn taking, who is talking [9:21] when, interruption handling, [9:24] orchestrating agentic activities like [9:26] tool calls and knowledge lookups that [9:29] are all happening during a conversation. [9:30] If you think about a human agent that [9:32] you're talking to, they're going to [9:34] handle those in different ways than if [9:36] you just kind of are blindly listening, [9:39] talking, listening, talking. An example [9:42] of a critical learning we've had is [9:44] around turn caking and interruption [9:46] handling where you can just use a simple [9:50] method to do that and stitch the metals [9:52] together. But we've been training just [9:54] like our speech text model generations, [9:57] turn model generations that are actually [9:59] deep trained models that are capturing [10:02] the nuances of turn in real life and [10:05] able to deliver a better experience. [10:07] Let's listen to an example. ready for [10:10] your talk? Isn't it like tomorrow? [10:14] >> Yeah. Uh I I was wondering if you can [10:16] help me. Should I show a demo [10:18] conversation with a voice agent for this [10:20] talk? [10:21] >> Oh, absolutely. Nothing proves how far [10:24] >> Wait, wait. Specifically, I want to see [10:26] like turn taking, interruption handling. [10:28] Is that kind of something that you think [10:30] needs a live demo? [10:33] >> Oh, see, you just did it. That's exactly [10:36] what modern voice agents do. They listen [10:38] while they speak, detect when you want [10:40] the floor, and stop mid-sentence to let [10:42] you jump in, just like you did with me. [10:45] It's the perfect thing to show live [10:47] because hearing that natural flow is way [10:49] more convincing than just describing it. [10:52] >> Awesome. Yeah, we're just doing the live [10:54] demo right now. [10:56] >> Wait, are you recording this right now? [11:00] Oh, no. Okay. Hi, everyone at scale. You [11:03] are going to crush this talk tomorrow. [11:06] So take a deep breath and let's make [11:08] some noise. [11:10] >> It's not just turn taking and just [11:12] agents. [11:14] Every use case we've found where we [11:16] chain together models or we're actually [11:18] trying to now serve users who are trying [11:20] to achieve the end goals they have with [11:21] audio whether that's agents dubbing [11:24] creative has these kinds of seams. And [11:27] we've been investing in identifying the [11:29] next frontier of models to train, not [11:32] just to help with each step, but to help [11:34] and cross over those seams. [11:38] Finally, I'm going to speak to how for [11:41] deployed engineers like myself are [11:43] working side by side with the builders [11:46] that are integrating these voice and [11:49] audio models into their products and [11:52] bringing them to the general public. So [11:54] we can actually see when the rubber [11:56] meets the road. How well does this work [11:58] in each particular use case in each [12:01] particular domain in each particular [12:03] user setting and discover through here [12:06] the next phase of ideas for our [12:09] researchers and for our product builders [12:12] on what we need to get done. [12:15] I'm going to use an example here on what [12:18] we've learned from our partnership [12:20] working together with Meta with the [12:22] teams at Instagram reels and Facebook [12:24] reels on dubbing content. [12:27] When you hire a human voice actor to [12:29] dub, it's always going to be a different [12:31] person who's voicing the dub compared to [12:33] the original speaker. But our vision for [12:36] AI dubbing lets us clone your voice in [12:39] one language and have it speak in the [12:42] target language and preserve all of the [12:45] aspects that make your voice yours. Your [12:48] pace, your inflections, your pitch range [12:52] and find the closest voice is going to [12:54] talk just like you in the target [12:56] language. It's 2026 and the pace of AI [13:00] is accelerating in every domain [13:02] including audio AI. [13:18] We were trying to see some patterns [13:20] where the realism was too real. So you [13:23] can imagine when you're taking languages [13:25] that are close together or a speaker [13:27] maybe that's voice sounds like it could [13:28] really work in multiple languages. This [13:30] can work really well and you get that [13:32] uncanny resemblance between the origin [13:35] and the target. But if you're taking [13:38] languages that are much farther apart, [13:40] you can get a trade-off between [13:43] retaining that original speaker's [13:45] character, but also retaining parts of [13:48] the speech that are not great in the [13:49] target language, like an accent. [13:52] And so accent retention really surfaced [13:55] as a problem as we're heading in to this [13:58] summer and expanding some of the [13:59] language sets that we were working. So [14:02] to address that, we develop the cloning [14:03] strength parameter. At a full cloning [14:06] strength, that's the same experience [14:07] that we were talking about before. We [14:09] just take your voice, we clone it, and [14:12] we make it talk in the target language. [14:14] And by the way, if we just have a sample [14:16] of you talking in a language like [14:17] Japanese, and we're going to have you [14:19] talk and render speech in English, that [14:22] voice clone, again, it might not be the [14:24] best most natural voice in English. It [14:26] might retain some of the accent. and it [14:28] might retain some of their [14:29] characteristics of Japanese speech when [14:31] it's talking in the target. At a client [14:34] of zero, we are generating a purely [14:36] synthetic voice that's of a pure natural [14:39] speaker in the target language that is [14:42] much farther away from your voice. And [14:45] so this toggle lets us fix these kinds [14:48] of issues, but also it's become a dial [14:51] for taste. [14:53] Low cloning strength sounds more like [14:55] that traditional dub with an expressive [14:57] theatrical voice actor coming in on top [15:00] speaking very cleanly and clearly on top [15:02] of what you said. And at full strength, [15:04] that's our original vision of realistic [15:08] audio. If you have poor quality audio [15:10] coming in, it will be reflected in the [15:12] output. If you have a weird way of [15:15] speaking, it will be reflected in the [15:16] output and it might not be so polished, [15:18] but it will be fully realistic. And we [15:20] actually started dialing our own [15:22] defaults down to be somewhere more in [15:24] the middle compared to what we were [15:26] doing before and introducing this as a [15:28] lever for our partners in dubbing across [15:31] all industries to be able to toggle to [15:33] get the right style and results that [15:35] they're looking for. Let's hear what [15:37] that looks like in practice. [15:50] audio. [16:01] To conclude, I did want to talk about [16:04] where we're going next. Like I was [16:06] saying, we're really seeing some of [16:08] these objective error-based metrics [16:12] saturate and finding the new frontier in [16:14] taste based and emotional intelligence. [16:19] We're having our next batch of model [16:20] announcements on 1111 and so you can [16:23] look forward to what some of those next [16:25] generation models will sound like. I [16:28] wanted to leave with a quick preview of [16:32] a dub that especially for me as a [16:34] musician was important to make and to [16:37] show a bit about what you can listen for [16:41] again to try try to understand where [16:44] we're at and hopefully where we're [16:45] going. [16:52] A lamps low and the night leans in. The [16:55] room is soft with a blue kind of light. [17:22] Thank you so much for listening to the [17:24] talk. [17:25] This is 2026 and hopefully next year [17:28] we'll have even more advances that we [17:31] can share. I wonder where we'll be with [17:34] the pace that everything is moving at, [17:36] but I hope you've learned something from [17:38] this and that you'll find your audio use [17:41] case moving forward for next year. [17:45] Thanks so much.