Hugging Face Launches Open TTS Leaderboard for Scalable Multilingual and Voice Cloning Evaluation

Hugging Face Launches Open TTS Leaderboard for Scalable Multilingual and Voice Cloning Evaluation Hugging Face has introduced the Open TTS Leaderboard to address fragmented text-to-speech evaluation, using objective metrics across intelligibility, latency, and speaker similarity to benchmark open-source models rapidly. Arena-based human evaluations cannot scale with the more than 8,000 text-to-speech models on the Hugging Face Hub, leaving open-weights models underrepresented. The new leaderboard reduces evaluation times from weeks to hours by using Qwen3 ASR for error rates, H200 GPU and CPU timings for speed, and WavLM embeddings for voice cloning similarity. Evaluations span English and multilingual datasets including Seed TTS Eval and CV3 Eval, while dedicated tabs assess streaming time-to-first-audio and user listening tests. Addressing Arena Scalability with Objective Metrics While human preference scoring remains the gold standard, arena leaderboards struggle to scale and host open models, resulting in open-weights systems making up only 16 of the 92 models on Artificial Analysis as of September 30, 2026. The Open TTS Leaderboard complements human preference by testing objective dimensions: intelligibility via word or character error rates using Qwen3 ASR, batched speed via inverse real-time factor (RTFx), and speaker similarity (SIM) using WavLM embeddings. This objective approach lowers model evaluation duration from weeks to hours. Multilingual Support and Voice Cloning Capabilities The leaderboard benchmarks macro-average WER across English splits from Seed TTS Eval and CV3 Eval, where hexgrad/Kokoro-82M, Supertone/supertonic-3, and fishaudio/s2-pro lead. To account for non-English performance, users can filter by multiple languages, applying character error rate (CER) metrics for Chinese, Japanese, and Korean. Enabling the voice cloning filter displays speaker similarity scores and tradeoffs, highlighting models such as k2-fsa/OmniVoice and FunAudioLLM/Fun-CosyVoice3-0.5B-2512. Streaming Latency Benchmarks and Interactive Listening A dedicated Streaming tab evaluates interactive latency by measuring median time-to-first-audio (TTFA) across English prompts from CV3-Eval at batch size 1 on H200 GPUs and CPUs, highlighting systems like kyutai/pocket-tts. Additionally, the Listen tab lets users inspect and vote on model audio outputs directly to bridge objective metrics with human perception, alongside plans to open-source the underlying evaluation scripts.

Hugging Face推出Open TTS Leaderboard：面向开源多语言语音合成与克隆的评测基准 面对开源文本转语音（TTS）模型快速增长而传统竞技场式人工评测难以扩展的现状，Hugging Face推出了Open TTS Leaderboard。该榜单采用客观指标大幅缩短评测时间，覆盖多语言支持、声音克隆以及流式推理延迟评测。 竞技场式人类偏好评估扩展受限，开源模型代表性不足，而Open TTS Leaderboard将模型评测耗时从数周缩短至数小时。 评测采用Qwen3 ASR计算WER/CER衡量清晰度，利用RTFx衡量离线推理速度，并基于WavLM计算说话人余弦相似度。 支持多语言及声音克隆功能对比，Kokoro-82M与OmniVoice等模型在不同场景下表现突出。 提供针对交互式场景的流式首包音频时间（TTFA）基准，并设有“Listen”标签页供社区试听与投票。 背景与客观评测维度 随着Hub上TTS模型数量激增，传统的竞技场（Arena）人类偏好评测受限于算力托管与投票周期，开源权重模型往往代表性不足。Open TTS Leaderboard通过引入客观指标作为互补，显著提升评测效率：使用Qwen3 ASR计算生成的文字错误率（WER/CER）评估清晰度；在H200 GPU上测试逆实时率（RTFx）衡量批处理速度；并通过WavLM嵌入特征的余弦相似度（SIM）评估声音克隆的说话人还原度。 多语言与声音克隆表现 该榜单默认基于Seed TTS Eval与CV3 Eval的英文切片进行宏平均WER排名，hexgrad/Kokoro-82M、Supertone/supertonic-3和fishaudio/s2-pro领跑英文表现。由于英文能力无法直接外推至其他语种，榜单支持多语言切换，中日韩等字符语言采用CER统计，k2-fsa/OmniVoice等模型在多语言上表现强劲。开启“Voice cloning”后，部分模型（如higgs-tts-3-4b和VoxCPM2）在引入参考音频时WER进一步改善。 流式推理与试听投票机制 针对语音助手等交互应用，榜单设置了“Streaming”标签页，使用批大小为1的统一英文提示词测量首包音频到达时间（TTFA），kyutai/pocket-tts在GPU和CPU上的流式指标表现优异。此外，榜单设立“Listen”页面允许用户对比实际音频并进行防刷票认证投票，官方计划随后开源全部评测脚本并依据社区反馈持续迭代。

TLDR 👉 new TTS leaderboard focused on open-source and multilingual The pace of open-source text-to-speech (TTS) model releases has been incredible. On the Hugging Face Hub (as of Sep 30, 2026) there are more than 8K TTS models available 🚀 Evaluation, however, hasn't kept pace: it remains fragmented and unstandardized. The gold standard is human preference scores such as MOS or MUSHRA (more on metrics ). To this end, several arena-based leaderboards have established themselves as useful reference points for the community: TTS Arena v2 Artificial Analysis Voice Arena These arenas compare models by presenting users with TTS outputs from two models, and asking them to choose one over the other. After collecting a sufficient number of votes, an Elo score is computed to rank models, typically with the Bradley–Terry model (see Voice Arena methodology ). While human preference is the ultimate decider, arenas cannot scale to keep up with the pace of TTS releases . This may partly explain why open-source models are underrepresented on arena-style leaderboards: as of Sep 30, 2026, only 16 of the 92 models on Artificial Analysis are open-weights, with a similar skew on Voice Arena . This likely reflects practical factors: adding an API model requires little more than an API key, whereas an open model must be hosted and served by the arena operator, and commercial providers have more reason to seek placement than open-source authors. Another limitation with arena-style evaluation is voter consistency: no arena can ensure that the same voters with the same criteria of “better” can consistently evaluate models over time. Even the preferences of a single person change over time (“A man cannot step into the same river twice” as famously said by Heraclitus). To this end, we've built the Open TTS Leaderboard , which uses objective metrics to evaluate models on complementary aspects of performance: Intelligibility : word/character error rate (WER and CER) between the prompt and the generated audio's transcript, using Qwen3 ASR (top ranking open-source model on the Open ASR Leaderboard ). Speed : inverse real-time factor (RTFx) for batched offline inference on an H200 GPU, and time-to-first-audio (TTFA) for quantifying streaming batch size 1 latency on an H200 GPU and CPU. Speaker similarity by computing the cosine similarity (SIM) between WavLM speaker embeddings of the generated audio and the reference clip. By relying on objective metrics evaluating a model drops from a couple weeks (for collecting votes) to a couple hours ⚡ Importantly, the Open TTS Leaderboard does not replace human preference ranking. ASR-based WER provides a proxy for intelligibility, while speaker similarity estimates voice identity preservation. Neither directly measures naturalness, expressiveness, or listener preference. Nevertheless, they can even inform voting-based leaderboards which models to include in their evaluations. Our intention with this leaderboard is for it to be shaped by the community ; we want to hear your feedback so the evaluations stay relevant and insightful. The next few sections give an overview of main features of the Open TTS Leaderboard. Multilingual + voice cloning evaluation From the default view of the leaderboard, models are ranked by macro-average WER on the English splits of Seed TTS Eval ( paper ) and CV3 Eval (zero shot) ( paper ). hexgrad/Kokoro-82M , Supertone/supertonic-3 , and fishaudio/s2-pro lead the pack on English WER when averaged on these two splits, while the Pareto plots visualize which models strike a good balance between WER, batched inference (RTFx), and size. English performance doesn't necessarily translate to other languages. Multiple languages can be toggled to rank models on multilingual performance. Seed TTS Eval only has audio for English and Chinese, so the other languages are simply the score on CV3 Eval (zero shot). Note that Chinese, Japanese, and Korean are character-based languages and so character error rate (CER) is reported, and the “Average WER” across languages is a macro-average across languages. k2-fsa/OmniVoice , fishaudio/s2-pro , and FunAudioLLM/Fun-CosyVoice3-0.5B-2512 are strong multilingual models. By toggling “Voice cloning”, the models that support this functionality (on the selected languages) can be compared. Moreover, a SIM column for speaker similarity now appears in the table, as well as two more Pareto plots for visualizing the tradeoff between SIM, batched inference, and size. The average WER of some models, such as bosonai/higgs-tts-3-4b and openbmb/VoxCPM2 , improve under voice cloning, namely when a reference audio is provided. Compare and vote on TTS outputs Numbers only tell part of the story, and as mentioned earlier human preference is the ultimate decider . From the “Listen” tab, you can compare the generated outputs that are behind the metrics, to find which model(s) you prefer! Pick the language/dataset you're interested in, whether you want to compare voice cloning , and optionally pick the models or listen to outputs from a random selection. The “Listen” tab fills an important gap in existing TTS leaderboards: a space to explore model outputs of various models. You can even give feedback on the generated outputs. As we collect more votes from the community, we may include this data on the leaderboard. So vote! But please login with your HF account to help us weed out spam/bots. Streaming performance The “Streaming” tab compares the streaming capabilities. Models are ranked by TTFA (time-to-first-audio), which quantifies how long a user waits after probing a model in order to obtain audio that can be played. This is important for voice agents and other interactive apps. For streaming models (✅ under “Streaming API”) it's the time until the first audio chunk arrives. For non-streaming models, it's the time until the whole utterance is generated, because playback can't start any earlier. Every model runs one audio at a time (batch size 1), on the same 50 English prompts from CV3-Eval, on the same hardware and in its default voice. We drop the first 3 runs as warm-up and report the median TTFA across the rest. The default view compares performance on an H200 GPU. Results are also available for CPU for a small (but growing) set of models! kyutai/pocket-tts is a great model for streaming on both GPU and CPU! Conclusion The goal of the Open TTS Leaderboard is not only to keep up with the incredible pace of TTS model releases, but to be shaped by the community; we want to hear your feedback so the evaluations stay relevant and insightful. Let us know which datasets, models, and metrics you want to see! For now, we've focused on: Open-source models , to put forward many great models that have been neglected by arena-style evaluations. Multilingual , since English performance is not a suitable proxy for other languages. We will soon open-source the evaluation scripts, much like the Open ASR Leaderboard repo , so that you can directly provide your feedback and suggestions via GitHub Issues and PRs! Let's shape TTS evaluations together 🤗