Microsoft Cuts Speech Errors 30% With Neural Nets in Live Chinese Demo

Microsoft Research

Deploying deep neural networks breaks a multi-decade plateau in speech recognition by cutting error rates to 15 percent, enabling real-time translation that speaks fluent Mandarin in a user's own voice.

AI-assisted summary of Speech Recognition Breakthrough for the Spoken, Translated Word — original source. Check the original for full context.

Source published: 2012-11-08

Key ideas

Microsoft Research and the University of Toronto applied deep neural networks (DNNs) to speech recognition, cutting word error rates by approximately 30%—reducing mistakes from roughly one in four words to one in seven or eight. (2:36)
The breakthrough overcomes a decades-long performance plateau in statistical Hidden Markov Models (HMMs), which had powered commercial systems like Siri and Kinect but struggled with unconstrained speech. (0:00)
A live translation pipeline integrates DNN speech transcription with a two-step Chinese translation engine that reorders syntax to match authentic native phrasing. (4:36)
Using roughly one hour of Rick Rashid's recorded English voice, researchers modulated a native Chinese text-to-speech engine to output spoken Mandarin that retained his unique vocal timbre. (6:31)

Detailed notes

## Background Published by Microsoft Research on YouTube on November 8, 2012, this video showcases Chief Research Officer Rick Rashid demonstrating a major speech recognition and machine translation breakthrough. The page description highlights how deep neural networks significantly reduce speech-to-text and translation errors, ultimately converting spoken English directly into computer-generated Chinese.

## TL;DR Rick Rashid traces the historical evolution of speech recognition from early brittle acoustic pattern matching through Hidden Markov Models to a major breakthrough using deep neural networks (DNNs) developed with the University of Toronto. This DNN architecture slashes word error rates by roughly 30%, shifting accuracy from 20–25% error down to approximately 15% or less. Rashid then details an automated machine translation pipeline that converts transcribed English into grammatically reordered Chinese text. In a live stage demonstration, the system processes Rashid's spoken English sentences and synthesizes them aloud in fluent Mandarin Chinese while preserving his personal vocal timbre using just one hour of training audio.

## Atmosphere & style The presentation carries the measured, academic gravitas of a high-stakes tech keynote, grounded in historical perspective before shifting into quiet astonishment. Rashid speaks with the calm, methodical clarity of a veteran computer scientist, acknowledging the technology's imperfections rather than over-hyping them. When the system replays his exact vocal timbre speaking natural Mandarin after a brief processing pause, the audience's spontaneous applause infuses the room with genuine awe and hopeful optimism for tearing down global communication barriers.

## Bulletin - [02:36](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=156s) Microsoft Research and the University of Toronto applied deep neural networks (DNNs) to speech recognition, cutting word error rates by approximately 30%—reducing mistakes from roughly one in four words to one in seven or eight. - [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) The breakthrough overcomes a decades-long performance plateau in statistical Hidden Markov Models (HMMs), which had powered commercial systems like Siri and Kinect but struggled with unconstrained speech. - [04:36](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=276s) A live translation pipeline integrates DNN speech transcription with a two-step Chinese translation engine that reorders syntax to match authentic native phrasing. - [06:31](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=391s) Using roughly one hour of Rick Rashid's recorded English voice, researchers modulated a native Chinese text-to-speech engine to output spoken Mandarin that retained his unique vocal timbre.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) Early speech recognition systems attempted to match acoustic waveforms directly against templates, failing because individual voices and contextual pronunciations vary widely. - [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) Carnegie Mellon University revolutionized speech research in the late 1970s by introducing statistical modeling via Hidden Markov Models (HMMs). - [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) HMM-based statistical models drove three decades of advancement and enabled widespread tools like automated banking agents, Apple's Siri, and Microsoft's Kinect. - [02:36](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=156s) State-of-the-art HMM systems eventually plateaued, exhibiting unconstrained word error rates between 20% and 25%. - [02:36](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=156s) Microsoft Research partnered with the University of Toronto to implement deep neural networks modeled after biological brain processes to handle massive volumes of training data. - [02:36](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=156s) Deploying deep neural networks dropped word error rates by roughly 30%, shifting speech recognition accuracy down to 15% error or less. - [04:36](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=276s) Modern machine translation utilizes big-data web corpora and statistical methods to power large-scale engines like Bing Translator. - [04:36](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=276s) Translating spoken English into Chinese requires a two-step process: mapping English terms to Chinese vocabulary, followed by syntactic reordering to accommodate divergent grammar. - [06:31](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=391s) Microsoft engineers synthesized personalized speech by using approximately one hour of recorded English audio to modulate a native Chinese text-to-speech model. - [06:31](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=391s) A live demonstration showed English spoken from the stage being transcribed, translated, and played back as fluent Mandarin Chinese preserving Rick Rashid's vocal identity. - [06:31](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=391s) While acknowledging that speech recognition and translation errors still occur, Rashid underscored the potential of speech-to-speech systems to break down global communication barriers.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) The Evolution of Speech Recognition: From Waveforms to Hidden Markov Models

Rick Rashid returns to the core subject of natural user interfaces, noting that spoken language represents one of the most intuitive ways humans interact with the world. Despite computer scientists dedicating over six decades to understanding and recognizing human speech, the early days of research stumbled over fundamental architectural challenges. The earliest systems approached speech purely as a pattern-matching problem, attempting to map raw acoustic waveforms from an individual’s voice directly onto pre-recorded template waveforms representing distinct words. This method proved extremely brittle and impractical in real-world scenarios. Beyond the reality that every individual’s voice sounds distinct, even a single speaker rarely pronounces the same word the same way twice, subtly shifting inflection, cadence, and acoustic signatures based on adjacent words, mood, or conversational context.

A pivotal turning point occurred during the late 1970s at Carnegie Mellon University, where researchers discarded pure acoustic template matching in favor of statistical modeling techniques, specifically Hidden Markov Models (HMMs). By training on large corpora of voice samples drawn from many different speakers, HMMs allowed computers to build far more robust statistical representations of speech sounds and language transitions. Over the following three decades, this approach drove substantial advances, elevating speech recognition from laboratory experiments to functional consumer and enterprise tools.

Although these systems still produced errors, they became sufficiently dependable to power constrained, task-oriented interfaces. Rashid points out how commonplace automated speech agents had become: calling a bank in the United States routinely pairs the caller with an automated system capable of resolving account queries or routing complex problems to a human representative. Similarly, commercial products like Apple's Siri answered user queries, and Microsoft's Kinect gaming peripheral enabled voice navigation and mid-game command execution.

- Early pattern-matching systems failed because speaker waveforms vary wildly across individuals and fluctuate depending on surrounding phonetic context. - Carnegie Mellon University pioneered the application of Hidden Markov Models (HMMs) to speech recognition in the late 1970s. - HMM-based statistical models powered widespread consumer applications, including automated banking call centers, Apple Siri, and Microsoft Kinect voice commands.

---

[02:36](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=156s) The Deep Neural Network Breakthrough and Error Rate Reductions

Despite thirty years of steady refinement, statistical HMM architectures eventually hit a performance plateau. When handling arbitrary, unconstrained speech, state-of-the-art systems routinely suffered from word error rates in the 20% to 25% range—meaning roughly one out of every four or five words was misidentified. To shatter this barrier, researchers from Microsoft Research joined forces with collaborators at the University of Toronto, turning to an architecture patterned after biological human brain processes: deep neural networks (DNNs).

By deploying multi-layered neural networks capable of ingesting and modeling dramatically higher volumes of training data than classical HMM setups could tolerate, the research team achieved an unprecedented performance leap. The deep neural network architecture reduced word recognition errors by roughly 30%. Rashid puts this achievement into concrete perspective: it moved speech recognition error rates from roughly 20–25% down to approximately 15% or less, shifting accuracy from one mistake every four or five words to only one mistake every seven or eight words.

While acknowledging that automated transcription remains imperfect and requires extensive continued development, Rashid underscores the magnitude of the improvement:

> “That one change, that particular breakthrough increased recognition rates by approximately 30%; that's a big deal.”

---

[04:36](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=276s) Machine Translation and the Two-Stage Chinese Text Pipeline

Running parallel to speech recognition, the problem of automated machine translation has engaged computer scientists for over sixty years. In recent years, pairing statistical methods with massive big-data web corpora unlocked unprecedented translation quality, enabling systems like Microsoft’s Bing Translator to serve millions of translated pages every day to users worldwide in their native tongues.

Applying these capabilities to live speech involves a structured translation pipeline. When an English speaker talks, the recognized text is sent through a two-step translation engine. First, the system performs an initial transformation of the English words into Chinese vocabulary. Second, because Chinese grammar and syntax diverge sharply from English, the engine parses and reorders the translated Chinese text to mirror authentic phrasing that a native Chinese speaker would naturally utter. Rashid explains this real-time pipeline while initiating a live screen demonstration that projects the transcribed English sentences alongside the rearranged Chinese character output.

```mermaid flowchart LR accTitle: Speech-to-Speech Translation Architecture accDescr: Pipeline showing raw English audio converting to text, translating and reordering into Chinese text, and synthesizing into cloned Chinese audio. subgraph ASR["1. Speech Recognition"] direction TB A["English Spoken Voice"] --> B["Deep Neural Network Model"] B --> C["Transcribed English Text"] end subgraph MT["2. Machine Translation"] direction TB C --> D["Word-by-Word Chinese Translation"] D --> E["Syntactic Reordering for Chinese Grammar"] end subgraph TTS["3. Voice-Cloned Synthesis"] direction TB E --> F["Native Chinese TTS Base Model"] G["1 Hour of Speaker Voice Audio"] --> H["Voice Modulation Model"] F --> H H --> I["Synthesized Chinese Output in Speaker's Voice"] end ASR --> MT MT --> TTS ```

---

[06:31](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=391s) Live Voice-Cloned Chinese Speech Demo and Vision for the Future

Rashid introduces the final and most dramatic capability of the system: moving beyond translated text on a display to speaking fluent Chinese aloud while preserving the speaker's personal vocal identity. To achieve this, Microsoft Research engineers built a high-quality Chinese text-to-speech (TTS) system trained on extensive voice data from native Chinese speakers. They then took approximately one hour of Rashid’s recorded English speaking voice and used it to modulate the timbre, pitch, and acoustic characteristics of the underlying Chinese synthesizer.

As Rashid speaks English sentences from the stage, the system processes his phrases, and after short pauses, his own voice speaks back to the audience in fluent Mandarin Chinese. The live demonstration elicits enthusiastic applause from the crowd as the synthesized audio mimics his vocal signature while pronouncing Mandarin tones.

Rashid readily admits that the combined pipeline of DNN-based recognition, statistical translation, and voice personalization still makes errors and leaves significant engineering work ahead. However, he emphasizes the transformative human potential of dismantling communication divides across the globe, closing the session with heartfelt remarks and a Mandarin sign-off:

> “Personally I believe this is going to lead to a better world.”

- The personalized text-to-speech engine required only approximately one hour of Rick Rashid's recorded English voice to modulate a native Chinese voice model into his own vocal likeness. - The presentation demonstrates a complete end-to-end loop: spoken English input $\rightarrow$ deep neural network transcription $\rightarrow$ grammar-adjusted Chinese translation $\rightarrow$ personalized vocal synthesis output. - Rashid concludes his live keynote address by thanking the audience and offering a warm "Xièxie" (谢谢).

## Mentioned Rick Rashid, Microsoft Research, Carnegie Mellon University, University of Toronto, Apple, Siri, Microsoft Kinect, Bing Translator

## Video description Chief Research Officer Rick Rashid demonstrates a speech recognition breakthrough via machine translation that converts his spoken English words into computer-generated Chinese language. The breakthrough is patterned after deep neural networks and significantly reduces errors in spoken as well as written translation.

For more information on Speech Recognition and Translation, visit http://www.microsoft.com/translator/skype.aspx

[Source](https://www.youtube.com/watch?v=Nu-nlQqFCKg)

中文简报

微软现场中文演示：神经网络将语音识别错误率降低30% 应用深度神经网络打破了数十年语音识别技术的停滞，将错误率降至15%，实现以用户原声进行流利普通话实时翻译。 Rick Rashid回顾了语音识别的历史演变，从早期脆弱的声学模式匹配，到隐马尔可夫模型，再到与多伦多大学合作利用深度神经网络（DNN）实现的重大突破。该DNN架构将词错误率降低了约30%，使错误率从20%–25%降至约15%或更低。Rashid随后详细介绍了自动化机器翻译流程，可将转录的英文转换成经过语法重排的中文文本。在现场舞台演示中，该系统处理了Rashid口述的英文句子，并仅使用一小时的训练音频，在保留其个人音色的同时，合成了流利的普通话语音朗读。微软研究院与多伦多大学将深度神经网络（DNN）应用于语音识别，将词错误率降低约30%——错误由大约每四词错一个减少至每七或八词错一个。 (2:36) 该突破打破了统计隐马尔可夫模型（HMM）长达数十年的性能停滞；HMM曾驱动Siri和Kinect等商业系统，但在应对非限制性语音时表现受限。 (0:00) 实时翻译流程将DNN语音转录与两步中文翻译引擎相结合，重构语序以契合地道的母语表达。 (4:36) 研究人员利用Rick Rashid大约一小时的英文录音，对原生中文文本转语音引擎进行调制，输出保留其独特音色的普通话语音。 (6:31)

Reading summary

Microsoft Research demonstrated an end-to-end speech-to-speech translation pipeline that combines deep neural networks, machine translation, and personalized voice synthesis to translate spoken English into Mandarin spoken in the presenter's own voice. From Acoustic Templates to Hidden Markov Models (0:00) Early speech recognition research treated audio processing as a direct pattern-matching challenge, attempting to match an individual's acoustic waveforms against pre-recorded word templates. This framework failed in real-world environments because human vocal signatures vary substantially between people, and even a single speaker alters pronunciation, inflection, and acoustic profiles depending on adjacent words and context. A critical transition occurred in the late 1970s at Carnegie Mellon University with the adoption of Hidden Markov Models (HMMs). By training on large multi-speaker voice datasets, statistical models captured general sound properties and linguistic transitions. Over the following three decades, HMM architectures enabled commercial speech tools for constrained tasks, including automated banking call centers, Apple's Siri assistant, and voice controls on Microsoft's Kinect. Breaking Error Plateaus with Deep Neural Networks (2:36) Despite decades of incremental improvements, statistical HMM systems eventually stalled on arbitrary, unconstrained speech, maintaining word error rates between 20% and 25%. This meant conventional systems misidentified roughly one out of every four or five words. To overcome this threshold, Microsoft Research collaborated with researchers at the University of Toronto to implement deep neural networks (DNNs), an architecture modeled on human brain processes capable of training on substantially larger data volumes. According to Rick Rashid, shifting to DNNs reduced word recognition errors by approximately 30%, dropping overall error rates to 15% or below and decreasing mistakes to roughly one in every seven or eight words. Two-Stage Syntactic Reordering for Chinese Translation (4:36) Translating transcribed speech in real time required leveraging statistical machine translation trained on large-scale web corpora, similar to the architecture powering Microsoft's Bing Translator. Moving from English to Mandarin Chinese required a specialized two-stage text pipeline. The system first converts recognized English vocabulary into equivalent Chinese terms. Because Chinese syntax diverges sharply from English, the engine then executes a second parsing pass to reorder the translated words into grammatically natural phrasing appropriate for a native Chinese speaker before displaying or synthesizing the output. Cross-Lingual Voice Cloning and Real-Time Demonstration (6:31) Rather than relying on a generic synthetic voice, the platform personalized text-to-speech output to preserve the original speaker's vocal traits across languages. Engineers constructed a base text-to-speech model trained on native Chinese speakers, then used approximately one hour of Rashid's recorded English speech to modulate acoustic features such as pitch and timbre. During a live demonstration, Rashid spoke English sentences from the stage, and the system transcribed, translated, and broadcast the sentences back to the audience in Mandarin Chinese mimicking his own vocal likeness. The complete loop connected spoken English input, DNN transcription, grammatical Chinese translation, and personalized vocal synthesis. Rashid acknowledged that despite the 30% accuracy gain, automated transcription and multi-stage translation pipelines remain imperfect and require substantial ongoing engineering. He argued, however, that removing language barriers through automated voice-preserving translation will fundamentally improve global communication.

精炼阅读摘要

微软研究院（Microsoft Research）的里克·拉希德（Rick Rashid）展示了一套端到端跨语言语音翻译系统，该系统结合深度神经网络语音识别、两阶段机器翻译与声音克隆合成，旨在打破跨语种口语交流障碍。从模板匹配到隐马尔可夫模型的技术演进 (0:00) 人类探索计算机语音识别已历经六十余载。早期的模式匹配架构试图将说话人的原始声学波形与预先录制的模板波形进行直接映射。这种方法极易失效且缺乏实用性，因为不仅不同说话者之间的音色存在巨大差异，即便是同一位发言者，其音调、节奏和声学特征也会随语境、情绪和相邻词汇产生细微变化。直到1970年代末，卡内基梅隆大学（Carnegie Mellon University）开创性地引入隐马尔可夫模型（HMMs），利用海量多说话人语音语料库建立统计表征，才促使语音识别从实验室走向实用化。在此后三十年中，基于HMM的统计模型逐步渗透至银行业呼叫中心自动问答、苹果Siri语音助理以及微软Kinect游戏体感导航等限定场景的消费级与企业级应用中。深度神经网络显著降低词错率 (2:36) 尽管HMM架构经过三十年的持续优化，但在处理任意且无限制的自然口语时，其词错率长期停滞在20%至25%的区间，相当于每讲四到五个词就会出现一次识别错误。为了突破这一瓶颈，微软研究院与多伦多大学合作，引入了模拟人脑处理机制的深度神经网络（DNN）。DNN具备容纳和建模超大规模训练数据的能力，使词识别错误率降低了约30%。拉希德指出，这一突破将整体错误率从原先的20%–25%压低至约15%或更低水平，相当于将出错频率减少到每七八个词仅错一次。拉希德坦言自动转录依然存在不完美之处且需持续研发，但他强调该幅度提升是一项极为重要的技术跃进。两阶段机器翻译实现句法重排 (4:36) 伴随语音识别同步发展的机器翻译技术同样经历了六十余年的探索。近年来，统计方法与互联网大数据语料库的融合大幅提升了翻译质量，支撑了微软必应翻译（Bing Translator）等面向全球用户的规模化翻译服务。在将该能力接入实时口语流程时，系统采用了两阶段处理管线：第一阶段将转录得到的英文词汇初步转化为对应的中文词汇；第二阶段针对中英文之间显著的语法和句法差异，对翻译出的中文词汇进行解析与重新排序，以符合母语使用者的自然表达习惯。在现场演示中，系统能够实时投射转录的英文句子并同步呈现重排后的中文字幕。声音克隆驱动的个性化中文语音合成 (6:31) 该系统在文本翻译的基础上进一步实现了保留说话人音色特征的语音输出。微软研究院的工程师首先基于母语发音人的海量语音数据构建了一套高品质中文文本转语音（TTS）基础系统，随后仅采集了拉希德约一小时的英文演讲录音，据此提取其音色、音调等声学特征，并对中文合成模型实施调制。在现场演示中，系统接收拉希德的英文发言后，经过短暂运算，即以其本人的原声声线播报出流利的普通话。该环节完整展示了从英文口语输入、深度神经网络识别、语法调整翻译，到个性化中文合成输出的全链路整合。拉希德承认，由深度神经网络识别、统计机器翻译及个性化语音合成构成的复合管线目前仍会产生错误，后续尚存大量工程优化工作；但他认为消除全球语言沟通壁垒具有深远价值，并坚信这项技术终将推动世界走向更紧密的联结。

Detailed notes — zh

## Background 该视频由微软研究院（Microsoft Research）于2012年11月8日发布在YouTube平台上。视频内容记录了微软首席研究官 Rick Rashid 现场展示的一项重大技术突破，系统依托深度神经网络与机器翻译，将他即时口述的英文演讲转译为具有其个人音色的中文语音，显著削减了语音识别与文本翻译的错误率。 ## TL;DR Rick Rashid 首先回溯了语音识别六十年的技术演进，指出从早期的波形模式匹配到卡耐基梅隆大学引入的隐马尔可夫模型，传统统计方案在应对不受限的任意自然语音时，词错误率长期停滞在 20% 到 25% 的瓶颈期。随后，微软研究院联合多伦多大学引入深度神经网络（DNN），大幅提升了声学模型的数据处理能力，将识别错误率锐减至 15% 左右，实现了识别率提升约 30% 的历史性跨越。结合必应翻译的统计重排算法，团队利用 Rashid 约一小时的英文演讲录音调制中文文本转语音（TTS）系统，成功克隆出其个人声线。在震撼的现场演示中，系统近乎实时地将他的英文演讲同步转化为同音色的中文语音播报，生动展现了未来彻底打破人类跨语言沟通壁垒的技术前景。 ## Atmosphere & style 整场演示散发着严谨扎实却又极富前瞻性的学术理想主义色彩，演讲者从沉静理性的技术演进历史切入，层层递进地展开底层算法的质变。当音响中骤然传出由计算机合成、却与台上的 Rashid 声线如出一辙的中文语句时，现场从全神贯注的聆听瞬间转为由衷的惊叹与雷动掌声。演讲者既不避讳系统尚存生硬语序的科学坦诚，又动情展望消弭全球语言鸿沟的愿景，让整场展示充满见证未来科技诞生的震撼与人文温情。 ## Bulletin - [03:07](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=187s) 微软研究院与多伦多大学合作引入深度神经网络（DNN），将语音识别错误率从长年停滞的20%-25%大幅拉低至约15%甚至更低，识别率提升约30%。 - [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) 传统基于隐马尔可夫模型（HMM）的语音识别在特定受限场景虽有落地，但在面对任意自然语音时遭遇瓶颈，每4至5个词就会出现一次识别错误。 - [04:34](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=274s) 微软展示了端到端跨语言语音克隆技术，仅需约1小时英文录音提取声学特征，即可调制中文TTS模型，实现用演讲者原声讲中文。 - [07:12](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=432s) 现场实时演示实现了英文语音输入、中文字幕生成与同音色中文合成输出，微软首席研究官 Rick Rashid 预言该技术将在未来几年打破人类语言隔阂。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) 语音被公认为最自然的人机交互界面，但计算机科学家探索语音识别已历经六十年曲折演进。 - [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) 早期的波形模式匹配技术极度脆弱，无法适应不同说话人以及同一个人在不同上下文中的发音波形差异。 - [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) 1970年代末卡耐基梅隆大学引入隐马尔可夫模型（HMM）等统计建模技术，为现代语音识别三十年的发展奠定了基石。 - [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) 统计建模技术在银行电话自动应答、苹果 Siri 及微软 Kinect 等限定领域取得了显著的商业化成功。 - [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) 在面对不受限制的任意日常语音时，传统统计模型的词错误率长期居高不下，维持在 20% 至 25% 之间。 - [03:07](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=187s) 微软研究院与多伦多大学合作，借鉴人脑神经机制引入深度神经网络（DNN），彻底突破了声学模型的吞吐与训练瓶颈。 - [03:07](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=187s) 深度神经网络将语音识别错误率从“每四到五个词错一个”骤降至“每七到八个词错一个”，实现了精度上质的飞跃。 - [04:34](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=274s) 跨语言机器翻译依托统计建模与海量大数据实现飞跃，必应翻译日均支撑数百万网页的实时转译请求。 - [04:34](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=274s) 英汉机器翻译采用“逐词转换”与基于语言模型的“语序重排（Reordering）”两阶段处理流程，以契合中文语法习惯。 - [04:34](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=274s) 个人音色克隆通过标准中文母语者 TTS 模型结合演讲者约 1 小时的自有英文音频，完成跨语言的声学特征调制。 - [07:12](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=432s) 现场端到端演示展现了近乎实时的英文转录、机器翻译与音色高度还原的中文语音播报全过程。 - [07:12](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=432s) 尽管系统当前仍存在机械停顿与语法瑕疵，但其实时跨语言语音互译的潜能预示着消弭人类沟通障碍的美好未来。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=0s) 语音识别六十年的演进与隐马尔可夫模型的局限微软首席研究官 Rick Rashid 回溯了人机交互界面的发展脉络。他指出，在探索自然用户界面（Natural User Interfaces）的漫长历程中，人类最自然的沟通媒介莫过于语音。计算机科学家尝试让机器理解并识别人类语音已有六十年的历史。在最初阶段，研究人员主要将语音识别视为一种波形模式匹配（Pattern Matching）问题，试图将说话人声带发出的波形直接与预先存储的特定词汇波形进行比对。然而，这种方案极其脆弱，因为不仅人与人之间的发音千差万别，即使是同一个人，在不同的上下文语境中说出同一个词时，发音波形也会截然不同。到了1970年代末期，卡耐基梅隆大学（Carnegie Mellon University）的研究团队带来了开创性的范式转换，开始将隐马尔可夫模型（Hidden Markov Models, HMM）等统计建模技术引入语音识别。这一改变使得系统能够吸收并处理来自大量不同说话人的语音数据，建立起更为鲁棒的统计模型。在接下来的三十年里，语音识别技术取得了长足的进步，并在限定领域（limited domains）实现了非常成功的商业应用。例如在美国，拨打银行客服电话时往往是由计算机应答，系统可以处理账户查询等基础问题，遇到复杂状况再转接人工；此外，苹果的 Siri 以及微软 Kinect 的语音控制系统，也都展现了成熟的指令识别能力。但面对不受限制的任意自然语音（arbitrary speech）时，传统系统的词错误率长期徘徊在 20% 至 25% 之间。 - 模式匹配阶段的缺陷：极度脆弱，无法适应不同说话人以及同一个人在不同上下文中的发音变化。 - 1970年代末统计学突破：卡耐基梅隆大学引入隐马尔可夫模型（HMM），奠定了现代语音识别三十年的技术基石。 - 经典落地应用：银行自动化电话应答系统、苹果 Siri 语音助手、微软 Kinect 游戏语音指令系统。 - 传统技术瓶颈：对于任意日常语音，系统的识别错误率长期居高不下，维持在 20% 到 25%（即每说四到五个词就会错一个）。 --- ### [03:07](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=187s) 深度神经网络带来的跨越式性能提升为了打破传统统计模型的瓶颈，微软研究院与多伦多大学（University of Toronto）的研究人员展开紧密合作，共同研发了语音识别领域的又一次重大突破。这项新技术借鉴了人脑神经元的工作机制，即深度神经网络（Deep Neural Networks, DNN）。相比于以往的隐马尔可夫模型，深度神经网络能够吞吐并处理体量大得多的训练数据，从而在本质上改善声学模型的识别精准度。这一技术突破使得语音识别率直接提升了约 30%。用具体的错误率来衡量，它成功将长年困扰学术界与工业界的 20% 到 25% 错误率，直接下拉到大约 15% 甚至更低的水平。Rick Rashid 强调，这意味着系统的识别表现从原先“每四到五个词就出现一个错误”，大幅跃升为“每七到八个词才可能错一个”。虽然这项技术距离完全成熟依然有很长的路要走，但这一飞跃标志着语音识别精度在数十年停滞后迎来了质的蜕变。关于这一技术突破的重大意义，Rick Rashid 评价道：“So that one change that particular breakthrough increased recognition rates by approximately 30% that's a big deal that's the difference between going from 20 to 25% errors or about one error out of every four or five words to roughly 15% or slightly less errors roughly one out of every seven words or perhaps even one out of every eight.” ```mermaid flowchart TD accTitle: 语音识别与跨语言合成工作流 accDescr: 展示从英文语音输入到深度神经网络识别、机器翻译、再到个性化中文语音合成的全过程 subgraph ASR ["1. 语音识别 (ASR)"] A["Rick Rashid 英文语音"] --> B["深度神经网络 (DNN) 模型"] B --> C["英文文本 (错误率降至 ~15%)"] end subgraph MT ["2. 机器翻译 (MT)"] C --> D["逐词跨语言转换"] D --> E["汉语语序重排 (Reordering)"] E --> F["规范中文文本"] end subgraph TTS ["3. 个性化语音合成 (TTS)"] G["海量中文母语者语音库"] --> H["标准中文 TTS 系统"] I["Rashid 个人约1小时英文录音"] -->|"声学特征调制 (Modulation)"| H F --> H H --> J["输出：Rashid 个人音色的中文语音"] end ``` --- ### [04:34](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=274s) 跨语言机器翻译与个人音色克隆技术在解决语音识别的同时，计算机科学界攻关了六十年的另一大难题是机器翻译（Machine Translation）。近几年，统计建模技术与大数据的结合使得机器翻译取得了突破性进展，例如微软研究院开发的必应翻译（Bing Translator），每天都在为全球用户将数百万个网页实时转译为其母语。要实现从英文发言到中文语音的实时转换，系统主要分为两步处理：首先将英文文本逐词转译为对应的中文词汇，接着通过语言模型对文本进行重排（reordering），以契合中文独特的语法结构与表达习惯。然而，更具颠覆性的一步在于让系统以演讲者本人的声音开口说中文。微软研究团队首先通过海量中文母语者的发音数据训练出一个高水准的标准中文文本转语音（Text-to-Speech, TTS）系统；随后，他们采集了 Rick Rashid 本人大约一小时的英文演讲录音，从中提取出独特的声学与音色特征，并以此去调制（modulate）底层的中文 TTS 模型。通过这种声学迁移，系统能够用带有一模一样个人音色的声音，将生成的中文文本声情并茂地念出来。 - 必应翻译支撑：基于统计与大数据技术，日均处理数百万网页的跨语言翻译请求。 - 文本转译双阶段：第一阶段完成英汉词汇映射，第二阶段依照汉语语法规律重排语序（Reordering）。 - 声音克隆机制：结合大量中文母语者样本构建通用 TTS 模型，再用演讲者约 1 小时的自有英文音频提取音色特征完成声学调制。 --- ### [07:12](https://www.youtube.com/watch?v=Nu-nlQqFCKg&t=432s) 现场双语实时演示与消弭语言隔阂的愿景在随后的现场演示环节，Rick Rashid 开始使用英文继续演讲，舞台大屏幕上几乎同步滚动出现英文转录与翻译后的中文字幕，紧接着音响中传出了由计算机合成、但音色与 Rashid 完全一致的中文播报。尽管合成语音伴随轻微的机械停顿，且中文表达中夹杂着部分生硬的语法错误，但精准的音色还原与流畅的跨语言复现依然引发了全场热烈的掌声与惊叹。面对观众的掌声，Rick Rashid 坦率地指出，这项技术目前绝非完美，现场输出中仍有相当数量的错误，整个领域还有大量的工作亟待推进。但毫无疑问，这项技术的潜能是无与伦比的。他憧憬在未来的几年内，随着技术的不断演进，人类将能真正打破语言障碍。在演讲的最后，他动情地表达了对技术未来的信念，并特意用刚学到的中文向全场致谢。对于这项技术的深远影响，Rick Rashid 动情地断言：“Personally I believe this is going to lead to a better world.” - 演示实况：英文发言即时转译为中文字幕，并由克隆了 Rashid 声线的系统以中文朗读。 - 技术现状：端到端演示虽伴随语法瑕疵与转译错误，但确立了实时跨语言语音互译的可行性。 - 结语致意：Rashid 表达了用技术打破人类沟通壁垒的信念，并以中文“谢谢”（see she）向现场致意结束演示。 ## Mentioned Rick Rashid, 微软 (Microsoft), 微软研究院 (Microsoft Research), 卡耐基梅隆大学 (Carnegie Mellon University), 多伦多大学 (University of Toronto), 苹果 (Apple), Siri, Kinect, 必应翻译 (Bing Translator), 隐马尔可夫模型 (HMM), 深度神经网络 (DNN), 文本转语音 (TTS) ## Video description Chief Research Officer Rick Rashid demonstrates a speech recognition breakthrough via machine translation that converts his spoken English words into computer-generated Chinese language. The breakthrough is patterned after deep neural networks and significantly reduces errors in spoken as well as written translation. For more information on Speech Recognition and Translation, visit http://www.microsoft.com/translator/skype.aspx [Source](https://www.youtube.com/watch?v=Nu-nlQqFCKg)

Original transcript

[0:00] [Music] [0:00] now I uh I want to come back to the [0:06] topic of natural user [0:09] interfaces one of the most natural [0:11] interfaces for people is human [0:15] speech and for the last 60 years [0:18] computer scientists have been trying to [0:20] find ways to [0:22] understand and recognize human [0:26] speech now in the beginning when people [0:29] first start of tackling this problem [0:31] they looked at it largely as a pattern [0:34] matching problem and the earliest [0:37] systems attempted to take the [0:40] waveforms that came out of a speaker [0:44] speaker's voice and matched them up to [0:47] waveforms that they knew represented [0:49] certain [0:51] words unfortunately that approach was [0:53] extremely [0:55] fragile partly because everyone speaks [0:58] differently but also Al even a single [1:01] speaker will often say the same thing [1:04] differently depending on the other words [1:07] or the context in which they're speaking [1:10] you've probably already noticed me doing [1:14] that now in the late [1:18] 1970s there was a major change in the [1:22] way people decided to [1:27] do speech recognition this was work [1:30] being done at Carnegie melon University [1:33] and the idea was to use a statistical [1:35] modeling technique in this case hidden [1:37] Markoff models to make the trans to [1:41] really be able to take a lot of data [1:43] from many speakers and produce more [1:46] robust statistical models of [1:49] speech [1:51] now that was a huge [1:53] Improvement and over the last 30 years [1:56] speech recognition systems have become [1:59] dramatically better than they used to [2:02] be they still make a lot of [2:05] mistakes but in in limited domains it's [2:09] possible to do very successful speech [2:13] interfaces so for example in the United [2:15] States when I call my [2:17] bank I'm talking to a computer I'm not [2:20] talking to a [2:21] person the computer can answer simple [2:24] questions about my bank account or if [2:27] necessary it can connect me to a real [2:30] person if I have a significant issue [2:33] that I want to [2:36] discuss I'm sure you've heard of of [2:38] Apple Siri product which answers simple [2:42] questions Microsoft Connect has a robust [2:46] speech interface that allows you to [2:48] control the interface and even allows [2:50] you to issue commands in the middle of [2:55] games still these systems have a lot of [2:58] errors and the error rates for arbitrary [3:02] speech have been in the 20 to 25% [3:07] range well a few years ago researchers [3:10] at at Microsoft research and at the [3:12] University of [3:13] Toronto came together to develop what's [3:17] another breakthrough in the field of [3:19] speech recognition [3:21] research the idea that they had was to [3:24] use a technology in in a way patterned [3:27] after the way the human brain works it's [3:30] called Deep neural [3:32] networks and to use that to take in much [3:36] more data than had previously been able [3:39] to be used with the hidden Markoff [3:41] models and use that to significantly [3:44] improve recognition [3:47] rates so that one change that particular [3:50] breakthrough increased recognition rates [3:53] by approximately 30% that's a big [3:56] deal that's the difference between going [3:59] from [4:00] 20 to 25% errors or about one error out [4:03] of every four or five [4:06] words to roughly [4:09] 15% or slightly less errors roughly one [4:13] out of every seven words or perhaps even [4:15] one out of every [4:16] eight so it's still not perfect there's [4:19] still a long way to [4:21] go but I think you can see that we're [4:25] have already made a significant amount [4:28] of additional progress [4:30] in the recognition of [4:34] speech now one of the problems that [4:36] we've also been trying to solve for 60 [4:38] years is machine [4:42] translation and again in just the last [4:44] few [4:45] years the combination of statistical [4:48] techniques and big data have allowed us [4:52] to do a much better job than we [4:54] previously were able to do in being able [4:57] to [4:58] translate we Pages or other kinds of [5:01] information into other [5:04] languages for example today being [5:07] translated which is Microsoft [5:08] translation system that comes out of [5:10] Microsoft research being translate [5:13] translates millions of pages a day for [5:16] users into their native language it's an [5:19] extremely heavily used [5:23] service now if I want to have what I'm [5:26] saying be translated into Chinese [5:30] we can take the text that comes from my [5:33] voice and put that through a translation [5:36] system it really happens in two [5:41] steps in the first [5:44] instance we take the English and we [5:47] convert it more or less word by word [5:51] into into Chinese [5:54] text and I think pretty soon we may see [5:57] that up on the screen [6:00] so what happens is we're basically [6:02] taking the English text and pushing it [6:04] through the translation [6:06] system we then have to reorder that text [6:10] in Chinese because the word order in [6:12] Chinese is not the same as the word [6:14] order in [6:14] English to produce something that begins [6:17] to resemble something that a Chinese [6:19] speaker might [6:21] say so now we're taking the things that [6:24] I'm saying and we're converting them [6:26] into Chinese text [6:30] now the last step that I want to be able [6:33] to take in this process is to actually [6:36] speak to you in [6:40] Chinese now the key thing there is we've [6:43] been able to take a large amount of [6:45] information from many Chinese speakers [6:48] and produce a text to speech system that [6:50] takes Chinese text and converts it into [6:53] Chinese [6:55] language and then we've taken an hour or [6:57] so of my own voice and we used that to [7:00] modulate the standard text to speech [7:03] system so that it would sound like [7:05] me so what you see now is the result of [7:10] that [7:12] change I'm speaking in English and [7:15] hopefully you'll hear me speak in [7:17] Chinese in my own voice [7:31] again the results are not [7:32] perfect there are in fact quite a few [7:39] [Applause] [7:42] errors there's much work to be done in [7:45] this [7:56] area but this technology is very [7:59] promising [8:07] and we hope in a few years that we'll be [8:09] able to break down the language barriers [8:11] between [8:25] people personally I believe this is [8:27] going to lead to a better world [8:40] I hope you enjoy the rest of the [8:41] presentations [8:55] today thank you [9:01] see she