UCSF Neurosurgeon Eddie Chang Decodes Brain-Machine Interface Progress and the Future of Neural Augmentation

UCSF神经外科医生Eddie Chang解读脑机接口进展与神经增强未来 本次讨论连接了学术临床在瘫痪患者言语重建方面的突破，以及商业认知增强领域尚未解决的伦理边界。 该节目阐述了作为认知框架的语言与作为复杂运动执行的言语之间的根本区别，后者需要声道进行精确且无意识的协调。Chang博士分享了Bravo临床试验的突破性成果，在该试验中，AI和机器学习成功将一名闭锁综合征患者的原始神经模式解码为完整的句子以及数字化身表情。讨论还探讨了商业脑机接口的未来前景，并明确指出口吃是大脑听觉-运动反馈回路中特定于发音的障碍，而非语言或焦虑导致的缺陷。 言语和语言在大脑中是两个不同的概念，言语是声音交流信号的物理生成，而语言则是涵盖语用、语义和句法的更广泛的认知框架。 (0:23) 生成言语是一项极其复杂的运动技能，需要声带高频振动（男性约100赫兹，女性约200赫兹）来产生原始声能。 (3:11) 哭泣和笑等原始发声与后天习得的言语截然不同，受非人类灵长类动物共有的更原始、独立的脑部结构控制。 (5:35) 闭锁综合征通常由脑干中风或渐冻症（ALS）等疾病引起，会导致自主运动和言语输出完全丧失，但认知和环境感知能力保持完全完好。 (6:37) Bravo临床试验成功利用植入在言语皮层上的电极阵列截获大脑信号，并将其传输至外部计算机，从而为瘫痪患者解码言语。 (9:14)

Neurosurgeon Dr. Eddie Chang distinguishes the cognitive architecture of language from the physical motor execution of speech, illustrating how mapping cortical speech representations enables brain-machine interfaces to restore communication for paralyzed individuals. His research demonstrates that combining direct neural decoding with commercial predictive language algorithms can successfully reconstruct full sentences, even as biological bandwidth limits and ethical dilemmas constrain the future of neural augmentation. Language Architecture Versus Motor Speech Execution (0:00) Dr. Chang separates speech, defined as the physical motor production of acoustic signals through the vocal tract, from language, which operates as an overarching cognitive framework encompassing semantics (meaning), syntax (grammar), and pragmatics (social context and gist). While language can be expressed through alternate modalities such as sign language and reading, speech requires an extraordinarily complex motor sequence. Vocalization begins with lung expiration, which the larynx intercepts via vocal folds that oscillate at roughly 100 Hz in men and 200 Hz in women due to anatomical differences in larynx size. The tongue, lips, and jaw then articulate this acoustic energy into distinct vowels and consonants. In contrast to learned speech, primitive vocalizations such as laughing, crying, and moaning depend on evolutionary conserved brain structures shared with non-human primates. Because these primitive vocalizations rely on distinct neural pathways, patients suffering from severe trauma to their speech and language networks often retain the ability to vocalize basic emotional sounds. Speech Restoration in Paralyzed Patients Using AI (6:37) Severe brain stem strokes and late-stage Amyotrophic Lateral Sclerosis (ALS) can result in locked-in syndrome, a state in which a patient's cognitive awareness and intellect remain fully intact while all voluntary motor output is severed. In the Bravo clinical trial, Dr. Chang's team mapped the cortical electrical patterns corresponding to individual vocal-tract movements and implanted an electrode array over the speech-motor cortex of a participant paralyzed for 15 years. The hardware connects through a skull-mounted transcutaneous port that the participant has used safely for three years, routing analog brain signals to computers for digital processing. Because raw neural decoding produces classification errors, the team integrated a contextual language model resembling smartphone autocorrect software. By analyzing sentence permutations from an initial 50-word target vocabulary, the predictive algorithm evaluates surrounding context to correct misclassified phonemes in real time. This combined system achieved the first direct decoding of full sentences from cortical activity in a paralyzed individual. Technical Bottlenecks and Ethics of Neural Augmentation (14:27) Addressing the commercial push toward brain-machine interfaces by ventures like Neuralink, Dr. Chang notes that the field is transitioning from academic motor-restoration studies to commercial medical devices. However, he urges caution regarding non-medical cognitive augmentation intended to create superhuman memory, communication, or athletic capabilities. He emphasizes that society has not adequately addressed the ethical questions, accessibility concerns, and potential disruptions associated with elective neural surgeries. Dr. Chang highlights that natural evolutionary biology remains the primary rate-limiting step for brain enhancement. Human communication networks possess a functional bandwidth across millions of neurons that no current laboratory or commercial neurotechnology can replicate. External digital tools like smartphones already supply universal information access, meaning physical interfaces do not yet offer an advantage over evolved natural biological systems. Decoded Avatars and Embodied Nonverbal Communication (19:24) Interpersonal communication relies heavily on nonverbal cues and visible jaw and lip movements, which directly aid the listener's brain in processing speech. To replicate these elements, Dr. Chang's laboratory is developing computer-animated avatars driven directly by decoded cortical commands. Rather than merely rendering decoded words as text, the system reconstructs intended facial expressions and articulatory movements. Dr. Chang forecasts that while virtual communication environments will increasingly serve general consumers, avatar-driven neuroprosthetics offer critical utility for paralyzed patients. Beyond enriching communication for listeners, an avatar provides dynamic visual feedback to the patient. This visual embodiment reinforces the user's perception of direct vocal-tract control, measurably accelerating their ability to operate the speech neuroprosthetic. Articulatory Coordination and Auditory Disruption in Stuttering (22:51) Dr. Chang classifies stuttering strictly as a speech articulation disorder rather than an impairment of language. Individuals who stutter retain intact grammatical, semantic, and conceptual capabilities, but experience situational disruptions in the automated motor machinery coordinating the larynx, lips, and jaw. While psychological anxiety can provoke or worsen dysfluent episodes, Dr. Chang clarifies that anxiety is not the root physiological cause. Instead, research indicates that stuttering stems from a communication breakdown within the brain's auditory-motor integration loop. Because the brain constantly monitors speech through auditory feedback, modifying that feedback can instantly alter the severity of a stutter. Consequently, clinical interventions emphasize non-invasive behavioral therapies that leverage early childhood neuroplasticity to resolve sound-initiation blocks without surgical intervention. While speech neuroprosthetics demonstrate significant clinical success in decoding cortical intention for locked-in patients, matching the speed and nuance of natural human communication remains constrained by the millions of neurons operating within natural auditory-motor circuits. As commercial interfaces advance toward broader consumer applications, society faces unresolved ethical and regulatory questions regarding access and non-medical cognitive augmentation.

神经科学家埃迪·张（Dr. Eddie Chang）阐明了人类语言认知架构与语音物理输出之间的神经解剖学区别，并展示了如何通过结合皮层电极阵列与人工智能上下文算法的脑机接口（BMI）技术，为严重瘫痪患者恢复言语生成与非语言沟通能力。 语音发声与语言认知的神经解剖差异 (0:00) 张博士明确区分了“语音”（Speech）与“语言”（Language）的概念。语言属于高级认知框架，涵盖提炼社交情境的语用学、理解词句涵义的语义学以及组合词汇的句法学，并包含手语和阅读等非口语模态；语音则仅是语言的一种物理交流信号。部分科学家认为生成语音是人类能执行的最复杂的运动行为。其发声过程本质是“对呼吸的塑形”：肺部呼出气流穿过喉部，促使声带以极高频率振动产生原声基调。因喉部尺寸差异，男性声带振动频率通常约为100 Hz，女性约为200 Hz；原声音频随后经咽部进入口腔，由舌头、唇部和下颌精确塑形为特定的元音与辅音。 与之相对，哭泣、呻吟或大笑等原始发声并不依赖这套习得性语言运动网络，而是由在非人类灵长类动物中同样保留的更原始脑结构控制。张博士指出，这种神经通路的分立解释了临床上的一种特殊现象：即便患者掌管语言和言语的脑区遭受严重创伤或肢体瘫痪，往往仍能保留哭泣或发出呻吟等本能声音的能力。 口吃障碍的运动协调机制与听觉闭环 (22:51) 张博士指出，口吃在医学上被严格界定为影响发音协调的语音运动障碍，而非语言理解或认知缺陷。口吃个体的内在思想、语法组织和语义表达完全完好，其核心问题在于执行发声时无法流畅协同喉部、唇部与下颌的精密运动。正常说话犹如一场高度同步、无意识自动运转的交响乐，一旦说话者对发音过程产生过度意识，其高度自动化的内部协调机制便容易发生情境性紊乱。 尽管个体可能具备遗传或生理易感性，但口吃并非持续存在。张博士澄清，焦虑情绪是激发或加剧口吃的重要情境诱因，但并非该障碍的根本致病因。其生理机制的核心线索在于大脑运动中枢与听觉反馈系统之间的回路解耦：大脑在说话过程中持续监测自身声音，一旦人为改变这种听觉反馈，患者的口吃程度便会显著加剧或减轻。在临床干预方面，通过在童年期利用活跃的神经可塑性开展行为语言治疗，即可通过矫正发音启动技巧来改善症状，无需诉诸侵入性脑部手术。 脑机接口与AI语言模型的临床突破 (6:37) 脑干作为连接大脑皮层（产生思考与运动意图）与脊髓及面部神经的桥梁，一旦因中风或肌萎缩侧索硬化症（ALS）受损，会导致患者陷入认知与思维完全正常却丧失所有运动与发音能力的“闭锁综合征”（Locked-In Syndrome）。为打破这种严重的社交与心理隔绝，张博士团队通过历时数年解码对应不同元音与辅音的皮层电波模式，发起了Bravo临床试验。首位受试者因脑干中风瘫痪达15年，此前仅能依靠帽子上固定的小木棒在键盘上逐字敲击；研究团队将电极阵列植入其负责控制喉、唇、舌和下颌的皮层表面，并通过固定于颅骨的接口与计算机相连，该患者已安全携带该植入接口生活了三年。 该突破的关键技术路径在于结合了原始神经信号解码与商业级上下文预测算法。团队最初建立了一个包含50个目标单词的系统，鉴于纯脑电信号直接解码难以达到100%准确率，研究人员引入了类似智能手机“自动纠错”的语言概率模型，实时通过上下文修正误读的音素。这是全球首例直接从瘫痪患者大脑皮层信号中解码出完整句子的临床案例。张博士回忆，患者首次用意念在屏幕上生成单词“outside”时因兴奋而全身颤笑，其引发的生理震颤甚至一度干扰了算法对后续脑电的识别。 虚拟化身交互与神经增强的现实瓶颈 (14:27) 人类面对面交流极度依赖非语言线索，观察发音者的唇部与下颌运动能够直接增强大脑对语音的理解。为此，张博士团队正将脑机接口技术由纯文字输出推进至数字化身（Avatar）系统：通过解码大脑皮层运动信号，在合成语音的同时驱动计算机虚拟化身重现面部微表情与口唇动作。数字化身不仅能帮助行动受限者更自然地参与数字与虚拟社交，还可作为实时视觉反馈机制，强化患者对言语神经义肢的具身控制感，从而加速脑电算法的学习与适应。 针对商业资本（如Neuralink）推动脑机接口向超常记忆或高速信息传递等“认知增强”方向延伸的趋势，张博士持谨慎态度。他指出，人类天然的语言交流系统依托经过数百万年演化、具备数百万神经元带宽的神经回路，当前实验室与商业技术在信息传输能力上均远远落后于生物系统，技术本身的通量依然是认知增强的主要瓶颈。此外，面向健康人群使用侵入性手术进行机能增强，还将面临深刻且尚未达成共识的技术准入公平性与社会伦理考量。 张博士的研究证实，通过神经电生理记录与上下文语言算法的结合，能够为闭锁综合征患者重建包含发音与微表情的完整交流通道；然而，当前技术仍面临脑电信号带宽受限、物理运动干扰以及词汇规模等工程限制，且针对健康个体的神经增强应用在生物学匹配度与社会伦理层面依然存在未解疑问。

[1:00] Dr. Eddie Chang: Speech corresponds to the communication signal [3:07] Dr. Eddie Chang: shaping the breath [11:07] Dr. Eddie Chang: got a kick out of that [14:27] Andrew Huberman: manipulating neural circuitry to achieve suprahuman or superhuman [16:27] Dr. Eddie Chang: augmentation [19:24] Dr. Eddie Chang: speech neuroprosthetic [27:34] Andrew Huberman: absolute thrill

[0:00] Welcome to Huberman Lab Essentials, [0:02] where we revisit past episodes for the [0:04] most potent and actionable science-based [0:06] tools for mental health, physical [0:08] health, and performance. [0:11] I'm Andrew Huberman and I'm a professor [0:13] of neurobiology and opthalmology at [0:16] Stamford School of Medicine. And now for [0:18] my discussion with Dr. Eddie Chang. [0:20] Eddie, welcome. [0:21] >> Hi. Hi, Andrew. [0:23] >> Great to be here with you. Your main [0:25] focus these days is the neurobiology of [0:28] speech and language. So for those that [0:30] aren't familiar, could you please [0:32] distinguish for us speech versus [0:34] language in terms of whether or not [0:36] different brain areas control them? When [0:38] I think about language, I think about [0:39] words and just talking. If I sit down to [0:41] do a long podcast or I think about [0:43] asking you a question, I don't even [0:44] think about the words I want to say very [0:46] much. I mean, I have to think about them [0:48] a little bit. one would hope, but I [0:50] don't think about individual syllables [0:51] unless I'm trying to, you know, accent [0:54] something or it's a word that I have a [0:56] particular difficulty saying or I want [1:00] to change the cadence, etc. So, what in [1:02] the world is contained in these brain [1:04] areas? What is represented? Um, to me is [1:07] is perhaps one of the most interesting [1:09] questions and I know this lands square [1:11] in your wheelhouse. [1:12] >> Sure. Let's get into this uh Andrew [1:15] because this is one of the most exciting [1:18] stuff that's happening right now is [1:19] understanding how the brain processes [1:21] these exact questions. And speech [1:24] corresponds to the communication signal. [1:27] It corresponds to me moving my mouth and [1:30] my vocal tract to generate words. And [1:33] you're hearing these as an auditory [1:35] signal. Language is something much [1:38] broader. So it refers to what you're [1:40] extracting from the words that I'm [1:42] saying. We call that pragmatics and sort [1:44] of are you getting the gist of what I'm [1:46] saying? There's another aspect of it [1:47] that we call semantics. Do you [1:49] understand the meaning of these words [1:52] and uh the sentences? There's another [1:54] part that we call syntax which refers to [1:56] how the words are assembled in a [1:58] grammatical form. So those are all [2:00] really critical parts of language and [2:03] speech is just one form of language. [2:06] There's many other forms like sign [2:08] language, uh, reading. Those are all [2:10] important modalities for reading. Our [2:13] research really focuses on this area [2:16] that we're calling speech. Again, the [2:18] production of this audio signal, which [2:22] you can't see, but your microphones are [2:25] picking up. There are these vibrations [2:27] in the air that are created by my vocal [2:30] tract that are picked up by the [2:31] microphone in the case of this [2:33] recording, but also picked up by the [2:35] sensors in your ear. The very tiny [2:37] vibrations in your uh ear are picking [2:40] that up and translating that into [2:42] electrical activity. It's such a complex [2:45] feat. Some people would say it's the [2:46] most complex motor thing that we do as a [2:48] species is is just speaking. not you [2:51] know the extreme feats of acrobatics or [2:54] athleticism but speaking [2:56] >> well and especially when uh one observes [2:58] you know uh opera or um people who you [3:01] know freestyle rappers you know and of [3:04] course it's not just the lips it's the [3:06] tongue [3:07] >> and you've mentioned two other [3:08] structures ferinx and larynx are the [3:10] main ones that um can you tell us just [3:12] just educate us at a a superficial level [3:15] what this ferinx and larynx do [3:17] differentially because I think most [3:19] people aren't going to Okay, sure. I'll [3:21] talk primarily about the larynx here for [3:24] a second, which is that if you think [3:26] about when we're speaking, really what [3:28] we're doing is we're shaping the breath. [3:30] So, even before you get to the larynx, [3:31] you got to start with the expiration. We [3:34] fill up our lungs and then we push the [3:36] air out. That's a normal part of [3:38] breathing. What is really amazing about [3:40] speech and language is that we evolved [3:43] to take advantage of that normal [3:44] physiologic thing at a larynx. And what [3:48] the larynx does is that when you're [3:50] exhaling, it brings the vocal folds [3:52] together. Some people call them vocal [3:54] cords. They're not really cords. They're [3:56] really vocal folds. They're two pieces [3:58] of tissue that come together and a [4:00] muscle brings them together. And then [4:01] what happens is when the air comes [4:03] through the vocal folds, when they're [4:05] together, they vibrate at really high [4:06] frequencies, like 100 to 200 hertz. And [4:09] the reason why men and women generally [4:11] have different voice qualities is it has [4:14] to do with the size of the larynx and [4:16] the shape of it. Okay? So in general men [4:20] have a larger voice box or larynx and [4:23] the vibrating frequency the resonance [4:25] frequency of the vocal folds when the [4:27] air comes through them is about 100 [4:29] hertz for men and about 200 for women. [4:32] So you take a breath in. As the air is [4:35] coming out, the vocal folds come [4:36] together. The air goes through. That [4:39] creates the sound of the voice that we [4:41] call voicing. It's not just your voice [4:44] characteristic. It's the energy of your [4:47] voice. It's coming from the larynx [4:50] there. It's a noise. And then it's the [4:52] source of the voice. And then what [4:54] happens is that energy that sound goes [4:58] up through the parts of the vocal tract [5:01] like the fairings into the oral cavity [5:04] which is your mouth and your tongue and [5:06] your lips. And what those things are [5:08] doing is that they're shaping this the [5:10] air in particular ways that create [5:14] consonants and vowels. That's what I [5:16] mean by shaping the breath. It just [5:18] starts with this exhalation. You [5:21] generate the voice in the larynx and [5:23] then everything above the larynx is [5:25] moving around just like the way my mouth [5:27] is doing right now to shape that air [5:30] into particular patterns that you can [5:33] hear is words immediately makes me [5:38] wonder about more um primitive or [5:41] non-learned vocalizations like crying or [5:44] laughter. Are those produced by the [5:47] language areas or do they have their own [5:49] unique neural structures? [5:51] >> We call those vocalizations. A [5:53] vocalization is basically where someone [5:55] can create a sound like a cry or a moan [5:59] that kind of sound. And it also involves [6:03] the exhalation of air. It also involves [6:06] some phonation at the level of larynx [6:09] where the vocal folds come together to [6:11] create that audible sound. But it turns [6:13] out that those are actually different [6:15] areas. So people who have injuries in [6:18] the speech and language areas oftentimes [6:20] can still moan. They can still vocalize. [6:23] And it is a different part of the brain. [6:25] I would say an area that uh even [6:27] non-human primates have that can be [6:30] specialized, you know, for vocalization. [6:32] It's a different form of communication [6:34] than than words, for example. Speaking [6:37] of storage of and ability to speak, you [6:40] are doing some amazing work and have [6:42] achieved some um pretty incredible [6:45] well-deserved recognition for your work [6:48] in bringing language out of paralyzed [6:51] people. essentially allowing people who [6:53] are locked in to a paralyzed state or [6:56] otherwise unable to articulate speech [6:59] using brain machine interface [7:02] essentially translating the neural [7:04] activity of areas of the brain that w [7:06] would produce speech into hardware [7:10] artificial non-biological tools in order [7:12] to allow paralyzed people to [7:14] communicate. [7:15] >> So there are a series of conditions um [7:18] they include things like brain stem [7:20] stroke. The brain stem is the part of [7:22] the brain that connects the cerebrum [7:24] which is the top part does our thinking [7:26] and a lot of the motor control, speech, [7:28] language, everything. And the brain stem [7:29] is what connects that to the spinal cord [7:31] and the nerves that go out to the face [7:32] and vocal tract. So if you have a stroke [7:34] there, you could be thinking all the [7:36] wild creative intelligent thoughts you [7:38] have in the mind and the cerebrum, but [7:40] you can't get them out into words or you [7:42] can't get them out to your hand to write [7:44] them down. So that's a very severe form [7:47] of paralysis called brain stem stroke. [7:49] There's another kind of conditions that [7:51] we call neurogenerative where the nerve [7:53] cells die basically or atrophy and a [7:57] condition called uh ALS. That's a very [8:00] severe form of paralysis. In its extreme [8:02] form, people essentially lose all [8:04] voluntary movement. The muscles to their [8:06] diaphragm and their lungs essentially [8:08] give out as well. They get weakness [8:10] there and then they can't breathe [8:11] anymore. In our field, these are kind of [8:12] like the most devastating things that [8:14] can happen. This condition of what we [8:16] call being locked in refers to this idea [8:20] that you can have completely intact [8:23] cognition and awareness but have no way [8:26] to express that. No voluntary movement, [8:30] no ability to speak and that is [8:32] devastating because uh psychologically [8:34] and socially you know you're completely [8:36] isolated. That's what we call locked in [8:38] syndrome and it's devastating. So we've [8:41] been studying this patterning of [8:44] electrical activity for consonants and [8:46] vowels. And essentially once we figured [8:48] out a lot of these codes for the [8:51] individual phonetic elements, part of [8:53] the lab started to focus on this very [8:55] specific question for people who have [8:57] these kind of paralysis. Could we [9:01] intercept those signals from the brain, [9:04] the cerebral cortex, as someone is [9:06] trying to say those words? And then can [9:09] we intercept them and then have them [9:12] taken out of the brain through wires to [9:15] a computer that are going to interpret [9:17] those signals and translate them into [9:19] words. So we started a clinical trial. [9:23] It's called the Bravo trial. It's still [9:24] underway. And the first participant in [9:26] the Bravo trial was a man who had been [9:29] paralyzed for 15 years. He was in a car [9:32] accident. He actually walked out of the [9:34] hospital the day after that car [9:35] accident, but the next day had a [9:37] complication related to it where he had [9:40] a very large stroke in the brain stem [9:43] and that turned out to be devastating. [9:46] He didn't wake up from that stroke for [9:48] about a week. He was in a coma for about [9:50] a week and when he woke up from that [9:52] coma, he realized that he couldn't speak [9:54] or move his arms or legs. As he told me [9:57] or communicated to us, that was [9:59] absolutely devastating. He wanted really [10:02] to die at that time. [10:03] >> Could he blink his eyes or move his [10:04] mouth in any way? [10:06] >> He could blink his eyes. He had some [10:08] limited mouth movements but couldn't [10:09] produce any intelligible speech. It was [10:12] like completely slurred and [10:13] incomprehensible. He survived this [10:16] injury. A lot of people who have that [10:17] kind of stroke just don't survive. The [10:19] way he actually communicates because he [10:21] has a little bit of residual neck [10:23] movements is that he improvised and had [10:27] his friends basically put a stick [10:30] attached to his baseball cap because he [10:33] could move his neck. He would [10:34] essentially type out letters on a [10:37] keyboard screen to get out words. In [10:39] fact, this is how he communicated was [10:42] through a device that he would [10:43] essentially peck out letters one by one [10:46] by moving his neck to control this stick [10:48] attached to his baseball cap. He hadn't [10:50] really spoken for about 15 years. [10:52] >> Oh, goodness. [10:53] >> Yeah. So, it was part of a clinical [10:55] trial. It was, you know, something that [10:57] our hospital and also the FDA, you know, [11:00] had to approve and looked at very [11:02] carefully. But given a lot of the work [11:03] that we had done, there was some basis [11:05] for for why this might work. And so we [11:09] did a surgery where we implanted [11:12] electrodes [11:13] onto these areas that control the vocal [11:16] tract, the areas that control the [11:17] larynx, the areas that control the lips [11:20] and tongue and jaw movements when we [11:22] normally speak. These are areas that [11:24] presumably may be active. That was our [11:26] hope. And he underwent a surgery, a [11:29] brain surgery. We put an electrode array [11:31] and we connected it to a port that was [11:34] sculled to uh screwed to his skull. And [11:36] the port actually goes through his [11:38] scalp. And he's lived with this now for [11:39] the last three years. So he has an [11:42] electrode array that's implanted over [11:44] the part of the brain that's important [11:46] for speech. It's connected to a port. [11:49] And then we connect a wire to that port [11:52] that translates those uh what we call [11:55] analog, you know, brain waves and [11:57] converts them into digital signals. We [12:00] put them through a machine learning or [12:02] artificial intelligence algorithm that [12:04] can pick up these very very subtle [12:07] patterns. You can't actually see them [12:08] with your eye uh in in the brain [12:11] activity and translate those into words. [12:14] And this is something that took weeks to [12:17] train the algorithm to interpret it [12:19] correctly. But what was incredible about [12:22] it was to see how he reacted. He would [12:25] be prompted to say a given word like you [12:27] know outside for example and then he [12:30] would think about it try to say it and [12:32] finally those words would appear on the [12:34] screen. And what was really amazing [12:36] about it was you could really tell that [12:39] he like got a kick out of that because [12:41] you know his body would shake in a way [12:43] and his head would shake in a way that [12:44] he would start to giggle. That was cool [12:46] to see, but then I also realized that [12:48] when he was giggling, it kind of screwed [12:51] up the next words. Decoding. [12:54] >> Is that a bug you've since uh fixed? [12:56] >> No, we haven't fixed that. It's easier [12:58] just to tell him to stop giggling. The [13:01] way this worked was we trained uh this [13:04] computer to recognize 50 words. We [13:06] started with a very small vocabulary [13:08] that's expanding as we speak. I think [13:10] that this is just a matter of time [13:12] before these vocabularies become much [13:14] much larger. But we started with a 50 [13:17] set of words. We created essentially all [13:21] the possible sentences that you could [13:22] generate from those 50 words. Why that [13:24] was important was you can use those all [13:27] those possible sentences to create a [13:29] computational model computer model of [13:32] all the different word combinations to [13:34] give different sentences given those 50 [13:36] words. And then you can essentially do [13:39] what we call autocorrect. It's the same [13:41] kind of thing that we do when you're [13:43] texting, for example, you get the wrong [13:45] letter in there. Your phone actually [13:47] knows, you know, because it's context [13:49] what to correct it. So because the [13:52] decoding is not 100% correct all the [13:54] time. In fact, it's far from that. It's [13:57] really helpful to have these other [13:58] features like autocorrect, the stuff [14:00] that we use routinely now with texting [14:02] that makes it correct and then updates [14:05] it. So, it's a combination of a lot of [14:06] things. It's the AI that is translating [14:09] those brain activity patterns, but it's [14:11] also things that we've learned from [14:13] speech and speech technologies that, you [14:16] know, you put all together and then all [14:17] of a sudden it starts to work. That was [14:19] the first time that someone was [14:21] paralyzed and could create words and [14:24] sentences uh that was just decoded from [14:26] the brain activity. [14:27] >> These days, we hear a lot about neural [14:29] link, Elon Musk's company. While brain [14:32] machine interface of the sort that you [14:34] do and that other laboratories do has [14:37] been going on for a long time, there's [14:39] been some press around neural link about [14:42] the promise of what brain machine [14:43] interface could do. What are your [14:45] thoughts about manipulating neural [14:47] circuitry to achieve suprahuman or [14:51] superhuman or super physiological [14:53] functions? And here we don't even have [14:55] to think about neural link in [14:56] particular. It's just but one example of [14:59] companies and people in laboratories [15:01] that are quite understandably [15:03] considering all this. [15:04] >> It's a really interesting time right [15:06] now. The science has been going on for [15:09] decades. The work that we've done in [15:11] this field that you call brain machine [15:13] interface. It's been going on for a [15:15] while and a lot of the early work was [15:17] just trying to restore things like arm [15:19] movement or having people or monkeys [15:22] control a computer cursor for example on [15:24] the screen. That's been going on for [15:26] decades. What's been really new is that [15:29] industry is now involved and some some [15:31] of this now becoming commercialized and [15:33] we're starting to see us now cross over [15:36] to this field where it's no longer just [15:39] research that we're talking about [15:41] medical products um that are designed to [15:43] be you know surgically implanted in some [15:46] cases you know there's people doing this [15:48] kind of work non-invasively as well that [15:50] don't require surgery. The specific [15:52] question that you were asking about is [15:54] an area that we call augmentation. So [15:57] can you build a device um that [16:00] essentially enhances someone's ability [16:03] beyond superan normal, super memory, [16:09] super communication speeds, beyond [16:11] speech for example, superior [16:15] uh precision athletic abilities. I think [16:17] that these are very serious kind of [16:19] questions to be asking now because as [16:22] you mentioned the pathway so far is [16:25] really to focus on these medical [16:27] applications. [16:29] I personally don't think that we've [16:31] thought enough actually about what these [16:33] kind of scenarios are going to look like [16:35] and I don't think we've thought through [16:36] all the ethical implications of what [16:38] this means for augmentation in [16:40] particular. There's part of this that is [16:43] not new at all. Humans throughout [16:46] history have been doing things to [16:49] augment our function. Coffee, nicotine, [16:52] all kinds of medications that cross over [16:54] from medical to consumer that is [16:58] everywhere. So the pursuit of [17:00] augmentation or performance or [17:02] enhancement is really not a new thing. [17:05] The questions really as they relate to [17:08] neurochnologies for example have to do [17:11] with the invasive nature. For example, [17:14] if these technologies require surgery, [17:17] for example, to do something that is not [17:19] for a medical application. Again, there [17:22] that is not exactly new territory [17:24] either. People do that routinely for [17:27] cosmetic kind of procedures for physical [17:30] appearance, not necessarily cognitive. [17:32] So, I do think that provided the [17:36] technology continues to emerge the way [17:39] that it does, that it's going to be [17:42] around the corner. And it probably is [17:43] not going to be in ways that are super [17:45] obvious. I don't think it's going to be [17:47] like can we easily memorize every fact [17:49] in the world, but in forms that are [17:52] going to be much more incremental and [17:54] maybe more subtle. In many ways, we [17:56] already have that now. Like for example, [17:58] you don't have to have a neural [17:59] interface embedded in your brain to get [18:01] information essentially access to all [18:03] information in the world. You just have [18:05] to have, you know, your iPhone. Whether [18:07] you could do it faster through uh a [18:10] brain interface, I definitely wouldn't [18:12] rule that out. But think about this that [18:14] the systems that we have already to [18:17] speak and to communicate have evolved [18:19] over, you know, thousands and millions [18:22] of years and they're supported by neural [18:24] structures that have bandwidth of [18:27] millions of neurons. [18:30] There's no technology that exists right [18:33] now that people are thinking about that [18:35] are in commercial form. sternly not even [18:37] in research labs that come anywhere [18:39] close to what has been evolved for those [18:42] natural purposes. So I'm essentially [18:46] saying two sides of this which is we're [18:49] already getting into this now. This is [18:52] not new territory. This topic of [18:54] augmentation both physical and [18:56] cognitive. We've already surpassed that. [18:58] That's part of what humans do in [18:59] general. But we are entering this area [19:02] of like enhanced cognition. um these [19:06] areas that I think the technology is [19:08] going to be the rate limiting step and [19:09] how far we can go and we have not had [19:12] the full conversations about number one [19:14] is this what we actually want is this [19:16] going to be good for society who gets [19:18] access to this technology these are all [19:21] things that are going to become real [19:22] world problems could you tell us what [19:24] you're doing in terms of merging the [19:26] brain machine interface with extraction [19:28] of speech signals from people who are [19:30] locked in like Poncho with facial [19:32] expressions [19:33] >> sure yeah I'm here with you in person. [19:35] We could have done this virtually [19:37] probably. It's pretty easy to do that. [19:39] We could have recorded this really [19:41] separate. But there is something about [19:43] being able to actually see your [19:44] expressions and to understand other [19:48] forms of communication. So, another [19:50] really important one is nonverbal the [19:53] expressions that you're making. For [19:55] example, if you have a quizzical look on [19:56] your face, if I'm saying something not [19:58] clear, that's a sign to me that I need [20:00] to rephrase it or to say it in a [20:02] different way or to slow down. Facial [20:05] expressions actually are really [20:06] important part of the way we speak. And [20:08] there's two things. It's not just the [20:10] expressions of like how you're feeling [20:12] and perceiving what I'm saying. But it's [20:15] also seeing my mouth move in your eyes. [20:19] actually see my mouth move and my jaw [20:21] move in a particular way that actually [20:23] allows you to hear those sounds better. [20:26] So having both the visual information [20:28] but also the sounds go into your brain [20:31] is going to improved intelligibilityly [20:33] also make it more natural. And the [20:36] reason why we're also very interested in [20:38] this idea of not just having text on a [20:41] screen, but essentially a fully computer [20:45] animated face like an avatar of the [20:49] person's speech movements and their [20:52] facial expressions is going to be a more [20:54] complete form of expression. Now you can [20:57] imagine right now that might just be [21:00] someone looking at a computer screen [21:01] interpreting these signals. But I think [21:04] the way things are going in the next [21:06] couple of years a lot more of our social [21:08] interactions more than even now are [21:11] going to move into this digital virtual [21:13] space. Of course most people are [21:14] thinking about what that means for most [21:16] consumers but it also has really [21:18] important implications for people who [21:20] are disabled right and whether how how [21:23] are they going to participate in that. [21:24] And so we were thinking really about for [21:27] people like Poncho and other people who [21:29] are paralyzed, what other forms of BCI [21:34] can we do in order to help improve their [21:36] ability to communicate. So one is [21:38] essentially building out more holistic [21:40] avatars, you know, things that can [21:42] essentially decode, you know, [21:44] essentially their their expressions or [21:47] the movements associated with their [21:49] mouth and jaw when they actually speak [21:51] to improve that communication. So, do [21:53] you envision a time not too long from [21:56] now where instead of tweeting out [21:57] something in text, my avatar will I'll [22:00] I'll type it out, but my avatar will [22:02] just say it. It'll be an image of my [22:04] avatar saying whatever it is I happen to [22:06] be tweeting at that moment. [22:07] >> That's what we're working on. That is [22:08] going to happen and it's going to happen [22:10] soon and there's a lot of progress in [22:12] that. And again, we're just trying to [22:14] enrich [22:16] um the the field of, you know, of [22:18] communication, expression um to make it [22:21] more normal. And we actually think that [22:24] having that kind of avatar is a way of [22:27] getting feedback to people learning how [22:28] to speak through a speech [22:30] neuroprothetic. That's the device that [22:32] we call it. It's a speech [22:32] neuroprothetic. That is going to be the [22:34] way that can help people learn how to do [22:36] it the quickest. Not necessarily like [22:38] trying to say words and having it come [22:40] on a screen, but actually have people [22:43] embody feel like it's part of themselves [22:46] or that they are directly controlling [22:47] that [22:48] >> that illustration or animation. I get a [22:51] lot of questions about stutter. What can [22:53] people with stutter do if they'd like to [22:55] relieve their stutter? Stutter is a [22:58] condition where the words can't come out [23:01] fluently. So, you have all the ideas, [23:03] you've got the language. In fact, you [23:05] know, remember we talked about this [23:06] distinction between language and speech. [23:08] Stuttering is a problem of speech, [23:10] right? So, the ideas, the meanings, the [23:13] grammar, it's all there in people's [23:15] stutter, but they can't get the words [23:17] out fluently. So that's a speech [23:20] condition and uh in particular it's a [23:23] condition that affects articulation [23:25] specifically about controlling the [23:28] production of words in this really [23:30] coordinated kind of movements that have [23:32] to happen in the vocal tract to produce [23:35] fluent speech [23:37] and um stuttering is a condition where [23:41] people have a predisposition to it. So [23:44] there's an aspect of stuttering. You are [23:47] a stutterer or you're not a stutterer. [23:49] But people who stutter don't stutter all [23:51] the time either. So you could be a [23:53] stutterer who stutters at sometimes but [23:56] not others. And really the the main link [24:00] between stuttering anxiety is that [24:03] anxiety can provoke it and make it [24:06] worse. That's certainly true, but it's [24:10] not necessarily caused by anxiety. It [24:14] can essentially trigger it or make it [24:16] worse, but it's not the cause of it, per [24:19] se. So the cause of it is still really [24:22] not clear, but it does have to do with [24:25] these kind of brain functions that we've [24:27] been talking about earlier, which is [24:29] that in order to produce normal fluent [24:33] speech, we're not even conscious of what [24:36] is going on in our mouths, in our [24:38] larynx, we're not conscious. And if we [24:40] were, we would not be able to speak [24:41] because it's too complex. It's too [24:43] precise. It's something that we have [24:47] really uh developed the abilities to do [24:49] and we do it naturally, right? It's part [24:51] of our programming and part of what we [24:53] learn inherently and you know it's just [24:55] through exposure. [24:57] So [24:59] stuttering is a is essentially a [25:02] breakdown at certain times [25:05] in that machinery being able to work in [25:07] a really coordinated way. [25:10] You can think about, you know, the [25:11] operations of these areas that are [25:13] controlling the vocal tract. Let's say [25:15] speech is like a symphony. In order for [25:16] it to come out normally, you've got to [25:18] have not just one part, the larynx, but [25:22] the lips, the jaw, they can't be doing [25:24] their own thing. They have to be very, [25:27] very precisely [25:29] activated and very, very precisely [25:31] controlled in a way to actually create [25:33] words. And so, in stuttering, there's a [25:35] breakdown of that coordination. If [25:38] somebody has a stutter, is it better to [25:40] address that early in life when there's [25:42] still neuroplasticity at is very robust? [25:46] And if so, what's the typical route for [25:48] treatment? I I have to imagine it's not [25:49] brain surgery typically. Um I'm guessing [25:52] there are speech therapists that that [25:54] people can talk to and and and they can [25:58] help them work out where they're getting [26:00] stuck in the relationship to anxiety. [26:02] >> Yeah, exactly. I mean, part of it is [26:04] about that anxiety, but a lot of it [26:06] really has to do with um therapy to sort [26:11] of like work through and think of tricks [26:13] basically sometimes to create conditions [26:16] where you can actually get the words to [26:18] come out. A lot of some forms of [26:20] stuttering are really initiation [26:21] problems. Just getting started itself is [26:24] is very hard. You want to start with [26:26] initial vowel or consonant, but it won't [26:29] emit. So a lot of the therapy is really [26:32] just focusing on like how do you create [26:34] the conditions you know for that to [26:37] happen. There's another aspect to it [26:39] that I find very interesting is that um [26:42] the feedback essentially what we hear [26:44] ourselves say. For example, every time [26:47] that I say a word I'm also hearing what [26:49] I'm saying. So that's what we call [26:50] auditory feedback. That turns out to be [26:52] very important. And sometimes when you [26:55] change that it can actually change the [26:57] amount someone stutters for better or [26:58] for worse. And it it's giving us a clue [27:02] that the brain is not just focused on [27:06] sending the commands out, but it's also [27:08] possibly interacting with the part that [27:10] is hearing the sounds. And there's [27:12] something might be going on in that [27:14] connection that that breaks down when [27:16] stuttering occurs. So there are [27:19] individuals that are stutterers, but [27:21] they don't stutter all the time. In [27:23] those instances, there's something [27:25] happening in those particular moments [27:27] where this very, very precise [27:29] coordination needs to happen in the [27:31] brain in order to get the words out [27:33] fluently. [27:34] >> Eddie, I have to say from the first time [27:36] we became friends, uh, 38 years ago, [27:39] >> something like that. [27:40] >> Something like that. To be sitting here [27:41] with you today for me is a absolute [27:44] thrill. Not just because we've been [27:46] friends for that long or that we got [27:47] reacquainted through the literally the [27:49] halls of medicine and science, but [27:51] because I really do see what you're [27:54] doing as really representing that front [27:57] absolute cutting edge of of exploration [27:59] and application. I mean, the story of [28:01] Poncho is but one of your many patients [28:04] that um has tremendous benefit from your [28:07] work and and now as a chair of a [28:09] department, you of course work alongside [28:11] individuals who are also doing [28:12] incredible work in the spinal cord etc. [28:15] So on behalf of myself and and everyone [28:17] listening, I just really want to thank [28:18] you for joining us today to share this [28:21] information, but also just for the work [28:23] you do. It's truly spectacular. So thank [28:26] you ever so much. [28:27] >> Thanks.