李沐称语音智能体商业化并非成熟 规模化仍存诸多挑战

李沐称语音智能体商业化尚未成熟 规模化仍存诸多挑战 大模型语音智能体在游戏与电销落地上面临技术局限与安全合规双重壁垒，从可用到真正规模化商业化仍需长期探索。 演讲通过“Stella”开放世界游戏和财富500强保险电销两大案例，深入探讨了语音智能体在实时互动与目标导向任务中的落地实践。李沐指出，团队通过自建数据中心和针对性的后训练攻克了早期GPT-4的高昂成本与开源模型的局限，并转向了能兼顾高智能与低延迟的双组件链架构。尽管语音智能体在特定领域已达到人类专家水平且展现出高度的架构可扩展性，但行业目前仍处于起步阶段，多角色复杂世界构建及任意新产品的快速适配仍是未来需要攻克的难关。 语音智能体（Voice Agent）的核心在于将端到端延迟严格控制在1秒以内，并以客户支持、电话销售或叙事推进等具体任务为导向。 (0:00) 在游戏场景实践中，团队发现从头预训练30B模型极易被开源大模型迭代超越，随后转向自建算力中心与专业人工标注后训练（Post-training），使特定剧情交互表现超越了GPT-4。 (4:59) 为打破传统ASR-LLM-TTS级联架构的高延迟与缺乏声学感知瓶颈，系统采用由30B理解模型与1B生成模型构成的双组件架构，实现边听、边说与异步深度思考。 (19:59) 面对合规严苛的保险电销场景，团队历时9个月将评估得分从早期GPT-4的55分提升至人类执照水平的80分，并能够通过语调实时识别客户是不耐烦还是认可。 (22:15) 李沐强调语音智能体商业落地仍处于早期阶段，当前在500至5,000美元价格带的特定产品表现良好，但多角色世界观构建与任意新品泛化仍需持续探索。 (24:59)

Deploying commercial voice agents requires balancing sub-second real-time responsiveness with rigorous domain control, necessitating specialized two-component architectures and targeted post-training over raw foundation model pre-training. Across both gaming and telemarketing, practical adoption depends on meeting human-level evaluation benchmarks while operating within strict latency and enterprise data security constraints. Two-Component Architectures for Sub-Second Latency (19:59) Real-time conversational agents must achieve an end-to-end latency under one second. The traditional three-stage pipeline—Automatic Speech Recognition (ASR), Large Language Model (LLM), and Text-to-Speech (TTS)—struggles with cumulative delay and cannot process acoustic nuances. Conversely, while end-to-end full duplex models allow fluid interruption, they remain difficult to control commercially. To balance speed, intelligence, and control, production systems deploy a two-component chain. In this architecture, a 30-billion parameter understanding model processes user audio and plans responses, while a lightweight 1-billion parameter generation model produces speech. For complex queries, the system asynchronously queries a fine-tuned larger model while speaking sentence-by-sentence. Training the understanding model requires a mixture of text and varied, real-world low-quality audio, whereas the generation component requires pristine acoustic data. Pivoting From Scratch Pre-Training to Domain Post-Training (4:59) Early development on the open-world sci-fi game 'Stella' exposed the limits of training custom base models. The team spent months pre-training a 30-billion parameter model on 5 trillion tokens enriched with fiction data, only to find it quickly outmatched by newer releases like Llama 2 70B. Furthermore, off-the-shelf GPT-4 proved economically unfeasible to operate at scale two years ago. The team resolved this by lowering GPU costs with internal infrastructure and shifting focus to targeted post-training. Twenty human labelers trained as game designers evaluated branching storylines to steer character behavior. The team established that specialized models must retain strong baseline general intelligence: if a leading closed API scores 90 on general tasks, a domain model must achieve at least 85 to prevent performance degradation on specialized tasks. Meeting Enterprise Compliance and Human-Level Benchmarks (22:15) Unlike gaming, outbound telemarketing in healthcare insurance operates under severe regulatory boundaries, prohibiting unsolicited calls and demanding extreme script precision. Deploying an AI telemarketer required achieving a score of at least 80 on professional licensing exams, matching human performance. Out-of-the-box GPT-4 scored only 55, requiring nine months of active engineering to hit the human-grade benchmark. Operational success relies on detecting acoustic intent—such as distinguishing an impatient tone from genuine agreement when a prospect says 'uh-huh'—and dynamically navigating playbooks exceeding 100,000 tokens. Strict data privacy rules prevent customer records from leaving corporate security perimeters, eliminating standard third-party API calls and compelling enterprises to deploy dedicated models in private cloud environments. Shared Core Workflows and Persistent Scaling Bottlenecks (24:59) Development revealed that underlying model architectures, pre-training methods, and post-training pipelines are interchangeable across contrasting verticals like gaming and insurance; the only adaptations needed are domain data and evaluation benchmarks. However, the speaker stresses that voice agents remain in an early 'day one' phase with unresolved operational limits. Dialogue coherence degrades sharply after 50 conversational turns, and managing multi-character interactions remains unsolved. In commercial telemarketing, deployment remains restricted to roughly five health insurance packages priced between $500 and $5,000. Generalizing to arbitrary product lines still demands extensive manual fine-tuning, and performance evaluation remains bottlenecked by the need for live human testing. Commercial voice agents demonstrate technical viability in focused single-character tasks, but broad enterprise scalability is limited by high evaluation costs, manual adaptation across new product matrices, and persistent multi-character narrative breakdowns.

[0:00] 李沐: just an agent with a voice [4:59] 李沐: huge loss of the revenue [4:59] 李沐: bad choice [9:59] 李沐: sitting floor [24:59] 李沐: day one

[0:00] Today I'm going to talk about voice agent. That's a very hard topic this year. And the reason is people think, okay, that technology is ready to use. And it's like it's ready to landing. So today I'm going to share two of our past projects based on voice agent. Show some lessons and the best practice how to help customers use it. And so let's get started. So first of all, what's a voice agent? Basically, that's just an agent with a voice. So user interacts with agent with a voice interface. So we hope that you have a more nature way to interact with larger language model. So two things here is that first of all, it's real time. It's a little bit different to video generation. video generation can be offline. Right now, the response must be within one second, for example, to end-to-end latency. The second one is like, most cases, people don't do chit chat. Chit chat is kind of companion, but in most cases you have particular tasks For example you want to do customer support you want to do information retrieval or you want to sell something Companion sometimes is a task It not just cheat chat. You have some particular goals. Maybe you want to tell a story. Maybe you want to do something else. So this is basically what's a voice agent. Then in the rest of the talk, we're going to have two examples show we how build it and what lessons we have. Here's the first example. And I really need to find a way out of here fast. Okay, I guess you're cutting straight to the chase here, but at least can you tell me a little bit more about yourself? Like... Okay, let's see. All right, I'm a sophomore at Mariana University studying astrophysics. So this is a game we started like two and a half years ago. So this is the app and this is the user. So this app is like this guy is called Stella. The user is going to do voice interaction with Stella to help her. So you can think that the whole story is driving by how the user inputs The task here is open game so like this this sentence is copied from the game which was already launched two months ago So basically Stella landed on an alien planet So the aircraft crashed here. So then she's asking the players to help. Like, okay, it's so So strange world, so many options, and I feel a lot of emotions here. So using voice to interact with player to let the player to help her to escape the planets. So that's a very large world setting. That's only the first game. It's a kind of trial game in a very large series. So what voice agent play here? The role here, you need to be both a game designer and actor. The game designer means that you want to design the story, which makes sense, which is fun to play. So then the agent one, you want to create a dialogue that matches the character setting. So this dialogue has a particular character setting like all the background she has or all the things she has like kinds of maybe 20 pages of setting Then that's the actor's thing. For the game designer, you need to guide. When the user integrates a game, you want to guide the storyline, which is if it's just a single line story, it's not agent, right? Not kinds of complex structure or even graph structure. and something have freedom here. The issue here is that you want to have a really good game. Like if you're familiar with how to write a book, how to write a game plot, it has a lot of principles there. Like you want to have all the stage, all the pace, all of the things, make the story look interesting. The other thing is like it's a game. You expect people to have arbitrary input with you, like the teeth and the player to try all the boundaries. The agent must be within their setting. Sometimes like this game is on a sci-fi world, like maybe 2000 year later, and you have a random chat setting. If you say, okay, what's a movie you watched recently? If you pick up a movie right now, it's maybe, wow, you watched it. [4:59] movie like 1000 years ago. But the things like all the language models are trained on the current data. Now you want to how to move all these settings to your future worlds. So I shared a particular like earlier log right here and show how some challenge task. So this is the log on the very earlier states, it's still in Chinese. So the idea here is that Stellar finds some food, asking the player to say, okay, which food I'm going to choose. The player settings here, I will help you. So I think you cannot eatable, you cannot eat anything. So then, oh, the Stellar first thing, like you need to find some meat to eat. So you have a rack to search how to catch animals here. But the answer is that you cannot, you didn't see any animal yet. So the stellar response, like, OK, I want you to admit, but I can only have a vegetable here. The player don't want to help. Stellar say, OK, I really need your help. Player still, like, I don't want to help. The thing is, like, if you start here, the story cannot move on So the problem is saying like okay after three trials just choose something by yourself And so, but then it's a random choice. Stella's dying. So I say, okay, he's dying. But the player say, okay, you're going to die. It's not a nice guy here, but you need to be nice here. So the challenge here is like, it's open with the game. But your response should make sense. Like, it's an open word game. It's like three years or maybe one, 2,000-year letter. Not every word setting is specified. A lot of things like you can, when you develop a game, the game designer cannot write anything for you. You need to think, okay, that thing maybe makes sense in a 2,000-year letter. Also, you need to be engaged in the form. So that's a game. That's not a chatbot. So that's all the challenges here. So what do we do is like the project launched two years ago, at that time you have GPT-4, but it's very expensive. We did some calculations thinking if you use GPT-4, like okay that a huge loss of the revenue And that time the best model is Lama 2 right now At that time Lama 2 is not strong enough So what we And at that time the best model is LAMA2 right now At that time LAMA2 is not strong enough So what we did at that time is we actually pre 30B model with Kendo 5 trillion tokens. But these tokens enriched on the friction gain roleplay data. The performance Kendo match on the LAMA2 on general tasks, a little bit better on the roleplaying. and but the lesson we got is like okay pre-trained model take a few months and even that you can't have outperformed NAMA 2 but you have NAMA 2 70b so like you spend too much time on pre-training and you maybe the the progress isn't so great so that's kind of the lesson we got that's we're gonna say why that's maybe a bad choice another thing we did is like okay because gpu so expensive. That's two years ago and we spent the effort to actually build data center by ourselves. So if you own a data center, the cost is much lower. And then we move to post training. The post training is that the key thing here, you have very complex storyline workflow. That's an example that not a real one And the real one is much more complicated And then we have kinds of 20 labelers We need to train the label to be a good game designer because like this particular way how you response Then ranking and evaluating all the model preference So using these two, we spend kind of quarter here and can outperform GPT-4 on this particular scenario. For all this game, we use human to play so you can outperform GPT-4. But a question here is like, okay, that's a single game. Again, that's the tiny bit of a whole open world game. What if you want to do multiple games? Then we get another phase, is that we want to expand to a broader range of games and characters. So you can less rely on a prompt engineer. At that time, a prompt engineer is very complicated right now. And even different versions of GPT-4 is a very sensitive prompt engineer. So the thing is, yes, we can help, But you want to design, game designer to write prompt engineer. So the idea here at that time is like, it's still like one half year ago. The idea at that time is like you need to pre-trained a real world model. Can distinguish which one's good, which one's bad. And because you cannot rely on humans to do it. So at that time we first trained a real [9:59] or model to tell you which response is good in this game setting, then you can post-train another model for it. The one important lesson we got is like, even this is for game, sounds like simple, but it's still a lot of things like instructing following, you need to make sense. So the model still need to be very general enough. So even that you train the in-domain model, the model must be good in general tasks. If you think, okay, the best open AI or best closed API, the score is 90. But in your application, the general task need to be 85. If it's lower than that one, you're going to think you have a sitting floor on your task. So that's the first guarantee. The general task is good. Then for your particular task, we kind of create an in-domain evaluation. So you have a lot of character settings, a lot of sync settings for different game settings. You want to make sure like under this setting, you follow all the settings and the response is good. Follow the instruction follow the sync And then once you have this benchmark you can tune the model so that you can be the best compared to others I think that a very general pattern that you care about in performance The key thing here, like you really want to develop a really good in-domain evaluation task. So you can say, I can see the model improve on this one. But at the same time, guarantee the model performs well on all these other general tasks. so the lessons we got is like the intelligence came from pre-training so after we finish the whole project but then we think backwards all the big improvements from pre-training on mass data so it makes us rethink maybe we give up on the pre-training maybe a bad idea we maybe spend another quarter on pre-training so that's kind of the lesson we got and but still like it's still limited the dialogue quality declines after 50 turns is still right now like given the complex setting after 50 turns conversation then you think that maybe the model can become the intelligence much lower also the models nowadays still struggle with complex word setting and you have multiple characters so it still hard right now even like if look at all this voice model all this video model kind of two to three characters that's the limit. Even for the text part, if you have four characters, it's very challenging right now. The other thing is like, in the demo you see, the latest is big. It's a turn-by-turn based. So all the projects we hear is focused on the larger language model itself. Then the lesson we got is like, if you really want to truly human-like interactions, you kind of need to tune the architecture a little bit. You're not by the traditional, like three component and architecture. So that's become our next project. Is this a good time to talk? Actually, I'm just about to head out the door. No problem, John. I know how busy things get. If you'd prefer, I can give you a call back at a time that's better for you. To be honest, I think I'm all set. I already have health insurance through my job. That's great. And I'm glad you're covered. A lot of folks I talk to do have something through work, but many don't realize there are options that could lower their out-of-pocket costs. Okay that a very different one It sale insurance So before it again right now we sell insurance At the first time I think maybe selling insurance like you can be very creative you can do whatever you want to sell something, but in reality, it's pretty, very formal. First of all, you cannot call anyone you want to call. The user you called must be submissive information, so it shows their interest on your product. Secondly, it's very highly regulated, the whole insurance industry. So let me explain this problem. So right now we do AI telemarketer. The agent rose like is telemarketer. So right now this particular example is like we sell health insurance through phone and but on multiple countries. The thing, the requirement, you have two requirements. You must pass the telemarketing certification. Like the human, you have 80 scores to pass. This one must be passed this certification before you can launch. Secondly, you have some performance metrics. You are able to sell at a particular threshold. Like giving you like 1,000 customers, you call them. You must be able to sell a particular number of sales. Also, the complaints must be low particularly. [14:59] If the people say, okay, I feel so bad, like you said, not true information, all the experience is bad, they're going to complain that insurance companies really care about it. So the capacity you need here, first of all, you need to be intelligent. You need to follow the sales playbook with precise answer. We're going to show what the precise answer means. You need to be able to use tools because insurance, you have a lot of internal tools to query and some math. like you need to do a lot of combinations here. Also, you need to be very human-like. And if you call someone, maybe this guy's out of the door, like a lot of noise, and maybe have some accent. And also, you need to have some, and the voice need to be realistic. It's not so robotic. And the last one is like end-to-end latency. When I finish my sentence, your response must be within one second. Otherwise you feel like it's not so responsive. So the precise response, like for example, if you respond, you can cover up to $600. That's wrong. Totally wrong. You failed this exam. Because the precise answer the precise one is like cover for some common one And like the only for the front teeth So that on their product information So if you have any issue with your teeth, that's not right. If you have particular like A, B diseases with your teeth, that's the right answer. If you respond to this one, it's going to be, you have failed the exam. The other thing, the other more challenging thing here is similar to the gaming. like when you call some customer and try to ask him, okay, can we grab time to talk about the events? You're going to try three times. If you cannot reschedule before three times, or you can reschedule later. So for example, I try the MR, try first one, the telemarkers try first one, no thank you. Then you try second time, no. Then you try the third one, If the user says, uh-huh. So you maybe think, uh-huh, maybe interesting. If you think the emotions interesting because you change the world, you change like, can you tell you, okay, like how it can benefit your personality, a personal, then you maybe think okay it a uh maybe the user interest but in reality it like you need to find out the voice is impatient Then given the contact you can think okay, I already have tried three times, I need to reschedule. So that's the difficulty when you have the audio as inputs. So then one key question here, how do we do real time? So I kind of show examples how different mode architecture we have right now. The first one is the fanciest one, it's called end-to-end full duplex. It means like you have user, you have your model, a single model, the user speaks to you, which is all the waveforms come in. Then you listen to the waveform and respond to anything during the interactions. So in this case, it's easy for users to interrupt. Also easy for the model to do some feeling words, like a user say something, say a long sentence, you can say, yes, yes, that's right. So that's the most natural way. But none of this system is deployed right now. There's one or two demos you can try, but it's very few, it's not so controllable. So in most cases like if you using even GPT I think they using the end half duplex It means that when the user speaks you have a voice active detector to detect if the user speaks or not So you have chunks of chunks go to the model, and the model is going to respond to the previous one. So that's the half duplex. Another one is a channel solution. Still similarly, you have turns, but here you have two models. not just single model. So these two models, the first is understanding model, give the audio in, generate the text response. Then the text goes to the generation, generate the audio outside. The last one is called chain three components. Like you have just do ASR, which is transcribed audio, go to the log generation model and just and then get a response, go to the TTS, which is to generate audio. So for this different one, this one's human-like, very human-like because the model can interrupt you. And the last one is like, if you go that direction, it's easy to customize because like you can much easier to adding a new [19:59] capacity into the agent. So what we typically use for customers is using the two-component chain solution. So you have, for example, we're using 30B understanding model to generate response. But if the user query is complex, maybe using a fine-tuned larger model to do syncing as a tool I use. So then it goes to 1B generation model to generate the response. Nowadays, all this model is based on single, is the same larger language model, all based on the same LLM. But you can either continue pre-trained or fine-tuned with different data mixture. For example, for the understanding, you need to have minutes of hours of really different quality of audios. You maybe want to have a lot of low quality audios. Also because you want the understanding model generated response, you want to have a lot of text tokens to continue to pre-trans as well. Otherwise, it's just the audio model. The generation model you want to have an even more high quality hours of audios The larger language model you kind of want to train on some domain data So this architecture makes it easy to customize. Because this is kind of like understanding and the generations, kind of general purpose one. You have this model you can maybe use in different scenarios, but if you go to a particular scenario, you just fine-tune this model. How to get both intelligence and low latency, that's a key for voice agent. There's a bunch of ideas here. First of all, you want to lessen, talk, and think at the same time. You lessen, and you generate a response, sentence, and then between that, while you call the large generation model to think, maybe I want to respond better, maybe I want to do some search better. But all the things can be asynchronous. The other one is like, you want to do context engineers. It's a one step beyond prompt engineer. That's because for your problem, you maybe have very non-context, like the product information, and also the other playbooks, kinds of like maybe 100K tokens. You want to dynamically generate, construct the context generate the prompt The other thing you have an organizer which is handle different strategies like, OK, what kind of user do you think this user is? And then think about different strategies, and also do intent analysis, for example, how to count the unhunt, and some do live task tracking. So all the things, together, you can get both intelligence and low latency. So that's kind of the project progress we did. It started this year. We partnered with Fortune 500 issues later. So we started here, January. February, you have, using ChappGD4, you got this kind of 55 score. But the thing is that you need to pass this line. It's a human performance. It's 80. You must pass this line to be launched. You can see it struggles a lot, but then you have steadily progress into and how you can match human. You take hands off half a year, or three quarters actually. The lesson here is that the evaluation of the end-to-end voice agents is pretty challenging. Like because you need to have a real human to make a call Once you have a call it much harder to do like automatic evaluation But that the key If you don have this one it really hard to know the whole end performance. And this is ongoing in that handling complex product compilation in real time, still pretty difficult. Like for insurance, you have a lot of product combinations, how to handle them, the price is different. Maybe I, okay, that's too expensive for me and I want a cheap solution, then you need to pick up the right one for that. The last one, the high security setting make the cost higher. We have the panel discussion talk about if there's only open and maybe dominant words. And the 2B area is not. The reason is because for insurance, if it launched in different country, the data cannot go out of this country or even more, the data cannot go outside the security group of the company. So either way, you can rent GPT model on your account, running on your account, or you need to develop your own model. So that's why all the things are struggling here. So also that's why we spent so many efforts to develop the whole model by ourselves, rather than just maybe proper engineer opening it. [24:59] just API. So I showed two examples how we developed voice agent in the past two years. The lesson we got is like the voice agent is pretty highly scalable. Even that the GAN settings, the insurance settings are very different, but the technology-wise same model architecture, same technologies here. Only things like maybe data is a little bit different and the evaluation is a little bit different. You need to spend a lot of people that one but um the model architecture and also the methods how you post trend how you pre-trend how you like all the things it's the same same thing from game to telemarketer it's very different one again you want to be fun telemarketing you want to be very precise but handle the user input very carefully but still like i think right now it's able to long landing in these areas but still on the day one setting. The reason is like for game, it's just a very simple game right now. Single character, a small world setting. But how about you want to do a really multiple character really large world settings That really hard right now For telemarketing right now we can maybe sell kind of five different health insurance for a particular company with some certain combinations It's hard to sell general purpose. I think generally this telemarketing is really good for any product between $500 to $5,000. That's really good for this telemarketing to sell. But right now, if you use this trend model to sell any arbitrary new products, you still need a lot of tuning right now. Also, there are a lot of other scenarios. Like before, it's like customer service, all the things based on just text, large-end-aggregate model. Right now, you can add in a voice interface to this application. So there are a lot of applications here. So I think that's why I think we're able to land to product right now, but still on day one. So we have maybe another few exciting years to go. Lastly, if you're interested to work with us or partner with us, just contact us. We have a booth, you're gonna be here, our co-founder will be here, welcome to talk to us. Yeah, that's all, thanks everyone.