微软CoreAI主管称大模型难独立履职 促企业设专属身份与评分标准

微软CoreAI主管称大模型难独立履职 促企业设专属身份与评分标准 通用模型与泛化指标无法识别真实业务缺陷，企业唯有为Agent建立独立身份以隔离越权风险，并用包含加权规则的业务评分标准持续迭代，才能使AI安全投入生产。 对话深入剖析了大模型在企业落地的现实鸿沟，强调模型本身仅是预测概率的引擎，必须依靠运行框架（Harness）和结构化上下文才能运转。针对落地痛点，访谈重点探讨了企业级 Agent 的独立数字身份与最小权限原则，以及宝马团队耗时半年整理数据字典以打通海量传感器数据的真实案例。面对业务评估盲区，Marco 提出了基于细粒度规则与权重的自然语言评分标准（Rubric）以及漫长试点期的必要性，并指出前置部署工程师（FDE）与专职评估员正成为行业新需求，呼吁开发者将 AI 视作一门手艺亲自动手实操。 大模型本身只负责 token 预测，必须依赖运行框架（Harness）组装上下文、工具与权限，通用模型不会轻易吞噬垂直应用与专用端侧模型。 (1:13) 企业级 Agent 必须具备独立数字身份并遵循最小权限原则；仅靠“扮演”用户会直接继承全局权限，带来误删本地或云端数据的灾难性风险。 (6:00) 宝马在改装测试车上部署了 5000 个传感器，单次运行产生超 1TB 遥测数据；团队耗时约半年梳理跨部门数据字典与业务术语，才让 Agent 具备跨越隐性知识查询刹车数据的能力。 (17:32) 传统“任务完成度”等通用评估指标无法捕获深层业务错误，行业正全面转向自然语言定义的评分标准（rubric），通过区分合规红线与业务容错权重，并将真实用户的报错原因写回标准闭环迭代。 (24:03) 企业落地 AI 应遵循“先写评分标准作为准出标准”与“预留异常漫长的试点期”两大原则，这也推动了评估员（Evaluator）和前置部署工程师（FDE）等新职业的诞生。 (38:02)

微软CoreAI副总裁Marco Casalaina指出，底层大模型仅是逐词预测概率的引擎，无法凭空解决实际业务；企业落地AI的关键在于构建运行框架、赋予智能体独立安全身份，并建立基于业务评分标准的持续验证闭环。 运行框架与专用模型破除基座全能神话 (1:13) 通用大模型能力日趋强大，但Marco明确否定了“模型将吞噬一切应用”的观点。他强调模型本身缺乏环境感知与执行能力，必须依赖UI、状态管理和工具链等运行框架（Harness）来承载上下文并调用API。微软的M365 Copilot等产品本质上都是高度复杂的运行框架。 在垂直场景中，端侧模型与专用模型展现出不可替代的壁垒。例如Marco本人在离线设备上深度依赖定制微调的Whisper语音转写工具以精准识别专有名词；在文档处理方面，微软Foundry虽然集成了超过11500个大模型，但处理数十亿文档的主力依然是专用的文档智能套件，以解决跨页表格缝合、复杂版式还原以及从右向左语言翻译时的版面镜像反转等现实脏活。 独立身份与最小权限原则构筑企业安全边界 (6:00) 消费级Agent在执行订票、发邮件等操作时通常只是在“扮演”人类用户，这导致它全盘继承了用户的全部系统权限。一旦模型出现幻觉或遭遇Prompt注入，Agent可能误删整台电脑的文件。企业级应用如微软推出的Autopilot，核心分水岭在于必须赋予Agent真正独立的数字身份（拥有专属Teams账号、企业邮箱与存储空间）。 通过将Agent与自然人用户物理隔离，企业能够在底层目录服务（如Entra ID）上贯彻“最小权限原则”。独立身份的Agent在未获显式授权时无法接触用户文件；它在组织架构中更像是接受委派的“数字雇员”，拥有独立的权限白名单、审计日志和责任边界。这种设计防范了越权灾难，但也伴随着对传统跨软件搬运“胶水岗位”和中层协调管理的重构冲击。 业务落地必须补齐数据字典与隐性知识 (17:32) 真实复杂的工业场景无法仅靠接入大模型就自动化运转。以宝马MDR（移动数据记录仪）项目为例，每辆工程测试车装载了5000个传感器，单次运行产生超过1TB的遥测数据，原先必须通过物理寄送硬盘并依赖少数被称为“高手”的专家手动写查询导出。大模型虽然擅长编写代码，但光看原始数据根本无法分辨刹车温度或判定急刹车事件。 为了将专家的隐性知识显性化，团队耗费了大约半年时间深入宝马各部门，梳理出包含每个传感器物理指标、阈值及业务术语的数据字典。只有当这些业务上下文被转化为AI可读的标准规范后，工程师才能够通过自然语言多轮查询潮湿天气下的刹车事件，真正实现技术赋能业务。 从通用测试演进至基于评分标准的动态评估 (24:03) AI智能体具备非确定性特征，无法采用传统软件工程的二元“通过/失败”单元测试。Marco指出，行业过去依赖的“任务完成度”等通用指标存在致命盲区。例如在公寓定价案例中，Agent返回的电子表格在通用任务指标下会全部亮绿灯通过，但人类业务专家介入审查后却发现三居室定价严重偏离市场，通用指标完全不懂业务规则。 当前的破局解法是转向“基于评分标准的评估”（Rubric-based Evaluations）。企业应在立项之初就用自然语言制定包含多维权重的业务规则（如银行严格合规为高权重、几美元扣费纠错为低权重），并将其作为生产准入阈值。当专家发现输出错误时，不能只报错误，而是必须将“为什么错”写回评分标准并直接作为Agent的上下文，形成双螺旋迭代；同时企业需顶住上线焦虑，预留出异常漫长的试点期。 Marco总结认为，应对AI快速迭代的有效方式是保持动手实践的手艺人心态，企业也正因此催生出专职的“评估员”（Evaluator）和前置部署工程师（FDE）团队。但技术落地的核心始终在于人：无论是隐性知识梳理、评分标准制定，还是权限审批与最终交叉验证，人类必须在每个关键节点履行“信任但验证”的把关责任。

[0:00] Marco Casalaina: 首先要记住，模型本身并不会做任何事。它们只是一个 token 接一个 token 地预测，本身并不执行动作。 [38:02] Marco Casalaina: As you're thinking about what it is you're gonna build, especially if that thing is AI centric, then consider that the rubric is your success criteria. Write that up front. [8:13] Marco Casalaina: 比如，我自己的电脑，我有权删除里面的所有文件。如果 Agent 以我的身份运行，真出问题时，它也可以删掉我电脑上所有文件。 [38:02] Marco Casalaina: for AI, you kind of need to have an abnormally long pilot period. [12:47] Marco Casalaina: 任何 AI 要取得成功，认可和授权都是关键。

[0:00] 模型本身，并不会做任何事。 Models by themselves don't do anything. [0:02] 每辆车上，真的有5000个传感器。 There are literally 5,000 sensors on these cars. [0:04] 他们花了大约半年，跑遍宝马各个部门， They spent like six months going all around BMW [0:07] 整理出一份数据字典。 and gathering a data dictionary. [0:10] 如果Agent是以我的身份运行， And if this agent is impersonating me, [0:12] 真出问题时，它也能删掉我电脑上的所有文件。 it too can delete all of the files off of my computer if it really goes sideways. [0:16] “任务完成度”会说：“没问题。” And task completion will be like, 'You're good.' [0:19] 但我打开表格，仔细一看， But then I open the spreadsheet, I look at the spreadsheet, [0:21] 一居、两居都还挺准， and I'm like, well, the one and two bedrooms, those are pretty good, [0:24] 可三居室，偏得离谱。 but those three bedrooms, man, they're way off. [0:25] 你不能只说：“这个价格不对。” And you can't just say, 'This price is wrong.' [0:28] 你要说：“这个价格不对，因为……” You have to say, 'This price is wrong, and here's why,' [0:31] 这个“为什么”，之后要写进评分标准。 because that why, that's what goes into the rubric down the line. [0:35] 信任，但要验证。 Trust, but verify. [0:40] 我和团队通常是最先试用微软新技术的人， So my job really is me and my team, we're the first people to try anything [0:44] 也会尝试全球其他地方的新东西。 new at Microsoft generally, although also in the rest of the world as well. [0:49] 我们会实际试用，判断这些技术能怎样服务客户。 And so we really try these things and determine how they're gonna apply to our customers. [0:56] 也要想清楚怎么向客户解释， how we can explain them to our customers because sometimes it's not [0:58] 因为新的AI能力能用来做什么，有时并不明显。 super obvious what these new AI capabilities are going to be good for. [1:02] 有时是拿着方案找问题，有时则真的解决了问题。 Sometimes they are a solution looking for a problem, but sometimes they are a real solution to a problem. [1:07] 我们需要把它们用起来， And we really just need to kind of break them in and understand [1:10] 弄明白它们怎么工作、有哪些细节。 like how they work, the nuances of them and that sort of thing. [1:14] 这正是我最想问、观众应该也很关心的问题之一。 That's actually one of my most important questions, and I think a question the audience cares a lot about. [1:21] 大模型会不会把一切都吃掉？ That is, would just would the big models just eat everything? [1:25] 你试过这么多新东西。 Because you try so many different new things, right? [1:28] 但有一种观点是， But the argument is you just need GPT, you just [1:31] 只要有GPT、Codex或者Anthropic的模型就够了。 need a Codex, you just need Anthropic model. [1:34] 上面就不再需要其他应用了。 there's no AI applications on top of that. [1:37] 你怎么看？ What's your opinion on that? [1:39] 首先要记住，模型本身并不会做任何事。 Well, first of all, let's bear in mind that models by themselves don't do anything. [1:43] 它们只是一个token接一个token地预测，本身并不执行动作。 I mean, models by themselves, they predict token by token, but they themselves actually do nothing. [1:47] 要让它们做事，必须把它们放进某种运行框架， So they have to be embedded within a harness [1:49] 也就是harness里。 of some sort for them to do anything. [1:52] 比如我们的M365 Copilot是一种， And so certainly our own M365 copilot is [1:55] GitHub Copilot是另一种。 one harness, GitHub copilot is another. [1:57] 还有很多大家熟悉的消费级运行框架。 There are plenty of consumer harnesses that many people are familiar with. [2:01] 是模型与这些运行框架的配合，产生了我们今天看到的效果。 and so really it's the interaction of the harness and the model that produces the results that we see today. [2:08] 正如你所说，模型确实在越来越好。 And these models are indeed, as you say, getting better and better. [2:11] 每天都有新模型，来自我们，也来自其他模型公司。 Every day there's a new model from us or from one of the other model building houses that are out there. [2:16] 它们当然还在持续进步。 And so they continue to improve, for sure. [2:20] 与此同时，模型定制仍然有用武之地。 At the same time, there is definitely a place for model customization. [2:25] 接下来你会越来越常见到端侧模型。 One of the things that you're going to start to see now, I think, more often is models on the edge. [2:29] 我正在用来和你通话的这台笔记本上，就有一个我经常用的模型。 And in fact, there is a model that I use on this laptop on which I'm talking to you quite frequently. [2:35] 我用的是Handy。 I use Handy actually. [2:37] Handy是一个开源语音转写项目。 Handy is an open source project for transcription. [2:39] 它的转写效果非常好。 It does transcription really, really well. [2:42] 录播客之前，我不是给你写了一封邮件吗？ And so, you know, I wrote an email to you in advance of this podcast. [2:46] 那封邮件不是我打字打出来的， I didn't type it. [2:47] 很可能是口述的。 I probably dictated it. [2:49] 我已经把语音输入调得很顺手了。 And I've gotten this dictation really dialed in. [2:53] 而且它在本地运行。 But it runs locally. [2:54] 这个模型完全跑在我的笔记本上。 This is a model that's running entirely locally on my laptop. [2:57] 不会上传到云端。 Does not go to the cloud, nothing like that. [2:59] 这样的东西会越来越多。 Now we're going to find more and more of these things. [3:01] 硬件能力在增强。我这台电脑比较新， As the hardware becomes more and more capable, I have a relatively [3:05] 跑这些模型已经相当有能力。 new computer here, so it's pretty capable at handling these models. [3:10] 随着硬件越来越能跑得动模型， But as the hardware becomes more capable of handling these models, I think you're going to start [3:13] 我认为专用模型会大量涌现，比如转写模型。 to see an explosion of models that have custom purposes that like the transcription model. [3:19] 它不是通用模型，不是什么都做，而是专门做转写。 It's not a general purpose model, it doesn't do whatever, it does transcription. [3:24] 我知道模型能做这些，但另一个问题是：我为什么需要这个专用的？ I think the model can do it, but the other question is why do I need this one? [3:31] 比如，我直接用Whisper或者GPT实时模型，不也可以吗？ when I can just have Whisper, for example, or GPT real time to do this for me. [3:37] 尤其你的客户很多是企业。 Especially like your clients or enterprise. [3:40] 现在大家也在争论主权AI：企业该拥有自己的模型， And I know it's actually a debate going on right now, like sovereign AI and you know, enterprise, [3:47] 还是把所有数据交给Anthropic和OpenAI？ shall we have our own model or shall we just give every data all the data to Anthropic and OpenAI? [3:53] 这背后的真实需求是什么？ So like yeah, what is the demand for this? [3:58] 说实话，我在本地用的这个模型就是Whisper。 Well, I mean, to be perfectly honest, what the model that I'm using locally here is Whisper. [4:02] 不过，我做了不少定制。 However, I have made a number of customizations to it. [4:05] 比如，假设我经常给你发邮件。 I mean, for example, let's say that I was commonly emailing you. [4:08] 我可能就得教它怎么识别“Yuzheng”。 I would probably need to teach it how to say Yuzheng, right? [4:12] 它不会天然就知道你的名字。 It like it wouldn't just know how to say your name, [4:15] 而且我自己都不一定读得对。 which I may or may not be pronouncing correctly. [4:17] 不管我怎么读，模型能不能把它拼对？ But in any case, regardless of how I pronounce it, you know, will the model spell it correctly? [4:22] 可能能，也可能不能。 Mm, I mean, maybe, maybe not, right? [4:24] 但如果像我这样稍微调整一下，它正确拼写的概率就会高很多。 But if I adjust it a little bit as I have been doing, then it's much more likely to spell it correctly. [4:31] 所以你的意思是，针对具体场景定制的模型，会比直接开箱使用更好？ So what argument is this customized model just works better for a scenario than out of the box? [4:38] 没错。 That's right. [4:38] 定制视觉模型这些领域，多年来也是如此。 And you'll see that, I mean, we have seen that for many years with things like custom vision models. [4:42] 比如我们的文档智能产品。 I mean, for example, in our own document intelligence. [4:45] 微软Foundry的模型目录有大量模型， So in addition to the general purpose, you know, of course, we have on Foundry, we have with Microsoft, [4:50] 大约11500个，现在可能更多了。 we have many different models, literally 11,500 models on the model catalog, probably more than that now. [4:56] 其中很多是GPT、Claude、Mistral这样的通用模型。 and most of them are general purpose models like GPT and Claude and Mistral and stuff like that. [5:01] 但我们还有文档智能套件，每天处理数十亿份文档。 But we also have this document intelligence suite, which processes billions of documents a day. [5:07] 它不仅专门读取文档， and it is made specifically to read documents, but [5:11] 还能识别文档的版式结构。 moreover, to find their layouts and stuff like that. [5:15] 而通用模型在这方面往往不够好。 And that's something that general purpose models have a tendency not to do wonderfully. [5:18] 比如文档里有一张跨页表格， Like if you have a document and it has a table in it, that table cuts across pages, [5:22] 通用模型经常没法把它正确拼起来。 then general purpose models have a tendency not to be able to stitch that together. [5:28] 但我们的文档智能模型可以。 But our document intelligence model can do that. [5:31] 它做得很好，我们还基于它做了文档翻译。 In fact, it can do that so well that we also have a document translator. [5:35] 翻译时可以保留原来的版式，只把内容换成另一种语言。 And the document translator will take the layout and reproduce it, but in some other language. [5:40] 如果从英语翻译成希伯来语、阿拉伯语这种从右往左的语言， And if I translate from like English to Hebrew or Arabic, [5:44] 它还会把版式方向翻过来。 like a right to left language, it'll also flip the layout. [5:49] 确实挺不可思议的。 it's pretty crazy, actually. [5:50] 实际看到翻译结果时， I mean, when you do it, it's it's kind of melts your mind a little bit when you [5:54] 你会有点惊讶：它到底怎么做到的？ you actually do these translations because it's like, well, how did you do that? [5:58] 太厉害了。 That's nuts. [6:00] 听你描述这些模型的应用，我意识到一件事。 And I think by describing the usage, like the application of this model I realized something. [6:06] 我们大多数人接触新技术时，都是以消费者的身份。 that is most of us like interact with new technologies because we are a consumer. [6:12] 但你在微软主要面对企业和大型组织， But Microsoft, your job is mostly like talking with enterprise [6:16] 它们的使用场景可能很不一样。 or large organizations and their use case can be very different. [6:21] 能不能带我们了解一下，它们都在做什么、怎么用AI？ Yeah, so help us understand a little bit about their world, like what are they doing, how they're using AI. [6:27] 可以。我能很具体地对比一下消费级技术和企业技术。 Well, actually this I can I can make a very specific contrast between consumer technology and business technology. [6:33] 当然也别忘了，微软也有规模很大的消费级业务。 And let's not forget, I mean, Microsoft does have a pretty large consumer offering also, right? [6:38] 比如Xbox、Bing、Windows这些。 In terms of like Xbox Bing, Windows, and stuff like that. [6:42] 所以我们也服务消费者。 So yes, we do have a consumer footprint as well. [6:44] 但确实，在我这个部门，我主要接触企业和政府客户。 But yes, it's true, at least in my division, that I am mostly talking to enterprises and governments. [6:49] 也有很多创业公司在用我们的技术。 Although, you know, there are a lot of startups as well that use our technologies. [6:54] 我这次在AIE上海会讲到一个主题： Now one of the things that I'm going be talking about at the AI [6:57] 拥有独立身份的Agent。 Engineers Conference in Shanghai is agents with their own identity. [7:01] 这是一个正在兴起的趋势。 And this is an emerging trend. [7:02] 现在到处都能看到。 You're starting to see this everywhere. [7:03] 最近新出现的消费级Agent也体现了这一点。 And you see this lately in some of the new consumer agents that have recently showed up on the scene. [7:09] 你登录进去，第一件事就是让你给Agent起个名字。 So there's these consumer agents, and when you log into them, the first thing they ask you is to give them a name. [7:15] 我自己就有一个。 So I have one of these things. [7:17] 我叫它Lily。 I named it Lily. [7:18] 其实是随便起的。 kind of randomly, actually, I don't know. [7:19] 就是突然想到这个名字。 I pulled that name right out of the air. [7:21] 这种Agent有一定的身份，但很有限。 So That kind of agent, it has a limited form of identity. [7:25] 微软也有我们称为Autopilot的Agent。 We at Microsoft, we also have what we call autopilots. [7:28] 我们是在5月，不，应该说6月的Build大会上发布的。 And we announced that just back in May at our or June, I should say, at our build conference. [7:32] 这就是Autopilot这个概念。 So we have this notion of autopilots. [7:34] 它也是拥有自己身份的Agent。 This is also an agent with its own identity. [7:36] 但这件事对企业和对消费者，含义很不一样。 But it means something very different for a business than it means for a consumer. [7:40] 比如我那个Lily去执行动作， So if I use my little Lily agent over here, I mean, ultimately, when it takes an action, [7:47] 预订旅行、写邮件之类的。 you know, if it goes and books travel or writes an email or any of that stuff. [7:51] 它是以我的身份在做。 It's doing it as me. [7:53] 也就是在代替我操作。 So it is impersonating me. [7:55] 它虽然有名字，但除此之外，身份其实很薄。 So it may have a name, but beyond that, it's a pretty light identity. [8:00] 执行动作的时候，它并没有真正独立的身份。 Like it doesn't really have its own identity for the purpose of taking actions. [8:05] 而Autopilot的一项基础能力， But one of the fundamental capabilities of our autopilots, which also have their own [8:09] 就是拥有自己的身份。当然，你可以给它起名字。 identity, I mean, yes, you can give them a name and that's all well and good, but they [8:13] 但它还会有自己的Teams账号、邮箱、文件空间等。 also get their own Teams box, their own email address, their own file space, and all of these kinds of things. [8:17] 它能以自己的名义执行操作。 They can take actions on their own behalf. [8:20] 对企业和政府尤其关键的是：它在自己的安全上下文中运行。 And crucially, for businesses in particular and governments, they run in their own secure context. [8:27] 如果Agent是在扮演你，它就运行在你的安全上下文里。 So when you have an agent that's impersonating you, it is running in your security context. [8:32] 它会继承你拥有的所有权限。 It inherits whatever permissions you have. [8:35] 比如，我自己的电脑， So for example, like if I have an agent impersonating me, [8:38] 我有权删除里面的所有文件。 I have the ability to delete all of the files off of my computer. [8:42] 如果Agent以我的身份运行， And if this agent is impersonating me, it too can delete all [8:45] 真出问题时，它也可以删掉我电脑上所有文件。 of the files off of my computer if it really goes sideways. [8:48] 但如果我建一个有独立身份的Autopilot， But if I make an autopilot and it has its own identity, [8:51] 它首先就不能直接删除我的文件。 then first of all, it can't just delete my files. [8:55] 它甚至看不到我的文件。它是另一个用户， It can't even see my files because it's a totally [8:57] 就像你和我不是同一个用户一样。 different user, just like you're a different user from me. [8:59] 除非我明确授权，把这些访问权限给它。 Unless I specifically give it that access, unless I affirmatively grant it that access. [9:05] 我们用Agent身份这个概念， And so the way that we're using this notion of agent identities is really [9:11] 真正要落实的是你可能听说过的“最小权限原则”。 towards the principle of least privilege, which you probably heard of that. [9:13] 也就是给人，以及现在的Agent， You know, this idea that you want to give people and now agents. [9:17] 只授予完成工作所必需的最少权限，不多给。 the least number of privileges as possible for them to be able to do their job and no more than that, [9:22] 这样它们才不容易出格。 so that they can't go sideways. [9:23] 在Agent时代， That becomes especially important in this age of agents, for the kind [9:27] 这对企业和政府需要的治理尤其重要。 of government the governance that businesses and governments are looking for. [9:32] 很有意思。我记得听过一期Acquired， That's interesting because I remember hearing Acquired episode and actually [9:37] 讲微软怎样基于目录服务建立优势。 talks about how Microsoft builds its advantage on top of the directory. [9:41] 有了目录服务， Like is the directory that once like once you have the directory, [9:46] 整套Office软件就能继承背后 all the Office suite software just inherits that very complex [9:53] 复杂的身份、权限和文件访问体系。 and behind the scene, you know, rights and permissions and files. [9:58] 这确实会强化微软的优势。 so That is actually playing in the advantage [10:01] 但员工可能就要担心了。 of Microsoft, but then the employees would be worried. [10:06] 现在来了一个有身份、能做我工作的Agent。 Now I'm seeing an agent with identity doing what I can do. [10:11] 这对职场意味着什么？ So like what does it mean for the workforce? [10:14] 比如你创建一个Autopilot实例， Well, when you declare an autopilot, for example, when you [10:16] 它会向你汇报。 create an instance of one of these things, it will report to you. [10:19] 它真的会向你汇报。 So it actually reports to you. [10:20] 设计思路是，你可以把一部分工作委派给它。 And the idea is that you can delegate some of your work to these autopilots. [10:24] 客户会构建一些专用Autopilot， And there will be autopilots that our customers will build for specific purposes, [10:29] 比如处理特定的客户支持或合作伙伴支持。 certain types of customer support or partner support, those kinds of things. [10:33] 也会有更通用的Autopilot， And then there will be more general purpose autopilots that we will use in the way like personal [10:37] 像个人助理一样，帮你管理日历等等。 assistance and stuff like that, that you use to manage your calendar and those kinds of things. [10:42] 其实，这次录制的日历安排，我很确定就是用它建的， Indeed. I think the at the time that we are recording this, I am pretty sure that I [10:47] 或者至少是用它接受了你发来的邀请。 used mine to make this calendar entry, or anyway, to accept the one that you sent to me. [10:53] 我觉得值得认真考虑一下：AI大规模替代劳动力， I think it's important to at least, I guess, steelman the case of AI seriously replacing a lot [11:00] 最有说服力的理由是什么？我想到两个。 of the workforce just because like I guess I can steelman with two cases. [11:10] 第一，知识工作中确实存在大量低价值、看起来在忙的工作。 The first case is there are just a lot of fake work in our like built into knowledge worker. [11:17] 比如在不同软件之间复制粘贴信息。 Like we copy paste information between different software. [11:23] 现在，一个Agent就能完成很多事情。 And now we have this single agent that can do anything. [11:26] 我不用为了办一件事，先去Notion， There I don't have to go to Notion And then go [11:29] 再去Google Sheets、Outlook， to Google Sheets and then go to Outlook and then go [11:33] 再去另一个SaaS产品。 to another SaaS product in order to get something done. [11:36] 这类工作会高效得多，耗时少得多。 So that work is just going to be much more efficient and requires much less time. [11:42] 这是第一种情况。 So that is scenario number one. [11:44] 第二，如果从意图就能直接走到结果， scenario number two is if I can go from my intent to the outcome, [11:50] 也许就不用开那么多会、反复争论了。 I don't need to have so many meetings and just people debating. [11:54] 很多流程之所以存在，是因为执行的成本很高。 A lot of process are in place because doing th things are expensive. [11:59] 我们要投入工程师的工时，把东西做出来，或者卖出去。 we have to like spend engineering hours to make it happen or, you know, sell it. [12:04] 但如果做产品变得很便宜， But now if I can produce products cheaply, then maybe I [12:08] 可能就不需要那么多配套环节和中层管理了。 don't have need other elements and other middle management. [12:12] 所以， So I feel like it is like reducing a lot of at least [12:18] 它会减少很多冗余、不必要的工作， redundant unnecessary work that used to actually I [12:23] 而这些工作过去其实支撑了不少岗位。 feel like responsible for lot of the imp employment. [12:28] 另一方面，AI也在变聪明。 and other hand, yeah, getting smarter, right? [12:30] 去年，它还做不了企业里大部分人能做的事； And I feel like last year it cannot do most of things that most people can do in like [12:37] 今年，它真的能完成很多工作了。 enterprise setting, but this year it can seriously like do get a lot of things done. [12:43] 这就是我想提出的挑战。 So with that, yeah, that's my challenge or [12:47] 工作肯定会变化。 Jobs are definitely gonna change. [12:49] 而且已经在变了。 They are already changing. [12:51] 就像我说的，我自己的工作和四年前刚加入微软时相比，已经变了很多。 as I said, my own job has changed quite dramatically even from four years ago when I first joined Microsoft. [12:56] 但我们也必须意识到，这些技术存在各种风险和外部影响。 So, I mean, we do need to be cognizant of the various risks and the externalities of these things. [13:01] 我觉得大家对此越来越重视了。 And I think we are increasingly cognizant of these things. [13:04] 风险包括工作将会怎样改变。 And So those risks include how jobs are going to be changed. [13:08] 也包括数据中心带来的外部影响。 they also include the externalities of things like data centers. [13:11] 现在很多人不愿意当地建数据中心。 You know, a lot of people are right now averse to having data centers in their locales [13:16] 我认为，无论是建设、使用这些技术， and I think it's key for us to gain permission from a community or from an employee [13:22] 还是代表别人部署它们，都需要获得社区或员工的认可。 to build these things or to use these things or present these things on their behalf. [13:27] 坦白说，任何AI要取得成功，认可和授权都是关键。 that I think is really key to the success of any AI is frankly permission. [13:33] 嗯。 Mm, mm. [13:36] 你提到要获得员工的认可。 You mentioned permission from employees. [13:38] 能展开讲讲吗？ Can you elaborate [13:42] 有些人会犯一个错误，现在少一些了， I mean I think one of the mistakes that some folks make, and we don't do this [13:46] 就是把AI硬推到你面前。 as much now, but I mean i it kind of gets pushed in your face, you know. [13:49] 你本来只想做件事，比如剪个视频， It's like, hey, try this AI thing, you know, it's like you're just trying to do something, [13:53] 结果它一直说：“来，试试这个AI功能！” trying to edit a video or try to do whatever you're doing, and it's like, hey, use this AI. [13:58] 有时候确实有用。 And sometimes it is useful. [14:00] 有时候没那么有用。 And sometimes it's not so useful. [14:02] 有趣的是，我自己做很多视频， I mean, the funny thing is, the video editing tool that I use, I make [14:04] 也有YouTube频道。 a lot of videos and now I got a YouTube channel and stuff like that. [14:07] 但我长期都在抵触剪辑软件内置的AI功能。 And I for a long time I resisted the AI video editing tool that was built into my the video editor that I use. [14:14] 直到有一天，我想：今天试试吧。 until one day I was like, you know what? Let me try it today. [14:17] 看看它到底好不好用。 I'm gonna try this thing, see if it's any good. [14:19] 结果，天哪，好用太多了。 And my god, it was so much better. [14:21] 我不喜欢它被硬塞到面前，但最后，我自己愿意给它一个机会。 Like, in a way, like I resisted having it pushed in my face, but then eventually I gave it that permission. [14:26] 用了之后发现， When I did, I found that it cut my edit time down to, you [14:30] 过去剪一条视频需要一小时， know, what would have taken me an hour to edit a video before. [14:33] 现在15分钟左右就能搞定。 Now I can do it in like 15 minutes. [14:34] 因为我可以直接编辑转录文字。 Because basically I can edit the text of my transcript. [14:37] 如果哪个词说错了， And if I misspeak a word or something. [14:39] 把那个词删掉，对应的视频也就删掉了。 I just delete the word out it deletes that part of the video and then we're done. [14:42] 然后继续往下做就行。 Like it moves on. [14:44] 我不是在用生成式视频。 Anyway, so yeah, I'm not using the I'm not using like generative video. [14:48] 视频还是我自己做，配音也是我自己来。 Like I make all of my videos, I voice over all my videos. [14:50] AI声音我用得并不多，虽然现在已经做得很好了。 I'm not really using AI voices very much, although they there's they're getting very good now. [14:54] 但在剪辑上，我现在认可它了。 but for editing, I mean it has my permission now. [14:58] 所以，认可和授权不只是员工的问题，而是所有人的问题。 So I mean I think permission and not just employees but everybody, right? [15:01] 使用消费级AI也一样。 I mean when you think about using consumer AI also. [15:04] 这里面有好几个层次的授权。 I mean, there's a number of layers of permission to that. [15:07] Agent能不能访问我的邮件、消息、日历？ Does the agent, you know, does it have my permission to access my email, [15:12] 我允许了吗？ to access my messages or my calendar or those kinds of things? [15:15] 这些权限必须由我明确授予。 I have to explicitly give it that permission. [15:18] 否则就会觉得被冒犯了。 Otherwise it feels like a violation. [15:23] 这一部分，我想到一个总结性的问题。 I guess now I can think about think of one question for this part. [15:28] 后面我们会讲机会，但先问： We're going to talk about the opportunity part, but for this part, I guess my summary [15:32] 有没有一条让大家都开心的AI转型之路？ will be summarized question will be can there be a happy AI transformation path? [15:39] 一条更愉快的AI转型路径。 Yeah, happy AI transformation path. [15:40] 比如企业适应AI，员工也学会使用AI。 Like we talk about, you know. Enterprise adapting to AI, employees adopting it to AI. [15:46] 但正如你说的，很多东西是被直接推到眼前的。 And like you said, like a lot of the things are just in our face. [15:50] 参加培训的时候，也不得不接受。 We have to accept it when we do training. [15:52] 有些人会兴奋，觉得终于能用新技术了，但很多人还是担心。 Like I feel like some people are excited about, I can have this new technology, but most people are worried. [15:58] 出于好的初衷， And from the good intention, I feel like, yeah, a lot [16:02] 很多企业都想让这场AI转型皆大欢喜。 of enterprise try to do this happy AI transformation. [16:05] 大家保住工作，也一起提升技能。 Like everybody keeps their job, we all level up level our skills. [16:09] 用AI做更有创造性的工作，不用再做繁琐的事情。 We use AI, we do the creative work, we don't have to do the tedious work. [16:14] 这是大家的承诺。 That's the promise. [16:16] 但你觉得做得到吗？ But do we think it's possible? [16:18] 最理想的路径当然是， Well, certainly the happiest AI path is when the AI is available to everybody, everybody [16:22] 组织里的每个人、一个国家或地区的每个人，都能用上AI。 in an organization, everybody in a country or in a region, and those kinds of things. [16:26] 今天AI的一个优势，就是它越来越容易使用了。 And one of the advantages of AI as we know it today is that it's becoming more and more accessible. [16:32] 回想不久前， that is to say, whereas where we started, and not even that [16:36] 很多编程Agent还是命令行界面。 long ago, I mean you think about coding agents and the CLIs. [16:38] 我们也有GitHub Copilot CLI。 I mean, we have our own GitHub Copilot CLI. [16:41] 但会用命令行的人毕竟有限。 But there's only so many people that know how to use a command line interface. [16:44] 可能只有1%的人，甚至更少， You know, it's probably 1% of the population or less could really [16:47] 真正会用，知道怎么打开终端启动它。 use that thing, even knows how to open a terminal to start it up. [16:50] 使用CLI时， you know, when you use it, it looks like one [16:53] 那个界面就像电影里的黑客场景。 of those hacker scenes in some movie or whatever you're using a CLI. [16:57] 但AI正在离开这种形式，变得更自然。 But AI is moving away from that and into something that's much more natural. [17:01] 你开始看到越来越多语音界面。 Now you start to see more and more voice interfaces. [17:03] 这就是一种自然得多的交互。 So but that's a much more natural interface, right? [17:06] 不用命令行。 I'm not using a command line interface. [17:07] 不用很技术性的操作。 I'm not using something very technical. [17:09] 我能和它说话，它也能和我说话。 It's something that I can talk to, that it talks to me. [17:12] 这样的交互越来越普遍。 And that becomes more and more common now. [17:14] 用一种非常自然的方式沟通。 that kind of really natural interaction in a natural way of being. [17:18] 但这些东西仍在演进。 But these things are still evolving. [17:21] 连我们用来和AI交互的设备，都还在变。 I mean, even the devices that we use to interact with AI are still evolving. [17:26] 未来真的需要这么多屏幕吗？ Do we need this many screens in the New World? [17:28] 我不知道。 I don't know. [17:28] 我们现在到处都是屏幕。 You know, we got screens all over the place. [17:30] 真的需要吗？ Do we need it? [17:31] 再看看吧。 I mean we'll see. [17:32] 前面问了不少推测性的问题， yeah, sorry for you know all these speculative questions because [17:35] 聊AI时确实很难不去想象未来。 you know when we talk about AI, we couldn't help but speculate. [17:39] 不过现在，我们可以转向你在一线的实际观察。 But now I think we can shift gears towards like what you actually observed on the field. [17:45] 已经在采用AI的企业，哪些策略好、哪些不好？ especially like this enterprise already adopting AI, what is a good strategy, what is a bad [17:50] 又产生了什么新的商业机会？ strategy, and what are the opportunities and new kind of business opportunities created? [17:57] 我认为，最好的策略是构建真正赋能人的AI。 Yeah, I mean, I think the best strategy is creating AI that enables people. [18:02] 我很喜欢的一个客户案例，是宝马。 I think one of one of my favorite customer stories is that of BMW. [18:05] 宝马大量使用我们的技术。 BMW uses our technology heavily. [18:07] 今年早些时候，我去慕尼黑拜访过他们。 And in particular, I went and visited them in Munich earlier this year, actually. [18:11] 他们带我参观了工厂，非常有意思。 And it was cool they gave me a tour of the factory and all this stuff. [18:15] 宝马有个叫MDR的团队， but there is a group at BMW called [18:16] 全称Mobile Data Recorder。 BMW MDR, Mobile Data Recorder. [18:19] 他们在全球部署了装有专门测量设备的宝马汽车。 And they have these specially instrumented BMWs all over the world. [18:24] 加入MDR项目，就能免费使用一辆宝马， So you basically you get if you're part of the MDR program, you get a free BMW, but your trunk [18:29] 但后备箱装满了仪器，它会记录几乎所有你能想到的数据。 is full of instrumentation because it's logging everything, everything you can think of. [18:33] 刹车、发动机、环境湿度， They got sensors on the brakes and the engine and like [18:36] 各种地方都有传感器。 for the ambient humidity and everything you can possibly think of. [18:39] 每辆车真的有5000个传感器。 There are literally 5,000 sensors on these cars. [18:42] 问题是，直到大概一年前， And the challenge was that until recently, and I mean like a year ago, these things, [18:48] 他们还得靠少数“高手”来查询数据。 they had these like wizards, basically these people that could query. [18:52] 数据量非常大， All of this data because they get so much data by the way that the way they [18:56] 大到不能直接通过网络传回来。 retrieve it is you can't just send it over the wire because there's too much. [19:00] 得把硬盘寄回去，再插进一台大型设备。 You literally send a drive to them and they put it into this big machine. [19:05] 如果你看过《捉鬼敢死队》， It looks like if you know Ghostbusters, it's like when they put the trap [19:08] 就像把捕鬼器放进收容装置里，再把里面的东西放出来。 in the thing and the containment unit and they get the ghost out. [19:11] 就是那个样子。 It looks like that. [19:12] 我可不是开玩笑。 I'm not even kidding. [19:12] 我看到时，就这么跟他们说。 That's what I told them when I saw it. [19:14] “这不就是那个收容装置吗？” I was like, it's it's the containment unit. [19:16] 车开一圈，就会收集到一TB甚至更多的数据。 It's like a terabyte or more of data that it gathers from driving around. [19:21] 遥测数据非常多。 So it's quite a bit of telemetry. [19:22] 这个项目做了很多年， And for many years, because they've had this program for many years, but for many [19:26] 但长期只有少数几位高手能查询这些信息。 years they've just had a few of these kind of wizards that could query this information. [19:30] 假设我在制动部门， So if I was working in the braking division and I wanted to know how the brakes on my new [19:35] 想知道新车型的刹车表现，就得去找其中一位。 car design were performing, I would have to go talk to one of these kind of wizards and [19:40] 排上队，等查询结果出来， eventually I'd get in the queue and eventually I'd get my query back [19:43] 才知道刹车性能怎么样。 and I'd get some information about how my brakes are performing. [19:45] 后来有一天，他们决定把这套系统现代化。 And they decided one day, they're like, we're gonna modernize this. [19:50] 毕竟现在已经有Agent，也有能力很强的模型了。 Like we have agents today, we have these models that are pretty capable. [19:53] 但这恰恰说明，光有模型还不够。 But again, I mean this is one of those places where the model by itself is insufficient [19:57] 5000个传感器，每一个的数值都不一样， because again, you got 5,000 sensors, and each one of these sensors has a different value, [20:03] 阈值不同，含义也不同。 different thresholds, a different meaning. [20:05] 比如刹车传感器，和发动机传感器完全是两回事。 I mean, you the brake sensors are totally different than the, you know, engine sensors, right? [20:10] 至少我们希望，刹车温度和发动机温度的变化规律是不一样的。 And brake temperatures will vary very differently than engine temperatures, one would hope. [20:15] 所以他们花了大约半年，跑遍宝马各个部门，整理出一份数据字典。 so they spent like six months going all around BMW and gathering a data dictionary. [20:22] 这些工作现在依然少不了。 And this is the work that's still required. [20:24] 想让AI处理结构化或非结构化数据， I mean, if you want your AI to work with your structured data [20:28] 看起来像魔法，但其实不是。 or your unstructured data or whatever, it's it seems magical, but it's not. [20:32] 你还是得告诉它，这些数据到底是什么。 You still have to tell it what this stuff is. [20:34] 他们花了很多时间和宝马其他团队一起整理： So they spent all this time with other groups at BMW compiling this information [20:39] 传感器在收集什么，该怎样与这些数据交互。 about what these sensors were gathering and also how to interact with them. [20:45] 比如查询刹车、后备箱或其他传感器时， so like how people are going to interact with them, like what words, what terminology, what acronyms might you [20:51] 人们会用什么词、什么术语、什么缩写。 be using when you're interacting with the brake sensors versus the trunk sensor versus the whatever sensor, you know? [20:57] 投入这些时间后， so they spent this time doing this stuff, but the result is they [21:01] 他们做出了一个真正好用的Agent，我也亲自试过。 have an extremely functional agent that I was able to try myself. [21:05] 我就是一个刚走进门的外人。 Like now, this is me walking in the door. [21:07] 完全不了解这些传感器。 I have no knowledge of any of these sensors. [21:10] 但我记得，我当时问的是： But I was able to actually the query I made, I remember it. [21:13] “给我看过去一周，潮湿天气下所有急刹车事件。” I said, show me all the hard braking events in wet weather in the last week. [21:19] 它就做到了。 And it did it. [21:20] 它执行了一系列查询， It like made a series of queries and it was like, okay, here [21:23] 然后把过去一周潮湿天气下的急刹车事件列了出来。 are all the hard braking events in the last week in wet weather. [21:26] 如果我想继续深挖， And so if I wanted to dig deeper on that and be like, okay, what was the temperature of the brakes when that happened, [21:30] 比如当时刹车温度多少、制动距离多长，都可以继续问。 or what was the you know, stopping distance, or these kinds of things, I could have gone deeper with that. [21:36] 即使我不是这个领域的专家，也能做得到。 And even as a lay person, I was able to get that done. [21:39] 他们花了很多时间构建这套系统，现在宝马所有工程师都可以用。 So, you know, they spent a lot of time to make this system, which now all of the BMW engineers can use. [21:46] 不用再排队等别人查数据， So rather than waiting in line for somebody to make them a query, anybody can go into [21:51] 任何人都可以像我一样，自然地与系统交互。 the system that I'm referring to and they can use it just like I did and in a natural way. [21:57] 所以，这不是魔法。 So again, it wasn't magical. [21:58] 背后是真的要下功夫。 They had to put the work in. [21:59] 他们花了半年时间。 These guys, they put the work six months. [22:01] 开发系统，整理数据，把需要的上下文补齐。 They spent developing this, putting together the data, the context that it needed, and that kind of stuff. [22:06] 最后做出的Agent，确实改善了宝马工程师的工作， But in the end, They have built an agent that materially improves the working lives [22:12] 也提升了产品本身的质量。 of these engineers at BMW and frankly improves the quality of the product itself. [22:16] 对，比如减少事故，造出更好的车。 Yeah, yeah, like reducing incidence and building better cars. [22:21] 我非常尊敬这样的人。 I have such high respects for these people. [22:24] 叫他们“高手”确实没错。 I think it's right to call them wizards. [22:26] 我以前在腾讯时， I did like when I was at Tencent, I did [22:29] 也做过广告系统的数据清洗。 kind of cleaning the data for ad system. [22:34] 很难，做了三个月。后端、前端、机器学习系统， It was so hard, three months and just like back end, front end machine [22:39] 还有各种写死的逻辑，清理起来非常费劲。 learning systems and people hard coding things so much so hard to do to clean. [22:44] 但我能想象，他们的工作还要难上一百倍。 But I can imagine their work is like a hundred times harder. [22:47] 对，这就是常常被忽略的部分： Yeah, and I think that's really the hidden part of, you know, really [22:50] 一个真正好用的AI，背后其实有大量工作。 functional AI behind this really functional AI is actually quite a bit of work. [22:55] 现在编程Agent的模型很强， And for as much as we have these coding agents now that have these really, really good [22:59] 当然能写各种查询， coding models and those kinds of things, and yes, of course they can write SQL queries [23:03] 比如SQL、KQL、GQL。 and KQL queries and GQL queries and all kinds of queries, yes, they could do that stuff. [23:08] 但这不等于你不需要做准备工作。归根结底，还是上下文。 And that still doesn't mean that you don't have to put in the work to because it really comes down to the context. [23:14] 再拿宝马MDR来说， I mean, again, like when I think about BMW MDR. [23:17] 他们给Agent补充的上下文， It really is about the context that they gave to this agent that tells it [23:21] 告诉了它这些数据是什么、代表什么、应该怎样理解。 what all this data is, what it means, what it's about, and all that stuff. [23:25] 这件事没有什么神奇的全自动办法。 And there is no like super ultra magical automated way to do that. [23:30] 他们必须向同事收集信息。 I mean, they had to gather this information from their colleagues. [23:34] 向制造传感器、使用这些系统的同事了解。 Their colleagues who were making these sensors, who were using these systems. [23:37] 不会凭空发生。 You know, it just wasn't going to be magical. [23:39] 世上没有哪个编程Agent，光看原始数据就能说： And no coding agent in the world was going to be able to look at the raw data and be like, [23:44] “对，这肯定是刹车数据。” yeah, that's definitely brakes. [23:46] “这肯定是一次急刹车。” That's a hard braking event for sure. [23:49] 不可能，做不到。 Like, no way, no way. [23:51] 对，一旦涉及隐性知识，就会出现一道鸿沟。 I think, yeah, it is a barrier or chasm, like when there is tacit information involved. [23:57] Agent不会自己知道这些。 Agents just cannot know that. [23:58] 你得把人脑中的知识提取出来，Agent才真正有用。 You have to extract that from human for agents to be any useful. [24:03] 这是输入端的原材料。 But that's part of the kind of the raw material. [24:06] 另一端则是评估，对吧？ And then the other end is evaluations, right? [24:08] 怎么知道Agent有没有把工作做好？ How can you know if the agent is doing the job properly? [24:13] 你向它查询以后， You queried the agent and I'm I'm sure it must have [24:16] 应该也有办法判断信息是不是真的准确。 some way to know this information is actually accurate. [24:20] 是不是给了你真正想要的结果。 And gives you what you ask for. [24:23] 对，最近这方面发生了一些变化。 Yeah, well, I mean, we could talk about how that has recently changed. [24:26] 大概过去三个月，评估Agent的方式其实变了很多。 The way that we evaluate agents has changed actually quite dramatically over the last three months or so, actually. [24:33] 只是最近才开始逐渐进入市场。 It's all it's only been relatively recently that this has started to penetrate the market. [24:37] 我们在Foundry里也支持了。 We certainly have support for it in Foundry. [24:39] 这叫基于评分标准的评估。 It's called rubric-based evaluations. [24:40] 这是整个行业的趋势。 and this is an industry trend. [24:42] 不只是我们在做。 We're not the only ones that are doing this. [24:43] 大家都在走向基于评分标准、基于结果的评估。 you know, all of us are kind of moving towards this notion of rubric based or outcome based evaluation. [24:50] 理解今天的方法之前，先回顾一下过去怎么做。 Yeah, what it means is so you know. Let's think about how it used to be before we even think about how it is today. [24:56] 过去三年半左右，我们评估Agent，主要依靠一系列指标。 So, how we have been evaluating agents for the last three and a half years or so, we've had these metrics. [25:04] 要注意，评估Agent不像单元测试，不是简单的通过或失败。 Now bear in mind that when you would evaluate an agent, it's not like a unit test, it's not a pass-fail. [25:08] 它是概率性的、统计意义上的衡量。 It's gonna be a probabilistic measure, a kind of a statistical measure, because they are [25:12] 因为Agent不是确定性系统，每次都会有些不同。 non-deterministic and they, you know, they every time it'll be a little bit different. [25:17] 我们有很多指标，最早的是“有据性”，groundedness。 So we have these metrics, and the original metric was groundedness. [25:21] 回想2023年， You know, you think about back in the day, the day being like [25:24] 距离我们今天录制，也就三年。 2023, which is only three years ago as of this recording. [25:29] 那时我们主要做问答Agent， but we were building these question-answering agents, and one of the primary concerns [25:31] 最担心的是幻觉，也就是模型凭空编造。 at that time was hallucination, this idea that these models would just make up whatever. [25:36] 不论当时还是现在，这些评估大多都是这样： So all of these evaluations, and this is true, this was true then, this is still true now. [25:41] 用另一个大语言模型来当裁判。 In most cases, we use a judge large language model. [25:44] 本质上就是再用一个模型进行审查。 So an adversarial large language model is basically another model. [25:48] 比如Foundry里，我们有专门适配过、用于评判的模型。 In our case, at least in Foundry, we have. A model that's specially adapted to be a judge. [25:52] 它就是专门当裁判的。 So it's a special kind of model that is specifically a judge. [25:56] 我们让它按这些指标来评判。 And so we run it on these metrics. [25:59] 比如有据性这个指标。 But if you think about the old style of metrics, so groundedness is a metric, and what you [26:03] 你把Agent的回答给裁判看。 give to the judge in this case is, well, here's my agent, it's making some responses and stuff. [26:08] 裁判能看到Agent说了什么。 So the judge in this case can see what the agent is responding. [26:11] 也能看到它检索到的数据：结构化数据、非结构化数据、文档等等。 It can also see whatever data it retrieved: structured data, unstructured data, documents, whatever. [26:16] 这些它都能看到。 So it can see all that stuff. [26:18] 裁判要回答的是：Agent说的每件事， And the question that the judge is trying to answer is: Is everything [26:21] 有没有检索数据作为依据？ that the agent said grounded in the data that it retrieved? [26:26] 有依据，就是grounded。 If it is, it's grounded. [26:28] 没依据，就是幻觉。 If it's not, it's a hallucination. [26:31] 所以我们有有据性、相关性、连贯性、流畅性等指标， So we have those types of metrics: groundedness, relevance, coherence, [26:34] 评估Agent怎么回答。 fluency, these kinds of things that are about how the agent responds. [26:37] 在问答时代，这很好用。 And that was great for the question-answering era. [26:39] 但现在，Agent不只回答问题，还会真正执行动作。 But now we have these agents that are actually doing things, they're not just answering questions anymore. [26:43] 于是就有了工具调用指标。工具就是Agent接入外部系统的方式。 So then we have these. Tool call metrics where a tool is an agent integration. [26:48] 它要连接某个系统。 It's integrating to something, right? [26:49] 比如微软的系统、SAP的系统，或其他系统。 A Microsoft system or an SAP system or whatever system. [26:53] 所以我们会评估工具选择。 And so now we have these metrics about tool selection. [26:56] 它选对工具了吗？ Is it selecting the right tools? [26:57] 输入准确吗？有没有正确使用？ Input accuracy? Is it using them right? [27:00] 还有输出利用率。 Is it output utilization? [27:01] 工具调用以后，返回的结果，它真的利用了吗？ It's calling these tools. Is it actually using what it gets? [27:04] 这些就是裁判要判断的东西。 And so that's what the judge is judging in those cases. [27:06] 还有一些指标是围绕任务本身的。 And then we have some metrics that are around the task. [27:09] 比如意图理解：它是否正确理解了你交给它的任务？ So there's intent resolution. Did it correctly understand the task you were giving it to do? [27:13] 任务完成度：它完成任务了吗？ task completion. Did it complete the task you were giving it to do? [27:17] 任务遵循度：它是否只做了这件事， Task adherence. Did it do only that thing [27:19] 没有在执行过程中顺手做其他无关的事？ and not some other random stuff in its quest? [27:22] 这些指标，我们已经用了好几年。 So those are the kinds of metrics that we have been using for some years now. [27:26] 但整套方法有一个根本缺陷： But there is a fundamental flaw with that whole method, which is [27:29] 刚才那些指标都是通用的。 that all of those metrics that I just described to you are generic. [27:34] 没有哪个指标知道，你的Agent具体应该做什么。 Like none of those metrics know what it is that your agent is supposed to be doing. [27:38] 比如我最近聊的一位客户， So, you know, let's say, for example, like let's say I was talking to one [27:42] 想做一个公寓定价Agent。 of our customers recently and they were looking to build An apartment pricing agent. [27:46] 他们有一个新的公寓项目， So let's say you have a new apartment complex and they have a bunch [27:49] 也有周边公寓等相关信息。 of information about, you know, area, apartments, and things like that. [27:53] 想把新项目的全部房源 So they want to make an agent where you have this new complex, [27:55] 整理成表格，交给Agent。 you give it a spreadsheet of all of the apartments in your new complex. [27:59] 里面有一居、两居、三居， You know, here's the one bedroom, two bedrooms, three bedrooms, and what their sizes [28:02] 面积、卫生间数量，以及配套设施等。 are, and how many bathrooms they have, and all those amenities and things like that. [28:05] 交进去一张表，Agent再返回一张填好价格的表。 So you give it a spreadsheet of that, and it gives you a spreadsheet back that has all the prices in it. [28:10] 问题来了。假设我们用“任务完成度”作为指标。 So now here's the problem: let's let's take task completion. As a metric. [28:14] 在评估里采用这个指标。 So let's say I'm using task completion in my eval, right? [28:17] 我给Agent一张公寓表，它回给我一张表。 So I give the agent a spreadsheet of my new apartment complex that gives me a spreadsheet back. [28:22] 任务完成度就会说：“没问题。” And task completion will be like, You're good. [28:26] 因为你要了一张表，它给了你一张表。 Because you asked for a spreadsheet, you got a spreadsheet, right? [28:30] 但我打开表格，仔细一看， But then I open the spreadsheet, I look at the spreadsheet, [28:32] 一居、两居都还挺准， and I'm like, well, the one and two bedrooms, those are pretty good, [28:34] 可三居室，偏得离谱。 but those three bedrooms, man, they're way off. [28:36] 明显有地方不对。 Like something here is not right. [28:38] “任务完成度”不会知道，因为它是通用指标。 But task completion as a metric would not know this because it's generic. [28:41] 它不懂公寓该怎么定价。 It doesn't know how to price apartments. [28:45] 根本不知道这方面的要求。 Like it doesn't know anything about that, right? [28:47] 而现在，我们有了基于评分标准的评估。 Now, fast forward to today, now we have these rubric-based evaluations. [28:53] 你可以自己定义指标，而且是用自然语言定义。 And so in a rubric, you define the metric yourself, and you define these metrics in natural language. [29:00] 这些指标会细得多，也更贴近这个Agent具体要做的工作。 And those metrics might be things like it. It would be much more granular and specific to what this agent is doing. [29:06] 比如公寓定价Agent的评分标准可以明确写： So the rubric for my apartment pricing agent might specifically state: you will take into account the number [29:12] 每次定价都要考虑卫生间数量、卧室数量和面积。 of bathrooms, the number of bedrooms, and the square footage of the apartment for every price that you make. [29:17] 每次定价，还要参考类似房源的可比价格。 And you will consider the comparisons of similar apartments for every price that you make and so on. [29:23] 你可以设定一系列这样的维度。 And you would have a whole bunch of these dimensions. [29:25] 每个维度，本质上就是Agent应当遵循的一条规则。 And each one of these dimensions is effectively like a rule that your agent is supposed to be following. [29:30] 用这样的评分标准来评估， And so When you do an evaluation with a rubric, you get something that's much, [29:34] 就能更细致、更具体地判断Agent有没有做好它的工作。 much more granular and much more specific to what your agent is supposed to be doing. [29:38] 你可以为每个Agent建立自己的评分标准。 and so you can make a rubric for every agent that you build. [29:41] 现在我会建议客户，在设计阶段就把这件事考虑进去。 In fact, now I have been recommending to customers that this become part of their design time concerns. [29:48] 也就是还在构思一个新Agent的时候， That is, when you are on the drawing board, you're looking to make a new agent [29:52] 就先把评分标准一起写出来。 to do whatever it might be, that right up front, you produce. Also the rubric. [29:57] 评分标准是什么？ What is the rubric? [29:58] 里面有不同维度，也就是一条条具体规则。 So we have this rubric, and in that rubric, there are these dimensions, which are the specific individual rules. [30:03] 至少在Foundry中， And each one of these dimensions, at least in the foundry system, [30:06] 每个维度还有权重，代表它的重要程度。 each one of these dimensions has a weight, that is to say, an importance. [30:09] 有些规则比其他规则更重要。 So some rules will be more important than others. [30:11] 比如银行，合规规则有时比正确性规则还重要。 You know, if you're a bank, for example, compliance rules may be more important sometimes than correctness rules. [30:16] 如果这是银行的Agent，我不能问它： Like, you know, like it can't, like, for example, if I if it's a banking agent, I should [30:22] “Yuzheng的账户里有多少钱？” not be able to ask the banking agent, hey, how much money is in Yuzheng's account? [30:26] 那绝对不行。 Like that's a that's an absolute no-no. [30:28] 所以“不能泄露其他客户信息”这条规则，就是必须满足的。 And that rubric that says you will not reveal other customers' information, you know, that's a must-have. [30:35] 相比之下，如果我说ATM收了4美元手续费， Whereas like, let's say that I'm like, hey, you know, this ATM charged for me a $4 fee [30:40] 它本来不该退，却错误地退了给我。 and it mistakenly grants me the $4 fee when it's not supposed to, or like it refunds it. [30:44] 那不是同等级别的合规问题。 I mean, you know, it's not a compliance fail. [30:47] 银行不会因此就被监管追责。 It's not they're not gonna get in trouble with the government for that. [30:50] 相对来说，重要程度就没那么高。 It's so it that relatively speaking has less importance. [30:52] 当然，你也不希望Agent退还根本不存在的费用。 You still don't want your agent Know refunding fees that don't exist or whatever. [30:57] 但相比之下，后果没有那么严重。 But on the scale of things, like you know, it's it's not as bad. [31:01] 所以评分标准应该提前做， So when you make your rubrics, you want to you want to do this [31:04] 因为它实际上就是成功标准。 right up front because these effectively become your success criteria. [31:08] 通过了这套标准，我们才认为结果达标。 Like it's like if this rubric is passing, then we're good. [31:12] 此外，也要提前定义好可接受的阈值。 and furthermore, you also have to define up front what is your threshold. [31:17] 别忘了，Agent不是确定性系统，会犯错。 Because bear in mind, these things are non-deterministic, they will make mistakes. [31:21] 你可以采取措施减少错误， Now there are Things that you can, you know, steps that you can take [31:23] 但不可能每次都考100分。 to mitigate their mistakes, but you're not gonna get 100 on the test. [31:27] 它还是会出错。 You know, they're they're still gonna make mistakes. [31:29] 偶尔总会有错误。 Every now and again, it will make a mistake. [31:31] 因此你要先决定：允许多少错误、允许哪一类错误， And so you have to decide what is your threshold, like what level of error and what [31:37] 才能把这个系统投入生产。 types of errors are you willing to accept to put this type of thing into production. [31:43] 这是一个权衡。想用更智能的系统， It's a trade-off between like yeah, like if you want to use something smart, [31:47] 就要容忍一些错误，但不同错误的容忍度不一样。 you need to tolerate some error, but in certain error more than other areas. [31:54] 我想追问，如果规则只有通过和失败两种状态， One follow-up question is if a rule is binary, [31:58] 其实比较容易判断。 you pass or you fail, I think it's very easy to follow. [32:01] 但回到定价： But in the example of did I price [32:06] 价格到底对不对，偏差有多大？ this correctly, how far off I am? [32:10] 即使考虑了面积， Like even if I considered the square footage, I considered okay, [32:13] 也考虑了这套房多一个卫生间、应该贵一点， this unit has one extra bathroom, so I should price a little bit higher. [32:19] 最后的价格仍可能偏得很离谱。 But maybe it can still be way off in my pricing. [32:22] 这种情况，评分标准该怎么定？ So how can I how come up with the rubric for that? [32:26] 这就涉及评分标准的持续改进了。 Yeah, ultimately that gets you to the notion of rubric refinement. [32:30] 先做出初始标准，运行评估， And so, you know, what you produce this initial rubric, you do [32:33] 难免还是会出现一些错误。 your evaluations, and inevitably it still will make some types of mistakes. [32:38] 你需要让真实用户接触AI的输出。 And what you want to do is have users interact with the outputs of your AI. [32:43] 在公寓的例子里，可以请公寓经理来审核这张价格表。 And so let's say that I might have the apartment manager in my apartment case reviewing this spreadsheet. [32:50] 但不能只说：“这个价格不对。” And you can't just say, This price is wrong. [32:53] 你要说：这个价格不对，原因是什么。 You have to say, This price is wrong and here's why, because [32:56] 这个“为什么”，以后要写进评分标准。 that why, that's what goes into the rubric down the line. [32:59] 可能是修改原有维度，也可能新增一项。 And so maybe it's an amendment of one of your existing rubric dimensions, or maybe it's an entirely [33:04] 比如没考虑到，这套房配了高档的电磁炉之类设施， new dimension that says, here's something that you hadn't taken into account, that this particular [33:09] 所以价格应该更高。 apartment has induction stoves or something that's super fancy and therefore it should be priced higher. [33:15] 相应的评分维度就可以写： And so in my rubric, my rubric dimension would be like, and if it [33:19] 如果有这类高档设施，定价应当有所提高。 has induction stoves, you know, make the price higher or something. [33:23] 你明白我的意思。 You see what I mean. [33:24] 所以，评分标准和Agent本身，都要持续改进。 So there is this ongoing process of improvement to your rubrics and to your agent itself. [33:29] 两件事是对应的。 And they are analogous, right? [33:31] 改进Agent的同时，也改进评分标准。 I mean, as you improve the agent, you improve the rubric. [33:33] 很多时候，评分标准本身就是Agent应该遵循的规则。 A lot of times, again, the rubric is the rules that the agent is supposed to follow. [33:36] 有时，你还会把评分标准直接作为上下文提供给Agent， And sometimes you will reveal the rubric itself to the agent as a form [33:39] 让它知道该遵守什么。 of context so it knows what rules it's supposed to be following. [33:43] 很好。 Great. [33:44] 我想帮观众总结一下，这里有一个很重要的思路转变。 And I want to summarize for my audience that this is an important message shift. [33:48] 如果我理解得不对，请纠正。过去， like correct me if I summarize it wrong, but the message shift is before [33:53] 很多人会告诉Agent，具体该怎么做。 a lot of people are trying to tell the agents how to do certain things. [33:56] 按这些步骤，走这个流程。 Like you follow these steps, you follow this process. [33:59] 现在则是告诉它规则、评分标准， Now you just tell the agents the rules, the rubrics, [34:04] 什么结果更好，什么结果不理想。 and what is the more desired outcome versus what's not. [34:09] 然后相信它足够聪明，能够自己找到实现目标的方法。 And you trust the agent to be intelligent enough to figure out how to get there. [34:14] 某种程度上，你还在相信它能更有创造力。 And you actually, I guess, implicitly trust the agent to be more creative. [34:20] 比你做得更好、更高效、更准确。 Like they can do things like better, more efficient and more correct than you are. [34:25] 所以你描述目标，而不是指定最佳流程， So you just tell the outcome, you don't specify the best process because you trust [34:30] 因为相信Agent能找到最好的方法，甚至比我们现有的更好。 the agent to find the best process or even the better process that was what we have today. [34:35] 是的，但要信任，也要验证。 Sure, but trust but verify. [34:37] 以M365 Copilot为例， That is to say that you know, in as much as you trust an agent, when you when you come [34:41] 我们对自己的产品做基于评分标准的评估， to trust an agent, you come to trust it in part because like if I think about M365 Copilot, [34:45] 已经一年多了。 you know, we have been doing rubric based evals on our own 365 Copilot for over a year now. [34:51] 一直在用这种方式评估。 We've we've been, you know, doing that style of evaluation. [34:54] 我对M365 Copilot的很多能力已经很信任了。 And I trust 365 Copilot, pretty implicitly for a lot of stuff. [34:58] 每天都会拿它做些事情。 I use it every day for something. [35:01] 我信任它，是因为背后有一整支评估团队， And the reason why I trust it is because we have this whole eval [35:04] 在测试各种场景。 team back here that's evaluating all of these different scenarios. [35:07] 同时，数百万用户每天都在用它， And meanwhile, all of us, the many millions of us that are using this thing every day, are stretching it [35:11] 从各种角度探索边界，让它做各种任务。 in 15 different ways all the time, you know, like all kinds of things that we're trying to make it do. [35:17] 顺便说，Copilot里也有点赞、点踩按钮。 And in Copilot, by the way, there is a thumbs up, thumbs down button too. [35:20] 如果某件事没做好，或者不让我满意，我会点踩， So if something doesn't work or doesn't work to my satisfaction, I'll give it the thumbs down and I'll give [35:24] 也会写明原因。即使你不在微软工作，我也希望你这样做。 a reason too, because I know you know, and I would hope that you would do it too, even though you don't work here. [35:29] 因为我知道， because I know that down the line there is this eval team and eventually what's gonna [35:35] 背后的评估团队最终会把这条负面反馈及其原因放进评估集， happen is my thumbs down and the reason that I give in there is gonna go into their [35:40] 让同类任务以后做得更好。 eval set so that in the future this thing that I am trying to do will work much better. [35:46] 对，这可能是管理者比较自然的直觉， Yeah, I think this is intuition for managers, but not like for [35:52] 但开发者或独立贡献者不一定马上意识到。 developers, individual contributors may not be immediately obvious. [35:56] “信任但验证”，重点是交叉验证，而不是重新做一遍。 That is, when you say trust but verify, you verify by cross check, not double check. [36:03] 还有很多开发者觉得， Like there are still a lot of developers as that I [36:05] 必须检查Agent写的每一行代码，可能并不需要。 need to review all the agents' code, and probably not. [36:09] 顺便说说我接下来最期待的事。 I will tell you what I'm looking forward to, by the way. [36:12] 今天我们聊了很多，但现在只是我坐在这里说。 a lot of these things that we're talking about here today, I mean, I'm kind of just [36:14] 到了AIE上海，我会把这些东西演示出来。 a talking head here, but when I'm at AI engineers in Shanghai, I'll be showing this stuff. [36:19] 会做很多演示。 I'll be showing this stuff a lot. [36:20] 至少会演示我们自己的实现。 You know, and I'll be at least our version of it. [36:22] 就像刚才说的，这些是行业趋势。 Like I said, these things that I'm talking about, these are industry trends. [36:25] 不只是微软在做。 It's not just us, Microsoft, that are doing these things. [36:28] 我会展示我们自己的版本。我觉得这些能力在产品里已经做得相当好了。 Obviously, I'll be showing our versions of these things, which I think have come out quite well in our products. [36:34] 所以我非常期待。 but yeah, I'm definitely looking forward to that. [36:36] 光讲的话，我可以一直讲下去。 I mean, I could talk about this till I'm blue in the face, but [36:38] 但这件事，真的是一图胜千言。 You know, this is one of those things where a picture says a thousand words. [36:41] 对，期待的不只是图片，还有现场Demo。 Yeah, look forward to the not only pictures, live demos. [36:45] 能亲眼看到，肯定很棒。 That's super cool to see. [36:47] 很好。 Great. [36:47] 回到刚才说的，建立评估能带来很大的提升， Yeah, yeah, like you mentioned, like the building evals, like you [36:52] 而对你来说，这已经是一种习惯。 get so much leverage before you but that is almost like a habit to you. [36:58] 很多人都应该养成这种习惯，它能大幅提升使用AI的能力。 And a lot of people should start adopting that habit because it can unlock so much leverage of using AI. [37:04] 这确实是个关键习惯。 That's a crucial habit. [37:05] 而且，越来越多的人把这当成专门的工作。 And increasingly you do find folks that are focused on that as a job. [37:09] 刚才说过，M365 Copilot有一整支评估团队。 As I said, we have in M365 Copilot an entire evals team. [37:13] 这就是他们的工作。 And that's what they do. [37:14] 建立评估，再根据客户反馈持续改进。 They build these evals, they improve these eval evals based on our customer feedback and that kind of thing. [37:20] 坦白说，这正成为一种新岗位。 So that is becoming kind of a new job, frankly. [37:23] 我们一直在讨论，工作将会怎样变化。 I mean, we talk about how jobs are gonna change and things like that. [37:26] 但过去几年也已经出现了全新的岗位类别， Meanwhile, there's an entirely new job category that has been [37:29] 比如评估维护者。 created over the past couple of years, which is eval maintainer. [37:34] 或者叫评估员。 Or something. Evaluator, I guess you might say. [37:36] 评估员。 evaluator. [37:38] 我听说，“计算员”以前也是一个职业。 Calculator used to be a job, what I heard is. [37:41] 没错，以前确实是。 That's right. [37:42] 类似这样的变化。 You know, stuff like that. [37:44] 新技术不断出现。 Yeah, and so we have these, you know, these new technologies that come on the scene. [37:48] 有时候，也会带来新工作。 And sometimes new jobs emerge from that. [37:50] 比如社交媒体运营。 I mean, social media coordinator also. [37:52] 没有社交媒体的时候，也就没有这个岗位，现在有了。 Like before social media there was no such thing, and now that is a job. [37:55] 总之，我们已经能看到全新的工作类别正在出现。 but yeah, now you see already you start to see entirely new job categories being created. [38:02] 最后一类问题，我想问问不好的策略， one I guess last question or categorical question I want to ask you is the bad strategies [38:08] 或者不算不好，但你觉得有更好做法的情况。 or maybe not bad strategies, but like there is a better way of doing this that you observed. [38:14] 不用点名。很多企业都在思考、尝试AI， Don't have to name anybody, but I'm sure there are a lot of enterprises that [38:19] 但可能还有更好的方式。 are thinking about this, they're doing things, but maybe there's a better way. [38:23] 能举几个例子吗？ Can you name a few? [38:25] 好，那我说一些通用的最佳实践。 Well, okay, in terms of I guess just general best practices. [38:29] 刚才说过，先写评分标准，就是很重要的一条。 I mean, and as I did say, what is certainly a best practice now is to start with a rubric. [38:34] 构思要做什么时，尤其是以AI为核心的产品， As you're thinking about what it is you're gonna build, especially if that [38:38] 要把评分标准视为成功标准。 thing is AI centric, then consider that the rubric is your success criteria. [38:44] 提前写出来。 Write that up front. [38:46] 很多企业其实已经有类似的东西了。 And many businesses, I mean you'll find that many businesses actually already have some analog of a rubric. [38:50] 比如银行的合规文档。 You know, for banks it'll be Their compliance documentation. [38:53] 它就是未来评分标准的一部分。 That's part of what will become their rubric. [38:55] 写法可能和给AI写的评分标准不完全一样，但也差得不远。 It may not be written exactly in the way that you would write a rubric for AI, but it's not that far off. [39:00] 可以据此调整。 It is adaptable in that way. [39:02] 这是第一点。 so that's one of them. [39:03] 第二，我经常说：AI需要一个格外长的试点期。 two, and I always say this, for AI, you kind of need to have an abnormally long pilot period. [39:10] 大家都有压力，恨不得明天就发布，因为现在做出东西很容易。 I mean, we have this pressure, we all want to release these things tomorrow because it's easy to put them together. [39:15] 正如前面讨论的，把一个想法做出来，甚至投入生产，都非常容易。 And as we were discussing earlier, it's super easy to put an idea. even into production. [39:20] 但不是每个想法都是好想法。 but not all ideas are good ideas. [39:22] 上线变得太容易， And so in a way, the fact is that the ease of putting these things into [39:26] 也让很多不好的想法被直接投入了生产。 production has resulted in people putting a lot of bad ideas into production also. [39:30] 所以，你需要试点。 So you want to have a pilot period. [39:31] 让真实用户使用，认真收集反馈。 You want to have your users interact with this stuff and really get their feedback. [39:36] 再把反馈纳入评分标准， And then go back and incorporate that feedback into the rubrics again so [39:40] 让评估能够检验系统是否符合用户真正想要的行为。 that when you do these evals, that it's acting the way your users want it to. [39:45] 试点期更长，AI又是概率性而非确定性的， With the long pilot period, with AI being probabilistic than [39:50] 这会不会意味着，我们需要更多服务，而不只是产品？ deterministic, do you think we need more service than product? [39:55] 比如前置部署工程师，也就是FDE。 Like such as FDEs, for example, the forward deployed engineers? [39:59] 是不是需要有人帮助企业采用AI、用好AI， Like do we need people to help these enterprises adopt AI, use it well, [40:04] 建立测试、评估和评分标准，最后把它真正落地？ like test like build evaluations, build rubrics, eventually like get it [40:10] 当然。 Sure. [40:11] 这就是另一个新岗位：前置部署工程师。 Well, there it is. Another job that didn't exist before the AI revolution is forward deployed engineers. [40:16] 微软也有这样的团队。 And we have that at Microsoft also. [40:17] 叫Microsoft Frontier Company。 That's called the Microsoft Frontier Company. [40:19] 一批FDE会进入客户现场， And it is a bunch of FDEs, forward deployed engineers that go into our customer sites [40:24] 帮助他们掌握使用AI和将AI投入生产的最佳实践。 and kind of help them work on the best practices for using AI and putting AI into production. [40:30] 这确实能帮上忙。 and so that can help. [40:31] 但与此同时， I mean, that helps, but at the same time, I think everybody wants to have, [40:34] 每个工程师现在都应该具备一些基本理解。 everybody should have, every engineer should have a baseline understanding now. [40:38] 知道AI是什么、怎么用、怎样给Agent提供上下文， What AI is, how to use it, how to now provide context for your agents, [40:41] 以及怎样写评分标准。 how to, you know, write these rubrics and stuff like that. [40:45] 挑战在于，变化太快了。 you know, the challenge is it is changing very quickly. [40:47] 我很理解。 I understand this, right? [40:48] 对我而言，这就是工作。 And now for me personally, it's my job. [40:51] 我整天都在接触这些东西。 I'm literally in this all day long. [40:54] 很多晚上也是。 And most nights also, right? [40:55] 我一直在实际使用、探索。 I'm here acting on this stuff all the time. [40:58] 但对其他人来说，变化这么快，确实是个挑战。 And I think that's challenging for people because it is changing so quickly. [41:01] 真的，快得不可思议。 Like, my word, like it's it's insane, right? [41:03] 每天都有新东西。 Every day it's something new. [41:04] 确实如此。 and that's true. [41:05] 但即便这样，为了自己，也值得尽量跟上。 but nonetheless I think I think you owe it to yourself to stay abreast of these things. [41:11] 我一直跟大家说：什么都去动手试。 And what I always tell people, mess with everything. [41:14] 我说的是真的什么都试。 And I mean everything. [41:15] 能接触到的，都去试试。 I mean everything you can get your hands on, right? [41:17] 试试微软Copilot，试试其他编程Agent、消费级产品， So if you know, try Microsoft Copilot and our things, try other [41:21] 或者用Suno给自己做首歌。 coding agents, try consumer stuff, try Suno and make yourself a song. [41:26] 我觉得这很好玩。 I think that's cool. [41:27] Suno很酷。 I think Suno is cool. [41:28] 我很喜欢Suno。 I mean, I love Suno. [41:30] 你看，我还在学弹身后这把吉他。 I still am learning to play this guitar back here, you know. [41:33] 钢琴我会弹一点，吉他还在学。 I play piano, okay, but the guitar I'm still learning. [41:35] 总之，去试这些东西。 anyway, you know, try all these things because it really gives you a feel for what [41:41] 亲身体验才能真正理解它们是什么、怎么工作； they are and how they work in a way that, you know, no amount of podcasts or YouTube [41:47] 听再多播客、看再多视频，都替代不了这种感受。 videos or whatever can really give you this sense that first hand experience will. [41:51] 对。今天早上我在哥伦比亚大学讲课，也告诉学生： Yeah, like just this morning I was at Columbia, like giving a lecture at Columbia and I tell the students, [41:57] 把学习AI当成学一门手艺，就像学游泳、滑雪。 you know, treat learning AI as a craft, like how you learn swimming, how you learn snowboarding. [42:02] 光看视频永远学不会。 You never can learn that still through watching video. [42:04] 必须自己动手。 You have to do it. [42:07] 总结一下，今天我们从一个问题出发：从前沿模型和运行框架， Yeah, so if I summarize our talk today, we started with the question of on top [42:13] 到真正的产品，中间还有哪些工作要做？ of frontier model and harness to the actual product, what are the jobs to do to be done? [42:18] 我们谈到了评估。 Like we talk about evaluations. [42:20] 谈到了身份和权限。 We talk about kind of the identity and permission. [42:25] 谈到了基础设施，怎样提取隐性知识、整理信息， We talk about you know, building the infrastructure or the extracting the tacit [42:31] 再把它们提供给AI。 information, building all the information, giving all the information to AI. [42:35] 我觉得核心还是人。 And I feel like at the center it's really people. [42:37] 是人在做这些工作，也由人来授权和认可。 It's really people who do all this and give permissions. [42:40] 还有遗漏吗？ did I miss anything? [42:42] 我觉得没有。 I don't think so. [42:43] 你总结得很完整。 I think you got everything. [42:45] 很好。 Nice, nice. [42:45] 非常感谢。 Thank you so much. [42:47] 期待你来到AIE上海， And yeah, look forward to having you in at AI Shanghai and look [42:51] 也期待现场看到你演示这些精彩的东西。 forward to watching live demos of other amazing things you do. [42:55] 我也很期待去现场。 I look forward to being there. [42:57] 谢谢。 Thank you. [42:57] 谢谢。 Thank you.