Microsoft’s Neville Finds Multi-Turn Chats Degrade AI Performance

微软研究员内维尔发现多轮对话会降低AI性能 大语言模型在实际工作流中容易受挫，因为用户逐步提供的澄清说明会复合上下文错误，而Transformer架构难以有效计算这些错误。 詹妮弗·内维尔回顾了自己从大学本科物理和数学专业辍学，到先后在普渡大学和微软研究院主导前沿AI评估研究的经历。她分析了当代大语言模型在贴近实际的多轮对话与长周期智能体工作流中性能下降的原因，并强调人类错误地将自身对难度的认知投射到了Transformer计算之上。为了克服复合错误和细微的语义流失，内维尔提出了保护隐私的遥测分析以及高保真用户反馈回路，最终展望了由人类和AI智能体组成的人机协同团队的未来。 詹妮弗·内维尔在微软研究院的团队开展的评估表明，与优化过的单轮提示词相比，大语言模型的准确率在多轮交互中大幅下降，这反映了用户在反复澄清指令时常遇到的日常阻碍。 (10:00) 内维尔指出了“令人意外的AI故障”背后的一个核心认知偏差：人类错误地认为对人而言简单的任务对Transformer也应该很简单，却忽视了诸如反复修改文档时的误差累积等机械性计算限制。 (20:00) 相比学术界，工业界AI研究实验室拥有独特优势，能够接触到规模化、保护隐私的遥测数据和真实活跃的用户工作流，使团队能够针对多轮任务构建产品内置引导和强化学习。 (7:30) 现代对话界面使用户能够就模型错误提供丰富、高保真的描述性反馈，从而将用户转变为积极的训练者，协助界定并拓展模型的运行边界。 (25:00)

Microsoft Research Partner Research Manager Jennifer Neville argues that artificial intelligence systems often fail in practice because model benchmarks reward fully specified, single-turn prompts that clash with how humans iteratively work. Bridging this gap requires designing evaluations for messy, multi-turn interactions, understanding the computational constraints of transformer architectures, and leveraging high-fidelity user feedback. The Breakdown in Multi-Turn and Long-Horizon Workflows (10:00) Neville's AI Interaction and Learning Team found that large language models experience substantial performance degradation during multi-turn interactions, despite excelling on complex single-turn benchmarks. In everyday practice, users rarely compose comprehensive system prompts on their first try; instead, they start with underspecified requests and incrementally supply clarifications. As this conversational context expands, models frequently lose coherence. When tasks extend into long-horizon agentic workflows, such as iterative document editing, early uncorrected mistakes compound and progressively corrupt the context window for subsequent operations. Human Assumptions Versus Computational Realities (20:00) The discrepancy between user expectations and model capabilities often stems from a fundamental cognitive bias: humans project their own perception of difficulty onto AI. Neville observes that tasks humans find easy—such as retaining semantic meaning across minor edits—can be exceptionally difficult for transformer computations and dense retrieval systems, whereas traditionally hard human tasks like code generation or complex math may be solved more easily. Recognizing this disconnect helps researchers distinguish between anticipated failures (such as factual hallucinations) and surprising degradations caused by the architectural mechanics of transformers. Harnessing Production Telemetry and High-Fidelity Feedback (15:00) To diagnose why models break down, Neville points to industrial telemetry as a vital tool for observing failure patterns at scale. To protect privacy, Microsoft researchers do not inspect raw personal text; instead, automated compute environments process aggregated data, and downstream engineering relies on voluntary user feedback like thumbs-down ratings paired with explanatory notes. Neville highlights that unlike historical recommender systems constrained to binary ratings, modern conversational interfaces allow users to articulate exactly what failed and what was expected, converting user oversight into rich training data for expanding model boundaries. Academic Theory Meets Industrial Scale (7:30) Reflecting on her career path—from exploring statistical relational learning at Purdue to working at Microsoft Research—Neville emphasizes the complementary strengths of academia and industry. While university research excels at developing high-level theoretical abstractions across diverse domains, industrial labs provide access to production-scale infrastructure, active systems, and live user interactions. Neville also cautions practitioners against assuming low benchmark metrics always signal algorithmic failure, noting her hardest career lesson has been realizing that missing metrics are repeatedly caused by data shifts, dirty inputs, or annotation errors. An open debate remains over whether transformer constraints can be mitigated through external agentic scaffolding and reinforcement learning, or if foundational architectures must be replaced entirely. Neville anticipates that future systems will achieve deeper grounding and move toward mixed-initiative organizations where human and AI agents collaborate on complex knowledge work.

微软研究院（Microsoft Research）合伙研究经理兼普渡大学教授珍妮弗·内维尔（Jennifer Neville）指出，当前人工智能的基准评测脱离了真实的协作场景；大语言模型在多轮交互和长流程任务中暴露出的退化，根源在于人类往往错误地将自身的认知难度投射到基于Transformer架构的模型运算之上。 学术界与工业界视角的碰撞与融合 (7:30) 内维尔的学术道路并非一蹴而就，她在经历了数学、物理以及认知科学的摸索并曾一度退学后，才转向计算机科学，并在毕业研究中涉足统计关系学习领域。在获得普渡大学终身教职后，她先后在加州大学伯克利分校西蒙斯计算理论研究所从事理论深造，随后于2021年赴微软研究院休假并留任。她认为学术界与工业界虽然秉持相同的求知欲，但在定位上差异鲜明：学术界通常依赖NSF和DARPA等机构的资金支持，偏好高层抽象与跨领域普适性；工业实验室则拥有大规模真实系统、海量实时数据与活跃用户环境，能够直接在高并发与实际产品约束下检验算法的行为表现，两者具有深度的互补协同价值。 多轮对话评测揭示模型性能退化 (10:00) 在工业界领导AI交互与学习团队期间，内维尔指出传统机器学习基准由于过于简化，无法准确反映模型在复杂工作场所中的真实表现。现实中的非专业用户很少能在首次尝试时就给出完备的单轮提示词，往往是目标表达不完全，并在后续对话中逐步添加条件和澄清。内维尔的团队通过模拟用户分步递交约束条件发现，在单轮基准测试中得分极高的模型，其表现在多轮交互中会出现严重退化。为应对这一困境，内维尔提出了一种实用妥协方案：用户可在多轮沟通致使模型困惑时终止对话、清空历史上下文，并将已理顺的要求汇整为单一的完备提示词重新输入；与此同时，团队正在研发专门的强化学习方法以增强模型在多轮环境中的稳定性。 规模化遥测与隐私保护的数据研究 (15:00) 依托工业实验室环境，研究人员能够与产品部门协作分析脱敏后的海量交互遥测日志。内维尔类比早期网络搜索引擎的发展过程指出，正如当时用户必须逐步摸索出适合检索的语法与词汇一样，当下的公众也正在经历适应对话式AI界面的学习期。研究团队通过分析成功与失败的双向行为模式，旨在设计出更优的产品内置引导机制，指引用户走向系统能够可靠执行的操作路径。内维尔强调，为杜绝侵犯隐私，研究人员不会用肉眼直接查阅用户的原始输入文本，所有分析均在严格受限的计算环境中以自动化方式提炼宏观模式；用户通过“点踩”并主动填写的文字说明等高保真负反馈信号，则被作为改进下游算法的重要依据，而非直接用于对个人数据的微调。 长流程任务瓶颈与人类难度投射偏差 (20:00) 在文档连续迭代修改等长流程Agent工作流中，智能体不仅需要持续跟踪用户意图与文档状态，还须判定过时信息并协同工具调用。然而，长流程极易引发误差累积：早期微小未被察觉的差错会在多轮交互中复合放大，最终使上下文受到污染并导致模型深度混乱。内维尔指出，虽然算术错误或法律幻觉等“预期内失败”已有既定基准进行缓解，但在反复编辑中语义静默丢失等“令人意外的失败”却更具破坏力。这种意外感源于人类的认知偏差：人类本能地将自身的难易标准强加给AI，误以为人类觉得简单的任务对模型也容易，人类觉得繁琐的任务对模型也必然困难；但Transformer架构和密集检索的计算逻辑截然不同，厘清其数学与结构上的计算边界才是诊断故障的核心。 高保真反馈价值与人机协同组织展望 (25:00) 针对用户层面的落差，内维尔提醒用户目前尚不能对AI进行完全放权的委托；当模型给出不佳输出时，往往通过重新构思提问方式便能取得显著改善。与以往搜索推荐系统中的二元点赞点踩不同，现代生成式交互允许用户提供详细的高保真语言反馈，精确指出系统在何处偏离预期以及实际应得的结果，从而协助工程人员校准模型边界。面对学界关于Transformer架构局限性是否必须依靠全新架构替换的争论，内维尔回顾了2000年在AT&T实验室时学者们曾预测围棋程序需要50到100年才能攻克、但最终被加速解决的历程，对技术演进速度保持乐观，并预期未来在办公场景中将形成人机混合主创团队共同协作与创新的组织架构。 内维尔在速答环节总结指出，从业者在基准指标未达预期时，往往下意识归咎于复杂的建模难题，但实际根源往往是脏数据、标注错误或分布偏移；同时她强调研究应大胆提问并持续从环境交互中汲取灵感。然而，长流程Agent系统的自我监控、错误回滚机制以及何时主动引入人类介入，仍是当前尚未解决的核心工程与理论课题。

## Background 本期内容由来自微软研究社区的研究人员于 2026 年 10 月 6 日在微软研究院播客平台发布。节目聚焦微软研究院合伙人研究经理兼普渡大学讲席教授 Jennifer Neville，围绕她曲折的求学历程，以及她如何通过识别 AI 面对复杂任务时的“意外失效”来推动系统突破展开深度探讨。 ## TL;DR 微软研究院应用科学家 Chad Atalla 对话 Jennifer Neville，深入拆解了 AI 评估与复杂交互的前沿瓶颈。Neville 从自己多次换专业与退学、最终投身数据科学与学术研究的经历谈起，重点指出现有主流模型在单轮基准测试下表现优异，但在真实多轮交互和复杂长文档协同中性能会发生严重滑坡。针对大模型在长程任务中的“意外失效”，她深入剖析了人类直觉与 Transformer 架构机制之间的错位，并展示了如何通过隐私保护下的大规模日志分析提取错误模式、引导用户提供高保真反馈。最后，她探讨了突破 Transformer 物理局限的未来架构走向，展望了人类与 AI 组队协同创新的愿景，并在问答环节分享了死磕数据、勇敢提问等深刻的人生洞察。 ## Atmosphere & style 整场对话在严谨的学术深度与轻松幽默的极客日常之间达成了极佳平衡，既有前沿技术交锋的烧脑感，又充满了平易近人的温情。主持人 Chad 作为资深评估从业者与 Neville 惺惺相惜，双方对话默契自然，从面对模型迷失时的工程挫败，谈到攻克围棋预言破灭时的释然与振奋。在回忆养娃期间拿下教职、甚至把婴儿翻身当成“正样本训练”的生活片段时，对话充满了自嘲与欢笑，让听众不仅能真切感受到顶级学者在人类认知前沿追逐未知的极致兴奋，也能体会到面对复杂科学问题时的谦逊与执着。 ## Bulletin - `10:00` 大语言模型在经过单轮优化的基准测试中表现出色，但在处理相同任务的多轮动态交互时性能会出现显著滑坡，主要由于长上下文累积造成的干扰和逻辑漂移。 - `15:00` 针对多轮对话性能退化，实用的规避策略是在模型出现混乱时打断对话并清空上下文，将摸索出的全部明确约束整理为一条详尽的单轮指令重新提交。 - `20:00` 长周期智能体与文档协作中频发的“意外失效”，本质上源于人类基于自身直觉对任务难度的预判，与 Transformer 底层依赖注意力机制和内部表征的计算几何特性存在认知错位。 - `25:00` 在聊天式交互范式下，用户能够提供高保真度的详细纠错反馈，取代传统的点赞或点踩，直接参与后端模型的优化与能力边界拓展。 ## Key takeaways - `00:00` Jennifer Neville 早期曾多次转换专业甚至从大学退学，最终因探索数据处理与理解的驱动力进入计算机科学领域。 - `05:00` 科研在人类认知最前沿破除未知的成就感促使 Neville 投身学术，并在普渡大学执教期间保持兼顾理论与落地的双栖路线。 - `05:00` 学术界注重高度抽象化以争取科研经费，而工业界在大规模真实系统、复杂数据及用户交互落地方面具有得天独厚的核心优势。 - `10:00` 传统基准测试过于简陋并脱离现实，评估多轮对话、协同交互及长周期任务是摸清模型短板并突破性能瓶颈的前哨。 - `10:00` 普通用户在真实交互中往往无法一次性说明需求，而是通过多轮追加约束，现行模型在多轮展开的任务中性能出现严重退化。 - `15:00` 研究团队通过海量用户日志的大规模统计分析提取失败模式，辅导用户提出更具成效的问题以适应聊天式交互范式。 - `15:00` 日志分析严格遵循隐私保护计算规范，禁止人工肉眼查看原始交互，仅在受控隔离环境中提取高阶抽象统计模式。 - `20:00` 长流程文档编辑中微小的语义错误会连锁放大，迫使研究团队探索智能体的外部工具调用、人机协同介入与自省回滚机制。 - `20:00` 所谓“意外失效”主要源于人类智能对“难易程度”的直觉预期，与 Transformer 底层计算几何及注意力机制的现实能力发生错位。 - `25:00` 用户应对当前 AI 建立合理预期，避免追求 100% 委托，通过将 AI 置于有监督、有验证的工作流中提升生产力。 - `25:00` 现代聊天界面支持高保真反馈，用户可通过详述错误与原本预期，实质性参与到模型的训练与迭代过程。 - `30:00` 当前主流通过外层包装和 Agentic 编排套件修补 Transformer 架构缺陷，未来需要支持更长程交互与实体化世界理解的底层突破。 - `30:00` 历史经验显示技术突破往往远超人类预料，庞大的人才密度与计算规模正大幅压缩 AI 跨越技术瓶颈的周期。 - `33:00` 长远终极愿景是构建人类与 AI 深度融合的混合型组织与团队，在复杂系统中共同完成任务并实现创新。 - `34:20` 在遭遇模型失效时，首要法则应是深入审查底层数据本身，而非单纯归咎于算法或任务难度。 ## Detailed walkthrough ### `00:00` 缘起与转折：从多次退学到走进计算机科学 本期播客由微软研究院（Microsoft Research）应用科学家 Chad Atalla 主持，邀请到的是微软研究院合伙人研究经理（Partner Research Manager）、普渡大学计算机科学与统计学 Samuel D. Conte 讲席教授 Jennifer Neville。Jennifer 的研究聚焦于交互式领域与结构化数据的机器学习和人工智能，重点探究用于训练 AI 系统的底层数据点如何影响系统行为，以及这些行为如何与用户的真实意图对齐。迄今为止，她已发表逾 130 篇论文，被引用超 10,000 次，并斩获美国国家科学基金会杰出青年学者奖（NSF CAREER Award）、IEEE“十位值得关注的 AI 学者”（AI's 10 to Watch）以及 ICDM 和 ICLR 最佳论文奖等荣誉。 谈及成长轨迹，Jennifer 回忆道，自己小时候从没想过会进入计算机领域，原因很简单：“我父亲是计算机科学家，作为十几岁叛逆的孩子，我唯一的想法就是绝不做我父亲做的事。”步入大学后，她最初选择主修数学，随后又转到物理学，但始终无法与这些专业产生共鸣，最终甚至一度从大学退学。再次重返校园时，她转投认知科学（Cognitive Science），却又感觉该专业“太软（too squishy）”，缺乏足够的数学深度。当时根本没有人引导她认识到，AI 恰恰是结合认知科学、数学与计算思维的最佳交汇点。再度工作一段时间后，Jennifer 终于选定了计算机科学，而促成这一决定的直接动力，单纯是因为她想探寻如何有效处理与思考数据，而当时甚至没完全意识到这本就是计算机科学的核心部分。 - 主持人 Chad Atalla 透露自己儿时的梦想曾是成为天体物理学家，后同样转向了计算机科学领域。 - Jennifer Neville 的早年求学路径经历多次反复：主修数学 $\rightarrow$ 主修物理 $\rightarrow$ 辍学工作 $\rightarrow$ 主修认知科学 $\rightarrow$ 再次工作 $\rightarrow$ 最终修读计算机科学。 - 促使她重返校园并选定 CS 的核心驱动力是“如何思考并处理数据（working with data and dealing with data）”。 ```mermaid flowchart TD accTitle: Jennifer Neville 早年求学历程流转图 accDescr: 展示 Jennifer 从最初排斥计算机，经历多次换专业和退学，最终找到数据与计算交汇点的曲折路径。 subgraph Early["早期探索与尝试"] A["父亲为计算机科学家<br/>（叛逆心理：绝不学计算机）"] --> B["主修数学与物理<br/>（无法产生共鸣）"] B --> C["大学退学并参加工作"] end subgraph Mid["重返大学与摸索"] C --> D["主修认知科学<br/>（过于偏软、数学不足）"] D --> E["再次离开校园参加工作"] end subgraph Final["锚定方向"] E --> F["重返校园主修计算机科学<br/>（核心驱动力：探索数据）"] F --> G["误打误撞进入 AI 与计算领域"] end ``` --- ### `05:00` 科研的“成瘾”快感与产学双栖的互补之路 Jennifer 之所以踏上学术研究之路，源于本科荣誉学位项目（Honors Program）的强制性毕业研究要求。当导师让她自主拟定研究课题时，她对数据和 AI 产生浓厚兴趣，并在广泛阅读文献后提出：希望探索如何对互联的网页数据进行数据挖掘（data mining on web data that was interconnected）。系里随即为她引荐了一位专注于当时新兴领域的导师，该领域被称为统计关系学习（Statistical Relational Learning, SRL）。在导师的指导下，她不仅完成了项目，还在研讨会（Workshop）上发表了人生首篇学术论文，从此彻底迷上了科研，改变了原本不打算读研的人生规划。 谈及究竟是什么特质吸引了她，Jennifer 将科研过程比作一种让人上瘾的“毒药”。在研究过程中，常常会经历漫长的停滞与挫败，陷入原地打转的沮丧情绪；但导师始终给予积极鼓励，坚信只要继续推进就能有所收获。Jennifer 动情地回顾道：“那种能够理解某种别人尚未理解的事物的感受——仅仅因为它正处于人类已知边界的最前沿——就像是一种毒品，这种对同一种感觉的追逐让我在研究领域坚守了这么久。”正是这种在人类认知前沿破除未知的成就感，深深锁定了她的学术人生。 在普渡大学执教和指导学生二十年间，Jennifer 的研究始终兼顾理论与应用。在拿到终身教职（Tenure）后的第一次学术休假（Sabbatical）中，面对许多同僚去工业界实验室后一去不回的先例，她为了确保自己能重返讲台，特意选择了更加理论化的加州大学伯克利分校西蒙斯计算理论研究所（Simons Institute at UC Berkeley）。而到了第二次学术休假，她选择走向实践的另一端——加入微软研究院（MSR），并于 2021 年正式兼顾两界身份。她深刻体会到理论算法在真实系统、海量数据与真实用户场景中落地检验的巨大魅力。 Jennifer 对高校与工业界的研究范式差异进行了精辟剖析：学术界更倾向于提炼高度抽象的问题，以覆盖多个不同领域，从而争取 DARPA 或 NSF 等机构的科研经费；而工业界的优势在于能在大规模真实系统与复杂数据中实操落地，研发目标直接服务于真实产品和企业战略利益。Jennifer 特别强调：“但眼下，如果你从事的是 AI 系统方面的研究，我认为工业界确实才是最核心的阵地。”Chad Atalla 也深有共鸣地表示，自己在微软负责 AI 评测六年多，长期面对该领域诸多令人望而生畏的基础难题，而 Jennifer 对复杂 AI 系统的评测洞察与解决方案，为业内从业者带来了极大的启发与解题曙光。 - Jennifer Neville 在本科期间的研究方向催生自早期万维网数据的关联性挖掘，直接对接了“统计关系学习（Statistical Relational Learning）”这一前沿领域。 - 两次学术休假的战略性选择：第一次前往 Simons Institute 专注纯理论，以便平稳回流大学；第二次前往微软研究院（MSR）深入工程系统与工业数据前线。 - 学术界与工业界的核心差异：学术界注重高度抽象化与跨领域通用性（申请科研基金导向）；工业界注重在真实规模、复杂系统与真实用户交互下的实操验证（产品落地与商业导向）。 - Jennifer Neville 认为，当前从事 AI 系统研究（AI Systems）的最佳场域正是工业界，但产学研之间的双向流动具有极高的协同价值。 ### `10:00` AI 交互与学习团队的研究愿景：为何评估是突破模型性能的前哨 对话由主持人的提问展开。主持人谈到 Jennifer Neville 既批判现有的 AI 评估体系，又能沿途给出务实的解决方案，但这仅仅是她庞大研究版图的一小部分，因而询问她所带领团队的核心使命。Jennifer Neville 介绍道，她的团队名为“AI 交互与学习团队”（AI Interaction and Learning Team），其核心目标是推动 AI 系统在真实工作环境中的前沿表现。这要求团队深入研究当前系统性能的边界究竟在哪里、普通用户在日常工作流中会有怎样的真实体验，并在此基础上开发出更先进的算法、模型和估计方法，以提升模型在处理复杂任务时的能力。 Jennifer Neville 强调，之所以将大量精力投向评估，是因为传统的机器学习（ML）与人工智能（AI）领域的基准测试过于简陋，完全脱离了现实世界中用户的实际用法。真实的复杂场景往往包含多轮对话交互、多人协作环境以及超长周期任务（long-horizon tasks）。面对这些差距，必须重新审视“我们究竟对系统有何期望”，设计出更接地气的评估框架，进而探明系统在何处需要补强。因此，评估本身并不是终点，而是理解模型短板、突破性能天花板的必由之路。 在谈及真实交互时，Jennifer Neville 重点剖析了团队近期关于大语言模型在多轮对话中“迷失自我”以及在文档协作中损坏内容的代表性成果。她指出，计算机科学家和软件工程师往往会下意识地假设用户能够给出完美的指令，但普通用户在实际交互中根本无法在首轮提示词里全面说明需求。用户通常是在多轮对话中边摸索、边追加约束与澄清信息的。为了在受控环境中精确测量这一现象，团队将原本复杂的单轮完整指令解构，使用模拟用户在多个对话轮次中逐步补充这些细节要求，并以此测试目前主流大语言模型的反应。测试发现，模型在专门针对单轮优化过的基准下往往表现优异，但在面对经过多轮展开的完全相同任务时，其性能竟然出现了严重的滑坡。 面对这一出人意料的性能衰退，Jennifer Neville 解释了背后的意义： > “And we found that the performance that we see in single turn which is very high because, of course, we're optimizing to that as model builders, actually degrades significantly over multiple turns.”（我们发现，由于模型构建者一直在针对单轮表现进行优化，模型在单轮对话中的性能非常高，但在多轮对话中却出现了显著的退化。） 这一研究成果公开发布后，在广大用户群体中引发了强烈的共鸣。大量用户反馈道：“这就是我每天的真实体验，我早就觉得它越聊越糊涂，到底该怎么解决？” ```mermaid flowchart TD accTitle: Multi-turn degradation evaluation flow accDescr: Shows how single-turn instructions are split across simulated turns leading to model degradation and the user workaround. subgraph EvaluationSetup ["评估设计思路"] A["复杂任务完整单轮指令"] -->|"拆解并逐步追加细节"| B["模拟用户多轮提示生成"] end subgraph ModelReaction ["模型表现差异"] A -->|"基准单轮测试"| C["高水准表现 (针对单轮过度优化)"] B -->|"多轮动态交互"| D["性能显著退化 (上下文累积干扰)"] end subgraph PracticalFix ["实用规避策略"] D --> E["停止并清除多轮上下文"] E --> F["直接输入汇总后的完整单轮提示词"] end ``` --- ### `15:00` 从用户日志洞察失败模式：隐私保护计算与“教用户提示”的演进 既然多轮性能滑坡普遍存在，如何才能修复这一问题？Jennifer Neville 指出，团队一方面正在探索强化学习（Reinforcement Learning）手段，以提高模型在多轮动态交互中的鲁棒性；另一方面，研究也带来了一个立竿见影的“用户侧技巧”：当发现模型在多轮交互中开始产生混乱和逻辑漂移时，最聪明的办法往往是直接打断对话，清空上下文重新开一局，利用此前摸索出的全部明确约束，将任务整理为一条详尽完整的单轮指令一次性喂给模型。这说明算法研究往往能反哺交互策略，教用户如何改变习惯来迁就当前技术局限。 主持人进一步追问，研究团队是如何洞察到用户这种“无法在首轮讲清楚需求”的痛点，又是如何通过模拟技术来进行此类长链条评估的。Jennifer Neville 回应道，身处微软这样拥有大规模工业级实验室的环境中，最大的优势之一就是能够接触到海量的消费级产品日志。团队与内部各产品部门紧密协作，通过对海量用户日志开展大规模统计分析，归纳出用户在交互中最常见的失败模式与成功模式。她回顾了互联网早期搜索引擎的历史：在搜索引擎刚刚问世时，大众并不知道如何检索，是在漫长的使用过程中摸索学会了提炼关键词；如今，交互范式正在经历从“搜索”向“聊天式 AI”的剧烈迁移，用户同样需要经历一段学习曲线。通过分析真实场景中的成功案例，团队可以设计出推荐策略或提示引导，像家庭教师一样辅导用户如何提出更具成效的问题。 主持人随即追问了隐私问题：在大规模研究真实日志时，研究人员到底能不能直接看到用户的私密对话？Jennifer Neville 对此做出了非常严谨的技术界定： > “We're not actually looking at the data. We are analyzing the data at scale to extract patterns of failures or successes in a privacy kind of way.”（我们实际上并没有人工查看这些数据。我们是在保护隐私的前提下，对数据进行大规模分析，从而提取出失败或成功的模式特征。） 她详细列举了保护用户隐私的具体边界和安全措施： - **严格禁止人工裸眼调阅**：研究员绝不能在没有防护的屏幕上肉眼查看（eyes-on）具体的交互文本，所有计算代码只能推送到受严格权限管控的隔离计算环境中自动跑批。 - **高阶抽象聚合**：最终留给算法研究团队的，全部是脱敏后、经过高阶抽象的宏观模式与统计特征，绝非底层原始记录。 - **不针对私人聊天进行微调**：系统绝不会直接拿普通用户的日常对话或回答来微调模型底座。 - **志愿捐赠通道**：只有当用户主动点击“踩（thumbs down）”并选择捐赠数据以附带文本反馈时，或者通过特定渠道同意时，这些详尽描述才会在安全聚合蒸馏后，作为模型对齐与算法迭代的参考依据。 --- ### `20:00` 智能体与文档协同中的“意外失效”：人类直觉与 Transformer 算力现实的错位 对话随后转向当下的前沿热点：AI 交互正在从简单的多轮问答向智能体（agentic）以及协同知识工作（collaborative knowledge work）拓展。主持人询问在长周期工作流和智能体任务中，团队又发现了哪些值得关注的现象。Jennifer Neville 介绍，团队目前双管齐下：既有偏理论的工作去论证哪些任务在数学和架构本质上是 Transformer 难以内部计算的；也有基于模拟的长工作流实验，例如模拟智能体对一篇复杂长文档进行反复修改与多轮编辑。 在反复修改文档的场景中，智能体面临着巨大的上下文维护压力：它不仅必须追踪用户在每一阶段的真实意图，还要精确拿捏文档当前的最新状态，辨识哪些信息相关、哪些已失效。尽管智能体在某些简单任务上表现不错，但当工作链条被拉长时，如果缺乏自检与纠错机制，细微的错误便会层层传导、连锁放大，最终导致模型彻底丧失对全局的把控。这就引出了一系列亟待解决的学术问题：何时该让智能体调用外部工具？何时应该打断执行并引入人机协同（human-in-the-loop）？智能体是否能够具备自我监控、自我反思能力，在发现走入死胡同后主动回滚到历史版本？ 针对长流程中频繁出现的“意外失效”（surprising failures），主持人提出疑问：“意外失效”是否意味着它打破了人们的预设预期？模型到底是犯了意料之中的错误，还是犯了更多让人摸不着头脑的怪异错误？Jennifer Neville 坦言，传统已知的错误（例如数学算术算错、法律咨询中捏造虚假案例判据）早已被学界广泛研究并设立了专门的评测基准，随着算法迭代，这部分预料之中的错误正在逐渐减少。真正令人头痛的，是在长篇文档编辑等任务中发生的一些极其微妙的语义内容流失——模型看似每一步都在执行命令，但在几个轮次之后，整篇文档的核心意图和逻辑语义已经被彻底篡改或破坏。 Jennifer Neville 深入剖析了“意外”这一感受的人类心理根源： > “And so the surprising kind of creeps in, I think, with our own expectations based on human intelligence about what's hard or what's easy.”（因此，我认为那种‘意外感’的悄然滋生，本质上源于我们基于人类智能对什么是难、什么是容易所产生的自我预期。） 人类常常理所当然地认为，那些人类觉得棘手的任务对模型肯定也很难，而那些人类觉得不假思索就能办妥的事模型也理应轻松拿捏。但大语言模型完全不同，它的运算逻辑依赖于 Transformer 底层的注意力机制、内部表征能力以及检索系统的召回机制。那些在人类看来显而易见的常识或简单的状态保持，在模型底层的计算几何中可能反而是极难跨越的障碍。这种人类认知与机器底层机制之间的巨大鸿沟，构成了智能体在现实复杂应用中屡屡让人大跌眼镜的深层原因。 ### `25:00` 用户在当下应如何面对 AI：建立合理预期与高保真反馈 谈及外界由于将人工智能类比为人类智能，再加上铺天盖地的宣传与炒作，导致对 AI 抱有过高期望这一现象，Jennifer Neville 为普通用户如何穿透噪音、建立合理预期提供了实用建议。她认为当前的 AI 系统已经具备许多强大功能，非常适合用于现有的工作环境中，但关键在于“不要期望它们在所有方面都能达到 100% 的成功率”。如果系统给出的回答有误，用户不必立即认定该模型完全无法胜任这项任务，而是应该尝试换一种问法或者重新提问，往往就会得到更好的答案。目前这些系统还不适合被 100% 委托所有事务，最佳的使用方式是将其纳入有监督、有验证的工作流中，这样既能大幅提升工作效率，也能保障结果的可靠性。同时，由于整个 AI 体系处于“边研发边上线”的实时迭代状态，用户的真实使用案例恰恰是帮助研究人员拓宽模型边界的关键输入。 ```mermaid flowchart TD accTitle: 高保真反馈闭环工作流 accDescr: 展示用户从提问、校验到高保真纠错，最终驱动模型权重更新与迭代的流程。 subgraph Workflow["人机协作工作流"] A["向模型输入任务 Prompt"] --> B["获取模型输出结果"] B --> C{"人工校验与监督"} C -->|"通过"| D["提升日常工作效率"] C -->|"不符合预期"| E["换种问法重新提问"] E --> B end subgraph Feedback["高保真反馈机制"] C -->|"纠错反馈"| F["详细说明错误所在与原本预期"] F --> G["取代传统的点赞/点踩低维反馈"] G --> H["进入后端模型更新与训练流程"] H -.->|"拓宽模型能力边界"| A end ``` 针对反馈机制，Neville 指出了一个很多人尚未意识到的代际差异：过去在搜索或推荐系统时代，用户只能提供点赞或点踩（thumbs up, thumbs down）这种极低维度的反馈；而在当下的聊天式交互界面中，用户可以给出保真度极高（high fidelity）的反馈。Jennifer Neville 直言： > “And so you can think of that as users, your role can be to train the models to do the things that you would have wanted to do, but it couldn't do right now.” 她鼓励用户在模型表现糟糕时，直接通过打字或语音清晰地表达出究竟哪里出了问题、用户最初的期望是什么、以及模型实际给出的内容是什么。这些详细的交互信息会被用于后续的模型更新，让用户实质上参与到了模型的“训练”过程之中。 - 核心原则：保持人工监督与验证（oversight and verification），不搞 100% 全权放手。 - 遇错策略：不要因一次失败就假定模型做不了，尝试重写提示词（prompt）再次提问。 - 角色转变：从被动的内容接收者，转变为利用高保真上下文纠错帮助模型迭代的协作者。 --- ### `30:00` Transformer 架构的边界与围棋预言的破灭 当被问及超大 AI 系统的底层科学走向何方时，Neville 提出了目前研究社区正在探讨的一个核心问题：Transformer 架构到底是不是最合适的底层技术基座。由于 Transformer 在计算机制本身存在一些已知的物理和理论局限，当前业界主要是依靠外层的“包装（wrapper）”、智能体编排套件（agentic harnesses）以及后端推理链来克服这些缺陷。 ```mermaid flowchart LR accTitle: 当前 AI 系统架构与未来演进对比 accDescr: 对比基于 Transformer 外包智能体框架与面向长程世界理解的未来架构。 subgraph Current["当前系统形态"] T["底层: Transformer 架构"] --> W["外层包装 / Agentic 套件"] W --> R["后端推理与编排"] end subgraph Future["下一代系统演进目标"] D["根本性的新型底层架构探索?"] L["长程交互能力 (Long-term Interactions)"] U["更具实体感的物理世界理解 (World Understanding)"] end Current -.->|"演进与互补"| Future ``` Neville 认为下一个研究阶段必须解决让模型处理更长期的交互（longer term interactions），并让模型以比现在更加具体、切实（tangible）的方式去理解它们所处的世界。尽管研究难度巨大，但她对未来的进展保持非常乐观（bullish）的态度。 为了说明人工智能技术迭代的速度往往远超人类预测，Neville 分享了自己在 2000 年于 AT&T Labs 进行第一次实习时的真实经历。当时在午餐时间，她和许多著名的机器学习界学术泰斗一边下围棋一边讨论 AI 何时能攻克这项游戏。因为 19×19 的大棋盘太难，她当时还在 9×9 的小棋盘上学习规则，并观察自己是如何学习的。当时周围包括诸多大牛在内的研究人员纷纷预测，让 AI 下围棋起码要花 50 年甚至 100 年的时间。然而，围棋早已被人工智能彻底攻克。Neville 感叹道，虽然研究人员在面对当前模型瓶颈时容易悲观地认为“这又要花上 10 年或 25 年”，但在如此多的人才投入和巨大的计算规模推动下，突破的到来将远远快于 2000 年时任何人的预料。 - 科学探索核心问题：外层 Agent 包装和推理机制究竟能多大程度修补 Transformer 的本质局限，何时需要引入颠覆性的底层架构。 - 历史对照：2000 年 AT&T Labs 专家曾断言 AI 攻克围棋需要 50 到 100 年，如今该问题已成过去式。 - 演进速度：得益于全球研发人员密度的激增和算力集群规模的扩大，AI 跨越瓶颈的速度远超传统经验。 --- ### `33:00` 科幻愿景：人类与 AI 组队协同与复杂系统创新 展望未来 10 到 20 年自己希望在科研领域留下的印记，Neville 指出，“复杂系统（complex systems）以及如何让 AI 和机器学习在复杂系统中发挥作用”贯穿了她整个职业生涯的核心脉络。目前她的团队重点正是探索人类与 AI 之间的多方协作（multi-human, multi-AI interaction）。 当谈到自己心中理想的、近乎科幻般的终极形态时，Jennifer Neville 阐述了自己的评判标准： > “If I think really in terms of sci-fi kind of outcomes, I think we'd be successful if in the work environment we start to see organizations of both humans and AI working together on teams to accomplish things and innovate.” 她坚信，如果未来能够真正建立起由人类与 AI 共同组成的混合型组织与团队，并在其中顺畅协作去完成任务、共同实现创新，那么她的科研工作就算取得了圆满的成功。 --- ### `34:20` 闪电问答：提问的勇气、死磕数据、终身成就与“带娃练模型” 访谈在尾声阶段进入了轻松快速的“闪电问答（Lightning Round）”环节，Neville 回顾了多段个人成长轨迹中的关键感悟与生活趣事： - **受用一生的建议（The best piece of advice）**： 在大学初涉计算机科学专业时，Neville 曾在参加 pre-coop 时感到手足无措，觉得周围所有人都懂而唯独自己什么都不知道。当时第一门计算机课的女助教（TA）对她说：永远不要害怕提问，很多人只是表面上看起来懂，其实大家心里也充满疑惑；只要有人勇敢提问，其他人其实会非常感激，因为他们心里往往想的是同样的问题。Jennifer Neville 将此条建议奉行至今： > “And it's very hard to, you know, you can be afraid and feel like you're going to appear stupid to ask the question. But undoubtedly, along the way, whenever I had these feelings of fear and, but had the courage anyways to ask the questions. I did find that people, were really appreciative.” 每次战胜恐惧提出问题后，课后或会后总会有人走过来说：“我刚才也是这么想的，太感谢你把这个问题问出来了。” - **吃尽苦头才学会的教训（Learned the hard way）**： “去看数据（Look at the data）。”在机器学习领域，研究人员往往习惯于只盯着基准测试（benchmarks）、测试集指标和跑出来的分数，一旦模型效果不好就下意识觉得是算法不行或者任务太难。但 Neville 多次吃亏后才领悟到，模型不工作时只要花时间去审查底层数据，往往就会发现数据本身存在错误、数据不符合预期，或是测试数据偏离了训练集的分布。即使她早已明白这个道理，在职业生涯中她依然“至少重新交了半打（六次）学费”才彻底记住这个教训。 - **最引以为傲的成就（Proudest accomplishment）**： 在抚养一个刚学会走路的幼儿（toddler）的同时成功拿到了终身教职（tenure）。由于她和同在高校任教的丈夫在入职教师岗位前夕迎来了儿子的出生，家庭与学术竞争的双重压力巨大。Neville 笑着总结道，不仅自己拿到了终身教职，儿子健康快乐地成长，而且她和丈夫“甚至成功保住了婚姻”，这绝对是人生中最了不起的成就。 - **艺术或自然对科研的启发（Art/Nature influence）**： 对自然界和生命体学习过程的观察始终深刻影响着她的 AI 研究。作为一个自嘲“极度极客（nerdy）”的母亲，在儿子还是婴儿、正在学习如何翻身时，Neville 竟然套用机器学习的概念，试图给儿子提供“正样本训练（positive training example）”——亲自动手摆弄儿子的腿来演示翻身动作，以此观察人类幼童的学习反馈，并将这些对于认知演进的现实观察反哺到算法的设计思考中。这种生命学习与机器算法之间的对照，贯穿了她的整个学术生涯。 ## Notable quotes > `05:00` 如果你从事的是 AI 系统方面的研究，我认为工业界确实才是最核心的阵地。 — Jennifer Neville > `05:00` 那种能够理解某种别人尚未理解的事物的感受——仅仅因为它正处于人类已知边界的最前沿——就像是一种毒品，这种对同一种感觉的追逐让我在研究领域坚守了这么久。 — Jennifer Neville ## Mentioned Jennifer Neville, Chad Atalla, 微软研究院 (Microsoft Research), 普渡大学 (Purdue University), 加州大学伯克利分校西蒙斯计算理论研究所 (Simons Institute at UC Berkeley), 美国国家科学基金会 (NSF), 美国国防部高级研究计划局 (DARPA), IEEE, ICDM, ICLR, AT&T Labs, Transformer ## Video description <p>Jennifer Neville did not want to go into computer science—but that’s exactly where she landed. Neville discusses the starts and stops that led to her professional sweet spot and her work identifying “surprising failures” making it hard for AI to handle complexity.</p><p><a href="https://www.microsoft.com/en-us/research/podcast/what-ai-gets-wrong-and-what-failure-teaches-us/">Show notes</a></p> [Source](https://media.blubrry.com/microsoftresearch/content.blubrry.com/microsoftresearch/MRP-2026-10-29-318.mp3)

[5:00] Jennifer Neville: 如果你从事的是 AI 系统方面的研究，我认为工业界确实才是最核心的阵地。 [5:00] Jennifer Neville: 那种能够理解某种别人尚未理解的事物的感受——仅仅因为它正处于人类已知边界的最前沿——就像是一种毒品，这种对同一种感觉的追逐让我在研究领域坚守了这么久。

[10:00] Jennifer Neville: The reason we focus on evaluation is that we find that the standard way that ML and AI people evaluate are through benchmarks that are fairly simple with respect to how people would actually use them in practice. [28:44] Jennifer Neville: I think we'd be successful if in the work environment we start to see organizations of both humans and AI working together on teams to accomplish things and innovate. [7:30] Jennifer Neville: Right now, if you're doing work on AI systems, I think that industry is really the place to be.

[0:00] There were times that I felt like I was spinning my wheels, things weren't working, I would get kind of dejected, and then we'd get to a point where we did learn something, and it was just the emotional thrill of that was really what hooked me, the being able to kind of understand something that no one else would. understood yet because it's just at the frontier of what we know about things. This is the Microsoft Research Podcast, where Microsoft researchers, driving advancement through fundamental science and technology research, explore the who, how, and what's next in computing and AI. Hello and welcome. I'm Chad Attulla, an applied scientist here at Microsoft Research. Today, I'm joined by Jennifer Neville, a Partner Research Manager at Microsoft Research, and the Samuel D. Conte Chair Professor of Computer Science and Statistics at Purdue. Her research examines machine learning and AI for interactive domains and structured data, looking at how the data points that AI systems are trained on affect their behaviors and how that aligns with what users actually want. Across that research, she's published more than 130 papers with over 10,000 citations and received honors such as a National Science Foundation Career Award, a spot on IEEE's "Tend to Watch in AI" list, and best paper awards from the International Conference on Data Mining and the International Conference on Learning Representations. What amazes me most about Jen's work is her foresight and ability to have a through line that stays consistent as the field of AI develops. And I'm excited to hear more about how she got to where she is today. Jen, thank you for joining me. Thanks for having me. Awesome. Well, I would love to learn about how you got here today. And let's rewind way back. When I was a kid, I wanted to be an astrophysicist. But of course, here I am with a computer science background. When did you know that you wanted to go into computer science? to do anything but what my dad did. So when I went to college, first I measured in math, and then in physics, but I really just couldn't vibe with those majors. So I dropped out of college for a while. And when I went back to college again, I majored in cognitive science instead, which also was too squishy for me, didn't have enough math in it. At that point, if somebody had told me, AI is the thing you should be doing because it combines the cognitive science with the math and computational thinking. I think I would have saved myself a lot of time, but that didn't happen. And so I worked for a while. And then when I went back to school, I decided to major in computer science because what I wanted to do was think about working with data and dealing with data. By happenstance, I didn't even really realize that it was part of computer science. Awesome. Well, it's one thing to go into computer science and want to work on data or AI, but And another thing to want to be involved in research on that front. So what sorts of questions or big ideas sparked your research drive? Yeah, that's actually an interesting story as well. I only got into research because I was in the honors program in my computer science degree. And as part of the honors program, you have to do a research project. It's mandatory. And so when I talked to professors about what their research project should be, they actually said, well, you have to do it. decide what topic you want to work on. And at that point, I was interested in data, and I was interested in AI, and I thought a lot, I did a lot of reading. And what I decided that I wanted to investigate was how to do data mining on web data that was interconnected. And so that when I decided that was the question I wanted to work on, I got pointed to a particular faculty member working in this nascent field at the time that was called statistical relational learning. And we did my project in that. I published a paper at a workshop and I was kind of hooked. And so I hadn't planned to go on to grad school, but that experience made me want to go into grad school and continue. Nice. What part of it do you want to do? [5:00] think hooked you? Was it like the thrill of doing the research? Was it the environment of the conference and what academic publishing looks like? It was really the... The... the thrill and the process of doing research. So it was a long project. There were times that I felt like I was spinning my wheels, things weren't working, I would get kind of dejected, but my advisor would be kind of like, you know, supportive and positive, saying, no, keep going, we're learning something. And then we'd get to a point where we did learn something. And it was just the emotional thrill of that was, that was really what hooked me. The, the being able to kind of understand something that no one else knows, understood yet because it's just at the frontier of what we know about things was, it's just, it's like a drug almost, right? So it's kept me in research for this long, that same chasing after that same feeling. Gotcha. Well, love it. Yeah, you joined Microsoft in 2021 from academia. And in fact, you're still a professor. And you've been teaching and advising for 20 years. What motivated the shift to part of your professional life being research in industry? And how would you characterize the difference between industry research and academia research? Yeah. My research kind of spans the spectrum from theory to application. When I did my first sabbatical after I got tenure, A lot of colleagues that I had that were at the same point in their career went off into industry labs for their sabbatical and never came back to Purdue. I thought about which direction I'd want to go, either more theory or more applied. And I thought at that point in my career, it'd be better to explore the theory side because then I might actually go back to Purdue. And so I went to the Simons Institute in Berkeley for my sabbatical. And that was very theoretical. a great experience, but it was kind of seamless to transition back to academia. On my second sabbatical, I went the other way, which was to come to MSR and do a sabbatical here. And of course, being able to see how algorithms and theory kind of hit where the rubber hits the road with respect to how they behave in practice and real systems and with real users and real data, that's hard to So I think the difference between research and academia and industry is... depends on really is affected by the target of where where you're aiming the research and so I think fundamentally it feels very similar the questions you would ask in academia and industry but the in industry your ability to apply it at scale in real systems on real data is really very different from academia. And in academia, I think you end up asking questions that are more abstractions that cover applications across a lot of different domains at once. And that's how you get funding from places like DARPA and NSF. But in industry, it's a little easier to just, you know, actually get your hands dirty and And then your kind of target is the products and the company's interests. that sort of changes maybe the types of questions that you would ask or, or investigate. So I think it's actually great to be in both places, right? Like there's lots of synergies across the two. And I think there's lots of opportunities to go, uh, to industry and come back to academia or to go start an industry and go to academia. But right now, if you're doing work on AI systems, I think that industry is really the place to be. Gotcha. Yeah, perhaps useful advice to folks who are deciding which way to take that decision in their career right now. I've been here at Microsoft for a little over six years and have been working on AI evaluation for much of that time. And there are, of course, a number of daunting fundamental questions and challenges in the space. And I initially came across your work and became aware of your work because... part of your work proposes some solutions to some of these problems and helps to point out these problems. And my immediate reaction was relief that we have someone wonderful like you working on these things and you're going to take care of it for us. [10:00] And I really appreciate how you balance critiquing AI evaluation and also providing solutions along the way. But I understand that that's just a small sliver of your broader research agenda. And so how would you describe the current research charter for the team that you're leading here? Our team is called the AI Interaction and Learning Team. What we really focus on is trying to push the frontier behavior of these AI systems in realistic work environments. And so what that entails is studying how, where's the boundary of the performance of the current systems? How do users experience that in practice with real, real workflows? And then how do we improve that and push the performance? of the models for these kind of technologies tasks, complex tasks that users are working on. The reason we focus on evaluation is that we find that the standard way that ML and AI people evaluate are through benchmarks that are fairly simple with respect to how people would actually use them in practice. And so we find that the what do we want out of these systems and design practical evaluations to put the systems in those kind of environments. So that means things like multi-turn behavior, collaborative environments, long horizon tasks that users are doing. Do we need to improve these systems? So we start with evaluation, but ultimately what we're trying to do is develop better algorithms, models, estimation methods to push the performance of the models. Yeah. It's a gateway to understanding and therefore being able to push the boundary and improve these systems. So you mentioned multi-turn interactions and collaborative environments. You have a couple of recent papers that examine these things specifically, like how LLMs may get lost in multi-turn conversations or how they may corrupt documents in these sort of agentic knowledge work scenarios. Can you tell me a little bit more about those research projects specifically? Sure. And those are examples of cases where we had to sort of have innovative insights as to how to set up the evaluation to really be able to control and test the model's behavior in those scenarios. often underspecify what they're trying to do. They don't know how to give a complete specification to begin with. We sometimes forget this as computer scientists and software engineers, but lay users might not be able to fully specify things in the first turn. And so they kind of figure out what they're doing over multiple turns and sort of add conditions and clarifications public, and we changed the fully specified single complex turn of instructions and had models that simulated users providing that clarification over multiple turns, and then studied how do the models, how do all of the current models that we have, how do they behave when you get this kind of task specified over multiple turns. And we found that the performance that we see in single turn which is very high because, of course, we're optimizing to that as model builders, actually degrades significantly over multiple turns. And so that was something that was a fairly surprising finding because nobody had evaluated in that way before. But when we released that research into the world, it was a fairly surprising finding. the users of the systems all resonated with, well, that's been my experience. I knew that that was happening. How do we fix that? And one of the, so we're working on... reinforcement learning methods to actually help the models learn better in these environments to kind of fix that behavior. But a practical solution that we talked about in the paper is if you end up having the model kind of get confused, if you specified something over multiple turns, just like stop the [15:00] chat, erase everything, go back, and now, given what you know, give it a fully specified single turn, and then it will behave better in that kind of interaction. So maybe that points out that although fundamentally we're looking for algorithmic improvements to the models, sometimes in these projects, we end up coming up with insights of how users could change their behavior to get better outcomes from the models like where they are right now with their performance. Yeah. Well, we noted that one of the themes of your research is how we can improve these systems to bring them better into alignment with what users actually want. And you mentioned a case here of, hey, well, we know that users realistically are often not fully specifying what they want in their first turn. And we do have these long multi-turn interactions. And after you put this paper out, they expressed resonance with this frustration about getting lost in multi-turn conversations. How do you think about understanding the user's wants and needs and experience? And how does that relate back to what you do with as you said simulating a user for example to run these longer multi evaluations Yeah that getting to see at scale what users are trying to do and what kind of failures and successes they have in the systems is one of the advantages I think you get if you're in an industry lab and you get to see data at scale like that. So we have done a lot of work with the product groups here internally at Microsoft, analyzing consumer logs at scale to figure out what are the main patterns of failures that users are experiencing. And that provides the kind of insight or motivation for a lot of the work that we do. And there's a lot of great product groups that are also analyzing the successes that users have, which would allow you to then even recommend to users what are the kind of tasks that they're going to be able to do successfully in the systems that we have now. I think if you think back when search engines first started people had to learn how to interact with search engines in a way to effectively get the information that they looking for We might have forgotten that that happened now because it was so much a part of what everybody would do before all these AI systems came out But now I think there's a transition away from the search engines to these chat based systems. And I think there's going to have to be some learning from the users as well in this new environment. And I think analyzing the things that are successful in terms of what users are doing is a great way to start recommending and tutoring or teaching the users in the same way. I love how you're pointing out the utility of looking at real data. But what does that actually mean? Are you genuinely looking at real data? How is privacy coming in here? What sort of processes do you have for responsibly leveraging data in the wild? Well, yeah, that's a good point. Thanks for bringing it up. We're not actually looking at the data. We are analyzing the data at scale to extract patterns of failures or successes in a privacy kind of way There also a lot of restrictions that we can actually see the data eyes on We can only sort of push methods to process the data in very restricted environments. And then these sort of higher level insights that are privacy preserved are the things that then fuel like what we would do algorithmically later on. So just to be clear, we're not fine tuning on your data or your responses, but the more details you can give in your responses, whether it's through donated information when you give a thumbs down and you give a description of it, or if it's in the sort of chat interaction you have with the models, those can eventually get distilled into things that are going to improve the models, not through people actually looking at what you've been doing. And you noted this shift from a search being the dominant modality to these multi-turn chat interactions. But now we're also seeing agentic and collaborative [20:00] knowledge work style tasks becoming even more relevant. I noted that you did some work on that front as well. Any other interesting takeaways that you'd love to share with our listeners? Yeah, so we have both some theory work that tries to characterize the types of tasks that are fundamentally hard for transformers to compute inside the model, as well as more simulation-based at complex tasks that are conducted over long horizon workflows where you take, for example, documents and you do repeated edits over those documents. In those environments, the complexity of the task for the models or even the agents is to track not only what the user's intending to do but also understanding what's the current state, what information is relevant, what is not relevant, how things have changed. And those are things that agents are starting to be fairly good at. But there's particular kinds of tasks that are easier than others. And so something that we look at is what are the types of tasks, where complexities become too much for the current agents, I think that's a good thing. tool usage with the agents. Is it better to bring a human in the loop? How can we know that things have gone awry? So even having the agents kind of monitor themselves and assess whether they answered something correctly or they should kind of roll back to a previous state. I think those are all open questions right now. But we do have some work showing that, again, We get longer and longer into these workflows because errors, if you don't catch them, can accumulate and start to confuse the AI even more as they have to carry out the task over repeated interactions. Yeah. That phrase surprising failures is interesting to me. It implies an expectation of, hey, this should do better. And I'm surprised that it failed in this specific way. you see more of these surprising errors or are a lot of the errors like expected, you would understand that it would fail in this way or that? Yeah, I think there definitely are expected failures because those, well, there's fewer expected failures, of course, as we work to make the systems better and better. If we can expect them, then we can fix them. that have been characterized for a long time in the community and that there are specific benchmarks and methods to try to reduce those things. incorrect answer to a math problem or a hallucinated case in a law, you know, a legal opinion, but are this sort of subtle loss of semantic content in the documents. And I think that if I look back on what we've seen, sort of investigated with the team, we have a kind of hypothesis that that we think as humans, the things that are hard for us to do, are going to be the things that are hard for the model to do. And in fact, That's not often the case because we have to think about what's hard for the transformers actually to compute, what's hard for their retrieval systems to retrieve from the underlying content. should be done correctly. But in fact, that's not necessarily the case with LLMs. And so the surprising kind of creeps in, I think, with our own expectations based on human intelligence about what's hard or what's easy. Yeah. And so between that, confusion that may exist for users based on what [25:00] They expect these systems to be good on perhaps based on reflecting on human intelligence and all of the... hype and marketing that exists out there. What do you want? real users to take away from this? Or how would you caution them to think about the capabilities and expectations that they may have for AI systems, given your work here? How can they kind of cut through that noise and build better expectations? Oh, that's a good question. I think that... I think the current AI systems have a lot of capabilities that are going to be very useful to people in work environments right now. I think that the takeaway from our work is that you shouldn't expect them to be 100% successful across the board. You should be checking the answers that you get back. And if you get back something that you think is incorrect, you shouldn't necessarily assume that the model can't do that at all. But you should think about asking again, or asking in a different way. And you might get a better answer the next time. That is a good question. is hard to work into the way that we work right now, because I think that you can't, they're not ready for things to be 100% delegated to them. But if you can figure out how to use them in a workflow with you with oversight and verification, I think it can, they can be very useful to do things, improve, you know, productivity and the speed with which you get things done. because we're developing these systems in real time as they're being released. And so your use helps us actually push the boundary of the models because it gives us examples of things that can and can't be done. Maybe another thing I would like to say is that something that maybe users don't understand is that in the past with search-based systems or recommender systems, the kind of feedback that we were able to give as users was just kind of like thumbs up, thumbs down. But now we're in an environment where actually we could give feedback with much higher fidelity that can be super useful to improving the models downstream. So if you're working in kind of a chat environment with a model, understand that if something has gone badly, it's actually helpful to say how it's gone badly and in your textual interaction or speech interaction, to really convey exactly what went wrong and what you expected and what you got instead, because that actually will be used as we update the models. And so you can think of that as users, your role can be to train the models to do the things that you would have wanted to do, but it couldn't do right now. Awesome. Interesting call to action there. And, I'm just curious, you talked about the transformer paradigm that we're in now for these sorts of large language systems. Where do you see the science of large AI systems going next? Broadly, maybe that's a really hard question, but At least in your interests and research directions, where do you see the science of science? these systems going next? That's a big question. Thank you. I think where the research community is simultaneously trying to figure out if the transformer architecture is the right underlying model to use, because there are certain known limitations of how computation can work in transformers. A lot of those limitations can be solved by the... the wrapper around the models, the agentic harnesses that we have around the models right now, and the reasoning that goes on on the back end. So I think an open question is... How much of... Thank you. Can we deal with the limitations of the transformer architecture with this sort of wrapper foundation around it? And what needs to be addressed with sort of a fundamentally different architecture underlying things? Thank you. wrappers around the current model. [30:00] that we have. I think that we will see in the next generation of work that we will have to focus on longer term interactions with the models and having the models understand the world within which they're working in a much more tangible way than they're doing right now. But I think I'm very bullish on the research that's happening. I think we will continue. you know, be successful at this. Maybe going back to when I started, my first internship was at AT&T Labs back in the year 2000. And I remember very clearly sitting at lunch in my internship, playing Go and talking about how we could get, you know, AI models to be able to, you know, play this game and how it was harder than chess and what could we do. And I was learning how to Yeah. And I was learning on a nine by nine board. I don't know if you've ever learned how to play Go, but you don't learn how to play on the big board because it's too hard even for humans. Right. So you start on this nine by nine board. I was kind of watching myself learn how to play and think about how the algorithms would learn how to play. And while we were doing that, we were, you know, sort of. conjecturing about how long it was going to take AI to be able to do this. And so it's funny looking back because, you know, at AT&T Labs, a lot of the sort of luminaries in, you know, machine learning and AI research were there. People were saying, oh, it's going to take 50 years to be able to do this. You know, some people say 100 years. But of course now that's already a solved problem. And so looking forward, there are things that I – might think as a researcher because it's so hard for us to get past this sort of boundaries we see right now in behavior. there's a tendency to think, oh, it's going to take 10 years, it's going to take 25 years. But I think given the pace of research and the number of people researching, you know, doing research in this field and the scale at which these models are running, I think it's actually going to happen much more quickly than I would have thought as that person, you know, that new AI student in the year 2000. So, wow. Well, in the spirit of that... Looking back, I'm going to be a little bit more. to the year 2000. Let's imagine now jumping 20 years forward in the future, or just 10, because things are moving so fast. What mark would you like to have left on research in this space? I think the... The things that have really been the sort of common thread through my research career is thinking about complex systems and how to get AI and machine learning to work in these complex systems. So and even right now, that's what we focus on with collaboration. And we're thinking about multi-human, multi-AI interaction. If I think really in terms of sci-fi kind of outcomes, I think we'd be successful if in the work environment we start to see organizations. of both humans and AI working together on teams to accomplish things and innovate. And if we can get that working correctly, then I will feel like my research has been successful. Awesome. Well, as we wrap up here, I thought it could be fun to do a lightning round, which means quick questions, quick, low stakes answers, just whatever first comes to your mind. Sound good? Sure. Awesome. What's one piece of advice that stuck with you or changed your perspective? Ah, OK. You said short answers, though. Yeah. When I first started in computer science, I actually doing a pre-coup was a TA in my first computer science class. And... She said to me when I was sort of being overwhelmed and feeling like I didn't, everybody else knew everything and I didn't know anything. She said, you know, don't be afraid to ask questions because people don't. It might appear that they know what's going on, but in fact, they don't know what's going on. And they will actually really appreciate that you're courageous enough to ask a question because they're probably thinking the same thing. And it would help both you and them if you will ask the question. And so I've taken that to heart over the course of my career. [35:00] And it's very hard to, you know, you can be afraid and feel like you're going to appear stupid to ask the question. But undoubtedly, along the way, whenever I had... these feelings of fear and, but had the courage anyways to ask the questions. I did find that people, We're really appreciative. And, you know, people would say afterwards, oh, I was thinking the same thing. I'm so happy that you asked that question. So be courageous and ask the question. Thank you. What's a lesson that you learned the hard way? Look at the data. Look at the data. So in machine learning, we have a tendency, this applies to the benchmarks, we have a test set, we run our model, we learn it, we apply it to the test set, we generally get a number that's our metric that we're trying to push. problem's too hard. But I have learned the hard way multiple times that when things are not working out, if you take the time to look at the data, you might actually find that there's errors in the data. The data is not what you expected. The data is different from the distribution that you train the model on. And so anytime, I guess maybe that makes me a data person, but often when things are understand. Yeah, makes sense. It's a good one. It's a hard one. Hard to remember. And maybe that's an important thing is even though I've learned that lesson, I've had to relearn that lesson at least half a dozen times throughout my career. Yeah. Well, on the flip side, what's an accomplishment that you're most proud of? I think I would say the fact that I was able to get tenure and raise a toddler at the same time. I had my son right before we started. My husband's also faculty, so before we started our faculty positions. And so I think the fact that I was... Able to get tenure. My son was healthy and happy and I managed to stay married. It was probably the biggest accomplishment. Big accomplishment, yeah. And last question here is, What's one way that art or nature has influenced your work? Hmm. I'm not sure if this would really count as nature, but one thing that I've always been observing as an AI person is how Kids, people, animals, insects seem to learn about the world and the environment and change their behavior. And so I fully admit that I was very nerdy as a person. parent with a young child, I would take, you know, ideas that we have for machine learning and try to see if I could help my son learn better. For example, if he's learning to try to like flip over when he was an infant, I'd say, okay, let me give you a positive training example. Like, take your legs and do this and then see the effect on, you know, how he learned, but also you know, might be worked into the algorithms and models that we developed. That's probably the biggest interaction that I've like sort of continually returned to over the course of my career. That's beautiful. Well, Jen, thank you for sharing your story and your insights. It's been a pleasure having you here. And to our audience, thank you for tuning in. If you would like to learn more about the work of my colleagues here at Microsoft, check out the Microsoft Research page at aka.ms slash research or check out the other episodes of this podcast. Thanks. you