Amazon Scientist Links AI Chatbot Hallucinations to Logic Failures

亚马逊科学家将AI聊天机器人幻觉归因于逻辑缺陷 亚马逊研究表明，企业聊天机器人的错误主要源于上下文误解以及检索文档中的逻辑缺陷，而非随机捏造。 Udita Patel 提出了一套专为企业级检索增强生成（RAG）应用定制的全局评估框架，超越了基础的准确率指标。该方法通过将度量锚定在用户查询、检索来源和生成响应这三要素上，能够诊断复杂的管线故障，例如重复回答以及由逻辑推理错误引发的幻觉。亚马逊的实际案例研究展示了分析各项指标间的相互作用如何为生产监控、知识库文档更新以及管线升级的离线护栏提供支持。 针对企业级 RAG 架构黑盒化与缺乏标注基准的问题，评估指标应聚焦于由用户提问、检索文档和生成回答组成的核心数据三元组。 (0:04) 该评估体系涵盖生产监控、知识库优化及离线评估三大场景，由于企业知识库频繁更新，系统采用基于大模型的匹配机制以应对人工标注无法扩展的挑战。 (5:27) 企业大模型的幻觉问题往往源于复杂逻辑推理失误与上下文误读，而非单纯凭空捏造事实。 (21:59) 通过交叉分析指标组合可推导出 30 至 40 种诊断结论；例如高检索精确度配合低回答自相异性，通常表明系统检索到了近似重复的内容并生成了冗余回答。 (22:57)

Udita Patel, a Senior Applied Scientist at Amazon, presents a holistic evaluation framework for enterprise Retrieval-Augmented Generation (RAG) systems that replaces static accuracy benchmarks with multi-metric diagnostic interplay. The approach addresses the reality that enterprise ground truth cannot scale alongside constantly shifting documentation, instead evaluating the relationships between user queries, retrieved sources, and generated outputs. Triplets and the Limits of Ground Truth (0:04) Building enterprise RAG pipelines—such as internal IT helpdesks, onboarding bots, and customer service tools—presents operational hurdles across product, engineering, and science teams. Large language models operate with unpredictable behaviors, retraining them directly is cost-prohibitive, and traditional user feedback is predominantly anecdotal. Furthermore, Patel notes that static human-annotated ground-truth datasets quickly become obsolete because corporate documentation updates constantly, such as when return policies adjust from 30 to 45 days. To establish reliable evaluation without depending on brittle intermediate modules, the framework evaluates the core data triplet: the user query, retrieved sources, and generated response. This triplet architecture is deployed across three operational workflows: real-time production monitoring with bot-specific thresholds, offline benchmarking for pre-deployment changes, and content-level auditing to identify gaps in enterprise source material. Logic Failures Rather Than Synthetic Hallucinations (21:59) Addressing model groundedness, Patel highlights an empirical finding: enterprise LLM hallucinations rarely manifest as completely invented numbers or fabricated facts out of thin air. Instead, hallucinations predominantly arise from logical reasoning failures when processing complex context. In an illustrative e-commerce case study where a baseline policy specifies a 30-day return window alongside a conditional 45-day exception for incorrect shipments, the model does not invent an arbitrary 60-day window. Rather, it errs by applying the conditional 45-day exception to standard customer return inquiries. Patel emphasizes that enterprise hallucination evaluation must therefore detect context misapplication and logical conflation rather than merely flagging ungrounded vocabulary. Diagnosing Pipeline Bottlenecks via Metric Interplay (22:57) Rather than evaluating metrics in isolation, the framework analyzes pairs and triplets of metrics to yield 30 to 40 distinct diagnostic conclusions. This diagnostic capacity was demonstrated when business analytics revealed that repetitive chatbot responses accounted for 34% of negative user survey feedback. Evaluating production metrics revealed that while Source Precision stayed high above the standard 0.8 threshold, Response Self-Distinctness fell below 0.8, and Source Query Coverage (SQC) fluctuated at low-to-medium levels. This specific combination indicated that the retriever was pulling near-duplicate documents covering one aspect of a multi-part query (such as the return window) while missing others (such as restocking fees). The repetition was further compounded by mandatory summarization prompts on short answers. In response, engineering teams introduced post-retrieval semantic clustering to deduplicate fetched context, worked with documentation teams to merge redundant source articles, and restricted summarization prompts strictly to longer generations. Automating Content Audits and Upgrade Guardrails (26:42) The framework also repurposes user interactions to audit source documentation health. When Source Precision, Groundedness, and Source Query Coverage all drop below the 0.8 threshold simultaneously on high-traffic articles, the root cause is diagnosed as missing or defective source content rather than algorithmic failure. Forcing the LLM to navigate incomplete articles causes it to fabricate details to fulfill user queries. Consequently, content teams use these alerts to restructure articles, add explicit FAQs, and publish concrete policy examples. For offline evaluations, the framework acts as a deployment gatekeeper. In one instance, before swapping out an active vector embedding model, Amazon evaluated the proposed retriever against a full month of historical interaction data. Deployment was conditioned on maintaining all metrics above the 0.8 baseline while demonstrating measurable gains in Source Precision and SQC, ensuring retrieval upgrades did not produce unintended regressions. Patel concludes that as enterprise systems advance from standard RAG pipelines into complex multi-agent architectures, holistic evaluation frameworks must continuously evolve their metric interplay to ensure operational reliability without relying on static reference data.

亚马逊（Amazon）高级应用科学家 Udita Patel 提出，企业级检索增强生成（RAG）系统因业务知识频繁变更而难以依赖静态标准答案（Ground Truth），必须依托“用户查询、检索来源、生成回答”数据三元组与指标联动机制，实现超越单纯准确率的系统诊断与全生命周期评估。 评估困境与数据三元组架构 (0:04) 企业构建 RAG 对话机器人的门槛较低，但规模化运营面临诸多摩擦：模型具有黑盒特性且重训成本高昂，管线中包含复杂的前后置模块，而终端用户反馈零散且无法定位根因。更为关键的是，由于企业知识库（如退货时限政策）处于动态更新中，人工标注的基准测试集极易过时，导致生产环境缺乏大规模的标准真实答案。 为摆脱对脆弱内部模块及静态标注的依赖，该评估框架将度量机制锚定在核心数据三元组——即用户查询（User Query）、检索来源（Retrieved Sources）与生成回答（Generated Response）。系统引入大模型作为裁判（LLM-as-a-Judge）来量化各维度匹配关系，为跨团队协作提供统一的方向性指标。 逻辑推理缺陷引发的幻觉特征 (21:59) 在真实可靠性（Groundedness）的度量中，Patel 指出企业级 RAG 的幻觉极少表现为凭空捏造离谱数字或虚构事实，其本质多源于复杂逻辑推理与上下文理解的失效。 以电商退货政策为例，知识库规定基准退货期为 30 天，但在发错商品的特殊例外条款下可延长至 45 天。大语言模型在回答常规退货咨询时，并未凭空制造 60 天或 90 天，而是错误套用了 45 天的例外条款。这表明评估重点必须转向对多重业务约束及条件分支的逻辑推理检测，而非单纯依赖关键词匹配。 指标联动诊断与检索冗余治理 (22:57) 单一评估指标往往无法定位具体架构缺陷，其核心诊断能力建立在多指标组合的动态分析上；不同指标组合可推导出 30 至 40 种具体的管线故障结论。在亚马逊的一项线上监控案例中，调查显示 34% 的用户负面反馈直接源于答案内容重复。 指标联动分析显示，系统来源精准度（Source Precision）处于高位，但回答独特性（Response Self-Distinctness）跌破 0.8 的门槛，同时来源查询覆盖度（SQC）偏低。这诊断出检索器拉取了多篇高度近似的近重复文档（如反复拉取不同版本的退货时效说明），占满了召回配额，导致重新上架费（Restocking Fee）等关键子问题未被检索覆盖。团队据此在检索后引入语义聚类去重，推动内容库排重，并仅在长文本生成时保留总结提示词，精准解决了冗余问题。 知识库缺陷治理与离线升级验证 (26:42) 该框架同时赋能知识库维护与离线模型升级。在知识治理方面，内容维护团队以 0.8 作为全指标准入阈值，当高频访问文档对应的来源精准度、可靠性与 SQC 同时下跌时，系统直接将故障定性为底层文档缺失或表述模糊，迫使模型产生代偿性编造。内容团队据此针对性重写文档结构、增设 FAQ 与明确用例，无需修改模型即可修复回答缺陷。 在离线升级场景中，工程团队若要替换向量检索底层的嵌入模型（Embedding Model），需基于一个月的历史交互数据进行仿真管线测试。升级准则明确要求全线指标必须维持在 0.8 基线以上，且来源精准度与 SQC 必须录得显著提升，以此为生产环境部署提供量化的安全保障。 Patel 总结认为，随着企业级 RAG 及 AI 智能体架构向多阶段与复合逻辑演进，评估体系必须超越孤立打分，通过数据三元组与指标联动形成覆盖监控、内容运维及离线部署的闭环诊断能力。

## Background 本内容由 YouTube 频道 Ai4 于 2025 年 8 月 27 日发布，主讲人为亚马逊资深应用科学家 Udita Patel。视频围绕企业级 RAG 聊天机器人展开，探讨传统 NLP 评估指标的局限性、大语言模型诱发误差的监控方案，以及如何通过多维指标联动实现规模化评估与流水线持续优化。 ## TL;DR 分享针对企业级 RAG 系统落地后缺乏无标注评估手段的痛点，提出了基于“用户查询-检索来源-生成回复”核心三元组的原子级解构与 LLM-as-a-Judge 评测体系。演讲详细拆解了源精度、源查询覆盖率、事实扎根度及自身独立性等六大对抗平衡指标，展示了如何通过 30 至 40 种指标联动组合精准诊断回答重复、近似文档冗余等生产故障。此外，该框架不仅服务于实时监控，还能逆向反哺企业知识库的内容重构，并在离线评测中为嵌入模型等核心组件的无损替换提供量化准入标准。 ## Atmosphere & style 整场演讲呈现出严谨、克制且极具实战工业质感的学术工程风格。主讲人语调沉稳务实，摒弃了空洞的宏观概念，直接切入生产环境中“黑盒模型偶发报错”与“人工标注失效”的现实困境。伴随着清晰的因果链条拆解与量化公式推演，听者能直观感受到一线工业界顶级科学家在面对复杂工程涟漪效应时，那种抽丝剥茧、化繁为简的系统性架构思维。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=4s) 企业级 RAG 的核心诉求在于严格依赖内部知识库生成事实，彻底规避大模型预训练内生知识的发散，因此需要超越单一准确率的全面评估框架。 - [02:31](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=151s) 大模型黑盒特性与线上零散的用户反馈无法支撑系统演进，必须建立能对齐业务需求、量化多维报错的监控告警机制以指导工程资源分配。 - [04:25](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=265s) 评估系统依托“用户查询、检索来源、模型回复”核心三元组数据，闭环赋能生产实时监控、知识库反向优化以及离线效果评测三大业务场景。 - [07:11](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=431s) 针对生产环境无静态标准答案（Ground Truth）且知识库高频变更的痛点，框架采用无需人工标注的 LLM-as-a-Judge 范式执行统一匹配判定。 - [10:21](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=621s) 通过原子级解构机制将复合查询与回复拆解为独立的不可分割单元，实现细粒度误差诊断，并支持针对不同场景配置差异化的容忍阈值。 - [14:01](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=841s) 亚马逊构建了包含源精度、源查询覆盖率、回复精度、回复查询覆盖率、事实扎根度及自身独立性在内的六大对抗平衡指标体系。 - [19:20](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1160s) 自身独立性（Self-Distinctness）指标通过量化生成内容中的语义重合度与换词复述，有效治理大模型在 RAG 场景下的冗余车轱辘话问题。 - [21:59](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1319s) 企业级幻觉多表现为将特定例外规则混淆为通用规则的逻辑推理偏差，事实扎根度（Groundedness）用于确保生成事实严格被检索文档支撑。 - [22:57](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1377s) 通过多指标联动组合（Metrics Interplay）分析成对或三元组指标的升降异动，工程师能够精准反推定位 RAG 管线中的具体故障根因。 - [23:50](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1430s) 高源精度但低覆盖率与低独立性的指标组合，直接揭示了检索召回近似重复文档与 Prompt 强制摘要导致的回答冗余缺陷。 - [26:42](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1602s) 精确率、扎实度与源查询覆盖率全线走低标志着知识库存在内容真空，评测指标可作为雷达反哺内容运营团队针对性重写高频关键文档。 - [29:09](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1749s) 在替换底层 Embedding 或生成模型时，离线评测通过长周期历史数据回放检验涟漪效应，设定严格的指标准入红线保障安全上线。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=4s) 企业级 RAG 架构的兴起与超越“准确率”的评估诉求 作为亚马逊（Amazon）负责对话系统生成式 AI（GenAI）倡议的资深应用科学家，Udita 开门见山地抛出了本次分享的核心主题：在企业级聊天机器人落地过程中，如何建立一套超越单一“准确率”（Accuracy）的全面评估框架。检索增强生成（RAG）已经成为当今信息传递与知识问答的事实标准，广泛应用于客户服务机器人、员工入职培训助手、IT 运维支持系统，以及各类作为搜索引擎功能延伸的智能 AI 助手。 Udita 以一个最典型的企业交互场景拆解了 RAG 的工作流：用户提问“我买的电子产品退货期限是多久？”，系统不会直接将问题裸传给大语言模型（LLM），而是先检索连接到向量数据库的企业内部知识库，抽取出最相关的权威资料切片（例如退货政策条款）。紧接着，LLM 将检索到的上下文与用户查询结合，生成有凭有据的答案。对于企业级场景而言，系统的命脉在于确保 LLM 严格依据公司内部知识库回答，而不是凭借预训练期间记住的内生知识进行未经授权的发散。通常情况下，各个团队都能以极快的速度搭建出一个基础可用、看起来运转良好的 RAG Demo，但真正严峻的问题是：“接下来该往哪走？” - **RAG 典型企业应用**：客户服务机器人、员工 Onboarding 助手、企业内部 IT 服务台、新一代智能化搜索增强系统。 - **企业核心诉求**：严格限制 LLM 仅基于企业自有知识库生成事实，彻底规避模型自身预训练权重中的过时或不可信先验信息。 ### [02:31](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=151s) 从原型到生产：黑盒 LLM、零散反馈与评测框架的迫切性 当 Demo 完成并初见成效后，团队内部往往会迅速分化出不同角色的诉求与矛盾：开发人员和产品经理脑海中充满了功能扩展的构想，急于快速迭代并推向上线；科学家们则摩拳擦掌，希望在检索算法、嵌入表征（Embeddings）、提示词工程（Prompt Tuning）上开展大量实验，但他们极其缺乏一种能准确度量这些实验真实业务收益的基准标尺。 更棘手的挑战来自于技术底层与真实用户场景。大语言模型本质上是“黑盒”（Black boxes），经常带来不可预测的怪癖与偶发性错误。即便排查出了根本原因，企业通常采用的是昂贵的超大规模预训练模型，绝不可能为了个别 badcase 去重新预训练（Retrain）。与此同时，线上收集到的用户反馈往往是“零散且诉求单一的轶事”（Anecdotal），通常只是某个用户在某一次交互中遇到了不爽的体验；企业若要投入昂贵的工程力量去修复问题，必须保证投入能够换来规模化的回报。这意味着系统必须拥有能对齐产品真实需求的监控告警机制，一旦指标跌破红线即刻触发行动。由于人类对话本身是多维度的，RAG 的报错形式也是多维度的，单凭一两个孤立的统计指标根本无法支撑系统的持续演进。 ```mermaid flowchart TD accTitle: 复杂 RAG 系统的交互与评估流 subgraph InputStage["输入端"] Q["用户查询 (Query)"] end subgraph Pipeline["RAG 复杂处理流水线"] PreR["检索前预处理 (Pre-retrieval)"] Ret["向量检索 (Retrieval)"] PostR["检索后处理 (Post-retrieval / Rerank)"] PreG["生成前处理 (Pre-generation)"] Gen["LLM 事实生成 (Generation)"] PostG["生成后过滤 (Post-generation)"] end subgraph OutputStage["输出端"] S["检索上下文 (Sources)"] R["生成回复 (Response)"] end Q --> PreR --> Ret --> PostR --> S S --> PreG --> Gen --> PostG --> R ``` ### [04:25](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=265s) 核心三元组数据与三大落地闭环：生产监控、知识库反哺与离线评测 随着业务深入，企业级 RAG 架构会迅速膨胀：检索前后会增加 Query 重写、重排序（Rerank）等模块，生成前后也会挂载防御注入提示词、输出对齐过滤等复杂处理。在如此高频迭代且高度动态的复杂流水线中，评测系统唯一能够恒定依赖并持续获取的，只有数据交互的核心“三元组”（Triplets）： 1. **用户查询（User Query）**：用户输入的真实意图； 2. **检索来源（Retrieved Sources）**：系统检索出的参考文档片段； 3. **模型回复（Generated Response）**：LLM 最终呈现给用户的生成内容。 基于这三个基础元素，该评估体系已成功在亚马逊融入三大核心业务闭环： - **生产实时监控（Production Monitoring）**：针对不同产品设定个性化的阈值。评测指标能够精准下钻诊断出错误归因，从而指引工程师优先修复最具规模效益的高频核心故障。该监控能够直接覆盖全量线上流量或高比例流量采样。 - **知识库反哺与优化（Knowledge Base Improvements）**：除了技术管线本身，该框架被创新性地用于监控高流量知识源。例如退货政策页面被高频访问，若指标捕捉到该文档屡屡导致机器人产出错误回答，系统会定位出内容本身的歧义或漏洞，反向推动企业文档团队（Content team）重写并优化这些基础政策条目。 - **离线效果评测（Offline Evaluation）**：在任何技术组件发生变更前——无论是更换底层 Embedding 模型还是切换生成式 LLM——利用离线评测集全面扫描其在整个 RAG 管线中引起的“涟漪效应”（Ripple effects），确保综合表现不出现负向漂移后方可准入上线。 ### [07:11](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=431s) 无标注困境与解法：为何放弃标准答案并采用 LLM-as-a-Judge 构建这套框架首先必须直面一个现实挑战：在真实的生产流量中，不存在任何现成的“标准答案”（Ground Truth）。虽然算法团队可以在测试集或验证集上进行人工标注，但面对千万级的实时生产流量，人工打标既不现实也无法规模化。在典型的交互链路中，如果依赖静态金标，每次评估检索片段是否正确、回复内容是否达标，都需要人工基准或黄金参考答案作为对比。 然而，企业真实场景的知识库处于高频更新中。Udita 举例说明：如果某一天公司的电子产品退货期限突然从“30 天”调整为“45 天”，旧有的标准答案便在瞬间失效变质（Stale），导致历史的来源标签、回复标签和参考文本全盘作废。因此，强依赖人工标注的传统评测在动态生产环境中是完全走不通的死胡同。 为此，框架全面转向采用大模型作为裁判（LLM-as-a-Judge）的技术范式。尽管 LLM-as-a-Judge 伴随着一定的推断计算成本，但它具备无需静态真值标注、极高扩展性以及评测标准可随提示词快速迭代的优势。在该框架中，评测逻辑被高度抽象为一个统一的匹配判定函数： $$M(X, Y) \in \{0, 1\}$$ 其中 $X$ 与 $Y$ 分别取自用户查询、检索来源或模型生成的回复。大模型裁判会根据预设的语义规则执行判定，匹配成功输出 1，不匹配输出 0。 ### [10:21](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=621s) 细粒度误差诊断：原子级事实解构与差异化容忍阈值 单纯依靠粗粒度的“好/坏”二元评分或 0/1 结果对企业系统而言远远不够。粗糙的判定不仅会掩盖局部优势，还会抹杀部分致命的缺陷。企业需要的不是笼统的“机器人表现很差”或“机器人输出了无关信息”，而是具有高度可操作性的工程诊断：“让我们优化文本简练度，因为 25% 的回复中至少包含了 3 个用户根本没问的多余事实。” 为了实现这种颗粒度，该框架引入了**原子级事实解构机制（Decomposition into atomic units）**。无论是查询、来源还是回复，都会通过 Prompt 模块先拆解为不可分割的原子单元： - **查询解构（Atomic Queries）**：例如将复合查询“退货期限是多久？退款需要多久到账？”精准切分为独立的原子子问题 Q1（退货期限）与 Q2（退款到账时间）。 - **回复解构（Atomic Facts）**：将生成的多句长文本拆解为各自仅包含单点信息的独立原子事实（如 R1, R2, R3, R4）。 通过对原子单元进行评估，诊断能力直接从“整体好坏”精细化为“事实重复”、“插入非必要的多余冗余信息”或“存在事实错误”。此外，框架允许针对不同业务场景定制告警阈值（Configurable Thresholds）。某些机器人定位是快捷问答，要求高度简练；而有些场景注重品牌亲和力，容忍适度详尽冗长；又或者有些机器人背后有真人坐席托底，能容忍回复不完全回答用户所有问题，而闭环自动化机器人则要求必须穷尽解决用户的所有子诉求。系统支持在单次交互粒度、周期聚合粒度以及全量交互上自由配置差异化的通过阈值。 ```mermaid flowchart TD accTitle: 基于三元组原子解构的 RAG 评估框架图 subgraph DecompStage["原子解构层 (Prompt-based Decomposition)"] Q["复合查询 (Query)"] --> AQ["原子子查询<br/>(Atomic Queries: Q1, Q2...)"] S["检索文档 (Sources)"] --> AS["原子文档切片<br/>(Atomic Sources: S1, S2...)"] R["生成回复 (Response)"] --> AR["原子事实切片<br/>(Atomic Facts: R1, R2...)"] end subgraph MatchingStage["判决匹配层 (Match Function: 0 或 1)"] direction TB M1["源精度 (Source Precision)<br/>Sources ➔ Query"] M2["源查询覆盖率 (Source Query Coverage)<br/>Query ➔ Sources"] M3["回复查询覆盖率 (Response Query Coverage)<br/>Query ➔ Response"] M4["事实扎根度 (Groundedness)<br/>Response ➔ Sources"] M5["自身独立性 (Self-Distinctness)<br/>Response ➔ Response"] end AQ -. 匹配判定.-> M1 AS -. 匹配判定.-> M1 AQ -. 匹配判定.-> M2 AS -. 匹配判定.-> M2 AQ -. 匹配判定.-> M3 AR -. 匹配判定.-> M3 AR -. 事实归因.-> M4 AS -. 支撑校验.-> M4 AR -. 冗余去重.-> M5 ``` ### [14:01](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=841s) 整体框架全貌：六大核心指标与类 Precision/Recall 的对抗平衡 很多团队在调优 RAG 流程时经常陷入“按下葫芦起了瓢”的困境：单独优化了检索模块，生成的文本质量却大幅劣化。Udita 强调，机器人的实际体验取决于各模块协作的综合水准。为此，亚马逊设计了包含 6 个独立但相互制约的指标体系，以模拟传统机器学习中经典的查准率（Precision）与查全率（Recall）的动态博弈（Tension metrics）： | 指标名称 | 评估流向（X $\to$ Y） | 核心定义与业务直觉 | |:--- |:--- |:--- | | **源精度 (Source Precision)** | Sources $\to$ Query | 检索到的文档片段中，真正解答用户问题所需的有效片段占比，排除仅有字面相似的无用片段。 | | **源查询覆盖率 (Source Query Coverage)** | Query $\to$ Sources | 用户的各个原子子查询中，有多少比例能从检索到的文档来源中找到对应答案。 | | **回复精度 (Response Precision)** | Response $\to$ Query | 模型生成的各个原子事实中，有多少是针对用户提问所必需的核心事实，压制无关废话。 | | **回复查询覆盖率 (Response Query Coverage)**| Query $\to$ Response | 用户的原子子查询中，有多少比例在模型最终生成的回复中得到了切实的解答。 | | **事实扎根度 (Groundedness)** | Response $\to$ Sources | 最终回复中的原子事实是否严格来源于检索文档，是衡量与杜绝大模型幻觉的生命线。 | | **自身独立性 (Self-Distinctness)** | Response $\to$ Response | 最终回复内部是否存在信息的换词复述、冗余总结或语义车轱辘话，量化文本的非冗余度。 | 在图解这套架构时，Udita 强调了方向箭头的含义：在源精度（Source Precision）中，系统拿着“检索来源”去对比“用户查询”，检验来源对于回答问题是否是必需的；而在源查询覆盖率（Source Query Coverage）中，则是拿着“用户子查询”去比对“检索来源”，检验用户的所有诉求是否在文档里得到了覆盖。 ### [17:10](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1030s) 指标实战拆解：源精度与回复查询覆盖率计算 Udita 结合具体案例详细推演了这套指标的判定过程。 **案例一：源精度（Source Precision）计算** - **用户查询**：“线上购买的电子产品退货期限是多久？有无重新入库费（Restocking fee）？退款多长时间能到账？” - **检索结果**： - 文档 A 为公司的《退货政策》正文； - 文档 B 是一份提及了“电子产品促销”和部分退货字眼的内部《新闻通讯》（Newsletter）。 - **判定过程**：匹配函数分析发现，文档 A 切实解答了问题，得分 1；文档 B 虽然在字面上与 Query 高度重合（TF-IDF 或向量相似度高），但完全没有回答任何实质问题，判定为 0。 - **得分**：检索返回 2 篇文档，有效命中 1 篇，最终源精度（Source Precision）为 $1 / 2 = 0.5$。 **案例二：回复查询覆盖率（Response Query Coverage）计算** - **用户复合查询**：包含两个原子子问题，即 Q1（退货期限多长）与 Q2（退款多久到账）。 - **模型生成的回复**：“线上购买的电子产品可在收货后 30 天内退货。” - **判定过程**：由于是在测量查询覆盖率，系统将 Q1 与 Q2 分别与生成回复匹配。Q1 得到了明确回应，匹配分值为 1；Q2 完全被遗漏未答，匹配分值为 0。 - **得分**：回复查询覆盖率（Response Query Coverage）为 $(1 + 0) / 2 = 0.5$。这一细粒度指标能立刻向系统警示：模型遗漏了用户的一半诉求。 ### [19:20](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1160s) 幻觉与冗余治理：事实扎根度与自身独立性评估 在演示的最后部分，Udita 重点探讨了最令企业头疼的两类生成错误：冗余复述与幻觉，并引出了自去重与扎根度度量。 **自身独立性（Self-Distinctness）与冗余消除**： 当 RAG 系统针对一个问题检索出多篇相关但内容重叠的文档时，生成模型极易出现“自作聪明”的问题——在末尾附带总结，或者换汤不换药地反复用不同句式复述同一件事。在小参数模型中甚至会出现完全逐字的死循环复述，而大模型则表现为隐蔽的同义换述（Paraphrasing）。这种多余的篇幅不仅不会带来新信息，反而极大伤害了用户的阅读体验与对话效率。 在实现上，为了复用 NLP 中高效的语义相似度计算模块，框架底层匹配函数实际上计算的是“句子间事实的相似重合度”（Similarity Match），然后再通过互补变换得到独立性得分： $$\text{Self-Distinctness} = 1 - \text{冗余重复比例}$$ 例如，一段包含 4 句话的回复中，第 1 句与第 3 句语义重复，第 2 句与第 4 句语义重复。当系统前序步骤匹配到该重复事实后，算法会剪枝优化后续判定，最终判定 4 个原子事实中存在 2 处重复，自身独立性得分为 0.5。 **事实扎根度（Groundedness）与幻觉监控**： 作为企业对话机器人的基石指标，事实扎根度度量的是“模型回复在多大程度上严格锚定在检索到的上下文上”。系统将回复拆解出的每一个独立原子事实，逐个比对检索文档。只有能找到直接源文档支撑的事实计为分子，总原子事实数为分母。通过精准抓取未扎根的事实碎片，运维团队得以实时阻断 LLM 利用内生幻觉“胡说八道”的行为。 ### [21:59](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1319s) 幻觉的本质与扎实度（Groundedness）评估 在企业级 RAG 系统中，幻觉往往不是大语言模型在凭空捏造数字，而是源于“复杂的逻辑推理偏差”。例如在退换货政策中，源文档明确规定普通退货窗口为 30 天，但如果因卖家发错商品，退货窗口可延长至 45 天。在这种情况下，大模型可能会直接给出一个 45 天的回答，将特殊条件下的例外期限错误地泛化为通用期限。这种事实存在但逻辑应用错误的现象，正是企业知识库中最常见的幻觉形态。扎实度（Groundedness）指标正是通过精细拆解回答中的声明并与源文档比对，计算模型生成内容被检索事实支撑的比例。 - **逻辑混淆型幻觉**：模型极少凭空生成无意义数字，更多是将源文档中的特定条件（如发错货延长至 45 天）与普通规则（标准 30 天）混淆。 - **扎实度量化**：通过比对生成文本中的事实单元与检索上下文，剔除无法在源文档中找到依据或错误引用的陈述。 --- ### [22:57](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1377s) 指标联动机制（Metrics Interplay）与指标组合推断 单项指标往往只能呈现孤立的切片，而在实际应用中，评估体系拥有 30 到 40 种通过“指标联动组合（Interplay）”推导结论的方法。工程师和科学家们通过观察成对（pairs）或三元组（triplets）指标的升降变化，能够极其精准地反推出系统管道中具体的故障点。 ```mermaid flowchart TD accTitle: RAG Metric Interplay Diagnostic Flow subgraph Signal["监控信号 / 指标三元组"] A["自身去重度 Self-Distinctness: 过低"] B["源文档精确率 Source Precision: 维持高位"] C["源查询覆盖率 SQC: 中低波动"] end subgraph Diagnosis["根因诊断"] D["检索召回大量近似重复文档<br/>仅覆盖单一子查询，遗漏其余问题"] E["Prompt 中的强制摘要指令<br/>导致短文本回答陷入自我复述"] end subgraph Action["工程修复方案"] F["检索后执行语义聚类与去重"] G["知识库内容源头去重"] H["动态 Prompt：仅长文本启用摘要"] end Signal --> Diagnosis Diagnosis --> Action ``` --- ### [23:50](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1430s) 生产监控实战：诊断回答重复与近重复文档 在生产环境监控中，用户满意度调查显示有 34% 的差评原因是“回答过于啰嗦和重复”。通过监控体系捕捉到的指标三元组呈现出清晰特征：回答自身去重度（Response Self-Distinctness）显著低于 0.8 的阈值，源文档精确率（Source Precision）保持高位，而源查询覆盖率（Source Query Coverage, SQC）则在低到中等水平之间剧烈波动。 这一反常组合揭示了检索与生成的深层缺陷：高精确率意味着所有被检索出来的文档都确实在回答用户的子问题；然而，检索器召回了 4 篇高度相似的近似重复文档（Near-duplicate Documents），它们全都在解释“30 天退货窗口”，却把用户同时询问的“重新入库费（Restocking Fee）”彻底遗漏，导致 SQC 走低。大模型接收到 4 篇内容雷同的上下文后，加之 Prompt 中设置了强制总结摘要的要求，便在生成回答时反复兜圈子，甚至将短短两句话重复换词复述，极大损害了用户体验。 - **生产痛点比例**：回答重复性问题直接影响了 34% 的负向用户体验反馈。 - **设定阈值**：关键指标通常设定 0.8 作为合格线。 - **落地修复策略**： - **检索后处理**：在检索后阶段引入语义聚类（Semantic Clustering），剔除语义冗余的切片。 - **知识库治理**：协同内容团队从源头上清理重复或高度重叠的历史版本文档。 - **Prompt 动态调优**：取消对简短回答的死板强制摘要，仅在长文本生成时开启摘要逻辑。 --- ### [26:42](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1602s) 知识库与内容团队的逆向优化实战 企业知识库（Knowledge Base）与内容运营团队不再依赖人工去翻阅每一篇枯燥的长篇政策，而是将 RAG 评测框架作为发现内容缺陷的核心雷达。由于企业级聊天机器人将源文档视为“唯一事实来源（Ground Truth）”，内容团队会针对所有指标设定 0.8 的高门槛。 在针对高频访问内容源的监控中，一旦出现三项指标全线跌破阈值的极端恶化现象——回答精确率（Response Precision）低、扎实度（Groundedness）低、源查询覆盖率（SQC）低——即意味着源文档中根本不存在能够回答该查询的事实支撑。此时，LLM 为了强行给出一个看似完整的回复而被迫开始捏造事实（幻觉）。这直接证明问题不在检索模型或生成 Prompt，而在内容源头存在知识真空。 ```mermaid flowchart LR accTitle: Content Gap Identification Flow subgraph Metrics["三低警报"] M1["精确率低"] M2["扎实度低"] M3["SQC 极低"] end subgraph RootCause["知识库缺陷"] R["源文档缺失答案<br/>LLM 被迫幻觉补全"] end subgraph ContentFix["内容优化与重写"] C1["高频文档优先重写"] C2["增加结构化总结与 FAQ"] C3["引入具体案例解析政策"] end Metrics --> RootCause RootCause --> ContentFix ``` - **高频文档聚焦**：通过统计日维度、周维度以及特定业务旺季（Seasonality）中被检索召回频次最高的文档，精准锁定最具重写价值的内容。 - **内容治理落地举措**：为重点政策文档增加清晰的语言表达、结构化摘要、常见问答（FAQ）板块，并通过具体示例深入解释复杂政策条款。 --- ### [29:09](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1749s) 离线评测（Offline Evaluation）与模型迭代部署 在对 RAG 架构进行升级换代（例如在检索器中替换全新的 Embedding 嵌入模型）时，必须通过严格的离线评测来防范负面连锁反应（Ripple Effects）。 上线准入标准被明确量化为两个硬性条件：第一，历史数据上的全套指标必须维持在生产阈值之上；第二，鉴于 Embedding 模型的本质是优化语义检索，源文档精确率（Source Precision）与源查询覆盖率（Source Query Coverage）必须展现出明确的增量提升。通过提取长达一个月的大规模历史交互真实数据进行离线回放与波及分析，确认新模型在各项指标均达标且检索表现显著改善后，方可获准发布至生产环境。随着 RAG 技术的不断演进，整套企业级评测框架亦在持续迭代扩展。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=4s) 企业级场景而言，系统的命脉在于确保 LLM 严格依据公司内部知识库回答，而不是凭借预训练期间记住的内生知识进行未经授权的发散。 — Udita Patel > [02:31](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=151s) 即便排查出了根本原因，企业通常采用的是昂贵的超大规模预训练模型，绝不可能为了个别 badcase 去重新预训练。 — Udita Patel > [02:31](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=151s) 线上收集到的用户反馈往往是“零散且诉求单一的轶事”，通常只是某个用户在某一次交互中遇到了不爽的体验；企业若要投入昂贵的工程力量去修复问题，必须保证投入能够换来规模化的回报。 — Udita Patel > [07:11](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=431s) 在真实的大规模生产流量中，不存在任何现成的“标准答案”。 — Udita Patel > [10:21](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=621s) 企业需要的不是笼统的“机器人表现很差”或“机器人输出了无关信息”，而是具有高度可操作性的工程诊断：“让我们优化文本简练度，因为 25% 的回复中至少包含了 3 个用户根本没问的多余事实。” — Udita Patel > [14:01](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=841s) 机器人的实际体验取决于各模块协作的综合水准，设计包含 6 个独立但相互制约的指标体系，以模拟传统机器学习中经典的查准率与查全率的动态博弈。 — Udita Patel > [21:59](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1319s) 在企业级 RAG 系统中，幻觉往往不是大语言模型在凭空捏造数字，而是源于“复杂的逻辑推理偏差”。 — Udita Patel > [22:57](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1377s) 工程师和科学家们通过观察成对或三元组指标的升降变化，能够极其精准地反推出系统管道中具体的故障点。 — Udita Patel > [23:50](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1430s) 在生产环境监控中，用户满意度调查显示有 34% 的差评原因是“回答过于啰嗦和重复”。 — Udita Patel > [26:42](https://www.youtube.com/watch?v=fk-Ws46-2N0&t=1602s) 一旦出现三项指标全线跌破阈值的极端恶化现象——回答精确率低、扎实度低、源查询覆盖率低——即意味着源文档中根本不存在能够回答该查询的事实支撑。此时，LLM 为了强行给出一个看似完整的回复而被迫开始捏造事实。 — Udita Patel ## Mentioned Udita Patel, Amazon (亚马逊), Ai4, RAG (检索增强生成), LLM (大语言模型), LLM-as-a-Judge, Embedding 嵌入模型, TF-IDF ## Video description Udita Patel, Senior Applied Scientist, Amazon Abstracts I will talk about limitation of traditional NLP evaluation metrics, followed by unique challenges faced by enterprise chatbot owners in measuring the quality of their applications. I will discuss about what already exists in current research and how existing flawed characteristics of LLM induce errors that must be identified and monitored in enterprise RAG chatbots. I will then focus on describing the scalable monitoring and evaluation approaches for RAG based question answering, and possibly AI agents. I will discuss about the metrics in detail (definition, implementation and example ) and how to use their interplay to identify the specific areas of improvement in basic and advanced RAG pipelines. [Source](https://www.youtube.com/watch?v=fk-Ws46-2N0)

[0:04] Udita Patel: 企业级场景而言，系统的命脉在于确保 LLM 严格依据公司内部知识库回答，而不是凭借预训练期间记住的内生知识进行未经授权的发散。 [2:31] Udita Patel: 即便排查出了根本原因，企业通常采用的是昂贵的超大规模预训练模型，绝不可能为了个别 badcase 去重新预训练。 [2:31] Udita Patel: 线上收集到的用户反馈往往是“零散且诉求单一的轶事”，通常只是某个用户在某一次交互中遇到了不爽的体验；企业若要投入昂贵的工程力量去修复问题，必须保证投入能够换来规模化的回报。 [7:11] Udita Patel: 在真实的大规模生产流量中，不存在任何现成的“标准答案”。 [10:21] Udita Patel: 企业需要的不是笼统的“机器人表现很差”或“机器人输出了无关信息”，而是具有高度可操作性的工程诊断：“让我们优化文本简练度，因为 25% 的回复中至少包含了 3 个用户根本没问的多余事实。” [14:01] Udita Patel: 机器人的实际体验取决于各模块协作的综合水准，设计包含 6 个独立但相互制约的指标体系，以模拟传统机器学习中经典的查准率与查全率的动态博弈。 [21:59] Udita Patel: 在企业级 RAG 系统中，幻觉往往不是大语言模型在凭空捏造数字，而是源于“复杂的逻辑推理偏差”。 [22:57] Udita Patel: 工程师和科学家们通过观察成对或三元组指标的升降变化，能够极其精准地反推出系统管道中具体的故障点。 [23:50] Udita Patel: 在生产环境监控中，用户满意度调查显示有 34% 的差评原因是“回答过于啰嗦和重复”。 [26:42] Udita Patel: 一旦出现三项指标全线跌破阈值的极端恶化现象——回答精确率低、扎实度低、源查询覆盖率低——即意味着源文档中根本不存在能够回答该查询的事实支撑。此时，LLM 为了强行给出一个看似完整的回复而被迫开始捏造事实。

[30:21] Udita Patel: Thank you so much for your time

[0:04] Hi everyone. Uh I am Udita. I work as a [0:07] senior applied scientist at Amazon [0:09] leading Genai initiatives in [0:12] conversational systems. [0:14] Today I'm here to talk about holistic [0:17] evaluation of enterprise chatbots. [0:20] So a quick summary on today's talk. I [0:23] will go through a quick introduction of [0:25] rag. Then I will discuss the need for a [0:27] comprehensive framework that goes beyond [0:30] just accuracy. Next we discuss the [0:33] details of the framework design. I will [0:36] then present the set of evaluation [0:39] metrics that we have built and then that [0:41] we use. And lastly we will also look [0:45] into some real world examples of this [0:48] eval framework in action. So rag that is [0:52] retrieval augmented generation has [0:54] become a new paradigm for information [0:56] propagation. So we see rag used in [0:59] building customer service chat bots [1:02] employee on boarding assistance [1:06] IT assistance and then AI assistants [1:09] that are just an extension of search [1:11] systems for better user experiences. [1:16] So let's quickly go through an example [1:19] to see how rag works. [1:22] I'm pretty sure all of you already know [1:24] about this. Uh so let's take for an [1:27] example the user asks something. For [1:29] example, what's the return window for an [1:32] electronic item that I bought? Um the [1:35] system reaches [1:38] the system searches a connected [1:40] knowledge base uh that is indexed in a [1:43] vector database [1:45] uh and pulls in the most relevant [1:47] sources. For example, uh a snippet from [1:51] the company's return policy. [1:54] A large language model then takes the [1:56] retrieve text plus the query and uses [1:59] them together to generate a response [2:02] grounded in those sources. [2:05] And for enterprise chatbots, the main [2:08] focus is always to make the LLM respond [2:10] according to the company's knowledge [2:13] base and not use its like internal [2:16] information. [2:18] So usually this is how teams start [2:21] building well a simple rack system. So [2:25] there's a rack chatbot that kind of [2:27] works. Where do we go next? [2:31] Let's quickly see what follows the [2:33] initial development of this. So de [2:37] developers and product have lots of [2:39] ideas um on where this can go uh and [2:42] they want to execute fast. [2:45] Scientists have ideas to experiment and [2:48] improve retrieval embeddings prompt [2:51] tuning and they need a way to measure [2:55] the true impact of these experiments. [3:00] However, LLM are blackboxes [3:03] which means that they will introduce [3:04] their own quirks and errors and so in [3:08] that case we know the root cause but we [3:11] can't really retrain them because most [3:14] of the time we are using large scale [3:17] LLMs that are too costly to build. [3:22] Next is feedback from users is anecdotal [3:27] and usually for a single interaction [3:30] about something that they didn't like, [3:32] right? Uh but the way we invest in these [3:36] fixes that should bring us like big [3:39] payoffs [3:41] which brings us to monitoring. [3:45] So we need alarms that aligns with [3:48] product requirements. So when something [3:51] breaks, actions can be taken. And [3:54] lastly, conversations are multifaceted. [3:58] So are errors and drag and then so [4:01] should be your evaluation. [4:05] As you can see, this is getting hard to [4:07] handle. [4:09] We need a comprehensive eval framework [4:12] that provides us with the directional [4:14] strategy for developers, for product [4:17] managers, for stakeholders to maintain [4:21] and improve our rack system. [4:25] And just to demonstrate, very soon most [4:28] enterprise chatbots are looking like [4:30] this or even a more complex version of [4:33] this. So you can see uh pre and post [4:37] modules for both retrieval and [4:40] generation. And even at this stage this [4:42] is a continuously evolving system where [4:45] all modules are iteratively experimented [4:47] on. So [4:50] the only thing that we can assume that [4:52] remains consistent and available are [4:55] these triplets of data which is [4:59] user query that was asked. second [5:03] sources that were retrieved for the [5:04] query [5:06] and third responses [5:09] a response that was generated by an LLM. [5:13] So we base our evaluation framework just [5:16] on these three things. [5:18] So what does clarity look like or maybe [5:23] just a peak into what we built? [5:27] So we have successfully integrated our [5:29] eval framework into three processes. So [5:32] first is production monitoring which is [5:35] we customize thresholds for every [5:37] chatbot. [5:39] Our metrics interpret the root cause [5:42] and we first fix things that matter the [5:45] most [5:47] because we monitor on all our production [5:50] traffic or at least a huge sample of it. [5:54] Second is we derive knowledge base [5:58] improvements as well from this [6:00] framework. We monitor high traffic [6:03] sources that are mostly used. A return [6:05] policy document on a website is usually [6:09] a policy that was that is being queried [6:12] upon a lot of times. So if you monitor [6:16] these high traffic sources, our metric [6:19] identifies issues in content that ended [6:22] up being an incorrect answer for the [6:24] chatbot. And then as a result of this [6:28] analysis, our content team pro performed [6:32] rewrites on those policies and articles. [6:37] And third is offline evaluation. [6:41] So we use eval metrics to define overall [6:44] success criteria before we launch [6:47] anything into production. By anything I [6:50] mean it could be a new embedding model, [6:53] change swap of an LLM or anything else. [6:57] So we analyze the ripple effects of that [7:00] change into the entire rack pipeline and [7:04] then once we ensure performance [7:06] maintenance we go and deploy. [7:10] So [7:12] let's dive into how to build this [7:14] framework. [7:16] I will first walk through the framework [7:18] design. But why not just a set of [7:21] metrics? Why do we need a framework? [7:24] Well, it gives you a systematic view and [7:27] then also you can tailor it to your own [7:29] needs because each chatbot is built a [7:32] little bit differently. [7:34] And for this framework, we were tasked [7:36] to build something that a lot of team [7:38] can use. So I will now discuss the cho [7:40] choices we make for made for this [7:43] framework and why. [7:46] So the first situation that comes up is [7:49] that there is no ground truth uh [7:51] especially for production traffic like [7:54] you could have a test set you could you [7:56] could have a validation set you can [7:58] create some human annotated data set but [8:01] for production traffic there is no [8:02] ground truth. So in order to evaluate [8:04] rag interactions we need that gold [8:07] standard data set and but then so let's [8:11] see how that goes. So in a typical [8:13] interaction you have a query retrieval [8:16] will fetch few sources. [8:19] We need annotations or labels on for [8:21] this query whether this source was [8:23] correct or incorrect things like that. [8:26] And then in the next step LLM will [8:28] generate a response. [8:31] Now to evaluate the response, we either [8:34] need labels that checks whether this [8:37] text is a correct response or not or we [8:40] need a reference response to compare. [8:44] So in real world scenarios, this is what [8:46] happens. The content in sources is [8:49] updated frequently. A quick example [8:51] could be the the return window changed [8:54] from 30 days to 45 days. So the ground [8:57] truth has now changed, right? So any [9:01] source labels will go stale, [9:05] response labels will go stale and then [9:08] also the reference responses are [9:09] invalidated. So ground truth is not an [9:12] option. So what do we do? We use LLM as [9:15] judge. I'm again pretty sure almost all [9:18] of you know a little bit about this. So [9:21] we opted we opted to use LLM as judges [9:24] for our evaluation framework. It uses a [9:27] large language model to evaluate the [9:29] output of another LLM or an AI system. [9:32] So it doesn't need doesn't need ground [9:34] trth data. It's scalable although a bit [9:37] expensive. Evaluation [9:40] criteria can also be changed by changing [9:43] the prompt. So it kind of moves as your [9:47] chat bots move and evolve. Um so our [9:50] framework implements a matching criteria [9:53] with this LLM as judge. All metrics are [9:57] defined through a prompt matching [9:59] function which outputs zero if it's [10:01] unmatched and one if it's matched and [10:05] over here in the diagram you can see [10:07] like X and Y could take could be parts [10:10] of query source and response. So on [10:14] either either side X or Y could be [10:16] either of the either of these three [10:18] values. [10:20] Enterprise system also need granular [10:23] error diagnosis. [10:25] So the first aspect of this is that we [10:28] need granularity in scale of a metric. [10:32] So good bad responses aren't enough. [10:34] Zero values aren't enough. It hides [10:37] partial weaknesses and it also hides [10:39] like partial strengths. [10:43] The second aspect of this is we also [10:45] need more granular diagnosis of a metric [10:49] which means that what are we measuring [10:52] also needs to be detailed and granular [10:55] enough. [10:56] So basically here the idea is to go from [11:00] the bot is doing bad. [11:02] The bot is giving out irrelevant [11:05] information [11:08] and ideally to have something like this. [11:10] Let's improve brevity. 25% of responses [11:15] have at least three extra facts that [11:17] user didn't ask for. So for these kinds [11:20] of details, how what do we do? [11:24] We decompose. [11:26] So [11:28] let's look at this diagram again. The [11:32] same three things that we base our [11:34] entire evaluation framework on which is [11:36] query, source, and response. uh these [11:40] get decomposed into atomic units of [11:43] fats. [11:45] So let's take an example here for a [11:48] query. Uh it's asking for a return [11:50] return window and then it's also asking [11:52] how long for a refund. So we decompose [11:55] this and create selfcontaining [11:58] queries. Uh this user query would create [12:02] two atomic queries. And similarly for a [12:06] response [12:08] um the response there are four different [12:11] response in this in this uh uh in these [12:15] sentences and they are all represented [12:17] with as atomic facts as R1, R2, R3 and [12:21] R4 and they include one piece of [12:24] information. [12:26] Another thing we did to introduce this [12:28] kind of granularity is just is just we [12:32] went from like having labels that would [12:36] just say this is a bad response or a [12:38] good response uh to now being able to [12:41] evaluate the decompositions of that [12:44] response as repetition of facts, [12:48] non-essential unnecessary information [12:50] that was included in the response or if [12:53] it's an incorrect act. Another thing [12:56] necessary for this framework since we [12:58] were building this for multiple chat [13:00] bots to evaluate [13:02] u [13:04] they are all built for a unique user [13:05] experience and they all have to [13:08] accommodate varying tolerance levels of [13:11] errors. So some chat bots require [13:14] concise responses while other require it [13:18] to be friendly and verbose. uh some [13:21] tolerate incomplete responses because [13:23] there is an human agent setting that can [13:26] answer the other parts of the query that [13:28] wasn't answered by the chatbot while [13:30] others need exhaustive resolution of [13:32] user query through that um AI chatbot. [13:38] So there are differences here. So what [13:41] do we do for that? So we solve this by [13:43] allow allowing for a configurable [13:46] threshold for every metric. [13:48] So thresholds can be defined for every [13:51] instance of interaction. Thresholds can [13:54] also be defined for an aggregated view [13:57] and of course for all interactions. [14:00] The last u aspect of building such a [14:04] framework is maintaining robustness like [14:06] usually teams tune one component but [14:08] they it ends up degrading another right. [14:12] Uh so chatbots are only as good as the [14:15] combined performance. So how do we [14:17] measure combined performance by [14:21] uh including everything that we have [14:23] seen until now? [14:25] So this is a technical diagram for our [14:29] evaluation framework. It consists of [14:32] three six independent metrics. I'm [14:34] sorry. Uh when used together, it gives [14:39] us a holistic view of the rack system. [14:41] So you would see here that there are [14:43] components that I have already [14:45] discussed. So there's decomposition here [14:49] which is like you decompose uh the three [14:52] textual components that you're using [14:54] into atomic facts or atomic queries with [14:58] a decompose module which is implemented [15:00] through a prompt and then there is this [15:03] match function which is just these [15:04] arrows that you see. Uh let's see how to [15:08] read this like for source precision we [15:11] are checking that we are checking that [15:14] the sources that are retrieved by a [15:17] retriever is actually answering a query [15:21] that we have asked for. So this is what [15:25] arrow means. So, so we are checking [15:27] sources against queries versus in source [15:29] query coverage we are checking queries [15:33] against sources like is this query [15:36] answered in this source something like [15:38] that. [15:40] So now I'm going to go through the [15:42] details of the metric design. [15:44] So for a scenario where ground truth is [15:47] not available, we introduce [15:50] concept of query coverage and precision [15:53] to mimic the precision recall metric [15:56] that we all are very uh uh familiar with [16:00] uh that we have came across in [16:02] traditional ML. So we replace that with [16:05] query coverage and precision. So it's a [16:07] tension metrics. So [16:11] if you look at this diagram, the left [16:13] one, these small shapes, they represent [16:15] the decompositions that we just saw. Uh [16:18] and here are how the two metrics that [16:20] are set up. [16:22] The green ones represent the [16:24] decompositions that got a match score of [16:26] one. [16:28] Which means like over here, it means [16:30] that these resources are answering a [16:33] query. And then for the query these [16:36] queries are getting answered by these [16:38] sources and then the one like this [16:41] particular query is unanswered and this [16:45] these two particular resources are not [16:48] needed. So even within a single [16:50] retrieval let's say we retrieve five [16:53] articles we we would know that these [16:55] three were useful and the other two were [16:57] not. uh and then we do the same thing [17:00] for uh uh response precision and [17:04] response query coverage as well. These [17:06] are very two very similar metrics. So [17:09] here is the intuition for these two [17:11] metrics. So query coverage measures to [17:14] what extent are subqueries addressed or [17:17] covered by the retrieved sources. So is [17:20] is there an answer that exists in these [17:22] sources versus source precision measures [17:26] to what extent retrieved sources are [17:29] necessary to answer answer the query or [17:32] are they just slightly relevant but [17:35] aren't actually answering the question [17:38] that the user asked for. So let's walk [17:41] through an example of source precision [17:43] to look at this. So, the user asks, [17:47] "What's the return window for online [17:49] electronic purchases? Is there a [17:51] restocking fee? And how long for a [17:53] refund?" [17:55] The retriever retrieves a return policy. [17:59] Text highlighted in colors, they [18:01] represent like similar words found in [18:03] sources and it's kind of a intuition of [18:07] why this particular source was [18:10] retrieved. [18:12] And then it retrieves another article [18:14] which is actually a newsletter. It also [18:16] has some word similarities with the user [18:18] query uh in purple and in red. So [18:24] we can see that source A addresses the [18:28] the question the answers while source B [18:30] doesn't. So our math score in our [18:33] matching function would be 1 and zero [18:36] and then hence our source precision [18:38] becomes.5. [18:40] Similarly, let's look at the query [18:42] coverage metrics, but this time for a [18:45] response. [18:47] Same question. The response is online [18:50] electronics can be returned within 30 [18:53] days of delivery. So this time we take [18:56] the query because we are measuring query [18:59] coverage. So it's the query that we [19:01] would decompose and match against. And [19:05] then we see here that one one of the [19:08] question which is Q1 was answered but [19:11] then the Q2 was not. So we have math [19:14] score of one and zero. And then our [19:16] response query coverage becomes.5. [19:19] We have few more response metrics over [19:22] here. Uh [19:26] self-distinctness to identify redundant [19:28] information in response and groundedness [19:31] to measure hallucinations. [19:33] Hallucinations again I think you're all [19:36] very aware that they exists [19:38] they exist. Um so the induction behind [19:41] self-distinct here is like to what [19:44] extent is the response non-redundant [19:47] usually we see u especially if we u [19:51] retrieve a lot of related sources for a [19:54] particular query we would see that the [19:57] LLMs tend to generate u a summary at the [20:00] end or uh they start repeating [20:03] information by paraphrasing it it's [20:05] generally not like same sentence [20:08] generation over and over again. Although [20:11] for like smaller weight models you would [20:13] see that as well but usually you would [20:15] see like them paraphrasing information [20:17] multiple times and this verbosity many [20:20] times hurts uh user experience and user [20:23] readability a lot. So that's why we [20:26] ended up adding this metrics. [20:28] Groundedness is something that is very [20:30] very important to almost all enterprise [20:32] chatbots because we want them to behave [20:35] based off of what knowledge they are [20:37] given uh and not uh using information [20:42] within themselves. So for groundedness [20:45] it is to what extent the response is [20:47] grounded in retrieved sources. So again [20:50] we do the same matching functions we see [20:53] uh and then we take the ratio of what [20:57] worked and what didn't work. Um and [20:59] that's how we measure these two. [21:02] Again a very quick example the same [21:04] response we have four sentences here. Uh [21:09] they get matched with the first sentence [21:12] get matched with third the second one [21:14] gets matched with fourth. They contain [21:16] the same information. So we match once [21:20] we got got a like repetition match for [21:23] the first two we end up not matching for [21:25] the third and the fourth one because [21:27] that's like to improve our optimization [21:30] on this and then our [21:33] self-distinctness score becomes.5. [21:38] So one thing I want to note here is that [21:41] the match function in this specific [21:43] metrics matches similarity because it is [21:46] easier to uh easier to calculate that u [21:50] with lots of NLP algorithms. So but what [21:54] we are measuring here is distinctness. [21:57] So so our formula then becomes like 1 [21:59] minus the fraction that matched uh and [22:03] that's the distinctness of this [22:04] response. [22:08] Um I have another one for groundedness. [22:14] It does the same thing. Um [22:19] here is an interesting uh fact here like [22:22] usually the hallucinations that we see [22:24] they are not just making up numbers. So, [22:29] so the source the source says that the [22:32] return window is 30 days but for [22:34] incorrect item send return window is [22:37] extended to 45 days. So here the model [22:40] actually ended up using 45 days instead [22:43] of 30 days. So it's this kind of comp [22:46] complex logical reasoning that usually [22:50] uh becomes hallucination. [22:55] And then [22:57] as as the last part of this presentation [23:00] I will go through some real real life [23:03] examples. So the ind interdependency of [23:06] these metrics it can be interpreted in [23:10] multiple ways. So we have at least 30 to [23:12] 40 ways of uh deriving conclusions from [23:16] the combination of how these metrics are [23:18] doing which means that what is going [23:20] higher which metrics is going lower and [23:24] sometimes these are like pair of metrics [23:26] sometimes these are triplets of metrics [23:29] and then then we assign like [23:30] interpretations on what that means and [23:33] then that particular interpretation will [23:36] show you what needs to get fixed. [23:42] So let's go through a realam real real [23:45] world examples uh to see how this [23:47] framework enables monitoring. [23:50] So production monitoring [23:53] the business insight was that 30% of the [23:56] time user dissatisfaction reasons reason [23:59] is answers feel repetitive. [24:04] metric interplay is that [24:07] response source uh response [24:10] self-distinctness is below the [24:12] threshold. Source precision value [24:15] remains above the threshold and while [24:18] query coverage goes from low to medium. [24:20] So it kind of fluctuates, [24:23] right? So what does it tell us now for [24:27] with this triplet of uh metrics? This [24:31] tells us that all sources are answering [24:34] a user subquery which is why the [24:36] precision is high right but the problem [24:39] is they may all be answering the same [24:42] query and leaving out the other two that [24:45] needed to be answered. So while some [24:49] subqueries may or may not have sources [24:51] address them which is why we see source [24:54] query coverage being low or medium over [24:56] here. [24:58] So the interpretation here is that [25:00] retriever is fetching multiple near [25:03] duplicate documents. So for a particular [25:05] query let's say again for simplicity our [25:08] query is the return window and let's say [25:10] there are like four different versions [25:13] of these documents and they all get [25:15] highly ranked in our top retriever and [25:18] they all get uh [25:21] included in our retrieved sources prior [25:24] to generation but they are all on [25:26] answering just one query whereas the [25:28] question about the res restocking fee [25:31] that just gets ignored right So this [25:34] kinds of tell tells us that we need to [25:36] we need we we have duplicated documents [25:39] and then also that maybe we have a [25:42] mandated summarization in prompt which [25:44] we have added for helping users but now [25:49] it is just causing repetition and that [25:52] those repetitions are now causing bad [25:55] user experience. [25:57] So now that so we had a threshold for [25:59] this for8 and we know knew from data [26:02] analysis that this was a great problem [26:05] to solve based because it was affecting [26:07] 34% of uh dissatisfaction user survey. [26:11] So the fix is to apply [26:14] semantic clustering post retrieval or [26:17] dduplicating by going to the content [26:19] team or if you if you were using [26:22] summarization in the prompt then maybe [26:25] just do that for longer answers because [26:28] maybe for shorter answer is just [26:29] repeating like the two sentences that [26:34] need not need repeating. So things like [26:36] that. So [26:42] now the next this is a very interesting [26:44] one. A content or knowledgebased team. [26:47] Um the team that publishes content, [26:50] maintains content, verifies content, [26:53] writes content, updates content. Uh they [26:57] use this framework in a completely uh [26:59] different way. they are focused on [27:01] identifying the improvement areas in [27:04] published sources using these chatbot [27:06] interactions right because most of the [27:08] time these days nobody's going and [27:10] clicking and reading the entire policy [27:12] people just go to AI assistance and say [27:15] help me solve this problem so [27:19] so for that again first comes the [27:21] thresholding they set thresholds of8 on [27:24] all metrics because sources are ground [27:26] truth for enterprise chat bots which [27:29] means that they are the most important [27:31] thing. Next is [27:35] we are observing a particular hight [27:37] traffic content sources right now um [27:41] that we could easily figure out like uh [27:44] what what articles get retrieved the [27:46] most in a day in a week on a particular [27:49] season things like that um and then the [27:52] and then we see that the metrics [27:54] interplay looks like this everything's [27:56] low response precision is low we are [27:58] hallucinating [28:00] questions aren't getting answered Third. [28:03] So for the first one when [28:06] uh response precision is slow this means [28:09] that response is not directly answering [28:12] the query. Why is it not answering the [28:15] query? Because ground and then we have [28:19] groundedness is low which means that [28:21] LLMs have to make up facts to give a [28:24] clear reply. Now SQC is low which which [28:28] means that answer isn't in even found in [28:30] the retrieved sources. So what ends up [28:32] happening here is that there's no answer [28:35] in sources. [28:37] Um so the model cannot find it. When it [28:40] tries to create a response, it just [28:42] hallucinates. [28:44] So here we identify that the issue is [28:48] actually content. [28:50] So again we prioritize these kinds of [28:52] impactful rewrites and updates. Uh we [28:56] change the language of these uh of these [28:58] articles. We add structured summaries to [29:00] it. We sometimes add frequently asked [29:03] questions. We sometimes explain policies [29:05] with examples. All sorts of things. [29:09] And then the last one is offline [29:11] evaluation of new improvements. [29:14] Uh so again over here to in order to [29:18] launch let's say a new embedding model [29:20] in retriever the success criteria is [29:23] defined as all metrics should remain [29:26] above the threshold as they are in [29:29] production and since embedding improves [29:31] retrieval so we need to have at least [29:34] some improvements in source precision [29:36] and source query coverage. [29:40] Now we take the historical data and we [29:43] analyze the ripple effects of this [29:46] change uh let's say for a month worth of [29:48] data. So we take historical interactions [29:51] analyze how switching the embedding [29:54] model is affecting other metrics [29:58] and in this case all metrics are [30:00] maintained above threshold and we also [30:02] see some improvements in uh source [30:05] metrics. So we now that we have ensured [30:08] that we have maintained performance in [30:11] improvement even if we use this new [30:14] embedding model then we just go ahead [30:16] and deploy [30:21] and this was this is it. I hoped you [30:24] like this talk. Uh as Rag is ever [30:27] evolving our framework is also ever [30:30] evolving. Um thank you so much for your [30:33] time.