JPMorgan Architect Urges Retrieval Fixes to Halt Production Failures

摩根大通架构师：优化检索方能遏制生产故障 由于难以承受的高昂计算成本，长上下文模型无法取代RAG；基于经验的检索优化与混合路由成为通往企业级AI可靠性的唯一可行路径。 Naresh Dulam系统分析了企业级RAG在摄入、检索和生成阶段的核心故障模式，揭示了生产系统在真实环境下的失效机制。为解决检索与上下文瓶颈，他提出了切实可行的技术手段，包括语义分块、超参数扫描、混合搜索（BM25结合向量搜索）、交叉编码器重排以及上下文压缩。此外，演讲还详细阐述了基于HyDE的查询扩展、结构化输出约束、智能体回退工具，以及Lynx等幻觉评估框架。最后，Dulam倡导采用一种在轻量级RAG检索与长上下文LLM回退之间动态选择的混合自路由架构，并强调优化检索能解决RAG面临的绝大多数难题。 企业将 RAG 系统从概念验证（POC）推向生产环境时面临严峻的可靠性鸿沟，系统故障往往分布在数据摄取、检索以及模型生成这三个核心阶段。 (4:45) 针对内部知识库缺失内容的问题，可通过前端结构化提取工具、在 Prompt 中设置严格的防御性防幻觉指令（如明确回答“我不知道”），以及启用具备联网搜索能力的 Agent 工具进行动态回退补充。 (6:28) 在企业生产场景中，基于语义边界或 Agent 驱动的高级分块策略显著优于固定字符切分；通过专家标注的黄金评估集对分块大小、重叠度及 Top-K 阈值进行超参数微调，能在不重构代码的情况下最大化检索召回率。 (15:14) 为应对高精度的技术查询，应结合密集语义向量与稀疏关键词搜索（如 BM25）实现混合检索，并配合 Cross-Encoder 重新排序器与上下文压缩技术以过滤无关噪音，防止大模型注意力分散。 (19:10) 长上下文大模型并不能完全替代 RAG，企业架构应采用自路由混合方案：优先通过低成本的精准 RAG 检索处理，仅在上下文不足时才回退至长上下文窗口，从而平衡计算成本与响应精度。 (28:23)

Naresh Dulam argues that the transition of Retrieval-Augmented Generation (RAG) systems from prototypes to enterprise production is hindered by severe reliability gaps rooted primarily in retrieval engineering failures. Rather than relying on default frameworks or massive context windows, organizations must adopt multi-stage retrieval pipelines, systematic hyperparameter tuning, and hybrid self-routing architectures. Enterprise Failure Points and the Retrieval Gap (4:45) While standard proof-of-concept RAG implementations succeed on basic datasets, production environments expose critical failure points that erode user trust. Drawing from academic research on RAG engineering, Dulam organizes these system breakdowns into three operational stages: ingestion failures (missing corpus content), retrieval bottlenecks (missed top-rank scoring, context dropped during consolidation, and incorrect query specificity), and generation errors (unextracted context obscured by noise and malformed output formats). Dulam contends that RAG is fundamentally a retrieval discipline. If engineering teams can resolve retrieval failures, they resolve approximately 90% of overall production RAG challenges, ensuring the generation model receives precise, uncluttered, and contextually grounded source text. Mitigating Missing Content with Guardrails and Agents (6:28) When enterprise knowledge stores lack the data required to address a user's prompt, unconstrained large language models (LLMs) frequently hallucinate responses with unfounded confidence. To prevent this, pipelines must incorporate upstream document parsers—such as Unstructured.io or LandingAI—to correctly extract text from complex enterprise formats and scans before chunking. At the prompt layer, defensive guardrails must explicitly instruct the LLM to reply with 'I do not know' if the context does not supply the answer, preventing parametric memory bleed. In advanced setups, agentic RAG workflows dynamically check whether vector store retrieval yielded adequate context. If internal documents are insufficient, the agent invokes external API tools, such as Tavily search, to retrieve live external data before proceeding to generation. Multi-Stage Retrieval, Chunking, and Contextual Compression (19:10) Rigid character-count chunking fractures contextual coherence. Dulam recommends transitioning to semantic chunking, which splits text based on sentence-level embedding distances, or LLM-driven agentic chunking. He notes that teams frequently neglect empirical hyperparameter sweeps across chunk sizes (e.g., 256, 512, 1024 tokens), chunk overlap, Top-K candidates, and similarity thresholds (such as 70% or 80% cutoffs), all of which improve yield without rewriting application logic. To bridge the gap between vague user phrasing and dense technical specifications, advanced pipelines use multi-query expansion and Hypothetical Document Embeddings (HyDE), which embed an idealized LLM-generated answer to match relevant documents. Dense semantic search is paired with sparse keyword retrieval (BM25) in a hybrid pipeline, followed by cross-encoder rerankers that score candidate passages against the query. Finally, contextual compression utilizes sub-chunk extractors and coarse filters to strip non-essential tokens, preventing semantic clutter from degrading the LLM's attention mechanism. Hybrid Self-Routing and Output Schema Governance (28:23) Dulam dismisses claims that million-token context windows eliminate the need for RAG, noting that context stuffing introduces unsustainable computing costs ('tokenomics'), operational latency, and attention degradation such as 'lost in the middle' phenomena. Instead, he proposes a hybrid self-routing architecture. A lightweight decision node evaluates whether low-cost standard RAG retrieved sufficient context; if adequate, generation proceeds, but if judged incomplete, the pipeline falls back to loading the broader document set into a long-context LLM. To prevent downstream automation pipelines from failing due to unstructured or invalid outputs, systems must enforce schemas using native foundation model JSON modes, framework parsers (such as LangChain Pydantic parsers), or constrained decoding libraries like Fructose. Output verification models, such as Patronus AI's Lynx, can inspect final outputs for hallucinations before delivery. Dulam emphasizes that long-term enterprise reliability requires treating RAG as an empirical science evaluated against domain-expert 'golden' test sets using tools like DeepEval. As architectures mature, organizations must prioritize localized embedding models to meet privacy constraints, fine-tune retrieval components, and implement tool-enabled agentic fallbacks rather than relying on monolithic context stuffing.

在 AI4 2025 大会上，平台架构师 Naresh Dulam 提出，企业级检索增强生成（RAG）从原型验证迈向实际生产的瓶颈主要在于检索工程而非生成模型；通过系统性的分块优化、超参数扫描、混合检索与重排，并结合长上下文回退的自路由架构，才能跨越生产环境的可靠性鸿沟。 生产环境下的七大失效点与落地鸿沟 (4:45) Dulam 指出，许多 RAG 原型在基于玩具数据集验证时表现优良，但部署至真实企业环境时，往往由于可靠性断裂导致工作流瘫痪并损害用户信任。借鉴学术界关于 RAG 工程中七大失效点的分类，他将核心故障归纳在三个生命周期阶段：首先是摄取阶段的内容缺失，即源文档未被索引入库，导致大语言模型产生极具欺骗性的高置信度幻觉；其次是检索阶段的未达前列排名、脱离上下文以及特异性错误，表现为关键信息因粗糙的评分机制跌出召回窗口、在合并过滤中被剔除，或因用户提问过于模糊而检索出缺乏针对性的宽泛段落；最后是生成阶段的信息未被提取与格式不完整，此时过多噪声或文档间的相互矛盾会分散模型的注意力机制，使其无法提炼关键事实，或输出无法被下游程序解析的残缺格式。 摄取防御、代理回退与输出结构化约束 (6:28) 针对内容缺失引发的模型捏造现象，Dulam 提出了三层递进的治理方案。在数据源头，企业应使用 Unstructured.io 或 LandingAI 等高级解析工具，从排版复杂的企业级 PDF 或扫描件中准确提取文本与结构；在提示词层面，必须设立防御性护栏，显式约束模型在上下文未包含答案时直接回答“不知道”，杜绝其擅自调取参数化训练记忆。若本地向量库确实缺乏关键事实，系统则应演进为 Agentic RAG 架构，借助 LangChain 等框架动态绑定外部工具（如 Tavily 搜索引擎）自主检索外部实时数据补充上下文。针对自动化业务流中常见的 JSON 语法错误或结构残缺问题，Dulam 建议结合现代基础模型的原生结构化模式、应用框架的输出解析器，或使用 Fructose 等通过函数调用与受限解码强制执行预设模式的专用库。 高级分块、超参数扫描与混合重排策略 (19:10) Dulam 强调“RAG 本质上是检索问题，解决了检索问题就解决了 RAG 挑战的 90%”。在分块维度，生产系统应告别固定的字符计数切分，转向根据句子语义距离断句的语义分块（Semantic Chunking），甚至采用由大模型判断主题边界的代理分块。在系统调优上，团队无需重构核心架构，即可对分块大小（如 256、512 与 1024 Token）、重叠比例、Top-K 数量以及严格的相似度阈值（如设定 70% 或 80% 余弦相似度截断）进行网格扫描，并依赖业务专家构建的黄金问答集和 DeepEval 框架量化验证。 针对单纯语义检索在面对型号、硬件参数等精准实体查询时表现不佳的缺陷，Dulam 主张建立多层级检索流水线：首先利用多查询生成拦截模糊提问并进行语义扩展；随后将密集语义向量与 BM25 稀疏关键词检索融合成混合搜索；最后引入交叉编码器（Cross-Encoder）重排模型对候选文档与用户提问进行联合注意力打分。此外，结合按意图提炼句子的上下文压缩技术与基于假设性文档嵌入（HyDE）的查询重写技术，能有效排除语义噪声并对齐文档分布。 代币经济学制约下的长上下文自路由 (28:23) 面对“百万 Token 长上下文模型将取代 RAG”的论调，Dulam 反驳称，盲目向提示词中塞入海量文档会导致严重的“代币经济学”（Tokenomics）算力成本失控，并引发端到端延迟增加以及模型在长文本中间区域注意力衰减（Lost in the Middle）的弊端。他倡导一种混合自路由 RAG 架构：系统优先执行成本低廉的标准 RAG 检索，并由轻量级决策节点评估召回的切片是否足以支撑回答；若判定证据充分则直接生成，若判定信息不足，流水线才会自动回退至长上下文模型，按需载入完整文档集合。在此架构的输出端，还可结合 GPT-4 原生检测或 Patronus AI 的 Lynx 等专用幻觉评估模型进行二次事实核验，在运行成本、响应延迟与生成准确率之间达成工程平衡。 Dulam 总结认为，企业级 RAG 的长期演化将依赖工具启用的代理工作流与领域微调检索模型，而非单一依赖无限制扩大的上下文窗口。不过，该方案在落地时依然面临依赖领域专家标注黄金基准集的高昂人力成本，以及在引入多查询扩展、重排序器和回退判定后，系统端到端累积延迟显著上升的工程折中挑战。

## Background 该内容由 Ai4 频道于 2025 年 8 月 27 日发布在 YouTube 平台上，为摩根大通（JP Morgan Chase）软件工程副总裁 Naresh Dulam 在大会上的技术演讲。视频简介指出，演讲聚焦于改善现实世界中的企业级 RAG 系统，深入剖析了从内容缺失、检索排名偏差到生成幻觉等关键故障，并提供了高级分块、混合搜索、上下文压缩、HyDE 以及智能体 RAG 等经过验证的工程解决方案。 ## TL;DR 演讲者直击 RAG 系统从概念验证原型走向企业级生产环境时面临的巨大鸿沟，并依托经典的“七大故障点”框架对摄取、检索和生成全链路问题展开系统拆解。针对内容缺失与检索失效，他提出了引入专业解析工具、严格提示词约束、Agentic 外部工具调用、语义/智能体分块以及基于 Cross-Encoder 的重排与 HyDE 增强等实操策略。在生成与架构层面，演讲详述了 LangChain 解析器与结构化框架在输出控制中的应用，并通过上下文压缩过滤冗余噪声。最后，演讲者从 Tokenomics（Token 经济学）角度反驳了长上下文将取代 RAG 的观点，主张采用带有判定节点的自路由混合架构，并重申 RAG 本质上是一项检索优化挑战。 ## Atmosphere & style 整场分享节奏明快、干货紧凑，散发着资深一线架构师务实且严谨的技术极客气质。演讲者摒弃了空洞的概念炒作，以直击生产“翻车”痛点的坦诚开场，通过层层递进的架构拓扑与清晰的代码逻辑呈现解题思路。听众能够深切感受到一种高度聚焦于工程落地、超参数微调与成本效益平衡的实战氛围，既有对底层细节的扎实把控，又不乏对大模型演进与 Agentic 架构的前瞻视野。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=4s) 企业级 RAG 系统在从 POC 原型走向生产环境时面临巨大鸿沟，经常因各类错误导致用户丧失信任，其实质是整个数据生命周期中的工程与检索挑战。 - [04:45](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=285s) RAG 系统的主要故障点可归纳为七大类：内容缺失（摄取层）、错失首选排名、未进入上下文、具体性偏差（检索层），以及未成功提取、内容不完整和格式错误（生成层）。 - [06:28](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=388s) 解决知识库“内容缺失”需从三方面入手：采用高质量数据提取工具从源头解析、在 Prompt 中严格要求‘不知道即明确拒答’以遏制幻觉，以及引入 Agentic RAG 调用外部工具补充实时信息。 - [13:32](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=812s) 分块策略应从基础的递归字符分块，逐步升级到语义分块乃至由大模型主导的智能体分块（Agentic Chunking），以提升分块的语义完整性。 - [13:32](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=812s) 超参数调优（如 Chunk Size、Top-K 召回量、相似度阈值）无需系统重构即可带来显著性能提升，必须结合业务黄金数据集与 DeepEval 等框架建立基准评估闭环。 - [17:59](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1079s) 切勿盲信 MTEB 等公开排行榜上的 Embedding 模型跑分，公有榜单第一在企业私有垂直数据上未必最优，需在本地业务数据集上实测验证。 - [19:10](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1150s) 单纯语义搜索无法精准匹配具体技术参数型号，必须结合 BM25 关键词检索构建混合搜索（Hybrid Search），并采用 Cross-Encoder 重排模型进行深度相关性打分。 - [21:59](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1319s) 通过上下文压缩技术（包含提取器 Extractor 和过滤器 Filter），可以在生成前剔除冗余干扰段落，避免大模型因信息过载而提取失败。 - [23:39](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1419s) 控制 LLM 输出格式可通过优先利用原生模型结构化输出、使用 LangChain 解析器（如 Pydantic/JSON OutputParser）或采用 Fructose 等专用结构化框架来实现。 - [26:13](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1573s) 采用假设性文档嵌入（HyDE）可有效改善含糊提问的检索效果：先让模型生成假想答案，再拿假想答案向量去检索，大幅拉近文档与查询之间的语义鸿沟。 - [28:23](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1703s) 长上下文大模型并不能完全取代 RAG，无节制消耗 Token 带来的成本和延迟不可持续；最佳实践是构建基于判定节点（Decide Node）的自路由混合架构。 - [28:23](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1703s) 生产系统应在最终交付前引入幻觉检测环节，例如利用 Petronus AI 开发的 Lynx 工具进行事实一致性核查。 - [31:01](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1861s) RAG 本质上是检索优化问题，未来演进重点将集中在结合工具调用与多步规划的 Agentic RAG，以及针对垂直领域微调的专用 RAG 系统。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=4s) 开场演讲与企业级 RAG 的生产落地鸿沟 在拉斯维加斯举办的 AI4 2025 大会上，演讲者开门见山地向现场观众发问：有多少人在实际业务中构建过真实的 RAG（检索增强生成）系统？多年以来，RAG 已经从概念验证（POC）和学术研究全面走向驱动真实世界的业务应用。然而，任何真正将 RAG 推向生产环境的工程师都会深切体会到其中的痛点：在企业级场景下，原型与生产系统之间存在着一道巨大的鸿沟。当系统从实验室搬到生产环境时，产品往往频繁出错，导致用户迅速丧失信任。 本次分享的核心目标，正是探讨如何构建能够在生产环境中稳定运行的可靠 RAG 方案。演讲者结合自身在企业一线的实战踩坑经验，并借鉴学术界的前沿研究成果，梳理出一套切实可行的应对方案。在进入正题前，演讲者简要介绍了自己的背景：他是一位拥有 16 年行业经验的 AI 与平台架构师，长期深耕于云计算、大数据与机器学习的交叉领域，兼具工程实现、技术战略与系统架构经验，曾多次在全球技术大会演讲并指导行业从业者。他同时给出了免责声明：本次分享仅代表个人观点，不代表前任或现任雇主的立场；由于演讲结束后需要立刻赶往一场初创企业的媒体发布会，现场可能无法留出问答时间，但他欢迎参会者通过 LinkedIn 与他交流。 --- ### [02:40](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=160s) RAG 系统基础工作流与企业级架构挑战 演讲者首先带领大家快速回顾了 RAG 的基本定义与运行机制。简单来说，RAG 是一种极其优雅的方式，它能够将企业专有的私有数据提供给大语言模型（LLM），使模型不再给出通用、泛泛而谈的模棱两可回答，而是能够使用“企业自己的业务语言”给出高度契合上下文的精准答案。 ```mermaid flowchart TD accTitle: 企业级 RAG 基础双阶段工作流 subgraph Ingestion["第一阶段：数据摄取 (Ingestion)"] A["企业原始文档"] --> B["数据分块 (Chunking)"] B --> C["向量嵌入 (Embedding)"] C --> D[("向量数据库 / 存储库")] end subgraph QueryStage["第二阶段：查询与生成 (Query & Generation)"] E["用户提问"] --> F["向量检索 (Retrieve)"] D -.-> F F --> G["上下文与 Prompt 拼装"] E --> G G --> H["大语言模型 (LLM)"] H --> I["生成业务答案"] end ``` 从高层架构来看，RAG 的处理流程可以清晰地划分为两个阶段： 1. **数据摄取阶段（Ingestion）**：加载企业数据，将其切分成较小的数据块（Chunks），通过 Embedding 模型将其转换为机器可理解的数值向量，最后存入存储介质中（通常是向量数据库，也可以是关系型数据库或对象存储）。 2. **查询与生成阶段（Query & Generation）**：当用户提出问题时，系统前往向量存储中检索出最相关的文档块，将检索到的上下文与用户的原始提问拼接在一起输入大语言模型，最终由模型生成回答。 整个流程原理看似简单明了，但在企业级现实落地中，各个环节都极易出现故障。 --- ### [04:45](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=285s) RAG 系统的核心故障点：从摄取到生成的七大挑战 演讲者指出，本次分享的挑战分类灵感来源于一篇题为《检索增强生成系统工程中的七大故障点》（*Seven Failure Points in Engineering Retrieval-Augmented Generation Systems*）的经典研究论文。企业在实际落地 RAG 时，常见的问题贯穿了数据流的整个生命周期： ```mermaid flowchart LR accTitle: RAG 系统的主要故障点分布 subgraph S1["1. 摄取阶段故障"] F1["内容缺失<br>(Missing Content)"] end subgraph S2["2. 检索阶段故障"] F2["错失首选排名<br>(Missed Top Rank)"] F3["未进入上下文<br>(Not in Context)"] F4["具体性偏差<br>(Incorrect Specificity)"] end subgraph S3["3. 上下文与生成阶段故障"] F5["未成功提取<br>(Not Extracted)"] F6["内容不完整<br>(Incomplete)"] F7["格式错误<br>(Wrong Format)"] end S1 --> S2 --> S3 ``` - **数据摄取层**：首要故障点是“内容缺失”（Missing Content），即知识库或数据存储中压根就没有能够回答用户提问的基础数据。 - **检索提取层**：包括“错失首选排名”（Missed Top Rank，正确文档虽被召回但排名过低）、“未进入最终上下文”（Not in Context，初始检索到了但被后续过滤掉）、以及“具体性偏差”（Incorrect Specificity，召回的内容太宽泛或太窄）。 - **生成响应层**：即知识库中确实存在信息，但由于上下文噪声过大或信息互相矛盾，模型无法正确提取答案（Not Extracted），或者生成的内容残缺不全、格式有误。 --- ### [06:28](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=388s) 挑战一：内容缺失（Missing Content）与三项破解之道 当数据存储中没有足够的信息来回答用户的问题时，大模型往往会以极其自信的口吻编造答案，即产生严重的“幻觉”（Hallucination），这是导致终端用户对系统产生挫败感和信任崩塌的主要诱因。针对内容缺失问题，演讲者提出了三项行之有效的解决方案： 1. **采用高质量的数据提取工具**：正所谓“垃圾进，垃圾出”（Garbage In, Garbage Out）。面对企业内部格式各异、散落各处的凌乱文档，基础解析器往往丢失关键信息。引入专业的数据提取工具（如 Unstructured.io 或 Landing AI 等多模态/文档提取模型），可以从源头大幅提升文档解析质量。 2. **优化提示词工程（Improved Prompting）**：严格约束模型行为。传统的 Prompt 仅将问题和上下文喂给模型，演讲者展示了一个生动的反例：上下文中提供的是关于板球（Cricket）运动的信息，但提问却是关于欧洲杯足球锦标赛（Euro Championship）的。如果使用常规 Prompt，模型会动用自身的先验知识来作答，而非基于给定的上下文。改进的做法是在 Prompt 中严厉声明：“若提供的上下文中未包含答案，严禁凭空假设或产生幻觉，必须明确回复‘不知道’。”这样能有效确保输出高度贴合上下文。 3. **引入 Agentic RAG（智能体 RAG）**：简单来说，Agent 的本质就是“LLM + 提示词 + 外部工具”。当向量数据库中缺失相关数据时，Agent 能够自主判断并调用外部工具获取信息。演讲者展示了一个代码实现示例：系统绑定了 Web 搜索工具（如网络检索 API），并在 Prompt 中指示“如果在提供的上下文中找不到答案，请调用工具前往外部搜索”。此时即便知识库中只有板球数据，系统在被问及欧洲杯时也能自动发起联网搜索，获取实时数据并生成准确答案。 --- ### [11:15](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=675s) 检索与上下文提取的多重陷阱：排名错失、未入上下文与噪声干扰 在实际检索过程中，即便数据仓库中存在正确信息，系统也常常因为各种策略失当而折戟： - **错失首选排名（Missed Top Rank）**：由于分块策略不良或相似度打分不准，最关键的文档块没有排在前列，导致被截断过滤。 - **未进入最终上下文（Not in Context）**：虽然在初检阶段命中了相关文档，但在后续的上下文合并与压缩（Consolidation）策略中发生偏差，导致关键信息未能被送入最终发给 LLM 的 Prompt 中。 - **信息未能成功提取（Not Extracted）**：信息虽然被成功送入了上下文，但由于上下文中充斥着大量无关的噪声信息，或者存在前后矛盾的数据，导致 LLM 被干扰，无法从中准确定位并抽取出答案。 - **问题表述过于宽泛（Vague Query / Incorrect Specificity）**：用户提出的问题过于笼统或模糊，导致系统检索出大量同样宽泛模糊的文档，最终生成毫无实际参考价值的空泛回答。 --- ### [13:32](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=812s) 提升检索精度的核心策略：分块、超参数调优与评估 针对检索阶段的各项顽疾，演讲者详细剖析了多维度的优化手段，重点强调了分块策略与超参数调优： ```mermaid flowchart TD accTitle: RAG 检索与上下文优化策略栈 A["检索与上下文优化"] --> B["分块策略演进<br>(Chunking Strategies)"] A --> C["超参数微调<br>(Hyperparameter Tuning)"] A --> D["先进检索与压缩<br>(Reranker & Compression)"] B --> B1["基础/递归字符分块 (Recursive Character)"] B --> B2["语义分块 (Semantic Chunking)"] B --> B3["智能体分块 (Agentic Chunking)"] C --> C1["数据块大小 (Chunk Size)"] C --> C2["Top-K 召回数量"] C --> C3["相似度阈值 (Similarity Threshold)"] D --> D1["上下文压缩 (Contextual Compression)"] D --> D2["重排模型 (Reranker Models)"] ``` 在分块策略上，复杂度自上而下递增：最基础的是递归字符分块（Recursive Character Chunking，常见于各类 POC 原型）；而在业务实践中表现更优越的是**语义分块（Semantic Chunking）**；最前沿的则是**智能体分块（Agentic Chunking）**，即使用 LLM 配合特定指令，让大模型自主理解并根据语义边界切分文档。 在超参数调优方面，演讲者观察到许多工程团队完全忽视了这项工作，但超参数调优往往无需重构系统架构就能带来立竿见影的效果： - **数据块大小（Chunk Size）**：必须针对业务数据进行多组尺寸测试，行业内没有放之四海皆准的“黄金尺寸”。 - **Top-K 与相似度阈值（Similarity Threshold）**：合理设定召回数量（如 3 条或 5 条），并搭配相似度阈值（例如仅保留相似度大于 70% 或 80% 的文档），过滤低质噪声。 - **评估闭环**：调优的前提是建立基准。团队应当邀请业务领域专家（SME）精心构建包含标准问题与基准答案的“黄金数据集”（Golden Dataset），并借助 DeepEval 等评估框架，量化对比 RAG 输出与专家标准答案的匹配度。 --- ### [17:59](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1079s) 嵌入模型选型与高级检索进阶 最后，演讲者探讨了嵌入模型（Embedding Models）的选型考量与高级检索机制。采用最新一代的 Embedding 模型是不言而喻的优化方向，因为新模型使用了更大规模、更高质量的数据集进行训练，生成的数值向量在表征能力上显著优于老旧模型。 然而，演讲者特别告诫：**切勿盲信公开排行榜（如 MTEB Leaderboard）的基准跑分**。公开榜单排名第一的模型，在企业的特定私有垂直领域数据上未必表现最佳，必须在自己的业务数据集上进行严谨的实测与对比。此外，如果企业受到严格的数据合规与隐私政策限制，无法直接调用商业化闭源模型 API，则必须在本地基础设施中部署高质量的开源嵌入模型。在高级检索策略上，结合相似度阈值过滤、上下文压缩（Contextual Compression）以及重排模型（Reranker Models，对候选文档与提问进行相关性深度重新打分），是确保送入 LLM 的上下文保持高纯度、高相关性的关键路径。 ### [19:10](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1150s) 多查询检索、混合搜索与 Cross-Encoder 重排策略 在提升检索阶段的召回质量时，首先可以通过设定相似度阈值（如 70% 相似度）来直接过滤掉低相关的噪声文档。然而，面对真实用户模糊、不精确的提问，系统需要具备多查询检索（Multi-query Retrieval）的能力：大模型能够识别出用户意图中的模糊点，并自动将原始问题改写、拆解成多个不同角度的问题变体，以此从向量数据库中拉取更全面的候选文档，显著提升最终生成的回答质量。 在搜索方式的选择上，单纯依赖语义向量搜索（Semantic Search）往往无法解决所有问题。演讲者指出，当用户查询具体的技术配置参数（例如特定的 RAM 大小或硬盘容量型号）时，语义距离计算并不能精准匹配，此时必须结合传统的关键字检索（Keyword Search）。将语义检索与关键词检索相结合的混合搜索（Hybrid Search）是提升召回率的关键手段。演讲者强调了一个核心观点：“RAG 从根本上说就是一个检索问题。如果你能解决检索问题，你就解决了 90% 的 RAG 挑战。” 为了进一步精确筛选候选文档，引入重排模型（Reranker Models）至关重要。重排器通常基于交叉编码器（Cross-Encoder）架构，与双编码器仅计算向量空间距离不同，Cross-Encoder 会同时接收原始查询和检索到的文档，直接评估该文档对回答该问题的实际相关性分数。结合链式检索（Chained Retrieval）策略——即依次执行关键词与语义混合搜索、Cross-Encoder 相关性重排、低分截断——能够确保送入大模型上下文的文档高度契合问题需求。 - **相似度阈值过滤**：设置特定比例阈值（如 70% 相似度），丢弃低于相关性基线的召回块。 - **多查询改写（Multi-query）**：利用大模型将单一模糊查询扩充为多个语义变体，拓宽召回覆盖面。 - **混合搜索（Hybrid Search）**：融合向量语义相似性与 BM25 等传统关键字匹配，解决精确规格与专业术语匹配难题。 - **Cross-Encoder 重排**：输入 (Query, Document) 联合打分，精确判定文档对回答问题的真实贡献度。 --- ### [21:59](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1319s) 上下文压缩：提取器与过滤器的实战机制 当送入大模型的上下文包含过多非必要信息时，模型会与人类一样产生认知过载与困惑。为了解决上下文冗余问题，可以采用上下文压缩（Contextual Compression）技术。上下文压缩主要分为两种模式：提取器（Extractor）和过滤器（Filter）。提取器负责通读召回的文档块，并提炼、抽取其中与问题直接相关的关键句子或段落；而过滤器则评估整个文档或章节，直接将判定为不相关的整段内容彻底剔除。 演讲者以印度首都为例生动阐释了上下文压缩的应用场景：在印度，除了通常意义上的首都外，可能存在行政首都、司法首都等不同概念。如果用户提出了一个表意宽泛的问题，系统可能同时召回关于多个首都维度的文档。通过引入上下文压缩层，大模型利用自身的先验推理能力进行信息综合，识别出最后两篇文档与用户寻找行政中枢的核心意图不符，从而直接过滤掉干扰信息，仅保留最精准的文档送入生成阶段，确保最终响应精准贴合上下文。 --- ### [23:39](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1419s) 输出格式控制：原生 LLM 支持、LangChain 解析器与结构化框架 在生成阶段，模型未能按照预期输出指定格式（如特定的 JSON 模式、XML 或结构化对象）是生产环境中的高频故障。针对格式错误问题，首选方案是选用具备强大原生结构化输出能力的前沿模型（例如 GPT-4o 等最新模型）。如果模型开箱即用支持结构化生成，只需配合精准的 Prompt 引导即可直接获得目标格式。 对于需要深度定制格式转换或使用开源模型的场景，可以借助成熟的解析库。例如 LangChain 提供了丰富的开箱即用解析组件，涵盖字符串解析器（String Parser）、CSV 解析器（CSV Parser）、Pydantic 解析器（Pydantic Parser）以及 JSON 解析器（JSON Parser），开发者既可以直接调用，也可以根据业务需求进行类扩展与深度定制。此外，采用专门的结构化输出框架也是极佳的解法，演讲者分享了自己在实际生产项目中采用 Fructose 框架的经验，该框架在保障响应结构严谨性方面表现优异。 ```mermaid flowchart TD accTitle: 结构化输出与格式控制方案 accDescription: 展示应对大模型输出格式错误的几种主要解决路径 A["LLM 生成原始文本"] --> B{"格式控制技术选型"} subgraph Native ["原生模型方案"] B -->|"原生支持"| C["GPT-4o 结构化输出 / 强化 Prompt"] end subgraph Parsers ["解析转换组件"] B -->|"LangChain 生态"| D["Pydantic / JSON / CSV Parsers"] end subgraph Frameworks ["结构化框架"] B -->|"专用框架"| E["Fructose 等结构化输出框架"] end C --> F["合规格式响应 (JSON / XML / 结构化对象)"] D --> F E --> F ``` - **原生输出能力**：优先利用前沿模型的 Native Structured Outputs 特性降低解析失败率。 - **LangChain Parsers**：通过 Pydantic/JSON OutputParser 约束输出并实现自动序列化校验。 - **专用框架赋能**：采用 Fructose 等轻量级结构化框架实现确定性的字段提取。 --- ### [26:13](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1573s) 不完整响应处理与 HyDE（假设性文档嵌入）检索增强 当生成的响应不够完整或遗漏关键事实时，根本原因通常在于用户提问过于含糊，导致模型无法进行充分推理，或未能从知识库中检索到完整的支撑材料。对于不完整响应问题，一方面需要升级底层模型，并结合思维链（Chain of Thought, CoT）与自洽性（Self-Consistency）等高级 Prompting 策略增强模型的推理解析步骤；另一方面，可以构建具备工具调用能力的 Agentic 系统，在内部知识库缺失数据时主动调用外部工具补充上下文。 另一种极其有效的检索优化手段是 HyDE（Hypothetical Document Embeddings，假设性文档嵌入）。HyDE 的运作逻辑是：首先让大模型根据用户的原始问题生成一份“假设性答案”（即假想中理想文档的样子，哪怕内容存在部分虚构），接着将这份假设性答案转换为向量 Embedding，再拿该 Embedding 去向量数据库中进行相似度匹配。由于文档与文档之间的向量表征匹配度远高于简短查询与详细文档之间的匹配度，HyDE 能够大幅拉近语义鸿沟，召回质量显著提升的高相关真实文档，最后再将这些真实文档送入 RAG 最终提示词生成准确答案。 --- ### [28:23](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1703s) RAG 与长上下文大模型：Tokenomics 考量与自路由混合架构 针对业内关于“长上下文 LLM（Long-Context LLMs）是否会让 RAG 走向消亡”的激烈争论，演讲者给出了明确的反驳。把数万甚至上百万字的内容一股脑塞入上下文窗口在经济性上是不可持续的，为此业内催生了一个新词汇——“Tokenomics（Token 经济学）”。无节制地消耗上下文 Token 会导致极高的推理调用账单与延迟瓶颈，因此最佳的生产实践并非二选一，而是采用 RAG 与长上下文 LLM 相结合的自路由混合方案。 在混合路由实现中，系统首先尝试通过标准 RAG 路径进行轻量级检索与初次评估。系统内包含一个判定节点（Decide Node），由 LLM 评估当前检索到的切片文档是否足以回答用户提问。如果文档支撑充分，则走标准 RAG 生成响应，将 Token 消耗与响应时间降至最低；一旦模型判定现有检索文档不足以支撑完整回答，系统则动态降级/回退至长上下文模式，将完整的大篇幅原始文档整体加载至上下文窗口中进行全局综合推理。此外，在响应交付前还应引入幻觉检测机制，既可以依赖 GPT-4 的内置一致性校验，也可以集成 Petronus AI 开发的 Lynx 等开源/商业评估工具对生成文本进行事实校验。 ```mermaid flowchart TD accTitle: RAG 与长上下文 LLM 混合自路由架构 accDescription: 展示根据判定节点动态选择标准 RAG 或长上下文模式的路由流程 subgraph Input ["输入与检索"] Q["用户原始查询"] --> R["轻量级检索模块"] R --> D1["检索候选文档块"] end subgraph Decision ["路由仲裁层"] D1 --> DN{"Decide Node (判定节点)<br/>文档是否足以回答?"} end subgraph Generation ["双轨响应生成"] DN -->|"是 (信息充分)"| SR["标准 RAG 流程<br/>(低 Token 成本 / 低延迟)"] DN -->|"否 (信息缺失)"| LC["长上下文 LLM 模式<br/>(全局文档灌入上下文)"] end subgraph Safety ["安全与幻觉检测"] SR --> HC["幻觉检测 (Lynx / 内置检查)"] LC --> HC HC --> Out["最终确定性答案输出"] end ``` - **Tokenomics 权衡**：无序扩展 Context Window 会带来二次方的成本与计算延迟增长，长上下文无法完全替代 RAG。 - **Decide Node 仲裁**：通过决策节点在轻量 RAG 与全量 Long-Context 之间实现动态自路由调度。 - **事实幻觉拦截**：在输出端配置 Lynx（来自 Petronus AI）等专业事实核查工具，杜绝编造内容。 --- ### [31:01](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1861s) 总结与展望：检索优化内核、评估驱动与 Agentic RAG 未来 演讲者在最后重申，RAG 在本质上依然是一个检索优化问题（Retrieval Optimization Challenge）。“只要能解决检索挑战，90% 的 RAG 难题就迎刃而解了”。在工程落地中，开发团队切忌直接套用框架的默认配置，必须建立持续的评估指标体系（Evaluation Metrics）与黄金基准数据集，对 Chunk 大小、Overlap、Top-K、相似度阈值等关键超参数展开系统性的调优实验。 展望未来技术演进，即使超长上下文窗口持续普及，RAG 依然是降本增效的核心架构支柱。RAG 的演进方向将高度聚焦于两大维度：其一是深度融合工具调用、自主规划与多步检索的 **Agentic RAG** 系统；其二是结合垂直行业私有知识库深度微调的 **领域专用 RAG（Domain-Specific Fine-Tuned RAG）**。演讲者在结尾表示由于需赶往支持一位创业者无法现场接单提问，欢迎参会者通过 LinkedIn 进行后续的技术交流与探讨。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=4s) 在企业级场景下，原型与生产系统之间存在着一道巨大的鸿沟。当系统从实验室搬到生产环境时，产品往往频繁出错，导致用户迅速丧失信任。 — Naresh Dulam > [02:40](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=160s) RAG 是一种极其优雅的方式，它能够将企业专有的私有数据提供给大语言模型（LLM），使模型不再给出通用、泛泛而谈的模棱两可回答，而是能够使用“企业自己的业务语言”给出高度契合上下文的精准答案。 — Naresh Dulam > [06:28](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=388s) 垃圾进，垃圾出（Garbage In, Garbage Out）。 — Naresh Dulam > [06:28](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=388s) 若提供的上下文中未包含答案，严禁凭空假设或产生幻觉，必须明确回复‘不知道’。 — Naresh Dulam > [06:28](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=388s) 如果在提供的上下文中找不到答案，请调用工具前往外部搜索。 — Naresh Dulam > [17:59](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1079s) 切勿盲信公开排行榜（如 MTEB Leaderboard）的基准跑分。公开榜单排名第一的模型，在企业的特定私有垂直领域数据上未必表现最佳，必须在自己的业务数据集上进行严谨的实测与对比。 — Naresh Dulam > [19:10](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1150s) RAG 从根本上说就是一个检索问题。如果你能解决检索问题，你就解决了 90% 的 RAG 挑战。 — Naresh Dulam > [31:01](https://www.youtube.com/watch?v=ZExf9aiD4R4&t=1861s) 只要能解决检索挑战，90% 的 RAG 难题就迎刃而解了。 — Naresh Dulam ## Mentioned Naresh Dulam, JP Morgan Chase, Ai4, Unstructured.io, Landing AI, DeepEval, MTEB Leaderboard, LangChain, GPT-4o, GPT-4, Fructose, HyDE, Petronus AI, Lynx, LinkedIn ## Video description Naresh Dulam, VP of Software Engineering, JP Morgan Chase Abstracts Improving Real-World RAG Systems explores key challenges and practical solutions for enhancing Retrieval-Augmented Generation (RAG) pipelines. The presentation identifies critical failure points including missing content, poor retrieval ranking, context consolidation errors, and LLM output issues (hallucinations, formatting errors, incompleteness). It addresses these with actionable strategies like advanced chunking techniques, hybrid search (semantic + keyword), context compression, and modern embedding models (highlighting the MTEB Leaderboard). The deck emphasizes agentic RAG systems that leverage live data tools and structured output frameworks (e.g., LangChain parsers) to mitigate missing context. It discusses tradeoffs between RAG and long-context LLMs (cost vs. performance), advocating hybrid approaches with self-routing systems. Practical solutions include Hyperparameter tuning (chunk size, top-K thresholds), HyDE for query rewriting, and hallucination detection tools like Lynx. The presentation stresses the importance of evaluation metrics, domain-specific fine-tuning, and recent research insights while maintaining cost-effectiveness. Key takeaways position RAG as fundamentally a retrieval optimization challenge, urging practitioners to prioritize experimentation with chunking/retrieval strategies over default configurations, despite advancements in long-context models. Target audience: AI engineers and enterprise teams implementing production-grade RAG systems. [Source](https://www.youtube.com/watch?v=ZExf9aiD4R4)

[0:04] Naresh Dulam: 在企业级场景下，原型与生产系统之间存在着一道巨大的鸿沟。当系统从实验室搬到生产环境时，产品往往频繁出错，导致用户迅速丧失信任。 [2:40] Naresh Dulam: RAG 是一种极其优雅的方式，它能够将企业专有的私有数据提供给大语言模型（LLM），使模型不再给出通用、泛泛而谈的模棱两可回答，而是能够使用“企业自己的业务语言”给出高度契合上下文的精准答案。 [6:28] Naresh Dulam: 垃圾进，垃圾出（Garbage In, Garbage Out）。 [6:28] Naresh Dulam: 若提供的上下文中未包含答案，严禁凭空假设或产生幻觉，必须明确回复‘不知道’。 [6:28] Naresh Dulam: 如果在提供的上下文中找不到答案，请调用工具前往外部搜索。 [17:59] Naresh Dulam: 切勿盲信公开排行榜（如 MTEB Leaderboard）的基准跑分。公开榜单排名第一的模型，在企业的特定私有垂直领域数据上未必表现最佳，必须在自己的业务数据集上进行严谨的实测与对比。 [19:10] Naresh Dulam: RAG 从根本上说就是一个检索问题。如果你能解决检索问题，你就解决了 90% 的 RAG 挑战。 [31:01] Naresh Dulam: 只要能解决检索挑战，90% 的 RAG 难题就迎刃而解了。

[6:28] Naresh Dulam: If you do not find the answer in the provided context, do not assume or hallucinate; reply with 'I do not know'. [15:14] Naresh Dulam: There is no universal rule of thumb—optimal sizes depend directly on the dataset. [17:59] Naresh Dulam: Public benchmark rankings do not always mirror domain-specific enterprise data performance; teams must run local experiments against their own internal datasets. [19:10] Naresh Dulam: RAG is a problem of retrieval. If you can solve the retrieval problem, you will solve your RAG challenges. [19:10] Naresh Dulam: Re-rankers function as cross-encoders that directly compute query-document relevancy scores rather than isolated semantic vector distance. [21:59] Naresh Dulam: Contextual saturation degrades LLM performance just as cognitive overload impairs humans. [26:13] Naresh Dulam: HyDE bypasses query-to-document embedding mismatches by performing document-to-document vector similarity search. [31:01] Naresh Dulam: If you can solve the retrieval challenge, the 90% of your RAG is solved. [31:01] Naresh Dulam: Long-context LLMs will not replace RAG; RAG remains critical for cost optimization, fine-grained access control, and latency management.

[0:04] It's great to be here at the Las Vegas [0:06] AI4 2025. [0:08] Um over the years you might have seen uh [0:12] the RAD retrieval augmented systems from [0:16] PC or the research to any real world are [0:20] powering the real world applications [0:22] right but as many of just for the sake [0:25] of it how many of you have an uh build [0:28] the real world rag applications uh not [0:31] knowing that building the real world ra [0:32] so then you know that what are the pain [0:34] points in building an applications at [0:37] the enterprise price at the real world. [0:41] So, so you you might have experience [0:44] right. So we are going to see how we can [0:46] build in a reliable rag solutions that [0:49] works in the production and not the PC's [0:53] and um whenever these like the PC's when [0:56] we do that when we move into the [0:58] productions most of the times we [0:59] experience that products get started [1:02] failing and users lose the trust and we [1:06] see the wide gap between the a reliable [1:08] product in the production versus the P. [1:12] gap is too too much. So today I wanted [1:16] to take you in a a path where what are [1:19] the common challenges you see in any [1:22] enterprise systems and how you can [1:24] address them. So these are the some of [1:26] the uh uh solutions that we have tried [1:29] myself I tried it and uh some of the [1:32] research work we uh we borrowed from the [1:34] other research papers as well. [1:38] So the these are the practical lessons. [1:40] Okay, before we get started, a quick [1:42] introduction about me. Uh, I'm an AI and [1:45] platform architect with 16 years in [1:47] industry. Um, I'm working in the [1:50] intersection of the cloud, big data and [1:52] machine learning. [1:54] So, my journey has taken me through the [1:57] blend of like engineering, strategy and [2:00] architecture and I have also presented [2:02] in uh multiple global conferences as [2:06] well as I mentored the people across the [2:08] globes. [2:10] So having said that one u okay a quick [2:14] disclaimer is that uh here what I'm [2:17] going to share is those do not reflect [2:19] my employer previous or my current [2:20] employer just for the disclaimer and uh [2:23] at the end I may not able to take the [2:25] questions because I have a press [2:26] announcement right after this uh session [2:29] one startup I'm helping a founder so I [2:32] may need to rush but I'll share my [2:33] contact so that you can reach out to me [2:35] in the LinkedIn [2:37] having said that So let's start with [2:40] understanding a rag systems. So if you [2:43] already know that rag what is the rag um [2:46] so it's it don't I don't be I don't give [2:49] the much details but it will quick [2:51] refresher for you. So [2:54] let's see that what is a rag system. RAG [2:58] is a nice way in which you can give your [3:02] enterprise data to your [3:06] language models so that they will give [3:09] you an answers that contextually [3:11] relevant means they don't give you any [3:14] generic vague answer they will they will [3:16] talk you the language of your enterprise [3:18] the data you have it so enterprise data [3:21] so that's what uh I was showing here [3:23] like you have a data you give it to the [3:25] model so that model will give you a [3:27] answer that talks your business. [3:31] So if we look at the rag at a very high [3:34] level, it's a two-step process. The [3:35] first step is like you take the data, [3:38] your enterprise data. You load it and uh [3:41] you break it into a small parts chunks [3:44] and then you convert them into a [3:46] numbers. Those are called the embeddings [3:48] so that the machine can understand and [3:50] then you store it into a stores kind of [3:53] vector stores. It could be a vector [3:55] store or can be any relational data [3:58] store as well or an object store. It [4:00] doesn't matter but it just a store where [4:03] it stores the information. Most of the [4:05] times we use an electra store. [4:08] And the second part of the system is [4:10] that you ask the question. So the [4:12] question goes to your uh store and it [4:15] will pull out the information and you [4:18] get the ret uh like the relevant [4:20] documents from your store and then you [4:22] combine that with your question and give [4:24] it to the LLM and it will generate an [4:26] answer. So that's a very very high level [4:29] of what is the rag systems. [4:32] So it's an easy right but a lot can go [4:35] wrong here in enterprise world. [4:39] So that's the talk today challenges and [4:42] practical solutions. [4:45] So [4:47] these are the very high uh very high [4:49] picture of high level picture of how the [4:52] real world uh what are the few [4:54] challenges in building any enterprise [4:58] rack systems. So these are the failure [5:01] points there but there could be more but [5:03] these are the most of the times we [5:05] experience for any rag system. [5:09] So by the way this um [5:12] uh talk was inspired from one of the [5:15] research paper uh by the called seven [5:18] failure points in engineering in a [5:21] retrieval ragumented systems. [5:24] So it starts with the uh top where the [5:27] injection process where you get an [5:29] injection the challenges there is that [5:32] you have an missing content the red box [5:35] red boxes represent the challenges. So [5:37] where you h you don't have the data to [5:40] answer your question in your data store [5:43] that's the first failure point and [5:45] sometimes there could be ch uh [5:47] challenges in the retrieval phase when [5:50] you are retrieving the information. So [5:52] those represent the misstop rank or not [5:55] in the context or incorrect specificity. [5:59] So these are like the challenges when [6:00] you are retrieing the information from [6:02] your vector store you might face these. [6:04] We'll go into the detail and sometimes [6:07] you have the information but you it's a [6:09] challenging to generate an response out [6:12] of it because of the various reasons [6:14] like the noise in the information or [6:16] contradicting information. So those [6:18] represent the uh incomplete not [6:21] extracted or wrong format. We go into [6:23] the detail and how can we solve them as [6:26] well. So the first challenging is the [6:28] missing content where you don't have the [6:31] information in your system to answer [6:33] your rag system uh you answer your [6:35] question. [6:37] So that's the f first challenge. Then if [6:40] there is no enough information in your [6:42] data store sometimes these models give [6:46] you an answers very confidently that's [6:49] called hallucination. you know that and [6:51] that most of the times end users getting [6:54] frustrated with incorrect or irrelevant [6:56] wrong responses. So that's the first [6:59] challenges. So what is the uh some of [7:02] the solutions for the missing content is [7:05] like these are the some these are the [7:07] three but there could be more depending [7:09] on your use case but most of the times [7:12] we the this is the best way to fix the [7:14] missing content. So the first challenge [7:17] is like it goes by the like the good [7:19] data good right so if you have an you [7:23] have a documents the data store like you [7:25] that's clumsy or like incorrect and it's [7:28] have a lot of uh messy data right so [7:31] with that data you cannot get a better [7:34] outcome so you use the better tools [7:37] extracting tools like unstructured IO or [7:40] like the recently I have experienc [7:42] experimented with the landingai AI these [7:45] are the some of the models will help us [7:47] to extract the good information from [7:49] your um u the documents of your uh store [7:53] like your enterprise where the document [7:56] is scattered in multiple places in [7:57] multiple format. So that goes by the [8:00] first solutions and the second solution [8:03] is like uh so the beta prompty. So as [8:07] you see that the model will hallucinate [8:09] right. So instead of that one we can [8:11] restrict the model to not to give an [8:13] answer so that users get in a more [8:16] contextual relevant answer and it gives [8:19] saying that I don't know in case of the [8:21] relevant information not is present not [8:23] present and the third one is the agentic [8:26] it's the most famous you might have [8:27] hearing about a lot about the agents so [8:29] the agent is agentic way is like pulling [8:31] the information wherever it is it's not [8:33] in your vector store but you can go [8:35] outside of your store and get the [8:37] information we'll see one example for it [8:40] as well. So here is a uh one a typical [8:44] rag solution where uh the f top part of [8:47] that is a prompt where you are giving a [8:50] question and context and saying that [8:52] give an answer. So in the middle part of [8:55] the slide you see that one uh I'm [8:58] providing an information about a [9:00] different sport like a cricket but I'm [9:02] asking a question about the Euro [9:03] championship. um in the bottom you might [9:06] seeing the right answer because the [9:08] model is using its knowledge its own [9:10] knowledge to answer the question it's [9:12] not using the context that we provided. [9:15] So the better way to uh handle this [9:18] solution is that improved rag prompt [9:22] where you are saying that if you don't [9:24] have the information in your store in [9:26] the context provided do not assume don't [9:30] hallucinate like do not assume just give [9:33] the information that you have you saw it [9:35] in the context. So that's way we can [9:37] better restrict the your rag systems [9:39] output so that the users will get an rag [9:43] response contextually relevant response. [9:46] So that's the second uh second solution [9:49] and the third one is the agentic rags. [9:51] So in a very very simplified way agent [9:55] is LLM with prompt and better tools. So [9:59] these tools help the models to get the [10:02] right information to answer your user uh [10:04] questions. So that's what uh way very [10:08] high level of agentic rag. It's a [10:10] complicated but it's a very simply way [10:12] of simply representing the agentic rack. [10:15] So here is a quick implementation where [10:18] um we are we are implementing as as we [10:21] know that agentic rack needs a tool. The [10:23] top part is in a tool. We implemented a [10:25] web search that's an attack search and [10:27] then we are binding the tool to the [10:29] model in the uh middle part of it and [10:33] then we are saying in the prompt in case [10:37] if you don't find the information in the [10:39] context provided to you go and use one [10:42] of the tools provided to get the [10:43] information wherever available. So in [10:46] case of this is a web search. So with [10:49] that we are asking the same question [10:51] like we are providing in a context about [10:53] a one sport but we are asking a question [10:55] about another sport and if you see in [10:57] the bottom the model the rag system is [11:00] able to make an internet uh website call [11:03] and able to get the information. So [11:05] these are the some of the ways you can [11:07] address the missing content challenges. [11:10] There could be more but these are few of [11:12] the better ways. [11:15] So the second challenge is you have the [11:18] data the correct information to answer [11:21] your questions in your s store but they [11:24] were filtered out because of various [11:27] regions maybe the chunking strategy or [11:30] consolidation strategy bad consolidation [11:33] strategies. [11:34] So this leads the model to again given a [11:37] incorrect response, irrelevant response [11:40] so that users get frustrated with the uh [11:42] answer generated by your lag system. [11:46] So [11:49] I'll combine a few challenges together [11:51] because we can tackle them together. So [11:53] the second other challenge is the not in [11:55] the context means the the pres the [11:59] information is was present in the [12:01] initial stages when it retrieved but it [12:04] did not make to the final stage when the [12:06] model is answering that question. That's [12:07] a final context that you sent to the [12:09] model. So that's what the not in the [12:11] context. So in the previous one it was [12:15] there but it was filtered out because of [12:17] the top ranking. Here it was there but [12:20] it was uh due to the wrong consolidation [12:24] strategies it was missing in the final [12:26] context that's sent to the uh model to [12:29] generate an assa. [12:31] So and the other one is other challenge [12:34] is not extracted means you have the [12:38] information and you send it to the you [12:40] have the information that sent to the [12:42] rag or model but the model is not able [12:45] to get the correct information from the [12:48] context you provided maybe because of [12:50] your we are providing an contradicted [12:53] information or the two v uh too much [12:56] noise in the information that context [12:58] that we sent it. So these are the some [13:00] of the reasons for that the model or the [13:03] rack system is not able to extract that [13:05] uh information out of your context to [13:08] answer the question. [13:11] So other other challenge is that um [13:14] sometimes the users questions may be too [13:16] vague that may lead to the uh vague [13:19] documents retrieved from the um data [13:22] store and that gives you a vague [13:24] responses they do not give any accurate [13:27] responses. So this is another challenge. [13:32] So the solutions for these challenges [13:36] Mr. top rank not in the context [13:38] incorrect specificity is [13:41] the first one is the better chunking [13:43] strategies. We'll see the some of the [13:44] chunking strategies uh that we can use [13:46] it and the second is a hyperparameter [13:49] tuning. I observed most of the people [13:52] will not doing the hyperparameter tuning [13:54] but that gives a better results without [13:57] doing anything. mean without rewriting [13:59] your application ra systems and uh third [14:03] one is the better embedding models on [14:04] latest embedding models they are trained [14:06] on the latest data so that you have a [14:07] better um uh better the embeddings the [14:12] numbers so that you get a better uh doc [14:14] uh when you search it you get a better [14:16] responses the other one is that using [14:20] using the advanced retrieval strategies [14:22] we'll see the hide uh and the wrong rag [14:25] cont long context versus the rag [14:27] solution conditions that how can we use [14:30] an advanced strategies to get a better [14:33] documents from your store so that your [14:34] model will give a better answers [14:38] and the other one is the uh contextual [14:41] compression. So you might we might seen [14:43] that in the previous slide that one of [14:45] the you might have a lot of noise in [14:48] your the context that you are building. [14:50] So how you can reduce that noise in your [14:53] context. So that's a comp some of the [14:55] contextual compression strategies. And [14:58] uh the final one is the reanker models. [15:00] Reranker models are the models they take [15:03] the information and they will give you [15:04] the relevant scores of the context the [15:07] documents to your question how how much [15:10] they are related or not. [15:14] So these are the some of the um uh out [15:17] of the box chunking strategies [15:18] available. Uh but you depends on your [15:21] use case you can try your own chunking [15:23] strategy as well. So these are the the [15:25] order in that from the top to the bottom [15:28] uh the complexity goes on as you go down [15:30] and the one uh one semantic ch the [15:34] chunking strategy we used most of it [15:36] time is the semantic chunking that gave [15:37] a better results for certain use cases [15:42] and the last one is an agentic chunking [15:44] you putting an agents to break your [15:46] documents you putting an LLM telling in [15:49] some of the instructions how to break [15:51] the document so the LLM itself will [15:54] break your documents into the uh [15:57] contextually relevant chunks. [16:00] So that's what the uh agentic chunk and [16:02] the top uh top one uh one to five are [16:05] the uh basic chunking strategies and the [16:08] second one is the recursive character [16:10] text chunking you might see in most of [16:12] the um PC's. [16:17] Okay. So the other solution is [16:21] hyperparameter tuning. So [16:24] you have the so there are [16:26] hyperparameters are basically the chunk [16:28] size you can try with the different [16:29] chunk size you can uh as you seen in the [16:32] slide like you can try with the [16:33] different chunk sizes and see that which [16:36] chunk size is giving you a better [16:37] answers to your uh questions and it [16:40] there is no thumb rule this is the chunk [16:42] size will work it depends on the uh your [16:45] data and your system [16:48] and the other hyperparameter turing is [16:50] like the top k top Okay, with the [16:52] similarity threshold saying that how [16:54] many like you have a you put a threshold [16:57] like the documents that you retrieve [16:59] from your system should have a minimum [17:01] like 80% or 70%age of similarity. So [17:04] that's what the similarity threshold. Um [17:07] so and the top K is like how many [17:09] documents you wanted to retrieve when [17:11] you retrieve from your vector. So like [17:13] the three documents or five documents. [17:15] So play with the different numbers on [17:17] your uh use case and see that which one [17:20] get the better results for you. So to do [17:22] that one you need to have in a first the [17:25] golden answers and questions where the [17:27] SMMES can craft you the golden question [17:30] and answers for you and you can see that [17:33] the answer generated by your system use [17:35] these settings and then evaluate with [17:38] the golden answers using the evaluation [17:41] metrics like the deep eval u I think [17:43] there was some more framework available [17:45] so that you can see that the how much [17:48] the answers are relevant. and the [17:51] generated by the C rag versus your SM [17:55] curated answers. [17:59] The second one is then um the better [18:01] embedding models there is no brainer. So [18:03] the latest models are trained on the [18:05] better uh data so that they will give [18:07] you a better results. So again these [18:10] don't go by these embedding models uh [18:12] because we have those are the um what is [18:15] the leaderboard models you have to [18:17] experiment with your uh system and see [18:20] that which one works for your use case. [18:23] So that's a better uh embedding models [18:27] and um yeah the new embedding models [18:30] they are trained on the new data so they [18:32] are works better always. So again the [18:35] don't go by the benchmarks use your data [18:37] to experiment and see that which one [18:39] works and sometimes if you are data [18:42] privacy is a concern for you you cannot [18:44] use the u commercial models you have to [18:47] use your own um like the open source [18:50] models to deployed onto your system. So [18:53] the second one uh other one is that [18:55] advanced retrieval strategies. [18:58] So the first one is like like applying [19:01] the some kind of similarity threshold [19:03] like saying that whenever I retrieve the [19:05] documents from your system I wanted only [19:09] the documents those are like 80 [19:10] percentage or 70%age similarity. So [19:13] those kind of similarity thresholds you [19:15] can put it so that you can filter out [19:17] the lower similar documents. [19:20] Uh the second one is the multiquery [19:22] retrieval. [19:24] So assume that your your users are [19:26] asking a very vague questions. They they [19:30] are not giving a right uh questions. So [19:33] your system should be able to understand [19:35] your system should be able to spot and [19:37] rephrase that user questions in such a [19:39] way that it will give you you can [19:42] generate a multiple variations of the [19:44] user question. So that these multiple [19:47] variations of the questions will help [19:49] you to get the better documents from [19:52] your store. so that you can uh your [19:55] response will be improved over the time. [19:57] So that's what the multi-query [19:59] retrieval. So and the other is in a [20:02] hybrid search. Um sometimes the semantic [20:05] search may not solve your challenge uh [20:07] solve your problems. Let's say you are [20:09] searching for a specific uh system [20:12] configuration say RAM or uh hard disk [20:15] size the semantic meaning will not help. [20:17] You need an keyword search. So the [20:20] hybrid search combining the semantic [20:22] search with the keyword search is in a [20:24] way to get the better retrieval. So if [20:27] you look at if you observe that one all [20:29] we are trying to solve the retrieval [20:30] problem. Rag is a problem of retrieval. [20:34] If you can solve the retrieval problem [20:36] you will solve your rag challenges. [20:39] So other one is the reanker models. So [20:42] reerankers are the these are the cross [20:44] encoder models. These are special [20:46] trained models. They will tell you that [20:49] the documents that you retrieve from [20:51] your store how much they're relevant [20:54] relevancy it's not the semantic meaning [20:57] they'll tell that how much the document [20:59] related to the question that you are [21:00] asking. So that's a reancher models you [21:03] can use that and so that you can filter [21:06] out the documents or like bring the [21:08] documents which are more relevant to [21:10] your answer your question to the top. So [21:13] that's all the reanking and the chain [21:15] retrieval that we saw that like hybrid [21:17] search like combining the keyword and [21:20] semantic search along with the re uh [21:22] reanking is chaining together these [21:25] steps so that you can use these [21:28] strategies to get a better documents [21:31] from your store so that the answers [21:33] generator will be improved. Yeah. So the [21:36] reanchankers are as a cross encoder [21:38] models. So they take the [21:40] documents and your question and will [21:43] tell that how much the relevant the [21:45] documents to answer your question so [21:47] that you can uh you you can either [21:50] consider the documents or leave the [21:54] documents while generating your answers [21:56] to your questions. [21:59] So the other one is the contextual [22:01] compression. So we we saw that like if [22:03] you have any too much information in [22:05] your um context the model will get [22:08] confused right even as a humans we get [22:10] confused too much information we cannot [22:12] take right so the same thing with the [22:13] model so what we can do that we can [22:15] apply some kind of uh contextual [22:17] compression techniques. So there are two [22:19] ways we can apply the contextual [22:21] compression one is an extractor other is [22:24] in a filter. Extractor is like you read [22:28] the documents from the documents you [22:30] will filter out the content which you [22:32] don't feel that is not related to your [22:34] question. So that's an extractor concept [22:37] and the filter out is like you [22:38] completely if certain document certain [22:42] section of the document is not relevant [22:43] then you filter out that one. Again this [22:45] is you have to experiment with your data [22:48] to see that which which technique works [22:50] for you. A a simple explan example given [22:53] here. So you if you are from India you [22:55] know that uh India has three or four [22:58] capitals. If you ask a question a very [23:00] vague questions it could be any one of [23:02] them but the what the user intent is [23:05] like actual capital like the [23:07] administrative capital of country. So [23:09] with the contextual compression we we [23:12] retrieve those three documents we send [23:14] to the contextual compression. So the [23:15] LLM uses its knowledge to synthesize and [23:18] okay the the last two documents are not [23:21] relevant so that it will filter out and [23:23] it will give an a right document so that [23:25] you send these right documents to your [23:27] final response generation so that it [23:29] will give a more contextually relevant [23:31] answer. Okay. Um so the other challenge [23:37] is [23:39] wrong format. So have you seen that you [23:42] are asking the model to generate a [23:44] response in a certain way but it's [23:46] generating the resp. [23:49] So let's say you wanted you wanted to [23:51] read some document and extract the [23:53] information in certain JSON or XML [23:55] format it's not able to do that one. So [23:58] that's an uh another challenge that's [24:00] what the wrong format in response [24:02] generation. [24:04] So the solutions for the wrong format is [24:07] like use the latest models like the GPD [24:11] 40 for they are inbuilt supporting these [24:14] uh different uh file output format. So [24:17] that if out of the box their models are [24:19] giving just leverage them. So the second [24:22] one is like the uh using the better [24:25] output parsers or uh better prompting or [24:27] output parsers. These are the libraries [24:29] you can use that to uh control the [24:32] output generated by the answers of your [24:35] rag systems. [24:37] Uh and the other one is the structured [24:39] output framework. There are a lot of uh [24:41] frameworks libraries available for you [24:42] to try it out. Uh in in fact uh I tried [24:46] some of the frameworks and it gave me a [24:48] better results. [24:50] So as I mentioned like the solution for [24:52] the native LLM support is wrong format [24:55] is the native LLM support. If your LLM [24:58] your model is able to do that one, use [25:00] the better prompt to generate uh [25:02] directly to give an output in the way [25:04] you expected. [25:07] Um [25:09] and this other one is like the lang [25:12] chain uh have a lot of uh parsers [25:16] converters. It will take the information [25:18] that uh generated by the LLMs and gener [25:21] convert into a format that you are [25:23] looking for it. So lang chain has an out [25:26] of the box string parser, CSV parser, [25:29] pyantic parser, JSON parsers. You can [25:31] experiment you can take them out of the [25:33] box of the lang chain to parse that or [25:36] else you can lchain have any giving [25:38] those implementation you can extend them [25:40] and then uh convert into your format [25:43] that you the way you wanted to do that. [25:49] Uh so the other solution is that um [25:52] these structured output frameworks. So [25:54] one framework I used is the fructose [25:56] that gave a better uh results for [25:58] generating a better responses for my [26:01] some of the use cases that I have worked [26:03] on it again. So these are the frameworks [26:06] you have to experiment and see that [26:08] which one works for you. [26:13] Yeah. So the next challenge is [26:16] incomplete response. So the model is [26:19] generating an answer but it's not an [26:21] accurate that user uh expected. So it [26:24] could be because of the user asked the [26:27] question very vague so that the model is [26:29] not able to do the correct reasoning and [26:32] it's not able to pull the right [26:33] information to answer your um answer the [26:37] questions. So that's another challenge. [26:39] This this challenge is in the output [26:41] generation stage. So some of the [26:44] solutions uh for the incomplete response [26:47] is like always having a better models [26:50] latest models will always help you and [26:53] use the advanced prompting techniques [26:55] like the chain of thought or [26:56] self-consistency to improve the [26:59] reasoning of your the user's question [27:01] before generating an answers and agentic [27:05] systems with the tools necessary so that [27:07] they will pull the information if it is [27:09] not available in your system where you [27:11] are [27:12] And the other one is a rewrite uh user [27:15] query improve the retrieval. So this is [27:17] an hide like hypothetical document [27:20] embedding. We'll see that how we can do [27:22] that one. So this is in a hypothetical [27:25] document embedding. So here you ask the [27:28] question and you you you take the help [27:32] of the model to generate some [27:34] hypothetical answers to the questions so [27:37] that it will generate some answers. So [27:40] you take that answers and you convert [27:42] them into the embeddings and then you [27:44] compare those embeddings with your data [27:46] store. Sometimes this kind of comparison [27:49] will help you to retrieve a better [27:51] documents from your store. So then you [27:54] take those documents and then send it to [27:56] the rag final prompt saying that [28:00] generate an answers given this contact. [28:02] So this is the one way you can improve [28:04] the answer generation. So basically [28:07] improving the retriever quality. [28:10] So some it it worked for some of the use [28:12] cases better way but again experiment [28:15] and see that it will works for. So these [28:17] are some techniques so that we have to [28:18] see that which one works for the use [28:20] case. [28:23] So um other practical solutions from the [28:26] recent uh research papers is like we [28:30] might have here that rag versus long [28:32] context LM. the long context LLMs a lot [28:35] of people saying that rags are absolute [28:38] but uh in my uh what I observed that [28:42] these long context LLMs are expensive in [28:46] terms of like if you put all the stuff [28:48] into the context then you will consume a [28:52] lot of tokens right there is a new [28:54] concept coming the tokconomics it's not [28:55] an economics so with LLMs people are [28:58] calling tokconomics so that you will [28:59] spend a lot of tokens if you put a lot [29:02] of information in your context text. So [29:05] the better approach I observed is that [29:08] hybrid approach combining the rank uh [29:10] rag versus long context LLM is the way [29:13] to generate a better build a better rag [29:16] systems. [29:21] So here is an a implementation how you [29:24] can implement an rag versus long [29:26] contextual element. It's a hybrid [29:28] approach where you first ask the [29:30] question and you generate an answer [29:33] using the system and see you you will [29:36] take that answer and see that one you [29:38] will ask and again the model whether [29:40] these documents are like these documents [29:43] help me to answer the user as the [29:46] question then the LLM will say that [29:49] using the decide node saying these [29:52] documents whatever you retrieved will [29:54] help you to answer the question then you [29:57] take those documents and give it to your [29:59] final model to generate a response. If [30:01] the LLM says that these documents will [30:04] not help you to give in a right [30:06] information, generate a right answers [30:09] then you combine all these documents [30:11] like you take the entire document stuff [30:12] it into the context and send it to that [30:14] that's what you are taking the help of [30:17] the long context LLM to generate a [30:20] response. So that that's an a is like a [30:23] hybrid approach combining the rag versus [30:25] long context lms and um some of these uh [30:29] models coming up with the hallucination [30:32] checks inbuilt like the GPT4 has [30:34] hallucination checks inbuilt so that's [30:37] you can leverage the models [30:38] hallucination checks or else you can use [30:40] some tools or libraries like the um [30:44] links from the petronus is a way you can [30:47] check the hallucinations of your system [30:49] that you you are link like the respond [30:52] generated before you give it to the [30:54] answers. [30:56] So this is another way you can check the [30:58] hallmission checks. [31:01] So um if you look at all these things [31:05] the rag is still a pretty much a [31:09] retrieval problem. The retrieval the [31:12] better the retrieval the better the rag [31:15] system generation. So if you can solve [31:17] the retrieval challenge the 90% of your [31:21] racks are solved [31:24] and u always have an evaluation metrics [31:28] with you. So the evaluation data sets to [31:31] evaluate your system continuously see [31:34] that whether they are uh improving their [31:36] performance or degrading. [31:39] Um and the other one is like uh try [31:43] experiment with the different chunking [31:44] retrieval strategies. That's what we saw [31:46] that hyperparameter tuning. So that uh [31:49] don't go by the default settings. Try [31:51] with the different values so that you [31:53] can uh see that which parame uh which [31:55] settings works for you. [31:58] And uh even with the long context LLMs, [32:01] rag is not going anywhere. Rag still uh [32:04] going to be there for long context LLM. [32:07] That will help you to save lot of cost. [32:11] And the final win is agentic rack [32:14] systems where you can um I mean it's [32:18] going we are seeing a lot of agentic [32:19] rack system these days and the future is [32:22] going to be more domain specific uh rack [32:24] system like the fine-tuned rack systems [32:27] they are coming so that's what the [32:28] future for the rack [32:31] um that's it from my side and in case if [32:34] you wanted to connect with me this is my [32:36] LinkedIn uh I may not able to take the [32:38] uh questions I have to rush through I [32:40] have an announcement of one of I'm [32:42] helping one of the founder so you can [32:44] connect with me and I'm happy to answer [32:46] over the LinkedIn. Thank you.