CDC, Tech Leaders Back RAG Over Giant Models for Enterprise Data

CDC与科技领袖支持企业数据采用RAG而非超大模型 超大上下文窗口无法取代检索增强生成，因为对数百万份专有文档建立索引需要针对性的潜在空间搜索与专用模型，以防止产生幻觉并保护私有数据。 专家小组探讨了企业级RAG在运营、数学及架构层面的现实情况，将其定义为严谨的上下文工程，而非单纯的语义向量匹配。结合实际应用案例——例如CDC符合FedRAMP标准的文档处理流程以及GB Corp的汽车备件咨询系统——发言嘉宾剖析了多项系统性故障点，包括简单粗暴的top-k上下文填充，以及基础向量搜索无法处理定量聚合查询的问题。他们详细介绍了多层次的评估框架，该框架相较于通用的LLM评判机制，更优先考虑遥测技术、向量追踪和人机协同验证。最后，专家组针对不断扩大的上下文窗口是否会让RAG过时的争论进行了探讨，并得出结论：对于兼具成本效益与安全性的企业工作流程而言，潜在索引、专用小模型以及严格的数据治理依然不可或缺。 RAG 本质上是一项上下文工程（context engineering），通过将企业私有数据精准投递到大语言模型的上下文窗口中，无需耗费高额成本调整基础模型权重即可实现专有信息增强。 (0:00) 传统向量 RAG 假设数据处于扁平且孤立的孤岛中，难以应对企业内部复杂的关系型与层次化结构，因此需要演进为知识图谱（Knowledge Graphs）和多步骤智能体 RAG 架构。 (4:33) GB Corp 展示了企业 RAG 的落地标准，包括三层基础数据架构及五维评估支柱（算法指标、用户反馈、A/B 测试、实时遥测大盘和自动化数据刷新），将汽配查询耗时从 4 至 8 分钟大幅缩短至 10 秒以内。 (7:55) 单纯的语义向量检索无法处理定量聚合（如“有多少份文档提及某内容”），CDC 与 Meibel 指出此类统计和跨语料综合必须与结构化实体抽取、表格增强技术（TAG）或数据库模板相结合。 (22:17) 即使超长上下文窗口普及，面对 PubMed 等 3000 万篇量级的庞大专有语料，向量索引与 RAG 依然不可替代；结合小语言模型（SLM）与本地化微调更是保障数据隐私与降本的关键路径。 (30:02)

Retrieval-Augmented Generation (RAG) remains the primary enterprise mechanism for grounding large language models in private data without retraining model weights, though practitioners face major hurdles in context sizing, evaluation, and data topology. Industry and public sector leaders emphasize that enduring value depends on rigorous data foundations, specialized architectures, and verifiable context engineering rather than foundation models alone. Context Engineering and the Mechanics of Retrieval (0:00) Kevin McGrath frames RAG as an exercise in context engineering—supplying an optimal context window to the foundation model so it has enough signal to answer queries without drawing attention to irrelevant noise. Travis Hoppe notes that beneath the application layer, multimodal assets are vectorized into a dense numerical latent space for mathematical similarity search. This dynamic injection bypasses the slow and costly process of training base models. Topological Constraints and Semantic Blind Spots (4:33) Traditional vector RAG assumes data exists in flat, isolated silos like individual PDF documents. Aditi Kothari points out that enterprise software environments involve complex graphs linking code, tickets, and architecture documentation, which require advanced structures like knowledge graphs and agentic workflows. When enterprise queries lack a single canonical answer, returning a standard top-10 result set can saturate prompts with misleading near-misses. A persistent point of user confusion occurs when users expect vector RAG to perform quantitative counts or corpus-wide synthesis, such as asking 'how many documents mention animal welfare?' Hoppe and McGrath explain that standard vector similarity cannot compute aggregates or handle exhaustive summarization without structured tabular extraction or predefined query routing, as seen when pre-processing chemical specification PDFs into structured databases. Enterprise Deployment and Real-World Evaluation (34:50) Real-world implementations demonstrate substantial efficiency gains alongside operational friction. Bishoy Sawiris highlights that GB Corp used a RAG service advisor to reduce automotive spare parts lookup times from 4–8 minutes to under 10 seconds. However, deployments surfaced challenges with semantic homonyms across business units, such as conflicting definitions of 'renewal' between leasing and insurance divisions, necessitating manual data curation. Measurement Frameworks Beyond Synthetic LLM Judges (15:22) Panelists caution against relying solely on generic LLM-as-a-judge scoring, which often fails to verify if proprietary context was truly utilized. Sawiris outlines GB Corp's five-pillar evaluation model, combining algorithmic metrics (Precision, Recall, F1), monthly user surveys, A/B testing, real-time query logging in Power BI, and automated ingestion refreshes. Hoppe observes that retrieval size selection ($K$) involves domain-specific trade-offs: the CDC can tolerate false positives in internal staff lookups ($K=3$), but public health systems cannot risk delivering irrelevant advice. In complex software engineering contexts, Kothari notes automated evaluations remain largely unstandardized. Potpie AI evaluates system performance through qualitative developer feedback and operational metrics, such as code acceptance and production merge rates. Architectural Evolution: Context Windows Versus Small Models (30:02) While expanding context windows beyond 100,000 tokens lead some to question RAG's longevity, Hoppe argues that scanning massive enterprise repositories—such as the CDC's PubMed collection of 30 million full-text papers—makes vector retrieval permanently necessary due to economic and attention limits. McGrath likens inference to the film *Memento*, noting that altering context prompts allows developers to update model behavior dynamically without the steep cost of altering baked model weights through fine-tuning. Addressing whether enterprise value will accrue exclusively to frontier model providers, McGrath and Hoppe advocate for task-specific small language models. For instance, the CDC deployed a lightweight BERT model running locally on a USB thumb drive to filter survey non-responses, illustrating that small on-premise models can satisfy strict privacy and regulatory standards without expensive GPU clusters. While Aditi Kothari suggests RAG may eventually be ablated as base architectures natively absorb retrieval tasks, panelists conclude that defensibility lies in proprietary data hygiene, domain-specific disambiguation, and user-centric workflows rather than generic monolithic models.

在由 Thomson Reuters Ventures 主持的 Ai4 研讨会上，来自 Meibel、Potpie AI、GB Corp 及美国疾控中心（CDC）的技术专家达成共识：检索增强生成（RAG）仍是企业在不改动模型底层权重前提下注入专有数据的核心架构，但其成功落地高度取决于底层数据治理、非孤立图谱拓扑的处理能力以及超越传统指标与大模型裁判的端到端评测体系。 上下文工程与隐空间的数学本质 (0:00) Meibel 首席执行官 Kevin McGrath 将 RAG 定义为一种“上下文工程”，即在工作流的关键节点精准向模型窗口投放最少但必要的信息，既能避免从头训练或微调基础模型的漫长周期与高昂成本，又能防止信息过载导致系统故障。McGrath 以电影《记忆碎片》作比：推理阶段的大语言模型就像每天失去短期记忆的主角，只能依赖身上的“纹身”理解世界；上下文工程便是动态生成的提示词与检索结果，业务逻辑变动时只需调整检索管道与提示词，无须逆向修改已固化的生产模型权重。 CDC 首席人工智能官 Travis Hoppe 则从物理学视角指出，RAG 并非魔法，模型无法原生解析多模态原始数据，而是依赖分词与向量化将企业异构信息压缩至共享隐空间中，形成稠密数值向量。这种数学表征构成了相似度计算的基石，也是后续所有动态检索与上下文编排的核心支柱。 突破孤立文档：拓扑结构与混合路由 (4:33) 传统向量 RAG 往往假定数据以孤立文件（如独立的 PDF 报告）形式存在。Potpie AI 首席执行官 Aditi Kothari 指出，真实企业数据具备深层层级与关联拓扑。在软件开发中，代码库由相互调用的函数、类与文件构成，并深度关联到 Jira 工单和系统架构设计；平面向量检索面对这种图状网络往往失效，必须升级为包含多步推理与知识图谱的智能体架构。 此外，向量检索天生无法应对聚合与定量查询。Hoppe 表示，用户向 RAG 提问“有多少篇文档提到了动物福利”等统计问题时，向量相似度无法实现穷举。针对此类盲区，McGrath 分享了其化工客户的案例：面对数十万份产品规格 PDF，团队通过前置实体抽取将物理参数存入结构化数据库；当系统捕获“在特定温度下干燥时间小于10分钟的产品有哪些”等定量意图时，智能路由层会自动调用预定义 SQL 查询，再结合定性语义上下文生成完整解答。 企业落地实践与五支柱质量评估 (7:55) GB Corp 首席数据官 Bishoy Sawiris 强调，RAG 必须依托操作层、基础设施层与消费层三级数据资产，并涵盖知识库、检索器、术语转换层、编排层和评估反馈五大模块。GB Corp 将该架构应用于汽车配件维修顾问，使跨系统查找零件的时间从 4 至 8 分钟大幅缩短至 10 秒以内。Sawiris 警告，初期落地常遭遇跨业务部门的语义歧义（例如“续签流程”在租赁与保险业务中定义完全不同），必须引入人工校验并严加审计基础数据卫生。 在质量验证上，Sawiris 提出了涵盖算法性能指标（查准率、查全率、F1 分数）、月度用户调研、A/B 测试、实时遥测监控（基于数据仓库与 Power BI 仪表盘）及自动化数据更新管道的五支柱框架。针对检索参数 Top-K，Hoppe 强调选取取决于容错场景：CDC 内部系统采用 K=3 可接受轻微噪声，而公众系统则承担不起虚假信息的风险。McGrath 与 Kothari 进一步提醒，不能过度迷信“大模型作为裁判”（LLM-as-a-judge），团队应跟踪输入向量在输出中的实际表征，并以代码采纳率、合并率等端到端实际业务指标作为衡量依据。 长上下文演进与专用小模型防御 (30:02) 针对超大上下文窗口是否会让 RAG 走向消亡，现场调研显示约 70% 的参会者依然认为 RAG 不可或缺。Hoppe 解释称，尽管现代模型上下文已突破 10 万词元，但面对 CDC 所拥有的 3000 万篇 PubMed 生物医学文献库，全量注入在算力与经济成本上均不可行，且过度填充上下文会导致注意力涣散并诱发幻觉。Kothari 则持有不同预测，从 AI 算法不断“消融”（Ablation）外部依赖的演进历程来看，当前的图检索脚手架未来可能被更强大的基础模型原生吸收。 面对投资人关于企业价值可能被 OpenAI 或 Anthropic 等上游模型商垄断的顾虑，McGrath 与 Hoppe 认为专用小语言模型（SLM）构成了企业的核心防线。受限于 FedRAMP 与严格的数据主权规定，敏感机构无法直接调用公有云接口。Hoppe 透露，CDC 曾训练一个仅有 USB 闪存盘大小的轻量级本地 BERT 模型，专门用于在本地高效过滤调查问卷中的非有效回复。McGrath 认为，采用 7B 或 40B 模型处理垂直任务，不仅能大幅节约基础设施与 GPU 开销，更是保护企业专有数据资产最安全、高性价比的途径。 专家一致认为，RAG 并非孤立的算法插件，而是贯穿数据清洗、拓扑建模与系统级验证的系统工程。尽管未来超大上下文与基础模型能力演进可能会吸收部分检索外壳，但在海量文献检索、严苛的数据隐私合规以及对事实确定性要求极高的企业场景中，以高质量专有数据为壁垒、结合结构化路由与本地专用小模型的混合架构，仍是不可替代的落地路径。

## Background 该内容由频道 Ai4 于 2025 年 8 月 29 日在 YouTube 平台发布，记录了一场题为“利用 RAG 解锁专有数据”的行业圆桌讨论。视频汇集了来自 Thomson Reuters Ventures、Potpie AI、GB Corp、Meibel 和美国疾控中心（CDC）的行业领袖与技术专家，共同探讨检索增强生成（RAG）如何赋能企业安全、高效地激活专有数据资产并转化为高价值业务洞察。 ## TL;DR 本次研讨会深入剖析了企业落地 RAG 系统的全链路实战经验与架构演进。专家们首先从数学潜空间与上下文工程的底层逻辑切入，探讨了层级网状数据、结构化数据与非结构化文档的处理痛点，并提出了融合知识图谱、TAG 以及智能路由的进阶方案。随后，嘉宾结合汽车制造、医疗公共卫生及软件研发等真实业务场景，分享了包含性能指标与用户调研的双轨评估体系、本地轻量小模型的落地优势以及防范上下文污染的工程实践。最后，研讨强调企业不应盲目迷信单一全能大模型，而应以高质量专有数据为护城河，构建敏捷、垂直且可度量的 AI 工作流。 ## Atmosphere & style 整场对话兼具严谨的技术深度与一线落地的实战智慧，现场互动活跃且不乏幽默机智的交锋。主持人节奏把控得当，四位来自政府、传统工业与 AI 初创企业的嘉宾各抒己见，既有物理学视角下的底层解构，也有踩坑后的真诚自嘲（如用户对 RAG 统计能力的认知偏差），生动展现了从理论构想到严苛工业化生产环境之间的碰撞与权衡。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=nm4apwB1pqs&t=0s) RAG 的核心本质是“上下文工程（Context Engineering）”，其目标不是盲目塞满窗口，而是寻求能使大模型获得准确答案的最小、最恰当上下文平衡。 - [04:13](https://www.youtube.com/watch?v=nm4apwB1pqs&t=253s) 传统 RAG 适合处理扁平孤立的文档，但面对大型企业内部具有深层级与网状互联特征的数据（如代码库、工单与架构规范），必须引入知识图谱或 Agentic RAG 解决拓扑依赖缺失的问题。 - [07:55](https://www.youtube.com/watch?v=nm4apwB1pqs&t=475s) 企业级 RAG 必须建立在扎实的数据三层底座（运营层、基础设施层、消费层）之上，并通过专业转换器（Converter）消除行业黑话与日常自然语言之间的语义鸿沟。 - [11:27](https://www.youtube.com/watch?v=nm4apwB1pqs&t=687s) RAG 用于锚定单一权威事实来源以抑制幻觉，但在大型复杂机构中，同一问题往往存在多个“情境正确”的答案，简单的 Top-K 检索存在抓取偏差与决策误导风险。 - [13:10](https://www.youtube.com/watch?v=nm4apwB1pqs&t=790s) 避免上下文窗口污染的关键在于克制；对于结构化数据应独立采用表格增强生成（TAG）或图谱检索，而非将所有未加工数据一股脑填入模型上下文。 - [15:47](https://www.youtube.com/watch?v=nm4apwB1pqs&t=947s) 工业级 RAG 需依靠五大支柱构筑可信底座：性能基准指标、终端用户反馈、双架构 A/B 测试、实时数仓监控看板以及自动化数据同步更新管道。 - [17:35](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1055s) 评估体系不能单一迷信“LLM-as-a-Judge”，必须追踪稀疏与稠密向量在最终生成中的映射、测算答案空间方差，并从全系统（Whole System）交付价值的角度综合评判。 - [21:48](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1308s) 业务用户常误将 RAG 当作聚合统计工具提出“有多少”或“全文概括”等诉求；原生 RAG 擅长语义检索而非数学聚合，必须引入结构化提取流水线或 MCP 协议协同解决。 - [27:36](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1656s) 面对海量非结构化资产（如长篇 PDF），成熟做法是离线阶段先行开展实体抽取转化为结构化属性，在线阶段通过意图路由结合结构化查询与定性生成。 - [29:49](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1789s) 超大上下文窗口无法淘汰 RAG；面对数千万篇规模的海量文献库，在数学潜空间中提取与查询匹配的核心表征（Latent Representation）是长期存在的刚需。 - [32:11](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1931s) 相较于将知识不可逆硬编码进权重的微调（Fine-tuning），RAG 如同电影《记忆碎片》中的动态刺青，允许企业在基础模型无状态的前提下敏捷更新与修正专有知识。 - [39:56](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2396s) 企业真正的护城河不是打造通晓一切的庞大全能模型，而是结合受保护的专有数据与严格护栏，采用按需选型的小模型（如 7B/40B 或 BERT）深耕垂直窄切口工作流。 - [43:02](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2582s) 面对技术快速演进，企业切勿与单一模型或阶段性架构深度绑定，应始终以沉淀与治理自身专有数据资产为核心，小步快跑落地并深度结合终端用户定性反馈进行迭代。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=nm4apwB1pqs&t=0s) 开场介绍与 RAG 的基本定义 论坛主持人、来自 Thomson Reuters Ventures 的合伙人 Joe Dormani 宣布本次专题讨论“利用 RAG 解锁专有数据（Unlocking Proprietary Data with RAG）”正式开始。在热场介绍环节中，Joe 逐一向观众介绍在场的四位重量级嘉宾：Potpie AI 首席执行官 Aditi Kothari、美国疾病控制与预防中心（CDC）首席人工智能官 Travis Hoppe、埃及 GB Corp 首席数据官 Bishoy Sawiris 以及 Meibel 首席执行官 Kevin McGrath。现场在念出几位嘉宾姓名时还穿插了幽默的发音校正小插曲。 ```mermaid flowchart TD accTitle: RAG 基础工作流程 subgraph S1["数据准备与检索"] A["企业专有数据源<br/>(文档 / 数据库 / 代码)"] --> B["向量化与切片检索<br/>(Retriever)"] end subgraph S2["上下文拼接与生成"] B --> C["上下文工程整合<br/>(Context Engineering)"] D["用户提示词<br/>(Prompt)"] --> C C --> E["大语言模型<br/>(LLM 生成)"] E --> F["高精准度输出结果"] end ``` Joe 为本次研讨奠定了基础定义：检索增强生成（RAG）是指在特定数据仓库中进行检索，并将检索到的事实信息注入大语言模型（LLM）的生成流程中。这一技术已成为业界帮助模型锚定事实（grounding）、提升生成质量并赋予其访问特定私有数据能力的核心手段。随后，Kevin McGrath 从生成式 AI 运行时平台（Runtime Platform）的实务视角进一步补充：RAG 的核心本质是“上下文工程（Context Engineering）”，即以最高效的方式为大模型供给最恰当的上下文信息。 - **上下文把控的关键平衡**：在调用大模型时，向上下文窗口注入的信息必须拿捏在最优状态，既不能提供过多冗余导致模型迷失，也不能提供过少而缺失关键依据。 - **无需重新训练模型的敏捷性**：企业无需承担高昂的自建模型或重新预训练成本，即可直接把自身私有数据融入工作流；但同时也要注意，“跑得太快也可能把系统搞崩（if you move fast, you can also break things）”。 --- ### [03:08](https://www.youtube.com/watch?v=nm4apwB1pqs&t=188s) 物理学与数学视角：RAG 的向量与潜空间表示 拥有物理学学术背景的 CDC 首席 AI 官 Travis Hoppe 从严谨的底层数学原理解释了 RAG 的工作机制。他指出，RAG 并不是什么黑盒魔法，其背后是极其精密的数学潜空间表示（Latent Representation）。机器学习模型唯一能理解的语言是数字向量（Vectors of Numbers），但现实企业中的数据却具有高度的多模态特性——既有文本、图像、音频，也有离散与连续的各类结构。 ```mermaid flowchart LR accTitle: 多模态数据向量化潜空间映射 subgraph MultiModal["多模态异构数据"] A1["文本文档"] A2["图像/音视频"] A3["结构化/离散数据"] end subgraph LatentSpace["潜空间映射"] MultiModal --> B["分词与嵌入<br/>(Tokenization & Embedding)"] B --> C["稠密数字向量<br/>(Latent Vectors)"] end subgraph VectorSearch["相似度检索"] C --> D["向量数据库检索<br/>(Vector Search)"] end ``` 为了让模型在海量数据库中快速定位相关内容，系统必须通过分词（Tokenization）与嵌入技术，将复杂杂乱的原始数据“揉成一串高维数字向量（mushes all the stuff together into a string of numbers）”。通过在统一的向量潜空间内计算用户输入向量与已有数据向量之间的相似度，系统才得以实现跨模态、跨语料的精准匹配。 - **多模态与数学本质**：输入给 AI 系统的本质始终是数值向量，RAG 是将非结构化现实数据映射到数学空间并完成近邻搜索的桥梁。 - **检索的实质**：利用数学算法对比待检索内容与知识库中各向量之间的几何距离，从而筛选出最贴切的相关切片。 --- ### [04:13](https://www.youtube.com/watch?v=nm4apwB1pqs&t=253s) 数据特性与企业级 RAG 的挑战：孤岛数据与层级互联数据 Potpie AI 首席执行官 Aditi Kothari 结合其在大规模企业代码库中构建自定义 AI 智能体的经验，深入剖析了 RAG 对数据形态的要求及其实际落地的硬伤。她指出，RAG 在处理孤立形式（Isolated / Siloed）的文档（如单一独立的 PDF 文件）时表现极佳，因为其逻辑是在不重新训练模型的前提下外挂检索源。然而，大型企业内部的真实数据形态几乎从不是彼此孤立的，而是呈现出极其复杂的层级化（Hierarchical）与网状互联（Interconnected）特性。 ```mermaid flowchart TD accTitle: 复杂代码库与系统元数据的网状层级关联 subgraph CodeHierarchy["代码内部层级"] F1["文件 (Files)"] --> C1["类 (Classes)"] C1 --> FN["函数 (Functions)"] end subgraph ProjectContext["项目与元数据关联"] J1["Jira / Linear 任务工单"] D1["技术文档 (Docs)"] S1["系统架构设计 (System Design)"] end CodeHierarchy <--> ProjectContext ``` 以企业软件研发为例，代码库中的函数、类与代码文件之间存在强烈的依赖关系，且代码之外还深度关联着 Jira 或 Linear 上的需求工单、技术架构设计文档以及部署规范。传统的朴素 RAG 仅做片段切块与向量匹配，根本无法理解并遍历这种深度的层级依赖与网状拓扑。Aditi 强调，要攻克这一企业瓶颈，必须引入知识图谱（Knowledge Graph）或智能体 RAG（Agentic RAG）技术来进行补充。 - **专有上下文对比通用互联网数据**：公网数据包含通用智能，但在企业级任务（如根据个人历史习惯制定定制化行程，或理解企业内部业务流程）中，必须依赖内部专有上下文。 - **传统 RAG 的局限场景**：面对具有明确从属关系、跨系统引用的网状数据时，扁平化的向量切片极易丢失上下游依赖关系，导致检索失效。 --- ### [07:55](https://www.youtube.com/watch?v=nm4apwB1pqs&t=475s) 企业级 RAG 系统的核心架构与关键组件 来自埃及 GB Corp（业务覆盖汽车制造与非银金融领域）的首席数据官 Bishoy Sawiris 结合一线实操经验，系统性梳理了构建企业级 RAG 方案所必需的数据底层基础设施与系统核心模块。他强调，要让 RAG 真正发挥业务价值，企业绝不能仅仅关注算法本身，而必须自底向上扎实搭建完整的数据三层支撑体系。 ```mermaid flowchart TD accTitle: 企业级 RAG 系统全栈分层架构 subgraph DataLayers["企业底层数据三层体系"] L1["1. 数据运营层 (Data Operational Layer)<br/>(客户/供应商/物料核心资产)"] L2["2. 数据基础设施层 (Data Infrastructure Layer)<br/>(数据质量、治理与架构)"] L3["3. 数据消费层 (Data Consumer Layer)<br/>(数据分析与业务即时赋能)"] L1 --> L2 --> L3 end subgraph RAGCore["RAG 核心执行组件"] L3 --> KB["知识库 / 数据仓库 (Knowledge Base)"] KB --> RET["检索器 (Retriever)"] RET --> CONV["生成转换器 (Generator / Converter)<br/>(专业术语与自然语言适配)"] CONV --> ORCH["编排协调层 (Orchestration Layer)<br/>(跨智能体与业务调度)"] ORCH --> EVAL["评估与反馈机制 (Evaluation & Feedback)"] end ``` 在数据地基之上，Bishoy 进一步拆解了 RAG 系统的核心组件，并特别提到了许多企业容易忽视的“转换器/生成器（Converter/Generator）”环节。在汽车行业，诸如车辆质保政策、发动机规格技术参数等文档充斥着密集的专业术语，转换器能够根据不同目标市场的业务需求，将生涩的工程术语精准翻译转换为易于理解的日常英语或阿拉伯语自然语言。最后，通过顶层的智能编排层（Orchestration Layer）平滑协调系统请求，并结合闭环的评估与反馈机制（Evaluation & Feedback），确保整体系统的高可用与可信赖度。 - **数据运营层（Data Operational Layer）**：聚焦构建高标准的核心数据资产，涵盖客户数据、供应商数据、物料与配件档案等。 - **数据基础设施层（Data Infrastructure Layer）**：负责全链路的数据治理（Data Governance）、质量监控与统一数据架构规范。 - **数据消费层（Data Consumer Layer）**：面向数据分析与终端业务，确保正确的数据能在恰当的时间交付给对应的用户。 - **术语转换器（Converter）的必要性**：解决行业内部专业术语与终端用户日常交互用语之间的语义鸿沟，实现多语言、跨语域的平滑适配。 ### [10:54](https://www.youtube.com/watch?v=nm4apwB1pqs&t=654s) RAG 系统的核心构建与评估双轨机制 Bishoy Sawiris 详细介绍了 GB Corp 在构建数据基础设施层之后，如何针对 RAG（检索增强生成）系统搭建两套并行的评估机制。为了持续收集使用者反馈并把控落地节奏，团队设立了月度与季度的评估闭环： - **月度用户调研**：按月向实际使用 RAG 系统的终端用户发放调研问卷，直接收集主观体验与准确度反馈。 - **季度实施评估**：基于季度周期推进工程实现层面的功能迭代与系统综合评测。 Bishoy 强调，在筑牢底层数据基础设施之后，这两套定期评估机制便构成了整个 RAG 体系平稳运行的核心骨架。 --- ### [11:27](https://www.youtube.com/watch?v=nm4apwB1pqs&t=687s) 为何需要 RAG：从单点事实到多重上下文真相的现实挑战 针对主持人 Joe Dormani 提出的基础组件补充问题，来自 CDC（美国疾病控制与预防中心）的首席 AI 官 Travis Hoppe 分享了他对 RAG 本质的深刻洞察。很多外界声音误以为 RAG 是一根“银弹”或万能魔法，但在技术底层，LLM（大语言模型）本质上是海量数据的预训练产物；模型记忆了海量事实，但也进行了大范围的推断泛化，而这正是幻觉（Hallucinations）的根本来源。企业和机构引入 RAG 的核心目的，就是将模型的输出锚定在权威的“单一事实来源”（Source of Truth）上。CDC 内部既有通用聊天机器人，也针对内部文档、技术规范等场景专门构建了 RAG 系统，旨在抑制模型幻觉并精准调配机构内部专属数据。 然而，现实场景远比技术演示复杂得多。Travis 指出，当问题只有一个明确答案时（例如“退货政策是什么”这类常见演讲示例），RAG 系统很容易发挥效用；但对于 CDC 这样的大型复杂机构，同一问题往往在不同业务情境下对应着多个“情境正确”（Contextually correct）的答案。如果系统缺乏精细的向量化切分与恰当的检索机制，直接提取前 10 个（Top 10）检索块，很有可能根本抓不到核心依据，最终生成一个看似相近却偏离正确的答案。在公共卫生等关键决策领域，这种“似是而非”的结果伴随着极高的潜在风险。 ```mermaid flowchart TD accTitle: RAG 与传统 LLM 检索生成对比流程 subgraph RawLLM["标准预训练大模型 (存在幻觉风险)"] A["用户复杂提问"] --> B["预训练权重记忆与泛化推断"] B --> C["生成看似合理但可能失真的答案"] end subgraph GroundedRAG["多上下文 RAG 架构 (事实锚定)"] D["企业复杂业务提问"] --> E["高精度切分与向量化检索 (Top K)"] E --> F["私有专属数据检索块 (上下文约束)"] F --> G["上下文窗口注入与受限生成"] G --> H["可溯源的高保真度输出"] end ``` --- ### [13:10](https://www.youtube.com/watch?v=nm4apwB1pqs&t=790s) 避免上下文污染：从 TAG 到多步推理的演进历程 Kevin McGrath 与 Aditi Kothari 随后从系统工程与认知演进的角度进行了深度探讨。Kevin 提醒大家警惕“上下文窗口污染”问题：企业数据极其繁杂碎片，在配置向量数据库、图数据库甚至带引用的混合知识库时，必须构建精巧的流水线以精确控制上下文注入。很多人误以为只要拼命拉大 Context Window、塞入尽可能多的内容就能解决问题，但神经网络很容易在冗杂信息的诱导下发生偏离，遍历到完全错误的分支节点。优秀的 RAG 系统核心在于“克制”——不给大模型喂入与其推理无关的信息。 此外，并非所有企业资产都适合直接做文本向量化检索。针对结构化数据，业内衍生出了 TAG（Table-Augmented Generation，表格增强生成）等独立架构。最佳实践是将结构化查询、图谱检索等能力置于标准 RAG 流程之外分别处理，在下游环节再完成结果组装，绝不能盲目把所有数据无脑丢进 LLM。 Potpie AI 首席执行官 Aditi Kothari 紧接着回顾了 RAG 范式的代际跃迁： - **一代 RAG（One-shot 单步范式）**：遵循简单的“读取、检索、响应”（Read, Retrieve, Respond）三步走，单次检索直接输出答案。 - **现代 RAG（多步推理范式）**：人类探寻信息从来不是一次到位的（例如了解一次大会内容，往往需要逐项切换板块、反复比对）。现代 RAG 已深度融合 Context Engineering（上下文工程），演进为带有推理思考的多轮迭代检索流程。 --- ### [15:47](https://www.youtube.com/watch?v=nm4apwB1pqs&t=947s) 衡量与信任：构建工业级 RAG 的五大工程支柱 面对主持人关于“如何科学评估并量化信任 RAG 系统”的提问，Bishoy Sawiris 体系化地分享了 GB Corp 在实践中坚守的五大指标支柱（Five Pillars）： ```mermaid flowchart LR accTitle: GB Corp 评估与监控 RAG 的五大支柱 subgraph Evaluation["评估与监控机制"] P1["1. 性能指标<br/>(精确率 / 召回率 / F1 分数)"] P2["2. 用户反馈<br/>(业务端直接评价)"] P3["3. A/B 测试<br/>(双系统架构效益对比)"] P4["4. 实时日志与监控<br/>(数仓落表 + Power BI 告警)"] P5["5. 持续数据更新<br/>(产品规约管道自动同步)"] end Evaluation --> Trust["建立可信的 RAG 业务底座"] ``` 1. **性能基准指标（Performance Metrics）**：在工程端严格追踪检索召回的单会话 Precision、Recall 及综合 F1 分数。 2. **终端用户反馈（User Feedback）**：直接捕捉实际业务人员在使用过程中的正负反馈信号。 3. **A/B 对照实验（A/B Testing）**：并行配置两套 RAG 架构方案，从业务收益转化和数据仓库性能消耗双重视角进行横向选型对比。 4. **实时日志与监控大屏（Real-Time Monitoring）**：数据分析师与科学家将每一次查询以及命中的检索文档，作为独立对象打标并留存于数据仓库；利用 Power BI 搭建监控看板，在业务方察觉到召回异常之前，先一步捕获指标下滑。 5. **数据持续更新流水线（Continuous Data Refresh）**：构建自动化数据管道，当产品规约（Product Specs）或客户条款发生变更时，自动化同步至数据仓库，避免模型基于陈旧信息作答。 --- ### [17:35](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1055s) Top K 权衡艺术与输出保真度校验（摒弃单一 LLM-as-a-Judge） Travis Hoppe 补充了评估体系中一个关键工程权衡点：输入给 LLM 的切片数量（Top K 参数）。Top K 的取值直接对应经典的信息检索权衡： - **K=1**：检索要求极高，极难确保精准度； - **K=10**：大幅提升了召回率，但引入了大量干扰噪声。 究竟选择哪个阈值，完全取决于业务场景所能容忍的错误类型： - **机构内部使用（例如 CDC 内部员工查询技术规范）**：设定 Top K = 3。内网用户拥有专业鉴别力，能够容忍稍高的假阳性（False Positives），多看两句冗余信息并不会造成破坏。 - **面向公众端（Client / Citizen Facing）**：必须严防假阳性；一旦向大众输出无关或误导性检索内容，将带来严重的公共声誉与合规灾难。 随后，Kevin McGrath 深入探讨了“LLM-as-a-Judge（大模型裁判）”在落地时的常见陷阱。当下行业流行直接用另一个裁判大模型来判定输出是否正确，但裁判模型根本无法核验内容是否真正吸收了企业的私有知识。如果企业 RAG 的目标是确保最终输出融入专有数据，团队就必须深入底层追踪输入上下文与最终输出之间的关联： - **稀疏向量与稠密向量对齐追踪（Sparse & Dense Vectors Tracking）**：严格检查注入上下文窗口的稀疏和稠密向量特征，是否真实映照在最终生成的文本中。 - **可能答案空间测算（Realm of Possible Answers）**：对同一问题调用 5 至 8 次，统计答案分布的方差与置信区间，确认其是否稳定锚定在核心私有数据内。 - **传统 NLP 与大模型交叉校验**：不要迷信单一的大模型评判，应结合经典自然语言处理指标对结果保真度（Fidelity）进行双向锚定，从而真正验证专有数据在上下文窗口到输出链路中的转化价值。 ### [20:30](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1230s) 评估的困境：定性反馈与全系统视角 评估（Evals）在生成式 AI 领域一直是一块难啃的硬骨头。Aditi Kothari 坦言，即使在与客户对接时，评估也极其棘手：“虽然生成式 AI 如今让大量系统实现了自主化，但评估本身却完全没有实现自动化，坦白讲，没人真正确切知道该怎么把这件事彻底解决。”评估理论上可以分为定性和定量两个维度，但现实中他们不得不极度依赖用户的定性反馈。在 Potpie AI 面向软件工程的实际业务中，系统最依赖的就是一线开发者的持续反馈——他们是否接受了 AI 生成的代码修改建议、有多少生成的代码被反复推翻（churn），以及最终有多少代码能够顺利合并并推向生产环境。 团队目前尚未建立一套完全自动化的评估框架，寄希望于未来技术能够填补这一空白。然而更关键的认知在于，评估的对象绝不能仅仅局限于 RAG 本身。在评估整个应用时，必须将注意力放在全系统（whole system）的表现上，单纯讨论 RAG 技术的独立功效并没有太大意义，真正的评判标准在于整个端到端系统能否交付预期的业务价值。 ### [21:48](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1308s) CDC 的落地实践与用户认知偏差：“到底有多少？” 主持人 Joe Dormani 随即引导讨论转向落地实操层面，邀请疾病控制与预防中心（CDC）的 Travis Hoppe 分享专有数据与 RAG 系统结合的具体案例。Travis 介绍，CDC 在 2023 年通过微软 Azure 向全体员工推行了生成式 AI，成为美国首个全面推行该技术的联邦机构。尽管这令人振奋，但由于政府机构有着严苛的 FedRAMP 认证以及运营授权（ATO）合规要求，市面上大量酷炫的商业现成工具他们完全无法使用，只能“眼睁睁看着别人用”，然后由内部团队基于底层模型从零自研前端界面以及整套 RAG 架构。 ```mermaid flowchart TD accTitle: CDC 内部 RAG 部署与查询局限对比 subgraph Ingestion["数据沉淀与自动化摄取"] A["专有共享文件夹<br/>(数千份技术规范/PDF)"] --> B["每周自动更新流水线"] B --> C["内网专用 RAG 知识库"] end subgraph QueryTypes["用户查询与系统表现"] C --> D{"用户查询意图"} D -->|"事实检索: 规范在何处?"| E["✅ 精确命中 Top-K 文本片段"] D -->|"统计聚合: 有多少份文档提到 X?"| F["❌ 原生 RAG 失效 (仅返回相关片段)"] end ``` 在落地过程中，CDC 最初由 IT 部门手动接入业务部门提供的上千份技术规范文档建立系统，随后开发了一条自动化流水线：员工只需将文档放置在特定文件夹中，系统便会每周自动拉取更新，并在内网直接生成专属问答机器人。然而，Travis 指出系统面临的最大痛点并非算法本身，而是用户认知的转变。RAG 极其擅长单点事实检索——例如在数万页的技术规范中精确定位某个规定；但用户最常提出的却是统计与聚合类问题：“比如当知识库有 2,000 或 20,000 份文档时，用户总喜欢问：‘到底有多少份文档提到了动物福利？’原生的 RAG 根本无法回答‘有多少（how many）’这种问题，它只会傻傻地返回前 10 个相关的片段。” - **联邦合规约束**：必须满足 ATO 和 FedRAMP 安全门槛，导致无法外采成熟 SaaS，所有交互前端与 RAG 底层均由 CDC 内部工程师独立开发。 - **自动化知识流转**：支持指定文件夹周级别增量更新，全天候向内网用户提供定制化 RAG 问答服务。 - **典型认知误区**：业务用户常常误将 RAG 当作聚合统计引擎或全文概括工具，频繁提出“统计总数”或“提炼两万份文档所有核心主题”的越界请求。 ### [24:25](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1465s) 破解聚合难题：结构化与非结构化数据的协同治理 针对 Travis 提到的“统计难题”，Kevin McGrath 补充指出，语言模型与原生 RAG 本质上是在向量空间中寻找语义接近的片段，除非语料库中刚好存在一句话明确写着“这里共有 20 份文档”，否则 LLM 绝不可能凭空统计出准确数字。用户在与模型交互时，面对的实际上是一个复合中间层（Interface）。如果要支撑“有多少”这类的提问，就必须在整体架构中引入结构化提取机制，例如借助模型上下文协议（MCP, Model Context Protocol）或外接结构化存储系统，将数据以可查询的形式固化下来，再将聚合计算结果作为上下文喂给大模型。 Travis 苦笑着表示赞同，他补充道，除了问“有多少”，用户还经常直接扔出整个文档库要求“帮我总结所有文档的主要核心思想是什么”，这几乎是“你能向 RAG 系统提出的最离谱、最错误的问题”。归根结底，如果答案没有以文本形式直接落在嵌入空间的语义匹配范围内，系统就只能靠运气猜测。想要实现高阶洞察，绝不能指望单纯依靠一次检索调用（one-shot）就能搞定，而是必须依靠跨系统协同的业务工作流。 ### [26:19](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1579s) 复杂工程上下文：从分块到智能路由 Aditi Kothari 随后从 Potpie AI 的业务场景切入，探讨了在极端复杂的软件开发生命周期（SDLC）中部署 RAG 所遭遇的技术阻碍。在他们的实际应用中，系统不仅要连接客户庞大的代码库，还必须打通 Jira（项目管理）、Notion（文档中心）、Sentry 以及 PagerDuty（故障警报与事件监控）等全套研发工具链。这种跨工具、跨模态的专有数据交织在一起，会产生极其庞大的上下文体量，“朴素 RAG（Vanilla RAG）在这里根本无法工作”。 为了解决海量上下文的干扰，团队采取了三步走策略：深度打标（Labeling）、精细切片（Chunking）以及智能路由（Intelligent Routing）。单纯对文档切片依然无法抵御无效信息的泛滥；特别是在多智能体（Agentic）系统架构中，一旦将无关上下文全部硬塞给 Agent，模型便会迅速迷失。因此，系统必须先对碎片数据进行语义与元数据标注，并在检索阶段通过路由器将上下文定向下发给负责具体任务的 Agent，以确保输出结果的高精度。 ### [27:36](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1656s) 工业级实践：以实体抽取与预处理重构 PDF 知识库 Kevin McGrath 以一家大型化工制造客户为例，详细解析了结构化改造的工程落地全貌。该企业拥有数十万份长年累月沉淀下来的产品规格说明书，基本全部以静态 PDF 文件的形式存放。他们曾多次尝试搭建配置管理数据库（CMDB），但均以失败告终。业务人员迫切需要在这批文档中快速查询高价值技术参数，例如不同产品适用的作业温度区间、干燥硬化速度等。 ```mermaid flowchart LR accTitle: 化工产品规范说明书结构化提取与混合查询流水线 subgraph Offline["离线预处理阶段"] A["数十万份化工 PDF 规范"] --> B["实体抽取 (Entity Extraction)"] B --> C[("结构化数据库 / 数值属性表")] end subgraph Online["在线交互与推理阶段"] D["用户提问<br/>(例如: 哪些产品在特定温度工作?)"] --> E["前端交互路由层"] E --> F["生成/匹配结构化查询语句"] C --> G["执行检索返回定量数据"] F --> G G --> H["融合上下文与定性提问"] H --> I["LLM 输出一致性答案"] end ``` 针对这种专业场景，成熟的做法是建立分层处理流程：首先对离线文档执行实体抽取（Entity Extraction），从原本非结构化的长篇技术文档中挖掘出结构化属性并妥善入库；当用户在前端发起“在某温度区间下能正常运作的产品有多少种”时，交互界面不会直接盲目地去向量数据库里扫全集，而是首先识别问题类别并触发预定义的结构化查询；最后，系统将数据库返回的定量数值与用户原始问题的定性语义合二为一，再交给大模型润色，呈现给用户一个连贯自然的终极答案。Kevin 强调，在表面上看似一次简单的“单次问答交互”，底层实际上运转着一套严谨的工程管线。很多团队习惯让数据静态沉睡，寄希望于每次提问都现场依赖向量相似度碰运气，这完全是本末倒置；提前对数据进行结构化预处理，或借助 MCP 灵活挂载补充信息，才是企业级 RAG 迈向可靠的必由之路。 ### [29:49](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1789s) RAG 的必要性与局限性：大上下文窗口时代的定位讨论 在谈到构建解决方案时，通过特定级别的信息预处理来精准匹配最佳上下文窗口是不可或缺的考量。主持人随即面向现场观众发起互动提问：“有多少人认为 RAG 系统在生成式 AI 的未来中是值得信赖、必要且不可或缺的？”现场大约 70% 的观众举手表示赞同。随后，主持人将这一话题抛给专家组：随着基础模型和生成式系统的飞速演进，RAG 到底有哪些局限性？在上下文窗口持续扩大的背景下，它会一直作为核心技术存在，还是终将被更先进的架构所淘汰？ Travis Hoppe（CDC 首席 AI 官）率先风趣地回应，自己来参加小组讨论前甚至冒出过“我要当个 RAG 黑粉、高呼 RAG 已死”的念头。他回顾了 RAG 诞生初期的背景：当时模型的上下文窗口极其狭小，只有大约 1000 个 token，RAG 的出现解决了无法向模型注入过多信息的燃眉之急。而如今上下文窗口动辄达到 100k 甚至更长，人们几乎可以在 Elasticsearch 检索后将整篇或大体量文档全部塞入。然而在实际生产中，现实往往介于两极之间： ```mermaid flowchart LR accTitle: RAG Latent Representation Workflow subgraph DataPool["海量数据源"] A["PubMed 语料库 (3000万篇全文章)"] end subgraph RAGProcess["RAG 检索层"] B["匹配查询 (Query)"] C["提取潜在表征 (Latent Representation)"] end subgraph LLMInput["大模型处理"] D["精炼上下文窗口 (Top Pieces)"] end A --> B B --> C C --> D ``` Travis Hoppe 指出，以 CDC 常用的 PubMed 数据库为例，其语料库包含 3000 万篇全文章，“永远不会有也绝不应该有那么大的上下文窗口来容纳所有数据”。因此，从海量数据中提取出与查询相匹配的潜在表征（Latent Representation）是永恒的需求。无论未来如何命名该技术，提取最核心片段的能力将始终存在。虽然围绕 RAG 建立的大量专属公司和庞大管线可能会随着技术演进而常态化，但 RAG 最终会转变为一个长期存在、不可或缺的成熟细分工具。 - **早期 vs. 现状 token 容量对比**：早期的上下文窗口仅约 1,000 tokens，而如今主流模型普遍支持 100k+ tokens 的超长上下文。 - **超大规模语料现实**：CDC 日常处理的 PubMed 医疗文献库包含超过 3,000 万篇全文，超出了任何无脑填充上下文窗口的可行范围。 --- ### [32:11](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1931s) 上下文工程对比微调：如同《记忆碎片》的动态记忆系统 Kevin McGrath（Meibel 首席执行官）完全赞同上述观点，并指出技术人员往往总想寻找一步到位的“单点魔法解决方案”（One-shot magic solution）。RAG 是当前将专有数据与 LLM 结合且无需重新训练、构建小模型（SLM）或自行微调的最佳路径。针对上下文窗口的扩大，他强调这是“一把双刃剑”： - **大上下文窗口的风险**：向模型输入过多无序上下文，反而会导致模型输出偏离预期、产生噪音，开发者应当追求“能够获得正确答案的最小上下文窗口”。 - **微调（Fine-tuning）的高门槛**：微调是将知识永久写入模型的“大脑”，一旦训练数据存在偏差，后续想要剔除或回滚旧知识将极其困难且成本高昂。 ```mermaid flowchart TD accTitle: Movie Memento Analogy for RAG subgraph Concept["《记忆碎片》类比机制"] A["主角每日醒来无短期记忆"] --> B["全身刺青与笔记作为当天上下文 (Context)"] B --> C["大脑首次根据刺青做出决策与行动"] end subgraph RAGSystem["RAG 动态管道优势"] D["LLM 基础模型保持纯净无状态"] --> E["检索动态注入最新私有数据"] E --> F["随时修改/修正检索数据 (无需重新训练模型)"] end Concept -.-> RAGSystem ``` 为了生动阐明 RAG 的灵活性，Kevin McGrath 引用了经典悬疑电影《记忆碎片》（*Memento*）作为类比：主角每天早晨醒来都没有短期记忆，全靠身上的刺青来重新拼凑当天的认知脉络，大脑每次都是“第一次”看到这些信息。RAG 管道就像这些可以随时修改的刺青——如果发现输出结果有误或者业务策略改变，工程师只需调整检索管道与上下文提示，第二天模型就能基于全新上下文运转，而无需耗费巨资推倒并重新微调模型权重。 --- ### [34:50](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2090s) 汽车工业实战指标、词义歧义性与技术演进反思 Bishoy Sawiris（GB Corp 首席数据官）结合汽车供应链场景分享了 RAG 的落地成效与现实瓶颈。在 GB Corp 内部，他们为售后服务顾问开发了备件咨询 Agent。过去业务人员需要跨多个孤岛系统、不同平台和股东数据库翻查匹配零件，平均查询耗时在 4 至 8 分钟；而接入 RAG 系统后，信息获取时间缩短至 **10 秒以内**，大幅提升了准确率与客户体验，显著解放了员工生产力。但与此同时，他也指出了两大核心限制： 1. **业务术语歧义（Understanding Issue）**：系统在冷启动阶段缺乏对跨业务线同名术语的理解能力。例如查询“续约流程”（Renewal process）时，在车险合作伙伴与车辆租赁合作伙伴的业务定义截然不同，必须引入人工介入（Human-in-the-loop）来规范与校准模型理解。 2. **底层数据依赖（Data Dependency）**：企业往往盲目追求上马 RAG，却忽视了自身私有数据资产的质量与治理水准，缺少高质量的数据输入就无法得到精准的输出。 ```mermaid flowchart LR accTitle: Evolution of AI Optimization subgraph PastEvolution["历史演进借鉴"] A["强化学习 (RLHF)"] -->|"DPO 论文提出消融机制"| B["直接偏好优化 (DPO)"] end subgraph CurrentRAG["当前与未来趋势"] C["基础 RAG"] --> D["知识图谱 + Agentic RAG 多步推理"] D --> E["更高维度的替代架构出现"] end ``` Aditi Kothari（Potpie AI 首席执行官）进一步从机器学习演进史的角度补充道，AI/ML 领域的发展本质上是一个不断“消融”（Ablation）的过程——过去人们一度认为 RLHF（基于人类反馈的强化学习）是模型对齐的唯一方式，直到 DPO（直接偏好优化）论文证明可以省去复杂的 RL 环节。因此，当前的 RAG 很大程度上也是一个阶段性的“补丁”（Patch）。虽然目前 RAG 已经融合了知识图谱（Knowledge Graphs）、Agentic RAG 以及多步推理流程，但未来极有可能出现全新的架构来彻底颠覆并替代现有的上下文处理模式。 --- ### [38:00](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2280s) 投资人视角的价值捕获担忧与垂类 Agent 护城河 这一讨论引发了主持人作为 AI 领域投资人的深层担忧：如果 RAG 仅是一个过渡阶段，且基础大模型能力持续跃升，是否意味着价值将全盘向上游的基础模型提供商（如 OpenAI 的 ChatGPT、Anthropic 的 Claude 等）汇聚？当现成的大模型能够解决绝大多数通用问题时，企业该如何通过专有数据（Proprietary Data）建立系统差异化（System Differentiation）与长期护城河？ Kevin McGrath 对此提出了不同的见解。他认为并非所有价值都会流向上游，主要原因在于： - **数据隐私与安全红线**：企业的核心私有资产绝对不希望也不可能直接上传暴露给上游大模型厂商作为训练数据。 - **拒绝单一全能 Agent（One Agent to rule them all）**：在绝大多数企业的真实业务工作流中，目标并不是在内部复制一个全知全能的 ChatGPT，而是需要高度定制、窄切口的执行单元。 - **垂直聚焦的生产力工具**：企业更需要的是带有强护栏（Guardrails）的专有 Agent，精准完成特定任务，例如实体抽取（Entity Extraction）、分类任务（Classification）或垂直精准检索，业务场景切口越收敛，应用落地的稳定性和成功率就越高。 ### [39:56](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2396s) 告别“单一全能模型”：专用小模型与垂直工作流的优势 在大模型研究不断演进的背景下，针对特定领域的专用模型与小语言模型（SLMs）正在迎来巨大机遇。Kevin McGrath 指出，随着前沿研究的深入，企业不仅能够更好地微调模型，还应尽可能选用参数量更小但能满足需求的模型——比如 70 亿（7B）或 400 亿（40B）参数的模型。这种做法直接减少了算力消耗、GPU 占用与基础设施成本，而模型的底层构建逻辑并不会因此受损。 Kevin 强调，企业在构建 AI 工作流时，应当避免试图打造一个“包办一切”的全能型 AI，而是将复杂流程拆解为细分的、专注特定动作的独立组件。将应用场景收窄不仅能大幅降低系统复杂度与试错成本，还能让开发团队灵活利用 Hugging Face 等开源社区最新、最前沿的技术成果，按需选型与组装。 - **按需选型（Right-sizing）**：优先评估 7B 或 40B 等轻量级模型是否能完成目标任务，从而大幅节省 GPU 算力与运营开销。 - **工作流模块化解耦**：将 AI 管道拆解为执行特定子任务的独立步骤，而不是依赖单一模型承载所有业务环节。 --- ### [41:05](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2465s) CDC 现场实践：本地部署与“杀鸡莫用牛刀”的实用主义 Travis Hoppe 风趣地调侃道，Kevin 的这套理念可能会让很多企业管理者和 CIO“感到头疼并直翻白眼”，因为大家都希望有一套“统领一切的单一模型（one model to rule them all）”可以直接全面铺开。但现实中，像美国疾控中心（CDC）这样的政府机构拥有大量来自各州和地方部门的专有数据，受限于极高的数据隐私法规与安全要求，必须具备在本地（On-Prem）部署和运行模型的能力。 ```mermaid flowchart TD accTitle: CDC 调查问卷分类微小模型架构 subgraph Input["用户问卷输入"] A["公众调查反馈文本"] end subgraph LocalModel["本地轻量模型 (On-Prem)"] B["BERT 分类小模型\n(可装入普通 U 盘)"] C{"回复有效性识别"} end subgraph Actions["后续处置路径"] D["正常业务数据入库"] E["无效/调侃回复标记\n('我是只猫' / '我讨厌CDC')"] F["引导用户补充有效信息"] end A --> B B --> C C -- "有效作答" --> D C -- "无效/非作答" --> E E --> F ``` Travis 分享了一个鲜活的案例：CDC 在两年前训练了一个轻量级的 BERT 模型，其唯一职责就是在公众调查中识别无效回复——例如分辨填表人是给出了真实回答，还是写了“我是一只猫”甚至“我讨厌 CDC”这类调侃与发泄言论。分类完成后，系统可以进一步引导用户提供有效答案。这个模型小到“能直接装进一个普通 U 盘”，完全无需占用昂贵且庞大的 GPU 资源。Kevin 对此高度赞同，称这正是“需要用凿子时绝不用大锤（not using a sledgehammer when a chisel is necessary）”的绝佳典范，避免了在本地硬扛类似 Llama 4 这种超大模型所带来的天价算力负担。 - **本地部署与数据隐私**：政府与受监管行业的专有数据严禁外泄，轻量化模型使纯本地（On-Premise）推理变得经济可行。 - **专用小模型的威力**：利用 BERT 等经典轻量架构解决高频、单一的分类任务，兼顾低成本与高稳定性。 --- ### [43:02](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2582s) 专家快问快答：落地与优化 RAG 系统的核心建议 在圆桌会议的最后环节，主持人 Joe Dormani 邀请四位嘉宾各自用简练的语言，为正准备实施或优化 RAG 系统的从业者提供核心建议： ```mermaid flowchart LR accTitle: 落地 RAG 系统的四项核心建议 subgraph Prep["准备阶段"] D["Bishoy Sawiris\n整理好数据底座\n从小目标启动并持续追踪"] G["Kevin McGrath\n锁定业务结果\n做好上下文与向量库工程"] end subgraph Exec["执行与评估阶段"] U["Travis Hoppe\n深入终端用户\n面对面调研与 UI/UX 结合"] F["Aditi Kothari\n守住专有数据核心\n不绑定单一特定技术或模型"] end D --> G G --> U U --> F ``` 1. **Bishoy Sawiris（GB Corp 首席数据官）**： - **第一要务是理清数据**：“在为技术进行任何投资或行动之前，务必先把数据底座整理好。” - **设定明确小目标并敏捷推进**：从具有明确业务目标的小型 RAG 项目切入，步步为营地追踪实际需求与运行状态，逐步迭代并扩展至更大规模的项目。 2. **Travis Hoppe（CDC 首席 AI 官）**： - **深入终端用户进行定性验证**：尽管可以使用“LLM as a Judge”等自动化与定性指标，但绝不能低估与实际终端用户面对面交流的价值。组织焦点小组座谈，邀请 UI/UX 团队共同参与，深入倾听人类用户的真实使用反馈，这是发现系统改进空间的关键。 3. **Kevin McGrath（Meibel 首席执行官）**： - **明确业务产出，反向度量**：切忌为了用 AI 而用 AI。必须先确定清晰的业务目标与预期结果，再从结果反向推导评估指标。 - **夯实工程架构与上下文设计**：在底层架构上，要认真规划向量数据库的存储与扩展机制、引用溯源（Citations）、Graph RAG 以及上下文工程（Context Engineering），确保输入给 RAG 的上下文质量可控且输出效果可被精准度量。 4. **Aditi Kothari（Potpie AI 首席执行官）**： - **守住核心专有数据，保持技术敏捷**：生成式 AI 领域的技术迭代极为迅猛，企业应将核心护城河建立在自身的专有数据资产上，切勿与某种特定的模型或单一技术形态过深绑定，因为今天的最佳方案明天就可能被更新的技术所取代。 ## Notable quotes > [00:00](https://www.youtube.com/watch?v=nm4apwB1pqs&t=0s) if you move fast, you can also break things — Kevin McGrath > [03:08](https://www.youtube.com/watch?v=nm4apwB1pqs&t=188s) mushes all the stuff together into a string of numbers — Travis Hoppe > [20:30](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1230s) 虽然生成式 AI 如今让大量系统实现了自主化，但评估本身却完全没有实现自动化，坦白讲，没人真正确切知道该怎么把这件事彻底解决。 — Aditi Kothari > [21:48](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1308s) 比如当知识库有 2,000 或 20,000 份文档时，用户总喜欢问：‘到底有多少份文档提到了动物福利？’原生的 RAG 根本无法回答‘有多少’这种问题，它只会傻傻地返回前 10 个相关的片段。 — Travis Hoppe > [24:25](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1465s) 帮我总结所有文档的主要核心思想是什么 — Travis Hoppe > [26:19](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1579s) Vanilla RAG 在这里根本无法工作 — Aditi Kothari > [29:49](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1789s) 我要当个 RAG 黑粉、高呼 RAG 已死 — Travis Hoppe > [32:11](https://www.youtube.com/watch?v=nm4apwB1pqs&t=1931s) 能够获得正确答案的最小上下文窗口 — Kevin McGrath > [38:00](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2280s) One Agent to rule them all — Kevin McGrath > [41:05](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2465s) one model to rule them all — Travis Hoppe > [41:05](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2465s) not using a sledgehammer when a chisel is necessary — Kevin McGrath > [43:02](https://www.youtube.com/watch?v=nm4apwB1pqs&t=2582s) 在为技术进行任何投资或行动之前，务必先把数据底座整理好。 — Bishoy Sawiris ## Mentioned Joe Dormani, Thomson Reuters Ventures, Aditi Kothari, Potpie AI, Travis Hoppe, CDC, Bishoy Sawiris, GB Corp, Kevin McGrath, Meibel, Microsoft Azure, FedRAMP, Jira, Linear, Notion, Sentry, PagerDuty, PubMed, Elasticsearch, Power BI, Hugging Face, OpenAI, ChatGPT, Anthropic, Claude, BERT, Llama 4, 《记忆碎片》 ## Video description Joe Dormani, Partner, Thomson Reuters Ventures Aditi Kothari, CEO, Potpie AI Bishoy Sawiris, Chief Data Officer, GB Corp Kevin McGrath, CEO, Meibel Travis Hoppe, Chief AI Officer, Centers for Disease Control and Prevention Abstracts In this panel, industry leaders will discuss how Retrieval-Augmented Generation (RAG) is transforming the way organizations leverage proprietary data to generate more accurate and contextually relevant insights. Panelists will share real-world examples of how RAG models are being implemented to combine internal data with powerful AI systems, driving improved decision-making and personalized solutions. The discussion will also cover best practices for integrating RAG into existing data ecosystems and customizing RAG systems for certain industry verticals. Finally panelists will address topics such as data privacy, security, and scalability, including techniques to test and evaluate RAG systems for accuracy and trust. Join this panel to learn how to unlock the full potential of your organization’s proprietary data through RAG to stay competitive in an AI-driven world. [Source](https://www.youtube.com/watch?v=nm4apwB1pqs)

[0:00] Kevin McGrath: if you move fast, you can also break things [3:08] Travis Hoppe: mushes all the stuff together into a string of numbers [20:30] Aditi Kothari: 虽然生成式 AI 如今让大量系统实现了自主化，但评估本身却完全没有实现自动化，坦白讲，没人真正确切知道该怎么把这件事彻底解决。 [21:48] Travis Hoppe: 比如当知识库有 2,000 或 20,000 份文档时，用户总喜欢问：‘到底有多少份文档提到了动物福利？’原生的 RAG 根本无法回答‘有多少’这种问题，它只会傻傻地返回前 10 个相关的片段。 [24:25] Travis Hoppe: 帮我总结所有文档的主要核心思想是什么 [26:19] Aditi Kothari: Vanilla RAG 在这里根本无法工作 [29:49] Travis Hoppe: 我要当个 RAG 黑粉、高呼 RAG 已死 [32:11] Kevin McGrath: 能够获得正确答案的最小上下文窗口 [38:00] Kevin McGrath: One Agent to rule them all [41:05] Travis Hoppe: one model to rule them all [41:05] Kevin McGrath: not using a sledgehammer when a chisel is necessary [43:02] Bishoy Sawiris: 在为技术进行任何投资或行动之前，务必先把数据底座整理好。

[0:00] Kevin McGrath: context engineering [3:08] Travis Hoppe: mushes all the stuff together into like a string of numbers [22:17] Travis Hoppe: how many documents mention animal welfare? [22:17] Travis Hoppe: summarize all 20,000 documents for main themes. [30:02] Travis Hoppe: I search off PubMed all the time. That is one of our corpuses. That's 30 million full-text articles... there will never be a context window large enough. Nor do you want to have a context window large enough. [30:02] Kevin McGrath: a blessing and a curse [41:05] Travis Hoppe: one model to rule them all [41:05] Travis Hoppe: I'm a cat [41:05] Travis Hoppe: I hate you CDC

[0:00] Right, [0:04] welcome everyone. Uh, my name is Joe [0:07] Dormani. I'm a partner at Thompson [0:08] Reuters Ventures and this session is [0:11] unlocking proprietary data with RAG. [0:14] We're going to be having a panel [0:16] discussion here. So, we'll do a quick [0:17] round of introductions. Then we have [0:19] some topics that we'd like to all [0:21] discuss uh and then we'll leave some [0:24] time for questions at the end. Um, so [0:27] unlocking proprietary data with rag is [0:29] is the name of the session. We have a [0:31] very uh great panel here. I'll I'll try [0:33] and get everyone's names right as as [0:35] close as possible. Uh, so we have Aditi [0:38] Kthari. [0:40] We have Travis Hope. [0:42] >> Hoppy. [0:43] >> Hoppy. All right. Close. One extra P. [0:45] Okay. We have uh Bashoy Saris. [0:48] >> Situs. [0:49] >> Zoitus. Okay. And then Kevin McGrath. [0:53] That one. That one I got. Okay, great. [0:56] And you guys can all introduce [0:58] yourselves when when we hand it over to [1:00] you guys. But just to start things off, [1:02] I think we wanted to start with a quick [1:04] definition of of rag. And and so when [1:07] we're talking about rag, what we're [1:09] talking about is the process of [1:10] searching a data repository uh and using [1:13] the data in that repository to uh power [1:17] the AI out generated outputs uh used for [1:20] for LLMs. And this is one of the [1:23] predominant techniques that [1:24] practitioners use to ground and improve [1:26] the quality of AI outputs by giving it [1:29] access to specific data that they want [1:31] LLM to to use. So that's what we're [1:34] going to be discussing today. And to get [1:36] the conversation started, um maybe [1:39] Kevin, you can start by telling us a [1:41] little bit more about what RAG is, why [1:43] it's valuable, and what has led to this [1:45] becoming one of the more popular [1:46] techniques in modern generative AI. [1:49] >> Uh sure, thank you. And uh Kevin again [1:51] Kevin McGrath uh CEO of Maybel uh we're [1:54] a runtime platform for generative AI um [1:58] and our customers you know rag is a part [2:00] of of what they use right it's it's [2:02] about contact uh engineering and getting [2:05] the context to the LLM in the most [2:07] efficient way possible right and rag [2:10] allows you to get the information you [2:13] want in the context and what you want to [2:15] do is provide it at the most optimal [2:18] point in the process that you're not [2:19] giving it too much and you're not giving [2:21] it too little to get the most desired [2:22] output. And why this is effective is you [2:25] can take your data that wasn't [2:27] necessarily, you know, trained in the [2:29] LLM beforehand. It's your proprietary [2:31] data. Ideally, it's something like that. [2:33] You can provide it to the LLM and you [2:35] can get the context back that you want. [2:37] Uh you can move fast with it. Uh but [2:39] like with all things, if you move fast, [2:41] you can also break things. Uh so rag is [2:44] a very very powerful tool for you to use [2:46] AI without you having to build models [2:48] yourselves right and so um I think [2:51] that's why it's become you know super [2:53] predominant over everything that we do [2:54] and uh rag pipelines are a big part of [2:57] you know how you successfully use [2:59] generative AI today. [3:01] >> Great. Any anyone want to add into that [3:05] definition or give some examples? [3:08] >> Feel like I want to jump in if that's [3:10] that's okay. So I'm Travis Hoppy. the [3:11] chief AI officer at the Centers for [3:13] Disease Control and Prevention, the CDC [3:15] as most of you know it. Um, that is a [3:17] perfectly good and valid definition. I'm [3:20] also a physicist on my background. It is [3:22] so I got to do the math. And I feel like [3:24] we need to have both definitions in here [3:25] because it's not magic underneath. It is [3:28] a latent representation of your data, [3:30] right? Machines understand things [3:32] through vectors of numbers and that's [3:34] how they need like that is the input to [3:36] machine learning models. That is the [3:37] input to AI, but your data is [3:39] multimodal. It's an image. it's text, it [3:42] is discreet or it's continuous or it's [3:43] audio. It's like all of these weird [3:45] things and you need to put it in a way [3:47] that it can understand it, right? We [3:48] have tokenization that helps bring it [3:50] into these things. But how do you find [3:51] something across a big database? You [3:54] make a latent representation. You make a [3:56] vectorzed thing that like mushes all the [3:58] stuff together into like a string of [3:59] numbers and then you can search your [4:01] mush to all the mush that you have, [4:03] right? You get this vectorzed latent [4:05] representation. And so there's like deep [4:07] math that underlines all the things [4:09] within a rag and that leads to all the [4:11] enablement that you were talking about. [4:13] >> Yeah, I think I think that's that's [4:15] great. It's a good foundation for us to [4:17] build a conversation on. Maybe building [4:19] off of that, we can talk a little bit [4:21] about um what type of data is useful um [4:26] and does the size of the the data does [4:28] the format of the data matter uh for it [4:31] to be useful in an AI system? Ad maybe [4:33] you can help us there. Am [4:37] I audible? Oh yeah, it works now. Hey [4:39] everyone, I'm Aditi Gotari, CEO of Party [4:42] AI. We build custom AI agents to [4:44] automate use cases across the software [4:47] development life cycle for enterprise [4:49] companies with very large code bases. So [4:52] you know to answer your question um rag [4:56] works really well with structured data [4:58] but it does not mean it cannot work with [5:00] unstructured data. Um however the major [5:04] constraint is that rag works with data [5:07] that is mostly isolated. So you know [5:10] think of a PDF right um LLMs are trained [5:13] on generic information available right [5:16] um but specifically for any enterprise [5:19] use case you would want to use rag [5:21] because you can train the same LLM or at [5:24] least retrieve information from a data [5:28] source without retraining that LLM and [5:30] that's where rag is useful unfortunately [5:33] that is not how data is present in any [5:36] enterprise company it's hierarchal It's [5:39] interconnected and rag is mostly useful [5:42] when the data that you want to retrieve [5:45] is you know in iso in silos isolated [5:49] form. So that's where you know rag has [5:52] limitations which obviously can be [5:54] solved by you know things like a [5:56] knowledge graph or an agentic rag but I [5:59] think more upon that later. Yeah, that [6:02] that's great. And and maybe just to to [6:04] build on that, what are some of the [6:07] types of data that can be used and why [6:09] would you use, you know, let's say [6:10] internal data versus maybe data that [6:13] exists externally on the internet? Like [6:15] what are some of the different pros and [6:17] cons of different types of data and how [6:18] they could be used can start and anyone [6:21] can build. [6:22] >> All right. Uh maybe I'll just uh take [6:24] this up. So uh this completely depends [6:27] on the use case. Of course, there's a [6:29] lot of general intelligence available as [6:32] information on the internet, but for any [6:34] specific use case that you might want to [6:37] work on with a LLM, let's say it's [6:39] about, you know, building an itinerary [6:42] for, you know, taking a trip to maybe [6:45] Spain. Um, the model really has to [6:48] understand what kind of traveler am I [6:51] and what do I really like doing in a [6:53] trip? Am I someone who likes a relaxed [6:55] trip? and that can be actually retrieved [6:58] from my previous travel history and [6:59] itineraries. That is where rag is super [7:01] useful. So um think of the same scenario [7:06] but in a more large enterprise company [7:08] where there is so much contextual [7:10] knowledge which is obviously proprietary [7:12] and not available on the internet and [7:15] you can le never leverage that [7:17] information unless you have that context [7:20] share which is where rag is super useful [7:22] but again this information is hierarchal [7:24] it's all interconnected there are [7:26] relationships specifically in our case [7:28] for a codebase uh there are [7:31] relationships between functions, files, [7:33] classes and even beyond that right your [7:36] codebase connects to all your um you [7:39] know let's say your Jira or linear [7:41] tickets your documentation your system [7:43] design but in that scenario rag is not [7:47] useful because it cannot um you know [7:50] retrieve through hierarchal data or [7:53] interconnected data [7:55] >> very very helpful maybe maybe now just [7:57] to sort of zoom out and help create a [7:59] bit of a framework for for folks [8:01] Can we boil down maybe the core [8:04] components of a rag system that are [8:06] necessary for it to be useful and and [8:09] valuable for for enterprise or for [8:10] consumer application or for whatever [8:12] type of AI system. Maybe Bisho you can [8:15] start help us frame that up. Uh I am shy [8:19] chief data officer of GB Corp uh in [8:22] Egypt and GB Corp it's an focusing in [8:25] automotive industry and and [8:27] non-financial banking sector [8:30] uh I believe the main uh components of [8:33] rag uh it's uh starting from the [8:35] knowledge base and data sources so uh I [8:39] will answer from my experience as a [8:42] chief data officer it's a a main [8:45] important things in implementation of [8:47] the RO in order to make it uh it happen [8:51] we need to start with the data itself. [8:55] So uh I will describe the three layers [8:58] of data in order to build any technology [9:02] and AI and rag system and whatever [9:04] technology starting from data [9:06] operational layer which is focusing in [9:08] data assets. how to build in data assets [9:12] in in in good way with a good quality [9:14] such mean uh customer data, supplier [9:17] data and item data and so on. Another [9:20] thing uh to build and data for data uh [9:24] infrastructure layer which is focusing [9:26] on data quality and data governance uh [9:29] and building data architecture inside [9:31] way. Another layer which is data [9:34] consumer layer which is building for the [9:37] data analytic and so on which is uh uh [9:40] to help the user for getting the right [9:43] information in the right time. After [9:45] building the three layer of the data you [9:47] can build the knowledge base such as [9:50] mean and data warehouse and so on. [9:52] Another thing to uh main component of [9:55] the retriever how to retrieve the data [9:57] from the data sources and the main [10:00] component and the missing in many [10:02] company which is generator or converter. [10:06] Uh I will give you an example from the [10:08] automotive industry we have uh document [10:11] for warranty uh uh policies uh and specs [10:15] techniques for the car. So when you have [10:17] a case to retrieve the rack system the [10:21] converter will change the technical [10:24] terminologies to the English or Arabic [10:26] language as per market uh uses. The last [10:29] one it's an orchestration layer which is [10:32] building for the high uh level of data. [10:36] So the main focus of the orchestra layer [10:38] to building uh how to smooth the the [10:42] same request to the uh the system [10:45] whatever agent is mostly to to help the [10:48] operation layer. The last one which is [10:51] uh evaluation and the feedback. So we [10:54] are building in our company uh two way [10:56] for the evaluation. We are sending u a [11:00] survey uh on monthly basis to the user [11:04] who are using the rock system and the [11:06] implementation way will be building on [11:08] the quarterly basis. I believe this is [11:10] the main uh components of the rock [11:12] system after building the data uh uh [11:15] infrastructure layer. [11:17] >> I think that that that's a great answer [11:19] very detailed. Any other foundational [11:21] components anyone would want to add? If [11:23] not, I think there's a there's a good [11:25] followup. [11:27] Yeah, I I got a really short one. Um all [11:30] of this and like that was a great [11:31] explanation I think of the core [11:33] component of this. Um something I think [11:36] is missing when people are like oh rag [11:38] it's a magic thing like why why rag like [11:40] why do we need it ultimately is that you [11:43] know if you ask a large language model a [11:45] question they are pre-trained and so [11:47] they have memorized a lot of facts they [11:49] have generalized even more facts and [11:52] that's where the source of [11:53] hallucinations come from. you ask [11:55] something is retrieving from those those [11:57] facts that are in there. You want to [11:58] ground it in a source of truth. So CDC, [12:01] we have an internal chatbot. Uh but we [12:03] also have a rag based system for things [12:06] like our documentations or our tech [12:08] specs or these things that like all the [12:10] other companies here are talking about. [12:12] Um and that is that is why we do it is [12:14] to reduce hallucinations on a generative [12:16] AI component and incorporate that that [12:19] specific data that's just for us. Um, [12:22] one thing I will say that is really [12:23] really tricky that comes up with our rag [12:25] based stuff is that rags work really [12:28] great when there's a single source of [12:29] truth for your answer. Yeah. You know, [12:31] like I saw like three talks today and [12:33] the three talks were like, "Oh, we're [12:34] going to look up uh what my return [12:36] policy is and we'll use a rag to find [12:38] the right thing." Yeah. There's like [12:40] maybe one answer to that, but like you [12:42] can imagine the complexity like of our [12:44] organization or some of yours that have [12:46] many different answers that are all [12:48] contextually correct if you ask the [12:50] right question. And if you don't have a [12:51] good rag that that does the [12:53] vectorization level at the right level [12:55] and all of these things that you need, [12:57] you'll return the top 10 results and [12:59] they won't actually answer your [13:01] question. They won't give the grounding [13:02] that you need and then you'll give an [13:04] answer that is close but maybe not [13:07] correct and that is a real danger in [13:09] these things. [13:10] >> Yeah, I think just just to expand on [13:11] that what Adidia said before um you know [13:15] rag is part of it. you your enterprise [13:17] data is all over the place optimizing [13:19] for that top K right that context window [13:21] if you throw too much into that context [13:23] window right you need this whole system [13:25] set up like you have a vector database [13:27] maybe you have a graph database maybe [13:28] it's not graph rag but maybe it's you [13:30] know has citations to different articles [13:32] and stuff like you need to build this [13:35] kind of efficient pipeline that gets the [13:37] right information at the right time and [13:39] brings it in because if you uh overthink [13:42] it or you throw too much information [13:44] into the context window everything in [13:46] that context window the brain is is I [13:49] it's it's going to see and there's a [13:51] chance that it traverses the neural net [13:54] to something that you don't want and so [13:57] the best rag as you're saying the best [13:58] rag system is not to confuse the LLM [14:00] with information that it shouldn't have [14:02] to bring up the answer and I think [14:03] that's one of the most difficult parts [14:06] of building a sophisticated rag system [14:08] that gets you the answers that you want [14:10] and then along your lines with the [14:12] enterprise data it's not everything [14:15] should be ragable, right? There's [14:16] structured information. Uh there's tag [14:19] out there, table augmented generation, [14:21] things like that. Like there are things [14:22] that you should handle outside of the [14:24] process of rag and then put those [14:26] components together at the end to get [14:28] your answer. And you shouldn't just be [14:29] throwing everything to the LLM at all [14:31] the time. [14:32] [Music] [14:36] >> I think that's spot on. Um when you know [14:39] Rack started initially, it was three [14:41] steps. You read, you retrieve, and you [14:44] respond. Right? And it was a oneshot [14:47] process. But that's not how humans ever [14:51] access information. If I were to ask [14:53] you, you know, um, tell me everything [14:56] about AI4, you would never give me a [14:58] one-shot answer because you would go [14:59] through the whole app, go through [15:02] different sections, and to make it more [15:04] humanlike. Today, Rag is more of a [15:06] multi-step process with reasoning [15:08] involved. So, it's never one shot. And [15:11] that's what you know that's where the [15:12] whole context engineering bit comes into [15:15] picture where you make it a multi-step [15:17] reasoning process. So that's where Rag [15:20] has moved forward in time. [15:22] So really great answers all built on [15:24] each other and I think we'll get to some [15:27] of the uh trust questions when it comes [15:30] to LLMs and RAG. But maybe just first we [15:34] can put some meat around how do you [15:35] evaluate a RAG system? How do you [15:37] measure its quality? How do you what [15:40] types of metrics or things do you [15:41] measure to make sure it is a trustworthy [15:44] system? Maybe bish you can get us [15:46] started there. [15:47] >> Yeah. Um [15:50] uh I will complete to um the last answer [15:53] of this question. Without the kind of [15:55] data foundation we can building any [15:58] trust for the system. So we are uh uh [16:02] working on five pillar starting with the [16:05] performance metrics. So we are building [16:07] a KPI like uh per session uh F1 score [16:12] and recall and so on in order to uh [16:15] measure the KPI of the rack system and [16:17] so on. Another one of the user feedback [16:21] for uh who is using the rack system and [16:25] also and we are building and uh working [16:28] in AB testing. We are configuring two [16:31] rack system in order to see what is the [16:32] benefit for the business and what is the [16:34] benefit for the data from the [16:36] performance perspective in hunting the [16:38] data warehouse. The third one which is [16:41] the real time monitoring which is uh the [16:44] the main role of data analytic and data [16:46] scientist. So uh every query and every [16:50] retired document is logged and stored in [16:52] data warehouse in a separate object and [16:55] we are building uh a data analytic [16:57] product using a data the powerbi in [17:00] order to see what we can enhance before [17:03] the business uh let us to know we have [17:06] an issue on retrieval type of number of [17:08] data and the last one which is a [17:11] continuous data refresh. We have a data [17:13] pipeline automated in order to uh change [17:18] and to give an update to our data [17:20] warehouse with a product spec change and [17:22] the customer section change and so on. [17:25] So uh we are trusting the rack system [17:28] when are using the five pillar of the uh [17:31] performance and so on. [17:35] >> Okay. Can I add on to you? Um I I love [17:38] following you because you have such [17:39] great detailed answers on what to do and [17:41] I could just Yes. And the next piece um [17:43] the one thing that that changes for us [17:45] on evaluation and I'm sure this is part [17:47] of it is how many results do you want to [17:50] take from your rag and then give to your [17:52] LLM and you're like oh we can optimize [17:54] for this right do you want to give just [17:56] one result so like if a user within CDC [17:59] asks a question for our technical specs [18:01] and we have a rag just based for that we [18:04] might actually we've decided that like [18:05] the top three answers are the ones we [18:07] want to return to the user because the [18:09] rate of a false positive there It's [18:11] fine, right? These are internal CDC [18:13] users and if they get a little bit too [18:15] much information, they're not going to [18:16] be mad. They can look through this. But [18:18] if you have like a client sidef facing [18:19] thing, if we ever built a rag that serve [18:22] to the American public, we don't want to [18:24] return results that are irrelevant to [18:25] their query. There's harm in a false [18:27] positive there. So choosing the top K, [18:29] like the number of results that then go [18:31] back to the AI model is critical because [18:34] it's really hard to get the right result [18:36] with a K equals one, right? But it's [18:38] really easy to get it with a K equals [18:39] 10, right? This is the trade-off on [18:40] precision and recall shows up in rag 2. [18:43] And so the answer to what that is is not [18:45] just an optimization of a metric. It's [18:47] highly dependent and highly contextual [18:50] on what sort of errors you can absorb, [18:52] whether they're false positives or false [18:53] negatives. [18:55] >> And in addition to that, not only, you [18:56] know, getting the top K right going in, [18:58] but are you actually evaluating? So if [19:00] it's your data that you want to see [19:02] following up in the output, like an LLM [19:04] judge isn't necessarily going to tell [19:05] you that. So what a lot of people like [19:07] we're talking about fidelity on the [19:08] output like are you going to use an LLM [19:10] judge to do that or are you actually [19:12] going to track the sparse and dense [19:13] vectors that you fed in in the rag [19:15] process and see if those same sparse and [19:17] dense vectors are showing up in the [19:18] output. So it depends what is your [19:21] outcome. If your outcome is you want to [19:22] use your proprietary data and you want [19:24] to see your proprietary data in the [19:26] output are you actually tracking that [19:28] the vectors that you gave it to in [19:30] context are actually showing up in the [19:32] output and then you also have to measure [19:34] things. You have to measure things like [19:36] realm of possible answers. Like are you [19:37] asking the LLM once or are you asking [19:39] the LLM five, six, seven, eight times? [19:41] And in the realm of possible answers, [19:44] are those an is the answer the best [19:46] answer? Is it in context with what you [19:48] have? Is it grounded in your data? Um [19:50] there's a lot of different you can use [19:52] just standard natural language [19:53] processing to figure out correctness of [19:55] things alongside LLMs and alongside uh [19:59] how you're setting up your rag system. [20:01] And it's really important because [20:02] setting up your RAG system correctly to [20:05] begin with is all about the fidelity of [20:07] the output. And I think that's something [20:09] that we actually see is a lot of people [20:11] are using these LLM judges and just [20:12] trying to say like, hey, here's the [20:14] output. I I threw a few LLM judges at it [20:16] and they say like I'm generally correct [20:18] or whatever, but you're not linking it [20:20] back to your proprietary data that went [20:22] into the context window in the first [20:23] place. If you don't make that [20:25] connection, you don't actually know how [20:27] well you're doing. [20:30] I think if I'm being honest um eval [20:36] tricky for us um even when we are [20:38] interfacing with our customers while [20:41] generative AI has enabled making a lot [20:43] of systems autonomous today evals are [20:46] something that have not been automated [20:48] and no one really knows like how we are [20:51] going to figure this out but um you know [20:55] eval could be both qualitative and [20:57] quantitative and fortunately or [20:59] unfortunately ely today we rely a lot on [21:02] qualitative feedback coming from our [21:04] users. So in our case because the use [21:06] case is around engineering we trust [21:08] developers to give us continuous [21:10] feedback on you know if this is working [21:12] if they're accepting the code changes um [21:15] and how much of that code is being [21:17] churned or pushed into production. Um so [21:20] yeah that's pretty much it. uh we don't [21:22] have an automated um you know eval [21:25] system yet but hopefully you know in [21:28] future there will be a technology that [21:30] enables this but more importantly when [21:33] we evaluate the system we evaluate the [21:35] whole system and not just rag so it's [21:38] not really about the technology or the [21:41] approach but the whole system itself so [21:44] it's not really the efficacy of rag as [21:46] much as a system [21:48] >> great and so I think we have a a good [21:50] foundation definition of reggg. We've [21:52] understood why it's important. We've [21:54] understood why we have to test and [21:56] monitor it and the role it plays in [21:57] these systems. Maybe now let's make this [21:59] a little bit more practical. We'll talk [22:00] about some examples. Um maybe maybe [22:03] Travis, you can start talking about some [22:05] specific examples of rag systems that [22:08] you've deployed and and maybe [22:10] specifically highlight the proprietary [22:12] data that played an important role in [22:15] those systems and why that was so [22:16] important. [22:17] >> Yeah, happy to discuss that. So in 2023, [22:20] CDC rolled out generative AI through [22:23] Azure um to all staff, which was the [22:26] first federal agency to do so. Super [22:28] jazzed about that. Um but we had to [22:30] implement everything else, everything [22:32] ourselves. It was through an ATO. It was [22:34] fed ramped. And so like there was no [22:36] front end. We used chatbt, but we had to [22:38] write our front end, which means we had [22:40] to write the ragbased system for all of [22:41] this. So we were basically developing it [22:43] along with everything else. You guys [22:44] have all these cool tools that you're [22:46] developing. And I'm like looking at it, [22:47] love it. we can't use it. So, we [22:49] basically watch what everybody else was [22:50] doing and bring them in. Um, so like our [22:53] ragbased models, they're not as fancy or [22:55] as great, but what we noticed is that [22:57] the grounding for what you need, like if [22:59] there's a bunch of technical specs, um, [23:02] this was a useful tool. And so, we [23:04] basically built a process where our IT [23:06] department, somebody would up give them [23:08] a thousand documents on like a on a on a [23:11] proprietary share and say, "Build a rag [23:12] off this." And we would. And just a [23:14] couple months ago, we kind of automated [23:15] that pipeline uh where somebody could [23:17] just say, "Hey, all the documents in [23:19] this folder, I want you to update them [23:21] once a week." And now we have a chatbot [23:22] that you can deploy internally as a rag [23:26] uh with your own data on our internet, [23:28] which is like a super cool thing. And I [23:30] actually see some of the companies doing [23:31] things just like this. Um what I will [23:34] say and the biggest thing that I we have [23:36] to educate our users on it is great for [23:38] information retrieval. Like if there's a [23:40] sole source of truth, you want to ask a [23:42] question and it's like where is this in [23:43] the tech technical specs? Our users ask [23:46] all the time how many, right? Like if [23:49] you have 2,000 documents or 20,000 [23:51] documents and like how many documents uh [23:54] mention animal welfare, [23:57] rags won't tell you that. They just give [23:58] you like a top 10 or a top something. [24:01] And like this is the biggest complaint [24:02] and this is a huge shift in education. [24:04] It's like people want to ask how many. [24:06] You give them a corpus of documents. [24:07] They need to know this. Uh, and rags [24:10] will not do that for you. They could you [24:12] could build a system where they could [24:13] find them and tag them and do a [24:15] classification scheme, but a standard [24:17] ragbased system, the one that we're [24:18] talking about, will not answer the how [24:20] many question. And that is like been the [24:21] biggest challenge is just educating [24:23] users on the how many question. [24:25] >> Yeah, I think that's super important [24:26] because unless there's an actual like [24:28] sentence in what you gave it that says [24:31] there are 20 documents, you know, I'm [24:33] just making up a number, right? The the [24:35] the rag and the LLM aren't going to come [24:37] up with that, right? And you need to be [24:39] able like in your full pipeline if you [24:42] know rag is part of your AI solution. [24:45] it's like part of the chatbot like so [24:46] when you go to um you know or when you [24:49] go to chat GPT you're not using the [24:51] model directly you're using an interface [24:53] that goes into the model right and then [24:55] what does that interface do between the [24:57] model and you and so if you want to ask [24:58] questions like how many there better be [25:00] a way maybe you're using an MCP maybe [25:02] you're using something else but there [25:04] needs to be an actual structurable way [25:06] that you're storing the data getting [25:07] that data and then maybe presenting it [25:09] upon retrieval but you're presenting [25:11] that answer with the context right with [25:14] that top Okay. And then it will come out [25:16] and you you force the LM to kind of use [25:18] that answer. But um marrying uh [25:21] structurable and unstructurable data, [25:23] finding structure and unstructured data [25:25] and then uh you know tabular data with [25:28] with rag. Uh there's a lot of really [25:30] cool things that you can do there. But [25:32] it's not one process, right? You're not [25:34] going to just get that with rag. You [25:37] have to have other systems that you're [25:39] working with in some sort of workflow [25:41] that's putting the answer together for [25:42] you at the end. And I think that is a [25:44] big thing. A lot of people ask like what [25:45] is the answer? If it's not in text that [25:48] that's the answer and those words don't [25:50] find each other in the embeddings and [25:52] you know it doesn't wind up in in the [25:54] space, you're not going to get it right. [25:55] You might get lucky, but you're not [25:57] going to get the answer. [25:58] >> Yeah. And I'll even add that people ask [26:00] how many and then then they also ask you [26:02] have a whole corpus of documents. [26:03] They're like summarize them all for me. [26:05] Like what are the main themes? And it is [26:06] the the wrongest question you can ask [26:08] about a rag. And again, back to the [26:10] education piece. [26:12] >> Great. Great. Does anyone want to add [26:14] any other examples of deployed rag [26:16] systems or maybe challenges they've [26:18] encountered deploying rag? [26:19] >> So for us um we connect with the code [26:23] base of um any company who is using our [26:27] product. Uh but also beyond that we [26:29] connect with multiple tools that you [26:32] know one might use across the software [26:34] development life cycle. It could be [26:36] something like Jira for project [26:38] management, notion for documentation, [26:41] sentry or pager duty for your incidents [26:44] and all these datas create very large [26:47] context and knowledge base and obviously [26:50] just vanilla rag wouldn't work here. So [26:53] one of the things that we tried which [26:54] works really well for us is labeling and [26:58] chunking this data. So chunking is [27:01] something that really helps in dealing [27:03] with large context. Uh which is pretty [27:06] much our case. Um but just chunking is [27:09] not enough. When someone is trying to [27:11] retrieve information, you have to also [27:14] intelligently route those chunks because [27:16] you cannot send all the information at [27:18] once. Um in our case, it's an agentic [27:22] system. So you cannot send all the [27:24] information to the agent. So I think [27:26] labeling, chunking and then routing that [27:28] information plays a very key role in you [27:32] know just giving accurate responses to [27:34] users. [27:36] >> Yeah. And just another example which is [27:38] uh similar uh working with a customer [27:41] picture hundreds of thousands of product [27:43] documentation uh company been around for [27:46] a long time in the chemical space. most [27:48] of their information is stored in PDF [27:50] files, right? They actually don't have [27:52] they've tried the CMDB thing I don't [27:54] know how many times it didn't work. If [27:56] you have a successful CMDB, [27:59] raise your hand. But um they want to get [28:02] a bunch of information. They want to ask [28:04] questions against this large document [28:05] set. It's very specific document set. [28:07] It's not all their documents. It's their [28:08] product documentation. There's a lot of [28:10] information in these spec sheets like [28:12] what temperatures it works at, how fast [28:14] does it dry, you know, all things like [28:16] this. So there's a workflow you do like [28:18] you can do entity extraction once you do [28:20] entity extraction on that you save that [28:22] in a structurable way you find the [28:24] structure and the unstructured document [28:26] you save that in a structured fashion [28:29] then when you ask the question how many [28:31] what happens is on that interface you [28:33] built you don't go directly to the model [28:35] but the interface then goes what type of [28:37] question am I being asked what type of [28:38] query do I need to run and then what's [28:40] the sentiment that I want to run against [28:42] the numbers that I get back from the [28:43] system so you do an entity extraction [28:46] you ask how many how many products do I [28:49] have that operate between certain [28:51] temperature settings the AI can come up [28:53] with a query if you're comfortable with [28:54] that it can select from a list of [28:56] queries that you have predefined [28:58] different things um and then you can [29:00] also now marry that you know qualitative [29:03] or quantitative sorry information with [29:05] your qualitative information that came [29:07] from the question to the user it can [29:09] look like one answer to the user looks [29:11] like a oneshot but it's not a oneshot [29:13] like you said it's not in the background [29:14] it's not a oneshot it's this process [29:16] that's happening. Um, and that's one [29:18] thing that we think about a lot is how [29:21] do you look at data? How do you store [29:23] that data in the correct way so when you [29:25] want to use AI, you're correctly [29:28] applying that data either to the LLM or [29:31] augmenting your RAG with the information [29:34] it needs at the right time. MCP is a [29:36] good way to do this on your own, but [29:38] also processing your data ahead of time [29:40] sometimes is the right way to do this [29:42] and not just let it sit at rest and say [29:44] like, "Oh, I'm gonna ask a question and [29:46] every single time I ask a question, I'm [29:48] just going to go find the right vectors [29:49] or I'm just going to go, you know, find [29:51] the right document." But certain levels [29:53] of pre-processing your information so [29:54] you can get to that best context window [29:57] um is something that you have to think [29:58] about when you're putting together uh [30:00] your solution. [30:02] >> Some great examples. I'm going to shift [30:05] gears slightly now and just ask the [30:07] question to to the audience. Can we do a [30:09] quick show of hands? How many people [30:12] believe that rag systems are [30:14] trustworthy, necessary, and essential [30:17] for the future of generative AI? Just [30:19] raise your hands quickly if that's what [30:21] you believe. [30:23] Okay, I'd say around 70% of the audience [30:26] believes that. So, similar question to [30:29] to the group. Um what are some of the [30:32] limitations of rag systems? Do we think [30:34] they are essential and necessary for the [30:37] future of generative AI given the [30:40] improvements in foundation models and [30:42] genetic systems? Do we think rag is is [30:45] going to be persistent or is it [30:47] something that we might evolve beyond? [30:50] You [30:50] >> go first. [30:51] >> Oh, I just spoke so you go for it. Yeah. [30:52] Um, you know, I I I came to this panel [30:55] actually kind of late uh with my other [30:57] co-panelists here and I was like, man, [30:59] I'm I'm going to hate on rag. Like, long [31:01] live rag rag is dead sort of thing. [31:03] Because when it first came out, it [31:04] solved such a big problem. Our context [31:06] windows were like a thousand tokens that [31:09] like that's all we got at the very [31:10] beginning. And so, you know, you could [31:12] only feed a tiny bit of information in [31:14] there. And and then now our context [31:16] windows are like 100k like ridiculously [31:19] stupid long context windows. And so you [31:21] could feed in all of the documents or a [31:24] huge piece of the documents after like [31:25] an elastic search. Um, and so I think [31:28] the answer is somewhere in between, [31:30] right? Rags are an essential piece. They [31:32] are a latent representation of your [31:33] data. And finding that, you know, I [31:35] search off PubMed all the time. That is [31:37] one of our corpuses. That's 30 million [31:38] full text uh uh articles. And like there [31:42] will never be a context window large [31:44] enough. Nor do you want to have a [31:45] context window large enough. So you do [31:46] need some way of pulling out a latent [31:49] representation out of that. always will [31:52] and like rag is that thing call it what [31:54] you will but a latent representation [31:56] that matches your query is how you [31:58] extract the top pieces um so it'll be a [32:01] component but I think we have put a lot [32:03] of importance on it and it's been a big [32:05] thing and we built companies and we [32:07] built pipelines around this and I think [32:08] it'll just become a niche tool that will [32:10] always exist. [32:11] >> Yeah. Yeah. I fully agree. I think I I [32:14] think we're always looking for the magic [32:16] solution like the ones shot solution. [32:18] We're looking at from a technical [32:19] perspective as well all the time, right? [32:20] And it's like rag again rag is is is [32:23] part of the solution set that you're [32:25] have to put together. Um I do think it's [32:28] important. It's it's it's one of the [32:29] best ways to take your data, put it [32:31] against the LLM, get it back without [32:32] having to retrain, create a SLM or a [32:35] model or fine-tune or do other things on [32:36] your own. Um I I think it's definitely [32:40] part of the puzzle. Um, I think context, [32:42] you know, context windows getting huge [32:44] are both a blessing and a curse, right? [32:46] You can send a lot of information, but [32:47] you know, as we spoke about earlier, if [32:49] you send too much context, there's a [32:52] higher and higher probability you're [32:54] going to get back an answer that you [32:55] don't want. And, you know, you're [32:57] gunning for sending the smallest [32:59] context, I would say, that'll get you [33:00] the correct answer. Now, what size is [33:03] that context window? It depends on the [33:04] use case, right? Um, but you're right. [33:06] when you have these super large document [33:08] sets, you're never going to be able to [33:10] send well, maybe I shouldn't say never, [33:12] but at this point in time, for the [33:13] foreseeable future, you're not going to [33:14] be able to send all of your documents in [33:16] at any given time. So again you you you [33:19] you want to understand how you prepare [33:21] that data to be presented but it is a [33:24] very very good mechanism and we find [33:25] customers can go a long way with context [33:28] engineering prompt management um uh you [33:33] know good rag and getting the outcomes [33:36] that they want uh without going to you [33:39] know the next step after that is [33:41] literally going into fine-tuning and [33:42] different things like that on your own [33:44] which is a which is a whole another step [33:46] And uh that's much more involved. And [33:49] you have to be really really certain [33:51] about the data you give to fine-tuning [33:53] because that's in the brain, right? Uh [33:56] and one of my favorite examples of this [33:57] is I don't know anybody seen the movie [33:59] Momento. Um we got some hands. It's [34:03] basically a guy who wakes up every [34:04] morning and has no short-term memory. Uh [34:06] so he has tattoos all over his body and [34:08] he puts together the day every day that [34:10] he wakes up. And that's that's kind of [34:12] like what you're doing is you are [34:13] presenting the context every time to the [34:15] brain. Uh and the brain is seeing this [34:17] for the first time. And what's good [34:20] about rag is if you screw up or you want [34:22] to change the outcome, you can change [34:24] the t like and he does in the mood. Like [34:25] I'm not giving away, but he changes some [34:27] tattoos and the context changes the next [34:29] day, right? And you can do that and you [34:31] can iterate and you can move with this [34:34] without fine-tuning and redoing your [34:36] models all the time and then realizing [34:38] you trained something into a model a [34:39] year ago that shouldn't be in there and [34:41] now you have to backtrack it out. Those [34:43] processes are usually a lot harder than [34:44] having some sort of rag pipeline um you [34:48] know that you've used. [34:50] >> Yeah. Uh I will I will add it to this [34:54] answer. Um uh I believe this is very [34:56] important uh the right system for any [34:59] organization. Yes. Yeah. We already [35:01] agree on this. I I will give you an [35:03] example. Uh we are in GB Corp. We are [35:07] building uh one of system to uh service [35:11] advisor for spare parts. It's an agent [35:14] for the customer when we getting to us [35:15] to get a new parts for the maintenance [35:19] or so on. uh before using the rack uh [35:22] the service advisor or the salesman will [35:24] getting the information from multiple uh [35:27] system and the platform and shareholder [35:29] and so on. So uh we are getting the [35:32] information from 4 to 8 minutes after [35:35] using the rack system we are getting the [35:38] right information on the right time [35:40] within less than 10 seconds. So I [35:43] believe uh we have a benefits of high [35:46] accuracy and time saving for the [35:48] customer and enhancing customer [35:50] experience and order to saving the time [35:52] for the employee. On another side we [35:55] have a limitation. I will give you [35:57] another example. Uh we have a [36:01] understanding issue starting in the [36:04] building the system especially in the [36:06] starting phase. If I asking the RA [36:08] system for the renewal process, it's a [36:12] different meaning from insurance partner [36:14] to leasing partner. This is uh a [36:17] definition and this is another [36:19] definition. So when we starting to [36:21] building a system, it's required a human [36:24] interference in order to build and to [36:27] train the model in the right information [36:30] for the building the right answer for [36:33] the same terminology. [36:36] This is the one of the limitation. [36:37] Another one it's a data dependency. I [36:40] believe if you have a right information [36:43] you will a right answer but all of us [36:46] are required to building and to get an [36:49] AI and direct system without saying what [36:52] is the data assets you have. So the data [36:55] dependency and understanding issue the [36:58] two main issues I faced when are [37:00] building the rock system before. [37:02] [Music] [37:05] I think u the best way to predict future [37:08] is to look at the history. For the [37:11] longest time for example we thought that [37:14] you know RLHF is the only way to train [37:16] AI models but then there was this paper [37:19] on DPO direct preference optimization [37:22] which said that you don't need RL [37:24] anymore. I think the premise of [37:26] innovation in AI and ML is pretty much [37:29] based on ablation. what we think is [37:32] ultimate today is something we not we [37:34] may not even need in the future. So I [37:36] strongly feel rag is a patch. In fact [37:40] what where rag started and where it is [37:42] today is very different. Today rag works [37:45] really well with knowledge graphs, agent [37:47] rag, multi-step process and reasoning [37:50] but I think soon enough we would be [37:52] having something which would you know [37:55] replace rag and help us deal with you [37:57] know larger context [37:59] >> right so I think we're sort of leaning [38:02] feature rather than bug but with some [38:05] diminishing returns. uh this this line [38:08] of questioning always gets me nervous as [38:09] an AI investor because we always look [38:11] for things like system differentiation [38:14] right how do you tell one solution from [38:16] another solution and you know your your [38:18] line of thinking means a lot of value [38:21] acrru to the large model providers so [38:24] and it's harder and harder to generate [38:26] value it's harder harder and harder to [38:28] get value from things like proprietary [38:30] data or content so maybe maybe the next [38:34] part part of this question is what what [38:36] do we Think about this line of [38:38] questioning and the implications for the [38:41] value of proprietary data and the [38:43] ability to to differentiate uh [38:46] applications versus you know what's just [38:48] going to be available off the shelf from [38:51] foundation model providers you know the [38:53] claws the chat GPTs of the world [38:57] I'll take a stab at that I I don't know [39:00] that it means that everything goes [39:01] upstream especially from a proprietary [39:04] data perspective because I [39:06] You don't want your proprietary data [39:08] going upstream. And I don't even know [39:10] that the future for what a lot of [39:13] businesses want to use AI for is [39:17] something that a large language model is [39:18] necessarily even needed, right? So if [39:22] you have a narrow use case, uh LLMs are [39:25] great because they can kind of do [39:26] everything, but you really want to you [39:28] you want to put guard rails on that. [39:30] Like you want very specific things done. [39:34] If you have a workflow in your business, [39:36] you probably rarely want one agent to [39:38] rule them all. You're not trying to make [39:40] another chat GPT in your organization. [39:42] You're looking to do something specific. [39:44] I want to do entity extraction. I want [39:46] to do classification. I want to do [39:48] search. I want to do I want to do [39:51] something. And the more narrow you can [39:54] make that use case, the better chance [39:56] you have of that outcome being [39:58] successful. At the end of the day, I [40:00] actually think as we get forward, the [40:03] research going on that helps large [40:06] language models are also going to help [40:08] building domain specific models, small [40:10] language models, it'll help people [40:12] fine-tune models for their [40:14] organizations. And I think the smaller [40:17] model you can use, if you can use a 7 [40:18] billion or 40 billion, you know, [40:21] parameter model and get the outcome you [40:23] want, that's better. That's less [40:24] compute, that's less GPU, that's less [40:27] cost. it doesn't mean that the [40:28] fundamentals of how you built that model [40:32] change and it has to all go you know to [40:34] the to the large LLM. So, um, I think [40:39] the the the more narrow use case you [40:42] have and splitting up things in your AI [40:44] workflow, so something does a specific [40:46] action within your workflow and you're [40:49] not trying to make one AI that does all [40:51] the actions across the board, is super [40:54] important and allows you to bring down [40:56] the scope and still take advantage of [40:58] the latest and greatest research and the [40:59] latest greatest technologies that are [41:01] out there and go nuts on hugging face. [41:05] I I think the way you describe it is [41:06] going to make all the enterprise people [41:08] just cry because there's like not one [41:10] model to rule them all. Not one solution [41:12] I could just like deploy out. I know my [41:13] CIO is just like rolling his eyes just [41:16] on this one. Um I think to your point [41:18] about proprietary data. Yeah, of course. [41:20] For us, there's we have a trove of [41:23] proprietary data, whether it's data from [41:25] states or from like local jurisdictional [41:27] departments or data that we can't share [41:29] for uh privacy rules and other sorts of [41:32] like it's a big deal and keeping the [41:34] ability to do things on prem cannot be [41:36] underestimated for the federal [41:38] government or for state governments or [41:40] local governments. Like this is a really [41:41] big thing and we will continue to want [41:43] to do this. We're really excited to have [41:46] the ability to use a foundation model, [41:49] but also to pull on a like small [41:52] language model. Like we trained a BERT [41:54] model two years ago, right? Like a BERT [41:57] model that like sounds so old these [41:59] days. And it's its only job was to look [42:02] for when people answered a survey, what [42:04] sort of non-response they give. Did they [42:06] give an answer that seems like a real [42:07] answer, or did they say, "I'm a cat." [42:09] Which is like a thing people might say [42:11] on a CDC response. or do they say I hate [42:14] you CDC another thing that they might [42:16] say and be able to classify those [42:17] non-responses and then maybe guide them [42:20] to you know getting a better response [42:22] from them that model is you know it's [42:25] BERT so it can fit on you know a small [42:28] thumb drive [42:29] >> I think that's I I think it's a great [42:31] example and I think it's I'm sorry I cut [42:33] you off but I think that's a great [42:34] example because you're using the tool [42:36] that's necessary for the job and you're [42:38] not using a sledgehammer when a chisel [42:39] is necessary [42:41] >> and it's on prem [42:42] Yeah, that's also a key, right? Like the [42:44] closer and closer you get on prem, you [42:46] don't want to be buying these huge [42:47] Nvidia GPUs machines when you don't have [42:50] to. Like heaven forbid you got to run, [42:52] you know, all of Llama 4 locally, right? [42:55] You want to try and avoid that when [42:56] possible. Um, so yeah, but yeah, great [43:00] great example. [43:02] >> Great. So maybe we'll just move into a [43:05] bit of a quick fire round here with the [43:07] last couple minutes of the session. [43:08] Maybe we could just go from right my [43:11] right to to left on if you had you know [43:14] two sentences or less uh advice for [43:17] folks in the audience people who are [43:19] considering implementing rag systems or [43:22] you know making improvements to existing [43:24] systems that they may be deploying. What [43:26] would what would be your what would be [43:27] your advice to to folks? [43:30] >> Okay. Um I will start with uh one one [43:34] statement for uh uh the advice when the [43:38] company using uh when the company start [43:40] using the rag system. [43:43] I believe um get your data right before [43:47] doing anything. [43:49] Uh this is uh I believe uh in this [43:52] statement we we need to see the data [43:55] before uh using or investing anything [43:58] for the technology. This is the number [44:00] one. And the number two, start with a [44:04] small business objective with a clear [44:08] goal. Start with a small project of [44:11] Rexist. And finally track whatever what [44:16] do you need and what do you want and [44:18] what do you do now with the Rexist in [44:23] order to take a step by step to enhance [44:25] and to step by step to scale up for the [44:28] highest project. This is my advice. [44:32] >> I will again yes and you um and just say [44:35] on top of that we have qualitative [44:37] metrics of success for your RAG. You [44:39] could look to see LLM as judge just to [44:41] look at this. But do not uh [44:43] underestimate the value of just talking [44:45] to the end users like whether they're [44:47] internal and like talking to them as [44:49] humans, right? Sitting down having a [44:51] focus group saying did it did it work [44:53] because it probably did but it probably [44:55] has room for improvement and like that [44:57] hard work of sitting down with your [44:59] users bring in your UIUX people like it [45:02] will go so far. [45:04] >> Yeah. Yeah. I think to uh you know just [45:07] echo on top of that um have a defined [45:10] use case. There's so many people you [45:12] talk to it's like they're just using AI [45:14] and trying to figure out a solution for [45:15] the technology. Like get that [45:18] well-defined solution first. I know I'm [45:19] like bringing it like way back, but like [45:21] I can't tell you how many people I [45:22] talked to who don't have the outcome [45:25] that they want to get out of the whole [45:26] thing. They're just like I was told I [45:28] need to use AI. Like let's use AI. Like [45:30] have the outcome you're looking for and [45:32] measure back from there, right? like [45:34] have a way to measure those results. [45:35] Have a way to put everything together. I [45:37] got to be a little I got to be a little [45:39] salesy heel. Like this is something that [45:40] we do at Maybel. Like this is we we our [45:44] platform looks at data, understands how [45:46] to catalog data, understands when rag or [45:48] other, you know, relational is necessary [45:50] and gets you the output. If you're not [45:52] going to use a platform like Maybel, [45:55] then you got to look into how am I going [45:57] to store how am I going to store and [45:58] scale my vector database? Do I need [46:00] citations? Do I need a graph rag? How do [46:02] I want to get that context engineering [46:05] to work in an optimal way so when the [46:08] context gets to the rag I can measure it [46:10] on the other side right like you got to [46:12] be thinking those things and please [46:13] please have a have a defined outcome [46:15] where you can't measure anything [46:17] [Music] [46:19] I think the only thing I have to add [46:22] here is that generative AI landscape is [46:25] moving really fast so you know keep your [46:28] fundamentals intact which is your [46:29] proprietary data don't get compare it to [46:32] a certain technology or a certain model [46:35] because you never know it may not be the [46:37] best tomorrow. [46:39] >> Well, great. I think we are right at [46:42] time. Uh, how about a big round of [46:45] applause for all of our panelists?