Morgan Stanley and Private Equity Firms Scope RAG Data to Cut Costs

摩根士丹利与私募机构精简RAG数据以削减成本 企业唯有将精准的向量搜索与严格的权限管理及结构化API相结合，而非将未经整理的原始数据直接视为护城河，才能成功落地RAG。 本次圆桌探讨了在生产环境中部署RAG架构的务实现状，不再局限于基础概念介绍，而是深入探讨高价值业务应用、数据规范治理与安全性。与会嘉宾分享了来自投资银行、私募股权、健身资产组合及企业软件领域的实战经验，重点介绍了自动化尽职调查问卷（DDQ）等明确的成功案例，同时也指出了数据孤岛和依赖Excel等长期存在的痛点。他们强调，企业原始数据只有经过妥善的版本控制、精细整理，并在严格的访问权限控制与智能体规范治理下，才能成为真正的竞争壁垒。关键在于，要将RAG从演示Demo演进为高可靠系统，必须将大量工程资源投入到后端回归测试、可审计性以及结构化API路由的建设中。 检索增强生成（RAG）正从基础概念普及，转向对企业实际落地执行、失效模式及商业价值权衡的深度评估。 (0:04) 摩根士丹利利用RAG架构激活并复用沉淀在数千份银行家PowerPoint推介文稿中的机构知识，同时执行严格的隐私管控。 (2:24) 企业原始数据天然杂乱且持续动态变化，这意味着数据模型必须像代码一样进行版本控制，才能在企业系统之上构建可靠的RAG层。 (8:28) 缺乏结构的海量原始数据毫无价值；高度规整和精心筛选的数据集，其实际表现始终优于杂乱无章的信息堆砌。 (11:10) 扩大的上下文窗口虽可支持在短期任务中进行简单的文档提示工程，但对于长期、复杂的特定领域业务场景，专用的RAG流水线依然必不可少。 (11:13)

Enterprise practitioners argue that real-world business value from Retrieval-Augmented Generation (RAG) depends far less on raw model capabilities than on disciplined data curation, strict security scaffolding, and back-end regression testing. Rather than treating proprietary data as an automatic competitive moat, organizations must build auditable software pipelines that balance vector search against structured operational systems. Curating Clean Data as an Enterprise Moat (1:10) Joe Gaska points out that raw enterprise data inside CRMs, ERPs, and HR systems is inherently messy, subject to continuous schema updates, and often constrained by third-party SaaS APIs. Simply possessing large volumes of corporate records does not provide a defensible advantage; organizations must treat their data models like software code that is versioned and governed over time. Ameen Kazerouni emphasizes this distinction with a library analogy, stating that a curated collection containing 40% of the world's books will consistently outperform a disorganized warehouse containing 100%. He cautions that proprietary, human-generated enterprise data must be guarded against synthetic AI-generated noise to prevent internal models from degrading. Enforcing Access Governance and Pipeline Selectivity (1:16) In regulated environments, Nick Ginsburg explains that Morgan Stanley's 5,000 bankers require strict data boundaries when querying historical pitch decks. Ginsburg rejects fine-tuning for internal knowledge sharing because it bakes enterprise information directly into model weights, breaking access permissions and exposing data to prompt-injection leaks. RAG with scoped embeddings ensures employees only retrieve documents they are explicitly authorized to view. Richa Singh notes that building and maintaining these RAG data pipelines with proper metadata tagging represents the costliest part of implementation. With context windows expanding from 4,000 tokens in early 2023 to over one million, Singh advises that RAG should be reserved for complex, multi-year repositories, while simpler ad-hoc tasks are more efficiently handled through direct long-context prompting or lightweight document-scrubbing setups. Engineering Discipline Versus Vibe-Coding Pitfalls (27:32) The panel highlights that Microsoft Excel remains the entrenched operational backbone across finance, legal, and HR because enterprise users demand absolute auditability and manual verification. Ameen Kazerouni, Joe Gaska, and Chirag Shah contrast senior engineers' architectural precision with the rise of unvetted "vibe coding," where junior developers frequently generate layered, unverified AI code from loosely defined prompts without understanding intermediate logic. Gaska explains that while AI drastically accelerates early proof-of-concept development, it pushes the engineering burden into back-end validation. At GRAX, initial underinvestment in verification proved costly; operating in regulated sectors requires automated regression testing against established "golden cases" to ensure document, schema, or prompt updates do not silently alter production outputs. Hybrid Architectures and Agent Constitutions in Practice (30:30) High-value enterprise RAG deployments succeed by integrating retrieval into deterministic, structured workflows. Richa Singh highlights Lexington Partners' automation of Due Diligence Questionnaires (DDQs), where a vector database matches investor inquiries against compliance-approved historical records, reducing a multi-week cross-departmental chore to a few hours while retaining mandatory human-in-the-loop review. Conversely, Kazerouni explains that Orangetheory Fitness deliberately avoids vector embeddings for dynamic operational details such as studio class times, coach rosters, or amenities to prevent customer-facing hallucinations. Instead, the company enforces an "agent constitution" that routes factual queries to structured APIs while reserving vector stores for tone and general marketing questions. For broader enterprise data silos, Gaska recommends implementing a unified federated search layer to govern distributed systems before granting AI models direct data access. The panel concludes that while RAG delivers massive labor reductions in targeted domains like diligence documentation and pitch deck reuse, it cannot bypass enterprise data engineering fundamentals. The primary operational bottleneck remains managing continuous schema drift, building rigorous regression testing suites, and governing distributed data silos without relying on unverified, AI-generated code.

来自摩根士丹利、Lexington Partners、Purpose Brands 与 GRAX 的技术领袖指出，检索增强生成（RAG）的商业价值不在于快速搭建概念验证，而在于底层数据的精细化治理与严格的权限隔离。随着模型上下文窗口扩展和“氛围编程”引发的质量隐患，企业必须依托受保护的人类真实数据，在向量检索、结构化 API 以及严密的后端回归测试之间审慎权衡。 数据护城河：精选结构与纯净人工数据 (1:10) GRAX 联合创始人 Joe Gaska 指出，尽管专有数据常被视作核心护城河，但 CRM、ERP 和 HR 等底层 SaaS 系统中的数据普遍存在质量缺陷，其业务流程与数据库模式也在持续变动。由于 API 配额限制与数据壁垒，许多企业甚至无法完全掌控自身数据。因此，企业若要在动态数据上稳定构建 RAG 语义层，必须将数据模型视同代码进行严格的版本控制。 Purpose Brands 首席技术官 Ameen Kazerouni 通过“图书馆比喻”强调，一个仅收录 40% 书籍但索引井然的图书馆，检索表现远超堆满全球书籍却杂乱无章的藏书库。原始数据若缺乏语义建模，只会放大检索噪声并加剧幻觉。Kazerouni 特别主张，企业必须全力捍卫未经 AI 合成污染的“人类生成专有数据”，避免模型陷入吞噬自生成噪声的崩溃循环。 权限隔离与安全：范围化嵌入对决模型微调 (1:16) 针对社交媒体宣称“数小时即可通过氛围编程（vibe-coding）取代传统企业软件”的言论，摩根士丹利执行董事 Nick Ginsburg 明确予以反驳，指出头部 AI 厂商自身依然重度依赖 Salesforce 和 Workday。在摩根士丹利，5,000 名投资银行家沉淀了大量非结构化演示文稿，其 RAG 系统的核心挑战在于精细的数据权限控制，必须杜绝处理特定交易的员工检索到未经授权的客户机密。 Ginsburg 反对“微调将彻底取代 RAG”的预测。他解释称，微调会将企业知识直接固化到模型权重中，从而瓦解原有的权限边界，使系统极易因提示词注入或越狱攻击而发生数据泄漏。相比之下，RAG 能够支持“范围化嵌入”（Scoped Embeddings），依据用户权限动态收缩检索范围（例如仅向用户开放其有权查看的 40% 数据），从检索层原生实现安全隔离。 架构权衡与混合路由：向量检索与结构化接口 (1:13) Lexington Partners 数据与 AI 负责人 Richa Singh 回顾指出，大模型上下文窗口自 2023 年初的 4,000 token 演进至百万级别，深刻改变了架构决策。构建企业 RAG 管线伴随着高昂的元数据打标与清洗维护成本。针对需要深厚领域背景、精确跨库检索与严苛元数据过滤的长期核心场景（如 Lexington 沉淀 30 年的私募股权数据），RAG 具备显著投资回报；但对于简单的临时文档处理，采用长上下文模型配合直接提示则更为经济高效。 此外，Kazerouni 警示企业切忌将所有业务数据强行塞入向量数据库。以健身品牌为例，品牌基调与通用常见问题适合向量检索，但排班表、教练名单或设施可用性等实时运营数据，若依赖非结构化检索极易产生误导性幻觉。Purpose Brands 采用明文设定的“Agent 宪法”（Agent Constitution）作为路由规则，强制要求模型在涉及动态运营事实时必须调用结构化数据库与 API。 生产落地的代价：尽调提效与回归测试陷阱 (24:49) 在实际落地成效方面，Singh 介绍了 Lexington Partners 运用 RAG 自动化处理尽职调查问卷（DDQ）的成功实践。面对机构投资者复杂多变的问卷，系统通过向量数据库匹配历史合规答案并生成草稿，再交由合规专家复核把关，成功将耗时数周的跨部门协作压缩至数小时，实现了效率与合规治理的平衡。 然而，Gaska 提醒道，初始概念验证的迅捷往往掩盖了投产后的繁重运维。在生产环境中，底层文档、向量模式或提示词的微调都会引发不可预测的模型漂移。GRAX 曾为此付出试错代价，因而强调必须建立金样自动化反向回归测试与全链路追踪机制。Shah 与 Kazerouni 亦补充，Excel 至今仍是金融与法务部门的基石，反映出业务对透明审计的依赖；开发者过度依赖缺乏校验的“代码废料”（slop），将埋下严重的系统债务。 破解企业数据孤岛：基于联邦检索的统一治理 (34:34) 面对大型机构内部普遍存在的数据孤岛，以及业务部门使用即席脚本或临时应用挖掘数据带来的合规风险，Gaska 提出了基于“联邦检索层”（Federation Layer）的系统性架构方案。 他认为，企业无需强行将所有异构数据物理汇聚至单一中心数据库，也不应赋予 AI 代理不受控的底层直接访问权。通过构建跨分布式系统的联邦检索层并在其上搭建统一语义层，企业能够在定义业务逻辑与检索机制的同时，集中实施上下文权限管控，确保各业务单元在接入 RAG 时均遵循一致的安全治理协议。 本场研讨表明，企业落地 RAG 已从早期的技术探索演变为深度的工程治理考验。尽管尽调自动化等场景带来了切实的投资回报，但面对动态模式漂移、长上下文模型演进以及非结构化检索的局限，企业仍需在向量数据库与结构化 API 之间建立混合路由。如何规避缺乏审查的氛围编程产物，并依托联邦检索与常态化回归测试筑牢安全底座，是决定企业 AI 能否规模化交付真实价值的关键。

## Background 该视频由人工智能行业峰会组织 Ai4 于 2026 年 8 月 22 日发布在 YouTube 平台。内容为一场题为《检索增强生成（RAG）用例：借助更聪明的 AI 解锁商业价值》的圆桌讨论，汇集了来自学术界、投资银行、私募股权以及实体消费连锁的技术高管，共同探讨企业如何利用 RAG 提升生成式 AI 的准确性与信任度，将专有数据转化为实际商业价值。 ## TL;DR 研讨会跳过了基础概念科普，深入剖析了 RAG 在金融尽调、合规审查与跨国品牌运营等真实企业环境中的成败得失。嘉宾们指出，未经治理与版本控制的混乱数据绝非企业的护城河，盲目将所有数据向量化反而会带来严重的幻觉风险与维护灾难。相较于追求社交媒体吹捧的“氛围编程”（vibe coding），企业级 RAG 的真正壁垒在于细粒度的权限隔离、三维时序语义层以及严格的全链路回归测试。专家们最终一致呼吁，RAG 的痛点本质上是传统数据治理的延续，决策者应打破技术锁定与热词焦虑，将 90% 的精力重新扎根于解决真实的业务问题。 ## Atmosphere & style 整场研讨会节奏紧凑、直击要害，洋溢着一线技术操盘手特有的犀利与坦率幽默。主持人与嘉宾频频调侃社交媒体上浮夸的技术神话，在对“手搓彭博终端”与盲目堆叠代码垃圾的讽刺中，碰撞出强烈的工程实战共鸣。投行风控的严谨克制与消费实体落地的敏捷探索相互交织，使整场对话既像是一场严肃的架构避坑指南，又透着一股给行业狂热“泼冷水”的清醒与务实。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=PTbKrnsB05s&t=4s) RAG 技术已跨过基础科普阶段，当前企业落地的核心转向深入评估其在具体业务场景中的真实收益、失效边界与工程局限。 - [08:15](https://www.youtube.com/watch?v=PTbKrnsB05s&t=495s) 企业原始专有数据大多残缺混乱，仅占有数据不构成竞争护城河；真正的壁垒在于通过语义层解构业务流程并实现如代码般严密的数据版本控制。 - [11:10](https://www.youtube.com/watch?v=PTbKrnsB05s&t=670s) “两座图书馆”比喻揭示：管理有序的 40% 高质量数据胜过混乱堆放的 100% 全量数据；人类生成的专有数据是抵御模型自我训练导致“模型崩溃”的核心资产。 - [13:08](https://www.youtube.com/watch?v=PTbKrnsB05s&t=788s) 大模型百万级长上下文窗口的普及改变了架构权衡：轻量临时需求无需过度投入维护繁重的 RAG 生态，系统核心成本已转向前端脚手架与元数据工程。 - [16:14](https://www.youtube.com/watch?v=PTbKrnsB05s&t=974s) 企业级应用无法仅凭轻量脚本解决，金融强监管场景依赖基于 RAG 的细粒度权限隔离与合规审计，而直接微调模型会丧失物理权限边界并面临越狱风险。 - [18:21](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1101s) 警惕缺乏底层推敲的“氛围编程”（vibe coding）与对 AI 结果的盲目信任，过度依赖生成代码会导致低质代码堆叠并累积长远技术债务。 - [24:49](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1489s) 私募与资管领域的尽职调查问卷（DDQ）是高回报落地场景，借助历史合规库 RAG 检索与人机协同，可将数周的繁琐流程缩短至数小时。 - [27:32](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1652s) 工程效率存在“转移悖论”：AI 极大加速了前端概念验证（POC），但整体工程复杂度转移至后端的全链路回测、版本控制与自动化回归测试。 - [30:30](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1830s) 严禁将所有数据盲目灌入向量库，应区分结构化与非结构化数据；通过“智能体宪法”约束意图路由，结合 Trace 追踪确保数据来源可靠性。 - [34:41](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2081s) 破解大型企业数据孤岛需构建分层治理架构（RAG of RAGs），在底层分布式系统之上建立统一联邦搜索接入层与语义层调度。 - [37:00](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2220s) 应对 SaaS 系统的模式漂移（Schema Drift），企业需将原始数据沉淀至自建云平台，构建三维时序语义层以解释历史时间点下的字段含义。 - [41:21](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2481s) 所谓“RAG 系统的失败”本质往往是上游的数据治理问题，保持原始数据保真度并夯实数据预处理基本功远比追逐最新模型算法更为关键。 - [43:35](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2615s) 避免将系统架构与任何单一技术方案或供应商深度绑定，保持解耦与弹性以便在快速技术迭代中敏捷替换组件。 - [44:27](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2667s) 决策者与工程师应走出“AI 焦虑”，停止对新概念标签的盲目追逐，将资源与精力重新聚焦于企业真实的痛点解决与数据资产建设。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=PTbKrnsB05s&t=4s) 圆桌开场：告别基础科普，探讨 RAG 的实战成败 主持人 Chirag Shah 宣布首场圆桌研讨正式开始。Chirag 拥有双重身份：他既是西雅图华盛顿大学的教授，专攻前沿 AI 技术与工业界落地合作，同时也正在经历自己的第一次创业历程。他一开场就幽默地调侃西雅图的凉爽天气，与现场的高温形成了鲜明对比。 谈及本次论坛的核心议题 RAG（检索增强生成），Chirag 直言不讳地指出：对于在场的所有从业者来说，RAG 绝非新鲜事物。“如果你来这里是想听‘什么是 RAG’以及‘如何搭建基础 RAG’，那你已经迟到了整整两年。”经过行业数年的探索与沉淀，现在正是深入探讨 RAG 究竟“在哪里有效、在哪里碰壁”的关键时刻。为了避免传统圆桌沦为轮流作答的枯燥“教科书式问答”或短采访，主持人强调本次讨论将聚焦于更具深度、更显微妙的实际挑战，由不同嘉宾针对具体问题展开争辩与互动。由于整场讨论仅有 45 分钟，时间非常紧张，他也极力建议听众在会后随时抓牢嘉宾继续深聊。 --- ### [02:25](https://www.youtube.com/watch?v=PTbKrnsB05s&t=145s) 投行与私募的落地考量：知识复用与演进步伐 圆桌首先由两位具有投资银行背景的技术领袖展开自我介绍： ```mermaid flowchart TD accTitle: 投行与私募机构 RAG 核心诉求 subgraph MorganStanley["Morgan Stanley (Nicholas Ginsberg)"] A1["全球 5,000 名投资银行家"] --> A2["海量历史 Pitch Decks / 提案演示"] A2 --> A3["提取非结构化 PPT 知识"] A3 --> A4["安全跨团队复用，避免重复造轮子"] end subgraph Lexington["Lexington Partners (Richa Singh)"] B1["全球顶级二级市场 PE 机构"] --> B2["早期落地 RAG (2023 年初)"] B2 --> B3["应对爆炸式迭代：3年演进堪比行业30年"] B3 --> B4["持续更新 RAG 定义与核心应用范式"] end ``` 摩根士丹利（Morgan Stanley）投资银行部执行董事兼约翰斯·霍普金斯大学量化金融客座教授 Nicholas Ginsberg 首先发言。在进入正题前，他特意插播了一条合规免责声明，幽默地提醒媒体若要引用其言论，必须事先经过他本人与摩根士丹利的法务审核。针对业务实际，Nick 强调摩根士丹利在全球拥有超过 5,000 名投资银行家，日常业务需要不断向客户竞标提案、制作演示文稿。这些精心排版制作的 PPT 中沉淀了极为庞大的非结构化数据。如果某个团队已经针对某款产品做出了极高水平的 Pitch Deck，其他团队理应能够安全、合规地复用这些智慧成果，而不是在全行范围内反复“重新发明轮子”。在确保数据隔离、严防信息越权访问的前提下，通过 RAG 释放大型跨国机构的规模效应，是极具价值的挑战。 随后发言的是全球最大二级市场私募股权投资机构之一 Lexington Partners 的数据与 AI 主管 Richa Singh。Richa 回顾了自己与 Nick 此前在高盛（Goldman Sachs）投行部共事的经历，正是从 2023 年初起，她便全身心投入到了 RAG 架构的研发中。她感叹，在 AI 领域狂飙突进的这两三年里，技术的迭代速度甚至抵得上传统行业走过 30 年，RAG 的内涵与技术边界已经发生了深刻的演进。Richa 在 2013 年便选择离开职场深造机器学习，拥有超过 13 年的 AI 积淀，目前还在哥伦比亚商学院讲授生成式 AI 在商业中的落地实践，RAG 正是她在课堂上最常剖析的核心模块。 --- ### [05:42](https://www.youtube.com/watch?v=PTbKrnsB05s&t=342s) 实体连锁与 SaaS 生态：代理销售编排与底层语义层 接下来的两位嘉宾分别从庞大的跨国消费连锁与企业级底层数据管理切入，展示了 RAG 完全不同的应用场景： Purpose Brands 首席技术官 Ameen Kazerouni 介绍了旗下庞大的实体健身与生活方式品牌矩阵，包括 Orange Theory Fitness、Anytime Fitness、The Bar Method 以及 Wax in the City 等。这些业务沉淀了极其丰富且独一无二的实体交互数据：例如 Orange Theory 拥有全球最大规模的情境化心率监测数据集，而 Anytime Fitness 则在覆盖全球 40 多个国家、十几种语言的 5,000 多个线下健身场馆中收集海量会员签到与运动数据。此前曾在 Zappos 掌管 AI 与机器学习平台的 Ameen 坦言，自己至今还在“随手撒点 AI 调味品”（sprinkle some AI on it）的盲目需求后遗症中恢复。他强调，生成式 AI 的核心目标必须是带来真实的商业价值。在 2024 年，他的团队正致力于将 Agent 编排（Agentic Orchestration）与 RAG 结合，赋能企业的最底层生命线——会员销售，在跨越不同语言、不同品牌语境的场景下，与潜在客户进行长期且自然的互动沟通。 GRAX 联合创始人兼首席产品官 Joe Gaska 紧接着介绍了他们在数据底座方面的专长。GRAX 专注于采集并留存 Salesforce、SAP、Workday、ServiceNow 等主流企业级 SaaS 工具的全部历史版本数据，并在其上构建统一的语义层（Semantic Layer）。Joe 认为，RAG 的商业价值在很大程度上取决于历史数据的动态性与变现周期。无论是在公司内部提升运维与研发效率，还是将 RAG 转化为产品功能提供给财富 500 强客户，其核心壁垒都在于应对数据的多态变异（Morphological Variance）以及数据模式（Schema）随时间的演化。 --- ### [08:15](https://www.youtube.com/watch?v=PTbKrnsB05s&t=495s) 企业专有数据是护城河，还是“脏乱差”的沉没成本？ 在进入核心提问环节后，主持人 Chirag 抛出了关于 RAG 基础立论的尖锐问题：业界普遍宣称，将企业专有数据灌入 RAG 架构是构建商业壁垒（Moat）的绝招，但现实中企业数据往往极为混乱而肮脏，以这种代价去换取护城河，究竟是不是一笔划算的买卖？ 针对这一问题，Joe Gaska 将数据比喻为商业运转的“原材料”。企业界心照不宣的一个小秘密在于：几乎所有企业内部的 CRM、ERP 或人力资源管理系统中的原始数据，从本质上讲都是残缺和肮脏的。更为棘手的是，业务流程、员工使用工具的方式每天都在变化，数据 Schema 也随着业务演化而不断漂移。因此，仅仅把 RAG 挂载在现有数据表面是远远不够的，系统自身必须像管理源代码一样进行严密的全版本控制。 ```mermaid flowchart LR accTitle: 企业专有数据转化为 RAG 护城河的路径 subgraph RawData["混乱原材料 (Dirty Data)"] D1["SaaS 锁定系统 (CRM / ERP)"] D2["业务流程变迁 & Schema 漂移"] end subgraph Transformation["价值转化瓶颈"] T1["打破 API 限制与生态围墙"] T2["语义层 (Semantic Layer) 解构"] T3["如代码般的版本演进控制"] end subgraph Moat["真·商业护城河"] V1["高置信度业务价值"] V2["动态自适应的 RAG 应用"] end RawData --> Transformation --> Moat ``` 针对数据壁垒的成色，Joe 进行了更深层次的拆解： - **数据并不等于价值**：原材料本身构不成最终护城河，真正的能力体现在如何利用语义层去精确理解、解释业务流程，并将其高效转译为下游应用可消化的信息。 - **“你真的拥有这些数据吗？”**：大量的现代业务跑在封闭的 SaaS 生态之内。严苛的 API 调用限制、封闭的数据存储机制，使得许多企业误以为自己拥有数据，实际上却连完整调取全量深度数据的权限都不具备。无法掌控全局数据，所谓的 RAG 壁垒便无从谈起。 - **高昂的转化摩擦成本**：从脏数据、API 围墙到可用的私有化 RAG 架构，中间伴随着巨大的工程改造开销与治理成本。 就在这一争论的节点，主持人转向 Ameen Kazerouni，准备深入探讨企业应当如何精准评估手中持有的数据是否真的具备转化为商业价值的潜质。 ### [11:10](https://www.youtube.com/watch?v=PTbKrnsB05s&t=670s) 数据护城河与“两座图书馆”的比喻 在探讨企业自研 RAG 系统如何实现并评估其业务价值时，Ameen Kazerouni 指出，无论当前用来激活和操作化数据的新工具是什么，底层面临的核心挑战始终未变。他分享了当年加入 Orangetheory 时对团队打过的一个生动比喻：现场许多人都知道是谁陪同山姆·詹吉（Samwise Gamgee）前往魔多，但如果把问题抽象出来——设想有两个地方，一间房间里杂乱堆放着世界上所有的书，所有人都可以声称“我们拥有全部数据，答案全在里面”；而另一边是一座井然有序的现代化图书馆，尽管馆藏只覆盖了世界上 40% 的书。当需要寻找精准答案时，任何人都会毫不犹豫地走向那座管理有序的图书馆。 这个比喻精准揭示了企业数据架构的核心：仅仅“拥有海量数据”并不等于具备竞争优势。无论是在上层搭建 RAG 检索层、构建语义模型，还是梳理精选数据仓库，数据的组织结构与可用性才是决定成败的关键。在生成式 AI 时代，合成数据的门槛越来越低，单纯的数据占有甚至正在变成一种准入门槛（Table Stakes），而非坚不可摧的壁垒。Ameen 强调，企业真正应该“拼命守住”的是由人类生成的专有数据，这种数据没有被 Agent 反复自我训练所污染，不存在模型崩溃（Model Collapse）的系统性风险。但光有原材料还不足以盖起大厦，唯有建立清晰的数据架构与结构化组织体系，让数据真正流转、服务于业务，才能将其转化为真正的战略护城河。 ```mermaid flowchart TD accTitle: 数据价值转化逻辑图 subgraph RawData["底层数据资产"] A1["未整理的专有数据<br/>(杂乱堆放的所有图书)"] A2["人类生成的专有数据<br/>(抵御模型崩溃风险)"] end subgraph Architecture["数据工程与结构化"] B1["语义建模与清晰架构"] B2["精细化元数据标注"] B3["受控检索管线 / RAG 层"] end subgraph Outcomes["业务产出对比"] C1["高幻觉 / 权限泄漏风险<br/>(空有全量数据)"] C2["高精度 / 真正业务护城河<br/>(40% 有序数据)"] end A1 --> C1 A2 --> B1 B1 --> B2 B2 --> B3 B3 --> C2 ``` --- ### [13:08](https://www.youtube.com/watch?v=PTbKrnsB05s&t=788s) 架构脚手架与长上下文窗口下的方案权衡 主持人转向 Richa Singh，探讨企业在自建复杂基础设施与选用现成轻量工具之间应如何权衡ROI。Richa 回顾了自 2023 年以来的技术演进：当时语言模型的上下文窗口仅有大约 4K token，面对大规模文档，搭建 RAG 管线是唯一的出路；同时，当时的 Agent 架构也远没有今天这般成熟。而如今，上下文窗口已暴涨至上百万 token，AI 的演进日新月异。因此，技术决策必须紧密跟随 AI 的底层演化，审视具体用例是否真的值得投入高昂的维护成本去维护一套庞大的 RAG 生态系统。 对于一次性、周期短的小型用例，如果只是需要对几份文档进行快速信息清洗与提取，盲目投入重度 RAG 往往是过度设计。Richa 以自己所在机构为例，公司拥有长达 30 年的投资历史数据，如果直接将这堆未经整理的陈年资料原封不动塞进系统，即便套上 RAG 框架，结果也必然充斥着严重的幻觉与错误。真正的核心成本不在于调用大模型，而在于前端的“脚手架”（Scaffolding）工程：如何筛选入库数据、如何赋予正确的元数据（Metadata）以及如何建立稳固的数据流水线。如果业务场景只是“自带文档”（Bring Your Own Documents, BYOD），用户很清楚具体要查哪几份材料且不依赖复杂的全局元数据过滤，那么采用轻量级直读方案远比投入重金建设 RAG 管线更具性价比。 - **上下文窗口跃迁**：从 2023 年初主流的约 4K token 迅速扩展至当前百万级 token，直接改变了企业检索架构的设计前提。 - **维护生态的隐形成本**：完整的 RAG 基础设施包含向量库、更新流水线、监控与评估，长期维护成本远超快速脚本或临时提取方案。 - **30 年投资数据教训**：未经元数据清洗的遗留数据直接灌入向量库，只会加剧模型幻觉，而不会凭空产生商业洞察。 --- ### [16:14](https://www.youtube.com/watch?v=PTbKrnsB05s&t=974s) 戳破“午后手搓 RAG”神话与金融风控合规 针对主持人打趣提到的“有市场人员或工程师宣称自己一个下午就搭好了整套 RAG 方案”，Nicholas Ginsberg 进行了幽默而犀利的剖析。他指出，社交媒体如 LinkedIn 上充斥着各种浮夸推文，宣称“彭博已死，我用 Claude Code 两个小时就凭感觉敲出了一个彭博终端（vibe coded Bloomberg）”，或者声称一个下午就手搓替代了 Salesforce。“如果真是这样，那你现在身价应该已经千亿美元了，但显然并不是这么回事。”他调侃道，即便强如 Anthropic，其内部也依然在重度依赖 Salesforce 和 Workday 这样的企业级基础设施。 ```mermaid flowchart LR accTitle: RAG 与模型微调在权限隔离上的对比 subgraph FineTuning["模型微调方案 (Fine-Tuning)"] direction TB F1["所有数据权重混合进模型"] F2["Prompt 越狱攻击风险"] F3["难以实施角色访问控制"] F1 --> F2 --> F3 end subgraph EnterpriseRAG["企业级受控 RAG"] direction TB R1["向量数据库与元数据隔离"] R2["按用户权限缩减检索范围 (40%)"] R3["细粒度安全与监管合规保障"] R1 --> R2 --> R3 end ``` Nicholas 进一步从金融机构的严苛监管视角指出，企业级方案的核心难点在于细粒度的访问控制（Granular Control）——系统必须严格受控，明确知道在何时、为了什么原因、向谁生成了什么内容。在投行业务中，绝不能让未负责某客户业务的银行家查阅该客户的交易数据，否则会引发巨大的法律和监管合规灾难。回到此前“两座图书馆”的比喻，企业级 RAG 的真正价值在于能够结合权限系统，在向量空间中直接将检索范围缩减为当前终端用户被允许查阅的那 40% 的向量嵌入（Embeddings）。Nicholas 直言，那些叫嚣“RAG 已死，微调才是未来”的论调完全站不住脚：一旦将数据通过微调写入模型权重，就丧失了物理权限隔离，任何精巧的提示词越狱（Jailbreak）都能把不该公开的数据套取出来。 --- ### [18:21](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1101s) Excel 的统治力与代码生成的认知鸿沟 主持人顺势抛出一个引发全场共鸣的问题：在 AI 技术如此突飞猛进的今天，金融、人力资源、法务等核心行业中，使用频率最高的工具究竟是什么？台下观众立即道出了事实：“Excel”。技术界热烈讨论 RAG、Agent 和前沿大模型，但 Excel 的统治地位依然稳固，这反衬出所谓“更聪明的 AI”在真正落地企业业务时所面临的复杂现实。 接着，主持人将话题引向方案评估：如果供应商或内部团队拿来一套已经搭建完毕的 RAG 管线，管理层该如何验收？Ameen 表示绝不能照单全收，他用日常研发管理中的真实体会作出了深入说明。在公司积极推行 AI 辅助编码 Agent 的过程中，团队确实观察到了开发速度的飞跃以及抽象能力的提升，但问题也随之暴露——初级工程师与资深工程师使用同样的 Coding Agent，产出的软件质量却存在天壤之别： | 维度 | 初级工程师（易产出代码垃圾） | 资深工程师（高质量生产力） | |:--- |:--- |:--- | | **交互提问方式** | 仅输入最终状态（“帮我做这个”），任由模型自由发挥 | 明晰构建路径，以精确步骤（A $\to$ B $\to$ C $\to$ D）下达指令 | | **底层直觉沉淀** | 缺乏实际踩坑经验，无法识别模型生成的隐蔽缺陷 | 具备数十年实战积累的直觉（Instinct），能敏锐把关架构 | | **产出代码特征** | “感觉编程垃圾”（Vibe-coded slop）层层堆叠，累积隐形成本 | 结构合理、抽象清晰的高质量软件工程资产 | Ameen 警示，过度依赖“只描述结果、任由 Agent 发挥”的模式，会导致大量粗制滥造的低质代码（Slop on top of slop）不断堆叠，企业未来必将为此付出沉重的技术债务代价。他半开玩笑地看向同在台上的学者嘉宾们，指出真正的隐忧在于教育与下一代人才培养：当前的高级工程师之所以能驾驭 AI，是因为他们曾在没有 AI 的年代写了几十年底层代码、建立了敏锐的工程直觉；而在代码生成工具普及的当下，下一代真正的软件专家究竟该从何而来，是全行业必须直面的深刻拷问。 ### [22:11](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1331s) “氛围编程”与盲目信任的隐患 在工程实践中，资深团队与初级开发者面对 AI 生成内容时往往持有截然相反的态度。经验丰富的工程师通常从“质疑与验证”（trust but verify）出发，而许多初出茅庐的年轻开发者则习惯直接进行“氛围编程”（vibe coding）——盲目轻信模型输出的每一个代码片段或分析结论，甚至在未经严密审查的情况下就急于将其推向生产环境。特别是在金融、合规和强安全监管等对审计能力要求极高的行业中，AI 偶尔产生的幻觉、逻辑漏洞或与现有系统架构严重脱节的产物，若未经甄别直接上线，将带来巨大风险。 这种过度依赖不仅存在于工程领域，在高校教育和商业分析中也屡见不鲜。华盛顿大学的 Chirag Shah 教授分享了一则生动的轶事：有学生在提交的代码作业中，赫然包含一段 AI 生成的注释：“找不到 XYZ 数据，所以我直接模拟了一段，我认为这对该公司的债券结构是一个很好的示意。”学生甚至没有在代码中按 `Ctrl + F` 检索或检查注释内容，就直接当成正确作业提交。这种缺乏对底层逻辑与技术细节推敲的现象，凸显了在生成式 AI 时代，培养下一代工程师不仅关乎编写代码的科学，更关乎在试错与挣扎中积累经验与审慎判断的“艺术”。 - 经验丰富的工程师往往秉持“从不信任开始”的严谨审查逻辑，而初级开发者容易直接信任 AI 生成物并急于上线。 - Chirag 教授指出，学生作业中甚至直接出现模型自述“我编造了这段数据”的注释，反映出使用 AI 时验证意识的严重匮乏。 --- ### [24:49](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1489s) 金融尽职调查问卷（DDQ）的 RAG 落地实践 尽管 RAG 的运维成本与边际效益递减常引发讨论，但其在特定高价值垂直场景中展现出了显著的业务回报。Lexington Partners 的 Richa Singh 分享了 RAG 在私募股权与资产管理领域的核心落地案例——尽职调查问卷（Due Diligence Questionnaire, DDQ）。当机构投资者考虑向基金注资时，会向管理团队发送大量涉及投资策略、风控和合规流程的标准化与定制化问卷。在受强监管的金融行业中，过往所有回答都必须经过法务与合规团队的严格审核，企业内部通常沉淀了庞大的历史问卷库。 在实际业务中，投资者提出的问题往往大同小异，但提问的角度、表述方式和具体上下文却千差万别。通过引入由嵌入模型（Embedding Model）、向量数据库及智能体（Agent）构成的 RAG 检索流程，系统能够自动扫描新上传的 DDQ 文件，提取具体问题，并在历史经法务审核的问卷知识库中精准匹配并生成最恰当的草案。引入该系统后，原本需要业务团队耗费数周时间逐条翻阅、撰写和复核的繁琐流程，被压缩到了短短几个小时内，最后仅需由专业人员（Human-in-the-loop）进行最终审核把关。 ```mermaid flowchart TD accTitle: 金融 DDQ 自动化生成 RAG 流程架构 subgraph Ingestion ["数据摄取与知识库构建"] A["历史已审核 DDQ 库"] --> B["文本分块与 Embedding 向量化"] B --> C["企业合规向量数据库"] end subgraph Processing ["智能体提取与检索匹配"] D["新上传投资者 DDQ"] --> E["Agent 自动解析与提取问题"] E --> F["向量检索与上下文召回"] C -.-> F end subgraph Output ["生成与人工协同"] F --> G["LLM 组装生成标准答案草案"] G --> H["合规人员人工审核 (Human-in-the-Loop)"] H --> I["输出最终回复文档"] end ``` --- ### [27:32](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1652s) 效率转移悖论：从快速原型到严苛的回归测试 AI 工具表面上极大地缩短了初期的开发周期，但并未消除整体工程的复杂度，而是将工作量从前端构建转移到了后端的保障与验证上。Joe Gaska 强调，从一个仅需一个下午就能搭建完成的概念验证（POC）或 Demo，跨越到能够经受生产环境检验、值得团队“为其背书”的正式系统，存在一条极其陡峭的学习与工程壁垒。 为了在生产环境中确保输出的一致性与合规性，团队必须对系统实施严格的变更控制与审计追踪体系： 1. **全链路变更回溯测试（Back-testing）**：无论是底层文档内容变动、Embedding 模型微调、向量索引更新，还是数据库模式（Schema）演进，每一处细微变更都必须进行回测。 2. **自动化回归测试集**：利用已知的成功历史用例构建自动化基准测试套件，验证每一次系统更新后模型能否始终产生一致且准确的答案。 3. **因果溯源与版本控制**：当生产环境输出出现偏差（go awry）时，系统必须具备端到端的审计与版本管理能力，以精准定位是哪一部分文档或检索权重引发了异常。 团队在初期往往低估了后端测试与验证工具链的投入，经历了惨痛教训后才加倍补足了变更控制与回归测试体系的建设。这种对可预测性与绝对掌控力的追求，也是金融与传统企业在面对 AI 浪潮时依然离不开 Excel 等确定性工具的核心原因。 - AI 提升了前期的开发效率，但后端的测试、版本控制与验证预算必须大幅增加。 - 必须针对已知正确案例进行持续回归测试，确保文档和模型更新不会导致输出质量漂移。 --- ### [30:30](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1830s) 智能体宪法与混合检索：结构化数据与向量库的边界 在现场问答环节，一位观众提问：在面对包含数千份文档的知识库时，Agent 常常在关键词检索与向量嵌入检索之间摇摆，如何有效设计决策路由，让 Agent 选用最恰当的工具？ ```mermaid flowchart LR accTitle: 智能体宪法驱动的混合检索决策机制 Q["用户查询请求"] --> Router{"Agent 宪法规则路由"} subgraph Unstructured ["非结构化数据层"] Router -->|"品牌语气 / 训练理念 / FAQ"| VDB["向量数据库 (Embeddings)"] end subgraph Structured ["确定性结构化数据层"] Router -->|"课程排期 / 教练姓名 / 淋浴与停车设施"| DB["结构化数据库 (SQL / APIs)"] end VDB --> Merge["上下文汇聚与生成"] DB --> Merge Merge --> Trace["链路追踪与回归测试 (验证数据来源)"] ``` Purpose Brands 的 CTO Ameen Kazerouni 以旗下健身品牌 Orangetheory Fitness（奥兰治健身）的实际架构为例进行了解答： - **内容形态与检索路径的解耦**： - **非结构化数据**：品牌调性（Brand Voice）、FAQ、训练机制、心率区间定义等高度标准化的内容，非常适合存入向量数据库并通过 Embedding 检索。 - **结构化数据**：课程排期、代课教练、具体门店是否配备干洗吧（Dry Bar）、淋浴间或停车场等硬性设施信息，必须由专门的 API 与结构化数据库提供。若用户满怀期待驱车一小时前往健身房准备课后直接上班，却因 AI 幻觉发现该门店没有淋浴间，会对品牌口碑造成灾难性打击。 - **引入“智能体宪法”（Agent Constitution）**：在多智能体体系中，为每个 Agent 确立严格的调用宪法，硬性规定哪类意图必须访问确定性的结构化工具，坚决避免将所有数据不加区分地塞入向量数据库。 - **全链路追踪与溯源回归测试**：企业正在引入第三方评估与回归测试软件，监控 Agent 的每一步调用链路（Trace）。系统不仅要验证模型是否给出了正确答案（如报出了正确的教练姓名），更要严格审查该信息是否来自权威的结构化排期接口，防止 Agent 偶然从过期的旧博客文章中“蒙对”答案。 ### [32:54](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1974s) 评估策略与结构化数据优先：不要把所有数据都塞进向量 小组讨论延续了系统质量与可信度的验证主题。发言者强调，在查询教练信息或门店数据等具体场景时，通过特定工具和明确来源验证信息至关重要。评估追踪（traces）和进行彻底的回归测试（regression testing）是系统的核心支柱。从设计的第一天起就必须纳入评估策略，甚至应该从评估出发“倒推构建”（build backwards），否则系统注定会在生产环境的边缘场景（edge cases）中崩溃。 关于是否所有场景都要使用 RAG，讨论引出了行业对流行术语泛化现象的剖析。业界往往热衷于把最新流行的缩写标签贴到所有有效的方案上以推动采纳与销售。但技术原则非常明确：当有结构化数据可用时，绝对不要硬把非结构化数据强塞进 Embedding 向量中。无论是构建整体架构并将其统称为 RAG，还是通过 MCP（Model Context Protocol）工具暴露结构化 API，名字并不重要，核心在于数据必须来自具备正规血统与来源（provenance）的可靠源头。 ```mermaid flowchart TD accTitle: 数据检索路由与架构选择 subgraph Input["用户查询与需求"] Q["用户请求"] end subgraph Routing["数据源评估与路由"] Check{"是否存在结构化数据？"} end subgraph Pathways["执行路径"] API["MCP 工具 / 结构化 API / 数据库"] Vector["向量化检索 (Vector Embedding / RAG)"] end subgraph Verification["质量与评估机制"] Eval["Trace 追踪评估与回归测试"] end Q --> Check Check -- "是 (如门店/人员精确信息)" --> API Check -- "否 (非结构化文档)" --> Vector API --> Eval Vector --> Eval ``` - 评估体系必须在架构设计初期就确立，避免在实际投产时遭遇极端案例失效。 - 结构化与非结构化数据应分流处理，精准 API 查询优于盲目的文本向量化。 - 数据血统（provenance）和来源溯源是保障企业级生成式 AI 真实性的基石。 --- ### [34:41](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2081s) 破解大型企业数据孤岛：联邦搜索与“RAG of RAGs” 现场观众提问：在大型企业内部普遍存在数据孤岛（data silos），单个员工自行开发小工具可能不足以支撑企业级方案，但在“氛围编码”（vibe coding）和新兴技术的普及下，个人挖掘数据依然具有巨大价值。面对这种情况，构建语义层和 RAG 的最佳实践是什么？ 专家指出，分布式数据架构具有多个层级。解决数据孤岛的关键在于向上构建“联邦搜索”（federated search）层与统一治理机制： 1. **联邦层（Federation Layer）**：在底层分布式系统之上建立统一联邦接入层，集中管理访问权限与合规治理。 2. **语义层（Semantic Layer）**：在联邦层之上叠加语义层，统一规定“何时、何地、为何访问何种数据”。 3. **分层 RAG 架构（RAG of RAGs）**：每个独立的数据孤岛建立各自专属的 RAG 管道，顶层的联邦协调层充当路由，自动识别意图并将请求分派给对应孤岛的子 RAG。 --- ### [37:00](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2220s) SaaS 系统演进与随时间漂移的数据：构建三维时序语义层 观众 Rajiv Mehrotra 提出了复杂企业环境下的数据演进难题：企业级 Agent 需要与 Salesforce、理赔系统等多种 SaaS 及遗留系统交互，而这些外部系统的 Schema 经常更新、发布说明模糊，且内部数据往往具备强时间序列特征（例如 2022 年未填报的数据可能只是当时字段不存在，而非事实缺失）。如何让 RAG 理解这种随时间演进的数据？ Joe Gaska 解释了应对数据漂移（schema drift）的核心逻辑： - **完整捕获历史数据**：SaaS 平台的 API 通常存在访问限制与权限约束，企业必须将全量历史数据抽取并存放到自己的超大规模云平台（hyperscaler）大数据设施中。 - **上下文版本化与时间戳绑定**：将上下文层（contextual layer）赋予版本号，把上下文与数据随时间演进的轨迹进行绑定。 - **三维时间序列数据库视角**：将数据系统视为带有历史时间戳的三维空间，Schema 在变，顶部的语义层必须能够准确解释“在那个特定的历史时间点，该字段代表什么含义”。 --- ### [39:22](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2362s) 原始数据的保真度与预处理：从金融场景看上下文提炼 面对观众关于“数据混乱（messy data）时如何评估投入产出并落地 RAG”的追问，小组给出了明确的技术准则： - **严禁破坏原始数据**：保持原始材料（raw materials）的最天然形态，不要在源头进行破坏性转换，因为一旦改动便无法找回原始保真度。数据的规范化（normalization）与反规范化（denormalization）应放在后续处理层完成。 - **建立强有力的预处理层**：Richa Singh 指出，高质量 RAG 依赖于“从原始材料到知识再到信息”的处理链条。在金融业务中，如果能在数据接入 RAG 之前，把交易信息（deal information）、市场动态关联起来，打上元数据，并处理好各司法管辖区的合规边界（如中国或瑞士的数据合规限制），系统输出的质量将显著高于直接塞入原始粗糙数据。 --- ### [41:21](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2481s) 30秒闭门总结：回归数据本质与治理底座 主持人邀请各位嘉宾用 30 秒进行最终总结，各位专家提炼了关于企业 AI 与 RAG 的核心认知： - **Joe Gaska**："Trust but verify"（相信但要验证）。从第一天起建立版本控制与流程框架，把企业全量数据以最干净的原始形态保存在自己的云端，业务流程虽会变化，但原始保真度不可丧失。 - **Ameen Kazerouni**：尽管这是一个 RAG 研讨会，但讨论的核心与十年前大家穿“数据是新石油”T恤时别无二致。当前所谓“RAG 失败”，本质上都是上游的数据治理问题（data governance problems）。模型固然重要，但数据的整理（curation）与评估（evaluation）重要性毫不逊色。 - **Richa Singh**：“坏食材做不出美味佳肴”。面对“上下文工程”（context engineering）、“循环工程”（loop engineering）等层出不穷的新术语，团队可以将 10% 的精力用于前沿探索（R&D），但 90% 的重心必须扎根于不变的数据基本功。 ### [43:35](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2615s) 拒绝技术锁定：敏捷迭代与架构灵活性的演进价值 嘉宾在总结发言中给出了至关重要的实践建议：切忌将技术架构锁定在任何特定的单一实现方案中。回顾两年前团队首次部署 RAG 解决方案的历程，从 2023 年底到 2024 年初起步探索第一个落地用例以来，底层技术与架构经历的变化与迭代升级堪称“超乎想象”（beyond comprehension）。如今系统所达到的成熟度与技术深度，早已远远超越了团队最初启动时所能设想的边界。 正是由于始终保持开放、避免被某一种特定技术栈或现成方案深度绑架，团队才具备了在技术狂潮中敏捷应变的能力。这种解耦且具弹性的策略让工程团队能够持续、快速地进行原型制作（prototyping）、升级替换组件，从而以最高效的节奏将前沿能力转化为切实的业务价值交付。 - **架构解耦策略**：坚决避免技术供应商或单一实现方案锁定（vendor/implementation lock-in），为模型替换与技术栈升级留出充分缓冲。 - **演进超预期**：自 2023 年末至 2024 年初首个用例落地以来的迭代速度证明，企业级 RAG 的长期收益取决于系统能否顺畅融入持续发布的技术突破。 ### [44:27](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2667s) 摘下浮躁滤镜：聚焦真实问题与缓解 AI 焦虑 华盛顿大学教授 Chirag Shah 在圆桌尾声特意戴上“教育者的帽子”向在场的所有企业决策者与开发者分享结语。他直言不讳地告诫大家：“停止盲目追赶最新技术潮流，将重心回归到问题本身。”（"Stop chasing the newest trends. Focus on the problems."）这一观点也是贯穿整场研讨会的核心主线。 当前行业中层出不穷的新名词、新热词接踵而至，无论是检索增强生成（RAG）、提示词工程（Prompt Engineering）还是闭环工程（Loop Engineering），技术标签永远只是工具手段而非终极目的。面对铺天盖地的技术宣传，企业切不可陷入许多人普遍面临的“AI 焦虑”（AI anxiety）之中。与其在技术狂热中盲目跟风，不如沉下心来深挖组织内部亟待解决的核心业务痛点与数据根基。在全场观众的热烈掌声与致谢声中，这场关于 RAG 商业实操与落地战略的圆桌研讨圆满收官。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=PTbKrnsB05s&t=4s) 如果你来这里是想听‘什么是 RAG’以及‘如何搭建基础 RAG’，那你已经迟到了整整两年。 — Chirag Shah > [11:10](https://www.youtube.com/watch?v=PTbKrnsB05s&t=670s) 一间房间里杂乱堆放着世界上所有的书，所有人都可以声称‘我们拥有全部数据，答案全在里面’；而另一边是一座井然有序的现代化图书馆，尽管馆藏只覆盖了世界上 40% 的书。当需要寻找精准答案时，任何人都会毫不犹豫地走向那座管理有序的图书馆。 — Ameen Kazerouni > [16:14](https://www.youtube.com/watch?v=PTbKrnsB05s&t=974s) 彭博已死，我用 Claude Code 两个小时就凭感觉敲出了一个彭博终端。 — Nicholas Ginsberg > [16:14](https://www.youtube.com/watch?v=PTbKrnsB05s&t=974s) 一旦将数据通过微调写入模型权重，就丧失了物理权限隔离，任何精巧的提示词越狱都能把不该公开的数据套取出来。 — Nicholas Ginsberg > [22:11](https://www.youtube.com/watch?v=PTbKrnsB05s&t=1331s) 找不到 XYZ 数据，所以我直接模拟了一段，我认为这对该公司的债券结构是一个很好的示意。 — Chirag Shah > [41:21](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2481s) Trust but verify（相信但要验证）。从第一天起建立版本控制与流程框架，把企业全量数据以最干净的原始形态保存在自己的云端，业务流程虽会变化，但原始保真度不可丧失。 — Joe Gaska > [41:21](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2481s) 当前所谓‘RAG 失败’，本质上都是上游的数据治理问题。模型固然重要，但数据的整理与评估重要性毫不逊色。 — Ameen Kazerouni > [41:21](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2481s) 坏食材做不出美味佳肴。面对‘上下文工程’、‘循环工程’等层出不穷的新术语，团队可以将 10% 的精力用于前沿探索，但 90% 的重心必须扎根于不变的数据基本功。 — Richa Singh > [44:27](https://www.youtube.com/watch?v=PTbKrnsB05s&t=2667s) Stop chasing the newest trends. Focus on the problems.（停止盲目追赶最新技术潮流，将重心回归到问题本身。） — Chirag Shah ## Mentioned Ameen Kazerouni, Chirag Shah, Nicholas Ginsberg, Joe Gaska, Richa Singh, Rajiv Mehrotra, University of Washington, Purpose Brands, Morgan Stanley, GRAX, Lexington Partners, Goldman Sachs, Johns Hopkins University, Columbia Business School, Orange Theory Fitness, Anytime Fitness, The Bar Method, Wax in the City, Zappos, Salesforce, SAP, Workday, ServiceNow, Bloomberg, Anthropic, Claude Code, Excel ## Video description Ameen Kazerouni — CTO, Purpose Brands Chirag Shah — Professor, University of Washington Nicholas Ginsberg — Executive Director, Morgan Stanley Joe Gaska — Co-Founder & Chief Product Officer, GRAX Richa Singh — Head of AI & Data, Lexington Partners Retrieval-Augmented Generation (RAG) is emerging as a key enabler of enterprise-grade generative AI, bridging the gap between static models and dynamic, real-time data. This use- case oriented panel will explore how organizations are using RAG to enhance accuracy, improve trust, and deliver meaningful business outcomes. From customer support and internal knowledge access to legal, finance, and operations, hear how leading enterprises are implementing RAG architectures to supercharge their generative AI applications. Learn about best practices, tooling options, and performance measurement frameworks to turn your unstructured data into strategic advantage. [Source](https://www.youtube.com/watch?v=PTbKrnsB05s)

[0:04] Chirag Shah: 如果你来这里是想听‘什么是 RAG’以及‘如何搭建基础 RAG’，那你已经迟到了整整两年。 [11:10] Ameen Kazerouni: 一间房间里杂乱堆放着世界上所有的书，所有人都可以声称‘我们拥有全部数据，答案全在里面’；而另一边是一座井然有序的现代化图书馆，尽管馆藏只覆盖了世界上 40% 的书。当需要寻找精准答案时，任何人都会毫不犹豫地走向那座管理有序的图书馆。 [16:14] Nicholas Ginsberg: 彭博已死，我用 Claude Code 两个小时就凭感觉敲出了一个彭博终端。 [16:14] Nicholas Ginsberg: 一旦将数据通过微调写入模型权重，就丧失了物理权限隔离，任何精巧的提示词越狱都能把不该公开的数据套取出来。 [22:11] Chirag Shah: 找不到 XYZ 数据，所以我直接模拟了一段，我认为这对该公司的债券结构是一个很好的示意。 [41:21] Joe Gaska: Trust but verify（相信但要验证）。从第一天起建立版本控制与流程框架，把企业全量数据以最干净的原始形态保存在自己的云端，业务流程虽会变化，但原始保真度不可丧失。 [41:21] Ameen Kazerouni: 当前所谓‘RAG 失败’，本质上都是上游的数据治理问题。模型固然重要，但数据的整理与评估重要性毫不逊色。 [41:21] Richa Singh: 坏食材做不出美味佳肴。面对‘上下文工程’、‘循环工程’等层出不穷的新术语，团队可以将 10% 的精力用于前沿探索，但 90% 的重心必须扎根于不变的数据基本功。 [44:27] Chirag Shah: Stop chasing the newest trends. Focus on the problems.（停止盲目追赶最新技术潮流，将重心回归到问题本身。）

[0:04] Chirag Shah: about two years too late [24:49] Richa Singh: what used to take multiple weeks, it is just done in couple of hours [34:34] Joe Gaska: creating a federation layer that is unified to access all the distributed sources... allows you to govern access [1:16] Nick Ginsburg: Bloomberg is dead [20:04] Chirag Shah: couldn't find xyz data so I'm just gonna simulate it... I am making this up

[0:04] [snorts] Thank you. Good morning everyone and  welcome to this first panel. I am Churak Sha. [0:10] Um I'm a moderator of this. Uh I'll get to that  in a minute. Uh but just a quick introduction. [0:15] I'm a professor uh at University of Washington in  Seattle where it's not this hot. Um but u glad to [0:24] be here. So a lot of my work is in Asian um AI.  I do a lot of work with industry as well as uh [0:31] working on my first startup. So we'll see how that  goes. Um but I'm excited here to talk about rag. [0:37] Now rag is not something that's new. Hopefully  not to this audience here. So we're not here to [0:42] talk about hey what is rag and how to do this.  But after and if you're looking for that kind [0:47] of panel you're like about two years too late. Um  after spending all this time now we are ready to [0:52] talk about where it works where it doesn't work  right and so we have this incredible panel here [0:59] um who would share their experiences from  their own work and what they've seen and [1:03] observed. Um so the way this is going to go  and this is a little bit of format. So first [1:09] uh we're going to go through our introductions.  The panelists will introduce themselves and just [1:14] very quickly briefly tell us how they  are doing rag in where they are. Um, [1:19] then we'll go through some uh set of questions.  Um, and I hate and we were talking about this [1:24] before that I hate doing panels where we are  just doing one-on-one short interviews. So, [1:29] we are intentionally doing questions that are a  little bit more nuanced rather than, you know, [1:34] asking some textbook questions. Um, and we'll try  to keep it lively. So, so we're not going to do [1:40] one question that everybody gets to answer. I'll  probably direct questions to individuals here. Um, [1:45] and then we'll try to leave some time for audience  questions. Um, and so as we're going through this, [1:51] be thinking about what you might want to ask this  panelist. Uh, we'll try to spare a few minutes [1:58] towards we only have 45 minutes total. Um, but  it's good possibility that we'll run out of time. [2:04] And in that case, I would strongly encourage  you to find us outside of this room at some [2:09] other time. Come and talk to us. We have a lot of  interesting things to say. At least I can tell you [2:13] that having interacted with this panelist um they  got lots of interesting things to say. So do take [2:19] advantage of it. So with that, let's uh start our  quick introductions uh with Nick. Awesome. Good [2:25] morning everybody. It's really great to be here.  I'm Nick Ginsburg. I'm an executive director at [2:29] Morgan Stanley in the investment banking division  using AI and machine learning and data science to [2:35] help deliver smarter analytics to our bankers and  to our clients. I'm also an adjunct professor of [2:41] quantitative finance at Johns Hopkins and uh where  I teach uh basically students how to use uh all of [2:48] those tools for uh financial analysis. A quick  legal thing um if there's anybody from the media [2:54] uh please no quotes without running them uh by  myself and Morgan Stanley uh first I would really [3:00] appreciate that. Um and then in terms of use  of RAG uh you know we have 5,000 bankers spread [3:06] across the world. We're constantly pitching deals.  We're constantly putting together presentations [3:10] and pitches. And you know, you can imagine how  many uh of those have overlap, how much of that is [3:17] just unstructured data um that is sitting in, you  know, very nicely formatted powerpoints. And so, [3:24] uh from an efficiency standpoint, from  a what are what is going on in the firm, [3:27] it's a huge challenge in terms of um we want to  make sure that we have the best information that [3:33] we're giving to our clients. Uh but we also don't  want to be, you know, uh redoing the same thing [3:38] multiple times if somebody's already done a really  great pitch deck on product XYZ. Uh wouldn't it be [3:44] great if others could reuse that and and do it in  a safe manner where we're not um exposing data to [3:49] to people who who shouldn't have it but can unlock  the gains uh of being a a global organization. [3:58] Good morning everyone. I'm Rich Singh. I  head data and AI department of Lexington [4:04] partner. It is one of the largest secondaries  private equity firm and I joined Lexington in [4:10] 2024 and before that Nick joined like when he  joined Morgan Stanley we both were together in [4:16] investment banking of Goldman Sachs. That was the  first time I got introduced to Rack. We have been [4:22] I've been working in rack since early 2023 and  you know in the course of last two and a half [4:29] years which is actually three and a half years  which is almost 30 years in industry I should [4:35] say the with the pace of AI is going on we have  seen a massive evolution of how things are moving [4:41] towards rag the definition of rag is changing  and what are the different use talk about those [4:49] um little bit of more of My intro uh my  journey started in the AI field before [4:55] even generative AI became a thing. In 2013 I  decided to take a break from career and went [5:01] for higher studies and studied machine learning.  So I spent over 13 14 years in this space. Um now [5:08] you know have seen the whole progression  going forward. Um besides my day job I'm [5:15] also not as professor at junk professor but I  do spend some time at Columbia business school [5:21] uh talking about taking couple of lectures  sorry thank you so much um taking a couple [5:29] of lectures every semester talking about genai in  business and rag again there is a very core part [5:35] of that keeps coming so I'm very excited about  the discussion today with this amazing panel [5:42] Hi, uh my name is Amin Kazeruni. I'm the chief  technology officer at Purpose Brands. Purpose [5:49] Brands is the parent portfolio company that  owns uh Orange Theory Fitness, Anytime Fitness, [5:56] the Bar Method, Wax in the City. So, uh lots of  data, very data richch environments. Uh I don't [6:02] know how many of you familiar with Orange Theory,  one of the largest kind of contextualized heart [6:06] rate data sets that are in the world. Anytime  Fitness has uh gym visitation data sets from [6:12] 5,000 locations all over the world. So 40 plus  countries, 10 12 plus languages. Uh I'm before [6:19] that I used to head up the AI and ML platforms  about five or six years ago for Zappos which was [6:24] I used to live here in Vegas. Uh we headquartered  in downtown Las Vegas. So um obviously I'm in the [6:32] thick of you know the demands to sprinkle some  AI on it and u still recovering from that but [6:38] finding real business value is is very challenging  and something that we take very seriously how we [6:43] deploy our data machine learning and statistics  has always been part of it but using generative [6:48] AI to drive automations to drive new insights to  drive new value is is obviously critical for our [6:53] business but we want to do it right. Uh my focus  this year is on trying to take a suite of agents [6:59] agentic orchestration rag to the heart of our  business which is membership sales trying to find [7:04] a way to nurture conversations with prospects in  many languages across many brand contexts across [7:10] the globe. So it's an exciting time to be working  on such awesome tools. Nice. Hi, I'm Joe Gazka. [7:18] I'm the chief product officer um of company called  Gra and the the direction of our product itself is [7:28] was all about collecting historical data from  SAS products and whether it's Salesforce SAP [7:34] workday service now collecting every version of  data and creating a semantic layer on top of the [7:39] data itself. So when you talk about rag and you  talk about the dynamics of historical data and [7:45] the value and the time to value and that's really  where our product has been focused. So whether it [7:49] be rag internal for our business from operational  efficiency product develop product develop [7:54] developments or within the product itself. I focus  more on the rag application of the productization [8:00] of the data for our customers in the fortune  500. the dynamic nature of the data itself, [8:04] you know, the the morphological variance of the  data, the evolution of the schema. That's really [8:10] where the rag power comes in for us and what  we're talking about. Thank you. So with that, [8:15] let's uh jump into some questions. So just to  uh quick reminder, this is the panel is titled [8:21] the rag use cases unlocking business value with  smarter AI. So what we're going to talk about are [8:27] some of the lessons we've learned in implementing  rag some of the things that where you hesitated [8:32] or or cancel rag right and so let's start with um  um data itself right because one of the things and [8:39] and Joe I'm going to ask you this um one of the  attractions behind rag is you can have your own [8:46] enterprise data that you can you know train the  the rack system on and that becomes your moat. So [8:53] is is that really the case that that you've seen  because we can also see that enterprise data could [8:59] be messy right and challenging to work with uh so  is is that a good trade-off. [snorts] So when you [9:06] think of data, data is you know the raw material  right the raw material going into business looking [9:12] for the looking for the output but how is that raw  material used and utilized to have impact right [9:18] and you spoke of the data by nature is dirty in  a lot of enterprises the little un little little [9:24] untold secret of most enterprise data whether it's  your CRM whether it's your ERP whether it's your [9:29] workday system the data itself and not only that  the business processes are changing all the time [9:35] how they're using it, how they're using the tools.  Not only that is the schema evolution over time, [9:41] right? So looking at this and saying not only do  I put a rag model on top of my data right now, [9:46] but that has to be versioned itself like code as  it evolves over time, right? All of that really [9:51] starts to come in. The moat is there. It's the  raw material, but when you really think of the [9:56] rag on top of it is how do you actually take  that raw material and transpose that to derive [10:01] value from it. That's really where we focus that  we focus right there and to how do we take that [10:06] raw material and for the for us the rag right  there and a lot of people say hey the semantic [10:11] layer right how do we understand or interpret our  data itself what are the business processes on top [10:16] of that that's really where it comes in for all  of our customers and our business internally right [10:20] there it's dynamic but it is the mode but how do  you actually use that raw material to drive the [10:26] value so so you have the mode you have the data  but it's not enough there's a cost associated with [10:31] you turning that into a proprietary rack system.  Um, that's what I'm hearing. Well, there's also do [10:39] you really have the data? That's another question  of you have a lot of SAS products of, you know, [10:44] these locked ecosystems, the API limitations  that you have, the access to that, making sure [10:49] that you have ownership and access to that data  and you have the full depth and breadth of that [10:53] data. That's the key to really what what's coming  in and making sure that you protect all that. So, [10:59] so maybe and I'll follow up with um I don't know  maybe Amin um you can take this um in that case [11:04] how do you determine that you have um that value  that you can generate you have the right data [11:10] that you can evaluate it and and say yes you know  we're going to have that um if you were to do the [11:16] in-house you know rag building that we will have  that value that we you know how do you determine [11:21] that you you know that's a great question I'm  going to answer it [clears throat] a little bit [11:25] generically because these problems have been the  same whatever the latest tool to operationalize [11:30] and activate your data is the problem's always  been the same and I when I first joined uh Orange [11:36] theory all those years ago I use this analogy with  the team is uh I don't know how many of you know [11:40] who accompanied Sam Wise GMG to uh to Mordor a lot  of you u so if you imagine a room with every book [11:49] in the world the answers in there and somebody can  own every book in the world and be like I've got [11:53] the data I have the answers to everything and then  somebody's about a library that's well organized [11:59] with 40% of the books in the world. Which one are  you going to to try and answer that question? The [12:05] place with all the data or the place with some  of the data that's organized well? And that's [12:09] the kind of name of the game here. Whether it's  a rag layer that you're putting on top of it, [12:13] whether you're building a semantic model,  curating a Korean data warehouse, the reality [12:18] is that data is yes, it's a moat, but it's also  becoming table stakes in the age of generative AI [12:24] where synthesizing data is getting easier and  easier. Human generated proprietary data that [12:28] you own that was not agentically generated  that's not in this kind of risk of collapse [12:34] of agents training on data other agents made.  Hold on to that for dear life. Keep that safe. [12:39] But operationalizing that data, building clean  architecture around that data um just creating [12:46] structural organization around it to where you  can truly leverage it is I think when it turns [12:51] into a real mo and I think the raw materials  is right without it you can't build but you got [12:57] to build just having it not necessarily enough.  So, so now let's talk about operationalizing it, [13:02] right? So, building this this kind of system.  Um, and Rich, I want to ask start with you. So, [13:08] how do you um, so, so you can do scaffolding,  right? You can build your own infrastructure, [13:13] but at some point you have to ask like is it worth  it, right? How do you decide that, right? How, [13:18] you know, do you have any examples? Do you can  you give us some best practices where you say [13:23] well it's worth investigating in investing into  this because it's going to pay off long time or [13:28] let's just uh you know put together with some  you know models and tools that we can get and [13:33] put some hard harnessing on it. So how do you  decide that? I think no we're touching at the [13:38] exact point what we what we must be discussing in  this panel. Oh thank you so much. uh so um for I [13:47] mean in my journey we have had all the use cases  when we started everything was around rag but as [13:53] we are saying we at the time I'm talking about  2023 we started with a very small token windows [13:58] as like sorry it's a context window was about  4k token now we talking in million so there was [14:04] no other option to go rack pipeline there was  also not that much of agentic architecture as [14:10] we have today so for me it's always been keeping  an eye on where the the evolution of AI is coming [14:18] along and we have to just keep that in mind what  are my use cases what are the best thing forward [14:24] and that's how we make the decision so speaking  of rag rag also comes with a massive maintenance [14:32] right if we are building the whole rag ecosystem  yes it's going to pay off in a long term but I [14:37] want to make sure if my use case is very quick use  case a small one I don't want to just go and going [14:43] to use for a long term. Do I really need drag or  do we really want to you know just have something [14:48] quick straightforward scrubbing of data? Um uh  a couple of thing I'll just touch base a little [14:54] bit on the data and scaffolding related to that.  So we spoke about this like the like my firm has [15:00] 30 years of investment data most of them like all  history and if I have to build something on top [15:05] of it if I just take it as is as we're talking  about an unorganized room full of books it's [15:12] even if I put the rag architecture as is on top  of it it's not going to give me the results I'm [15:18] expecting. They're going to be hallucinations.  They're going to be use cases where you know [15:22] it's not giving the right result. So building the  whole the scaffolding for first it's like how we [15:29] are selecting the right data to input how we are  attaching the right metadata to that data to just [15:36] make sure we have the good pipeline. So the whole  data pipeline is actually for me is the main stuff [15:43] that is just going to be the foundation to the  rag and sometime we have seen is like building [15:47] the pipeline is so expensive that not for every  use case you would like to go rag route sometime [15:53] you want to just you know what we call as bring  your own documents is bod if you have those stuff [15:59] you know which documents and you don't need  those kind of use cases where the metadata is [16:04] important you just want something quickly out  of those documents. ment there are much more [16:09] straightforward you know solutions available  instead of going to rack so so you're talking [16:14] about the cost of building the pipeline but wait a  minute I've I I learned about this engineer this u [16:20] the marketing person who actually built the whole  rack thing in one afternoon [snorts] I mean like [16:25] how hard can it be I mean what do you think you  know uh it's funny I mean I'm sure many people out [16:32] here especially those in the financial services  industry I see this on LinkedIn all the time like [16:36] there you get the jazzy LinkedIn post that's like  Bloomberg is dead. I vibe coded Bloomberg in two [16:42] hours using Claude code. It's like cool. So now  you're worth hundred billion dollars. Like no, [16:48] you're you're very much not. Um and I've I've  seen a ton of variations of of that. You know, [16:54] Salesforce is dead. I vibe coded in an afternoon  like you know there's we were talking earlier and [16:59] u there's a reason Anthropic uses Salesforce and  Workday, right? Um and and so uh those companies [17:06] are still very very very much u very very strong  in those cases. From um from my standpoint in [17:12] terms of looking at this from a how do you  actually build this out? What's critical for [17:16] us is being in a very regulated industry we need  to know that we have that granular control over [17:24] what the system generates and to who and why and  when. And so um you know we can't expose let's [17:31] say deal info from one client to a banker who is  not covering that client is not on that deal. That [17:37] would be you know of of massive regulatory and  and legal impact. So to your point earlier on [17:43] you know the the two libraries one that has  everything and one that only has 40%. Well, [17:48] if you can scale down your embeddings and say,  well, this is my end user, these are the 40% of [17:54] the embeddings that they are allowed to even  see, then your rag actually has, you know, [17:59] that security and that layer built in, which is  hugely valuable to us. And I've heard people say [18:05] rag is dead. It's all about fine-tuning. Well,  when you fine-tune it, you lose that entire [18:10] ability because now somebody could come up with  some crafty jailbreaky thing to to get access [18:16] to data that that the model knows that it that  person shouldn't necessarily have. All right, [18:21] thanks for crushing my dreams of building a rap  rack pipeline in afternoon. Um, cloud six maybe [18:28] cloud six. [laughter and gasps] Actually, it's  funny like we were talking about this earlier. Um [18:32] and then maybe you know I can ask this question  to audience like you know we're doing all these [18:36] amazing we have access to this amazing technology  and all these tools. Um but still like what is [18:43] the one of the you know top most used tools used  in these industries if not the number one tool. [18:50] Does anyone have any guesses in finance, in HR,  in legal, the tool that is used the most? Yes, [19:04] there you go. Excel after all these things, you  think that's going away? What happened to all the [19:11] AI advancement? We were talking about this rag  and agents and stuff. There's a reason for that, [19:17] right? and and this is why we want to kind of  get into a little bit of nuance um behind some [19:22] of these smarter AI that we are talking about.  So let's talk about why is that case that rag or [19:28] something like rag hasn't solved all our problems  you know so so let's get into and maybe I mean I [19:34] want to ask you um about going back to some of the  evaluation things right um tell us about how you [19:42] would assess um let's say somebody comes up to you  like here's either there are vendors or somebody [19:47] in your company is like builds this um uh pipeline  forget about the cost but now you have this [19:53] solution Would you just take it? You would you ask  some questions? Would you do some evaluation some [19:59] way? What would be your approach? So, [clears  throat] I think I wouldn't I wouldn't I'm not [20:04] going to try and design a evaluation framework  on the fly because that's very, you know, that [20:10] would be very uh agentic of me to just take chat  GPT and be like build me an evaluation framework. [20:15] But I think the simple answer is no. Because the  number of times I spend my day with my product [20:22] teams build me bring me fully built out features  as product requirements. Our HR team will bring us [20:28] fully built out solutions as HR requirements.  And the one thing I'm realizing as we've made [20:34] a legitimate push, I don't want to sound like a  skeptic, made a legitimate push for our software [20:38] engineers to take on coding agents. It drives  velocity. It allows them to create abstractions, [20:44] let them focus on real thinking. But the important  piece is are they focused on real thinking and I [20:50] think that's where it differentiates vcoded slop  from you know higher velocity output but genuinely [20:57] good software and I found that my junior most  engineers and my senior most engineers when using [21:05] the same coding agents still produce different  qualities of work. And the reason is it's not [21:12] just the agent, it's the way you ask the question.  And one as we've been this is a hypothesis not [21:16] validated. But what I'm exploring now is the idea  of how did you ask for the thing? Did you stick [21:22] in the end state and just be like make this for me  or did you know how you would want to build it and [21:28] put in the instructions and just skipped over  to doing the next thing because like you need [21:32] to do A then B then C then D. And that balance  between overprescribing instructions versus just [21:38] being like, I'm going to describe an end state. Do  whatever you need to get there is the one we need [21:44] to strike. Cuz the the costs that get fabricated  on this slop on top of slop on top of slop is it's [21:51] a price we'll pay later. And I'm glad we've got so  many different levels of professors on this panel [21:57] because the real problem is going to be with y'all  because our senior engineers are good at this [22:02] because they have instinct. they developed over  decades of writing code. I don't know where the [22:07] next generation of experts is coming from is just  the reality of where we are right now. You know [22:11] what's odd is I've I've debated this with some of  our engineers is I start with a place of distrust [22:18] versus trust and the a lot of our junior engineers  come right out of the out of the gate vibe code [22:25] something completely trust what's produced and I'm  ready to ship it right and it's absolutely amazing [22:30] where it's obviously you start from a place of  skepticism you know trust but verify and it's [22:36] inherent with how you work and how you operate but  when AI has come so fast and so furious and people [22:42] are so excited about it. they come with automatic  trust of what's the what's being produced and it [22:48] and it comes up and when someone's knowledgeable  that reads it and it's amazing you're like well [22:51] this is completely false right or the artifact  that it produces in the code or whatever it's [22:56] completely outside of your patterns that you've  been built and then you have the some of the [23:00] engineers that come through and like oh well we're  ready to ship it oh no no you know especially with [23:05] security and auditability and compliant industries  like yourselves the inherent just oh I trust [23:10] everything that's produced or what it's given me  back that has been a shocking revelation when I [23:14] see the depth of the you know it's not just the  techies it's you know the business people that ask [23:19] a question about you know some of the investment  materials you're saying and it's producing you're [23:24] like no that's wrong and go ahead yeah to to that  point you know I was telling this story earlier [23:30] that I have brilliant brilliant students don't  get me wrong but um I have had students who have [23:35] submitted code as part of their assignments where  it literally says you know couldn't find xyz data [23:42] so I'm just gonna simulate it and I think that  this would be a good illustrative you know bond [23:48] structure for this company and then that makes it  into their submitted assignment. I'm like did you [23:52] even it's not even a matter of being a subject  matter expert. It's like the code comment says [23:58] I am making this up and you know did you you know  they they just inherently like yep looks good to [24:03] me here you go and I'm like did you control F in  the code for like making this up? [laughter] Yeah, [24:09] and I think this is becoming a big big issue and  again uh I can go on and on about it you know as [24:14] a professor as an educator how challenging it  has become to train the next generation in some [24:20] of this uh not just the science but the art of you  know coding and but you know coding related things [24:26] right because um it's very easy to vibe code and  and not know like some of the things that only [24:33] comes through the experience and struggle and and  and working through it you know some things just [24:38] comes through trying so Anyway, like I don't  want to get into the whole um these people, [24:43] these kids these days thing, right? Um but  let's talk about the business cases here. So, [24:49] one of the things um and and Rich, I want to come  back to you on this um as we make choices, right, [24:56] about productionizing some of these things. Um  earlier you mentioned there are there are some [25:02] diminishing returns, right? to you know rag  um you know as you add more like data more [25:08] documents seems like it it it goes up you know the  performance but at some point it doesn't and so [25:14] um are there trade-offs are there like ways you  decide where uh the value is is good enough and it [25:22] doesn't pay off like you know in terms of uh you  know investing into that or continuing with that. [25:29] Yeah, I think uh we we spoke about cost a little  bit and we spoke about like all the overhead of [25:35] the rag but I just want to tune on the other  side. I just want to talk about the positive of [25:41] the racks is like there are many use cases where  I found it's like so useful. That's one of the use [25:45] case we have um I'm not sure if how many people  are from finance industry but it's like DDQs. Do [25:51] you have familiar with DDQs? No, it's like due  diligence questionnaire. So if an investor is [25:56] investing in your fund, they send out DDQs and  you have to just fill it up. So the DDQs is like [26:03] originally most of the enterprise they have the  whole database. It's like if this question is [26:08] asked answer this it is approved by legal and  compliance again highly regulated industry. [26:14] So we have the whole DDQ database and then it's  not just that every time you don't get the same [26:19] question it's the same flavor of the question  but the answer is very different. Rag was just [26:26] the best there like you know we just brought in  all the historical DDQs we just you know we had [26:33] through you know the whole the same thing is just  like embedding model vector database the whole you [26:38] know the retrieval cycle and now we are realizing  is like we just have to update update our or [26:44] just upload our DDQs we have this whole agent  that scan through it extract the questions for [26:52] each questions it goes ahead and find the right  answers. Boom. What used to take multiple weeks, [26:58] it is just done in couple of hours. And then we  have a human in the loop. They review the answers [27:02] and everything. But that is like one of the use  case I'm just talking about. And there are many [27:06] we can say where it's actually getting in the you  know the production and and doing some good. So [27:12] sorry I just take I want to take the positive  angle not saying only the negative side. And I [27:17] think that's a good so um before I continue I just  want to again remind the audience that we are open [27:21] to taking questions. So um after the next one I'll  I'll see I'll scan the audience if there is any [27:27] question u you can raise your hand and we'll uh  figure out how to get the microphone to you. Um [27:32] but to to continue on that point and one thing you  know the both the positive negative side and and I [27:38] guess that's why this is a nuance conversation  because obviously there are benefits right [27:42] uh but then there are times when either the the  cost is uh kind of shifted right so you can get [27:50] some things done fast but now you have to spend  more time on something else right and so one thing [27:56] um I want to come to you Joe uh you were  talking about this earlier about how [28:01] um the this doesn't necessarily make some things  overall things fast, right? It it it's just kind [28:10] of shifting. So, can you like elaborate on that  where we can make things fast where we now need [28:15] to budget more? Sure. So the one thing that we  realized the absolute critical piece of what [28:22] you were just speaking about was you have to back  test every single change that you have including [28:30] anything with your documents and your model  and your vectors and your schema changes over [28:34] time. So the critical piece is of upfront you can  develop things faster but you still have to go to [28:40] the process of you know trust but verify. There's  a lot of automations you can use for back testing, [28:45] version control, running regression testing as  all these as all this go as all this processes [28:50] how quickly you can kind of go to market. So our  testing framework in the back end doing continuous [28:56] validation and version management of everything  and back testing known success cases to make sure [29:02] you get the same answer every single time with  a change of your documents. That's been the most [29:07] critical piece that we needed to invest in and we  learned that the hard way, right? So making sure [29:13] that right up front when you're actually putting  these in production you have the change management [29:19] control test verification and auditability of all  those changes cuz also when something goes ary how [29:26] do you trace it back to the cause and effect. So  the investment of upfront time savings you also [29:31] have to put it down in the back end saying all  the change controls change process validation [29:36] regression version controls of all these going  out to make sure that the same output especially [29:41] in regulated industries you know that's the  most critical piece that you have to invest [29:46] in we underinvested at first learned the tough  way and obviously had to double down and reinvest [29:50] in the back end so that is definitely something  that the shift over in the back test there and [29:54] treating it like that and and I think this is very  common trend that I've seen across the industry [29:58] that this is one of the things it's very easy to  get in have something produced very quickly. Some [30:04] demos, some proof of concept seems so promising.  If if this is what you can do in one afternoon, [30:09] imagine what you can do in one week and one  month. But then we realize that it's a steep [30:13] curve to go from that initial set of demos to  something that's productionizable and something [30:19] that you can stand by, right? Uh stand behind.  And this is one reason why we love Excel still, [30:26] right? All right. So, we're going to see if  there are questions. We have one in the front [30:30] and then one we'll take from the back if  we can. Yeah. Can help with the microphone. [30:44] Hi, morning. Morning. So I've been working with uh  rag and I started with uh relying almost entirely [30:53] on embeddings and then started to do more where  uh search with embeddings was one tool among many [31:00] for the agents and wonder how you have your agents  decide what's the right way to go when it's kind [31:07] and think think of a you know repository of a  couple thousand documents uh and it's looking [31:12] through them and it seems to be more often going  choosing keywords and uh finding that and saying, [31:19] "Yep, I'm on the right kind of thing here." And  then looking through and grabbing it from there. [31:22] So, just wondering if you have a a way that  you think about which way is better. Want to [31:29] take Yeah, I can I can take a stab at this. We've  we've actually played with this very question and [31:35] uh when you think of a brand like Orange  Theory for example the FAQs, the brand tone, [31:39] the brand voice, what is the workout, what are the  heart rate zones, it's all very well doumented, [31:45] highly curated information that you can leverage  embeddings and vector databases to retrieve. [31:50] But then also being extremely diligent about  schedules, who's a coach. You don't want someone [31:56] that's expecting a studio to have a dry bar an  hour away from home because they're going to [32:00] go work out and then go straight to work and then  not have a dry bar or not have showers or not have [32:04] parking. You can really piss off your consumers  that way. So protecting from hallucinations is [32:10] is super critical. Relying on your structured  data with the right tools in front of it when [32:14] you have structured data over trying to stick  everything in vector databases and embeddings [32:18] has been a hard learned lesson. And every agent  in our agentic suite has a constitution and that [32:25] constitution kind of dictates where do you  go to get what type of information and how [32:29] do you answer and then leveraging and this is  where the buy comes in. We are exploring a lot [32:34] of different software providers that effectively  let you regression test any change you make to an [32:40] agent. Not only did it give you the right answer,  but it get the information from the right place. [32:44] you can monitor that trace. Like maybe it got the  coach's name, right? But if it went and tried to [32:49] use some embedding that's got a coach mentioned  in a blog post, don't do that. You don't know if [32:54] that coach works there anymore. This is where  you get coach information by studio and you do [32:58] it via this tool. So validating the source of the  info and the quality of the answer by evaluating [33:04] the traces and doing really thorough regression  testing has been pretty critical. And you know, [33:09] we discussed this earlier. evaluation strategy  should be part of your design from day one almost [33:17] build backwards from it or you are destined to  kind of die in the edges of a production solution [33:22] is at least my my opinion on on that. So actually  one thing not exactly pushing back but you know [33:28] clarifying it's almost seems like you're saying  not don't do rag here. I I think there is like I [33:35] mean I feel like there's people in this room who  would argue that oh what you just described is [33:39] optimization of your rag pipeline anywhere and  I think that's been the flavor of this entire [33:44] revolution is take the latest acronym that's  very popular and just slap it on anything that's [33:47] working and that's great because adoption is very  important and we want to make it easy to adopt. [33:52] We need names to sell. There's businesses that  need to you know drive the the boom and I get it. [33:58] But what I'm saying is don't use unstructured  data jammed into an embedding when there is [34:04] structured data available. Now whether you build  an architecture and call using all of it rag or [34:09] you treat it as like uh you know rag is only if  you've got vectors in place and if you you know [34:15] get it through an MCP tool exposing structured  APIs that's not rag. I don't I don't care what we [34:20] call it like that's kind of what's important. uh  what I care about is that it's sourced from data [34:24] with the right provenence if it requires the right  provenence you know take question from the back [34:34] and then there was yeah we'll  come back to the front after that [34:41] so [snorts] uh my question is I I work for a  very large firm and in a lot of these large [34:46] firms there are data silos uh where if you if  you get one person to try to mine that data [34:53] might not be powerful enough for the firm to come  up with an enterprise solution but there's true [34:57] value driven from someone developing something  to mine that data. So my question becomes you [35:03] know in the age of vibe coding in the age of  uh you know creating something to to mine that [35:08] leveraging this new technology and creating  semantics layers and rags what are some of [35:12] the best practices and lessons that you would uh  you know give someone that uh is in that position [35:22] So with distributed data there's I mean there's  multiple layers to this to this cake right and [35:27] there's there's a million different ways to solve  it. So as you move up the stack you know the key [35:32] is to think about federated search you know you  can have distributed systems you know throughout [35:38] the enterprise and your access controls to the  distributed systems have to be obviously managed. [35:43] Um creating a federation layer that is unified  uh to access all the distributed sources. So [35:49] you control so you can govern access of that. Um  creating a semantic layer on top of the federation [35:54] layer allows you to to create you know how do I  access and where do I access and why do I access [36:00] that layer itself. So the first thing if you're  talking about distributed systems like yourselves, [36:04] the the the best way that we found is find the  right federation uh approach that you're going to [36:10] take for your data before you get to direct access  and control because if you want to govern who can [36:15] access that data, why they're accessing it, who  they're going through a single federation source, [36:19] and there's a ton of them out there that do it.  That has definitely been the the one that we've [36:23] seen across, you know, both our business and  we have and and our customers businesses who [36:27] actually do this. And and to quickly add to  that, I think what you end up in that model [36:32] sometimes is sort of a rag of rags, right?  You have each silo has its own rag and then [36:38] you have your federated uh layer above each  of those and it sort of knows like, oh, okay, [36:43] I have to direct it to this rag or this one or  or what have you. Question from the front here. [36:54] I'm coming. We're getting good workout  in the morning. There you go. Questions. [37:00] My name is Rajiv Mehotra. Uh a related  question to the point that you had earlier [37:05] uh brought up uh complex enterprises where you  want to use agentic for the natural benefits [37:13] uh have to operate talking to a salesforce a claim  system and so on and so forth. And each of these [37:19] systems if they are SAS are evolving and you don't  know how they are evolving how fast you don't get [37:24] the release notes and even if you do you don't  know how to interpret them and if they are your [37:28] systems legacy systems you don't know what change  when how do you is a practice evolving how to make [37:35] sense of the data that rag is pulling because it's  a time series in 2022 that data may have if just [37:43] because something was not available does not mean  people didn't fill it was not available right we [37:48] could probably spend spend about two more hours  talking about this but we can talk afterwards but [37:51] I'll so the one the most critical piece is the  first thing is capturing your historical data [37:57] even as if it evolves over time right schemas  drift object creation user of the data so that [38:04] data is going to evolve over time and the APIs are  limited to you can only access certain those some [38:09] people might use some of these SAS platforms just  so you know there's there's data access controls [38:13] where you have you don't have full complete  access to your complete data set itself so one [38:18] you have to get it in a big data facility where  you have access to all of that you write time [38:23] series of data as it evolves over time. So what  actually happens in that world is your contextual [38:29] layer has a version much much like your your  data access over time. So there's timestamps on [38:38] the actually context that is bound to your data  as it's evolved over time because think of it [38:44] as a historical time stamp or three-dimensional  database and the schemas are changing over time [38:48] and you have to have that schema semantic layer  explaining what it meant at that point in time. [38:56] Um I can explain a little more detail but that  is that is one of the most crit but the most [39:00] critical point is capture the data all of it get  it and govern it within your hyperscaler right [39:05] and putting the rag on top that's actually trivial  when you actually get down to the layer of it but [39:09] the first first and foremost is get all of your  SAS data with full depth and breath inside of your [39:16] hyperscaler itself then the the other become much  easier to solve with the rag we'll take one last [39:22] quick question because we're going to run out of  time I Do you still have a question? Are we good? [39:27] Yeah. Quick question, quick answer. Okay. Uh so  get all of your data, right? Uh but messy data, [39:36] right? Uh how do you get to that point where you  get all of your data, you have your business value [39:42] in your data and you lay your rag on top. Where do  you what is that level of effort or how do you get [39:51] to that point? I'll take one little bit of Sure.  Quick, quick. I'm sorry to try to make it quick. [39:57] So, first and foremost, don't ever translate your  raw material. Keep it in its most natural form. [40:03] There is a way to get it where you can normalize  or denormalize that data to unify how that is. And [40:08] we that's a different piece. But don't change your  raw materials because you can never get it back [40:12] to the fidelity that you had. Getting the layer  normalized and denormalized so you can actually [40:17] improve the accuracy of the answers on top of  that. That is a different that is a different [40:21] problem. But keep your raw material the way it is.  Mick or go ahead quick. Uh so for every rag to be [40:29] done I think we need to have a very strong data  prep-processing layer and what I call is like raw [40:34] material to the knowledge to the information and  if you are ingesting like some kind of knowledge [40:40] management to be done and then you're ingesting  the data it will have very different result than [40:45] the getting the raw data as is. So I'll take an  example like for example if you know all of your [40:50] let's say I'm just using the financial term is  like all your deal information all of your you [40:55] know the the market information if you can connect  them together up front if you can add the metadata [41:02] or like who can access the we have so much  federation going on different jurisdiction have [41:06] different law I cannot take you know data from  China or Switzerland into US like all those things [41:11] are there if you can do all those pre-processing  before layering the rag on top of it. That's [41:17] where you're going to get the results. If you just  throw it as is, it's not going to work. All right, [41:21] on that note, we're going to start wrapping this  up. So, I'm going to give our panelists one last [41:25] chance to give us their last word in 30 seconds or  less. And that could be about what you've learned, [41:33] what you were wrong about, what you see the future  going, whichever pick, whichever one that you want [41:38] to, you know, go with. 30 seconds or less. Joe,  we'll start with you. So the biggest learning is [41:44] trust but verify right version control put in the  framework put in the processes right from the very [41:51] start capture as much as you can of the data that  is in your enterprise and keep it as clean as raw [41:57] material as you can building on top of that  as your business processes change over time [42:02] but keep the raw material in your cloud itself  but everything you put on trust but verify and [42:08] version control wonderful I mean I would say  it's kind of stunning to see that, you know, [42:13] this is a rag panel, but it's it's the same it's  always been for the last 10 years. It was 10 years [42:18] ago when people started wearing the data as a new  oil t-shirts. We're not we're not talking about [42:22] rag. We're still talking about data governance.  We're still talking about how to organize your [42:26] information, how to own your information, how to  own your own destiny. So, I would say that what [42:31] you're seeing is rag problems, those failures go  upstream of it. They're data governance problems. [42:36] Focus on your data. Get your data right. And when  you're building your AG solutions, curation and [42:42] evaluation is just as important if not more than  the models you're deploying on top of your data. [42:48] That's where I'd leave it. Richard, I'll just I'll  just say on the same line is like you cannot make [42:56] a flavorful recipe out of bad ingredient. So the  data is still the same. And and the last one is [43:02] like I would say just like there's so many jarens  like we spoke about the newer generation coming [43:07] and we can talk but there's so many jarens  also coming along like context engineering [43:11] loop engineering even in the rag there's so many  flavors of rag right now I still believe at the [43:17] core the fundamentals are still the same so you  know sometime I find my team chasing that hey this [43:24] architecture is not going to work let's chase  the latest one and you should have that R&D I [43:29] just keep like 10% R&D is like keep seeing what's  latest is happening but the fundamental is still [43:35] the same and we should not deviate from that. So  that's what my take is. take I would say my main [43:42] takeaway would be to not lock yourself into any  one particular implementation and you know just [43:48] in the last two years we deployed our first rag  solution um about two years ago and the just the [43:55] amount of change and upgrades that we've done  is is beyond comprehension and so I think what [44:03] we first when we first started down this road you  know late 2023 early 2024 with with this first rag [44:09] use case. Um, where we are now is just so far  beyond where we even envisioned we would get [44:16] uh back then. And so, you know, by not locking  ourselves into any one approach, any one solution, [44:22] um, we've been able to just continuously  and and rapidly upgrade and and prototype [44:27] and and deliver value there. And for my last  word, I will put on my educator hat and I will [44:32] say stop chasing the newest trends. Focus on the  problems. Right? And this has been a reinforced [44:38] uh throughout this panel. Um so as you hear  all these words like whether it's rag or prompt [44:42] engineering or loop engineering is focus on the  problems right and so stop that AI anxiety that [44:49] many of us have as you start these journey today  here as you uh go on with this uh conference. [44:55] So thank you so much for joining us. Uh please  thank the uh speakers again panelist [applause] [45:03] enjoy your day.