Fannie Mae, Orbis AI Leaders Fault Infrastructure for Deployment Failures

房利美与 Orbis AI 高管将部署失败归咎于基础设施 企业级检索增强生成系统之所以失败，主要原因在于被忽视的网络、存储和治理瓶颈，而非机器学习模型存在缺陷。 由企业级 AI 和基础设施专家组成的小组分享了构建生产级 RAG 系统的实战策略与架构模式。讨论重点聚焦于克服数据处理瓶颈、平衡成本与性能，以及在开源灵活性与商业厂商锁定之间进行权衡。专家们还探讨了关键的运维挑战，包括遗留系统遥测、影子 AI、隐性治理延迟，以及利用隔离飞地架构保障智能体工作流的安全。 企业 AI 基础设施高度依赖底层网络和存储基石，这使得突发性 AI 客户端流量成为 IT 系统面临的一项独特挑战。 (2:30) 在开源网络情报（OSINT）和复杂数据环境中，KV 缓存增强与滚动假设生成等专用 RAG 架构有助于管理庞大的数据规模与快速变化的关系。 (2:30) 在开源与商业解决方案之间进行技术选型时，必须优先考虑业务价值而非盲目跟风，并通过松耦合架构在工具迭代时尽量降低组件替换成本。 (7:46) 为了控制生产级 RAG 系统的运营支出，企业应考虑使用更小、经过领域适配的模型，而非完全依赖大型通用基础模型。 (11:27) 最终用户的信任与采纳是衡量 RAG 成功与否的关键指标；用户可以为准确的结果容忍适度延迟，但会迅速弃用响应迅速却不可信的输出。 (11:27)

Deploying enterprise Retrieval-Augmented Generation (RAG) systems requires prioritizing foundational infrastructure, governance, and loose coupling over model-centric optimization, according to industry practitioners at an Ai4 panel. Infrastructure Foundations and Retrieval Architectures (2:30) Panelists argue that enterprise RAG bottlenecks center on infrastructure rather than model ingestion. Sam Newnam notes that the primary hurdle in systems like Nvidia's AI Enterprise Data Platform is filtering out noise and personally identifiable information (PII) to isolate high-value intellectual property before vectorization. Mark Menger likens enterprise AI to an iceberg, warning that vector databases and GPUs represent only the visible peak, while bursty, unpredictable network and storage demands make AI clients some of the most challenging workloads enterprise infrastructure handles. To manage fast-moving data such as open-source intelligence, Luis Romero-Sevilla outlines specialized retrieval methods developed at Orbis Operations. These include a Key-Value (KV) cache-augmented architecture coordinated by a supervisory orchestration layer, as well as rolling hypothesis generation, which segments large document corpuses into batches to iteratively update findings when full-graph traversal is too slow. Loose Coupling and Evaluating Operational Trade-Offs (7:46) When navigating build-versus-buy decisions, Dr. Selim Mimaroglu advises enterprises to align tool selection strictly with business value and establish internal standardization layers to prevent vendor lock-in. Citing Nicole Forsgren's 'Accelerate', Menger emphasizes that loose coupling predicts delivery performance and reliability, urging teams to make early architectural choices that keep future replacement costs manageable amidst rapid component churn. Balancing cost, latency, and adoption requires distinguishing between development experimentation costs and runtime user token consumption. Mimaroglu suggests testing whether smaller, fine-tuned, or domain-adapted models can replace general-purpose foundation models. Romero-Sevilla observes that millisecond latency overheads compound rapidly on cloud billing. However, Newnam maintains that end users prioritize trustworthy answers over pure speed, recommending that engineering complexity mirror user risk tolerance through targeted pipelines rather than single, all-encompassing systems. Attribution in Legacy Systems and Root-Cause Diagnostics (16:43) Integrating modern retrieval pipelines with legacy systems poses tracing and diagnostic challenges. Mimaroglu shares an experience at Oracle Utilities where an AI model operated alongside a legacy rule-based system named 'Simple'. Because the AI routed unhandled edge cases back to the legacy tool, users blamed the new AI model for legacy errors whenever explicit lineage tracking was absent, demonstrating the need for end-to-end request logging. Mimaroglu also argues that organizations mistakenly fault underlying machine learning models when production failures arise. Implementing a top-down diagnostic approach—evaluating production, then deployment, and lastly the model itself—revealed that most operational failures stem from deployment bottlenecks rather than model flaws. Scaling beyond initial pilots of 1,000 to 15,000 files consistently encounters friction across IT, data science, and CISO risk management regarding customer data obfuscation and chain of custody. Governance Delays, Shadow AI, and Enclave Security (39:02) Underinvesting in foundational architecture risks forced rearchitecting as regulatory mandates shift. Mimaroglu notes that vendor support delays and sluggish governance bodies create hidden costs, with enterprise approval cycles stalling technically sound deployments for weeks or months. Frustrated employees frequently turn to 'shadow AI' by testing tasks on personal subscriptions when corporate vendor stacks fail. To manage third-party software risks without manually auditing every library or asynchronous partner update, Romero-Sevilla highlights an emerging trend toward on-premise setups and micro-secure 'enclaves' modeled after federal security practices. This architectural pattern establishes strict firewall controls over data ingress and egress, securing the system perimeter while treating the internal open-source tools and dependencies as an acceptable 'messy middle.' While enterprises face persistent friction from vendor lock-in, legacy fallbacks, and lengthy compliance reviews, the panel indicates that managing RAG risk increasingly relies on strict perimeter defense and modular architectures rather than inspecting every internal model component.

在企业级检索增强生成（RAG）系统的构建中，成败关键在于底层基础设施的可靠性、数据治理策略及生产部署的解耦架构，而非单纯依赖核心算法模型的性能。来自多个行业的专家强调，企业必须在成本、延迟与用户信任之间寻求工程平衡，并借助隔离机制应对遗留系统整合与供应链合规风险。 高价值数据过滤与自适应检索架构 (2:30) Sam Newnam指出，构建生产级RAG流水线的关键瓶颈并非数据摄取能力，而是在数据注入向量数据库之前，如何有效识别并提取高价值的企业专有知识，同时彻底剥离噪声与个人身份信息（PII）。在基础设施底层，Mark Menger将企业级AI系统比作冰山，强调GPU与向量数据库仅是浮在水面上的显性部分，水面之下的网络与存储才是系统基石；由于AI客户端具备高度突发且不可预测的资源需求特征，传统IT基础设施正面临前所未有的负载压力。 为应对公开来源情报（OSINT）高动态与强关联的特性，Luis Romero-Sevilla介绍了Orbis Operations的两类前沿检索方案：其一是KV缓存增强生成架构，依托主管编排层将领域知识分发至多个键值缓存中；其二是面向海量文档集的“滚动假设生成”架构。该架构规避了在大规模语料库上全量图遍历的高昂成本，通过按批次切分数据，在新证据到达时持续对照并更新初始假设，以此生成高可靠的综合研判。 松耦合设计与成本及采纳率权衡 (7:46) 在开源工具与商业平台的选型中，Dr. Selim Mimaroglu认为技术决策应完全由业务价值驱动，建议企业在建立内部标准化抽象层的前提下并行评估两者，以防范供应商锁定。Mark Menger引用《Accelerate》一书的研究指出，松耦合是系统交付性能与运行稳定性的强力预测指标，在组件更迭极度频繁的AI发展初期，合理的架构必须将未来的替换成本降至最低。 面对运营层面的多维取舍，Selim Mimaroglu建议将成本细分为实验研发成本与规模化运行成本，并主张评估使用微调领域小模型替代通用基座大模型的可行性。Luis Romero-Sevilla提示，云环境下的计算延迟与调用账单直接成正比，毫秒级开销会在高并发场景下迅速累积。Sam Newnam则将视角转向文化采纳，指出终端用户宁愿接受合理范围内的延迟以换取高度准确的结果，但对充斥低级错误的快速响应零容忍；因此，针对特定业务痛点构建小型流水线的投资回报率与准确度，普遍优于单一全能的庞大系统。 生产故障定位与治理流程的隐形成本 当AI系统在生产环境中表现异常时，企业往往本能地质疑底层大语言模型。然而，Selim Mimaroglu依据实战排障经验指出，通过“生产环境、部署链路、核心模型”自顶向下的诊断流程可以发现，绝大多数系统失效根植于生产基础设施与集成缺陷，而非模型本身。在旧系统升级改造时这一矛盾尤为尖锐；例如在Oracle Utilities项目中，当AI模型遇上数据覆盖盲区而将请求回退给遗留的“Simple”规则系统时，因缺乏端到端链路遥测追踪，用户习惯性地将遗留系统的错误结论全盘归咎于新上线的AI模型。 此外，基础设施投入不足不仅会导致被迫重构，还会诱发“影子AI”。当官方采购的企业级工具因性能缺陷受阻、且高阶技术支持响应冗长时，员工往往会私下将任务转移至个人环境运行。与此同时，受监管行业的合规审批流程往往需耗费数周乃至数月，这种制度性等待所导致的算力与人力闲置，构成了企业事前极少预算的沉重隐形成本。 第三方风险控制与飞地隔离边界模式 (44:10) 针对受监管企业接入智能体（Agentic AI）和开源依赖时面临的代码黑盒与异步更新风险，Selim Mimaroglu直言，追踪未经验证的第三方底层API和代码库目前仍是一项极度依赖人工核查的繁重工作。企业必须对系统底层调用的各项组件保持完全透明，并要求外部供应商明确签署合规规范并提交合规凭证。 Luis Romero-Sevilla针对该困境提出，当前企业正加速借鉴联邦政府安全规范，推动工作负载重返本地或迁入微型安全“飞地”（enclaves）。该模式放弃了对内部每一个开源库、模型或数据调用进行微观审计的低效做法，转而将内部调用视为受信任的“混乱中间层”（messy middle），集中资源在数据出入口构建绝对严格的边界防火墙。Luis认为，这种强边界隔离策略是目前大型组织在保证安全底线的同时吸纳前沿开源组件的切实路径。 专家一致表明，企业级RAG落地的首要瓶颈并非算法逻辑本身，而在于底层数据清洗、基础设施解耦以及跨部门的安全协同。在第三方生态高速演进与合规审批周期冗长的现实冲突下，如何构建兼具敏捷迭代与严密边界隔离的防御体系，仍是企业迈向规模化应用的核心考验。

## Background 本内容由 Ai4 于 2026 年 8 月 22 日发布在 YouTube 平台上。根据页面描述，这是一场关于企业级检索增强生成（RAG）工具与解决方案的专家座谈会，邀请了来自 Nordstrom、Fannie Mae、Orbis Operations 等企业的专家，探讨数据摄取、向量数据库、成本延迟及落地架构等关键课题。 ## TL;DR 在实际的企业级 AI 系统落地与运维中，当出现故障或性能不如预期时，人们往往第一反应是归咎于底层的 LLM 或机器学习模型。然而经验表明，引发问题最频繁的恰恰是 AI 生产环境与工程部署层，而模型本身出现故障的概率反而最低。因此，团队在排查问题时应建立“先审查生产环境、次查部署环节、最后才看模型”的优先级顺序。 ## Atmosphere & style 对谈呈现出浓厚的实战视角与客观理性的技术反思风格。演讲者以资深业内人士的坦诚语气，剖析了企业在推进 AI 落地时面临的认知误区与沟通阻力。整体氛围务实且具启发性，深刻传达出将理论模型转化为可靠生产系统过程中的不易与曲折。 ## Key takeaways - [20:24](https://www.youtube.com/watch?v=NWYf14HbYmU&t=1224s) 当 AI 系统在生产环境中发生故障时，企业往往倾向于归咎于大型语言模型（LLM）本身，但实际经验表明模型引发的问题占比最少。 - [20:24](https://www.youtube.com/watch?v=NWYf14HbYmU&t=1224s) 排查与归因 AI 系统问题时，应优先审查 AI 生产环境，其次排查 AI 部署环节，最后才去检查 AI 模型本身。 ## Detailed walkthrough ### [20:24](https://www.youtube.com/watch?v=NWYf14HbYmU&t=1224s) 生产部署与模型问题的归因辩析 Selim Mimaroglu 强调，企业在遇到 AI 系统故障时，往往倾向于将责任推卸给大型语言模型（LLM）或底层的机器学习模型，认为“这一定是模型出了问题”。然而根据他的经验，事实恰恰相反。 在实际推动落地时，Selim 的团队非常艰难地与工程师和管理层沟通，引导大家改变视角的优先级：先审查 **AI 生产环境（AI Production）**，再检查 **AI 部署（AI Deployment）**，最后才去排查 **AI 模型本身（AI Model）**。 事实证明，系统中最频繁发生故障的环节恰恰是 AI 生产阶段，其次是 AI 部署，而模型本身引发的问题反而是最少的。Selim 叹息道，这是一段“ Sorry, something went wrong. Please try your request again. ## Notable quotes > [20:24](https://www.youtube.com/watch?v=NWYf14HbYmU&t=1224s) 这一定是模型出了问题 — Selim Mimaroglu ## Mentioned Shubham Shukla, Selim Mimaroglu, Luis Romero-Sevilla, Mark Menger, Sam Newnam, Nordstrom, Fannie Mae, Orbis Operations, F5, Secuvy, RAG, LLM ## Video description Shubham Shukla — Data Scientist II, Nordstrom Selim Mimaroglu — Author, Director of GenAI and Data Science, Fannie Mae Luis Romero-Sevilla — VP of AI, Orbis Operations Mark Menger — Solutions Architect, F5 Sam Newnam — CMO, Secuvy Retrieval-Augmented Generation (RAG) has become a cornerstone for enterprise AI, enabling models to ground outputs in proprietary knowledge and deliver trustworthy results. This panel will break down the critical components of a RAG system, from data ingestion and indexing, to vector databases, embedding models, retrieval pipelines, and response optimization layers. Panelists will explore the evolving vendor landscape, comparing open-source and commercial RAG platforms, as well as strategies for integrating them into enterprise environments. Expect discussion on cost, latency, scalability, and governance trade-offs, plus real-world use cases where RAG has unlocked measurable business value. Attendees will walk away with a blueprint of the cutting-edge RAG tools and strategies needed to move beyond experimentation and build production-grade AI applications. [Source](https://www.youtube.com/watch?v=NWYf14HbYmU)

[20:24] Selim Mimaroglu: 这一定是模型出了问题

[2:30] Luis Romero-Sevilla: We segment the data in batches... formulate a hypothesis on that first batch, then when we receive the new batch, new evidence is presented. So we compare that new evidence against the initial hypothesis and then we eventually build out our final response based on updating that hypothesis. [7:46] Mark Menger: The question shouldn't be necessarily 'Do I build or buy?', but what architectural choices can I make today that make the cost of replacement later higher?...We're at the early stages, so the likelihood of component churn is exceptionally high. [42:22] Selim Mimaroglu: I just wish it's not easy.

[0:04] Hello everyone, welcome to React tools and  solutions session. Thanks everyone here to join [0:10] just before the lunch. So thank you a lot. Thanks  a lot. Um I'll start with the introduction. I'll [0:16] start with myself. Uh I am Shubhham. I'm a applied  scientist at Nordstrom leading the development of [0:23] generative AI systems uh with prior stints at  Amazon. um super excited about this session. [0:30] I'll start with you. Yeah, good morning. My  name is Sam Nunham. I'm the CMO and head of [0:35] growth at a company called Suvi AI that does uh  data readiness and onboarding and governance for [0:40] AI pipelines. Uh previously been in storage and  infrastructure and AI for 1015 years in the data [0:46] center and it's fun to watch what's happening  especially with pipelines getting in and out of [0:50] these rooms. Uh hello everyone. My name is Mark  Manganger. I'm a solution architect with F5. Um, [0:56] some of you are probably wondering what is  a load balancing company doing in this panel [1:02] discussion. Uh, well, I've been working with our  uh, partners and with our customers on their AI [1:07] implementations from a computer networking and  security perspective and hopefully we've got a [1:12] few things that we can share with you with regard  to the pipelines and the like. Sounds good. Hi, [1:18] I'm Lu Romero Savvija. I'm the VP of AI at Orbeez  Operations. We are a Washington DC based firm that [1:25] focuses on open source intelligence and data  fusion and strategy consulting and yeah and f [1:32] my focus is on uh strategy and also advancing AI  research and operationalizing that work intools [1:42] that help analysts to uh make sense of distributed  information and complex relationship between data. [1:50] Good morning everyone. Uh I'm Dr. Selim Mimi  Marovo. I have been at Fenime for about a year. [1:56] I'm director of geni and data science. Before that  I was at Oracle for eight years. I was director [2:02] of AI. I built AI center of excellence at Oracle  utilities, another highly regulated industry. I [2:09] put the AI organization together. I have a book on  the topic. It's called AI enterprise productivity [2:16] success beyond the high. And before all that  I worked as an academic for about a decade. [2:23] I'm happy to be here. Uh excited to see you all.  Thanks for having us. Cool. Okay. So let's start [2:30] with the first question. Why don't you start maybe  describing one of the rack systems that you might [2:36] have recently worked on and is in production  or you are thinking of actively working on it. [2:42] What was the problem statement and what was the  impact of it? Yeah. Uh well I can start with that [2:48] a little bit. We um we've worked with a lot of  leading storage providers in the market around [2:53] Nvidia's AIDP solutions, right? So the idea of  taking data to an AI factory was hard. How do we [3:00] bring GPUs and stuff to data to work on it? Right?  And the big challenge wasn't just around can we [3:05] load a pipeline? Can we look at a blueprint  and and process a bunch of data? The the real [3:10] challenge that we had to work on was was what  is the data? What's the right data that goes in [3:14] that particular thing? Right? We've all heard the  garbage in garbage out. A lot of what we're going [3:17] to talk about today is that you know there's a lot  of tooling that comes out there right open source [3:22] branded etc. But to design an enterprise system,  right, that can take a massive amount of data, [3:27] whether that's PDF files or textual documents  or customer support emails and be able to really [3:32] understand not just the data within them, not just  PII and context, but real true context, right? [3:37] Like the the alpha intellectual property that's in  those things and how does that make it to a vector [3:42] database, right? That's what people are really  trying to find is is not the lowhanging fruit, [3:45] but it's like how do you find the right pieces  of data that help you take that next edge, right? [3:50] that's the first to discovery, first to answer,  first to IP, that type of stuff without bleeding [3:54] that IP into that particular chatbot or internal  session. So well uh in in Orbis we have built two [4:04] uh what's uh two retrible architectures well  both of them they were in response to some of the [4:11] challenges of building uh working with open source  intelligence sorry sorry open source data and [4:18] usually that data comes in large volumes and not  only in large volume they're deeply interconnected [4:25] and the I think the biggest aspect of it is that  it quickly becomes obsolete sometimes in a matter [4:31] of minutes and hours. So the first uh architecture  that we built it's an extension of cache augmented [4:39] generation just a system that distributes that  knowledge into different KV caches and then a [4:46] supervisory layer that orchestrates the questions  and synthesize the answer. The second one uh that [4:54] uh one that I'm most excited about something  we called uh rolling hypothesis generation. So [5:01] uh imagine you have a massive amount of documents  and in order to answer a question about those [5:07] documents you need to traverse all of them. uh you  can potentially build a knowledge graph with them [5:15] uh and you can map out their relationship  but sometimes you have to traverse the [5:21] entire knowledge graph to answer that particular  question. So what we did we segmented the data [5:26] in batches and we take for instance the same the  the first batch we formulate a hypothesis on that [5:34] first batch then when we receive the new batch  new evidence is presented. So we compare that [5:41] new evidence against the initial hypothesis  and then we eventually build build out our [5:48] uh our final response based on updating that  hypothesis. And not only at the end of processing [5:57] all those batches, you get your final response,  but there's also very interesting details that [6:02] you can map out by uh revising all the hypo the  the hypothesis how how it has changed over time. [6:15] Yeah. Um, so what we've been working  on is kind of like the underpinning [6:20] uh infrastructure that that that undergirds  all of this uh in in our conversations with [6:25] our customers over the last say 18 months.  What we found is a couple of things. One is [6:30] and I'm not going to say anything I think overly  surprising to this audience right is that AI is [6:34] essentially an iceberg and above the waterline  is are all the GPUs and the rag databases and [6:39] the vector databases that are are are in uh in  use. And what happens occasionally is people [6:45] tend to lose sight of what's below the waterline,  the larger mass of the system. But this is where [6:50] most of the shall we say failure that uh uh  that that causes the ROI not to be delivered [6:57] uh occurs. And in that space is where we've been  making a lot of investments to make sure that we [7:02] understand better how we can help contribute to  the reliability of the underpinning infrastructure [7:07] so that the services that are being so excitedly  described are actually available when you need [7:12] them to be. [snorts] Um and so we've made some  investments with our uh alliance partners with [7:17] storage vendors and the like and creating very  elaborate enterprise scale and exoscale labs [7:22] so that we can test to make sure that we have  an understanding of how that performance and [7:26] reliability landscape uh exists because uh again  probably not too much of a surprise to you guys [7:32] but we just uh reinforced it with testing  which is uh AI clients are essentially the [7:38] worst clients your AI infrastructure has ever  frankly your IT infrastructure has ever seen. [7:46] Now that's all good. [laughter] Yeah. So when  you described your examples and all the answers, [7:53] all I one thing I was uh looking at was there  are so many rack tools and so many components and [8:00] it's just hard to keep up with them, right? And  there's always a question and a tug of war between [8:07] open-source systems and uh commercial systems and  how do we make choice between what to pick. So my [8:15] next question is around what is the framework  that you use when it comes to picking whether [8:19] the open-source systems or or commercial systems  will solve a particular problem and move it move [8:25] your rack system to production. Yeah, I want to  I want to take a stab at this question. So in my [8:30] mind like what's the most important thing? It is  to provide value and the most important value is [8:36] the business value. How can we provide business  value? Most of the time the infrastructure, [8:42] the open source, the private vendors, they're  available and that's not actually what's holding [8:48] you. What's holding you is the organization, the  AI organization, the management should be value [8:55] oriented and they should enable uh the tools and  products. That's how I see that. Open source, free [9:02] source, there are many products from big names,  there are many products from small companies, [9:08] midsize companies that you heard of that you have  never heard of that there are many big products [9:14] as well. So in my mind the most important thing is  not to lock yourself into any vendors and in this [9:22] quickly changing ecosystem I would like you to  try everything try the free open and the private [9:30] vendors for your products and they will tell you  what works they will also tell you what doesn't [9:36] work but it is only available to you what works  and what doesn't work in practice when you put it [9:44] to use and you will learn from it. Of course,  another aspect of this to keep in mind is to [9:54] make it standard so that you can actually move  from one vendor to another vendor. Unfortunately, [10:01] there are not set of rules to do that. So it  depends on the developers and the management [10:07] to set up these standards and to put them so that  they can interchange the tools that are available. [10:15] You have it. Yeah, I just like to jump on that  because um what what I find very fascinating about [10:20] that and it's been reinforced in our experience  with our customers is that these patterns actually [10:25] exist for a while. Um if you re have read the  book accelerate by Nicole Forsgrren which is [10:30] a lot of research on uh effective patterns for  delivering applications loose coupling is one [10:37] of the key predictors for success performance  reliability and uh we're seeing the same thing [10:43] with regard to AI applications is that you want to  enable that application uh loose coupling because [10:49] coming back to the the question here it reinforces  optionality it creates opportunities for you to [10:54] switch some things out. And so the question  shouldn't be necessarily do I build or buy, [10:59] but what architectural choices can I make today  that make the cost of replacement later higher? [11:05] Because in the previous presentation, we heard  some really interesting new ideas about how to [11:10] implement a rag system that were frankly divergent  from anything that I'd heard before. And we're at [11:16] the early stages. So the likelihood of component  churn is exceptionally high. And so you need to [11:21] do the kind of loose coupling that that's uh  being made that's being recommended here. Cool. [11:27] So again uh I I think we are touching on the  realities of the production system in a way [11:34] and if you look at the rack system there are  various components in it like data injection [11:38] embedding models and what I would like to  discuss is what are the biggest tradeoffs [11:43] uh that you make that you find yourself making in  terms of cost and in terms of scalability latency [11:50] while developing these systems. In terms of uh in  terms of the pipelines in terms of the components [11:58] there is a very big component of data. So you need  to take the data from the original data source and [12:04] ingest it put put it in the rack when we are  talking about rack put it into a vector database. [12:11] So both sides are going to be costly and there  is going to be latency depending on what you [12:17] do. So you don't want latency in chatbot related  systems which may have rag component behind and [12:25] uh you may avoid it if you really design it with  smart architectures but there are ways uh there [12:33] are actually use cases which you cannot really get  rid of this latency. So latency uh for example if [12:39] you generate data if you generate documents it's  going to be there. So it's an agreement between [12:45] you and your users to understand that what they  should be expecting from uh the product that [12:52] you have. Of course cost this is the biggest uh  this is the biggest part from the business sensor [13:00] point. So when you look into the cost there are  many types of cost. One is the development cost. [13:07] So how many tokens did you spend? So this is well  known and people think think about it carefully. [13:13] There's also runtime cost. So when you  have the product available to your users, [13:19] how much tokens are they using? That's another  cost. So you can't have a big large language model [13:26] having trillions of parameters behind. Uh it's  going to consume tokens. It's going to be costly. [13:34] Another thing for me to think about it is like  can we actually do it with a smaller large [13:39] language model instead of putting something  huge there which actually had some transfer [13:45] uh learning and some other components that is  actually built on the use case can it be more [13:52] efficient so uh at least in our experience uh  I think the biggest lesson that we have learned [14:00] when evaluating those trade-offs it's the first  one uh there's no one rack strategy fits all and [14:09] uh and it for a lot of experts that sounds like  very obvious but it's very tempting to try to find [14:14] the most accurate strategy and then bring that to  to production. So uh I would say the thing that we [14:23] should focus before implementing a red strategy  is to evaluate the risk tolerance of the user [14:29] and then look for the implementation that meets  that criteria before implementing something like [14:36] highly complex and the other thing it's uh latency  and adding to that latency and cost ratio because [14:45] uh usually there's a sorry that latency and cost  ratio it's basically uh directly proportional so [14:58] whenever I usually hear the argument that it's lns  are relatively slow so few milliseconds they don't [15:04] uh matter that much but when you're deploying  stuff to production in cloud those milliseconds [15:10] those milliseconds start to count to  your bill so also [snorts] evaluating [15:14] uh taking care of those milliseconds  when deploying your regable strategies [15:18] production and not overlooking them just because  lens are relatively slow. Uh so far, yeah, [15:27] I think I'll add to this a little bit. Part of  this we we look at it as well as this is kind of [15:32] also like an infrastructure view, right? We keep  talking about latencies and how we're doing stuff, [15:35] but you both hit on an important topic which is  really cultural adoption of your app, right? And [15:40] so we've seen a lot more organizations focus on  accuracy and actually producing an app and then [15:45] going back and tuning for latency, right? because  everyone kept trying to solve the equation before [15:50] they know what the outcome is, right? It's so easy  to deploy a pipeline, create this vector database, [15:54] throw a whole bunch of data through it. But the  cultural adoption part is the piece that matters, [15:58] right? When you launch that application and your  senior engineer or senior support technician goes [16:03] back and says, I'm not using that. My old way was  faster, right? All of that was for not. And so [16:08] what we're seeing a lot of customers do is really  start focusing on what's in the pipeline. To your [16:12] point, what the outcome of that? Every other  application we designed had a business outcome, [16:16] right? But in AI, we've been forced to push this  stuff. Hey, get something launched the boards. [16:20] It's in my 10K. Like, how do we do this? But the  reality is what I would say is narrow your focus, [16:25] right? It doesn't cost a lot to launch a lot of  small pipelines versus one really big pipeline. [16:30] Accuracy and making sure that that pipeline does  exactly what it's supposed to do. And in a time [16:35] frame, users will wait for great answers. They  won't wait for bad answers. Nice. Nice. Um well [16:43] it's one thing to build rack system in isolation  and it's completely different story to build a [16:50] production grade rack system and what I want to  know from all of you is um do you have a good for [16:59] story when it comes to integrating a retrieval  pipeline and what kind of roadblocks you faced [17:05] while developing that in retrieval pipeline into  an enterprise system. Yeah, I want to I want to [17:12] start by saying that this is so common and I have  a section in my book. It's called AI with legacy [17:20] systems. So most of you may be thinking that AI  is going to be standalone. Maybe it's going to be [17:26] biggest part in your product and you may be right.  But more than often it's going to work with what [17:33] you had working previously. So I talked about  Oracle utilities. When I started a decade ago, [17:41] we had a solution named as simple. So simple is  related to energy disagregation. It was a set [17:49] of rules to tell you what was the energy usage  in a household for EV for the large appliances [17:57] that you can think of uh with only given the total  amount of energy in a household. So customers were [18:05] not happy about simple and actually I was hired to  put a new system AI that is better than the simple [18:15] and that was my starting point. [clears throat]  So once you have that you have a starting point [18:22] you have a baseline one way is to say like oh  let's get rid of simple okay let's get rid of the [18:29] legacy system altogether start from scratch but  you need to realize it is generally speaking not [18:36] possible because AI cannot uh score every customer  in every geography the training input must be seen [18:48] So within the training data set if there is a  customer with the input data that is not being [18:53] seen this customer will not be scored. So what was  done at that point of time instead of using AI we [19:02] kept using simple. So they were intermingled and  without all the details uh the customers depending [19:10] on their input some of them actually were scored  with simple and some of them actually were scored [19:16] with AI and unbeknownst to us meaning like we  didn't know who was scored with what and we kept [19:25] getting some complaints less complaints but still  the problem was if you don't have a logging if [19:33] you cannot trace what was the root cause. People  tend to think AI is the culprit. They will not [19:41] blame the simple even though the starting point  was the customer is not really happy with the [19:47] simple. They will say this is AI. So we put  this into place and for a very large system [19:53] that's not easy. It requires a lot of effort to  do the logging and backtracking and after that [20:01] uh it was actually easily more I would say like  more easily managed because once you have that [20:09] you can actually pinpoint what's the uh what's  the problem is it related to the legacy system is [20:15] it related to the new AI system uh you're not done  because when we talk about AI and this may be geni [20:24] You have AI research, you have AI deployment and  you have AI production. And people will blame AI [20:32] models for that. They will say this is either  a large language model or this is the model, [20:37] the modern model, machine learning model that you  have underneath. This must be the problem. And in [20:42] my experience, that's not the case. You start from  the top. So it was actually very challenging for [20:49] us to talk with the engineers to talk with the  managers and to say okay I know what you think [20:57] but let's start looking into the problem from AI  production then let's go into AI deployment and [21:06] finally let's go into the AI model. So we put this  system in place and find out most of the problems [21:13] actually were related to AI production then later  AI deployment and least of the problems were [21:20] uh related to the AI model itself. So so it is  a war story that is ongoing which doesn't have [21:26] a happy ending but it is better now. anyone  else. I mean, I think when you talk about [21:36] blockers and going into production, like for me,  there's always three ICPS in these pipelines, [21:42] right? There's the storage and IT teams that have  have held this data in hierarchal systems forever. [21:47] There's the business unit, the data scientists who  are trying to put this stuff together. And then [21:50] there's this nice team, you know, called security,  right, with the CISO who's whose entire job is [21:55] to protect the company from risk. And so without  chain of custody and you mentioned risk earlier, [22:00] right? Like this is the whole thing is that we  see every PC take the gardener stat, IDC stat, [22:05] whatever about projects failing, but it's always  a data problem. It's like the first thousand files [22:08] are fine. The first 15,000 files are fine. Hey,  I want to go hook up to this system. Nope. Like [22:14] how do we get past that particular point, right?  And and that's really where this whole, you know, [22:18] source of origin governance thing comes in  because that's what we saw everybody race [22:22] through this with really easy pipelines, right?  It's like, hey, look, we got this thing launched. [22:26] production is a whole different thing right  especially in regulated industries in you know [22:31] uh publicly traded companies these sorts of things  like this this becomes real and that's where we're [22:35] seeing enterprise we've had the same trouble  internally right is you know how do we obuscate [22:40] customer data how do we do certain IP stuff how  do we look at these things like it's so easy to [22:43] see that intellectual property and stuff leak and  I think that's where you know we're seeing a lot [22:48] of enterprises really focus and that's where  we've had a hard time launching things too is [22:52] it really comes to how do you make sure it's  absolutely safe right? When you read the news, [22:57] you've got, you know, agents jumping sandboxes and  attacking other corporate like there is this fear, [23:02] right, of how does AI get away from us? And  so that's back to where we've, you know, [23:06] I mentioned earlier, but it's all about narrow  and focus. It's about, hey, do a really small app, [23:09] do a really small rag pipeline, prove the  model, prove your your chain of custody, [23:14] right? And then continue to build and orchestrate.  I think my next I hopefully I can keep this really [23:21] quick but it's on hopefully it's on topic in terms  of what we've seen with our customers and it goes [23:26] specifically to the point of you know I'm I'm  scaling I'm scaling everything's fine and then [23:29] all of a sudden I'm not and what we've seen and  this kind of basically ties back to my point a [23:34] moment ago which is uh AI clients are the worst  clients of your IT infrastructure you've ever [23:38] seen is we have customers buying DOS solutions to  protect their underpinning infrastructure because [23:45] of the the the fan out behaviors, the uh uh the  highly federated behaviors of AI clients. And so [23:52] that is something that you're not going to see in  your pilot. It's going to be very similar to maybe [23:57] you've heard this this kind of apocryphal story  before about the the the the experience of the [24:02] turkey, right? The turkey thinks that humans  love them for the first x number of years of [24:06] its life up until Thanksgiving Eve, right?  And that's your experience with your pilot, [24:11] right? your pilot's fine fine fine until it's not.  And so this is where a lot of these patterns where [24:17] the accelerate work where you know anticipating  the kind of workloads that he was just describing [24:22] uh or should be part of your planning from day  one. Nice. Um you touched upon data governance [24:32] and regulatory things. That's where my next  question is. um you have a large enterprise [24:38] and huge amount of proprietary data and that comes  with its own uh interesting uh problems and you're [24:48] ingesting all of that in a vector database. So how  do you uh take care of access controls, compliance [24:55] and regulatory aspects of that? Yeah, I think  there's there's two ways you look at it. You know, [25:01] I walked expo floor for a minute. It's agent agent  agent security and and while I think guard rails [25:06] and permissioning from the app side has always  been important we've got a different focus which [25:10] is all about if we're building these massive race  cars what fuel's going into them right and so you [25:14] know what we focused out of our tooling around and  discovery around is that intellectual property is [25:19] that contextual data like there's tons of tools  that can do a reject expression and track what [25:23] goes in there but there's there's two really hard  pieces to solve is where is intellectual property [25:28] within my data is that in a PDF in a title block  of a drawing is that buried in some sort of text [25:33] file and the first four paragraphs were benign,  the fifth one's a problem. And then it's tracking [25:38] the linearity of that. It's like once I find that  particular piece of data, where else does it exist [25:42] in my organization? How did I make sure that if  that did make it into that vector database and I, [25:47] you know, ejected it at some point later that  it doesn't slip back in or that I made sure it [25:51] was actually obsuscated from that particular  thing. And so for us, the the governance piece [25:56] and the visibility of that is where we've kind  of been focused lately because we see that's [26:00] enterprises biggest challenge, right? is is that  governance layer. There's plenty of like point [26:04] tools. Nobody's looking at this as like who's your  trusted like layer for AI security. And you know, [26:11] I think while we've got a a strive to  be like I mean product plug, right? But [26:15] there's also we're not going to be the end all  of either, right? Every storage manufacturer [26:19] has this responsibility, every tool has this  responsibility. I think you mentioned standards [26:23] earlier and I think that's where we still haven't  seen the industry as a whole mature is to how do [26:28] we hand this data to each other, right? There's  always this there's this new layer of security [26:31] to say oh I handed this off or yes I can touch it  or you know the father agent said the mother agent [26:36] the like there's going to be a whole new world  right of communication all around data security [26:41] and data traceability that's not just in these  core rag pipelines and adding to that uh there's [26:48] in terms of standards there's a lot of things  that we can take inspiration from for instance [26:53] uh what we uh some of the things that we have  tried it's taking inspiration from zero trust [26:59] uh when it's a middle medium scale application,  you can just uh compartmentalize all the different [27:07] data and make sure they're isolated depending  on their domain and you can assign uh domain [27:13] base access all that stuff. At least that work  for medium scale data. Then when you get to the [27:18] larger scale, it gets a bit more complicated.  But there's a lot of uh uh things we can take [27:24] uh in terms of standards take inspiration  from security standards that already exist. [27:30] Two weeks ago, I was at Boston giving a speech at  Chief Data Officer Conference and I listened to [27:38] Mike Stormbreaker who is a well-known father  of databases and he talked about his work [27:45] his experience with large language models and he  talked about the data warehouse at MIT and he he [27:52] mentioned that he didn't really like the schema  it is unmanageable which is a reality. So your [28:00] question is like how do you manage user access  and uh why did I bring Mike Stone Breaker? Because [28:08] relational databases have made a lot of progress  for that. They do for the last I would say four [28:16] decades. They have all the bells and whistles all  the features for managing access controls. And now [28:22] we are looking into vector databases part of rag  like what do they have? This is an evolving space. [28:30] Some of the vector databases are more advanced  than the others and some of them are following [28:35] behind and some of them maybe will catch up and  will be the front runner. But the point is for the [28:42] time being they don't have all the features that I  and you are used to that comes let's say free in a [28:50] relational database. A very you related use case  that I have been working on recently is a rack [28:59] system that has three types of data in it. One is  static let's say almost static and which is open [29:09] to every user and [snorts] the other one is what  the user brings into the rack the vector database. [29:18] Let's say you bring something to the database.  It should be only available to you and not to [29:23] me. So this is an access control and there is in  between this is the third type. So how did we look [29:31] into this problem and how did we solve this uh at  the application layer. So for each user we need [29:40] to look into what are the files that they have  and how do they uh how can they access it? Well, [29:48] we are talking about latency. How can they access  having uh let's say millions of uh data files, [29:56] resources? It's a challenge, but it is a borrowed  technology from relational databases using [30:03] indexing. And over there like there are a lot  of indexing techniques which work very fast. in [30:10] here you need to reimplement it uh if the vector  database doesn't have that features uh coming with [30:17] it. Amazing examples. Um the next question is  about a pretty interesting one. Uh what is the [30:27] most common architectural mistake you have seen  organizations make when scaling their rack system? [30:35] Can you give us an example on that? Um, yeah,  I'm just going to bang on the same drum I've been [30:40] banging on all along, which is the the notion of  loose coupling versus tight coupling. So, when the [30:46] solutions are tightly coupled from the beginning,  um, then what you end up with, and we've seen [30:51] this with our customers and we end up helping them  rearchitect, uh, is uh unconstrained blast radius. [30:59] So what ends up happening is that misbehavior on  the client side ends up being a failure mode for [31:04] the uh underpinning services whether it's rag or  or or um an LLM runtime and conversely if there [31:12] are like component failures within the the service  uh that gets propagated back into the client. Um, [31:18] we've seen this with our testing, we've seen this  with our our uh uh with our alliance partners. Um, [31:24] in in one particular lab scenario where we were  trying to replicate uh a highly federated uh [31:32] inference use case, we had a tightly coupled uh  environment and all it took was one node to fail [31:39] fail on the storage backend as part of the rag  infrastructure and the client started failing. [31:45] The entire cluster ended up failing. um because  of the tightly coupled solution is as soon as [31:50] we introduced a loose coupling interstitial  layer then we ended up having we were able to [31:57] have failures on either side of the fence and that  that there was blast radius control. So that was [32:04] long answer. The short answer is tight coupling.  Sounds good. Uh I think in my case and also well [32:12] not something that we have seen a lot of people  do also have done in the past and learned from it. [32:18] It's not uh considering what type of questions  uh or queries you're going to be receiving or [32:24] your rack system. There are sometimes where the  the right the data pipeline that you're going to [32:30] be developing they will answer the same question  over and over again. There are sometimes where the [32:35] questions are user based or they're dynamic  and not considering what type of questions [32:40] usually it can leads you to a wrong path when  developing your right uh infrastructure. Yeah, [32:47] I want to bring a different angle. So uh you  both talked about technical components and you [32:54] are right in bringing these into play. So I want  to talk more about like the business side of it. [33:00] So in my opinion the algorithm the technology the  components are pretty good. They are working very [33:07] well especially rack components are very well  known but it is an or it is the organizational [33:14] problem. What do I mean by that? Like we talked  about prototypes. We say that oh prototype is [33:20] great, right? So prototypes have been hiding  places for too long and [snorts] prototypes [33:27] are where people overpromise. They will say,  "Oh, this is what I can do. This is it looks [33:34] great." When they don't use the real data sets,  they may look into a toy data set, a much simpler [33:40] problem which you don't have and it's like this is  overpromising. then it is really hard to catch up [33:46] with the reality. Now it goes south once you have  prototypes and once you don't have the alignment [33:54] and the management to put this project into the  right order. So in my opinion AI organization [34:04] management is the big problem like if you don't  have it in place it is really hard to get anything [34:12] in production whether it is rack or not and maybe  to open it a bit more uh what what does it mean [34:20] like what is the organization what is like I talk  about it in detail but without going into too much [34:26] details there are several components that must  be there one is AI research, AI de deployment, [34:35] AI production, AI product management, innovation  and all put together like what do you have like [34:43] do you have an AI center of excellence which  is going to manage all this or do you have an [34:48] organizational structure which will say well these  are the resources that we have the skills that we [34:54] have and this is how we are going to put it into  production that's what I think it's a very honest [35:00] and candid answer I would say. Um, you have  anything to add as well? Yeah, mine mine isn't [35:06] infrastructure related. It's who owns it. Every  every problem and architectural trouble we run [35:12] into is it these AI pipelines feel like products,  but we don't assign a product owner to them, [35:17] right? We assign this AI center of excellence and  there's a storage guy and a network guy and this [35:21] like data scientist person. And the trouble is is  no one actually has authority to say yes or no. [35:26] Can you use that data source? Like we teach our  salesforce to do this in our discovery questions. [35:30] We're like, who says yes to your data? And half  our enterprise tables sit there and they're like, [35:36] right, it's BU1, it's BU2, it's this database  owner, it's this particular problem. And so our [35:41] challenge, I think, is not, you know, where  the failures come from is that these things [35:45] were never set up properly to succeed because they  didn't follow the same structures if we've learned [35:49] as enterprises and software developers forever is  that it requires an ownership structure, right? It [35:54] requires success metrics to go launch. And so part  of it's like before you dive into infrastructure, [35:58] before you do that stuff, think about who owns  that particular data set. Can it be used in [36:02] those particular workflows because that for me  is the biggest architectural problem that we see [36:07] every enterprise fail at. Nice. This one is the  last and then if we have time we can open it to [36:13] audience as well. Um for business leaders trying  to budget for these systems, what are the hidden [36:20] costs of building these systems, these pipelines,  rack pipelines that most people don't anticipate. [36:31] I think a lot of it is the human capital that goes  into this, right? is that we think about pipelines [36:36] are easy and kind of what we mentioned before like  you know we think about finding intellectual data [36:40] dark data within these data sets that sort of  stuff like we see people contract and spend [36:45] millions of dollars on outsourced data tagging  and data soy travels and trying to put all this [36:50] particular stuff together because you know I  think we've stuck stuff in the the sock drawer [36:55] of the enterprise for too long right is that  we don't even know that John's still here and [36:59] John owned that product management directory  two years ago what's in it no idea who knows [37:03] about that product I don't know we deprecated it  two years ago but it needs to be in our support [37:06] engine and so I think that's that hidden cost as  you start doing data discovery is that's where [37:12] people it's not tokens it's not that sort of most  enterprises can't tokconomics is a foreign concept [37:16] right they understand it from a billing concept  from not from a planning concept but I think most [37:21] people it really comes to data discovery and  unpacking what's been in their organization [37:25] what's important what's not what's valuable to  that particular pipeline nice anything else to [37:35] um [snorts] [37:40] the hidden costs from my perspective. Early on  I I talked about the iceberg and how, you know, [37:45] we're all like highly distracted by what's above  the waterline, all the bright and shiny GPUs and [37:50] and and and what have you. And I'm not saying  that that isn't important and from a knowledge [37:54] and skill set perspective, these are things that  need to be developed. Um but if you look at the [37:58] research from MIT, Harvard Business Review,  NANDA, Rand and you look at uh and mind you [38:06] there is an organizational dimension to a lot of  these failures but where the technical failure [38:11] occurred was rarely above the waterline. the  technical failure that occurred often was below [38:17] the waterline and the traditional API security,  API load balancing and all of that stuff that [38:22] frankly is not as glitzy, not as exciting, not  as interesting, but unfortunately or fortunately [38:28] maybe for us it's fortunate is extraordinarily  critical to be able to deliver the ROI on AI. And [38:35] so so my point is like you know take the time to  read that stuff and validate that what I'm saying [38:40] is true that these failure modes are actually  critical to your organization being able to make [38:46] the transition that was talked about earlier right  your your pilot's going to be fine and you're [38:51] going to think everything's fine and then you're  going to transition to production then you're [38:53] going to go to enterprise scale then you're going  to go to whatever you know scale is required for [38:57] your organization whether it's exa or petta and  then all of a sudden things are going to not work [39:02] anymore and then the regulatory churn is going  to catch and the scale churn is going to catch [39:06] you and um so so underinvesting below the water  line is going to um often cause rearchitecting. [39:16] I want to add uh I have two points of view  in here. One is shadow AI. So we talked about [39:27] components. We said like what would you get free  open source? Would you get something from a vendor [39:34] big or small? Like you may be locked into a vendor  and the business may be telling you this is our [39:41] ecosystem and this is what you need to build on  top of this. Okay, that's a requirement. Everybody [39:48] has uh subscriptions, personal subscriptions. Your  when they're at business will not work, okay, for [39:56] one reason or the other uh and you wouldn't know  this. And when you go home, okay, you will try it. [40:03] You will see that but it works on my private  system. So people will do things on the side [40:09] and they will find out like oh the vendor doesn't  work. And then you have sponsorship. You have some [40:16] uh help from the vendor that you have let's say  gold platinum you will call them and they will not [40:22] immediately say oh it doesn't work. Okay. They  will pull you in to discussions and meetings. [40:29] they will say okay and maybe they will later on  confess and say the piece that you you want to do [40:36] doesn't work and we put a bug ticket and you are  lucky if this is resolved in the next month or [40:43] so this is one angle the other angle we talked  about it very uh a little bit I would say this [40:51] is the governance okay especially in regulated  industries the governance is a big piece Okay. [41:00] And it starts with good intentions. Okay. One is  like, oh, I need to do this because it's required. [41:07] And governance also has another angle. It's like,  oh, let's be helpful to AI and data science. But [41:14] generally speaking, it's like add-on work. It's  like, oh, you need to do this. So, another hidden [41:20] cost is going to come from the governance piece  of it. So you may say like oh I have an AI system [41:26] that works that does everything that are part of  the requirements and the governance will say the [41:32] governance bodies at your organization if you are  medium or large company you know what I'm talking [41:38] about they will say how long no you cannot go  ahead with this until we approve you and this [41:44] approval is not going to take hours so I have  witnessed it takes several weeks in some cases [41:52] more than a couple of months. It's maybe wait time  and you may say like I'm not developing anything [41:58] but it is a hidden it's another hidden cost. Okay,  we have we have time for maybe one question. Yeah. [42:22] [snorts] [42:22] Hi, it was very nice u uh set of panelist and  lot of information to digest. One question [42:28] which I think you alluded to as well. Um when  we are talking about uh regulatory businesses, [42:37] legacy enterprises and use of agentic AI  for the productivity it brings and promises [42:42] uh use of open-source which also brings third  party risk right it's it's not comprehendable [42:50] when everything is running inside and  when you are relying on partners and [42:56] they are changing at their own pace you  don't know what is coming back to you are [43:00] there any patterns that any of you have  seen which people here can take and try [43:07] to emulate before everyone answer when anyone  answers we have two around 2 minutes left yeah [43:17] uh I think the pattern that I'm aware of like  you need to know what is underneath okay like [43:24] as you are saying the APIs the libraries that uh  must be approved Okay. And when you get something [43:32] from the third party, you are talking about like,  oh, there may be some third party libraries and [43:37] they may not be approved. I don't think I have  a direct solution for that other than like maybe [43:43] talk with your partner and make sure that they  also know what is regulated in in your part, [43:50] what they need to uh what they need to  provide to you. I think it's mostly manual. [44:02] I just wish it's not easy. [44:10] Yeah, we're seeing a lot of people come back on  prem and I don't mean that in a silly way, but [44:15] they're starting to run these in enclaves, right?  In small microsecure systems because to your [44:19] point, I think, you know, and you guys do a lot  with this too, like there's this idea of, well, [44:23] what happens if I use this neocloud and where do I  exchange data here and what about these particular [44:27] libraries and so everybody's almost going back  to what the federal government's done forever, [44:30] right? And said, hey, we're going to put this in a  small enclave in its own self-secure stuff. I know [44:34] exactly what went into that particular system,  what came out and what bounces in the middle. [44:38] I care less about, right? And that's that's the  pattern because your point, I don't think there's [44:42] any one tool, any one thing that solves any of  that stuff at a data layer, at an app layer, at [44:48] an open source library layer, an open source model  layer, right? Those types of things. But they're [44:52] literally doing it to control firewall and data  in, firewall and data out, and letting the rest be [44:57] a messy middle because I think that's the only way  it's solved today. Yeah, we are out of time now.