Nvidia Integrates NeMo RAG on HPE Cloud to Curb Model Retraining Costs

NVIDIA 在 HPE 云端集成 NeMo RAG 以降低模型重训成本 通过容器化微服务将企业模型锚定在动态数据存储中，既能保障运行中工作负载的安全，又能防止准确度漂移并免去持续微调。 演讲详细阐述了企业如何结合 NVIDIA 的全栈软件与 HPE 基础设施，从生成式 AI 实验过渡到可扩展的生产级智能体系统。Dave Salvator 重点介绍了检索增强生成（RAG）和 NeMo Retriever 流水线，以在无需昂贵且持续重训的情况下防止模型准确度漂移。他解释了 NVIDIA NIM 容器与 NeMo Operator 如何简化跨 Kubernetes 及 HPE GreenLake 环境的安全部署与编排。此外，演讲还涵盖了 Blackwell 架构上基于硬件实现的机密计算，并概述了 NVIDIA 截至 2028 年的技术路线图，重点聚焦机架级液冷 Grace Blackwell 系统与模块化 MGX 参考设计。 企业 AI 正在向 Agentic AI 加速演进，多智能体工作流将推理转化为高密度计算挑战，推动基础设施向多节点及机架级架构转型以优化总体拥有成本（TCO）。 (2:09) 检索增强生成（RAG）通过将模型与动态企业私有数据相连，有效解决了模型精度漂移难题，在无需昂贵重复微调的情况下大幅延长了 AI 应用的生命周期。 (4:04) NVIDIA NIM 微服务提供针对前沿与开源模型的 Day-0 容器化支持及 CVE 漏洞扫描，结合 HPE 本地架构与 HPE GreenLake 混合云，在确保数据护栏的同时消除供应商锁定。 (8:11) NVIDIA NeMo Retriever 整合了基于 VLM 的文档 OCR 解析、cuVS 向量加速及语义重排微服务，并能通过 Kubernetes 上的 NeMo Operator 实现动态弹性调度。 (11:21) 为保障运行中数据的安全，Blackwell 架构引入硬件级机密计算，在 CPU 与 GPU 内存间构建可信执行环境（TEE），实现接近零性能损耗的数据防护。 (15:01)

NVIDIA and HPE present a full-stack strategy for enterprise agentic AI, arguing that production adoption requires moving beyond raw GPUs to tightly integrated software, retrieval pipelines, and hardware-level security. They emphasize that Retrieval-Augmented Generation (RAG) and optimized microservices lower total cost of ownership by eliminating frequent model retraining and solving inference bottlenecks at rack scale. Scaling Compute for Agentic Inference (2:09) Enterprise AI is shifting from conversational text generation toward autonomous agentic workflows that write code, generate market reports, and design molecular structures. According to NVIDIA, this transition fundamentally alters computational bottlenecks. While compute was historically concentrated in pre-training and fine-tuning, agentic multi-step reasoning and continuous tool use turn inference into an intensive workload requiring distributed multi-node and rack-scale infrastructure. To move enterprise deployments beyond experimental proofs of concept, organizations must achieve high operational efficiency. Hardware and algorithmic co-design aims to maximize throughput per node, which directly curbs total cost of ownership (TCO) and latency, ensuring applications justify their infrastructure investment. Preserving Accuracy via RAG and NeMo Retriever (4:04) Deployed enterprise models face continuous accuracy drift as external conditions evolve. Traditional periodic fine-tuning cycles resolve this drift but remain computationally expensive and operationally demanding. In contrast, RAG links models dynamically to live proprietary databases at query time, extending application shelf life without altering underlying weights—such as immediately updating an automotive configurator when new vehicle trims are introduced. To process complex enterprise records, the NVIDIA NeMo Retriever architecture applies NeMo Retriever Parse, a Vision-Language Model (VLM)-based OCR microservice that converts multimodal documents into structured Markdown while preserving tables and reading layout. Chunks are embedded and indexed in vector databases using cuVS acceleration, then evaluated by a Reranking NIM before submission to the foundation model. Deployment and auto-scaling across Kubernetes environments are managed via the NeMo Operator. Protecting Proprietary and In-Flight Data (6:32) Connecting foundation models directly to internal data repositories introduces significant data leakage risks. Enterprise RAG implementations require strict guardrails and output filters to prevent proprietary information from unauthorized exposure. Infrastructure partnerships, such as NVIDIA's integration with HPE GreenLake and on-premises HPE servers, provide hybrid cloud flexibility so enterprises can maintain control over proprietary data stores without re-architecting applications. To protect information beyond static storage and transit encryption, NVIDIA incorporates Confidential Computing to address runtime vulnerabilities. Linking the CPU, GPU, and memory into a hardware-enforced Trusted Execution Environment (TEE), the Blackwell architecture uses dedicated hardware encryption engines to protect in-flight data against malicious code injection with what the speaker characterizes as near-zero performance loss. Standardized Microservices and Hardware Roadmaps (8:11) To prevent enterprise vendor lock-in, NVIDIA Inference Microservices (NIM) package popular community and open-source models—including GPT-OSS and the Llama 4 family (such as Scout and Maverick)—into standard, production-ready containers. These microservices come pre-scanned for Common Vulnerabilities and Exposures (CVEs) and are optimized to reduce per-query inference costs across on-premises, hybrid, and multi-cloud environments. On the hardware level, NVIDIA maintains an architecture roadmap extending through 2028 to give server manufacturers predictable planning cycles. The liquid-cooled Grace Blackwell platform—exemplified by the GB200 and upcoming GB300—interconnects 72 Blackwell GPUs inside a single NVLink domain. To streamline manufacturing across the industry, NVIDIA contributes modular MGX reference designs to the Open Compute Project (OCP), covering both standard PCIe configurations and dense NVLink rack architectures. The presentation frames full-stack co-design—spanning microservices, dynamic retrieval, and liquid-cooled rack hardware—as essential for enterprise agentic AI. However, realizing these TCO and efficiency gains in practice depends heavily on successfully orchestrating distributed Kubernetes pipelines and managing the operational demands of high-density, multi-node infrastructure.

NVIDIA与HPE联合阐述了企业级人工智能从生成式向智能体（Agentic）演进的系统方案，指出应对智能体多步推理带来的机架级算力开销、模型精度漂移与私有数据安全问题，企业必须依赖软硬件协同的全栈架构与检索增强生成（RAG）流水线。 智能体演进与机架级推理算力 (2:09) 企业AI正从单纯的内容生成迈向具备自主多步骤分析能力的智能体AI，并延伸至机器人与边缘计算等物理AI领域。在智能体应用场景中，系统能够自主撰写代码、生成复杂市场报告或设计蛋白质链，但这种连续调用工具的多步骤推理过程，使得算力重心发生转移，推理计算需求已从单节点扩展至分布式多节点乃至整机架级集群。 为避免AI项目受阻于实验阶段并实现商业价值，企业必须严格控制总体拥有成本（TCO）。演讲者强调，单纯的硬件堆砌无法解决生产难题，需依靠算法与硬件协同设计，在确保低延迟响应的前提下最大化单节点吞吐量，从而优化基础设施规模与部署经济性。 检索增强生成与NeMo微服务流水线 (4:04) 面对模型因市场与业务变化而出现的“准确率漂移”，传统的周期性微调面临高昂的计算与运维开销。检索增强生成（RAG）通过将模型动态挂接至企业内部实时数据源，使应用无需反复重新训练即可全天候运行，显著延长了模型生命周期，例如汽车配置助手仅凭更新底层参数库即可支持最新上市年款。 在架构实现上，NVIDIA NeMo Retriever构建了端到端检索流水线：首先利用基于视觉语言模型（VLM）的NeMo Retriever Parse解析企业多模态文档，在保留版面与表格结构的同时输出结构化Markdown；随后通过Embedding NIM生成向量并利用cuVS加速索引；当用户发起查询时，NeMo Retriever Reranking NIM会对候选内容按上下文相关度重排，最后送入大模型生成。整个检索系统可通过Kubernetes平台上的NeMo Operator实现依据查询负载的弹性自动化伸缩。 NIM微服务与HPE混合云部署 (6:32) 当AI直接访问企业私有数据时，必须建立严格的安全护栏以防机密泄露。NVIDIA软件栈与HPE基础设施紧密集成，支持工作负载在本地HPE服务器与HPE GreenLake混合云之间灵活迁移，使企业在不重构应用逻辑的情况下保持对数据湖的完全自主掌控，避免云厂商锁定。 为加速软件交付，NVIDIA推出了推理微服务（NIM），将主流开源及前沿模型打包为经过吞吐量优化和自动化CVE漏洞扫描的生产级容器，并对GPT-OSS和Llama 4系列（包括Scout与Maverick版本）提供首日即时优化支持。此外，NVIDIA开发者门户还提供了涵盖多轮对话RAG与数据飞轮的Jupyter Notebook开发范例，降低从零构建企业应用的工程门槛。 运行时机密计算与MGX硬件路线图 (15:01) 针对推理计算期间数据在加速器与内存间流转时可能遭遇的代码注入风险，NVIDIA提供了机密计算方案。该技术在GPU、显存与CPU之间建立硬件级可信执行环境（TEE），Blackwell架构更集成了专用加解密硬件引擎，据称能在“近乎零性能损耗”的状态下防范未授权访问。 在底层基础设施层面，NVIDIA维持着延展至2028年的年度架构更新路线图。依托液冷机架级工程，Grace Blackwell平台的GB200及计划年内发货的GB300可将72颗Blackwell GPU整合于单一NVLink域内。同时，NVIDIA通过开放计算项目（OCP）公开包含标准PCIe与高带宽NVLink架构的MGX模块化参考设计，协助服务器OEM厂商加速落地整机架计算方案。 NVIDIA与HPE勾勒了软硬件协同推进企业级RAG落地的工程蓝图，但其长期成效仍取决于整机架液冷系统在大规模部署中的稳定性、混合云环境中私有数据的合规管理，以及自动化检索流水线在异构复杂文档解析中的实际表现。

## Background 本内容由 Ai4 频道于 2025 年 8 月 29 日发布在 YouTube 平台上。视频记录了 NVIDIA 加速计算产品总监 Dave Salvator 的演讲，主要探讨企业如何结合 NVIDIA AI Enterprise（NVAIE）平台与 HPE 基础设施，利用 RAG、微调和安全护栏等技术在专有数据上优化企业级 AI 应用。 ## TL;DR 演讲梳理了从生成式 AI 迈向智能体 AI（Agentic AI）的行业趋势，指出大规模推理正逐渐从单机扩展至整机柜级别。针对企业在知识时效性、数据隐私与成本控制上的痛点，NVIDIA 提出了由 NeMo Retriever、NIM 微服务及硬件级机密计算构成的全栈 RAG 落地架构。同时，NVIDIA 依托 MGX 模块化参考设计及持续到 2028 年的算力路线图，与 HPE 等 OEM 伙伴深度协同，助力企业在私有云与混合云环境中构建高安全、低 TCO 的生产级 AI 系统。 ## Atmosphere & style 整场演讲呈现出专业严谨且极具行业前瞻性的技术布道风格，主讲人条理清晰、语速稳健，善于将宏观算力趋势与底层工程架构无缝衔接。现场氛围务实而高效，既有对软硬件协同与安全边界的深入拆解，也有针对 MGX 设计细节的即兴互动解答，给观众带来扎实、可信赖的企业级技术落地指引感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=zUjLK11cEqo&t=4s) AI 行业正从生成式 AI（Generative AI）快速向智能体 AI（Agentic AI）及未来的物理 AI（Physical AI）演进，全栈软硬件协同是推动生产力跃升的关键底座。 - [02:09](https://www.youtube.com/watch?v=zUjLK11cEqo&t=129s) 在智能体 AI 时代，大规模推理（Inference at scale）已超越传统训练成为核心算力挑战，工作负载扩展至多节点乃至整机柜规模（Rack-scale），直接影响企业的总体拥有成本（TCO）。 - [04:04](https://www.youtube.com/watch?v=zUjLK11cEqo&t=244s) RAG（检索增强生成）通过直接连接实时企业数据库，有效解决了模型上线后的准确率漂移（Accuracy drift）问题，大幅延长了 AI 应用的货架期并避免频繁重训的昂贵成本。 - [06:32](https://www.youtube.com/watch?v=zUjLK11cEqo&t=392s) 企业落地 RAG 面临准确性、数据安全护栏（Guardrails）、防止过度共享及避免供应商锁定等多重挑战，需依托开放且具高度可移植性的技术架构。 - [08:33](https://www.youtube.com/watch?v=zUjLK11cEqo&t=513s) NVIDIA NIM 微服务为开源及前沿基础模型提供“零日支持（Day 0 Support）”，通过标准化容器封装加速模型落地，并支持跨本地、私有云（如 HPE GreenLake）及公有云的随处运行能力。 - [11:21](https://www.youtube.com/watch?v=zUjLK11cEqo&t=681s) 针对企业私有知识库的高敏感度，NIM 容器在发布前进行全方位 CVE 自动化漏洞扫描与安全加固，确保数据检索与模型访问链条的合规与安全。 - [11:56](https://www.youtube.com/watch?v=zUjLK11cEqo&t=716s) NeMo Retriever 通过基于 VLM 的 Parse 文档解析、cuVS 底层加速的 Embedding 向量检索以及 Reranking 重排序微服务，重构了高精度的企业级 RAG 数据链路。 - [13:49](https://www.youtube.com/watch?v=zUjLK11cEqo&t=829s) NeMo Operator 基于 Kubernetes 实现了多节点环境下的弹性伸缩编排，结合官方提供的多轮对话范式与数据飞轮示例，大幅降低了企业从 PoC 到生产上线的门槛。 - [15:01](https://www.youtube.com/watch?v=zUjLK11cEqo&t=901s) Blackwell 架构引入了硬件级机密计算（Confidential Computing）与专用加解密引擎，在受信任执行环境（TEE）中以接近零的性能损耗保障运行期（Data in Use）数据的绝对安全。 - [17:17](https://www.youtube.com/watch?v=zUjLK11cEqo&t=1037s) NVIDIA 保持“一年一迭代”的技术演进节奏并规划至 2028 年，通过全液冷 GB200/GB300 等整机架方案与共享至 OCP 的 MGX 模块化参考设计赋能 OEM 合作伙伴。 - [19:00](https://www.youtube.com/watch?v=zUjLK11cEqo&t=1140s) MGX 参考设计全面兼容包括 PCIe 与 NVLink 在内的多种高速互联技术，为 OEM 厂商从单节点服务器到机架级系统集成提供了统一且灵活的工程规范。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=zUjLK11cEqo&t=4s) 从生成式 AI 到智能体 AI 的演进与全栈解决方案 演讲者首先提到，上一场演讲深入探讨了检索增强生成（Retrieval-Augmented Generation，简称 RAG）的具体技术细节，而本次演讲则更偏向于高层宏观的综述，并由 NVIDIA 与慧与科技（Hewlett Packard Enterprise，HPE）联合呈现。回顾整个 AI 行业的发展脉络，大约一年前大家的关注焦点还是生成式 AI（Generative AI），而如今整个行业的重心已全面转向智能体 AI（Agentic AI）。未来，这一浪潮还将进一步延伸至与机器人、自主设备及边缘计算密切相关的物理 AI（Physical AI）。Agentic AI 展现出了巨大的应用潜力，能够显著放大人类的工作能力——不论是自动生成市场分析报告、编写高质量代码、设计蛋白质链，还是对这些生成内容执行更深层次的复杂分析，都带来了前所未有的生产力跃升。 面对 Agentic AI 带来的机遇，NVIDIA 秉持的是全栈式解决方案（Full Stack Solution）。外界常常有一种刻板印象，认为 NVIDIA 仅仅是一家制造高端 GPU 的硬件厂商。NVIDIA 确实打造了业界顶尖的 GPU，但硬件本身只是起点；如果没有庞大的生态系统支撑，缺乏成熟、稳健且持续优化的软件栈，单凭一块 GPU 根本无法解决任何实际业务问题。NVIDIA 在开源软件生态、主流框架以及最前沿基础模型（Frontier Models）上的支持力度几乎是无可比拟的，正是这种全栈能力在推动整个 AI 行业向前发展。 ```mermaid flowchart TD accTitle: AI 发展浪潮与全栈技术底座演进 subgraph S1["AI 范式演进阶段"] A["生成式 AI\n(Generative AI)"] --> B["智能体 AI\n(Agentic AI)"] B --> C["物理 AI\n(Physical AI / 机器人与边缘)"] end subgraph S2["NVIDIA 全栈赋能底座"] D["顶尖 GPU 硬件算力"] --> E["成熟稳定的全栈软件栈"] E --> F["开源框架与前沿模型生态"] end S2 --> S1 ``` --- ### [02:09](https://www.youtube.com/watch?v=zUjLK11cEqo&t=129s) 生产环境落地的核心挑战：成本、规模与推理瓶颈 将 Agentic AI 从概念验证（PoC）阶段推进到实际生产环境时，企业面临着多重严峻挑战。首要问题是成本控制与总体拥有成本（TCO），企业不仅需要确保投入产出比合理，还要保证系统性能足以向最终用户交付优秀的使用体验，使底层基础设施的价值最大化。过去业界普遍认为计算负载最重的是模型训练（尤其是预训练和微调阶段），但在 Agentic AI 时代，大规模推理（Inference at scale）正在迅速演变成一个巨大的算力挑战。复杂的智能体推理早已不再是单卡或单机多卡的问题，而是需要跨多节点（Multi-node），甚至需要扩展到整机柜规模（Rack-scale）才能有效支撑。 除了硬件层面的横向扩展，算法层面的持续创新同样至关重要，目的是在相同硬件规格下榨取更高的计算吞吐。NVIDIA 正与各大模型开发商、学术界研究员以及内部团队紧密合作以突破这些瓶颈。此外，系统的互操作性（Interoperability）、运行速度与执行效率直接决定了硬件的利用率。企业在部署生产级 AI 时，必须摆脱“仅做科学实验（Science Project）”的阶段，切实构建能为组织持续创造实际业务价值的生产系统。 - **推理扩展新常态**：Agentic AI 的推理工作负载正从单机多卡（Multi-GPU）迅速升级为多节点集群乃至整机柜规模（Rack scale）。 - **价值与效率绑定**：最大化利用现有基础设施算力，在控制总体拥有成本的前提下保障高吞吐与低延迟体验。 - **产学研协同优化**：通过算法创新与软硬件协同设计，持续提升单位硬件资源的产出效率。 --- ### [04:04](https://www.youtube.com/watch?v=zUjLK11cEqo&t=244s) RAG 的核心价值：延长模型生命周期与业务场景落地 在解决知识更新与精准回答的问题上，RAG 提供了一条无需频繁重新训练模型的理想路径。在传统的 AI 应用生命周期中，模型部署上线初期表现良好，但随着业务环境与市场需求的变化，用户提出的问题日益复杂新颖，模型的回答准确率便会出现漂移（Accuracy drift）。传统做法是将新收集的问题与数据作为训练集进行微调（Fine-tuning），这种方法虽然有效，但成本高昂且周期冗长。引入 RAG 后，模型可以直接连接企业内部数据库或外部公开数据源，只要底层数据库保持实时更新，AI 应用就能持续获取最新事实，从而大幅延长应用的“货架期（Shelf life）”，避免频繁陷入重训模型的循环。 ```mermaid flowchart LR accTitle: 传统微调与 RAG 知识更新机制对比 subgraph G1["传统模型维护流程"] A1["模型部署上线"] --> B1["遇到新问题/市场变化"] B1 --> C1["准确率漂移下降"] C1 --> D1["重新微调/训练模型"] end subgraph G2["RAG 增强维护流程"] A2["AI 应用服务"] --> B2["外接实时数据库"] B2 --> C2["检索最新业务数据与知识"] C2 --> D2["无需重训保持高准确率与长生命周期"] end ``` RAG 在企业各行各业中拥有极广的应用前景。以汽车选配器（Car Configurator）为例，车企每个车型年都会推出大量新车型，伴随着全新的配置、技术参数与规格详情。如果每次产品更新都要重训整个模型，维护成本将难以承受；而借助 RAG 架构，只需将最新的规格参数写入数据库供应用检索，同一套 AI 应用便可无缝跨越多个车型年继续稳定运行。 --- ### [06:32](https://www.youtube.com/watch?v=zUjLK11cEqo&t=392s) 企业落地 RAG 的核心顾虑与生态协同 尽管 RAG 带来了显著收益，但企业在落地过程中仍需面对诸多技术与治理层面的挑战： 1. **准确性与防漂移**：确保检索到的外部知识能够精准注入并正确理解。 2. **数据安全性与护栏机制**：当检索涉及企业内部的核心私有数据时，必须建立严格的安全护栏（Guardrails），防止数据在生成回答时泄露给未授权的公众或客户，杜绝“过度分享（Overshare）”。 3. **系统复杂性与成本控制**：应对多组件集成带来的架构复杂度，并在保障创新迭代速度（Innovation velocity）的同时压低运维成本。 4. **避免供应商锁定（Avoid Vendor Lock-in）**：这也是企业架构选型中最关键的一环，确保技术栈具备开放性与可迁移性。 在基础设施落地层面，NVIDIA 平台层及软件栈（包括 NVIDIA NeMo 等核心套件）已与 HPE 深度整合。企业不仅可以在本地部署的 HPE 物理服务器上平稳运行 NVIDIA 全套 AI 技术方案，还能够直接在 HPE GreenLake 云平台上灵活调用，实现从底层计算硬件、私有云平台到上层 AI 框架的协同交付。 ### [08:33](https://www.youtube.com/watch?v=zUjLK11cEqo&t=513s) 跨平台可移植性与 NVIDIA NIM 微服务生态 在 NVIDIA 平台上构建的企业级 AI 应用具备极高的可移植性与自由度，无论底层基础设施部署在本地数据中心（On-Prem）、公有云还是混合云环境，应用都可以实现“随处运行”（Run Anywhere）。这种多云与混合架构支持，充分契合了企业因历史遗留系统或数据合规而导致数据湖分散分布的现实需求。在多代际的长期技术规划中，NVIDIA 依托在 MLPerf 等行业标准基准测试中经受考验的软硬件能力（涵盖训练与推理环节，以及最新的 Blackwell 架构硬件），为企业提供了领先的性能支撑。在商业语境下，更强的计算性能不仅是技术优势，更直接转化为企业总体拥有成本（TCO）的显著下降。 为了进一步加速开源与社区模型的落地，NVIDIA 推出了 NVIDIA NIM（前身为 NVIDIA Inference Microservices）。当各大基础模型开发者发布最新成果时，NVIDIA 能够通过 NIM 提供“零日支持”（Day 0 Support），例如对 GPT-OSS 以及 Llama 系列前沿模型实现即刻适配。NIM 通过标准化的容器封装、算子深度优化以及严格的安全漏洞扫描，让开发者能够即开即用地部署最新的大语言模型，大幅压缩了应用的研发与上市周期（Speed to Market）。 - **全域可移植性**：一套代码无缝迁移于本地部署、公有云与多云架构之间，破除数据湖位置与算力孤岛的限制。 - **MLPerf 基准领跑**：在行业标准性能评测中持续验证推理与训练算力，结合新一代 Blackwell 架构大幅降低 TCO。 - **NIM 核心优势**：针对前沿开源模型提供即插即用的优化微服务容器，涵盖性能调优、依赖打包与生产级稳定性保障。 --- ### [11:21](https://www.youtube.com/watch?v=zUjLK11cEqo&t=681s) 企业级容器安全与数据合规防线 当生成式 AI 应用与企业专有知识库深度结合时，安全性成为了架构设计中最不容妥协的一环。NIM 容器在对外分发之前，均必须经过全面的自动化安全扫描机制，针对已知的通用漏洞与披露（CVE）及潜在的攻击面进行深度排查。这种机制让企业在将模型容器接入生产环境时拥有充分的安全信心。 这种安全保障在检索增强生成（RAG）系统中尤为关键。在典型的 RAG 工作流中，模型推理流程需要频繁调取并处理企业的核心内部数据库与私有资产。如果使用的第三方容器或开源镜像未经严格校验，可能会给企业带来严重的数据泄露或代码注入风险。通过对 NIM 进行常态化的漏洞检测与安全加固，NVIDIA 确保了应用从模型层到数据访问层的端到端安全边界。 - **CVE 漏洞扫描**：每个 NIM 镜像发布前均完成系统级安全检测，消除潜在的恶意暴露点与系统弱点。 - **内网数据接入保障**：当 AI 系统直连企业内部敏感数据库进行知识检索时，提供企业级可信合规环境。 --- ### [11:56](https://www.youtube.com/watch?v=zUjLK11cEqo&t=716s) NeMo Retriever 驱动的全流程 RAG 架构 为了系统化解决企业检索增强生成的工程化难题，NVIDIA 推出了 NeMo Retriever 解决方案。该方案通过一系列专用的 NIM 微服务重构了数据摄取与检索管线。数据摄取阶段首先通过基于视觉语言模型（VLM）的微服务——NeMo Retriever Parse 对复杂文档进行光学字符识别（OCR）与语义结构化解析。它不仅能准确提取文本和复杂表格，还能识别阅读顺序与内容分类，最终输出保留完整空间排版结构的 Markdown 文档。随后，Embedding NIM 结合底层 CUDA 加速库 cuVS 将文本向量化并写入向量数据库中，实现极致的高并发检索。 ```mermaid flowchart TD accTitle: NeMo Retriever RAG 工作流程图 subgraph DataIngest["1. 数据摄取与解析"] Doc["企业私有文档<br/>(PDF / 扫描件 / 复杂表格)"] --> Parse["NeMo Retriever Parse<br/>(基于 VLM 的 OCR 与版式解析)"] Parse --> MD["结构化 Markdown 数据<br/>(保留空间排版与表格结构)"] MD --> EmbedNIM["Embedding NIM<br/>(向量化计算)"] EmbedNIM --> VecDB[("向量数据库<br/>(基于 cuVS 算子加速)")] end subgraph QueryPipeline["2. 用户查询与检索优化"] UserQ["用户输入查询 / Prompt"] --> VecDB VecDB --> Candidate["初步候选文本块 (K-NN)"] Candidate --> RerankNIM["Reranking NIM<br/>(重排序微服务)"] RerankNIM --> TopK["上下文精准相关信息"] end subgraph Generation["3. 大模型生成"] TopK --> LLM["LLM 推理服务 (NIM)"] UserQ --> LLM LLM --> FinalAns["准确、可溯源的业务回答"] end ``` 当在线用户发起查询时，系统检索出的初步候选数据会进入 NeMo Retriever Reranking NIM 进行二次重排序。重排序微服务会根据语义关联度对候选结果进行精准打分筛选，确保注入大模型上下文窗口（Context Window）中的参考材料具有最高的准确度与相关性，从根源上遏制模型幻觉。NeMo Retriever 与 NIM 深度集成，使企业能够快速构建直连私有数据库的企业级 RAG 应用。 - **NeMo Retriever Parse**：依托 VLM 视觉大模型能力进行版式解析，无损保留表格排版与阅读顺序，输出高保真 Markdown。 - **cuVS 硬件加速**：利用 NVIDIA 底层 CUDA 向量计算库加速 Embedding 向量生成与索引检索。 - **Reranking NIM 精准重排**：过滤无关噪声，提升 Top-K 检索上下文的质量与准确性。 --- ### [13:49](https://www.youtube.com/watch?v=zUjLK11cEqo&t=829s) 基于 Kubernetes 的 NeMo Operator 运维与示例生态 在生产级基础设施管理方面，NVIDIA 提供了 NeMo Operator 工具，专门用于在 Kubernetes 平台上实现极速部署与智能编排。鉴于 Kubernetes 已经成为公有云和 HPE 私有云平台（如 HPE Private Cloud AI）的标准底座，NeMo Operator 能够为企业 AI 工作负载提供弹性的横向与纵向伸缩能力（Scale up / Scale down），根据业务流量波动动态调整计算资源分配，兼顾服务高可用性与算力成本控制。 为了降低开发者的上手门槛，NVIDIA 在官方文档站点（docs.nvidia.com）上提供了完备的技术指南和端到端示例。官方提供了多轮对话检索（Multi-turn RAG Pipeline）架构范式，以及基于 Jupyter Notebook 的“数据飞轮”（Data Flywheel）实践教程。开发者无需从零手写所有底层集成代码，通过交互式 Notebook 即可快速跑通整套检索、生成与数据闭环流程，快速完成概念验证（PoC）并推向生产。 - **Kubernetes 自动化运维**：支持全生命周期编排与多节点弹性伸缩，兼容各大公有云及 HPE 混合私有云底座。 - **多轮对话 RAG 范式**：内置针对复杂连续多轮业务问答的上下文管理与检索逻辑。 - **数据飞轮快速上手**：提供开箱即用的 Jupyter Notebook 示例资产，加速模型验证与微调迭代。 --- ### [15:01](https://www.youtube.com/watch?v=zUjLK11cEqo&t=901s) 硬件级机密计算保障运行期数据安全 在常规安全架构中，静态存储（Data at Rest）和网络传输中（Data in Transit）的数据往往已有成熟的加密手段，但在数据计算（Data in Use）——即数据从主存加载到加速器（GPU）显存并进行矩阵运算的过程中，依然存在被底层恶意代码注入或特权恶意进程窥探的风险。针对这一痛点，NVIDIA 引入了硬件级“机密计算”（Confidential Computing）技术。 ```mermaid flowchart LR accTitle: 机密计算 TEE 内存隔离与保护架构 subgraph Host["主机系统环境"] CPU["CPU (支持硬件加密)"] CPUMem["系统内存"] CPU <-->|硬件加密链路| CPUMem end subgraph PCIeBus["通信总线"] Bus["PCIe / NVLink 数据链路<br/>(端到端加密传输)"] end subgraph GPUArch["Blackwell GPU 隔离执行环境 (TEE)"] Engines["专用硬件加解密引擎<br/>(Dedicated Crypto Engines)"] GPUMem["GPU 显存 (HBM)"] GPUCore["GPU 计算核心"] Engines <--> GPUMem Engines <--> GPUCore end Host <--> Bus <--> GPUArch ``` 机密计算在 GPU 与显存之间构建起受硬件保护的受信任执行环境（Trusted Execution Environment, TEE）。配合支持加密特性的 CPU，系统能够实现计算链路上的全密态处理。尤其在新一代 Blackwell 架构中，硬件内部集成了专用的加解密引擎，使系统在开启全链路安全加密的同时实现“接近零的性能损耗”（Near-zero performance loss）。在多租户云环境或对商业机密极度敏感的企业 RAG 场景中，该技术为专有数据的运行期安全构筑了终极防线。 - **受信任执行环境（TEE）**：在 GPU 计算核心与高带宽显存（HBM）之间建立隔离安全区，阻断运行期内存窥视与提权攻击。 - **Blackwell 专用加密引擎**：采用硬件专用逻辑单元实现线速加解密，在保障密态计算的同时确保吞吐性能不受影响。 - **全生命周期安全闭环**：结合传输加密与静态存储加密，补齐“运行期数据安全”最后一块拼图。 ### [17:17](https://www.youtube.com/watch?v=zUjLK11cEqo&t=1037s) NVIDIA 算力路线图与 MGX 机架级系统创新 NVIDIA 当前将其算力路线图一直规划到了 2028 年，其背后的核心逻辑非常清晰：为了让 HPE 等关键生态合作伙伴清楚地了解未来技术演进的方向与时间节点。如今 NVIDIA 的硬件架构更新保持着“一年一迭代”（Annual Cadence）的极快节奏，在 GPU 侧保持每年推出新架构的同时，在 CPU、网络互联以及机架级（Rack-level）系统设计上进行全方位协同创新。特别是对于 Grace Blackwell 产品线而言，从当前的 GB200 到后续的 GB300，整机架系统设计已成为平台落地的关键推手，使得单个 NVLink 域内能够无缝聚合 72 颗 Blackwell GPU，并全面采用全水冷（Liquid Cooled）散热方案。 为了赋能整个硬件生态，NVIDIA 构建了名为 MGX 的模块化参考设计，并主动将其共享给开放计算项目（Open Compute Project, OCP），使其成为行业通用标准之一。这种开放策略极大地帮助了以 HPE 为代表的 OEM 合作伙伴加快硬件工程落地与系统集成进度，显著缩短新产品推向市场的时间（Time to Market）。对于数据中心与企业级计算行业而言，这种迭代速率是前所未有的，因此长期的路线图透明化与深度的工程协同，能确保 OEM 厂商提前做好产线与研发准备，最终让终端企业更快地获取算力资源来构建智能化应用。 ```mermaid flowchart TD accTitle: NVIDIA 机架级架构与生态赋能流程 subgraph ComputeInnovations["技术创新架构"] GPU["GPU/CPU 年化迭代"] Liquid["全机架水冷散热"] NVLink["NVLink 域互联 (72x GPU)"] end subgraph StandardDesigns["标准化参考设计"] MGX["MGX 模块化参考设计"] OCP["Open Compute Project (OCP) 行业标准"] end subgraph EcosystemAdoption["生态与客户落地"] OEM["OEM 合作伙伴 (如 HPE)"] Enterprise["企业级 AI 应用落地"] end GPU --> MGX Liquid --> MGX NVLink --> MGX MGX --> OCP OCP --> OEM OEM --> Enterprise ``` - **高密度互联与散热**：通过 NVLink 域将 72 颗 Blackwell GPU 互联，并配合整机架全液冷散热设计，释放极致算力。 - **开放生态协作**：MGX 参考设计不仅自用，更开放给 OCP 联盟，协助 OEM 加速产品开发周期。 - **超前路线图指引**：明确公开至 2028 年的规划，保持紧密沟通，确保上下游软硬件研发步调一致。 --- ### [19:00](https://www.youtube.com/watch?v=zUjLK11cEqo&t=1140s) 现场问答与会议结语 在演讲接近尾声的问答环节中，现场观众针对 MGX 的设计范围提出了疑问，询问其是否仅适用于 PCIe 方案。Dave Salvator 明确澄清，MGX 并非仅限于 PCIe，而是全面覆盖了包含 NVLink 在内的多种高速互联架构。他进一步解释道，过去 NVIDIA 曾主推 HGX 等方案，而随着机架级系统（Rack-level systems）时代的到来，参考设计的多样性大幅增加，因此公司扩展了设计体系并统称为 MGX，以适应从单一服务器节点到大型整机架系统的不同场景需求。 最后，由于演讲时间用尽，Dave 对全场参会者的到来表达了感谢，祝愿大家在本次大会中有所收获，并表示自己将在演讲台旁继续留步数分钟，以便与有进一步技术疑问的观众进行线下交流。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=zUjLK11cEqo&t=4s) 大约一年前大家的关注焦点还是生成式 AI（Generative AI），而如今整个行业的重心已全面转向智能体 AI（Agentic AI）。 — Dave Salvator > [00:04](https://www.youtube.com/watch?v=zUjLK11cEqo&t=4s) 硬件本身只是起点；如果没有庞大的生态系统支撑，缺乏成熟、稳健且持续优化的软件栈，单凭一块 GPU 根本无法解决任何实际业务问题。 — Dave Salvator > [02:09](https://www.youtube.com/watch?v=zUjLK11cEqo&t=129s) 复杂的智能体推理早已不再是单卡或单机多卡的问题，而是需要跨多节点（Multi-node），甚至需要扩展到整机柜规模（Rack-scale）才能有效支撑。 — Dave Salvator > [02:09](https://www.youtube.com/watch?v=zUjLK11cEqo&t=129s) 企业在部署生产级 AI 时，必须摆脱“仅做科学实验（Science Project）”的阶段，切实构建能为组织持续创造实际业务价值的生产系统。 — Dave Salvator > [04:04](https://www.youtube.com/watch?v=zUjLK11cEqo&t=244s) 只要底层数据库保持实时更新，AI 应用就能持续获取最新事实，从而大幅延长应用的“货架期（Shelf life）”，避免频繁陷入重训模型的循环。 — Dave Salvator > [08:33](https://www.youtube.com/watch?v=zUjLK11cEqo&t=513s) 在商业语境下，更强的计算性能不仅是技术优势，更直接转化为企业总体拥有成本（TCO）的显著下降。 — Dave Salvator > [08:33](https://www.youtube.com/watch?v=zUjLK11cEqo&t=513s) 当各大基础模型开发者发布最新成果时，NVIDIA 能够通过 NIM 提供“零日支持”（Day 0 Support）。 — Dave Salvator > [11:56](https://www.youtube.com/watch?v=zUjLK11cEqo&t=716s) 重排序微服务会根据语义关联度对候选结果进行精准打分筛选，确保注入大模型上下文窗口中的参考材料具有最高的准确度与相关性，从根源上遏制模型幻觉。 — Dave Salvator > [15:01](https://www.youtube.com/watch?v=zUjLK11cEqo&t=901s) 在数据计算（Data in Use）——即数据从主存加载到加速器（GPU）显存并进行矩阵运算的过程中，依然存在被底层恶意代码注入或特权恶意进程窥探的风险。 — Dave Salvator > [15:01](https://www.youtube.com/watch?v=zUjLK11cEqo&t=901s) 硬件内部集成了专用的加解密引擎，使系统在开启全链路安全加密的同时实现“接近零的性能损耗”（Near-zero performance loss）。 — Dave Salvator > [17:17](https://www.youtube.com/watch?v=zUjLK11cEqo&t=1037s) 如今 NVIDIA 的硬件架构更新保持着“一年一迭代”（Annual Cadence）的极快节奏，在 GPU 侧保持每年推出新架构的同时，在 CPU、网络互联以及机架级系统设计上进行全方位协同创新。 — Dave Salvator > [19:00](https://www.youtube.com/watch?v=zUjLK11cEqo&t=1140s) MGX 并非仅限于 PCIe，而是全面覆盖了包含 NVLink 在内的多种高速互联架构。 — Dave Salvator ## Mentioned Dave Salvator, NVIDIA, HPE (Hewlett Packard Enterprise), NVIDIA AI Enterprise (NVAIE), HPE Private Cloud AI, HPE GreenLake, NVIDIA NIM, NeMo Retriever, NeMo Retriever Parse, NVIDIA NeMo, NeMo Operator, Blackwell, Grace Blackwell (GB200, GB300), MGX, HGX, NVLink, cuVS, Kubernetes, Open Compute Project (OCP), MLPerf, Llama, GPT-OSS ## Video description Dave Salvator, Director of Accelerated Computing Products, NVIDIA Abstracts Join us for where we explore how HPE and NVIDIA transform enterprise AI with the NVIDIA AI Enterprise (NVAIE) platform. Learn how techniques like Retrieval-Augmented Generation (RAG), fine-tuning, guard railing, and prompt tuning can enhance AI effectiveness for business applications. Leveraging HPE’s infrastructure and NVIDIA’s AI software, enterprises achieve faster, secure, and efficient AI outcomes. Discover how integrating NVIDIA AI Enterprise with HPE Private Cloud AI maximizes AI investments. [Source](https://www.youtube.com/watch?v=zUjLK11cEqo)

[0:04] Dave Salvator: 大约一年前大家的关注焦点还是生成式 AI（Generative AI），而如今整个行业的重心已全面转向智能体 AI（Agentic AI）。 [0:04] Dave Salvator: 硬件本身只是起点；如果没有庞大的生态系统支撑，缺乏成熟、稳健且持续优化的软件栈，单凭一块 GPU 根本无法解决任何实际业务问题。 [2:09] Dave Salvator: 复杂的智能体推理早已不再是单卡或单机多卡的问题，而是需要跨多节点（Multi-node），甚至需要扩展到整机柜规模（Rack-scale）才能有效支撑。 [2:09] Dave Salvator: 企业在部署生产级 AI 时，必须摆脱“仅做科学实验（Science Project）”的阶段，切实构建能为组织持续创造实际业务价值的生产系统。 [4:04] Dave Salvator: 只要底层数据库保持实时更新，AI 应用就能持续获取最新事实，从而大幅延长应用的“货架期（Shelf life）”，避免频繁陷入重训模型的循环。 [8:33] Dave Salvator: 在商业语境下，更强的计算性能不仅是技术优势，更直接转化为企业总体拥有成本（TCO）的显著下降。 [8:33] Dave Salvator: 当各大基础模型开发者发布最新成果时，NVIDIA 能够通过 NIM 提供“零日支持”（Day 0 Support）。 [11:56] Dave Salvator: 重排序微服务会根据语义关联度对候选结果进行精准打分筛选，确保注入大模型上下文窗口中的参考材料具有最高的准确度与相关性，从根源上遏制模型幻觉。 [15:01] Dave Salvator: 在数据计算（Data in Use）——即数据从主存加载到加速器（GPU）显存并进行矩阵运算的过程中，依然存在被底层恶意代码注入或特权恶意进程窥探的风险。 [15:01] Dave Salvator: 硬件内部集成了专用的加解密引擎，使系统在开启全链路安全加密的同时实现“接近零的性能损耗”（Near-zero performance loss）。 [17:17] Dave Salvator: 如今 NVIDIA 的硬件架构更新保持着“一年一迭代”（Annual Cadence）的极快节奏，在 GPU 侧保持每年推出新架构的同时，在 CPU、网络互联以及机架级系统设计上进行全方位协同创新。 [19:00] Dave Salvator: MGX 并非仅限于 PCIe，而是全面覆盖了包含 NVLink 在内的多种高速互联架构。

[2:09] Dave Salvator: a science project [14:23] Dave Salvator: go full bore into writing code from zero [15:01] Dave Salvator: near zero performance loss

[0:04] We there was just a talk previous to [0:05] mine in this room uh from a company [0:07] doing a lot of work in retrieval [0:08] augmented generation or rag for short. [0:11] Um lot of good information in that talk. [0:13] My talk frankly is going to be at a [0:15] higher kind of overview level. This is [0:17] sort of being in done in conjunction [0:18] with Hullet Packard Enterprise HPE uh [0:21] who asked me to come in and kind of give [0:23] this overview talk. So here I am. Um, as [0:27] we've been talking here at this show, [0:28] you know, a lot about Agentic and also [0:30] generative AI. I mean, it was probably a [0:32] year or so ago that generative AI was [0:34] sort of the thing that really had [0:35] everyone buzzing and that has now [0:37] shifted to Agentic. And as we move [0:39] forward, that will make its way into [0:40] physical AI as well as it relates to [0:42] things like robotics, autonomous [0:44] devices, things at the edge and things [0:45] like that. Um, but Agentic really [0:48] represents a phenomenal opportunity. uh [0:51] the capabilities that Agentic is going [0:53] to bring us in terms of our ability to [0:54] kind of amplify our own capabilities as [0:56] to what we can do in our even in our [0:58] working lives. Whether that's you know [1:00] generating market analysis, market [1:01] reports, generating code, generating [1:03] protein chains and then being able to do [1:06] complex analysis on those things to [1:08] generate even sort of more sophisticated [1:10] information about the things that have [1:11] been generated um holds a tremendous [1:14] amount of promise. [1:17] And so uh you know one of the ways we [1:19] help solve for that is that we come at [1:21] this with a full stack solution. And I [1:23] won't spend too much time on this but [1:24] the main takeaway is that while uh there [1:27] is a tendency sometimes for people to [1:28] think oh Nvidia yeah they make those big [1:30] GPUs um we do make GPUs we make really [1:33] good GPUs. That said the GPU is just the [1:35] beginning. You know a GPU absent [1:38] ecosystem support absent great mature [1:41] robust constantly optimized software [1:44] will not solve any problems. So we [1:46] really come at this with a full stack [1:47] approach and that's actually super [1:49] important and I believe that our [1:50] ecosystem support our support for open [1:52] source software our support for open [1:54] source frameworks and and frontier [1:55] models is essentially unmatched and [1:58] that's really I think what's allowed us [2:00] um to help really move AI forward and [2:02] enable you to build the kinds of [2:03] applications you want to build both now [2:05] and in the future. [2:09] So as we think about taking AI, agentic [2:11] AI into production, it does pose [2:13] challenges, right? Obviously, uh cost is [2:16] a big one. How do we do this in a [2:17] cost-effective way? uh and so but in [2:20] addition to that beyond just cost and [2:22] sort of the business consideration [2:23] there's other sort of technical [2:24] considerations around getting [2:26] performance which actually truthfully [2:27] does map back to TCO but it also means [2:30] being able to deploy in a way um that [2:32] the app is delivering a great experience [2:34] to your users and that you're able to [2:36] really make best use of your available [2:38] infrastructure in a way that really um [2:40] helps you get the value out of it right [2:42] scalability super important especially [2:44] for Agentic while we historically have [2:47] thought that the really heavy part of AI [2:50] was things like training and and [2:51] admittedly pre-training is still a very [2:53] heavy operation. Fine-tuning can be a [2:55] pretty heavy operation, but increasingly [2:58] with uh Agentic AI, what we're seeing [3:00] now is that even inference at scale on [3:03] these agentic AI applications is now [3:04] becoming really an atscale problem. You [3:06] know, it's not just multiGPU, it's [3:09] actually multi-node and in some cases [3:10] it's like entire rack scale to really [3:12] make these things deploy and be [3:14] effective. And so um you know there will [3:16] be algorithmic innovations over time [3:17] that help uh help drive that forward and [3:20] help us do more on the same hardware. [3:22] And that's something we're constantly [3:23] working with model makers and and [3:25] researchers and our own researchers at [3:27] NVIDIA to help improve upon. But those [3:30] challenges nonetheless remain. And so [3:33] you know relating to uh inter [3:35] interoperability and of course speed is [3:37] efficiency which again relates to [3:39] getting the most mileage out of the [3:41] hardware that you're operating on so as [3:43] to deliver the most value for the least [3:45] amount of cost. Right? So these are just [3:46] some of the considerations that we all [3:48] wrestle with as we think about taking [3:50] these applications from say the PC stage [3:52] to actually deployed applications where [3:54] you know the expectation is is that this [3:56] should be not a science project but [3:58] something that actually generates value [4:00] for your organization in whatever [4:01] capacity it's meant to do that. [4:04] So one of the tools in our toolbox and [4:07] again this was talked about in the [4:08] previous talk. So you know a lot of this [4:10] material will be available on demand as [4:11] it's being recorded. So I would [4:13] encourage you to also check out that [4:14] talk. Um but retrieval augmented [4:17] generation you know sort of at the [4:18] highest of high levels really is about [4:20] using uh you know getting you more data [4:22] into your AI application frankly without [4:25] having to retrain it. In other words you [4:26] build the application and it then has [4:28] the ability to talk to your own internal [4:30] databases or other publicly available [4:32] databases as additional resources to [4:35] formulate answers. So in other words, if [4:37] you think about, you know, the typical [4:39] what I'll call life cycle of AI [4:41] application is that you deploy it, you [4:43] know, and this is sort of think of this [4:44] as pre- rag, right? You deploy it, it [4:47] works great for a while, you know, it's [4:49] delivering you the accuracy you need, [4:50] but then over time you start to see your [4:52] accuracy start to drift. And some of [4:54] that can be driven by usually the fact [4:55] that the questions that or the problems [4:57] that are being asked of your application [4:59] are changing and evolving as the market [5:00] changes and evolves. at which point, you [5:02] know, you want to take maybe some of [5:04] those inputs, use that as a new set of [5:06] training data, do another fine-tuning [5:07] run to bring your accuracy back up. [5:10] That's a, you know, that's a very sort [5:11] of valid technique and it does work. [5:13] That said, the use of of rag or [5:16] retrieval augmented generation allows [5:17] you to use these databases, which if [5:19] they're kept up to date and sort of the [5:20] information in those is, you know, the [5:23] most relevant, the most recent, it [5:24] allows that application to have sort of [5:26] a longer shelf life before really have [5:27] to think about retraining it. So, you [5:29] know, there's really so there's kind of [5:30] a technical component to this certainly, [5:32] but then there's also the component of [5:34] of just, you know, thinking about it [5:35] from the cost-effective deployment [5:36] perspective of keeping that application [5:39] up and running 247 for a longer period [5:41] of time before you have to think about [5:43] going back and and perhaps retraining [5:44] it. [5:47] So, here's some example use cases. I [5:49] mean, and there are many, but this is [5:51] just a couple, right, that you can see. [5:53] I mean, I won't read the names to you. [5:54] you can read them yourself, but you you [5:56] can see that it's a pretty broad swath [5:57] of potential applications here that uh [5:59] that you know where RAD can be [6:01] applicable. And again, if you think [6:02] about I'll just take the car [6:02] configurator for example. You know, [6:05] every model year, there's a whole new [6:07] army of cars that are brought to market [6:08] that have new features, new speeds and [6:11] feeds, new information, new sort of, you [6:13] know, specifications about that vehicle [6:15] that need to be updated. So if that can [6:17] be put into a database rather than [6:19] having to retrain the whole application [6:21] that allows you to keep using that same [6:22] application for multiple model years [6:24] that's just sort of one example as to [6:26] how rag has a role to play for that [6:27] particular application right [6:32] so you know but rag also you know within [6:34] rag it has its own actual challenges [6:36] right and so again I'm not really going [6:38] to you know read this to you as though [6:39] it was story time you guys obviously can [6:41] read so I'm not going to I'm not going [6:42] to sit here and try and read this all to [6:44] you but I will highlight a few things [6:45] I'm just going to do this this way so I [6:47] don't have to crane my neck to look at [6:48] it. So um and again these are similar [6:51] concerns to what we talked about before [6:53] relating to accuracy. Security of the [6:55] data is a really important one actually [6:57] particularly if the databases you're [6:58] using are touching data that your [7:00] company or your organization considers [7:01] proprietary and really shouldn't be led [7:03] out into the wild. Maybe it can be used [7:05] to inform answers, but you may want to [7:07] have it on some kind of a guard rail so [7:08] that you know that the information being [7:10] shared publicly or with your customer [7:11] base is at the level you want to have [7:14] information shared as opposed to sort of [7:16] an I'll call it an overshare, right? Uh [7:18] you know, complexity also, you know, [7:20] that's just a kind of a constant with AI [7:22] application uh development. And then of [7:25] course cost and then the innovation of [7:26] uh uh velocity. And this gets back to [7:29] kind of that comment I made a moment ago [7:30] about extending the life of your [7:32] application by using rag databases as [7:34] opposed to having to go back and sort of [7:36] re-engineer your model you know every [7:38] say six months or so or something. So if [7:40] we think about then you know how we [7:42] Nvidia are coming at this and by the way [7:44] of course all the work that we do at the [7:46] platform level and the software work [7:47] that we do this all um can be brought [7:50] and run on HPE servers you know for [7:52] onrem they also have their own cloud [7:54] platform called green lake where it can [7:56] also be run so you know as I'm as I sort [7:59] of talk through this yes these are [8:00] NVIDIA technologies but many of these [8:01] you know because this is a session that [8:03] we're sort of doing in conjunction with [8:04] HPE can be thought of as also running [8:06] very well on the on the HPE platform [8:10] So, as we look at, you know, the ways we [8:13] solve for some of these problems, you [8:14] sort of see it listed out here. Um, one [8:17] of the key ones I want to touch on real [8:18] quickly is is the uh avoid vendor lockin [8:21] piece. That one's super important in my [8:23] mind. And and I'll tell you why. because [8:25] with the solutions that we develop with [8:28] things like uh Nemo and some of the [8:30] other things we're going to talk about [8:31] here in just a moment um those [8:33] applications if they're built using [8:35] NVIDIA development tools can be run [8:36] pretty much anywhere on an NVIDIA [8:38] platform whether that's in the cloud [8:39] onrem hybrid what have you. So in other [8:41] words you really have maximum [8:42] flexibility or if you're running some [8:44] mix of say uh onrem as well as [8:48] multicloud and again different [8:49] organizations want to come at it [8:50] different ways. Sometimes there are [8:52] historical reasons why certain data [8:54] links live in certain places as an [8:56] example. But the point is if you're [8:57] developing on the NVIDIA platform that [8:59] ability to run the application is highly [9:01] portable and that's another important [9:03] consideration in terms of degrees of [9:04] freedom as you think about you know sort [9:07] of the multi-year sort of horizon of [9:09] building these applications right um [9:12] again you know really delivering [9:14] industryleading performance. We've [9:15] proven this time and time again uh in [9:17] places like the ML Perf industry [9:19] standard benchmark for both inference [9:20] and training. Um and so you know our [9:23] platforms continue to deliver that level [9:24] of leadership. Current uh generation of [9:26] product of course being called Blackwell [9:28] right so again you're really coming on [9:30] to a very strong platform to help solve [9:32] for a lot of these challenges. Uh with [9:34] rag being again one more way to help [9:36] solve for these challenges. Another [9:38] important tool we bring is what's called [9:40] NVIDIA NIM. Um NIM used to stand for [9:42] NVIDIA inference microservices. Now we [9:44] just call it NIM. And uh you know again [9:47] a lot of really great features in this [9:49] but the main thing that we do with these [9:51] particularly for community models which [9:52] are being developed by large model [9:54] makers you know when they release a new [9:55] model we have day zero support for that [9:58] model and we often have day zero support [10:00] for that model um using our NIM [10:02] technology which basically containerizes [10:04] optimizes security scans that container [10:07] and then is made available right so um [10:09] it's a way to get access to the latest [10:11] greatest community models recent [10:12] examples being things like GPTOSS [10:15] um as well as the the newest Llama 4 [10:17] models, you know, uh both Scout and [10:19] Maverick, which are all available as [10:21] NIMS, right? But it's a way to sort of [10:24] get your application built and deployed [10:26] faster. And then of course with these [10:28] NIMS, we are constantly working to [10:30] improve them, improve their performance [10:32] and uh and so bringing you more [10:34] performance. And performance in this [10:36] context can be thought of as a reduction [10:37] in TCO, right? I mean, more performance [10:40] is great. I love more performance. But [10:42] uh when you think about it in the [10:43] context of of you know sort of uh [10:46] essentially justifying the project and [10:48] saying why are we doing this? Well, one [10:49] of the answers is it's going to give us [10:50] additional capabilities in a way that's [10:52] cost effective. Right? [10:55] So here are some of the benefits of NIM. [10:57] Um you know and again I'm not going to [10:59] try try and read the whole slide to you [11:00] but you know I talked about deploy [11:01] anywhere. That's certainly an important [11:03] one. Um speed speed to market is a big [11:06] consideration there. uh running nims in [11:08] production uh of course super important [11:11] and then um and then you know again [11:13] simplifying deployment and I'm going to [11:14] talk a bit more about some additional [11:16] sort of facets of this here in just a [11:18] moment. [11:21] So one other piece because uh we [11:23] promised we'd talk about this in the [11:24] talk is security. I mentioned the fact [11:25] that these containers are security [11:27] scanned. Every time we do a new [11:29] instantiation of these they're security [11:30] scanned before being made available. And [11:32] so you know you have you have sort of [11:34] the confidence of knowing that what [11:35] you're getting with this containerized [11:36] version of the model is a model that has [11:38] been scanned for you know CVE and [11:40] potential uh threats or exposures in [11:42] such a way that you can deploy with [11:44] confidence right that's actually super [11:46] important um and it gets even more [11:47] important when you bring rag into the [11:49] equation since now this application may [11:51] be touching your internal databases [11:55] so the next thing I wanted to touch on [11:56] um is actually called Nemo retriever [11:59] which is sort of our sort of rag [12:01] solution that we have been offering now [12:02] for some time. I think we introduced [12:04] this back in 2024. Um I'm going to make [12:07] my best attempt to walk you through this [12:09] diagram. Um there are multiple [12:11] components here. Um I'm going to start [12:13] with uh what's called the embedding nim [12:15] and just see if I've got a laser pointer [12:17] here even if I can just find the darn [12:18] thing. Um but in any event um basically [12:22] this starts by taking uh uh the [12:24] embedding and converting that into we [12:26] actually convert for the moment into [12:27] vectors into a vector database using [12:29] something called KVS which is one of our [12:31] our CUDA libraries. Right from there [12:33] there we have what's called Nemo [12:34] retriever parse which is a VLM based OCR [12:37] microser for taking text and tables [12:39] extractions and preserving its semantic [12:42] structure and then uh taking that and [12:45] then putting it into essentially a [12:47] reading order to classify content types [12:49] and then output structured markdown to [12:51] retain spatial layout and formatting. [12:54] Bit of a mouthful but that's that's a [12:56] lot of what happens there. And then uh [12:58] when a query comes in, we will then be [13:00] able to do uh reranking using part of [13:02] neur retriever called the reranking nim, [13:04] which will rerank results for accuracy [13:06] with the most pertinent information that [13:08] the nim generates being part of the [13:10] response to make sure that it is sort of [13:12] informed, accurate, contextually [13:13] relevant. So this is a really great [13:16] solution for building rag applications [13:18] where you can then have them basically [13:19] pointed at the databases you want to be [13:21] referencing as part of this application. [13:24] And um so again it's just one more way [13:26] we help speed time to market not only [13:28] with NIMS but in addition with the Nemo [13:30] retriever um which sort of integrates [13:32] with NIM in fact NIM in some ways is [13:34] part of Nemo um they're very interlin so [13:38] um it basically allows you to build your [13:41] model your your application I should say [13:43] faster get it deployed faster generate [13:45] value faster right an additional piece [13:49] is what's called Nemo operator and so [13:51] what this does is this allows you to [13:53] deploy very quickly uh onto Kubernetes [13:55] platforms which has some obvious [13:56] advantages around things like [13:58] orchestration, scaling and the like. Um [14:01] this is super important obviously [14:02] because a lot of organizations like [14:04] using Kubernetes it's supported on all [14:05] the cloud platforms of course HPE [14:08] supports it as well. So it's another [14:10] great way to deploy where you can able [14:12] you're able to get that orchestration [14:13] that scalability and also that ability [14:15] to scale up and down on an as needed [14:16] basis depending on you know workload [14:18] demand right so it gives you a number of [14:20] pieces that really help you to deploy [14:23] and manage that application once it is [14:24] deployed um we actually have um quite a [14:27] bit of documentation about both Nemo [14:29] retriever and uh nim uh nvidia nim [14:32] operator available on docs.invidia.com [14:33] invidia.com. Uh in fact for operator in [14:37] particular, we actually have a couple of [14:38] sample applications. Uh one is a [14:40] multi-turn rag pipeline and then the [14:43] other one is actually a data flywheel [14:44] with a Jupyter notebook which of course [14:46] is a nice way to sort of dip your toes [14:48] uh sort of gently into this without [14:50] having to go full boore into you know [14:52] writing code from zero. So it sort of is [14:53] a guided way to help you understand to [14:55] kind of build your first application [14:57] using this and see how it runs. Right. [15:01] So another piece I want to touch on, [15:03] this is not so much related to rag but [15:05] it is a security piece which is also [15:06] quite important. Um it's not as relevant [15:09] necessarily to onrem although it [15:11] certainly could be. It's maybe more of a [15:13] cloud related thing but there's a [15:14] technology in our platform called [15:16] confidential computing which some of you [15:18] may have heard of right uh and what that [15:20] solves for is what kind of what you see [15:22] laid out here. You know if data is at [15:23] rest or is in transit it's generally [15:26] thought to be pretty secure. That said [15:28] when the data is being processed say [15:30] which is to say being moved between an [15:32] accelerator and its memory there is a [15:34] potential there for you know injection [15:36] of malicious code um or just other sort [15:39] of you know bad actors sort of coming to [15:41] try in and do not nice things right so [15:44] what we have uh in the form of [15:45] confidential computing is essentially we [15:48] create what's called a trusted execution [15:49] environment between the GPU its memory [15:52] um and then uh in conjunction with [15:54] particularly our systems uh that can [15:57] also have the CPU view which has a [15:58] similar technology for encrypting its [16:00] data between processor and its [16:02] respective memory. It allows you while [16:04] data is being processed to maintain that [16:07] data securely and that that application [16:09] data securely so that you are not really [16:12] having to worry so much about that would [16:13] be bad actor coming in and trying to do [16:15] something to your platform or your [16:16] application while you're trying to run [16:18] it. Right? So this is another feature. [16:20] You can also enable confidential [16:22] computing um with near zero performance [16:24] loss. This is another important feature [16:26] of it. We have actually dedicated [16:28] encrypt decrypt engines particularly in [16:29] the Blackwell architecture um that allow [16:32] for that near zero performance loss [16:33] while you get that encryption to create [16:35] that trusted execution environment. So [16:37] that's another important aspect. So [16:39] again um certainly something can be used [16:41] in conjunction with an application [16:42] that's using uh retrieval augmented [16:45] generation. Uh that said you know but [16:47] it's not necessarily you know only [16:49] pertinent to that. that can be used for [16:50] really any application, but it can help [16:52] add an additional layer of security to [16:54] your applications that are using Rag. [16:59] Okay. Uh this is actually my last slide [17:01] and if if um if I have a few minutes, I [17:03] might be able to take a few questions [17:04] and I'll do my best to answer them. But [17:06] I just wanted to touch on our roadmap. [17:07] This is the other thing that we do to [17:09] help continue to drive the platform uh [17:11] and just the industry really forward [17:13] with what we're building with AI. Um as [17:15] you see, we actually now carry our [17:17] roadmap out to 2028. We do this for a [17:20] pretty simple reason. Um, we want our [17:22] partners like HPE to know what's coming [17:25] and when it's coming because we're [17:27] moving at an annual cadence now with our [17:29] new architectures. Basically doing a new [17:31] architecture just about every year on [17:33] the GPU side while at the same time [17:35] innovating on CPU networking and even [17:37] rack level design. That's actually super [17:40] important because as we've moved to [17:41] particularly our Grace Blackwell [17:42] products, GB200 today, GB300 shipping [17:45] later this year. Um that that rack [17:48] design is a critical enabler to that [17:50] platform to be able to give you those 72 [17:53] Blackwell GPUs all in one NVLink domain [17:55] and the whole thing is liquid cooled. So [17:57] we actually build reference design [17:59] called MGX and we've even shared those [18:01] reference designs with the industry with [18:03] what's called the open compute project [18:04] or OCP where that is now one of their [18:06] reference designs as well and it's gone [18:08] a long way towards helping our OEM [18:10] partners like HPE and others uh to be [18:13] able to deploy and get their systems [18:15] built and into market more quickly. And [18:18] so, you know, but again, the main thing [18:20] you see here is that we're really we [18:21] continue to move at a pretty fast pace. [18:23] And this is actually for the kind of the [18:25] the data center side of the computing of [18:27] computer industry. We're moving at a [18:29] pace that is a very fast pace for for [18:31] those those segments. And so, we [18:33] actually are showing our road map out to [18:34] 2028 precisely because we want them to [18:36] understand what's coming and when it's [18:38] coming. And we're in constant [18:39] conversation with them about what it is [18:42] we're delivering and when we're planning [18:43] to bring it to market so they can [18:44] prepare, work with their teams, and then [18:46] give them the tools. as I talked about [18:48] the MGX reference designs as an example [18:50] as a way to help them uh basically speed [18:52] time to market and help us get these [18:54] products uh into your hands so you can [18:56] build your applications. And with that I [18:59] think I have about one minute left so I [19:01] might be able to take like a question or [19:02] two. So uh if not I will say go ahead. [19:05] Yeah. [19:06] >> So you mentioned the designs [19:11] for the PC. [19:14] >> Nope. They are um they are for all we do [19:17] them for NVLink as well. [19:18] >> Okay. [19:19] >> Absolutely. I mean if you think about [19:20] HGX as an example that was sort of a we [19:23] we we came up with the term MGX because [19:25] we have now multiple different reference [19:27] designs. It used to be primarily for the [19:28] NVL link and the PCA as you say but with [19:31] the advent of those rack level systems [19:33] you know we kind of expanded that [19:34] program. Okay. Um I think I'm about out [19:38] of time so I'm going to go ahead and [19:40] wrap it up and say thank you for [19:41] attending. Enjoy the rest of the [19:42] conference. I'll be available for a [19:44] couple of minutes afterward here if you [19:45] have any other questions. Thank you.