Nvidia Shifts Flagship Blackwell Chips to Handle AI Inference Load

英伟达转向使用旗舰Blackwell芯片承接AI推理负载 智能体AI的推理需求产生多达100倍的中间Token，促使企业直接将顶级硬件部署于推理，而非仅用于模型训练。 Dave Salvator概述了AI从生成式AI向智能体AI及物理AI的快速演进，重点指出了多步推理引发的算力激增如何促使NVIDIA Blackwell GB200等旗舰硬件转向以推理为主的部署模式。他详细介绍了包括NVIDIA NIM微服务、通过Dynamo实现的解耦服务以及每月容器更新在内的软件创新，这些创新大幅降低了总体拥有成本，并使现有硬件性能提升高达2.5倍。结合HPE的服务器生态系统与GreenLake云平台，上述解决方案及年度硬件更新周期有助于企业客户实现投资回报率的最大化。该环节最后以问答收尾，探讨了CPU与GPU工作负载的适用性、并行化分析和实时AI安全护栏等议题。 AI 发展正从生成式 AI 迅速转向智能体 AI（Agentic AI），并进一步扩展至自动驾驶和机器人等具身智能（Physical AI）领域。 (0:04) 智能体 AI 的多步推理过程生成的中转 token 数量是传统问答的 100 倍，这促使企业采取“推理优先”策略，将 NVIDIA Blackwell GB200 等旗舰硬件直接部署于推理任务。 (4:15) 数据中心通常将 GPU 利用率设定在 60% 至 80% 之间以优化 ROI；而液冷架构 GB200 NVL72 相比上一代 Hopper 架构每兆瓦计算容量提升了高达 50 倍。 (5:42) NVIDIA 通过月度容器化软件更新使同一硬件生命周期内性能提升 2 至 2.5 倍，并通过 Dynamo 的解耦式推理（Disaggregated Serving）为现有 GPU 集群带来 30% 到 40% 的吞吐量提升。 (11:54)

Speaking on behalf of Hewlett Packard Enterprise, NVIDIA's Dave Salvator argues that the rapid enterprise progression toward agentic AI is shifting compute pressure from training to inference. To achieve operational return on investment, organizations must balance high-density rack hardware with continuous software compilation gains and disaggregated inference serving. Agentic Reasoning and the Inference-First Infrastructure Shift The rapid enterprise transition from basic conversational generative AI to agentic and physical AI fundamentally alters compute demands. While pre-training foundational models from scratch remains cost-prohibitive for most individual organizations, operationalizing agentic workflows introduces heavy computational complexity. Unlike basic prompt-and-response queries, multi-step agentic tasks execute intermediate reasoning steps that generate up to 100 times more tokens behind the scenes before delivering an output. Because reasoning models consume immense compute at runtime, enterprises are altering their infrastructure acquisition strategies. Historically, organizations purchased premier GPU hardware exclusively for model training, later offloading depreciated systems to inference tasks. With architectures like the Blackwell GB200, enterprises are deploying flagship platforms directly as inference-first infrastructure. Compute footprints are expanding further through techniques like Retrieval-Augmented Generation (RAG), which connects models to live enterprise databases to expand context and preserve application utility without requiring costly model retraining. Hardware Density and Compounding Software Efficiency (5:42) To balance operational efficiency against latency requirements, data center teams generally target GPU utilization rates between 60% and 80%, reserving the remainder as headroom for transient traffic spikes. NVIDIA maps these trade-offs on Pareto efficiency curves, evaluating compute throughput per megawatt against latency response across various form factors. These include the 8-GPU air-cooled HGX B200 for standard data centers, the 4-GPU GB200 NVL4 for scientific workloads, and the liquid-cooled GB200 NVL72, which clusters 72 Blackwell GPUs via NVLink into a single domain providing up to 50 times more compute capacity per megawatt than previous Hopper generation systems. Salvator stresses that software optimization is the primary driver for lowering total cost of ownership (TCO) across hardware lifecycles. NVIDIA ships updated containerized software releases roughly every month. Through compiler and runtime refinements, these updates routinely deliver a 2x to 2.5x performance boost on identical physical hardware over the course of a single GPU generation. In enterprise deployments, this software stack integrates across HPE platforms, including B200, GB200, and upcoming B300 and GB300 systems managed under HPE GreenLake. Disaggregated Serving and Open Infrastructure Standards (11:54) To lower software friction, NVIDIA packages base models into standardized, security-scanned runtimes called NVIDIA Inference Microservices (NIMs), supporting community models such as customized Meta Llama variants. For real-time inference execution, the NVIDIA Dynamo framework applies disaggregated serving. This technique decouples the inference workflow into distinct prefill (input processing) and decode (token output) phases across dedicated, specialized GPUs, yielding a 30% to 40% performance improvement on existing hardware clusters. Additional tools, such as NVIDIA Agent IQ and TensorRT-LLM with speculative decoding, assist engineers in profiling and accelerating agentic latency. To keep pace with an aggressive annual release cycle across GPUs, CPUs, networking, and liquid cooling, NVIDIA distributes modular MGX reference blueprints. These validated designs reduce server manufacturers' development timelines by 6 to 12 months. To promote broader industry adoption, NVIDIA contributed its liquid-cooled GB200 MGX rack blueprint to the Open Compute Project (OCP) as an open standard. Workload Profiling and Parallel Execution Suitability (16:00) Addressing workload placement, Salvator notes that allocating tasks between CPUs and GPUs depends on algorithmic structure. Highly parallelizable, 'N-threaded' workloads that scale across hardware cores—such as deep learning and 3D graphics rendering—achieve substantial acceleration on GPUs. In contrast, serialized code paths execute more efficiently on high-clock-speed CPU architectures with lower core counts. When evaluating legacy enterprise applications for GPU acceleration, developers must profile workloads using tools like Intel VTune to detect whether latencies stem from memory saturation, cache misses, or compute limits. Only compute-bound, parallelizable applications justify GPU porting. Concurrently, GPUs are taking on real-time observability roles in production AI pipelines, executing guardrails and safety filtering to mitigate model hallucinations and filter inputs dynamically. Realizing concrete ROI from agentic AI depends not merely on adopting flagship GPUs, but on pairing hardware with continuous compiler updates, profiling workloads for parallel execution, and splitting inference pipelines to absorb massive reasoning token expansion.

NVIDIA的Dave Salvator代表Hewlett Packard Enterprise（HPE）指出，企业级AI正快速从基础生成式对话转向多步推理的代理型（Agentic）与物理AI，中间推理Token激增促使基础设施向“推理优先”转型，必须依托全栈软硬件协同来优化帕累托效率并加速投资回报（ROI）。 AI范式演进与推理算力激增 从基础生成式AI向代理型AI与物理AI（涵盖自主设备、机器人与自动驾驶）的演化正在深刻重塑计算架构。传统的生成式AI仅处理基础的提示词输入与文本输出问答，而代理型工作流需要执行复杂任务（例如自动生成多变量市场分析报告），其依赖的多步骤“思维链”（Chain-of-Thought）推理过程在生成最终结果前，后台产生的中间推理标记（Tokens）数量可达传统问答模式的100倍。 由于从头预训练模型需要耗费巨大的资金与算力，大多数企业倾向于采用开源社区基础模型并进行微调。同时，庞大的推理计算负担打破了以往“新旗舰芯片仅用于训练、淘汰旧硬件用于推理”的传统惯例。为了承载高密度的推理需求，众多企业开始将NVIDIA Blackwell架构和GB200等最新旗舰系统直接作为“推理优先”的基础设施进行部署。此外，企业利用检索增强生成（RAG）技术连接实时内部数据库，在避免昂贵的全量模型重新训练的同时拓展了应用寿命，但也进一步推高了生产环境的持续推理算力负载。 硬件架构演进与数据中心能效平衡 (5:42) 在数据中心运营中，IT团队通常将GPU的目标利用率设定在60%至80%之间，旨在最大化基础设施投资回报率的同时为突发流量峰值预留响应缓冲。NVIDIA通过帕累托（Pareto）曲线评估每兆瓦电力产出与响应延迟的最佳平衡点，并推出了多款适配不同工作负载的硬件方案：其中单液冷NVLink域连接72颗Blackwell GPU的GB200 NVL72系统，每兆瓦计算容量相较上一代Hopper架构提升高达50倍；面向传统标准数据中心提供8卡风冷的HGX B200；4卡配置的GB200 NVL4则专注于科学计算、分子动力学与流体仿真。 为应对硬件年度快速迭代对数据中心部署带来的挑战，NVIDIA与HPE展开深度合作，采用MGX预验证模块化服务器蓝图，帮助硬件制造商将机箱与冷却方案的设计周期缩短6至12个月。同时，NVIDIA已将GB200 MGX液冷机架设计捐赠给开放计算项目（OCP），推动业界形成统一的标准化基础设施参考规范。 全栈软件优化与分拆服务架构 (11:54) 单纯依靠物理硬件无法完全解决落地成本瓶颈，持续的软件栈优化是降低总体拥有成本（TCO）的关键手段。NVIDIA大约每月发布一次更新的容器化推理微服务（NIM）与数百个CUDA-X加速库，全面集成于HPE各代服务器（涵盖B200、GB200及未来的B300和GB300）与HPE GreenLake云平台。得益于编译器与运行时的持续优化，同一代物理GPU在其生命周期内通常能实现2至2.5倍的性能提升，无需企业追加硬件采购支出。 在服务部署层面，NVIDIA Dynamo框架引入了“分拆服务”（Disaggregated Serving）技术，将推理过程细分为处理上下文输入的Prefill阶段与生成输出Token的Decode阶段，并将其分别调度至针对该任务特别优化的专用GPU上。这种解耦机制使现有硬件集群的推理性能直接提升30%至40%。此外，配合投机解码（Speculative Decoding）技术、寻找帕累托最优配置的Agent IQ调优工具，以及基于Meta Llama开源的三种尺寸社区微调模型，企业能够更加高效地优化代理工作流。 计算负载选型准则与安全可观测性 (16:00) 针对应用在CPU与GPU之间的选型问题，核心判定标准在于算法架构是高度可并行还是高度序列化。具备高度并行特征的任务（如3D图形渲染与深度学习模型）在GPU架构上能够实现量级式的加速提升；而高度串行化的任务则依然在具备高时钟频率的传统CPU核心上运行更为高效。对于现有x86传统软件的代码现代化评估，开发者需借助Intel VTune等性能分析工具进行画像，明确其延迟瓶颈究竟源于内存压力、缓存饥饿还是原始算力不足；只有属于计算受限型且具备“N线程扩展”能力的任务，迁移至GPU后才能真正释放性能收益。 另一方面，随着生成式模型进入大规模实时生产，运维安全与可观测性成为不可忽视的关键环节。为了在生产环境中动态减少模型幻觉并过滤违规提示词，系统需要部署实时安全护栏（Guardrails）。这些实时的内容监控与安全审计机制本身对算力有着严苛的时效要求，正越来越依赖GPU强大的实时并行计算能力以维持合规保障。 Salvator强调，将AI转化为实际ROI的核心在于软硬件深度协同与端到端的工程化落地；然而，随着代理型推理对中间算力消耗呈百倍爆发，企业在实际扩张过程中仍面临如何在推理延迟瓶颈、动态安全护栏开销与高密度能耗极限之间达成长期平衡的现实考验。

## Background 该视频由 Ai4 于 2025 年 8 月 26 日发布在 YouTube 平台上，由 NVIDIA 加速计算产品总监 Dave Salvator 主讲。视频内容聚焦于 AI 的商业价值与投资回报率（ROI）分析，展示了如何结合 HPE 的企业级经验与 NVIDIA 全栈 AI 技术来加速企业 AI 落地并最大化投资回报。 ## TL;DR 演讲指出 AI 范式正迅速迈向 Agentic AI 与物理 AI，推理过程中的中间思考链导致 Token 消耗激增 100 倍，推动顶级硬件转向“推理优先”部署。通过 Blackwell 架构平台以及 NVIDIA 每月持续迭代的软件栈，企业能在相同硬件生命周期内获得 2 至 2.5 倍的免费性能红利，显著降低整体拥有成本（TCO）。此外，结合 Dynamo 分离式服务架构、推测解码技术以及 HPE 基础设施与 MGX 机架标准化设计，企业能够打破算力利用率与延迟的博弈困境。在问答环节中，演讲者进一步强调了任务的“N 线程可并行性”是判断业务应运行在 GPU 还是 CPU 上的核心决策标准。 ## Atmosphere & style 演讲兼具硬核技术深度与清晰的商业洞察，Dave Salvator 以幽默自嘲且务实的工程师风格，将原本晦涩的数据中心能效与软硬件架构拆解得生动透彻。现场节奏紧凑明快，通过帕累托前沿曲线与生动的运维博弈比喻，营造出专业、高效且极具说服力的行业研讨氛围。最后的现场问答互动直击性能分析与硬件选型的底层逻辑，展现出深厚的技术底蕴与实践指导价值。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=4s) AI 技术范式正迅速从生成式 AI 演进至智能体 AI（Agentic AI）和物理 AI（Physical AI），NVIDIA 通过开源社区模型与安全容器化的 NIM 推理微服务框架提供“零日支持”，大幅降低企业部署门槛。 - [02:35](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=155s) 智能体推理（Reasoning）在构建复杂思考链时消耗的 Token 数量激增多达 100 倍，使得顶级算力架构从以往的“模型训练优先”历史性地转变为“重度推理优先”。 - [02:35](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=155s) 利用检索增强生成（RAG）技术，企业可以在不重新训练基础模型的前提下连接最新业务数据库，有效控制迭代成本并延展 AI 应用的生命周期。 - [05:02](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=302s) 数据中心基础设施在追求 60% 至 80% 高算力利用率以最大化 ROI，与预留冗余以保障终端用户实时低延迟交互体验之间存在天然的资源权衡拉扯。 - [06:35](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=395s) 基于 Blackwell 架构的 GB200 NVL72 在每兆瓦（MW）功耗预算下可提供较上代高达 50 倍的容量提升，大幅外推了算力利用率与响应速度的帕累托最优边界（Pareto Curve）。 - [08:30](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=510s) NVIDIA 软件栈保持按月迭代发布，工程团队持续榨取既有硬件潜力，在不更换硬件设备的前提下直接摊薄单次任务成本，降低整体拥有成本（TCO）。 - [09:21](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=561s) 硬件芯片必须依赖端到端全栈生态才能创造业务价值，NVIDIA 现已开放数百个 CUDA-X 专用加速库，全面覆盖前沿 AI 与各类科学仿真计算。 - [10:16](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=616s) HPE 基础设施与 HPE GreenLake 混合云深度整合 NVIDIA 软硬件栈，同一代 GPU 硬件在生命周期内可通过持续软件优化累计获得高达 2 至 2.5 倍的免费性能红利。 - [11:54](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=714s) 推理服务平台 Dynamo 采用分离式服务架构（Disaggregated Serving），将计算密集型的预填充（Prefill）与访存受限型的解码（Decode）物理分发解耦，使同等硬件上的 LLM 性能直接提升 30% 至 40% 以上。 - [11:54](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=714s) 结合 TensorRT-LLM 中的推测解码（Speculative Decoding）与 Agent IQ 性能画像工具，开发者能够以极低算力开销达成更优的吞吐与延迟指标。 - [14:19](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=859s) NVIDIA 维持一年一代的全栈架构迭代，并通过向 OCP 捐赠 MGX 模块化机架参考设计规范，帮助服务器厂商缩减 6 至 12 个月的研发验证周期并加速整机落地。 - [16:01](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=961s) 工作负载的可并行性（Parallelizability）与 N 线程无限制线性扩展能力，是评估任务是否适合迁移到 GPU 还是保留在 CPU 的核心技术判定标准。 - [17:02](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=1022s) 在生产环境中部署 AI 时，模型防护栏（Guardrails）与实时可观测性对于抑制幻觉和拦截不当输入至关重要，这些实时安全监控任务同样依赖 GPU 算力进行加速。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=4s) 从生成式 AI 到智能体与物理 AI：社区模型与 NIM 框架的赋能 在科技行业深耕近三十年的演讲者开场提到，尽管行业向来因宣传炒作（hype）而闻名，但眼下 AI 技术的进化速度确实令人震撼。仅仅大约一年前，生成式 AI（Generative AI）还是全行业最核心的话题；而转眼之间，技术焦点已迅速转向智能体 AI（Agentic AI），并进一步向物理 AI（Physical AI，如自主设备、机器人、自动驾驶汽车等）延伸拓展。这种前所未有的创新节奏不仅体现在底层计算平台能力的跃升上，也体现在上层 AI 应用的爆炸式增长中。 从历史上看，从零开始预训练（Pre-training）大模型曾是 AI 落地的最大门槛，因为它不仅需要承担高昂的算力基础设施成本，还需要耗费海量的训练时间。如今，具备雄厚资源的模型开发者正在构建并向社区开放各类优秀的开源社区模型（Community Models）。企业和开发者能够获取这些模型，通过微调自身数据集、采用各类优化算法来激发下一轮业务创新。为了确保生态效率，NVIDIA 为这些社区模型提供“零日支持”（Day 0 Support），并将其封装进 NIM 框架（NVIDIA Inference Microservices，NVIDIA 推理微服务）。NIM 提供经过安全漏洞扫描的容器化服务，内置深度性能优化，使企业能够跨本地数据中心（On-premises）或公有云环境无缝部署高性能推理。 ```mermaid flowchart LR accTitle: AI模型从预训练到企业部署的演进路径 subgraph S1["模型生产与优化"] A["基础模型预训练"] --> B["开源社区模型发布"] B --> C["企业专有数据微调/优化"] end subgraph S2["NVIDIA NIM 交付与部署"] C --> D["NIM 容器化与安全扫描"] D --> E1["本地数据中心 (On-Prem)"] D --> E2["公有云环境 (Public Cloud)"] end ``` --- ### [02:35](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=155s) 推理算力爆炸：从简单问答到 100 倍 Token 的复杂推理 随着 AI 范式从生成式转向智能体（Agentic），算力需求的核心瓶颈正在发生历史性转移。过去的生成式 AI 主要是“输入 Token — 输出 Token”的简单问答模式；而具备复杂推理能力（Reasoning）的智能体，不再只是回答事实性问题，而是承接“依据特定标准撰写完整市场深度分析报告”等高难度复杂任务。模型在得出最终结论前，会在内部生成长链条的中间思考过程（Chain of Thought），导致推理过程消耗的 Token 数量激增多达 100 倍，对计算算力提出了极高的即时响应要求。 由于推理负载激增，传统上“训练吃重算力、推理吃轻算力”的格局彻底逆转。诸如 NVIDIA Blackwell 系列（如 GB200 旗舰平台）等最新架构，过去通常优先部署于大规模训练集群，现在许多客户却选择将其作为“推理优先”（Inference-first）的平台进行上线。与此同时，在企业落地层面，如何兼顾系统响应速度、互操作性以及复用现有数据资产成为核心挑战。通过检索增强生成（RAG）技术，企业可以在不重新训练模型的前提下，让应用持续查询最新的业务数据库，既延展了 AI 应用的生命周期，又显著控制了迭代成本。 - **推理模式对比**：传统生成式 AI 是单轮问答（Q&A），而智能体推理（Reasoning）需要拆解子任务并规划求解路径。 - **计算倍率剧增**：推理思考链阶段生成的中间 Token 量可达传统生成的 100 倍（100x more tokens）。 - **工作负载翻转**：顶级算力芯片从以往的“首选用于模型训练”转变为现在的“优先部署于重度推理业务”。 - **RAG 降本增效**：利用企业现有数据库进行知识增强，避免高昂的模型重训练成本，延长模型业务周期。 --- ### [05:02](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=302s) 基础设施的双向博弈：利用率与用户体验的平衡 在推理算力需求激增的背景下，数据中心面临着严峻的资源权衡挑战。演讲者幽默地提到自己像“电视台天气预报员一样频频转头看大屏幕”，随后深入拆解了基础设施架构师的核心痛点：如何在“维持高算力利用率”与“提供极致终端用户体验”之间找到平衡点。 任何基础设施运维管理者（Infra Manager）的核心 KPI 都是服务器算力利用率。如果算力集群长期闲置，就意味着企业未能最大化投资回报（ROI）。行业通常追求将集群利用率维持在 60% 至 80% 的高负载区间，预留 20%~40% 的冗余缓冲（Headroom）以应对瞬时流量洪峰。然而，极高的利用率往往会导致排队延迟增加，进而损害终端用户的实时交互体验与响应速度。这两者在实际运维中往往是相互拉扯的“拔河”关系，不同企业和业务场景必须在延迟与成本之间进行精准取舍。 ```mermaid flowchart TD accTitle: 基础设施利用率与用户体验之间的平衡博弈 subgraph TradeOff["数据中心资源权衡矩阵"] direction TB A["高算力利用率 (60% - 80%)"] --- VS["⚖️ 权衡与取舍"] --- B["极致用户响应体验 (低延迟)"] A --> C["优势: 提升硬件投资回报 (ROI)<br/>劣势: 瞬时高并发易产生排队延迟"] B --> D["优势: 保证复杂推理即时交互<br/>劣势: 需预留高昂算力冗余成本"] end ``` --- ### [06:35](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=395s) Blackwell 矩阵与帕累托前沿：打破每兆瓦算力极限 为彻底解决利用率与响应速度的冲突，NVIDIA 推出了基于 Blackwell 架构的全系列算力平台，并与 HPE 等合作伙伴共同推向市场： 1. **GB200 NVL72**：将 72 颗 Blackwell GPU 通过统一的 NVLink 互联网络连接在一个域内，逻辑上表现为一颗拥有统一内存池的“超大 GPU”。 2. **HGX B200**：采用更为标准传统的 8 卡（8-way）模组形态，可适配标准风冷服务器，满足主流机房的快速升级需求。 3. **GB200 NVL4**：专为 HPC 与科学模拟（如计算流体力学 CFD、分子动力学等）量身定制的算力形态。 演讲者通过帕累托曲线（Pareto curves）阐述了硬件跃迁带来的业务变革。帕累托曲线反映了在给定功耗（例如每兆瓦 MW）限制下，基础设施利用率与终端响应性能的最佳平衡边界。相较于上一代 Hopper 架构，GB200 NVL72 能够在每兆瓦功耗预算下提供高达 50 倍的容量提升。这不仅意味着极限性能的飞跃，更为企业在帕累托最优前沿上提供了更大的战略自由度——企业可以根据自身的预算、延迟容忍度和并发需求，自由选择最理想的业务落点。 ### [08:30](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=510s) 软件持续迭代释放隐性收益：以更高性能降低整体拥有成本（TCO） 每个企业在规划 AI 落地时，都必须面临一项关键决策：如何以兼具成本效益的方式进行系统部署。随着时间推移，业务需求和技术环境都在不断演化，而 NVIDIA 平台的独特优势在于其容器化软件保持着大约每月一次的发布节奏。在每个月的新版本中，工程团队不仅在排查并修复 Bug，更关键的是在“完全相同的硬件上榨取并交付出更高的性能”。 从商业价值的角度来看，单纯追求更高性能虽然令人兴奋，但当它被转化为财务指标时，其核心本质在于“相同的硬件投资能够交付更多的业务吞吐能力”，从而直接拉低部署应用的整体拥有成本（TCO）。企业无需更换物理基础设施，仅凭软件升级就能使单次推理或任务调用的成本持续摊薄。 - **按月更新节奏**：NVIDIA 保持约每月发布一次容器化软件镜像，在生命周期内持续深挖硬件潜力。 - **商业本质转化**：在算力投资固定不变的前提下，性能的提升直接等同于 TCO 的降低。 --- ### [09:21](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=561s) 硬件本身无法解决问题：纵贯全栈的 CUDA-X 软件生态 仅仅拥有强大的硬件是无法直接解决实际业务痛点的。计算芯片本身并不能凭空产出价值，企业必须依赖一个向上贯穿整个技术栈的健壮平台。NVIDIA 与 HPE 的合作正是基于这一整套端到端架构。以 NVIDIA 的 CUDA-X 库为例，其数量现已扩充至数百个，覆盖的领域不仅包括尖端的人工智能，还延伸至极其广泛的科学仿真计算。 所有这些加速库均已对外开放，开发者可以直接从 NVIDIA NGC 网站下载，或者在 GitHub 上获取开源实现。如果某项极其小众或新兴的应用场景在当前库目录中尚未找到现成工具，NVIDIA 的工程团队也在持续评估开发者诉求，并以极快的速度补充新的专用加速库，确保上层开发者能够以最低的工程阻力构建应用并获取所需的极致吞吐。 ```mermaid flowchart TD accTitle: NVIDIA与HPE全栈协同架构 subgraph AppLayer["业务与应用层"] A["企业生成式 AI / 科学仿真 / Agent 应用"] end subgraph SoftwareStack["NVIDIA 平台软件栈"] B["NIM / Agent IQ / Dynamo 推理服务"] C["CUDA-X 专用加速库群（数百个）"] D["底层 CUDA 驱动与月度优化容器"] end subgraph HardwareLayer["HPE 基础设施与云服务"] E["HPE GreenLake 混合云服务"] F["HPE AI 硬件系统（B200 / GB200 / GB300 等）"] end A --> B B --> C C --> D D --> E E --> F ``` --- ### [10:16](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=616s) HPE 基础设施与 NVIDIA 平台协同：硬件生命周期内的 2~2.5 倍免费红利 当 NVIDIA 的软件平台落实在 HPE 构建的本地服务器上时，协同效应开始显现。无论企业部署的是基于当前世代的 B200、GB200，还是面向未来的 B300 与 GB300 架构，这些设备均能原生获得软件持续优化所带来的全部红利。企业在采购硬件时，实际上是在对 HPE 的企业级基础设施与 NVIDIA 的软件平台进行双重投资。 在以往的 GPU 世代中，凭借工程团队在每月容器迭代中持续进行的“无休止软件调优”，客户往往能在不改动任何硬件的前提下，让既有系统跑出相比初始交付状态 2 倍、甚至高达 2.5 倍的性能提升。这种性能飞跃对企业而言相当于获得了一笔持续的“免费 TCO 缩减红利”，构成了企业与 HPE 及 NVIDIA 长期绑定的核心财务逻辑。与此同时，HPE 还通过其 HPE GreenLake 云平台提供了涵盖监控、管理在内的丰富企业级附加服务，使企业不仅获得了 NVIDIA 的核心算力，也完整拥有了 HPE 构建的私有云与混合云运营保障。 - **同代硬件性能增益**：通过容器化软件持续优化，同一代 GPU 硬件在生命周期内可累计实现高达 2 至 2.5 倍的性能提升。 - **多代产品演进支持**：全面覆盖从 Blackwell 世代的 B200、GB200 到后续 B300、GB300 的全线服务器阵列。 - **HPE 增值服务整合**：借助 HPE GreenLake 提供弹性的云化交付与运营支持，降低本地机房的管理复杂度。 --- ### [11:54](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=714s) 突破推理吞吐瓶颈：开源社区模型、Dynamo 分离式服务与推测解码 随着应用逐渐从单次 Prompt 交互走向复杂的 Agentic AI（智能体应用），对计算性能的需求呈现指数级爆发。为了协助开发者将应用精确调优至帕累托最优曲线（Pareto curve），NVIDIA 提供了多种进阶工具与前沿架构： 在模型与调优层面，NVIDIA 基于 Meta 的开源 Llama 架构微调并开源了三个不同尺寸的社区模型供生态免费使用；配合 Agent IQ 等性能画像与优化工具，开发者能够精准微调业务应用，以最优算力开销达成既定延迟与质量目标。 在系统级服务层面，NVIDIA 引入了专为生成式 AI 和 Agent 设计的全新推理服务软件平台 Dynamo，其核心杀手锏是**分离式服务架构**（Disaggregated Serving）。传统的 LLM 推理过程被划分为计算密集型的“预填充”（Prefill）阶段和访存受限型的“解码”（Decode）阶段。由于二者对硬件资源的敏感维度截然不同，Dynamo 将这两个阶段在物理层面上解耦，分别调度到不同 GPU 节点上独立执行。 仅通过转向分离式服务架构，在完全相同的硬件设备上，大语言模型的实际性能即可迎来大约 30% 乃至超过 40% 的直接跃升。此外，集成在 TensorRT-LLM 中的**推测解码**（Speculative Decoding）技术也发挥了关键作用：系统通过轻量级的草稿模型（Draft Model）先行推测候选 Token，再交由主模型一次性验证，以极低的算力开销换取更高的推理生成速度。 --- ### [14:19](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=859s) 一年一度的架构跨越与机架级工程标准：MGX 规范加速落地 NVIDIA 目前保持着每年推出一套全新计算架构的极快演进节奏。这种升级不仅仅局限于 GPU 芯片本身，而是全方位涵盖 CPU、GPU、网络互联以及至关重要的**机架级系统设计**（Rack Design）。在以 Grace Blackwell 为代表的高密度、全液冷系统中，机架工程对于系统的正常运转起到了决定性作用。 针对服务器厂商长达 6 到 12 个月的底盘设计与工程验证周期，NVIDIA 推出了模块化参考设计架构 **MGX**（包括针对 GB200 及未来 GB300 的专用方案）。它为 HPE 等系统制造商提供了一套已经过完整优化与压力验证的工程蓝图，使合作伙伴可以快速推出整机系统、加速上市。NVIDIA 甚至将包括 GB200 在内的多项 MGX 规范无偿捐赠给了开放计算项目（Open Compute Project, OCP），将其转化为行业开放参考标准。面对传统数据中心行业相对保守的演进习惯，NVIDIA 正试图通过整机设计标准化，“带动整个数据中心产业跟上每年一次的大步快跑节奏”。 - **全要素年度迭代**：芯片（GPU/CPU）、高速互连网络与机架结构全面保持一年一代的同步刷新。 - **液冷工程蓝图**：MGX 参考规范有效省去了服务器厂商耗时 6~12 个月的机箱底盘重新研发周期。 - **生态开放共享**：将 GB200 的 MGX 核心规范捐献给 OCP，确立全球数据中心基础设施的标准化范本。 --- ### [16:01](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=961s) 现场互动：任务的可并行性决定底层芯片选型 在演示主体内容提前五分钟结束后，演讲者开放了现场问答。面对台下观众关于“如何评估一项业务任务究竟适合跑在 GPU 还是 CPU 上”的提问，演讲者风趣地调侃称：“在科技行业，所有问题的标准答案通常都是‘视情况而定’（It depends）。” 随后，演讲者给出了一个直击本质的技术判断标准——**任务的可并行性**（Parallelizability）： 如果一个业务计算任务可以被拆解成众多相互独立的子任务，并在海量计算核心上同时分发处理，那么 GPU 展现出的吞吐优势将是碾压式的，性能差距往往会达到数倍甚至数十倍的数量级。相反，如果业务流程高度依赖串行逻辑、前一步输出直接决定下一步分支，那么 CPU 依然是更合理的选择。因为 CPU 虽然核心总数较少，但每个单核的主频通常更高，底层微架构也是专门为了快速处理高度串行化的任务而设计的。随着未来各类业务可观测性数据分析与算法逻辑向并行化重构，GPU 适用的场景边界还将继续向传统 CPU 阵地扩展。 ### [17:02](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=1022s) 模型防护栏、可观测性与 GPU 的安全角色 在模型运行与部署过程中，对大语言模型进行实时监控与状态追踪（Observability）是一项至关重要的任务。这不仅包括设置防护栏（Guardrails）以最大限度地抑制和避免幻觉（Hallucinations），还包括对输入提示词进行实时审查，确保模型在遭遇不当提示（Inappropriate Prompts）时能够明确拒绝回答。戴夫·萨尔瓦托（Dave Salvator）打着手势引号强调，必须确保模型“表现规矩”（"behaving itself"），而在处理这些防护栏算法与实时可观测性任务时，GPU 算力同样扮演着不可或缺的关键加速角色。 ```mermaid flowchart TD accTitle: AI 模型安全与可观测性流程 subgraph InputControl["输入控制"] A["用户 Prompt 输入"] --> B["Guardrails 防护栏检测"] B -->|"拦截恶意/不当请求"| C["拒绝响应"] end subgraph Execution["模型推理与监控"] B -->|"安全通过"| D["LLM 核心推理"] D --> E["幻觉抑制与输出审查"] E --> F["合规输出结果"] end subgraph Acceleration["底层算力支撑"] G["GPU 算力加速安全监测与推理"] -.-> B G -.-> D G -.-> E end ``` --- ### [17:33](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=1053s) 传统性能分析法与 GPU 适用性：如何判断任务是否适合并行加速 在演讲进入最后两分钟的问答环节时，针对现场关于“是否利用 AI 本身来分析大型应用内部各个子任务”的提问，戴夫表示英伟达内部通常更依赖传统的性能特征分析（Performance Characterization），而非直接使用 AI 去做代码拆解。判断一个应用是否适合移植到 GPU 的核心首要问题，在于该算法是否具备“$N$ 线程化”（N-threaded）的能力——即能否随着处理器核心数的增加无限制地扩展线程数量并获得线性的性能提升。有些程序天生受限，最多只能扩展到 2 到 4 个线程，而另一些程序则是“你给它多少线程，它就敢吞下多少线程”。对于后者而言，迁移到 GPU 获得巨大加速是毫无悬念的选择。 现代计算任务中，AI 和 3D 图形渲染均属于典型的“易并行”（Embarrassingly Parallelizable）工作负载，这正是 GPU 架构最擅长的领域。对于传统的 CPU（如 x86 架构）应用程序，业界通常使用英特尔经典的 VTune 等工具来进行深度剖析，以此诊断程序的性能瓶颈究竟是卡在内存带宽、缓存命中率不足，还是属于纯粹的计算受限（Compute-bound）。 如果分析表明应用程序具备高度的 $N$ 线程并行能力，并且数据流与缓存调度得当、计算单元不会发生饥饿等待，那么将其迁移到 GPU 上必定会带来显著的性能跃升。相反，如果算法在架构上只能扩展两三个线程，最理智的策略则是将其留在 CPU 上处理，仅把高度并行的子任务卸载给 GPU。在回答完该问题后，戴夫向现场观众致谢并结束了本次演讲。 ```mermaid flowchart LR accTitle: 工作负载特征分析与硬件选型决策流程 A["应用程序性能分析<br>(例如使用 Intel VTune)"] --> B{"是否可 N 线程并行扩展<br>(N-threaded)？"} B -- "高并行度 / 易并行<br>(AI、图形计算)" --> C{"瓶颈是否为计算受限<br>(Compute-bound)？"} C -- "数据供给充分 / 计算受限" --> D["迁移至 GPU 加速<br>(获得显著性能提升)"] C -- "内存带宽受阻" --> E["优化缓存与数据流后<br>再迁移至 GPU"] B -- "仅支持 2-4 线程<br>(串行度高)" --> F["保留在 CPU 运行<br>(仅卸载局部并行任务)"] ``` - **任务并行特征分类**： - **易并行任务（Embarrassingly Parallel）**：如 AI 大模型计算、3D 渲染，对线程数需求极高，是 GPU 加速的最佳场景。 - **低并行/串行任务**：仅能扩展 2 至 4 个线程，强行迁移到 GPU 无法发挥架构优势，更适合保留在 CPU。 - **性能诊断工具**：推荐使用如 Intel VTune 等工具定位程序瓶颈（分析程序是卡在缓存、受限于内存压力，还是纯粹的计算受限）。 ## Notable quotes > [02:35](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=155s) 输入 Token — 输出 Token — Dave Salvator > [02:35](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=155s) 推理优先 — Dave Salvator > [08:30](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=510s) 完全相同的硬件上榨取并交付出更高的性能 — Dave Salvator > [08:30](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=510s) 相同的硬件投资能够交付更多的业务吞吐能力 — Dave Salvator > [10:16](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=616s) 无休止软件调优 — Dave Salvator > [10:16](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=616s) 免费 TCO 缩减红利 — Dave Salvator > [14:19](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=859s) 带动整个数据中心产业跟上每年一次的大步快跑节奏 — Dave Salvator > [16:01](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=961s) 在科技行业，所有问题的标准答案通常都是‘视情况而定’（It depends） — Dave Salvator > [17:02](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=1022s) 表现规矩 — Dave Salvator > [17:33](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=1053s) 你给它多少线程，它就敢吞下多少线程 — Dave Salvator > [17:33](https://www.youtube.com/watch?v=5QLF2jgJKWU&t=1053s) 易并行 — Dave Salvator ## Mentioned Dave Salvator, NVIDIA, HPE, HPE GreenLake, NIM, Blackwell, GB200 NVL72, HGX B200, GB200 NVL4, B300, GB300, Hopper, CUDA-X, NVIDIA NGC, GitHub, Agent IQ, Dynamo, TensorRT-LLM, Meta Llama, MGX, Open Compute Project (OCP), Intel VTune ## Video description Dave Salvator, Director of Accelerated Computing Products, NVIDIA Abstracts This session will explore AI's business value. Attendees will learn to conduct ROI analysis for AI use cases, using NVIDIA's success stories. The session will compare the ROI of on-premises AI infrastructure versus cloud deployment, highlighting the benefits of on-prem solutions compared to the public cloud. Combining HPE’s enterprise expertise with NVIDIA’s AI stack, the session demonstrates how integrated solutions accelerate AI adoption and maximize ROI [Source](https://www.youtube.com/watch?v=5QLF2jgJKWU)

[2:35] Dave Salvator: 输入 Token — 输出 Token [2:35] Dave Salvator: 推理优先 [8:30] Dave Salvator: 完全相同的硬件上榨取并交付出更高的性能 [8:30] Dave Salvator: 相同的硬件投资能够交付更多的业务吞吐能力 [10:16] Dave Salvator: 无休止软件调优 [10:16] Dave Salvator: 免费 TCO 缩减红利 [14:19] Dave Salvator: 带动整个数据中心产业跟上每年一次的大步快跑节奏 [16:01] Dave Salvator: 在科技行业，所有问题的标准答案通常都是‘视情况而定’（It depends） [17:02] Dave Salvator: 表现规矩 [17:33] Dave Salvator: 你给它多少线程，它就敢吞下多少线程 [17:33] Dave Salvator: 易并行

[0:04] Dave Salvator: bold new frontier [16:00] Dave Salvator: It depends. [16:00] Dave Salvator: behaving itself [17:33] Dave Salvator: N-threaded [17:33] Dave Salvator: embarrassingly parallelizable

[0:04] Well, thanks for being here, guys. Good [0:05] to be with you. Um, HPE, one of our [0:08] partners, has asked me to give this [0:10] presentation. So, here I am. [0:13] So, uh, as we've been talking about here [0:15] at this conference, you know, AI has [0:16] been moving at some incredible speed in [0:18] terms of how the advances that we're [0:19] making. Um, you know, it was literally [0:21] just about a year or so ago where [0:23] generative AI was the thing that was [0:24] really, you know, kind of the real topic [0:27] of conversation. Now that's very quickly [0:29] moved to agentic and then as we move [0:30] forward a lot of that agentic AI um in [0:33] different shapes and forms will find its [0:34] way into physical AI you know autonomous [0:36] devices robotics autonomous vehicles [0:39] etc. Um and so we we continue to [0:41] innovate at just this amazing really [0:43] unprecedented pace both in terms of the [0:45] the capabilities of the platforms that [0:47] we're building and then the the AI [0:49] applications that are being built on [0:51] those platforms. So this is a really [0:53] kind of amazing time. I mean, I've been [0:55] in and around the tech industry for [0:56] about 30 years, and um this really [0:59] represents just a, you know, kind of a [1:02] bold new frontier. I mean, I know we in [1:03] the tech industry are a little bit [1:05] infamous for our hype, but um I think [1:07] we've all, you know, probably used AI [1:09] applications as end users as well, and [1:11] the capabilities they bring to us um and [1:14] will continue to bring to us are pretty [1:15] astounding. [1:17] That said, um there are a lot of [1:20] challenges around AI, right? And it's [1:21] not just on the training side. [1:23] Historically [1:24] uh it's been sort of the case that [1:25] things like pre-training building a [1:27] model essentially from the beginning uh [1:29] represented one of the big barriers and [1:30] challenges because of uh the cost not [1:33] only of owning the infrastructure but [1:34] the amount of time it takes to train [1:35] these models. It is it is a very heavy [1:38] heavy task. That said, we have a number [1:41] of community models now that are being [1:42] built by uh model makers, many of whom [1:44] have the resources to do that [1:46] pre-training to build these models to [1:48] kind of bring innovation to the [1:49] community and then they release those [1:51] models to the community which can then [1:52] be um basically modified in in really [1:55] almost any way you want to in terms of [1:57] applying your own data sets uh doing [1:59] different types of optimization [2:00] techniques and what have you to kind of [2:02] spur that next round of innovation. And [2:04] so one of the things that we're doing at [2:06] NVIDIA is making sure that we have day [2:08] zero support for those community models. [2:10] Not only in terms of all the [2:11] optimizations to make them performant [2:13] because we work with all these model [2:14] makers, but as well we also put them [2:16] into what's called our NIM framework or [2:18] NVIDIA inference microservices. We just [2:20] call them NIM now. Um but um and the the [2:23] idea there is that you have a [2:24] containerized um and security scanned [2:27] containerized service that brings a [2:29] highly optimized version of that model [2:30] to you that's easily deployed on any [2:32] platform whether that's on prem or in [2:33] the cloud. [2:35] But you know the thing of it is is as we [2:38] see the models grow the other thing of [2:39] course that's happening now is with [2:41] aentic AI is that we have this massive [2:43] increase in compute requirement right in [2:45] the form of reasoning is what we're [2:47] calling it. used to be with generative [2:48] AI, you had your input tokens, your [2:50] output tokens, and you know, it was sort [2:51] of a question and answer kind of a [2:52] model. Uh, and still is. But that said, [2:55] with reasoning, you've got this big [2:57] giant chunk in the middle called [2:58] reasoning, which can generate up to a [3:00] 100x more tokens that you're going to [3:02] have to work with to get to that more [3:04] sophisticated answer to that more [3:05] complicated question. Because a lot of [3:07] times it isn't even question answer [3:09] anymore. It's build me a market report [3:11] or a market analysis report about this [3:13] market based on these criteria. And so [3:16] suddenly, you know, the task is much [3:17] more complicated. And so, you know, the [3:19] amount of compute required to bring that [3:21] to bear in a reasonable amount of time [3:23] is really significant. [3:27] And so those challenges then extend into [3:29] a number of other areas. And again, [3:31] apologies, I'm sort of having to do the [3:32] weatherman thing here and look at the [3:33] screen because I don't have a monitor. [3:36] Um, but you know, there's a number of [3:37] challenges here as you can see. you [3:39] know, speed, interoperability, [3:41] um, you know, that the ability to use [3:42] your existing data sets. That's another [3:44] big one. You know, there's a technique [3:46] called RAG or retrieval augmented [3:48] generation where you can use existing [3:49] data from your own databases as a way to [3:52] make the model smarter and more aware of [3:54] your particular informationational [3:55] domain, right? And what's great about [3:57] that is as the database is updated and [3:59] the app can continue to query that [4:01] database as part of how it formulates an [4:03] answer, it gives you that ability to [4:05] extend the life of your AI powered [4:06] application without having to go back [4:08] and retrain it. And so these you know [4:10] there's any number of techniques being [4:12] developed there to help sort of reach u [4:14] you know the efficiency and sort of the [4:16] cost effectiveness we need to be able to [4:17] scale these things and bring them to the [4:19] market in a way that is cost-ffective [4:21] because cost is another piece of this [4:22] equation. And the other thing to bear in [4:24] mind of course with Agentic is that [4:26] while we used to talk about the really [4:27] the heavy lifting of AI was on the [4:29] training side as you're deploying [4:31] Agentic AI applications what we're [4:33] seeing is that there's a fair amount of [4:34] heavy lift now happening on the [4:36] inference side as well as an example our [4:38] Blackwell platform even our GB200 which [4:40] is sort of our current flagship [4:41] platform. Uh we're seeing many uh [4:44] customers who are taking that and [4:45] deploying it as an inference first [4:47] platform. You know, historically our our [4:49] biggest newest platform was typically [4:51] first used for training and then sort of [4:52] eventually used for inference. We're now [4:55] seeing in some cases that that's [4:56] actually flipping and it's being used [4:57] for inference first because of those [4:59] demands. [5:02] So, you know, I touched on the 100x [5:04] thing and so this is just sort of a [5:06] simple example to sort of give you an [5:07] idea between this notion of question and [5:09] answer which is more kind of what we're [5:11] doing with LLMs and kind of in the [5:12] generative space uh versus reasoning [5:15] where you actually have to take a [5:16] problem that actually involves literal [5:19] reasoning and figure out a solution. And [5:21] again, one of the nice things with this, [5:22] of course, as you've probably seen in a [5:24] number of the popular applications is [5:26] you'll get this window into sort of the [5:28] chain of thought is how is the model [5:29] sort of thinking about this, breaking [5:31] the problem into smaller component [5:33] problems that it can solve, bring them [5:34] together to ultimately bring you a [5:37] solution, a report, an ultimate answer [5:39] to the thing you've asked it to do, [5:41] right? [5:42] But the but the tricky bit here is on [5:44] the far right hand side which is really [5:46] striking a balance between keeping your [5:48] infrastructure sufficiently lit up which [5:51] is to say keep your utilization [5:53] sufficiently high because that's [5:54] something that every infra manager on [5:56] the planet will tell you is that if [5:58] we're not keeping our our utilization [6:00] sufficiently high we are not getting [6:01] sufficient value from this platform and [6:03] that's something that they're often [6:04] measured on. And so their goal is, you [6:05] know, the number will vary depending on [6:07] who you talk to, but typically it's on [6:09] the order of 60 to 80% utilized. They [6:11] always want to allow a little bit of [6:12] headroom for potential sort of transient [6:14] spikes in in compute demand, but [6:17] ultimately they want to keep that system [6:18] pretty lit up versus the ability to [6:21] deliver great experiences to every user. [6:23] And those two things kind of tend to [6:25] pull at each other. And you usually are [6:26] kind of having to figure out how you're [6:28] going to trade one for the other to [6:29] arrive at the right balance for your [6:31] particular company, your particular [6:32] application. [6:35] So to help solve for that um and these [6:38] are basically our current platforms. [6:39] Blackwell Ultra is coming into market [6:41] this year. Um and you'll start seeing it [6:44] very very soon from from a number of our [6:46] customers including HPE. And then uh [6:49] what we have today is the three [6:50] platforms you see here around GB200 or [6:52] Grace Blackwell uh NVL72 which brings 72 [6:56] of our Blackwell GPUs together in a [6:58] single NVLink domain which can [7:00] essentially act as one giant GPU, right? [7:03] And then we have what is can be thought [7:04] of as to some degree a slightly more [7:06] traditional HGXB200 which is a BY8 [7:09] configuration. It can go into an air [7:10] cooled server. And then we also have the [7:12] GB200 NVL4. Um this is more targeted [7:16] probably for scientific simulation in [7:18] the HPC uh segment where they're doing [7:20] more um you know things like uh [7:22] computational fluid dynamics, molecular [7:24] dynamics and things of that nature. Um, [7:27] and so they're they have sort of [7:28] different compute requirements and the [7:30] NVL4 a lot of times is a better better [7:32] fit for those particular uh types of [7:34] customers. [7:35] So what I'm showing you here are what [7:37] are called paro curves. And this sort of [7:39] gets back to that tug-of-war dynamic I [7:41] was describing just a moment ago where [7:44] you're ultimately trying to get the most [7:46] performance you can out of every [7:48] megawatt or out of every sort of you [7:50] want to think of it as a unit of compute [7:51] you have in your data center versus [7:53] delivering responsiveness and great [7:55] experiences to your end users. Now what [7:58] we're plotting here are basically the [7:59] ideal configurations at each one of [8:01] those sort of stops if you want to think [8:02] of them that way as you strike a balance [8:04] between those two things. So if you look [8:06] at where Hopper is in terms of the PTO [8:08] curve we could generate with it versus [8:10] the PTO curve we can now generate with [8:12] this is sort of a curve for GD300 NVL72 [8:15] which as I mentioned will be in market a [8:16] little later this year. Um we have an [8:19] ability to deliver up to 50x more sort [8:21] of capacity per megawatt but more [8:24] importantly it gives you a better set of [8:26] choices on that PTO curve as to where [8:28] you want to land your application. Uh [8:30] and this is actually a really important [8:32] decision that every organization has to [8:34] figure out about you know how to deploy [8:36] cost- effectively right and of course [8:38] this will change over time and one of [8:40] the great things about the NVIDIA [8:41] platform and of course HPE benefits from [8:43] this is that we are publishing our [8:45] containerized software on about a [8:47] monthly cadence and with every month [8:49] we're improving our performance we are [8:51] of course chasing down and squishing [8:53] bugs uh but but the main thing is is [8:55] that we're delivering more performance [8:56] on the same hardware which as you think [8:58] about more performance in this context [9:00] equals lower TCO. Essentially, that's a [9:02] way to think about it. I mean, more [9:04] performance was great. I love more [9:05] performance for its own sake. But that [9:07] said, when you're trying to translate [9:08] that into business value, it really [9:10] comes down to you're delivering more [9:12] capability on the same investment you've [9:14] already made in such a way that you can [9:16] reduce TCO of any given application that [9:18] you've deployed. [9:21] That said, we've talked a lot about [9:23] hardware, but the hardware by itself [9:25] solves no problems. What you have to [9:26] have is a robust platform that goes all [9:28] the way up the stack, which is what we [9:30] have and have developed and brought to [9:32] HPE. And of course, they have their own [9:34] stack, which I'll show you here in just [9:35] a moment. But our CUDA X libraries, [9:37] which now number in the hundreds, help [9:39] solve a wide diversity of problems, not [9:41] only for AI, but also for scientific [9:42] simulation, as I touched on a moment [9:44] ago. Um, so you know, these are of [9:47] course all available uh and uh can be [9:49] get gotten from uh either our NGC [9:52] website or from uh some of these are [9:54] even available on GitHub. now. And so, [9:56] as you can see, you know, they cover a [9:57] diverse set of potential things you may [10:00] be working on. And so, um, you know, and [10:02] if there isn't a library there that that [10:04] will solve for your particular problem, [10:06] odds are we'll be making it pretty soon. [10:07] We are constantly looking at what other [10:09] libraries can we bring to developers to [10:11] help them build the applications and get [10:12] the performance that they need to build [10:14] those applications. [10:16] So as we think about that stack then in [10:18] the context of HPE what you can kind of [10:21] see here is how our stack ultimately [10:23] then is going to reside on those HPE [10:25] servers that you may be uh deploying on [10:27] prem right so those HP servers that [10:29] they're building whether that's based on [10:31] uh B200 GB200 what have you or going [10:34] forward B300 and GB300 [10:37] um you know are going to be able to take [10:38] advantage of all of this platform [10:39] goodness that I've been talking about [10:41] and so that's a really important thing [10:42] to be considering as you think about [10:44] making investments in platform platforms [10:46] because you're not just making an [10:47] investment with an HPE platform, you're [10:49] making an investment in the Nvidia [10:50] platform as well. And as I touched on, [10:52] as we grow our performance over time, [10:55] that ultimately translates into free [10:58] reduction in TCO for you. I mean, we [11:00] have seen often times that in any given [11:03] generation of GPU, we can generally get [11:06] about two to sometimes even as much as [11:07] 2.5x more performance out of the same [11:10] hardware through our relentless software [11:12] optimizations that we're doing on those [11:13] monthly containers. So again, that [11:16] translates into ongoing value for your [11:18] organization when you make the [11:19] investment in the NVIDIA platform and in [11:21] this case making that with uh with HPE. [11:25] So speaking of HPE's platform, this just [11:27] sort of gives you a picture of of kind [11:28] of the additional services that they [11:30] bring in as well. And a lot of this can [11:32] also be landed on their HPE Green Lake [11:34] cloud which you see listed here. But [11:35] this just gives you a sense of the [11:37] additional pieces that HPE is bringing [11:39] to the uh to the overall sort of value [11:41] that they want to offer to you with the [11:43] NVIDIA platform. So in addition to all [11:44] that Nvidia platform that I've been [11:46] telling you about, this is additional [11:48] aspects of the platform that come on the [11:50] HPE servers. [11:54] So the other thing I want to talk about [11:56] now is additional ways that we help [11:58] realize more performance because again [12:00] as we've talked about, you know, more [12:02] performance lets you do more things. It [12:04] either lets you deliver your existing [12:05] applications with better TCO or it lets [12:08] you build that next new class of [12:09] applications that have that much more [12:10] demand for compute and performance. [12:13] Right? So just three things I want to [12:15] touch on briefly. Um you know we talk [12:17] about agents and and you know how NIM [12:19] can help accelerate those. We also have [12:21] our own community models that we've [12:22] built. Um these are basically based on [12:24] the llama uh community model from meta [12:27] and we have three different sort of [12:28] sizes of those you see listed here right [12:30] and these are available as well uh [12:32] basically you know for free to the [12:34] community and then we have our have our [12:35] agent IQ which also helps do profiling [12:38] and optimization to help you get your [12:40] application dialed in in such a way that [12:41] you will get optimal performance from [12:43] it. Um again somewhat relating to where [12:45] you want to land on that PTO curve I [12:47] showed a minute ago. [12:50] And of course, another uh important [12:52] piece we've added to the equation is [12:53] what's called Dynamo. This is our new [12:55] inference serving software. We [12:56] introduced this a little earlier this [12:57] year. Uh it is expressly built to help [13:00] take advantage and accelerate uh both uh [13:02] generative and agentic AI applications. [13:05] Uh it has a quite uh quite an array of [13:07] features. The main one I want to talk [13:08] about for uh for this talk because it's [13:10] a very short talk is disagregated [13:12] serving which some of you may be [13:14] familiar with. It basically is the [13:15] process of taking the prefill portion of [13:17] inference and the decode portion and as [13:19] the name implies separating them and [13:21] actually landing them on different GPUs. [13:23] The purpose being is that each has sort [13:25] of a different performance sensitivity. [13:27] And by basically separating them out and [13:29] then running them on different GPUs, you [13:31] actually can get pretty significant [13:33] performance bumps on the same platform. [13:35] we've actually seen already, you know, [13:37] and again giving you some ballpark [13:39] numbers, 30 even upwards of 40% uh more [13:42] performance on the same hardware by [13:43] going to disagregated serving on LLMs. [13:46] And so this is a really interesting area [13:48] to kind of look into. And then another [13:50] area which um is more covered by our [13:53] Tensor RT LLM software, it's not on the [13:55] slide, but I'll give it a quick mention [13:56] is what's called a speculative decoding [13:58] as well, which again, if you're [13:59] developers, you may already be familiar [14:01] with this technique. involves using [14:02] what's called a draft model to help kind [14:04] of basically do some of this as the name [14:06] implies speculative decoding to uh to [14:09] help again sort of get to the same [14:11] answers with less compute. So again [14:13] you're getting more performance out of [14:14] your same platform. [14:18] Let's see. [14:20] Okay, there we go. So the other thing [14:21] that we're doing and again HPE is very [14:23] much on this ride with us is we are on a [14:26] one-year cadence delivering new [14:27] architectures to the industry. And it's [14:29] not just the GPUs. I mean, as you can [14:31] see from the slide, it's GPUs, CPUs, [14:33] networking. It's also rack designs. [14:35] That's actually an incredibly important [14:37] thing. Most people don't think of racks [14:38] as being, you know, particularly [14:41] exciting. But without the rack design, [14:43] you can't deploy these systems, [14:45] particularly the the the Grace Blackwell [14:46] systems. The rack design is incredibly [14:48] important because these are liquid [14:49] cooled systems. And we actually have [14:51] reference designs called uh MGX. And uh [14:54] we have even one now for Grace Blackwell [14:56] 200 going forward Grace Blackwell 300 [14:59] which will uh basically allow a server [15:02] maker to get to market faster because [15:03] rather than having to go back and [15:04] basically you know build and engineer [15:06] their own chassis design which can take [15:08] anywhere from 6 to 12 months they have a [15:10] readym made design that we've already [15:11] kind of put through the test done a lot [15:13] of the optim optimization work on that [15:15] they can basically take as sort of a [15:17] blueprint to go build their servers and [15:18] get to market faster right we've even [15:21] donated a number of these MGX designs [15:23] the one for GB200 in particular to uh [15:26] the open compute project or OCP and it [15:28] is now one of their reference designs as [15:29] well. So, it's another way that we're [15:32] helping our partners sort of accelerate [15:34] their cadence because, you know, we're [15:35] moving at this year-long, you know, this [15:37] basically annual cadence with our GPU [15:39] architectures. And as many of you [15:40] probably know, you know, the data center [15:42] world is not really used to moving at [15:44] that speed. So, we're uh we're sort of [15:46] trying to encourage them to uh to move [15:48] along a little faster with us uh on this [15:50] ride. And so, I believe that is my last [15:53] slide. I'm not sure how I'm doing on [15:55] time. I got Oh, I got five minutes left. [15:57] Wow. Okay. Uh I'm I'm happily surprised. [16:00] So that's my slides. Um I do thank you [16:02] for attending. I'm I'm available to [16:04] answer a few questions since we have [16:05] five minutes here. Uh I don't know if we [16:07] have a have a mic with runners here. If [16:09] not, just Yeah, please go ahead. That's [16:11] a great question. I my answer would be u [16:14] a little simplistic, but I see if this [16:16] helps answer the question. The the [16:19] answer is it depends, which is the [16:20] answer to every tech question. But no, [16:22] why it depends is if the task is [16:24] parallelizable, which is to say a task [16:25] that can be broken into component tasks [16:27] and run across many processors, GPUs [16:29] will absolutely do it dramatically [16:31] faster by like probably X factors. If [16:33] it's a highly serialized process, that's [16:35] where CPUs are really usually a better [16:37] answer because they've got, you know, [16:39] fewer cores, but those cores generally [16:42] are uh going to run at slightly higher [16:43] clocks and are just they're very much [16:45] designed for those serialized tasks. But [16:47] to the extent it can be parallelized, [16:49] GPUs will actually be able to do that [16:50] much much better, much much faster. [16:53] Yeah, I mean again is if it can be [16:55] parallelized, it will run much better on [16:58] GPUs. So it, you know, I think you'll [16:59] see an evolution there. Um because a lot [17:01] of that will come because observability [17:02] is a super important thing as I'm sure [17:04] you're aware. So the ability to kind of [17:05] keep tabs on the model, you know, to do [17:07] things like guard railing and some of [17:08] the other techniques we do to make sure [17:09] that, you know, we're avoiding [17:10] hallucinations or at least keeping them [17:12] to a minimum, making sure that answers [17:14] are appropriate. try and avoid you know [17:16] if inapp inappropriate prompts are being [17:18] put into the model that the model kind [17:20] of knows okay I can't answer that [17:21] question u and then observing all that [17:23] and making sure that frankly the model [17:25] is I'll use quoty fingers behaving [17:27] itself right then yeah then I think GPUs [17:29] absolutely have a role to play there [17:33] okay I got two minutes left any other [17:35] questions I can answer for you while I'm [17:36] here huh that's a great question I can't [17:42] think of a specific example where we've [17:43] used AI to do that analysis analysis. [17:45] The analysis we generally do is more at [17:47] the what I'll call sort of traditional [17:49] performance characterization which is [17:51] looking at you know take the the first [17:52] real fundamental question is can [17:54] something be uh what's called nth [17:56] threaded which is to say you can spin up [17:58] as many threads as you have cores to run [18:00] that particular application that [18:02] particular algorithm and see performance [18:04] scale on it because not everything is [18:06] necessarily amenable to being threaded [18:07] and parallelized as you probably know [18:10] some things maybe go are grayed out to [18:12] maybe two maybe four threads and some [18:13] things will take every damn thread you [18:15] can give them, right? That latter [18:17] category about give me all the threads [18:19] you got, it's it's kind of a no-brainer [18:21] that those are going to run faster on a [18:23] GPU. Um, [18:27] so let me see if I understand the [18:29] question. So that have we used it to [18:31] sort of do task analysis of like [18:32] specific inner tasks of a larger [18:35] application? Is that what you're asking [18:36] about? You certainly could. I'm not [18:38] aware that we've specifically done that [18:40] kind of characterization internally. We [18:42] certainly, you know, know what kinds of [18:43] applications are what we like to call [18:45] embarrassingly parallelizable. AI being [18:47] one of them, 3D graphics being another, [18:49] which is what GPUs are really good at. [18:51] Um, but, you know, I'd say, you know, [18:53] there's certainly all kinds of [18:54] applications that you can like if you [18:55] have a traditional CPU application. I'll [18:57] I'll just use x86 as an example. Um, [18:59] there's a very good tool that Intel has [19:01] made for years and years called Vtune, [19:02] which you may be familiar with, right? [19:04] So, Vtune is often used to do that [19:06] performance characterization to [19:07] understand where is an app spending its [19:08] time. You know, is it bound in memory? [19:10] Is it is it getting stuck in the caches? [19:12] Um or is it computebound, you know, or [19:14] are you basically having too much memory [19:16] pressure? So, you're kind of starving [19:17] your compute. Vtune can help you kind of [19:20] kind of basically root cause all that [19:21] and solve for where is your application [19:23] basically getting, you know, stuck right [19:26] now. If what you find is it's an [19:27] endthreaded application and it's [19:29] primarily computebound because they [19:30] figured out how to keep it fed. In other [19:32] words, you know, you figured out your [19:33] caching, you figured out how to stream [19:35] data into those compute engines so that [19:37] they're not waiting. there's a really [19:39] good chance that when you walk that over [19:40] a GPU, you're going to see a significant [19:42] bump in performance. But again, I'm sort [19:45] of repeating myself a little bit, but if [19:46] if the app is only really to scale out [19:48] to a couple of threads, odds are it's [19:51] probably best left on CPU and other more [19:53] parallelizable tasks are then put over [19:54] onto the GPU. And I think with that, I'm [19:57] out of time. So, I'm going to hand it [19:59] back to our host here. And I want to say [20:00] thank you for attending and enjoy the [20:02] rest of the conference.