Nvidia, HPE Pitch On-Prem Hardware to Slash Surging AI Agent Costs
Ai4
Autonomous AI agents consume up to 100 times more tokens per action than chatbots, forcing enterprises to deploy on-premises AI factories to eliminate variable inference fees.
## Background
Published on YouTube by Ai4 on August 15, 2026, this session features Steve Stein from NVIDIA Product Marketing exploring how HPE and NVIDIA are co-architecting enterprise AI factory architectures. According to the description, the presentation details how accelerated compute, networking, storage, NVIDIA AI Enterprise software, and HPE Private Cloud AI enable organizations to make cost per token, data gravity, and governance foundational platform design principles for production-scale agentic AI.
## TL;DR
The transition from standard conversational chatbots to autonomous agentic loops drives a massive inflection in token consumption, escalating per-task costs by up to 100x and threatening enterprise IT budgets. To solve this tokenomics crisis, NVIDIA and HPE propose on-premises AI factory architectures running local open-weight models like the Nemotron family to eliminate variable per-token API expenses. The joint strategy combines specialized hardware—including high-throughput Vera Rubin GPUs and dedicated monolithic NVIDIA Vera tool-execution CPUs in HPE ProLiant DL394 servers—with dynamic routing software like NeMo Relay and SwitchYard alongside HPE's enterprise-grade observability and security frameworks.
## Atmosphere & style
The presentation carries the polished, authoritative tempo of an enterprise technology briefing, balancing high-level business urgency with deep systems-architecture design. Steve Stein speaks in a clear, pragmatic, and approachable tone, incorporating moments of self-deprecating humor while explaining data center economics and silicon-level bottlenecks. The delivery evokes an industrialized, forward-looking mindset that treats autonomous AI scaling not as an open-ended software experiment, but as a disciplined manufacturing process governed by power efficiency, memory throughput, and infrastructure economics.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=4s) Gartner projects that by 2029, 70% of large enterprises lacking dedicated AI factory infrastructure will cease to exist.
- [01:29](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=89s) GenAI 2.0 marks a transition from human-driven chatbot queries to autonomous machine harnesses that pair LLM brains with real-world tool execution.
- [04:14](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=254s) Autonomous agentic tasks consume roughly 47,000 tokens ($4) per action compared to 400 tokens ($0.04) for basic chats, multiplying operational costs by 30x to 100x.
- [06:40](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=400s) Hosting open-source and open-weight models on-premises eliminates per-token API billing, reducing variable token costs to zero.
- [06:40](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=400s) NVIDIA's upcoming Vera Rubin architecture is projected to deliver up to a 10x token-per-watt efficiency gain over Blackwell GPUs.
- [09:46](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=586s) Agentic tool calling creates a host CPU bottleneck, leading NVIDIA to engineer the monolithic Vera CPU specifically for fast OS command and tool execution.
- [09:46](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=586s) HPE is bringing Vera CPU architecture to enterprise data centers via the ProLiant DL394 platform to handle dedicated agent tool execution.
- [12:55](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=775s) Software tools like NeMo Relay dynamically route intermediate agentic sub-tasks to right-sized open models (e.g., 30B or 120B) rather than expensive frontier LLMs.
- [15:24](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=924s) SwitchYard evaluates prompt complexity to intelligently steer tasks between external frontier APIs and locally hosted open Nemotron models.
- [18:13](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1093s) Token generation efficiency is outpacing raw compute growth, exemplified by NVIDIA Blackwell delivering a 50x token efficiency gain over Hopper despite a 2x compute increase.
- [19:00](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1140s) The AI factory paradigm models enterprise infrastructure as a five-layer stack that takes in electricity and proprietary data to produce cost-efficient tokens and intelligence.
- [20:05](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1205s) HPE partners with NVIDIA to deliver turnkey Private Cloud AI infrastructure equipped with agent observability, governance, and data security controls.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=4s) The Tokenomics Crisis and the AI Factory Imperative
Tokenomics has rapidly become one of the most pressing challenges across enterprise IT, steadily consuming budgets as organizations scale up their use of large language models. While previous discussions highlighted the sheer difficulty of financing token consumption, NVIDIA and Hewlett Packard Enterprise (HPE) are framing this not merely as an access problem, but as a systems-level architecture challenge requiring purpose-built on-premises infrastructure. Organizations that fail to rethink their computing backbone risk severe consequences as generative and autonomous AI evolve.
Citing industry projections, Gartner estimates that by 2029, 70% of large enterprises that do not build or use dedicated "AI factories" will cease to exist. In the emerging era of generative and agentic AI, these specialized AI factories serve as the foundational engines required to manufacture intelligence and serve tokens with maximum efficiency, preventing enterprise IT spending from collapsing under runaway operational costs.
- **Enterprise AI Adoption Warning:** Gartner projects that by 2029, 70% of large enterprises lacking AI factory capabilities will cease to exist.
- **Strategic Partnership:** NVIDIA and HPE are co-architecting full-stack AI solutions to make enterprise intelligence generation economically sustainable.
```mermaid
flowchart LR
accTitle: The AI Factory Paradigm
subgraph Input["Inputs"]
P["Power & Data"]
end
subgraph Factory["Enterprise AI Factory"]
H["HPE & NVIDIA Infrastructure"]
S["Accelerated Compute & Software"]
end
subgraph Output["Outputs"]
T["Cost-Efficient Tokens"]
I["Autonomous Intelligence"]
end
Input --> Factory --> Output
```
---
[01:29](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=89s) Defining Agents: Brains, Harnesses, and "GenAI 2.0"
To understand why token consumption is accelerating exponentially, one must examine how autonomous agents actually function. An LLM acts essentially as a brain: it can generate vast amounts of insight when interacting with a human via tools like ChatGPT or Claude, but on its own, a brain cannot take real-world action. An agent pairs that brain with a "body," implemented as a software execution harness capable of running code, issuing API commands, editing slides, and triggering workflows. Once the harness executes an action, it feeds the output back into the LLM for evaluation, initiating a continuous "agentic loop."
```mermaid
flowchart TD
accTitle: The Agentic Loop
subgraph AgenticLoop["The Autonomous Agentic Loop"]
LLM["LLM (The Brain)<br/>Analyzes state & generates instructions"]
Harness["Software Harness (The Body)<br/>Executes code, tools & actions"]
LLM -->|"Commands & code actions"| Harness
Harness -->|"Execution results & environmental feedback"| LLM
end
```
This dynamic marks the fundamental transition into what can be called "GenAI 2.0." Between 2022 and 2024, token consumption was driven almost entirely by human users querying chatbots. In the agentic era, machines and execution harnesses are consuming tokens autonomously. As agents leverage external tools, build memory systems across sessions, and ingest skills, enterprise productivity rises dramatically—but so do security attack surfaces, vulnerabilities, and computational expenses.
---
[04:14](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=254s) The Tokenomics Inflection: Chatbots vs. Autonomous Agents
Enterprises are facing a steep agentic inferencing inflection point. When an employee interacts with a standard conversational chatbot, a typical prompt-and-response turn consumes approximately 400 tokens, costing roughly 4 cents. Autonomous agents, however, operate at a completely different order of magnitude because each discrete task requires ingesting vast context: long-term memory, modular skill definitions, and foundational persona guidelines ("soul files") stored as dense markdown text files.
Consequently, a single agentic task consumes an average of 47,000 tokens—translating to nearly $4 per action. Gartner estimates that individual agentic tasks can cost 30 times more per action than standard generative AI chats, while internal NVIDIA data indicates that complex workloads can reach up to 100 times the cost. Because agentic loops execute repeatedly and can spin into infinite automated loops, organizations relying on public cloud frontier APIs have reported exhausting entire annual token budgets within months.
[06:40](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=400s) Eliminating Variable Token Costs with On-Premises GPU Architectures
The architectural solution to runaway token expenditure lies in on-premises infrastructure running local, open-source, open-weight models. While organizations must account for upfront hardware capital acquisition or financing, running open-weight models on private systems reduces the "variable cost per task" to zero. NVIDIA is addressing this challenge across four pillars: hyper-efficient GPUs, purpose-built CPUs, specialized agentic software toolkits, and optimized open-weight model runtimes.
On the GPU hardware front, token generation efficiency is accelerating at an unprecedented pace. The NVIDIA Hopper architecture established the generative AI industry, yet the Blackwell generation substantially outpaced it. The upcoming Vera Rubin architecture advances this trajectory even further, delivering up to a 10x improvement in tokens generated per watt compared to Blackwell. NVIDIA evaluates AI factory throughput by graphing tokens per second per user (measuring latency and interactivity) against tokens per second per megawatt (measuring power efficiency), where the area under the curve represents total AI factory output.
- **Zero Variable Marginal Cost:** Hosting open-source and open-weight models on-prem eliminates per-token API billing.
- **Generational Leaps:** Vera Rubin achieves up to a 10x token-per-watt efficiency gain over the already high-performance Blackwell architecture.
---
[09:46](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=586s) Breaking the Execution Bottleneck: The NVIDIA Vera CPU and HPE ProLiant DL394
As GPU token generation has surged, frontier AI labs have uncovered a severe architectural bottleneck on the CPU side of the agentic loop. Once GPUs generate instructions, those commands are offloaded to host CPUs to execute shell commands, orchestrate APIs, and manage software tools. Traditional x86 processors were never designed for this specific workload, creating a severe processing backlog. While current data center deployments operate at roughly a 4:1 GPU-to-CPU ratio—with some predicting a potential shift toward 7:1—simply adding more x86 CPUs is an inefficient workaround.
```mermaid
flowchart LR
accTitle: GPU and CPU Division of Labor in Agentic AI
subgraph GPU_Node["GPU Acceleration Layer"]
VR["NVIDIA Vera Rubin GPUs"]
Gen["High-Efficiency Token Generation<br/>(Open-Weight LLM Inference)"]
VR --- Gen
end
subgraph CPU_Node["Standalone CPU Execution Layer"]
VC["NVIDIA Vera CPU (ProLiant DL394)"]
Exec["Fast Monolithic Tool Calling<br/>(Code Execution & OS Actions)"]
VC --- Exec
end
GPU_Node -->|"Tokens & Action Commands"| CPU_Node
CPU_Node -->|"Tool Results & State Data"| GPU_Node
```
To resolve this imbalance, NVIDIA developed the NVIDIA Vera CPU—a standalone processor engineered specifically to accelerate agentic tool execution rather than host model inferencing. Inferencing remains strictly on GPU silicon where it is most efficient. NVIDIA Vera delivers nearly double the tool-execution efficiency of standard x86 CPUs. Unlike multi-chiplet x86 designs that suffer from erratic memory access latencies across stacked dies, Vera is constructed on a single monolithic die, ensuring predictable, ultra-fast memory throughput. NVIDIA is partnering with HPE to bring this architecture to enterprise data centers later this year via the HPE ProLiant DL394 server platform.
- **Workload Specialization:** GPUs handle pure model inference; Vera CPUs exclusively execute the operating system commands, harnesses, and tools.
- **Architectural Advantage:** Vera's monolithic single-die design avoids the memory access latency bottlenecks common to stacked x86 chiplet architectures.
- **Hardware Platform:** HPE ProLiant DL394 systems integrate Vera CPUs to serve as dedicated agentic tool-execution nodes.
---
[12:55](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=775s) Optimizing Agentic Throughput with NVIDIA Agent Toolkit and NeMo
Beyond hardware innovation, the software layer represents NVIDIA's core differentiator in maximizing end-to-end agentic throughput. The NVIDIA Agent Toolkit addresses runtime execution, model optimization, and the verification of safe skills and tools used by agent harnesses. Because agents chain together diverse sub-tasks, optimizing the entire workflow requires intelligent orchestration rather than routing every single operation to an expensive frontier model.
Within the toolkit, software components such as NeMo Relay and NeMo Switchyard dynamically allocate compute resources across model tiers. Not every agentic step demands a massive, frontier-class model with hundreds of billions or trillions of parameters; simpler intermediate tasks, parsing, or data formatting can be routed to nimble 30-billion or 120-billion parameter models. By dynamically offloading less complex tasks to faster, right-sized open models, NeMo Relay substantially drives down cumulative agentic pipeline costs while improving system uptime and total processing velocity.
- **Dynamic Task Routing:** NeMo Relay routes sub-tasks to right-sized models (e.g., 30B or 120B parameter models) instead of defaulting every sub-action to frontier-scale LLMs.
- **Comprehensive Agent Toolkit:** Covers model runtime acceleration, harness execution optimization, and safety verification for tools and skill libraries.
[15:24](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=924s) Intelligent Prompt Routing with SwitchYard and the Nemotron Family
When managing agentic workflows at scale, intelligent orchestration becomes essential for balancing performance and operational expenses. Tools like SwitchYard address this by inspecting complex incoming prompts to determine the optimal execution path. SwitchYard evaluates whether a query truly demands an expensive external frontier model—such as a cloud-hosted flagship LLM—or whether it can be routed directly to on-premises compute running local open models like Nemotron 120B. This intelligent routing optimizes end-to-end agentic throughput across the enterprise.
```mermaid
flowchart TD
accTitle: Intelligent Prompt Routing via SwitchYard
subgraph Ingestion ["Ingestion & Routing"]
A["Complex Prompt / Agent Task"] --> B["SwitchYard Router"]
end
subgraph Execution ["Execution Target"]
B -->|"High-complexity frontier task"| C["External Frontier Model\n(Cloud API / Flagship LLM)"]
B -->|"Domain / high-throughput task"| D["On-Premises Infrastructure\n(NVIDIA Nemotron 30B / 120B / 550B)"]
end
subgraph Output ["Outcome"]
C --> E["Enterprise Agent Response"]
D --> E
end
```
To support local on-prem execution, NVIDIA develops Nemotron, an open-source, open-weights model family trained and released entirely by NVIDIA. Rather than restricting development to a single monolithic architecture, NVIDIA provides a tiered portfolio tailored for diverse workloads:
- **Nemotron Nano (30B):** Lightweight model suitable for low-latency agent tasks and edge deployments.
- **Nemotron Super (120B):** Mid-tier model balancing heavy reasoning capacity with fast token generation.
- **Nemotron Ultra (550B):** Frontier-class open model designed for massive multi-agent coordination and deep reasoning.
- **Specialized Modalities:** Dedicated open models for vision, information retrieval, safety guardrails, and speech processing.
Steve Stein remarks humorously on his own experience testing these models locally: *"I interact with these on my own personal machine, and I love them. I'm a little biased, you know, I work for NVIDIA, but I really enjoy working with these. And if I can figure it out, by the way, I think you can figure it out. I'm not exactly any top-tier engineer coming out of NVIDIA. I'm just a little lowly product marketer that's trying to get some of this stuff done."* Across standard industry benchmarks, the Nemotron family consistently matches or leads competing open-source models.
---
[17:35](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1055s) Taming Enterprise Tokenomics with Local LLMs and Zero Variable Cost
The core economic hurdle of autonomous agentic systems is the "tokenomics challenge." Chained model invocations, recursive loops, and continuous retrieval can make proprietary cloud API calls up to 100 times more expensive than hosting a local LLM. The enterprise remedy is architecting on-premises AI systems powered by NVIDIA and HPE that leverage open models to fundamentally shift the cost curve.
Because open models carry no per-token API licensing fee, the marginal variable cost for every generated token on-premises drops to essentially zero. By pairing owned infrastructure with high-performance local open-weight models, enterprises transform unpredictable, escalating API bills into predictable, depreciable capital infrastructure.
---
[18:13](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1093s) Generational Breakthroughs: Why Token Efficiency Outpaces Raw Compute
Generational hardware upgrades in AI are no longer measured purely in raw floating-point operations; they are defined by token generation throughput. Comparing NVIDIA Blackwell to the prior-generation Hopper architecture illustrates this paradigm shift:
This massive 50x gain in token generation efficiency stems from a combination of underlying silicon architectural redesigns, specialized inference engines, memory bandwidth enhancements, and full-stack software optimizations. For high-volume agent swarms generating millions of intermediate tokens, running on purpose-built NVIDIA GPU platforms delivers orders-of-magnitude lower cost-per-token than legacy compute footprints.
---
[19:00](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1140s) The Five-Layer AI Factory and the NVIDIA Hardware Roadmap
NVIDIA views modern enterprise infrastructure as an "AI factory" modeled around a cohesive "five-layer cake" that spans facilities, compute hardware, networking, platform acceleration, and software runtimes. In this paradigm, raw inputs—electricity and enterprise data—flow in at the bottom, and intelligence in the form of processed tokens flows out at the top.
```mermaid
flowchart TD
accTitle: The AI Factory Processing Model
subgraph Inputs ["Factory Inputs"]
InPower["Electricity & Data Center Power"]
InData["Enterprise Proprietary Data & Gravity"]
end
subgraph Factory ["Five-Layer AI Factory (NVIDIA & HPE)"]
F1["Accelerated Infrastructure & Networking"]
F2["NVIDIA AI Enterprise Software Runtime"]
F3["Governed Agent Platforms & Fine-Tuned Models"]
end
subgraph Outputs ["Factory Output"]
OutTokens["Actionable Tokens / Machine Intelligence"]
end
NVIDIA maintains a predictable, transparent hardware cadence to sustain this scaling trajectory:
- **GPU Progression:** Blackwell transitioning into the upcoming Rubin GPU generation.
- **CPU Progression:** NVIDIA Grace CPUs evolving into the Vera CPU architecture.
- **Full-Stack Turnkey Stacks:** Deep joint engineering with HPE to package networking, compute, and management into turnkey on-prem deployments.
---
[20:05](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1205s) Partnering with HPE for Enterprise Observability, Security, and Scale
While NVIDIA provides the underlying silicon, networking, and core AI models, Hewlett Packard Enterprise (HPE) shoulders the operational burden of delivering and maintaining enterprise-grade on-prem AI deployments. HPE holds deep enterprise credibility, maintaining 18 MLPerf benchmark records and consistently placing systems at the very top of the TOP500 supercomputer rankings.
- **Agent Observability:** Tracking, tracing, and logging runtime activities across autonomous agent swarms deployed within the corporate boundary.
- **Enterprise Security & Policy Controls:** Ensuring agents are safely spawned, bound to internal governance boundaries, and prevented from leaking proprietary data.
- **Integrated Co-Engineering:** Deep architectural integration across HPE Private Cloud AI and NVIDIA AI Enterprise to streamline the transition from pilot agent demos to production-scale operations.
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=4s) By 2029, 70% of large enterprises that do not build or use dedicated 'AI factories' will cease to exist. — Gartner
> [01:29](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=89s) An LLM acts essentially as a brain: it can generate vast amounts of insight when interacting with a human via tools like ChatGPT or Claude, but on its own, a brain cannot take real-world action. — Steve Stein
> [04:14](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=254s) A single agentic task consumes an average of 47,000 tokens—translating to nearly $4 per action. — Steve Stein
> [06:40](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=400s) Running open-weight models on private systems reduces the 'variable cost per task' to zero. — Steve Stein
> [06:40](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=400s) Vera Rubin achieves up to a 10x token-per-watt efficiency gain over the already high-performance Blackwell architecture. — Steve Stein
> [09:46](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=586s) NVIDIA Vera delivers nearly double the tool-execution efficiency of standard x86 CPUs. — Steve Stein
> [12:55](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=775s) Not every agentic step demands a massive, frontier-class model with hundreds of billions or trillions of parameters; simpler intermediate tasks, parsing, or data formatting can be routed to nimble 30-billion or 120-billion parameter models. — Steve Stein
> [15:24](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=924s) I interact with these on my own personal machine, and I love them. I'm a little biased, you know, I work for NVIDIA, but I really enjoy working with these. And if I can figure it out, by the way, I think you can figure it out. I'm not exactly any top-tier engineer coming out of NVIDIA. I'm just a little lowly product marketer that's trying to get some of this stuff done. — Steve Stein
> [17:35](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1055s) Because open models carry no per-token API licensing fee, the marginal variable cost for every generated token on-premises drops to essentially zero. — Steve Stein
> [18:13](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1093s) Generational hardware upgrades in AI are no longer measured purely in raw floating-point operations; they are defined by token generation throughput. — Steve Stein
> [19:00](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1140s) Raw inputs—electricity and enterprise data—flow in at the bottom, and intelligence in the form of processed tokens flows out at the top. — Steve Stein
> [20:05](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1205s) While NVIDIA provides the underlying silicon, networking, and core AI models, Hewlett Packard Enterprise (HPE) shoulders the operational burden of delivering and maintaining enterprise-grade on-prem AI deployments. — Steve Stein
## Video description
Steve Stein - Product Marketing, NVIDIA
Agentic AI changes enterprise architecture from a model-access problem into a systems-design challenge: autonomous agents chain model calls, retrieve context, invoke tools, and coordinate with other agents, multiplying token consumption while expanding the security and governance boundary across the business. In this session, HPE and NVIDIA will show how they are co-developing enterprise AI factory architectures that make cost per token, data gravity, sovereignty, observability, and policy enforcement core platform design principles. Attendees will learn how accelerated compute, networking, storage, NVIDIA AI Enterprise software, HPE Private Cloud AI, and governed agent runtime controls can help enterprises move from pilot agents to production-scale agentic AI systems that are secure, observable, and economically sustainable.
As autonomous AI agents replace conversational chatbots, enterprise token consumption is surging by orders of magnitude, turning token economics into a critical IT challenge. Hewlett Packard Enterprise and NVIDIA argue that mitigating these unsustainable cloud expenses requires dedicated on-premises infrastructure combining specialized inference silicon, execution CPUs, and tiered open models.
The Agentic Shift and Exponential Token Consumption (4:14)
The transition from conversational chatbots to autonomous agents marks a major architectural shift. Standard chatbot interactions between 2022 and 2024 averaged roughly 400 tokens and cost around 4 cents per prompt-and-response turn. In contrast, autonomous agentic loops pair an LLM with a software execution harness that runs code, calls APIs, and ingests long-term memory and persona guidelines. This continuous multi-step feedback loop averages 47,000 tokens per action, costing approximately $4.00.
According to Gartner, individual agentic actions cost 30 times more than standard conversational chats, while NVIDIA internal data shows complex tasks can reach 100 times baseline costs. Because autonomous loops execute repeatedly and can run into unbounded cycles, enterprises relying entirely on public cloud APIs risk depleting annual budgets within months. Gartner further projects that by 2029, 70% of large enterprises that fail to build or adopt dedicated AI factories will cease to exist.
On-Premises Infrastructure and Generational Token Efficiency (6:40)
To counteract unpredictable cloud API billing, HPE and NVIDIA propose shifting inference to on-premises systems running local, open-weight models. While requiring upfront capital acquisition, hosting open models privately reduces the marginal variable cost per generated token to zero. This turns escalating, variable operational expenses into predictable, depreciable capital investments.
Silicon performance metrics have correspondingly shifted from raw floating-point operations to token generation efficiency. While NVIDIA's Blackwell architecture delivers a 2x raw compute increase over Hopper, it yields a 50x increase in token generation efficiency due to inference-specific architectural and memory optimizations. NVIDIA projects that its upcoming Vera Rubin architecture will further advance this trajectory, providing up to a 10x improvement in tokens generated per watt compared to Blackwell.
Overcoming the CPU Execution Bottleneck with Vera (9:46)
As GPU inference throughput expands, agentic workflows face a hardware bottleneck on the host CPU. Once GPUs produce action instructions, host processors must execute shell commands, manage software tools, and coordinate external APIs. Standard x86 processors were not built for continuous tool-calling workloads, and their multi-chiplet designs introduce latency variations across stacked dies.
To decouple inference from tool execution, NVIDIA developed the Vera CPU, a standalone processor dedicated exclusively to OS actions and software harnesses while leaving model inference on GPUs. Vera utilizes a single monolithic die to avoid inter-die memory latency, achieving nearly double the tool-execution efficiency of conventional x86 chips. HPE is integrating this processor into enterprise data centers via the HPE ProLiant DL394 platform.
Dynamic Routing and Tiered Open-Weight Models (12:55)
Optimizing end-to-end agentic pipelines requires software-level orchestration to prevent unnecessary calls to costly frontier models. Within the NVIDIA Agent Toolkit, tools like NeMo Relay and NeMo Switchyard inspect incoming prompts and dynamically route sub-tasks across model tiers, reserving cloud-hosted frontier models for deep reasoning while routing routine parsing and data formatting to local open-weight models.
For on-premises execution, NVIDIA provides the Nemotron model family across multiple tiers: Nemotron Nano (30B) for low-latency tasks, Nemotron Super (120B) for balanced reasoning, and Nemotron Ultra (550B) for multi-agent coordination. Concurrently, HPE supplies runtime observability, security boundaries, and policy governance through HPE Private Cloud AI to monitor agent swarms, trace outputs, and prevent proprietary data leaks.
HPE and NVIDIA outline a comprehensive, full-stack alternative to public cloud API dependency, but the strategy hinges on enterprises successfully absorbing substantial upfront capital expenses, managing physical data center constraints, and integrating complex multi-tier software stacks to achieve expected operational efficiencies.
## Background
该内容由 Ai4 于 2026 年 8 月 15 日发布在 YouTube 平台，由 NVIDIA 产品营销部门的 Steve Stein 主讲。页面介绍指出，智能体 AI 将企业架构演进从单纯的模型接入转变为复杂的系统工程挑战，NVIDIA 与 HPE 携手共建企业级 AI 工厂架构，旨在将每 Token 成本、数据重力、可观测性与治理机制融入平台设计，助力企业安全且经济地迈向生产级智能体系统。
## TL;DR
随着人工智能进入机器自主消费 Token 的 Agentic AI 时代，智能体循环与复杂上下文导致任务成本飙升高达百倍，代币经济学（Tokenomics）成为企业 IT 预算的核心挑战。为破解成本困局，NVIDIA 与 HPE 联合提出本地私有化 AI 工厂方案，通过运行 Nemotron 等开源开放权重模型将边际 Token 成本降至归零。在架构上，Blackwell 及下一代 Vera Rubin 架构实现了能耗比的代际跨越，专用的 NVIDIA Vera CPU（搭载于 HPE ProLiant DL394）彻底化解了传统 x86 处理工具调用的性能瓶颈，配合 NeMo 软件栈的智能分流路由，全链路压缩算力开销。依托 HPE 顶级超算工程实力与全链路可观测性平台，该联合架构为企业提供了兼顾安全性、成本可控与高吞吐的开箱即用 AI 基础设施。
## Atmosphere & style
整场演讲呈现出专业严谨、直击行业痛点且极具工程实操感的企业级技术宣讲氛围。主讲人风格沉稳务实，适时穿插自嘲式的幽默来化解深奥硬件架构的枯燥感，同时借助 Gartner 的严峻行业预测营造出向 AI 工厂转型的战略紧迫感。叙事逻辑层层递进，从 Token 成本失控的微观机制平滑过渡到底层芯片、服务器硬件与软件调度的宏观全栈方案，带给观众清晰、权威且令人信服的系统级技术认知。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=4s) Tokenomics 成本挑战正严重吞噬企业 IT 预算，Gartner 预测到 2029 年，全球 70% 未能构建或使用 AI 工厂的大型企业将面临淘汰。
- [01:29](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=89s) Agent 系统的本质是由 LLM 认知大脑与 Harness 执行框架共同驱动，通过动作指令触发外部环境并在反馈中形成闭环的 Agentic Loop。
- [03:00](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=180s) 在 GenAI 2.0 阶段，Token 的核心消费者已从人类转变为计算机与 Harness，机器自治的高频调用使单任务成本膨胀达传统对话的 30 至 100 倍。
- [05:03](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=303s) 自主 Agent 单次任务需摄入角色设定、技能集与长期记忆，平均消耗约 47,000 个 Token（约 4 美元），若缺乏运行时治理极易陷入失控的无限循环。
- [06:40](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=400s) 本地私有化部署结合开源权重模型可使边际 Token 生成可变成本归零，下一代 Vera Rubin 架构更将每瓦 Token 产出能效推向 Blackwell 的 10 倍。
- [09:46](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=586s) 针对 x86 处理器处理 Agent 高并发工具调用造成的性能瓶颈，NVIDIA 推出单片裸晶设计的 Vera CPU 及 HPE ProLiant DL394，使工具调用执行效率翻倍并将 CPU 与 GPU 推理完全解耦。
- [12:55](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=775s) NVIDIA Agent Toolkit 配合 NeMo Relay 与 Switchyard 智能路由，可根据复杂度将非关键任务自适应分流至 30B 或 120B 轻量模型，有效抑制算力浪费。
- [15:24](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=924s) NVIDIA 自研开源的 Nemotron 模型家族覆盖 Nano (30B)、Super (120B) 到 Ultra (550B)，并构建了涵盖视觉、安全与语音的全套开源生态矩阵。
- [17:35](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1055s) 通过软硬件深度协同优化，Blackwell 架构相比 Hopper 在算力提升约 2 倍的基础上，实现了高达 50 倍的 Token 生成效率飞跃。
- [19:00](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1140s) NVIDIA 与 HPE 联合定义了软硬件全栈五层 AI 工厂架构，以数据资产和电力为底层输入，端到端交付高价值的业务智能与 Token 产出。
- [20:05](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1205s) 依托 HPE 在全球超算领域的工程实力与 HPE Private Cloud AI，企业能够获得全链路 Agent 运行态可观测性与治理保障，实现从试点到生产级架构的跨越。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=4s) Tokenomics 挑战与 AI 工厂的战略必然性
在当今企业推进人工智能落地的过程中，一个核心矛盾正逐渐显现：业内称之为“代币经济学”（Tokenomics）的挑战正在迅速吞噬各大企业的 IT 预算。随着大模型调用量的激增，企业如何支付这些海量的 Token 账单成为了生死攸关的难题。为此，NVIDIA 与 HPE（慧与科技）展开深度合作，旨在从系统架构层面彻底重构解决方案。在探讨具体的应对策略前，企业必须认识到构建或接入“AI 工厂”（AI Factory）的紧迫性。正如行业分析机构 Gartner 所预测，到 2029 年，全球 70% 未能建立或使用 AI 工厂的大型企业将面临被市场淘汰的命运。在生成式 AI 与 Agentic AI（智能体 AI）爆发的时代，AI 工厂是企业以最高效率提供 Token 与智能算力的核心基础设施。
- **IT 预算危机**：企业调用模型产生的 Token 成本呈现爆发式增长，传统的云端按量付费模式难以维系。
- **Gartner 严峻预测**：到 2029 年，70% 不采用或不构建 AI 工厂的大型企业将“停止存在”（cease to exist）。
- **AI 工厂定位**：企业级 AI 基础设施转型的核心，专注于以极高能效与极低单位成本大规模分发智能。
---
### [01:29](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=89s) 定义 Agent：LLM 大脑与 Harness 执行框架构成的智能体循环
要理解 Token 消耗为何失控，首先需要从工程实践角度明确什么是真正的 Agent（智能体）。大型语言模型（LLM）本身如同一个功能强大的“大脑”，在与人类（如 ChatGPT 或 Claude 用户）交互时能够输出大量有价值的信息，但大脑本身无法直接干预现实世界。为了执行具体任务，LLM 必须连接到一个“躯体”——即 Harness（执行框架）。Harness 负责解析 LLM 的意图，执行具体的代码、系统命令、调用 API 工具或自动生成演示文稿（PowerPoint）等操作。
在 Harness 完成物理世界的任务执行后，它会将执行结果和上下文环境数据回传给 LLM。LLM 对这些返回的信息进行二次分析，进而推导并生成下一阶段的动作指令。这一过程构成了典型的“智能体循环”（Agentic Loop）。不论是瞬时触发的轻量级任务，还是需要长时间自主运行的复杂流程，这种自主往复的闭环架构正是现代 Agent 系统的底层运作逻辑。
```mermaid
flowchart LR
accTitle: Agentic Loop Workflow
subgraph Cognitive ["认知层 (Brain)"]
LLM["大语言模型 LLM<br/>(规划/决策/分析)"]
end
subgraph Execution ["执行层 (Body)"]
Harness["执行框架 Harness<br/>(代码/命令/工具调用)"]
Env["外部环境 / 工具 / API"]
end
LLM -->|"动作与代码指令"| Harness
Harness -->|"触发调用"| Env
Env -->|"执行结果"| Harness
Harness -->|"环境反馈与状态更新"| LLM
```
---
### [03:00](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=180s) 从人类对话到机器自治：GenAI 2.0 的 Token 消耗爆炸
人工智能的演进经历了两大鲜明阶段：在 2022 至 2024 年初的“ChatGPT 时代”，全行业的焦点在于如何让人类通过与大模型交互获取高质量信息，随后逐步引入检索增强生成（RAG）以降低幻觉并增强上下文感知。然而，这两个阶段的共同特征是：**Token 的最终消费者是人类**，系统吞吐受限于人类阅读和提问的生理极限。
进入 Agentic AI（被称作“GenAI 2.0”）时代后，Token 消耗的本质发生了颠覆性转变——**计算机与 Harness 本身成为了 Token 的主要消费者**。为了维持自主运作，Agent 需要实时加载技能模块（Skills）、调用专业工具（Tools），并通过持久化记忆系统跨会话维持状态。这种从“人类消费”向“机器自主消费”的跃迁，不仅带来了生产力的倍增与全新的攻击面，更引发了单位任务成本的几何级数飙升。
- **消费主体变更**：从人类单向问答演变为机器 Harness 之间高频度、自治化的 Token 吞吐。
- **攻击面与系统负担**：自主调用外部技能与长期记忆机制，同步放大了安全脆弱性与系统开销。
- **成本倍数级膨胀**：Gartner 预计单个 Agent 动作的成本可达传统聊天机器人的 30 倍；而根据 NVIDIA 内部测算，某些极端场景下的消耗倍率甚至高达 100 倍。
---
### [05:03](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=303s) 算力成本失控：任务级 Token 消耗与失控循环模型
传统聊天机器人单次交互大约生成 400 个 Token，平均成本仅约 4 美分。但当一个自主 Agent 执行单项复杂任务时，情况发生剧烈变化。Agent 必须将系统的角色设定文件（Soul File）、技能描述文档（Skills）以及海量的历史记忆作为上下文全部吞入——这些本质上是由大量文本和 Markdown 格式构成的长上下文数据。
数据表明，一个 Agent 执行单次任务平均需摄入并消耗约 **47,000 个 Token**，直接导致单次任务的成本逼近 **4 美元**。更为致命的是，在 Agentic Loop 的机制下，任务执行完毕返回数据后，模型会再次生成新的推理指示，循环往复。如果缺乏严格的运行时治理机制，Agent 极易陷入无限执行循环（Infinite Loop），进而导致企业的年度 AI 预算在数月甚至数周内被彻底耗尽。
| 交互形态 | 单次任务平均 Token 消耗 | 预估单次成本 | 典型交互机制 |
|:--- |:--- |:--- |:--- |
| **传统对话式 AI (Chatbot)** | ~400 Tokens | ~$0.04 | 单轮/少轮问答，受限于人类输入速度 |
| **自主智能体 (Agentic AI)** | ~47,000 Tokens | ~$4.00 | 摄入记忆、Soul 设定与技能集，多步自治循环 |
---
### [06:40](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=400s) 本地私有化破局与 Vera Rubin 架构：实现 Token 生成能效的代际跃升
要从根本上解决 Tokenomics 危机，关键路径在于构建**本地私有化系统（On-Prem Systems）**并运行开源/开放权重模型（Open-Weight LLMs）。在完成前期的硬件资产购置或融资成本后，本地部署运行开源模型的“边际可变 Token 成本”（Variable Cost per Task）几乎为零。NVIDIA 通过软硬件协同创新，致力于降低整个系统的单位 Token 生成能耗。
在 GPU 架构层面，继奠定生成式 AI 产业基石的 Hopper 架构以及性能大幅跃升的 Blackwell 架构之后，NVIDIA 即将推出下一代 **Vera Rubin** 架构。Vera Rubin 的每瓦 Token 生成效率（Tokens per Watt）相比 Blackwell 实现了高达 **10 倍**的飞跃。NVIDIA 采用以“用户交互延迟（Tokens/sec/user）”为横轴、“能效产出（Tokens/sec/MW）”为纵轴的曲线下面积来量化衡量 AI 工厂的综合输出能力，证明其代际升级并非行业常见的 10%~30% 微调，而是数量级的能效跨越。
- **边际成本归零**：本地私有化 + 开源权重模型，消除了云端 API 调用的按量计费陷阱。
- **能效跃迁**：Vera Rubin 架构实现高达 Blackwell 10 倍的能效比（Tokens per Watt）。
- **AI 工厂产出模型**：综合权衡单用户低延迟交互性与每兆瓦电力的极限 Token 产出。
---
### [09:46](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=586s) 突破 CPU 工具执行瓶颈：NVIDIA Vera 与 HPE ProLiant DL394
随着 GPU 端 Token 生成效率的大幅提升，前沿实验室的数据揭示了一个全新的架构瓶颈：GPU 能够极速吐出海量指令，但承载工具调用、代码执行与业务逻辑的传统 x86 CPU 却成为了严重的系统卡点。x86 架构并非为处理密集的 Agent 工具调用而设计，其复杂的 Chiplet（小芯片）堆叠设计导致了内存访问延迟不一致且不可预测。
这导致业界预测 GPU 与 CPU 的配比关系正在发生剧烈震荡（从原先的 4:1 甚至倒退到需要更多 CPU 支撑的 7:1）。NVIDIA 明确指出，**绝不能在 CPU 上运行模型推理**（推理必须保留在最高效的 GPU 上），CPU 必须专门优化以承担高并发的工具调用执行。为此，NVIDIA 推出了专用的独立 CPU 服务器架构——**NVIDIA Vera CPU**，并与 HPE 联合打造了 **HPE ProLiant DL394** 系统。
- **单片全集成设计（Monolithic Die）**：Vera CPU 采用单片金属裸晶设计，彻底摒弃 x86 的 Chiplet 架构，提供极其强劲的核心算力与高度一致、极速的内存访问响应。
- **工具调用能效翻倍**：在执行 Agent 工具调用与命令执行时，Vera CPU 的执行效率达到传统 x86 处理器的近 2 倍。
- **职责解耦分离**：Vera Rubin GPU 专职进行极致高效的 Token 矩阵推理，独立的 Vera CPU 服务器则全力执行工具与系统调用，彻底粉碎 Agentic 架构瓶颈。
---
### [12:55](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=775s) 软件定义效能：NVIDIA Agent Toolkit 与 NeMo 智能模型路由
除了底层的顶尖算力基础设施，NVIDIA 的核心竞争优势还在于其全栈软件优化能力。为了最大化整个 Agentic 吞吐效率，NVIDIA 推出了 **NVIDIA Agent Toolkit**，覆盖开源模型加速、Harness 安全执行、已验证技能库分发以及运行时环境控制等全流程。
在 Agent Toolkit 中，两款关键产品——**NeMo Relay** 与 **NeMo Switchyard** 扮演着智能调度核心的角色。在实际业务流中，并非所有 Agent 步骤都需要调用万亿参数级别的顶级前沿模型。许多辅助性推理、格式校验与中间指令完全可以卸载给轻量级的 30B 或 120B 参数模型。NeMo Relay 通过动态评估任务复杂度，实现智能分流与模型降级路由，在确保任务精准完成的同时大幅缩减计算开销与延迟，从全链路软件层面遏制 Token 成本的无序膨胀。
- **分层计算优化**：避免“大炮打蚊子”，将非核心任务自适应分流至更轻快、低成本的 30B/120B 模型。
- **端到端流程提速**：通过软硬件协同将 Agent 全生命周期的 Token 消耗降至最低，大幅提升系统可用性与总体吞吐量。
### [15:24](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=924s) 智能路由与 NVIDIA Nemotron 开源模型家族
在多智能体系统（Agentic AI）的实际运行中，海量的复杂提示词（Prompt）如果无差别地直接发送给云端前沿模型，将带来不可承受的调用开销。演讲者介绍了类似 SwitchYard 这样的智能路由机制：当系统接收到复杂提示词时，SwitchYard 能够实时评估并决策是将其路由至像 ChatGPT 等顶级前沿大模型，还是直接卸载（Offload）到企业本地部署的算力资源上，运行诸如 NVIDIA Nemotron 120B 等本地模型。这种灵活分流的核心目标，是在保证任务完成质量的前提下最大化端到端的智能体吞吐效率（Agentic Throughput）。
```mermaid
flowchart LR
accTitle: "SwitchYard 智能路由分流机制"
subgraph Input["任务输入"]
P["复杂任务 Prompt / Agent 协作请求"]
end
subgraph Routing["决策中心"]
SY{"SwitchYard 智能路由器"}
end
subgraph Execution["执行环境"]
Cloud["云端前沿大模型<br/>(Frontier Models / GPT 等)"]
OnPrem["本地本地算力部署<br/>(Nemotron Nano / Super / Ultra)"]
end
P --> SY
SY -- "高复杂度 / 关键决策" --> Cloud
SY -- "本地处理 / 降本增效" --> OnPrem
```
针对本地算力部署，NVIDIA 推出了 Nemotron 系列开源开放权重（Open-weight）模型。Nemotron 是完全由 NVIDIA 自主训练并发布的模型家族，覆盖了从小型到超大型的多样化规模：Nano 版本为 30B（300 亿参数），Super 版本为 120B（1200 亿参数），而近期发布的 Ultra 版本则达到了 550B（5500 亿参数）。演讲者幽默地提到，他在自己的个人电脑上日常使用这些模型且体验极佳，自嘲道："我只是个试图把事情做好的普通产品营销人员（lowly product marketer），如果我都能搞明白并顺利运行，各位肯定也能掌握。" Nemotron 不仅涵盖文本基准大模型，还构建了包含视觉（Vision）、信息检索（Information Retrieval）、安全防护（Safety）和语音处理（Speech）在内的全套开源生态，在各项主流评测基准中均展现出位列第一梯队的强劲性能。
- **Nemotron 模型家族规模**：涵盖 Nano（30B）、Super（120B）及 Ultra（550B）等多元尺寸。
- **开源生态矩阵**：除通用基础模型外，同步提供针对视觉理解、信息检索、安全对齐和语音处理的专用模型。
- **评测表现**：在开源模型多项基准测试（Benchmarks）中，Nemotron 各版本表现均逼近或达到开源领域最高水准。
---
### [17:35](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1055s) 解决代币经济学难题：本地大模型与架构代际跃迁
企业规模化应用 Agent 的最大瓶颈在于代币经济学（Tokenomics）危机——在频繁调用商业 API 的模式下，每 Token 的累积费用极其高昂，其综合成本可能达到运行本地轻量级大模型的 100 倍。解决这一难题的终极路径，是将推理负载迁移至企业私有云与本地系统（On-prem Systems），通过运行开源大模型彻底消除边际调用费。演讲者强调："运行本地大模型时，每生成一个 Token 的可变成本（Variable Cost）是零，因为这些开放模型本身是免费的。"
除了模型本身的免费属性，底层硬件架构的代际跃迁更是驱动 Token 成本骤降的关键杠杆。以 Blackwell 与上一代 Hopper 架构进行对比，单纯从原始计算能力（Compute）来看，Blackwell 是 Hopper 的约 2 倍；但如果评估核心的 Token 生成效率（Token Generation Efficiency），Blackwell 实现了惊人的 50 倍提升。这一跨越式的效率飞跃不仅归功于芯片底层微架构的革新，还得益于软件栈与系统级优化的协同，也是企业将 Token 生成全面部署在 NVIDIA GPU 上的核心动因。
- **Token 边际成本对比**：商业前沿 API 调用成本最高可达本地运行模型的 100 倍；本地开源模型的 Token 生成可变成本为 0。
- **代际效率飞跃**：Blackwell 架构相比 Hopper 架构，算力提升约 2 倍，而 Token 生成效率实现 50 倍（50x）剧增。
- **软硬件协同设计**：50 倍效率提升由底层架构改造与 NVIDIA AI Enterprise 软件栈共同赋能。
---
### [19:00](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1140s) AI 工厂理念与软硬件全栈五层架构
演讲者指出，NVIDIA 从未将自身局限为一家单一的硬件基础设施公司，而是致力于打造涵盖软硬件协同的"五层蛋糕"（Five-layer cake）全栈体系，为业界提供标准化的企业级 AI 工厂（AI Factory）架构。在 AI 工厂的运转模型中，最底层输入的是海量数据与电力（Data and Electricity），经过中间层层加速与调度，顶层最终输出的是高价值的智能与 Token（Intelligence or Tokens）。NVIDIA 与 HPE 紧密协作，正是为了向企业交付能够开箱即用、轻松在本地搭建与运维的 AI 工厂交钥匙方案。
```mermaid
flowchart TD
accTitle: "企业级 AI 工厂（AI Factory）分层输入输出架构"
subgraph Output["工厂产出层"]
Tokens["业务智能 / Token 交付 (Intelligence / Tokens)"]
end
subgraph Stack["NVIDIA & HPE 全栈协同体系（五层架构）"]
S5["智能体应用与编排 (Agentic Runtime & Workflow)"]
S4["模型与框架层 (Nemotron, NeMo Framework, NIMs)"]
S3["企业级平台软件 (NVIDIA AI Enterprise / HPE Private Cloud AI)"]
S2["高速集群网络与存储 (InfiniBand / NVLink / Spectrum-X)"]
S1["加速计算硬件 (Blackwell / Rubin GPU, Grace / Vera CPU)"]
end
subgraph Input["基础要素输入"]
Raw["企业数据资产 & 算力电力基础设施 (Data & Electricity)"]
end
Raw --> S1
S1 --> S2
S2 --> S3
S3 --> S4
S4 --> S5
S5 --> Tokens
```
与此同时，NVIDIA 保持着高度公开、透明且极具节奏感的产品路线图，让企业客户能够清晰规划自身的架构演进路径。从 GPU 维度的 Blackwell 演进至下一代 Rubin，到 CPU 维度的 Grace 演进至 Vera，NVIDIA 持续推进计算与网络互连的边界，使整个 AI 基础设施的性能与能耗比按年实现指数级递进。
- **AI 工厂核心范式**：以数据和电力为原材料输入，以可落地的业务智能与高效 Token 为最终产出。
- **全栈五层架构**：覆盖底层硬件、网络存储、系统软件、AI 运行时平台及顶层应用框架。
- **技术路线图演进**：GPU 产品线由 Blackwell 稳步迈向 Rubin；CPU 产品线由 Grace 升级为 Vera。
---
### [20:05](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1205s) HPE 与 NVIDIA 深度联合：企业级 AI 系统的落地与可观测性保障
在 AI 工厂的物理落地与运维实践中，HPE 扮演了极其关键的基石角色，切实为企业承担起端到端交付的运维负担（Operational Burden）。作为企业级计算领域的深耕者，HPE 拥有建造全球最强超算系统的工程实力，不仅在 MLPerf 评测中保持着 18 项基准测试纪录，在权威的全球超算 TOP500 榜单中也稳居榜首与前列。通过 HPE Private Cloud AI 等方案，企业能够以最合规、最安全的方式将 NVIDIA 最新计算与网络架构接入本地机房。
随着自主多智能体在企业内部的大规模繁衍，治理与追踪成为核心挑战。HPE 软件栈在联合方案中提供了至关重要的 Agent 可观测性（Observability）能力。企业不仅需要在源头上生成安全受控的智能体，还必须实时监控各 Agent 跨部门、跨系统的工具调用、数据检索与协作链路。HPE 与 NVIDIA 通过深度的系统协同研发（Co-engineering & Co-developing），共同构建了兼具极致经济性、严密安全性与全链路可观测性的企业级 Agentic AI 生产级平台。
- **超算与基准测试标杆**：HPE 在 MLPerf 中斩获 18 项基准纪录，并长期位列全球 TOP500 超算前茅。
- **全链路 Agent 可观测性**：提供针对智能体运行态的细粒度追踪工具，掌控 Agent 在组织内的行为边界与安全状态。
- **联合研发交付**：双方通过软硬件一体化共研，助力企业平滑渡过从试点验证（Pilot）到生产规模（Production-scale）的落地鸿沟。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=4s) 到 2029 年，70% 不采用或不构建 AI 工厂的大型企业将停止存在（cease to exist）。 — Steve Stein
> [01:29](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=89s) 大语言模型 LLM 本身如同一个功能强大的大脑，在与人类交互时能够输出大量有价值的信息，但大脑本身无法直接干预现实世界。为了执行具体任务，LLM 必须连接到一个躯体——即 Harness 执行框架。 — Steve Stein
> [03:00](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=180s) Token 消耗的本质发生了颠覆性转变——计算机与 Harness 本身成为了 Token 的主要消费者。 — Steve Stein
> [05:03](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=303s) 一个 Agent 执行单次任务平均需摄入并消耗约 47,000 个 Token，直接导致单次任务的成本逼近 4 美元。 — Steve Stein
> [06:40](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=400s) 在完成前期的硬件资产购置或融资成本后，本地部署运行开源模型的边际可变 Token 成本（Variable Cost per Task）几乎为零。 — Steve Stein
> [09:46](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=586s) 绝不能在 CPU 上运行模型推理，CPU 必须专门优化以承担高并发的工具调用执行。 — Steve Stein
> [15:24](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=924s) 我只是个试图把事情做好的普通产品营销人员（lowly product marketer），如果我都能搞明白并顺利运行，各位肯定也能掌握。 — Steve Stein
> [17:35](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1055s) 运行本地大模型时，每生成一个 Token 的可变成本（Variable Cost）是零，因为这些开放模型本身是免费的。 — Steve Stein
> [17:35](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1055s) 单纯从原始计算能力来看，Blackwell 是 Hopper 的约 2 倍；但如果评估核心的 Token 生成效率，Blackwell 实现了惊人的 50 倍提升。 — Steve Stein
> [19:00](https://www.youtube.com/watch?v=r2l5ReP6jTk&t=1140s) 最底层输入的是海量数据与电力（Data and Electricity），经过中间层层加速与调度，顶层最终输出的是高价值的智能与 Token（Intelligence or Tokens）。 — Steve Stein
## Mentioned
Steve Stein, NVIDIA, HPE, Gartner, ChatGPT, Claude, Vera Rubin, Blackwell, Hopper, NVIDIA Vera CPU, HPE ProLiant DL394, NVIDIA Agent Toolkit, NeMo Relay, NeMo Switchyard, Nemotron, Nemotron Nano (30B), Nemotron Super (120B), Nemotron Ultra (550B), NIMs, NVIDIA AI Enterprise, HPE Private Cloud AI, MLPerf, TOP500
## Video description
Steve Stein - Product Marketing, NVIDIA
Agentic AI changes enterprise architecture from a model-access problem into a systems-design challenge: autonomous agents chain model calls, retrieve context, invoke tools, and coordinate with other agents, multiplying token consumption while expanding the security and governance boundary across the business. In this session, HPE and NVIDIA will show how they are co-developing enterprise AI factory architectures that make cost per token, data gravity, sovereignty, observability, and policy enforcement core platform design principles. Attendees will learn how accelerated compute, networking, storage, NVIDIA AI Enterprise software, HPE Private Cloud AI, and governed agent runtime controls can help enterprises move from pilot agents to production-scale agentic AI systems that are secure, observable, and economically sustainable.
[Source](https://www.youtube.com/watch?v=r2l5ReP6jTk)
Source quotes
[0:04] Steve Stein: 到 2029 年，70% 不采用或不构建 AI 工厂的大型企业将停止存在（cease to exist）。
[1:29] Steve Stein: 大语言模型 LLM 本身如同一个功能强大的大脑，在与人类交互时能够输出大量有价值的信息，但大脑本身无法直接干预现实世界。为了执行具体任务，LLM 必须连接到一个躯体——即 Harness 执行框架。
[3:00] Steve Stein: Token 消耗的本质发生了颠覆性转变——计算机与 Harness 本身成为了 Token 的主要消费者。
[5:03] Steve Stein: 一个 Agent 执行单次任务平均需摄入并消耗约 47,000 个 Token，直接导致单次任务的成本逼近 4 美元。
[6:40] Steve Stein: 在完成前期的硬件资产购置或融资成本后，本地部署运行开源模型的边际可变 Token 成本（Variable Cost per Task）几乎为零。
[9:46] Steve Stein: 绝不能在 CPU 上运行模型推理，CPU 必须专门优化以承担高并发的工具调用执行。
[15:24] Steve Stein: 我只是个试图把事情做好的普通产品营销人员（lowly product marketer），如果我都能搞明白并顺利运行，各位肯定也能掌握。
[17:35] Steve Stein: 运行本地大模型时，每生成一个 Token 的可变成本（Variable Cost）是零，因为这些开放模型本身是免费的。
[17:35] Steve Stein: 单纯从原始计算能力来看，Blackwell 是 Hopper 的约 2 倍；但如果评估核心的 Token 生成效率，Blackwell 实现了惊人的 50 倍提升。
[19:00] Steve Stein: 最底层输入的是海量数据与电力（Data and Electricity），经过中间层层加速与调度，顶层最终输出的是高价值的智能与 Token（Intelligence or Tokens）。
Source quotes
[0:04] Gartner: By 2029, 70% of large enterprises that do not build or use dedicated 'AI factories' will cease to exist.
[1:29] Steve Stein: An LLM acts essentially as a brain: it can generate vast amounts of insight when interacting with a human via tools like ChatGPT or Claude, but on its own, a brain cannot take real-world action.
[4:14] Steve Stein: A single agentic task consumes an average of 47,000 tokens—translating to nearly $4 per action.
[6:40] Steve Stein: Running open-weight models on private systems reduces the 'variable cost per task' to zero.
[6:40] Steve Stein: Vera Rubin achieves up to a 10x token-per-watt efficiency gain over the already high-performance Blackwell architecture.
[9:46] Steve Stein: NVIDIA Vera delivers nearly double the tool-execution efficiency of standard x86 CPUs.
[12:55] Steve Stein: Not every agentic step demands a massive, frontier-class model with hundreds of billions or trillions of parameters; simpler intermediate tasks, parsing, or data formatting can be routed to nimble 30-billion or 120-billion parameter models.
[15:24] Steve Stein: I interact with these on my own personal machine, and I love them. I'm a little biased, you know, I work for NVIDIA, but I really enjoy working with these. And if I can figure it out, by the way, I think you can figure it out. I'm not exactly any top-tier engineer coming out of NVIDIA. I'm just a little lowly product marketer that's trying to get some of this stuff done.
[17:35] Steve Stein: Because open models carry no per-token API licensing fee, the marginal variable cost for every generated token on-premises drops to essentially zero.
[18:13] Steve Stein: Generational hardware upgrades in AI are no longer measured purely in raw floating-point operations; they are defined by token generation throughput.
[19:00] Steve Stein: Raw inputs—electricity and enterprise data—flow in at the bottom, and intelligence in the form of processed tokens flows out at the top.
[20:05] Steve Stein: While NVIDIA provides the underlying silicon, networking, and core AI models, Hewlett Packard Enterprise (HPE) shoulders the operational burden of delivering and maintaining enterprise-grade on-prem AI deployments.
Original transcript
[0:04] Today what I want to talk to you about
[0:05] is something that's kind of on all of
[0:06] our minds. The previous panel did a
[0:08] great job talking through this challenge
[0:10] that we have that we call tokenomics.
[0:13] And how tokenomics is kind of eating all
[0:15] of our IT budgets in terms of being able
[0:18] to pay for these tokens. So I'm going to
[0:20] talk to you a little bit today about why
[0:22] this is happening, how agents are
[0:24] changing the equation with how many
[0:25] tokens you're consuming, but more
[0:27] importantly
[0:28] I want to talk about solutions. And
[0:30] specifically how NVIDIA
[0:33] is partnering with HPE to provide
[0:36] solutions to solve this problem. And I
[0:38] talked a little bit about agents
[0:41] uh yesterday. So I'm going to go through
[0:43] a little bit of that before I get into
[0:45] the tokenomics channel cuz I want to be
[0:46] able to define
[0:48] what agents are. But before I do that, I
[0:50] always love to throw up this quote.
[0:52] And that's because if you're not
[0:53] thinking about building an AI factory,
[0:55] if you're not thinking about accessing
[0:57] an AI factory,
[1:00] there's a high probability they're going
[1:01] to be left behind.
[1:03] Gartner projected, you know, by 2029,
[1:06] 70% of large enterprises that don't
[1:08] build or use AI factories will cease to
[1:12] exist. And that's because in this moment
[1:14] of agentic AI, in this moment of
[1:16] generative AI, it does provide that much
[1:18] more value to your organization to be
[1:20] leaning on these tools. And AI factories
[1:22] are the primary tool to be able to serve
[1:25] tokens, be able to serve intelligence as
[1:27] efficiently as possible.
[1:29] And before I get into
[1:31] kind of what agents are doing for your
[1:32] organization, the token cost that
[1:33] they're generating, I love showing this
[1:37] because this is the most practical
[1:38] definition of what an agent actually is.
[1:43] Cuz what an agent is actually doing is
[1:45] it's able to take the information
[1:48] generated from an LLM
[1:51] and do something with it.
[1:53] So an LLM is great. It can feed you a
[1:55] lot of great information. If you
[1:57] interact with chat GPT or Claude, it can
[1:59] give you give you a human a lot of great
[2:02] information, but it can't do anything by
[2:03] itself. It needs connections to like a
[2:06] body. Just like how an LLM is like a
[2:08] brain. A brain can give you a lot of
[2:09] great information, but if you don't
[2:11] connect it to something, if you don't
[2:12] connect it to a body to go off and do
[2:14] things, it can't do anything. And that's
[2:15] what a harness
[2:17] is. The harness is the capability to
[2:20] actually go off and execute code,
[2:22] execute commands, execute actions that
[2:24] the LLM is telling it. And then what
[2:25] happens is after that harness goes off
[2:27] and does something, it executes codes,
[2:29] it builds a PowerPoint, whatever you
[2:31] need, it can send that information back
[2:33] to the LLM, the LLM can analyze it, come
[2:34] up with new actions, and that creates
[2:37] what we call now this agentic loop.
[2:39] The LLM is sending actions over to the
[2:41] harness to execute, the harness executes
[2:43] it, sends information back to the LLM,
[2:46] and then the loop continues. That's
[2:47] what's happening here. That's what's
[2:48] actually practically happening when
[2:50] we're talking about agents. And some of
[2:51] these can be long-running agents, some
[2:52] of these can be simple agents that it's
[2:54] asking it to do,
[2:56] but this is what an LLM is trying to get
[2:58] done.
[2:59] And then we're then we'll also need to
[3:00] talk about how this era is different
[3:03] from past eras. So, back in the chat GPT
[3:05] era of 2022 to 2024, we're all focused
[3:08] on how can we use LLMs to get better
[3:10] information. Last year was mostly
[3:12] focused on how can you make these LLMs
[3:13] more context-aware,
[3:15] decrease the the hallucination rate. But
[3:18] what's interesting about these two eras
[3:20] is the consumption of the tokens
[3:24] was us.
[3:25] Was humans.
[3:27] Us interacting with these chatbots to
[3:29] get information.
[3:30] What's different about agentic AI
[3:34] is these harnesses, these computers, are
[3:37] consuming the tokens as well.
[3:40] So, we have a new system that's
[3:42] consuming the tokens. And that opens up
[3:46] a whole new world of productivity, it
[3:47] opens up a whole new world of what these
[3:49] harnesses are actually trying to get
[3:50] done. They're they're grabbing skills,
[3:52] they're executing with certain tools,
[3:54] they have a memory system associated
[3:56] with them so they remember what they're
[3:58] trying to get done over certain
[3:59] sessions, but it also opens up a whole
[4:00] new layer of attack surfaces and costs.
[4:03] I like to call this new era kind of
[4:04] GenAI 2.0
[4:06] because it's going to provide a lot more
[4:07] value to us, a lot more interesting
[4:09] things to get done, but it's going to
[4:10] give us a lot more challenges to
[4:12] execute.
[4:14] And agents generally, as you're
[4:16] executing them, have two challenges that
[4:18] we need to solve.
[4:20] Yesterday, I went through the right-hand
[4:22] side of this chart looking at the
[4:23] securities and vulnerabilities. Today, I
[4:25] want to dive a lot deeper into this
[4:26] tokenomics problem, or we call runaway
[4:29] token costs. I'm sure all of you have
[4:30] experienced, if you haven't already, how
[4:32] expensive interacting with frontier
[4:34] models can be. There's a lot of news
[4:36] reports out there about companies
[4:37] running out of their running out of
[4:39] their entire yearly budgets within
[4:40] months
[4:42] because of how expensive tokens can be
[4:44] when you're accessing some of these
[4:45] frontier models. Uh Gartner projects
[4:47] that each individual task can be upwards
[4:50] of 30 times more expensive per action
[4:54] compared to, say, standard generative AI
[4:56] chatbots. Here at Nvidia, we have some
[4:57] information that that tells us that
[4:58] number could be as high as 100x.
[5:01] It's a real challenge that we're trying
[5:01] to get solved. So, let's dive into what
[5:03] this challenge is for a second, then I
[5:04] want to talk about solutions, too. So,
[5:07] we are truly at this point of this
[5:09] agentic inferencing inflection. What
[5:11] that means is we have these autonomous
[5:13] agents running,
[5:15] we have ability to spawn them and use
[5:17] them. Uh we have we're else in this era
[5:20] now AI is definitely useful giving us,
[5:22] you know, information back, and so we're
[5:23] seeing this huge boom in inferencing
[5:26] demand. So, you see a huge spike. And to
[5:28] put some numbers to it, if you interact
[5:31] with a chatbot today, it's about 4 cents
[5:33] per interaction. You know, it goes on
[5:35] average, you're going to be generating
[5:36] 400 tokens when you're just doing a chat
[5:38] session
[5:39] with an LLM. Now, if you have an agent,
[5:42] what an agent is doing per task,
[5:45] it's ingesting all of this memory.
[5:48] It's ingesting skills. It's ingesting a
[5:51] soul file. And what those are are
[5:54] they're markdown files. They're text
[5:55] files filled with a bunch of text. And
[5:58] what that means is the agent is
[6:00] consuming on average 47,000 tokens.
[6:05] And if you cost that out,
[6:07] that's somewhere in the neighborhood of
[6:08] just shy of $4 a task. And what's also
[6:11] happening you know with this agentic
[6:12] loop is a task completes, information
[6:15] gets sent back to the LLM.
[6:17] More information is generated, more
[6:18] tokens are generated back to the
[6:19] harness, and round and round it goes.
[6:21] You can have these agents running in an
[6:23] infinite loop.
[6:25] And that's how you get the tokenomics
[6:27] problem where you have agents that are
[6:29] one more expensive to run per task, but
[6:32] run much more often and can infinitely
[6:34] run. And so you have an issue where
[6:37] token budgets can just spiral out of
[6:38] control.
[6:40] So, let's talk about how we can fix this
[6:41] a little bit.
[6:43] And uh spoiler alert, it's all about
[6:45] on-prem systems.
[6:47] If you can have an on-prem system
[6:48] running a local LLM, an open-source
[6:50] open-weight LLM,
[6:52] I'll get to this in a in a bit, the
[6:54] variable cost per task
[6:57] is zero.
[6:59] That's because you're running an
[7:00] open-source open-weight model
[7:02] on your on-prem system. Now you got you
[7:04] got to take into account, you know,
[7:05] financial costs in terms of acquiring
[7:07] the system or or financing costs to be
[7:09] able to pay for that system on-prem. But
[7:10] if you have the hardware and you have an
[7:12] open open-weight open-source LLM,
[7:15] variable cost is zero. So, I'm going to
[7:17] go I'm going to go through these four
[7:18] categories in a second. I'm going to
[7:19] talk about how NVIDIA creates, you know,
[7:20] incredibly token efficient GPU systems.
[7:23] I'm going to show you some data on that.
[7:25] We have new CPUs coming out, and I'll
[7:28] explain why.
[7:30] It has nothing to do with running
[7:32] inferencing on the CPU.
[7:35] I'll tell you why in a second, but you
[7:36] do not want to be running inferencing on
[7:38] the CPU. But it's all about trying to
[7:39] think about how we to this entire
[7:40] agentic throughput more efficient, the
[7:42] entire system more efficient, rather
[7:45] than just looking at how we generate
[7:46] efficient tokens, how can you make the
[7:47] entire agentic system more efficient? We
[7:49] also have software focused on that
[7:51] problem as well with our agentic AI
[7:53] toolkit.
[7:54] And that's going to allow look at how we
[7:56] can make the entire agentic process more
[7:58] efficient, let alone, you know, also
[7:59] making sure we have efficient execution
[8:01] of the models. And then finally, I'll
[8:03] get into talking about some of the
[8:04] open-source models that that exist out
[8:06] there and why you should be choosing
[8:07] them and running them on prem uh
[8:10] compared to some of the frontier models
[8:12] out there.
[8:13] So, first, we have our Vera Rubin
[8:14] systems coming out here fairly soon.
[8:17] Some important stats to think about with
[8:18] this Blackwell architecture, which is
[8:20] largely in the market today, that's the
[8:22] successor to the Hopper generation.
[8:25] Blackwell was an incredible architecture
[8:29] and brought a lot of performance, a lot
[8:31] of, you know, performance gains over
[8:32] Hopper. By the way, Hopper was the most
[8:35] successful GPU architecture in history.
[8:39] Hopper is what built the generative AI
[8:41] industry
[8:43] in the first place. And Blackwell
[8:44] completely blows that out of the water.
[8:45] And now we got Vera Rubin coming online.
[8:48] And Vera Rubin is up to 10 times more
[8:50] efficient in to in generating tokens per
[8:52] watt than Blackwell. We are focused on,
[8:55] you know, generating, you know, systems,
[8:57] creating GPU systems that generate
[8:58] tokens incredibly efficiently. And these
[9:01] aren't little 10, 20%, 30% gen on gen
[9:03] gains. These are massive leaps in
[9:05] performance year on year.
[9:08] We are creating systems that are
[9:09] incredibly token efficient.
[9:11] And when you kind of look at some other
[9:12] charts here, we like we like to use
[9:14] these charts to kind of talk about AI
[9:16] factory output. I know it's a little
[9:17] hard to read, but the bottom axis here
[9:19] is tokens per second per user. So, think
[9:20] of that as like interactivity, latency
[9:23] per user. Y axis is tokens per second
[9:26] per megawatt.
[9:27] So, efficiency. We kind of we kind of
[9:29] measure this curve. And the area under
[9:32] the curve is the AI factory output. And
[9:35] so, we can kind of measure this output
[9:37] gen on gen, and that's where we get
[9:38] these numbers from of how much more
[9:39] efficient Vera Rubin architecture is
[9:42] going to be over Blackwell architecture.
[9:46] CPUs.
[9:48] I talked about that agentic loop.
[9:51] One major problem with this agentic loop
[9:54] is we have some data coming back from
[9:56] our Frontier Labs right now that show us
[9:57] that our token generation capability
[10:01] from our GPU systems is so efficient,
[10:03] we're sending so much information to
[10:05] CPUs that are executing the tools,
[10:08] executing the commands. Like a lot of
[10:10] the execution layer of the agentic AI
[10:12] workflow exists on the CPU side. There's
[10:15] a huge bottleneck happening at that CPU
[10:17] side.
[10:19] And it's because
[10:20] of x86 architectures.
[10:23] And so, we're going to see a lot of news
[10:24] reports, you probably already have,
[10:25] about how this GPU-to-CPU relationship
[10:27] is going to change. Right now, it's
[10:28] about 4:1. Some people project it might
[10:30] be 7:1 the other way here pretty soon.
[10:33] I want to be clear. It's because the x86
[10:36] ecosystem, the x86 CPUs don't execute
[10:39] these tools as efficiently as they
[10:40] could. That's not what they're built to
[10:42] do. They're built to do a lot of great
[10:43] things. They're not great at executing
[10:45] these tools that the LLMs are giving
[10:47] them.
[10:48] So, we're coming out with a brand new
[10:49] CPU. We call it Nvidia Vera. Standalone
[10:53] CPU servers. This is not just serving as
[10:56] a head node for GPUs.
[10:58] It's going to be almost almost twice as
[11:01] efficient as these x86 CPUs in executing
[11:03] these tools. So, that's how you're going
[11:05] to bust through this bottleneck.
[11:06] You're not going to You know, you can
[11:08] bust through the bottleneck if you want
[11:10] by buying more CPUs and flipping that
[11:12] CPU-to-GPU ratio if you want. Or, you
[11:14] can buy a better CPU to do it.
[11:17] And that's exactly what Nvidia Vera is
[11:18] going to do. And we're partnering very
[11:20] closely with HPE, releasing the ProLiant
[11:22] DL394 system later this year.
[11:26] And the reason why we were able to bust
[11:27] through this bottleneck mostly is how
[11:29] we've architected and built this new
[11:32] NVIDIA Vera CPU. It's very powerful
[11:35] cores,
[11:36] very fast memory access, very consistent
[11:39] memory access compared to x86. It's
[11:42] built on a monolithic die, which means
[11:44] it's built on a single sheet of metal.
[11:46] A lot of x86 CPUs have these chiplet
[11:48] designs where they're stacked on top of
[11:49] each other and that can give you very
[11:50] inconsistent memory access and slow down
[11:52] your system, not to mention give you
[11:54] very unpredictable performance when it
[11:56] comes to memory access. So,
[11:58] we're focusing on building a CPU that
[11:59] can bust through all that and give you
[12:01] very fast tool calling so you can kind
[12:02] of fix this bottleneck issue of sending
[12:05] tokens over to the to the, you know, the
[12:07] token consumption layer,
[12:09] which is the CPUs. That's something I
[12:11] didn't point out on the previous slide,
[12:14] in the top right, no inferencing. We're
[12:17] not running inferencing on the CPU here.
[12:19] We're running inferencing on the GPU
[12:21] systems. That's where you get the most
[12:22] efficient token generation,
[12:25] and you want to focus the CPU on
[12:26] executing those tool commands as much as
[12:29] possible.
[12:31] That's why we built NVIDIA Vera CPU. I
[12:33] want to show a quick, you know, video of
[12:35] how this comes together just to kind of
[12:36] visualize it a little bit. So, we're
[12:37] going to have Vera Rubin GPUs
[12:41] generating the tokens and then separate
[12:44] Vera servers,
[12:46] standalone servers, executing the tool
[12:50] calls.
[12:51] So, that's how we're going to be
[12:52] bringing this together here later this
[12:54] year.
[12:55] Now, that's the hardware side of the
[12:57] equation. Let's talk about the software
[12:58] side of the equation a little bit. So,
[13:00] we have something called NVIDIA Agent
[13:01] Toolkit,
[13:02] and it's focused across all these
[13:03] different layers, you know, optimizing
[13:05] open models, optimizing harness
[13:08] execution, giving you tools and skills
[13:10] because harnesses rely very heavily on
[13:13] tools and skills. You want those skills
[13:15] to be verified and safe as you're using
[13:17] them with your harnesses. We're focused
[13:19] very much on that, too, and is also on
[13:20] runtime and execution. One of the secret
[13:22] sauces of NVIDIA,
[13:24] you know, we build amazing GPUs. We're
[13:26] going to be building amazing CPUs. We
[13:28] build amazing networking equipments. Our
[13:31] infrastructure layer is is world-class.
[13:34] But, what separates Nvidia, in my
[13:35] opinion, from others is our software
[13:37] layer and how we optimize token
[13:39] generation, how we interact with these
[13:41] different open players, we optimize
[13:42] actual tools that people are using to
[13:44] give you the most efficient systems
[13:46] possible. And we're focused here mostly
[13:48] again on how we make the entire, you
[13:49] know, agentic throughput
[13:51] uh faster.
[13:52] And in this case, with the agentic
[13:54] toolkit,
[13:55] we have a lot of different kind of tools
[13:56] and and skills on the right here uh that
[13:58] we're trying to get executed out to be
[14:00] able to make this whole agentic workflow
[14:02] much more efficient.
[14:04] And here's what's inside of it. It's a
[14:06] bit of an eye chart. I'm going to focus
[14:07] on two separate components
[14:09] to kind of show you how we're focusing
[14:10] in on that agentic throughput, not just
[14:13] efficient open model, you know,
[14:14] execution.
[14:16] Although we're very very very focused on
[14:17] making sure these open models are as
[14:19] performant as possible on the GPU
[14:20] systems. Uh two of those are going to be
[14:23] Nemo Relay and Nemo Switchyard.
[14:26] Nemo Relay In both these products, what
[14:28] they're going to be trying to be focused
[14:29] on is how to allocate your resources the
[14:32] most efficient way possible. What I mean
[14:33] by that is not every prompt, not every
[14:36] action your agent is executing needs
[14:38] frontier level intelligence.
[14:40] Not everything that's executing needs a
[14:42] 10 trillion parameter model. Some of
[14:44] these actions can be offloaded to like a
[14:45] 30 billion parameter model or 120
[14:47] billion parameter model. Those are much
[14:49] more efficient models for what you're
[14:51] trying to get done.
[14:52] And that's what Nemo Relay is really
[14:54] trying to think about is how can you
[14:56] offload some of these more less complex
[14:59] tasks to models that can be executed
[15:01] faster and much more efficiently, so
[15:03] your overall throughput is just that
[15:04] much faster. And we have some data here
[15:06] that shows how how Nemo Relay is like
[15:08] reducing costs, it's you know, it's it's
[15:10] improving uptime. Um
[15:13] but again, in this world of tokenomics
[15:14] and reducing costs for agentic AI,
[15:16] anything you can do to reduce that
[15:18] overall cost is incredibly important.
[15:20] So, we have some numbers here coming out
[15:21] where we're able to reduce the overall
[15:22] cost in the entire workflow, not just
[15:24] the token generation part of it.
[15:26] SwitchYard is a very similar product
[15:28] where again, you have a complex prompt
[15:30] coming in. SwitchYard is going to be
[15:32] able to determine, okay, does this need
[15:33] to go that frontier level model? Do I
[15:35] need to offload this to a Fable or or or
[15:37] or ChatGPT 5.6?
[15:39] Or can I offload it to maybe some of my
[15:41] on-prem resources?
[15:43] Uh and run it on a NeMo Triton 120B or
[15:46] name your your model of choice for an
[15:48] on-prem deployment. And so, we're trying
[15:50] to focus again on the overall agentic
[15:52] throughput of all of this. And finally,
[15:55] open models.
[15:57] Nvidia has a family of open-source
[15:59] open-weight models called NeMo Triton.
[16:02] And NeMo Triton, it's a 100% trained and
[16:04] released model by Nvidia. And we're not
[16:07] just focused on frontier-level type of
[16:10] type of models, we're focused on a
[16:11] family
[16:12] of models. So, Nano is a 30B, Super is a
[16:15] 120B, Ultra is a 550B model that we just
[16:19] released a few months ago. I interact
[16:21] with these on my own personal machine,
[16:22] and I love them.
[16:23] I'm a little biased, you know, I work
[16:24] for Nvidia, but I I I really enjoy
[16:27] working with these. And so, um and if I
[16:28] can figure it out, by the way, I think
[16:30] you can figure it out. I'm not exactly
[16:31] any uh top-tier engineer coming out of
[16:34] Nvidia. I'm just a little lowly product
[16:36] marketer that's trying to get some of
[16:37] this stuff done, but very, very powerful
[16:39] and a family of models. And this is what
[16:41] I mean when I say like we're trying to
[16:42] enable on-prem local LLM models to be
[16:46] used so you can solve this tokenomics
[16:48] challenge of being able to offload, you
[16:51] know, a lot of these tasks or even the
[16:53] entire the entire process
[16:55] onto on-prem systems to solve this
[16:58] tokenomics issue.
[16:59] And we have a family of models. Again,
[17:01] it's not just about the Super, Nano, and
[17:04] Ultra, but
[17:05] vision models, information retrieval
[17:07] models, safety models, speech models
[17:10] that we come out with. Uh we're really
[17:11] focused on making sure this open-source
[17:13] ecosystem is as robust as possible.
[17:16] And we have have of This is one that's
[17:18] going to be hard to read, but basically
[17:19] this one's trying to look at what are
[17:21] the different benchmarks that we're
[17:22] seeing across NeMo Tron compared to
[17:24] other open source models out there. And
[17:26] if we're not the best, we're pretty
[17:28] close to being the best for all these
[17:30] different benchmarks that are being
[17:31] measured across NeMo Tron for both nano,
[17:33] super, and ultra.
[17:35] And so this is what we're trying to get
[17:36] to. This is the solution where we have
[17:40] tokenomics challenge. It can be 100
[17:41] times more expensive than running a a
[17:43] simple local LLM model. So the solution
[17:46] is to buy on-prem systems and use local
[17:49] LLMs with those systems. Use all the
[17:51] great technologies here from NVIDIA and
[17:53] our partners over at HPE to help you
[17:55] build on-prem systems
[17:57] that can help you run local LLMs because
[17:59] the Again, the variable cost
[18:02] of running
[18:03] a local LLM for each token that is
[18:05] generated is zero because these open
[18:07] models, another word for that, is free.
[18:10] You can run these local models on-prem.
[18:13] Another data point I'd like to throw out
[18:14] there is the difference in efficiencies
[18:16] between our generations. I mentioned the
[18:18] the the
[18:19] architectural, you know, differences and
[18:21] the performance differences that we have
[18:22] between Blackwell and Rubin. If we take
[18:25] a step back and look at Blackwell
[18:26] compared to Hopper,
[18:28] the compute differences between
[18:30] Blackwell and Hopper is about 2x,
[18:32] but the token generation efficiency was
[18:35] 50x.
[18:36] That's because of a lot of different
[18:37] architectural, you know, changes that we
[18:38] put in, software improvements that we
[18:41] have of Blackwell versus Hopper. So it's
[18:42] not just about compute improvements on
[18:45] gen, it's about actual token efficiency.
[18:48] And this is why you want to be running
[18:50] token generation on GPUs.
[18:53] Specifically, NVIDIA GPUs because of how
[18:56] much efficiency we're building into this
[18:58] over time.
[19:00] And And something else we'd like to
[19:01] point out, you know, we're not just an
[19:02] infrastructure company. We we we we
[19:04] focus on what we call the five-layer
[19:05] cake. And this is all the things you
[19:07] need to put together to release AI
[19:08] correctly. I'm going to go a little bit
[19:09] faster because I'm running out of time
[19:10] here. But we have an AI factory story as
[19:13] well, and this is how you basically
[19:15] architect being able to release like a
[19:17] full stack into the into the ecosystem.
[19:19] You have data and electricity being
[19:20] input in on the bottom, and the output
[19:22] on the top is intelligence or tokens.
[19:24] And everything in between is what NVIDIA
[19:27] and HPE are putting together to make
[19:28] sure we can actually deliver for you an
[19:30] easy way to, you know, build an an
[19:33] on-prem AI factory for your needs.
[19:35] Uh HPE has a very similar view on this
[19:37] and how we have data and models being
[19:38] adjusted into the AI factory, creating
[19:40] the AI at scale, and then and that's
[19:42] deployed across the entire ecosystem.
[19:44] And we have we have a very consistent
[19:46] rhythm with which we're coming out to
[19:47] market. One thing about NVIDIA, we're
[19:48] very open about our road map.
[19:51] We We generally don't keep anything
[19:52] secret. So, we have Blackwell, you know,
[19:55] being graduated into Rubin this year.
[19:57] Same thing with our Grace CPUs coming
[19:59] into Vera and finally GPUs coming out
[20:01] next year
[20:03] or the year after that. And then uh with
[20:05] our partners here at at HPE, again,
[20:07] they're the ones They're They're going
[20:08] to bear the operational burden
[20:10] of bringing this to market for you.
[20:12] They're the ones that are going to be
[20:13] able to take our technology, be able to
[20:15] install it, and you know, and run those
[20:16] on-prem systems to help solve that
[20:18] tokenomics challenge. And they have a
[20:19] number of different a wide range of
[20:21] solutions. And something I keep in mind
[20:23] about HPE, they're a very trusted name
[20:25] in bringing, you know, safe, secure
[20:27] systems on-prem so they can help solve
[20:28] that security challenge while also
[20:31] they're known for building some of the
[20:32] most powerful systems in the world. They
[20:34] hold an 18 benchmarks in MLPerf. They're
[20:36] constantly at, you know, number one or
[20:37] number two at the top of the top 500
[20:40] list for supercomputers around the
[20:41] world. Very effective company at
[20:43] building incredibly powerful systems
[20:45] that work on-prem.
[20:47] And a great software stack as well.
[20:49] Something that they provide that I went
[20:50] through in my in my security talk, that
[20:52] talks about how HPE provides a lot of
[20:54] observability for these agents as well.
[20:58] And we have a lot of different great
[20:59] value props from HPE as well, how they
[21:01] help solve that tokenomics challenge,
[21:03] help you build secure systems, give you
[21:04] observability of your agents, by the
[21:06] way. Something we didn't talk about
[21:07] today, or at least in this talk, is how
[21:09] important it is to track what's
[21:10] happening with all your agents inside
[21:12] your organization. HPE gives you the
[21:14] tools to do that. And I believe they're
[21:16] one of the only companies that can
[21:17] really do that. They're working really
[21:18] closely with us to make sure we can
[21:19] spawn safe agents in the first place,
[21:21] and they're very, very deeply integrated
[21:23] co-engineering, co-developing systems
[21:25] with NVIDIA.
[21:27] So, that's all the time I have for
[21:27] today.
[21:28] Thanks. And if you have any questions,
[21:30] I'm going to stand outside for a second
[21:32] to be able to talk further. Thanks for
[21:33] your time.
[21:34] >> [applause]
