Nvidia Stein Says AI Factories Essential as Agent Costs Surge
Ai4
Enterprise AI agents consume up to 100 times more tokens than chatbots, forcing companies to deploy on-premises infrastructure to avoid budget depletion and security breaches.
## Background
Published on YouTube by the Ai4 channel on August 17, 2026, this session features Steve Stein from NVIDIA Product Marketing. The presentation focuses on how enterprise AI agents drive operational value, while detailing strategies to mitigate risks related to token consumption costs and security vulnerabilities using HPE and NVIDIA solutions.
## TL;DR
Steve Stein outlines the shift from generative chat to autonomous agentic AI, which combines LLM intelligence with execution harnesses to perform complex business workflows. While agents promise massive ROI, their iterative loops dramatically increase token consumption by 5x to 30x and introduce significant security risks like prompt injection and credential exposure. To address these bottlenecks, Stein advocates for on-premises AI factories utilizing open-weight local models, model routing, NVIDIA NeMo Guardrails, and HPE observability tools to make agent deployments predictable, secure, and cost-effective.
## Atmosphere & style
The presentation carries an urgent, executive, and highly strategic tone, opening with a stark warning that companies lacking AI factories risk obsolescence within years. Stein speaks with authoritative clarity and structured precision, framing complex technical architectures into crisp, digestible metaphors like 'the brain' and 'the body.' The brief session moves at an energetic pace, balancing high-stakes economic projections with practical architectural solutions, creating a compelling primer for his subsequent deep-dive sessions.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=4s) Companies without access to an AI factory risk becoming obsolete within a few years as enterprise AI becomes mandatory.
- [01:07](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=67s) AI has evolved from Deep Learning and Perception AI into Generative and Agentic AI, with Physical AI expected as the next future frontier.
- [01:07](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=67s) Enterprise value realization from AI investments experienced a significant surge, jumping from 1% to 28% year-over-year.
- [03:36](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=216s) An agent is defined structurally as a combination of an LLM serving as the reasoning brain and a harness acting as the execution body.
- [03:36](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=216s) Unlike standard text-based chat, agentic loops shift token usage as computers become the primary consumers and generators of tokens.
- [06:46](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=406s) Agentic tasks introduce severe tokenomics challenges, consuming on average 47,000 tokens per job compared to ~500 tokens for standard chat.
- [06:46](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=406s) Security risks and broad access in long-running agentic loops have led to 65% of enterprises operating agents reporting security incidents.
- [11:43](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=703s) Deploying open-weight local models inside on-premises AI factories helps mitigate token costs by driving variable token costs toward zero.
- [11:43](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=703s) Enterprise solutions like NVIDIA Vera CPUs, model routing to smaller language models, and NeMo Guardrails provide the performance, efficiency, and sandboxing required for secure agent execution.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=4s) Introduction and the Imperative of AI Factories
Steve Stein introduces himself as a product marketing professional from NVIDIA. Although this presentation is brief, he frames it as an essential overview of agentic AI—defining where the industry stands, distinguishing this moment from prior Generative AI eras, and teasing critical operational challenges around token costs and security. Stein notes that he will dive much deeper into these topics during two dedicated sessions scheduled for later in the day and the following day.
To emphasize the urgency of the moment, Stein presents a stark data point for enterprise leaders: organizations that fail to build or access an AI factory risk ceasing to exist within a few short years. Engaging directly with agentic AI is no longer optional, making the rapid establishment of AI infrastructure a top strategic priority.
- **Speaker:** Steve Stein (Product Marketing, NVIDIA)
- **Core Warning:** Companies without access to an AI factory risk becoming obsolete within a few years.
---
[01:07](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=67s) The Evolution of AI: From Perception to Useful Agentic Systems
Tracing the historical evolution of artificial intelligence, Stein maps out the major paradigm shifts over the past decade and a half:
1. **Deep Learning & Perception AI (2012–Present):** Beginning around 2012 with image recognition breakthroughs like AlexNet, deep neural networks spawned an era focused on perception, image classification, and smart voice assistants (e.g., Amazon Alexa, Google Home).
2. **Generative & Agentic AI (Current Era):** Labeled by Stein as the true "unlock for enterprise value," this stage transitions systems from simple classification to actively generating useful business information and executing complex workflows.
3. **Physical AI (Future Horizon):** The next boundary, where autonomous decision-making moves into robotics—first simulated inside virtual environments and then deployed into the real world.
Stein argues that we have officially entered the era of "useful AI." Citing platform metrics like GitHub code pushes, he highlights a massive surge in 2026 where developers are heavily leveraging AI not merely to generate text paragraphs for websites, but to produce operational production code running global infrastructure. Furthermore, survey metrics show a dramatic shift in enterprise ROI: while only 1% of organizations reported receiving tangible value back from their AI investments a year ago, that figure has spiked to 28%.
```
+-----------------------------------------------------------------+
| EVOLUTION OF AI ERAS |
+-----------------------------------------------------------------+
| 2012: Deep Learning (AlexNet, Image Recognition) |
| │ |
| ▼ |
| Perception AI (Voice Assistants: Alexa, Google Home) |
| │ |
| ▼ |
| Generative & Agentic AI ("Useful AI", Autonomous Operators) |
| │ |
| ▼ |
| Physical AI (Simulated Robotics & Real-World Decision-Making) |
+-----------------------------------------------------------------+
```
- **Economic Impact:** AI is projected to add $22 trillion in economic value globally by 2031.
- **Enterprise Reach:** 40% of employees in Global 2000 companies are expected to interact with AI agents by the end of the current year.
- **Value Realization Spike:** Enterprise AI ROI reports increased from 1% to 28% year-over-year.
Stein contrasts previous generative paradigms with the agentic approach. The ChatGPT wave in 2022 was driven by human-to-chatbot text interactions. The subsequent phase in 2023–2024 focused on Retrieval-Augmented Generation (RAG) to supply context and mitigate hallucinations. However, in both prior eras, humans were the sole consumers of tokens. Agentic AI shifts this fundamentally: computers are now generating *and* consuming tokens in continuous, autonomous operational loops.
At a practical level, Stein defines an agent through a clear structural metaphor:
$$\text{Agent} = \text{Large Language Model (LLM)} + \text{Harness}$$
The LLM serves as the "brain," excelling at reasoning, planning, and writing code, but remaining inherently powerless on its own. The "harness" acts as the "body"—the runtime environment capable of invoking tool calls, maintaining persistent memory, executing terminal commands, and interacting with enterprise software (such as generating slide decks or querying APIs).
- **Harness Examples:** Tools and frameworks such as OpenClaw, Hermes, Codex, and Claude execution harnesses.
- **Compute Distribution:** LLMs process token generation primarily on GPU clusters, while execution harnesses handle system calls and logic primarily on CPU infrastructure.
---
[06:46](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=406s) The Dual Challenges: Tokenomics and Security Risks
Despite the immense utility of agents, two major scaling bottlenecks threaten to derail enterprise deployments:
1. **The Tokenomics Problem:** Agents require iterative reasoning loops, system prompts, memory injection, and multiple tool iterations per query. While a standard human chatbot conversation consumes around 500 tokens (costing ~$0.04), an agentic task averages **47,000 tokens** and can consume between 5 to 30 times (and occasionally up to 100 times) more tokens per task. Unbounded feedback loops between the LLM and the harness can cause corporate token budgets to be wiped out in months.
2. **Security & Governance:** Long-running, broad-access agents expand the attack surface. Issues include prompt injection, credential exposure, ungoverned APIs, shadow agents, and rogue execution loops. Stein notes that 65% of organizations using agents have already experienced a security-related incident over the past year.
- **Task Token Multiplier:** Agents consume 5x to 30x (up to 100x) more tokens per task compared to static chat prompts.
- **Average Token Consumption:** Standard chat interactions utilize ~500 tokens (~$0.04), whereas agent interactions average ~47,000 tokens per job.
- **Incident Rate:** 65% of enterprises operating agents report having experienced security incidents.
---
[11:43](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=703s) On-Prem AI Factories and Technical Solutions
To solve both token cost unpredictability and security risks, Stein advocates for building **on-premises AI factories**. Deploying open-weight local models on owned infrastructure changes the marginal cost structure: once hardware is capitalized, the variable cost of each newly generated token drops to virtually zero dollars. Additionally, on-prem setups prevent sensitive operational data and corporate credentials from leaking to external model vendors.
NVIDIA and HPE provide an integrated hardware and software stack to make local agentic enterprise deployments practical:
- **Hardware Acceleration:** Utilizing GPU clusters optimized for rapid token generation alongside advanced CPUs—specifically highlighting the **NVIDIA Vera CPU**—to eliminate host-side bottlenecks in tool calling and token consumption.
- **Model Routing & Small Language Models:** Software layer optimizations that dynamically route low-complexity prompts to smaller models (e.g., 30B or 100B parameter models) rather than running every query through massive multi-trillion parameter systems, resulting in drastic compute savings.
- **Open-Weight Ecosystem:** Open models (such as Nemotron and Gemma) are rapidly catching up to closed alternatives (like ChatGPT or Claude) in intelligence and are already outpacing closed models in raw token generation volume.
- **Security & Observability (NVIDIA NeMo Guard & HPE):** Tools like **NVIDIA NeMo Guardrails** spawn agents inside secure, sandboxed environments with strict explicit whitelisting. Complementing this, HPE provides enterprise observability software to track token usage, control tool access, and monitor agent behavior across the infrastructure.
- **Upcoming Detailed Sessions:**
- **Security Deep-Dive:** Today at 3:00 PM.
- **Tokenomics & On-Prem AI Factories Deep-Dive:** Tomorrow at 4:00 PM.
## Notable quotes
> The Agentic Enterprise: Unlocking AI Value Without Breaking the Token Budget or the Trust Boundary
> Steve Stein - Product Marketing, NVIDIA
> [03:36](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=216s) Agent = Large Language Model (LLM) + Harness — Steve Stein
## Mentioned
Steve Stein, NVIDIA, HPE, ChatGPT, AlexNet, Amazon Alexa, Google Home, GitHub, Global 2000, OpenClaw, Hermes, Codex, Claude, NVIDIA Vera CPU, Nemotron, Gemma, NVIDIA NeMo Guardrails
## Video description
Steve Stein - Product Marketing, NVIDIA
AI agents are moving enterprise AI from simple question-answering to autonomous operators that can plan, reason, use tools, coordinate sub-agents, and execute work across business systems, creating major value in productivity, service quality, and operational speed. But as agents scale, two challenges can undermine that value: tokenomics and security. Long-running reasoning loops, memory, retrieval, and tool calls can make inference costs unpredictable, while prompt injection, exposed credentials, shadow agents, ungoverned APIs, and weak observability expand the enterprise attack surface. This session will show how organizations can capture the value of agentic AI responsibly by starting HPE and NVIDIA solutions enabling safe agent creation, using on-prem systems with Local LLMs, using HPE observability to track agents, enforcing zero-trust access, governing agent identity and tools, and preserving human oversight for agents.
Nvidia Stein称随着智能体成本激增，AI工厂至关重要
企业级AI智能体消耗的Token数量最高可达聊天机器人的100倍，迫使企业部署本地基础设施以避免预算耗尽和安全违规。
Steve Stein概述了从生成式对话向自主智能体AI的转变，该技术将LLM的智能与执行框架相结合，以执行复杂的业务工作流。尽管智能体有望带来巨大的投资回报率，但其迭代循环使Token消耗量显著增加5到30倍，并引入了诸如提示词注入和凭证泄露等重大安全风险。为解决这些瓶颈，Stein主张利用开放权重本地模型、模型路由、NVIDIA NeMo Guardrails以及HPE可观测性工具构建本地AI工厂，从而使智能体部署更具可预测性、安全性和成本效益。
NVIDIA 专家 Steve Stein 警告，未能建立或接入 AI 工厂的企业可能在数年内被淘汰；随着 AI 进入实用阶段，企业从 AI 投资中实现切实价值的比例已从去年的 1% 激增至 28%。 (0:04)
Agent 在架构上由作为“大脑”的 LLM 与负责工具调用、持久记忆的“身体”Harness 构成，推动计算机成为自主循环中 Token 的主要生成者与消耗者。 (3:36)
Agent 任务平均每次作业消耗约 47,000 个 Token，远超传统聊天的 500 个；复杂的长程运行循环不仅推高推理成本，还导致 65% 的部署企业报告了安全事件。 (6:46)
通过在本地 AI 工厂部署开源权重模型，企业可将边际 Token 可变成本降至接近零，并可结合 NVIDIA NeMo Guardrails 沙箱与 HPE 可观测性系统构筑安全边界。 (11:43)
Reading summary
NVIDIA product marketer Steve Stein argues that enterprise adoption of agentic AI delivers substantial operational value but introduces critical token cost and security bottlenecks that require specialized infrastructure and governance.
The Shift to Useful and Agentic AI (1:07)
Stein outlines an AI evolution starting with 2012 deep-learning perception models and progressing through generative chatbots and retrieval-augmented generation toward autonomous agentic systems, with physical robotics marking the future horizon. He asserts that the enterprise sector has entered a phase of practical utility, highlighting a surge in developers generating production code for global infrastructure in 2026. According to cited survey metrics, enterprises reporting tangible ROI from AI investments rose from 1% to 28% year-over-year. Projections presented also indicate that AI will generate $22 trillion in global economic value by 2031, with 40% of Global 2000 employees expected to interact with AI agents by year's end.
Agent Architecture: Brains Paired With Execution Harnesses (3:36)
Stein defines an agent using the structural formulation that an agent equals a large language model combined with a runtime harness. Unlike earlier models where humans acted as the sole token consumers, agentic architectures operate in continuous loops where software both generates and consumes tokens. The LLM serves as the reasoning engine for planning and writing code, running primarily on GPU clusters, while the harness functions as the operational body executing terminal commands, managing tool calls, and maintaining persistent memory across CPU environments using frameworks such as OpenClaw, Hermes, Codex, or Claude execution harnesses.
Escalating Tokenomics and Operational Security Vulnerabilities (6:46)
The agentic framework creates severe economic and security hurdles. Standard human chat prompts consume roughly 500 tokens at approximately $0.04 per task, whereas autonomous agents average 47,000 tokens per job due to iterative reasoning loops, tool interactions, and memory injections. Stein notes that agents generally consume 5 to 30 times—and up to 100 times—more tokens than static queries, creating risks that corporate budgets could deplete in months. Additionally, extended autonomy expands the enterprise attack surface through prompt injection, exposed credentials, ungoverned APIs, shadow agents, and rogue loops, with 65% of enterprises operating agents already reporting a security incident.
Mitigating Cost and Risk Through On-Premises Infrastructure (11:43)
To address variable token costs and data exposure, Stein recommends deploying on-premises AI factories using open-weight models such as Nemotron and Gemma. Under capitalized local infrastructure, the marginal cost per generated token approaches zero dollars while preventing proprietary data leaks to external vendors. Stein highlights combined hardware and software approaches from NVIDIA and HPE, including dynamic model routing that diverts low-complexity tasks to smaller models (e.g., 30B or 100B parameters), the NVIDIA Vera CPU to alleviate host-side tool bottlenecks, NVIDIA NeMo Guardrails for secure execution sandboxing, and HPE observability tools to monitor agent behavior and token usage.
Stein cautions that organizations without an accessible AI factory face obsolescence within a few years, leaving enterprise deployment contingent on resolving the tension between autonomous utility, uncapped token expenses, and infrastructure isolation.
## Background
该视频由频道“Ai4”于 2026 年 8 月 17 日发布在 YouTube 平台，主讲人为 NVIDIA 产品营销负责人 Steve Stein。内容聚焦于企业如何从被动问答转向具备自主规划与执行能力的智能体企业，并重点探讨如何借助 NVIDIA 与 HPE 的本地化解决方案应对 Tokenomics 成本失控与安全边界失守两大核心挑战。
## TL;DR
随着 AI 进入实用主义与智能体时代，交互范式已彻底从“人类消耗 Token”转向“计算机自主消耗 Token”的闭环，大幅拉动了生产力。然而，由 LLM 大脑与 Harness 执行体构成的智能体也带来了成本黑洞（单任务 Token 消耗激增数十倍）与安全失控风险（超六成企业遭遇安全事件）。演讲指出，解决这一困境的关键在于构建本地“AI 工厂”，结合 NVIDIA Vera CPU、NeMo Guard 安全沙箱以及 HPE 的可观测性管理平台，在将边际推理成本降至零的同时守牢企业安全信任底线。
## Atmosphere & style
整场演讲开门见山、危机感十足，开篇便以“不建 AI 工厂的企业几年内可能消失”的技术现实奠定了极具紧迫感的基调。Steve Stein 的表达干练务实且条理分明，用详实的数据对比和软硬件架构图撕开云端按量计费的算力陷阱。观看过程既能让管理者对失控的 Token 账单与越权安全风险产生警惕，又能从软硬件协同的本地化全栈解法中获得笃定的落地信心。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=4s) 构建 AI 工厂已成为企业生存的紧迫要求，企业 AI 正从被动的问答生成加速跃迁为具备自主行动与执行能力的“智能体企业”。
- [01:07](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=67s) AI 已经全面跨入“实用 AI”时代，GitHub 生产级代码提交量激增，AI 从演示玩具进化为实质性提升交付速度与生产力的核心工具。
- [02:53](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=173s) 企业从 AI 获取投资回报的比例从 1% 激增至 28%，其技术本质在于交互范式从“人类阅读消耗 Token”转变为“计算机生成并消耗 Token”的自主机机循环。
- [04:54](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=294s) 智能体的工程架构可拆解为“LLM 大脑 + Harness 躯体”，LLM 负责规划决策并在 GPU 上生成 Token，Harness 则在 CPU 端调度 API 并执行实际业务操作。
- [06:52](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=412s) 智能体落地面临 Tokenomics 成本失控与安全失控的双重陷阱，智能体单任务 Token 消耗量通常是传统对话的 5 至 30 倍甚至 100 倍，且 65% 的企业已遭遇过安全事故。
- [09:45](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=585s) 面对复合任务中平均超 47,000 Token 的单次调用底噪，本地部署（On-Prem）开源模型能将边际 Token 成本降至零，避免公有云 API 账单被长程循环击穿。
- [12:36](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=756s) NVIDIA Vera CPU 专为智能体的工具调用与 Token 消费端进行硬件加速，打通 CPU 执行环境与 GPU 推理之间的吞吐瓶颈。
- [12:36](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=756s) 通过模型路由将琐碎任务卸载到轻量级小模型，配合 Nemotron、Gemma 等开源权重模型的快速演进，为本地高效运行智能体奠定了算法基础。
- [14:28](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=868s) NVIDIA NeMo Guard 提供沙箱隔离与白名单规则，严格限制智能体在未授权环境下的 API 与凭据访问，杜绝流氓智能体越权外泄数据。
- [14:28](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=868s) NVIDIA 与 HPE 联合打造全套本地 AI 工厂与可观测性平台，实时监控智能体行为拓扑与 Token 消耗效率，确保在安全边界与预算可控下释放生产力。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=4s) 智能体企业与 AI 工厂的生存紧迫性
NVIDIA 的产品营销负责人 Steve Stein 在开场时直言不讳地指出了当前企业面临的残酷现实：如果企业现在还不着手构建自己的“AI 工厂”（AI Factory），或者无法接入 AI 工厂，那么“可能在短短几年内就会不复存在”。这绝非危言耸听，而是技术范式剧烈转换下的生存法则。当前整个产业正处于一个极其特殊的历史节点，企业不仅需要积极拥抱 AI，更需要深刻理解它如何正在从最初的生成式对话演进为真正具备自主行动力的“智能体企业”（Agentic Enterprise）。
尽管这是一场简短的概览演讲，但他明确表示，后续还将通过两场分论坛进一步深入拆解智能体带来的技术挑战与落地路径。核心议题非常清晰：企业必须认清什么是智能体 AI（Agentic AI），当下到底正在发生什么，以及如何以实用、可控的方式将其落地到实际业务运转当中。
### [01:07](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=67s) 从 AlexNet 到实用主义 AI 的演进历程
回顾现代人工智能的发展路径，技术演进经历了清晰的阶段跃迁。2012 年 AlexNet 在深度神经网络图像识别上的突破，正式拉开了这一轮技术革命的序幕。随后技术进入了感知 AI（Perception AI）时代，深度学习开始广泛赋能图像识别，并催生了以 Alexa、Google Home 为代表的智能语音交互设备。而当技术跨入生成式 AI 乃至今天的智能体 AI 阶段时，真正的企业级价值才被彻底解锁——AI 不再只是被动识别模式，而是开始主动生成对业务运营、组织决策真正有价值的实用信息。未来，AI 还将进一步迈向具身物理 AI（Physical AI），在虚拟环境中模拟机器人行为，并最终让机器人在现实物理世界中自主决策。
当前业界已经全面跨入“实用 AI”（Useful AI）时代。过去人们对生成式 AI 是否真能创造实际价值的疑虑已经被彻底打破。来自 GitHub 代码提交的数据给出了有力印证：2026 年迎来了爆发式的代码推送增长，开发者们正在大规模借助 AI 编写、部署实际运行在各行各业的生产级代码，而不再仅是用它生成网页上的营销文本段落。交付速度与生产力被提升到了前所未有的高度。
- **2012 年 AlexNet 诞生**：开启深度神经网络与感知 AI 时代，代表应用包括图像识别与智能音箱。
- **物理 AI 的终局愿景**：从数字世界的智能体进化为物理世界具备自主决策能力的机器人。
- **2026 年代码提交激增**：GitHub 上由 AI 驱动的代码提交量呈现断崖式上升，AI 从玩具全面演变为生产力工具。
### [02:53](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=173s) 范式转移：从“人类消耗 Token”到“计算机消耗 Token”
多项重磅宏观数据印证了商业价值的实质性回流。据预测，到 2031 年，AI 将为全球经济贡献高达 22 万亿美元的经济价值；全球 2000 强企业中，40% 的员工在今年年底前就会直接与智能体进行协作。更关键的变化发生在投资回报率上：去年各项调研显示，全球仅有 1% 的企业表示能从其 AI 投资中看到切实回报，而今年这一比例大幅跃升到了 28%。虽然这仍意味着约四分之三的企业尚未收回成本，但从 1% 到近三成的飞跃，足以证明实用 AI 正在释放巨大的经济动能。
这种爆发背后的技术本质，是 Token 消耗主体发生的根本性转移。2022 年 ChatGPT 掀起的 GenAI 浪潮，是人类用户与聊天机器人对话并阅读其返回的文本；随后的技术迭代重点在于引入检索增强生成（RAG），为大模型注入上下文以减少幻觉。但在这两个阶段，Token 的最终消费者始终是人类自己。智能体 AI 则完全重构了这一逻辑：计算机自身开始消耗 Token。
```mermaid
flowchart TD
accTitle: Generative AI vs Agentic AI Token Flow
subgraph GenAI["传统生成式 AI / RAG"]
A["人类提出 Prompt"] --> B["大语言模型 LLM"]
B --> C["人类阅读并消耗 Token"]
end
subgraph AgenticAI["智能体 AI 闭环"]
D["目标任务输入"] --> E["LLM 大脑生成规划与代码"]
E -->|GPU 生成 Token| F["Harness 执行体调用工具"]
F -->|CPU 消耗 Token / 执行结果反馈| E
F --> G["执行外部业务系统操作"]
end
```
这种“计算机生成 Token、计算机消耗 Token”的自主循环，解锁了工具调用（Tool Calling）、特定技能执行以及带持久化记忆的任务编排系统。然而，它在极大提升自治能力的同时，也推高了推理成本，并放大了企业的受攻击面。
- **经济价值预测**：到 2031 年为全球经济贡献 22 万亿美元。
- **价值兑现率剧增**：企业从 AI 投资中获取回报的比例从去年的 1% 激增至今年的 28%。
- **核心差异**：交互范式从“人机对话”转变为“机机循环”，计算系统兼具 Token 的生产者与消费者双重身份。
### [04:54](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=294s) 智能体的工程解构：LLM 大脑与 Harness 身体
究竟什么是智能体？从最接地气的工程实现角度来看，智能体的架构可以归结为一个极其直观的公式：选定的大语言模型（LLM）加上外挂的执行底盘（Harness）。LLM 负责扮演智能中枢的角色，用于生成思考计划、行动策略和可执行代码，但它本身无法对外部系统发起实际操作。这就像人类的大脑：大脑能够进行复杂的构思与推演，但如果脱离了躯体，它什么也做不了；只有为大脑接上身体，才能跑动、搬运或敲击键盘。
Harness 正是智能体的“躯体”。它负责实际运行环境指令、调度外部 API、执行生成好的代码，甚至为你自动生成一份 PowerPoint 演示文稿。在工程实践中，像 OpenClaw、Hermes、Codex 以及特定模式下的 Claude 都可以被视作具备动作执行能力的 Harness。在底层硬件协同上，这两者有着清晰的分工：LLM 主要部署在 GPU 算力系统上负责生成 Token，而各类 Harness 和工具执行逻辑则主要运行在 CPU 端。由于 GPU 生成 Token 的速度极快，系统架构中频繁在 CPU 工具调用与环境执行之间产生瓶颈，这种系统分层架构将对整个工作流的性能带来深远影响。
### [06:52](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=412s) 智能体落地的双重陷阱：Tokenomics 崩溃与安全失控
伴随智能体能力释放而来的，是两个可能瞬间吞噬业务价值的核心瓶颈：Tokenomics（Token 经济学）与安全性。关于 Tokenomics 的失控，市场上关于企业在短短几个月内就烧光全年 Token 预算的新闻屡见不鲜。其原因在于，智能体每处理一项任务所生成的 Token 量通常是传统对话的 5 到 30 倍，NVIDIA 内部的实测数据甚至显示某些复杂任务会飙升至 100 倍。由于智能体拥有自我修正机制，LLM 制定方案后交由 Harness 执行，执行结果再回传给 LLM 进行复盘调整，这种长程推理循环在没有人工干预或熔断机制的情况下，甚至可能无限循环下去，导致推理账单呈指数级失控。
另一个致命挑战是安全性。近期媒体频繁报道了智能体脱离控制在企业内网“暴走”（Rogue Agents）的案例。智能体可能在缺乏审计的情况下静默调用内部敏感数据、暴露凭据或引发严重的系统漏洞。数据显示，在过去一年常规使用智能体的企业中，高达 65% 的企业已经遭遇过与智能体相关的安全事件。这表明安全防护绝非空中楼阁，而是关乎组织信任的当务之急。对于这两大痛点，最彻底的破局方案在于“将 AI 工厂搬回本地”（On-Prem）。
- **Token 暴增乘数**：单任务平均产生 5 至 30 倍 Token，极限情况下可达到常规对话的 100 倍。
- **长程死循环隐患**：缺乏熔断的智能体会持续在“规划-执行-回传-调整”中循环消耗算力。
- **安全事故频发**：已有 65% 的受访企业在部署和使用智能体时遭遇了明确的安全事故。
### [09:45](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=585s) Tokenomics 成本失控背后的数学账与本地化解法
通过具体数字对比，传统对话与智能体运作之间的成本鸿沟令人触目惊心。人类与 ChatGPT 或 Claude 这类聊天机器人进行一次常规问答，平均消耗约 500 个 Token，折合成本仅约 4 美分；但当智能体开始介入时，它为了理解全局不仅要读入庞大的历史上下文记忆（Memory），还要载入被称为“灵魂文件”（Soul File）的系统角色设定及各种技能规则定义，导致单次交互的平均 Token 消耗直接飙升到 47,000 个左右。更糟糕的是，如果智能体持续处于自主运转状态，这些调用量会无限累积，任何基于公有云 API 按量计费的企业预算都会被迅速击穿。
这也是为什么本地部署（On-Prem）是解决 Tokenomics 难题的核心钥匙。虽然本地搭建 AI 工厂需要一定的前期资本支出（CapEx）或设备融资成本，但由于运行的是本地开源权重模型，其理论边际变量成本（Variable Cost）为零美元——企业每多生成一个 Token，都不再需要向第三方模型提供商支付调用费用。这种成本结构让企业能够放开手脚去跑高频长程的复杂智能体工作流。Steve Stein 特别强调，自己在日常工作中从三月起就深度依赖智能体来处理高强度事务，智能体作为“GenAI 2.0”，其生产力价值是无可置疑的，前提是不能被高昂的云端账单卡住脖子。
| 交互形态 | 典型任务 Token 规模 | 预估单次基础成本 | 计费特性与潜在风险 |
|:--- |:--- |:--- |:--- |
| **传统聊天机器人 (GenAI 1.0)** | 约 500 Tokens (单次问答) | 约 4 美分 ($0.04) | 线性成本，单次交互完结，成本可预测 |
| **自主智能体 (GenAI 2.0)** | 平均 47,000+ Tokens (复合长程) | 随循环与工具调用激增 | 指数级累加；记忆与设定文件拉高底噪，易陷无限循环 |
| **本地 AI 工厂 (On-Prem)** | 自主长程海量 Tokens | 边际成本趋近 $0 (固定 CapEx) | 彻底规避云端按量计费陷阱，适合超高频自动化任务 |
### [12:36](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=756s) 软硬件协同架构：Vera CPU、小模型卸载与开放生态
要实现高吞吐、低成本的本地智能体落地，需要针对智能体工作流重构计算硬件体系。NVIDIA 不仅通过其 GPU 系统提供极致的 Token 生成效率，还针对智能体的工具调用（Tool Calling）与执行瓶颈推出了专用的 CPU 架构。特别是 NVIDIA Vera CPU，专门针对智能体环境中的 Token 消费端进行硬件加速，打通 CPU 执行环境与 GPU 推理之间的吞吐瓶颈。此外，软件层面的模型路由优化同样至关重要：许多琐碎的智能体中间提示根本不需要 10 万亿参数级别的超大模型，完全可以智能卸载到 300 亿（30B）或 1000 亿（100B）参数的轻量级模型上处理，从而成倍缩减计算开销。
同时，开源模型生态正在迎来历史性拐点。从数据趋势来看，开源权重模型所生成的 Token 总量正在快速追赶甚至反超以 ChatGPT、Claude 为代表的闭源商业模型。以 NVIDIA Nemotron 和 Google Gemma 为代表的开放模型正在以惊人的速度演进，其智力水平已经完全足以支撑企业本地复杂智能体的运行。这种高质量开放模型的爆发，为本地自主运行智能体奠定了坚实的算法基础。
- **NVIDIA Vera CPU**：专为智能体工具调用与环境交互优化的全新处理器，打破 Token 消费瓶颈。
- **动态模型卸载（Offloading）**：根据任务复杂度在不同体量模型间灵活分发，大幅压低能耗与算力消耗。
- **开源模型崛起**：以 Nemotron、Gemma 为代表的开源权重模型在生成吞吐与智力表现上正逐步超越闭源系统。
### [14:28](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=868s) 安全沙箱防护与 HPE 联合赋能的本地生产力闭环
在安全性维度，本地化部署同样提供了天然的信任边界。针对智能体失控与权限滥用风险，NVIDIA 推出了核心安全组件 NVIDIA NeMo Guard（基于 NeMo Guardrails 构建的安全机制）。它能让智能体在完全受控的沙箱隔离环境（Sandbox）中启动生成；除非管理员显式配置了白名单规则，否则沙箱内的智能体绝对无法触碰任何未授权的外部 API、系统凭据或内网环境，从底层杜绝了“流氓智能体”引发的越权与数据外泄。
这一整套本地智能体生态的落地，离不开 NVIDIA 与 HPE 的深度联合工程研发。在这套协同体系中，NVIDIA 负责提供领先的 GPU 算力、Vera 处理器与 NeMo 安全框架，而 HPE 则带来了全套企业级 AI 工厂解决方案与可观测性（Observability）管理平台。借助 HPE 的工具链，企业能够实时监控本地所有运行中智能体的行为拓扑，清晰了解哪些智能体正在创造价值、哪些在浪费 Token，以及是否有任何未授权的工具调用企图。HPE 还承担了本地机房准备、系统集成部署与全生命周期的运维服务，帮助企业在坚守安全底线与财务预算的前提下，真正释放智能体企业的商业潜能。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=4s) 如果企业现在还不着手构建自己的“AI 工厂”，或者无法接入 AI 工厂，那么“可能在短短几年内就会不复存在”。 — Steve Stein
> [00:04](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=4s) 核心议题非常清晰：企业必须认清什么是智能体 AI（Agentic AI），当下到底正在发生什么，以及如何以实用、可控的方式将其落地到实际业务运转当中。 — Steve Stein
> [01:07](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=67s) 当前业界已经全面跨入“实用 AI”（Useful AI）时代。过去人们对生成式 AI 是否真能创造实际价值的疑虑已经被彻底打破。 — Steve Stein
> [02:53](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=173s) 这种爆发背后的技术本质，是 Token 消耗主体发生的根本性转移……智能体 AI 则完全重构了这一逻辑：计算机自身开始消耗 Token。 — Steve Stein
> [04:54](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=294s) 究竟什么是智能体？从最接地气的工程实现角度来看，智能体的架构可以归结为一个极其直观的公式：选定的大语言模型（LLM）加上外挂的执行底盘（Harness）。 — Steve Stein
> [06:52](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=412s) 智能体每处理一项任务所生成的 Token 量通常是传统对话的 5 到 30 倍，NVIDIA 内部的实测数据甚至显示某些复杂任务会飙升至 100 倍。 — Steve Stein
> [06:52](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=412s) 数据显示，在过去一年常规使用智能体的企业中，高达 65% 的企业已经遭遇过与智能体相关的安全事件。 — Steve Stein
> [09:45](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=585s) 这也是为什么本地部署（On-Prem）是解决 Tokenomics 难题的核心钥匙。虽然本地搭建 AI 工厂需要一定的前期资本支出（CapEx）或设备融资成本，但由于运行的是本地开源权重模型，其理论边际变量成本（Variable Cost）为零美元 — Steve Stein
> [12:36](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=756s) 许多琐碎的智能体中间提示根本不需要 10 万亿参数级别的超大模型，完全可以智能卸载到 300 亿（30B）或 1000 亿（100B）参数的轻量级模型上处理，从而成倍缩减计算开销。 — Steve Stein
> [14:28](https://www.youtube.com/watch?v=Q2n6G5pJiiw&t=868s) 除非管理员显式配置了白名单规则，否则沙箱内的智能体绝对无法触碰任何未授权的外部 API、系统凭据或内网环境，从底层杜绝了“流氓智能体”引发的越权与数据外泄。 — Steve Stein
## Mentioned
Steve Stein, NVIDIA, HPE, Ai4, AlexNet, Alexa, Google Home, GitHub, ChatGPT, Claude, OpenClaw, Hermes, Codex, NVIDIA Vera CPU, NVIDIA Nemotron, Google Gemma, NVIDIA NeMo Guard, NeMo Guardrails
## Video description
Steve Stein - Product Marketing, NVIDIA
AI agents are moving enterprise AI from simple question-answering to autonomous operators that can plan, reason, use tools, coordinate sub-agents, and execute work across business systems, creating major value in productivity, service quality, and operational speed. But as agents scale, two challenges can undermine that value: tokenomics and security. Long-running reasoning loops, memory, retrieval, and tool calls can make inference costs unpredictable, while prompt injection, exposed credentials, shadow agents, ungoverned APIs, and weak observability expand the enterprise attack surface. This session will show how organizations can capture the value of agentic AI responsibly by starting HPE and NVIDIA solutions enabling safe agent creation, using on-prem systems with Local LLMs, using HPE observability to track agents, enforcing zero-trust access, governing agent identity and tools, and preserving human oversight for agents.
[Source](https://www.youtube.com/watch?v=Q2n6G5pJiiw)
Source quotes
[0:04] Steve Stein: 如果企业现在还不着手构建自己的“AI 工厂”，或者无法接入 AI 工厂，那么“可能在短短几年内就会不复存在”。
[0:04] Steve Stein: 核心议题非常清晰：企业必须认清什么是智能体 AI（Agentic AI），当下到底正在发生什么，以及如何以实用、可控的方式将其落地到实际业务运转当中。
[1:07] Steve Stein: 当前业界已经全面跨入“实用 AI”（Useful AI）时代。过去人们对生成式 AI 是否真能创造实际价值的疑虑已经被彻底打破。
[2:53] Steve Stein: 这种爆发背后的技术本质，是 Token 消耗主体发生的根本性转移……智能体 AI 则完全重构了这一逻辑：计算机自身开始消耗 Token。
[4:54] Steve Stein: 究竟什么是智能体？从最接地气的工程实现角度来看，智能体的架构可以归结为一个极其直观的公式：选定的大语言模型（LLM）加上外挂的执行底盘（Harness）。
[6:52] Steve Stein: 智能体每处理一项任务所生成的 Token 量通常是传统对话的 5 到 30 倍，NVIDIA 内部的实测数据甚至显示某些复杂任务会飙升至 100 倍。
[6:52] Steve Stein: 数据显示，在过去一年常规使用智能体的企业中，高达 65% 的企业已经遭遇过与智能体相关的安全事件。
[9:45] Steve Stein: 这也是为什么本地部署（On-Prem）是解决 Tokenomics 难题的核心钥匙。虽然本地搭建 AI 工厂需要一定的前期资本支出（CapEx）或设备融资成本，但由于运行的是本地开源权重模型，其理论边际变量成本（Variable Cost）为零美元
[12:36] Steve Stein: 许多琐碎的智能体中间提示根本不需要 10 万亿参数级别的超大模型，完全可以智能卸载到 300 亿（30B）或 1000 亿（100B）参数的轻量级模型上处理，从而成倍缩减计算开销。
[14:28] Steve Stein: 除非管理员显式配置了白名单规则，否则沙箱内的智能体绝对无法触碰任何未授权的外部 API、系统凭据或内网环境，从底层杜绝了“流氓智能体”引发的越权与数据外泄。
Source quotes
The Agentic Enterprise: Unlocking AI Value Without Breaking the Token Budget or the Trust Boundary
Steve Stein - Product Marketing, NVIDIA
[3:36] Steve Stein: Agent = Large Language Model (LLM) + Harness
Original transcript
[0:04] My name is Steve Stein, but thank you for the
wonderful intro. I'm I work at Nvidia, and today I
[0:08] want to talk to you about this idea of the agentic
enterprise, but more specifically, I want to
[0:11] educate you a little bit on what agents are, what
moment we're in right now, how is this different
[0:16] from the previous, you know, GenAI eras that we've
been in, and then I'm going to introduce a couple
[0:20] major problems with agentic AI, and then I'm going
to tease a couple different solutions. So, this is
[0:26] a pretty short talk. I can't go very deep. I have
two other talks, one today, one tomorrow, that'll
[0:31] go a little bit deeper into these challenges, but
I want to talk to you mostly about what agentic
[0:35] AI is, and how you can use it, and more and more
practically, exactly what's happening with it.
[0:41] And so, I always like to show this data point
that says, if you're not building your AI factory,
[0:48] if you don't have access to an AI factory,
you might cease to exist in a few short years.
[0:53] This is real. You need to be in this moment. You
need to be thinking about how is your company,
[0:58] how are you engaging with AI, more specifically,
agentic AI, as you're getting closer to it,
[1:02] but it is a serious problem that we need to think
about solving and quickly with AI factories.
[1:07] And so, let's talk a little bit about how AI's
evolved a little bit over the years. We started
[1:11] back in 2012, training images to training image
recognition on deep neural nets. That's AlexNet.
[1:18] And from there, we spawned a whole new
era of types of of of evolution of AI.
[1:23] That era evolved into what we now call perception
AI. This is this era of deep learning, recognizing
[1:30] images, interacting with Alexas and Google Homes
and things like that. But this next era, where we
[1:34] got into generative AI, more specifically, agentic
AI, I view that as the unlock for enterprise value
[1:41] because we're actually generating information
that's useful to you, useful to your business,
[1:47] useful to your organization. This is where
we're starting to receive a lot of value back.
[1:52] And eventually, we're going to progress to this
era of physical AI, where we're, one, simulating
[1:58] robotics inside of virtual environments but also
allowing robotics to make autonomous decisions
[2:02] out in the real world. But today, we're going to
zero in a little bit more on this agentic era.
[2:06] Um more specifically, we are now fully in the era
of useful AI. If people have questioned whether or
[2:12] not generative AI was going to be useful or before
that with perception AI, those questions have been
[2:17] answered. What this graph here that you're looking
at here is basically, you know, commits to GitHub,
[2:22] you know, code pushes. And we see this gigantic
push in 2026 that says so many people are using AI
[2:30] to push a lot more code, a lot getting a lot more
things done. And we're not just talking about like
[2:35] generating paragraphs that'll go up on a web page.
We're talking about code that's being generated
[2:39] that's running things around the world. So, and
people are able to get these things done so much
[2:43] faster than they used to. And now we're seeing
this era of useful AI. Some other data points.
[2:53] By 2031, you know, AI will add 22 trillion dollars
of economic value to the world. 40% of us in the
[3:00] Global 2000 companies will interact with agents
by the end of this year, if you haven't already.
[3:06] But a more interesting data point is, you
know, last year we we saw some studies that
[3:10] said only 1% of the world's organizations were
receiving value back from their AI investments.
[3:15] This year, it's gone up to 28%. Now, it's still
only roughly a quarter of the organizations report
[3:20] that they're receiving value back from their
AI investments, but that's a massive spike.
[3:25] And so, we really are in this era now where
we call useful AI. It is adding value back
[3:31] into the economy, adding value back into our
organizations. Uh some other data points here.
[3:36] So, if you look at the different
eras that we've had over the years,
[3:40] so we, you know, back in the the the the the
ChatGPT, you know, the ChatGPT boom of 2022
[3:45] really ushered in this era of generative
AI of us interacting with a, you know,
[3:49] and getting, you know, humans interacting with uh
with chatbots and getting information back. Last
[3:53] year was all about how to make these chatbots and
LLMs a lot more intelligent by injecting it with
[3:57] rag and making them more context aware, reducing
hallucinations. But in both of these eras,
[4:04] guess who was consuming the tokens? It's us,
it's humans. What's different about Agentic AI?
[4:11] It's computers that can consume
the tokens now. Not just humans.
[4:17] So, we have essentially computers generating
the tokens, computers consuming the tokens,
[4:22] you're going to get this loop of work
being done. So, that and that's also going
[4:27] to introduce this new concept of, you know,
tool calling, skills, command executions by
[4:33] by AI now, persistent memory inside the harnesses,
but we also get to unlock, unfortunately, a lot
[4:38] new a lot more attack surfaces and token costs to
come along with that. But, it is incredibly useful
[4:45] for the computers to be consuming the
tokens now as well. So, what is an agent?
[4:54] This is the most practical explanation, the most
practical definition of what an agent really is.
[5:01] An agent is when you have a large language
model of your choice, connect that to a harness.
[5:08] And what's actually happening here, the LLM
is generating the intelligence of what to do,
[5:12] it's generating plans, actions, code, but it can't
execute anything by itself. Think of it like your
[5:18] brain. Your brain is really good at coming up with
a lot of great information, but by itself, inside
[5:23] your head, your brain can't do anything. You need
to attach it to a body to do stuff, and that's
[5:28] what the harness is. It's actually able to execute
code, execute commands, build PowerPoints for you.
[5:33] That's the execution side of this. And what's
also, you know, a lot of people ask questions,
[5:39] okay, great, what is a harness practically? A
harness is Open Claw. A har- A harness is Hermes,
[5:44] Codex, Claude can be considered a harness.
These are the things that are executing actions,
[5:50] not just generating information for
you to consume. It's executing actions.
[5:55] Another data point here, too, is like what systems
are these things actually executing on? So, for
[6:01] LLMs, these are largely GPU systems, generating
the tokens. The harnesses are run on the CPU side,
[6:11] executing the commands. And I'll get more relevant
here in a second when we kind of talked about this
[6:16] agent loop and bottlenecks that exist in the CPU
architecture because of how fast tokens are being
[6:21] generated, but there's different systems now
between token consumption and token generation.
[6:29] That's an important point to kind of
keep in mind. And on top of all of this,
[6:32] of course, here at NVIDIA, we're we're we're
thinking about where are the different skill
[6:36] skill software paths, you know, the
actual different harness optimizations.
[6:39] We're investing in all of this to make the
entire agentic workflow efficient and useful,
[6:46] not just the token generation part of it all. So,
what are some challenges that we have with agents?
[6:52] I think we've all seen the tokenomics
problem. Why is that a problem?
[6:58] I'm sure you've seen all the different
types of news reports out there of companies
[7:01] blowing through their entire yearly budgets of
tokens within months. That's because agents,
[7:09] on average, generate between, you know, 5 to 30
times more tokens per task. We have some data here
[7:15] at NVIDIA that says like that number can climb
to 100 times per task. And that's just per task.
[7:21] Agents, they can they can be long-running.
You can have an LLM come up with a plan,
[7:26] computer executes it, the harness executes it, it
sends the results back to the LLM, it modifies,
[7:30] it sends it back to the harness, and this loop
can continue infinitely if you don't stop it.
[7:37] Each task is more token expensive, and each task
can run infinitely. So, you can see how these
[7:42] token costs can spiral out of control really fast.
Second challenge is security. We've seen some
[7:49] reports in the news lately, too, of how agents
can go rogue, how agents can be running inside
[7:54] your environment without you knowing, how they can
cause security vulnerabilities inside those. Uh we
[7:59] have some major challenges that we need to solve.
And I'm going to go a little bit deeper into both
[8:03] of these for a second, but I'm also going to talk
about solutions. And I'm going to go deeper into
[8:08] solutions in future talks. I'm going to have one
later today, one tomorrow that's going to go a
[8:12] little bit deeper. The one later today is going
to be a little bit deeper on the security side.
[8:16] One tomorrow is going to be a little bit deeper on
the tokenomics side. But these are things that we
[8:19] need to solve. And spoiler alert, the solution's
on prem. If you can build an on-prem AI factory,
[8:27] you can solve both of these. Cuz you can run local
LLMs, which are open weight, technically free,
[8:35] on your systems that you own. Each additional
token generated is technically zero dollars,
[8:42] right? Now, there's some CapEx and other things
you need to think about along the way, but
[8:45] we're not sending money away to other these
other LLM providers. You're generating all the
[8:50] information on prem. Also, if you have your own
on-prem AI infrastructure, your own AI factory,
[8:56] you're able to protect it. And we have a lot
of the technologies here at NVIDIA to help
[8:59] you do that. Uh we're partnering very closely
with HPE. HPE has a wonderful software suite,
[9:05] as well as the ability to operationalize
all this and bring it on prem for you.
[9:09] They're one of our best partners in making sure
you can build up these on-prem AI factory systems
[9:14] so that you can solve both of these challenges.
But these are very real challenges. And something
[9:18] I want to note on the security side is we've
seen a couple different escapes recently,
[9:22] but over the past year, people have been using
agents pretty regularly. And 65% have experienced
[9:29] security-related incidents already. So, this is
not a theoretical problem. This is happening.
[9:34] We need to solve this. We need to fix this for
organizations to trust agents running inside
[9:39] their organizations. We can't have rogue
agents running inside their organization.
[9:43] So, let's talk a little bit about the token
cost first. So, I mentioned before that hey,
[9:46] this agentic inflection point really is here.
We have autonomous agents. AI is useful now.
[9:52] That's causing this huge inference scene boom
because of how many tokens are being generated.
[9:57] And to put some numbers to it, so when you and I
interact with chatbots, if I'm consuming tokens
[10:03] from a chatbot, it costs about 4 cents per
chat. You know, 500 tokens. It's not that
[10:07] much. If you're not familiar with what a token is,
by the way, a token is essentially a word. It's a
[10:12] unit of measurement for how much the AI system is
producing. So, in this case, we're generating 500
[10:17] words in an average conversation with ChatGPT or
Claude or name your own of choice. For an agent,
[10:26] this is where it spikes because an agent is
going to ingest all this memory. It's going
[10:31] to ingest a soul file. It might ingest skills.
And every single task compounds that token cost.
[10:39] And you can be using 47,000 tokens on average per
interaction. Sometimes it can be more, sometimes
[10:47] it can be less. Also, again, tokens, you know,
or the the agents here are running continuously.
[10:52] And as you can see, all the costs can stack up
really, really fast. So, you need to be thinking
[10:57] about on-prem solutions, hopefully from HPE, that
can help you build up these these systems to kind
[11:03] of solve this. But these costs can spiral pretty
quickly. However, agents are incredibly useful.
[11:09] You know, I I like to call you know, you know,
agentic AI, you know, GenAI 2.0 because I feel
[11:14] like now we're in this era where it's we're
actually getting value back again from AI,
[11:18] but it's expensive. There's some security issues
we got to solve, but it's incredibly useful. If
[11:22] you If you've interacted with agents, I've I've
I've been I've been using them since about March
[11:27] or so of this year. Um very regularly. I use them
every single day for a lot of different tasks that
[11:32] I get done and it is incredibly powerful what
it can do now. And so I would recommend you know
[11:36] these if you haven't interacted with an with agent
yet, I highly recommend getting more getting more
[11:43] getting deeper on that. So again,
what's the solution? The solution here
[11:46] is to build on-prem infrastructure because every
if you're running local LLMs, you're running it
[11:52] on your on-prem AI factory, theoretically,
the variable cost of each token is zero.
[11:59] Again, CAPEX, you know, or as financing
costs come to come into play here, but
[12:04] this is a massive win for anyone that's trying to
solve this tokenomics problem is to build up this
[12:09] this on-prem infrastructure. Now, that's that's
not easy, but HPE has the experts in-house,
[12:14] NVIDIA has the experts in-house to help you
build these systems and make them perform it,
[12:18] make them run the LLMs that you want, install the
right harnesses that you want, make your agents
[12:23] safe. There's a lot of different things that
we're working on to to to you know, make on-prem
[12:28] AI factories possible, but that is the solution
here. If you want to solve the tokenomics problem,
[12:33] build on-prem AI factories. And so I'm going to
go through each of these in a later talk, but this
[12:39] is the these are the solutions we have. We have
the GPU systems for incredibly efficient token
[12:46] generation. We have CPUs, NVIDIA has new CPUs for
tool calling. If you haven't heard of NVIDIA Vera,
[12:55] start reading about it. It's an incredible CPU
that's going to help with the tool calling side
[13:00] of the equation. So the token consuming part
of the equation so you kind of bust through the
[13:05] bottleneck and so you get great agentic workflow
happening even faster. We have a lot of software
[13:10] that makes, you know, optimizations of open
source models happen. We have uh you know,
[13:15] things that improve your agentic throughput. We're
thinking about the entire system and how you might
[13:19] even be able to be able to offload models to lower
parameter models for certain prompts that don't
[13:24] require 10 trillion parameter model intelligence.
You can offload those to 30 billion, 100 billion
[13:30] parameter models and get that done. That saves a
lot of cost and we have numbers that show that.
[13:34] Also, open models will become incredibly important
here. If you haven't seen the the charts out there
[13:39] of how many tokens are being generated on
open models versus closed models lately,
[13:44] the open models are starting to overtake the
closed models in terms of how many tokens
[13:48] have been generated using open models recently
versus closed models. When I say closed models,
[13:52] I mean things like chat GPT, you know, Claude.
Those are the open or closed source models and
[13:57] then things like NemoTron, Gemma, these are the
open open weight models and we're starting to see
[14:02] more more to you know, tokens being consumed
with these open source models and so they're
[14:07] becoming incredibly intelligent. Something to pay
attention to and these are some of the topics I'm
[14:11] planning to go through uh tomorrow talking about
these specific topics. So, if you're around
[14:17] tomorrow at I can read that, right? It's
really hard to read. 4:00. Tomorrow at 4:00,
[14:22] I'll be going through a lot of these topics in
a lot more detail so you can learn about them.
[14:28] Second challenge, security. There's two things
we're going to focus on here. One is there's a
[14:34] product called NVIDIA Nemo Guard. If you haven't
heard of that, it will be essentially allows
[14:38] you to spawn safe agents inside of a sandbox
environment. Inside that sandbox environment,
[14:43] they can't access anything unless you explicitly
whitelist it. It's our solution for saying,
[14:47] how do you create an agent that can't go rogue
inside your organization? And there's a number
[14:53] of different AI factory solutions from HPE, by
the way. They're going to give you observability,
[14:58] give you control over your systems. These are
things you can't necessarily get if you're just
[15:03] you know, buying agents or using agents through
other sources. If you can own your own AI factory,
[15:07] HPE also gives you a lot of observability tools
and ability to control what's happening with your
[15:11] infrastructure and now you actually know what's
happening with all your agents, you know which
[15:14] ones are productive, which ones aren't, which
ones are consuming more tokens, which ones aren't,
[15:19] uh which ones might have be accessing tools and
skills you don't want them to, but you have the
[15:23] insight now. HPE is going to give that to you.
So, I'm going to go through that later today.
[15:29] Later today at around 3:00, I'll be back. And
I'm going through a lot more of the security
[15:34] side of this challenge and figuring out how can we
actually solve the security side of the of a GenAI
[15:40] together. Let me go through. And so, here
is um just some some value prop information
[15:47] around around why HPE can solve this for
you. So, they're going to give you systems
[15:52] that help you solve this tokenomics problem,
eliminating this runaway token cost issue.
[15:57] You're also going to have solutions
that are going to allow you to secure
[16:00] your AI factories and give you solutions that are
going to allow you to observe what's happening
[16:04] with all your agents on prem. And all of this
is co-engineered, co-developed with Nvidia.
[16:10] We work very closely together to make sure we're
creating systems that work on prem. And HPE has
[16:14] the operational burden, frankly, of making sure it
actually works on prem with you. And they have a
[16:18] lot of resources, services, installation services,
uh prep services, as well as a lot of different,
[16:25] you know, software tools that will help you get up
and running. So, that's all I have for you today.
[16:30] I want to stop a little bit early so we
can hopefully get get get caught up on
[16:32] on time here. Thanks for having me. I'll be
outside for a second if you have questions.
