Amazon, PayPal Execs See Energy Grid Limits Forcing AI Chip Overhaul

Amazon与PayPal高管：电网极限将倒逼AI芯片架构革新 随着人工智能转向指数级的推理需求爆发，物理电网的限制将耗尽吉瓦级站点容量，迫使开发者用超高效的模块化架构取代依靠蛮力堆砌的算力集群。 在Fusion Fund的Lu Zhang主持下，来自Amazon、PayPal、Aligned和TensorWave的基础设施与工程负责人深入探讨了支撑生成式AI时代所需的技术基石。圆桌讨论详述了从单体模型向模块化、智能体驱动架构的演进，并权衡了公有云、本地部署以及采用替代硬件的新型专有云（neoclouds）之间的战略取舍。尤为关键的是，随着算力需求从训练转向激增百倍的实时推理，发言嘉宾重点分析了日益迫近的物理能源与电网瓶颈。最后，专家们给出了切实可行的建议，敦促AI构建者应立足于具体业务问题进行逆向设计，打造具备适应性的事件驱动型系统，而非为了当下的完美而过度工程化。 随着 AI 从传统单向预测演进为具备自主执行力的代理（Agent）系统，基础设施的重心正在向代理间通信协议、身份验证及安全交易机制转移。 (5:00) 为缓解对单一 GPU 厂商的依赖风险，以 AMD 集群为代表的多元算力与 Neocloud 快速崛起，通过对底层驱动和编译器的抽象封装，降低企业采用异构硬件的门槛。 (12:09) 企业在架构选型时应从具体业务决策倒推，优先考虑混合部署方案：在本地 CPU 上运行任务专用的小语言模型（SLM），并将弹性研发与多模态负载交由云端处理。 (16:46) 制约 AI 扩展的瓶颈已逐渐从纯芯片产能转向电网传输能力、数据中心液冷及电力供给限制，能源的可用性与可持续性正成为大模型持续迭代的关键物理防线。 (27:13)

Scaling artificial intelligence beyond experimental software requires shifting industry focus from front-end models to foundational infrastructure, spanning heterogeneous silicon, hybrid workload hosting, agentic protocols, and physical power constraints. Industry leaders across cloud computing, venture capital, and enterprise commerce argue that energy limits and operational integration friction will increasingly dictate the viability of AI deployments. Architecting Heterogeneous Silicon and Agentic Stacks (5:00) Modern enterprise AI architectures are moving away from monolithic GPU designs toward modular, multi-tier systems. As Rajesh Sura illustrates through Amazon's conversational platforms, resilient stacks separate concerns by pairing orchestration layers (such as LangChain and AWS Bedrock Agents) with specialized hardware, dividing compute between standard GPU clusters and cost-efficient ASICs like AWS Inferentia. High-level software abstractions are emerging to reduce integration friction, enabling developers to bypass low-level compiler and driver hurdles (such as bridging AMD ROCm and Nvidia CUDA ecosystems). Darrick Horton notes that provisioning large-scale AMD clusters offers an essential hedge against single-vendor supply concentration. Beyond hardware diversification, Prakhar Mehrotra argues that generative systems upend traditional machine learning by turning data storage into latent intelligence storage and expanding post-inference requirements. Autonomous models require new foundational rails—including the Model Context Protocol (MCP), identity verification, and 'agent of record' mechanisms—to safely authenticate transactions as autonomous agents interact directly without human intermediaries. Resolving Workload Topology Across Cloud and On-Premises (16:46) The choice between public cloud and on-premises hosting is governed by workload scale, security constraints, and data gravity rather than a simple binary preference. Sura and David Stanfill emphasize that teams should work backward from specific business decisions before deploying capital, noting that routine classification and automation often run more efficiently on high-end CPUs or Small Language Models (SLMs) rather than large models. While public cloud provides unmatched velocity for rapid experimentation and bursting, on-premises infrastructure remains mandatory for latency-critical operations and sovereign AI initiatives where external dependencies are prohibited. For enterprise-scale model workloads, infrastructure alternatives have expanded beyond traditional hyperscalers. Horton highlights that managing clusters of tens of thousands of GPUs introduces extreme networking and total cost of ownership challenges that internal IT teams cannot easily sustain. In response, 'neocloud' providers build specialized hardware topologies, colocation links, and proprietary interconnects—such as Aligned's NeuralSync DPU—to bridge legacy compute with intensive AI pipelines. Physical Grid Limits and the Inference Power Shift (27:13) As computational scaling laws expand from pre-training into inference-time compute and chain-of-thought reasoning, physical energy availability is displacing chip supply as the primary operational bottleneck. Mehrotra describes the generative era as fundamentally converting electricity into tokens, forecasting an inference demand surge that could reach 100 times historical levels. This explosive growth triggers Jevons' paradox, where efficiency gains merely accelerate aggregate power consumption across data centers. This continuous baseload demand creates both strain and opportunity for power utilities. Stanfill notes that steady AI power consumption justifies long-delayed investments in transmission upgrades and clean power generation, including nuclear energy. However, Horton cautions that the industry will soon exhaust readily available stranded power. Once access to multi-gigawatt sites saturates, organizations will face a hard physical ceiling, accelerating the necessity for direct liquid cooling, renewable load balancing, and fundamentally more efficient paradigms like neuromorphic or analog computing architectures. Edge Distillation and the Rise of Agentic Systems (33:02) Deploying frontier models at scale is increasingly tied to model distillation, compressing large foundation models into specialized SLMs capable of local execution. Panelists emphasize that running fleets of dozens of local SLMs on consumer hardware will require a fundamental redesign of mobile operating systems and device hardware architectures. This migration also raises unresolved commercial questions regarding whether the edge inference energy cost will be borne by technology providers or end consumers. Looking ahead over a three-year horizon, panelists forecast a rapid shift toward embodied and agent-driven workflows. Sura expects automated systems to converge around real-time, event-driven loops, while Mehrotra anticipates autonomous 'agentic commerce' outnumbering human-driven web traffic. Stanfill predicts agentic models will cross over into physical robotics, while Horton projects extreme organizational leverage, forecasting that five-person startups using dense AI enablement will attain billion-dollar valuations and scale revenue in record time. Long-term infrastructure sustainability hinges on resolving the severe friction between model development velocity and physical world limits. While algorithmic scaling laws remain unbroken, enterprise success will ultimately depend on whether power grids, interconnect hardware, and modular software architectures can adapt to an exponential explosion in inference workloads.

由Fusion Fund合伙人Lu Zhang主持的技术研讨指出，人工智能若要从概念热潮转化为可持续的商业生产力，核心诉求必须转向底层基础设施建设。来自Amazon、PayPal、TensorWave及Aligned的技术专家共同认为，长效AI基础设施的关键在于解耦硬件依赖、构建务实的混合云拓扑、克服电网物理约束，并为自主智能体（Agent）建立可信的身份与通信协议。 异构算力扩张与技术栈解耦 (5:00) TensorWave首席执行官Darrick Horton指出，整个行业过度依赖Nvidia GPU带来了严重的芯片供给瓶颈与商业集中度风险，因此TensorWave专注于构建基于AMD硬件的大规模GPU集群，旨在推动硬件层面的多元化竞争并降低算力门槛。在具体工程架构上，Amazon数据工程负责人Rajesh Sura介绍了其对话式AI系统的模块化分层方案：利用AWS Bedrock集成基础模型，采用OpenSearch作为向量数据库、Titan处理密集向量，并将计算任务按需分流到GPU集群与高性价比的AWS Inferentia专有ASIC芯片上，同时依赖Prometheus、Grafana和SageMaker模型注册表构建全栈可观测性。 面向重度企业级负载，Aligned首席技术官David Stanfill强调，企业AI不仅处理文本对话，更涵盖LiDAR点云、地质传感数据与超算模拟等多模态重型负载，这促使Aligned研发了专有的NeuralSync DPU（数据处理单元），以打通传统计算流水线与现代AI集群的高吞吐数据流。Horton与Lu Zhang一致强调，降低企业整合成本的关键在于高阶软件抽象层；企业开发者不应被迫处理AMD ROCm或Nvidia CUDA等底层驱动与编译器的异构差异，抽象软件是实现跨芯片平滑部署的必要抓手。 混合云拓扑与工作负载决策分流 (16:46) 针对公有云、本地数据中心与新型“新云”（Neocloud）之间的架构权衡，Amazon的Rajesh Sura建议企业放弃盲目训练或托管超大规模模型的冲动，坚持从业务决策需求逆向推导架构设计。Sura主张常规分类与自动化任务应优先采用可在高端CPU上高效运行的小语言模型（SLM）或模块化Agent框架，建立兼顾合规与超低延迟的本地部署，同时利用公有云的弹性资源处理突发性算力需求，形成高互操作性的混合云架构。 关于数据主权与硬件规模，PayPal全球AI负责人Prakhar Mehrotra指出，涉及国防、国家主权AI或需自主掌握合规底层（如跨多国法律的专用金融大模型）的实体必须构建本地超算集群，但主流非模型研发型企业通过提示工程和上下文工程即可创造业务价值，依靠云端更利于紧跟科学扩展律的快速演化。在规模界限上，TensorWave的Horton给出了明确参数：数十至数百张GPU的推理或轻量微调完全可由企业内部精简团队在本地掌控；但当集群规模达到1,000至100,000张GPU时，运营难度与总体拥有成本（TCO）急剧升高，这使得专注于大规模调优的专业Neocloud成为高性价比的替代路径。 电力转换为代币与电网物理天花板 (27:13) PayPal的Prakhar Mehrotra从物理本质剖析称，生成式AI的核心是“将电力转化为代币（Tokens）”。信息时代仅利用电力传输静态比特，而智能时代则耗费电能计算衍生出全新智能；随着扩展律向推理期计算延伸，受杰文斯悖论（Jevons' paradox）影响，推理端的能耗倍数预计将激增至以往的100倍。当前制约算力调用和导致API限流的根本瓶颈，已经从单纯的先进制程芯片短缺，转移到了物理数据中心容量、液冷散热能力以及电网输电负荷的硬性制约。 针对电网承受力，Aligned的David Stanfill提出了相对乐观的产业视角：AI数据中心产生的全天候持续稳定基荷需求，为公用事业部门升级输电基础设施提供了长期的确定性资本流入，并加速了核能与清洁能源的商业化周期。然而，TensorWave的Horton警告称，利用现有闲置电力的红利期即将结束，一旦全美难以寻获具备1吉瓦（1GW）供电能力的站点，能源硬天花板将不可避免地逼迫工程界放弃“单纯增加电力”的粗放模式，转而倒逼架构颠覆，例如推动号称可降低100倍能耗的模拟计算及实现1000倍能效跃升的神经拟态计算结构商业化落地。 智能体信任机制与端侧系统重塑 (33:02) 随着AI从静态单次推理全面转向自主多步执行的Agent时代，系统工程的重心正在向代理交互安全迁移。PayPal的Prakhar Mehrotra指出，传统机器学习仅向人类提供辅助决策建议，而在由数百万自主Agent执行真实交易的网络中，核心挑战在于确立“主体记录代理”（Agent of record）、身份认证机制以及模型上下文协议（MCP）等交互标准，以确保自主商业交易的可靠性与合规性。 在边缘交付层面，从庞大前沿模型中蒸馏出的小语言模型（SLM）部署正引发底层计算重构。Mehrotra与Lu Zhang探讨指出，未来移动终端若需常驻数十个专用SLM协同运作，将迫使操作系统架构经历系统级重写，同时衍生出端侧推理能耗账单应由用户还是服务商承担的商业问题。面向未来创业范式，Horton预测随着AI杠杆率的提升，行业将出现5人规模便实现10亿美元估值与1亿美元年度经常性收入（ARR）的超级微型团队；Sura与Stanfill则强调，构建系统应优先保障模块化演进能力与真实物理场景（如具身智能与机器人）的契合度，避免在缺乏业务实际支撑的情况下盲目堆砌AI组件。 该讨论表明，AI基础设施的竞争重心正在由单一的GPU算力堆砌，转向由异构硅芯片、混合云架构、电网能源拓扑及Agent信任协议构成的综合系统工程。然而，边缘端侧计算的成本归属机制、超大规模1吉瓦级供电站点的物理短缺，以及跨生态自主智能体通用安全标准的缺失，仍是全行业在追求真正AGI与生产力落地的道路上亟待解决的关键难题。

## Background 本内容由 Ai4 频道于 2025 年 8 月 29 日发布在 YouTube 平台上。视频记录了一场由 Fusion Fund 创始人兼管理合伙人 Lu Zhang 主持，汇聚了 Amazon、TensorWave、Aligned 与 PayPal 行业专家的圆桌讨论，核心探讨如何从硬件算力、MLOps 软件架构、能源瓶颈以及商业落地的全方位视角，构建兼具扩展性、高能效比与长远生命力的 AI 基础设施。 ## TL;DR 本场圆桌围绕“智能体时代（Agentic Era）”下 AI 基础设施的范式跃迁展开了深入剖析。嘉宾们指出，企业 AI 落地正在从简单的对话式 UI 转向深度业务工作流的端到端自动化，架构设计必须从具体业务决策逆向推导，优先保障系统的可靠性、模块化解耦与可观测性，破除盲目追求单一大模型的迷思。同时，讨论重点聚焦于“算力转化为 Token”背后的物理硬约束，剖析了超大集群运维、公有云与新兴算力云（Neo-Cloud）的分流、电网负荷与散热瓶颈，并展望了未来神经形态计算、类脑架构、Agent 间通信协议以及 5 人极简超级团队的爆发式商业前景。 ## Atmosphere & style 整场研讨会节奏紧凑且干货密集，充斥着硬核技术洞察与犀利商业现实的碰撞。主持人 Lu Zhang 控场敏锐、提问切中要害，而来自巨头大厂、新兴云厂商与风投视角的嘉宾们各抒己见，既有对万卡集群工程复杂度和加州停电等现实“阵痛”的坦率调侃，又有对人脑能效比、核能复兴与 Scaling Law 终局的宏大思辨。现场交流务实、理性且极具前瞻性，带给观众一种在物理极限与智能爆发交汇点上，冷静洞察未来技术范式转移的沉浸感。 ## Key takeaways - [05:24](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=324s) 企业级对话式 AI 架构正在向模块化和事件驱动演进，结合高弹性无服务器计算、专用向量库与混合算力芯片（如 GPU 与 AWS Inferentia 协同调度）以优化端到端 TCO。 - [07:17](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=437s) 从传统 ML 到大模型时代，数据存储本质从存储客观信息转变为高压缩比的智能表征；同时推理层催生了面向自主代理的身份认证（Identity）、交互信任（Trust）与通信协议（如 MCP）等新型基础设施需求。 - [11:17](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=677s) 打破单点硬件垄断至关重要，以 AMD 为代表的替代算力方案通过底层软件栈的高级抽象屏蔽异构性，让非 AI 原生企业无需感知芯片细节即可落地业务。 - [12:06](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=726s) 企业 AI 项目在迈向生产环境时常因管理层急躁和缺乏长远架构规划而受挫；技术评估必须计入系统集成与理解成本（脑力成本），并坚持“针对可靠性构建优于盲目追求扩展性”。 - [19:14](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1154s) 算力选型应坚决破除参数规模迷信，常规确定性业务与分类任务可下放至运行在高端 CPU 上的专用小语言模型（SLM）或轻量智能体框架，避免滥用昂贵稀缺的 GPU 集群。 - [20:21](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1221s) Scaling Law 依然充满生机并向推理端扩展（Inference-Time Scaling）延伸；企业自建本地机房（On-Prem）的核心驱动力已从数据隐私转向战略级模型主权与摆脱供应链依赖。 - [25:26](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1526s) 基础设施呈现 On-Prem、超大规模公有云与专业新兴算力云（Neo-Cloud）三层格局；数十至数百卡级集群适合本地或公有云管理，而千卡至万卡以上的超大规模集群则高度依赖 Neo-Cloud 的深度调优与 TCO 控制。 - [27:14](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1634s) 算力能耗正在从模型训练侧向推理侧发生上百倍的剧烈爆发（符合杰文斯悖论），推动全模态无感架构、废热回收、液冷技术以及基于 AI 的跨区域动态绿电负载调度的普及。 - [38:00](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2280s) AI 带来的确定性基荷用电需求反向倒逼核能复兴与国家级电网改造，但未来 1GW 级供电硬瓶颈将全面倒逼底层模型向神经形态计算（降耗 ~1000x）与模拟计算等极高能效方向突破。 - [41:13](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2473s) 未来三年核心趋势：智能体商业（Agentic Commerce）爆发、微型团队（如 5 人初创团队）依靠 AI 杠杆创造数十亿美元估值，以及基础设施向事件驱动、具身机器人及 Agent-to-Agent 通信协议栈深度演进。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=4s) 圆桌讨论开场与嘉宾背景介绍 主持人 Lu Zhang（Fusion Fund 创始人兼管理合伙人）为整场讨论定下基调：当前行业虽然对 AI Agent 和各种 AI 落地应用讨论热烈，但从根本上讲，要让 AI 成为兼具卓越性能与高成本效益的方案，核心在于构建具有可扩展性且在成本与能耗上均可持续的 AI 基础架构（AI Infrastructure）。 随后各位嘉宾依次介绍了各自的业务重心与技术方向： * **Darrick Horton**（TensorWave 联合创始人兼 CEO）：TensorWave 是一家专注于 AMD 生态的云服务提供商（AMD CSP），致力于构建以 AMD GPU 为主的大规模 AI 算力集群，专门支撑大模型的训练、分布式推理及微调任务。 * **David Stanfill**（Aligned CTO）：Aligned 拥有自营数据中心并构建了 Neo-Cloud。团队自主研发了 NeuralSync DPU 定制互连芯片以调控 AI 集群内部及进出的专用数据流，专注于服务具备定制化基础设施需求、需要将既有高性能计算（HPC）架构与数据流水线无缝迁移到 AI 的企业和科研团队。 * **Prakhar Mehrotra**（PayPal 全球 AI 负责人）：作为应用型 AI 企业，PayPal 目前核心聚焦于“Agent 时代”（Agentic Era）的基础设施演进——当互联网上涌现数以百万计代表用户执行任务的自主代理时，如何在推理层之上建立身份认证（Identity）、信任（Trust）以及代理记录（Agent of Record）机制。 * **Rajesh Sura**（Amazon 数据工程与分析负责人）：拥有 16 年以上云原生 AI 及分析平台构建经验，目前主管 Amazon 北美电商业务的对话式 AI。他强调“对话式 AI 绝不仅仅是一次 UI 界面升级，而是一场端到端决策工作流自动化的范式转变”。 Lu Zhang 在总结时也特别提及了 Amazon 新近发布的 Nova 基础模型，并指出更多超大规模云厂商（Hyperscalers）在模型层注资能够让生态更加活跃健全；而作为管理着逾 5 亿美元规模的风投机构，Fusion Fund 过去十年间跨越硬件、推理及系统架构等多个层级广泛布局底层设施，重点赋能创业者在竞争与合作中平衡性能与单位成本。 --- ### [05:24](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=324s) Amazon 的对话式 AI 全栈云架构 Rajesh Sura 深入剖析了 Amazon 内部构建对话式 AI 的底层技术体系。该架构整体采用“模块化（Modular）”与“事件驱动（Event-Driven）”相结合的设计哲学，打通了从交互前端到推理硬件的全链路： ```mermaid flowchart TD accTitle: Amazon Conversational AI Stack subgraph Frontend_Backend["交互与编排层"] A["React 前端 UI"] --> B["API Gateway"] B --> C["TypeScript Lambda 函数"] C --> D["LangChain 编排引擎"] D --> E["Amazon Bedrock Agents<br/>(推理步骤链式调用)"] end subgraph Core_Data["模型与数据存储层"] E --> F["Amazon Bedrock 平台"] F --> G["Amazon Titan 嵌入模型"] F --> H["OpenSearch 向量数据库"] F --> I["Amazon S3 对象存储"] end subgraph Compute_Infra["算力与可观测性"] F --> J["混合算力集群"] J --> K["GPU 算力集群"] J --> L["AWS Inferentia<br/>(成本优化推理芯片)"] J --> M["可观测性套件<br/>(Grafana + Prometheus + Model Registry)"] end ``` 该技术栈包含以下核心组件： - **模型与推理编排**：依托 AWS Bedrock 构建生成式 AI 能力，结合 Amazon Titan 嵌入模型与 OpenSearch 向量数据库；通过 LangChain 进行 LLM 编排，并配合 Bedrock Agents 实现复杂推理步骤的链式拆解与执行。 - **业务中台与前端**：前端采用 React 框架确保交互体验与安全性，后端深度依赖 API Gateway 和基于 TypeScript 的 AWS Lambda 无服务器函数。 - **存储与混合算力**：采用高弹性的 Amazon S3 作为底层对象存储应对业务波峰；在计算端创新性地将传统 GPU 计算集群与专用低成本推理芯片 AWS Inferentia 进行混合调度。 - **全生命周期可观测性**：系统从第一天起就内置了 Grafana 监控看板、Prometheus 时序指标追踪，以及 Amazon Model Registry（用于模型版本追溯、血缘管理与运行状态监控）。 --- ### [07:17](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=437s) 从传统 ML 到 Agentic 时代的范式跃迁 Prakhar Mehrotra 梳理了 AI 基础设施在生成式 AI 与大模型出现前后的本质演进。他指出传统机器学习与 AI 基础设施通常划分为四大板块，而大模型的引入重构了每一个环节： | 基础设施模块 | 传统 ML 时代架构 | 生成式 AI / 大模型时代演进 | |:--- |:--- |:--- | | **数据存储 (Data)** | 存储结构化/非结构化的客观**信息 (Information)** | 存储**智能 (Intelligence)**：大模型本质是高压缩比的智能表征 | | **模型训练 (Training)** | 经典分布式训练流程 | 超大规模参数训练，必须在架构中引入深度的**检查点 (Checkpoints)** 容错机制 | | **指标优化 (Optimization)** | 传统指标（如准确率、召回率）调优 | 围绕模型综合推理能力与资源消耗的权衡优化 | | **推理层 (Inferencing)** | 基于过去十年成熟的**水平扩展 (Horizontal Scaling)** 架构 | 受到 GPU 硬件物理特性的强约束，需将传统分布式传感理念转化为分布式智能调度 | Prakhar 进一步强调，在推理层之上正在诞生一个全新的“Agency 代理层”。传统 ML 系统只是向人类提供一条决策建议（由人类完成最后一公里执行），而在 Agentic 时代，AI 代理必须代表人类在互联网上自主执行复杂任务。这一转变直接催生了对**身份认证（Identity）**、**交互信任（Trust）**、**代理间通信协议**（如 Model Context Protocol）以及工具调用（Tool Usage）的全新基建需求。这正是 PayPal 过去 20 年在电商信任领域沉淀能力并正在全力攻坚的下一代网络底座。 接续此话题，David Stanfill 补充指出，除了对话式文本与常规音视频外，工业界正面临海量专有的多模态数据挑战（如 LiDAR 激光雷达点云数据、岩土工程与地理空间数据等）。这类数据体量极为庞大，在融入 AI 流水线的前后均需要极其密集的计算预处理与后处理支持。 ### [10:25](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=625s) 超越对话式 AI：多模态、实时智能体与复杂计算对底层架构的新要求 David Stanfill 深入探讨了企业 AI 基础设施演进的深层挑战。他指出，业界目前看到的很多早期落地场景主要集中在对话式 AI、CRM 和销售系统等与人类交互极其紧密的领域；但企业如果想要将 AI 基础设施拓展至更深层的业务系统，就必须支持那些与现有重度算力基础设施、超级计算机以及仿真模拟系统紧密交织的复杂负载。这类任务无论在数据侧、训练侧，还是在以实时智能体（Agentic）形式运行的推理侧，都有着截然不同的严苛诉求。 在这种高阶架构中，检索增强生成（RAG）的生命周期早已脱离了“从知识库检索零散文本片段”的初级阶段。它所面对的往往是规模极其庞大、结构极其复杂的数据集检索，甚至需要与运行中的其他进程和底层系统进行深度实时交互。因此，构建现代 AI 基础设施的核心命题，在于如何让底层架构既能支撑轻量级的对话交互，又能无缝承载高吞吐、多模态与复杂计算交织的深层企业级任务。 --- ### [11:17](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=677s) 打破单点垄断：为市场提供 AMD 等多元化算力替代方案 针对底层基础设施的供给现状，TensorWave 联合创始人兼首席执行官 Darrick Horton 表明了其鲜明的差异化立场。他直言，TensorWave 的核心定位是向市场规模化交付替代性算力基础设施，“我们刻意完全不碰 NVIDIA”。这一决策背后有着深刻的市场逻辑：无论是大型基座模型构建者、传统企业、初创团队还是小型 AI 实验室，在算力基础设施层拥有多元化选择都至关重要。 Darrick Horton 强调，推动以 AMD（以及未来更多算力硬件提供商）为代表的多元算力进入生态，不仅能够打破单点依赖，更能直接为最终用户带来更高的性价比与商业价值。让算力获取真正实现民主化，让客户不再受制于单一芯片巨头的供货与定价约束，是构建健康行业生态的基石。 --- ### [12:06](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=726s) 应对“高管催促”：非原生 AI 企业的落地困局与抽象层化繁为简 主持人张璐（Lu Zhang）向 Darrick Horton 和 David Stanfill 抛出了一个极其现实的商业落地难题：像 AMD 或云巨头这类前沿技术虽然成熟，但大量传统企业并不具备 AI 原生基因，既不懂最新架构，也没有懂技术的专家团队对接，基础设施服务商究竟该如何帮助这类客户平稳落地？ ```mermaid flowchart TD accTitle: 传统企业落地 AI 面临的断层与抽象解决路径 subgraph Gap["企业内部断层"] A["董事会 / 管理层指令:<br/>'我们需要 AI 战略'"] --> B["业务/技术团队在公有云上<br/>小规模快速试水实验"] B -.->|"缺乏生产环境规划导致脱节"| C{"落地生产瓶颈:<br/>性能失真 / 成本失控 / 架构崩溃"} end subgraph Solution["基础设施抽象破局"] D["高级抽象平台层<br/>屏蔽底层驱动与硬件异构性"] D --> E["业务按需分流:<br/>敏感负载在本地/托管，突发上云"] end C ==> D ``` David Stanfill 指出，非 AI 优先的传统企业最常遇到的陷阱是“管理层脱节”。董事会或 C 级高管往往直接下达“我们必须有 AI 战略、必须赶紧上线项目”的行政命令。执行团队领命后，匆忙寻找现成的公有云工具开始做原型实验，却从未通盘考虑过“一旦实验成功，部署到生产环境会是什么样子”。这种缺乏长远架构规划的做法，正是无数企业 AI 项目在迈向生产时彻底翻车的原因。服务商前期的核心价值，就是为客户理清边界：哪些部分适合放在公有云快速试错，哪些核心链路必须部署在本地数据中心或自主可控的托管机房内。 Darrick Horton 对此深有共鸣并幽默地还原了客户的典型场景：许多企业揣着董事会的死命令跑过来，张口就是“嘿，我们想买点 AI”。但深究下去，他们既不清楚自己要解决的具体业务问题，更不知道实现路径。他指出，服务商的责任不仅是稳定、可靠、可扩展地运营底层机架，更在于提供极致简洁的高层抽象：“这才是最伟大的平衡器”。只要底层软件栈将复杂的 AMD 驱动、NVIDIA 驱动及底层细节彻底屏蔽，客户只需要对接统一的接口来解决业务问题，他们根本不在乎底层具体是哪家厂商的芯片在运转。 --- ### [15:13](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=913s) 降维适配与教育客户：技术成本之外的“脑力与集成成本” 张璐对两人的观点表示高度赞同，并提出了技术商业化中极具洞察的成本构成理论。她指出，评估新技术方案时，企业不能只看账面上的硬件或软件采购成本，更要核算“集成成本”。这种成本不仅包含真金白银的资本投入，更包含了沉重的“脑力成本”——客户根本不想费心弄清后台复杂的架构逻辑，他们要的只是稳定可靠的交付结果。 张璐分享了 Fusion Fund 自 2018 年起建立的 CXO 智囊网络（汇聚了 45 位来自财富 1000 强企业的首席技术官）的真实变化：大约两年前开始，这些 CTO 频繁接到 CEO 和董事会成员的催促电话，被要求在董事会上专门汇报“AI 驱动的数字化转型进展”。在强大的外部压力下，企业往往在“自主研发还是对外采购（Buy vs. Build）”之间摇摆不定，通常会先拿出预算在内部小范围自研试水，随后才逐步明确哪些自研、哪些采购第三方。因此，基础设施供应商在早期保持足够的耐心，承担起客户教育的角色并做好利益绑定，是建立长期深厚客户信任的生命线。 --- ### [16:45](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1005s) 云端、本地与混合架构博弈：从业务决策反推与“先可靠后扩展” 随后，话题转向了行业内争论不休的核心架构选择：公有云 vs. 本地部署（On-prem）。鉴于敏感行业的合规限制、数据隐私要求以及各异的基础设施成熟度，张璐邀请来自亚马逊的 Rajesh Sura 分享混合架构的平衡之道。 Rajesh Sura 首先向初涉 AI 的企业与初创团队提出了一条中肯建议：“不要一上来就急着造大模型”。企业应当停下盲目投入的脚步，先退后一步深度拆解：“AI 究竟需要替你做出什么具体的决策？”从目标业务决策进行逆向推导，充分利用现存丰富的开源工具链，避免过早陷入资产与厂商锁定。更关键的是，开发架构时必须“优先针对可靠性构建，而非盲目追求可扩展性”——因为 AI 的输出本身存在不可控的幻觉、偏差与不确定性，只有在系统稳定可靠之后，扩展规模才有实际意义。 在架构选型上，Rajesh 细致拆解了不同形态的利弊： - **公有云环境**：为追求极高实验迭代速度的 AI 研发团队提供了无与伦比的弹性伸缩能力、动态算力分配与全球化网络覆盖； - **本地部署（On-prem）**：牢牢契合严苛监管行业的数据合规要求，并能完美满足极低时延的边缘与核心任务； - **混合部署（Hybrid）黄金平衡点**：将极其敏感的核心负载约束在本地受控环境中运行，同时在面临高 CPU/GPU 算力峰值时无缝向云端突发溢出（Bursting）。而这套体系能够平稳运转的真正基石，在于跨环境的“互操作性”（Interoperability），即负载与数据在本地与云端之间自由迁移调度的能力。 --- ### [19:14](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1154s) 务实分级与解耦：不必凡事都上大模型（LLM vs. SLM） 面对张璐关于“本地与云端模型尺寸差异及落地实现细节”的追问，Rajesh Sura 进一步阐述了他的业务逆向拆解法。他极力反对任何不加思考就盲目引入庞大语言模型的盲从行为。 在实际业务场景中，大量高频次、常规性的分类任务和确定性业务流程完全需要自动化，但它们根本不需要动用消耗巨大资源的大语言模型（LLM）。架构师应当在选型初期就做出清晰的架构划界：评估使用小语言模型（SLM）或是轻量级的模块化智能体框架（Modular Agent Frameworks）是否就能圆满交付任务。许多经过专项优化的轻量级小模型甚至直接在高端 CPU 上就能高效飞速运行，完全无需堆砌昂贵稀缺的顶级 GPU 集群。识别业务本质，按需匹配算力阶梯，才是 AI 基础设施长期可持续运行的理性之道。 - **核心原则提炼**： - 架构设计要“由终见始”，从最终需要 AI 承担的决策点逆向倒推技术栈； - 研发初期“可靠性优于扩展性”，收敛不确定性远比单纯扩大并发量重要； - 破除“参数规模迷信”，常规任务下放至运行于 CPU 的 SLM 或模块化 Agent，将重度 GPU 资源留给真正产生高价值的复杂计算。 ### [20:21](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1221s) 后 GenAI 时代的基础设施抉择：Scaling Law、自主大模型与上云灵活性 关于企业在生成式 AI 时代究竟该选择自建本地机房（On-Prem）还是拥抱云端，PayPal 全球 AI 负责人 Prakhar Mehrotra 直言“目前尚无定论”（the jury is still out）。他将这一决策拆解为几个核心变量：企业是否以构建或训练模型为核心业务。对于军工国防、强监管行业或是追求“主权 AI”（Sovereign AI）的国家而言，摆脱对外部云厂商的依赖、实现技术自主可控是首要诉求，此时自建 On-Prem 算力并深耕模型训练是不二之选。但对于绝大多数普通企业而言，决策逻辑则截然不同。 ```mermaid flowchart TD accTitle: AI Infrastructure Decision Framework accDescription: Enterprise infrastructure choice based on model scale and sovereignty needs subgraph Decision["决策分流"] A["企业核心需求"] --> B{"是否有主权/强监管训练诉求?"} B -- "是 (军工/国家主权/专属合规模型)" --> C["自建本地机房 (On-Prem)"] B -- "否 (通用企业创新与业务落地)" --> D{"算力规模与迭代速度"} end subgraph CloudTiers["云端多层架构"] D -- "快速实验 / 灵活适配模型" --> E["公有云 (Hyperscalers)"] D -- "大规模集群 (1,000+ GPU) / 高 TCO 优化" --> F["新兴算力云 (Neo-Clouds)"] end ``` Prakhar 强调，科学文献中没有任何迹象表明 Scaling Law（缩放定律）正在走向饱和，相反，“Scaling 范式依然充满生机”——从最初的预训练 Scaling，演进到当下的推理端扩展（Inference-Time Scaling）与思维链推理（Reasoning）。虽然迈向超级智能仍有距离，但沿着这条路径通往 AGI 的方向已十分清晰。为了跟上这一演进，企业研发需要极高的灵活性，必须频繁横跨不同尺寸与架构的模型进行实验。云端天然具备这种弹性，而 On-Prem 往往会因硬件锁定导致研发迭代速度急剧下降。企业必须清醒评估：自己究竟是要做模型微调（Fine-tuning），还是仅凭 Prompt 工程与上下文工程（Context Engineering）就能达成业务目标。 在回应主持人张璐关于“金融高隐私敏感场景是否必须上本地机房”的追问时，Prakhar 澄清道，现代云服务已经从技术上解决了隐私隔离问题。真正的决策考量在于：企业是否需要打造完全自主可控的专属模型？例如，PayPal 是否需要构建一个能深度理解全球每个地区、每个国家合规政策的“金融合规大模型”，且绝不能受制于任何外部云供应商？一旦确定有这种战略级自研诉求，On-Prem 才有其存在价值，否则盲目自建只会拖累业务创新。 - **Scaling Law 的持续演进**：已从单纯的预训练阶段算力堆叠，扩展至推理端 Scaling（Inference-Time Scaling）与复杂推理能力（Reasoning）。 - **企业核心考量维度**：区分自身是处于模型训练赛道，还是处于应用层（Prompt/Context 工程），避免过早陷入重资产硬件锁定。 - **隐私与合规解耦**：云端基础设施已成熟解决常规数据隐私问题，自建机房的核心驱动力已转变为战略级模型自主权与去除供应链依赖。 --- ### [23:33](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1413s) 数据重力、混合架构与 Neo-Cloud 的生态位 针对部署架构的考量，Aligned CTO David Stanfill 指出，所有系统设计的根本出发点始终是“你的数据在哪里，以及数据能否合规、高效地流动”。现实中的选择并非只有“非云即本地”的二元对立，而是呈现出三层格局：自建机房（On-Prem）、超大规模公有云（Hyperscale Cloud，如 AWS），以及专注于特定优化与定制化方案的新兴算力云（Neo-Cloud）。 David 坚定支持混合架构（Hybrid Approach），认为在现代技术生态中，几乎所有企业的系统都必须在某种程度上连接云端，与外部第三方系统进行交互。因此，架构设计的关键在于“在何处架设数据桥梁”。另一个决定性因素是企业内部团队的技术储备。诸如 AWS 这类超大规模云厂商，通常只对极少数超大体量客户提供深度的专属工程支持；而像 Aligned 这样的新兴云厂商，则能为客户直接派驻工程师、输出技术专长并协同攻坚定制化方案。当被问及混合架构是否会导致系统集成周期变长时，David 表示数据在不同算力服务商之间的迁入与迁出（Ingress/Egress）早在传统高性能计算（HPC）时代就已得到充分解决，技术上非常成熟，绝非落地阻碍。 - **基础设施三层形态**：On-Prem、Hyperscale Cloud（超大规模公有云）与 Neo-Cloud（定制化新兴算力云）。 - **混合架构成为常态**：企业系统无法脱离外部生态孤立运行，混合部署是兼顾数据驻留与外部系统连接的必由之路。 - **工程服务差异化**：中小规模及特定需求企业在超大厂商处难以获得定制服务，新兴云厂商通过输出工程专家与定制安全配置填补生态空白。 --- ### [25:26](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1526s) 算力规模的分水岭：为什么千卡以上集群需要专业 Neo-Cloud TensorWave 联合创始人兼 CEO Darrick Horton 从算力规模（Scale）的角度切入了本地与云端的界限。他指出，AI 工作负载与传统的 CPU 计算截然不同。如果企业的业务负载仅需要数十张或数百张 GPU（例如单节点推理、单卡推理或小规模微调与轻量训练），那么由内部小型工程团队在本地机房搭建和运维是完全可行的，技术门槛并不算高。 然而，一旦算力规模跨入数千张甚至数万张 GPU 的门槛，保持集群在高负载下的可靠性、高吞吐性能并实现优异的总体拥有成本（TCO），就会变成一项极其复杂且门槛极高的系统工程。Darrick 强调，运营这种超大规模集群所必需的软硬件调优与排错技能，需要庞大的专业团队历经数年才能沉淀出来，普通企业在合理的时间和预算内“几乎不可能在内部完成复制”。这也正是 Neo-Cloud 迅速崛起的核心价值： > “我们可以去构建 10,000 张甚至 100,000 张 GPU 的超级集群，确保它们高效运转，让你每一分钱都花在刀刃上（get the best bang for your buck）。” 正因如此，TensorWave 将自身业务牢牢锚定在 1,000 张 GPU 以上的大规模算力需求上，凭借深厚的工程经验和调优知识帮助客户最大化发挥算力效能，降低实际综合 TCO。 | 算力规模 | 典型工作负载 | 推荐部署模式 | 关键考量因素 | |:--- |:--- |:--- |:--- | | **数十至数百卡 (Tens/Hundreds of GPUs)** | 单节点推理、单卡推理、轻量微调、小规模实验 | 本地机房 (On-Prem) 或公有云实例 | 小团队可轻松维护，系统运维复杂度较低 | | **千卡至万卡级 (1,000 ~ 100,000+ GPUs)** | 超大模型预训练、大规模分布式推理与对齐 | 专业新兴算力云 (Neo-Cloud) | 涉及复杂的网络拓扑、容错治理与 TCO 优化，自建门槛极高 | --- ### [27:14](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1634s) 能耗瓶颈与未来架构演进：AI 动态负载调度与全模态统一 在谈及技术演进对底层架构的冲击时，主持人张璐分享了一段生动的亲身经历：就在上周她休假期间，AI 领域迎来重大密集发布（包括 Gemini 2.5、DeepSeek 系列模型以及关于 GPT-5 的广泛讨论），她不得不身处非洲通过 Starlink 卫星网络连线召开紧急 Zoom 会议。张璐指出，除了模型层的高频迭代，AI 庞大的电力消耗使得“能源层”（Energy Layer）成为全行业热议的重心。 亚马逊数据工程与分析负责人 Rajesh Sura 赞同这一观点，并指出能源正同时扮演着“瓶颈”与“创新机遇”的双重角色，未来能够胜出的必然是可持续的基础设施（Sustainable Infrastructure）。在硬件与物理层面，余热回收（用于二次供暖或工业用途）与浸没/冷板式液冷技术正在加速普及；在软件与调度层面，**基于 AI 的动态负载均衡（AI-based Load Balancing）** 正在成为颠覆性的技术手段——系统能够实时感知电网负荷，动态将高吞吐计算任务“即时调度”（on the fly）到拥有丰富可再生能源（如风电、水电）或实时电价更低廉的数据中心节点。 展望模型与基础设施的协同演进，Rajesh 提出了“全模态无感架构”（Modality-Agnostic Infrastructure）的概念。未来的 AI 底层架构必须打破传统的数据孤岛（Data Silos），在单一端到端工作流中无缝吞吐和处理包括结构化平表（Flat Tables）、文本、图像、视频在内的所有结构化与非结构化数据，彻底告别为特定模态单独定制割裂烟囱式系统的历史。 - **能源层技术突围**：硬件端采用废热捕获与液冷散热，降低数据中心 PUE。 - **动态算力调度**：通过 AI 驱动的负载均衡算法，依据绿电可用性与分时电价，跨地域无缝迁移计算任务。 - **基础设施模态解耦**：从底层消除数据烟囱，支持跨文本、表格、多媒体的全模态统一数据流与计算流。 ### [29:49](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1789s) 小模型组合与面向智能体的基础架构演进 在探讨企业级 AI 架构时，Rajesh Sura 提出了关键的第二点观察：**小语言模型（SLM）与大语言模型（LLM）的权衡**。企业并不需要依赖单一庞大的模型来处理每一项任务，更明智的路径是构建由专用小模型驱动的可组合 AI 智能体（Composable AI Agents）。每个小模型针对特定任务进行微调，再通过上层编排系统（Orchestration）将这些智能体串联成一条协同流水线，这将彻底改变技术交付的范式。 为了承载这种自动化智能体生态，底层的 AI 基础架构必须做到“面向智能体就绪（Agent-ready）”。具体而言，系统架构必须具备三大核心属性：**模块化（Modular）**、**互操作性（Interoperable）**以及**全面可观测性（Observable）**。唯有满足这三点，基础设施才能在技术快速迭代的浪潮中保持弹性扩容与长期可持续性。 ```mermaid flowchart TD accTitle: 可组合 AI 智能体流水线架构 subgraph S1["底层模型与微调"] M1["专用小语言模型 SLM A"] M2["专用小语言模型 SLM B"] M3["大模型蒸馏 SLM C"] end subgraph S2["智能体编排层"] A1["AI 智能体 1"] A2["AI 智能体 2"] A3["AI 智能体 3"] O1["中央流水线编排系统"] end subgraph S3["支撑底座要求"] INF["Agent-Ready 基础设施<br/>(模块化 / 互操作 / 可观测)"] end M1 --> A1 M2 --> A2 M3 --> A3 A1 --> O1 A2 --> O1 A3 --> O1 O1 --> INF ``` - **架构三要素**：模块化确保组件可按需插拔；互操作性保障不同框架与模型无缝对接；可观测性则为复杂多智能体系统提供实时监控与故障排查能力。 - **任务分工**：放弃“万能单体大模型”迷思，转向“专业微调小模型 + 编排流水线”的精细化协作模式。 --- ### [31:00](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1860s) “电力转化为 Token”：Scaling Law、推理爆发与杰文斯悖论 Prakhar Mehrotra 从物理最底层对 AI 算力本质进行了剖析：“在最基础的原子层面，你所做的本质上就是把电力转化为 Token。”如果相信未来 AI 将承担人类 50% 甚至绝大部分的工作，那么无论通过推理期计算扩展（Inference-time Scaling）、复杂推理还是思维链（Chain of Thought），生成的 Token 总量都将呈现指数级爆炸。在过去的信息时代，电力的作用只是将数据从 A 点搬运到 B 点；而智能时代是对数据做“求导”（Derivative），通过连接数据点来驱动智能，这带来了截然不同的能量密度需求。 当前整个产业的能耗大头仍集中在模型训练侧。Prakhar 直言，如果能源问题已被解决，GPT-5 发布时甚至可以完全免费或不设任何速率限制（Rate Limits），而现实中各种配额限制的根本原因在于数据中心匮乏、电力输送受阻以及散热瓶颈。眼下没有任何迹象表明 Scaling Law（扩展定律）触顶，全人类将继续把规模推向极致。随着模型逐渐接近具备经济可行性任务的 AGI 阶段，能耗重心将发生剧烈转移——推理侧能耗可能会迎来上百倍（100x）的爆发式增长。这势必引发典型的“杰文斯悖论（Jevons' Paradox）”：效率的提升不仅不会减少消耗，反而会带来更庞大的整体需求，AMD、Intel 等芯片厂商也将深度卷入这一轮推理硬件革新。 - **基础设施短板**：即使 GPU 芯片短缺得以缓解，美国依然面临电网输送能力不足、高密度数据中心机房短缺与散热系统跟不上的严峻物理制约。 - **推理成本分摊思考**：主持人 Lu Zhang 补充指出，当小模型被下沉部署到边缘或客户端时，推理能耗与计算成本究竟应由技术提供商承担还是由终端客户买单，将直接决定未来商业模型与基础设施的可持续性。 - **SLM 的依赖悖论**：Prakhar 进一步强调，高精度的小模型往往依赖超大模型进行知识蒸馏，而非平地起炉灶；即便拥有海量小模型，在移动端或操作系统级别编排 50 个微型模型，也将彻底重构底层软件工程与 OS 形态。 --- ### [35:08](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2108s) 能源视角的分歧：电网改造的历史机遇与边缘功耗隐忧 面对能源焦虑，长期从事大型能源项目的 David Stanfill 带来了截然不同且更乐观的产业视角。他认为，当前由训练驱动并即将转向推理的庞大电力需求，实际上是全美乃至全球电网基础设施升级的“历史性机遇”。以往电网之所以难以说服监管与投资方升级输电线路，是因为传统消费端与普通工业用电波动剧烈、极不稳定；而 AI 带来的则是持续、高速增长且极具确定性的基荷电力需求，这为整个国家级电网注入了前所未有的巨额资本与投资动能。 针对能耗优化的节奏，David 认为当前整个行业仍在争分夺秒抢占先机，“大家都在全力奔向未来，目前还不太在乎具体花了多少钱”。但随着发展进入深水区，成本控制与能源优化必将提上日程。针对小模型边缘化部署，他提出了一个极具现实意味的隐忧：无论计算发生在云端还是边缘设备，物理电耗并没有消失；当消费者购买微波炉或冰箱时会有明确的能耗能效标签，但未来家里跑着专属 AI 智能体时，却没有标签会告诉你它每小时在吞噬多少度电。 - **地域痛点与现实反差**：主持人 Lu Zhang 调侃自己身处加州，深知本地电网脆弱，甚至有高校或科技园区因训练 AI 大模型而直接导致整个园区跳闸停电，凸显了升级的紧迫感。 - **成长的烦恼**：David 将此类停电事件定性为任何高速增长事物必然伴随的“阵痛（Growing Pains）”，长远来看基础设施红利将覆盖整个生态。 --- ### [38:00](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2280s) 核能复兴、闲置电力套利与未来 1GW 站点的极限挑战 Darrick Horton 同样认同 AI 负载对能源产业的倒逼推动效应。他指出，由于 AI 带来了稳定且巨大的电力采购预期，当前核电项目（Nuclear Projects）迎来了强劲复兴，绿能建设的推进力度甚至超过了过去十年的总和。这种稳定的能源需求正在推动全美乃至全球电网的实质性升级。 然而，Darrick 也对中长期的物理瓶颈提出了预警。目前各家算力构建商正在利用“闲置电力（Stranded Power）”等低垂果实，采用灵活创新的方式寻找电力切入口。但在不远的将来，各区域电网的冗余容量将被彻底榨干，届时市面上将几乎无法在合理周期内交付新的 1 吉瓦（1GW）级超大型电力站点。当这个临界点到来时，整个算力体系的重压将全面倒逼到底层计算架构与系统效率的极致优化之上。 - **能源结构转型**：核能与绿色能源投资因 AI 的确定性用电需求被全面盘活，成为最大的资本承接地。 - **1GW 瓶颈临界点**：随着闲置电力红利吃紧，吉瓦级数据中心供电将成为不可逾越的物理硬约束，倒逼芯片与模型架构进行能效革命。 ### [39:11](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2351s) 算力瓶颈倒逼架构革新：从人脑能效到神经形态计算 Darrick Horton 强调，所谓“需要是创新之母（necessity breeds innovation）”。当前整个行业并没有迫切的动力去彻底重构底层架构，核心原因在于大家仍然能靠“堆算力、堆电力”来掩盖低效。但只要能源和物理极限一旦撞墙，我们势必会经历好几轮颠覆性的基础架构大洗牌。他指出，顶尖 AI 模型的能耗与人类大脑相比，存在着“许多个数量级的巨大差距”。这毫无疑问地证明了：在能效比和计算架构上，留给人类的优化空间依然深不见底。Darrick 预测，未来的 AI 行业每隔几年就会迎来一次底层逻辑的范式转移，并随之掀起新一轮的爆发式繁荣，在真正迈向可扩展的 AGI（通用人工智能）之前，整个行业还有极其漫长而充满潜力的长跑要走。 Lu Zhang 随即呼应了这一观点，感叹人脑确实是全球已知最高效的智能机器——潜意识、表意识与逻辑思维协同运作，能以微不足道的能耗解决不可思议的复杂问题。她分享了自己近期接触的前沿团队案例：有团队正在采用类脑的神经学方案（neurological approach）设计全新的模型结构，尽管模型尚未公开发布、暂无第三方 Benchmark，但他们声称能将能耗降低至少 1,000 倍（1,000x）；同时，还有初创公司正在探索模拟计算（analog computing），试图将 AI 落地的能耗压低 100 倍以上。模型层的这些架构突破，将反哺底层能源与算力设施，推动形成更加高效的混合基础设施。 ```mermaid flowchart TD accTitle: AI架构能效演进路径 accDescription: 从当前蛮力推算力到未来模拟计算与神经形态计算的能效演化路径 subgraph Current["现状：算力暴力扩张"] A["堆叠电力与集群规模<br/>(Brute-force Compute)"] --> B["面临能源与物理极限瓶颈"] end subgraph BrainInspired["灵感源泉：生物智能"] C["人脑神经机制<br/>(潜意识 + 逻辑协同)"] -.->|高出数个数量级的能效差距| D["架构重构目标"] end subgraph FutureArch["破局：底层模型与硬件革新"] B --> D D --> E["神经形态模型架构<br/>(降耗 ~1000x)"] D --> F["模拟计算技术<br/>(降耗 ~100x+)"] E & F --> G["混合高效基础设施<br/>迈向可扩展 AGI"] end ``` - 当前行业缺乏颠覆算力架构的紧迫感，是因为仍处于“靠加电加卡硬抗”的粗放增长期。 - 顶尖大模型与人类大脑的能耗差高达数个数量级，证明算力利用率与模型架构存在巨大的重构空间。 - 神经形态新架构团队声称可实现最低 1,000 倍的能耗降低；模拟计算路线团队目标将能耗降低 100 倍以上。 --- ### [41:13](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2473s) 终极预测与创业忠告：事件驱动与模块化演进 在进入圆桌尾声之际，Lu Zhang 向各位嘉宾抛出了收官问题：跳过遥远的 2030 年，聚焦未来三年内，给出一个最大胆的 AI 基础设施预测；同时面向台下的创业者，给出一项关于技术选型与 AI 原生解决方案构建的核心避坑指南。 来自 Amazon 的 Rajesh Sura 首先作答。在预测层面，他最看好 AI 与“事件驱动系统（event-driven systems）”的深度融合，让智能体（Agents）能够在真实的业务流中实现端到端的实时学习、自适应调整与即时执行。在给创业者的忠告上，Rajesh 直言：“不要为完美而构建，要为演进（evolution）而构建（Don't build for perfection, build for the evolution）。”鉴于底层技术的日新月异，团队绝不能指望一步到位训练出完美模型，而必须采用模块化架构，并严格保障工作流的“可观测性（observability）与互操作性（interoperability）”。最后，他警示大家千万不要妄图构建能解决所有开放性场景的万能系统，必须始终以具体用例（use case）为导向倒推技术路线。Lu Zhang 随即打趣道，这种通吃所有场景的庞大方案还是留给像 Amazon 这样的科技巨头去啃吧，这恰恰折射出当下大厂与初创公司之间既微妙竞争又相互依存的生态位关系。 - **未来三年预测**：AI 与事件驱动系统（Event-Driven Systems）全面结合，赋能具备实时感知、自适应与行动能力的智能体。 - **架构方法论**：拒绝完美主义，采用模块化解耦，优先构建链路的可观测性与跨系统互操作性。 - **产品落地**：坚持由具体用例逆向推导（Working backwards from a use case），切忌贪大求全去构建开放式通用大一统系统。 --- ### [43:09](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2589s) 智能体协议网络与物理世界的具身扩张 紧接着发言的 Prakhar Mehrotra（PayPal）给出了极快且干脆的回答。他调侃称自己眼下最强烈的诉求就是“赶紧来个 Agent 帮我把度假旅行彻底规划好”，因此他最大胆的预测是“智能体商业（Agentic Commerce）”将迎来全面爆发。而给创业者的核心建议则是：当未来的互联网上智能体的数量压倒人类数量时，大家必须思考承载这些智能体交互的底层基础设施到底是什么——“智能体之间的通信将如何发生？它们彼此对话的协议标准会演进成什么样？”提前押注这些通讯与交互协议才是高价值创业的方向。 随后，Aligned 的 CTO David Stanfill 接过话茬，他预测智能体 AI（Agentic AI）将在未来三年大举突破数字世界的围墙，全面蔓延至物理世界，与机器人硬件深度结合，带来颠覆人类社会的具身化变革。谈到建议时，David 与 Rajesh 产生了强烈的共鸣：务必牢牢盯紧业务用例本身，先搞清楚核心要达成的业务目标，再看 AI 能够如何赋能辅助，千万不要在现有的旧业务流程上盲目“生搬硬套硬贴 AI（throw AI at your existing business）”。 - **智能体网络协议**：未来网络上的自主 Agent 数量将超越人类，重点基础设施赛道在于“Agent-to-Agent（A2A）”通信协议与协调调度层。 - **具身智能破圈**：Agentic AI 深度融入物理世界与具身机器人（Robotics），从单纯的软件自动化跃升为物理实体的自主操作。 - **反向技术绑架**：从业务实质目标出发找工具，坚决避免拿着 AI 锤子满世界找钉子的盲目套用模式。 --- ### [44:14](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2654s) 超级个体团队的崛起与模型硬件形态的终局思考 TensorWave 的 CEO Darrick Horton 给出了极具震撼力的商业演进预测：未来三年将见证“微型团队爆发超级效能”的奇迹。极少数人组成的小团队甚至能够达成人类历史上从未有过的商业成就，他预测行业将很快目睹“仅有 5 名员工的初创公司在创纪录的超短时间内斩获数十亿美元估值”。事实上，这种趋势眼下已露端倪，行业内部分顶尖 AI 初创企业仅用几个月就狂揽 1 亿美元 ARR（年度经常性收入），这一超高效率的创新飞轮将在短期内引发密集的创新浪潮。 圆桌最后，Lu Zhang 进行了系统性总结。未来的技术浪潮将呈现多维度的跃迁：不仅有生成式 AI（Agentic AI）与物理世界 AI（Physical AI）的并行演进，更伴随着底层硬件与模型形态的深刻重组——小模型正在向 CPU 端迁移以榨取更高的性价比，而眼下人人都在谈论的 Transformer 架构，在三年之后是否仍占据绝对统治地位同样存在巨大变数。她总结道，追求顶尖 AI 性能固然重要，但“成本效益（Cost Efficiency）”永远是商业落地的生命线。创业者必须时刻保持“演进型思维”，敏锐适应新型 AI 基础设施层的迭代。在全场观众的热烈掌声中，本场关于长期 AI 基础设施扩展性的研讨圆满收官。 | 核心维度 | 传统软件 / 早期阶段 | 未来三年大趋势（嘉宾共识） | |:--- |:--- |:--- | | **组织规模与产出** | 庞大人力堆砌研发运营 | 5 人极简超级团队依靠 Agent 杠杆冲刺独角兽，数月达成 $100M ARR | | **智能体形态** | 数字界面的被动问答、单点任务 | 物理具身机器人化（Physical AI）与自动化商业（Agentic Commerce） | | **通信与协同模式** | 人机交互界面（UI/API）为主 | Agent-to-Agent 协议栈主导的大规模自主网络 | | **模型与硬件生态** | GPU 暴力堆叠、Transformer 一统天下 | 架构向模拟计算/类脑架构发散；CPU 运行小模型；能效比成为第一准绳 | ## Notable quotes > [00:04](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=4s) 对话式 AI 绝不仅仅是一次 UI 界面升级，而是一场端到端决策工作流自动化的范式转变 — Rajesh Sura > [11:17](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=677s) 我们刻意完全不碰 NVIDIA — Darrick Horton > [12:06](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=726s) 这才是最伟大的平衡器 — Darrick Horton > [16:45](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1005s) 优先针对可靠性构建，而非盲目追求可扩展性 — Rajesh Sura > [20:21](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1221s) Scaling 范式依然充满生机 — Prakhar Mehrotra > [25:26](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1526s) 我们可以去构建 10,000 张甚至 100,000 张 GPU 的超级集群，确保它们高效运转，让你每一分钱都花在刀刃上（get the best bang for your buck）。 — Darrick Horton > [31:00](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=1860s) 在最基础的原子层面，你所做的本质上就是把电力转化为 Token。 — Prakhar Mehrotra > [35:08](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2108s) 大家都在全力奔向未来，目前还不太在乎具体花了多少钱 — David Stanfill > [39:11](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2351s) 需要是创新之母（necessity breeds innovation） — Darrick Horton > [41:13](https://www.youtube.com/watch?v=Jd4u5RnL5kw&t=2473s) 不要为完美而构建，要为演进（evolution）而构建（Don't build for perfection, build for the evolution） — Rajesh Sura ## Mentioned Lu Zhang, Rajesh Sura, Darrick Horton, David Stanfill, Prakhar Mehrotra, Fusion Fund, TensorWave, Aligned, PayPal, Amazon, AWS, AMD, NVIDIA, Intel, AWS Bedrock, AWS Bedrock Agents, AWS Inferentia, Amazon Titan, Amazon Nova, Amazon S3, Amazon Model Registry, NeuralSync DPU, OpenSearch, LangChain, Grafana, Prometheus, React, TypeScript, AWS Lambda, API Gateway, DeepSeek, Gemini 2.5, GPT-5, Model Context Protocol, Starlink, Zoom ## Video description Lu Zhang, Founder & Managing Partner, Fusion Fund Rajesh Sura, Head of Data Engineering and Analytics, Amazon Darrick Horton, Co-Founder and CEO, TensorWave David Stanfill, CTO, Aligned Prakhar Mehrotra, Global Head of AI, PayPal Abstracts This session will focus on the critical components required to build and scale AI infrastructure, addressing both hardware and software needs for long-term AI success. Experts will discuss how to architect flexible, efficient systems that manage growing data volumes and computational demands, covering both hardware (GPUs, chips) and software platforms (ML Ops, AI frameworks) that enable organizations to fully leverage their compute power. Learn about the relationship between AI hardware and the software that maximizes its potential and discover strategies for future-proofing your infrastructure to ensure it supports ongoing AI innovation. [Source](https://www.youtube.com/watch?v=Jd4u5RnL5kw)

[0:04] Rajesh Sura: 对话式 AI 绝不仅仅是一次 UI 界面升级，而是一场端到端决策工作流自动化的范式转变 [11:17] Darrick Horton: 我们刻意完全不碰 NVIDIA [12:06] Darrick Horton: 这才是最伟大的平衡器 [16:45] Rajesh Sura: 优先针对可靠性构建，而非盲目追求可扩展性 [20:21] Prakhar Mehrotra: Scaling 范式依然充满生机 [25:26] Darrick Horton: 我们可以去构建 10,000 张甚至 100,000 张 GPU 的超级集群，确保它们高效运转，让你每一分钱都花在刀刃上（get the best bang for your buck）。 [31:00] Prakhar Mehrotra: 在最基础的原子层面，你所做的本质上就是把电力转化为 Token。 [35:08] David Stanfill: 大家都在全力奔向未来，目前还不太在乎具体花了多少钱 [39:11] Darrick Horton: 需要是创新之母（necessity breeds innovation） [41:13] Rajesh Sura: 不要为完美而构建，要为演进（evolution）而构建（Don't build for perfection, build for the evolution）

[0:04] Rajesh Sura: paradigm shift in automating end-to-end decision workflows [0:04] Prakhar Mehrotra: agent of record [12:09] Darrick Horton: would like to buy some AI [12:09] Darrick Horton: the great equalizer [12:09] Lu Zhang: cost of integration [23:32] David Stanfill: where is your data and where can you move your data? [27:13] Prakhar Mehrotra: converting electricity into tokens [27:13] Prakhar Mehrotra: too big to fail [35:08] Darrick Horton: low-hanging fruit [35:08] Darrick Horton: throw more power at it [41:13] Rajesh Sura: build for evolution

[0:04] Yeah, thank you very much for joining us [0:05] today. You know, there's lots of [0:08] discussion about AI agent recently and [0:10] lots of the discussion about the AI [0:12] application. But fundamentally, if we [0:14] want to make AI not only amazing [0:16] solution but also cost efficient [0:18] solution, it's very important to talk [0:21] about how to make AI infrastructure more [0:23] scalable and also sustainable in terms [0:26] of the cost, energy consumption, etc. So [0:28] that's the thing we wanted to discuss [0:30] today and today we have a pretty diverse [0:32] the panel and with different [0:34] backgrounds. So we're going to dive into [0:36] different aspect of the AI [0:37] infrastructure. So to get started we [0:40] want to ask the panelists to introduce [0:42] themsel very quickly about your [0:44] background and also what you're building [0:46] right now within AI info within your [0:48] company and organization. We want to [0:50] start with uh Derek. [1:02] testing works now. All right. Uh hello [1:05] everyone. My name is Derek. I am [1:07] co-founder and CEO of TensorWave. [1:10] TensorWave is a AMD CSP. So we focus on [1:14] building large scale uh AI GPU clusters [1:18] primarily with AMD GPUs. So we're a [1:20] little different in the market. Most [1:22] AICSPs use uh Nvidia. We have a lot of [1:26] good reasons for branching out and going [1:28] with AMD, which I'm sure we'll get into [1:30] a little bit later. But uh we are [1:34] primarily focused on building large [1:36] scale clusters for training uh [1:38] inference, distributed inference, and uh [1:40] fine-tuning. [1:42] >> Perfect. Thank you, David. [1:46] >> Hi, I'm David Stanfel. I'm the CTO of [1:48] Aligned. We're a high performance [1:50] computing company with uh our own data [1:52] centers and a neocloud as well that we [1:56] build out for end customers. We also [1:58] develop our own custom interconnect [2:00] technology in the form of our neural [2:02] sync DPU that we use to control the data [2:04] flows in more specialized ways in and [2:07] out of our AI clusters. And we focus on [2:09] working with enterprise customers and [2:11] research teams that have more bespoke [2:14] infrastructure needs outside of what [2:15] they might find in uh traditional [2:17] hyperscaler cloud and helping them adapt [2:20] those existing compute infrastructure [2:23] and existing data pipelines to AI [2:26] opportunities that they're looking to [2:28] pursue. [2:29] >> Great. Thank you. [2:31] >> Uh I'm Par. Uh uh I'm from PayPal. I [2:34] lead AI for PayPal. uh at PayPal it's an [2:38] applied AI company uh and we are [2:40] interested in uh uh exploring the [2:43] infrastructure for AI that operates in [2:45] the agentic words like what happens when [2:48] you have uh millions of agents on the [2:50] internet and trying to do a task on your [2:52] behalf. Uh uh so things after the [2:54] inferencing layer uh uh is what we are [2:57] very much interested in and uh building [3:00] uh getting uh if if you believe that the [3:02] future is going to be agentic and agents [3:05] are going to do task on your behalf. How [3:07] do you build trust and identity in that [3:10] uh in that network of agents? Uh uh [3:13] PayPal as a company was started uh 25 [3:15] years back building trust in the [3:17] e-commerce era. uh and so now uh how do [3:20] you build trust like this concept of [3:22] agent of records that uh uh for the [3:24] agentic works so that's what we are [3:26] interested in [3:27] >> perfect so last but not least [3:30] >> hello I'm Rajes uh head of data [3:32] engineering and analytics at Amazon and [3:35] uh have over 16 years of experience in [3:37] designing and building uh cloudnative [3:39] platforms that power both AI and [3:42] analytics and currently my focus is on [3:45] uh building conversational AI uh for [3:47] North America source within Amazon. Uh [3:50] and um yeah uh when I say conversational [3:53] AI uh I always believe this is not just [3:56] a UI upgrade and it is a paradigm shift [3:59] uh in you know like uh automating end [4:01] to-end decision uh workflows. [4:04] >> I'm looking forward to deep dive more [4:06] into it. [4:06] >> Yeah, I also heard Amazon going to have [4:08] your own model launch Nova, right? It's [4:10] already launched. Yeah. So [4:11] congratulations. It's always better to [4:13] see more like hyperscalers start to [4:15] deploying capital to the model layer and [4:17] we can make it more dynamic and robust [4:19] the ecosystem. So this is Lou. I'm the [4:21] founder managing partner of a fusion [4:22] fund. We're Silicon Valley based VC firm [4:25] and had around total a little bit over [4:27] half a billion dollar size. Uh I started [4:30] for about 10 years ago and across the AI [4:32] category we invest in both AI agent [4:35] application. We also invest a lot of on [4:37] AI infrastructure from the hardware [4:40] layer to info layer to inference layer [4:42] to on the architecture layer. So it's [4:44] quite important to really help a founder [4:47] to understand how to navigate within the [4:49] ecosystem with collaboration competition [4:51] and be able to provide a solution to the [4:54] customer make it not only just optimize [4:57] performance but also cost efficiency. So [5:00] to kick off the discussion today, you [5:02] guys all mentioned about briefly about [5:05] the important of the AI info and also [5:07] the effort you're putting in. So what is [5:09] the unique piece you're building here [5:11] and also which type of no matter tools, [5:14] hardwares and info architecture design [5:17] you're adopting with your company to [5:19] enable it to have uh lots of sustainable [5:21] scalable features. You want to get [5:23] started? Oh [5:24] >> yeah, sure. Uh so as I said my primary [5:27] focus is on building conversational AI [5:29] and uh under our hood under the tech [5:31] hood right uh our stack is both modular [5:34] and at the same time event driven. uh we [5:37] start with AWS bedrock uh for building [5:39] the generative AI applications and we [5:41] use open search as our vector DB and [5:44] Amazon titan plays the role for uh [5:46] embedding models and then uh we leverage [5:49] of course uh lang chain uh for LLM [5:51] orchestration uh and uh use bedrock [5:55] agents for chaining the reasoning steps [5:58] and at the same time uh we're moving on [6:00] to our back end we work a lot on API [6:03] gateways and uh of course um uh the uh [6:07] lambdas you know like TypeScript lambdas [6:09] uh and on the front end we build React [6:11] applications in order to provide both [6:13] secure and uh you know like uh better [6:16] user experience. Uh coming to the [6:18] storage we rely on uh data store objects [6:20] S3 which is super easy to scale uh [6:24] depending on the demand variability and [6:26] at the same time for compute we blend uh [6:29] the GPU backed you know like computing [6:31] clusters with the uh inference optimized [6:34] chips such as AWS inferentia uh which is [6:37] really cost efficient and of course [6:39] observability is another thing that we [6:41] built into our infrastructure from day [6:43] one. We use tools like uh uh graphana uh [6:47] for dashboarding, Prometheus for time [6:49] series metric tracking and at the same [6:51] time uh um the uh Amazon you know like u [6:55] model registry in order to track the [6:57] model lineage versioning and uh you know [7:00] basically how every AI component how [7:02] every component in your AI is actually [7:04] working uh and you know like uh [7:06] producing results. So that's that's uh [7:09] you know like a brief about the uh AI [7:11] tech stack that we use uh behind [7:14] building this conversational AI. [7:15] >> Nice. Yes. [7:17] >> Yep. Uh as we were thinking about the AI [7:20] stack, right? Like if you see what has [7:21] changed between pre-generative AI and [7:24] postgenerative AI era. Um uh you can [7:28] think of AI infrastructure in four [7:30] buckets, right? There's there's [7:31] infrastructure that is needed for data, [7:34] storing data. There is infrastructure [7:35] that is needed for training the models. [7:37] There's infrastructure that is needed [7:38] for uh optimization of the metrics that [7:41] you care for and there's an [7:42] infrastructure that is needed for [7:43] inferencing layer. And so that's what [7:45] the classical uh machine learning [7:46] paradigm has been about. uh uh in [7:49] generative era in in in with the advent [7:51] of large language models playing into [7:53] fifth uh uh all these four buckets have [7:56] gone some changes right like earlier you [7:58] went from storing information to now you [8:01] have to somehow store intelligence [8:03] because LLMs are representation [8:04] compressed representations uh of [8:06] information that you can derive [8:08] intelligence on so that's a shift how do [8:10] you store that uh how do you uh uh when [8:13] you're training these large language [8:14] models right I mean you have to in the [8:16] modeler architecture that Rajes talked [8:17] about you have to introduce checkpoints [8:19] because now I'm training a much much [8:20] larger model than not so your [8:22] architecture looks much different now uh [8:25] uh in the inferencing layer uh uh you [8:28] are now serving these models I mean look [8:30] the last 10 years in technology taught [8:32] us horizontal scaling that was a thing [8:34] right like but but in AI era it's not [8:36] about it's about like serving the GPUs [8:39] are at constraint here uh and so how do [8:41] you scale like how do you take the [8:43] distributed sensor concepts that was [8:45] invented for information era and bring [8:47] it to to intelligence era. Uh but [8:49] there's something more uh that is [8:51] happening which is like the the these [8:53] agents are going to have agency. So like [8:57] in the entire ML stack you would serve a [8:59] recommendation to a human and you would [9:01] would basically carry out the last mile [9:03] but in AI era the we are right now or [9:06] we'll see in the near future is agentic [9:08] right and agentic era needs to carry out [9:10] the task on your behalf which means [9:12] issues like identity trust uh like uh uh [9:16] communication between the two agents [9:18] right like agents requiring tool usage [9:19] model context protocol that was so all [9:22] those becomes very important and so I [9:23] call this like agency [9:25] >> like the layer after inferencing that [9:27] has to worked out and so at PayPal uh uh [9:30] it's a company that understands identity [9:32] and trust uh over last 20 years and so [9:34] we have invested a lot of time in [9:36] building that agency layer uh so that so [9:38] that when uh agents do become a thing on [9:41] the internet which we hope is is very [9:43] soon uh uh you have this agency you have [9:46] this infra that loc protocols that like [9:48] two agents can talk to each other and [9:50] bring trust to the transaction [9:53] >> great so David [9:55] >> so I think We're looking uh a lot of the [9:58] multimodal data that's beyond [10:00] conversational AI that's out there which [10:02] is not just audio and video but you have [10:04] to look at things like LAR point cloud [10:06] data geotechnical data all kinds of [10:08] enterprises [10:09] excuse me all have this very specialized [10:12] data and we come from a high performance [10:14] computing pedigree where we're dealing [10:16] with this data and this data is voluous [10:18] it has substantial compute that's [10:20] happening before and after in [10:22] incorporated in the AI pipeline and we [10:25] have to look at how you build [10:26] infrastructure that supports those kinds [10:28] of enterprises beyond just [10:30] conversational AI cloud enterprises [10:32] which we're seeing a lot of early [10:33] adoption there in CRM and sales systems [10:36] and those things that interact very [10:37] closely with humans but as you start [10:39] working on the things that interact very [10:41] closely with existing heavy compute [10:43] infrastructure supercomputers simulation [10:47] that's a whole different set of needs [10:48] both on the data side and the training [10:50] side but also on the inference side when [10:53] you have these applications operating in [10:54] real time in an agentic form. Your rag [10:59] recycle is not just going out and [11:00] retrieving pieces of information from a [11:02] knowledge store. It's retrieving [11:04] potentially very large, very complex [11:06] data sets or even interacting with other [11:09] processes and systems. So how we look at [11:11] how you build infrastructure that can [11:12] support all of those modalities as well [11:15] as just the conversational AI. [11:17] >> Yeah, makes sense. with Eric. [11:20] >> Yeah, from our perspective, you know, [11:22] we're we're very focused on bringing [11:23] alternative infrastructure to the market [11:25] at scale. So, um, like I said before, we [11:28] don't do anything with Nvidia and we do [11:30] that on purpose because, uh, we think [11:32] it's really important for all adopters, [11:34] whether they're large scale foundation [11:37] model builders, whether they're [11:38] enterprise, whether they're, you know, [11:40] small AI labs or startups to have more [11:43] than one option in their AI [11:44] infrastructure. So not just Nvidia but [11:47] also to have in our case AMD and over [11:49] time other infrastructure providers. So [11:51] you know we're very focused on that [11:52] aspect of uh bringing differentiated [11:54] infrastructure to market and ultimately [11:57] that drives uh better value for the end [12:00] users right to have more options and [12:02] more uh democratized access to uh [12:05] compute. [12:06] >> Yeah. So back to both of you um Derek [12:09] and David both of you are building [12:10] really deep tech infrastructure solution [12:13] and uh of course you can work with AMD [12:16] and video like a very techsavvy [12:18] hyperscaler company but also I'm sure [12:20] there's also client and company they're [12:23] not ready uh to integrate I understand [12:25] the latest uh new infrastructure [12:27] architecture design they even don't have [12:29] a technology savvy company tech savvy [12:32] team to work with you how do you really [12:35] integrate with them what is kind of the [12:37] lesson learned. What is the best [12:38] practice based on your experience? [12:42] >> So, one of the things that uh [12:44] particularly AI or existing enterprises [12:47] that aren't AI first and aren't AR savvy [12:50] have to grapple with is a lot of times [12:52] you'll have a seuite that's dictating we [12:54] need to have an AI strategy. We need to [12:56] put something out there. They'll task a [12:58] team. That team will start looking at [12:59] options and tools and they'll start [13:01] doing experiments. They'll start [13:03] building um on AWS tool sets. they'll [13:06] start building on other things but they [13:07] haven't really looked at what if this [13:10] experiment is successful what does that [13:12] look like in production [13:14] >> and that's where a lot of these projects [13:16] fall can fall down and they can get [13:18] tripped up uh and so I think that's some [13:20] of the guidance we can provide early on [13:22] is okay if you want to do your [13:24] experimenting in a cloud existing cloud [13:27] infrastructure uh if you want to have [13:28] part of your application there but what [13:30] parts might you need to have on prem [13:32] what parts might you need to have in a [13:33] localized data center or in a data [13:35] center that you control that's [13:36] essentially coll-located. So I think [13:38] those are the challenges that we try to [13:40] guide early teams that are not labs and [13:43] AI focused teams on. [13:46] >> Yeah. Um you know echoing some of those [13:48] sentiments uh we find especially the [13:50] enterprise [13:52] um oftentimes they have this mandate [13:54] from their board hey go do AI right and [13:57] they'll come to us and say like hey like [13:59] we'd like to buy some AI and we're like [14:01] okay uh we can help you with that but a [14:04] lot of times they they don't know [14:05] exactly what they want they don't and [14:07] they know even less about how to get it [14:09] right and so uh we have to not only [14:13] operate infrastructure in a stable [14:16] reliable, scalable way, but we also have [14:17] to guide especially enterprise into what [14:20] solutions are going to be best for, you [14:22] know, their unique business and what [14:23] opportunity they're trying to apply AI [14:25] to. So, you know, we find it's it's [14:28] incredibly important not only to have [14:29] stable infrastructure, but to also make [14:32] the experience for the end user as [14:35] simple as possible, as seamless as [14:37] possible. And that is the great [14:39] equalizer. If you abstract away the [14:41] experience so that the end customer [14:44] doesn't really need to know like oh I [14:46] need to use these AMD tools or these [14:48] Nvidia tools or some low-level driver [14:51] thing right uh if they can just use uh [14:54] some higher level abstraction that lets [14:57] them achieve their their AI goals then [15:00] they don't really care what hardware is [15:02] powering it uh at the end of the day and [15:04] so it's it's really important especially [15:06] if you're providing some alternative [15:08] infrastructure that you make that [15:09] experience as easy as possible and as [15:11] seamless for the end user. [15:13] >> Yeah, especially we always talk about [15:15] it's important for fun to understand [15:17] there's not only the cost of technology [15:19] there also cost of integration. You [15:21] wanted to make sure integration cost is [15:22] as low as possible for your client not [15:26] only just the cost of the capital but [15:28] also cost of the brain power they don't [15:30] want to know too much what is going on [15:32] the background as the result really [15:34] deliver and happen. I think it's also [15:36] interesting what you mentioned earlier [15:37] on that the board is all of a sudden [15:39] asking the questions we need to buy AI [15:41] we need to adopt AI we have a network of [15:44] 45 CTO for Fortune 1000 company we call [15:47] the CXO network working with us since [15:49] 2018 and most of them are CTO and until [15:52] recently I would say two years ago [15:54] they're like oh we're got getting calls [15:56] from the CEO of the board member we need [15:58] to go to the board meeting to really [16:00] talk about what is the progress of our [16:01] AI powered the digital transformation so [16:04] I I think they're asking questions that [16:06] they may not directly jump into [16:08] conclusion in terms of buy versus build [16:11] right they may try build a little bit [16:13] and with some capital and later on [16:16] figure out what is the best combination [16:17] of certain building house if they have [16:19] the capacity and uh most of the other [16:22] part would rely on the third party so [16:24] based on that sentiment I think both of [16:26] you mentioned that it's important to be [16:28] a little bit patient at the beginning to [16:30] try to educate your client and customer [16:32] and coming really support and help them [16:34] allow with your interest is very [16:36] critical to help them navigate the AI [16:40] fast moving evolving trend and also help [16:42] you build up a strong you know customer [16:44] relationship. Okay. So on top of that [16:46] another questions uh and also [16:48] interesting topic want to discuss today [16:50] is the trade-off between cloud and [16:53] onprime you know when we talk about AI [16:55] infrastructure also we go back to talk [16:57] about you can deploy large model on the [17:00] cloud have everything running on cloud [17:02] upload everything to the cloud for [17:04] certain industry either they don't have [17:05] the infrastructure ready to upload [17:07] everything to the cloud or they don't [17:09] want to because some industry like very [17:11] sensitive in terms of data privacy and [17:14] uh regulation compl clients. Uh so [17:16] there's the different reason and how to [17:19] really have a balance point of uh [17:21] deciding whether we should run it on [17:23] cloud on prime and also how to run a [17:26] hybrid solution. So want to start with [17:29] you Raj. uh before jumping into the [17:32] cloud and on right like uh for all those [17:34] startups who are first time [17:36] experimenting with AI uh don't jump into [17:39] you know like building models rather [17:41] take a step back and first decide on [17:44] what decisions does AI have to make for [17:46] you you need to get a clarity on that [17:48] and then work backwards from there and [17:51] don't try to you know like uh lock in [17:53] early investments uh use the open-source [17:56] tools there are a lot of uh open source [17:58] tooling that is available and build for [18:00] reliability and not for scalability you [18:02] know like don't try to you know like [18:03] okay go ahead and build a scalable [18:05] solution right away uh because AI is [18:08] very unpredictable you know uh there can [18:10] be bias there can be you know like uh [18:13] unexpected responses out of it so build [18:16] for reliability and then try to you know [18:18] like uh scale on top of it and coming to [18:21] cloud you know uh of course for AI teams [18:23] who are having very high experimentation [18:26] velocity uh right uh cloud provides them [18:29] elastic icity uh dynamic computer [18:31] allocation and at the same time global [18:33] reach uh at the same time uh the on-prem [18:37] it makes sense for highly regulated [18:39] industries and uh for workloads which [18:41] are very uh you know like which have [18:43] ultra low latency but the sweet spot [18:46] here is the hybrid approach uh you know [18:49] like which allows you to you know like [18:52] have highly sensitive workloads in a [18:54] very controlled environment but at the [18:56] same time burst into cloud uh for tasks [18:58] that require high CPU, GPU, you know, [19:01] like resourcing and uh uh the key here [19:04] is the interoperability uh like being [19:07] able to shift these workloads from prom [19:09] to you know like cloud and vice versa. [19:11] So that interoperability is the key. [19:14] Yeah. [19:14] >> And can you elaborate more how to [19:16] achieve that and especially you know the [19:19] size of the model running on the prime [19:21] and the local on the cloud will be very [19:23] different. [19:24] >> Yeah. So um here is what uh my approach [19:28] is as I said earlier always work [19:30] backwards from what you want to do. [19:32] There are certain you know like routine [19:33] classification tasks uh which needs full [19:36] automation and that doesn't require an [19:38] large language model. So you need to [19:40] choose whether a small language model or [19:43] simple you know like modular agent [19:45] frameworks can automate those tasks and [19:47] that doesn't require you know like high [19:49] GPU like even these some of the small [19:52] language models can even run on you know [19:54] like uh uh CPUs high-end CPUs. So th [19:58] those are the you know like uh [19:59] demarcations that you need to make. Uh [20:01] don't rely always uh you know like [20:03] trying to uh deploy a large language [20:06] model. figure out what's the use case, [20:08] work backwards from this, work backwards [20:10] from there and uh you know choose [20:12] whether an small language model or a [20:15] model or agent framework would uh fit [20:17] your use case. [20:18] >> Great. Thank you. [20:20] >> Huh? [20:21] >> Yeah. Uh so in my opinion, I think the [20:24] jury is still in in the in the post [20:26] Genai era, I think the jury is still out [20:28] on this balancing between on-prem and [20:30] cloud. uh uh now if you are uh doing uh [20:35] I mean there are three variables right [20:36] one is like just the pure whether you [20:39] are in the business of creating models [20:41] or training models or not uh uh uh right [20:43] and so for like if if I am military if [20:46] I'm highly regulated industry right like [20:48] you want to or I'm a a country like [20:50] sovereign in AI you want to have this [20:52] own dependency you actually don't want [20:53] to depend on any uh outside provider and [20:56] so you might want to do on-rem and then [20:57] you might want to enter the get business [20:58] of model training and and you have [21:00] panelist on the on this panel who will [21:02] be become very important for you, right? [21:04] Like so just by that very aspect, right? [21:06] Like uh uh but for if you're enterprise [21:08] setting uh uh uh like the the number one [21:11] decision battle today is is uh uh look [21:15] uh there's nothing that is saying in the [21:17] uh in the scient science literature that [21:20] scaling laws are not saturating. They [21:21] are not saturating. Scaling paradigm [21:23] still exists. It's well alive. Uh uh you [21:26] had pre-trained scaling, now you have [21:27] inference time scaling and now you have [21:29] reasoning. And then and and and uh if [21:31] you take if you go on this extension the [21:33] path for AGI is much very clear right [21:35] like like you can get to AGI with this [21:37] like you can't get to super intelligence [21:38] but you can get to AGI right like but [21:40] but for that experimentation you need [21:42] crazy flexibility to be able to operate [21:45] that like you need to go across the [21:47] models you need to try out different [21:48] model size and as I just talk about like [21:50] cloud just gives you that flexibility [21:52] like on-prem just locks I mean the the [21:54] rapid iteration speed of R&D just [21:57] dramatically goes down on onrem right [21:59] like and So if you are not in that like [22:00] if you if so first of all if you're not [22:02] so in AI business category the [22:04] discussion is different but if you are [22:05] in enterprise setting you have to ask [22:07] the question like uh what model size are [22:09] you using is it will you ever get into [22:11] the world of fine-tuning uh or is it [22:14] just prompt engineering plus context [22:15] engineering and then you get the job [22:16] done uh uh uh uh and so those decision [22:19] starts and the jury is still out on in [22:21] my mind on the best practice because we [22:23] are in the year three of this this wave. [22:25] So Picar you mentioned early on that the [22:27] PayPal you know for the past more than a [22:29] decade that you guys are really [22:30] emphasized on the data privacy. So do [22:32] you do you think like any industry with [22:35] the sensitivity to data privacy and [22:37] compliance has to go with on prime? I [22:39] >> I think uh I mean cloud has solved the [22:41] privacy problem. So that's not a uh not [22:44] a issue. issue I think uh uh even in [22:47] PayPal uh uh is is whether do we want to [22:51] build our own models and at what speed [22:53] do we want to build our like for example [22:55] will there be ever a fintech LLM [22:57] >> that understands compliance of every uh [22:59] every geography every country uh uh and [23:02] how will that work like and if if if you [23:05] and if you think that okay I need to [23:06] build that compliance LLM that is [23:08] heavily regulated which I don't need [23:10] dependency on any of the cloud providers [23:12] then you need your on-prem [23:13] >> yeah Yeah. And so you have to go into [23:15] that thought and a very similar argument [23:16] for militaries for elsewhere and so you [23:19] can just keep going on this chain of [23:20] thought and the decision becomes clear. [23:22] >> Yeah. I think there's also lots of [23:24] discussion about adoption of the federal [23:26] computing right on top of the sensitive [23:28] data especially for healthcare and [23:30] financial industry. Okay. Come to you [23:32] David. What do you think? [23:33] >> At the end of the day a lot of it [23:35] depends on where's your data. [23:36] >> Where's your data and where can you [23:38] where can you move your data? Um so and [23:41] you have kind of the three tiers I would [23:43] say not just onrem versus cloud but [23:45] on-prem to you know hypers scale cloud [23:47] versus a neo cloud that may offer [23:49] certain combinations may be able to [23:51] customize build configurations or [23:53] security configurations a little [23:55] differently. So you have to look at [23:57] those decisions, but it comes down to [23:58] where are where is your data and where [24:00] are the processes and other systems that [24:02] you need to interact with and can you [24:05] can you do that? And also how fast is [24:08] your team iterating? Um to echo [24:10] something that was said there and I I [24:12] would agree with what was said about a [24:13] hybrid approach. A hybrid approach is [24:15] almost always going to be the best [24:17] approach. Some part of your process, [24:19] your systems is always going to need to [24:22] be cloud in this modern world. you're [24:24] going to have to interact with outside [24:25] systems. So, where do you build those [24:28] bridges? How do you make those [24:30] decisions? Um, [24:32] >> it depends on your internal team's [24:33] expertise. [24:35] >> A lot of times on prem again versus [24:38] Neocloud work like u we're doing, we may [24:42] have more bandwidth to give you [24:44] expertise. We may be able to give you [24:46] lend you engineers, lend you support, [24:48] help you develop a solution. Um, at a [24:50] certain scale, maybe AWS is going to do [24:52] that, but a different scale. they don't [24:54] they're not in that business right so I [24:57] think that's another where's your [24:58] expertise is another way to look at it [25:00] >> based on your experience with customer [25:02] David so do you think when the customer [25:04] wanted to have a hybrid solution will [25:05] take longer for them to integrate your [25:07] infrastructure [25:09] >> uh solution [25:10] >> no I wouldn't think that that's a [25:12] barrier because we regularly are used to [25:15] ingress and egress outside of [25:17] >> cloud various cloud compute providers [25:20] we've done that for years in different [25:21] HPC applications that's not a new [25:23] problem to solve. [25:24] >> Great. Yeah, Derek. [25:26] >> Yeah. So, from our perspective, I I [25:29] think the the key determining factor [25:31] whether you should do cloud or onrem [25:34] really boils down to scale in this in [25:36] this industry in particular. Um, you [25:39] know, it's it's a different question in [25:41] traditional cloud, CPU cloud, [25:43] traditional workloads. But in AI [25:45] workloads, I if your uh you know if your [25:50] workload needs tens or hundreds of GPUs, [25:53] >> you can do it on prem. [25:55] >> That's not that difficult [25:57] >> mostly. [25:58] >> Um if it's inference, if it's single [25:59] node inference, single GPU inference or [26:01] small scale fine-tuning, small scale [26:03] training, these are all things that can [26:05] be achieved on prem by a small team [26:07] pretty uh pretty easily. But as soon as [26:09] you start getting into thousands or tens [26:11] of thousands of GPUs, operating that [26:14] infrastructure reliably uh in a [26:16] performant way, actually getting good [26:18] TCO from it becomes incredibly [26:20] non-trivial, [26:22] >> right? It's a very difficult problem. [26:24] And that's why uh Neoclouds are [26:27] successful is because that that's the [26:29] role that they fill is uh you know we [26:31] can go out there we can build a 10,000 [26:33] GPU cluster or 100,000 GPU cluster and [26:35] we can ensure that it's working well and [26:38] that you're getting the best bang for [26:39] your buck out of it. Um that skill set [26:43] takes many years to develop. it takes a [26:45] huge team and the the premise of um your [26:49] average enterprise being able to [26:50] replicate that in-house in any [26:52] reasonable time frame is it's it's not [26:55] possible. So that's the key [26:56] differentiator and that's why we focus [26:58] on large scale opportunities you know [27:00] 1,00 plus GPUs uh because that's really [27:03] where our value at is. We can make that [27:05] infrastructure operate more efficiently [27:07] and get better TCO from it because we [27:10] have the experience and the knowledge uh [27:12] to do that. [27:13] >> Amazing. So now let's think about the [27:16] future. You know we all know how fast [27:17] the AI ecosystem is evolving. Like I [27:20] remember was uh last week I was actually [27:22] taking a short vacation and um bingo you [27:25] know this is a major AI launch week. I [27:28] have to do all this zoom call from [27:29] Africa with the Starlink in order to you [27:32] know have the discussion with the team. [27:33] Okay, Gemini 2.5 launches and also the [27:36] steep syninko model and then come come [27:39] to the GBD5 and meanwhile beyond the [27:42] model layer is evolving there are also [27:44] lots of discussion about the energy [27:46] layer right we all know we're consumer [27:48] tons of energy for supporting the future [27:50] of AI and there's a different solution [27:53] uh being proposed in order to solve it [27:55] and also including how to make AI itself [27:58] more energy uh efficient so there's a [28:00] lots of thing happening on the model [28:02] layer there are lots of things happening [28:03] on the energy layer. So what do you [28:05] think this evolution of the model the [28:08] energy layer even including the hardware [28:10] layer genic AI everything small language [28:12] model have the impact on the shift of [28:15] the infrastructure needs so I will start [28:17] with you uh Rajes. [28:19] Yeah. Uh definitely energy is becoming [28:22] both a bottleneck and at the same time [28:24] an opportunity. Uh you know so uh I uh [28:28] in my perspective uh the sustainable [28:30] infrastructure is what is going to you [28:32] know like win at the end and um the [28:36] technology in this space is evolving uh [28:38] really fast. Uh today we have uh methods [28:42] to recapture the heat uh and you know [28:44] like use it for secondary use cases and [28:47] at the same time the liquid cooling and [28:50] um at the same time um [28:53] what are the things sorry [28:56] um yeah uh like recapturing that and AI [28:59] load balancing you know like uh AI AI [29:01] based load balancing is a gamecher uh [29:04] which is able to shift your compute uh [29:06] to you know like uh regions which have [29:08] renew renewable energy or cheaper [29:11] options uh on fly uh dynamically is [29:14] definitely a gamecher in this space and [29:16] coming to the uh you know like uh what's [29:19] in future like how this is uh this whole [29:21] AI models are evolving what I see is uh [29:24] number one um modality agnostic you know [29:28] like uh you should be your [29:29] infrastructure should be able to support [29:32] uh uh to build a model that can handle [29:35] unstructured and structured data like [29:37] coming from flat tables text, images, [29:39] videos, everything end to end in one [29:41] workflow. Uh rem eliminating the need [29:44] for you know like bpoke silos. So that's [29:46] going to be uh one big thing. And uh [29:49] number two uh SLM versus LLM, you know, [29:52] like small language models versus large [29:55] language models. You don't need to have [29:57] that one large language model to perform [30:00] every task. rather you should be [30:02] building composable AI agents you know [30:04] like using small language models which [30:06] are uh tuned uh for specific tasks and [30:09] then there should be an orchestration uh [30:11] to coordinate uh all these tasks in one [30:14] uh all these composable AI agents in one [30:16] pipeline so that's that's going to be [30:18] another gamecher and uh uh last but not [30:22] the least your infrastructure should be [30:23] able to support uh this autonomous uh [30:26] agents which means your infrastructure [30:28] should be should be should be uh ready [30:30] for agent uh uh like uh should be able [30:34] to support the agents you know like [30:36] agent ready. So uh those are the three [30:39] things and you you need to build your [30:41] infrastructure uh to be you know like [30:44] modular and it it also needs to be [30:46] interoperable and it also needs to be [30:49] observable. So those are the three [30:51] things that would uh you know like keep [30:52] your infrastructure scaling and uh [30:55] sustained as the technology evolves. [30:58] >> Great. Thank you Pak. [31:00] >> I think uh there were two questions in [31:02] your question. One is what is the energy [31:03] need? What happens to energy? And the [31:05] second is like evolution of the models. [31:07] Uh at the lowest atomic level, I think [31:10] uh you are converting electricity into [31:12] tokens. That's pretty much it, right? [31:14] Right? Like and so if you believe uh [31:16] that uh uh uh future AI is going to [31:20] basically carry out most of the task or [31:22] even let's say 50% of the tasks that [31:24] humans do today uh uh uh then the amount [31:28] of tokens that were going to get [31:29] produced either uh uh uh at the [31:32] inference time scaling or through [31:33] reasoning or through chain of thoughts [31:35] like whatever is the paradigm uh they're [31:37] just going to exponentially explode [31:39] right and so uh uh so like that's the [31:41] core hypothesis right because [31:42] intelligence uh uh is a different beast [31:45] like we have we have lived in [31:47] information age where the job of [31:49] technology was or the job of energy [31:51] electricity was to move data from point [31:53] A to point B that's pretty much it right [31:55] like but but intelligence is about der [31:57] there's a derivative on the data uh it's [31:59] like you're driving you're connecting [32:00] data points to drive intelligence and so [32:02] I think energy uh uh then so it then [32:06] connects to the to the training versus [32:08] inferencing laws uh most of the energy [32:11] that you see consumption today uh uh is [32:13] is on training side, inferencing side is [32:16] just picking up like if the energy [32:18] problem was solved, you talked about [32:19] GPT5 launch. I mean literally they would [32:21] give it for free or they would uh they [32:24] would basically have no rate limits on [32:25] GPT5 but there are rate limits like the [32:28] reason why there are rate limits is [32:29] because like with the data centers are [32:30] not there electricity is not there even [32:32] if you have GPU problem solved uh US [32:34] doesn't have the transmission grid of [32:36] like transferring actual electricity to [32:38] the data center uh and then you have all [32:40] sorts of issues on cooling and all and [32:42] so as I said we are in early innings of [32:44] this how does this evolve as I said in [32:47] my earlier remarks there's there's no [32:50] stopping to scaling laws right now. [32:52] There's there's no there's no there's [32:53] nothing in in the data points come out [32:55] that scaling laws are getting such so as [32:57] long as that paradigm still exist and [33:00] AIS have become too big to fail right [33:02] now. So you will like humanity will [33:04] continue to push the limits of scaling [33:06] laws till you see the first saturation [33:07] points and the moment so that means the [33:09] energy consumption is going to [33:10] dramatically explode at the training [33:12] layer also. I mean we thought that [33:13] training will saturate but it's not it's [33:15] not saturating and then as you get [33:18] closer to like closer to AGI we might [33:20] never theoretically get there but let's [33:22] say as you get closer to like [33:23] economically viable tasks as open AI [33:25] called about it you will see inference [33:27] explosion and so the load will shift [33:29] right and so you would then have like [33:32] companies like like AMD getting into [33:34] involved like Intel will do some [33:35] innovation uh and so you you have this [33:37] like Jivvon's paradox is going to play [33:39] out like dramatically you have like a [33:41] full faith on that uh so right now [33:44] energy consumption is on the training [33:45] side uh most of it uh it'll continue to [33:48] be there uh uh uh you'll see some shift [33:51] on energy on the inferencing side but [33:53] inferencing is a different beast [33:54] altogether it's like I don't even know [33:56] what is a multiple on that maybe 100x we [33:59] haven't seen anything like this in our [34:00] lifetime [34:01] >> exactly I think we had lots of [34:02] discussion earlier this year that it is [34:04] dramatic increasing of the cost and [34:06] energy consumption computer consumption [34:08] on the infrance side but also it will be [34:10] an interesting topic to think about if [34:12] is a small language model deploy on the [34:14] customer side deploy on the edge who's [34:16] going to pay for the inference cost is [34:17] that a technology provider or the [34:19] customer I think that's another way to [34:21] really calculate the cost of uh and also [34:24] sustainability cost of the AI [34:25] infrastructure [34:26] >> on that SLM comment I think uh uh one [34:29] can argue that look I need a larger [34:30] model to distill it down to create an [34:33] SLM rather than creating SLM from [34:35] scratch so so you still need even if [34:38] let's say you had hundreds of SLMs out [34:40] there you still need to deploy them [34:41] there it's going to be a different [34:42] infrastructure altogether to there's [34:44] needs to be a different engineering [34:46] paradigm just to orchestrate 50 SLMs [34:48] like I mean does your mobile phone [34:50] instead of apps are they all SLMs so the [34:53] Apple as a company look fundamentally [34:55] different the OS looks fundamentally [34:56] different right and so you keep on going [34:57] there and so the question is where [34:59] should uh where are we as a community [35:02] investing time and energy it's on [35:03] scaling loss [35:04] >> that's good [35:05] >> and that's where the energy needs are [35:07] >> exactly David [35:08] >> I might have a different I might have a [35:10] different perspective active on the [35:12] energy side. I've been involved in [35:13] largecale energy projects for for more [35:15] than a decade uh well before the AI poll [35:18] on those and really uh what we're seeing [35:21] in terms of the growth in energy demand [35:23] from training primarily right now and as [35:25] we're saying moving forward in inference [35:27] is really an opportunity. It's an [35:29] opportunity for the the US grid. It's an [35:31] opportunity for other energy [35:32] infrastructure around the world because [35:34] the reason that we have so much strain [35:37] on those transmission lines and systems [35:39] is that it's difficult to justify making [35:42] the investments in those upgrades [35:44] because consumer demand, regular [35:46] industry demand is very volatile. It's [35:47] up and down. It's all over the place. [35:49] But what we're seeing is a sustained [35:51] growing and without signs of stopping [35:54] need for this power and this [35:56] infrastructure which translates into a [35:59] lot of capital and a lot of investment [36:01] that really benefits the entire company [36:04] country's infrastructure and grid. So I [36:06] I view that as a tremendous opportunity [36:08] and just like the cost side uh [36:10] optimizations on the cost side of AI, [36:13] we're not quite there yet, right? [36:14] Everybody's still running for the [36:16] future. we're not overly concerned about [36:18] what we're spending. That will come and [36:20] when that comes, we will f we will focus [36:22] even more than we are now on energy [36:24] optimizations, cost optimizations, and [36:26] then we'll reap the benefits of those [36:28] investments that we made during the [36:29] growth period. So, I think it's net a [36:31] positive for the overall uh ecosystem. [36:35] Beyond that, um, on the model side, I do [36:38] agree that we're going going to see more [36:40] and more shift towards these specialized [36:42] small models and we are going to see [36:45] them running more on the edge. That [36:47] energy, whether it runs in cloud or on [36:49] the edge, that energy cost is still out [36:51] there. It may better be distributed with [36:54] the existing grid and the existing [36:56] systems, but it's still going to take [36:57] the power. Um so there's a challenge of [37:01] whether on-prem and especially consumers [37:03] are prepared for that. Um if you buy a [37:06] new microwave or a refrigerator, you it [37:08] has a sticker on it how much energy it [37:10] uses. Well, nobody is going to tell you [37:12] what your on in-house uh specialized AI [37:16] agent is going to take in terms of [37:18] electricity. So that will be an [37:19] interesting thing to see evolve as time [37:21] passes. I would say I would really like [37:24] how positive you are because we always [37:25] talk about being entrepreneur you need [37:27] to always stay positive but I live and [37:29] based in California and we have lots of [37:31] complain about the local electrical [37:33] grilled and uh and also heard [37:35] interesting stories some of the campus [37:37] just went power outage because they were [37:39] training AI model. So we hope that there [37:41] will be new technology as you said more [37:43] incentive is also sense of urgency now [37:46] we really need to upgrade it to support [37:47] the future of AI. I think every anything [37:50] any rapidly growing opportunity is going [37:52] to have growing pains and that's one of [37:54] the growing pains but I think the [37:55] long-term benefits will be there. [37:58] >> Exactly. Yeah. Derek. [38:00] >> Yeah. I I also think this is a you know [38:02] an excellent opportunity for uh the US [38:05] grid and uh really the worldwide grid to [38:08] uh level up. We're already seeing uh [38:11] resurgence in uh nuclear projects [38:13] happening now. uh green energy [38:15] generation like um more than the past 10 [38:19] years combined right suddenly people [38:21] want to do because there's this new [38:23] stable demand for energy so that's [38:25] really exciting to see and I think [38:26] that'll be a net benefit for everyone um [38:30] however at this point the US and most [38:33] countries are not yet out of power um [38:37] we're getting a little more creative in [38:38] how to find power there's a lot of [38:40] stranded power um and right now AI labs [38:44] large builders are taking advantage of [38:46] those opportunities. That's the [38:47] lowhanging fruit. There will be a time [38:49] in the not too distant future when that [38:51] becomes a lot more difficult and all of [38:54] the grids actually essentially run out [38:56] of power and there are no more one [38:58] gigawatt sites that can be developed in [39:00] any reasonable amount of time and at [39:04] that point we're going to see a lot more [39:05] stress uh I think on making the [39:08] architectures more efficient right [39:11] ultimately necessity breeds innovation [39:13] right now there's not a a huge push [39:16] there's not a huge reason to um really [39:19] rethink the architecture because we can [39:21] just throw more power at it. But once [39:24] that stops being the case, uh I think we [39:26] will see some foundational shifts in how [39:29] we do AI, how we do clusters, uh [39:31] multiple times we'll have that type of [39:33] shift. And we know that's going to [39:34] happen because if you look at the power [39:37] consumption of the best AI model, uh and [39:40] the power consumption of the human [39:41] brain, they're like, you know, many many [39:44] orders of magnitude different. So we [39:46] know for sure without a doubt there are [39:48] optimizations that can be made. And so [39:51] uh we think we'll start to see those [39:53] happen. You know every few years there's [39:54] going to be a foundational shift in how [39:56] we do AI. Um and with that it's going to [39:59] come a whole new boom right. Uh so we [40:02] have many of those cycles left before we [40:04] even approach something that we would [40:06] describe as like a scalable AGI which [40:09] ultimately means u there's a a long road [40:11] left in this business. [40:13] >> Yeah. I'm glad you mentioned the example [40:15] of a human brain because most of people [40:17] didn't realize the most energy efficient [40:19] machine are like a intelligent machine [40:21] we have in the whole world is actually [40:22] our human brain the subconscious [40:24] cautious logic work perfectly together [40:26] and will be able to handle so much [40:28] complex the problem and I recently [40:30] talked with the team that they actually [40:32] are adopting this neurological approach [40:35] design the new model structure and of [40:37] course it's still very early at least [40:39] based on the they don't have benchmark [40:41] comparation yet because model is not [40:42] launched but they claim they can reduce [40:44] energy consumption by minimum 1,000x and [40:47] also we're seeing some of the other [40:49] company trying to do analog computing [40:51] try to use analog computings to see [40:52] whether they can reduce energy [40:54] consumption 100x or more for AI adoption [40:56] so I think there's some promising [40:58] opportunity within model layer as well [41:00] and coming back to help us make the [41:02] challenge of the energy layer uh [41:04] slightly small because the adoption [41:06] going to be bigger uh meanwhile we can [41:08] have more hybrid efficient you know AI [41:10] infrastructure to support the future So [41:13] yeah, we're coming to I got a signal [41:15] that we're almost done with the panel. [41:17] We still have a three more minutes to [41:19] go. So my last question to all the [41:21] panelists. Let's really kind of uh [41:24] looking at the future. I want to ask you [41:26] guys to give some really bold prediction [41:28] about the future. Uh maybe not in 2030 [41:31] that's too far away. Consider how fast [41:33] AI evolving just in the next three [41:35] years. give me one or two bold [41:37] prediction about uh evolution of the AI [41:40] infrastructure and also probably one [41:42] suggestion I think we have lots of the [41:44] entrepreneur here in this conference [41:47] what is the one key advice you want to [41:49] give entrepreneur in terms of the key [41:51] decision they need to make when they're [41:53] no matter building their AI native [41:54] solution and uh and also in terms of the [41:57] how to choose the right infrastructure [41:59] to start with so I will start with you [42:01] Rajash [42:02] >> I'm definitely excited for the [42:04] convergence of AI and uh event-driven [42:07] systems where uh the agents can learn, [42:10] adapt and action in real time, you know. [42:13] So that's that's what I'm looking out [42:15] for and at the same time uh yeah uh [42:18] build don't build for uh you know like [42:21] perfection uh build for the evolution [42:23] you know because this technology is [42:25] going to change super fast. It is [42:27] evolving every day. So uh don't try to [42:30] you know like uh perfect uh your your [42:32] models uh rather uh build in a modular [42:36] way and uh you know like um uh have [42:40] obserability and interoperability as I [42:42] said earlier in your workflows uh and [42:45] always work backwards from you know like [42:47] uh a use case rather thinking okay hey [42:49] I'm going to build uh something to solve [42:52] every open-ended use case you know so [42:54] that's not how uh AI works. Yeah. [42:57] >> Yeah. want to leave that that solution [42:59] to Amazon, right? To solve it. Yeah, [43:02] there's an interesting collaborating and [43:04] competitive uh relationship between a [43:06] large uh tech company and a smaller [43:08] startup right now. Okay, [43:09] >> I'll be quick. There's a minute left. Uh [43:11] I think the future will be agentic. I [43:12] really want an agent that can plan my [43:15] vacation. [43:16] >> So, uh uh agentic commerce will become a [43:19] thing. So, that's my bold prediction. uh [43:22] uh the advice is uh think about uh if [43:25] the future there going to be more agents [43:27] than humans on the internet then think [43:29] about the infrastructure for that agency [43:31] that I talked about right like how will [43:33] agentto agent communication happen what [43:35] will those protocols look like sort of [43:37] those those that's my advice [43:38] >> right [43:41] >> on the bold prediction I think that [43:42] we're going to see agentic AI expand [43:44] into the physical world where those [43:45] agents aren't interacting digitally but [43:47] they're interacting in robotics and in [43:49] other pieces like that and I I think [43:51] that's going to be very profoundly [43:52] altering to a lot of the world. Um, as [43:55] as far as advice, um, I I would say [43:59] echoing on much of what has been said [44:01] here, [44:03] follow your use case. Look at what you [44:05] need to accomplish. Start there and then [44:08] think about how AI can help. Don't try [44:10] to just throw AI at your existing [44:12] business as it is today. [44:14] >> Great. Thank you, Derek. Uh yeah, I [44:17] think what we're going to see, we're [44:18] going to see uh small companies, small [44:21] groups of people suddenly be able to [44:23] achieve uh much much more than they have [44:25] been able to in history. We're going to [44:27] start seeing, you know, companies that [44:28] have five people uh achieve billion [44:30] dollar valuations in record time. And [44:33] we're already starting to see some of [44:34] this, right? Some of the AI startups are [44:36] are hitting a hundred million ARR in [44:38] months, right? Um so I think that trend [44:41] is going to continue for some time at [44:43] least and we're going to see uh a bunch [44:45] of innovation in a very short amount of [44:47] time. [44:48] Yeah, I think as we all discussed, you [44:51] know, there will be lots of evolution on [44:52] the gent AI side and there will be lots [44:54] of evolution on the physical AI side and [44:56] also think about in the future. I think [44:58] some of you mentioned that the new small [45:00] model may not only run on top of a GPU, [45:02] some of them will run on CPU more [45:04] efficiently and also on top of that uh [45:07] we're talking about transformer model [45:09] all the time right now. Three years from [45:10] now is it the major model we're going to [45:12] discuss whether there's a new model [45:14] architecture gonna came out launching [45:16] the market. This is all the possibility [45:18] going to happen. That's also the beauty [45:20] of this AI ecosystem. And uh go back to [45:22] what I mentioned at the beginning. Yes, [45:24] we wanted to have the best AI solution, [45:26] but I also want to make it cost [45:28] efficiently. So making the right [45:29] decision on the AI infrastructure, be [45:31] aware of the evolving mindset of [45:33] building your product and adapt to the [45:35] new AI infrastructure layer is really [45:37] critical. So hopefully you have uh lots [45:40] of good learning from this panel. It's a [45:42] very dynamic and inform informative [45:44] discussion. So thank you for all the [45:46] panelists for joining us today and feel [45:48] free to reach out to them after the [45:49] panel to ask questions. Thank you. [45:52] [Applause]