Enterprises Pivot to Older GPUs, Edge AI as Power Bottlenecks Mount

电力瓶颈加剧，企业转向旧款GPU与边缘AI 急剧增加的电力限制与昂贵的前沿API正迫使基础设施负责人延长旧款GPU的使用寿命，并将复杂的企业工作流引导至本地化小模型。 专家小组探讨了扩展企业AI算力的运营与经济现实，权衡了激进的现代化进程与诸如DORA等严格监管要求以及物理数据中心瓶颈之间的平衡。与会嘉宾剖析了NeoCloud扩展机制、配电限制以及多租户安全障碍，强调了在边缘设备、本地私有集群与外部基础模型之间划分工作负载的重要性。他们打破了有关GPU迅速过时的普遍论调，阐明了为何旧款硬件和小语言模型在构建可持续、高性价比的企业AI架构中依然发挥着至关重要的持续作用。 企业推进 AI 落地的首要瓶颈在于数据治理与合规审查，导致斥资数百万美元采购的高性能算力集群在通过安全控制和数据流验证前长期处于闲置状态。 (6:29) 受经济效益驱动，企业正加速从每百万 Token 约 10 美元的高昂基础模型 API，转向单位成本仅 0.20 至 2.00 美元的 NeoCloud 及自托管 Serverless 推理方案。 (10:08) 供电瓶颈已取代物理空间成为数据中心的主要制约因素，电力限制迫使运营商每机架往往仅能部署 1 至 2 台高密 GPU 机箱，且变压与交直流转换损耗高达总能耗的 13% 至 30%。 (13:42) 智能体（Agentic AI）正推动边缘、本地与云端模型的动态任务分拆，但自主执行循环与 MCP 协议调用使遥测数据量激增百倍，为企业带来高额的云端网络出站（Egress）成本压力。 (20:05) 算力淘汰恐慌被严重夸大：GPU 服务器在经历最初两周的老化测试（Burn-in）后硬件故障率骤降，H100 等主力加速卡在生产环境中的实际服役寿命可长达 8 年。 (24:27)

Scaling enterprise AI requires shifting from raw compute expansion to disciplined workload decomposition, multi-tier hardware architectures, and hybrid edge-to-cloud governance. Industry practitioners emphasize that operational economics, power limits, and regulatory scrutiny now dictate infrastructure strategy more than frontier chip access. Governance and Regulation as Compute Bottlenecks (2:21) Enterprises often face operational paralysis despite heavy infrastructure investments. Pratik Jain points out that organizations frequently deploy millions of dollars in compute capacity that remains idle because security, compliance, and legal teams cannot audit agentic execution or data provenance. The emerging agentic paradigm compounds this complexity by decomposing workflows across edge devices, private data centers, and external foundation models, requiring an integrated control plane covering context management, Model Context Protocol (MCP) interactions, and tracing. Regulated industries operate under even narrower margins for error. Frank Kartmann of Deutsche Bank highlights that European banks must comply with the Digital Operational Resilience Act (DORA), which mandates strict ICT risk controls and operational resilience. To protect sensitive financial data while continuing to scale compute, banks must build multi-cloud and hybrid environments that preserve data sovereignty and avoid costly operational stalls. Physical Facility Limits and Power Conversion Losses (13:42) At the data center level, delivering adequate electrical power has superseded physical rack capacity as the primary bottleneck. Greg Wester of RunPod notes that while modern racks can physically house up to seven high-density server chassis, operators are often restricted to one or two per rack due to facility power delivery constraints. While future data center concepts showcase 200 kW liquid-cooled cabinets, operators today navigate real-world constraints by standardizing fleets into practical tiers, such as NVIDIA RTX 6000 Ada models for developers and flagship platforms like the H-series and B300 for sustained enterprise demand. Energy waste further deteriorates unit economics before power even reaches silicon. Stepping high-voltage alternating current (AC) down to direct current (DC) creates transformer conversion losses between 13% and 30%. Infrastructure operators must optimize Power Usage Effectiveness (PUE) and redesign power delivery stages to curb electricity overhead, especially as enterprises transition away from subsidized API pricing toward cost-managed production tiers. Workload Partitioning and Debunking Hardware Obsolescence (20:05) The panel challenges the premise that enterprise AI demands the newest flagship accelerators across all workflows. Instead of deploying top-tier chips for non-critical tasks, enterprises are tiering hardware across development, testing, and production. Mainstream scale-out inference relies heavily on utilitarian GPUs like the NVIDIA L4 and L40S. Industry data also indicates that hardware failure rates drop sharply once accelerators pass an initial two-week burn-in window, allowing workhorse GPUs like the H100 to remain productive for upwards of eight years despite short financial depreciation cycles. Deployments increasingly follow workload-specific placement models. Bursty inference and managed experimentation default to public clouds, continuous high-volume and compliant pipelines remain on premises, and low-latency or personalized tasks reside on edge hardware. Local edge parsing using Small Language Models (SLMs), vector stores, and knowledge graphs also mitigates context degradation seen in long-window foundation models while avoiding the ballooning egress fees generated by continuous agentic tool calling. Predictions: Continuous Fine-Tuning and Hybrid Architectures (33:13) Looking toward 2027, panelists project a transition toward automated, localized, and economically viable systems. Greg Wester predicts the emergence of automated, continuous retraining loops that empower standard software engineers to sample, label, and fine-tune specialized smaller models daily, reducing reliance on premium frontier APIs. Pratik Jain anticipates a cyclical return to client-server paradigms, shifting execution away from centralized H100 'mainframes' toward on-device neural silicon. Architecturally, Frank Kartmann forecasts fluid infrastructure fabrics where enterprise workloads shift dynamically among private clusters, public hyperscalers, and edge devices. However, Kartmann cautions that the industry has yet to hit its financial breakeven point, leaving open the question of when broad enterprise AI deployments will demonstrate sustained business profitability. While physical power delivery and hardware availability remain acute operational hurdles, the sustainable scaling of enterprise AI ultimately hinges on software governance, workload right-sizing, and proving long-term unit profitability.

在Ai4研讨会上，来自金融机构、新兴云服务商及数据中心基础设施领域的专家指出，企业级AI计算正面临电力交付短缺、高昂API成本以及严格合规审计的三重瓶颈，正促使行业从盲目采购顶级算力转向端云协同、分级复用和精细化治理的务实架构。 金融级合规与物理供电的双重制约 (2:21) 企业落地AI的首要障碍往往不在于模型本身，而在于基础设施与监管合规的冲突。Deutsche Bank副总裁Frank Kartmann指出，一级银行必须遵循欧洲《数字运营韧性法案》（DORA）等严苛的ICT风险管理与审计要求，无法像初创公司一样无约束地调用外部云端。自建本地算力集群需要高密度电力配电、液冷系统及低延迟NVMe存储网络，但硬件供应链经常出现延宕。KONST系统架构师David Wacker进一步揭示了机房物理层面的结构性瓶颈：由于现有数据中心单机柜供电能力不足，即使物理空间能容纳7台高密度GPU服务器，运营商通常也被迫限制在每机柜仅部署1至2台。业内展示的200 kW液冷机柜仍属未来愿景，而非当前常态。 API成本倒逼企业向自托管与边缘推理迁移 (10:08) 尽管前沿模型实验室验证了AI潜力，但高昂的接口调用价格引发了企业的经济核算危机。RunPod产品管理总监Greg Wester透露，企业客户极力寻求将每百万Token成本从托管API的10美元压低至0.20至2.00美元。为维持每年200%的扩张速度，RunPod每周需接入约1 MW电力与50 PB存储，但若一次性投放1,000至10,000张NVIDIA H100显卡而未锁定预付订单，全网利用率可能瞬跌15%，直接吞噬营业利润。同时，随着推理负载超越模型训练，利用小语言模型（SLM）结合边缘解析正成为控制成本的主流手段。Auvern.ai创始人Pratik Jain指出，长上下文窗口在多轮对话中存在注意力衰减，采用端侧解析配合向量库与知识图谱，仅在必要时回退至前沿大模型，能大幅削减昂贵的云端Token消耗。 智能体架构驱动控制平面治理与数据分流 (6:29) 计算范式正经历从互联网、云计算向智能体（Agentic）时代的第三次迁移。Pratik Jain根据其在大型投行的管理经验分析，企业购买的数百万美元算力之所以闲置，核心在于安全与法律团队无法审查数据流动与模型控制机制。智能体执行任务需要将工作负载拆解并路由到边缘端、私有集群与云端大模型，打通模型上下文协议（MCP）、调用跟踪和语义上下文层。此外，自主智能体在持续工具调用中产生的中间遥测与操作数据量可能暴增100倍以上，极易引发巨额的云网络出站传输费（Egress Fees），迫使金融及受监管机构将高敏感数据严格锁定在本地，形成按数据敏感度动态路由的混合云治理框架。 打破硬件淘汰神话与存量算力梯次复用 (24:27) 针对算力折旧与硬件损坏的恐慌，专家们给出了反驳证据。数据显示，企业级加速卡在度过前两周“老化测试（Burn-in）”并更换初始不良品后，硬件故障率会呈数量级下降。业内大规模推理生产环境目前仍大量依赖NVIDIA L4和L40S等实用型显卡，而非一味追求最新架构。即便财务报表按3至5年加速折旧，类似H100的稳定芯片在生产周期中往往能持续服役8年以上，云平台上甚至仍有9年前的旧卡在平稳出租。Pratik Jain与Frank Kartmann共同建议企业将开发、测试与生产环境分级，通过软件编排层构建异构算力集群，复用旧代硬件，以此对冲不断上涨的硬件资本开支。 小组专家预测，到2027年，AI算力将呈现持续微调流水线自动化、负载在边缘与云端动态流动，以及计算形态回归端侧芯片的趋势；但Kartmann强调，全行业目前仍未跨越投资回报的盈亏平衡点，广泛的企业级部署能否转化为可持续的长期商业利润仍是未解难题。

## Background 该视频由 Ai4 于 2026 年 8 月 22 日发布在 YouTube 平台。视频记录了一场围绕“扩展企业级 AI 算力”的圆桌讨论，汇聚了来自投资机构、德意志银行、NeoCloud 服务商（RunPod、KONST）以及 AI 治理平台（Auvern.ai）的行业领袖，重点探讨了芯片选型、机房能耗与成本优化、分布式部署以及如何为企业构建跨云、本地与边缘的算力底座。 ## TL;DR 本场圆桌聚焦于企业 AI 基础设施扩展中的现实工程与架构挑战。嘉宾们深入探讨了金融级强监管（如欧盟 DORA 法案）下的混合多云决策、数据中心机架功耗与 AC/DC 电力转换损耗瓶颈，以及多租户环境下的 KV Cache 隔离与治理。讨论破除了 GPU 短期内必然淘汰的误区，强调通过按工作负载拆分和异构算力复用来提升投资回报率，并分析了边缘端上下文管理及轻量化小模型的协同机制。最后，嘉宾们展望了模型微调全流程自动化的普及趋势，并呼吁行业加快跑通商业化与财务盈利闭环。 ## Atmosphere & style 整场对话风格务实、硬核且充满产业一线的真实感，彻底剥离了浮躁的 AI 概念炒作。主持人的节奏把控精准，来自银行架构师、云基础设施操盘手与创业者的互动自然流畅，既有对电网损耗与供应链物流的严肃工程剖析，又不乏将液冷机架调侃为“水上乐园”、把成熟芯片比作“耐开的丰田老车”的幽默风趣。聆听体验如同置身于一场高密度、高信息量的行业闭门技术复盘，兼具前瞻视野与脚踏实地的工业级洞察。 ## Key takeaways - [02:21](https://www.youtube.com/watch?v=a81Xc1zACB0&t=141s) 企业在强监管环境（如欧盟 DORA 法案）下扩展 AI 时，平台现代化与合规性是核心痛点，促使金融机构采用混合多云架构并依据数据敏感度动态调度算力。 - [06:29](https://www.youtube.com/watch?v=a81Xc1zACB0&t=389s) 智能体时代（Agentic Era）引发了工作负载切分的深层裂变，企业落地 AI 最致命的瓶颈往往不在于硬件采购，而在于构建完善的合规与治理控制平面（Control Plane）。 - [10:08](https://www.youtube.com/watch?v=a81Xc1zACB0&t=608s) NeoCloud 算力服务商面临超高速硬件迭代与极复杂的物流运营挑战，算力交付爬坡必须与客户确定性需求精准匹配，避免集群空置导致利用率暴跌。 - [13:42](https://www.youtube.com/watch?v=a81Xc1zACB0&t=822s) 数据中心受限于单机架电力输入瓶颈与高达 13% 至 30% 的交直流转换损耗，未来通过新型直供电架构与优化 PUE 降低能耗是算力降本的关键路径。 - [17:32](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1052s) 端侧 AI（Edge AI）和小模型流水线正成为缓解算力压力的重要方向，在多租户云推理环境中，严格隔离各租户的 KV Cache 是保障数据资产安全的关键底线。 - [20:05](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1205s) 混合部署决策应兼顾敏捷度与合规性：公有云适合弹性突发任务，本地机房适配高敏感可预测负载，而边缘端承载低延时隐私需求；智能体操作性数据激增正重构网络传输成本。 - [24:27](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1467s) “GPU 短期迅速淘汰论”并不符合实际，硬件故障大多集中在前两周老化期（Burn-in），成熟芯片如 H100、L4/L40S 仍将在长达数年的周期内持续提供性价比与生产价值。 - [26:56](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1616s) 企业无需在所有场景使用顶配算力（“丰田汽车隐喻”），应按照开发、测试、生产不同环境精细化拆分负载，并通过聚合管理平台实现退役硬件的二次复用。 - [29:42](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1782s) 算力重心正加速向边缘推理倾斜，利用端侧小模型与向量图谱进行本地上下文管理（Context Management），能有效规避大模型长对话上下文丢失并降低前沿大模型调用成本。 - [33:13](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1993s) 行业未来预测与期许聚焦于模型微调全链路自动化的普及、算力周期从集中式大型机向端侧边缘回归，以及全行业亟需跨越狂热期实现真实的财务盈亏平衡（Break-even）。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=a81Xc1zACB0&t=4s) 圆桌讨论开场与嘉宾背景介绍 主持人 Ryan Wang 首先对论坛主题进行了开篇介绍，本次圆桌的核心聚焦于“扩展 AI 基础设施”（Scaling AI Infrastructure）。Ryan 提到自己在硅谷从事风险投资已有 11 年，目前掌管两只基金：一只是专注于早期前沿科技投资的 VC 基金 Outpost Capital；另一只是两年前联合创立的私募股权基金（PE），主要投资方向正是 AI 基础设施与数据中心。由于自身投资实践与当前 AI 基础设施的爆发紧密相连，他对本场讨论非常期待，并邀请四位嘉宾依次做自我介绍： - **Greg Wester**：RunPod 产品管理总监。RunPod 属于新一代云服务商（NeoCloud），主要提供 Serverless 推理与模型训练等算力产品。 - **David Wacker**：KONST 系统架构师。KONST 是一家总部位于中国台湾的 NeoCloud 算力企业，业务主要是面向全球构建和部署 AI 数据中心。 - **Pratik Jain**：Auvern.ai 创始人兼 CEO。公司致力于打造企业级控制平面（Enterprise Control Plane），用于大规模运行、管理和治理 AI 智能体（Agent）。 - **Frank Kartmann**：德意志银行（Deutsche Bank）大数据与云架构副总裁、平台架构师。德意志银行作为总部位于德国的欧洲知名银行，在美洲（纽约、北卡罗来纳、佛罗里达等地）也设有大量机构。他主要负责运营大数据平台，为全行提供 AI 与数据分析能力支撑。 --- ### [02:21](https://www.youtube.com/watch?v=a81Xc1zACB0&t=141s) 金融业的基础设施挑战：强合规、平台现代化与供应链痛点 Ryan 将首个问题抛给来自德意志银行的 Frank，询问在强监管、高门槛的金融服务业中，扩展 AI 基础设施意味着什么，以及当前面临的核心硬件与算力优先级是什么。 Frank 坦言，对于银行和保险公司而言，当下面临的最大痛点是“平台现代化”（Platform Modernization）与“AI 赋能利用”（AI Utilization）。金融业的数据资产规模正以极快速度膨胀，机构不仅要收集海量数据，更渴望深度挖掘和处理这些数据，这对底层的存储与算力规模提出了前所未有的要求。但与此同时，银行业对平台稳定性有着近乎苛刻的要求。企业必须在持续向原有系统追加存储与计算资源的同时，兼顾平台的稳健性、网络安全、韧性（Resilience）以及商业层面的经济可行性（Economic Viability）。 在欧洲金融业，监管约束尤为关键。Frank 特别提及了欧盟出台的《数字业务弹性法案》（DORA，Digital Operational Resilience Act），该法案强制要求金融机构严格执行 ICT 风险管理和运营韧性保障。为了应对业务与合规挑战，各大银行目前正集中推进以下几项核心架构转型： ```mermaid flowchart TD accTitle: 金融机构 AI 混合多云部署决策流 subgraph Governance["合规与风险评估 (DORA 等法规)"] A["业务负载 / 数据输入"] --> B{"敏感度与合规审查"} end subgraph Deployment["多云与本地协同架构"] B -- "高合规 / 超敏感数据" --> C["本地数据中心 (On-Prem)"] B -- "弹性 / 通用负载" --> D["公有云多云架构 (Multi-Cloud)"] C --> E["本地高密机房设施<br>(高功耗 / 液冷 / NVMe / 低延迟网络)"] D --> F["云原生弹性架构<br>(跨云迁移 / 算力动态调度)"] end ``` - **全面拥抱云原生与混合多云**：构建灵活、可扩展且能够在不同公有云之间平滑迁移的云原生应用，同时结合本地环境打造混合多云（Hybrid & Multi-Cloud）架构。 - **依据数据属性动态调度负载**：根据数据敏感度（Data Sensitivity）、合规约束以及成本结构，在不同云厂商及本地基础设施之间动态分配算力负载。 - **本地机房高门槛基建改造**：若在本地部署并运行 AI 负载，就必须自行解决数据中心的物理机房配电、高密散热制冷（Cooling）、电力供应等基础设施问题；而在数据平台层，必须采购昂贵的高性能低延迟存储（如 NVMe）、超低延迟通信网络以及高算力 GPU 集群。 - **严峻的供应链断裂问题**：当前硬件与算力市场极度扭曲，供应链高度受限，采购与交付难度极高。 --- ### [06:29](https://www.youtube.com/watch?v=a81Xc1zACB0&t=389s) 从摩根大通到创业：智能体时代的企业控制面与分层治理 Pratik 曾在摩根大通（JPMorgan Chase）和富国银行（Wells Fargo）等华尔街头部金融巨头工作数十年。Ryan 顺着银行业的话题提问，好奇是什么契机促使他离开传统银行体系下海创业，以及在金融业等企业场景中扩展 AI 算力遇到了哪些具体困难。 Pratik 深入分析指出，企业在落地 AI 基础设施时，最致命的障碍往往不是底层硬件的采购，而是“治理”（Governance）与“合规控制”（Compliance）。企业即使斥巨资把 GPU 机架和集群搭建完毕，但如果缺乏完善的合规与治理层，这些基础设施最终只会空转闲置，根本无法交付给业务部门投入生产。打造一层强大的控制平台，让团队在合规前提下快速迭代，是他创立 Auvern.ai 的核心驱动力。 在 Pratik 看来，技术演化至今经历了三次重大技术范式转移： 1. **互联网时代**：引发了数据存储与管理方式的全面重构； 2. **云计算时代**：推动了企业将本地 IT 基础设施大举向云端托管迁移； 3. **智能体时代（Agentic Era）**：引发了算力与工作负载切分模式（Workload Splitting）的深层次裂变。 在如今的智能体范式下，架构设计不再是简单地把任务整体搬到云端某台服务器，而是需要将复杂的复合链路彻底拆解：部分工作负载部署在边缘端（Edge），部分部署在企业私有云（On-Prem），而底座基础大模型（Foundation Models）则可能托管在第三方云端。此外，不同环节的技术指标截然不同——模型预训练依赖于特定的高密算力拓扑与存储吞吐，而模型推理和实时 Agent 调度则有着完全相异的延迟与吞吐要求。 Pratik 强调：“正如你常听到的比喻，你没必要为了解决所有的业务场景都去开一辆法拉利（You don't need a Ferrari to drive and solve for every business case）。”企业必须在算力基础设施、模型规格（从小模型到大模型）以及治理层之间寻找最优性价比组合。 在系统落地的技术复杂度上，Pratik 指出当前的难点在于异构信息流的高效缝合与重构： - **可观测性数据（Observability）**：智能体执行时产生海量的调用追踪（Traces）； - **模型与协议交互层**：频繁的大模型推理调用、MCP（Model Context Protocol）工具调用以及操作型业务数据； - **上下文管理（Context Management）**：全套语义层（Semantic Layer）的动态构建与维护。如何将这些复杂交错的组件融合在同一治理体系内，是当前企业落地 Agent 的核心痛点。 --- ### [10:08](https://www.youtube.com/watch?v=a81Xc1zACB0&t=608s) NeoCloud 视角的演进：台湾硬件前沿与硅谷超高速扩张的现实挑战 随着话题转向新兴云服务商（NeoCloud），讨论聚焦于两位分别来自中国台湾与美国硅谷的行业实践者。Ryan 询问 KONST 的 David：立足台湾这一全球半导体与算力制造中心，KONST 感受到的产业痛点是什么，与亚洲生态又有怎样的协同。 David 阐明了地缘近邻带来的技术洞察优势。因为身处中国台湾，KONST 与各大硬件原始设备制造商（OEM）保持着极近的物理与研发距离。由于所有 AI 服务器与算力硬件基本都在当地生产制造，他们不仅深度扎根于制造生态，更能够直接紧跟全球最前沿的硬件工程研发节奏，提前把握下一代架构脉搏。 但这也带来了严峻的生命周期挑战。当前 AI 硬件与 GPU 架构的迭代迭代周期被压缩得极短：团队眼下正在全球规模化交付基于 Blackwell 架构的 B300 集群（当前主力部署代次），但产业链和原厂已经紧锣密鼓地研讨起下一代计算平台。在这种狂暴的技术迭代压力下，数据中心架构师必须深思熟虑：如何规划数据中心的机柜空间、供电管线、冷板与冷却系统，确保所建成的基础设施在未来 3 至 5 年内依然具备向下兼容与技术可用性，而不至于刚建成就面临淘汰并被迫推倒重来。 随后，视线转回硅谷的 RunPod。Greg 描述了当下算力租赁与云服务市场的狂热现状。他直言来自客户的算力需求已经“爆表”（Off the charts），同时对 OpenAI 和 Anthropic 等模型厂商表示敬意，正是这些先锋实验室向全球企业和大众普及了 AI 模型的无限潜力。然而，狂热伴随着极度务实的成本焦虑——客户的真实心声是：模型能力确实惊艳，但他们无法承受每百万 Token 支付 10 美元的高昂代价，客户急迫地探寻如何将成本打压至 2 美元乃至更低。 为了支撑每年高达 200% 的业务复合增长，RunPod 面临着近乎极限的扩张指标： - **每周并网接入 1 兆瓦（MW）算力功率**； - **每周上线 50 PB 的存储资源**。 Greg 指出，过去这种扩张主要受限于资本与资金，而现在最残酷的挑战变成了**极其复杂的物流与运筹调度（Logistics & Operations）**。如果企业一次性把 1,000 张甚至 10,000 张 NVIDIA H100 GPU 仓促上线投产，瞬间涌入的大规模供给可能会导致机房算力利用率骤降 15%，造成巨大的资本闲置消耗。因此，如何在算力交付爬坡与企业客户确定性需求之间建立毫厘不差的动态平衡，已经成为一个高难度的供应链运营课题，其复杂程度甚至超过了单纯的计算机体系结构工程。针对 Ryan 追问的现有基础设施瓶颈问题，Greg 赞同全球选址时虽然某些地区拥有充沛的电力供给，但网络基础设施与全链路配套条件往往会成为限制算力集群有效铺设的关键瓶颈。 ### [13:42](https://www.youtube.com/watch?v=a81Xc1zACB0&t=822s) 数据中心能耗瓶颈、机架功率与能源转换损耗 在分布式多区域数据中心部署中，虽然选址具有灵活性，但网络带宽和电力供应往往是最大的短板。客户把数据传入传出数据中心时很容易遇到瓶颈，因此构建能够保障服务质量（QoS）的多区域、多数据中心架构至关重要。谈及能效与单位经济效益（Unit Economics）的平衡时，Greg Wester 指出，当前已有数据中心面临的最大物理限制在于机架供电能力——物理空间虽然能放下多达 7 台 GPU 服务器，但现有基础设施的电力供应往往只能支撑 1 到 2 台。尽管英伟达在发布会上展示了功率高达 200 千瓦、配备华丽液冷管道的全新一代机架（被调侃为“看起来就像个水上乐园”），但这代表的是未来的愿景，并非当下所有数据中心的现状。为了应对这一挑战，基础设施平台通常会在硬件 SKU 上进行标准化规划：针对开发者与专业级消费者（Prosumer），将 RTX 6000 Pro 这类显卡按显存规格进行逻辑切分；针对企业级客户，则主推 B300 等核心规格，以实现稳定的供需预期。 随后，David Wacker 补充了从商业周期与物理基础设施维度的观察。目前 AI 领域的许多服务对最终用户而言几乎是免费的，用户在获客阶段缺乏考虑能耗和成本的动力。但随着商业模式从“圈地获客”转向“实现盈利”，企业将不得不主动审视并削减算力与能源成本。除了依靠算法层面的模型轻量化，基础设施本身也蕴藏着巨大的降本潜力，例如优化 PUE 指标（即机房总能耗中实际供给 GPU 算力的比例与支撑性基建能耗的比例）。此外，电网高压电经由变电到交直流（AC/DC）转换过程中，因反复转换所造成的电能损耗高达 13% 到 30%。随着新型无缝直供电等架构技术的成熟，未来无需经过多重转换即可直接为计算设备供电，能效将在不牺牲算力的情况下迎来跃升。 - **机架电力密度瓶颈**：传统机房单机架物理空间虽可容纳约 7 台服务器，但受限于电力输入，通常仅能部署 1-2 台高密 GPU 设备。 - **下一代机架规格**：英伟达新一代液冷机架功率达 200 kW。 - **硬件分级策略**：开发者生态主打 RTX 6000 Pro 及切分显存实例；企业端则以高端专用 GPU 作为主力支撑。 - **电力转换隐形损耗**：从高压输电线路到机房交直流转换环节，电能损耗目前约为 13% 至 30%。 --- ### [17:32](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1052s) 端侧 AI 崛起、数据主权与多租户 KV Cache 隔离 随着大模型落地进入深水区，端侧 AI（Edge AI）和小语言模型（SLM）正在成为缓解中心化算力压力与合规顾虑的关键趋势。Pratik Jain 结合自身在金融与强监管行业的背景指出，企业部署大模型时最核心的关切在于：数据存放在哪里、归谁所有、是否会被外部模型调用甚至用于训练。面对专有数据资产的保护诉求，端侧推理展现出巨大潜力，微软等科技巨头已开始研发专属推理芯片并将其直接内置到个人电脑和本地服务器中。 与此同时，构建 Agent 智能体及机器学习流水线并非仅依赖昂贵的超大语言模型（LLM），整个流程需要组合嵌入模型（Embedding）、稀疏编码（Sparse Encoding）等多种轻量化模块，这些任务完全可以在更具性价比的异构算力上运行。Pratik 还特别提出了一个关键的安全架构细节——多租户环境下的 KV Cache 隔离： ```mermaid flowchart TD accTitle: 多租户大模型推理与 KV Cache 隔离架构 subgraph ClientLayer["客户端与端侧部署"] A["端侧设备 / SLM<br/>(本地推理 / 离线运行)"] B["企业敏感数据 / 专有资产"] end subgraph PipelineLayer["混合模型处理流水线"] C["轻量嵌入与编码<br/>(Embedding / 稀疏编码)"] D["核心大语言模型<br/>(LLM / Agent 推理)"] end subgraph CacheSecurity["多租户算力平台安全边界"] E["租户 A 专属 KV Cache"] F["租户 B 专属 KV Cache"] G{"严格隔离层<br/>(防跨企业数据渗透)"} end A --> C B --> C C --> D D --> G G --> E G --> F ``` 在大规模推理服务中，KV Cache 能显著提升响应速度，但基础设施平台必须从底层确保不同租户之间的缓存严格物理或逻辑隔离，绝不能出现跨公司混用或信息残留交叉的情况。 - **监管核心诉求**：金融等垂直行业对私有数据归属权、合规性及“是否会被大模型隐式训练”极为敏感。 - **多模型流水线搭配**：利用低成本硬件运行向量嵌入（Embedding）与稀疏编码，避免全链路占用昂贵显卡。 - **KV Cache 安全风险**：多租户共享底层 GPU 实例时，加速缓存中的上下文数据隔离成为云平台的硬性安全指标。 --- ### [20:05](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1205s) 混合部署决策矩阵：云端、本地与边缘的选型最佳实践 企业在决定工作负载应该放在公有云、本地机房（On-Prem）还是边缘端（Edge）时，需要综合衡量敏捷度、合规性与数据传输成本。Frank Kartmann 从德意志银行等金融机构的实战经验出发，梳理了明确的部署原则： | 部署环境 | 核心适用场景 | 关键驱动因素与技术考量 | |:--- |:--- |:--- | | **公有云 (Cloud)** | 托管式 AI 服务、弹性伸缩任务、突发算力需求 | 摆脱本地硬件采购周期；应对高度弹性、不可预测的算力峰值。 | | **本地机房 (On-Prem)** | 高敏感合规数据、可预测的高并发负载 | 严格遵守金融数据主权（如 GDPR、区域法规）；长期高吞吐负载下自建更具单位成本优势。 | | **边缘/端侧 (Edge)** | 用户个性化体验、生物识别（Face ID / Touch ID）、离线运行 | 极低时延；无网络环境下的离线推理；生物隐私数据不出设备。 | Pratik Jain 进一步强调了“操作型数据（Operational Data）”爆炸对架构决策的影响。过去企业管理的数据量如果以 10 倍速增长，那么在 AI Agent 时代，由于智能体频繁调用模型上下文协议（MCP）、外部工具和 API，其操作性流量将激增 100 倍甚至更多。在公有云环境下，每一次网络调用和跨区数据传输（Egress Traffic）都会产生费用，这种调用量的指数级膨胀将迫使企业重新评估数据传输账单，甚至推动 SaaS 软件与云厂商重新设计其商业授权与定价体系。 --- ### [24:27](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1467s) 算力硬件的真实生命周期与“GPU 淘汰论”的破灭 针对半导体行业推陈出新速度加快、老旧芯片是否迅速过时的 CapEx（资本支出）疑问，Greg Wester 直言“AI 崩溃论者/末日论者（AI Doomers）”存在两大认知误区。其一是关于 GPU 故障率的夸大：GPU 的故障几乎都集中在最初高负荷运行的两周“老化筛选期”（Burn-in），只要熬过这两周并及时对不良品执行 RMA（退换货），其后续的故障率会呈现数个数量级的断崖式下降。 其二是关于“GPU 在两三年内就会被彻底淘汰”的说法，Greg 认为这与实际业务严重脱节。他戏称 RunPod 所支持的长尾芯片列表就像一个“计算机历史博物馆”——平台上甚至仍有 9 年前发布的英伟达老卡在运转，学生和个人开发者为了完成大学课程或低成本试验机器学习，对这些旧卡依然有着旺盛的租赁需求。在企业规模化推理与微调中，NVIDIA L4 和 L40S 仍是绝对的性价比主力。 谈到市场对先进制程的追逐时，现场爆发出笑声：Greg 调侃道，英伟达 CEO 黄仁勋此前频繁开会推销最新的 Blackwell 芯片，但广大客户实际上“只想继续加订 H100”，直到英伟达停止生产 H100 并要求客户至少订购 H200。很多企业已经在特定硬件架构和模型上跑通了业务逻辑，并不愿意冒风险频繁迁移底层栈。Greg 大胆预测，即便再过 8 年，业内依然能看到 H100 处于生产运行状态。即便企业在财务账面上以 3 年或 5 年加速折旧，折旧年限也丝毫不影响这些算力硬件在长周期内持续创造实际价值。 - **老化测试定律**：GPU 硬件风险主要集中在上架前两周的高温 Burn-in 阶段，度过初期后硬件失效率将大幅降低几个数量级。 - **长尾算力生态**：算力平台支持如 9 年车龄般的旧款 GPU，精准匹配低预算实验、学术教学等长尾需求。 - **实用主义主流**：NVIDIA L4/L40S 依然是扩展推理业务的中流砥柱；客户更青睐成熟稳定的 H100 系列而非盲目追逐 Blackwell。 - **超长生命周期**：预测 H100 在 8 年后仍将保有实际生产价值，财务层面的加速折旧与硬件的物理可用性并不挂钩。 ### [26:56](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1616s) 硬件策略：老旧硬件复用与按工作负载划分的异构算力 讨论延续了前文关于算力性价比的权衡。嘉宾用生活中的场景打了个生动的比方：如果买车的目的仅仅是每周两次接送孩子上下学以及去超市购物，根本没必要买昂贵的新车，一辆开了六年的丰田（Toyota）老车完全能够胜任。对于大多数企业而言，采购上一代甚至落后两代的硬件反而能获得更高的性价比（"bang for the buck"）。Pratik Jain 对此表示完全赞同，并指出核心在于如何最大化利用现有硬件资产，并在此基础上进行构建。随着行业技术的快速更迭，企业在评估具体业务场景时，应当按照工作负载类型、环境阶段（开发环境 vs 测试环境 vs 生产环境）以及实际的响应时间（延时）和运维需求来精细化拆分硬件决策。 面对市场上硬件价格飙升的剧烈波动，嘉宾补充强调了硬件复用策略的重要性。企业绝不能轻易报废或丢弃旧硬件，而应构建异构加速计算平台（Heterogeneous Accelerator Platforms）。当某些硬件从关键生产线退役（Decommission）时，完全可以将其降级重用于测试、开发或低优先级负载中，或者利用像 Runpod 正在构建的编排层，将分散的旧硬件聚合成统一的计算 Pod 单元，从而有效平抑供应链暴涨带来的成本风险。 - **丰田汽车隐喻**：日常轻量级任务无需顶配算力，上一代或两代前的 GPU 往往具有更高的综合投资回报率。 - **工作负载精细化切分**：根据 Dev（开发）、Test（测试）、Prod（生产）对延时与 SLA 的差异化要求配置异构算力。 - **资产寿命延长与异构纳管**：建立退役硬件二次利用机制，结合聚合管理层形成统一的硬件资源池。 --- ### [29:42](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1782s) 边缘推理的兴起与本地上下文管理（Context Management） 主持人 Ryan Wang 观察到，在过去一年多时间里，行业计算重心明显正在从“模型训练”向“模型推理”倾斜，并且边缘推理（Edge Inference）在实时、低延迟敏感型场景中展现出颠覆性的影响力。Pratik Jain 结合自身产品开发实践分析道，以广泛使用的编程辅助工具（如 Claude Code）为例，大量运算正在直接发生在开发者本地终端。开发者在边缘运行代码，仅将模型作为推理引擎，并通过精细的上下文管理机制与远程大模型交互。 Pratik Jain 深入剖析了边缘推理中的核心痛点——上下文管理（Context Management）。尽管像 Anthropic 等头部厂商不断宣传 100 万 token 的超长上下文窗口（1 million context），但在实际连续对话中，往往聊到第 15 条消息之后模型就开始丢失上下文，不得不重新读取原始数据。而企业与开发者的海量数据实际上就沉淀在本地的 CPU、GPU、笔记本或私有数据库中。为了高效召回并注入上下文，系统需要使用特定的模型解析向量数据（Vector Data），构建知识图谱（Knowledge Graphs）或本体库（Ontologies）。在整个端到端系统中，小语言模型（SLMs）、大语言模型（LLMs）与嵌入模型（Embedding Models）各自承担不同职责，大量计算将直接在边缘端完成，而无需将所有请求都扔给极其耗费昂贵 GPU 资源的前沿基础大模型（Foundation Models）。 ```mermaid flowchart TD accTitle: 端侧智能与上下文管理架构 accDesc: 展示从本地数据到端侧小模型处理再到云端大模型调用的上下文管理链路 subgraph EdgeLocal["边缘终端与本地环境 (Edge & Local)"] RawData["本地原始数据 / 数据库 / 代码"] EmbModel["嵌入模型 (Embedding Models)"] KG["向量数据解析 & 知识图谱 (Knowledge Graphs)"] SLM["端侧小模型 (SLMs) 快速处理与过滤"] end subgraph CloudInfra["云端中心算力 (Cloud)"] LLM["前沿大基础模型 (Frontier LLMs)"] end RawData --> EmbModel EmbModel --> KG KG --> SLM SLM -- "精准上下文摘要与低频复杂任务" --> LLM ``` - **上下文衰减痛点**：即便大模型宣称支持百万级上下文，长对话中依然会出现信息丢失或反复拉取数据的问题。 - **混合模型分工**：利用本地的小模型与嵌入模型完成数据过滤、本体构建与检索，基础大模型仅用于最终高阶推理。 - **数据隐私与本地驻留**：企业核心业务数据保存在本地数据库，通过边缘计算实现就地处理。 --- ### [33:13](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1993s) 2027年预测与行业愿景：从算法民主化到算力周期与商业闭环 在圆桌会议尾声，主持人 Ryan Wang 向四位嘉宾提出了最后一个问题：“如果一年后我们坐在 Ai4 2027 的现场，AI 基础设施与部署格局最大的变化会是什么？同时，如果给你一根魔法棒，你对这个行业最大的期许是什么？” Greg Wester（Runpod）率先给出预测与期许： - **预测**：通过工具链创新，将模型微调（Fine-tuning）和训练的能力从极少数拥有数学和统计学博士学位（PhDs）的专家群体，彻底普及给“每一位普通软件开发者”。构建一个自动化闭环：从持续运行推理、分层采样数据（Stratified Sampling）、自动标注、推入训练流水线，到重新将新模型部署发布回共享存储中。当前整个流程包含约 93 个步骤，即便硅谷最顶尖的 AI 公司也仅自动化了 40 到 50 步。一旦实现全自动化闭环，企业将大幅降低对前沿超大模型的依赖，转而使用在特定垂直任务上精确率和召回率（Precision and Recall）极高的小模型。 David Wacker（KONST）指出： - **预测**：整个现代 AI 浪潮实际上才起步几年时间，未来一年行业将迎来更深层次的“专业化与工业化”，迭代速度与基础设施建设将进一步提速，当前依然处于扩张的早期阶段。 - **愿景**：“希望所有人继续利用 AI 让世界变得更美好，而不是仅仅用来制造猫咪表情包（not just to produce cat memes）。” Pratik Jain（Auvern.ai）从技术周期的角度进行了解读： - **预测**：技术演进呈周期性轮回。历史从早期的大型机（Mainframe），走向客户端-服务器架构（Client-Server），再走向云计算（Cloud）；如今算力周期正在重演——配备 H100 的庞大计算集群就像当年的大型机，而随着边缘算力的兴起，架构再次向客户端/服务器混合模式靠拢，未来随着芯片微缩或量子技术突破，计算将再次全面回归个人笔记本端。 - **愿景**：希望人们将 AI 视为一种强大的工具去主动学习和利用，而不是对技术变革产生恐惧，人类本身就是高度进化的物种。 Frank Kartmann（德意志银行）总结了金融机构视角的观察： - **预测**：未来将出现更加灵活、动态的混合云架构（公有云与私有云无缝漂移），随着端侧 AI 处理器能效的大幅提升，更多工作负载将转移至边缘。银行业在移动端 App 中已经集成了几乎全谱系的金融服务，未来端侧智能化将进一步深化。 - **愿景**：全行业与市场正在狂热拥抱 AI，但目前尚未真正达到财务上的“盈亏平衡点（Break-even）”。他由衷希望行业能尽快看清商业化与盈利闭环的路径，获得更多的确定性（Clarity），验证 AI 究竟能否以可持续盈利的方式长久立足。 会议在全场观众对嘉宾的掌声中圆满结束。 ## Notable quotes > [06:29](https://www.youtube.com/watch?v=a81Xc1zACB0&t=389s) 你没必要为了解决所有的业务场景都去开一辆法拉利（You don't need a Ferrari to drive and solve for every business case）。 — Pratik Jain > [10:08](https://www.youtube.com/watch?v=a81Xc1zACB0&t=608s) 模型能力确实惊艳，但他们无法承受每百万 Token 支付 10 美元的高昂代价，客户急迫地探寻如何将成本打压至 2 美元乃至更低。 — Greg Wester > [13:42](https://www.youtube.com/watch?v=a81Xc1zACB0&t=822s) 看起来就像个水上乐园 — Greg Wester > [24:27](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1467s) 平台上甚至仍有 9 年前发布的英伟达老卡在运转，学生和个人开发者为了完成大学课程或低成本试验机器学习，对这些旧卡依然有着旺盛的租赁需求。 — Greg Wester > [24:27](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1467s) 广大客户实际上‘只想继续加订 H100’，直到英伟达停止生产 H100 并要求客户至少订购 H200。 — Greg Wester > [24:27](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1467s) 即便再过 8 年，业内依然能看到 H100 处于生产运行状态。即便企业在财务账面上以 3 年或 5 年加速折旧，折旧年限也丝毫不影响这些算力硬件在长周期内持续创造实际价值。 — Greg Wester > [26:56](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1616s) 如果买车的目的仅仅是每周两次接送孩子上下学以及去超市购物，根本没必要买昂贵的新车，一辆开了六年的丰田（Toyota）老车完全能够胜任。 — Greg Wester > [29:42](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1782s) 在实际连续对话中，往往聊到第 15 条消息之后模型就开始丢失上下文，不得不重新读取原始数据。 — Pratik Jain > [33:13](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1993s) 希望所有人继续利用 AI 让世界变得更美好，而不是仅仅用来制造猫咪表情包（not just to produce cat memes）。 — David Wacker > [33:13](https://www.youtube.com/watch?v=a81Xc1zACB0&t=1993s) 全行业与市场正在狂热拥抱 AI，但目前尚未真正达到财务上的‘盈亏平衡点’。由衷希望行业能尽快看清商业化与盈利闭环的路径，获得更多的确定性。 — Frank Kartmann ## Mentioned Ryan Wang, Frank Kartmann, David Wacker, Pratik Jain, Greg Wester, Solaris Venture Partners, Outpost Capital, RunPod, KONST, Auvern.ai, Deutsche Bank (德意志银行), JPMorgan Chase (摩根大通), Wells Fargo (富国银行), NVIDIA (英伟达), Jensen Huang (黄仁勋), OpenAI, Anthropic, Microsoft (微软), Blackwell, NVIDIA B300, NVIDIA H100, NVIDIA H200, NVIDIA L4, NVIDIA L40S, NVIDIA RTX 6000 Pro, Claude Code, MCP (Model Context Protocol), DORA (数字业务弹性法案), Ai4 ## Video description Ryan Wang — GP, Solaris Venture Partners & Outpost Capital Frank Kartmann — Vice President of Big Data & Cloud Architecture, Deutsche Bank David Wacker — Systems Architect, KONST Pratik Jain — Founder/CEO, Auvern.ai Greg Wester — Product Manager, Runpod As AI workloads grow more complex and compute-hungry, infrastructure leaders must rethink how they scale resources across cloud, on-prem, and edge environments. This panel brings together executives and infrastructure leaders to discuss how they’re building flexible, future-ready compute foundations for enterprise success. Topics will include selecting next-gen chips, planning for distributed training at scale, optimizing for energy and cost efficiency, and aligning infrastructure decisions with evolving business needs. Whether you’re running large language models or deploying edge inference at scale, this session will offer actionable insights into building the compute core of your AI strategy. [Source](https://www.youtube.com/watch?v=a81Xc1zACB0)

[6:29] Pratik Jain: 你没必要为了解决所有的业务场景都去开一辆法拉利（You don't need a Ferrari to drive and solve for every business case）。 [10:08] Greg Wester: 模型能力确实惊艳，但他们无法承受每百万 Token 支付 10 美元的高昂代价，客户急迫地探寻如何将成本打压至 2 美元乃至更低。 [13:42] Greg Wester: 看起来就像个水上乐园 [24:27] Greg Wester: 平台上甚至仍有 9 年前发布的英伟达老卡在运转，学生和个人开发者为了完成大学课程或低成本试验机器学习，对这些旧卡依然有着旺盛的租赁需求。 [24:27] Greg Wester: 广大客户实际上‘只想继续加订 H100’，直到英伟达停止生产 H100 并要求客户至少订购 H200。 [24:27] Greg Wester: 即便再过 8 年，业内依然能看到 H100 处于生产运行状态。即便企业在财务账面上以 3 年或 5 年加速折旧，折旧年限也丝毫不影响这些算力硬件在长周期内持续创造实际价值。 [26:56] Greg Wester: 如果买车的目的仅仅是每周两次接送孩子上下学以及去超市购物，根本没必要买昂贵的新车，一辆开了六年的丰田（Toyota）老车完全能够胜任。 [29:42] Pratik Jain: 在实际连续对话中，往往聊到第 15 条消息之后模型就开始丢失上下文，不得不重新读取原始数据。 [33:13] David Wacker: 希望所有人继续利用 AI 让世界变得更美好，而不是仅仅用来制造猫咪表情包（not just to produce cat memes）。 [33:13] Frank Kartmann: 全行业与市场正在狂热拥抱 AI，但目前尚未真正达到财务上的‘盈亏平衡点’。由衷希望行业能尽快看清商业化与盈利闭环的路径，获得更多的确定性。

[6:29] Pratik Jain: You don’t need a Ferrari to drive and solve for every business case. [10:08] Greg Wester: I love what it can do, but not for $10 per million tokens. How can I do that for $2 or 20 cents? [10:08] Greg Wester: it's not a technical problem as much—we know how to solve computers [13:42] looks like a water park [15:39] free to use [24:27] AI doomer [24:27] computer history museum [33:13] David Wacker: make the world better, not just to produce cat memes.

[0:04] This morning we are going to talk about scaling  AI infrastructure. And here with me I have four [0:11] amazing panelists um Frank Practic, David and  Greg. My name is Ryan Wong. I have been working [0:20] as a um Silicon Valley venture capitalist for  the past 11 years. Currently I manage uh two [0:27] fund. One is a VC fund called Outpost Capital  investing early stage frontier tech and two [0:33] years ago I also co-ounded a private equity firm  where we invest in AI infra and uh data centers. [0:40] So this topic today is very much um um uh very  tied to you know everything that's uh happening [0:49] in my world. So um with that I think uh you know  we only have 35 minutes. Um without further ado, [0:57] let's give uh each of the panelists opportunity to  introduce themselves. Maybe start with you, Greg. [1:03] Hi, I'm Greg Wester. I'm a director of product  management at RunPod, which is a NeoCloud that [1:09] offers serverless inference and training products.  David, hey. Um I'm David. I'm a system architect [1:16] at Constant. We build uh AI data centers all over  the world basically. Hey everybody. Uh I'm Pratik [1:24] Jen. I'm the CEO and founder of a.ai where we're  building the enterprise control plane to run, [1:30] manage and uh govern agents at scale. Um, one  of the key things today in the industry is [1:38] how do you really leverage the compute uh and  bring the business value out from there. Um, [1:45] and that's where we try to help. Uh, yeah. Hello  everyone. Uh, thanks for coming. My name is Frank [1:53] Kman. Um I work for Deutsche Bank. That's a um  European bank headquartered in Germany. Also [2:01] with facilities here in the US, uh New York, um  North Carolina, Florida and there I am a platform [2:10] architect and yeah we run big data platform and  enabling um AI and yeah data analysis. Awesome. [2:21] Frank maybe we can uh start from you right so  since you work in a financial service industry [2:26] it's a very special very regulated industry  and scaling AI infrastructure can mean very [2:33] different things in this particular industry right  so what do you think are some of the most critical [2:38] compute or hardware priority you have in your  work and also um maybe you can give us some of [2:44] the examples on some of the you know case studies  that you've recently did you feel like um um you [2:51] have some kind of challenges and you overcome  some kind of challenges through uh this kind of [2:55] implementation. Uh sure. Yeah. In the financial  industry uh for banks and insurance companies [3:04] um right now um the yeah biggest challenge is  platform I would say platform modernization and AI [3:13] utilization. Um data volumes keep growing. uh we  collect always more more data and want to process [3:23] it and for that we need uh even more storage  capacity and compute capacity at the same time [3:32] uh a bank uh needs a stable platform. So uh we  have to to build and maintain a secure uh reliance [3:44] and also economically viable um data platform and  we have to put al all this yeah elements together. [3:56] That means um adding uh storage and compute  capacity to the platform that was there and at [4:06] the same time keeping an eye on uh profitability  on reliance and uh stability and and security [4:16] uh for a especially in Europe and in the financial  industry because it's highly regulated reliance is [4:23] a a big issue In Europe we have the Dora digital  operational reliance act and um it enforce um yeah [4:34] ISD risk management and reliance management um  for challenges go we'll go very quickly um yeah [4:44] our major projects uh what are we doing all banks  are just now uh building cloud native applications [4:53] um that are flexible scalable All um can be moved  among uh clouds and um um are building also hybrid [5:06] um and multi- um multi cloud architectures and  uh and then moving workloads um depending on [5:16] um data data sensitivity. Sometimes you have um  regulatory requirements or or costs that have to [5:26] be yeah you have to to to keep an eye on yeah you  have to consider and uh I've finalized now um when [5:35] leveraging AI on on prem then we have to we have  to provision um data center capacities you need [5:46] uh building storage sorry uh building uh cooling  uh power all the things and at the same time you [5:53] need an efficient data data platform and you need  um you need um expensive low latency storage NVME [6:02] uh for instance you need um data storage you need  um um networking very low latency uh you need u [6:12] um yeah compute GPUs and all the things and the  biggest challenge uh right now is the that market [6:20] um is very yeah is not working and we have like um  broken supply chain. I can tell more later. Yeah, [6:29] definitely. Practic you spent decades in the  financial service industry. though I'm sure you [6:36] can resonate to a lot of things that Frank  mentioned and what's more interesting most [6:41] recently you left banking and then started  your own company right so tell us a little [6:47] bit about that journey what incentivize you to  um step on this new journey and what are some [6:54] of the challenges and the case studies that you  are implementing in scaling AI infrastructure [7:00] especially in the financial service industry since  Frank bring up the topic Um so as um as mentioned [7:09] right I've worked at JP Morgan WS Fargo some  of the top banks and I clearly what uh he said [7:15] here is clearly resonates with me um some of the  biggest challenges sometimes when we think about [7:22] um from an implementation perspective is really  about governance. You might have infrastructure [7:31] actually built out sitting there but without  the right compliance and governance uh most of [7:38] that infrastructure is just sitting idle and the  key thing there is how can you build the right [7:44] stack the right control layer uh on top of that to  enable your teams to move faster uh what prompted [7:52] me uh was exactly this uh I've been trying to  solve this problem Now this is the third time [7:59] we if we think back starting with the internet it  was a big shift everybody had to rethink how the [8:06] data was stored how it's managed um the second  big bigger shift was cloud um and now the third [8:14] is Agentic um and with Agentic how you split up  your workloads is even more critical uh where now [8:22] it is no longer just shifting the workloads to  let's say a cloud to one set of server ers now [8:28] to break up your workloads let us say running at  the edge versus maybe running on your onrem and [8:34] then third now you have your foundational models  that could be hosted somewhere else so that split [8:39] setup along with that now you have multiple  use cases like for using uh model training [8:46] you require different type of infrastructure CPU  versus for inference you need a different um so [8:53] this is where it becomes very interesting um and  how do you really leverage the different models [8:59] again that are hosted on those infrastructures for  your agent use cases to deliver business value. [9:05] Um as probably you might have heard you don't  need a Ferrari to drive uh and solve for every [9:12] business case right uh and that's where you have  to optimize infrastructure the model all of the [9:18] layers on top along with the governance built in  into it to be able to solve it and that's what I'm [9:23] trying to address here uh with what I'm building  um some of the biggest challenges I would say has [9:30] been really thinking or rethinking actually where  or how um you're allocating or structuring the [9:41] information together. I mean if you look at the  big blocks there's observability which generates [9:45] traces but then you have LLMs there you have  MCP calls there's operational data that's being [9:50] captured on the other end you have whole context  management now that's building a whole semantic [9:55] layer and how do you bring this all together  for especially an agent tech use case [10:01] um are some of the big challenges that I see.  Awesome. Um, thank you practic. And now if we [10:08] can switch from the financial service industry to  the traditional NeoCloud site and I think we have [10:16] uh two great representatives here. Um, uh, Greg is  um, managing Rampot which is a NeoCloud based in [10:24] Silicon Valley and David is managing const which  is a new cloud based in Taiwan. Right. So um from [10:33] maybe David let's start from you. So from your  perspective, what's some of the most critical [10:39] thing um the industry need to solve in scaling  the AI infrastructure side and also you know um [10:47] especially since you are from Taiwan and you have  a global understanding about different regional [10:54] differences maybe you can also talk a a little bit  about how you know const is differentiating from [11:00] all the other um neo cloud and uh what's special  in the Asia new cloud uh ecosystem as well. Okay, [11:07] thank you so much. I will uh pick up the  last question first. I think uh what is [11:11] really interesting for a company based in Taiwan  is uh we're basically so close to all the OEMs, [11:16] right? All the hardware is manufactured there.  So we are really tied into the ecosystem and [11:23] um are kind of close to to the whole engineering  development in that part and then in that sense [11:30] are also always looking forward then what is  the next step and that also kind of leads to [11:35] uh your uh question on what we need to solve right  now the iterations um of AI hardware the GPUs is [11:44] so fast so we're currently deploying uh B300's  the current generation and they already talking [11:49] about the next generation. So you kind of have to  look on um how can you build a data center and a [11:57] setup that is still usable in in three to four  or five years and you don't have to start from [12:02] uh scratch again. Yeah, makes sense. Greg, um now  let's go back to the the heart of Silicon Valley. [12:09] What's uh what's happening in NeoCloud? What's  trendy? What's a critical things we need to solve? [12:16] Um, I mean the the the demand is off the charts  and I kind of like pay respect to anthropic and [12:22] open AI for showing consumers and businesses  what, you know, machine learning and AI models [12:27] can do for you. Um, and that's driving a lot of,  you know, interest from customers about, you know, [12:33] I love what it can do, but not for $10 million or  for $10 per million tokens. How can I do that for [12:38] $2 or 20 or something? And so um when we think  about how are we going to grow 200% we have to [12:44] bring [snorts] on like a megawatt of compute a  week. We have to bring on 50 pabytes of storage [12:48] a week. And that would have been like maybe you  know financially challenging before but now it's [12:53] also logistically challenging. Um what if I bring  on you know a thousand or 10,000 H100s all at once [13:01] and my utilization drops you know 15% because it  was dumping a lot of supply on. So you have to [13:06] kind of start to match enterprise customer  demand with that. So it's more operational [13:12] than I thought it would be. It's not a technical  problem as much. We know how to solve computers, [13:16] right? But solving for the challenges of growing  200% a year is super hard. Yeah. Yeah. I think uh [13:21] you know yesterday we also talked about the  challenges around existing infrastructure, [13:27] right? The network and everything. Do you think uh  do you think currently the existing infrastructure [13:32] is also another bottleneck? Um, for sure when  when you look at a global footprint of where [13:38] you're going to put all this compute, uh, you  can go to places that have a lot of power, uh, [13:42] but often they don't have a lot of internet and  network bandwidth. Um, and so you really have to [13:47] to manage that if you build a really great site  and your customer has trouble getting their data [13:52] and information to it or getting it out of it. Um,  so having kind of a multi-reion, multi-data center [13:58] footprint and then offering some kind of quality  of service across that where it shouldn't matter [14:03] where my workload lands. I shouldn't notice the  difference but you know maybe if you're really [14:08] measuring you can notice right right and one of  the thing you just mentioned is really around [14:12] the unique economics of uh you know this uh this  business right um the other aspect is really the [14:18] um the the energy efficiency in running um AI  infrastructures so in your business how do you [14:26] typically balance what's happening you know on  the energy consumption side versus you know the [14:32] unit unit economics takes and also the quality of  the service to the customer. Um yeah, interesting [14:41] question. Um maybe the biggest limitation in  existing data center footprints is you can [14:45] only fit maybe, you know, one or two of these GPU  machines per rack. It can actually fit seven or [14:51] something, but uh you can't get enough power into  that rack, right, in a in an existing site. Now, [14:56] if you go to the NVIDIA conference, you'll see  them show, you know, new generation racks that can [15:00] do 200 kilowatts and they have beautiful plumbing.  It looks like a water park. Um, so that all looks [15:04] like a great future, but that's not necessarily  the present. Um, yeah, that's most and then I [15:10] think about how can I actually standardize some of  the business on a few SKs that customer wants. So [15:15] like in a proumer community developer type thing,  we might do like variants of the RTX 6000 Pro and [15:22] split them into different um like logical GPU GPU  memory sizes. And on the enterprise side, I might [15:28] try to, you know, center around like a B300 and  R300 type strategy. Mhm. um and sort of push that [15:35] as the workhorse so that we can have predictable  supply, predictable demand. Makes sense. Makes [15:39] sense. Anything to add? Yeah, sure. So um I think  another driver that will be there is currently [15:46] AI is basically like free to use, right? So um  users they themselves have no motivation on on on [15:53] uh considering that parts because there's no cost  for them of using it for for a large part of the [15:58] users. But once um I say the customer acquisition  phase is over and uh companies actually start to [16:06] make profit right you will see these numbers  and they will think more proactively on on how [16:11] can they reduce their cost and uh reduce power  consumption and these things and there's a lot [16:17] of potential on that part not just software wise  right you can you can have more efficient models [16:22] or whatever but um also infrastructure wise so the  the POE share so how much of the power you consume [16:29] actually goes to your GPU load and how much is a  supporting infrastructure that can change and then [16:36] uh I think what we briefly talked about before  is currently there's a huge and I think almost [16:42] no one is aware of that just converting from  like transmission lines uh the high voltage [16:47] down to DC ACDC again you lose like 30% of the  power just from uh transforming and uh there [16:55] will be definitely technology coming up that uh  basically makes it not necessary anymore to to do [17:01] these transformations and uh you will have greatly  um more efficient setups just by these changes [17:09] without any uh offset so to say. Yeah, it's one of  the things that it's crazy to think about the 13% [17:17] loss just because of the AC/DC transformation, but  then that's just the state of the industry right [17:23] now, which supposed to be easily um um changed in  the next few years. Pretik. Yeah. Um I think one [17:32] of the big shifts that's happening is going to be  AI at the edge. um that will that will or should [17:39] help with some of these things and also with the  trend towards SLMs. Um coming from a financial [17:45] background and having uh a highly regulated  industry, one of the biggest challenges is where [17:52] does your data sit from a customer perspective?  Who owns it and then how is it being used? [17:58] And these are some of the biggest questions that  uh financials regulated or even regular industries [18:06] also are wondering and is my data going to be get  trained um or used by some of these models right [18:14] because that's proprietary data and that is where  as we think through the architectures of building [18:22] um new agentic applications rethinking the  business processes Um I feel there will be [18:29] a big shift especially towards edge AI. I think  you can see Microsoft and lot of others building [18:35] their own inference chips and starting to put  them right there into your um desk laptops or [18:45] uh possibly servers. And then um another big thing  that we also I know we always think about LLM but [18:53] then when you think about the entire process of  converting data into u machine learning pipelines [19:02] and stuff there's different types of models again  there's embedding models there is uh split there [19:08] is sparse encoding all these different variations  of things that you need to bring that together [19:14] into uh and not all of those need to run again  on super uh expensive hardware. Um the other [19:22] big thing also sorry one quick thing is about  this data that sec how and where it is stored uh [19:29] especially as I was just talking to Greg about is  around KV cache and how does your data get shared [19:36] across multiple different um tenants and that's  something that uh I'm sure I think these guys [19:43] are looking into and building as part of their  platform but highly critical that the cash that's [19:50] for the LLMs does not get shared across different  companies and you don't get a mix of that. That's [19:57] a great point. Many many different type of models,  many different type of use cases. So, what's some [20:05] of the best practices in deciding what's best  to deploy on the cloud versus, you know, on the [20:14] on prim or uh on the edge? Any thoughts? Yeah, I  could take it. Of course, um there are like some [20:24] um uh good practices uh for deciding uh what  to move. Uh normally uh and I I know best the [20:34] financial industry, but probably are all of them  uh similar. uh you could uh put workloads on cloud [20:43] uh when you um when you want uh manage AI services  or um or you need highly scalable workloads or you [20:58] just don't have the compute capacity on prem just  to run them then you would go to public cloud. [21:07] uh what you would have on on prem uh having  um or keeping workloads on Prem is sometimes [21:15] a matter of data sensitivity because um special  especially in the financial industry we have [21:23] uh regulatory constraint for some data and has  to be there. Um other points here could be like [21:31] um predictable um high volume workloads that could  be much expensive on cloud or um you just have [21:40] some yeah whatever reasons um for keeping your  workloads on yeah other reasons on your private [21:48] cloud and on edge. Um at the moment um even if on  device AI uh processors are developing very fast [22:05] uh at the end you could have only um uh user  uh experience. You have um uh personalization [22:16] on edge. you have um biometric um identity um  face ID, touch ID and the things. So you mean [22:27] on [laughter] device local inference you have um  can be implemented on on edge. Um you could have [22:36] um offline operation is also an advantage of  prem and um um yeah of line inference those are [22:50] yeah the biggest ones I would say. Um I I think  there's two or three things that we should keep [22:57] in mind. One again is data um location where  what is critical like um Frank said and where [23:06] do you want to um locate it? There's different  GDPR versus Asia versus different regulations [23:13] that come into from a data governance perspective  that drive your decision making. The second big [23:18] thing which is going to be the major decision  factor is going to be your operational data. [23:24] um till now let's say you were you had like 10x  of data that you were managing and controlling [23:32] with AI and the speed of with which agents  LLMs are making MCP calls making different [23:39] um tool calls uh invoking certain things your data  operational data is going to grow 100 possibly [23:48] 100x or even more and that is something that is  going to be highly critical well on how you manage [23:54] and place your workloads um in the cloud where  every network um call or every data that's passing [24:04] through that gets charged and some of this it I  think maybe the pricing will evolve or how things [24:12] uh especially SAS are also rethinking their  whole licensing model will some will play into [24:18] uh over the next few years. So that will be  interesting to watch. Makes sense. Um so let's [24:27] talk about the capex question here. Um we know all  the in the semiconductor industry right now there [24:34] are a lot of new new progress new development  on the hardware side new compute new chips right [24:40] going to the market. What's happening to the  old hardware? Are they becoming obsolete and [24:46] um um and um that you know very soon the the  new hardware going to come to the market and [24:53] uh make the make the existing hardware um  uh no longer usable or do you think there's [25:00] um long-term value here as well? I think there's  two things like the AI AI doomers get wrong. Um, [25:08] one is that like GPUs have a high failure rate  and after you get through your initial burn-in, [25:13] the two weeks where you run it really hot. Um,  you can RMA anything that that that failed and [25:18] you know after that the the failure rate drops off  considerably few orders of magnitude. Um, and the [25:24] other thing that they'll say is like you know um  a GPU is obsolete in two or three years and that [25:30] like could not be further than the truth. I joke  that like the long tale of cards that we support [25:34] on RumPod, it's like a bit of a computer history  museum, meaning there could be like a 9-year-old [25:39] Nvidia card on there, but someone finds value  out of that. A student wants to rent that and [25:44] use that to experiment with machine learning for  their coursework beyond what they can get at the [25:48] university. Um, and frankly today, a lot of the  world runs their scale out workloads on the NVIDIA [25:54] L4 and L40S if you need a little more memory.  Um, you know, and for a long time, Jensen kept [26:01] having a conference every month because he was  trying to get people to buy, you know, Blackwell [26:05] chips and everyone just wanted to keep reordering  H100s. Yes. They stopped making them and said, [26:09] you have to buy an H200 at least. [laughter] But  people sometimes, you know, have success on a [26:15] certain architecture, have success on a certain  model, and they want to go with that for a long [26:18] time. Yeah. But I I predict we'll still be running  H100s in eight years. Okay. So, our hardware are [26:23] safe. it's gonna they're not going to go obsolete.  Even if you depreciate it in five years or three [26:28] years on accelerated basis, the depreciation has  nothing to do with uh you know yield yield ability [26:36] to use it. Yes. Um thank you. That's a very  optimistic view. Everybody agrees. [laughter] [26:45] Yeah, I totally agree. I I I really like the an  analogy to the car market. Uh it is a way Yeah. [26:51] Of course, the salesperson, he will try to sell  you the newest, fanciest car where he has the [26:56] most profit. But, but that's not what most people  need. So, same thing if you only drive your kids [27:00] to school and go to the supermarket twice a week,  you're not going to buy buy a new expensive car. [27:05] The I don't know 6 years old Toyota just does  the job fine. So, that's totally my standpoint, [27:12] too. you you probably uh for most people actually  um they are better off with buying the uh last or [27:21] or even two uh generations older hardware and um  probably have a better bang for the buck. Yeah, [27:30] I totally agree there and it comes back again to  the use case uh where how do you best leverage [27:36] what you have already and how do you build on  on it and as as we all know this industry is [27:44] shaping and reshaping itself very very quickly um  with new and friendships new ways of rethinking [27:53] uh but then when you look at your business  cases and you're trying to approach it. Um, [28:01] split it by maybe type of workflow, split it  looking it by dev environment versus test versus [28:06] production and what is the response times, what's  the operational needs uh to be able to solve that [28:15] business use case. Um, should actually drive some  of those um decisions around hardware cuz there's [28:22] a lot of money has already gone in. There's a lot  of hardware actually sitting out there and again [28:26] as some of these is getting op operationalized  I'm sure there will be layers coming that can [28:31] help group together some of these into um pods  like uh what Greg building right and leverage [28:41] it more as one consistent um hardware unit may I  add something okay okay [laughter] I think [gasps] [28:51] that has become increasingly important in the  current uh market uh situation because uh hardware [28:59] prices are rising very very fast and then you need  like some strategies to protect yourself a little [29:05] bit against u this um wild yeah market development  and one of these um elements of this strategy [29:14] could be not to throw away everything uh your  old hardware just but uh creating a hogenous like [29:22] accelerator platforms and then if you decommission  something then there will be for sure a good [29:28] uh um yeah purpose um for this decommission it  so um you can reuse it for instance yeah right [29:42] let's talk about edge inference right for the  past a year or so we see the computation shift [29:48] from the training side to the inference side  and Now with the edge being another option, [29:56] we see edge inference is making a lot of impact  to a lot of you know um especially um real time [30:04] um and and latency sensitive type of applications  right so can you share some real world example um [30:14] you know pushing the compute to edge inferences is  fundamentally changing how the business operates. [30:25] Yeah, I I see a big shift there for sure. Um  I can speak to from my own product development [30:31] perspective and how we're rethinking um each of  these. Let's take cloud code as example. One of [30:37] the biggest uh use cases which everybody's  using, right? A lot of it is happening right [30:43] there with the developers running their code  against cloud code and it's right on the edge. [30:50] you're only using the LLM. you're  sending the data back to them in a [30:55] uh in a context managed uh way and then using  that and then building on it right so similarly [31:06] uh as we think about all of this there's going to  be I feel there's going to be a major major shift [31:12] in how inference happens how you leverage and  build some of these workflows um and how do How [31:21] does it really um play into the business case use  case? Um when we think about uh edge inference, [31:37] one of the key things that's going to play in  also is your context management. So today you [31:42] probably hear u anthropic and all those talking  about your 1 million context but you can also see [31:50] in the number of chats that are happening you are  losing that context. um you start with the first [31:56] chat or or within the chat 15 messages later cloud  has to go read the data back again and the data [32:06] is actually sitting right there within your CPU GP  or laptop wherever it is and similarly enterprises [32:15] also have all of their data sitting maybe in  local databases and all of that has to be bought [32:20] together now to do the context there's different  ways Again there's vector data that you need to [32:26] parse which requires another set of models you  need to parse it into knowledge graphs or build [32:31] ontologies when bring that together from a recall  perspective where you now start using LLMs again. [32:38] So as you start to put these systems together,  there are different points within which you're [32:44] going to use the SLMs or the LLMs or the embedding  models, whichever the use cases might be. And a [32:50] lot of that is going to end up happening on the  edge. uh with the foundation models, the big ones [32:57] which you probably don't even want to run because  that's just a ton of hard GPU or uh the things um [33:06] versus your local uh business use case relevant.  Got it. We are running a little bit short on time [33:13] as of the last question, right? Um for and and  launch is coming up. So as some thought for food [33:22] uh [laughter] uh what so um a year later if we  are sitting here at AI4 2027 what's going to [33:33] be the biggest change in your industry in the AI  infrastructure site and um what's going to change [33:40] in the uh scaling factor the uh deployment  dynamic right and also um I want you to also [33:49] to answer uh this. So if you have a like a magic  wand, what do you want to change in the industry [33:55] as well? What's your what's your u wishful  thinking for the industry? So one prediction [34:01] and one wishful thinking. Let's go. Okay, really  quick. [clears throat] Um, if we work very hard, [34:09] we can expand the universe of people who can be  successful with fine-tuning and training from just [34:17] people who have PhDs in math and statistics to  every software developer. I think the company and [34:23] the companies and the community that unlocks this  and enables people to do a continuous pipeline of [34:30] running inference, continuously sample sampling  data in a stratified way, labeling that data, [34:37] getting that data into a a training pipeline that  then re-releases the model onto the same storage [34:43] like we have in RunPod, like one big circle of  that that could be happening every single day and [34:48] there's like 93 steps in it and it's tremendous  and maybe like 40 50 of them are automated at the [34:52] best AI houses in Silicon Valley. Um, but whoever  gets there first, I think it'll be a very big [34:58] unlock and really change the way that we interact  with these models, bending them so that they're [35:04] better at precision and recall for the tasks  that are important to us. And that's how we can [35:09] rely less on frontier models and more on small  models that are very good at what we need them [35:15] to do. I look forward to that, David. Uh, yeah,  sure. So I think uh what we often forget is like [35:22] we're actually only a few years into the whole AI  thing. So so also next year I think we will take [35:27] another major step in professionalizing things and  then iterations will become even faster maybe and [35:35] uh also the buildup. So I think we're only  at the beginning of the expansion. That is my [35:39] uh my prediction. And then for a wish maybe uh  I hope everyone continues to AI to to to make [35:47] the world better not just to produce cat memes  and that's the personal wish. I have the same [35:52] wish [laughter] practic yeah um I I think the  technology comes always in like a circle. Um [36:02] if we look back of starting with technology there  were huge mainframes then we went to client server [36:08] architectures and then we went to the cloud things  I feel we're going through that same cycle again [36:13] where now instead of the CPUs now it's the GPUs  you can see the H100's as massive compute boxes [36:21] as big as those mainframe that you're rolling  out and putting out there and your architectures [36:26] are with the edge especially are going to  a little bit more towards that cliententric [36:32] um architecture client server um kind of approach  where you mix that and then as this evolves maybe [36:39] with quantum or however they're able to reduce  that down to a chip and you have it running like [36:44] in your laptops at some point um it'll be that  cycle is what I see. Uh what I wish I would say is [36:54] um I AI is evolving um really fast but I would  wish that people use it more as a tool and [37:04] not more just to be scared about it learn to  leverage it. Um and I tell that to everybody. [37:13] uh when humans are highly evolving creatures.  Oh, sorry, we're already over, but I I'll stop [37:21] there. [laughter] Go ahead. Yeah. Yeah. Okay.  Final words. Uh what I expect is probably more [37:28] um flexible and dynamic um architectures um public  cloud um private cloud and and uh applications [37:40] running uh on both. and being uh like shiftable  and specifically on on edge was already mentioned [37:48] uh we are seeing how um this AI processors become  more efficient. So I expect that more workloads [37:56] will be moved to um the on on edge applications.  Um we have seen uh already that many banks [38:04] financial industry uh are offering mobile apps  and already have contain [clears throat] almost [38:11] all range of banking services there. But  probably I expect more even more. And what I [38:19] um what I wish um I see [laughter] the whole well  and and the market is embracing AI very quickly [38:33] but we don't see we don't know exactly to where  this developing is heading because we haven't [38:41] reached a life. So um there is no break event  yet and until then I don't know how it's going to [38:49] continue. It's going to be here to stay in the way  that it's profitable or not and I would um yeah [38:58] for myself wish a little more clarity here. So I  I hope we reach this uh state soon. Thank you very [39:08] much. Yep. Well guys, now you have your thoughts  for food and I think we are ready for lunch. Uh [39:14] let's give it up another round of applause for  our amazing panelist. Thank you guys so much.