Nvidia-Backed Upscale Targets AI Fabric to Halt GPU Idle Time

How Networks Shape the New AI Economy With Upscale — Ai4

Ai4

Because frontier model sizes outpace semiconductor memory growth, fabric latency directly dictates GPU utilization and token economics, turning network architecture into the primary driver of AI profitability.

AI-assisted summary of How Networks Shape the New AI Economy With Upscale — original source. Check the original for full context.

Source published: 2026-08-14

Detailed notes

## Background Published on YouTube by Ai4 on August 14, 2026, this session features Faisal Hanif from Product Management at Upscale discussing how networking defines the new AI economy. The presentation outlines why the transition of enterprise AI into production shifts the cost structure toward token unit economics, positioning network efficiency as the primary factor in tokens generated per dollar at scale.

## TL;DR As frontier AI models outgrow single-GPU memory capacities, networking has moved directly into the GPU execution cycle, turning communication latency into the primary bottleneck for compute utilization. Because synchronization patterns in data and expert parallelism mean a cluster is only as fast as its slowest link, network performance directly dictates key business metrics like tokens per second and tokens per watt. Upscale AI addresses these distributed computing challenges by integrating proprietary intra-node scale-up silicon with open, standards-based scale-out fabrics powered by NVIDIA Spectrum-X Ethernet. This unified, silicon-agnostic platform is architected to optimize heterogeneous AI workloads and prevent multi-million-dollar clusters from idling on stalled data transfers.

## Atmosphere & style The presentation carries a crisp, analytical, and authoritative tone structured for infrastructure engineers and enterprise leadership. Hanif delivers a measured, highly technical breakdown that moves methodically from the physical constraints of GPU high-bandwidth memory to top-line P&L balance sheets. The talk feels like an unvarnished masterclass in modern AI factory economics, emphasizing the hard physics of latency and interconnect bottlenecks over generic industry hype.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=4s) Networking has shifted from background infrastructure plumbing to sitting directly within the GPU execution loop, making it the primary economic lever in modern AI systems. - [04:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=244s) State-of-the-art frontier models (>1T parameters) vastly outpace single-GPU memory capacity (<300 GB HBM), making massively distributed cluster computing an inescapable baseline requirement. - [05:59](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=359s) Maintaining the standard 400 ms conversational response window requires an inference decode loop that touches billions of parameters per token, where any network stall immediately halts compute execution. - [10:03](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=603s) Distributed training and serving are tightly bound by Amdahl's Law and tail latency, meaning an entire cluster's execution speed is throttled to the pace of its slowest network link. - [10:03](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=603s) Diverse workload architectures create distinct network bottlenecks, ranging from All-Reduce barrier synchronization in Data Parallelism to All-to-All bandwidth flooding in Mixture-of-Experts (MoE) models. - [14:24](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=864s) Infrastructure metrics have shifted from traditional enterprise IT metrics like port count and uptime to direct business P&L KPIs: tokens per second, tokens per watt, and Model Flops Utilization (MFU). - [17:27](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1047s) The explosion of specialized heterogeneous accelerators (GPUs, LPUs, TPUs, and DPUs) demands open, unified fabric architectures to prevent costly hardware forklift upgrades as new ASICs emerge. - [02:29](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=149s) Adopting open-standard frameworks such as the Ultra Ethernet Consortium (UEC) and SONiC is essential to prevent proprietary vendor lock-in across AI fabrics. - [20:30](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1230s) Upscale AI addresses fabric fragmentation by coupling custom intra-node scale-up silicon with open 800G Spectrum-X scale-out Ethernet managed under a unified software substrate.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=4s) Networking as the AI Execution Loop

The AI industry is quietly learning the hard way that networking is no longer mere background plumbing. Instead, the network sits directly within the execution loop of every single GPU cycle. Every dollar invested in compute, every token generated, and every millisecond of latency traces back to how effectively the underlying fabric moves data between accelerators. Networking has emerged as the single largest economic lever in modern AI infrastructure, requiring a ground-up redesign rather than incremental adjustments to legacy data center switches.

Four primary shifts define this new landscape: AI workloads place fundamentally distinct demands on infrastructure compared to traditional enterprise computing; parallelism has become an inescapable physical requirement; the network dictates GPU execution cycles; and an explosion of heterogeneous silicon demands a unified, vendor-agnostic fabric.

```mermaid flowchart LR accTitle: Network within the GPU Execution Loop subgraph Compute["Compute & Memory"] A["GPU / Accelerator Cycles"] B["HBM / KV Cache"] end subgraph Network["Fabric Interconnect"] C["Scale-Up Interconnect\n(Intra-Node)"] D["Scale-Out Fabric\n(Spectrum-X / UEC)"] end subgraph Output["AI Economy KPIs"] E["Tokens per Second"] F["Tokens per Watt / Dollar"] end

A <--> C B <--> C C <--> D D --> E D --> F ```

[02:29](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=149s) The Upscale AI Mission and Architecture

Founded in 2025 with a singular focus on purpose-built networking for AI infrastructure, Upscale AI approaches the challenge across both scale-up and scale-out domains. The goal is to support heterogeneous compute environments—encompassing GPUs, LPUs (Language Processing Units), XPUs, and DPUs—without locking customers into proprietary silos.

Upscale AI tackles scale-up interconnects using proprietary purpose-built silicon and systems for intra-node communication, while standardizing scale-out fabrics on NVIDIA Spectrum-X Ethernet. The company is backed by NVIDIA as an investor and leverages open standards such as SONiC and the Ultra Ethernet Consortium (UEC). With a team of over 275 engineers, Upscale AI delivers a full stack covering silicon, systems, software, orchestration, services, and support.

- Founded in 2025 with a 275+ person engineering team. - Architecture spans both intra-node scale-up silicon and scale-out Ethernet fabrics under a unified software layer. - Strategic partnership and investment from NVIDIA utilizing Spectrum-X technology. - Strict adherence to open standards (SONiC, UEC) to eliminate vendor lock-in.

[04:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=244s) The Physics of Model Size and the 400ms Response

Modern AI infrastructure faces a rigid foundational constraint: state-of-the-art models simply do not fit into a single GPU. Frontier language models exceed one trillion parameters, while even flagship accelerators provide less than 300 GB of high-bandwidth memory (HBM). Because model sizes continue to expand much faster than semiconductor memory density—outpacing Moore's Law—distributing workloads across thousands of processors is the baseline starting condition for any serious AI deployment.

When a user interacts with a conversational interface like ChatGPT, receiving an answer within roughly 400 milliseconds feels instantaneous. However, that half-second window conceals billions of parameters across multi-layer models running on multi-million-dollar clusters coordinating in real time. The seamless user experience is the ultimate product, and network performance either guarantees or breaks that illusion.

[05:59](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=359s) Anatomy of an Inference Request: Pre-Fill, Decode, and Workload Diversity

Deconstructing a typical 400 ms inference turn reveals the operational burden placed on the interconnect:

1. **Front-End Pipeline (10–15 ms):** API gateway ingress, load balancing, tokenization, and model routing consume roughly 5% of the total request time. 2. **Inference Engine (Remaining 95%):** - *Pre-fill Phase:* Ingests the full prompt context and constructs the Key-Value (KV) cache. - *Decode Phase:* Executes an iterative forward pass generating one token at a time across billions of parameters.

This decode process represents a hyper-sensitive loop of raw compute, high-bandwidth memory access, and fabric interconnects. If the network stalls during data exchange, the entire execution pipeline halts. Furthermore, modern infrastructure cannot be tuned for a single behavior; Large Language Models (LLMs), Mixture-of-Experts (MoE), diffusion architectures, and dense embedding models each place radically different bandwidth and communication demands on the same shared cluster.

- One token forward pass touches 175B+ parameters in fractions of a second. - Scaling to 10,000 concurrent users generates trillions of operations per second across a shared distributed fabric. - Inefficiencies in scheduling, routing, memory placement, or congestion translate directly into lost revenue for GPU-as-a-Service and neo-cloud providers.

[10:03](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=603s) Parallelism Bottlenecks: Data, Expert, and Distributed Serving

Training a frontier model requires managing upwards of 25 terabytes of model state. Because a top-tier accelerator provides under 300 GB of capacity—addressing less than 1.4% of the requirement, leaving a massive 70%+ structural memory gap—parallel partitioning is unavoidable. Distributing workloads across thousands of chips forms a rigidly coupled system governed by Amdahl's Law and tail latency: the entire cluster moves only as fast as its slowest link.

```mermaid flowchart TD accTitle: Parallelism Communication Patterns subgraph DataParallel["Data Parallelism"] DP1["GPU 1..N"] -->|All-Reduce Synchronization| DP2["Global Barrier Wait"] end subgraph ExpertParallel["Mixture of Experts (MoE)"] EP1["GPU 1..N"] -->|All-to-All Token Exchange| EP2["Fabric Flooding / Cross-Traffic"] end subgraph InferenceServing["Distributed Serving"] IS1["Aggregator"] -->|Scatter-Gather| IS2["Worker GPUs"] IS2 -->|Tail Latency Bottleneck| IS1 end ```

- **Data Parallelism:** Imposes global `All-Reduce` synchronization barriers; if a single GPU or transceiver lags, all participating accelerators idle. - **Expert Parallelism (MoE):** Demands dense `All-to-All` communication matrices that saturate and flood cross-sectional network bandwidth. - **Distributed Inference Serving:** Employs scatter-gather patterns where a single delayed response link holds up token generation for the entire client session. - As workloads transition from memory-bound (fitting weights onto local chips) to communication-bound (exchanging states across nodes), the physical network boundary forms the hard performance ceiling.

[14:24](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=864s) The AI Factory and the Shift to Business P&L

In an AI factory, GPUs, high-speed memory, storage, and networking operate in continuous synchronization to manufacture tokens. Inadequate synchronization directly suppresses tokens per second; sub-optimal routing introduces unpredictable latency spikes; and fabric congestion exacerbates tail-end stragglers, burning capital on idling compute hardware.

This architectural shift has fundamentally altered infrastructure Key Performance Indicators (KPIs). Traditional metrics like port density, uptime percentages, and packet counts have been supplanted by direct business metrics:

- **Time to First Token (TTFT) & Cluster Onboarding:** Taking 3 to 6 months to validate and deploy a new GPU cluster results in severe revenue loss during critical demand windows. - **Tokens per Second & Tokens per Watt:** The ultimate efficiency metrics for operational throughput and power economics. - **Effective GPU Utilization (MFU/HFU):** Mitigating idle time caused by communication wait states. - Infrastructure investments have officially migrated out of overhead IT budgets and straight into executive P&L statements.

[17:27](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1047s) Silicon Explosion and Unified Open Fabrics

Over a five-year trajectory, AI workload profiles have diversified dramatically. In 2023, infrastructure centered on dense pre-training workloads with predictable synchronization patterns across standard GPU/CPU nodes. By 2025, environments incorporated MoE architectures, disaggregated inference pipelines, LPUs, TPUs, and DPUs, culminating in persistent, always-on autonomous agents.

Because specialized accelerator silicon will continue multiplying, clusters cannot undergo full architectural redesigns or disruptive "forklift upgrades" whenever a new ASIC emerges. Networks must deliver open, multi-silicon interoperability across compute, memory, and storage fabrics.

```mermaid flowchart LR accTitle: Upscale AI Architectural Model subgraph HeterogeneousCompute["Heterogeneous Compute"] G["GPUs"] T["TPUs / XPUs"] L["LPUs / NPUs"] D["DPUs / Storage"] end subgraph UpscalePlatform["Upscale Unified Software Substrate"] SU["Scale-Up: Purpose-Built Silicon & Interconnects"] SO["Scale-Out: 64-Port 800G Spectrum-X Fabric"] end G & T & L & D <--> SU SU <--> SO ```

[20:30](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1230s) Upscale AI's Product Architecture and Core Tenets

Upscale AI resolves fabric fragmentation by coupling purpose-built intra-node scale-up interconnects with open scale-out systems under a single, unified software substrate. For scale-out deployments, Upscale delivers hardware such as a 64-port 800 Gbps switching platform powered by NVIDIA Spectrum-X Ethernet, incorporating hardware-accelerated load balancing, congestion-aware dynamic routing, and fine-grained telemetry specifically optimized for collective operations.

The Upscale AI architectural strategy is anchored on five core tenets: 1. **Full-Stack Ownership:** End-to-end integration across custom silicon, switching hardware, software stacks, orchestration tools, and enterprise services. 2. **Purpose-Built for AI:** Designed from scratch for high-throughput AI traffic patterns rather than repurposed from legacy enterprise or general cloud switches. 3. **Silicon-Agnostic Heterogeneity:** Native support across any mix of GPUs, XPUs, and LPUs without picking silicon winners. 4. **Open Standard Foundations:** Built upon Ethernet, Ultra Ethernet Consortium (UEC) frameworks, and SONiC to eliminate proprietary hardware lock-in. 5. **Unified Scale-Up and Scale-Out:** Consolidating intra-node and inter-node networking under one management and orchestration platform.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=4s) networking is no longer mere background plumbing. Instead, the network sits directly within the execution loop of every single GPU cycle. — Faisal Hanif

> [00:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=4s) Every dollar invested in compute, every token generated, and every millisecond of latency traces back to how effectively the underlying fabric moves data between accelerators. — Faisal Hanif

> [00:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=4s) Networking has emerged as the single largest economic lever in modern AI infrastructure, requiring a ground-up redesign rather than incremental adjustments to legacy data center switches. — Faisal Hanif

> [04:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=244s) state-of-the-art models simply do not fit into a single GPU. — Faisal Hanif

> [04:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=244s) Because model sizes continue to expand much faster than semiconductor memory density—outpacing Moore's Law—distributing workloads across thousands of processors is the baseline starting condition for any serious AI deployment. — Faisal Hanif

> [05:59](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=359s) If the network stalls during data exchange, the entire execution pipeline halts. — Faisal Hanif

> [10:03](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=603s) Distributing workloads across thousands of chips forms a rigidly coupled system governed by Amdahl's Law and tail latency: the entire cluster moves only as fast as its slowest link. — Faisal Hanif

> [10:03](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=603s) As workloads transition from memory-bound (fitting weights onto local chips) to communication-bound (exchanging states across nodes), the physical network boundary forms the hard performance ceiling. — Faisal Hanif

> [14:24](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=864s) In an AI factory, GPUs, high-speed memory, storage, and networking operate in continuous synchronization to manufacture tokens. — Faisal Hanif

> [14:24](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=864s) Traditional metrics like port density, uptime percentages, and packet counts have been supplanted by direct business metrics — Faisal Hanif

> [14:24](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=864s) Infrastructure investments have officially migrated out of overhead IT budgets and straight into executive P&L statements. — Faisal Hanif

## Mentioned Faisal Hanif, Upscale AI, Ai4, NVIDIA, NVIDIA Spectrum-X, SONiC, Ultra Ethernet Consortium (UEC), ChatGPT, Moore's Law, Amdahl's Law

## Video description Faisal Hanif - Product Management, Upscale

Enterprise AI has moved past experimentation and into production, and the cost structure has fundamentally changed. Tokens are now the primary unit of AI value, and the network sits at the center of every decision around how many are produced, how fast, and at what cost. As inference workloads come to dominate AI spending, network efficiency directly determines tokens per dollar at scale. In this session, I will explore how organizations that architect for agentic workloads, disaggregated inference, and open fabric infrastructures today will be best positioned to compete in the AI economy of tomorrow.

[Source](https://www.youtube.com/watch?v=su1ZPDOd_fY)

中文简报

英伟达支持的Upscale发力AI网络架构，力图消除GPU空转由于前沿模型规模的增速超越了半导体存储的增长，互联延迟直接决定了GPU利用率与Token经济效益，使网络架构成为AI盈利能力的核心驱动力。随着前沿AI模型的规模超出单颗GPU的显存容量，网络通信已直接切入GPU的执行循环，通信延迟因而成为算力利用率的主要瓶颈。由于数据并行与专家并行中的同步机制意味着整个集群的速度取决于其最慢的链路，网络性能直接决定了每秒Token数和每瓦Token数等关键商业指标。Upscale AI通过将专有的节点内纵向扩展（scale-up）芯片与基于NVIDIA Spectrum-X以太网的开放标准横向扩展（scale-out）网络织构相结合，应对上述分布式计算难题。这一统一且独立于具体芯片的平台旨在优化异构AI工作负载，避免价值数百万美元的集群因数据传输停滞而陷入空转。网络已从传统的底层基础设施管道演进为直接嵌入 GPU 执行循环的关键环节，成为决定现代 AI 经济中每美元 Token 产出效率的核心杠杆。 (0:04) 万亿参数级前沿模型远超单卡 HBM 内存容量，使得大规模分布式集群成为刚需，而系统的生成吞吐与 400 毫秒交互响应窗口受到阿姆达尔定律和网络长尾延迟的严格制约。 (4:04) 基础设施评估指标已从端口数和运行时间等传统 IT 维度，全面转向首字延迟（TTFT）、每秒 Token 数、每瓦 Token 数及模型算力利用率（MFU）等直接挂钩企业损益的商业 KPI。 (14:24) 为应对 GPU、LPU、TPU 等多样化异构芯片的爆发，采用基于 UEC 与 SONiC 等开放标准的统一网络架构是避免硬件全面推倒重建和专有厂商锁定的关键路径。 (17:27)

Reading summary

Networking has shifted from passive background infrastructure into the direct execution loop of AI hardware, serving as a primary determinant of cluster economics and GPU efficiency as models exceed single-chip memory capacities. Physical Memory Gaps Force Parallel Execution (4:04) State-of-the-art frontier models now exceed one trillion parameters and require more than 25 terabytes of model state for training. Because flagship accelerators offer under 300 GB of high-bandwidth memory (HBM)—covering less than 1.4% of requirements and leaving a 70% or greater structural memory deficit—distributing workloads across thousands of discrete chips has become a physical necessity. As model sizes continue to outpace semiconductor memory density, AI workloads transition from being memory-bound to communication-bound, leaving network fabrics as the hard limit on performance. Inference Latency and Interconnect Bottlenecks (5:59) User interfaces typically demand response times around 400 milliseconds to feel instantaneous. In a standard inference turn, front-end routing and tokenization take 10 to 15 milliseconds, while the remaining 95% of the window is consumed by prompt pre-fill and iterative decoding across billions of parameters. During the decode phase, high-bandwidth memory access and inter-chip exchanges form a continuous loop; if communication stalls, execution stops entirely. Workload diversity amplifies these physical constraints. Data parallelism depends on global All-Reduce barriers that force all GPUs to wait for the slowest transceiver. Mixture-of-Experts (MoE) models generate dense All-to-All communication matrices that flood cross-sectional bandwidth, and distributed serving utilizes scatter-gather calls vulnerable to tail-latency stragglers that delay complete client sessions. AI Factory Economics and P&L Metrics (14:24) The operational paradigm of the AI factory treats clusters as synchronized token-manufacturing systems rather than conventional data centers. Fabric congestion, imperfect routing, and synchronization delays idle expensive compute capacity, which directly degrades tokens per second and tokens per watt. Consequently, operational priorities have moved away from traditional IT networking metrics like port counts and uptime percentages. Infrastructure performance is now evaluated through Model Flops Utilization (MFU), Hardware Flops Utilization (HFU), Time to First Token (TTFT), and the 3-to-6-month validation delays required to onboard new clusters, pushing fabric efficiency directly onto corporate P&L statements. Heterogeneous Silicon and Unified Fabrics (17:27) AI infrastructure has evolved from uniform pre-training clusters in 2023 to heterogeneous setups incorporating MoE architectures, LPUs, TPUs, DPUs, and persistent autonomous agents. To prevent costly forklift upgrades whenever new ASICs enter production, architectures require open interoperability across compute, memory, and storage fabrics. To address this heterogeneity, Upscale AI—founded in 2025 with over 275 engineers and backing from NVIDIA—combines purpose-built intra-node scale-up silicon with open scale-out systems under a unified software substrate. For inter-node networking, the architecture utilizes 64-port 800 Gbps platforms running NVIDIA Spectrum-X Ethernet alongside open standards like SONiC and the Ultra Ethernet Consortium (UEC), attempting to integrate full-stack hardware acceleration while avoiding proprietary vendor lock-in. The source maintains that purpose-built, open-standard fabrics spanning both scale-up and scale-out topologies can resolve communication bottlenecks across diverse silicon. However, the operational complexity of managing custom scale-up silicon alongside multi-vendor accelerators and nascent Ultra Ethernet standards remains an ongoing execution challenge.

精炼阅读摘要

根据Ai4发布的专题内容，Upscale AI指出网络已直接进入GPU计算执行循环并成为AI基础设施的核心经济杠杆，在万亿参数模型带来的严苛内存缺口与通信壁垒下，网络正推动数据中心向开放标准和软硬件统一的架构转型。物理内存鸿沟倒逼分布式并行通信 (4:04) 现代前沿大语言模型的参数量已超过一万亿，而顶级加速器的高带宽内存（HBM）容量仍低于300 GB。在训练前沿模型时，系统需处理超过25 TB的模型状态，单个芯片容量仅能满足不足1.4%的需求，造成超过70%的结构性内存缺口。由于芯片内存密度的增长速度明显滞后于模型规模扩展，分布式多芯片部署成为必然基准，这也迫使系统瓶颈从局部内存受限转向节点间的物理通信受限。根据阿姆达尔定律和长尾延迟机制，集群整体处理速度受制于最迟缓的链路。在数据并行中，全局All-Reduce同步屏障会导致单点延迟拖垮所有算力；在混合专家（MoE）架构中，All-to-All通信矩阵极易造成横截面网络拥塞；而在分布式推理服务中，分散-收集（scatter-gather）通信模式使得单条链路的拖延便会阻塞整体客户端会话的令牌生成。推理严苛时延与AI工厂损益重塑 (5:59) 在典型约400毫秒的对话推理交互中，前端API网关与路由调度仅耗时10至15毫秒（约占5%），其余95%的时间均消耗在推理引擎中。推理阶段包含构建KV缓存的预填充阶段，以及逐字迭代的解码阶段。解码过程涉及高敏感度的算力、高带宽内存与互联交互，单次前向传递需在零点几秒内触碰超过1750亿参数；一旦网络出现数据交换停顿，整套流水线将即刻挂起，而在万级并发下，共享网络每秒需承载数万亿次操作。这种架构特征推动了数据中心从传统技术指标向商业损益（P&L）的根本转变。新GPU集群验证部署若产生3至6个月的延误，会直接导致关键需求窗口期的收入流失。端口密度与在线率指标已被首字生成时间（TTFT）、每秒令牌数（Tokens/s）、每瓦特令牌数（Tokens/Watt）及模型浮点利用率（MFU/HFU）所取代，网络调优和拥塞控制直接关系到闲置算力损耗与运营成本。异构算力激增催生开放互联标准 (17:27) 从2023年至2025年，AI工作负载形态发生了显著分化，从规则同步的稠密模型预训练，演化为涵盖MoE架构、解耦推理流水线、专用语言处理单元（LPU）、TPU、XPU、DPU及常驻自主智能体的复合环境。随着专用加速芯片持续增加，数据中心无法在每引入一款新型ASIC时就实施推倒重来的“叉车式升级”（forklift upgrades）。为避免专有硬件绑定的技术孤岛，网络架构必须支持多厂商硬件的无缝互操作。通过遵循超以太网联盟（UEC）规范和开源网络操作系统SONiC等开放标准，底层网络能够在保持中立的前提下，跨计算、内存与存储层提供多芯片兼容的高吞吐数据传输能力。 Upscale AI的全栈统一互联架构 (20:30) 成立于2025年的Upscale AI拥有超过275人的工程团队，获得了NVIDIA的投资与战略支持。该公司提出覆盖自研芯片、交换硬件、软件栈、集群编排与专业服务的全栈方案，旨在统一解决节点内纵向扩展（Scale-Up）与节点间横向扩展（Scale-Out）的通信瓶颈。在纵向互联上，其采用专用定制芯片支持异构计算加速；在横向扩展上，则采用基于NVIDIA Spectrum-X以太网技术的64端口800 Gbps交换硬件平台。该平台结合了硬件加速负载均衡、拥塞感知动态路由与细粒度集合通信遥测。Upscale AI将技术路径归纳为五项核心原则：全栈所有权、专为AI高吞吐流量打造、芯片中立兼容、基于以太网开放标准，以及跨节点纵向与横向维度的统一调度管理。来源展示了网络从辅助通信设施上升为决定生成吞吐与商业回报核心要素的产业判断，以及基于开放以太网打造全栈互联的实践路径；但针对专用纵向扩展芯片与开放横向以太网在未来万卡超大规模集群协同中的长期工程兼容性，以及异构算力调度开销的实际量化表现，仍有待更长期的产业落地验证。

Detailed notes — zh

## Background 本内容由频道 Ai4 于 2026 年 8 月 14 日发布在 YouTube 平台上，演讲者为 Upscale 的产品管理负责人 Faisal Hanif。页面描述指出，企业 AI 已跨越实验阶段步入生产环境，Token 成为衡量价值的核心单位，而网络则处于决定产出速率与成本的中心位置，演讲旨在探讨如何通过 Agent 工作负载、解耦推理及开放 Fabric 架构在未来的 AI 经济中构筑核心竞争力。 ## TL;DR 演讲指出网络已不再是简单的数据管道，而是深嵌于每一个 GPU 执行周期的核心瓶颈，直接决定了企业的损益（P&L）与每美元 Token 产出。由于前沿大模型规模远超单卡显存物理极限，分布式计算已从显存受限全面转向通信受限，集合通信延迟直接绑架了集群算力利用率。随着 MoE、扩散模型及 LPU、TPU 等异构芯片大爆发，现代数据中心亟需摆脱厂商锁定与推倒重来式升级。Upscale AI 提出基于开放以太网与统一软件底座的全栈方案，将自研 Scale-Up 互联芯片与基于 NVIDIA Spectrum-X 的 Scale-Out 架构深度融合，致力于打造适应异构算力且稳定平滑的 AI 原生互联 Fabric。 ## Atmosphere & style 整场演讲呈现出一种高度务实、直击底层工程物理约束的硬核技术布道风格。演讲者 Faisal Hanif 摒弃了浮夸的行业黑话，用清晰而严肃的逻辑剖析了 400 毫秒推理延迟背后的精密齿轮咬合与巨额资产空转风险，叙事层层递进、发人深省。现场氛围专业而沉稳，在揭示网络成为算力最大杠杆的同时，传递出一种以系统工程驯服异构复杂度、“让网络运维重归枯燥稳定”的确定感与技术自信。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=4s) 网络已不再只是底层管道，而是直接嵌入到每个 GPU 的执行闭环中，成为决定 AI 商业效益与 Token 产出效率的核心杠杆。 - [02:29](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=149s) Upscale AI 致力于构建面向 AI 的专用网络，在 Scale-Up 领域自研互联芯片，在 Scale-Out 领域与 NVIDIA Spectrum-X 深度合作，全面拥抱以太网等开放生态。 - [03:58](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=238s) 模型规模的扩张速度远超物理显存增长，万亿参数大模型无法塞入单卡，分布式并行计算成为现代 AI 基础设施的物理法则。 - [05:03](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=303s) 在 400 毫秒的推理响应中，95% 的时间消耗在推理引擎内部，计算、显存与跨卡互联紧密咬合，网络抖动或丢包会直接导致整个流水线停顿。 - [05:03](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=303s) 大语言模型、混合专家模型（MoE）及扩散模型等通信特征各异的工作负载在同一集群共存，对底层互联网络提出了严苛挑战。 - [08:10](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=490s) AI 架构的核心瓶颈已从单纯的“显存受限”全面转向“通信受限”，分布式集群的整体性能取决于最慢的那条通信链路。 - [08:10](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=490s) 数据并行（All-Reduce）与专家并行（All-to-All）等集合通信模式高度依赖跨节点数据交换，网络拥塞会使算力设备陷入大面积等待。 - [14:24](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=864s) 现代 AI 数据中心被重新定义为“Token 工厂”，其核心 KPI 已转变为首个 Token 交付时间、每秒 Token 产出以及真实有效 GPU 利用率。 - [14:24](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=864s) 网络基础设施不再属于传统 IT 运维预算科目，其效率直接决定单 Token 成本，成为左右企业利润与损益表（P&L）的关键驱动力。 - [17:27](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1047s) 专用加速芯片（GPU、TPU、LPU、DPU）种类持续爆发，数据中心亟需构建无需推倒重来、支持异构计算的统一开放网络 Fabric。 - [19:37](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1177s) Upscale AI 采用统一软件底座协同管理 Scale-Up 专用互连与 Scale-Out 800G 以太网交换系统，实现全局遥测与自适应拥塞控制。 - [19:37](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1177s) Upscale AI 坚持全栈自研、AI 原生定制、异构芯片支持、开放标准以及双域统一平台五大核心信条，破除专有协议的厂商锁定。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=4s) 重新定义网络：网络已深嵌于每个 GPU 周期中 Faisal Hanif 开宗明义地指出，AI 产业界正在通过惨痛的教训认识到一个事实：“网络不再只是水暖管道（Networking is no longer plumbing）。”在现代 AI 架构中，网络已经直接介入到每一个 GPU 周期的执行闭环（execution loop）中。企业在硬件上投入的每一美元、系统产出的每一个 token 吞吐量，以及产生的每一毫秒延迟，本质上都取决于底层网络架构在加速器或 GPU 之间转移数据的效率。网络已经成为驱动整个 AI 经济效益的最大单点杠杆，必须从底层对其进行彻底重构。本次演讲围绕四大核心议题展开：第一，AI 工作负载对基础设施的需求发生了根本性颠覆，与传统数据中心负载截然不同；第二，并行计算不再是可选方案，而是现代 AI 基础设施的物理法则；第三，网络直接置身于 GPU 的执行闭环内；第四，芯片异构化（Heterogeneity）迎来大爆发，对通用互联网络提出了严苛挑战。 ```mermaid flowchart LR accTitle: Network in AI Execution Loop subgraph Compute["计算与存储层"] A["GPU / 加速器周期"] B["高带宽显存 (HBM)"] end subgraph Network["互联网络Fabric"] C["Scale-Up / Scale-Out Fabric"] end subgraph Metrics["商业与性能指标"] D["每美元 Token 产出"] E["端到端延迟与吞吐"] end A <--> C B <--> C C --> D C --> E ``` --- ### [02:29](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=149s) Upscale AI 创立使命与开放架构布局 Upscale AI 成立于 2025 年，其核心使命是为 AI 基础设施打造专用的网络体系（purpose-built networking）。团队目前已拥有超过 275 名成员，全栈业务涵盖芯片设计、硬件系统、软件平台、编排调度以及支持服务。Upscale AI 采取纵向扩展（Scale-Up）与横向扩展（Scale-Out）双管齐下的战略：在 Scale-Up 领域自研专用互连芯片与硬件系统；在 Scale-Out 领域则与 NVIDIA 达成深度合作，基于 NVIDIA Spectrum-X 以太网平台构建系统，NVIDIA 同时也是该公司的投资方之一。 Upscale AI 的技术路线全面建立在开放标准之上，彻底摒弃专有协议的厂商锁定（vendor lock-in）： - **开放操作系统与生态**：全面拥抱 SONiC、以太网（Ethernet）及超级以太网联盟（UEC, Ultra Ethernet Consortium）等开放行业标准。 - **异构加速器支持**：能够同时连接 GPU、LPU、XPU、DPU 等各类异构计算芯片。 - **全栈交付能力**：从自研硅片到端到端集群网络编排，提供完整的 AI 互连交付。 --- ### [03:58](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=238s) 基础设施的硬约束：模型规模超越单卡显存 AI 工作负载与传统企业级负载最本质的区别在于基础物理约束：大语言模型根本无法被塞进单颗 GPU 中。当前最前沿的旗舰大模型参数量早已突破 1 万亿（1 Trillion），而即便业界最高规格的单颗加速芯片，其显存容量也仅能容纳整个模型极小的一部分。摩尔定律或单纯的单卡显存提升无法弥合这一鸿沟，因为模型规模扩张的速度远远超过了物理显存容量的增长斜率。这种物理现实决定了：将模型拆分并分布式部署在成百上千颗 GPU 上，不再是一种优化选项，而是运行任何严肃 AI 工作负载的绝对前提起点。 --- ### [05:03](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=303s) 400 毫秒的背后：推理流水线与复杂工作负载的共存在终端用户使用诸如 ChatGPT 这类 AI 对话应用时，输入问题后约 400 至 500 毫秒即可获得响应，体验看似即时且无缝。然而在这不到半秒的时间窗口内，后端触发了极其庞大的精密协作：跨越 14 层以上的模型结构、数百亿参数的运算，以及价值数百万美元的算力基础设施在实时协同。Faisal Hanif 强调：“这种无缝感本身就是产品，而网络决定了它的成败。” 拆解这 400 毫秒的生命周期可以发现：API 网关、负载均衡、分词（Tokenization）与模型路由仅占用前 10 到 15 毫秒；剩余 95% 的时间全部消耗在推理引擎内部。推理过程首先经历读取输入并构建 KV 缓存（KV Cache）的预填充阶段（Pre-fill phase），随后进入按 token 生成的解码循环（Decode loop）。每一次前向传播（forward pass）都需要调用数十亿甚至千亿级参数，计算、显存访问与节点间互联紧密咬合，任何一处发生停顿（stall），整个流水线就会立刻陷入停滞。此外，AI 绝非单一类型的工作负载。现代数据中心需要同时承载多种计算特征迥异的模型： - **大语言模型（LLM）**：计算密集与内存受限交织。 - **混合专家模型（MoE）**：对通信带宽极度饥渴，依赖高频的全对全数据交换。 - **扩散模型（Diffusion Models）与向量嵌入（Embeddings）**：具有完全不同的吞吐与数据传输特征。 - **共存挑战**：企业期望这些差异极大的模型运行在同一套 AI 平台上，这直接将压力转移到了底层互联网络上。 ```mermaid flowchart TD accTitle: Breakdown of 400ms Inference Request subgraph Frontend["前端处理阶段 (10-15 ms)"] A["用户请求进入"] --> B["API 网关 / 负载均衡"] B --> C["分词 Tokenization & 路由"] end subgraph Backend["推理引擎阶段 (剩余 95% 耗时)"] C --> D["Pre-fill 阶段: 读取输入并构建 KV Cache"] D --> E["Decode 循环: 逐 Token 推进前向计算"] E --> F["高频闭环: 算力计算 <--> 显存读取 <--> 网络跨卡互联"] end subgraph Impact["网络抖动风险"] F -.-> G["互联阻塞/丢包导致流水线停顿 (Stall)"] end ``` --- ### [08:10](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=490s) 并行化的物理法则：从显存受限走向通信受限在处理单次推理请求时，单个 token 的生成就意味着一次前向计算，需在 400 毫秒内触达超过 1750 亿个参数；当并发扩展到 10,000 名用户同时在线、每秒消耗数万个 token、执行数万亿次操作时，系统就彻底演变为一个复杂的分布式生态系统。调度策略、显存放置、容错机制、网络拥塞以及路由决策中的任何微小低效，都会直接转化为高昂的商业成本。对于 Neo Cloud 提供商或 GPU 即服务（GPUaaS）运营商而言，每一毫秒的额外延迟都在侵蚀其净利润。在前沿训练场景中，这种鸿沟更为惊人：一个前沿模型的训练状态（State）可能需要高达 25 TB 的显存空间，而顶级单卡显存往往不足 300 GB，单卡容量仅占所需容量的约 1.4%，存在高达 70% 以上的物理容量缺口。这意味着数以千计的加速器必须紧密耦合在一起。在分布式系统物理学中，“系统的运行速度完全取决于最慢的那条链路”。单纯堆砌更大显存的 GPU 固然能缓解显存墙（Memory Wall），但跨芯片的数据移动依然是核心瓶颈。在具体并行模式中，网络瓶颈表现出相同的底层特征： - **数据并行（Data Parallelism）**：例如 8 颗 GPU 协同执行 All-Reduce 操作，任何单卡的轻微落后都会迫使所有其他 GPU 处于闲置等待状态。 - **专家并行（Expert Parallelism, MoE）**：采用 All-to-All 交换模式，每颗 GPU 都要向其他所有 GPU 发送数据，极易引发突发流量淹没网络。 - **分布式推理服务**：采用 Scatter-Gather（分发-汇聚）模式，单台服务器或链路的延迟会直接拖慢整个响应。整个架构的发展路径呈现出清晰的演进：模型超限 $\rightarrow$ 引入并行切分 $\rightarrow$ 构建集合通信（Collectives）结构 $\rightarrow$ 系统性质从“显存受限（Memory-bound）”全面转向“通信受限（Communication-bound）”。一旦计算跨越单机节点边界，网络便成为整套算力集群的性能天花板。 --- ### [14:24](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=864s) 算力资产新核算体系：AI 工厂的 P&L 革命在当前的 AI 时代，数据中心被重新定义为“代币工厂（Token Factory）”。其解剖学本质是 GPU、显存、存储与网络在步调一致的锁步（lock-step）机制下高效运转以生产 Token。网络如同粘合剂将所有组件绑定，一旦出现同步不良，Token 产出速率就会骤降；路由算法不佳会导致延迟激增；网络拥塞更会引发恶性连锁反应，导致企业花费重金租赁的 GPU 陷入空转烧钱。这种技术变化彻底颠覆了基础设施的考核指标（KPI）： - **传统指标**：过往关注设备正常运行时间（Uptime）、端口数量、网络带宽以及每秒数据包数（PPS）。 - **现代 AI 核心指标**： 1. **首个 Token 交付时间（Time to First Token）**：集群从上架到投入生产的速度。若搭建和调优一个集群需要耗费 3 到 6 个月，闲置资产将造成巨额资本浪费。 2. **生产效率与能效**：每秒 Token 生成量（Tokens/sec）、每瓦特 Token 产出（Tokens/watt）。 3. **硬件有效利用率**：真实有效 GPU 利用率（GPU Utilization）。基础设施因此完成了一次深刻转型：它已不再属于传统的企业 IT 运维预算科目，而是直接挂钩企业商业利润的核心损益表（P&L）。 --- ### [17:27](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1047s) 异构芯片大爆发与“让网络运维变得枯燥” 过去五年内，AI 负载形态经历了剧烈分化： - **2023 年**：以稠密模型（Dense Model）训练为主，通信模式相对单一可预测，主要依赖 GPU 与 CPU 协同。 - **2025 年及未来**：混合专家模型（MoE）训练、解耦推理（Disaggregated Inference）成为主流，LPU（语言处理单元）、TPU、DPU 大量涌现，更演进出永远在线的持久化智能体（Persistent Agents）。专用芯片的种类不会收敛，只会加速繁殖，且每种芯片都会带来独特的通信模式。然而，数据中心不可能在每次引入新芯片时都拆除并重构一次网络。因此，行业亟需一种能够连接各类 GPU、TPU、LPU、NPU、DPU 及存储与显存域的统一网络 Fabric。该统一 Fabric 必须具备四大特征： 1. **开放无锁定（Open & Vendor Agnostic）**：基于开放标准设计，避免被单一硬件厂商绑定。 2. **原生异构支持**：从设计之初便兼容多种不同形态的硅片。 3. **AI 原生优化**：针对 AI 通信特征定制，而非沿用传统企业级云网络的旧技术。 4. **让运维变得枯燥（Operationally Boring）**：当引入新型芯片或扩容时，无需进行颠覆性的推倒重来式升级（No forklift upgrades），保持底层架构的长期稳定平滑。 --- ### [19:37](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1177s) Upscale AI 的技术实现：Scale-Up 与 Scale-Out 的一体化融合为了应对异构演进，Upscale AI 聚焦于 Scale-Up 与 Scale-Out 这两大关键领域，并坚持将其置于统一的软件底座（Software Substrate）下进行管理，而非机械地将两套独立产品拼凑在一起： ```mermaid flowchart TD accTitle: Upscale AI Architecture subgraph Substrate["统一软件底座 (Unified Software Substrate)"] SW["统一遥测、自适应路由、拥塞控制与集群调度编排"] end subgraph ScaleUpDomain["Scale-Up (纵向扩展)"] A["自研专用互连硅片与系统"] --> C["节点内/近节点极低延迟数据交换"] end subgraph ScaleOutDomain["Scale-Out (横向扩展)"] B["NVIDIA Spectrum-X 以太网系统"] --> D["64 端口 800G 交换系统"] end SW --- ScaleUpDomain SW --- ScaleOutDomain C --- E["异构计算集群 (GPU, TPU, LPU, DPU, XPU)"] D --- E ``` Upscale AI 现场展示了用于 Scale-Out 网络的 64 端口 800G 交换系统。结合 Spectrum-X 以太网技术，该系统具备动态负载均衡、拥塞感知路由（Congestion-aware routing）以及面向集合通信操作（Collectives）深度优化的精细化网络遥测能力。 Faisal Hanif 在总结时提炼了 Upscale AI 坚守的五大核心信条（Five Core Tenants）： 1. **全栈自研与交付（Full Stack）**：完整掌控从芯片、硬件系统、软件平台、集群编排到技术支持与服务的整个纵向链条。 2. **AI 原生定制（Purpose-Built for AI）**：从研发第一天起就纯粹面向 AI 负载通信特征设计，绝不妥协使用传统企业级网络设备改装。 3. **异构计算支持（Heterogeneous Compute）**：跨芯片全面兼容各类 XPU，“我们不去预设谁是唯一的芯片赢家”。 4. **坚持开放标准（Open & Standard-Based）**：基于标准以太网、超以太网技术，消除专有生态锁定。 5. **双域统一软件底座（One Platform）**：通过单一软件平台统筹 Scale-Up 与 Scale-Out，实现全集群的高效协同。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=4s) Networking is no longer plumbing. In AI, networking is in the execution loop of every single GPU cycle. — Faisal Hanif > [05:03](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=303s) The seamless experience is the product, and networking makes or breaks that product. — Faisal Hanif > [08:10](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=490s) In distributed physics, you are only as fast as the slowest link. — Faisal Hanif > [14:24](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=864s) Today, a data center is a token factory. If you don't synchronize, tokens per second drops. If you have bad routing, latency spikes. If you have congestion, money literally burns as GPUs sit idle. — Faisal Hanif > [14:24](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=864s) Infrastructure is no longer an IT cost center. It is a P&L line item. — Faisal Hanif > [17:27](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1047s) Specialized chips are not going to converge. They are going to proliferate. — Faisal Hanif > [17:27](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1047s) Make networking operationally boring. When you bring in a new chip, you shouldn't have to forklift upgrade the network. — Faisal Hanif > [19:37](https://www.youtube.com/watch?v=su1ZPDOd_fY&t=1177s) We don't pick the silicon winner. We connect them all. — Faisal Hanif ## Mentioned Faisal Hanif, Upscale AI, Ai4, NVIDIA, NVIDIA Spectrum-X, SONiC, Ultra Ethernet Consortium (UEC), ChatGPT, GPU, TPU, LPU, DPU, XPU, NPU, HBM, Ethernet, MoE (Mixture of Experts) ## Video description Faisal Hanif - Product Management, Upscale Enterprise AI has moved past experimentation and into production, and the cost structure has fundamentally changed. Tokens are now the primary unit of AI value, and the network sits at the center of every decision around how many are produced, how fast, and at what cost. As inference workloads come to dominate AI spending, network efficiency directly determines tokens per dollar at scale. In this session, I will explore how organizations that architect for agentic workloads, disaggregated inference, and open fabric infrastructures today will be best positioned to compete in the AI economy of tomorrow. [Source](https://www.youtube.com/watch?v=su1ZPDOd_fY)

Source quotes

[0:04] Faisal Hanif: Networking is no longer plumbing. In AI, networking is in the execution loop of every single GPU cycle. [5:03] Faisal Hanif: The seamless experience is the product, and networking makes or breaks that product. [8:10] Faisal Hanif: In distributed physics, you are only as fast as the slowest link. [14:24] Faisal Hanif: Today, a data center is a token factory. If you don't synchronize, tokens per second drops. If you have bad routing, latency spikes. If you have congestion, money literally burns as GPUs sit idle. [14:24] Faisal Hanif: Infrastructure is no longer an IT cost center. It is a P&L line item. [17:27] Faisal Hanif: Specialized chips are not going to converge. They are going to proliferate. [17:27] Faisal Hanif: Make networking operationally boring. When you bring in a new chip, you shouldn't have to forklift upgrade the network. [19:37] Faisal Hanif: We don't pick the silicon winner. We connect them all.

Source quotes

[0:04] Faisal Hanif: networking is no longer mere background plumbing. Instead, the network sits directly within the execution loop of every single GPU cycle. [0:04] Faisal Hanif: Every dollar invested in compute, every token generated, and every millisecond of latency traces back to how effectively the underlying fabric moves data between accelerators. [0:04] Faisal Hanif: Networking has emerged as the single largest economic lever in modern AI infrastructure, requiring a ground-up redesign rather than incremental adjustments to legacy data center switches. [4:04] Faisal Hanif: state-of-the-art models simply do not fit into a single GPU. [4:04] Faisal Hanif: Because model sizes continue to expand much faster than semiconductor memory density—outpacing Moore's Law—distributing workloads across thousands of processors is the baseline starting condition for any serious AI deployment. [5:59] Faisal Hanif: If the network stalls during data exchange, the entire execution pipeline halts. [10:03] Faisal Hanif: Distributing workloads across thousands of chips forms a rigidly coupled system governed by Amdahl's Law and tail latency: the entire cluster moves only as fast as its slowest link. [10:03] Faisal Hanif: As workloads transition from memory-bound (fitting weights onto local chips) to communication-bound (exchanging states across nodes), the physical network boundary forms the hard performance ceiling. [14:24] Faisal Hanif: In an AI factory, GPUs, high-speed memory, storage, and networking operate in continuous synchronization to manufacture tokens. [14:24] Faisal Hanif: Traditional metrics like port density, uptime percentages, and packet counts have been supplanted by direct business metrics [14:24] Faisal Hanif: Infrastructure investments have officially migrated out of overhead IT budgets and straight into executive P&L statements.

Original transcript

[0:04] So my name is uh Fisel Hanife. I am responsible for product management at Upscale AI. Uh we've [0:13] been building purpose-built networking for AI. Uh but today we're going to focus on [0:20] something that the AI industry is quietly learning the hard way that networking is [0:28] no longer plumbing. Actually network is in the execution loop of every GPU cycle. Every dollar [0:36] of GPU spent and every token of throughput and every millisecond of delay or latency [0:45] it traces back to how well your fabrics move the data between your accelerators or GPUs. [0:53] So by the end of this hopefully you'll be able to see that network is the single [0:58] biggest lever for the AI economies and why it needs to be rethought from grounds up. [1:10] So here's the arc of the talk. We will start with a very brief quick intro into what a [1:19] upscale AI is and then we'll walk through the four main big ideas. Uh first of all, [1:28] how AI is fundamentally changing what's expected from the infrastructure. It's nothing like the [1:35] traditional data center workloads. Secondly, why parallelism is no longer optional. [1:44] It's actually the physics of the modern AI infrastructure. And third, which most people miss, [1:53] network is literally in the execution loop of every GPU cycle. And fourth, heterogenity. [2:05] That's the explosion in the different types of silicon and why it's makes it harder and demands [2:15] a new kind of fabric. So we'll close this how at upscale AI we are addressing this to solve this. [2:29] So as I said a brief intro into upscale AI. We were founded in 2025 with a singular mission [2:39] purpose-built networking for AI infrastructure. Right? So we are building across two domains. [2:47] So scale up and scale out and also providing the heterogeneous compute across your GPUs, [2:56] LPUs, XPUs or DPUs and all and everything under the sun. We're scale up interconnect. [3:05] We're building the scale up systems or scale up uh with the purpose-built silicon and systems [3:11] scale out fabrics based on Nvidia Spectrum X. So we have partnered with Nvidia and you [3:18] can also see Nvidia is listed as one of the one of the investor as well and all of this [3:24] is based on open standard base like Sonic, EAN, UEC. So no lock in everything's based [3:35] on open standards and we are a 275 plus strong team today building across the silicon system [3:48] software orchestration services and support. So the the whole end to end for your AI networking [3:58] let's start with the bird's eye view what actually AI demands from the infrastructure [4:04] because as I said earlier this is nothing like the traditional data center workloads AI fundamentally [4:10] changes or AI has AI has fundamentally changed the infrastructure so here is the foundational [4:18] constraint models do not fit into a single GPU period. The largest language models today, [4:27] they are well north of a trillion parameters and a single GPU or single accelerator even the best one [4:36] only holds a fraction of that. And this is isn't something that M's law or any other law can solve. [4:43] Model sizes are growing faster than the the memory capacity. So distribution across the many GPUs [4:52] isn't optional. It's the starting condition for pretty much every serious AI workload. [5:03] Let's start with a with a user experience. So we have all uh we have our favorite AI [5:12] chat bots. Let's look at maybe like chat GPD here. You type a question, [5:18] you get a response within half a second. It feels instant. But behind that 400 millisecond, there [5:28] is an extraordinary amount of machinery. There is 14 layers of models, billions of parameters, [5:35] multi-million dollar infrastructure, all pretty much coordinating in real time. [5:41] So the seamlessness is the product and the network is what makes it possible [5:50] or what breaks it. So that's a very different way of thinking about the AI. [5:59] Let's unpack that 400 millconds do we? All right. So there is you asked a question. So there is [6:09] an API gateway, a load balancer, tokenization, model routing. So all of that pretty much takes [6:20] about 10 to 15 milliseconds. Where is the other 95%. Wait, right, that's your inference engine, [6:32] right? The pre-fill phase [snorts] reads your entire input. It builds the KV cache. Then the [6:41] decode loop runs. One token per forward pass. The billions of parameter executed each time. [6:50] But if you look at look at it under the hood, there's a tight loop of compute, memory access, [6:57] and interconnects. If any of those stall, the whole pipeline stalls. So that's why the hardware [7:06] layer, especially the network that's tying the GPUs matter so much even for a single inference [7:14] request. So here's the reality. Not all models are the same. Not all workloads behave the same way. [7:24] And we have established the fact like AI isn't a single workload. We have as listed here large [7:32] language models, mixture of experts, diffusion models, they are really they are like bandwidth [7:41] hungry um embeddings. They all pretty much behave differently. They stress the infrastructure very [7:50] differently. Yet interestingly, many of us expect all of them to coexist [7:58] on the same AI platform. That's a networking challenge as much as the compute challenge. [8:10] So let's put that into perspective. We'll talk about we'll start with just one token generation [8:19] equal to one forward pass. That's equal to almost 175 billion plus parameters touched in [8:28] roughly 400 milliseconds like a half a second. Now imagine that happening across like 10,000 [8:39] times simultaneously like you have thousands of concurrent users, thousands of tokens consumed [8:45] per second and trillions of operations per second. So suddenly this is not just a compute problem, [8:56] it's a distributed ecosystem problem. You have scheduling, memory placement, fall tolerance, [9:04] congestion, routing. Every in inefficiency here has a measurable business cost. So [9:13] every millisecond, every millisecond of delay costs money. If you're a Neo cloud provider [9:22] or you're selling GPU as a service, this is impacting your bottom line. That's a reality. [9:34] So let's establish this this for now. So if we have already established that the models do not [9:42] fit into a a single GPU. So we need hundreds and thousands of them working together. So [9:49] how do we actually partition that work? The answer is parallelism. and it's not optional. [10:03] So let's put this into perspective. So you look at a frontier training model that typically needs [10:15] like up to 25 terabytes of state. If you look at a single GPU, maybe the best one, [10:23] it has less than 300 GB of memory, that's about 1 1.4% of what's needed. What's the gap? 70%. [10:36] There's a 70% gap. So we often describe AI cluster as a collection of GPUs. We [10:43] already established that. So there's this is a distributed ecosystem. One slower link pulls the [10:54] whole system down because it's a distributed but everything needs to work together in in [10:59] real time. The solution is as shown here all of these accelerators are GPUs working together. [11:10] So that's parallelism that's distributed system. So distribute the the and if you [11:17] look at the distributed AI is only as fast as the the the slowest link or slowest dependency. [11:27] If you have bigger GPUs, they reduce the memory wall, but the data data movement across them [11:35] still remains can be the bottleneck. So that's a network problem that's often overlooked. [11:49] So once you distribute across thousands of the GPUs, so we have created something [11:55] fundamentally new, a tightly coupled system where no part can run ahead of any any other [12:04] part. And the slowest link the slowest link sets the pace for the entire system. [12:16] Let's look at this through the training and the inference separately. On the training side, you [12:23] have top side data parallelism creates a global synchronization pattern. Here in the example, [12:31] you have like let's say eight GPUs doing an all reduce function. One one runs slow, every [12:38] every other GPU waits. If you look at the expert parallelism, it's an allto-all exchange. Every [12:46] GPU sends to every other GPU and it flirts the network. On the inference side, the distributed [12:57] serving is scatter gather. One slower server or a link delays the entire response. So every one [13:11] of these bottlenecks is a network bottleneck. Different pattern but has the same root cause. [13:22] Here's the progression. So kind of pretty much bringing everything together that we have gone [13:27] through. You start with a model. It's too big for one GPU. So you partition it. That's parallelism. [13:36] You add a structured you add a structured communication around it. That's collectives. [13:46] And now you're managing a distributed ecosystem, not just a chip. So if you start to dig deeper, [13:59] the nature of the problem now shifts from the memory bound meaning [14:05] Can I fit the model to communication bound? Can I move the data fast enough between my GPUs, [14:12] XPUs? Once you cross the node boundary, the network becomes the performance ceiling. [14:24] So here's the core thesis. The network isn't adjacent to just [14:30] the GPU compute. It's in the it's in that execution loop of every GPU cycle. [14:38] Every token decode depends on the network performance. [14:45] So we've been I think here talking about the token factory. So if you look at the overall [14:52] what is a token factory under the hood the the token factory if you look at the anatomy of a [15:00] token factory is nothing but a GPU memory storage networking all working in lock step [15:09] to produce tokens that's a token factory AI factory different names same Meaning [15:20] bad synchronization means how low tokens per second we're producing. [15:27] Bad routing means high latency and if you have congestion that makes everything worse. [15:39] So network is pretty much binding everything together in this AI factory. [15:44] when it underperforms, the entire factory, [15:47] the AI factory underperforms and you're burning money on idle GPUs. [15:56] Now, let's look at the new balance sheet. The the KPIs have changed. We used to measure [16:04] infrastructure by uptime, ports, bandwidth, packets per second. AI has pretty much changed [16:11] all that. Today we are interested in let's say if you are a new cloud provider an enterprise [16:21] you're building a new cluster you want the first measure that you have KPI you have how quickly I [16:28] can onboard my infrastructure which is time to first token if it takes like three months or [16:37] six months for you to deploy and onboard a cluster that's losing money on on an asset that should be [16:46] producing as a GPU as a service provider should be uh running and producing um ROI for you. So what [16:58] are the KPIs for today? So how many tokens per second from the from the optimization efficiency [17:06] perspective tokens per watt how efficient we are utilizing the GPUs that's your GPU utilization so [17:16] infrastructure has moved from the IT budget into business P&L that's a profound shift [17:27] so if network isn't in the critical path of every GPU cycle. The next question is [17:32] for for which silicon which chip? Because if you look at the silicon landscape is already [17:40] complex and it's about to get more complex. So we have five workload shapes in the five years. [17:48] In 2023 when we started it was all about dense training. One shape pred predictable [17:55] synchronization mostly across your GPUs and CPUs. By 2025 uh we started to look at thee [18:06] training the disagregated inference and then we were introduced the TPUs LPUs the language [18:14] processing units and the DPUs and now we're talking about the persistent agents always [18:22] on so the bottom line is the specialized chips aren't slowing down they are actually multiplying [18:29] And each one brings a new communication pattern or different communication pattern and network [18:35] has to handle all of that. So this is the crux. Silicon will keep changing but the [18:41] network fabrics cannot be rearchitected every time a new silicon shows up. We need a unified [18:48] fabric that connects your GPUs, TPUs, LPUs, NPUs, DPUs or whatever comes next across your compute, [18:55] storage, memory domain. And that fabric needs to do four things. Number one, it needs to be open, [19:08] no lock in, tech agnostic, should be supporting multiple silicon by design, AI optimized, not [19:17] just repurposed from your traditional data center workloads or cloud networking. And the finally, [19:24] we also want to make it operationally boring. No forklick upgrades every time a new silicon [19:30] shows up. So that's what it means by we want to make it operationally boring. [19:37] So here's the upscales point of view. The silicon will continue to evolve. New GPUs [19:43] need new NPUs. We already established that. So we operate across two domains that matter [19:49] the most. The scale up and scale out. On the scaleup side, we're building purpose-built [19:54] interconnect silicon and systems for intra node and near node communication. On the scale out [20:01] we are building we're partnering with Nvidia to build out systems of AI optimized fabrics based [20:07] on Nvidia spectrum X Ethernet which is an open interoperable and designed for traffic patterns [20:14] we just talked about and the bottom line is these don't look at these at two two separate [20:21] products bolted together it's under one software substrate which is spanning across both domains. [20:30] So this is the system in context showing you a I'm here I'm showing you a picture of a 64 [20:39] port 800 gig system for scale out that's we are working with Nvidia we partnered [20:44] with Nvidia Spectum Ethernet so that combination gives you load balancing [20:49] congestion aware routing telemetry designed for your collective operations [20:57] So this is my final slide. Five core tenants for what we are building at Upscale AI. There's [21:04] one full stack silicon system, software, orchestration, support and services. We own [21:09] the whole column. Number two, purpose-built for a AI engineered from day one for AI workloads, [21:17] not repurposed from the enterprise gear. Number three, we talked about the heterogeneousity. [21:24] So heterogeneous compute across any XPUs, we don't choose the winners. Number four, [21:30] which is the important one, it's based on open and standard based Ethernet, [21:34] EAN and all the open standards. No lock in. Number f and number five, scale up and scale [21:40] out under one software substrate. So we have one platform across both domains. Thank you.