Red Hat Teams With Nvidia, Google on llm-d to Scale AI Inference

Red Hat 联手英伟达和谷歌推进 llm-d 以扩展 AI 推理 随着智能体工作负载推动 Token 需求增长 20 倍，单节点推理服务器逐渐失效，使跨混合硬件的 Kubernetes 原生解耦成为控制企业 AI 成本的关键。 Rob Greenberg 深入探讨了 LLM 推理的技术机制与经济学，阐明了在推理模型需要多达 20 倍 Token 的背景下，推理运营成本如今为何远超训练成本。他将 vLLM 视为支持与硬件解耦执行的标准化开源引擎，并结合了通过 llm-compressor 库实现的 FP8 量化技术。为克服单节点瓶颈，他介绍了 llm-d——这是一个与 Google、NVIDIA 和 Hugging Face 共同打造的开源 Kubernetes 倡议，实现了预填充与解码解耦以及具备 LLM 感知能力的请求路由。演讲最后概述了智能体调用链的可观测性，以及 Red Hat 针对生产环境部署的企业级封装方案。 Transformer 模型推理分为计算密集的 Prefill（填充 KV cache）与内存带宽受限的 Decode（自回归逐 token 生成）两个阶段；随着生成式 AI 规模化生产落地，推理已取代训练成为企业生命周期成本的主要开销。 (1:32) 推理模型与 Agent 智能体工作流显著加剧了算力压力，相比传统单次前向传递的 LLM，这类迭代任务每个请求的 token 消耗提升高达 20 倍，计算需求增加 150 倍。 (5:25) vLLM 已成为支持 NVIDIA、AMD、Intel 和 Google TPU 等多硬件供应商的事实开源推理标准；结合 llm-compressor 将模型从 BF16 量化至 FP8 可将显存占用减半，同时精度损失控制在 1% 以内。 (11:14) 传统 Web 负载均衡无法应对输入长度波动极大的 LLM 请求，Red Hat 联合 Google、NVIDIA 等开源了 Kubernetes 原生分布式推理项目 llm-d，通过智能路由与 Prefill/Decode 物理分离解耦实现多节点集群扩展。 (17:00)

Rob Greenberg of Red Hat AI argues that inference has overtaken training as the primary generative AI operational cost, necessitating vendor-neutral serving architectures like vLLM and distributed orchestration frameworks like llm-d to handle volatile enterprise workloads and agentic scaling. The Economic Shift Toward Production Inference (5:25) Greenberg emphasizes that enterprise generative AI spending has shifted from initial model training to continuous production inference. This change is driven by the growing scale of open-weight models—advancing from early architectures like LLaMA and Falcon to massive models like Kimi and Nemotron—which intensifies hardware accelerator shortages. Compounding this is enterprise demand for high token throughput, termed 'token maxing.' This compute demand increases substantially with reasoning models and agentic workflows. While a standard forward pass without reasoning might generate approximately 439 tokens, agents that pause to iterate, call tools, and evaluate decision trees can consume up to 20 times more tokens and 150 times more compute. Greenberg notes that despite the severe cost increase, organizations accept this computational burden because reasoning loops yield substantially more accurate results. Unified Engine Architecture and Hardware Portability (11:14) Transformer inference operates in two core mechanical stages: a compute-bound prefill stage that processes the input prompt into a key-value (KV) cache, and an autoregressive decode stage that generates subsequent tokens one by one using the cache. To prevent enterprises from maintaining fragmented serving stacks across varying architectures, Red Hat centers its inference platform on vLLM, an open-source engine originating from UC Berkeley's Sky Computing Lab. vLLM provides techniques like PagedAttention and continuous batching while abstracting hardware across NVIDIA, AMD ROCm, Intel, AWS Trainium, and Google TPUs. Greenberg highlights that Red Hat is the leading corporate contributor to vLLM, coordinating with model creators to provide day-zero runtime enablement whenever new open models are released. Memory Compression Through FP8 Quantization (14:59) To mitigate accelerator scarcity, Red Hat utilizes model optimization techniques like quantization via its open-source library, llm-compressor. Models released in 16-bit brain floating point format (BFloat16) are compressed into smaller numerical representations, such as FP8, effectively halving memory requirements while keeping accuracy degradation below 1% across Red Hat's validated Hugging Face releases. This footprint reduction lowers time per output token (TPOT) and improves overall cluster throughput. In response to audience questions, Greenberg clarifies that FP8 performance nuances depend on whether dynamic or block quantization is applied across weights versus activations, with specific tuning strategies applied to maximize speedups on particular hardware targets like NVIDIA GPUs. Distributed Workload Scheduling with llm-d (17:00) Traditional microservice load balancers fail in LLM serving because conventional HTTP requests are lightweight and uniform, whereas generative AI inputs exhibit radical variance in context length and computational cost—ranging from a single short question to an entire ingested Jupyter notebook. To manage this volatility, Red Hat co-developed llm-d alongside Google, NVIDIA, Hugging Face, and CoreWeave. Described as combining Kubernetes orchestration with vLLM execution, llm-d uses an LLM-aware router to schedule requests to appropriate compute pods based on their computational profile. The architecture natively implements disaggregated prefill and decode, separating compute-bound prompt ingestion from memory-bandwidth-bound token generation across distinct pod configurations to prevent cluster memory exhaustion. Failure Recovery and Enterprise Packaging (20:16) While baseline infrastructure metrics like Time to First Token (TTFT) and TPOT are monitored via Prometheus, Greenberg points out that autonomous agents require deeper observability. To address multi-step agent failures, the open-source community is building agentic tracing tools that capture tool parameters, track structured inputs and outputs, identify where an execution derailed, and facilitate recovery routing. Greenberg confirms that llm-d remains completely open source, fitting into two enterprise packaging options: the comprehensive Red Hat AI Enterprise bundle, which integrates OpenShift AI for training, tuning, and governance, and the dedicated Red Hat AI Inference SKU, designed specifically for organizations seeking only the vLLM and llm-d serving stack. While open serving stacks like vLLM and llm-d address hardware lock-in and multi-node throughput, operationalizing resilient production systems remains an ongoing industry effort, particularly around standardizing agentic failure recovery and fine-tuning quantization formats across diverse accelerator architectures.

Red Hat 人工智能首席产品经理 Rob Greenberg 认为，随着推理取代模型训练成为企业生命周期的主要支出，开源技术栈已成为降低服务成本与消除硬件绑定的核心支柱；通过结合 vLLM 推理引擎与基于 Kubernetes 的分布式编排框架 llm-d，企业能够在异构硬件集群上实现高吞吐、解耦化的生产级推理。 推理成本结构转变与智能体算力激增 (5:25) 自 2022 年底 ChatGPT 及开源大语言模型兴起以来，企业人工智能的成本重心发生了根本性逆转。Greenberg 指出，早年模型训练曾占据大部分预算，但如今推理作为“生产环境中的 AI”已占据全生命周期成本的绝大部分。由于 Transformer 架构采用自回归机制逐个生成 Token，计算流程被严格划分为处理输入上下文的 Prefill（预填充）阶段和依赖 KV 缓存逐步预测的 Decode（解码）阶段，上下文长度与并发量的增加对底层硬件加速器形成了持续的供给压力。 在企业追求 Token 吞吐最大化的背景下，推理模型与自主智能体（Agent）工作流进一步加剧了算力消耗。普通的单次前向传递可能仅消耗约 439 个生成 Token，但具备自主思考、多轮工具调用、自纠错与分支决策的智能体系统，其消耗的 Token 数量可达普通模型的 20 倍，计算需求更激增高达 150 倍。尽管这种反复迭代的推理循环显著提升了最终输出的准确度与可靠性，但也给企业带来了严峻的算力成本与硬件短缺挑战。 vLLM 标准化引擎与 FP8 模型量化 (11:14) 面对快速迭代的开源模型架构（如 Qwen、Mistral、Nemotron 等），企业无法为每类模型单独维护互不兼容的推理服务，亟需在“任意模型、任意服务器、任意硬件”间建立统一的标准服务层。Red Hat 将其企业级生成式 AI 平台锚定在开源推理引擎 vLLM 上，借助其 PagedAttention 与连续批处理（Continuous Batching）技术消除厂商锁定，全面适配 NVIDIA、AMD、Intel 及 Google TPU 等多款加速芯片，并在新模型发布首日提供零延迟兼容支持。 为了在单节点层面压榨硬件性能，Red Hat 通过开源库 llm-compressor 对模型实施量化压缩。针对主流的 16 位浮点（BF16）格式，Red Hat 验证并发布了 FP8 格式的模型权重。Greenberg 表示，将模型压缩至 FP8 能够直接降低一半显存占用，在所有 Red Hat AI 发布的模型中将准确率损失控制在 1% 以内，并大幅缩短每输出 Token 所需时间（TPOT），从而成倍提升单机吞吐上限。 llm-d 分布式架构与解耦调度机制 (17:00) 单节点推理无法支撑高并发的大规模生产场景，而传统针对 HTTP 统一请求的微服务负载均衡机制在大模型场景下会彻底失效。Greenberg 说明，传统微服务的网络请求相对轻量且计算负载均衡，但大模型请求的提示词长度与计算开销极度分化——从单句简单查询到长达数万字的完整 Jupyter 笔记本代码分析，计算强度差异极大。若沿用轮询或最少连接等传统算法，极易导致局部节点显存耗尽和集群崩溃。 为此，Red Hat 联合 Google、NVIDIA、Hugging Face 和 CoreWeave 等产业伙伴共同发起了完全开源的 llm-d 项目。该项目将 Kubernetes 的容器编排与 vLLM 引擎结合，配备感知大模型负载特征的智能路由调度器，并原生支持 Prefill 与 Decode 阶段解耦（Disaggregated Prefill and Decode）。该架构将计算密集型的预填充任务与显存带宽受限的解码任务分配至不同 Pod 执行，显著提升了集群吞吐效率并保障了响应延迟目标。 智能体追踪可观测性与企业交付路径 (20:16) 在复杂智能体多步执行的容错场景中，基础架构面临着新的运维挑战。针对智能体中间步骤崩溃或执行偏差问题，Greenberg 指出，除 Prometheus 导出的首字生成时间（TTFT）、每 Token 生成时间（TPOT）等基础吞吐指标外，开源社区正重点推进“智能体追踪”（Agentic Traces）能力建设。该技术专门记录多轮工具调用的具体参数与输入输出数据，协助系统在执行故障后定位断点，进而规划恢复推理并将任务重定向至合适计算节点。 在商业落地方面，Greenberg 阐明了开源组件向商业化产品线（SKU）的转化结构。针对需要覆盖模型训练、微调、治理与分布式推理全生命周期的客户，Red Hat 提供基于 OpenShift 的 Red Hat AI Enterprise 套件；而对于仅需部署生产推理的企业，则提供轻量级且成本更低的 Red Hat AI Inference 独立产品线，仅打包预优化模型库、vLLM 与 llm-d 调度器以满足特定推理需求。 Greenberg 总结表明，开放协作是支撑企业级 AI 推理规模化落地的关键路径；然而，随着智能体长链路调用的普及，如何在复杂断点状态下实现健壮的自动恢复与动态资源调度，仍是 vLLM 与 llm-d 社区当前持续探索与基准测试的核心课题。

## Background 该视频由 Ai4 频道于 2026 年 8 月 19 日发布在 YouTube 平台上。内容记录了 Red Hat 首席产品经理 Rob Greenberg 的主题演讲，主要探讨大语言模型（LLM）在生产环境中的推理挑战，并系统介绍如何结合开源推理引擎 vLLM 与 Kubernetes 原生分布式平台 llm-d 实现企业级的大规模高效部署。 ## TL;DR 演讲指出随着 Agentic 思考与工具调用普及，推理阶段的算力与 Token 消耗呈爆发式增长，推理已成为企业 AI 生命周期的核心成本重心。Red Hat 倡导全栈开源，依托事实标准的 vLLM 推理引擎实现跨 NVIDIA、AMD、Intel、TPU 等异构硬件的统一抽象，并结合 FP8 量化大幅降低延迟与显存开销。面对单节点瓶颈，开源分布式项目 llm-d 结合 Kubernetes 提供了 LLM 感知智能路由以及 Prefill 与 Decode 的阶段解耦，有效化解了长短负载不均的流量调度难题。整套推理方案已整合进 OpenShift AI 商业套件，为企业提供从全生命周期到纯推理部署的高可用解决方案。 ## Atmosphere & style 演讲整体呈现出务实、严谨且充满技术洞见的工程布道风格，讲者在底层矩阵计算原理与顶层云原生架构设计之间深入浅出地自如切换。通过现场问答与即兴调研，讲者以轻松诙谐的方式直观揭示了行业从“重训练”向“重推理”的认知剧变，极大地调动了现场听众的共鸣。问答环节针对 Agent 容错与 FP8 硬件调优的讨论直接切中生产痛点，交流氛围浓厚且极具开源极客色彩。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=4s) Red Hat 倡导全栈开源的 AI 战略，涵盖模型、推理引擎、基础设施与部署架构，并以开源生态作为推动生成式 AI 发展的核心基石。 - [01:32](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=92s) LLM 的自回归生成机制在架构上分为 Prefill（并行写入 KV Cache）和 Decode（自回归单步生成）两个阶段，KV Cache 是避免重复计算并实现长文本生成的关键技术突破。 - [05:23](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=323s) 随着 Agent 智能体和深度思考链（CoT）工作流的普及，推理阶段的 Token 消耗量暴增 20 倍，算力需求提升 150 倍，使推理成为模型生命周期中成本和算力消耗的核心来源。 - [08:10](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=490s) 面对多模型共存的业务现实，企业亟需解耦硬件与模型的统一标准化推理服务栈，避免为每个新模型维护专属推理架构所带来的高昂运维开销。 - [11:12](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=672s) vLLM 已经成为行业事实上的开源推理引擎标准，支持跨 NVIDIA、AMD、Intel、AWS Trainium 及 Google TPU 等异构算力部署，并提供首日即用的 Day-0 模型兼容能力。 - [14:59](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=899s) 通过 LLM Compressor 工具对模型进行 FP8 动态量化，可将模型体积减半并显著降低单 Token 延迟，同时在主流基准测试中将精度损失严格控制在 1% 以内。 - [16:44](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=1004s) 由 Red Hat 联合 Google、NVIDIA 等共同发起的开源项目 llm-d，将 Kubernetes 的调度能力与 vLLM 深度结合，提供 LLM 感知路由和 Prefill 与 Decode 解耦架构，有效突破单机推理瓶颈。 - [20:02](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=1202s) 针对 Agent 智能体及工具调用（Tool Calling）场景，推理栈正进一步整合 Prometheus 原生可观测性指标，并探索多步推理的状态追踪、容错重试及断点续传机制。 - [22:51](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=1371s) Red Hat 通过 OpenShift AI 将整套大规模推理架构产品化，并提供覆盖全生命周期的 Red Hat AI Enterprise 以及专注高并发生产推理的 Red Hat AI Inference 两种商业化订阅方案。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=4s) 开场介绍：Red Hat 的开源 AI 愿景与 vLLM Red Hat AI 的首席产品经理 Rob Greenberg 向与会观众作开场介绍。他回顾了自己约一年半前加入 Red Hat 的经历——当时他所在的 Neural Magic 专注于大语言模型（LLM）的压缩与优化，起初主要针对 CPU 上的 NLP 和计算机视觉模型进行压缩，而在 ChatGPT 问世后团队迅速完成业务转型。Greenberg 明确了本场 20 分钟演讲的核心目标：梳理从推理基本原理到 vLLM 与 llm-d 在生产环境中的部署实践，并介绍 Red Hat 的开源推理技术栈。 Red Hat 坚信“AI 的未来是开放的”（The future of AI is open），这种开放性必须贯穿整个技术栈：开放模型（Open Models）、开源推理服务引擎（Open Inference Servers）、开放基础设施（Open Infrastructure）以及开放部署架构（Open Serving）。Red Hat 坚信开源生态是推动 AI 持续向前发展的核心动力，并通过主导和支持开源推理引擎 vLLM，践行将开源 LLM 普惠全球的使命。 - **演讲者背景**：Rob Greenberg，Red Hat AI 首席产品经理，前 Neural Magic 团队成员。 - **核心主张**：全栈开源（Open Stack），覆盖模型、推理引擎、调度与底层硬件适配。 - **关键引擎**：vLLM 作为事实上的开源推理服务标准引擎，是 Red Hat 推理战略的核心支柱。 --- ### [01:32](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=92s) 大语言模型推理的基本机制：自回归、KV Cache 与两阶段拆分 从底层逻辑来看，推理（Inference）即使用已训练完成的模型根据输入的 Prompt 生成文本的过程。在硬件层面上，输入提示词进入部署在 API 端点、本地 GPU 或笔记本上的推理服务器，模型进行矩阵运算后返回响应。本质上，基于 Transformer 架构的 LLM 并非传统的线性回归或随机森林等预测型 AI，而是一个通过大量线性代数运算来预测序列中下一个 Token 的“黑盒函数”（Black Box）。 LLM 文本生成的核心机制是**自回归（Auto-regressive）**，即模型必须单步逐个预测下一个 Token，再将它们依次拼接为人类可读的连续句子。为了避免生成每个新 Token 时重复处理所有历史 Token 所带来的巨大算力浪费，**KV Cache（键值缓存）** 应运而生。它是大模型推理中最重要的突破之一。 整个 LLM 推理流程在架构上被清晰地拆分为两个截然不同的阶段： 1. **Prefill（预填充阶段/Prompt 阶段）**：一次性接收并并行计算输入 Prompt 的所有 Token，并将其 Key/Value 状态写入 KV Cache。 2. **Decode（解码阶段/生成阶段）**：进入自回归循环，结合已有的 KV Cache，每次只迭代计算并生成一个新 Token，直到命中终止符。 ```mermaid flowchart LR accTitle: LLM Inference Lifecycle with KV Cache subgraph S1["1. Prefill 阶段"] A["输入 Prompt"] --> B["并行计算上下文"] B --> C["写入 KV Cache"] end subgraph S2["2. Decode 阶段"] C --> D["单步自回归迭代"] D --> E["预测下一个 Token"] E --> F{"是否结束？"} F -- 否 --> D F -- 是 --> G["生成最终文本"] end H["调度器 (Scheduler) / Kubernetes Pods"] -.-> S1 H -.-> S2 ``` - **黑盒本质**：Transformer 依赖庞大的数据与参数堆叠，核心运算目标高度聚焦于“预测序列中的下一个 Token”。 - **KV Cache 价值**：消除历史上下文的重复计算，使长时间对话与长文本生成在算力上可行。 - **调度器（Scheduler）角色**：在 Kubernetes 等容器化架构中充当“流量交警”，智能管控 Token 与请求的流向，是大规模扩展的核心前提。 --- ### [05:23](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=323s) 生产落地挑战：算力瓶颈、Token 需求爆发与 Agent 推理扩展 推理是生成式 AI 在生产环境中的生命线，也是系统真正产生成本、面对延迟和吞吐考验的地方。当前企业面临着严峻的现实制约：模型参数规模持续攀升（从早期的 LLaMA、MPT、Falcon，演进至如今架构更庞大的顶尖开源模型），专用硬件加速器（GPU）供应极度紧张。与此相伴的是企业对“Token 极致利用（Token Maxing）”的狂热追求——管理层普遍期望借助 AI 生成更多代码、更多产出，推动业务极速创新。 更关键的技术转折点在于**智能体与推理模型（Agentic Reasoning）**的崛起。传统 LLM 往往只需单次前向传播（例如输入一个 Prompt 生成 439 个 Token）；而具备思考链（CoT）与工具调用能力的 Agent 模型（如 Claude 思考模式、DeepSeek-R1、OpenAI o1 等）会进行多轮自我校验、分支路径探索与循环调用。这种推理扩展机制带来了显著的代价与回报： - **Token 消耗量**：相比传统单次调用暴增高达 **20 倍**。 - **算力需求**：推理阶段消耗的算力可激增达 **150 倍**。 - **产出收益**：换取了最终答案准确率与复杂任务解决能力的质的飞跃。 --- ### [08:10](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=490s) 企业级推理诉求：多模型生态与统一推理服务栈的价值 回顾开源模型的演进历程，自 2022 年底 ChatGPT 爆发后，Meta、Mistral、阿里巴巴（Qwen）、NVIDIA 等机构持续开源了一系列强大的权重模型。如今开源社区每周都会涌现刷新排行榜的顶尖模型，在逻辑推理和 Agent 工作流中已能与顶级闭源提供商（如 OpenAI、Anthropic）正面抗衡。 面对开源模型的爆发式增长，企业不可能依赖“单模型通吃”的策略，而是必然采用多模型协同体系。这就对底层的推理服务栈提出了严苛要求： | 核心维度 | 自建/碎片化推理栈的痛点 | 现代化统一推理栈 (vLLM / llm-d) 的优势 | |:--- |:--- |:--- | | **硬件与模型解耦** | 为每种模型或专用硬件开发适配逻辑，维护负担极重 | 实现 **Any Model, Any Server, Any Hardware** 的全通用抽象 | | **扩展性 (Scalability)** | 单节点吞吐受限，难以应对超大并发与长文本 | 原生支持分布式推理、智能请求路由与多模型协同并发 | | **稳定性与维护成本** | 团队无法针对企业内 20 种不同的推理服务逐一维护和排错 | 标准化推理服务端，降低企业拥抱新开源模型的集成与运维门槛 | 正如 Greenberg 所强调的，企业无法承受每引入一个新模型就要重新学习一套全新推理架构的代价；建立跨硬件、跨云环境的标准化开源推理层，是企业将 AI 真正规模化落地的必由之路。 ### [09:58](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=598s) 成本重心转移与红帽的企业级 GenAI 推理架构 在现场进行的一个快速举手问卷中，讲者向观众抛出了一个经典问题：从长远来看，模型的训练（Training）和推理（Inference）究竟哪个成本更高？台下观众几乎一致选择了推理。讲者笑着指出，如果倒退五六年前问同样的问题，答案会完全不同；这种认知的颠倒清晰地展示了整个 AI 生命周期的演进速度——随着模型走向实际生产应用，推理已经成为成本和资源占用的绝对大头。 红帽（Red Hat）致力于提供一套完整的企业级 GenAI 推理平台。虽然红帽在模型训练、微调（Fine-tuning）和可观测性（Observability）等全流程均有布局，但本次分享的核心重点在于推理层。对于企业而言，底层存在各种异构算力，上层有层出不穷的新模型与新架构，而推理服务层正是连接所有新模型能力与底层物理硬件的关键枢纽。 ```mermaid flowchart TD accTitle: LLM Inference Ecosystem accDescription: Architectural connection from models through the inference engine to heterogeneous hardware. subgraph Models["上层模型生态 (Models)"] M1["开源/专有模型<br/>(Gemma, Nemotron, Kimi 等)"] M2["量化/压缩模型<br/>(FP8, Speculative Decoding)"] end subgraph Serving["核心推理层 (Inference Layer)"] VLLM["vLLM 推理引擎<br/>(性能优化 / 统一抽象)"] LLMD["llm-d 分布式调度<br/>(K8s 原生 / 智能路由 / 解耦 Prefill-Decode)"] end subgraph Hardware["底层异构算力 (Hardware)"] H1["NVIDIA / AMD / Intel GPU"] H2["Google TPU / AWS Trainium / IBM"] end Models --> Serving Serving --> Hardware ``` --- ### [11:12](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=672s) vLLM：连接任意模型与异构硬件的标准引擎 在整个推理平台的中枢，正是扮演核心角色的 vLLM。如果市场上每一个模型都需要自己专属的推理服务器，那么支撑各种模型和架构所带来的运维管理开销将变得不可承受。vLLM 仅用了大约两年时间，就从加州大学伯克利分校 Sky Computing Lab 的一个学术研究项目，跃升为整个行业的事实标准，在 GitHub 标星数和实际使用量上均处于领先地位。vLLM 不仅解决了模型落地的兼容性问题，更在性能优化层面带来了质的飞跃。 vLLM 实现了"任意模型运行在任意硬件上"（Any Model to Any Hardware）的愿景，打破了单一硬件厂商的锁定。它不仅深度适配 NVIDIA 架构，还全面支持 AMD、Intel、AWS Trainium 以及 Google TPU。红帽拥有全职的推理工程团队，与 Gemma、Nemotron 以及 Kimi 等前沿大模型提供方直接合作，实现所谓的"Day-0 兼容支持"（Day-zero Enablement）。这意味着开源社区发布新模型的第一天，用户就能直接在 vLLM 上将其跑起来并用于构建实际应用。红帽是 vLLM 社区的第一大贡献者（Number One Contributor），并定期举办双周 vLLM Office Hours 进行技术交流与优化探讨。 - **多硬件解耦**：企业可自由选用 NVIDIA、AMD、Intel、IBM 或公有云定制芯片（如 AWS Trainium、Google TPU），底层无需重写推理逻辑。 - **混合部署自由**：支持公有云与本地私有化（On-prem）部署，特别契合金融、电信等强监管行业对数据控制、成本与延迟的严苛要求。 - **生态驱动力**：作为去年增长最快的开源 AI 项目之一，vLLM 正在驱动新一轮开源 AI 创新的爆发。 --- ### [14:59](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=899s) 模型层压缩优化：FP8 量化与精度无损压缩 除了推理引擎本身的算力调度，模型本身的轻量化也是降本增效的核心抓手。红帽在 Hugging Face 上维护了公开的 Red Hat AI 仓库，团队不仅专注于 vLLM 引擎工程，也是模型优化与压缩领域的专家。通过量化（Quantization）与投机采样（Speculative Decoding）等技术，模型可以在各类硬件上以更高的吞吐量和更低的延迟运行。 传统模型从实验室发布时通常采用 BF16（BFloat16）格式，权重以全精度表示。量化本质上是一种模型压缩技术，讲者打了一个生动的比方：就像把原本精确记录的圆周率 $\pi$（3.141592...）直接简化表示为 3。借助红帽开源的 `LLM Compressor` 工具库，团队能够对模型进行 FP8 等低精度量化。FP8 格式的模型体积仅有 BF16 的一半，但红帽发布的每一个 FP8 模型在主流基准测试中的精度损失均被严格控制在 1% 以内，以极微小的精度代价换取了单 Token 输出时间（Time Per Output Token）的显著缩短和硬件利用效率的大幅提升。 --- ### [16:44](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=1004s) llm-d：Kubernetes 与 vLLM 结合的分布式推理平台 当面对更大规模的模型以及极高的并发请求时，单节点推理引擎逐渐暴露出瓶颈，这正是分布式推理项目 `llm-d` 应运而生的背景。讲者将 llm-d 形容为"Kubernetes 的全部优势与 vLLM 的全部优势的完美联姻"（A marriage between all the benefits of Kubernetes and all the benefits of vLLM）。llm-d 是红帽与 Google、NVIDIA、Hugging Face、CoreWeave 等生态伙伴共同发起的开源项目，旨在突破单机推理的极限。 传统微服务架构中基于统一 HTTP 请求的负载均衡在 LLM 场景下彻底失效。传统的 Web 请求成本低廉、尺寸均匀、QPS 极高；而 LLM 请求的计算负载差异极其悬殊——可能一个实习生只发来一句话问"法国的首都在哪里？"，而一位资深工程师则直接粘贴进整篇几十页的 Jupyter Notebook 代码。针对输入负载高度不均的特性，llm-d 引入了具备 LLM 感知能力的智能路由器（LLM-aware Router），能够根据输入特征将请求精准调度至最合适的 Pod，并原生支持 Prefill（预填充）与 Decode（解码）的解耦分离（Disaggregated Prefill and Decode），大幅提升并发承载能力并满足严苛的 SLO。 ```mermaid flowchart LR accTitle: llm-d Architecture and Intelligent Routing accDescription: LLM-aware router dispatches heterogeneous requests into specialized pods or disaggregated stages. subgraph Ingress["客户端请求接入"] R1["短文本请求<br/>(例如: 简单问答)"] R2["超长文本请求<br/>(例如: 完整 Notebook)"] end subgraph Router["llm-d 分布式调度层"] LMR["LLM 感知智能路由器<br/>(LLM-Aware Router)"] end subgraph Execution["解耦与异构执行集群"] P_Prefill["Prefill 计算密集节点<br/>(加速处理长 Prompt)"] P_Decode["Decode 显存带宽密集节点<br/>(自回归生成 Token)"] end R1 --> LMR R2 --> LMR LMR -->|"按负载/阶段拆分"| P_Prefill P_Prefill -.->|"传递 KV Cache"| P_Decode ``` --- ### [20:02](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=1202s) 现场互动：智能体容错机制与可观测性度量 在演讲最后的问答环节，现场观众提问关注分布式环境下 Agent（智能体）工作流的容错与恢复能力：当 Agent 在执行多步推理任务中途失败或走偏时，负载均衡器和推理集群如何处理容错，是否支持从失败状态回滚或恢复。 讲者对此作出了积极回应： - **原生可观测性指标**：vLLM 开箱即集成了基于 Prometheus 的遥测与指标采集系统，能够实时监控诸如首字延迟（Time to First Token, TTFT）、每个输出 Token 耗时（Time Per Output Token, TPOT）、整体延迟和吞吐量等核心指标。 - **Agent 追踪与容错探索**：针对多轮交互与 Agent 状态的失败重试与轨迹回放（Agentic Traces），团队目前正在进一步深入探索与研发，以帮助开发者在复杂工作流发生异常时实现更智能的断点续传与调度容错。 ### [20:58](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=1258s) Agent 与工具调用（Tool Calling）的推理与路由优化 随着大语言模型逐步拓展到工具调用（Tool Calling）和智能体（Agent）工作流，推理系统的核心诉求发生了显著变化。在复杂的 Agent 交互中，开发者需要清晰洞察模型究竟调用了哪项工具、输入与输出参数是什么、执行在哪个环节出现了偏差，以及系统如何更高效地对推理路径进行逻辑推导与错误定位。当涉及实际的计算工作负载时，推理架构必须确保每一个解析任务（Resolution）都能被精准路由到最合适、最高效的计算节点上。 针对这些前沿挑战，Red Hat 正依托 llm-d 和 vLLM 开展大量活跃的研究与落地工作。团队一方面致力于保障主流前沿模型在工具调用场景下具备高效的执行力，另一方面积极与开源社区紧密协作，持续投入并完善针对 Agent 与工具调用能力的标准化评测体系，以便向行业公开发布更多可量化的基准测试数据。 ```mermaid flowchart TD accTitle: Tool Calling and Routing Flow subgraph AGENT ["Agent 推理与工具调用"] direction TB A["用户请求 / 任务输入"] --> B["LLM 推理与决策"] B --> C{"是否需要工具调用?"} C -- 是 --> D["调用外部 Tool / API"] D --> E["捕获输入/输出与可观测性追踪"] E --> B C -- 否 --> F["生成最终响应"] end subgraph ROUTING ["llm-d & vLLM 智能调度层"] direction TB G["请求分析与上下文评估"] --> H["KV Cache 感知路由"] H --> I["分发至最佳计算节点 (GPU/TPU)"] end B -. 动态分发.-> G ``` --- ### [21:40](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=1300s) FP8 模型量化差异与 Red Hat 模型版本解析 在现场问答环节中，一位与会者提出了具体的技术疑问：用户在 Hugging Face 上既看到了 Qwen 官方发布的 `Qwen-36-27B-FP8` 模型，也看到了 Red Hat 发布的同规格版本，两者之间到底存在什么技术差异，使用 Red Hat 维护的版本是否有额外优势？ 讲师对此进行了详细澄清：Red Hat 所采用的 vLLM 内核与开源社区上游（Upstream）保持完全同步，而在模型权重方面，如果原始模型提供方已经发布了标准的 FP8 动态量化版本，Red Hat 通常不会重复制作。FP8 量化主要细分为两种常见实现方式：**FP8 Block 量化**与 **FP8 Dynamic 量化**。两者的主要区别在于针对底层硬件架构所做的微调与取舍： - **权重与激活值处理**：Red Hat 在制作特定模型的 FP8 量化版本时，可能会针对模型权重（Weights）或激活值（Activations）采用针对性的量化策略，以期在 NVIDIA 等特定硬件加速卡上榨取更高的执行速度与吞吐量。 - **性能等价性**：如果原始厂商发布的模型与 Red Hat 发布的模型均基于完全相同的 FP8 Dynamic 方案且未做特殊硬件调优，那么两者在 vLLM 引擎上的运行性能是完全一致的。 --- ### [22:51](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=1371s) llm-d 开源属性与 Red Hat AI 产品商业化布局（OpenShift AI） 针对现场关于开源许可与企业级产品集成的提问，讲师明确指出 **llm-d 是一项完全彻底的开源项目**。随后，讲师梳理了演讲中介绍的各项技术组件在 Red Hat 整体企业级 AI 产品矩阵中的落地形式： 演讲中展示的整套大规模推理架构目前已全面整合进 OpenShift AI 体系，并以 **Red Hat AI Enterprise** 这一商业套件形式对外交付。该套件打包了完整的端到端 AI 生命周期能力，为企业提供从模型训练、微调（Tuning）、可观测性监控到底层高性能分布式推理（包含 vLLM 与 llm-d）的一站式支持。 为了满足不同企业的差异化诉求，Red Hat 提供了两种灵活的产品形态（SKU）： | 产品方案 (SKU) | 包含的核心组件 | 适用场景与定位 | |:--- |:--- |:--- | | **Red Hat AI Enterprise** | OpenShift AI + OpenShift Container Platform (OCP) + AI 加速器授权 | 覆盖模型全生命周期的完整平台，涵盖训练、微调、全链路可观测性以及完整推理技术栈。 | | **Red Hat AI Inference** | 经优化与量化的官方模型目录 + vLLM 引擎 + llm-d 分布式扩展层及核心推理组件 | 专为生产级纯推理场景设计的轻量级订阅方案，成本更低、性价比更高，适合专注于模型高并发部署的企业。 | 演讲最后，讲师向全场观众致谢，并邀请感兴趣的开发者移步展台（Booth）进行更深入的一对一技术交流。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=4s) The future of AI is open — Rob Greenberg > Training gets the headlines, but inference is where generative AI meets production — and where the costs, latency, and scaling challenges actually live. — Rob Greenberg > [08:10](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=490s) Any Model, Any Server, Any Hardware — Rob Greenberg > [11:12](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=672s) Any Model to Any Hardware — Rob Greenberg > [16:44](https://www.youtube.com/watch?v=cDTxNxzDIvM&t=1004s) A marriage between all the benefits of Kubernetes and all the benefits of vLLM — Rob Greenberg ## Mentioned Rob Greenberg, Red Hat, Neural Magic, vLLM, llm-d, Kubernetes, Google, NVIDIA, Hugging Face, AMD, Intel, Google TPU, AWS Trainium, IBM, CoreWeave, Meta, Mistral, Alibaba, Qwen, Gemma, Nemotron, Kimi, Claude, DeepSeek-R1, OpenAI o1, LLM Compressor, Prometheus, OpenShift AI, Red Hat AI Enterprise, Red Hat AI Inference ## Video description Rob Greenberg - Principal Product Manager, Red Hat Training gets the headlines, but inference is where generative AI meets production — and where the costs, latency, and scaling challenges actually live. This session demystifies LLM inference from the ground up: how models generate tokens, why the KV cache and the prefill/decode split matter, and what an inference server actually does to make GPUs go "brrrr." From there we turn to the hard part — running inference at scale. As reasoning and agentic workloads drive token demand 20x higher (and compute 150x higher), single-node serving stops being enough. We'll show how Red Hat approaches distributed inference with llm-d, a Kubernetes-native, open-source initiative built jointly with Google, NVIDIA, Hugging Face, and others. You'll see how disaggregated prefill and decode, intelligent request scheduling, KV-cache-aware routing, and hardware-agnostic serving across NVIDIA, AMD, Intel, and Google TPUs come together to maximize GPU utilization and hit demanding SLOs — across edge, private, and public cloud. We'll connect the dots from vLLM as the de facto inference engine to llm-d as the scaling layer, rounding out the picture with validated and quantized models that stretch more tokens out of fixed hardware. Attendees will leave understanding not just what inference is, but how to deploy it efficiently and reliably at enterprise scale. [Source](https://www.youtube.com/watch?v=cDTxNxzDIvM)

[0:04] Rob Greenberg: The future of AI is open Rob Greenberg: Training gets the headlines, but inference is where generative AI meets production — and where the costs, latency, and scaling challenges actually live. [8:10] Rob Greenberg: Any Model, Any Server, Any Hardware [11:12] Rob Greenberg: Any Model to Any Hardware [16:44] Rob Greenberg: A marriage between all the benefits of Kubernetes and all the benefits of vLLM

[0:04] Rob Greenberg: the future of AI is open [1:32] Rob Greenberg: predict the next token in a sequence [1:32] Rob Greenberg: Where is the capital of Victoria, Australia? [5:25] Rob Greenberg: AI in production [5:25] Rob Greenberg: token maxing [8:09] Rob Greenberg: any model, any server, any hardware [11:14] Rob Greenberg: Day Zero enablement [17:00] Rob Greenberg: marriage between all the benefits of Kubernetes and all the benefits of vLLM kind of married into one project [17:00] Rob Greenberg: What's the capital of France? [20:16] Rob Greenberg: agentic traces [21:39] Rob Greenberg: llm-d is 100% open source

[0:04] Well, good afternoon, um, everybody. Um,  so yeah, my name is Rob Greenberg. I'm a [0:08] principal product manager at Red Hat AI. Um,  I joined Red Hat about a year and a half ago, [0:14] um, from a company called Neural Magic, where  we did, um, LLM compression and optimization. [0:20] Um, we started as a company that compressed models  for CPUs, so think NLP and computer vision models, [0:25] and then um, chat GPT came about, and  we really pivoted. Um, and now, yeah, [0:30] I'm here to talk to you all today about vLLM  and LMD, um, and inference, uh, in production. [0:36] So, an intro So, this this talk will be kind of,  you know, we have 20 minutes, so it'll be pretty [0:39] quick, but I wanted to cover a few different, um,  topics to just set the baseline with everyone. Uh, [0:44] so the first one will be inference fundamentals,  so understanding, you know, what inference is, [0:48] why it's important, um, why inference matters, and  then we'll very briefly at the end touch upon Red [0:53] Hat's AI stack, um, and our inference stack. So,  at Red Hat, we believe the future of AI is open, [1:00] right? That means the entire stack, open models,  open inference servers, open infrastructure, [1:06] um, open serving, right? We want, you know, at  Red Hat, we really believe that the open source [1:10] ecosystem is what's going to drive AI forward,  um, and we're big proponents of that message. [1:15] Um, we are also in a, um, our mission is to bring  open source LLMs, um, to the world via vLLM. So, [1:23] vLLM is an open source inference server, um, that  we're really, really excited about, and we've been [1:27] working on for many years, and we'll kind of get  into, uh, talking a little bit more about that. [1:32] Okay, so I'm sure we have different levels  of understanding of inference, so very, [1:36] very basically, um, inference is kind of this  this diagram sums it up really nicely, right? [1:41] Inference is the process of using a trained model,  uh, to generate text in response to a prompt, [1:46] right? So, prompt goes in, everyone here has used  chat GPT, we've all probably used Claude as well, [1:51] um, but really the process of inference is prompt  goes in, it gets processed, um, by a model, right? [1:57] Um, which is served somewhere. You have  a model, you have an inference server, [2:00] you have hardware, whether it's any an API  endpoint or you're doing yourself on your own [2:04] GPU or on your own laptop. We can kind of get into  those details. And then the model will then go in [2:09] and kind of give you a response back. So that's  what inference is at a high level. What is a [2:13] model, right? The next question is, okay, I  understand what inference is. Prompts go in, [2:17] responses go out. How does a model work? All  a model really is, it's a function to predict [2:22] the next token in a sequence, right? I mean,  I say model, I mean LLM, large language model, [2:27] generally based off the transformers architecture.  But really all it is, right, is words are going [2:32] in, it's a bunch of linear algebra, really  to just predict the next token in a sequence. [2:39] Here's just a little meme, right? So this  is your machine learning and it's the idea [2:44] that data goes in and it's just a pile. You  don't really know what's going in and out. [2:48] And this just kind of highlights, yeah,  just keep doing it again until, you know, [2:52] it's doing the right things. And this really  just kind of goes to show that like these models [2:56] are traditionally known as black boxes, right?  These aren't predictive AI models like linear [3:01] regression models or random forest models. These  are black box transformer models. So what goes in, [3:07] we don't know what's going to come out. All we  know is the model's predicting the next token. [3:11] In terms of generating text with the model itself,  right? The model functions The model functions [3:16] auto-aggressively. So what auto-aggressively means  is basically one at a time the next token will be [3:22] generated. And what you see on the right is a  visual representation of what's called the KB [3:27] cache, right? A lot of us know what caching is  because of the internet, because of browsers, [3:31] right? You have cache, you have cookies.  It's the same kind of idea. So as you go in [3:36] and you prompt the model and get a response back,  the model's then predicting sequential tokens. [3:41] Right? And And these then get chained together  and these form human-readable sentences. And [3:46] this is how we understand how text is how models  generate text. Um, talked about this a second ago, [3:52] but key optimization is caching, right? So,  pro- uh processing all previous tokens is really [3:58] expensive, right? You can imagine if you have to  keep loading in the context for a short sentence, [4:03] um it just keeps getting longer and you're  throwing away all the previous computation. So, [4:08] caching was really a um a really big breakthrough  in terms of how these LLMs actually generate text. [4:14] Um, this is just a really high-level  view of the architecture, right? So, [4:18] there are two stages of inference as well that  we can talk about. So, there's the prefill stage, [4:22] which is that prompt stage, right? So, a  prompt goes in, goes into the inference server, [4:27] um that then gets uh cached, right? So,  where is the capital of Victoria, Australia? [4:31] Um, and then you can see um the decode stage,  decode you can think of as the generation stage, [4:38] where the model's actually generating text  back to you. Um, this is the auto-regressive um [4:43] nature of multiple iterations predicting the next  token. With caching in play, this is how you get [4:48] uh better performance. Right? And none of this  is possible without what's called the scheduler, [4:53] and the scheduler will go in and basically like  controlling the traffic, um right? Of like what [4:57] tokens are being sent where, um as you start  thinking about like a Kubernetes architecture [5:02] with different pods, um and kind of how to  actually do this thing at scale. So, we we kind of [5:07] covered a basic overview of of what inference is,  right? It's the process of feeding in a prompt, [5:13] um and then getting a uh generated token, um  which is then read as text by us humans back, um [5:18] and these two different stages of prefill, right?  Which is that prompt stage, and then decode. Um, [5:23] so, we we talked a little bit about that. Why  does inference really matter though, right? [5:26] Under the hood. Um, and really inference is AI in  production. Right? At its at its absolute core, [5:33] this is what is powering the applications.  This is what makes AI feel alive. Um, [5:38] this is what's actually running when you get that  prediction, right? And then this has been the way [5:43] that AI's worked since it was, you know, started  many, many years ago, um over a decade ago, [5:49] uh even with predictive AI. Um but inference is  really the the real-live action of AI happening. [5:55] Now, while inference is super important, right? Um  it's also really expensive. Um so, models are only [6:01] getting bigger. Um I'm sure all of you have, you  know, many of you have been keeping up with what's [6:05] going on in the open source um with models, you  know, we started out with Llama models, MPT model, [6:11] Falcon. These are models from 2022, 2023 uh time  frame. And now, you know, Kimmy K3, NeMo-1 Ultra, [6:19] a lot of these models are just keep getting bigger  and they're advancing in capability, but this [6:23] creates constraints, right? And and cost is one  of them. Um specialized hardware accelerators, I'm [6:28] sure everyone in this room has felt the pain of  trying to get a GPU, whether it's on the cloud or, [6:34] you know, at your company, to figure out how do we  actually take advantage of the latest and greatest [6:39] models? And we can talk a little bit about some  different methods on how to do that in just a [6:43] little bit. Um but then the demand for tokens,  right? Everyone's heard about this concept of [6:47] token maxing. Um every company wants to you move  fast. Every company wants to innovate with AI. [6:53] Um and to do that, right? It's every every CEO  is saying, "Use more tokens, generate more code, [6:59] generate more output. How do we move faster?" Um  agents and agentic reasoning, um which is kind [7:05] of a newer um group of LLMs that has come out and  have really evolved over the last few years, um is [7:12] actually driving inference scaling significantly.  Right? So, what, you know, here's just a sample [7:17] input to a traditional LLM to a traditional  LLM. And what you can see is without reasoning, [7:22] right? The model may just be going through once,  just a singular forward pass auto regressively. [7:27] Um and you might get 439 tokens that the model  is kind of using to generate, right? So, the [7:32] model's getting this prompt in, then generating  439 tokens. That has a cost. For age for agents, [7:39] that cost could be up to 20x more, right? We've  all seen Claude think, {quote} and {unquote}, [7:44] where basically it will go in and do these  kind of iterative loops of checking itself, [7:48] whether that be doing tool calls, whether that be  assessing different pathways, right? It's going [7:54] down different trees. This is really expensive.  So, more tokens, 150x more compute potentially, [8:01] but the the big breakthrough here is  that the answer is correct, right? So, [8:05] we've all seen this kind of happen in real time.  These models getting better. [sighs and gasps] So, [8:10] now we'll talk about a little bit why  this really matters to the enterprise, [8:13] right? So, we talked a little bit about open  models, the explosion of open weight models [8:18] back in March of 2022 three, we were kind of  at the very early days. You know, we had Llama [8:24] in 2022 in November. I'm sure we all remember when  ChatGPT came out from OpenAI, and that really kind [8:30] of proliferated a long tail of releases from a lot  of key model providers, right? So, Meta, we have [8:36] Quan, we have Mistral, Nvidia, all these different  companies kept putting out open source models. And [8:42] today, you know, in the news, it feels like you  can go and check every single week there's a new [8:46] model at the top of the leaderboard, right? That  is now crushing in agentic workflows, in reasoning [8:53] that can compete with a lot of the closed model  providers like an Anthropic, like a an OpenAI. [8:59] So, what a well-optimized inference serving  stack really enables you to do is, we talked [9:05] about before, right? Any model, any server,  any hardware. The inference server layer is [9:10] really what allows you to do this, right? So,  this is what gives you the speed. It gives you [9:15] the scalability, right? How do I actually, you  know, serve multiple models potentially at once? [9:20] We'll talk a little bit about the end like  distributed inferencing. So, as these models get [9:23] bigger and the workloads get larger, how do you  actually route and do intelligent scheduling and [9:28] things like that? Stability, right? So, a lot of  companies will kind of build their own hardware, [9:34] build their own inference server on top,  but what happens as a as an enterprise when [9:38] you want to now use a new model, right? You  can't learn 20 different inference servers [9:44] and each of their kind of ins and outs as a  large company that wants to actually scale AI, [9:50] right? We don't think there's going to be a  one model solution at Red Hat. We see kind of [9:54] a combination of multiple models as the way this  is going to evolve and standardizing the inference [9:58] server layer is really what can help connect all  those new models, all those new capabilities, [10:03] all those advancements in architecture down  to your actual hardware that's available. [10:08] Um this is a quick poll. I'm sure we all  know the answer. What do we think costs [10:13] more to run in the long term, training or  inference? Who thinks Who thinks training? [10:20] Okay, good. No one was asleep. Who thinks  inference? [snorts] Okay, that's correct. [10:25] The answer is inference. Um but yeah, this is  basically this turned the world on its head if [10:30] you look back five or six years. If you had asked  this question five or six years ago in this room, [10:35] the answer would be very different, right?  So, I think this just this goes to show [10:39] the speed and the evolution of where  we are in kind of this AI life cycle. [10:44] Um very quickly, I'll just kind of give  an overview of Red Hat's inference stack [10:49] and then I'll I'll save a few minutes at the end  for some some questions as well. So, what Red Hat [10:55] does is basically we deliver an enterprise GenAI  inference platform as part of our inference piece [11:03] as a as part of the entire story. So, we also do  training, we do fine-tuning, we do observability, [11:09] but today we're talking about inference and this  is kind of a high-level view of our inference [11:12] stack. So, um at the middle level, we talked a  lot about inference already. Let's talk about the [11:17] inference server that makes this all happen and  that's vLLM, right? So, this is what the key is to [11:24] unlock that picture of all models to all hardware,  right? If every single model had its own inference [11:31] server, the amount of overhead you would have to  deal with to serve all these different models, [11:36] all these architectures, would be too much to deal  with. And this is what vLLM critically solves, [11:41] and it doesn't just solve it from a enablement  perspective, but from an optimization perspective. [11:46] So, vLLM, for those of you that don't know,  in about 2 years went from a research project [11:51] to an industry uh standard. Um and basically,  this came out of the Berkeley Sky Computing Lab, [11:56] um very quickly gained steam as one of the  leading inference servers in the market, [12:01] um and now is kind of at the lead in terms  of stars and GitHub usage over the last um [12:08] probably last year at least or so. Um and really  the benefit here is, you know, what we've been [12:13] talking about already, high performance. Right?  So, vLLM kind of is highly optimized for you know, [12:20] specifically the Nvidia stack as well as AMD, um  Intel, um as well as uh Trainium. We we have we've [12:27] done worked with um AWS. We've also worked with  Google on TPU enablement, which we talked about [12:32] um at a Google summit um about a year or so back.  Um and really the benefits here are that you're [12:37] not kind of locked in to any one specific hardware  vendor, whether you want to buy from Nvidia or you [12:42] want to buy from Intel or you want to buy from  IBM, you can do that, and vLLM can still serve as [12:46] that core inference server for your platform.  Um it's also really easy to use, right? So, [12:51] at Red Hat, we have a full inference engineering  team that actually partners with a lot of the [12:56] model providers themselves to ensure what's  called day zero enablement. So, when a new model [13:02] comes out of a lab, call it, you know, Gemma 4,  call it NeMo Tron, call it Kimmy K2 or Kimmy K3, [13:08] we work with those original providers to ensure  that on day zero, that model will run on vLLM. [13:14] Right? A model on itself coming to the market in  the open source ultimately doesn't mean much if [13:20] no one can run it. And vLLM kind of and our team  at Red Hat, along with the core contributors to [13:25] vLLM across many companies, help make that reality  possible of like how do we get this thing live? [13:30] How do I test this in my cloud code setup? How  do I run this with open code? How do I start [13:34] doing the reasoning? How do I start building  applications? This is what vLLM enables, [13:38] right? So that work in the community really pays  dividends throughout the whole ecosystem. Um I [13:43] talked about all the companies that contribute to  vLLM and there are many. Um Red Hat notoriously is [13:48] is the number one contributor um to vLLM. Uh we  also do a lot of work within the vLLM community. [13:54] So we offer a bi-weekly LLM uh vLLM office hours  where we talk about all sorts of different topics [14:00] um related to vLLM and inference and optimization.  So definitely stay tuned for those. Um [14:08] And yeah, kind of you know, it's nice to  see different shout-outs and different kind [14:11] of highlights. This is a a really nice one  that came out from the vLLM project team of [14:15] one of the fastest-growing open-source AI projects  of the year last year. Right? So this isn't small [14:21] small pennies. Um vLLM is really powering a  whole new wave of open-source AI innovation. [14:28] Um this is just a really nice summary picture. So  if you've been taking photos, this is a good one [14:32] to take pictures of, but really just anchors  home that vLLM is that middle connector layer [14:37] of any model to any hardware, right? Wherever you  want. So you can run these things in the cloud, [14:42] you can run these models on prem, right? Red  Hat, we do a lot of work with regulated customers [14:48] who like to run AI models on prem and have  full control over cost, over performance, [14:52] right? And these are things that vLLM can enable  you to do with this open-source model ecosystem. [14:59] Um the next layer we talk very briefly  about is the model layer. Um so Red Hat AI [15:06] is a uh public-facing repo on Hugging Face. So our  team at Red Hat, along with becoming vLLM experts, [15:14] we are also model optimization experts. So we  do quantization, um we do other optimization [15:20] techniques such as speculative decoding as well  to models to basically make them more performant [15:26] on VLLM on whatever hardware that  you have of your choice. Right? So, [15:30] when models come out from labs, traditionally  they come out in what's called BFloat16, [15:35] um which is just full representation of each of  the weights. Um with techniques like quantization, [15:40] it's really just model compression. Right?  So, instead of representing pi like 3.141592, [15:46] you can just say three. And it turns out that  with a lot of these cutting-edge algo- rithms [15:50] that we've been able to productize in the open  source um through a library called LLM Compressor, [15:55] we're able to emulate the same baseline  performance as a full precision model. Right? [16:00] And this was a really big breakthrough that you  can have a model that's represented say as FP8, [16:05] um so half the size of the BF16 model. However,  the accuracy hit is less than 1% for every single [16:12] FP8 model that Red Hat puts out on the Red Hat AI  Hugging Face. So, the trade-off of performance for [16:18] accuracy is you you know, it's it's quite light.  And this is a kind of a really big benefit of [16:23] what VLLM enables you to speed up and what Red Hat  does in terms of model compression and research. [16:29] Here's just a graph um showing kind of  the impact of quantization. Right? So, [16:34] you can just kind of see on the left it looks  like we're looking at uh time per output token. [16:38] And you can just kind of see the performance of  the baseline model compared to the FP8 model. [16:44] Um the last piece we'll talk about is kind of  the uh the biggest one. And this is the one I [16:48] I would say I'm most excited about as we think  about the future of open models, open source, [16:54] um as well as kind of inference serving.  And that's distributed inference. Um [17:00] so, LLMD is a distributed LLM inference platform.  The way I like to explain it is a marriage between [17:07] all the benefits of Kubernetes and all the  benefits of VLLM kind of married into one project, [17:12] right? And it's really, you know, the goal  of LLMD is to enable this wide scale and this [17:19] distributed inferencing framework so that you can  run larger models, serve more users concurrently, [17:25] properly schedule where your workloads are  actually running, and it comes with some really, [17:31] really awesome techniques out of the box  such as disaggregated pre-fill and decode [17:37] that are really kind of unlocked a new tier  performance that you can unlock as these [17:42] models continue to get bigger and we start  thinking about serve inference more widely. [17:47] So, why distributed inference, right? On the left,  you can kind of see up here modern HTTP requests. [17:54] So, the traditional like microservice model  is kind of dead when it comes to LLMs and AI [18:00] inference serving mostly because HTTP requests  generally more uniform, they're load balanced [18:07] because they're cheap, and you have like a high  QPS, right? Like there's not a lot of cost that [18:13] you're paying to process workloads in that sense.  LLM requests are really different, right? Requests [18:18] come in at very different sizes. You might have  an intern ask a sentence, "What's the capital of [18:23] France?" You might have a senior engineer copy and  paste an entire Jupiter notebook in, right? So, [18:29] you're having variable inputs, and really what  what LMD can help do is it has an LLM-based [18:36] aware router that will help you load balance  and route the right requests to the right pods [18:42] so that you're able to more cheaply process and  generate these inputs to get stronger outputs. [18:51] This is just another view of kind of  some of the big benefits of LMD around um [18:58] operatization Sorry, operation I can't even read  that word from here. Flexibility, performance, [19:06] and how to operationalize your AI. [19:09] This is like summary slide, but really so Red Hat  is not the only one that works on um LLMD. We work [19:16] with a ton of different ecosystem partners as  well, such as Google, Nvidia, Hugging Face, [19:21] CoreWeave um are all kind of part of this LLMD  initiative. Um, and really it's all about kind [19:26] of the future of how do you optimize inference  even further beyond a single inference server, [19:30] marry the benefits of ELM with Kubernetes to  achieve this kind of new scale. Um, so yeah, that [19:36] was a quick overview. Um, all of this is kind of  do POCs at Red Hat uh for the Red Hat AI platform. [19:43] Today we just covered the inference piece. There's  a lot more around training, tuning, security, [19:48] governance, um tons of different layers that we  can kind of get into. Um, but yeah, hopefully [19:53] this was educational and informative for you all  and thank you very much. Um, thanks. [applause] [20:02] So I'm at the clock, but if there's like one  or two questions, I can try to do them fairly [20:06] quickly. Otherwise, I'm at the Red Hat booth. Um,  we're right in the main exhibition hall, so feel [20:11] free to find me today, tomorrow. Um, I'm here  until Thursday. So yeah, question in the front. [20:17] Are you guys also have mechanical tools  for fault tolerance and if the agent goes [20:24] in the wrong direction or fails, then  can we rewind that using the 40 spots? [20:29] And how would the load balancer handle like  fault tolerance and uh you know, issues such as [20:35] agent failed and now it has to resume from  where it failed, like the Yeah, so I mean, [20:41] so traditionally VLM out of the box ships with  telemetry uh via Prometheus. Um, so there are [20:47] logs that look at traditional LLM metrics like  time per output token, time to first token, [20:52] latency, throughput. Um, we are investigating now  more into like what's called like agentic traces, [20:58] right? So as you get into different capabilities  like tool calling, right? You want to see [21:01] what tool did this model call, um what was  the input and output, where did it go wrong, [21:07] and how do you more efficiently like reason  through that? And then when you get to like the [21:11] compute workload, how do you make sure that like  the resolution is routing to the right place? Um [21:16] so we're we're doing a lot of active research with  LLMD and tool calling as well as just with VLLM [21:21] and making sure that the top models are able to do  that effectively and working within the community [21:27] um to also like invest in more evaluations there  so that we can kind of publish more of those [21:31] results. But that's like an active area that we're  invested in. Yeah. It's a quick question. Yeah. [21:40] Go ahead, go ahead. Okay, I'm sorry, quick  question. Uh also you're using VLLM with uh [21:44] Qwen-36-27B-FP8. Yeah. Wanted to know is there  any advantage to running your version cuz I just [21:52] you know Yeah. So our ver So and yeah, that you  guys have a version. So our our version of VLLM [21:58] is the same version that's upstream. However,  the model. I mean, I'm talking about the model [22:03] on Hugging Face. Right now I'm literally using the  Qwen version of three uh uh 36-27B-FP8, but I just [22:10] noticed that you guys also have a Qwen-36-27B-FP8.  I mean, what's the difference between So Qwen's [22:16] version and your version. So there's two types of  FP8. There's FP8 block and FP8 dynamic. Um there's [22:22] minor tradeoffs that are being had depending  on the hardware that you're running it on so we [22:25] could look at that. Uh but no, like if it's an FP8  dynamic model that Red Hat puts out versus I mean, [22:30] we try not to make them if the original provider  puts it out. The only difference would be um if [22:34] it's a different type of FP8 quantization,  maybe to the weights or the activations, [22:38] we might do something slightly different uh to  try to get some speed up on Nvidia hardware or [22:42] something like that. But otherwise, no, if it if  they're both FP8 dynamic, you would get the same [22:46] performance on VLLM. Okay. Uh real quick, LLMD,  [clears throat] is that also open source? LLMD is [22:51] fully open source. Yep. How does OpenShift AI fit  into this landscape that you just uh All of what [22:59] I just shared is part of OpenShift AI. It's now  delivered as what's called Red Hat AI Enterprise, [23:05] um which is really just it's a bundle of  OpenShift AI, um OpenShift Container Platform, [23:10] and and an AI accelerator entitlement. So, really  you get the entire platform. So, we just went into [23:14] the inference stack. However, OpenShift AI comes  with everything. So, we get everything, training, [23:19] tuning, observability, plus everything I just  covered, LLMD, and the whole inference side. Um [23:24] we also have an inference skew called Red Hat AI  Inference. So, if you just wanted the bits I spoke [23:29] about, just, you know, the model catalog with  our quantized optimized models, VLAM and LLMD, [23:35] um and a few other components, you could get  that just with a Red Hat AI Inference, which [23:40] is another skew, a little bit more affordable,  um focused just at going after the inference. [23:45] So, those are your like kind of the two options.  Yeah. All right, we're probably at time, um but [23:51] thank you all very much. And again, I'll be at the  booth if you have any further questions. Thanks.