Red Hat Pushes Quantized Models to Fix Multi-Step AI Agent Bottlenecks

Small but Mighty: Quantized LLMs for Real-Time Agentic Systems With Red Hat — Ai4

Ai4

Autonomous AI agents trigger compounding latency and unsustainable token costs in production unless enterprises adopt quantized models that preserve reasoning accuracy on leaner hardware.

AI-assisted summary of Small but Mighty: Quantized LLMs for Real-Time Agentic Systems With Red Hat — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 27, 2025, this presentation features Taylor Smith, Senior AI Advocate at Red Hat. According to the video description, the session explores how model quantization techniques compress large and small language models to power low-latency, cost-effective agentic systems using runtimes like vLLM without sacrificing output quality.

## TL;DR Taylor Smith challenges the industry assumption that enterprise agentic architectures require massive frontier models, highlighting how autonomous multi-step reasoning loops compound token costs and latency bottlenecks. She demonstrates how quantizing model weights, activations, and KV caches enables smaller, cost-effective models to retain up to 99% of baseline reasoning accuracy across architectures like IBM Granite and DeepSeek. By combining compression workflows like LLM Compressor with high-throughput inference runtimes such as vLLM, enterprises can reliably deploy low-latency, scalable agentic pipelines on cost-effective hardware.

## Atmosphere & style The presentation carries an energetic, pragmatic, and approachable tone, blending technical rigor with conversational humor about the AI agent hype cycle and lighthearted personal nods to Beyoncé and cats. Smith delivers a grounded, practitioner-oriented talk that avoids vanity metrics, openly addressing the latency pitfalls and operational hurdles of autonomous agents. The vibe feels like an insightful peer briefing, guiding developers past peak hype into the practical realities of high-throughput model deployment.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=qflAu_ztE1A&t=4s) The prevailing industry assumption that "bigger is better" is impractical for enterprise deployments constrained by on-premise hosting, compliance mandates, and operational costs. - [01:48](https://www.youtube.com/watch?v=qflAu_ztE1A&t=108s) Enterprise AI is shifting beyond passive, single-turn chat and RAG pipelines toward autonomous agents that independently formulate plans and execute actions across external systems. - [02:50](https://www.youtube.com/watch?v=qflAu_ztE1A&t=170s) Production agent architectures face an adoption bottleneck due to skyrocketing token costs and unpredictable latency caused by multi-step iterative reasoning loops. - [06:24](https://www.youtube.com/watch?v=qflAu_ztE1A&t=384s) Deploying massive 405B+ parameter models demands steep baseline infrastructure investments, making smaller architectures (70B and below) far more cost-effective for enterprise private AI. - [08:02](https://www.youtube.com/watch?v=qflAu_ztE1A&t=482s) Quantization eliminates the size-versus-accuracy trade-off by compressing model weights, KV cache, and activations down to lower precisions such as FP8 and INT8. - [10:41](https://www.youtube.com/watch?v=qflAu_ztE1A&t=641s) Rigorous benchmarking demonstrates that quantized models, including IBM Granite and DeepSeek reasoning models, can retain up to 99% of their baseline accuracy. - [12:38](https://www.youtube.com/watch?v=qflAu_ztE1A&t=758s) Effective quantization requires domain-specific calibration datasets to accurately map continuous tensors to lower-precision formats without degrading task performance. - [13:59](https://www.youtube.com/watch?v=qflAu_ztE1A&t=839s) Serving quantized models requires optimized inference runtimes like vLLM, TensorRT-LLM, or SGLang to maximize token throughput and hardware efficiency. - [13:59](https://www.youtube.com/watch?v=qflAu_ztE1A&t=839s) Because agentic loops trigger multiple sequential inference calls per user request, reducing base-model latency is critical to prevent compounding delays under high concurrent load. - [16:07](https://www.youtube.com/watch?v=qflAu_ztE1A&t=967s) Open-source ecosystems and communities increasingly provide Day 0 pre-quantized weights, lowering the barrier to deploying energy-efficient, scalable agent backends.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=qflAu_ztE1A&t=4s) Introduction and Challenging the "Bigger is Better" Narrative

Taylor Smith opens the session by introducing herself as an AI and open-source advocate at Red Hat, working across the company's AI product portfolio from her home base in Raleigh, North Carolina. After sharing a brief personal aside—noting her deep love for cats and giving a shout-out as an unapologetic member of the Beyoncé "Beyhive"—she dives directly into the prevailing industry mindset surrounding AI architecture. The central goal of the talk is to dismantle the pervasive assumption that "bigger is better" when selecting large language models (LLMs), particularly within the context of agentic AI systems, which she terms the "hot buzzword of the year."

While massive frontier models have distinct strengths, organizations frequently encounter practical boundaries where relying on multi-hundred-billion parameter models becomes unsustainable or counterproductive. Smith highlights that many real-world enterprise deployments require hosting models locally on-premises due to compliance mandates or demand domain-specific fine-tuning. In these scenarios, deploying massive generalized foundation models introduces unnecessary friction, creating a compelling need for smaller, specialized, and quantized models tailored directly to the workload.

- **Speaker Background**: Taylor Smith, AI and Open Source Advocate at Red Hat (Raleigh, NC). - **Core Premise**: Dispelling the myth that bigger models are inherently superior for all enterprise deployments. - **Key Constraints**: On-premises hosting, regulatory compliance, operational expenses, and task specialization require smaller-scale models.

---

[01:48](https://www.youtube.com/watch?v=qflAu_ztE1A&t=108s) The Evolution from Simple Chat Interfaces to Agentic AI

Reflecting on the generative AI adoption curve, Smith notes that most practitioners began with a straightforward chatbot paradigm: a user submits a prompt, the model processes the context against its pre-trained knowledge base, and it returns a predicted text response. Even with the addition of Retrieval-Augmented Generation (RAG) pipelines to inject external document context, the interaction remains a rigid, single-turn request-and-response loop. While this setup works well for ad-hoc personal tasks, it falls short when applied to multifaceted enterprise workflows.

To address complex business problems, enterprise AI must actively interact with external infrastructure—such as reading logs, manipulating environments, sending notifications in a Slack workspace, or executing commands within a Kubernetes cluster. This necessity has driven the rapid transition from passive chat interfaces to autonomous agentic architectures capable of taking real-world actions independently.

```mermaid flowchart TD accTitle: Evolution from Chatbot to Agentic System subgraph Traditional["Traditional Chat / RAG Pattern"] User["User Prompt"] --> Model["Base LLM + RAG Context"] Model --> StaticAnswer["Generated Static Response"] end

subgraph Agentic["Agentic Architecture"] AgentUser["Complex User Objective"] --> Agent["Autonomous Agent"] Agent --> Plan["Task Planning & Reasoning"] Plan --> Execution{"Tool Invocation"} Execution --> Slack["Slack Workspace"] Execution --> K8s["Kubernetes Cluster"] Execution --> CustomAPI["Enterprise APIs"] Execution --> Observation["Observe State & Iterate"] Observation --> Agent end ```

---

[02:50](https://www.youtube.com/watch?v=qflAu_ztE1A&t=170s) Anatomy of an Agent and the Enterprise Production Chasm

Highlighting that the conference agenda features over 60 dedicated agentic sessions, Smith jokes that presenting on anything other than agents would make a speaker feel completely irrelevant. She clarifies the core functional pillars that define a genuine agentic system versus standard generative chat:

1. **Tool Utilization**: Equipping the model with functional interfaces to external software ecosystems, APIs, communication channels (like Slack), and underlying compute infrastructure. 2. **Autonomous Planning and Reasoning**: Empowering the system to formulate dynamic step-by-step task plans, execute intermediate thoughts, evaluate its own progress, and adjust course without requiring human micromanagement at every phase.

``` +-----------------------------------------------------------------------------+ | ENTERPRISE AGENTIC HYPE CYCLE | | | | Peak of Inflated Expectations | | [ CURRENT STAGE ] | | * Model Context Protocol (MCP) experiments | | * Reasoning & coding agent pilots | | /\ | | / \ | | / \ | | / \ Trough of Disillusionment | | / \ * Skyrocketing token costs | | / \ * Unpredictable multi-call latency | | / \ * Operational maintainability hurdles | | / \ | | / \___________ Production Reality | +-----------------------------------------------------------------------------+ ```

Smith cautions that the industry currently sits at the very peak of the hype cycle, with a predictable "dip" on the horizon. While developers and organizations are actively experimenting—such as spinning up Model Context Protocol (MCP) servers and exploring agentic coding extensions—the vast majority of enterprises have not successfully pushed autonomous agents into full-scale production. Agents introduce significant operational overhead, complex governance requirements, and severe performance bottlenecks: - **Cost Escalation**: Autonomous reasoning loops make numerous iterative model calls under the hood, dramatically increasing token consumption per user request. - **Latency & UX Degradation**: Waiting for an agent to traverse several multi-second reasoning and execution chains leads to frustrating user delays if powered by sluggish foundation models. - **Scalability**: Maintaining low-latency performance across concurrent production users multiplies infrastructure demands exponentially.

---

[06:24](https://www.youtube.com/watch?v=qflAu_ztE1A&t=384s) The High Cost of Massive Foundation Models

Frontier LLMs continue to expand exponentially in parameter volume with each generation. However, deploying these massive models carries steep infrastructure costs. Smith emphasizes that hosting multi-hundred-billion parameter models (such as 405B-scale systems) on cloud infrastructure demands substantial monthly base commitments just to maintain an idle or minimally utilized endpoint, entirely excluding high-concurrency traffic spikes.

``` +--------------------------------------------------------------------------------+ | LLM SCALE & DEPLOYMENT SPECTRUM | | | | [ Small-to-Mid Models: <= 70B ] [ Massive Models: ~405B+ ] | | * Practical for Private AI & On-Prem * Prohibitive Baseline Infra Cost | | * Predictable Latency Control * High Cloud Token Overhead | | * Lower Memory Footprint * Severe Concurrency Bottlenecks | +--------------------------------------------------------------------------------+ ```

For enterprises relying on external commercial APIs, direct infrastructure hosting is avoided, but organizations remain exposed to surging per-token billing rates and unpredictable API latency that cannot be tuned or guaranteed. Furthermore, businesses governed by strict privacy or compliance regulations must keep operations on-premise, where procuring and managing hardware for 405B+ parameter models is frequently cost-prohibitive. In contrast, models sized at 70B parameters or below represent a far more manageable, cost-effective target for enterprise hardware.

---

[08:02](https://www.youtube.com/watch?v=qflAu_ztE1A&t=482s) Demystifying Quantization: Mechanisms and Memory Efficiency

When engineering teams transition from massive models to smaller parameter variants to lower costs and latency, they inevitably run into an accuracy penalty. Smith introduces **quantization** as the primary solution to break this trade-off: compressing an existing high-performing model so that it retains the reasoning fidelity and accuracy of a larger model while drastically shrinking its memory footprint.

Smith uses a simple analogy: quantization functions much like compressing a heavy, uncompressed RAW photograph into a JPEG, or packing files into a ZIP archive. Rather than storing model parameters at full 16-bit floating-point precision (FP16), quantization algorithms map and compress these numerical weights down to lower-precision formats, such as 8-bit integers or floating points (FP8/INT8), without severely degrading model performance.

```mermaid flowchart LR accTitle: Quantization Target Areas in an LLM subgraph QuantTargets["Key Quantization Target Areas"] Weights["Model Weights / Parameters<br/>(Core Network Layers)"] KVCache["KV Cache<br/>(Conversation Memory Context)"] Activations["Activations<br/>(Intermediate Calculation Layers)"] end

subgraph Impact["Operational Benefits"] Mem["Dramatically Reduced VRAM / Memory"] HW["Deployment on Smaller GPUs & CPUs"] Speed["Higher Concurrency & Token Throughput"] end

QuantTargets --> Impact ```

Compression can be applied across several distinct elements of an LLM pipeline: - **Weights**: Compressing the core learned parameters of the network. - **KV Cache**: Quantizing the key-value cache that holds multi-turn conversational history and working context in memory. - **Activations**: Reducing the precision of dynamic intermediate computation layers during the forward pass.

Applying these compression techniques allows quantized models to run on significantly smaller, cheaper GPU hardware configurations—and in some setups, directly on CPUs—while maintaining robust token throughput and managing latency gracefully under concurrent load.

| Optimization Vector | Standard Small-Parameter Model | Quantized Large Model | |:--- |:--- |:--- | | **Model Size Origin** | Native small architecture (e.g., 8B) | Compressed larger architecture (e.g., 70B $\rightarrow$ FP8/INT4) | | **Baseline Accuracy** | Lower baseline knowledge capacity | Preserves higher reasoning capabilities of base model | | **Memory Footprint** | Low VRAM requirement | Substantially reduced VRAM requirement | | **Hardware Fit** | Standard edge / commodity hardware | Fits cost-effective GPUs and select CPU environments | | **Concurrency Scaling**| Fast, but accuracy-limited for complex reasoning | Superior throughput, lower latency under concurrent loads |

[10:41](https://www.youtube.com/watch?v=qflAu_ztE1A&t=641s) Benchmarking Quantized Models: Accuracy and Latency

When implementing quantization independently, rigorous and explicit testing is essential to ensure that model degradation is minimal. Benchmarking should cover core performance metrics—such as inter-token latency, time per output token (TPOT), overall end-to-end latency, and throughput—alongside standardized accuracy evaluations against the original unquantized base model. Real-world results demonstrate that compressed architectures can maintain exceptional fidelity; for instance, IBM Granite models compressed under various integer (INT) and floating-point (FP) quantization formats retain up to 99% of their baseline accuracy while drastically shrinking model size and slashing inference latency.

```mermaid flowchart LR accTitle: Quantized Model Evaluation Framework subgraph Benchmarks["Evaluation Suite"] A["Accuracy Benchmarks<br/>(Task & Reasoning Evals)"] B["Performance Benchmarks<br/>(Inter-token Latency, TPOT, Throughput)"] end subgraph Target["Validation Targets"] C["Maintain ~99% Base Accuracy"] D["Substantial Latency & Footprint Reduction"] end A --> C B --> D ```

This retention of capability extends to advanced reasoning models like DeepSeek. Benchmark charts demonstrate that despite the perception of complex reasoning tasks being fragile, quantized DeepSeek variants sustain up to 99% accuracy across standard test suites. Acknowledging that the evaluation charts can look like "boring graphs," the speaker emphasizes that reaching this tier of quality is not a trivial out-of-the-box operation: "If, I don't know, the HR person wants to go and quantize a model... they're not going to get the best output right away." Quantization remains a specialized machine learning and data science task, but with proper expertise, achieving near-lossless compression is entirely feasible.

- **IBM Granite Benchmarks**: Preserves up to 99% accuracy across INT and FP quantization formats while dramatically reducing disk footprint and inference execution time. - **DeepSeek Reasoning Evaluations**: Successfully preserves up to 99% baseline reasoning performance under quantized configurations. - **Core Latency Metrics**: Essential tests include inter-token latency, time per output token (TPOT), end-to-end latency, and request throughput.

---

[12:38](https://www.youtube.com/watch?v=qflAu_ztE1A&t=758s) Quantization Workflow and Tooling with LLM Compressor

To streamline the quantization lifecycle, practitioners can leverage dedicated frameworks such as LLM Compressor, which provides tooling across multiple quantization algorithms and precision targets. The workflow begins by selecting a starter base model and determining the targeted layers—such as quantizing weights only versus both weights and activations (W&A)—before matching those requirements to the appropriate algorithmic strategy.

```mermaid flowchart TD accTitle: LLM Compressor Quantization Pipeline subgraph Preparation["1. Configuration"] M["Starter Base Model"] --> A["Select Algorithm & Target Layers<br/>(Weights vs. Activations)"] end subgraph Calibration["2. Data Calibration"] D["Calibration Dataset<br/>(Domain & Task Representative)"] end subgraph Execution["3. Compression"] A --> C["LLM Compressor Engine"] D --> C C --> Q["Quantized Model Artifact<br/>(Preserved Accuracy)"] end ```

A pivotal component of this pipeline is selecting a representative calibration dataset that closely mimics the actual production prompts and domain data the model will encounter. The calibration dataset prevents the algorithm from "guessing willy-nilly" when mapping continuous values to lower-precision representations; instead, it dynamically validates and calibrates tensor transformations to ensure that response quality and reasoning patterns remain intact throughout the compression process.

- **LLM Compressor**: A framework providing multi-algorithm support for compressing model weights and activations. - **Target Selection**: Precision configurations (e.g., INT8, INT4, FP8) chosen based on hardware support and layer sensitivity. - **Calibration Datasets**: Domain-specific sample data used during quantization to fine-tune scaling factors and prevent degradation.

---

[13:59](https://www.youtube.com/watch?v=qflAu_ztE1A&t=839s) High-Performance Inference Runtimes for Multi-Step Agents

Once a model is quantized, it must be served through an optimized inference runtime. Open-source engines like vLLM—as well as NVIDIA TensorRT-LLM and SGLang—stand out for their native support of diverse quantization formats, allowing teams to load compressed checkpoints directly and benchmark real-time execution. In side-by-side prompt comparisons between full-precision and quantized variants, the performance gains appear directly in measured end-to-end latency, inter-token generation speed, and overall system throughput.

```mermaid flowchart TD accTitle: Agentic System Latency Accumulation subgraph AgenticLoop["Multi-Step Agent Pipeline"] Step1["Step 1: User Prompt Decomposition"] --> Step2["Step 2: Tool Selection & Call"] Step2 --> Step3["Step 3: Environment Feedback Parsing"] Step3 --> Step4["Step 4: Final Synthesized Response"] end subgraph RuntimeEngine["vLLM / TensorRT-LLM / SGLang"] Quant["Quantized Low-Latency Serving"] end Quant -. Reduces step overhead.-> Step1 Quant -. Reduces step overhead.-> Step2 Quant -. Reduces step overhead.-> Step3 Quant -. Reduces step overhead.-> Step4 ```

While watching side-by-side token streaming in a demo can seem subtle, the cumulative impact is massive in enterprise environments handling hundreds of concurrent sessions. Enterprise deployments frequently require serving "500 concurrent users" within tight, millisecond-level SLAs. In agentic architectures—where a single user request triggers multi-step loops, tool invocations, and background reasoning chains—latency penalties compound exponentially at each hop. Driving down latency at the base model level is critical to keeping background agentic workflows responsive and practical.

- **Inference Runtime Engines**: vLLM, NVIDIA TensorRT-LLM, and SGLang provide broad support for INT/FP quantization formats. - **High Concurrency Demands**: Built to satisfy enterprise scale targets (e.g., 500 concurrent users with strict millisecond SLA thresholds). - **Compounding Latency**: Agentic systems execute multiple consecutive inference calls behind the scenes; reducing per-token and inter-token latency prevents agent loops from stalling.

---

[16:07](https://www.youtube.com/watch?v=qflAu_ztE1A&t=967s) Production Advantages and Open-Source Ecosystem

Deploying compact, memory-efficient quantized models unlocks substantial operational advantages for modern agentic pipelines: faster inner execution loops, reduced hardware spend, higher concurrency thresholds, and superior energy efficiency—all while maintaining the accuracy levels expected of larger models. For teams that do not wish to quantize checkpoints from scratch, open-source repositories provide pre-quantized weights for leading model architectures with Day 0 availability upon upstream release.

The central takeaway is clear: "Bigger does not equal better." As autonomous agents introduce latency and cost bottlenecks unseen in simpler single-turn AI systems, quantized models paired with toolsets like LLM Compressor and runtimes like vLLM provide the necessary foundation for scalable, high-throughput agent deployments. Attendees working on agent architectures are encouraged to connect with the Red Hat team to share findings and collaborate on real-world implementations.

- **Operational Benefits**: Lower inference cost, expanded concurrent capacity, decreased power/energy consumption, and preserved accuracy. - **Pre-Quantized Repositories**: Red Hat and community hubs provide open-source quantized artifacts with Day 0 support for newly released models. - **Tooling Stack**: LLM Compressor for model compression combined with vLLM for high-throughput serving.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=qflAu_ztE1A&t=4s) bigger is better — Taylor Smith

> [00:04](https://www.youtube.com/watch?v=qflAu_ztE1A&t=4s) hot buzzword of the year — Taylor Smith

> [02:50](https://www.youtube.com/watch?v=qflAu_ztE1A&t=170s) dip — Taylor Smith

> [10:41](https://www.youtube.com/watch?v=qflAu_ztE1A&t=641s) boring graphs — Taylor Smith

> [10:41](https://www.youtube.com/watch?v=qflAu_ztE1A&t=641s) If, I don't know, the HR person wants to go and quantize a model... they're not going to get the best output right away. — Taylor Smith

> [12:38](https://www.youtube.com/watch?v=qflAu_ztE1A&t=758s) guessing willy-nilly — Taylor Smith

> [13:59](https://www.youtube.com/watch?v=qflAu_ztE1A&t=839s) 500 concurrent users — Taylor Smith

> [16:07](https://www.youtube.com/watch?v=qflAu_ztE1A&t=967s) Bigger does not equal better. — Taylor Smith

## Mentioned Taylor Smith, Red Hat, Beyoncé, Model Context Protocol (MCP), Slack, Kubernetes, IBM Granite, DeepSeek, LLM Compressor, vLLM, NVIDIA TensorRT-LLM, SGLang

## Video description Taylor Smith, Senior AI Advocate, Red Hat

Abstracts

LLMs now drive agents that handle multi-step reasoning, tool use, and user interaction—but many struggle to scale those systems affordably. In this session, we’ll dive into quantization: a critical technique that enables deployment of fast, cost-effective models without sacrificing output quality.

The talk will illustrate how different quantization methods compress large LLMs and SLMs while preserving accuracy, and how quantized models improve inference throughput in agentic pipelines. We’ll explore real-world use cases and guide you through inference runtimes like vLLM that make quantized LLMs practical for low-latency, high-scale deployments.

[Source](https://www.youtube.com/watch?v=qflAu_ztE1A)

中文简报

Red Hat力推量化模型以破解多步AI智能体瓶颈 自主AI智能体在生产环境中会引发复合延迟和不可持续的Token成本，除非企业采用能在更精简硬件上保持推理精度的量化模型。 Taylor Smith对企业智能体架构必须依赖超大规模前沿模型的行业假设提出质疑，并指出自主多步推理循环会如何加剧Token成本和延迟瓶颈。她展示了对模型权重、激活值和KV缓存进行量化，如何使更小、更具成本效益的模型在IBM Granite和DeepSeek等架构上保持高达99%的基线推理精度。通过将LLM Compressor等压缩工作流与vLLM等高吞吐量推理运行时相结合，企业能够在高性价比硬件上可靠部署低延迟、可扩展的智能体流水线。 业界普遍认为“越大越好”的假设，对于受本地部署、合规要求和运营成本限制的企业落地而言并不切实际。 (0:04) 企业级AI正在超越被动的单轮对话和RAG流水线，迈向能够在外部系统中自主制定计划并执行操作的自主智能体。 (1:48) 由于多步迭代推理循环导致Token成本激增且延迟不可预测，生产级智能体架构面临应用瓶颈。 (2:50) 部署405B+参数的超大模型需要高昂的基础设施基线投资，这使得较小架构（70B及以下）在企业私有AI中更具成本效益。 (6:24) 量化通过将模型权重、KV缓存和激活值压缩至FP8和INT8等更低精度，消除了尺寸与精度之间的权衡取舍。 (8:02)

Reading summary

Red Hat AI advocate Taylor Smith argues that massive foundation models are economically and operationally impractical for autonomous enterprise agents due to compounding multi-step latency and hardware costs. Instead, quantizing smaller models to lower-precision formats like FP8 or INT4 allows organizations to preserve up to 99% of baseline reasoning accuracy while meeting strict real-time production SLAs. Compounding Latency in Multi-Step Agent Workflows (2:50) While simple generative chat and retrieval-augmented generation (RAG) function as single-turn request-and-response interactions, enterprise agentic systems require dynamic planning, self-evaluation, and tool execution across infrastructure such as Slack workspaces and Kubernetes clusters. Smith contends that this operational shift exposes an enterprise production chasm where multi-call iterative reasoning loops trigger severe token cost escalation and performance bottlenecks under massive foundation models. Because an agent must invoke multiple sequential inference steps behind the scenes to decompose prompts, select tools, and parse environmental feedback, per-token delays compound exponentially at each hop. If driven by sluggish foundation models, these background chains lead to severe user-experience degradation when scaled across enterprise workloads, such as serving 500 concurrent users within strict millisecond latency thresholds. Infrastructure Barriers of Massive Foundation Models (6:24) Deploying massive frontier models in the 405-billion-parameter range requires substantial monthly baseline financial commitments to maintain even idle or minimally utilized cloud endpoints. While commercial APIs eliminate direct hardware hosting, they leave organizations exposed to escalating per-token pricing and variable third-party latency that cannot be tuned or guaranteed under service-level agreements. Furthermore, Smith emphasizes that regulatory compliance and privacy mandates frequently require organizations to deploy workloads locally on-premises or in private environments. Because procuring and managing compute clusters for 405B+ parameter models is often cost-prohibitive for enterprise IT budgets, models sized at 70B parameters or below represent a far more practical, cost-effective target for enterprise hardware. Compression Mechanics and Accuracy Preservation (8:02) To bypass the trade-off where smaller native models sacrifice reasoning capability, quantization compresses higher-performing models from standard 16-bit floating-point precision (FP16) into lower-precision formats like FP8, INT8, or INT4. This compression can be applied across model weights, intermediate activation layers, and the key-value (KV) cache that stores conversational context in memory, allowing compressed networks to run on smaller GPUs and select CPUs. Citing benchmark evaluations, Smith points out that compressed IBM Granite models retain up to 99% of baseline accuracy across integer and floating-point quantization formats while significantly shrinking disk footprints and inference execution times. Similarly, quantized DeepSeek reasoning models sustain up to 99% accuracy on standard benchmarks, demonstrating that complex reasoning capabilities remain intact when models are properly compressed. Tooling Workflows and Low-Latency Serving Engines (12:38) Achieving near-lossless compression requires specialized machine learning workflows rather than default conversions. Tooling frameworks like LLM Compressor require teams to select targeted layers and utilize domain-representative calibration datasets. These calibration samples ensure tensor transformations do not assign scaling factors arbitrarily, preserving response fidelity across domain-specific prompts. Once compressed, models require optimized open-source inference runtimes such as vLLM, NVIDIA TensorRT-LLM, or SGLang to realize tangible performance gains. These serving engines provide native support for loading quantized weights, directly slashing inter-token latency and time per output token (TPOT) to ensure background agentic reasoning loops complete without stalling concurrent user operations. Smith concludes that the industry assumption of 'bigger is better' fails in production agentic systems, where multi-step execution loops amplify token expenses and latency penalties. While quantized models and optimized inference runtimes resolve these bottlenecks, she cautions that quantization remains a specialized engineering discipline requiring calibration datasets and empirical benchmarking rather than functioning as an effortless, out-of-the-box solution.

精炼阅读摘要

Red Hat AI 与开源倡导者 Taylor Smith 指出，在企业级智能体（Agentic AI）系统中，“模型越大越好”的观念并不适用；采用量化技术压缩后的中小型模型能够在保留原有推理能力的同时大幅削减内存占用、硬件成本与延迟。 智能体落地的瓶颈与大模型局限 (1:48) 从传统的单轮聊天或基础检索增强生成（RAG）转向自主智能体架构，系统需要频繁调用外部工具（如 Kubernetes 集群、Slack 或企业级 API）并进行多步自主规划与推理。Smith 认为当前智能体技术正处于炒作周期的顶峰，绝大多数企业尚未实现真正的生产级部署，其主要障碍在于推理循环带来的成本飙升、多轮调用积累的延迟以及并发扩展困难。 部署数百亿至数千亿参数规模（如 405B）的超大前沿模型需要极高的基础设施底线投入，即便在云端处于低负载或空闲状态也需支付高昂成本；依赖外部商用 API 则面临无法控制的计费增长与网络延迟。此外，受到合规性约束与本地私有化部署要求的企业，往往难以承担 405B 以上模型所需的硬件，因此 70B 及以下规模的模型在企业私有环境中更具现实可行性。 量化压缩机制与多维度显存优化 (8:02) 当工程团队为了缩减延迟与成本而切换至原生小参数模型时，通常会遭遇显著的准确度下降。Smith 建议采用模型量化来打破这一权衡：将高精度参数（如 16 位浮点数 FP16）映射并压缩至低精度格式（如 FP8、INT8 或 INT4），从而在大幅减小显存占用的同时保留大模型的推理与基准精度。 量化不仅可以针对网络核心的模型权重（Weights），还能应用于保存多轮对话历史上下文的键值缓存（KV Cache）以及前向传播中的动态中间计算层（Activations）。这种多层次的压缩使得原本依赖昂贵集群的模型能够在配置更低廉的小型 GPU 上运行，甚至在特定场景下可直接部署于 CPU，为高并发请求提供了更充裕的吞吐空间。 精度保留基准与规范化压缩流程 (10:41) 通过对量化模型实施基准评测，企业可以验证压缩后的性能表现。实测数据显示，IBM Granite 模型在经过不同 INT 与 FP 格式量化后，在显著缩减磁盘占用与推理延迟的前提下，能够保留高达 99% 的原始基准精度；在复杂推理测试中，DeepSeek 量化版本同样维持了高达 99% 的基础推理性能。Smith 同时强调，量化并非普通人员即可随意实施的开箱即用过程，依然需要专业的数据科学与机器学习技术支持。 在工作流工具方面，开发者可借助开源框架 LLM Compressor 推进端到端的压缩流程。工程师首先选定基座模型并确定针对权重或激活层（W&A）的压缩算法，随后必须引入贴近实际生产场景的校准数据集（Calibration Dataset）。校准集能够在映射数值范围时防止算法盲目估算，从而确保张量转换后输出质量与复杂推理逻辑依然保持稳定。 高性能运行时与复合延迟消除 (13:59) 完成量化后的模型需要部署在适配的高性能推理引擎上，例如 vLLM、NVIDIA TensorRT-LLM 与 SGLang，这些运行时原生支持各类 INT 和 FP 压缩格式，能够优化令牌间延迟（Inter-token Latency）、单输出令牌时间（TPOT）及整机吞吐量。面对企业级应用常见的“500 名并发用户”且维持毫秒级 SLA 的场景，运行时的吞吐能力至关重要。 在智能体系统内部，单一用户目标往往会拆解为多轮后台推理、工具调用及环境反馈解析。若底层模型存在推理延迟，每一步的开销将在智能体循环中呈指数级累加；利用量化配合高性能运行时从基座降低单步延迟，是防止复杂智能体工作流因响应迟缓而停滞的关键手段。 Smith 总结认为，随着自主智能体将延迟与算力成本推向新的极限，量化技术结合 LLM Compressor 与 vLLM 等开源工具为企业提供了切实可行的解决方案，使组织无需盲目追逐超大模型即可实现高效扩展；同时社区与开源仓库也已开始提供 Day 0 级别的预量化模型权重支持。

Detailed notes — zh

## Background 该视频由 Ai4 于 2025 年 8 月 27 日发布在 YouTube 平台上，主讲人为 Red Hat 高级 AI 布道师 Taylor Smith。视频简介指出，本次演讲深入探讨了模型量化技术如何帮助开发者在不牺牲模型质量的前提下大幅降低成本，并结合 vLLM 等高性能推理运行时，解决 Agentic 智能体在多步推理与实时交互中的延迟和规模化落地难题。 ## TL;DR 演讲直面企业落地 Agentic 智能体系统时遭遇的调用成本高昂、端到端延迟过大和扩展性受限等现实瓶颈，有力驳斥了盲目崇拜超大参数模型的观念。讲者深入拆解了量化技术对权重、KV 缓存与激活值的压缩机制，并展示了 IBM Granite 与 DeepSeek 在量化后保留 99% 原始精度的基准评测数据。通过 LLM Compressor 工具链配合贴近真实业务的校准数据集，开发者可以实现标准化压缩，并借助 vLLM 等高性能运行时完成极速部署。最后，Red Hat 提供了 Day-0 开源量化模型资源，为企业在私有化和算力受限场景下构建实时智能体提供了高性价比路径。 ## Atmosphere & style 演讲风格务实干练且充满亲和力，讲者开场以幽默自嘲的破冰方式迅速拉近与听众的距离，随后直奔企业生产环境中最残酷的工程痛点。叙述节奏紧凑清晰，避开了华而不实的功能演示，而是以扎实的基准数据、架构流向图和底层工程指标说话。整场分享兼具清醒的行业审视与扎实的技术落地指导，让听众在冷静穿透智能体炒作周期的同时，切实感受到小尺寸量化模型所蕴含的巨大工业实用价值。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=qflAu_ztE1A&t=4s) 破除“模型越大越好”的迷思：在企业私有化部署、合规限制及垂直业务微调场景下，盲目追求超大模型成本极高，小尺寸与量化模型是更理性的工程选择。 - [01:48](https://www.youtube.com/watch?v=qflAu_ztE1A&t=108s) Agentic 智能体架构跃迁：系统从传统的单轮对话与 RAG 演变为具备外部工具调用、内置规划推理以及自主执行闭环三大核心能力的复杂架构。 - [04:38](https://www.youtube.com/watch?v=qflAu_ztE1A&t=278s) 智能体生产落地三大瓶颈：多步推理与反复调用会导致 API/算力成本呈指数级飙升，多环节累积的长延迟严重损害用户体验，并带来高并发下的扩展性困境。 - [06:24](https://www.youtube.com/watch?v=qflAu_ztE1A&t=384s) 私有化部署的算力分水岭：400B+ 超大模型硬件与托管开销巨大，对于有私有 AI 与监管合规需求的企业，70B 及以下模型是更具实操性的选型区间。 - [08:02](https://www.youtube.com/watch?v=qflAu_ztE1A&t=482s) 量化压缩技术的核心原理：量化类似于图像或文件压缩，将 FP16/BF16 等高精度浮点数转为 INT8、FP8 或 INT4，大幅缩减显存占用并提升高并发吞吐。 - [08:02](https://www.youtube.com/watch?v=qflAu_ztE1A&t=482s) 精准多维度的量化对象：量化可分别施加于固有模型权重（Weights）、缓解长文本显存暴涨的 KV 缓存（KV Cache）以及前向传播的激活值（Activations）。 - [10:41](https://www.youtube.com/watch?v=qflAu_ztE1A&t=641s) 实测约 99% 的高精度保持：在 IBM Granite 及 DeepSeek 推理模型的工业级量化基准实测中，量化后在大幅降低延迟的同时保留了原始模型近 99% 的准确度。 - [12:38](https://www.youtube.com/watch?v=qflAu_ztE1A&t=758s) LLM Compressor 工具链与校准数据：量化过程依赖贴近线上真实业务分布的校准数据集持续校验模型响应，而非凭空降精度，从而锁定输出质量。 - [13:59](https://www.youtube.com/watch?v=qflAu_ztE1A&t=839s) 高性能推理引擎对冲累积延迟：配合 vLLM、TensorRT-LLM 或 SGLang 等原生支持丰富量化格式的运行时，能够有效压低 Agent 链式多轮调用的毫秒级时延。 - [16:07](https://www.youtube.com/watch?v=qflAu_ztE1A&t=967s) 量化模型对 Agent 系统的综合价值：显著加速多步推理闭环、以更低显存支撑更高并发、极大改善能效与碳排放，且维持高质量输出水准。 - [16:07](https://www.youtube.com/watch?v=qflAu_ztE1A&t=967s) Red Hat 开源量化资源：开发者无需从零构建量化管道，可直接拉取 Red Hat 社区在业界新模型发布当日（Day 0）上线的开源量化版本快速试验。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=qflAu_ztE1A&t=4s) 破除“模型越大越好”的迷思与会议议程 Red Hat 的 AI 与开源布道师 Taylor Smith 登台开场。她常驻北卡罗来纳州罗利市，在演讲伊始幽默地提及自己是个“狂热的猫奴”以及碧昂丝粉丝团（Beyhive）的一员。本次分享的核心主题聚焦于小型模型（SLMs）、量化模型（Quantized Models）及其在当下的年度流行热词——智能体系统（Agentic AI）中的巨大价值。 长期以来，行业内普遍弥漫着“模型越大越好（Bigger is Better）”的叙事，认为模型参数量越大、性能与效果就必然越优越。虽然在特定全能场景下大模型确有优势，但在实际落地中，许多企业受制于计算资源、数据合规性要求或本地私有化部署（On-Premise）的硬性限制，盲目追求超大基础模型不仅不切实际，而且成本极高。对于需要针对特定垂直业务进行精调（Fine-Tuning）的场景，小尺寸模型往往是更理性、高效的选择。在本次紧凑的演讲中，Taylor 规划了核心议程：深入分析智能体在生产环境中的落地挑战、破除大模型盲目崇拜、用最通俗易懂的语言拆解模型量化的本质，并提供实际的基准测试数据与工具链建议。 ### [01:48](https://www.youtube.com/watch?v=qflAu_ztE1A&t=108s) 从传统对话式 RAG 到 Agentic 智能体的架构跃迁 回顾生成式 AI 的发展起点，最基础的交互形态莫过于在聊天界面（Chatbot UI）中输入 Prompt，由模型结合预训练知识库并辅以检索增强生成（RAG）管道获取上下文，进而预测生成回复。这种单轮或简单多轮的输入-输出闭环在处理日常琐碎任务时已经足够好用，但在面对复杂的企业级商业场景时却显得力不从心。 现代企业期望 AI 能够走出单纯的文本对话框，与真实的系统和工具（如 Slack 工作区、Kubernetes 集群、内部基础设施等）进行深度交互，具备独立决策与采取行动的能力。这催生了智能体（Agentic AI）的爆发式发展。Taylor 笑称，本次大会期间有超过 60 场关于智能体的专题分享，“如果我不做一个关于 Agent 的演讲，我在这里可能就完全跟不上潮流了”。她剖析了 Agentic 系统的三大核心要素： - **工具调用（Tool Utilization）**：赋予大语言模型或生成式模型调用外部 API 与基础设施工具的能力。 - **规划与推理（Planning & Reasoning）**：模型不再是一次性吐出结果，而是通过内置思考链路（Thinking Process）制定自主任务计划。 - **自主执行（Autonomous Execution）**：无需人类在每个细枝末节上微操作（Micromanage），模型自主拆解并推进任务闭环，从而显著提高复杂任务的输出准确度。 ```mermaid flowchart TD accTitle: 从基础 RAG 到 Agentic 架构的演进 subgraph Traditional["传统对话 / RAG 模式"] P1["用户输入 Prompt"] --> RAG["检索增强上下文"] RAG --> LLM1["单次 LLM 推理"] LLM1 --> Out1["预测生成答案"] end subgraph Agentic["Agentic 自主智能体模式"] P2["复杂业务目标"] --> Plan["规划与推理 (Task Planning)"] Plan --> Tools["多工具调用 (Slack / K8s / API)"] Tools --> MultiLLM["多轮 LLM 反复调用与反思"] MultiLLM --> Out2["自主执行并交付结果"] end ``` ### [04:38](https://www.youtube.com/watch?v=qflAu_ztE1A&t=278s) 智能体的炒作周期与生产环境落地三大瓶颈 尽管 Agentic 概念热度空前，Taylor 指出当前技术正处于 Gartner 炒作周期的顶峰（Peak of Hype Cycle），随后必然会经历幻灭的低谷。当前绝大多数团队和客户仍停留在实验探索阶段——例如搭建模型上下文协议（Model Context Protocol / MCP）服务器来验证可行性，真正将复杂智能体工具投入大规模生产环境（Production）的案例仍属凤毛麟角。由于涉及大量的动态组件，智能体系统的可维护性与运维开销极其庞大。 在诸多治理与工程挑战中，企业需要优先解决三大致命瓶颈： - **成本失控（Cost）**：传统交互通常只需一次模型调用，而智能体为完成规划、工具调用、反思修正，会触发多次模型请求与庞大的推理（Reasoning）Token 消耗，导致 API 账单呈指数级飙升。 - **延迟与用户体验（Latency & UX）**：多步调用叠加庞大基础模型的高耗时，使得系统响应极慢，用户面临漫长等待甚至失去耐心放弃交互。 - **扩展性困境（Scalability）**：随着并发用户数增长，高昂的算力消耗和低吞吐使得系统在规模化部署时难以为继。 ### [06:24](https://www.youtube.com/watch?v=qflAu_ztE1A&t=384s) 算力成本与私有化部署下的“大模型困局” 业界新发布的基础模型体积持续成倍扩张。尽管更大的参数规模带来了某些通用能力提升，但伴随而来的基础设施托管成本也达到了令人咋舌的地步。在云端托管顶尖规模的基础模型，即使在完全不考虑多用户高并发、仅满足最基础运行（Minimum Baseline）的前提下，每个月固定产生的基础设施支出也是一笔巨款。一旦面临生产级并发，算力成本会瞬间被无限放大。 ``` +-------------------------------------------------------------------+ | 企业私有化 / 合规部署面临的算力分水岭 | | | | [ 405B+ 参数超大模型 ] ------------> 算力黑洞 (极度昂贵/难以落地) | | [ 70B 及以下模型 ] ------------> 务实落地区间 (成本可控/易优化) | +-------------------------------------------------------------------+ ``` 虽然直接调用第三方云端托管的 API 可以转嫁部分底层运维烦恼，但开发者依然要全额承担高昂的 Token 费用，且完全无法掌控服务端波动造成的长延迟。更重要的是，受到严格行业监管（如金融、医疗等）或有私有 AI（Private AI）硬性需求的企业，必须在本地数据中心（On-Premise）自建环境，面对动辄数百上千亿参数的庞然大物，其硬件采购与维护成本往往让项目直接陷入僵局。Taylor 强调，通常 400B+（如 405B）属于难以驾驭的超大级别，而 70B 及以下参数规模才是更务实、合理的选型区间。 ### [08:02](https://www.youtube.com/watch?v=qflAu_ztE1A&t=482s) 量化技术原理：如何以极小精度代价压缩模型？ 面对“模型变大带来高成本与高延迟，模型变小导致准确度断崖式下降”的权衡博弈，量化（Quantization）成为破局的核心关键。Taylor 用通俗的比喻解释道：量化的本质类似于文件压缩，就像将无损原始图片（Raw Image）转换为 JPEG 格式，或者将大型文件夹打包成 ZIP 压缩包一样，在大幅削减体积的同时尽可能保留核心内容。 与直接采用小参数模型（可能牺牲过多的固有推理能力）不同，量化技术旨在**保留超大模型的先验精度，同时大幅缩减模型的整体显存占用（Memory Footprint）**。其核心机制是将原本高精度的浮点数表示进行降精度压缩——例如从 16 位浮点数（FP16/BF16）降低为 8 位整数（INT8）、8 位浮点（FP8）甚至 4 位整数（INT4）。在量化管道中，算法可以针对模型的不同组成部分实施精准压缩： - **权重（Weights）量化**：直接压缩模型自身的固有静态参数。 - **KV 缓存（KV Cache）量化**：针对存储多轮上下文与长对话历史的键值缓存进行压缩，极大释放长文本运行时的显存暴涨。 - **激活值（Activations）量化**：压缩计算过程中的前向传播中间计算层。 量化模型不仅能显著降低内存与显存门槛，甚至使得在更小规模的硬件乃至普通 CPU 上高效运行大模型成为可能。基准测试表明，量化模型在高并发环境下展现出了极佳的吞吐性能，能够以更低的延迟和更高的 Token 生成速率稳定应对真实生产负载。 ### [10:41](https://www.youtube.com/watch?v=qflAu_ztE1A&t=641s) 模型基准测试与精度保持：IBM Granite 与 DeepSeek 实测 在完成量化模型的构建或引入之后，严谨、规范且明确的基准测试是不可或缺的环节。评估指标主要涵盖三大维度：字词间延迟（inter-token latency）、每输出 Token 耗时（time per output token），以及端到端延迟和吞吐量。除了速度层面的指标，模型精度的对比测试同样必须同步进行。 演讲者展示了一组针对 IBM Granite 模型进行量化的基准数据。从图表左侧的模型体积差异可以看出，无论采用 INT 还是 FP 等不同的量化格式，量化后的模型与原始基础模型相比，均能维持高达 99% 的精度，而图表最右侧则直观呈现了推理时延的大幅缩减。 紧接着，演讲者进一步展示了广受欢迎的推理模型 DeepSeek 在经过量化后的推理能力评测。尽管基准测试图表看起来略显枯燥，但数据非常确凿：在绝大多数基准测试场景下，量化模型都保留了原始模型 99% 的准确度水平。演讲者特别指出，这种精度的保持绝非随手可得，“比如让 HR 去量化一个模型，显然不可能一上来就获得最理想的效果”；这属于典型且专业的机器学习与数据科学任务，但只要有相应专业背景的人才投入，99% 的精度保留是完全可以达成的工业级预期。 - **核心基准测试指标**：端到端延迟、字词间延迟（inter-token latency）、每输出 Token 时间、系统整体吞吐量。 - **精度保持水平**：在 IBM Granite 及 DeepSeek 推理模型的多格式量化（INT / FP）实测中，均实现了约 99% 的精度保留。 --- ### [12:38](https://www.youtube.com/watch?v=qflAu_ztE1A&t=758s) LLM Compressor 工具链与量化校准工作流 为了帮助开发者和数据科学家标准化地实施量化，LLM Compressor 提供了成熟的框架与工具集，内置对多种量化算法的支持。实施量化的标准流程始于选定初始模型，随后根据拟压缩的目标层（例如仅针对权重，或是权重与激活值同时压缩，对应 INT8、W4A16 等不同精度设定）挑选适宜的量化算法。 该工作流中最为关键的一步是选择“校准数据集”（calibration dataset）。校准数据必须尽可能贴近模型最终在实际业务中所面临的真实数据特征，模拟大语言模型在线上部署时的真实交互体验。量化算法并非靠凭空猜测（“willy-nilly”）来粗暴降低数值精度，而是依靠该数据集持续校验模型的响应表现，确保量化过程校准得当，并在最大程度上锁定并维持模型原有的输出精度。 ```mermaid flowchart TD accTitle: LLM Compressor Quantization Pipeline subgraph S1["准备阶段"] A["选定初始模型 (Base Model)"] B["选择校准数据集<br/>(接近真实业务分布)"] end subgraph S2["算法配置与压缩"] C["明确量化目标<br/>(权重/激活值/精度位数)"] D["应用对应量化算法<br/>(通过 LLM Compressor)"] end subgraph S3["验证与输出"] E["基于校准数据验证精度响应"] F["导出量化模型成品"] end A --> C C --> D B --> E D --> E E --> F ``` --- ### [13:59](https://www.youtube.com/watch?v=qflAu_ztE1A&t=839s) 高性能推理引擎对比与 Agent 场景下的低延迟刚需 生成量化模型后，必须依托高效的推理运行时（Inference Runtime）进行加载与部署。当前市场上存在多种开源与商业化引擎，其中 vLLM 是兼具极高性能与开源生态代表性的运行时。将量化模型加载至 vLLM 后，即可针对性开展端到端延迟、Token 生成时延及精度的实测。 vLLM 的核心优势之一在于其对极其广泛的量化格式的原生支持。演讲者指出，这种广谱兼容性并非所有运行时的标配，不过像 NVIDIA 的 TensorRT-LLM 以及 SGLang 等主流高性能运行时，也同样提供了对丰富量化格式的良好支持。在现场对比演示中，未量化模型与量化模型被置于同等 Prompt 输入下进行实时侧对侧（Side-by-side）较量。 虽然展示延迟测试数字的 Demo 不如花哨的功能演示那样具有视觉冲击力，但对于企业级客户而言，这类硬核指标关乎生死。企业往往提出极其严苛的要求，例如必须支撑 500 个并发用户，且响应必须控制在指定的毫秒级阈值内。在 Agentic 系统中，Agent 的工具调用、多轮规划与反思几乎全在后台以链式或循环管道的方式进行，各个环节累积的时延十分惊人。如果每个节点的响应无法大幅压缩，整个系统的端到端延迟就会迅速失控崩溃，因此“必须尽可能地削减系统延迟”。 --- ### [16:07](https://www.youtube.com/watch?v=qflAu_ztE1A&t=967s) 总结：量化模型对 Agent 系统的价值与 Red Hat 现成资源 演讲者全面归纳了采用低内存占用、高能效的量化小尺寸模型所带来的核心商业与工程价值： - **更快的 Agentic 闭环**：多步推理与工具调用的串行等待时间大幅缩短。 - **更高的并发容量与更低成本**：显存占用减少使得单节点能够支撑更高并发（concurrency）。 - **极高的能效表现**：大幅削减能源消耗与碳排放，迎合企业严苛的算力能耗诉求。 - **不妥协的精度表现**：在获得高速低耗的同时，仍旧维系堪比原始大模型的输出水准。 演讲者向与会者强调，企业和开发者并不一定非要亲自从零执行上述量化流程（除非涉及私有领域的自研或微调模型）。Red Hat 团队维护了一个开源的量化模型仓库，业界流行的新模型一旦推出，团队通常会在“Day 0”（发布当日）迅速上线对应的量化版本，供社区直接拉取和实验。 最后，演讲者重申了核心论点：“在现代复杂系统中，大并不等同于好（Bigger does not equal better）。”面对智能体系统中前所未有的延迟与成本路障，借助 LLM Compressor 与 vLLM 等成熟工具部署量化模型是极具性价比的落地路径。演讲者展示了包含讲义与 LinkedIn 联系方式的二维码，并邀请现场正在构建 Agent 应用的参会者前往 Red Hat 展台进一步交流经验与实际业务痛点。 ## Notable quotes > [01:48](https://www.youtube.com/watch?v=qflAu_ztE1A&t=108s) 如果我不做一个关于 Agent 的演讲，我在这里可能就完全跟不上潮流了 — Taylor Smith > [16:07](https://www.youtube.com/watch?v=qflAu_ztE1A&t=967s) Bigger does not equal better — Taylor Smith > [10:41](https://www.youtube.com/watch?v=qflAu_ztE1A&t=641s) 比如让 HR 去量化一个模型，显然不可能一上来就获得最理想的效果 — Taylor Smith > [13:59](https://www.youtube.com/watch?v=qflAu_ztE1A&t=839s) 必须尽可能地削减系统延迟 — Taylor Smith ## Mentioned Red Hat, Taylor Smith, IBM Granite, DeepSeek, LLM Compressor, vLLM, TensorRT-LLM, SGLang, NVIDIA, Kubernetes, Slack, Model Context Protocol ## Video description Taylor Smith, Senior AI Advocate, Red Hat Abstracts LLMs now drive agents that handle multi-step reasoning, tool use, and user interaction—but many struggle to scale those systems affordably. In this session, we’ll dive into quantization: a critical technique that enables deployment of fast, cost-effective models without sacrificing output quality. The talk will illustrate how different quantization methods compress large LLMs and SLMs while preserving accuracy, and how quantized models improve inference throughput in agentic pipelines. We’ll explore real-world use cases and guide you through inference runtimes like vLLM that make quantized LLMs practical for low-latency, high-scale deployments. [Source](https://www.youtube.com/watch?v=qflAu_ztE1A)

Source quotes

[1:48] Taylor Smith: 如果我不做一个关于 Agent 的演讲，我在这里可能就完全跟不上潮流了 [16:07] Taylor Smith: Bigger does not equal better [10:41] Taylor Smith: 比如让 HR 去量化一个模型，显然不可能一上来就获得最理想的效果 [13:59] Taylor Smith: 必须尽可能地削减系统延迟

Source quotes

[0:04] Taylor Smith: bigger is better [0:04] Taylor Smith: hot buzzword of the year [2:50] Taylor Smith: dip [10:41] Taylor Smith: boring graphs [10:41] Taylor Smith: If, I don't know, the HR person wants to go and quantize a model... they're not going to get the best output right away. [12:38] Taylor Smith: guessing willy-nilly [13:59] Taylor Smith: 500 concurrent users [16:07] Taylor Smith: Bigger does not equal better.

Original transcript

[0:04] Um, I am Taylor Smith. I work at Red [0:07] Hat. And we're going to talk about small [0:09] models, quantized models, and why they [0:12] are beneficial and helpful, especially [0:14] when we're thinking about agentic AI [0:16] systems, the hot buzzword of the year. [0:20] So, we're going to tackle this bigger is [0:23] better narrative because it's still very [0:25] much what people think. I think in a lot [0:27] of cases that the bigger the model is, [0:30] the better it is. And in some cases, [0:32] that's true. But in a lot of cases, we [0:34] can't or it doesn't make sense to use [0:37] those huge foundation models. Maybe we [0:39] need to host the models ourselves on [0:41] premise or we need models that are more [0:43] fine-tuned to specific use cases. [0:47] that will lend itself to needing more of [0:49] those smaller scale models for our [0:51] specific use case. But a little bit [0:53] about me first. So like I said, I work [0:54] at Red Hat. I am an AI and open-source [0:57] advocate at Red Hat, working with our AI [0:59] product portfolio. I am based in [1:02] Raleigh, North Carolina. Really into [1:04] cats. Love a good cat. And member of the [1:07] Beehive represent. Okay. [1:11] So today's schedule we are going to be [1:13] talking about agents in production and [1:15] the challenges that are faced there [1:17] especially as we're trying to scale up [1:19] and scale out why bigger isn't always [1:22] better and talking about those smaller [1:23] models quantized models explaining to [1:26] you and hoping you get a sense of what [1:28] quantization is and what it's beneficial [1:31] for in as plain English as possible [1:33] today and we'll give you some benchmark [1:36] data etc and you'll see this um in real [1:38] time as well and I'll hopefully give you [1:41] some tools that you can take and look [1:43] into this a little bit more after the [1:45] talk since we only have a few minutes [1:46] together today. [1:48] So this is where we all started, right? [1:50] And a lot of us are still in this space [1:53] where we have our prompt in a chatbot [1:56] user interface. The model consumes the [1:58] context that we share with it. It has [2:00] its own level of knowledge based on its [2:03] pre-training. Maybe you have a rag [2:06] pipeline that's giving it some [2:07] additional context, but there isn't much [2:09] outside of this workflow. It receives [2:12] our context and our question and it [2:14] gives us a predicted generative answer [2:18] in response to that. This is good and [2:21] fine in a lot of cases. It's good for me [2:23] in a lot of cases when I'm doing, you [2:25] know, various little tasks. But when we [2:27] start to approach these complex business [2:30] level use cases, maybe we want the AI to [2:33] be able to interact with different [2:35] tooling like our Slack workspace or [2:38] maybe our Kubernetes cluster, things [2:40] like that. We want to give it access to [2:42] these different tools to be able to also [2:45] do actions and operate a little bit more [2:47] independently. So that's where this [2:50] Agentic rise has come into play. There [2:53] are about 60 over 60 agentic sessions um [2:58] over the next couple of days. So it's [3:00] definitely as we all know a huge topic. [3:02] Everybody's talking about agents. I felt [3:05] that if I did not um produce an agent [3:07] talk I would not be relevant here. So [3:09] here we are. Um the main components of [3:12] what an agentic AI system is comprised [3:15] of. This is a little bit people seem to [3:17] kind of say different things when they [3:19] talk about agents or they throw the word [3:21] out and they don't maybe know what they [3:22] mean. Generally speaking, tool [3:25] utilization is a big one. So, giving [3:27] your model, your AI model, large [3:30] language model or generative AI model [3:32] access to different tools like I was [3:34] saying, maybe Slack, maybe whatever [3:36] infrastructure that you have set up. The [3:39] ability to plan and reason as well. So [3:43] if you've interacted with some agentic [3:45] like code extensions or things like that [3:48] um or even different reasoning models [3:50] you'll see that it goes through this [3:51] thinking process. It's able to make um [3:53] in some cases a task plan for itself and [3:56] it's able to operate a bit autonomously. [3:59] Um you're not having to micromanage [4:02] every step of its process and this leads [4:05] to generally also more accurate output. [4:08] However, it does mean there are a lot of [4:10] different components now involved and a [4:12] lot of time and resources involved. It [4:15] might be making multiple large language [4:17] model calls out. You're experiencing a [4:20] wait period now at this point. You've [4:22] probably experienced this with um [4:24] whatever you know foundation model chat [4:26] interface. There's some semblance of [4:28] this now. So, there's definitely more [4:30] complexity here, but it's more relevant [4:32] for complex tasks that we are facing in [4:35] the in the enterprise. [4:38] So in terms of hype cycle, like I said, [4:41] there are tons of sessions on this this [4:42] week. We are at the top of the hype [4:44] cycle. We're going to experience a dip, [4:46] right? We are in the midst of learning [4:48] about what Agentic can do and trying [4:51] things out. There are tons of customers [4:54] and people doing Agentic [4:56] experimentation, maybe building out [4:58] model context protocol servers to kind [5:00] of just get things rolling and [5:02] experiment what they can do. But the [5:04] reality is that most people are not in [5:07] production with Agentic tooling in this [5:10] moment. Um, and there are a lot of [5:12] challenges with Aentic as well because [5:15] of all of those different components [5:17] involved. Um, the maintainability, [5:20] uh, the overhead required to get that up [5:23] and running, especially at scale, [5:25] especially in production. We're a ways [5:27] out from understanding how to do that. [5:29] Well, um there's so there's various [5:32] those kind of governance concerns, but [5:35] there are a few concerns we can start to [5:37] address immediately and start to kind of [5:39] talk about and tackle more quickly. [5:41] Cost. So you're you like I mentioned [5:43] there might be multiple large language [5:45] model calls out now at this point versus [5:48] just one. You're using reasoning [5:50] capabilities. There's a lot more um [5:53] token usage happening within that [5:55] process now which means cost is going to [5:57] skyrocket. Latency and the user [6:00] experience with low latency is another [6:02] big concern. Um especially if you're [6:05] using a larger beefier model and you [6:07] have all these different steps. How can [6:09] we get that latency down as much as [6:11] possible so that if it's a user [6:13] interaction type of experience they [6:15] aren't like [6:16] waiting and annoyed and done. Um and [6:19] then the ability to scale with those [6:21] considerations as well. [6:24] So the challenges of is bigger is is [6:28] bigger better what's the deal with like [6:29] is do I have to use these huge [6:31] foundation models we are still seeing [6:34] with GPT as an example but every single [6:36] foundation model that's released it is [6:38] exponentially bigger than the last one [6:40] there are benefits to that but there are [6:43] certainly drawbacks to that as well [6:45] mostly in the cost area. Um so this is [6:50] an example if you are hosting these size [6:52] models on cloud infrastructure these are [6:55] some of the monthly costs that you might [6:57] expect. Um and this is not accounting [6:59] for any level of concurrency. This is [7:02] just at minimum you need to set this up [7:05] for very basic usage. So this can of [7:08] course increase quite a bit when you're [7:10] dealing with um many concurrent users at [7:12] a time making requests. [7:15] If you are just consuming from hosted [7:19] foundation models, there's some of this [7:21] that you are maybe less concerned about, [7:24] but you're still bearing the brunt of [7:26] the token cost and then not being able [7:28] to fully control the latency um of the [7:32] experience as well when you're relying [7:33] on a hosted service. Um, and if you have [7:36] regulatory concerns or you need a [7:38] private AI setup, you have to figure [7:41] this out on prem and you have to figure [7:43] out, oh, I got to host this huge model. [7:45] What's my infrastructure cost going to [7:47] be over x amount of time, etc. There's [7:50] it becomes more complex more quickly, [7:51] especially when you're dealing with [7:53] these humongous models. I would say for [7:56] 450 being the humongous, 70 and below [7:59] being in the more reasonable size. [8:02] So most people have come to terms with [8:04] this and realize okay I need to start [8:06] looking at smaller models. We are seeing [8:08] many small models released out there in [8:10] the wild today and there are always [8:13] going to be trade-offs. As you increase [8:15] the size of the model that's going to [8:17] increase cost, increase latency. As you [8:20] decrease the size of the model you're [8:22] going to have an accuracy tradeoff by [8:24] default. So you have to balance how do [8:27] we deal with those trade-offs and get [8:29] the best output that we possibly can for [8:32] what our use case is. So that's where [8:34] quantization comes into play. So [8:37] quantization you can think of it's [8:39] simply a way of compressing the model [8:41] files. So like a raw image to a JPEG or [8:45] a zip file. It's this kind of just [8:47] compression idea. And the goal here with [8:49] quantization instead of just using a [8:52] small parameter size model is that [8:54] you're preserving the accuracy of a [8:56] larger model but reducing the overall [8:59] memory footprint footprint required by [9:02] the model. So you compress the large [9:05] language models with various [9:07] quantization algorithms and you can [9:09] target kind of specific areas of the [9:12] model for that compression um [9:15] compression quantization. You can [9:17] quantize the weights, so the parameters [9:20] of the model itself. You can quantize [9:23] the KV cache, which is like the [9:25] conversation memory. And you can [9:27] quantize the activations and which is [9:29] like um intermediate calculation layer. [9:32] You can quantize multiple of these and [9:35] there's different algorithms and [9:37] approaches to do so. But the goal is [9:40] compress. So you're looking at these [9:42] floating point numbers, right? and [9:43] you're going from a 16 bit floating [9:45] point to an 8 bit floating point as an [9:48] example. So it's simply just reducing [9:50] the pre precision of each of those [9:52] numerical values within the model. And [9:54] the goal is to preserve accuracy as much [9:57] as you possibly can in that process. [10:00] So, some of the trade-offs or kind of [10:03] explaining the differences between using [10:05] a small model from the get-go versus [10:07] quantization, it's largely going to be [10:09] around the memory footprint and the [10:12] speed and the accuracy, which you can [10:15] see here in this table. And quant the [10:18] quantized models are much more friendly [10:20] to smaller hardware sizes as well, [10:22] including CPU in some cases. [10:26] So, in terms of impact, there are some, [10:28] you know, graphs. It's nothing amazing [10:30] to look at. Generally speaking, as [10:32] concurrency increases, um it handles [10:35] latency and output tokens and better and [10:38] faster essentially. And we do have we do [10:41] very exp or you should do if you're [10:42] doing this independently, very um [10:44] prescriptive and explicit testing of [10:46] these models as well. Um in and there's [10:49] various benchmarking tests you can do to [10:51] measure things like inter token latency, [10:53] time per output token, overall endto-end [10:56] latency and throughput. And you do these [10:58] benchmarks after you've created or built [11:01] or consumed your quantized model. And [11:03] then you also do the same accuracy [11:05] benchmarks as well. So this is an [11:08] example that shows more of the accuracy [11:10] and as well as the benchmarks on the far [11:13] left side for the inference speed. So [11:15] this is an example of a granite IBM [11:18] granite model that's been quantized. So [11:20] you can see the difference in size on [11:22] the far left hand side. You can see that [11:24] it's maintained 99% accuracy. um with [11:28] the different quantization formats the [11:30] int and the FP those are different [11:32] quantization formats um so you can see [11:35] that it's maintained accuracy up up to [11:38] the base model that was quantized and [11:41] then the inference time on the far right [11:44] hand side [11:48] so this particular chart is showcasing [11:50] the reasoning capabilities of deepseek [11:54] so deepseeek being a very popular [11:55] reasoning model. We quantized this [11:58] particular reasoning model and these are [12:00] the results of the reasoning [12:02] capabilities in those benchmark tests. [12:04] So you can see generally speaking, I [12:05] know it's like boring graphs, um, but [12:08] it's do it we're able to maintain up to [12:10] 99% in most cases of that accuracy level [12:13] and that's what you want and should [12:15] expect. It's not necessarily easy to do [12:17] that out of the gate. Like if I don't [12:19] know the HR person wants to go and [12:22] quantize a model like that's not [12:24] necessarily they're not going to get the [12:25] best output right away. It's certainly [12:27] an ML like data science task but the [12:31] expectation is if we have that level of [12:33] talent behind the task that this is what [12:34] they're able to achieve with [12:35] quantization. [12:38] LLM compressor is a specific framework [12:41] that you can use to help you along this [12:44] process. So it supports multiple [12:46] quantization algorithms and it gives you [12:50] a framework and tool set to use. The [12:52] general workflow is you have your [12:54] starter model that you've chosen to [12:55] quantize. You um pick an algorithm. So [12:59] depending on what layer you're looking [13:01] to quantize. Remember I mentioned the [13:03] weights and activations and the [13:05] different sizing like int w all these [13:07] numbers. Um you kind of choose what is [13:09] the best target there and you choose the [13:11] algorithm that supports that. You apply [13:14] that algorithm. You choose a um [13:16] calibration data set which is a data set [13:19] that you have to choose that's close to [13:21] whatever your use case data is going to [13:24] be. So whatever your use case might be [13:26] um data set that mimics what that [13:29] experience will be with the large [13:30] language model. And that calibration [13:33] data set is used within the quantization [13:36] algorithm process to be able to make [13:39] sure and check that it's responding [13:42] appropriately. It's not just guessing [13:44] willy-nilly um about like oh this seems [13:47] like a lower precision that's good. It [13:50] is using that calibration data set to [13:52] ensure that it is um calibrating [13:54] appropriately and maintaining accuracy [13:56] as much as possible. [13:59] So once you have your model, you need a [14:02] way to run your model and load your [14:04] model and inference your model. There [14:06] are several inference runtimes on the [14:08] market today, both proprietary and open [14:10] source. VLM is an example of an [14:13] open-source um higherforming inference [14:15] runtime. So when you have your quantized [14:18] model, you would load it into something [14:20] like VLM and you could do your benchmark [14:23] testing and your accuracy benchmarks at [14:25] that point as it's um loaded with VLM. [14:29] The main point I want to call out about [14:30] VLM is its support of a broad range of [14:33] quantization formats. um which is not [14:36] the case with every inference runtime, [14:37] but for some of the other very popular [14:40] ones like TRTLM um from NVIDIA and SG [14:43] lang um they also support a wide range [14:46] of quantization formats, but it's not [14:48] necessarily a given. Um but basically [14:51] what I'm trying to show here is [14:52] literally the latency comparisons in [14:54] real time. So I have a prompt input. I'm [14:59] measuring endto-end latency, inter token [15:01] latency, and throughput. Um, and I'm [15:03] comparing a quantized model and a not [15:05] quantized model side by side so you can [15:07] see in real time um the performance [15:10] differences there. Um, but I'm showing [15:12] the difference comparisons there because [15:14] it's not necessarily like a fun demo to [15:17] show of like exactly the the amazing [15:20] implications. I mean, we're talking [15:21] about, you know, businesses that need, I [15:23] don't know, 500 concurrent users and [15:25] they need this amount of milliseconds of [15:27] response. Like, these are the types of [15:29] engagements we're getting asked about [15:30] and working on. [15:32] um they need to have us prove that they [15:35] can reach those um specific uh [15:38] performance targets, but it's hard to [15:40] see sometimes. We just kind of can show [15:42] graphs and we can you can get a visual [15:44] of um the chat output, you know, coming [15:47] out more quickly. But when we think [15:49] about agentic systems, a lot of these [15:51] actions are happening in the background [15:53] and the speed is happening in the [15:54] background and it's very critical [15:56] because of all these different steps [15:58] taking place in these agentic pipelines. [16:00] um that adds up very quickly. So we want [16:03] to reduce the amount of latency as much [16:05] as possible. Okay. So to kind of [16:07] summarize just the general value of [16:09] using these smaller um memory efficient [16:12] quantized models, you have these faster [16:14] agentic loops generally lower cost. [16:17] You're able to handle more concurrency [16:19] at a given time, much more energy [16:21] efficient as well, which is a huge [16:23] concern for many, and you also preserve [16:25] that high accuracy that you would expect [16:27] in a larger sized model. [16:30] I wanted to point out as well, you don't [16:32] have to go through this quantization [16:33] process yourself. You would if you are [16:36] fine-tuning your own models or building [16:37] your own models and things like that, [16:39] but we also have an open- source um [16:41] repository of quantized models that you [16:44] can pull from and experiment with. And [16:46] we release um the quantized versions of [16:50] popular models as they come out, usually [16:52] with day zero support. So, that's [16:54] something cool to check out. I have a um [16:55] a QR code for the slides after this as [16:57] well so you can get all this [16:58] information. [17:00] But general takeaways um agents face [17:04] latency and cost roadblocks that we did [17:07] not experience before in more simple AI [17:09] systems. Bigger does not equal better. [17:12] So looking at these quantized models [17:14] depending again on what your use case is [17:17] is definitely going to be better in in [17:19] most cases than even the smaller size [17:21] parameter models. Um, and we have tools [17:24] to make this easier. Tools like LLM [17:26] compressor to help you get started with [17:28] the quantization process and VLM [17:30] inference runtime that supports a myriad [17:32] of quantization formats. [17:36] So, I have the QR code uh for the slides [17:39] and then for LinkedIn, I'm very curious [17:41] is a Red Hat has a booth this week. I'm [17:44] very curious to talk with anybody who's [17:46] working on Agentic use cases. We're [17:47] trying to kind of gather a lot of this [17:49] information. So, if anybody wants to [17:51] talk to me about what they're doing with [17:52] agents, happy to talk to you later. [17:55] Thank you.