FriendliAI Cuts Agentic Model Costs in Half via Continuous Batching

Scaling Inference for Agentic AI With FriendliAI — Ai4

Ai4

With inference consuming up to 90% of lifetime AI budgets, multi-step agentic workloads require token-level continuous batching and dynamic infrastructure to prevent latency bottlenecks and GPU cost overruns.

AI-assisted summary of Scaling Inference for Agentic AI With FriendliAI — original source. Check the original for full context.

Source published: 2025-08-24

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 24, 2025, this presentation features Byung-Gon Chun, CEO of FriendliAI, addressing how to overcome inference bottlenecks in agentic AI. According to the page description, the Friendli Suite simplifies the deployment and operation of complex AI agents by providing scalable, high-performance endpoints without requiring custom backend infrastructure.

## TL;DR As generative AI evolves into autonomous agentic systems characterized by multi-turn reasoning, external tool calls, and model routing, inference emerges as the primary bottleneck, consuming 80% to 90% of lifetime compute budgets. To tackle spiraling GPU expenses and latency issues, FriendliAI introduces an optimized inference stack powered by continuous batching, speculative decoding, and instant on-the-fly quantization. The solution is delivered through flexible deployment modalities—including one-click Friendli Endpoints integrated with Hugging Face, a Serverless API, private VPC/on-premises Friendli Containers, and the Friendli Agent orchestration layer. Real-world enterprise validations, including Twelve Labs cutting compute costs by over 50% and LG Electronics deploying their 32B EXAONE model in under 24 hours, demonstrate substantial gains in throughput and cost efficiency.

## Atmosphere & style The presentation carries the authoritative, pragmatic tone of an enterprise tech keynote that bridges academic computer systems research with real-world infrastructure engineering. Chun speaks with calm, methodical clarity, moving deliberately from systemic industry pain points and GPU economics to live platform demonstrations and architectural breakdowns. Watching it feels like a concise, no-nonsense masterclass designed for engineering leaders seeking to turn fragile AI agent prototypes into scalable, production-grade systems.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=zl6FizzYC4M&t=4s) Generative AI is rapidly shifting from static conversational chatbots to autonomous agentic systems capable of multi-step reasoning, planning, stateful memory, and tool execution. - [00:04](https://www.youtube.com/watch?v=zl6FizzYC4M&t=4s) In real-world enterprise deployments, live model inference accounts for 80% to 90% of an AI model's total lifetime compute expenditure. - [00:04](https://www.youtube.com/watch?v=zl6FizzYC4M&t=4s) Agentic workflows place heavy pressure on serving infrastructure due to multi-turn interactions, round-trip tool calls, and multi-model routing that compound latency. - [02:40](https://www.youtube.com/watch?v=zl6FizzYC4M&t=160s) Enterprise AI pilots often fail to scale when engineering teams become bogged down reinventing custom backend infrastructure instead of developing core product features. - [02:40](https://www.youtube.com/watch?v=zl6FizzYC4M&t=160s) Production-ready agentic infrastructure must deliver ultra-low time-to-first-token (TTFT), rapid dynamic auto-scaling, high throughput, and private VPC deployment options. - [05:17](https://www.youtube.com/watch?v=zl6FizzYC4M&t=317s) Continuous batching, invented by the FriendliAI team in their OSDI 2022 Orca paper, eliminates idle GPU cycles by inserting and evicting requests at token-level granularity. - [05:17](https://www.youtube.com/watch?v=zl6FizzYC4M&t=317s) The Friendli Inference engine integrates advanced optimizations including speculative decoding, optimized KV-cache management, and zero-setup on-the-fly quantization. - [07:23](https://www.youtube.com/watch?v=zl6FizzYC4M&t=443s) Friendli Endpoints integrates natively with Hugging Face Hub to enable one-click deployments of over 423,000 models, delivering over 500 tokens per second for Llama 3.1 8B on NVIDIA Hopper GPUs. - [11:34](https://www.youtube.com/watch?v=zl6FizzYC4M&t=694s) Deployment options range from a managed Serverless API to dedicated Endpoints and self-hosted Friendli Containers packaged as native Amazon EKS add-ons. - [11:34](https://www.youtube.com/watch?v=zl6FizzYC4M&t=694s) Friendli Agent sits directly atop the inference engine to orchestrate multi-model workflows, session memory, and external API tool calls with minimal latency overhead. - [13:03](https://www.youtube.com/watch?v=zl6FizzYC4M&t=783s) Enterprise case studies demonstrate significant operational efficiency, including Twelve Labs cutting infrastructure costs by more than 50% and LG Electronics deploying their custom EXAONE model in under 24 hours.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=zl6FizzYC4M&t=4s) From Prototypes to Production: The Rise and Cost of Agentic AI

Generative AI is shifting rapidly from static tools like copilots and simple conversational chatbots into active, autonomous agentic systems. Rather than merely offering single-turn responses, agentic AI operates as an active problem solver capable of reasoning, planning, maintaining memory across multi-step interactions, and executing external tool calls. This transformation is further accelerated by the surging adoption of open-source foundational models such as Llama, DeepSeek, and Qwen, which allow enterprises to fine-tune and achieve custom domain functionality at significantly lower initial development costs.

```mermaid flowchart TD accTitle: Agentic AI Inference Demands

subgraph AgentLayer ["Agentic AI Workflow"] A["User Goal / Task"] --> B["Planning & Reasoning"] B --> C["Memory & Session State"] C --> D["Multi-Step Tool Calls & Model Routing"] end

subgraph InfraDemands ["Infrastructure Requirements"] D --> E["High-Throughput Serving"] D --> F["Ultra-Low Latency"] D --> G["Dynamic Cost Optimization"] end ```

While creating a proof-of-concept AI agent is straightforward, the real business value emerges only when agents transition into production. In real-world enterprise deployments, inference represents the live touchpoint where customers interact with models, accounting for 80% to 90% of an AI model's total lifetime expenditure. Agentic workloads place unprecedented pressure on serving stacks because they involve longer multi-turn sessions, complex multi-model orchestrations, and frequent round-trip tool integrations, demanding tight latency budgets, massive throughput, and extreme cost efficiency.

- **Open-source surge**: Widespread adoption of base and fine-tuned open-source architectures like Llama, DeepSeek, and Qwen. - **Inference cost share**: Serving and operational inference consume 80% to 90% of total lifetime AI compute budgets. - **Agentic requirements**: High throughput, ultra-low response latency, and strict cost containment across multi-step reasoning cycles.

---

[02:40](https://www.youtube.com/watch?v=zl6FizzYC4M&t=160s) Production Pitfalls and Business Priorities for Agentic Infrastructure

Many enterprise AI initiatives begin with promising pilot programs but stall completely when scaled to real users. Typical operational failure modes include delayed product launches, spiraling GPU bills caused by over-provisioning for unpredicted bursty traffic, architectural complexity, security and data privacy hurdles, and slow inference latencies that lead directly to user churn. Product engineering teams frequently find themselves bogged down trying to "reinvent infrastructure instead of building product features."

To overcome these barriers, business leaders and executives require a standardized set of operational capabilities from their inference layer:

---

[05:17](https://www.youtube.com/watch?v=zl6FizzYC4M&t=317s) FriendliAI Engine: Continuous Batching and Optimization Innovations

FriendliAI was built from the ground up to solve these production bottlenecks by making model inference fast, scalable, and cost-efficient. The core engine—Friendli Inference—combines novel research innovations with industry best practices across language and multimodal models. It incorporates a suite of advanced optimizations including parallel model execution, KV-cache management, optimized scheduling, automatic quantization, and speculative decoding.

```mermaid flowchart LR accTitle: Friendli Inference Platform Stack

subgraph CoreEngine ["Friendli Inference Engine"] A["Continuous Batching (Orca OSDI 2022)"] B["Speculative Decoding & Caching"] C["On-the-Fly Quantization"] end

subgraph Delivery ["Deployment Formats"] CoreEngine --> D["Friendli Endpoints (Dedicated Cloud)"] CoreEngine --> E["Friendli Serverless API"] CoreEngine --> F["Friendli Container (EKS / On-Prem)"] end ```

A foundational breakthrough powering the engine is **continuous batching**, a pioneering technique invented by the FriendliAI team and introduced in their Orca paper at OSDI 2022. Because real-world inference traffic arrives irregularly and each generation sequence finishes at different times, traditional static batching results in idle GPU cycles. Continuous batching solves this by dynamically inserting incoming requests into an ongoing batch and evicting completed sequences at token-level granularity. This maintains near-total GPU compute utilization regardless of uneven request dynamics.

---

[07:23](https://www.youtube.com/watch?v=zl6FizzYC4M&t=443s) Friendli Endpoints: One-Click Hugging Face Integration and High-Speed Deployment

Friendli Endpoints packages the underlying inference engine into a fully managed serving service featuring intelligent traffic routing, automated scaling, real-time metrics (tokens processed, latency traces), model versioning, live rolling updates, and 99.99% uptime availability. A notable feature is zero-setup "on-the-fly quantization" and speculative decoding, which allows users to select optimized quantization schemes with a single click without running external compression tooling or degrading output accuracy.

Through native platform integrations with Hugging Face Hub and Weights & Biases, users can pull models into Friendli or deploy straight from model repositories:

- **Model versatility**: Instant, one-click deployment for over 423,000 models cataloged on Hugging Face (out of nearly 2 million total available models). - **Throughput performance**: Generates over 500 tokens per second for Llama 3.1 8B on an NVIDIA Hopper GPU. - **Cost efficiency**: Delivers greater than 2x throughput compared to alternative inference frameworks, halving operational costs. - **Reliability track record**: Proven at production scale across trillions of processed tokens.

A live platform demonstration illustrates selecting an open model—such as the Flux.1-dev image generation model or Llama 3.3 70B—choosing the target GPU profile, and launching a live endpoint within minutes. Once initiated, the dashboard provides a built-in playground to interactively test prompts, inspect deployment logs, manage version configurations, and monitor latency metrics over time.

---

[11:34](https://www.youtube.com/watch?v=zl6FizzYC4M&t=694s) Flexible Deployment: Serverless API, Friendli Container, and Friendli Agent

To meet varied enterprise architecture and compliance needs, FriendliAI provides multiple consumption pathways:

- **Friendli Serverless API**: A fully managed, pay-as-you-go API that instantly scales across a growing portfolio of open foundation models without dedicated GPU provisioning. - **Friendli Container**: An enterprise-grade, self-hosted container for on-premises clusters or private VPC environments with strict privacy, air-gap, or governance standards. On AWS, it functions as a native Amazon EKS add-on deployable via a single configuration command. - **Friendli Agent**: A dedicated agentic orchestration layer positioned directly above the model inference engine. It coordinates agentic workflows, tracks contextual memory, handles external tool calls and API integrations, and routes multi-model requests with minimal overhead.

---

[13:03](https://www.youtube.com/watch?v=zl6FizzYC4M&t=783s) Real-World Case Studies and Getting Started

FriendliAI's production engine has been validated across diverse commercial environments:

- **Twelve Labs**: The San Francisco Bay Area multimodal video search startup reduced its overall production infrastructure costs by more than 50%. - **LG Electronics**: Deployed their proprietary LG EXAONE 3.0/3.1 foundation model (a 32-billion-parameter custom architecture rivaling frontier models like Anthropic's Claude 3.5 Sonnet / Claude Opus) to production in under 24 hours. - **Enterprise Deployments**: Major regulated enterprises migrated to the self-hosted Friendli Container inside their private cloud footprints to fulfill internal security constraints while reducing required GPU counts.

Engineering teams can get started immediately by spinning up dedicated Friendli Endpoints, integrating the Serverless API, or installing the Friendli Container on-premises. FriendliAI also offers a $10,000 GenAI Inference Credit Program with a simple two-minute application process to help organizations migrate from prototype to scalable production without building bespoke inference infrastructure.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=zl6FizzYC4M&t=4s) In real-world enterprise deployments, inference represents the live touchpoint where customers interact with models, accounting for 80% to 90% of an AI model's total lifetime expenditure. — Byung-Gon Chun

> [02:40](https://www.youtube.com/watch?v=zl6FizzYC4M&t=160s) reinvent infrastructure instead of building product features. — Byung-Gon Chun

> [05:17](https://www.youtube.com/watch?v=zl6FizzYC4M&t=317s) FriendliAI was built from the ground up to solve these production bottlenecks by making model inference fast, scalable, and cost-efficient. — Byung-Gon Chun

> [05:17](https://www.youtube.com/watch?v=zl6FizzYC4M&t=317s) Continuous batching solves this by dynamically inserting incoming requests into an ongoing batch and evicting completed sequences at token-level granularity. — Byung-Gon Chun

> [07:23](https://www.youtube.com/watch?v=zl6FizzYC4M&t=443s) Generates over 500 tokens per second for Llama 3.1 8B on an NVIDIA Hopper GPU. — Byung-Gon Chun

> [07:23](https://www.youtube.com/watch?v=zl6FizzYC4M&t=443s) Delivers greater than 2x throughput compared to alternative inference frameworks, halving operational costs. — Byung-Gon Chun

> [11:34](https://www.youtube.com/watch?v=zl6FizzYC4M&t=694s) A dedicated agentic orchestration layer positioned directly above the model inference engine. — Byung-Gon Chun

> [13:03](https://www.youtube.com/watch?v=zl6FizzYC4M&t=783s) Deployed their proprietary LG EXAONE 3.0/3.1 foundation model (a 32-billion-parameter custom architecture rivaling frontier models like Anthropic's Claude 3.5 Sonnet / Claude Opus) to production in under 24 hours. — Byung-Gon Chun

## Mentioned Byung-Gon Chun, FriendliAI, Friendli Inference, Friendli Endpoints, Friendli Serverless API, Friendli Container, Friendli Agent, Orca, OSDI 2022, Llama, Llama 3.1 8B, Llama 3.3 70B, DeepSeek, Qwen, Flux.1-dev, LG Electronics, EXAONE 3.0/3.1, Twelve Labs, Claude 3.5 Sonnet, Claude Opus, Hugging Face, Weights & Biases, NVIDIA Hopper, Amazon EKS

## Video description Byung-Gon Chun, CEO, FriendliAI

Abstracts

As generative AI evolves into more complex, agentic systems, inference has become the key bottleneck. These systems require dynamic interactions, tool use, memory handling, and coordination across multiple models—placing new demands on infrastructure.

Friendli Suite eliminates this complexity and empowers teams to define, deploy, and operate AI agents tailored to their business needs. With scalable, high-performance endpoints, organizations can move faster from idea to production without building custom backend infrastructure. This enables shorter development cycles, reduced engineering overhead, and a robust foundation for delivering and scaling intelligent agents in production environments.

[Source](https://www.youtube.com/watch?v=zl6FizzYC4M)

中文简报

FriendliAI通过连续批处理将智能体模型成本减半随着推理占据高达90%的AI全生命周期预算，多步智能体工作负载需要Token级别的连续批处理与动态基础设施，以避免延迟瓶颈和GPU成本超支。随着生成式AI向以多轮推理、外部工具调用和模型路由为特征的自主智能体系统演进，推理已成为主要瓶颈，消耗了80%至90%的生命周期算力预算。为应对不断攀升的GPU支出与延迟问题，FriendliAI推出了一套由连续批处理、推测解码和即时动态量化驱动的优化推理栈。该解决方案支持灵活的部署方式，包括与Hugging Face集成的一键式Friendli Endpoints、Serverless API、私有VPC/本地部署的Friendli Containers，以及Friendli Agent编排层。包括Twelve Labs将计算成本降低超50%、LG电子在24小时内完成32B EXAONE模型部署在内的实际企业验证，充分展示了其在吞吐量和成本效益方面的显著提升。生成式 AI 正加速从被动对话助手向具备多步推理、记忆与工具调用能力的自主 Agent 演进；在企业实际落地中，在线模型推理已占其全生命周期算力支出的 80% 到 90%。 (0:04) Agentic 工作流的多轮交互与工具调用大幅放大了网络延迟与成本开销，许多企业试点往往因自建复杂后端基础设施而陷入停滞，无法顺利推向生产。 (2:40) FriendliAI 依托团队在 OSDI 2022 开创的连续批处理（Continuous Batching）技术，结合投机解码与即时量化，在 token 粒度动态调度请求，大幅消除了 GPU 空转浪费。 (5:17) Friendli Endpoints 原生集成 Hugging Face Hub，支持一键部署超 42.3 万个模型，在 NVIDIA Hopper GPU 上让 Llama 3.1 8B 达到 500 tokens/s 以上吞吐，已帮助 Twelve Labs 等企业削减超 50% 基础设施成本。 (7:23)

Reading summary

FriendliAI argues that as generative AI transitions to multi-step agentic systems, inference becomes the dominant operational expense—consuming 80% to 90% of total lifetime compute budgets. To prevent latency degradation and spiraling hardware costs, the company advocates for specialized serving infrastructure centered on continuous batching and flexible deployment architectures. Inference Pressures in the Agentic AI Era (0:04) Generative AI is shifting beyond static copilots and chatbots toward autonomous agentic workflows capable of reasoning, planning, maintaining memory across multi-turn sessions, and executing external tool calls. While the proliferation of open-source architectures such as Llama, DeepSeek, and Qwen has reduced initial development costs, the multi-step nature of agents compounds latency and compute demands. In production, serving models accounts for 80% to 90% of lifetime AI expenditure. Enterprises scaling from pilot prototypes to production frequently face operational bottlenecks. These failure modes include escalating cloud bills driven by over-provisioning for bursty traffic, architectural complexity that distracts engineering teams from product development, data privacy constraints in regulated environments, and sluggish token generation that leads directly to user churn. Continuous Batching and Core Engine Optimization (5:17) To counter serving inefficiencies, the Friendli Inference engine relies on continuous batching, an innovation introduced by FriendliAI researchers in the Orca paper at OSDI 2022. Because inference requests arrive unpredictably and finish at varying intervals, standard static batching causes GPUs to sit partially idle. Continuous batching resolves this by dynamically injecting new requests and evicting completed sequences at token-level granularity, maintaining near-total GPU compute utilization. The core engine supplements continuous batching with parallel model execution, KV-cache management, speculative decoding, and on-the-fly quantization. These mechanisms enable organizations to apply compression schemes directly without running separate external conversion pipelines or degrading model output accuracy. Delivery Models and Performance Benchmarks (7:23) The engine is offered across several deployment modalities: Friendli Endpoints, a managed cloud service with 99.99% availability guarantees; a pay-as-you-go Serverless API; and Friendli Container, an enterprise-grade package designed for private VPC or on-premises deployments (including an Amazon EKS add-on). To handle orchestration, the platform incorporates Friendli Agent directly above the inference engine to manage tool calling, memory tracking, and multi-model routing. In benchmark claims, FriendliAI reports over 500 tokens per second for Llama 3.1 8B on NVIDIA Hopper GPUs, asserting that its platform delivers more than twice the throughput of alternative frameworks to halve operating expenses. The system also supports one-click deployment for over 423,000 models cataloged on the Hugging Face Hub, featuring rolling updates, automated right-sizing, and built-in latency tracking. Enterprise Adoption and Operational Impact (13:03) The provider cites several commercial deployments to validate its throughput and cost claims. Video search startup Twelve Labs reported reducing its production infrastructure spend by more than 50% after moving to the Friendli engine. Additionally, LG Electronics deployed its proprietary 32-billion-parameter foundation model, EXAONE 3.0/3.1, to production within 24 hours. Other regulated enterprise clients have adopted the self-hosted container to meet internal data governance rules while decreasing required GPU hardware footprints. FriendliAI contends that tailored inference optimization is critical to making agentic AI economically viable in production. However, achieving claimed cost and throughput improvements in practice depends on specific model architectures, workload predictability, and an organization's willingness to integrate specialized serving stacks into their existing infrastructure.

精炼阅读摘要

FriendliAI指出，随着生成式人工智能从单轮交互工具转向具备多步推理与工具调用能力的Agentic AI，模型推理已占据系统全生命周期计算支出的80%至90%；为解决突发流量与延迟累积导致的算力瓶颈，该机构推出了以动态连续批处理为核心的优化推理引擎与多形态部署架构。 Agentic AI落地痛点与算力成本 (0:04) 随着Llama、DeepSeek和Qwen等开源基础模型的普及，企业能够以更低成本针对特定领域微调模型，推动生成式AI从简单的对话辅助工具迈向自主解决问题的智能体系统。然而，Agentic AI涉及复杂的长周期推理、会话状态记忆保留、外部工具调用与多模型路由。这种高频往返的多步工作流大幅推高了推理计算开销，演讲者指出，实际部署中模型推理已占到AI全生命周期总支出的80%至90%，对低延迟和高吞吐提出了严苛要求。许多企业在完成原型验证后难以顺利推进至生产环境。典型的运营困境包括：为应对不可预测的突发流量而过度配置硬件，导致昂贵的GPU算力大比例闲置；多步骤串联放大了推理延迟，首字生成时间（TTFT）与Token生成速度不足直接导致终端用户流失；此外，数据安全边界与复杂的底层技术栈也拖累了产品工程团队。企业管理者亟需一套兼顾细粒度算力整合、动态自动伸缩、全链路可观测性与99.99%高可用性的推理基础设施。连续批处理机制与核心引擎创新 (5:17) 针对企业在吞吐与延迟上的瓶颈，FriendliAI基于其团队在OSDI 2022发表的Orca系统论文，将“连续批处理”（Continuous Batching）作为底层引擎的核心支柱。在真实业务场景中，推理请求到达时间不规律且生成序列长短各异，传统的静态批处理极易因短序列等待长序列而出现GPU计算单元闲置；连续批处理技术则在Token级别进行细粒度迭代调度，允许新请求动态插入当前批次，并在特定序列完成时即刻移出，使GPU计算利用率维持在接近饱和的状态。在此基础上，Friendli Inference引擎还整合了并行模型执行、KV缓存（KV-cache）精细管理、调度算法优化、投机解码以及运行时即时量化（On-the-Fly Quantization）技术，用户无需配置外部压缩工具即可一键选择量化方案且不损耗精度。演讲者表示，该引擎在不同场景下能实现相比其他主流框架高出2倍以上的吞吐量，从而直接降低一半的基础设施运营支出，其高可靠性已在数万亿处理Token规模的生产实践中得到验证。灵活部署矩阵与Agent编排整合 (7:23) 为了适配多样化的企业架构与合规要求，平台提供了多层级的服务交付形态。托管型的Friendli Endpoints支持与Hugging Face Hub和Weights & Biases原生打通，覆盖了Hugging Face收录的近200万个模型中超过42.3万个开源模型的一键部署；在NVIDIA Hopper架构GPU上运行Llama 3.1 8B模型时，其生成速率可突破每秒500个Token，并支持在线加载Flux.1-dev等图像生成模型与Llama 3.3 70B等大参数模型。在部署形态方面，团队提供了无需预先配置专有GPU、按量计费的Friendli Serverless API；面向要求数据留存在私有边界内的受监管场景，则提供支持本地集群及私有VPC的Friendli Container，在AWS上可作为Amazon EKS原生插件一键配置。此外，针对智能体链路编排，平台在推理引擎上方构建了Friendli Agent抽象层，直接管理上下文记忆、工具调用接口和多模型路由，以此规避在模型层与外部编排框架间反复切换带来的系统开销。行业验证成果与算力迁移支持 (13:03) FriendliAI展示了其推理栈在真实商业生产环境中的效能数据。多模态视频搜索初创企业Twelve Labs通过采用该推理引擎，将其生产环境的总体基础设施硬件成本削减了50%以上；LG电子则利用其技术栈，在24小时内完成了320亿参数的专有基础模型LG EXAONE 3.0/3.1的生产级上线部署，验证了引擎承接超大规模私有架构的能力。此外，针对合规监管要求严苛的行业客户，通过在私有云内运行自托管Friendli Container，在满足内部数据治理标准的同时减少了物理GPU采购基数。为了降低工程团队从自建底层基础设施向成熟推理层迁移的门槛，FriendliAI还推出了10,000美元的生成式AI推理抵免额度计划，协助企业跳过底层管线研发，直接将Agent系统推向规模化商用。该报告系统论证了智能体化工作流对推理算力利用率的极高要求，并展示了通过连续批处理与即时量化实现降本增效的技术路径；但内容主要基于FriendliAI自身架构与基准测试，未深入讨论超大模型长文本推理下KV缓存暴涨所面临的物理显存极限，亦未提供与其他同类商业优化引擎在极端突发流量下的横向对比数据。

Detailed notes — zh

## Background 本内容由频道 Ai4 于 2025 年 8 月 24 日发布在 YouTube 平台，主讲人为 FriendliAI 的首席执行官 Byung-Gon Chun。视频简介指出，随着生成式 AI 向更复杂的智能体系统演进，推理已成为核心瓶颈；Friendli Suite 旨在消除底层基础设施的复杂性，助力团队快速将 AI 智能体从原型推向规模化生产。 ## TL;DR 演讲深入剖析了 Agentic AI 在走向生产环境时面临的高昂推理成本、长会话多轮调用及延迟等核心挑战。FriendliAI 依托其团队率先开创的连续批处理（Continuous Batching）技术与 Friendli Engine，构建了涵盖 Friendli Endpoints、Serverless API、私有化 Container 以及 Friendli Agent 的全栈产品矩阵。通过即时量化和推测解码等前沿优化，平台实现了超竞品 2 倍的高吞吐量与极速响应，并在 Twelve Labs、LG AI Research 等头部企业中验证了削减 50% 成本及 24 小时极速交付的商业价值。 ## Atmosphere & style 整场演讲呈现出典型技术领袖会议的严谨、干练与极高信息密度。演讲者以冷静且切中要害的工程视角开篇直击行业痛点，随后通过架构解析与流畅的现场实操演示展现出对底层性能优化的绝对自信。整体氛围务实而具号召力，既有学术顶会级硬核技术的沉淀感，又充满了帮助企业跨越落地鸿沟的商业推力。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=zl6FizzYC4M&t=4s) AI 发展正经历开源基础模型（如 LLaMA、DeepSeek、Qwen）与智能体（Agentic AI）的双重爆发，AI 角色正从被动的问答辅助转向能自主执行复杂任务的行动者。 - [01:22](https://www.youtube.com/watch?v=zl6FizzYC4M&t=82s) AI 商业价值必须在生产环境中体现，而推理（Inference）占据模型全生命周期总成本的 80% 至 90%，是决定产品能否规模化落地的核心瓶颈。 - [01:22](https://www.youtube.com/watch?v=zl6FizzYC4M&t=82s) Agentic AI 涉及多轮对话、记忆管理、工具调用与多模型协同，导致算力开销和延迟激增，许多原型项目因高昂的 GPU 账单和性能问题停滞在原型阶段。 - [03:29](https://www.youtube.com/watch?v=zl6FizzYC4M&t=209s) 企业在选择推理基础设施时面临降本（避免 GPU 过度配置）、毫秒级低延迟响应、高弹性伸缩、合规安全与极简运维等关键诉求。 - [05:17](https://www.youtube.com/watch?v=zl6FizzYC4M&t=317s) FriendliAI 核心推理引擎源于 OSDI 2022 突破性论文 Orca，其发明的连续批处理（Continuous Batching）在细粒度迭代级别动态增删请求，彻底解决了传统静态批处理导致的 GPU 算力闲置。 - [07:23](https://www.youtube.com/watch?v=zl6FizzYC4M&t=443s) Friendli Endpoints 平台提供即时无损量化（On-the-fly Quantization）与推测解码，在 NVIDIA Hopper GPU 上运行 LLaMA 3.1 8B 吞吐可超 500 tokens/s，整体吞吐量达到同类方案的 2 倍以上。 - [09:14](https://www.youtube.com/watch?v=zl6FizzYC4M&t=554s) 平台与 Hugging Face 及 Weights & Biases 深度集成，支持在 Hugging Face 模型页面一键快速部署至专用端点，并内置实时测试 Playground 和全链路运维监控看板。 - [11:34](https://www.youtube.com/watch?v=zl6FizzYC4M&t=694s) 提供多元产品形态：面向快速调用的免运维 Serverless API、面向合规私有化并作为 AWS EKS 官方 Add-on 的 Friendli Container，以及针对多步交互优化的 Friendli Agent 智能体层。 - [13:03](https://www.youtube.com/watch?v=zl6FizzYC4M&t=783s) 商业落地成效显著：视频多模态搜索初创公司 Twelve Labs 削减了超 50% 的推理成本，LG AI Research 则在 24 小时内完成了 32B EXAONE 模型的生产级部署上线。 - [13:03](https://www.youtube.com/watch?v=zl6FizzYC4M&t=783s) FriendliAI 推出最高 10,000 美元的 GenAI 推理额度计划，帮助企业团队免去重复造基础设施轮子的负担，加速从 Demo 原型迈向规模化生产。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=zl6FizzYC4M&t=4s) 开源浪潮与 Agentic AI 的崛起随着人工智能技术的演进，AI 正在从早期的代码辅助（Co-pilots）、简单自动化和基础客服聊天机器人，深刻重塑企业的整体运营方式。当前技术范式正经历着以智能体 AI（Agentic AI）和开源模型为核心的双重加速。开源生态中，包括 LLaMA、DeepSeek、Qwen 在内的基础模型及各类微调模型呈现爆发式增长；而 Agentic AI 的兴起，则标志着 AI 正从传统的“被动问答者”转变为能够自主解决复杂业务问题的“主动行动者”。这种转变不仅使企业能够以更低成本构建针对特定场景的功能，还催生了更加复杂的复合工作流乃至多智能体协作系统（Multi-agent systems）。然而，构建 AI 智能体原型（Prototype）仅仅是迈出转型的第一步。原型验证并不直接产生商业价值，AI 真正驱动业务增长、产生可衡量回报的场景必须发生在生产环境（Production）中。推理（Inference）是终端用户与 AI 产品产生交互的核心触点，也是模型在大规模实际应用中释放价值的关键环节。 - **开源模型爆发**：LLaMA、DeepSeek、Qwen 等开源基础与微调模型的大规模采用，大幅降低了企业获得顶尖 AI 能力的门槛。 - **智能体范式转变**：AI 演变为具备自主解决问题能力的 Agent，支持复杂工作流与多智能体系统协同。 ```mermaid flowchart TD accTitle: Agentic AI 生产化面临的核心挑战 subgraph Agentic_Loop["Agentic AI 核心运行时循环"] Reasoning["规划与自主推理 Reasoning & Planning"] --> Memory["上下文与记忆检索 Memory & Context"] Memory --> ToolCall["外部工具调用与 API 协同 Tool Calling & Action"] ToolCall --> MultiModel["多模型协同推理 Multi-turn Model Calls"] MultiModel --> Reasoning end subgraph Infra_Bottlenecks["底层推理基础设施瓶颈"] Bottleneck1["极端延迟敏感 Low Latency Requirement"] Bottleneck2["长会话与多轮调用成本 Long Sessions & GPU Cost"] Bottleneck3["突发性流量与动态批处理 Bursty Traffic & Inefficient Batching"] end Agentic_Loop -.-> Infra_Bottlenecks ``` --- ### [01:22](https://www.youtube.com/watch?v=zl6FizzYC4M&t=82s) 生产落地瓶颈：Agentic AI 推理的真实代价在真实的商业部署中，推理成本极其庞大，通常占据 AI 模型全生命周期总成本（Lifetime Cost）的 80% 至 90%。Agentic AI 本质上是一个能够自主追求目标的高级系统，它深度结合了模型推理、逻辑规划、长期记忆管理、外部工具调用以及多步骤动作执行能力。这一复杂架构对底层基础设施提出了极其严苛的要求：极低延迟（Low Latency）、超高吞吐量（High Throughput）以及高度的成本效益。许多充满前景的 AI 试点项目在迈向面向真实用户的大规模生产阶段时往往陷入停滞。Agentic 系统的多轮交互和长会话使得单次任务调用的模型次数成倍增加，带来了严重的系统性难题：新产品上线周期被无休止地推迟；由于 GPU 资源消耗过快，云账单出现灾难性失控；基础设施的编排与管理复杂度急剧攀升；此外，响应延迟过高直接导致终端用户流失。企业工程团队常常被迫陷入重新发明底层基础设施轮子的泥潭，反而无法专注于核心产品功能的研发。 - **成本分布特征**：推理阶段占据 AI 模型全生命周期成本的 80%–90%，成为决定商业可持续性的核心瓶颈。 - **Agentic 推理难点**：多轮会话拉长、高频模型互调、工具链集成与编排使计算开销呈指数级扩散。 - **常见落地阻碍**：产品交付延误、GPU 账单超支、响应缓慢导致用户流失、团队陷入基建内耗。 --- ### [03:29](https://www.youtube.com/watch?v=zl6FizzYC4M&t=209s) 业务决策者的核心诉求与理想推理平台面对生产化转型的困境，企业业务负责人在评估 AI 推理基础设施时，关注的焦点集中在几个关键维度上。首先是**成本控制**：GPU 硬件极其昂贵，而由于业务流量天然的不确定性，企业往往被迫进行严重的过度配置（Over-provisioning）；其次是**响应能力与弹性**：用户期待毫秒级的实时反馈，但 AI 流量具有极强的突发性（Bursty），基础设施必须能够实现无缝弹性伸缩；再次是**极简性与数据掌控力**：基础设施复杂度不能拖慢产品团队迭代，同时涉及敏感数据的场景必须支持私有化或本地（On-prem）部署；最后是全链路的**可观测性**与 **99.99% 的企业级可靠性**。基于这些严苛诉求，理想的企业级 AI 推理平台应具备在几分钟内一键部署任意开源、微调或自定义模型的能力，根据实时流量自动伸缩，在底层透明地进行全栈性能优化，原生支持云端与本地环境，并与主流模型生态（如 Hugging Face、Weights & Biases）无缝集成。 | 核心诉求维度 | 传统部署痛点 | 理想推理平台标准 | |:--- |:--- |:--- | | **成本与资源利用** | 流量预测困难导致过度配置 GPU，闲置浪费严重 | 依靠底层算子与调度优化实现高吞吐，按需缩容降低 GPU 开销 | | **延迟与响应速度** | 长上下文和多步推理导致延迟飙升，用户体验差 | 极致的内核级优化，实现超高速 Token 生成与流式响应 | | **弹性与运维复杂度** | 扩缩容反应滞后，基建维护占用大量工程师精力 | 秒级弹性伸缩，全托管运维，屏蔽底层硬件异构性 | | **数据安全与合规** | 公共 API 无法满足金融、医疗等敏感场景要求 | 原生支持 VPC 私有化部署与 On-prem 容器化交付 | | **生态打通** | 模型拉取、转换与上线流程繁琐，耗时数周 | 深度集成 Hugging Face 与 W&B，数分钟内一键完成上线 | --- ### [05:17](https://www.youtube.com/watch?v=zl6FizzYC4M&t=317s) FriendliAI 核心架构与连续批处理（Continuous Batching）为了彻底解决上述瓶颈，FriendliAI 构建了专为高性能大模型推理设计的底层技术栈——**Friendli Engine（Friendli Inference）**。该引擎从硬件底层重新设计，集成了业界最先进的优化技术与自主创新，全面支持大语言模型及多模态模型，涵盖批处理调度、量化、张量/流水线并行、KV 缓存优化以及推测解码（Speculative Decoding）等多个维度，旨在使 AI 推理变得“不可见、高可扩展且极具成本效益”。在诸多突破性技术中，最核心的基石便是 FriendliAI 团队率先发明并在 OSDI 2022 顶会论文（Orca 系统）中提出的**连续批处理（Continuous Batching）**技术。由于真实世界的推理请求到达时间极不规律，且各请求的输入输出长度差异巨大，传统的静态批处理（Static Batching）会导致计算单元等待最长序列完成，产生严重的 GPU 算力闲置。连续批处理技术通过在极细粒度的迭代（Iteration）级别动态插入新到达的请求并剔除已完成的请求，始终保持批处理单元处于高饱和填充状态。FriendliAI 在生产环境中应用了进一步深度优化的连续批处理机制，使 GPU 利用率和吞吐量达到了前所未有的高度。 - **连续批处理先驱**：源自 OSDI 2022 突破性论文 Orca，已成为全行业大模型推理服务的标准范式。 - **细粒度动态调度**：突破传统 Batch 必须统进统出的限制，在 Iteration 粒度动态增删序列，极大压榨 GPU 算力。 - **全栈性能加速**：融合先进量化、高并发缓存、高效并行策略与推测解码技术。 --- ### [07:23](https://www.youtube.com/watch?v=zl6FizzYC4M&t=443s) Friendli Endpoints 产品特性与性能突破在底层 Friendli Engine 之上，FriendliAI 推出了面向企业的全托管推理平台 **Friendli Endpoints**。该平台整合了现代生产系统所需的全套运维能力：用于最大化利用率的智能路由、应对流量剧烈波动的自动伸缩、细粒度实时监控（实时统计处理 Token 量、首字延迟、P99 延迟等）、在线模型版本管理与热更新、高达 99.99% 的高可用保障，以及独创的“即时量化（On-the-fly Quantization）”与推测解码功能。用户无需在部署前手动运行复杂的量化工具链，只需在界面上一键开启量化开关，系统即可在不损失模型精度的前提下自动应用最优量化策略，显著提升执行速度。在性能与生态集成方面，Friendli Endpoints 展现出强大的竞争力。平台与 Hugging Face Hub 和 Weights & Biases 深度整合，支持从 Hugging Face 的 42 万余个主流模型库中进行一键拉取与发布。在硬件性能基准上，Friendli Endpoints 在单张 NVIDIA Hopper GPU 上运行 LLaMA 3.1 8B 能够实现超过 500 tokens/s 的超高速生成，整体吞吐量达到同类方案的 2 倍以上，平台已在大规模生产环境中稳定支撑了数万亿（Trillions）Token 的计算吞吐。 - **即时模型量化**：支持零门槛一键无损量化，无需离线转换模型权重即可获得极限推理加速。 - **极限推理速度**：LLaMA 3.1 8B 在 Hopper 架构 GPU 上达到 >500 tokens/s 吞吐表现。 - **双倍成本效益**：吞吐量达到竞品方案的 2 倍以上，直接将模型推理运行成本压缩一半。 - **生态无缝对接**：支持直接从 Hugging Face 模型页面一键分发并建立专用端点。 ```mermaid flowchart LR accTitle: FriendliAI 全栈产品与部署形态全景 subgraph Hubs["模型生态 Hubs"] HF["Hugging Face 400k+ 模型"] WB["Weights & Biases 模型仓库"] end subgraph Friendli_Platform["FriendliAI 平台核心能力"] Engine["Friendli Inference Engine Continuous Batching | 即时量化 | Speculative Decoding"] AgentLayer["Friendli Agent 智能体层 Workflow 编排 | 记忆管理 | API/工具集成"] end subgraph Deploy_Modes["多元化部署形态"] Dedicated["Friendli Endpoints 专用托管端点 (99.99% SLA)"] Serverless["Friendli Serverless API 极速试用与轻量调用"] Container["Friendli Container 私有化/本地化部署 (AWS EKS Add-on)"] end Hubs --> Engine Engine --> AgentLayer AgentLayer --> Dedicated AgentLayer --> Serverless AgentLayer --> Container ``` --- ### [09:14](https://www.youtube.com/watch?v=zl6FizzYC4M&t=554s) 现场演示：从 Hugging Face Hub 到模型游乐场在演讲现场演示中，演讲者展示了通过 Friendli Endpoints 部署 Flux Dev 文生图模型与 LLaMA 3.3 70B 大语言模型的全流程。整个过程极其直观：在控制台中直接选定目标模型（可直接从 Hugging Face 搜索拉取），配置所需的 GPU 硬件类型，点击创建端点后，系统在数秒内便实时完成了 GPU 资源调度、模型拉取挂载与推理引擎初始化。部署完成后，控制台内置的 Playground 允许开发者立即进行交互测试，无论是图像生成还是复杂的文本生成推理均可秒级响应。此外，平台仪表盘全面集成了版本控制、实时运行日志以及长周期的延迟与吞吐监控指标，消除了传统自建基础设施需要维护 Prometheus、Grafana 等监控系统的繁琐负担。演讲者进一步演示了直接嵌入在 Hugging Face 模型主页上的 FriendliAI 部署入口——开发者只需在 Hugging Face 页面点击 FriendliAI 图标，即可一键直跳完成端点搭建。 - **极简部署流程**：无需编写部署配置脚本，选择模型与 GPU 规格后数秒内即可自动拉起服务。 - **内置 Playground**：支持实时在线测试各类大语言模型与图像生成扩散模型（如 Flux Dev）。 - **全方位运维看板**：原生集成实时监控仪表盘、调用日志追踪与多版本模型迭代管理。 --- ### [11:34](https://www.youtube.com/watch?v=zl6FizzYC4M&t=694s) 全场景产品矩阵：Serverless、Container 与 Friendli Agent 为了满足不同企业在架构设计与合规安全方面的差异化诉求，FriendliAI 提供了多层次的产品形态。对于希望以最低门槛快速调用的开发者，**Friendli Serverless API** 提供了免运维、按用量计费的即时调用能力，支持的模型矩阵正在持续扩充；而针对金融、医疗等对数据隐私与合规有严苛要求的场景，则推出了 **Friendli Container** 本地化与私有云部署方案。在 AWS 生态中，Friendli Container 已作为 AWS EKS 的官方 Add-on 提供，企业工程师只需运行一行安装命令，即可将其无缝嵌入现有的 Kubernetes 集群工作流中。在基础设施之上，FriendliAI 进一步推出了 **Friendli Agent** 智能体编排层。该层直接构建于高性能推理引擎之上，负责智能体工作流编排、上下文记忆管理、外部 API 集成以及多模型间的无缝路由。通过将深厚的推理优化经验注入到智能体执行层，Friendli Agent 能够确保复杂的多步动作与多模型交互依然保持极低的端到端延迟，为企业交付简单、通用、高速、高性价比且极其可靠的智能体基础设施。 - **Friendli Serverless API**：免除算力管理负担，开箱即用，按实际 Token 消耗计量。 - **Friendli Container (On-prem)**：专为高合规私有化场景打造，提供 AWS EKS Add-on 一键式集成。 - **Friendli Agent**：位于模型层之上的智能体执行系统，解决多步骤推理中的记忆、工具链与模型协同延迟。 --- ### [13:03](https://www.youtube.com/watch?v=zl6FizzYC4M&t=783s) 商业落地实战案例与开发者计划在真实商业应用中，FriendliAI 的技术方案已在多家前沿 AI 独角兽与跨国企业中落地验证： 1. **Twelve Labs**：位于旧金山湾区的知名视频多模态搜索初创公司，通过迁移至 FriendliAI 推理平台，将其整体基础设施计算成本大幅降低了 50% 以上； 2. **LG AI Research（LG 人工智能研究院）**：LG 研发了参数量为 32B 的自研基座大模型 LG EXAONE，在性能表现上能够匹敌 Anthropic 的 Claude 3 Opus 等超大模型。LG 团队借助 FriendliAI 平台，在不到 24 小时内便完成了该定制模型的全量上线与生产部署； 3. **大型企业级客户**：多家严苛合规要求的大型企业全面采纳了 Friendli Container，在完全满足内部数据安全隔离标准的同时，显著削减了所需的 GPU 服务器规模。在演讲最后，Byung-Gon Chun 总结道，AI 产品的核心价值绝不能困在早期的 Demo 阶段，企业工程团队应当聚焦于打磨卓越的 AI 产品体验，而非将宝贵的工程资源消耗在底层基础设施的搭建与维护上。FriendliAI 正式宣布推出“10K GenAI 推理额度计划（FriendliAI 10K GenAI Inference Credit Program）”，为准备迈向生产化扩张的团队提供最高 1 万美元的免费推理算力支持，助力企业跨越从原型验证到规模化生产的鸿沟。 - **Twelve Labs 案例**：视频搜索领军企业成功削减超 50% 的 GPU 基础设施账单。 - **LG EXAONE 案例**：32B 高性能定制模型在 24 小时内完成高效生产级部署。 - **10K 开发者计划**：提供最高 10,000 美元免费推理 Credit，加速企业从 PoC 到生产落地。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=zl6FizzYC4M&t=4s) 推理（Inference）是终端用户与 AI 产品产生交互的核心触点，也是模型在大规模实际应用中释放价值的关键环节。 — Byung-Gon Chun > [00:04](https://www.youtube.com/watch?v=zl6FizzYC4M&t=4s) AI 正从传统的‘被动问答者’转变为能够自主解决复杂业务问题的‘主动行动者’。 — Byung-Gon Chun > [01:22](https://www.youtube.com/watch?v=zl6FizzYC4M&t=82s) 在真实的商业部署中，推理成本极其庞大，通常占据 AI 模型全生命周期总成本（Lifetime Cost）的 80% 至 90%。 — Byung-Gon Chun > [01:22](https://www.youtube.com/watch?v=zl6FizzYC4M&t=82s) 企业工程团队常常被迫陷入重新发明底层基础设施轮子的泥潭，反而无法专注于核心产品功能的研发。 — Byung-Gon Chun > [03:29](https://www.youtube.com/watch?v=zl6FizzYC4M&t=209s) 用户期待毫秒级的实时反馈，但 AI 流量具有极强的突发性（Bursty），基础设施必须能够实现无缝弹性伸缩 — Byung-Gon Chun > [05:17](https://www.youtube.com/watch?v=zl6FizzYC4M&t=317s) 旨在使 AI 推理变得‘不可见、高可扩展且极具成本效益’。 — Byung-Gon Chun > [05:17](https://www.youtube.com/watch?v=zl6FizzYC4M&t=317s) 连续批处理技术通过在极细粒度的迭代（Iteration）级别动态插入新到达的请求并剔除已完成的请求，始终保持批处理单元处于高饱和填充状态。 — Byung-Gon Chun > [07:23](https://www.youtube.com/watch?v=zl6FizzYC4M&t=443s) 用户无需在部署前手动运行复杂的量化工具链，只需在界面上一键开启量化开关，系统即可在不损失模型精度的前提下自动应用最优量化策略，显著提升执行速度。 — Byung-Gon Chun > [07:23](https://www.youtube.com/watch?v=zl6FizzYC4M&t=443s) 在单张 NVIDIA Hopper GPU 上运行 LLaMA 3.1 8B 能够实现超过 500 tokens/s 的超高速生成，整体吞吐量达到同类方案的 2 倍以上 — Byung-Gon Chun > [11:34](https://www.youtube.com/watch?v=zl6FizzYC4M&t=694s) 通过将深厚的推理优化经验注入到智能体执行层，Friendli Agent 能够确保复杂的多步动作与多模型交互依然保持极低的端到端延迟 — Byung-Gon Chun > [13:03](https://www.youtube.com/watch?v=zl6FizzYC4M&t=783s) AI 产品的核心价值绝不能困在早期的 Demo 阶段，企业工程团队应当聚焦于打磨卓越的 AI 产品体验，而非将宝贵的工程资源消耗在底层基础设施的搭建与维护上。 — Byung-Gon Chun ## Mentioned Byung-Gon Chun, FriendliAI, Friendli Suite, Friendli Engine, Friendli Endpoints, Friendli Serverless API, Friendli Container, Friendli Agent, Continuous Batching (连续批处理), Orca, OSDI 2022, NVIDIA Hopper, LLaMA, LLaMA 3.1 8B, LLaMA 3.3 70B, DeepSeek, Qwen, Flux Dev, Hugging Face, Weights & Biases, Twelve Labs, LG AI Research, LG EXAONE, Claude 3 Opus, AWS EKS ## Video description Byung-Gon Chun, CEO, FriendliAI Abstracts As generative AI evolves into more complex, agentic systems, inference has become the key bottleneck. These systems require dynamic interactions, tool use, memory handling, and coordination across multiple models—placing new demands on infrastructure. Friendli Suite eliminates this complexity and empowers teams to define, deploy, and operate AI agents tailored to their business needs. With scalable, high-performance endpoints, organizations can move faster from idea to production without building custom backend infrastructure. This enables shorter development cycles, reduced engineering overhead, and a robust foundation for delivering and scaling intelligent agents in production environments. [Source](https://www.youtube.com/watch?v=zl6FizzYC4M)

Source quotes

[0:04] Byung-Gon Chun: 推理（Inference）是终端用户与 AI 产品产生交互的核心触点，也是模型在大规模实际应用中释放价值的关键环节。 [0:04] Byung-Gon Chun: AI 正从传统的‘被动问答者’转变为能够自主解决复杂业务问题的‘主动行动者’。 [1:22] Byung-Gon Chun: 在真实的商业部署中，推理成本极其庞大，通常占据 AI 模型全生命周期总成本（Lifetime Cost）的 80% 至 90%。 [1:22] Byung-Gon Chun: 企业工程团队常常被迫陷入重新发明底层基础设施轮子的泥潭，反而无法专注于核心产品功能的研发。 [3:29] Byung-Gon Chun: 用户期待毫秒级的实时反馈，但 AI 流量具有极强的突发性（Bursty），基础设施必须能够实现无缝弹性伸缩 [5:17] Byung-Gon Chun: 旨在使 AI 推理变得‘不可见、高可扩展且极具成本效益’。 [5:17] Byung-Gon Chun: 连续批处理技术通过在极细粒度的迭代（Iteration）级别动态插入新到达的请求并剔除已完成的请求，始终保持批处理单元处于高饱和填充状态。 [7:23] Byung-Gon Chun: 用户无需在部署前手动运行复杂的量化工具链，只需在界面上一键开启量化开关，系统即可在不损失模型精度的前提下自动应用最优量化策略，显著提升执行速度。 [7:23] Byung-Gon Chun: 在单张 NVIDIA Hopper GPU 上运行 LLaMA 3.1 8B 能够实现超过 500 tokens/s 的超高速生成，整体吞吐量达到同类方案的 2 倍以上 [11:34] Byung-Gon Chun: 通过将深厚的推理优化经验注入到智能体执行层，Friendli Agent 能够确保复杂的多步动作与多模型交互依然保持极低的端到端延迟 [13:03] Byung-Gon Chun: AI 产品的核心价值绝不能困在早期的 Demo 阶段，企业工程团队应当聚焦于打磨卓越的 AI 产品体验，而非将宝贵的工程资源消耗在底层基础设施的搭建与维护上。

Source quotes

[0:04] Byung-Gon Chun: In real-world enterprise deployments, inference represents the live touchpoint where customers interact with models, accounting for 80% to 90% of an AI model's total lifetime expenditure. [2:40] Byung-Gon Chun: reinvent infrastructure instead of building product features. [5:17] Byung-Gon Chun: FriendliAI was built from the ground up to solve these production bottlenecks by making model inference fast, scalable, and cost-efficient. [5:17] Byung-Gon Chun: Continuous batching solves this by dynamically inserting incoming requests into an ongoing batch and evicting completed sequences at token-level granularity. [7:23] Byung-Gon Chun: Generates over 500 tokens per second for Llama 3.1 8B on an NVIDIA Hopper GPU. [7:23] Byung-Gon Chun: Delivers greater than 2x throughput compared to alternative inference frameworks, halving operational costs. [11:34] Byung-Gon Chun: A dedicated agentic orchestration layer positioned directly above the model inference engine. [13:03] Byung-Gon Chun: Deployed their proprietary LG EXAONE 3.0/3.1 foundation model (a 32-billion-parameter custom architecture rivaling frontier models like Anthropic's Claude 3.5 Sonnet / Claude Opus) to production in under 24 hours.

Original transcript

[0:04] Welcome. I I'm gone, CEO of Friendly AI. [0:07] Today I'm going to talk about uh scaling [0:09] inference for agentic AI. Uh this talk [0:12] is designed for business leaders who are [0:15] navigating from uh AI prototypes to [0:18] production systems. [0:23] As you well know, AI is transforming [0:26] business. From co-pilots to automation [0:29] to customer chatbots, AI is reshaping [0:32] how businesses operate. [0:34] Recent trends on agentic AI and open [0:37] source are accelerating this shift. [0:41] Agents can be complex workflows or even [0:44] multi- aent systems. [0:52] So as uh you may heard open source and [0:55] agentic are taking off. Adoption of open [0:58] source AI models such as llama deepseek [1:01] and quen both space and fine tuned is [1:04] surging agentic AI drives business [1:07] impact by turning AI into active problem [1:09] solver not just a passive responder. So [1:13] as a result companies are able to get [1:15] the functionality they want at a much [1:18] lower cost. [1:22] So building AI agent prototypes is just [1:24] the beginning. The real value doesn't [1:27] come from AI prototypes. It comes from [1:30] production. So inference is where your [1:33] customers interact with your AI products [1:35] and where AI drives business outcomes at [1:38] scale. [1:40] In real world deployments, inference [1:42] often accounts for 80 to 90% of an AI [1:46] model's total lifetime cost. [1:51] So um agentic AI is uh a system that can [1:57] autonomously pursue goals by combining [1:59] reasoning and planning with models [2:01] memory tool calls and the ability to [2:03] execute multi-step actions and this [2:06] agentic AI inference has tight [2:09] requirements low latency high throughput [2:12] and cost efficient execution [2:15] and agentic AI inference is challenging [2:19] uh as beyond generative AI [2:23] motor inference remains a core challenge [2:26] but it becomes more challenging because [2:28] of longer sessions more AI model calls [2:32] tool inte integration and orchestration [2:38] so you know uh many AI projects they [2:43] have many promising pilots but they [2:46] store when it's time to run with real [2:49] users at scale. So common symptoms [2:52] facing are delays in launching AI [2:55] products. [2:57] um when you want to deploy and cost [3:01] overruns especially if you are running [3:03] models GPU bills skyrocketing [3:06] infra complexity to manage these agents [3:10] and security concerns slow responses can [3:13] cause user turns and your yous [3:18] uh your team spending time on [3:20] reinventing infrastructure instead of [3:22] building product features. So all these [3:25] common symptoms are happening. [3:29] So what business leaders really care [3:31] about? Uh first cost control. If you're [3:35] using GPUs, GPUs are expensive and [3:38] you're often doing over provisioning [3:40] since you don't really expect traffic [3:44] patterns. [3:46] Responsiveness users expect low [3:48] latencies. This is very challenging. [3:51] Elasticity AI traffic is bursty in front [3:54] needs to be flex uh and simplicity in [3:58] complexity shouldn't slow product teams [4:01] and data control sensitive use cases [4:03] require private or onrem [4:06] and uh another thing is observability [4:09] exacts need visibility into usage spend [4:12] and uptime and lastly reliability 247 [4:16] reliable services. [4:20] So what an ideal AI inference platform [4:23] looks like [4:25] uh if it can deploy any model open [4:27] source fine-tuned custom within minutes [4:30] not weeks or month uh scales up down [4:34] seamlessly with your traffic provides [4:37] fast responses cuts cost by optimizing [4:40] performance under the hood. You don't [4:42] need to re really think about optimizing [4:45] performance. Works where you need it [4:48] whether it's cloud or onrem. Tracks [4:51] usage and cost in real time. Runs with [4:54] 99.99% [4:56] reliability. And lastly, many AI [5:00] products are using motor sharing [5:01] platforms like hoging face or race and [5:04] biases. So we definitely want to [5:06] integrate with the such motor sharing [5:08] platforms. So this is an AI inference [5:11] prep platform uh an ideal AI inference [5:14] platform. [5:17] So MIT friendly AI it's an ideal AI [5:21] inference platform we built is built [5:23] from the ground up for high performance [5:25] AI inference. We make AI inference [5:28] invisible, scalable and cost efficient. [5:33] At the heart of friendly AI is friendly [5:37] inference. It's our inference stack for [5:41] scalable high performance gen AI [5:43] inference built from the ground up. Uh [5:46] it includes all the advanced [5:47] optimization techniques combining our [5:49] own innovations with industry best [5:52] practices and supports both language and [5:55] multimodal models. So it delivers both [5:58] fast and cost efficient per performance. [6:01] So there are optimizations on batching, [6:04] scheduling, quantization, uh [6:06] parallelism, caching and spec decoding. [6:09] I can talk about all these techniques [6:11] for hours. But since we don't have much [6:13] time, I'm going to just highlight one [6:15] technique we invented. [6:18] So continuous batching is the technique [6:21] we invented. Uh so it's introduced in [6:25] Orca in OSTI 2022. [6:28] Um, if you think about inference [6:30] traffic, it's very dynamic and uneven. [6:33] Uh, requests come in at irregular times [6:36] and they don't all take the same amount [6:38] of time to finish. That's why [6:40] traditional batching uh, grouping of set [6:43] of requests isn't enough. The industry [6:46] standard for handling this is continuous [6:49] batching. And with continuous batching, [6:52] you can dynamically add new requests to [6:54] the batch or remove completed ones from [6:57] the batch at a much fine grain level, [7:00] keeping the batch efficiently packed. [7:03] So this allows the system to maintain [7:06] high GPU utilization even when traffic [7:08] is dynamic and uneven. And we are using [7:11] a more optimized version of continuous [7:13] batching. [7:15] So it's this is just one of the [7:17] techniques uh many techniques we are [7:19] using [7:23] and friendly endpoints is providing [7:26] everything you need intelligent routing [7:29] for maximizing performance autoscaling [7:31] for handling traffic variation realtime [7:34] monitoring like tokens process latency [7:38] etc. live model updates and model [7:40] versioning high availability with 99.99% [7:45] uptime and on the-ly quantization and [7:48] spec decoding. So you don't need to run [7:50] any quantization tool uh before running [7:54] the model. Uh you can just click [7:56] quantization but button and that's it. [8:00] And we are applying the best available [8:02] quantization. reinvent it to the model [8:05] and you don't lose any model quality and [8:08] you know the model execution gets f [8:10] faster and the integration with hogging [8:13] face and race and biases with the [8:15] integration you can pull motors from [8:17] friendly or push to deploy directly from [8:20] hogging face or race and biases [8:25] so why friendly endpoints first it's [8:28] simple and versatile with oneclick [8:30] deployment for over 4 uh 123k models uh [8:37] directly from hugging face. I'm going to [8:39] show you a demo in a minute. Second is [8:41] blazingly fast. For example, our [8:43] platform supports over 500 tokens per [8:46] second for llama 3.18 billion on a [8:49] hopper GPU. Third, it's cost efficient, [8:53] delivering more than two times [8:55] throughput [8:57] of other solutions, so you can reduce [9:00] your cost. And finally, it's reliable at [9:02] scale, supporting trillions of tokens [9:05] process with 99.99% [9:08] availability. [9:14] So, uh can you play the video? [9:20] So, now let me show you how this works. [9:23] uh in this case we are uh choosing flux [9:26] dev model and we are fetching from [9:28] hogging face and we chose GPU and then [9:32] uh that's it uh it's creating an end [9:34] point [9:35] um now it's deploying uh in real time uh [9:40] it's really showing the real time [9:42] deployment [9:44] uh it's getting GPUs it's getting the [9:47] models and it's initiating the engine [9:50] and that's it and it's [9:53] And once it's deployed, you can test the [9:55] model with the playground. So you can [9:59] run the image generation in this case or [10:02] language model like llama 3.3 70 [10:06] billion. Uh you can check uh versions [10:10] logs and then uh you can also check [10:13] metrics over time. Um so it's all in the [10:17] platform. So you can you don't need to [10:19] worry about any of those. [10:22] Okay, great. [10:25] So now um [10:29] >> I'm going to show you deploy models [10:31] directly from hugging face hub with [10:33] friendly endpoints. [10:37] >> You go to page hugging face model page [10:39] and select our logo there. Right, [10:41] >> it's that simple. [10:45] >> You can deploy a model with just one. So [10:46] you click that, you go to the this page [10:49] and then just click the button [10:53] and that's it. [10:54] >> Once the deployment is complete, head [10:56] over to the friendly suite to access [10:57] even more powerful features. [11:01] Now you can enjoy the fastest and cost [11:03] effective inference with friendly AI's [11:05] GPU optimized inference engine. [11:09] So in hoging phase there are almost two [11:12] million models now and among two million [11:15] models you can try out like 400,000 [11:19] models right with friendly AI endpoints [11:24] you can just click on the motor page and [11:27] it's very powerful you can explore all [11:29] different types of models [11:34] and uh yeah uh of course we also have uh [11:38] friendly sublist API. With the API, you [11:41] can run any substantly. [11:44] We are expanding the models we are [11:47] supporting with sublex uh APIs. [11:52] And I talked about like privacy and [11:54] security requirements. Uh if you are [11:57] running companies with strict security [11:59] or privacy requirements, we also offer [12:02] friendly container. I our high [12:04] performance on-prem AI inference [12:06] solution [12:08] in particular if you're on AWS friendly [12:11] container is also available as an EKS [12:14] add-on. So you can just run a single [12:16] command to install and integrate it into [12:19] your EKS workflow and it's very simple. [12:25] Now friendly agent uh it's our agent [12:29] layer on top of model layer. It [12:32] orchestrates agent workflows, manages [12:35] memory, integrates with APIs, handles [12:37] model cores seamlessly [12:40] and with that uh we allow to deploy your [12:43] agent and scales it with low latency. It [12:47] brings our inference expertise to the [12:49] agent layer making agent AI simple, [12:51] versatile, fast, costefficient and [12:53] reliable. So if you are interested, you [12:56] can uh you know scan the QR code and uh [12:59] yeah sign up. [13:03] Uh and I'd like to share some real world [13:05] success stories. Uh first uh tech [13:08] companies like 12 labs uh I'm not sure [13:12] whether you heard about it. It's a video [13:14] search company in San Francisco Bay [13:16] area. Uh and uh startups they have cut [13:20] infrastructure cost by over 50% with our [13:23] solution. Um the second example is uh [13:26] LG. Uh you probably know LG Electronics. [13:29] They have a foundation model LG XR1. [13:33] It's a 32 billion custom model matching [13:36] clothes for OPUS anthropics model and [13:40] it's only 32 billion and they uh use our [13:44] platform to launch their model in under [13:46] 24 hours and it's all happening pretty [13:49] fast. [13:51] And third example is major enterprise [13:55] clients they switch to friendly [13:56] container to meet strict internal [13:58] security requirements and reduce GPUs. [14:02] So we have all these real world success [14:04] stories. [14:06] So getting started with friendly is [14:08] easy. You can um try deploying a model [14:11] with dedicated endpoints or try sublist [14:14] API with the sublist endpoints for [14:17] instance scale to try out different [14:19] models or you can try out onrem [14:22] deployment with our friendly container. [14:25] Just sign up and you're ready to go. [14:29] So to wrap up uh many AI products start [14:32] with promising pilot but too often they [14:35] store when it's time to run with real [14:37] users at scale. [14:39] And uh we think that your team should [14:41] focus on building great AI products, not [14:43] wrestling with infrastructure and we [14:46] will make sure your AI run fast, [14:49] affordable and reliable at scale and you [14:51] can leave the hassle of AI inference to [14:53] us. [14:56] Okay, so one more thing I have some uh [14:59] news for you. Uh we recently launched [15:01] the friendly AI 10K geni inference [15:03] credit program. If you are ready to [15:06] scale your AI with uh us, take advantage [15:09] of our program offering up to 10K in [15:12] free inference credits. Scan the QR code [15:14] to apply. It only takes two minutes. [15:19] Okay, thank you. I'm happy to take any [15:21] questions if there's time. I'll also be [15:23] around after session. Uh feel free to [15:25] come chat if you have any questions. [15:27] Thank you.