FriendliAI Rewires Inference Cloud as Multi-Step Agents Multiply Costs

The Frontier AI Inference Cloud for Agents With FriendliAI — Ai4

Ai4

Inference infrastructure replaces raw model scale as the core competitive battleground because multi-

AI-assisted summary of The Frontier AI Inference Cloud for Agents With FriendliAI — original source. Check the original for full context.

Source published: 2026-08-18

Detailed notes

## Background Published by Ai4 on YouTube on August 18, 2026, this talk features Brian Yoo, Chief Business Officer at FriendliAI. The presentation explores how autonomous agents have transformed AI inference economics from linear request scaling into multiplicative token execution, highlighting how FriendliAI rethinks inference infrastructure to overcome compounding cost and latency bottlenecks.

## TL;DR Brian Yoo argues that the primary frontier of AI competition has shifted from training larger models to optimizing inference execution, driven by open-weight models achieving quality parity with closed models at an 80% cost reduction. Unlike traditional chatbots, agentic workflows introduce iterative execution loops with input-to-output token ratios reaching 100:1, rendering standard request-level optimizations obsolete. To solve these compounding latency and prefix-accumulation challenges, FriendliAI developed a native C++ inference engine that combines continuous batching, advanced KV caching, custom quantization, and speculative decoding. Benchmark results demonstrate dramatic end-to-end performance gains, including a 7x speedup on coding agents and a 100-second latency reduction in multi-step game generation tasks.

## Atmosphere & style The presentation carries an authoritative, pragmatic, and systems-engineering-focused tone, trading generic AI buzzwords for concrete architectural realities and unit economics. Yoo speaks with measured clarity and technical conviction, structuring the talk around clean operational comparisons and real production benchmarks. It feels like an incisive industry briefing tailored for builders and enterprise leaders seeking practical solutions to the mounting latency and cost walls of scaling autonomous agents.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=8-NqZpStqhM&t=4s) The competitive advantage in AI has shifted from expanding model parameter size to optimizing the inference execution layer for speed, cost, and reliability. - [00:04](https://www.youtube.com/watch?v=8-NqZpStqhM&t=4s) FriendliAI pioneered continuous batching five years ago, creating what is now the global standard across LLM serving architectures. - [01:51](https://www.youtube.com/watch?v=8-NqZpStqhM&t=111s) Leading open-weight models now match closed frontier models on complex tasks while delivering up to an 80% reduction in inference costs. - [03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) AI interaction models have evolved through three distinct phases: single-turn chatbots, self-reflecting reasoning models, and autonomous multi-step agentic loops. - [03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) Agentic workflows break traditional request-level inference economics, fanning out single tasks into hundreds of interdependent tool and model executions. - [03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) Iterative agent execution loops cause rapid context accumulation, often driving input-to-output token ratios up to 100:1. - [05:41](https://www.youtube.com/watch?v=8-NqZpStqhM&t=341s) End-to-end task completion time and total task cost have replaced Time to First Token (TTFT) as the primary metrics for agent workloads. - [05:41](https://www.youtube.com/watch?v=8-NqZpStqhM&t=341s) FriendliAI built a ground-up C++ engine integrating prefix KV caching, speculative decoding, continuous batching, and custom quantization to optimize agent throughput. - [05:41](https://www.youtube.com/watch?v=8-NqZpStqhM&t=341s) Inference reliability is vital for agentic pipelines, where a single dropped call or rate limit can invalidate the full context history and force a complete restart. - [09:31](https://www.youtube.com/watch?v=8-NqZpStqhM&t=571s) Real-world benchmarks demonstrated that FriendliAI's agent cloud sped up coding agents like Kilocode by 7x and finished a game-generation task 100 seconds faster. - [09:31](https://www.youtube.com/watch?v=8-NqZpStqhM&t=571s) Latency improvements at the inference infrastructure layer directly impact business metrics, such as driving a 15% increase in player retention for interactive LLM games.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=8-NqZpStqhM&t=4s) The Shift from Model Quality to Inference Execution

Over the past three years, the artificial intelligence landscape has been dominated by a single race: who can build the better model. Every few months, organizations unveiled models boasting higher benchmark scores, larger parameter counts, and new frontiers of raw intelligence. Referencing Jensen Huang's keynote at GTC, the industry has now reached a critical inflection point where building larger models is no longer the sole priority. Instead, competitive advantage belongs to whoever can serve these models best, shifting the core challenge directly to the execution layer.

FriendliAI was founded five years ago with a distributed team spanning Seoul and San Francisco. The team originally invented continuous batching—a technique that has since evolved into the global industry standard across inference engines. In conversations over the past year with both enterprise customers and investors, the key questions have moved away from raw model architecture toward practical operational realities: how to deploy agents at scale, how to maintain reliability, and how to aggressively control operational costs.

- **Founding and Heritage**: Founded 5 years ago; invented continuous batching for LLM serving. - **Geographic Footprint**: Core engineering and business operations across San Francisco and Korea. - **Industry Inflection**: The narrative has pivoted from model parameter expansion to execution speed, cost efficiency, and serving infrastructure.

---

[01:51](https://www.youtube.com/watch?v=8-NqZpStqhM&t=111s) Open-Weight Models and the Economic Divergence

The transition toward execution-centric optimization is accelerated by open-weight models rapidly catching up to frontier proprietary offerings. Over the past twelve months, leading open-weight architectures such as Kimi, GLM, MiniMax, and NeMo-Megatron have matched or exceeded the performance of closed frontier models from OpenAI and Anthropic across a wide variety of production tasks.

```mermaid flowchart TD accTitle: Open-Weight vs Closed Model Economics Comparison accDescription: Flowchart illustrating the comparative cost difference between running GLM 5.2 and Opus 4.8 for building a tower defense game.

subgraph Task["Task: Build Tower Defense Game"] T["Identical Prompt & Requirements"] end

subgraph Execution["Model Execution"] T --> O["Claude Opus 4.8"] T --> G["GLM 5.2 (Open-Weight)"] end

subgraph Outcome["Economics & Quality"] O --> O_Res["Baseline Cost (100%) Target Quality: Complete"] G --> G_Res["80% Cheaper Cost (20%) Target Quality: Complete"] end ```

To demonstrate the dramatic shift in unit economics, FriendliAI evaluated both GLM 5.2 and Claude Opus 4.8 on an identical software development benchmark: generating a fully functional tower defense game. Both models successfully completed the task with virtually identical output quality, but GLM 5.2 achieved this at an 80% cost reduction compared to Opus 4.8. When baseline outputs achieve parity, inference unit economics dictate what developers can realistically build and scale.

- **Frontier Open Models**: Kimi, GLM (e.g., GLM 5.2), MiniMax, and NeMo-Megatron match closed frontier models. - **Cost Differential**: Up to 80% cost reduction on equivalent coding and generation tasks compared to high-end closed models like Opus 4.8.

---

[03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) The Evolution of AI Workloads: From Chatbots to Agents

The paradigm of AI interaction has undergone three distinct architectural waves since 2022:

1. **Chatbots (2022)**: Sparked by ChatGPT, users asked discrete questions and received direct answers via a simple, linear token-consumption model. 2. **Reasoning Models**: Introduced by DeepSeek, models began iterative self-reflection—asking whether an initial draft contained flaws and refining answers before responding. 3. **Agentic Workflows**: Emerging frameworks and platforms like OpenClaw, Hermes, and Kilo execute autonomous, multi-step tasks that trigger sprawling, complex execution graphs.

```mermaid flowchart LR accTitle: Agentic Task Execution Loop accDescription: Flowchart showing the cyclical loop of planning, tool and model execution, observation, and context accumulation in AI agents.

subgraph AgentLoop["Agent Execution Session"] direction TB P["Plan Next Steps"] --> A["Take Action (Tools, DBs, Model Calls)"] A --> O["Observe Output & Feedback"] O --> C{"Task Complete?"} C -- "No" --> P end

subgraph Memory["Context Overhead"] A -.-> CA["Accumulate KV History (Up to 100:1 Input:Output)"] CA -.-> P end

C -- "Yes" --> F["Final Delivery"] ```

Agentic workflows fundamentally break traditional request-level inference assumptions. A single session expands into an unpredictable cascade of hundreds or thousands of model inferences, database queries, and external tool calls. The model plans, takes an action, evaluates the output, processes environment or user feedback, and loops iteratively until completion. This causes the context window to accumulate continuously, turning token consumption into an exponential curve rather than a flat, predictable line.

- **Workload Shift**: Transition from isolated request-response calls to multi-turn task loops. - **Token Dynamics**: In traditional chat, token usage is linear; in agentic environments, tokens scale multiplicatively with execution steps. - **Context Accumulation**: Long-running agent sessions continuously append tool outputs, intermediate code, and system states to the prompt prefix.

---

[05:41](https://www.youtube.com/watch?v=8-NqZpStqhM&t=341s) Full-Stack Inference Engine for Multi-Step Workloads

In agentic workloads, conventional metrics like Time to First Token (TTFT) become secondary. Instead, end-to-end task completion time and total task cost determine practical viability. Because agent pipelines interleave pure model inference with non-model tasks (such as executing code or querying APIs), and input-to-output token ratios frequently reach 100:1, achieving high throughput requires a ground-up engineering approach rather than a single isolated trick.

```mermaid flowchart TD accTitle: Full-Stack Agentic Inference Engine Stack accDescription: Structural view of FriendliAI's custom C++ inference engine coordinating continuous batching, KV caching, speculative decoding, and quantization.

subgraph CoreEngine["FriendliAI Custom C++ Engine (Non-vLLM Wrapper)"] CB["Continuous Batching (Native Scheduling)"] KVC["Prefix & KV Cache Reuse (Eliminates Recompute on Shared Prefixes)"] SD["Speculative Decoding (Draft Models & Pattern Matching)"] Q["Hardware Quantization"] end

InputTokens["Agent Prompt & Context (100:1 Input-to-Output Ratio)"] --> CoreEngine CoreEngine --> HighThroughput["Fast, Reliable End-to-End Task Output"] ```

Rather than building a wrapper around generic open-source engines like vLLM, FriendliAI implemented a native C++ inference engine specifically tailored for agent-scale workloads. The engine tightly integrates four complementary pillars:

- **Advanced KV Caching**: Capitalizes on the 100:1 input-to-output ratio by persistently caching and reusing accumulated context prefixes across turns, bypassing expensive memory and compute recalculations. - **Speculative Decoding**: Employs smaller draft models and pattern-matching heuristics to predict upcoming tokens, maximizing useful token generation per GPU clock cycle. - **Custom Quantization & Memory Management**: Optimizes weight storage and throughput without sacrificing generation fidelity. - **Continuous Batching Orchestration**: Schedules dynamic, asynchronous agent calls efficiently across GPU clusters.

Reliability is equally vital: encountering a standard `HTTP 429` (rate-limiting) error in a simple chat application is an inconvenience, but in an agentic loop, a single dropped call can invalidate the entire context chain and force a complete restart of the task from scratch.

---

[09:31](https://www.youtube.com/watch?v=8-NqZpStqhM&t=571s) Production Benchmarks and Enterprise Results

To demonstrate the real-world impact of infrastructure-level optimization, FriendliAI conducted a head-to-head benchmark using an identical agent architecture, model weights, and prompt instructions to generate a playable *Candy Crush*-style game, changing only the underlying inference provider.

While a 100-second delta on a single coding run might seem incremental, scaled across tens or hundreds of thousands of daily automated tasks, it translates into massive compute cost reductions, superior user experiences, and measurable business retention. As Yoo concluded, open-weight models are fully enterprise-ready, and unlocking their capabilities requires an inference tier engineered specifically for the compounding demands of agentic workflows.

## Notable quotes > A chatbot's cost scales roughly linearly with the number of requests; an agent's scales multiplicatively. — Brian Yoo

> [00:04](https://www.youtube.com/watch?v=8-NqZpStqhM&t=4s) building larger models is no longer the sole priority. Instead, competitive advantage belongs to whoever can serve these models best, shifting the core challenge directly to the execution layer. — Brian Yoo

> [01:51](https://www.youtube.com/watch?v=8-NqZpStqhM&t=111s) When baseline outputs achieve parity, inference unit economics dictate what developers can realistically build and scale. — Brian Yoo

> [03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) A single session expands into an unpredictable cascade of hundreds or thousands of model inferences, database queries, and external tool calls. — Brian Yoo

> [03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) In traditional chat, token usage is linear; in agentic environments, tokens scale multiplicatively with execution steps. — Brian Yoo

> [05:41](https://www.youtube.com/watch?v=8-NqZpStqhM&t=341s) In agentic workloads, conventional metrics like Time to First Token (TTFT) become secondary. Instead, end-to-end task completion time and total task cost determine practical viability. — Brian Yoo

> [05:41](https://www.youtube.com/watch?v=8-NqZpStqhM&t=341s) encountering a standard HTTP 429 (rate-limiting) error in a simple chat application is an inconvenience, but in an agentic loop, a single dropped call can invalidate the entire context chain and force a complete restart of the task from scratch. — Brian Yoo

> [09:31](https://www.youtube.com/watch?v=8-NqZpStqhM&t=571s) While a 100-second delta on a single coding run might seem incremental, scaled across tens or hundreds of thousands of daily automated tasks, it translates into massive compute cost reductions, superior user experiences, and measurable business retention. — Brian Yoo

> [09:31](https://www.youtube.com/watch?v=8-NqZpStqhM&t=571s) open-weight models are fully enterprise-ready, and unlocking their capabilities requires an inference tier engineered specifically for the compounding demands of agentic workflows. — Brian Yoo

## Mentioned Brian Yoo, Jensen Huang, FriendliAI, Ai4, GTC, OpenAI, Anthropic, DeepSeek, Kimi, GLM 5.2, Claude Opus 4.8, MiniMax, NeMo-Megatron, ChatGPT, OpenClaw, Hermes, Kilo, Kilocode, vLLM, Candy Crush, Tactile Story

## Video description Brian Yoo - Chief Business Officer, FriendliAI

Agents have changed the economics of AI inference. A chatbot's cost scales roughly linearly with the number of requests; an agent's scales multiplicatively. A single task can fan out into hundreds of model calls, each carrying a repeated context prefix and adding latency that compounds across tool calls and reasoning steps. As open-weight models keep improving and agentic workloads grow, this shift exposes the limits of traditional request-level optimization. Inference infrastructure becomes a first-class concern, one that often shapes performance and cost as much as the model itself. I'll explore what changes when you optimize for the whole task rather than the individual request, and how FriendliAI is rethinking the inference cloud for the era of agentic AI. If you're building agents and hitting the wall on cost or latency, this talk will give you a new lens to think about your inference layer.

[Source](https://www.youtube.com/watch?v=8-NqZpStqhM)

中文简报

多步智能体推高成本，FriendliAI重构推理云推理基础设施取代原始模型规模成为核心竞争战场，因为多…… Brian Yoo认为，AI竞争的主要前沿已从训练更大规模的模型转向优化推理执行，其推动因素在于开放权重模型在降低80%成本的同时达到了与闭源模型相当的质量水平。与传统聊天机器人不同，智能体工作流引入了输入与输出Token比例高达100:1的迭代执行循环，使传统的请求级优化方法失效。为应对不断复合的延迟和前缀累积挑战，FriendliAI开发了原生C++推理引擎，结合了连续批处理、先进KV缓存、定制量化和推测解码技术。基准测试结果显示其端到端性能取得显著提升，包括使编程智能体提速7倍，并在多步游戏生成任务中减少了100秒的延迟。 AI 行业的竞争优势已从模型参数规模转向推理执行层的速度、成本和可靠性优化；FriendliAI 五年前开创的连续批处理（Continuous Batching）现已成为服务架构的全球标准。 (0:04) 主流开源权重模型在复杂任务中已能匹敌闭源前沿模型，在塔防游戏代码生成等场景下实现了高达 80% 的推理成本缩减。 (1:51) 自主多步 Agent 工作流打破了传统的单请求推理经济学，一次任务往往派生出数百次相互依赖的工具与模型调用，导致输入与输出 Token 比例高达 100:1。 (3:03) 端到端任务完成时间与全流程总成本已取代首字延迟（TTFT）成为 Agent 负载的核心指标，底层原生 C++ 推理引擎通过前缀 KV 缓存与投机解码有效消除了复合延迟。 (5:41)

Reading summary

FriendliAI argues that competitive advantage in artificial intelligence has pivoted from scaling raw model parameters to optimizing the inference execution layer. As open-weight models achieve parity with closed proprietary systems, purpose-built inference infrastructure is required to handle the compounding compute and memory demands of multi-step agentic workflows. Inference Execution and Open-Weight Economic Parity (0:04) Referencing Jensen Huang's keynote at GTC, FriendliAI states that expanding parameter counts is no longer the sole determinant of AI advantage. The primary industry challenge has shifted to the execution layer: reliably deploying models at scale while strictly controlling operational costs. This transition is reinforced by open-weight models—such as Kimi, GLM, MiniMax, and NeMo-Megatron—matching or exceeding the performance of closed frontier models from OpenAI and Anthropic across practical tasks. To demonstrate this economic shift, FriendliAI tested GLM 5.2 against Claude Opus 4.8 on a software development benchmark requiring the generation of a functional tower defense game. Both models completed the task with virtually identical output quality, but GLM 5.2 achieved this at an 80% cost reduction. FriendliAI argues that once baseline outputs reach parity, inference unit economics dictate what developers can realistically build and scale. Structural Bottlenecks in Multi-Step Agentic Workloads (3:03) AI workload architectures have progressed across three waves: 2022 chatbots marked by linear token consumption, reasoning models using iterative self-reflection, and emerging agentic frameworks like OpenClaw, Hermes, and Kilo. Agentic workflows break standard request-level serving assumptions by triggering cascading execution loops involving hundreds or thousands of model inferences, database queries, and external tool calls. Because agent sessions iteratively plan, act, observe environment feedback, and append tool outputs to the context window, token consumption scales multiplicatively. Input-to-output token ratios regularly reach 100:1. System reliability also becomes paramount in this environment: while an HTTP 429 rate-limit error is an inconvenience for a chatbot, a dropped call inside an agent loop invalidates the accumulated context chain and forces the entire multi-step task to restart from scratch. Engine Optimization for Asymmetric Token Ratios (5:41) In agentic systems, conventional metrics such as Time to First Token (TTFT) become secondary to end-to-end task completion time and total execution cost. To address these demands without relying on open-source wrappers like vLLM, FriendliAI developed a native C++ inference engine designed specifically for the compounding overhead of multi-turn agent pipelines. The engine coordinates four technical pillars: continuous batching to schedule asynchronous agent calls across GPU clusters; advanced KV caching to eliminate recomputing shared prefixes across the 100:1 input-heavy context history; speculative decoding using draft models and pattern heuristics to maximize token throughput per clock cycle; and custom hardware quantization to optimize weight storage without degrading output fidelity. Task-Level Performance and Enterprise Production Benchmarks (9:31) To evaluate infrastructure-level impacts, FriendliAI benchmarked identical agent architectures, model weights, and prompts while alternating only the underlying inference provider. In an automated test generating a playable Candy Crush-style game, FriendliAI's cloud completed the task 100 seconds faster than a standard inference provider by reducing compounding tool-call latency. In domain-specific applications, the open-source coding agent Kilocode ran 7x faster (a 700% speedup) during multi-turn code synthesis and testing. In interactive gaming deployments with Tactile Story, lower inference latency drove a 15% increase in player retention, showing that execution speed directly impacts end-user engagement in automated, real-time workflows. FriendliAI concludes that open-weight models are fully enterprise-ready, but unlocking their cost and performance potential depends on replacing generic serving layers with inference infrastructure engineered specifically for compounding, multi-step agent workloads.

精炼阅读摘要

FriendliAI指出，人工智能行业的竞争焦点已从单纯扩大模型参数转向推理执行层的效率与可靠性；在开源权重模型追平闭源模型的背景下，专为Agent复杂工作流设计的推理架构成为决定规模化落地经济效益的关键。推理执行成为核心竞争壁垒 (0:04) 过去三年AI行业的竞争主要聚焦于模型规模与跑分表现，但随着技术发展，行业已到达关键转折点。引用黄仁勋在GTC大会上的演讲观点，单纯构建更大模型的收益正在减弱，竞争优势开始转向如何以最高效率将模型部署和服务落地，底层执行架构成为关键。作为连续批处理（continuous batching）技术的发明团队，FriendliAI在首尔与旧金山设有团队，专注于推理工程已有五年。在与企业客户和投资者的交流中，核心关注点已从原始的模型架构转向实际的运营挑战，即如何在大规模场景下可靠部署Agent并严格控制运营成本。开源模型追赶与推理经济学分化 (1:51) 开源权重模型（如Kimi、GLM、MiniMax和NeMo-Megatron）在过去一年中快速追平甚至超越了OpenAI和Anthropic等闭源头部模型在生产任务中的表现。模型质量的趋同使得推理环节的单价成本（unit economics）成为制约开发者规模化落地的核心变量。在生成塔防游戏的基准测试中，FriendliAI对比了开源模型GLM 5.2与Claude Opus 4.8的表现。测试结果显示，两者在完全相同的提示词和需求下均能完整生成功能齐备的游戏，产出质量基本一致，但GLM 5.2的运行成本较Opus 4.8降低了80%，证明了开源权重模型在生产环境中具备显著的成本优势。 Agent架构引发推理负载本质变革 (3:03) AI工作负载已从2022年ChatGPT式的单轮对话，演变为引入自我反思的推理模型，并进一步进化为OpenClaw、Hermes和Kilo等自主多步骤Agent。传统的对话式交互呈现线性的Token消耗，而Agent通过“规划-行动-观察-评估”的循环执行网络，会带来不可预测的复杂模型调用、数据库查询和外部工具操作。在这种工作流中，长期运行的会话持续将工具返回结果、中间代码及系统状态追加至提示前缀中，导致输入与输出的Token比例经常高达100比1。这种连续累积的上下文使得计算开销呈指数级扩张，彻底打破了传统单次请求级别的推理假设，使得常规的首字延迟（TTFT）指标退居次要地位，端到端任务完成时间和总体成本成为衡量系统的实际标准。全栈推理引擎的技术实现与稳定性要求 (5:41) 面对Agent工作流的高上下文累积和高频输入特征，通用开源框架的简单封装已难以满足吞吐需求。FriendliAI构建了自研C++推理引擎，整合了前缀与KV缓存复用、投机解码（利用草稿模型和模式匹配）、定制量化以及原生连续批处理调度等技术，从而在不损害生成精度的前提下，避免重复计算并最大化GPU时钟周期的利用率。此外，工程可靠性在多步骤任务中呈现出完全不同的风险形态。在普通聊天应用中，遭遇HTTP 429限流仅是暂时的体验不佳，但在依赖复杂依赖链的Agent循环中，单次接口失败就可能导致整个执行上下文失效，迫使整个自动化任务从头重启，因此高并发下的执行稳定性成为基础设施的核心要求。生产基准测试中的性能与业务收益 (9:31) 在保持相同的Agent架构、模型权重与指令输入的测试中，将推理底座替换为FriendliAI平台后，生成可运行的类Candy Crush游戏任务耗时缩短了100秒，消除了多次工具调用带来的延迟累加效应。在其他生产案例中，开源代码Agent项目Kilocode的端到端运行速度提升至基准供应商的7倍；在LLM驱动的游戏应用Tactile Story中，交互延迟的降低直接推动玩家留存率提升了15%，证明了底层推理延迟和吞吐量的优化能够直接转化为企业的计算成本缩减与业务指标增长。 FriendliAI认为开源权重模型已完全具备企业级生产能力，但要真正释放其潜能，基础设施层必须针对Agent工作流中上下文持续累加、高并发调用与高输入输出比的特性进行全栈重构，以应对复合延迟与成本扩张的挑战。

Detailed notes — zh

## Background 本视频由 Ai4 于 2026 年 8 月 18 日发布在 YouTube 平台上。内容为 FriendliAI 首席商务官 Bryan Yoo 的主题演讲，主要探讨 Agent（智能体）工作流如何颠覆 AI 推理的经济模型与工程范式，并介绍了 FriendliAI 如何重构面向 Agent 时代的全栈推理云基础设施。 ## TL;DR 随着开源权重模型在性能上逼近前沿闭源模型并带来高达 80% 的成本降幅，AI 行业的核心叙事已从“模型参数竞争”转向“工程服务化交付”。Agent 的自主规划与工具调用循环引发了高达 100:1 的输入输出 Token 比例与级数累积的上下文，使传统请求级优化面临瓶颈。FriendliAI 采用 C++ 从零自研底层推理引擎，通过长上下文 KV Cache 复用与投机解码等技术实现端到端任务加速。实测表明，该方案不仅在复杂游戏代码生成中提速 100 秒，还为编程 Agent 带来了 7 倍的速度优势，并显著提升了终端产品的用户留存率。 ## Atmosphere & style 整个演讲节奏紧凑务实，充满硬核工程派的技术自信与商业洞察力。Bryan Yoo 的讲解条理清晰且层层递进，从行业宏观范式演进自然过渡到底层架构细节与真实客户案例。现场演示与数据对比直观硬核，将原本晦涩的推理系统吞吐、KV Cache 机制与时延成本优化讲得极具说服力与冲击感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=8-NqZpStqhM&t=4s) AI 行业迎来核心拐点，竞争焦点已从单纯追求模型参数规模与基准跑分，全面转向推理服务化交付（Serving）与工程执行力。 - [00:04](https://www.youtube.com/watch?v=8-NqZpStqhM&t=4s) FriendliAI 团队首创了 Continuous Batching（连续批处理）技术，该技术目前已成为大模型推理系统的行业标准架构。 - [01:51](https://www.youtube.com/watch?v=8-NqZpStqhM&t=111s) 开源权重（Open-weight）模型在实际业务场景中迅速崛起，其综合表现已在诸多任务上逼平甚至超越顶尖闭源前沿模型。 - [01:51](https://www.youtube.com/watch?v=8-NqZpStqhM&t=111s) 开源与闭源模型的单位经济模型差异巨大，在构建塔防游戏等复杂任务中，开源模型在同等产出质量下可降低 80% 的推理成本。 - [03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) AI 交互范式经历了从 Chat 单轮对话、Reasoning 自我反思到 Agent 自主工作流的三阶段断层演进。 - [03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) Agent 场景下单次任务会衍生出成百上千次推理与工具调用，上下文以前缀形式指数累积，输入与输出 Token 比例高达 100:1。 - [03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) 推理性能评估指标发生根本性转变，企业不再单纯关注单次请求的首字延迟（TTFT），而是聚焦于端到端任务完成速度与总成本。 - [06:25](https://www.youtube.com/watch?v=8-NqZpStqhM&t=385s) 单靠单一优化技巧无法解决 Agent 推理瓶颈，FriendliAI 放弃了开源框架封装，使用 C++ 从零构建了全栈自研推理引擎。 - [06:25](https://www.youtube.com/watch?v=8-NqZpStqhM&t=385s) 通过对长上下文 KV Cache 的深度复用与投机解码（Speculative Decoding），能够大幅缩减 Prefill 阶段开销并最大化 GPU 计算效率。 - [06:25](https://www.youtube.com/watch?v=8-NqZpStqhM&t=385s) 在 Agent 长链条多步调用中，系统可靠性至关重要，必须消除 429 报错及中断异常，以避免单点失败导致高昂的上下文全流程重算。 - [09:31](https://www.youtube.com/watch?v=8-NqZpStqhM&t=571s) 实测对比显示，在运行相同 Agent 与模型生成游戏代码时，FriendliAI 相比竞品能够将端到端完成时间缩短 100 秒。 - [09:31](https://www.youtube.com/watch?v=8-NqZpStqhM&t=571s) 落地成效显著：开源编程 Agent Kilocode 实现了 7 倍的推理提速，互动游戏开发商 Tactile Story 则借由延迟降低提升了 15% 的玩家留存率。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=8-NqZpStqhM&t=4s) AI 拐点：从“模型之争”转向“工程交付” FriendliAI 首席商务官（CBO）Bryan Yoo 登台发表演讲。他回顾了过去三年 AI 领域的发展主线：整个行业的核心叙事一直是“谁能做出更好的模型”，每隔几个月就会有参数更大、基准跑分更高的新模型问世，不断刷新智能的前沿。然而正如黄仁勋在 GTC 上指出的那样，行业已经迎来了拐点——如今的核心命题不再是谁能训出更强的模型，而是谁能以最优方式进行服务化交付（Serving）。工程执行力（Execution）成为了决定胜负的下一层关键。 Bryan 介绍了 FriendliAI 的背景：团队分布在韩国首尔和美国旧金山，公司创立已有 5 年。他们团队最初发明了 Continuous Batching（连续批处理技术），如今这项技术已成为各大 AI 公司的行业标配。在过去的 5 年中，团队始终专注于一个根本问题：“如何让 AI 运行得更快、更可靠、更高效？”在过去一年与众多客户和投资者的交流中，Bryan 发现大家探讨的核心问题已全面转向：“我该如何部署？”“如何在大规模场景下运行 Agent？”以及“如何控制推理成本？” - **团队背景**：FriendliAI 成立于 5 年前，团队分布在韩国与旧金山。 - **行业贡献**：首创 Continuous Batching 技术，如今已成为大模型推理的标准架构。 - **行业叙事转移**：从追求模型参数与跑分，全面转向推理部署、大规模 Agent 运行与成本控制。 --- ### [01:51](https://www.youtube.com/watch?v=8-NqZpStqhM&t=111s) 开源权重崛起与单任务经济学差异推动行业叙事转向执行层的核心原因之一，在于开源权重（Open-weight）模型的性能已经大幅追赶上来。在过去 12 个月里，包括 Kimi、GLM、MiniMax 以及基于 NeMo-Megatron 等顶级开源模型，在实际表现上已经逼平 OpenAI 和 Anthropic 的前沿闭源模型，在诸多实际业务场景中甚至打平或超越了闭源产品。为了直观展示经济效益的差距，Bryan 分享了一组对比实验：让开源模型 GLM 5.2 与闭源模型 Claude 3 Opus 4.8 分别执行“构建一个塔防游戏”的复杂任务。两者的最终生成质量几乎一致，但两者的单位经济模型（Unit Economics）却截然不同——GLM 5.2 完成该任务的推理成本比 Opus 4.8 降低了整整 80%。同样的任务、同等的产出质量，却能节省 80% 的开销，这直接重塑了企业应用 AI 的商业可行性。 ```mermaid flowchart LR accTitle: Open-weight vs Frontier Model Economics subgraph S1["任务：构建塔防游戏"] direction TB T["输入相同 Prompt/需求"] end subgraph S2["模型执行对比"] direction TB M1["开源模型: GLM 5.2 输出质量等同"] M2["前沿闭源: Claude Opus 4.8 输出质量等同"] end subgraph S3["成本结果"] direction TB C1["成本降低 80% 单位经济模型大幅优化"] C2["高昂基线成本"] end T --> M1 T --> M2 M1 --> C1 M2 --> C2 ``` - **开源模型表现**：Kimi、GLM、MiniMax 等开源模型在多样化任务中已不输顶尖闭源模型。 - **80% 成本降幅**：在开发塔防游戏的端到端测试中，GLM 5.2 相比 Opus 4.8 实现了 80% 的成本削减。 --- ### [03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) 从 Chat 到 Agent：推理范式的断层演进 AI 应用的交互范式经历了解构与重塑。2022 年 ChatGPT 的诞生展现了 AI 的可能性，其模式为简单的单轮问答（Chat-based），即“用户提问，模型吐出答案”，Token 的消耗呈现线性增长。随后以 DeepSeek 为代表的推理模型（Reasoning Models）带来了第二次突破，模型在输出前会进行“自我反思”，反复质问自身答案是否最优、是否存在逻辑漏洞，从而解决更复杂的问题。到了当前阶段，以 OpenClaw、Hermes、Kilo 等为代表的自主 Agent 成为主角，交互形式从单一对话升级为复杂的自动化工作流（Workflow）。从 Chatbot 跨越到 Agent 彻底颠覆了推理底层的运行逻辑。单次 Agent Session 会衍生出数十甚至上百个子任务，触发成百上千次推理调用、工具调用（Tool Calls）及数据库查询。Agent 在“规划（Plan）- 行动（Action）- 观察结果/接收反馈（Observe）- 迭代循环”的闭环中持续运行。这种模式带来极高挑战：上下文（Context）以前缀累积的方式呈指数级暴涨，模型工作与非模型工作频繁切换，且调用次数完全动态不可测。用户关心的指标不再是单次请求的首字延迟（TTFT），而是完成整个任务的**端到端耗时与总体成本**。 ```mermaid flowchart TD accTitle: Agentic Loop and Context Growth subgraph Loop["Agent 循环工作流"] direction TB P["1. 规划 (Plan)"] --> A["2. 行动: 工具调用/DB查询 (Act)"] A --> O["3. 观察结果/获取反馈 (Observe)"] O --> I{"任务是否完成?"} I -- 否: 累积上下文并迭代 --> P I -- 是 --> F["任务完成输出"] end subgraph InfrImpact["推理基础设施影响"] direction TB CTX["上下文 Context 级数累积 输入/输出比例高达 100:1"] COST["调用次数不可预测 端到端任务延迟与成本激增"] end Loop -.-> InfrImpact ``` - **交互演进三阶段**： 1. *Chat 时代（2022）*：单轮 Prompt-Response，Token 线性消耗。 2. *Reasoning 时代*：模型内置思考与自我纠错机制。 3. *Agent 时代*：自动化工作流，单次任务分发成百上千次推理与工具调用。 - **推理诉求转变**：从关注 TTFT（Time to First Token）转向关注“端到端任务完成速度与总成本”。 --- ### [06:25](https://www.youtube.com/watch?v=8-NqZpStqhM&t=385s) 全栈重构：FriendliAI 面向 Agent 的底层推理引擎面对客户和投资者“为什么你们能跑得这么快？有什么独门秘方？”的疑问，Bryan 强调推理加速绝非单一技巧，而是全栈工程体系的协同。单靠 Continuous Batching 远远不够，必须将 KV Cache 优化、量化（Quantization）以及投机解码（Speculative Decoding）等技术深度打通。为了实现极致性能，FriendliAI 放弃了基于 vLLM 等开源框架进行封装的常规做法，而是使用 C++ 从零（From scratch）构建了整套底层推理引擎。针对 Agent 工作流中高达 100:1 的输入输出 Token 比例（Input-to-Output Token Ratio），FriendliAI 进行了关键优化： 1. **长上下文 KV Cache 深度复用**：Agent 频繁携带相同的前缀历史，通过高效 Cache 机制避免重复计算，实现“能读缓存绝不重复算显存”，大幅缩减 Prefill 阶段的计算与内存开销。 2. **投机解码（Speculative Decoding）**：利用小型模型或模式匹配机制预判后续 Token，最大化榨干 GPU 计算周期，使每个周期产出更多有效 Token。 3. **极致系统可靠性**：在长链条 Agent 工作流中，API 报 `429 Too Many Requests` 或瞬时失败是灾难性的，单点报错往往意味着前面累积的大量计算全部作废并必须重跑。FriendliAI 针对端到端 API 稳定性进行了专项加固。 - **自研引擎**：全 C++ 自研推理底层，非开源框架二次包装。 - **100:1 极端比例**：针对 Agent 场景超大 Context Prefill 特性进行针对性缓存重用优化。 - **可靠性优先**：消除 429 报错及中断异常，避免 Agent 长链条任务因单步失败导致全流程重算。 --- ### [09:31](https://www.youtube.com/watch?v=8-NqZpStqhM&t=571s) 实测数据验证与客户落地成效 Bryan 现场展示了一组同屏对比实测：运行完全相同的 Agent、相同的模型与相同的任务指令——“编写一个类似 Candy Crush（消消乐）的游戏”，唯一的变量是底层推理提供商。结果显示，在生成代码功能与质量完全一致的情况下，FriendliAI 驱动的推理流程比竞品快了整整 100 秒。在单日执行数十万次复杂任务的企业生产环境中，节约 100 秒不仅改善了终端交互体验，更能直接颠覆业务的盈利模型。此外，Bryan 分享了两个典型的客户实测案例： - **Kilocode（顶级开源编程 Agent）**：在评测了市场上的主流推理服务商后，FriendliAI 展现出 **7 倍的速度优势**（“并非 7% 或 70%，而是足足 7 倍”），且服务可用性明显更高。 - **Tactile Story（LLM 驱动的游戏开发商）**：对于互动叙事类游戏，推理延迟直接关乎生命线。在切换至 FriendliAI 的推理底座后，其游戏终端用户的次日/长周期留存率提升了 15%。曾任游戏产品经理的 Bryan 强调，在游戏行业中 15% 的留存率提升足以决定一款产品的生死存亡。最后，Bryan 总结道：开源权重模型已经完全具备生产可用性，企业若想将 Agent 工作流落地于实际生产环境，必须选用能够将推理各层高效整合的专业基础设施。 | 客户/场景 | 对比维度 | FriendliAI 表现与业务收益 | |:--- |:--- |:--- | | **消消乐代码生成实测** | 相同 Agent 与模型生成游戏代码 | 任务完成时间缩短 **100 秒** | | **Kilocode (代码 Agent)** | 对比多家第三方推理服务提供商 | 端到端推理速度达到竞品的 **7 倍** | | **Tactile Story (LLM 游戏)** | 游戏内模型响应延迟与稳定性 | 终端玩家留存率（Retention）提升 **15%** | ## Notable quotes > [00:04](https://www.youtube.com/watch?v=8-NqZpStqhM&t=4s) 如今的核心命题不再是谁能训出更强的模型，而是谁能以最优方式进行服务化交付（Serving）。工程执行力（Execution）成为了决定胜负的下一层关键。 — Bryan Yoo > [00:04](https://www.youtube.com/watch?v=8-NqZpStqhM&t=4s) 如何让 AI 运行得更快、更可靠、更高效？ — Bryan Yoo > [01:51](https://www.youtube.com/watch?v=8-NqZpStqhM&t=111s) GLM 5.2 完成该任务的推理成本比 Opus 4.8 降低了整整 80%。 — Bryan Yoo > [03:03](https://www.youtube.com/watch?v=8-NqZpStqhM&t=183s) 用户关心的指标不再是单次请求的首字延迟（TTFT），而是完成整个任务的端到端耗时与总体成本。 — Bryan Yoo > [06:25](https://www.youtube.com/watch?v=8-NqZpStqhM&t=385s) 推理加速绝非单一技巧，而是全栈工程体系的协同。 — Bryan Yoo > [06:25](https://www.youtube.com/watch?v=8-NqZpStqhM&t=385s) 能读缓存绝不重复算显存 — Bryan Yoo > [06:25](https://www.youtube.com/watch?v=8-NqZpStqhM&t=385s) 在长链条 Agent 工作流中，API 报 429 Too Many Requests 或瞬时失败是灾难性的，单点报错往往意味着前面累积的大量计算全部作废并必须重跑。 — Bryan Yoo > [09:31](https://www.youtube.com/watch?v=8-NqZpStqhM&t=571s) 并非 7% 或 70%，而是足足 7 倍 — Bryan Yoo > [09:31](https://www.youtube.com/watch?v=8-NqZpStqhM&t=571s) 在游戏行业中 15% 的留存率提升足以决定一款产品的生死存亡。 — Bryan Yoo ## Mentioned Bryan Yoo, FriendliAI, 黄仁勋, NVIDIA GTC, Continuous Batching, Kimi, GLM, GLM 5.2, MiniMax, NeMo-Megatron, OpenAI, Anthropic, Claude 3 Opus 4.8, ChatGPT, DeepSeek, OpenClaw, Hermes, Kilo, vLLM, Kilocode, Tactile Story, Candy Crush ## Video description Brian Yoo - Chief Business Officer, FriendliAI Agents have changed the economics of AI inference. A chatbot's cost scales roughly linearly with the number of requests; an agent's scales multiplicatively. A single task can fan out into hundreds of model calls, each carrying a repeated context prefix and adding latency that compounds across tool calls and reasoning steps. As open-weight models keep improving and agentic workloads grow, this shift exposes the limits of traditional request-level optimization. Inference infrastructure becomes a first-class concern, one that often shapes performance and cost as much as the model itself. I'll explore what changes when you optimize for the whole task rather than the individual request, and how FriendliAI is rethinking the inference cloud for the era of agentic AI. If you're building agents and hitting the wall on cost or latency, this talk will give you a new lens to think about your inference layer. [Source](https://www.youtube.com/watch?v=8-NqZpStqhM)

Source quotes

[0:04] Bryan Yoo: 如今的核心命题不再是谁能训出更强的模型，而是谁能以最优方式进行服务化交付（Serving）。工程执行力（Execution）成为了决定胜负的下一层关键。 [0:04] Bryan Yoo: 如何让 AI 运行得更快、更可靠、更高效？ [1:51] Bryan Yoo: GLM 5.2 完成该任务的推理成本比 Opus 4.8 降低了整整 80%。 [3:03] Bryan Yoo: 用户关心的指标不再是单次请求的首字延迟（TTFT），而是完成整个任务的端到端耗时与总体成本。 [6:25] Bryan Yoo: 推理加速绝非单一技巧，而是全栈工程体系的协同。 [6:25] Bryan Yoo: 能读缓存绝不重复算显存 [6:25] Bryan Yoo: 在长链条 Agent 工作流中，API 报 429 Too Many Requests 或瞬时失败是灾难性的，单点报错往往意味着前面累积的大量计算全部作废并必须重跑。 [9:31] Bryan Yoo: 并非 7% 或 70%，而是足足 7 倍 [9:31] Bryan Yoo: 在游戏行业中 15% 的留存率提升足以决定一款产品的生死存亡。

Source quotes

Brian Yoo: A chatbot's cost scales roughly linearly with the number of requests; an agent's scales multiplicatively. [0:04] Brian Yoo: building larger models is no longer the sole priority. Instead, competitive advantage belongs to whoever can serve these models best, shifting the core challenge directly to the execution layer. [1:51] Brian Yoo: When baseline outputs achieve parity, inference unit economics dictate what developers can realistically build and scale. [3:03] Brian Yoo: A single session expands into an unpredictable cascade of hundreds or thousands of model inferences, database queries, and external tool calls. [3:03] Brian Yoo: In traditional chat, token usage is linear; in agentic environments, tokens scale multiplicatively with execution steps. [5:41] Brian Yoo: In agentic workloads, conventional metrics like Time to First Token (TTFT) become secondary. Instead, end-to-end task completion time and total task cost determine practical viability. [5:41] Brian Yoo: encountering a standard HTTP 429 (rate-limiting) error in a simple chat application is an inconvenience, but in an agentic loop, a single dropped call can invalidate the entire context chain and force a complete restart of the task from scratch. [9:31] Brian Yoo: While a 100-second delta on a single coding run might seem incremental, scaled across tens or hundreds of thousands of daily automated tasks, it translates into massive compute cost reductions, superior user experiences, and measurable business retention. [9:31] Brian Yoo: open-weight models are fully enterprise-ready, and unlocking their capabilities requires an inference tier engineered specifically for the compounding demands of agentic workflows.

Original transcript

[0:04] Good morning, everyone. My name's Bryan. I'm a CBO of an AI. Um so, over the last 3 years, [0:11] um AI has been about who can build a better model. Right? Every few months, new model would come out [0:17] that scores better on benchmark, um that's bigger parameters, uh and then it pushes a boundary [0:24] pushes a frontier of intelligence. Um but as as Jensen Huang mentioned you doing a GT GT [0:30] this year, I think we are at an inflection point. It's no longer about who can build a better model, [0:36] it's about who can serve it the best. So, execution is the is the next next layer. [0:43] So, um I'm originally from Korea from I'm I live in San Francisco, but my team is in Korea and [0:49] San Francisco. So, we started a company 5 years ago. Um our team invented continuous patching, [0:55] which is now the industry standard in in in a lot of new firms. Um and over the last 5 years, [1:02] um we really focused on one question. How do we make AI run faster, reliably, uh and efficiently? [1:11] And now, you know, that question is becoming one of the big uh big pillars of AI. [1:19] Yeah, so over the last year, um I've talked to many customers, investors alike, and the and the [1:25] question is is shifting from is this that who can build a better model to who can serve it the best, [1:33] um and uh question becomes um how do I deploy it? How do I run this agent at scale? Um how do [1:40] I control the cost? So, narrative is is quickly shifting from the model quality to execution. [1:51] And And that's largely because these open way models have largely caught up on performance. [1:57] So, over the last 12 months, um top open main models like Kimi, GLM, uh MiniMax, NeMo-Megatron, [2:04] they've largely caught up on performance with frontier models from OpenAI and and Anthropic. [2:09] Um and in and in many use cases, it's it's performing at on par or or even better than uh [2:17] those those closed models. So, take a look. So, we gave the same task to GLM 5.2 and Opus 4.8. [2:26] Um the task was to build a tower defense game. [2:29] And and they they largely complete the complete the task in similar quality. [2:37] But the economics are very different. Um with GLM 5.2, it completes the task 80 80% cheaper [2:46] than than Opus 4.8. So, same task, almost identical quality of output, [2:53] very different unit economics. So, this enables this this this changes what you can do with AI. [3:03] Um and then and the next big thing is the a the agents. So, in 2022, [3:08] when ChatGPT came out, um it was a it was a revolutionary uh breakthrough. [3:14] Um so, it it um ChatGPT showed the world what was possible with AI. [3:19] It was uh chat-based, so uh user would ask a question, model would spit out an answer. [3:26] And then the next big breakthrough was the reasoning model introduced by Deep Seek. [3:31] So, instead of just giving a first output, the model started thinking or or reasoning. [3:37] So, model started asking itself, "Hey, is this the best answer? Can I do better? [3:43] Are there any flaws with my first answer?" So, it kept on iterating until it find the best answer. [3:49] So, it was able to solve more complex problem. Then this year came the agents, [3:55] like Open Claw, Hermes, Kilo. These agents enabled um solving a a task or more more complex problems. [4:06] Um and it's the instead of just being a a chat base, now there are workflows. Uh [4:15] so as you move from the chatbots to agents, how you how you do inference changes dramatically. [4:26] Because this is what actually happens um in the in the workflow. In the chat GPT era, um usually what [4:33] I ask question, get an answer, very linear usage of tokens. But in agentic era, user inputs a task [4:42] or prompt, and it triggers hundreds or thousands of inference calls, tool calls, or whatnot. So it [4:49] becomes a very challenging uh problem to solve because the context context gets accumulated. [4:57] And in the number of tokens you use or or agent uses exponentially increases with usage. [5:07] And this is this is why um because an agent runs a session of tasks. So a single session turns into [5:14] many tasks. Each each task consisting of many inference calls or tool calls. [5:20] Um so a model first a agent first plans and take actions, whether that's [5:25] database queries, model calls, or whatnot. And it observes the outcome or gets feedback from user, [5:32] and iterates, and repeats the whole thing over and over again until the task is completed. [5:41] So it becomes a very challenging or different inference problem [5:45] um because context accumulates over and over again. Uh, uh [5:50] And, um, the agent has to constantly shift between model work versus non-model work. [5:58] And number of inference calls becomes highly unpredictable because it it [6:03] Sorry. It depends on the, um, on the on the input. So, in the end, what user wants [6:10] is not time to first token or how fast I can get my first answer. It's [6:16] how much does it cost and how long does it take to get a get a task completed end to end. [6:25] So, it's a it's a full stack problem. It's not any single optimization. And I get question from, [6:31] um, customers and investors like, "How do you guys do it? How do you guys do it make it so fast?" [6:37] They expect a secret sauce, like one thing that that makes it fast. [clears throat] [6:42] I mean, yeah, we reinvented continuous batching, but that's only one of the pillars. [6:47] Um, so you have to coordinate across KV caching, quantization, and speculative coding. [6:53] All of these has to work, um, in in sync in order to make agentic workflow fast. [7:03] So, that's why we built our entire inference engine from scratch. [7:08] Um So, we're not a kind of wrapper over some open open source, uh, framework like vLLM. We built [7:16] everything from scratch on C++ to make it fast and make it work better for agentic workflows. [7:27] So, this is the one of the way we we optimize. As I said, in agentic workflow, [7:32] context context accumulating is a is a big challenge. And in many workflows that we run, [7:38] uh, we often see 100 [snorts] to 1 input to output token ratio. [7:43] Um so optimizing for long context is is a key challenge. Um and but many of those input [7:50] um is largely reused um because um they're the same context uh built over time. [7:57] Um so instead of recomputing it's always faster and cheaper to to uh fetch it from [8:04] from a cache. So when when possible don't don't compute that from uh the memory. [8:16] And the next thing is uh speculative decoding. So um [8:21] either by using smaller model or just um um I'm taking a guess from a pattern matching or whatnot. [8:28] You try to guess what might come next. Um and and um that uh enables the uh us to use GPU [8:36] uh more efficiently. So every GPU cycle we can produce more tokens and more more useful output. [8:47] And many people just think um uh inference performance uh as in TPS or latency um but [8:56] reliability is also key pillar. Um because every error 429 We've all gotten 429 errors before. [9:05] Um they they add up to bad user experience. Um and and for agentic workflow it's even worse because [9:12] a single failure uh might mean you have to do the entire calculation and compute again. [9:19] So we focused on delivering the best end-to-end API experience so that we uh [9:25] our our time uh reliability is is very high. [9:31] So take a look. So this is a result of all these optimization added up. So we gave the uh [9:41] Yeah. [music] The same model, same agent, we gave the same task. [music] The only difference [9:48] is our inference provider. The task was to build a Candy Crush-like game. [music] [10:11] So, same outcome, same output, same model, [music] same task, [10:17] same agent, the only difference is our inference. So, inference can make all the difference. [10:24] So, we we can complete the task 100 seconds faster for this particular task. [10:28] 100 seconds might not sound like a lot, but if you think about agents, [10:33] they're running 100,000 of uh these tasks every day. So, it can quickly add up. Uh and me it means [10:40] uh it it impacts your user experience, and it can ultimately impact your business outcome. [10:49] Yeah, two other testimonials from our customers. Kilocode, another top open source coding agent. [10:55] They tested many inference providers, and we consistently were seven times faster. [11:01] Not not 7% or 70%, we're seven seven times faster and more reliable. [11:08] And Tactile Story uh they make LLM-based games. So, so for them, latency is is critical. And [11:13] they also tested uh many providers, and users that were using our inference had 15% higher retention. [11:21] So, I was a gaming PM once, so I know that 15% difference in retention for a game is a make or [11:27] make or break difference. A huge difference uh that can impact your business outcome. [11:33] So, last but not least, the takeaway for today, so Open Wave model is ready. [11:37] Um if you haven't used it, you should try it now. Um and if you want to run that in production, [11:44] you need a inference provider that can run agent taking on workflows um in a a cohesive way. [11:51] Um so, give that a try. Um and we are uh we have a booth at at the exhibition hall, so find us at [11:59] booth 40401 if you have more questions. And uh yeah, thank you. Thank you for listening.