Baseten Routes 1,800 H100 GPUs Across Clouds to Solve Inference Spikes

Inference in the Wild: Lessons from Scaling Real-time AI in Production With Baseten — Ai4

Ai4

Standard generative AI deployments fail in production because real-time latency and scaling bottlenecks stem from multi-cloud capacity limits, custom speculative decoding, and client-side concurrency rather than raw GPU count alone.

AI-assisted summary of Inference in the Wild: Lessons from Scaling Real-time AI in Production With Baseten — original source. Check the original for full context.

Source published: 2025-08-24

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 24, 2025, this presentation features Philip Kiely, Developer Advocate at Baseten. According to the talk description, it examines the four essential components of production-ready inference platforms—applied performance research, multi-cloud infrastructure, model management observability, and AI engineering expertise—highlighting common pitfalls and resilient architectural patterns for real-time generative AI workloads.

## TL;DR Kiely examines how modern AI-native startups reach $100M ARR with teams under 100 people by relying on automated AI infrastructure alongside an explosion of versatile open-source models. He defines the four pillars of inference strategy—quality, latency, economics, and differentiation—and explains why naive off-the-shelf deployments fail in real-world environments. Drawing on practical customer engagements, he details how custom speculative decoding slashed Sourcegraph Amp latencies to 200 ms, how multi-cloud routing provisioned over 1,800 H100 GPUs during a massive launch, and how fixing client-side bottlenecks unlocked a 12x throughput gain, ultimately showing that production inference requires rigorous software engineering rather than just raw GPU hardware.

## Atmosphere & style The presentation maintains a pragmatic, direct, and technically grounded tone typical of high-caliber engineering conferences, opening with casual personal anecdotes before quickly transitioning into rapid-fire architecture breakdowns. Kiely speaks with energetic clarity and authoritative realism, systematically dismantling AI hype to highlight unglamorous production realities like TLS handshakes, Python GIL bottlenecks, and regional cloud capacity exhaustion. The experience feels practical and empowering, offering systems builders battle-tested blueprints and empirical validation directly from the operational front lines.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=xYn20Yv_giE&t=4s) Baseten's core operational thesis is that 'inference is everything' when scaling production-grade generative AI applications. - [01:12](https://www.youtube.com/watch?v=xYn20Yv_giE&t=72s) AI-native companies are scaling from $1M to $100M ARR in 1 to 2 years with tiny teams of fewer than 100 people, aiming for an informal benchmark of $1M ARR per employee. - [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) Companies evaluating AI infrastructure focus on four core pillars: domain-specific quality, sub-second latency, shift from token 'price taker' to infrastructure 'price maker' economics, and proprietary differentiation. - [06:32](https://www.youtube.com/watch?v=xYn20Yv_giE&t=392s) The open-source model landscape has exploded across modalities, including text models like DeepSeek, Qwen 3, and OpenAI GPT-OSS, alongside specialized vision, embedding, and speech architectures. - [07:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=470s) Open-source architectures enable powerful composability across modalities, demonstrated by fine-tuning Llama 3.2 3B into Orpheus TTS to emit audio tokens directly. - [09:15](https://www.youtube.com/watch?v=xYn20Yv_giE&t=555s) Off-the-shelf deployments of engines like vLLM, SGLang, or TensorRT-LLM reliably fail in production without custom optimizations for multi-tenancy, traffic spikes, and strict latency targets. - [10:31](https://www.youtube.com/watch?v=xYn20Yv_giE&t=631s) Sourcegraph Amp reduced tab auto-completion latency from 700 ms down to roughly 200 ms by engineering over 1,000 lines of custom speculative decoding code in TensorRT-LLM. - [12:43](https://www.youtube.com/watch?v=xYn20Yv_giE&t=763s) Multi-cloud capacity management and global control planes are essential to burst compute dynamically across regions and providers, as demonstrated by scaling an active workload past 1,800 Nvidia H100 GPUs. - [14:37](https://www.youtube.com/watch?v=xYn20Yv_giE&t=877s) Client-side bottlenecks such as connection renegotiation and Python GIL saturation can severely limit throughput, with multi-core client execution delivering a 12x throughput gain on identical GPU infrastructure. - [16:33](https://www.youtube.com/watch?v=xYn20Yv_giE&t=993s) Production inference is more than just securing raw GPUs; it requires balancing model runtime optimizations, distributed multi-cloud systems, and foundational software engineering rigor.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=xYn20Yv_giE&t=4s) Introduction: The Definition and Role of Inference in Modern AI

Philip Kiely, Developer Advocate at Baseten, opens his talk by referencing Baseten's bold billboard campaign declaring that "inference is everything." Presenting a photo of himself with his team on Cape Cod, Kiely explains that Baseten focuses on building production-grade inference infrastructure specifically tailored for fast-growing startups. To understand why robust inference infrastructure is critical today, he outlines a three-part agenda: examining macro market trends, analyzing the operational challenges at their intersection, and studying real-world customer case studies.

```mermaid flowchart LR accTitle: Three-Part Agenda Flow subgraph S1["1. Market Trends"] A["AI-Native Growth"] B["Open Source Explosion"] end subgraph S2["2. Core Needs"] C["Scale, Reliability & Latency"] end subgraph S3["3. Execution"] D["Production Case Studies"] end A --> C B --> C C --> D ```

- Baseten's core thesis: "Inference is everything" in production AI. - Focus: Building high-performance AI infrastructure and inference platforms for high-growth startups. - Agenda framework: Evaluating two defining industry trends, finding their intersection in real-time inference requirements, and examining concrete customer case studies.

---

[01:12](https://www.youtube.com/watch?v=xYn20Yv_giE&t=72s) Trend 1: The Exponential Growth of AI-Native "Tiny Teams"

The first major trend is the unprecedented trajectory of AI-native companies. In the traditional SaaS era, taking 5 to 10 years to scale from $1M to $100M in Annual Recurring Revenue (ARR) was considered absolute best-in-class performance. Today, AI-native startups are achieving that same $100M ARR threshold in just 1 to 2 years, adding millions or tens of millions of dollars in revenue month-over-month despite operating in very young markets.

What makes this financial scale even more remarkable is the emergence of "tiny teams." Historically, reaching $100M ARR required hundreds or thousands of employees. AI-native startups are reaching it with fewer than 100 staff members—and informally targeting a metric of $1M ARR per employee. This massive revenue growth paired with minimal headcount expansion is only possible because heavy operational and computational leverage is offloaded to automated AI infrastructure.

- **ARR Scaling Comparison**: - *SaaS Era*: 5–10 years to reach $100M ARR (required 100s–1,000s of employees). - *AI-Native Era*: 1–2 years to reach $100M ARR (achieved with fewer than 100 employees). - **Tiny Team Benchmark**: Unofficial target metric of $1,000,000 ARR per employee.

---

[03:00](https://www.youtube.com/watch?v=xYn20Yv_giE&t=180s) Real-World AI-Native Workloads: Descript and Zed Industries

Kiely illustrates how AI-native products deliver value through concrete customer examples. As a video creator, he relies on **Descript** daily. Previously, editing raw video in iMovie required manually cutting clips, typing captions, and arranging title cards—a tedious workflow consuming roughly one hour of editing per minute of finished video. With Descript powered by Baseten's transcription and AI pipeline, creators type text edits to cut video automatically, instant captions are injected, and an entire video editing session takes an hour or less in total.

A second example is **Zed Industries**, creators of the high-performance text editor Zed (similar to Cursor). Designed in Rust to maximize speed and minimize local system resource usage, Zed extends its performance-first philosophy to its integrated AI code assistant. By partnering with Baseten, Zed delivers lower-latency code autocompletion and rapid model responses directly inside the editor.

- **Descript Workflow Savings**: Reduced editing time from 1 hour per finished minute of video down to ~1 hour for an entire project. - **Zed Industries**: Rust-based text editor built for extreme performance, leveraging Baseten for low-latency code completion.

---

[04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) The Four Core Pillars of AI Infrastructure Strategy

AI-native companies evaluate their underlying AI infrastructure against four distinct priorities:

1. **Quality**: The output must be top-notch. For instance, **Open Evidence** (described as "ChatGPT for doctors") requires flawless accuracy, as medical responses carry zero tolerance for hallucinations. 2. **Latency**: For voice agents or rapid interactive loops where feedback is measured in milliseconds, winning or losing market share comes down to "literally the blink of an eye." 3. **Economics**: Startups want to transition from being passive "price takers" accepting rigid per-token rates to "price makers" who directly control their hardware and inference trade-offs. 4. **Differentiation**: Leading AI startups reject being simple "ChatGPT wrappers." They demand deep customization and control over the model layer to build defensible IP and own their own product destiny.

---

[06:32](https://www.youtube.com/watch?v=xYn20Yv_giE&t=392s) Trend 2: The Explosion and Composition of Open-Source Models

The second macro trend is the explosive growth of open-source models, with nearly 2 million models hostable on Hugging Face across every modality and parameter size. In open-source LLMs, the landscape exploded six months ago with DeepSeek, followed by powerful international models including the GLM series by Zhipu AI, Kimi, and Qwen 3 (Qwen Coder). Meanwhile, Western labs continue releasing competitive weights like Mistral, Google's Gemma, and OpenAI's recently released GPT-OSS—marking OpenAI's first open-weights release since GPT-2 in 2019.

Beyond text, open-source models lead across multiple modalities: - **Vision/Image**: Qwen Image leads diffusion transformer architectures (similar to the viral Studio Ghibli generation models). - **Embeddings**: Models like Qwen Embed act as the connective tissue for RAG and vector retrieval systems. - **Speech/Transcription**: OpenAI's Whisper remains the audio transcription standard. - **Text-to-Speech (TTS)**: Orpheus TTS represents a "Llama moment" for open-source speech, allowing developers to own and fine-tune hyper-realistic custom voices.

```mermaid flowchart TD accTitle: Open Source Base Models to Fine-Tuned Modalities subgraph Base ["Base Open Weights"] L3["Llama 3.2 3B (Language)"] end subgraph Adapted ["Adapted Modality Models"] Orph["Orpheus TTS (Speech Output)"] QEmb["Qwen Embed (Vector Search)"] end L3 -->|"Architecture fine-tuning"| Orph L3 -->|"Weight adaptation"| QEmb ```

A key technical advantage of open source is composability. For instance, Orpheus TTS was built by fine-tuning **Llama 3.2 3B**—a text-based LLM—to output audio tokens directly, demonstrating how base architectures can be adapted into entirely new modalities.

- **LLM Open Weights**: DeepSeek, GLM (Zhipu AI), Kimi, Qwen 3, Mistral, Gemma, and OpenAI GPT-OSS. - **Multimodal Open Models**: Qwen Image (Diffusion Transformer), Qwen Embed, Whisper (Speech-to-Text), and Orpheus TTS (Text-to-Speech).

---

[09:15](https://www.youtube.com/watch?v=xYn20Yv_giE&t=555s) Production Reality: Why Off-the-Shelf Deployments Fail

While getting an API key from OpenAI was the historical path to shipping products, relying on open-source models introduces operational complexity. Beginners often throw an off-the-shelf inference engine like vLLM, SGLang, or TensorRT-LLM onto a few GPUs and expect production readiness.

In practice, simple deployments crumble under actual production traffic. Standard setups fail to satisfy the combined requirements of high quality, ultra-low latency, strict unit economics, and custom differentiation. Overcoming these limitations requires solving hard engineering challenges across model performance research, cloud infrastructure, observability tooling, and systems engineering.

> "The path to production used to be: go to openai.com, get an API key, put it in your product, and ship. Now there's a lot more things you have to do in order to leverage these open source models effectively."

- Common inference frameworks: vLLM, SGLang, TensorRT-LLM. - The Core Problem: Basic deployments cannot handle real-world traffic spikes, latency constraints, or multi-tenant demand without specialized optimizations.

---

[10:31](https://www.youtube.com/watch?v=xYn20Yv_giE&t=631s) Case Study 1: Sub-Second Latency Optimization for Sourcegraph Amp

For real-time developer tooling, latency is paramount. Software engineers type rapidly, leaving only a tiny millisecond window between keystrokes. If an AI auto-completion tool delays output, it disrupts the developer's flow. **Sourcegraph** encountered this bottleneck with their **Amp** tab auto-completion feature, where latencies were spiking up to **700 milliseconds**—an unacceptable delay that rendered the product frustrating to use.

```mermaid flowchart LR accTitle: Speculative Decoding Optimization Flow subgraph SpecDec ["Speculative Decoding Mechanism"] Draft["Small Draft Model<br/>(Generates candidate tokens)"] Verify["Large Target Model<br/>(Verifies candidate tokens)"] end Draft -->|"Fast proposal"| Verify Verify -->|"High acceptance rate"| Output["Accepted Completion"] ```

To solve this, Baseten collaborated with Sourcegraph to overhaul their inference stack. They implemented TensorRT-LLM, optimized KV caching, and engineered a custom speculative decoding pipeline. Baseten wrote over **1,000 lines of code** for a new draft decoding algorithm, producing higher-quality candidate tokens that achieved significantly higher acceptance rates by the main model. This reduced completion latency down to a couple hundred milliseconds, aligning output speed directly with human typing reaction times.

- **Problem**: Sourcegraph Amp tab completions were spiking up to **700 ms**, breaking user workflow. - **Solution**: Deployed TensorRT-LLM, KV caching optimizations, and custom speculative decoding. - **Engineering Effort**: Authored 1,000+ lines of custom decoding code to improve draft token acceptance. - **Outcome**: Cut latency down to ~200 ms, matching human reaction speed.

---

[12:43](https://www.youtube.com/watch?v=xYn20Yv_giE&t=763s) Case Study 2: Auto-Scaling Across Cloud Providers (The 1,800 GPU Launch)

Scalability creates a different set of infrastructure hurdles. During a major launch for a key customer, traffic surged far beyond baseline levels. Starting from a steady state of roughly 400 GPUs, Baseten's automated scaling platform dynamically provisioned over **1,800 Nvidia H100 GPUs** during peak demand before automatically scaling back down later in the day.

```mermaid flowchart TD accTitle: Multi-Cloud Capacity Management Topology subgraph Control ["Global Control Plane"] Router["Traffic Router & Capacity Manager"] end subgraph Primary ["Primary Cloud Provider"] Reg1["Primary Region (H100 Pool Exhausted)"] end subgraph Spillover ["Spillover Infrastructure"] Reg2["Secondary Region (H100 Pool)"] Cloud2["Alternative Cloud Provider (H100 Pool)"] end Router --> Primary Router -->|"Traffic Surge Spillover"| Reg2 Router -->|"Capacity Exhaustion Spillover"| Cloud2 ```

Critically, the customer's peak GPU demand exceeded the total physical inventory available inside their primary cloud provider's single data center region. Baseten's **Multi-Cloud Capacity Management** architecture resolved this bottleneck by employing a global control plane that orchestrated workloads across multiple cloud regions and secondary cloud vendors in an active-active setup. This prevented data center capacity exhaustion, ensured high availability without regional single-point failures, and minimized latency by routing requests to geographically optimal hardware.

- **Peak Compute Deployment**: Scaled automatically from ~400 GPUs to over **1,800 Nvidia H100 GPUs** during a live launch. - **Capacity Management**: Mitigated physical data center limits by spilling traffic over into alternative regions and cloud providers seamlessly. - **Architecture**: Separated a global control plane from distributed execution planes for active-active reliability and optimal pricing.

---

[14:37](https://www.youtube.com/watch?v=xYn20Yv_giE&t=877s) Case Study 3: Eliminating Client-Side Bottlenecks for 12x Throughput

The final case study highlights how performance bottlenecks often lie entirely outside the GPU or model runtime. Developers frequently report latency discrepancies—such as expecting a 200 ms response time based on model benchmarks but experiencing 300 ms end-to-end. Kiely notes that these hidden delays often stem from traditional software engineering issues, such as failing to reuse HTTP sessions (where establishing a new connection adds several dozen milliseconds per call).

```mermaid flowchart LR accTitle: Client Parallelism Bottleneck vs Baseten Performance Client subgraph Standard ["Standard Client"] PyGIL["Python GIL Bottleneck<br/>(Single CPU Core)"] -->|"Unsaturated pipe"| StandardGPU["GPU (Idle Cycles)"] end subgraph Optimized ["Baseten Performance Client"] MultiCPU["Multi-Core Parallel Execution"] -->|"Saturates pipeline"| FullGPU["GPU (100% Utilization)"] end ```

To fix client-side bottlenecks, Baseten created the **Baseten Performance Client**. In standard Python environments, the Global Interpreter Lock (GIL) restricts execution parallelism, preventing the client from issuing concurrent requests quickly enough to saturate powerful remote GPUs. By bypassing GIL restrictions across multi-core CPUs on massive embedding workloads (benchmarked on a corpus backfill of 2,000,000 embeddings), Baseten achieved a **12x increase in throughput** on the exact same model and GPU hardware purely by optimizing client request delivery.

- **HTTP Overhead**: Failing to reuse HTTP sessions adds dozens of milliseconds in TLS handshake and connection overhead per call. - **Python GIL Bottleneck**: Standard Python SDKs bottleneck single-threaded client processing, leaving server GPUs underutilized. - **Benchmark Result**: Processing **2,000,000 embeddings** achieved a **12x throughput gain** using the Baseten Performance Client without altering the underlying GPU hardware or model runtime.

[16:33](https://www.youtube.com/watch?v=xYn20Yv_giE&t=993s) Core Engineering Fundamentals and the True Nature of Production Inference

While novel AI breakthroughs and cutting-edge models are exciting to explore and discuss, engineering teams must not lose sight of foundational software engineering discipline. The core basics and fundamentals of building reliable software become even more critical when deploying real-time generative AI products into production environments under intense real-world demand.

```mermaid flowchart LR accTitle: Core Pillars of Production AI Inference subgraph InferenceSystem["Production Inference Engine"] direction TB RT["Model Runtime & Optimization"] INF["Distributed Infrastructure"] ENG["Core Software Engineering"] end InferenceSystem --> SCALE["High-Velocity Scaling with Lean Teams"] ```

Scaling AI successfully in production requires understanding that inference is fundamentally "more than just GPUs." It is a delicate combination of runtime performance optimization, resilient distributed infrastructure, and the foundational engineering practices that bind the entire architecture together. When this stack is engineered correctly, it enables next-generation applications to scale at unprecedented velocity while maintaining remarkably lean engineering teams.

- **Foundations Over Hype:** Teams must prioritize traditional software engineering excellence alongside cutting-edge AI experimentation. - **Triad of Inference:** Production inference demands a balance of optimized runtime, distributed infrastructure, and sound engineering practices rather than just raw GPU capacity. - **Expo & Follow-Up:** Attendees are invited to connect outside the session room or visit the Baseten team at Booth 600 in the expo hall to discuss production workloads and pick up Baseten t-shirts.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=xYn20Yv_giE&t=4s) inference is everything — Philip Kiely

> [01:12](https://www.youtube.com/watch?v=xYn20Yv_giE&t=72s) tiny teams — Philip Kiely

> [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) ChatGPT for doctors — Philip Kiely

> [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) literally the blink of an eye — Philip Kiely

> [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) price takers — Philip Kiely

> [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) price makers — Philip Kiely

> [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) ChatGPT wrappers — Philip Kiely

> [06:32](https://www.youtube.com/watch?v=xYn20Yv_giE&t=392s) Llama moment — Philip Kiely

> [09:15](https://www.youtube.com/watch?v=xYn20Yv_giE&t=555s) The path to production used to be: go to openai.com, get an API key, put it in your product, and ship. Now there's a lot more things you have to do in order to leverage these open source models effectively. — Philip Kiely

> [16:33](https://www.youtube.com/watch?v=xYn20Yv_giE&t=993s) more than just GPUs — Philip Kiely

## Mentioned Philip Kiely, Baseten, Descript, iMovie, Zed Industries, Zed, Cursor, Rust, Open Evidence, ChatGPT, Hugging Face, DeepSeek, GLM, Zhipu AI, Kimi, Qwen 3, Qwen Coder, Mistral, Google, Gemma, OpenAI, GPT-OSS, GPT-2, Qwen Image, Qwen Embed, Whisper, Orpheus TTS, Llama 3.2 3B, vLLM, SGLang, TensorRT-LLM, Sourcegraph, Sourcegraph Amp, Nvidia H100, Baseten Performance Client, Python

## Video description Philip Kiely, Developer Advocate, Baseten

Abstracts

Real-time generative AI applications promise transformative user experiences—but scaling inference in production is a constant battle. Demand spikes, infrastructure meltdowns, and unpredictable model behaviors lurk behind every release. Drawing from direct experience deploying mission-critical AI workloads at startups and large enterprises, this talk covers the four essential components of a production-ready inference platform: applied model performance research, distributed multi-cloud infrastructure, model management and observability tooling, and AI engineering expertise. I'll highlight common pitfalls and share concrete advice on how to build robust, resilient systems that thrive under real-world demand.

[Source](https://www.youtube.com/watch?v=xYn20Yv_giE)

中文简报

Baseten 跨云调度 1800 块 H100 GPU 应对推理流量激增标准生成式 AI 部署在生产环境中往往面临失败，因为实时延迟与扩缩容瓶颈源于多云容量限制、定制推测解码和客户端并发问题，而非仅取决于 GPU 的纯粹数量。 Kiely 探讨了现代 AI 原生初创公司如何依托自动化 AI 基础设施以及爆发式增长的多功能开源模型，凭借不足百人的团队实现 1 亿美元 ARR。他界定了推理策略的四大支柱——质量、延迟、经济效益与差异化，并阐释了简单的开箱即用部署在真实环境中屡屡失效的原因。结合客户实际案例，他详细说明了定制推测解码如何将 Sourcegraph Amp 的延迟大幅缩短至 200 毫秒、多云路由如何在一次重大发布期间调配超过 1800 块 H100 GPU，以及修复客户端瓶颈如何带来 12 倍的吞吐量提升，最终表明生产级推理需要严谨的软件工程，而非仅仅依赖原始 GPU 硬件。 Baseten 提出“推理即一切”，指出 AI 原生初创公司正以不足百人的极小团队在 1 至 2 年内将 ARR 从 100 万美元扩展至 1 亿美元，普遍将人均 100 万美元 ARR 视作效能标杆。 (1:12) 企业评估 AI 推理基础设施需聚焦四大支柱：特定领域的高质量输出、亚秒级交互延迟、从 Token“价格接受者”转向基础设施“定价者”的成本掌控力，以及基于开源权重的差异化产品壁垒。 (4:50) 开箱即用的推理引擎难以承受高并发生产流量，Sourcegraph Amp 通过重构 TensorRT-LLM 与编写逾千行定制化投机解码（Speculative Decoding）代码，将代码补全延迟从 700 毫秒压降至约 200 毫秒。 (10:31) 为应对单一数据中心算力耗尽的风险，Baseten 通过多云容量管理与全局控制平面实现跨区域跨厂商调度，在实际客户发布中将算力动态弹性扩展至超 1,800 张 Nvidia H100 GPU。 (12:43) 推理性能瓶颈常源于客户端而非 GPU 本身，通过绕过 Python GIL 并解决 HTTP 连接复用等传统工程问题，Baseten 在 200 万条 Embedding 处理测试中达成了 12 倍的端到端吞吐量提升。 (14:37)

Reading summary

Baseten developer advocate Philip Kiely argues that production AI success hinges on specialized inference infrastructure rather than raw model access, enabling lean software teams to scale rapidly while meeting stringent latency, reliability, and economic demands. Hyper-Growth Startups and Modern Infrastructure Demands (1:12) Kiely contrasts the traditional SaaS era—where reaching $100M in Annual Recurring Revenue (ARR) took 5 to 10 years and hundreds to thousands of workers—with the AI-native era, where startups achieve that scale in 1 to 2 years with fewer than 100 employees, targeting an informal benchmark of $1M ARR per employee. Products like Descript, which reduces video editing time from an hour per finished minute to about an hour per project, and Zed Industries' Rust-based code editor rely heavily on external inference systems to sustain operational leverage. To support such workloads, AI companies evaluate infrastructure across four pillars: domain quality (such as hallucination-intolerant medical tools like Open Evidence), millisecond-level interactive latency, shifting from per-token price takers to infrastructure price makers, and architectural differentiation beyond generic wrappers. The Operational Realities of Open-Source Models (6:32) The open-weights ecosystem has broadened beyond proprietary APIs, spanning international LLMs such as DeepSeek, Zhipu AI's GLM, Kimi, and Qwen 3, alongside Western releases from Mistral, Google's Gemma, and OpenAI's GPT-OSS. Open modalities also encompass diffusion models like Qwen Image, vector search models like Qwen Embed, OpenAI's Whisper for transcription, and speech synthesis like Orpheus TTS, which adapts Llama 3.2 3B to output audio tokens directly. However, Kiely emphasizes that placing default engines like vLLM, SGLang, or TensorRT-LLM on arbitrary GPUs falls short under production conditions, failing to balance latency, unit economics, and multi-tenant traffic spikes without tailored engineering. Latency Engineering and Speculative Decoding (10:31) Interactive applications demand sub-second turnaround times to maintain user workflows. Sourcegraph faced an operational bottleneck when its Amp tab autocompletion feature experienced latency spikes up to 700 milliseconds. Baseten addressed this by rearchitecting the inference pipeline using TensorRT-LLM, Key-Value (KV) cache tuning, and custom speculative decoding. By writing more than 1,000 lines of custom draft decoding code, Baseten generated higher-quality candidate tokens that improved main-model acceptance rates, lowering end-to-end autocompletion latency to approximately 200 milliseconds to match human typing reaction speeds. Multi-Cloud Dynamic Provisioning and Capacity Limits (12:43) Scaling unpredictable workloads can exceed regional cloud infrastructure. During a customer launch, demand surged from a steady-state footprint of roughly 400 GPUs to a peak of more than 1,800 Nvidia H100 GPUs before scaling back down. Because peak compute requirements exceeded the physical hardware available in a single data center region, Baseten utilized a Multi-Cloud Capacity Management system. This architecture separated a global control plane from distributed execution planes, routing spillover traffic across multiple regions and secondary cloud vendors in an active-active setup to prevent regional capacity exhaustion. Overcoming Client-Side Throughput Bottlenecks (14:37) System latency and throughput bottlenecks often arise outside GPU hardware or model engines. Kiely observes that basic integration flaws, such as failing to reuse HTTP sessions, introduce dozens of milliseconds in TLS handshake overhead. Additionally, standard Python client libraries are constrained by the Global Interpreter Lock (GIL), preventing single-core processes from saturating available server compute. Using the Baseten Performance Client to bypass GIL limitations across multi-core CPUs during a 2,000,000 embedding backfill workload, throughput improved by 12x over standard clients on identical GPU hardware and model runtime environments. Kiely concludes that sustainable production inference requires balancing runtime optimizations, distributed capacity management, and standard software engineering fundamentals, though managing these multi-cloud architectures introduces significant systems complexity beyond standard off-the-shelf deployments.

精炼阅读摘要

Baseten开发者布道师Philip Kiely指出，在AI原生企业爆发式增长与开源模型普及的浪潮下，生产级推理已成为应用落地的核心瓶颈；构建可靠的高性能推理体系不仅依赖GPU算力，更取决于模型运行时优化、多云弹性编排与基础软件工程的深度协同。 AI原生团队超速增长与基建四要素 (1:12) Kiely表示，传统SaaS企业实现1亿美元年度经常性收入（ARR）通常需要5至10年以及数百至上千名员工，而当下的AI原生初创企业仅耗时1至2年、在团队不足百人的情况下便能达到同等规模，并普遍以每名员工创造100万美元ARR为运营目标。这种高人效增长依赖于自动化计算杠杆，例如视频编辑工具Descript将每分钟成品所需的人工剪辑时间由1小时压缩至全流程约1小时，Rust语言编写的高性能编辑器Zed亦通过低时延模型调用实现实时代码补全。随着业务规模扩张，企业评估底层推理架构的核心维度呈现为四大支柱：输出质量（如医疗问答应用Open Evidence对内容准确性与零幻觉的极高要求）、交互时延（语音代理等交互界面依赖毫秒级响应）、经济效益（从被动接受Token调用定价转向自主调度硬件以控制单位经济成本），以及差异化竞争壁垒（拒绝仅作为基础模型的轻量封装，转向掌握专有权重与微调工作流）。开源模型生态与开箱即用部署的现实瓶颈 (6:32) 相较于早期单纯调用闭源商用API，当前开源模型生态呈现爆发态势，Hugging Face平台托管了近200万个跨模态与多尺寸模型，涵盖DeepSeek、智谱AI的GLM系列、Kimi、通义千问Qwen 3系列、Mistral、Google Gemma以及OpenAI发布的GPT-OSS。同时，开源架构展现出高度的可组合性，例如Orpheus TTS通过对文本模型Llama 3.2 3B进行微调以直接生成音频Token，打破了既有模态界限。但Kiely强调，直接采用vLLM、SGLang或TensorRT-LLM等开源推理框架挂载至少量GPU的简易部署方案，在实际生产环境中极易失效。开箱即用的配置往往无法承受真实的突发流量洪峰，也无法同时兼顾亚秒级响应、严苛的成本结构与定制化需求，企业仍须攻克模型性能研究、云基础设施调度与系统工程层面的诸多挑战。极低时延优化与多云千卡弹性伸缩实践 (10:31) 针对开发者实时编码场景，代码补全时延的微小波动会直接打断工作心流。Baseten协助Sourcegraph解决了其Amp功能时延偶发飙升至700毫秒的严重瓶颈。工程团队引入TensorRT-LLM并重构KV缓存，自主编写了超过1,000行投机解码算法代码，利用小型草稿模型快速生成候选Token并交由主模型并行验证，最终将补全时延压缩至约200毫秒的人类反应极限范围内。在应对极端流量激增方面，Baseten在某核心客户业务发布期间实现了算力规模从平稳期的约400张GPU动态扩展至超过1,800张Nvidia H100 GPU。面对单家云厂商单个数据中心物理库存耗尽的困境，其多云容量管理架构通过全局控制平面调度，在多区域与其他云服务商之间建立双活架构无缝承接溢出流量，避免了区域单点故障并保障了高可用性。消除客户端隐藏损耗与坚守工程底座 (14:37) 生产环境中的性能瓶颈往往潜藏在GPU之外。Kiely指出，开发者常遇到模型基准测试为200毫秒但端到端实际耗时却达300毫秒的脱节现象，其主要诱因包括未复用HTTP连接导致的TLS握手额外开销，以及标准Python环境下的全局解释器锁（GIL）对CPU多核并发的制约，导致客户端发送请求速率不足，使远端高性能GPU处于算力饥饿状态。通过开发绕过GIL限制、利用多核CPU并发执行的Baseten高性能客户端，团队在200万条嵌入向量的回填基准测试中，在硬件与模型完全不变的前提下实现了12倍的吞吐量提升。Kiely总结认为，生产级AI推理绝非简单等同于堆砌GPU，而是由模型运行时调优、分布式基础设施与基础软件工程规范共同构建的严谨工程体系。 Kiely总结指出，团队在追求前沿AI模型能力的同时，绝不能忽视传统软件工程的底层基本功；唯有在运行时调优、弹性分布式架构与系统工程之间保持平衡，小型工程团队才能支撑起AI原生业务的指数级增长。

Detailed notes — zh

## Background 本视频由 Ai4 官方频道于 2025 年 8 月 24 日在 YouTube 平台发布，记录了 Baseten 开发者布道师 Philip Kiely 的主题演讲。根据页面简介，演讲结合了 Baseten 在初创企业与大型企业中部署关键 AI 生产负载的实战经验，系统拆解构建生产级推理平台的核心要素与避坑指南。 ## TL;DR 演讲探讨了在 AI 原生企业极速崛起与开源模型大爆发的交汇下，如何破解生产级实时 AI 推理的工程瓶颈。Philip 指出单纯依靠“GPU + 现成推理框架”无法满足企业对质量、低延迟、成本经济性及差异化护城河的严苛要求。通过 Sourcegraph 投机采样优化延迟至 200ms、跨云弹性调度 1,800 张 H100 GPU 应对突发流量，以及优化客户端绕过 Python GIL 提升 12 倍吞吐等真实案例，他论证了生产级推理必须深度融合模型运行时调优、分布式多云基础设施与扎实的传统软件工程基本功。 ## Atmosphere & style 整场演讲节奏明快、务实且极具工程师洞察力，既有对宏观行业趋势（如微型团队创造过亿营收）的宏大叙事，又有深入底层代码和网络握手协议的硬核剖析。Philip 的分享风格接地气且充满热情，将抽象的架构理论与生动的实战踩坑故事紧密结合，带给观众专业、严谨而又富有启发性的技术临场感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=xYn20Yv_giE&t=4s) AI 原生时代的推理基础设施需求，由“AI 原生初创公司的爆发式增长”与“开源模型生态的全面繁荣”两大核心趋势共同驱动。 - [01:12](https://www.youtube.com/watch?v=xYn20Yv_giE&t=72s) AI 原生企业打破了传统 SaaS 增长规律，仅需 1 至 2 年即可实现 1 亿美元 ARR，并呈现出人均创造 100 万美元 ARR 的“微型团队”特征。 - [03:00](https://www.youtube.com/watch?v=xYn20Yv_giE&t=180s) 音视频剪辑工具 Descript 与高性能编辑器 Zed 证明，底层极低延迟的实时 AI 推理能力能够直接重塑产品的交互与交付体验。 - [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) AI 原生企业对底层基础设施有着四大核心诉求：垂直领域的极致质量、毫秒级交互延迟、成本自主掌控能力，以及摆脱“ChatGPT 包装工”的技术差异化护城河。 - [06:34](https://www.youtube.com/watch?v=xYn20Yv_giE&t=394s) 开源模型已覆盖全模态且具备模块化组合特性（如基于 Llama 架构微调改造的 Orpheus TTS），成为企业构建差异化壁垒的关键底座。 - [09:15](https://www.youtube.com/watch?v=xYn20Yv_giE&t=555s) 简单的“GPU + 现成开源推理框架（如 vLLM）”拼凑方案在严苛生产环境中极易崩溃，无法单独解决全方位的性能、弹性与工程难题。 - [10:31](https://www.youtube.com/watch?v=xYn20Yv_giE&t=631s) 通过引入 TensorRT-LLM 与重构投机采样算法，Sourcegraph 将代码补全延迟从 700 毫秒压低至 200 毫秒左右，突破了实时交互的心流阈值。 - [12:43](https://www.youtube.com/watch?v=xYn20Yv_giE&t=763s) 依托多云容量管理架构的全局控制面与分布式工作负载面，系统可实现单次发布跨云自动弹性调度超过 1,800 张 H100 GPU，消除硬件库存断点。 - [14:37](https://www.youtube.com/watch?v=xYn20Yv_giE&t=877s) 模型之外的客户端与网络握手损耗常是不可见的性能瓶颈，通过绕过 Python GIL 限制等客户端优化，可在同等硬件配置下实现 12 倍的吞吐量提升。 - [16:33](https://www.youtube.com/watch?v=xYn20Yv_giE&t=993s) 生产级推理绝不仅仅关乎算力芯片（GPU），而是模型运行时优化、分布式多云基础设施与扎实系统软件工程基本功的深度结合。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=xYn20Yv_giE&t=4s) 什么是推理：AI 原生时代的两大核心趋势来自 Baseten 的开发者布道师 Philip Kiely 以公司标志性的户外广告标语“推理就是一切”（inference is everything）开场，深入探讨在当前技术周期下构建生产级 AI 基础设施的核心意义。Baseten 团队专注于为创业公司打造高性能推理平台。Philip 指出，要理解现代 AI 推理平台的重要性，必须把握当前行业发展的两大交汇趋势：第一是“AI 原生”（AI-native）初创公司的爆发式增长，第二是开源模型生态在全球范围内的全面繁荣。当这两大趋势相互碰撞时，企业对运行在开源模型之上的高可靠性、高弹性扩展能力以及极低延迟推理基础设施产生了前所未有的迫切需求。Philip 强调，本次分享将通过实际客户案例，拆解 Baseten 如何协助前沿团队解决真实生产场景中的工程瓶颈，并最终打通从模型实验到规模化商用的完整链路。 ### [01:12](https://www.youtube.com/watch?v=xYn20Yv_giE&t=72s) 趋势一：AI 原生企业的极速崛起与“微型团队”现象在第一大趋势中，AI 原生企业的扩张速度正在彻底颠覆传统商业软件的增长模型。在传统 SaaS 时代，一家顶尖企业从 100 万美元年度经常性收入（ARR）增长到 1 亿美元 ARR 通常需要 5 到 10 年的时间，这已被视为行业标杆；然而在 AI 原生时代，这一周期被压缩到了惊人的 1 至 2 年。这些新兴企业在极其年轻、甚至尚未完全成熟的全新市场中，以月复一月的节奏新增数百万乃至数千万美元的营收。更引人注目的是伴随高增长而来的“微型团队”（Tiny Teams）现象。传统 SaaS 企业达到 1 亿美元 ARR 时往往需要数百甚至数千名员工，而当下的非官方标准则是人均创造 100 万美元 ARR。许多 AI 原生企业在团队总人数不足 100 人的情况下，就实现了过亿美元的收入规模。“巨额收入增长伴随着极低的人员扩张”，这种反常识的生产力跃迁正是由高密度算力与高度自动化的生产级 AI 系统所驱动的。 - **ARR 扩张周期**：从 100 万美元到 1 亿美元 ARR，传统 SaaS 需 5–10 年，AI 原生企业仅需 1–2 年。 - **人均产出基准**：微型团队以人均 100 万美元 ARR 为特征，百人以内团队即可支撑过亿业务。 ### [03:00](https://www.youtube.com/watch?v=xYn20Yv_giE&t=180s) 真实生产案例：Descript 与 Zed 的极速体验背后 Baseten 支撑了许多头部 AI 原生团队的核心生产负载，其中最具代表性的是音视频编辑工具 Descript 和高性能代码编辑器 Zed Industries。作为内容创作者，Philip 分享了自己的切身体会：过去使用传统剪辑软件如 iMovie 制作视频时，需要手动剪切素材、逐行添加字幕和标题卡，每产出 1 分钟成品视频往往需要花费 1 个小时进行剪辑；而在 Descript 中，用户只需像编辑文本文档一样直接修改转录脚本，系统便会自动剪辑视频并生成字幕，将原本耗时数小时的工作缩短至不到 1 小时。Baseten 在底层为其提供了强大的音频转录与 AI 功能推理支持。另一个典型案例是主打极致性能的代码编辑器 Zed Industries。与 Cursor 等竞品类似，Zed 在代码编辑中深度集成了 AI 能力，但其独特之处在于整个编辑器完全基于 Rust 语言构建，对系统资源要求极低且运行极其流畅。为了将这种对“速度”的执着延伸到 AI 交互体验中，Zed 联合 Baseten 优化了后台模型与推理链路，实现了极低延迟的代码自动补全与实时模型响应。 ### [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) AI 原生企业对基础设施的四大诉求：拒绝只做“包装工” 随着业务的演进，AI 原生企业对底层基础设施的要求高度聚焦在四个核心维度： ```mermaid flowchart LR accTitle: AI Native Infrastructure Priorities A["AI 原生企业底层诉求"] --> B["1. 质量 (Quality)"] A --> C["2. 延迟 (Latency)"] A --> D["3. 经济效益 (Economics)"] A --> E["4. 差异化 (Differentiation)"] B --> B1["垂直领域极致精准（如 OpenEvidence）"] C --> C1["毫秒级交互反馈（语音与代码补全）"] D --> D1["从接受定价转变为自主掌控成本"] E --> E1["摆脱 ChatGPT 包装工，自研/微调专属模型"] ``` 在质量层面，以被誉为“医生专用 ChatGPT”的 OpenEvidence 为例，医学场景下的每一个回答都关乎重大决策，对任务特定领域的质量要求容不得丝毫偏差。在延迟层面，特别是在语音代理（Voice Agent）或强交互场景中，用户的反馈循环以毫秒为单位计量，“赢下或输掉市场的差距就在眨眼之间”。在经济效益与差异化层面，初创企业在高速增长期或许可以暂时容忍高昂的 Token 费用，但商业逻辑终究会回归单位经济学。企业迫切希望从被动接受公有 API 定价的“价格接受者”（price taker），转变为能够自主权衡性能与成本的“价格制定者”（price maker）。更重要的是，AI 原生企业坚决拒绝仅仅做一个简单的“ChatGPT 包装工”（ChatGPT wrapper），他们追求在底层模型与 AI 架构层面构建核心壁垒，全面掌控自身产品的技术命运。 ### [06:34](https://www.youtube.com/watch?v=xYn20Yv_giE&t=394s) 趋势二：开源模型大爆发与多模态组合创新为了摆脱对单一闭源 API 的依赖，开源模型成为了企业构建技术护城河的关键支撑。目前 Hugging Face 平台上托管的开源 AI 模型已接近 200 万个，覆盖了几乎所有能够想象到的任务、尺寸和模态。在大型语言模型（LLM）领域，几个月前 DeepSeek 的横空出世震动了整个行业生态；与此同时，智谱 AI 的 GLM 系列、Kimi、阿里通义千问（Qwen / Qwen-Coder / Qwen 2.5）、Mistral、Gemma 以及 OpenAI 最新重返开源赛道推出的 GPT-OSS 等，使得开源模型的竞争力已全面比肩顶级闭源产品。开源生态的魅力不仅限于文本，还在非 LLM 模态上展现出惊人的创新速度与复合衍生能力： - **图像生成**：基于 Diffusion Transformer 架构的 Qwen-Image 等模型已达到行业 SOTA 水平，引发了广泛的视觉创作潮流。 - **向量表征（Embedding）**：微调与开源向量模型已成为现代 AI 应用不可或缺的底层连接枢纽。 - **语音转录与合成（TTS）**：Whisper 稳居行业转录基准；而在语音合成领域，Orpheus TTS 的诞生被誉为开源语音领域的“Llama 时刻”。 Philip 特别指出，这些开源模型展现出强大的“模块化积木”特性。例如专门输出音频波形的语音合成模型 Orpheus TTS，其底层架构竟然是基于一年前用于处理纯文本的 Llama 3.2 3B 模型微调改造而来。开发者可以基于成熟的开源底座进行激进的架构重塑，直接满足特定业务场景的严苛需求。 ### [09:15](https://www.youtube.com/watch?v=xYn20Yv_giE&t=555s) 生产级推理之痛：为何“GPU + vLLM”并非终极解法尽管开源模型生态繁荣，但将开源模型真正推向生产环境的复杂度却呈指数级上升。过去将 AI 接入产品的路径极其单线条：前往 openai.com 注册账号、获取 API Key、写入业务代码并直接上线；而自建开源推理流水线时，企业必须在保证快速上线的同时，兼顾高并发扩展与高可用可靠性。许多工程团队早期的通用做法是：采购一批 GPU 服务器，拉取开源推理框架（如 vLLM、SGLang 或 TensorRT-LLM），部署后直接对外提供服务。然而 Philip 直言，这种“GPU + 现成推理框架”的拼凑式方案在严苛的生产环境中往往迅速崩溃。面对质量、延迟、经济效益和差异化这四重挑战，简单的技术堆叠无法解决模型性能调优、跨云基础设施编排、系统观测与 AI 工程落地等全方位的现实难题。 ### [10:31](https://www.youtube.com/watch?v=xYn20Yv_giE&t=631s) 案例一：Sourcegraph 毫秒级代码自动补全与投机采样优化在实时人机协同场景中，代码自动补全对延迟有着极其严苛的要求。以普通开发者每分钟约 60 个单词的打字速度计算，击键之间的间隔仅在几百毫秒之内，AI 补全必须跟上键盘输入的节奏才能形成无缝心流。Sourcegraph 在为其代码补全功能（Amp Tab）构建推理引擎时，就曾遭遇严重的延迟尖峰，响应时间一度飙升至 700 毫秒。对于开发者而言，每次敲击 Tab 键都需要停顿等待，这种迟滞足以彻底摧毁产品体验。 ```mermaid flowchart LR accTitle: Speculative Decoding Architecture subgraph SpeculativeDecoding["投机采样 (Speculative Decoding)"] direction TB A["输入提示词 (Prompt)"] --> B["小型草稿模型 (Draft Model)"] B -->|"快速生成候选 Token"| C["草稿候选 (Draft Tokens)"] C --> D["大型目标模型 (Target Model)"] D -->|"单次前向传播批量验证"| E{"验证是否接受?"} E -- "高接受率通过" --> F["输出最终 Token (大幅缩短耗时)"] E -- "拒绝/修正" --> B end ``` 为了突破这一瓶颈，Baseten 团队与 Sourcegraph 深度协同展开了一系列底层优化： 1. **运行时优化**：引入 TensorRT-LLM 作为高性能推理引擎，并配合深度的 KV Cache（键值缓存）管理。 2. **投机采样（Speculative Decoding）算法重构**：编写了超过 1,000 行核心解码算法代码，设计了质量更高的草稿 Token 生成逻辑，显著提升了大模型验证时的命中接受率。通过由轻量级草稿模型快速生成候选词、大模型单次前向传播统一验证的协作机制，系统将端到端延迟大幅压低至两百毫秒左右，达到了人类反应时间的极限阈值，成功让自动补全从“卡顿等待”转变为“实时行云流水”。 ### [12:43](https://www.youtube.com/watch?v=xYn20Yv_giE&t=763s) 案例二：1,800 张 GPU 的极速弹性与多云容量调度除了极致的单请求延迟，面对爆发式流量洪峰时的规模化供给同样是一场严峻考验。某客户在重大产品发布会前夕向 Baseten 提出算力储备诉求，团队面临的是一场横跨千卡级别的动态扩容实战。 ```mermaid flowchart TD accTitle: Multi-Cloud Capacity Management subgraph ControlPlane["全局控制面 (Global Control Plane)"] CP["统一流量路由与弹性扩缩容控制器"] end subgraph WorkloadPlanes["分布式工作负载面 (Workload Planes)"] R1["云厂商 A - 主可用区 (400 GPUs 稳态)"] R2["云厂商 A - 溢出可用区"] R3["云厂商 B / C - 跨云溢出节点 (峰值 1,800+ GPUs)"] end CP -->|"流量监控与跨区调度"| R1 CP -->|"算力打满自动溢出"| R2 CP -->|"跨云故障转移与容量补充"| R3 ``` 在发布当天，该客户的日常算力稳定在约 400 张 NVIDIA H100 GPU 左右，随后突发流量迅速触发自动化弹性扩容，瞬间飙升并跨地域调度了超过 1,800 张 H100 GPU，并在活动结束后平稳缩容回基准线。整个伸缩过程完全基于实时流量自动触发，无需人工介入。该案例的核心挑战在于，突发流量所需的 GPU 总量直接击穿了客户主用云服务商在单一地域内的物理机房库存。Baseten 依托“多云容量管理”（Multi-cloud Capacity Management）架构，建立了全局统一的控制平面（Control Plane）与分布于不同云厂商、不同地域的众多工作负载平面（Workload Planes）： - **主动-主动（Active-Active）高可用**：单一数据中心或云厂商出现故障时，业务流量无缝切换，避免全局瘫痪。 - **跨云弹性溢出**：本地算力耗尽时，动态调度其他地域与异构云资源，彻底消除硬件供应断点。 - **就近接入与成本优化**：将计算实例部署在物理位置更靠近终端用户的节点，在降低网络往返延迟的同时获取更具优势的算力定价。 ### [14:37](https://www.youtube.com/watch?v=xYn20Yv_giE&t=877s) 案例三：消灭不可见的性能瓶颈——客户端与全链路优化在追求极速推理的过程中，真正的性能瓶颈往往潜藏在模型之外的隐蔽角落。Philip 分享了开发者之间经常发生的技术争论：开发者阅读官方博客后预期获得 200 毫秒的端到端延迟，但实际测试却测出 300 毫秒，因而质疑官方数据造假。经排查发现，模型本身的执行耗时与基准测试完全一致，多出的 100 毫秒完全损耗在客户端与网络协议层的“最后一公里”中。经典的传统软件工程缺陷（如未复用 HTTP 连接、每次请求重新建立 TCP/TLS 握手）会凭空为每次调用增加几十毫秒的开销。此外，Python 的全局解释器锁（GIL）也是限制客户端高并发请求的典型元凶。针对这些痛点，Baseten 工程师专门开发了高性能客户端（Baseten Performance Client）： - **绕过 Python GIL 限制**：利用客户端多 CPU 核心并发调度，突破单线程吞吐极限。 - **高负载压测表现**：在处理 200 万条向量嵌入（Embedding）的大规模数据集回填（Corpus Backfill）与微调预处理基准测试中，在完全相同的硬件配置与模型条件下，仅通过优化客户端发送吞吐与 GPU 饱和度，便实现了整整 12 倍的吞吐量提升（12x improvement in throughput）。 ### [16:33](https://www.youtube.com/watch?v=xYn20Yv_giE&t=993s) 回归软件工程本质：生产环境推理绝非仅仅关乎 GPU Philip Kiely 在演讲的尾声部分特别强调，尽管当前的生成式 AI 领域充满了各种新奇、好玩且引人入胜的技术概念，但在落地过程中，开发者绝不能忽视扎实可靠的“传统软件工程基本功”。在构建此类面向真实业务场景的产品时，扎实的软件工程实践不仅没有过时，反而比以往任何时候都更加至关重要。为了给整场演讲做核心总结，Philip 指出，在生产环境中扩展 AI 工作负载时必须认识到：“推理绝不仅仅关乎 GPU（inference is more than just GPUs）”。一个真正具备生产级韧性的推理平台，本质上是模型运行时优化（runtime）、分布式多云基础设施（infrastructure），以及将这两者紧密编织在一起的高质量系统工程实践（engineering work）的综合体。如果把这一套技术底座构建到位，它将释放出惊人的生产力杠杆——能够支撑起那些“以空前速度爆发式增长、背后却仅由规模极小的敏捷团队（unprecedentedly small teams）所打造”的前沿 AI 产品。在致谢环节，Philip 对现场观众的聆听表达了感谢，并表示演讲结束后会在会场外停留以回答大家的具体提问。此外，他热情邀请参会者前往展厅的 Baseten 600 号展台（Booth 600）进行深入交流。他和团队同事在当天剩余时间以及次日全天都会驻守展台，现场参会者只需向工作人员说明自己参加了本场演讲，便可免费领取一件 Baseten 的定制 T 恤周边。 - **推理平台的真正构成**：生产环境下的 AI 推理绝非只是购买或租用算力芯片，而是模型运行时优化、弹性和多云基础设施、以及连接两者的系统工程三者的深度结合。 - **技术杠杆与团队规模**：优秀的推理基础设施能够抹平底层复杂性，让极其精简的工程团队承载起前所未有的超大规模用户流量与业务增长。 - **展台交流与福利**：Baseten 在大会展厅设有 600 号展位（Booth 600），听众可前往与工程师交流并凭参会凭证领取定制 T 恤。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=xYn20Yv_giE&t=4s) 推理就是一切 — Philip Kiely > [01:12](https://www.youtube.com/watch?v=xYn20Yv_giE&t=72s) 巨额收入增长伴随着极低的人员扩张 — Philip Kiely > [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) 赢下或输掉市场的差距就在眨眼之间 — Philip Kiely > [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) ChatGPT 包装工 — Philip Kiely > [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) 价格接受者 — Philip Kiely > [04:50](https://www.youtube.com/watch?v=xYn20Yv_giE&t=290s) 价格制定者 — Philip Kiely > [06:34](https://www.youtube.com/watch?v=xYn20Yv_giE&t=394s) Llama 时刻 — Philip Kiely > [09:15](https://www.youtube.com/watch?v=xYn20Yv_giE&t=555s) GPU + 现成推理框架 — Philip Kiely > [14:37](https://www.youtube.com/watch?v=xYn20Yv_giE&t=877s) 12x improvement in throughput — Philip Kiely > [16:33](https://www.youtube.com/watch?v=xYn20Yv_giE&t=993s) 传统软件工程基本功 — Philip Kiely > [16:33](https://www.youtube.com/watch?v=xYn20Yv_giE&t=993s) inference is more than just GPUs — Philip Kiely > [16:33](https://www.youtube.com/watch?v=xYn20Yv_giE&t=993s) 以空前速度爆发式增长、背后却仅由规模极小的敏捷团队所打造 — Philip Kiely ## Mentioned Philip Kiely, Baseten, Descript, Zed Industries, OpenEvidence, Sourcegraph, Cursor, iMovie, Rust, Hugging Face, DeepSeek, 智谱 AI (GLM), Kimi, 通义千问 (Qwen / Qwen-Coder / Qwen 2.5 / Qwen-Image), Mistral, Gemma, OpenAI (GPT-OSS / ChatGPT), Whisper, Orpheus TTS, Llama (Llama 3.2 3B), vLLM, SGLang, TensorRT-LLM, Amp Tab, NVIDIA H100, Python GIL, Baseten Performance Client ## Video description Philip Kiely, Developer Advocate, Baseten Abstracts Real-time generative AI applications promise transformative user experiences—but scaling inference in production is a constant battle. Demand spikes, infrastructure meltdowns, and unpredictable model behaviors lurk behind every release. Drawing from direct experience deploying mission-critical AI workloads at startups and large enterprises, this talk covers the four essential components of a production-ready inference platform: applied model performance research, distributed multi-cloud infrastructure, model management and observability tooling, and AI engineering expertise. I'll highlight common pitfalls and share concrete advice on how to build robust, resilient systems that thrive under real-world demand. [Source](https://www.youtube.com/watch?v=xYn20Yv_giE)

Source quotes

[0:04] Philip Kiely: 推理就是一切 [1:12] Philip Kiely: 巨额收入增长伴随着极低的人员扩张 [4:50] Philip Kiely: 赢下或输掉市场的差距就在眨眼之间 [4:50] Philip Kiely: ChatGPT 包装工 [4:50] Philip Kiely: 价格接受者 [4:50] Philip Kiely: 价格制定者 [6:34] Philip Kiely: Llama 时刻 [9:15] Philip Kiely: GPU + 现成推理框架 [14:37] Philip Kiely: 12x improvement in throughput [16:33] Philip Kiely: 传统软件工程基本功 [16:33] Philip Kiely: inference is more than just GPUs [16:33] Philip Kiely: 以空前速度爆发式增长、背后却仅由规模极小的敏捷团队所打造

Source quotes

[0:04] Philip Kiely: inference is everything [1:12] Philip Kiely: tiny teams [4:50] Philip Kiely: ChatGPT for doctors [4:50] Philip Kiely: literally the blink of an eye [4:50] Philip Kiely: price takers [4:50] Philip Kiely: price makers [4:50] Philip Kiely: ChatGPT wrappers [6:32] Philip Kiely: Llama moment [9:15] Philip Kiely: The path to production used to be: go to openai.com, get an API key, put it in your product, and ship. Now there's a lot more things you have to do in order to leverage these open source models effectively. [16:33] Philip Kiely: more than just GPUs

Original transcript

[0:04] Hey everyone, thank you so much for [0:05] being here today. So, I'm going to talk [0:07] about inference. What is inference? [0:10] Well, if you've seen our billboards, [0:11] you'll know that inference is [0:13] everything. And today, I'm going to [0:15] break down what that actually means. By [0:18] way of a quick introduction, I'm Philip. [0:20] The uh big green arrow is pointing at [0:22] me. These are a bunch of my friends at [0:23] base 10. We're all hanging out at uh [0:25] Cape Cod together. So, we work on AI [0:28] infrastructure. We are building a [0:30] influence platform for startups. What [0:33] does that mean? I'm going to talk [0:35] through this idea in three parts. Number [0:38] one, there are two trends that we really [0:40] need to cover to understand why this [0:42] matters. The first is the rise of the AI [0:44] native startup and the second is the [0:46] proliferation of open-source models in [0:49] the industry. Once we kind of evaluate [0:52] those two trends, we can see where they [0:54] intersect in the need for highly [0:56] reliable, highly scalable, and very low [0:59] latency inference for these products on [1:01] these open models. I'm going to talk [1:03] through a few case studies about really [1:05] cool things that we've been able to do [1:07] with our customers, and then at the end, [1:09] we're going to see how it all comes [1:11] together. [1:12] So the first trend and I don't think [1:15] this is news to anyone if you're sitting [1:17] here at this conference is the rise of [1:19] AI native companies. And what's really [1:21] special about this is how fast all of [1:24] these companies are growing. So this [1:26] chart, okay, it's not as big as it [1:28] looked on my computer when I was making [1:29] these slides. Sorry about that. Uh this [1:32] chart is going to show the number of [1:34] years it takes to scale from 1 million [1:36] to hund00 million of ARR. And before the [1:40] the record, you know, five to 10 years [1:42] in the SAS era was absolutely [1:44] bestin-class. If you could go from 1 [1:46] million to 100 million ARR in 10 years, [1:49] you were doing something right. Today [1:52] with AI native companies, that's less [1:54] that's down to one or two years. So [1:57] these companies are growing at an insane [1:59] pace where they're adding millions to [2:02] tens of millions of dollars of revenue [2:04] in very immature, very new companies in [2:07] new markets month over month over month. [2:11] So with that scale, they're also not [2:14] really adding that many employees. [2:16] There's this new trend of kind of a [2:18] trend called tiny teams where the sort [2:22] of definition of a tiny team company is [2:24] unofficially one where they you have $1 [2:27] million of ARR per employee but in [2:31] general these companies that are scaling [2:33] to again a 100 million AR that sort of [2:35] golden target are doing so with fewer [2:38] and fewer staff than ever before. Again [2:41] in that SAS era you are looking at [2:43] hundreds maybe even thousands of [2:45] employees to scale to this size where [2:48] today many AI native companies are [2:50] getting there with less than 100 [2:52] employees. So massive revenue growth [2:55] less headcount growth. How does that [2:57] work? How is that possible? [3:00] Well we've seen it up close with many of [3:02] the top AI native companies who have [3:04] chosen to trust B 10 with their [3:06] production workloads. some examples of [3:08] what these kind of companies are [3:09] building. Dscript. Oh, that's the wrong [3:12] button. Dcript over here. One of my [3:14] favorites. I use this every single day [3:16] as a content creator where before when I [3:20] wanted to make a video, I would go in [3:22] iMovie. I would load it in. I would [3:24] manually cut everything, manually add [3:26] captions, title cards, all that kind of [3:29] stuff. It could take me an hour per [3:31] minute of video to do my editing. Now, [3:35] when I want to edit a video, I just load [3:37] it into dscript. I can type and [3:39] automatically have it cut the video to [3:41] the script. I can have it automatically [3:44] inject captions. And I can basically get [3:47] through this entire workload workflow [3:50] that used to take me again one hour per [3:52] minute of finished video in an hour for [3:55] the whole video or many times even less. [3:58] So with this with this tool, you know, [4:00] we're we're powering many of their [4:02] transcription flows and many other of [4:04] these AI features that are allowing me [4:05] as a creator to work much faster, which [4:07] is why I and so many other people are [4:09] very happy to pay Dcript every month to [4:11] use their product. Another great [4:13] example, [4:15] one day I'm going to figure out where [4:16] the buttons are on this thing. Another [4:18] great example is Zed Industries. If [4:20] you've heard of Kosero, uh Zed is [4:22] another text editor uh that's with with [4:25] AI features that's a very similar space. [4:28] What Zed really focuses on is [4:30] performance. They have their own editor [4:33] written entirely in Rust so that it's [4:35] really fast and not so demanding on the [4:37] computer. And they've extended that [4:40] philosophy of speed to the AI that's [4:43] actually powering their editor working [4:44] with B 10 for faster autocomp completion [4:46] and faster models. [4:50] Now, that's actually the slide that I [4:51] wanted to go to. So, these new [4:53] companies, these new AI native [4:54] companies, what do they care about in [4:56] their AI infrastructure? Well, number [4:58] one is quality. If the product isn't [5:01] good, no one's going to use it. Great [5:03] example of this is open evidence. [5:04] They're basically chat GPT for doctors. [5:07] And when you're working in the medical [5:09] field, every single response has to be [5:11] top-notch. So for them quality task [5:14] specific quality is the number one thing [5:18] for a lot of companies then it's going [5:19] to be latency. So if you're building a [5:22] voice agent or if you're building [5:24] something where the user loop is [5:26] measured in milliseconds or hundreds of [5:28] milliseconds then the difference between [5:30] winning and losing the market can be [5:32] literally the blink of an eye. A third [5:35] thing is economics. Now, to be perfectly [5:37] frank, some of these companies are [5:39] scaling so fast that they don't really [5:40] care about their unit economics, but it [5:43] catches up with everyone eventually. And [5:45] a big part of economics actually goes [5:47] into the fourth point. But with the unit [5:49] economics in their AI, they want to move [5:51] from being a price taker where they just [5:54] accept whatever the protoen pricing is [5:56] to a price maker where they get to [5:58] control their own trade-offs. And that [6:00] goes all the way into the fourth point, [6:02] differentiation. This is what AI native [6:04] companies really care about. AI native [6:06] companies are not taking off-the-shelf [6:08] models and just saying that they're [6:10] they're good enough. Uh quick show of [6:13] hands. Uh who has ever heard of the idea [6:15] of a chat GPT wrapper? I'm sure that's a [6:17] a pretty pretty common thing. And uh [6:20] keep your hand up if you've ever built a [6:21] chat GPT wrapper like I have. Uh AI [6:24] native companies don't want to be chat [6:26] GPT rappers. So they want [6:28] differentiation at the AI layer and [6:30] control over their own destiny. So how [6:32] are they getting that? Well, that comes [6:34] to trend number two, which is open [6:36] source models. There are, honestly, this [6:39] is probably not even accurate anymore [6:41] cuz I took the screenshot like two weeks [6:43] ago. It's probably more like 1.9 or 2 [6:45] million open source AI models on Hugging [6:47] Face at this point for every single [6:50] thing you could possibly imagine. Every [6:52] modality, every size, every use case. [6:56] Some of these models are really now [6:58] state-of-the-art in their field. So, on [7:00] the LLM side, that's what everyone loves [7:02] to talk about. [7:03] Six months ago, we had Deep Seat come [7:05] out and blow up an entire news cycle, [7:07] tank Nvidia's stock. Uh, that was that [7:10] was a fun time. But now, we have so many [7:12] more models like that. We have the GLM [7:14] series of models by ZAI. We have Kimmy. [7:17] We have the new Quen Quen Coder Quen 3. [7:20] We also have a lot of really great open [7:22] source models out of Europe and America. [7:24] Mistral has great models. Gemma has [7:26] great models. GPT OSS, OpenAI just [7:29] launched that last week. So now OpenAI [7:32] even is getting back in the open source [7:34] model game for the first time since 2019 [7:36] with GPT2. [7:38] So LLMs, there are a lot of really [7:41] competitive options out there in the [7:42] open source space. But what's so cool [7:44] about open source is that there are so [7:46] many other modalities as well. You have [7:48] image models where stuff like Quen image [7:50] is state-of-the-art in image generation [7:52] as a diffusion transformer model just [7:54] like the model that blew up on Twitter [7:56] when everyone was making Studio Gibli [7:58] images a few months ago. You have [7:59] embedding models which are the [8:01] connective tissue for all kinds of AI [8:03] applications where custom fine-tuned and [8:06] open source models are making a huge [8:07] impact. Transcription models like [8:09] whisper are still the industry standard [8:12] and with TTS which is the speech or the [8:15] voice part of a voice agent. You have [8:18] Orpheus TTS which is an exceptional open [8:20] source model. We're lucky to work with [8:22] the foundation model lab that actually [8:24] created that that has really been a [8:25] llama moment in open speech. So today [8:28] you can take and fine-tune your own open [8:31] voice model that you own and get, you [8:33] know, very realistic sounding voices for [8:36] your product. [8:38] And with that, one thing that's really [8:41] cool about these models is they're all [8:42] building on each other. So Orpheus TTS, [8:44] Quen Embed, two of those models in nonLM [8:47] modalities that I was just talking about [8:49] are actually built on top of other open [8:52] models. Or if TTS, again, this is a [8:54] speech synthesis model. This is a model [8:56] that outputs sound. It was built on [8:58] Llama 3.23B, [9:00] which is a large language model from a [9:03] year ago that is made to do text. So, [9:06] you can take these large language models [9:07] or any of these open source models and [9:10] adapt them really far to stretch them to [9:12] get to your use case. [9:15] But this has also introduced a new [9:17] complexity. Before the path to [9:19] production was you go to openai.com, you [9:22] get an API key, you put it in your [9:24] product and you ship. Now there's a lot [9:26] more things you have to do in order to [9:28] leverage these open source models [9:30] effectively. You need to scale up [9:31] inference, you need to do so quickly, [9:33] and you need to do so reliably because [9:35] this can't be a blocker to getting to [9:37] market and it can't be the reason your [9:39] product doesn't work. So let's talk [9:42] about these problems. Like why is this [9:43] hard? The issue is what most people [9:46] start with is they get some GPUs and [9:48] they get a framework say VLM. Although I [9:50] don't want to pick on them. I like VLM [9:52] quite a bit. You know, people do the [9:54] same thing with SG Lang or Tensor RTL [9:56] LLM. They take an off-the-shelf [9:58] inference server. They put it on the [9:59] GPUs and they say, "All right, let's [10:01] ship." The problem is that just doesn't [10:03] really work in production. Uh because of [10:06] these things that everyone cares about. [10:07] They care about quality, latency, [10:09] economics, and differentiation. This [10:12] equation doesn't get you there. [10:14] Because all of these are really hard and [10:16] doing them at scale is even harder. The [10:19] challenges you're going to face cover [10:21] performance, cover infrastructure, [10:23] tooling, and engineering. And I'm going [10:25] to talk through three success stories [10:27] now about companies overcoming these [10:30] challenges. [10:31] So the first story I want to tell is one [10:34] of instant autocomplete. I am not the [10:37] world's fastest typer. I I I try my best [10:39] to to type pretty quick. I've gotten up [10:41] to 60 words per minute, which I feel [10:43] pretty good about. But even still, the [10:46] time in between keystrokes is is not [10:48] that long. We're talking about [10:49] milliseconds. And you know, most of my [10:52] colleagues as as software engineers type [10:53] even faster. So if you're going to [10:56] insert AI into their workflow as we're [10:58] typing, this model cannot trip us up. It [11:01] has to be as fast as our keyboard to be [11:03] able to be something that we can rely on [11:05] in our daily work. And that's exactly [11:08] the problem that uh source graph was [11:11] facing with amp tab. So what we have [11:14] here is a chart on this side of [11:16] latencies on their autocomplete model. [11:19] The model that uh you know amp tab you [11:21] hit tab to autocomplete. They were [11:24] seeing latencies spike even as high as [11:26] 700 milliseconds. If I hit tab and I [11:30] think one Mississippi and I've got to do [11:32] that every single time that I that I [11:34] take a take a pause or a new line when [11:36] I'm writing code, I'm never going to use [11:38] this product. But they were able to work [11:40] with us to roll out some optimizations [11:43] which I'll explain in detail on on the [11:45] next slide here and get those latencies [11:47] down to as low as a couple hundred [11:49] milliseconds. This is a massive [11:51] difference in product experience where [11:53] now the autocomplete is happening as [11:56] fast as human reaction time and it feels [11:58] like a seamless experience. The way we [12:01] work together to get there is through a [12:03] couple things. Number one was the [12:05] runtime layer. We were adding Tensor [12:07] RTLM as an inference engine. We were [12:09] adding KV caching, but we also did a lot [12:12] of work on the speculative decoding [12:14] side. We wrote over a thousand lines of [12:15] code for a new decoding algorithm that [12:18] would offer better, higher quality draft [12:20] tokens so that there would be a higher [12:22] acceptance rate. Speculative decoding, [12:24] by the way, basically it works by using [12:26] a small draft model to generate [12:28] suggested tokens and then the large [12:30] model comes in and verifies them. So if [12:32] your small model can create accurate [12:34] suggestions quickly, then the large [12:36] model doesn't need to run inference. You [12:38] save a ton of time and you get, you [12:40] know, the the orange arrow on this [12:42] graph. [12:43] So that's an story of latency, but [12:46] latency alone, as we just saw, doesn't [12:48] solve the entire problem. You also need [12:50] scale. So here's a story of the uh 1800 [12:53] oh wait, I mean 1,800 GPU launch. So we [12:58] had a customer come to us and said, [13:00] we're we're doing a big launch and how [13:02] many GPUs do you have? People love [13:04] asking us how many GPUs we have. And the [13:06] answer was, how many GPUs do you need? [13:09] What we can see here is a graph of [13:13] Nvidia H100's uh which we were scaling [13:16] up across different regions. Every [13:18] single one of these colors is a [13:20] different region. And you can see as the [13:22] launch day went, they were, you know, [13:24] kind of in a steady state around 400 [13:26] GPUs, spiked to over 1,800 GPUs, and [13:30] then scaled back down through the day. [13:32] This scaling process was entirely [13:35] automated. It was entirely trafficbased. [13:37] And what was really cool about this [13:39] launch was that they actually exceeded [13:42] the number of GPUs that were physically [13:45] located in the region of the cloud [13:48] provider that they were using as a [13:49] primary. But what we were able to do is [13:52] spill those GPUs into other regions and [13:54] other cloud providers to be able to hit [13:57] that spike and you know not just [13:59] literally run out of GPUs in the data [14:01] center. [14:04] We were able to do this with something [14:05] called multi cloud capacity management [14:07] which is the idea of having a single [14:09] global control plane and numerous [14:10] workload planes across different cloud [14:12] providers and different regions. This [14:14] allows you to scale your workloads both [14:16] reliably with active active reliability [14:19] so that if one cloud provider or one [14:20] region goes down, your entire workloads [14:23] don't go down with them, but also to be [14:25] able to access GPUs everywhere to [14:26] improve latency by putting GPUs closer [14:29] to your customers as well as improving [14:31] scale by being able to access more [14:33] hardware and get better pricing in some [14:35] cases. [14:37] The final story I want to tell is one [14:39] that that hits a little bit closer to [14:40] home as a developer and that is one of [14:43] invisible bottlenecks. So I've had this [14:46] conversation a few times with other [14:48] developers where they say, "Hey, Philip, [14:50] your blog post told me that I was going [14:52] to be able to run this model and see [14:54] endto-end latency of say 200 [14:56] milliseconds and I'm seeing 300 [14:59] milliseconds and so you're a liar. Uh [15:02] you know, go fix your blog post." And [15:04] I'm like, well, let me let me check your [15:06] model. Okay, everything looks so good on [15:07] the model. Let me check your client [15:09] code. You know, actually, a lot of the [15:12] problems that we end up running into [15:14] while we're scaling these AI models in [15:17] production are less about the model [15:18] themselves. You have the optimized [15:20] runtime. You have the optimized [15:22] infrastructure setup. How do you get [15:24] that last mile? Sometimes it go just [15:26] goes back to good oldfashioned software [15:28] engineering stuff like session reuse [15:30] because every time you establish a new [15:32] HTTP session that takes a few dozen [15:34] milliseconds. If you add that on every [15:36] single call that's definitely going to [15:38] be noticeable in the user experience or [15:40] maybe your client is the bottleneck. We [15:43] actually had one of our engineers put [15:44] together this really great thing. We [15:46] call it the B 10 per performance client [15:49] and it uh it basically allows you to [15:52] bypass the Python global interpreter [15:54] lock which limits parallelism and hit [15:56] high concurrency by going across [15:59] multiple CPU cores on your client with [16:02] you know for example an embedding [16:04] workload. We benchmarked it on [16:06] processing 2 million embeddings which is [16:08] a realistic workload if you're doing say [16:10] a corpus back fill of rag or maybe [16:12] you're pre-processing a bunch of data to [16:14] do a fine-tuning job. we were actually [16:16] able to see on the same model on the [16:19] same GPU a 12x improvement in throughput [16:22] just because the client was able to [16:24] actually send enough traffic to fully [16:26] saturate the GPU. So again, while all of [16:29] these topics around model performance [16:31] and model infrastructure are very [16:33] exciting to talk about, very fun, very [16:35] novel, don't don't ignore the sort of [16:39] basics and fundamentals of good solid [16:41] software engineering because those are [16:43] more important than ever when building [16:44] these products. So to review, when [16:48] scaling AI in production, inference is [16:50] more than just GPUs. Inference is a [16:54] mixture of the runtime, the [16:56] infrastructure, and the engineering work [16:59] that ties it all together. And when done [17:01] right, it powers these products that are [17:03] growing at unprecedented rates with [17:06] unprecedentedly small teams. Thank you [17:08] so much for your time today. Really [17:10] appreciate you joining me and I'll be [17:12] available outside if you have any [17:13] questions. Uh, B 10 has booth 600. We're [17:17] over in the expo hall, so I'll also be [17:19] hanging out there rest of today, all day [17:21] tomorrow with my friends uh friends and [17:23] colleagues. Definitely stop by booth 600 [17:26] and uh grab one of these shirts. Just [17:28] tell them you're at the talk and we'll [17:29] make sure we grab one for you. So, thank [17:31] you so much. Have a great day.