Simplismart Slashes Voice AI Latency 72% via Custom Inference Stacks

Building The Right Inference For You: Balancing Latency, Throughput, and Cost With Simplismart — Ai4

Ai4

Replacing generic serverless endpoints with tailor-made model infrastructure allows engineering teams to meet strict real-time SLAs while cutting compute costs in half.

AI-assisted summary of Building The Right Inference For You: Balancing Latency, Throughput, and Cost With Simplismart — original source. Check the original for full context.

Source published: 2025-08-24

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 24, 2025, this presentation features Amritanshu Jain, CEO and Co-founder of Simplismart. According to the video description, the session presents a modular framework for designing scalable, SLA-compliant AI inference systems that balance latency, throughput, and cost across diverse workloads rather than relying on rigid monolithic stacks.

## TL;DR Amritanshu Jain demonstrates why generic serverless endpoints often dictate suboptimal trade-offs between cost, latency, throughput, and accuracy for production AI workloads. Through benchmarks on models like Llama 3.1 8B and a voice agent case study, he shows how tailor-made inference dropped end-to-end response latency from 1,050 ms to 290 ms while maintaining tight cost boundaries. He highlights the complexities of navigating 'grid search hell'—from KV caching to speculative decoding—and advocates for purposeful, customized infrastructure configurations that match specific application SLAs.

## Atmosphere & style The presentation is brisk, pragmatic, and deeply technical, delivered with the grounded confidence of an engineer experienced in high-stakes infrastructure. Jain uses concrete numbers, architectural breakdowns, and realistic trade-off dilemmas to demystify deployment challenges. The tone is engaging, rational, and problem-oriented, making complex systems optimization feel approachable and actionable.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=61S2_r3p3mY&t=4s) Generic serverless infrastructure often creates bottlenecks for production applications deploying open-source models that require custom latency, throughput, and cost SLAs. - [01:30](https://www.youtube.com/watch?v=61S2_r3p3mY&t=90s) Standard serverless endpoints force infrastructure constraints to dictate application trade-offs, rather than allowing business requirements to determine infrastructure design. - [01:30](https://www.youtube.com/watch?v=61S2_r3p3mY&t=90s) AI inference optimization fundamentally revolves around balancing four interdependent dimensions: cost, latency (TTFT/TTFB), throughput, and model accuracy. - [02:15](https://www.youtube.com/watch?v=61S2_r3p3mY&t=135s) The same model architecture can exhibit radically different economics depending on configuration; for instance, Llama 3.1 8B can run at $0.03 per million tokens for batch workloads or $0.20 per million tokens for high-throughput streaming. - [03:30](https://www.youtube.com/watch?v=61S2_r3p3mY&t=210s) Workloads broadly split into latency-critical tasks like conversational voice agents and cost-critical tasks like bulk document processing or contact center batch transcriptions. - [05:26](https://www.youtube.com/watch?v=61S2_r3p3mY&t=326s) Multi-stage agent pipelines—such as Speech-to-Text to LLM to Text-to-Speech—compound latencies across stages, often making generic serverless endpoints too slow for interactive use. - [06:15](https://www.youtube.com/watch?v=61S2_r3p3mY&t=375s) Custom backend engineering can dramatically accelerate models, demonstrated by optimizing Whisper to 1,300x Real-Time Factor on an H100 to reduce speech-to-text TTFB from 500 ms to 70 ms. - [07:00](https://www.youtube.com/watch?v=61S2_r3p3mY&t=420s) Extreme inference efficiency can drastically alter unit economics, allowing an organization to transcribe a continuous month of audio for approximately $1. - [07:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=465s) Tailoring inference infrastructure across pipeline components can drop end-to-end response times from over 1,000 ms down to 290 ms to enable natural voice interaction. - [08:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=525s) Engineers face 'grid search hell' when tuning production models across competing variables such as serving frameworks, execution backends, KV caching, and speculative decoding. - [09:15](https://www.youtube.com/watch?v=61S2_r3p3mY&t=555s) Optimizing beyond an SLA target can be counterproductive; choosing a 120 ms TTFT instead of an 80 ms TTFT preserved the sub-300 ms voice SLA while cutting serving costs by 50%. - [10:30](https://www.youtube.com/watch?v=61S2_r3p3mY&t=630s) High-performance production AI requires moving away from generic, shared serverless APIs toward tailor-made inference systems where developers control their exact placement on the trade-off curve.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=61S2_r3p3mY&t=4s) Introduction and Background in ML Infrastructure

Amritanshu Jain, co-founder of Simplismart, opens the session with a lighthearted greeting to attendees on day one of the conference. Before diving into inference, he shares his engineering background managing and optimizing machine learning training and inference infrastructure across several organizations. During his time at Oracle Cloud, he built low-latency simulation systems for the Red Bull Racing Formula 1 team, alongside building and deploying extensive suites of natural language processing and computer vision models.

```mermaid flowchart LR accTitle: Target Audience for Tailor-Made AI Inference A["Teams Running Open-Source Models in Production"] --> D["Tailor-Made Inference Solutions"] B["Applications Facing Shared Serverless Bottlenecks"] --> D C["Engineers Needing Custom Latency/Cost SLAs"] --> D ```

He frames the discussion around the realities of deploying open-source models in production. Whether an engineering team is relying on shared endpoints, struggling against scaling bottlenecks, or looking for deeper architectural control, generic infrastructure often fails to meet production demands.

---

[01:30](https://www.youtube.com/watch?v=61S2_r3p3mY&t=90s) The Inference Dilemma: Four Key Trade-offs

A common assumption in modern AI engineering is that one can simply plug a serverless endpoint into an application and move on. Amritanshu challenges this assumption by posing a fundamental architectural question: "Do your trade-offs dictate your infra or does your infrastructure actually dictate your trade-offs?" With standard serverless APIs, the infrastructure invariably dictates the trade-offs, leaving teams unable to achieve an optimal deployment point tailored to their unique workloads.

```mermaid flowchart TD accTitle: The Four Core AI Inference Trade-Offs T["Core Inference Trade-offs"] --> C["Cost (Cents per Million Tokens / Minute)"] T --> L["Latency (TTFT / TTFB Response Times)"] T --> TH["Throughput (Tokens per Second / Concurrency)"] T --> A["Accuracy & Model Quality"] ```

Inference optimization revolves around balancing four core metrics: cost, latency, throughput, and accuracy. To illustrate how stark these trade-offs can be, Amritanshu shares concrete benchmark numbers for deploying a Llama 3.1 8B parameter model across different operational profiles:

Workloads fall across a wide spectrum between latency-sensitive and cost-sensitive tasks: - **Latency-Sensitive Workloads:** Voice agents requiring sub-second audio turnarounds, real-time diffusion pipelines and video generation (such as workloads run by companies like NVIDIA), and complex reasoning architectures where multiple mixture-of-experts (MoE) routing calls occur sequentially. - **Cost-Sensitive Workloads:** High-volume document parsing (e.g., analyzing 1,000,000 documents per day where raw execution time is secondary to cost per document), contact center batch transcriptions, and mass image, text, or video content generation.

---

[05:26](https://www.youtube.com/watch?v=61S2_r3p3mY&t=326s) Case Study: Optimizing an End-to-End Voice Agent Pipeline

To demonstrate the real-world impact of tailored inference, Amritanshu walks through a case study involving a customer building a debt collection voice agent. The system utilizes a standard three-stage streaming pipeline: Speech-to-Text (STT) feeding into a Large Language Model (LLM), which streams into a Text-to-Speech (TTS) engine. The two make-or-break constraints were voice responsiveness and achieving viable economics—targeting an operational threshold of around 4 cents on the dollar per conversation minute.

```mermaid flowchart LR accTitle: Voice Agent Pipeline Optimization Comparison subgraph InitialStack["Initial Serverless Stack: Total ~1050 ms TTFT/TTFB"] direction LR S1["Serverless STT API 500 ms TTFB"] --> L1["Serverless Gemma 34B 450 ms TTFT"] --> T1["Serverless TTS API 100 ms TTFB"] end

subgraph OptimizedStack["Simplismart Tailor-Made Stack: Total ~290 ms"] direction LR S2["Custom OpenAI Whisper on H100 70 ms TTFB"] --> L2["Tuned Gemma 34B on H100 120 ms TTFT"] --> T2["Proprietary TTS API 100 ms TTFB"] end ```

In the customer's initial implementation using commercial third-party serverless APIs, Time-to-First-Byte (TTFB) for STT was 500 ms, Gemma 34B Time-to-First-Token (TTFT) was 450 ms, and TTS TTFB was 100 ms, totaling a sluggish 1,050 ms end-to-end response time. Simplismart overhauled the infrastructure:

- **Speech-to-Text Overhaul:** Deployed an optimized version of OpenAI Whisper. While standard Whisper lacks native streaming support and achieves roughly 40x Real-Time Factor (RTF) on standard serverless APIs, Simplismart pushed performance to 1,300x RTF on a single NVIDIA H100 GPU. This means 1,300 seconds of audio are transcribed in 1 second of H100 compute time, bringing STT TTFB down from 500 ms to 70 ms. At this efficiency, an organization can transcribe an entire month of continuous audio (24 hours × 30 days) for just $1. - **LLM Optimization:** Deployed Gemma 34B on dedicated H100 hardware, dropping TTFT from 450 ms down to 120 ms. - **Text-to-Speech:** Retained the proprietary serverless TTS endpoint at 100 ms TTFB, acknowledging that proprietary voice engines still outperform open-source TTS models in natural vocal quality. - **Overall Result:** End-to-end response latency dropped from 1,050 ms down to 290 ms, creating a natural, conversational interaction loop.

---

[08:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=525s) Navigating "Grid Search Hell" with Tailor-Made Inference

During the Gemma 34B deployment, Simplismart faced an explicit trade-off decision: one configuration achieved an ultra-low TTFT of 80 ms (and could even reach 60 ms), while another achieved 120 ms. However, running at 80 ms doubled the total serving cost. Because the voice agent's SLA target was anything under 300 ms end-to-end, they deliberately chose the 120 ms configuration, staying well within the client's SLA while cutting compute costs by 50%.

```mermaid flowchart TD accTitle: Navigating ML Grid Search Decisions G["Production AI Deployment"] --> F["Serving Framework Selection"] G --> B["Model Execution Backend"] G --> K["KV Caching Architecture"] G --> LC["Context Length Handling (30k-40k tokens)"] G --> SD["Speculative Decoding & Batching"] G --> S["Trade-off Alignment (SLA vs Cost vs Throughput)"] ```

Amritanshu describes the overwhelming matrix of choices that engineers face as "grid search hell." When moving an AI model into production, teams must evaluate dozens of competing variables: - Selecting serving frameworks and execution backends. - Choosing memory-efficient Key-Value (KV) caching strategies. - Handling long-context prompt processing (e.g., 30,000 to 40,000 input tokens). - Implementing speculative decoding techniques versus standard dynamic batching.

He concludes by emphasizing that high-performance products require "not generic inference but tailor-made inference," where the developer controls where their workload falls on the cost-latency-throughput spectrum. Attendees are invited to visit Booth 524 to test interactive deployment games and explore an automated copilot designed to determine the optimal serving strategy for any custom model workload.

## Notable quotes > [01:30](https://www.youtube.com/watch?v=61S2_r3p3mY&t=90s) Do your trade-offs dictate your infra or does your infrastructure actually dictate your trade-offs? — Amritanshu Jain

> [00:04](https://www.youtube.com/watch?v=61S2_r3p3mY&t=4s) generic infrastructure often fails to meet production demands. — Amritanshu Jain

> [01:30](https://www.youtube.com/watch?v=61S2_r3p3mY&t=90s) Inference optimization revolves around balancing four core metrics: cost, latency, throughput, and accuracy. — Amritanshu Jain

> [05:26](https://www.youtube.com/watch?v=61S2_r3p3mY&t=326s) 1,300 seconds of audio are transcribed in 1 second of H100 compute time — Amritanshu Jain

> [05:26](https://www.youtube.com/watch?v=61S2_r3p3mY&t=326s) an organization can transcribe an entire month of continuous audio (24 hours × 30 days) for just $1. — Amritanshu Jain

> [08:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=525s) grid search hell — Amritanshu Jain

> [08:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=525s) running at 80 ms doubled the total serving cost. — Amritanshu Jain

> [08:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=525s) not generic inference but tailor-made inference — Amritanshu Jain

## Mentioned Amritanshu Jain, Simplismart, Ai4, Oracle Cloud, Red Bull Racing Formula 1 team, Llama 3.1 8B, NVIDIA, NVIDIA H100, OpenAI Whisper, Gemma 34B

## Video description Amritanshu Jain, CEO & Co-founder, Simplismart

Abstracts

How can inference systems be designed to balance latency, throughput, and cost across diverse AI workloads? Rigid, monolithic stacks often lead to inefficient trial-and-error. This session introduces a modular framework for architecting SLA-compliant, scalable systems tailored to specific use cases. Gain a structured methodology to make informed trade-offs and deploy performant AI systems at scale.

[Source](https://www.youtube.com/watch?v=61S2_r3p3mY)

中文简报

Simplismart通过定制推理栈将语音AI延迟降低72% 以量身定制的模型基础设施取代通用Serverless端点，让工程团队在计算成本减半的同时满足严格的实时SLA。 Amritanshu Jain阐明了通用Serverless端点为何往往迫使生产级AI工作负载在成本、延迟、吞吐量和准确率之间做出次优权衡。通过对Llama 3.1 8B等模型的基准测试以及语音智能体案例研究，他展示了定制化推理如何在保持严格成本界限的同时，将端到端响应延迟从1,050毫秒降低至290毫秒。他指出了从KV缓存到推测解码等应对“网格搜索地狱”的复杂性，并倡导采用匹配特定应用SLA的有针对性的定制基础设施配置。通用的 Serverless AI 基础设施往往迫使架构约束决定业务权衡，而真正的推理优化应根据业务 SLA 在成本、延迟（TTFT/TTFB）、吞吐量和模型精度之间取得平衡。 (1:30) 同一模型在不同配置下经济效益差异巨大；例如部署 Llama 3.1 8B 时，离线批处理成本仅为每百万 Token 0.03 美元，而面向实时交互的高吞吐流式处理成本则为 0.20 美元。 (1:30) 多阶段语音 Agent 会产生延迟叠加，Simplismart 通过在单张 H100 上对 Whisper 进行深度定制优化，实现了 1,300 倍实时率（RTF），并将首字节时间（TTFB）从 500 毫秒压缩至 70 毫秒，整套 Agent 端到端延迟从 1,050 毫秒降低至 290 毫秒。 (5:26) 面对服务框架、KV Cache 和推测解码等复杂的“网格搜索地狱”，工程团队应采用定制化推理（Tailor-Made Inference），在满足业务 SLA 的前提下避免过度配置，从而降低 50% 的算力成本。 (8:45)

Reading summary

Amritanshu Jain, co-founder of Simplismart, argues that generic serverless endpoints force rigid operational trade-offs onto machine learning teams, whereas tailor-made infrastructure allows developers to balance latency, throughput, and cost against specific application requirements. The Core Trade-offs in Model Serving (1:30) Production model deployment centers on balancing four core metrics: cost, latency, throughput, and accuracy. Jain states that relying on off-the-shelf serverless endpoints lets the provider's infrastructure dictate these trade-offs rather than the application's actual operational needs. To illustrate the divergence between deployment profiles, he highlights benchmarks for serving a Llama 3.1 8B parameter model: a cost-optimized batch configuration yields 50 tokens per second at $0.03 per million tokens, whereas a latency- and throughput-optimized setup yields 1,200 tokens per second at $0.20 per million tokens. Workloads naturally divide between latency-sensitive and cost-sensitive profiles. Latency-sensitive applications include real-time voice agents requiring sub-second response loops, real-time diffusion pipelines, and sequential mixture-of-experts (MoE) reasoning calls. Conversely, cost-sensitive workloads—such as high-volume document ingestion parsing one million documents daily or large-scale batch audio transcription—prioritize low per-unit compute expenses over immediate execution speeds. Case Study: Optimizing a Voice Pipeline (5:26) Jain demonstrates the financial and operational impact of custom infrastructure through a debt collection voice agent requiring low latency and an operational budget threshold near 4 cents per conversation minute. The system used a three-stage pipeline: Speech-to-Text (STT), a Large Language Model (Gemma 34B), and Text-to-Speech (TTS). Using standard third-party serverless endpoints, the initial pipeline suffered an unviable 1,050 ms total response time: 500 ms Time-to-First-Byte (TTFB) for STT, 450 ms Time-to-First-Token (TTFT) for Gemma 34B, and 100 ms TTFB for TTS. Simplismart overhauled the architecture by shifting the first two stages onto dedicated NVIDIA H100 hardware. By deploying an optimized implementation of OpenAI Whisper, STT performance jumped from standard 40x Real-Time Factor (RTF) to 1,300x RTF, dropping STT TTFB to 70 ms while enabling continuous transcription for roughly $1 per month. Gemma 34B was tuned to achieve a 120 ms TTFT, and the proprietary TTS engine was retained at 100 ms TTFB due to superior audio naturalness. The end-to-end response time dropped to 290 ms, achieving conversational viability. Resolving Grid Search Configuration Dilemmas (8:45) Engineering custom inference requires navigating what Jain characterizes as "grid search hell," where teams must evaluate competing serving frameworks, execution backends, memory-efficient Key-Value (KV) caching strategies, long-context inputs ranging from 30,000 to 40,000 tokens, and speculative decoding techniques versus dynamic batching. This optimization process requires aligning infrastructure choices directly with service-level agreements (SLAs) rather than chasing raw performance benchmarks. During the voice agent deployment, Simplismart benchmarked a Gemma 34B configuration that achieved an ultra-low TTFT of 80 ms (with potential down to 60 ms), but found it doubled overall serving costs. Because the customer's end-to-end latency SLA was 300 ms, the team deliberately selected the 120 ms configuration, fulfilling the latency requirement while cutting compute costs by 50%. Jain concludes that achieving production-grade AI economics requires abandoning one-size-fits-all serverless APIs in favor of tailored serving stacks, though doing so demands that engineering teams carefully navigate the architectural matrix of backends, hardware, and operational trade-offs.

精炼阅读摘要

Simplismart 联合创始人 Amritanshu Jain 指出，通用型无服务器（Serverless）推理接口迫使应用去被动适应底层基础设施的预设限制；企业若要在生产环境中兼顾性能与商业可行性，必须根据特定业务的服务等级协议（SLA），定制化权衡延迟、吞吐量、成本与精度。四维权衡体系与吞吐成本分化 (1:30) Amritanshu Jain 提出，模型推理优化的核心在于平衡成本、延迟、吞吐量与模型精度这四个维度。开发者在采用标准 Serverless API 时，往往是由现成设施决定了应用的性能上限与成本结构，无法根据具体负载进行灵活调优。他援引部署 Llama 3.1 8B 模型的基准数据指出，针对离线文档处理等成本优先型任务，系统可通过每秒 50 个 Token 的低吞吐配置，将每百万 Token 成本压低至 0.03 美元；而对于实时交互式聊天机器人或流式智能体等延迟优先型任务，系统吞吐量提升至每秒 1,200 个 Token，成本相应上升至每百万 Token 0.20 美元。 Jain 分析认为，生产环境中的工作负载在指标敏感度上呈现两极化分布。语音智能体（要求亚秒级响应）、实时扩散与视频生成链路（如 NVIDIA 处理的负载），以及涉及多跳混合专家模型（MoE）路由的复杂推理系统，属于对延迟极度敏感的应用；而在大规模文档解析（如日均处理百万份文档）、联络中心离线语音转写以及大批量内容生成等场景中，吞吐与单次运行成本则成为决定业务可行性的核心约束。端到端语音智能体低延迟实操案例 (5:26) 在优化某客户债务催收语音智能体的案例中，系统由“语音转文本（STT）- 大语言模型（LLM）- 文本转语音（TTS）”构成的流式串联链路组成，业务要求具备逼真的实时对话响应能力，并将运营成本控制在每分钟通话约 0.04 美元之内。客户原先采用第三方商用 Serverless API，其中 STT 首字节时间（TTFB）为 500 毫秒，Gemma 34B 模型的首 Token 时间（TTFT）为 450 毫秒，TTS 的 TTFB 为 100 毫秒，导致端到端总延迟高达 1,050 毫秒，存在显著卡顿。 Simplismart 对该推理链路进行了全栈定制改造。针对原生缺乏流式支持且在常规 Serverless 下仅达到约 40 倍实时率系数（RTF）的 OpenAI Whisper 模型，团队在单张 NVIDIA H100 GPU 上将其 RTF 提升至 1,300 倍（即 1 秒硬件计算可处理 1,300 秒音频），使 STT TTFB 降至 70 毫秒；Jain 称在此效率下连续转写整月音频仅消耗 1 美元计算成本。针对 LLM 环节，通过在专用 H100 硬件上部署调优后的 Gemma 34B，TTFT 从 450 毫秒降至 120 毫秒。TTS 环节因商用专有接口在音色自然度上仍优于开源方案而予以保留（维持 100 毫秒 TTFB）。整套系统端到端延迟最终降至 290 毫秒，实现了自然的交互节奏。摆脱网格搜索困境与精准成本折衷 (8:45) Jain 将工程师将 AI 模型推向生产环境时面对的庞大配置矩阵称为“网格搜索困境”。技术团队必须在 Serving 框架、模型执行后端、KV 缓存机制、长上下文（如 30,000 至 40,000 Token）处理方式，以及推测解码与动态批处理技术之间进行大量评估测试。若缺乏针对具体场景的精确定位，盲目追求极致性能往往会导致硬件成本大幅攀升。在 Gemma 34B 的实际部署中，Simplismart 面临过明确的折衷决策：一种配置可将 TTFT 压低至 80 毫秒（极限可达 60 毫秒），但服务成本会直接增加一倍；另一种配置的 TTFT 为 120 毫秒。考虑到该语音智能体的全局端到端响应 SLA 设定在 300 毫秒以内，团队权衡后主动选择了 120 毫秒配置，在确保完全达标的前提下为客户降低了 50% 的算力成本。Jain 强调，满足高性能产品需求的关键在于根据业务目标实施“量身定制的推理”，让开发者主动掌握成本、延迟与吞吐之间的平衡控制权。 Amritanshu Jain 总结认为，开源模型生产落地的瓶颈主要源于通用 Serverless 架构在资源调度上的僵化，转向基于业务 SLA 深度定制的推理方案是兼顾性能与成本的必要路径。然而，在执行引擎、缓存架构与批处理机制等繁多维度中筛选最优方案仍给工程团队带来极高的试错门槛，亟需通过自动化辅助工具与模拟评测来加速最优服务策略的构建。

Detailed notes — zh

## Background 该视频由 Ai4 频道于 2025 年 8 月 24 日发布在 YouTube 平台上。视频简介指出，Simplismart 联合创始人兼 CEO Amritanshu Jain 探讨了如何在多样化的 AI 工作负载中平衡延迟、吞吐量和成本，并介绍了一套模块化框架，旨在替代僵化低效的单体技术栈，帮助团队按具体业务场景构建符合 SLA 的可扩展推理系统。 ## TL;DR 演讲直击当下许多开发者被第三方通用 Serverless API 绑架架构权衡的痛点，指出 AI 推理必须在成本、延迟、吞吐量和精度四个维度之间做出针对性取舍。通过对比 Llama 3.1 8B 在离线批处理与高并发实时场景下的巨大架构跨度，讲者清晰划定了延迟敏感型与成本敏感型工作负载的不同设计哲学。结合一个催收语音智能体的真实改造案例，他展示了团队如何通过定制化重构 Whisper 与 Gemma 34B，将端到端首字延迟从 1050ms 压降至 290ms，并在满足 300ms SLA 的前提下避免无谓的算力成本翻倍。最后，他呼吁企业摆脱在框架、后端与缓存策略中盲目试错的“网格搜索地狱”，采用量体裁衣的定制化推理架构。 ## Atmosphere & style 整场演讲呈现出典型的高密度、务实且直击痛点的工程技术分享风格。主讲人 Amritanshu 凭借深厚的底层算力调优背景侃侃而谈，语调自信从容、毫不拖泥带水，开篇便抛出反思性诘问直击现场工程师的心声。分享中穿插了生动形象的类比（如“网格搜索地狱”）与极具冲击力的真实落地数据，使原本枯燥复杂的系统底层权衡变得生动直观，营造出既专业硬核又高度关注商业落地性价比的峰会现场技术交流氛围。 ## Key takeaways - [01:32](https://www.youtube.com/watch?v=61S2_r3p3mY&t=92s) AI 推理本质上是成本（Cost）、延迟（Latency）、吞吐量（Throughput）与精度（Accuracy）四个核心维度之间的工程权衡取舍，不存在放之四海而皆准的默认方案。 - [01:32](https://www.youtube.com/watch?v=61S2_r3p3mY&t=92s) 同一个模型（如 Llama 3.1 8B）在不同部署架构下呈现截然相反的性能与成本表现，从 50 tokens/s（$0.03/1M tokens）到 1200 tokens/s（$0.20/1M tokens）不等。 - [03:46](https://www.youtube.com/watch?v=61S2_r3p3mY&t=226s) AI 工作负载可分为延迟敏感型（如流式语音智能体、实时生成、MoE 模型）与成本敏感型（如百万级文档解析、离线音频转录），两者的架构设计哲学完全对立。 - [05:28](https://www.youtube.com/watch?v=61S2_r3p3mY&t=328s) 级联式语音智能体（STT + LLM + TTS）若依赖通用 Serverless API，各环节延迟累加会导致端到端首字耗时高达 1050ms，产生严重停顿感。 - [05:28](https://www.youtube.com/watch?v=61S2_r3p3mY&t=328s) 通过在单张 H100 上深度优化 Whisper 引擎，可实现 1300x RTF 的极高并行吞吐与 70ms 超低首字延迟，大幅降低算力成本（1 美元可处理约 720 小时音频）。 - [05:28](https://www.youtube.com/watch?v=61S2_r3p3mY&t=328s) 针对全链路进行定制化部署与编译加速后，语音助手端到端延迟可从 1050ms 压低至 290ms，成功跨入人类自然交谈的 300ms SLA 门槛。 - [08:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=525s) 推理性能优化应受业务 SLA 约束，盲目追求极致低延迟（如将 LLM 从 120ms 压到 80ms）可能带来算力成本翻倍的代价，必须在满足整体 SLA 的前提下寻求最优性价比。 - [08:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=525s) 工程师在推理落地中常陷入“网格搜索地狱”，需要在推理框架（vLLM/TensorRT-LLM）、KV 缓存管理、投机解码与批处理策略等繁杂变量中反复试错。 - [10:49](https://www.youtube.com/watch?v=61S2_r3p3mY&t=649s) 企业应当告别泛化的通用基础设施，转向“量体裁衣”（Tailor-made）的定制化推理架构，结合推理 Copilot 工具针对具体模型与 SLA 快速生成最优部署策略。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=61S2_r3p3mY&t=4s) 讲者背景与演讲初衷：告别泛化的 AI 推理瓶颈 Simplismart 联合创始人兼 CEO Amritanshu Jain 登台分享了关于大模型推理架构优化的核心见解。在创立 Simplismart 之前，Amritanshu 曾在多家企业担任机器学习工程师，主要负责底层 ML 基础设施的管理与优化，涵盖大规模训练与推理架构。在 Oracle Cloud 任职期间，作为 F1 赛车爱好者的他，曾主导为 Red Bull（红牛车队）构建低延迟模拟仿真系统，并在自然语言模型和计算机视觉模型的训练与生产部署领域积累了深厚的实战经验。 Amritanshu 开门见山地指出，这场演讲专门面向三类人群：在生产环境中运行开源模型的开发者、依赖共享 Serverless API 端点却遭遇扩展瓶颈的架构师，以及在峰会第一天就想深入底层硬核技术的从业者。随着 AI 基础设施成为全行业热议的焦点，推理（Inference）层面的架构设计正成为决定应用成败的关键分水岭。 - **讲者从业背景**：曾任职于 Oracle Cloud，为红牛 F1 车队构建低延迟仿真系统，长期专注 NLP 与视觉模型的训练与推理优化。 - **适用业务场景**：正在将开源大模型落地到生产环境，或正在使用第三方共享 API 并遇到性能瓶颈与扩缩容限制的团队。 --- ### [01:32](https://www.youtube.com/watch?v=61S2_r3p3mY&t=92s) 推理架构的核心拷问：四个维度的权衡取舍与 Llama 3.1 8B 实例许多开发者习惯于直接将公有云上的 Serverless API 端点接入应用程序，认为推理只是简单的 API 调用。对此，Amritanshu 抛出了一个直击本质的问题：“到底是你的业务权衡决定了基础设施，还是你的基础设施反过来绑架了你的权衡？”（"Do your trade-offs dictate your infra or does your infrastructure actually dictate your trade-offs?"）。现实中，使用通用 Serverless API 往往属于后者——由于底层架构固定，开发者无法根据业务特性精细调节系统，最终导致扩展受阻、成本失控。 AI 推理系统本质上是在四个核心指标之间做权衡取舍（Trade-off）：成本（Cost）、延迟（Latency）、吞吐量（Throughput）和精度（Accuracy）。以部署开源模型 Llama 3.1 8B 为例，业务需求的不同会导致完全截然相反的部署形态： 1. **低吞吐/极致成本方案**：以 50 tokens/s 的吞吐运行，处理 100 万 tokens 的成本仅需 3 美分（$0.03/1M tokens），非常适合离线批处理或非实时文档处理。 2. **高吞吐/低延迟方案**：以 1200 tokens/s 的超高吞吐运行，成本则上升至每 100 万 tokens 20 美分（$0.20/1M tokens），适用于高并发实时交互场景。同一个模型在不同架构下展现出跨越数倍成本与数十倍吞吐的光谱，这证明通用型基础设施无法兼顾所有需求，唯有“量体裁衣”（Tailor-made）的推理架构才能实现最优投资回报比。 ```mermaid flowchart LR accTitle: Trade-off Spectrum for Llama 3.1 8B subgraph S1["低成本/离线批处理"] A["吞吐量: 50 tokens/s"] --> B["成本: $0.03 / 1M tokens"] end subgraph S2["高吞吐/实时在线交互"] C["吞吐量: 1200 tokens/s"] --> D["成本: $0.20 / 1M tokens"] end ``` - **四大权衡维度**：成本（API 费用上限）、延迟（响应耗时）、吞吐量（并发处理能力）、精度（模型质量与量化损失）。 - **Llama 3.1 8B 部署对比**：成本从 $0.03 到 $0.20 每百万 tokens 不等，吞吐量从 50 tokens/s 跨越至 1200 tokens/s。 --- ### [03:46](https://www.youtube.com/watch?v=61S2_r3p3mY&t=226s) 业务场景分类：延迟敏感型 vs. 成本敏感型工作负载根据业务对四项指标的侧重，AI 工作负载大致可划分为“延迟敏感型”与“成本敏感型”两大阵营。不同阵营的设计哲学截然不同，强行套用同一种架构必然导致资源浪费或用户体验崩塌。在**延迟敏感型（Latency Sensitive）**场景中，毫秒级的响应速度决定了产品的生死： - **语音智能体（Voice Agents）**：机器响应人类声音时必须达到人类交谈级别的自然流利度，极短的延迟才能避免令人尴尬的停顿。 - **实时扩散模型生成（Diffusion Pipelines）**：例如英伟达等公司推进的实时视频与图像生成流，更低的延迟能显著提升交互沉浸感。 - **推理模型与专家混合模型（Reasoning & MoE Models）**：由于单次推理涉及多次内部调用或多专家路由协作，底层的单次调用延迟必须压到极致。在**成本敏感型（Cost Sensitive）**场景中，系统往往处理海量数据，单价的微小差异会被无限放大： - **海量文档批量解析**：例如每天解析上百万份非结构化文档，业务方完全不关心是 1 秒还是 10 秒返回，核心诉求是将单份文档的处理成本压到以美分为单位。 - **呼叫中心录音离线转录**：优先采用离线批处理模式转录长音频，最大限度降低每分钟音频的处理单价。 - **大规模内容离线生成**：批量生成海量营销图片、视频或文本物料。 - **延迟敏感典型**：流式语音助手、实时视频渲染、MoE 复合模型调用。 - **成本敏感典型**：百万级文档提取、呼叫中心离线转录、大规模物料批量生成。 --- ### [05:28](https://www.youtube.com/watch?v=61S2_r3p3mY&t=328s) 实战案例剖析：催收语音智能体的端到端优化（从 1050ms 到 290ms） Amritanshu 分享了一个 Simplismart 近期服务的真实客户案例：该客户正在开发一套用于债务催收的语音智能体（Debt Collection Voice Agent）。语音系统的基本架构由三个级联模型构成：语音转文字（STT）接收音频输入，大语言模型（LLM）处理语义并生成文本，文字转语音（TTS）将回复合成为音频流。对于催收语音助手，核心挑战在于平衡语音交互质量与严苛的运营经济模型——目标是将整体服务成本压低至每分钟 4 美分左右。该客户最初完全采用市面上的第三方通用 Serverless API，其端到端首字延迟（Time to First Byte, TTFB / Time to First Token, TTFT）表现令人沮丧： 1. **STT 阶段**：使用第三方流式 Serverless API，首块音频转录的 TTFB 为 **500 毫秒**。 2. **LLM 阶段**：使用 Serverless Gemma 34B，首 Token 延迟（TTFT）为 **450 毫秒**。 3. **TTS 阶段**：使用主流商业 TTS Serverless API，音频合成首字节耗时 **100 毫秒**。三者相加，端到端首字节延迟高达 **1050 毫秒**（超过 1 秒），在真实通话中表现为明显的机械延迟与卡顿。 ```mermaid flowchart TD accTitle: Voice Agent Pipeline Optimization subgraph Before["优化前通用 API 栈 (总延迟: 1050ms)"] A1["STT (第三方 API) TTFB: 500ms"] --> B1["LLM: Gemma 34B (Serverless API) TTFT: 450ms"] B1 --> C1["TTS (第三方 API) TTFB: 100ms"] end subgraph After["Simplismart 定制优化栈 (总延迟: 290ms)"] A2["STT: 专属 Whisper H100 (1300x RTF) TTFB: 70ms"] --> B2["LLM: Gemma 34B (专用 H100 部署) TTFT: 120ms"] B2 --> C2["TTS (商业 API) TTFB: 100ms"] end ``` 针对这一瓶颈，Simplismart 对整个链路进行了深度重构与定制部署： - **重构 STT（OpenAI Whisper 极致优化）**：Whisper 原生缺乏流式支持，若直接调用 OpenAI 官方 Serverless API，其性能约为 40x RTF（Real-Time Factor，实时倍率）。Simplismart 在单张 NVIDIA H100 GPU 上重构了 Whisper 推理引擎，实现了惊人的 **1300x RTF**（即 1 秒 GPU 算力可转录 1300 秒音频）。经测算，这相当于 **1 美元的算力即可转录整整一个月份（24小时 × 30天）的音频内容**。STT 首字节延迟由 500ms 锐降至 **70ms**。 - **重构 LLM（Gemma 34B 专属部署）**：将 Gemma 34B 模型部署在单张 H100 上，结合底层编译与计算优化，将首 Token 响应时间从 450ms 大幅压低至 **120ms**。 - **保留商业 TTS API**：鉴于目前开源 TTS 模型的声音自然度与表现力相较头部闭源方案仍有差距，TTS 环节继续沿用第三方 API（100ms）。最终，整个语音助手的端到端首字节延迟从 **1050ms 骤降至 290ms**，成功跨入人类无感交谈的 300ms SLA 门槛以内。 - **初始整体延迟**：500ms (STT) + 450ms (Gemma 34B) + 100ms (TTS) = 1050ms。 - **优化后整体延迟**：70ms (STT) + 120ms (Gemma 34B) + 100ms (TTS) = 290ms。 - **Whisper 1300x RTF 经济效益**：单张 H100 支撑超高并行转录，1 美元可处理约 720 小时（1个月连续）音频数据。 --- ### [08:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=525s) 逃离“网格搜索地狱”：SLA 约束下的最优成本决策与复杂推理技术栈在优化 Gemma 34B 的过程中，Simplismart 团队实际上遇到了一个典型的工程决策分支：通过进一步激进调优，他们其实可以将 LLM 的 TTFT 压到 **80ms**（甚至更极限的 60ms），但代价是 80ms 方案所需的算力部署成本是 **120ms** 方案的整整两倍（Double the cost）。团队没有盲目追求纸面极限性能，而是回归业务本质：客户设定的整体端到端 SLA 阈值是 300ms。在 120ms 的 LLM 延迟下，全链路总耗时已达到 290ms，完全满足 SLA 指标。因此团队果断选择了 120ms 的配置，在确保极致交互体验的同时，为客户保住了最优的运营成本。 Amritanshu 指出，这种在多维参数中摸索平衡的过程，正是每位机器学习工程师日常面临的“网格搜索地狱”（Grid Search Hell）。当前的推理技术生态极其碎片化且更迭迅速，工程师必须在繁杂的技术栈中做出无数选择： - **框架与模型后端**：选择哪个推理框架（如 vLLM、TGI、TensorRT-LLM）与计算后端？ - **KV 缓存策略**：长上下文场景下（例如处理 30,000 至 40,000 tokens 的输入 Prompt）采用何种 KV Caching 管理方案？ - **加速算法选择**：当前用例是否适合引入投机解码（Speculative Decoding）？ - **计算模式配置**：是偏向高吞吐批处理（Batching）还是纯流式实时交互？由于不同业务场景的 SLA 要求与成本预算天差地别，不存在任何一套“放之四海而皆准”的默认参数。 - **工程决策案例**：放弃两倍成本的 80ms TTFT，选择性价比更高的 120ms TTFT，使整体 290ms 延迟精准落入 300ms SLA 内。 - **“网格搜索地狱”的变量**：推理框架选型、模型加速后端、KV 缓存算法、超长上下文优化、投机解码、批处理与调度策略。 --- ### [10:49](https://www.youtube.com/watch?v=61S2_r3p3mY&t=649s) 量体裁衣的定制化推理（Tailor-Made Inference）演讲最后，Amritanshu 强调，企业的 AI 产品不应妥协于平庸通用的基础设施，而应当拥有“量体裁衣”（Tailor-made）的定制化推理架构。Simplismart 的核心使命就是帮助开发者终结“网格搜索地狱”，让终端客户能够自由掌控质量、延迟、成本与吞吐量构成的技术光谱——无论是追求 3 美分/百万 tokens 的极低成本批处理，还是追求 1200 tokens/s 的超高性能实时集群，都能通过结构化的方法快速交付。为了帮助企业工程师摆脱盲目试错，Simplismart 开发了一款专门的推理副驾驶工具（Inference Copilot），能根据用户特定的模型、业务用例和 SLA 约束，自动推荐最佳的部署策略与硬件参数配置。Amritanshu 邀请与会者前往 524 号展位体验互动工具与演示，共同探索专属的高性能推理架构。 - **核心主张**：拒绝 Generic Inference（通用推理），拥抱 Tailor-made Inference（定制化推理）。 - **工具支持**：推出推理 Copilot 与配置辅助工具，针对特定工作负载自动生成最佳部署架构。 ## Notable quotes > [01:32](https://www.youtube.com/watch?v=61S2_r3p3mY&t=92s) Do your trade-offs dictate your infra or does your infrastructure actually dictate your trade-offs? — Amritanshu Jain > [01:32](https://www.youtube.com/watch?v=61S2_r3p3mY&t=92s) 到底是你的业务权衡决定了基础设施，还是你的基础设施反过来绑架了你的权衡？ — Amritanshu Jain > [01:32](https://www.youtube.com/watch?v=61S2_r3p3mY&t=92s) 同一个模型在不同架构下展现出跨越数倍成本与数十倍吞吐的光谱，这证明通用型基础设施无法兼顾所有需求，唯有“量体裁衣”（Tailor-made）的推理架构才能实现最优投资回报比。 — Amritanshu Jain > [03:46](https://www.youtube.com/watch?v=61S2_r3p3mY&t=226s) 不同阵营的设计哲学截然不同，强行套用同一种架构必然导致资源浪费或用户体验崩塌。 — Amritanshu Jain > [05:28](https://www.youtube.com/watch?v=61S2_r3p3mY&t=328s) Simplismart 在单张 NVIDIA H100 GPU 上重构了 Whisper 推理引擎，实现了惊人的 1300x RTF，相当于 1 美元的算力即可转录整整一个月份的音频内容。 — Amritanshu Jain > [05:28](https://www.youtube.com/watch?v=61S2_r3p3mY&t=328s) 整个语音助手的端到端首字节延迟从 1050ms 骤降至 290ms，成功跨入人类无感交谈的 300ms SLA 门槛以内。 — Amritanshu Jain > [08:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=525s) 这种在多维参数中摸索平衡的过程，正是每位机器学习工程师日常面临的“网格搜索地狱”（Grid Search Hell）。 — Amritanshu Jain > [08:45](https://www.youtube.com/watch?v=61S2_r3p3mY&t=525s) 由于不同业务场景的 SLA 要求与成本预算天差地别，不存在任何一套“放之四海而皆准”的默认参数。 — Amritanshu Jain > [10:49](https://www.youtube.com/watch?v=61S2_r3p3mY&t=649s) 企业的 AI 产品不应妥协于平庸通用的基础设施，而应当拥有“量体裁衣”（Tailor-made）的定制化推理架构。 — Amritanshu Jain ## Mentioned Amritanshu Jain, Simplismart, Ai4, Oracle Cloud, Red Bull (红牛车队), Llama 3.1 8B, Gemma 34B, OpenAI Whisper, NVIDIA H100, vLLM, TGI (Text Generation Inference), TensorRT-LLM, Inference Copilot ## Video description Amritanshu Jain, CEO & Co-founder, Simplismart Abstracts How can inference systems be designed to balance latency, throughput, and cost across diverse AI workloads? Rigid, monolithic stacks often lead to inefficient trial-and-error. This session introduces a modular framework for architecting SLA-compliant, scalable systems tailored to specific use cases. Gain a structured methodology to make informed trade-offs and deploy performant AI systems at scale. [Source](https://www.youtube.com/watch?v=61S2_r3p3mY)

Source quotes

[1:32] Amritanshu Jain: Do your trade-offs dictate your infra or does your infrastructure actually dictate your trade-offs? [1:32] Amritanshu Jain: 到底是你的业务权衡决定了基础设施，还是你的基础设施反过来绑架了你的权衡？ [1:32] Amritanshu Jain: 同一个模型在不同架构下展现出跨越数倍成本与数十倍吞吐的光谱，这证明通用型基础设施无法兼顾所有需求，唯有“量体裁衣”（Tailor-made）的推理架构才能实现最优投资回报比。 [3:46] Amritanshu Jain: 不同阵营的设计哲学截然不同，强行套用同一种架构必然导致资源浪费或用户体验崩塌。 [5:28] Amritanshu Jain: Simplismart 在单张 NVIDIA H100 GPU 上重构了 Whisper 推理引擎，实现了惊人的 1300x RTF，相当于 1 美元的算力即可转录整整一个月份的音频内容。 [5:28] Amritanshu Jain: 整个语音助手的端到端首字节延迟从 1050ms 骤降至 290ms，成功跨入人类无感交谈的 300ms SLA 门槛以内。 [8:45] Amritanshu Jain: 这种在多维参数中摸索平衡的过程，正是每位机器学习工程师日常面临的“网格搜索地狱”（Grid Search Hell）。 [8:45] Amritanshu Jain: 由于不同业务场景的 SLA 要求与成本预算天差地别，不存在任何一套“放之四海而皆准”的默认参数。 [10:49] Amritanshu Jain: 企业的 AI 产品不应妥协于平庸通用的基础设施，而应当拥有“量体裁衣”（Tailor-made）的定制化推理架构。

Source quotes

[1:30] Amritanshu Jain: Do your trade-offs dictate your infra or does your infrastructure actually dictate your trade-offs? [0:04] Amritanshu Jain: generic infrastructure often fails to meet production demands. [1:30] Amritanshu Jain: Inference optimization revolves around balancing four core metrics: cost, latency, throughput, and accuracy. [5:26] Amritanshu Jain: 1,300 seconds of audio are transcribed in 1 second of H100 compute time [5:26] Amritanshu Jain: an organization can transcribe an entire month of continuous audio (24 hours × 30 days) for just $1. [8:45] Amritanshu Jain: grid search hell [8:45] Amritanshu Jain: running at 80 ms doubled the total serving cost. [8:45] Amritanshu Jain: not generic inference but tailor-made inference

Original transcript

[0:04] Good morning folks. Thanks for coming [0:06] and taking the time today. Uh I'm [0:08] Abdanchu the co-founder of simply smart [0:11] and uh today we are going to talk about [0:14] inference. Uh but just before jumping [0:16] into that a bit about me. So before [0:19] building simply smart I worked as a [0:21] machine learning engineer in multiple [0:23] organizations where I used to manage [0:24] their machine learning infrastructure. [0:27] So I used to optimize their training [0:29] infrastructure, their inference [0:30] infrastructure. I did this for Oracle [0:34] cloud. Interestingly the if someone is [0:36] an F1 fan here, I built the low latency [0:40] uh simulation systems for Red Bull uh [0:42] when I was in Oracle cloud and we used [0:45] to work a ton on natural language models [0:48] like building them, deploying them. Same [0:51] with vision models. Anyway, uh just [0:54] jumping into this uh this talk will be [0:57] relevant for you. If you're using open [1:00] source models in production, if you're [1:03] using an API, a shared endpoint and you [1:06] are like facing any bottlenecks to scale [1:09] or you are just nerding out here and [1:11] it's just day one of the conference and [1:12] you are already tired of the all the [1:14] people. [1:16] So yeah, let's talk about the talk of [1:18] the town AI Infra which everyone is [1:20] going to talk about but specifically in [1:22] AI infra we're going to talk about [1:24] inference because I happen to know a bit [1:27] about it [1:30] right so [1:32] why do we even need to talk about [1:34] inference right when you can just plug a [1:37] serverless endpoint into your [1:38] application and go about it what's [1:41] what's there to talk about [1:44] so let me ask you this do your [1:47] trade-offs dictate your infra or does [1:50] your infrastructure actually dictate [1:52] your trade-offs right [1:57] [Music] [1:58] if you're using serverless APIs we have [2:00] generally seen the former is true and [2:04] generally people face issues in scaling [2:08] or like bottlenecks in scaling they're [2:11] not able to essentially have an optimal [2:14] deployment point for the kind of [2:16] application they're building. Basically, [2:18] tailor made inference. We're going to [2:20] talk more about tailor made inference. [2:22] But before that, what actually is a [2:24] trade-off? [2:26] So, a trade-off can be any of the four [2:28] metrics. Cost, latency, throughput, or [2:32] accuracy? How cheap do you want this API [2:35] to be? What at what latency do you want [2:38] this API to be? How much throughput do [2:40] you want from this API? Or how accurate [2:42] you want your model to be? [2:45] So just to kind of put things into [2:48] perspective what this actually means [2:49] from an example is let's say you want to [2:52] take a llama 3.18 billion and deploy it [2:55] for let's say a chatbot that you're [2:56] building right now you can use it for [3:00] like a chatbot you can use it for [3:01] document processing you can use it for [3:03] online uh real-time use cases or you can [3:05] also use it for offline transcript like [3:08] offline bad jobs right so this is the [3:11] spectrum that generally we for a llama [3:14] 3.18 billion that you can serve a llama [3:18] 3.18 billion model at 50 tokens per [3:20] second for 3 cents a million tokens or [3:24] you can serve a llama 3.18 billion at [3:26] 1200 tokens per second for 20 cents per [3:29] million tokens. This is the exact [3:31] trade-off that we're talking about. You [3:33] can you can deploy it for 3 cents per [3:36] million tokens. You can deploy it for 20 [3:38] cents per million tokens. But what you [3:40] will be compromising here is the latency [3:42] or the throughput of the model that you [3:43] get from it. [3:46] So talking a bit more about what are the [3:50] some what are some of the latency [3:51] sensitive use cases what are some of the [3:53] costs sensitive use cases that we have [3:54] seen we can generally either optimize [3:58] for one particular trade-offs of the [4:00] four that we talked about right so some [4:02] of the latency sensitive use cases like [4:04] voice agents right where we're going to [4:07] talk about voice agents with an example [4:09] but like with voice agents the idea is [4:11] that you need to be really really quick [4:14] really really fast and getting the [4:16] response to the human ear when the voice [4:18] agent is responding [4:21] or if you are building diffusion [4:22] pipelines let's say you are like [4:24] generating real-time content or videos [4:27] like companies like Nvidia they they [4:29] want their pipelines to be faster in [4:31] order to get enhance your user [4:34] experience as to say raising models of [4:36] course because there are a ton of calls [4:39] there are so many mixture of experts [4:43] working at once Then some costs [4:45] sensitive use cases. Let's say if you [4:47] are parsing a million documents a day, [4:50] you do not care about latency, right? [4:51] What you care about is for how many [4:54] cents can I pass this documentation. If [4:56] you're [4:58] if you're building something for contact [5:00] centers, if you're building something [5:01] that is transcription heavy, you would [5:03] put generally prefer it to kind of do [5:06] offline transcription in order to [5:08] optimize your cost per minute of audio [5:10] transcribe. And of course, content [5:12] generation [5:14] is uh one of the cases where you would [5:17] want to generate tons of content whether [5:20] it be like image content, video content [5:23] or even text content. [5:26] So let's consider an example. So this is [5:29] one of the customers we recently worked [5:30] with in the past and they were building [5:33] this debt collection voice agent. U they [5:36] were using of course the usual suspects [5:38] to build a voice agent speechtoext model [5:41] combined with an LLM and then a [5:43] texttospech model. Basically every model [5:45] is streaming in this case. And the key [5:47] metrics that we want to optimize for [5:49] here are latency and cost. Right? Again [5:53] because when you're building a voice [5:55] agent there are two major things that [5:58] you generally look at. One is the [6:00] quality of the voice delivered by an [6:02] agent and the second thing is for how [6:05] much for like can you [6:08] can you build a voice agent that can [6:10] give you the economics of 4 cents on [6:12] dollar per minute. Right? [6:16] So this is how their initial stack [6:18] looked like. For speechtoext they were [6:21] using a third party serverless API. They [6:24] were using Gemma 34 billion as their LLM [6:27] and for text to speech they were using a [6:29] serverless API from a company which most [6:32] of the people take their serverless APIs [6:34] from. And this is what their end to-end [6:37] latency looked like for time to first [6:39] bite. basically to transcribe the first [6:42] chunk of the audio that they received [6:44] through this speechto text streaming API [6:46] it was 500 milliseconds for a gemma 34 [6:50] billion they were again using a a [6:52] serverless API API which was giving them [6:54] a TTF of 450 mconds and for TTS the time [6:59] to first bite was 100 milliseconds [7:02] this is what we did so we were able to [7:06] optimize this speechto text streaming [7:08] and get a time to first bite to 70 [7:12] milliseconds. For Gemma 34 billion, we [7:14] got the time to first token to 120 [7:17] milliseconds. We actually got it to 60 [7:19] mconds, but we'll talk about why we [7:21] actually compromised on 120 mcond TTFT. [7:24] And then for TTS because they were still [7:26] using serverless APIs generally because [7:29] open source TTS haven't been that [7:31] successful as compared to the [7:33] proprietary ones and basically their end [7:36] to-end latency came to 290 milliseconds [7:39] down from 1 down from 1,50 millconds to [7:42] 290 milliseconds. [7:44] So how did we help? So for speech to [7:47] text we deployed openAI whisper but like [7:51] a couple of problems if you have ever [7:53] tried to use whisper are it does not [7:56] have native streaming support and if you [7:58] use the openi serverless API it'll give [8:00] you 40x RTF. What is RTF? Right. So RTF [8:04] means if if I'm saying that we can do [8:06] 1300x RTF on one H100 that means I can [8:09] transcribe 1300 seconds of audio in 1 [8:12] second of GPU time in 1 second of H100 [8:15] time that essentially translates to you [8:18] can do the math but uh that essentially [8:20] translates to that for a dollar you can [8:23] actually transcribe a month of audio [8:26] that's 24 into 30 hours of audio [8:28] transcribed in just $1. And then for LLM [8:31] we took GEMA 34 billion model deployed [8:34] on NH100 [8:36] and we were able to finally get the end [8:39] toend latency for 300 milliseconds. [8:45] So when we were deploying Gemma 34 [8:48] billion for this company uh we also were [8:52] standing at an intersection of this [8:55] trade-off that we are talking about. So [8:58] at for one configuration we were able to [9:01] achieve a latency of 80 millconds. For [9:03] the other one we were able to achieve a [9:06] latency of 120 mconds. Sorry for the [9:09] typo here but the latency was 120 mconds [9:12] here and the other one was 80 mconds. [9:14] The difference was for the 80 mcond [9:17] latency the the cost that they were [9:20] getting it was double of the cost that [9:22] they got from the latency of 120 mconds. [9:26] Now the idea is because their SLA was [9:28] within 300 millconds, we did not want to [9:31] go with the fastest one because we [9:33] wanted to optimize for cost as well. So [9:35] we went for the latter one where the [9:37] latency was still within the SLAs's but [9:40] provided like a better cost option for [9:43] them. And this is exactly what every [9:46] machine learning engineer goes through [9:48] when deploying a model, right? So we [9:51] call this grid search hell in our [9:52] language. But the idea is for every [9:54] model, for every use case, the [9:57] deployment has to be really really [9:58] tailor made. You want to optimize for [10:02] your SLAs that your customers want. You [10:04] want to manage the cost that your [10:06] customers would see beneficial in taking [10:09] the product. [10:11] And whenever you're deploying these [10:12] models there like so many options [10:15] there's a new inference technique every [10:17] day in town and you have to select okay [10:20] which framework do I choose? which model [10:22] backend do I choose? How do I choose [10:24] which KV caching technique do I want? [10:26] Right? Do I have like a long context [10:29] problem with my pro u LLM? Which [10:31] basically means am I giving them giving [10:34] it an input prompt of like 30 or 40,000 [10:37] tokens? Am I using it for a real-time [10:39] use case? Do I want to get into [10:42] speculative decoding? Am I running bash [10:44] use case? Blah blah blah. [10:47] So [10:49] we basically help you navigate this grid [10:52] search shell through tailor made [10:54] inference. This is exactly what tailor [10:56] made inference looks like right? If what [10:59] do you want to optimize for? Is it [11:01] quality? Is it latency? Is it cost? Is [11:03] it throughput? And what are whatever [11:06] trade-offs you want to wherever in the [11:08] spectrum you want to lie into the [11:10] spectrum that we talked about how we [11:12] deployed llama 3.18 billion at 3 cents [11:14] per million tokens at 50 tokens per [11:16] second. But we also were able to deploy [11:19] it for 20 cents per million tokens at [11:22] 1200 tokens per second. This is the [11:24] exact spectrum that needs to be [11:26] controlled by the end customer that [11:29] needs to be controlled by you in order [11:31] to build the most optimal deployment in [11:33] order to build the most optimal [11:35] application for your customers. [11:39] So that's why we think your product [11:41] deserves not generic inference but [11:44] tailor made inference. But honestly I [11:47] can go on and on about how we have [11:50] solved for like scaling rapid [11:52] autoscaling. How we have solved for [11:54] different companies for different use [11:56] cases. How do we optimize their models [11:59] but because we are so much obsessed [12:01] about grid search hell and tailor made [12:03] inference. We have actually planned a [12:05] few games around that and we have [12:08] planned like a co-pilot which will help [12:10] you determine what is the best [12:12] deployment strategy if you're deploying [12:13] a model for a particular use case. So, [12:15] please give us a visit at booth 524. [12:18] Thank you so much, folks.