Asurion Routes Enterprise AI Across Clouds to Avert Outages, Cap Costs

Asurion跨云路由企业级AI以规避中断并控制成本 企业级生成式AI部署面临灾难性预算超支与区域性服务中断的风险，除非团队执行无缝并发测试，并在竞争对手云服务商之间动态路由推理流量。 基于Asurion服务逾3亿用户的实践经验，演讲者提出了一份将企业级生成式AI超越基础原型、实现规模化扩展的蓝图，其依托三大基石：知识检索、模型服务与持续评估。他们重点阐述了在部署前如何通过合成量化负载测试，揭示按需付费与预置云容量之间在经济成本与SLA层面的权衡。为保障业务韧性，该架构采用了具备自动化多云故障转移功能的统一模型抽象层，并结合双目标摄取流水线，将实时语义搜索与下游使用分析相互分离。该框架还通过配置驱动的评估抽样完成整个生命周期闭环，在不产生难以承受的计算成本的前提下，平衡常态化基线监控与有针对性的限时审查。 企业级GenAI的成功取决于坚实成熟的数据平台与多云互操作性，而非将底层数据管理视作事后考虑。 (1:35) 彻底的架构抽象能够将客户端应用程序与云厂商更迭、API模式差异以及后端基础设施混乱隔离开来。 (1:35) Asurion将其大规模生成式AI平台构建于三大核心支柱之上：知识表示、模型托管与横向评估。 (5:11) 在投产部署前必须进行量化负载测试，以便为非确定性系统引入可预测性，并建立切合实际的P50与P95延迟基线。 (7:14) 严格的基准测试为不可预测的按需付费延迟峰值与昂贵的前期预置吞吐量承诺之间的关键经济权衡提供依据。 (11:12)

Asurion engineering leaders Rajesh Gundugollu and Ansh Gandhi assert that scaling enterprise generative AI requires applying mature data engineering principles, decoupling model providers via radical abstraction, and continuously benchmarking infrastructure cost against strict latency and quality thresholds. Quantitative Benchmarking and Infrastructure Economics (7:14) To introduce predictability into non-deterministic large language models, Asurion built an internal load-testing harness that parameterizes target requests per second (RPS), duration, and concurrency caps using standardized 10:1 input-to-output token datasets (ranging from 1,000/100 to 4,000/400 tokens). Benchmarking averages three runs across escalating load tiers to smooth network jitter and vendor queue fluctuations. The harness enforces continuous, gap-free concurrency, eliminating the brief recovery intervals found in standard open-source tools that can skew latency readings and hide endpoint breaking points. Gundugollu highlights that quantitative load testing dictates the financial and architectural trade-off between pay-as-you-go endpoints and provisioned throughput. While pay-as-you-go carries lower baseline commitments, it lacks latency SLAs and suffers from multi-tenant traffic spikes. Conversely, provisioned capacity provides guaranteed throughput and latency profiles, but requires capital commitments often reaching $100,000 regardless of utilization. Synthetic stress-testing allows teams to empirically verify whether transaction volumes justify provisioned capacity before committing capital. Decoupled Abstraction and Resilient Traffic Routing (15:58) Asurion isolates client microservices from vendor SDKs and breaking schema changes by routing all inference calls through a centralized model abstraction layer. A real-time routing path (Workflow A) consults tenant metadata to enforce authorization and quota isolation, preventing internal applications from exhausting shared capacity. It simultaneously checks active endpoint usage across providers and regions to avoid HTTP 429 rate-limit errors. In parallel, a background health-check loop (Workflow B) runs synthetic probes against external endpoints, instantly updating model metadata to bypass failing providers. Because cloud provider cross-region failovers can still exhaust account-level quotas at enterprise scale, Asurion implemented cross-cloud routing to divert overflow traffic from Amazon Bedrock directly to alternate providers like Anthropic Claude or Azure OpenAI. The routing engine also differentiates between real-time interactive traffic and background asynchronous tasks, routing eligible workloads to batch inference endpoints to capture up to 50% in infrastructure savings. Dual-Destination Ingestion and Retrieval Architecture (21:59) To augment models with enterprise knowledge without degrading retrieval speeds, Asurion's ingestion pipeline separates semantic search execution from analytical processing. Inbound document changes from content management systems pass through a centralized processor that handles keyword extraction, metadata enrichment, and vector embedding generation. The processed artifacts are then dispatched to two distinct endpoints: a semantic search engine optimized for low-latency similarity queries, and an isolated data platform for heavy analytics. This separation prevents deep analytical queries from degrading live customer interactions while allowing teams to measure document utility. The analytics engine tracks precise context contribution, determining whether a retrieved document accounted for 50% or 5% of an LLM's final response and identifying unhelpful source references. The retrieval lifecycle also incorporates pre-launch indexing benchmarks (such as NMSLIB versus Apache Lucene) and ongoing post-launch monitoring to detect approximate nearest neighbor (ANN) retrieval drift. Config-Driven Sampling and Lifecycle Quality Monitoring (30:58) While pre-launch testing focuses strictly on operational throughput and latency, post-launch evaluations assess qualitative correctness, factual grounding, and sentiment. Because evaluating every interaction across millions of daily sessions is computationally and financially unfeasible, Asurion employs an event sampler that inspects a 5% to 10% baseline sample. To avoid excessive overhead during multi-turn chats, the sampler evaluates only terminal lifecycle events, such as completed customer support sessions. Asurion balances continuous, long-lived baseline sampling with short-lived, targeted evaluation windows. When engineering teams deploy deliberate changes—such as prompt revisions—the platform temporarily increases sampling to a 100% capture rate for 24 to 72 hours. These events pass through automated LLM-as-a-judge scoring pipelines and an event distributor into a unified analytics platform, allowing engineers to validate feature updates before the sampler reverts to its lower-cost baseline rate. Gundugollu and Gandhi emphasize that scalable enterprise AI relies on treating security, PII redaction, and role-based access control as foundational architecture rather than post-launch additions, enabling platforms to maintain compliance and cost efficiency as transaction volumes expand.

Asurion工程团队指出，将企业生成式AI从原型推向服务数亿用户的规模化生产，必须摒弃追逐新技术的盲目性，通过多云抽象层、严苛的离线定量压测、检索与分析双轨分离以及基于生命周期事件的动态采样，在非确定性的模型行为中建立可控的成本与工程确定性。 多云基线与抽象层隔离：在数据基座上封装混沌 (1:35) Asurion目前在15个国家服务超过3亿用户，其技术支持覆盖全美88%以上人口。面对如此庞大的业务体量，工程总监Rajesh Gundugollu强调，生成式AI平台必须建立在扎实的数据工程基座之上。由于前沿模型能力割裂在不同云厂商生态中——微软Azure托管OpenAI、谷歌云托管Gemini、亚马逊AWS托管Claude与Bedrock——多云架构已成为企业不可避免的基础底线，企业不应陷入单一云厂商的技术选型争论。 为了防止底层模型与云基础设施的频繁变动影响上层应用，Asurion设计了统一的抽象层。Rajesh将这一抽象层比作建筑工程中的“干式墙”（drywall），核心目标是将底层模型调用模式的差异、供应商API变更以及基础设施故障彻底隔离在墙后。下游微服务只需对接内部统一的数据契约，从而避免外部第三方接口的调整引发内部系统的连锁故障。 定量压测与算力经济学：厘清成本与时延平衡点 (7:14) 针对大语言模型输出不确定性与公有云排队机制不透明的痛点，资深软件工程师Ansh Gandhi介绍了团队自研的定量基准测试框架。该框架利用标准化的10:1输入输出Token比例数据集（覆盖从1,000输入/100输出到4,000输入/400输出），在受控环境下对第三方API及自托管模型进行压力测试。每次测试取三次运行的平均值，以平滑公网抖动与云端排队波动，绘制P50和P95时延曲线。 这项测试直接决定了企业的基建预算路径。按需付费（Paygo）模式缺乏时延保障且面临严重的并发排队毛刺，而预留吞吐量（Provisioned Throughput）虽然能锁定SLA，却往往需要预先承诺高达10万美元左右的固定资金。通过量化压测，工程团队可以用数据证明何种业务规模适合按需调用，何种关键业务必须采购预留算力。此外，该框架采用“持续并发”（consistent concurrency）机制，消除开源测试工具中因微小停顿让服务端排队得以缓冲的缺陷，逼出端点的真实崩溃极限。 双工作流模型路由：动态容灾与推理成本分流 (15:58) 在模型托管层面，平台通过两条协同工作流保障高可用性。工作流A负责实时的在线推理路由，在请求进入时结合模型元数据与实时使用指标进行评估：模型元数据负责隔离租户权限与配额，防止内部不同应用相互挤占容量；实时指标则监控各区域端点的并发饱和度，避免触发HTTP 429限流错误。一旦主用预留节点饱和或故障，系统会自动将流量平滑引流至其他区域或备用云厂商的按需实例。 工作流B则是完全独立的后台健康巡检循环，持续向各个模型端点发送探测请求。当监控报警触发时，工作流B即时更新注册中心的健康状态，使工作流A能以零额外时延旁路绕过不健康节点。在运营策略上，团队严格区分实时交互与异步任务，将总结等非即时任务路由至批处理端点，从而节省约50%的推理基础设施成本。 知识检索双轨架构：解耦语义搜索与利用率分析 (21:59) 在知识检索增强（RAG）管道中，Asurion采用增量捕获机制提取内容管理系统的新增与修订文档，完成文本预处理、关键词提取和向量嵌入。平台架构的一大核心突破在于将处理后的数据分流至两个独立目标：面向毫秒级低时延推理的语义向量检索引擎，以及完全物理隔离的分析数据平台。这种设计避免了复杂的分析统计查询直接冲击在线向量库，进而拖垮线上客服交互的响应时延。 独立的分析平台能够精准量化知识的实际利用率：评估被召回的文档最终是在生成答案时提供了有效支撑，还是被模型过滤丢弃；同时精确分析单篇文档在最终上下文中的贡献权重（例如是贡献了50%的上下文还是仅有5%），帮助优化知识库信噪比。在扩展性方面，当AWS Bedrock发生跨区域配额耗尽时，系统支持跨云无缝切换至Anthropic原生接口，并结合带有20%缓冲余量的限流阈值，确保海量并发下的系统弹性。 配置驱动的事件采样：化解全量质检与安全开销 (30:58) 面对日常数百万次的客服交互，全量运行基于大模型（LLM-as-a-Judge）或人工的质量审查在算力与财务成本上无法维系。团队设计了基于配置的动态事件采样器，将常态化检测流量设定在5%至10%。更为关键的是，系统不再对多轮对话的每一次往返进行碎片化评估，而是监听会话终结事件（如`chat complete`），仅对完整闭环的服务会话进行成套评分，涵盖亲和度、问题解决率、客户情绪及知识库引用有效性等维度。 质检机制进一步划分为“长期持续评估”与“短期高频评估”。前者以5%至10%的低采样率在后台长期运行，监测基线漂移；后者则用于系统发生重大变更（如提示词修改）之后，在发布后的24至72小时内开启100%全量捕获，以便快速验证调整效果，并在窗口期结束后自动回落。全流程贯穿端到端的数据脱敏（PII屏蔽）、运行时防注入护栏与细粒度基于角色的访问控制（RBAC），确保合规审计透明可溯。 Asurion的实践表明，企业生成式AI平台的真正壁垒不在于调用尖端模型的能力，而在于围绕成本、时延和容灾所构建的系统工程确定性。然而，供应商底层的静默升级（如模型微调或参数漂移）以及近似最近邻（ANN）检索固有的语义漂移，依然是企业在实现长期生产稳定性时必须持续监控与校准的深层挑战。

## Background 本内容由 YouTube 频道 Ai4 于 2025 年 8 月 28 日发布。视频记录了 Asurion 的数据与机器学习工程总监 Rajesh Gundugollu 与高级软件工程师 Ansh Gandhi 的技术演讲，重点分享了如何在企业级规模下构建支撑每日数百万次 AI 交互的生成式 AI（GenAI）平台，涵盖知识平台、模型托管与评测体系三大支柱。 ## TL;DR Asurion 团队系统阐述了超越“Hello World”玩具级项目、实现工业级 GenAI 平台落地的完整工程架构。分享聚焦于三大核心子系统：通过双重目的地实现检索与分析解耦的知识表征（RAG）体系、具备多云智能路由与高可用容灾的模型抽象托管层，以及贯穿全链路的数量负载压测与基于事件采样的低成本质量评估（LLM-as-a-Judge）机制。整套方案在严守 PII 脱敏与安全护栏的前提下，有效平衡了企业级高并发场景下的延迟 SLA、运营成本与生成质量。 ## Atmosphere & style 演讲整体呈现出扎实硬核、务实接地气的工业级工程风格。两位主讲人配合默契，Rajesh 擅长从宏观业务视角剖析成本算账与架构决策的残酷现实，Ansh 则深入微观技术细节拆解压测参数与路由状态机；台上没有任何空洞的概念炒作，处处透露着从数百万真实流量踩坑中提炼出的实战清醒感。 ## Key takeaways - [01:35](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=95s) 多云架构已成为规模化 GenAI 的新常态，企业 AI 系统的成功高度依赖于成熟的数据工程地基与实时分析能力。 - [04:37](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=277s) 设计坚固的抽象层（Abstraction Layer）是核心原则，它能抹平多云供应商协议差异，将底层的技术演进与重构混乱与上游业务彻底隔离。 - [04:37](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=277s) 规模化 GenAI 平台由三大支柱构成：知识表示与检索（RAG）、模型托管（Model Hosting）以及横向贯穿全局的评测体系（Evaluations）。 - [07:14](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=434s) 数量评测体系通过标准化的“10:1 Token 黄金法则”与阶梯 RPS 压力测试，消除 LLM 调用的非确定性并建立 P95 延迟可预测基线。 - [14:11](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=851s) 通过一致性并发（Consistent Concurrency）压测消除传统工具的请求间隙，在上线前主动探明系统端点的极限崩溃点。 - [11:12](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=672s) 基于压测数据理性权衡按量付费（Pay-as-you-go）与预置吞吐量端点（Provisioned Throughput），在延迟 SLA 保证与高昂成本之间找到盈亏平衡点。 - [15:58](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=958s) 模型抽象层结合模型元数据与实时使用指标，实现业务配额隔离、单点水位监控以及跨多云供应商的无缝智能路由与故障降级。 - [21:59](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=1319s) 知识表征流水线采用“双重目的地（Dual Destinations）”设计，将处理数据同时写入语义搜索向量库与分析数仓，在不干扰线上检索的前提下深度量化知识利用率与贡献度。 - [30:58](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=1858s) 生产环境质量评估引入事件采样器（日常 5%-10% 采样率），精准捕获 chat complete 等关键终态事件，结合 LLM-as-a-Judge 解决全量质检的高昂成本瓶颈。 - [33:26](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=2006s) 采用“长期基线采样”与“发布后短期 100% 全量质检”相结合的配置驱动评估模式，既能长期把握质量脉搏，又能在 Prompt 调优等变更发布后密集排查风险。 - [35:57](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=2157s) 将安全视作一等公民，在全链路中深度集成上游 PII 自动脱敏、端到端模型安全护栏（Guardrails）与细粒度 RBAC 访问权限治理。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=4s) 演讲开场与 Asurion 业务规模概况 来自 Asurion 的数据与机器学习工程总监 Rajesh Gundugollu 与高级软件工程师 Ansh Gandhi 共同带来了主题为“工程化构建规模化 AI 平台”（Engineering AI Platforms for Scale）的演讲分享。Rajesh 开门见山地阐明了当天的议程框架：团队究竟是谁、做了什么、如何落地的、从中汲取了哪些深刻经验，以及参会者可以带走哪些可复用的实践方案，并预留时间进行互动讨论。 作为背景，Asurion 自定义为一家“技术关怀”（tech care）公司，其服务网络覆盖全球超过 3 亿用户。公司的服务宗旨通俗而直接——为“任何带有电源按钮的设备”提供保障和保险方案。从智能手机到各类家用电器，团队致力于无论何时何地、以何种形式为客户及其拥有的科技产品提供全方位支持。 - 业务覆盖全球 15 个国家，拥有逾 18,000 名员工。 - 超过 9,000 名专业技术专家通过虚拟线上、店内服务以及移动上门等多种途径响应客户需求。 - 整体服务网络触达全美超过 88% 的人口。 --- ### [01:35](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=95s) 回顾 2024 年基石原则：数据地基与多云生态 在深入当下的技术细节之前，Rajesh 特意回顾了他们在 2024 年同个大会上分享的核心基石原则。他敏锐地指出，在生成式 AI 热潮席卷的当下，业界往往急于追逐每一个“光鲜亮丽的新事物”（the next shiny thing），而忽略了最为根本的工程地基。AI 系统的成功在很大程度上取决于此前数年甚至数十年沉淀的数据工程能力。 ```mermaid flowchart TD accTitle: 规模化 AI 平台的五大基础原则 subgraph DataFoundation["数据基石与基础设施"] A["高可靠数据平台\n准实时分析与事件处理"] --> B["多云战略 (Multicloud)\nAzure / GCP / AWS 各取所长"] end subgraph GovernanceAbstraction["治理与架构隔离"] C["借鉴成熟数据治理\n复用数据工程经典准则"] --> D["安全与全链路可观测性\n作为一等公民内置"] D --> E["健壮抽象层 (Abstraction Layer)\n将工程混乱隐藏在石膏板后"] end DataFoundation --> GovernanceAbstraction ``` 第一个原则是大力投资健壮且高可靠的数据平台，特别是具备近实时（near real-time）能力的数据分析基础设施，因为海量 AI 业务交互每天都会产生天文数字级的事件和指标，必须依赖分析平台快速识别系统趋势与异常；第二个原则是毫不犹豫地拥抱多云架构（Multicloud），Rajesh 认为技术选型已不再是“AWS 还是 GCP”的单选题，OpenAI 依托 Azure，Gemini 扎根 GCP，而 AWS 同样拥有繁荣的模型生态，“多云已成为新常态”（multicloud is the new norm）；第三，直接将成熟的数据管理与治理经验平移到大语言模型（LLM）领域，无需从零等待生态规范成熟；第四，将安全合规与可观测性视作“一等公民”；第五则是重中之重的“抽象层设计”。 --- ### [04:37](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=277s) 抽象层与规模化挑战：三系统架构的确立 在强调抽象层的重要性时，Rajesh 打了一个极其形象的比喻：抽象层就像是“将所有工程混乱隐藏在一面石膏板之后”（hiding all your chaos behind a drywall）。AI 生态系统与底层模型日新月异、频繁迭代，如果不借助抽象层实施有效的解耦，工程师在后端所经历的重构与技术演进风暴就会不断冲击并破坏所有下游业务系统。 过去一年中，Asurion 将这些基石落地演进，应对真正意义上的工业级高并发规模：系统每天不仅要处理数百万次交互事务，还要支撑数百万次面向真实客户的客服聊天（chat）与电话语音（call）交互。在承担如此大规模流量的同时，团队必须严格守住成本、安全与合规等各项硬性红线。为此，他们将整个 AI 平台精炼为三大核心子系统： | 核心子系统 | 职责与技术范畴 | 关键技术组件 | |:--- |:--- |:--- | | **知识表示与检索**<br>*(Knowledge Representation & Retrieval)* | 负责非结构化与企业知识的处理、索引与上下文增强 | 向量数据库（Vector DB）、RAG 增强检索链路、数据清洗管道 | | **模型托管**<br>*(Model Hosting)* | 统一纳管闭源商用大模型、开源自研模型及功能型模型 | Claude/Anthropic、OpenAI、Gemini、开源 Llama/Qwen、Embedding 与 Re-ranker 模型、SageMaker 端点 | | **评测体系**<br>*(Evaluations)* | 贯穿全周期的系统质量与数量双维度校验 | 数量评测（吞吐量、延迟、并发压力测试）与质量评测（准确性、幻觉、业务指标达成） | 尽管在架构列表上“评测”常被排在最后，但团队在整理实践时发现，数量评测（Quantity Evaluations）绝非像模型托管或知识库那样孤立的垂直模块，而是一条横向贯穿整个架构全局的生命线（horizontal layer），因此整个技术拆解决定打破常规，由 Ansh Gandhi 率先从“数量评测”展开深度剖析。 --- ### [07:14](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=434s) 数量评估体系：构建确定性与负载压力测试 Ansh Gandhi 接过话茬，向全场介绍了评估体系中的关键维度——数量评测。由于大语言模型具备天然的非确定性（nondeterministic），且系统交互链路中充斥着各类未知变量，数量评测的首要目标就是打破黑盒，“让系统表现变得高度可预测”。这涵盖了针对外部商业 API 端点、自建 SageMaker 端点、嵌入（embedding）模型乃至重排序模型的全方位压力测试。 在执行压力测试时，平台定义了一套标准化的输入参数控制矩阵：首先是预期业务负载的每秒请求数（RPS，例如从 10 RPS 到 50 RPS 递增）；其次是测试持续时长（几秒至几分钟）；接着是并发参数设置，该参数对于调用商用第三方 API 尤为重要，以避免在压测自身系统的同时意外打瘫依赖服务；再者是目标端点选择，跨越 Bedrock、Gemini、SageMaker 开源端点等不同环境；最后是精细化的推理参数，如 Temperature 和最大输出 Token 数。 ```mermaid flowchart LR accTitle: 负载测试与指标分析流程 subgraph Config["测试配置输入"] A["目标端点选择\n(Bedrock / Gemini / 开源模型)"] B["负载参数设定\n(RPS 10-50 / 持续时间 / 并发)"] C["测试数据集设定\n(10:1 黄金比例样例集)"] end subgraph Execution["测试执行与平滑计算"] D["基准压测运行\n(单用例连续 3 次压测)"] E["网络波动剔除\n(计算 P95 均值)"] end subgraph Output["预测模型输出"] F["P95 / P50 延迟曲线图"] G["生产成本与容量基线"] end Config --> Execution Execution --> Output ``` 为了应对各家供应商截然不同的请求/响应协议格式，并降低各业务团队的测试门槛，平台建立了一套基准测试数据体系。他们提出了独特的“10:1 Token 黄金法则”，即每 10 个输入 Token 对应 1 个输出 Token，并以此构建阶梯样例集（从 1000 输入/100 输出一路扩展至 4000 输入/400 输出）。如果没有私有数据集，团队可直接使用基准集；若有特定场景，亦可传入定制数据。测试结果会在平台内部统一呈现为直观的可视化图表，横轴为 RPS 阶梯，纵轴展示测得的 P50 及 P95 延迟曲线。为了消除公网调用中的偶发网络延迟与服务抖动，平台在每一档位均执行三次测试并对 P95 延迟取平均值，从而为模型上线后的生产表现建立坚实的可预测基线。 --- ### [11:12](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=672s) 商业视角：成本预测、SLA 保证与预置端点决策 Rajesh 在此处插话，深入阐述了研发这套数量评测体系背后的商业痛点与残酷现实。在许多企业内部，高层领导往往会极具热情地号召“全面拥抱 AI”（let's go full AI），但当工程团队拿着方案汇报“这个系统跑起来一年需要耗资 1000 万美元”时，所有人都会瞬间被拉回现实。 现实的矛盾在于 SLA（服务等级协议）与成本的权衡： - **按量付费端点（Pay-as-you-go）：** 像 OpenAI 等公有端点虽然承诺服务可用性（uptime SLA），但绝不承诺延迟 SLA。因为你的流量正在与全球其他海量用户共享资源池，一旦出现流量拥堵，延迟就会急剧飙升。 - **预置吞吐量端点（Provisioned Throughput）：** 如果要在前台为终端用户提供响应敏捷的实时对话机器人，就必须给出毫秒或极短秒级的延迟保障。这意味着企业必须提前包下专用端点，即便毫无流量，每个月也可能要为预置实例预先支付 10 万美元的真金白银。 面对巨额支出，工程团队不能凭感觉猜测，必须依靠数据说话。该评测框架的核心价值，就是让团队在极短周期内跑通多组不同 Token 规模与 RPS 组合，清晰比对按量付费与预置端点的盈亏平衡点，做出经得起推敲的理性架构决策。现场有观众提问：“你们在压测过程中难道不会被供应商计费收取高额费用吗？”Rajesh 与 Ansh 坦诚回应：压测确实会产生费用，但短则 5 分钟的高压基准测试所耗费的微小成本，比起盲目上线后在数小时内烧掉数万美元、或遭遇生产事故的巨大代价，完全是一笔极其划算的必要投资。 --- ### [14:11](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=851s) 压测一致性并发与模型托管抽象层初探 在谈及主流开源压测工具时，Ansh 指出市场上很多框架存在致命缺陷：在声称以恒定速率（如 4 RPS）进行极限压力测试时，实际监控时间轴会发现请求之间存在隐蔽的空隙（pockets），导致端点获得了非预期的短暂喘息机会。这些看似微不足道的误差在规模化大并发场景下会被无限放大，严重扭曲延迟评估与成本测算。 因此，团队构建了“一致性并发”（Consistent Concurrency）压测机制，确保端点始终承受无间隙的持续极限高压，不留丝毫喘息空间，其核心逻辑在于“在上线前主动把端点打爆，提前探明系统的极限崩溃点，绝不能把意外留给生产环境”。 随后，Ansh 将焦点转移至模型托管（Model Hosting）架构。整套架构被清晰地拆解为两大协同工作流： - **A 工作流（推理请求路径）：** 位于架构图左侧，处理实时进入的推理请求。 - **B 工作流（后端管控路径）：** 位于架构图右上侧，处理背后的配置与动态调度。 当外部推理请求涌入时，首先抵达核心抽象层（Abstraction Layer）。这一抽象层肩负着三大核心使命：首当其冲的是消除供应商锁定与格式碎片化，将各类模型提供商五花八门、各不相同的请求/响应协议格式在入口处完全抹平规范化，从而向上层业务屏蔽一切底层异构性。 ### [15:58](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=958s) 模型抽象层与智能路由：高可用与容灾闭环 为了保证系统在企业级规模下的高可用与容错能力，模型抽象层在处理推理请求时依赖两大核心支柱：模型元数据（Model Metadata, A2）与模型使用指标（Model Usage Metrics, A3）。当外部应用发起请求时，抽象层首先校验元数据，确认该应用是否具备访问目标模型的合法权限以及模型当前的健康状态；紧接着通过使用指标评估当前的流量水位，避免单点过载。例如，如果某个模型的限流上限是每秒 100 次请求（100 RPS），系统必须能够实时判断当前到来的请求是否是导致超限的第 101 个请求，并在必要时触发故障转移（Failover）。 元数据与使用指标在设计定位上各有侧重：模型元数据主要服务于应用层面的配额隔离与治理，确保同一底层模型端点（如 Amazon Bedrock）被多个业务应用同时调用时，彼此不会相互争抢资源或“踩到对方的脚趾”；而模型使用指标则专注于底层端点本身的实时健康状况与承载力，例如监控 Anthropic 在 `us-east-1` 区域或 OpenAI 在加拿大区域的可用性。抽象层据此在运行时实现动态智能路由：若预置吞吐量端点（Provisioned Throughput）达到上限或发生异常，请求可无缝降级至同服务商或跨多云环境的按量付费端点（Pay-as-you-go）。此外，独立的健康监控工作流（Workflow B）会持续轮询各端点并触发健康告警，实时将故障状态反馈回元数据中心，构建起自动切流与故障自愈的闭环体系。 ```mermaid flowchart TD accTitle: 模型抽象层智能路由与健康反馈架构 subgraph ClientLayer["客户端请求"] App["应用服务请求"] end subgraph AbstractionLayer["模型抽象层"] Router{"智能路由引擎"} Meta[("A2: 模型元数据\n权限 / 状态 / 降级策略")] Metrics[("A3: 使用指标\n实时 RPS / 水位监控")] end subgraph Providers["多云模型提供方"] Cloud1["Cloud 1 (预置吞吐量)"] Cloud2["Cloud 2 (按量付费)"] Vendor2["备用服务商 / 跨云降级"] end subgraph MonitorLoop["健康巡检闭环 (Workflow B)"] HealthCheck["端点健康轮询器"] Alarm["健康告警系统"] end App --> Router Router <--> Meta Router <--> Metrics Router -->|"主路由"| Cloud1 Router -.->|"容量超限/降级"| Cloud2 Router -.->|"供应商宕机"| Vendor2 Cloud1 & Cloud2 & Vendor2 -.-> HealthCheck HealthCheck --> Alarm Alarm -->|"状态反馈更新"| Meta ``` 在大规模工程落地中，模型生命周期管理遵循三阶段原则： - **上线前评估（Pre-launch Evaluation）**：重点在于负载测试框架的压测结果。系统需全面量化输入/输出 Token 规模、不同模态（文本、图像、视频、文档）对生产环境 P95 延迟的影响。若聊天机器人业务场景在压测中出现 10 秒的 P95 延迟，则必须在模型选型或业务逻辑上进行取舍；同时需综合考量合规自建与成本预算。 - **运维弹性（Operational Resiliency）**：建立明确的自动扩缩容（Auto-scaling）机制，对应用端和底层端点实施双向速率限制（Rate Limiting），防止产生 `429 Too Many Requests` 报错；区分实时场景与离线批处理（Batch Inference，延迟换取最高可达 50% 的成本节省），并配置跨云与多供应商冗余兜底。 - **上线后评估（Post-launch Evaluation）**：利用合成测试（Synthetic Tests）持续监测准确率、相关性与延迟指标，监控因供应商底层静默升级模型版本（如 Gemini/OpenAI 自动切版本）而引发的响应漂移风险。 --- ### [21:59](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=1319s) 知识表征与检索架构：双重目的地与大规模 RAG 落地 知识表征（Knowledge Representation）负责将企业内容管理系统（CMS）中的静态资产转化为可支撑检索增强生成（RAG）的高质量上下文。摄取流水线通过轮询或事件推送捕获 CMS 增量变更进入暂存存储，随后由处理核心执行清洗、分块、关键词提取、后处理及向量嵌入（Embeddings）生成。此架构最关键的创新在于输出端设计了**双重目的地（Dual Destinations）**：处理后的数据会同步分流写入“语义搜索引擎”与“分析数据仓库”。 ```mermaid flowchart LR accTitle: 知识检索与双重目的地写入流水线 CMS["内容源 (CMS)"] -->|"增量 Push/Pull"| Inbound[("入站暂存存储")] Inbound --> Processor["知识处理器\n清洗 / 分块 / 关键词 / Embedding"] subgraph Destinations["双重输出引擎"] Search[("语义搜索目的地\n(Vector DB / OpenSearch)")] Analytics[("分析引擎目的地\n(数据仓库 / 指标系统)")] end Processor -->|"检索索引写入"| Search Processor -->|"元数据/血缘写入"| Analytics ``` 将分析存储与语义搜索物理隔离的核心原因在于：企业必须深度洞察知识文章在实际生成中的真实效用，但绝不能让繁重的分析查询压垮面向线上的向量引擎。系统需要回答一系列精细化指标： - 检索到的知识文章在最终生成回复中究竟占了多大比重？是贡献了 50% 的关键信息，还是仅占 5%？ - 在召回的多个参考源中（如 PDF 1、2、3），是否存在某些文档长期无用、仅有单篇文档起作用的现象？ - 最终生成的回复是被客服直接采纳、经过大幅编辑还是被彻底丢弃？ 在大规模支撑高并发检索时，同样需要围绕三大阶段进行精细工程治理： - **上线前评估**：在预发阶段快速横向评测多种向量引擎与检索算法（如 NMSLIB 与 Lucene 的对比）、确定最佳 Top-K 召回阈值（如 $K=2$ 还是 $K=10$），并评估内存数据库与 S3 Tables / PostgreSQL 等持久化存储在成本与延迟间的平衡点。 - **运维弹性**：设置带有严格上限的自动扩缩容，并在业务协议限流基础上预留约 20% 的安全缓冲（Buffer）。演讲者特别提到，AWS Bedrock 在跨可用区（如 `us-east-1` 与 `us-east-2`）算力耗尽后官方通常建议新建 AWS 账号，但这在生产环境极难落地；Asurion 在实际遭遇算力耗尽时，直接通过抽象层将流量无缝降级转移至 Claude/Anthropic 独立端点，保障终端用户体验绝不中断。 - **上线后评估**：由于近似最近邻（ANN/KNN）搜索本质上是用精度换取速度的非确定性算法，微小的输入变动可能导致召回结果大幅偏离。团队必须通过持续的自动化功能性测试，严密监控检索相关度与召回准确率的衰减。 --- ### [29:10](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=1750s) 从数量到质量评估：基于事件采样的低成本质检机制 演讲者明确区分了“数量评估”（Quantity Evaluations）与“质量评估”（Quality Evaluations）的本质差异。在系统前期的压测阶段，关注点完全聚焦于吞吐量与时延等数量指标——测试团队只需生成上千条合成对话记录并要求大模型在 400 Token 内完成摘要，即可统计 P95 延迟，完全无需人工或算法去核验摘要内容本身的正确性。 然而，当系统进入生产环境后，质量评估成为了决定业务成败的核心，其面临的最大痛点是**全量评估的高昂成本与算力黑洞**。当企业每天产生数百万次交互对话（Chat Events）时，若对每一次对话都调用大模型评审员（LLM-as-a-Judge）或复杂评估流水线进行全面质检，财务成本与耗时将不可承受。 针对这一瓶颈，团队引入了**事件采样器（Event Sampler）**架构作为质量评估的入口守门人： - 放弃全量质检策略，改用科学的事件采样逻辑，将评估范围收敛在整体流量的可控百分比内。 - 在系统上线初期，采用较高的采样比例（如 10% 或更高）进行高频监测与密集验证；待系统指标稳定后，逐步将采样率回调至 5% 或更低水位。 - 采样器内部根据预设的特定事件特征与业务标签进行分层过滤，在确保评估统计学显著性的同时，大幅压缩评测成本。 ### [30:58](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=1858s) 基于事件采样与 LLM 评分的高效质量评估架构 在规模化生产环境中，生成式 AI 应用会源源不断地产生海量交互事件。如果对每一次调用的每个瞬间都进行全量质量评估，计算成本与系统开销将难以承受。因此，系统在架构设计上引入了事件采样器（Sampler），通常将采样率设定在 5% 到 10% 之间，并可根据需要动态上下调节。整个系统左侧是产生海量事件的 GenAI 应用，右侧则是强大的数据与分析平台；演讲者特别强调，如果没有坚实的数据与分析底座，企业在 AI 运维中就如同“在盲跑（running blind）”。 采样器并非盲目抓取所有细碎事件，而是专门针对具有业务代表性的关键生命周期节点进行过滤。以客服代表（CSR）处理外部客户咨询的场景为例，系统不会在会话的每分每秒都去评估交互效果，而是精准捕获会话结束时触发的 `chat complete` 事件。如果一名客服每天处理 50 通客户对话，系统仅抽取约 5 通进行分析，就足以形成具有统计指导意义的“脉搏检测（directional pulse check）”。这些被抽中的事件会被分发到专用队列中，由后端的异步评估进程展开全方位的质量分析。 在评估阶段，开发者可以灵活运用大语言模型（LLM-as-a-Judge）执行极其细致的自动化多维打分。评估流程会检查整场对话，并就客服的友好度（按 0 到 5 分打分）、客户问题是否彻底解决、客户的情绪基调、客服代表的语气态度、知识库文章的检索深度与调用相关性等维度进行全方位审计。所有评估进程产生的大量质检指标数据，会再次汇入以 Kafka 为代表的分布式事件分发总线，最终沉淀到统一的分析平台中，为业务调优与模型迭代提供洞察支撑。 ```mermaid flowchart LR accTitle: GenAI 评估与事件采样架构流程图 subgraph Apps ["业务应用层"] A["GenAI 应用"] --> B["全量业务事件"] end subgraph Sampling ["事件过滤与采样"] B --> C{"采样器 (5%-10%)"} C -- "关键事件 (如 chat complete)" --> D["评估专用队列"] end subgraph Eval ["评估与质检层"] D --> E["LLM 质量评估进程"] E --> F["多维评分与质量元数据"] end subgraph Analytics ["数据洞察层"] F --> G["Kafka 事件分发"] G --> H["统一数据分析平台"] end ``` - **采样策略**：日常运行维持在 5% 至 10% 的动态采样区间，兼顾监控代表性与计算资源开销。 - **关键事件对齐**：专注于终端状态事件（如 `chat complete`），避免对单次会话中的中间轮次做冗余计算。 - **多维度 LLM 质检打分**：涵盖 0-5 分友好度评分、问题解决状态识别、双向情绪基调分析、知识库文章匹配度与回答深度等。 --- ### [33:26](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=2006s) 规模化评估策略：长期采样与短期全量监控 (Long-Lived vs. Short-Lived) 为了在兼顾成本的同时敏捷响应系统变更，整套评估体系完全由配置驱动（Config-driven）。根据不同的业务场景，团队沉淀出预设配置，使评估进程能够与生产线上部署的实际业务应用并行运行。在工程实践中，团队总结出两大核心评估模式：**长期评估（Long-Lived）**与**短期评估（Short-Lived）**，两者结合打破了单纯依赖固定采样的局限。 ``` +------------------+------------------------------------+------------------------------------+ | 评估模式 | 长期评估 (Long-Lived) | 短期评估 (Short-Lived / Time-Bound)| +------------------+------------------------------------+------------------------------------+ | 运行周期 | 伴随应用生命周期永久运行 | 变更发布后短期窗口 (如 24~72 小时) | | 采样比例 | 5% ~ 10% 抽样 | 指定时间窗口内 100% 全量捕获 | | 典型触发场景 | 日常质量基线监控、趋势分析 | Prompt 调优、新功能上线、修复验证 | | 核心目标 | 持续把握系统整体质量脉搏 | 密集捕捉变更后的实际反馈与潜在风险 | +------------------+------------------------------------+------------------------------------+ ``` 长期评估属于通用型的质量基线测试，只要应用在生产环境处于活跃状态，它就会按照预设的 5% 或 10% 采样率无休止地运行，持续提供宏观质量脉搏。然而，当工程师调整了提示词（Prompt）时——例如为了引导模型更加通俗易懂、去除生硬的技术行话（Tech jargon）并提升解答亲和力——团队往往需要在变更刚生效的短时间内进行极高密度的观测，以验证新 Prompt 的实际表现。 这时短期评估机制便发挥作用：团队可以通过配置设定一个具有明确起止时间的时间窗口（例如“在美东时间 1 月 10 日下午 4:15 发布新特性，并在接下来的 72 小时内运行”）。在此时间段内，采样器会切换策略，针对该特定版本或事件进行全量（100%）抓取并执行专项评估。这种“平时低频常态采样、发布后短期密集质检”的混合机制，既避免了永久全量检查带来的算力浪费，又杜绝了漏判线上故障的风险，依托一体化、实时的统一分析框架实现了高效的规模化运营。 - **配置化驱动**：所有评估任务与采样规则均由预先定义的配置文件动态管理，无缝与生产服务并行。 - **精细化时间控制**：支持按具体时间戳（如特定日期与具体分秒）设定 24 小时至 72 小时的短期集中监控窗口。 - **分析平台集成**：所有评估链路汇聚于一站式实时分析体系，为工程与产品团队提供敏捷的决策闭环。 --- ### [35:57](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=2157s) 企业级 GenAI 安全防线：PII 保护、护栏与访问治理 演讲最后着重阐述了保障企业级 GenAI 平台稳健运行的安全与合规支柱。在全流程架构中，安全性必须作为“一等公民（first-class citizen）”贯穿始终，首要任务便是对个人敏感信息（PII）的严防死守。系统内置了端到端的 PII 检测与自动脱敏（Detection and Redaction）机制，确保在上游尽可能彻底地阻断敏感数据流向底层大模型，从源头杜绝数据泄露隐患。 除数据层面的脱敏外，平台在推理与交互各环节均尽可能广泛地部署模型安全护栏（Guardrails），彻底封堵恶意提示词注入、违规输出及模型被滥用的空间。在权限治理维度，平台构建了严格的身份认证（Authentication）与权限授权（Authorization）机制，对数据流、知识库检索路径以及模型调用实施细粒度的基于角色的访问控制（RBAC）。配合端到端的调用能见度追踪与日志审计体系，平台全面满足企业级治理与合规要求。演讲在此画上圆满句号，随后进入现场提问环节。 - **PII 自动化检测与脱敏**：内置脱敏引擎，在请求触达模型前剔除敏感个人信息。 - **全方位安全护栏**：在输入与输出两端设置 Guardrails，防止越狱、滥用及非合规内容生成。 - **RBAC 与合规治理**：对数据源、模型资产与接口进行严格的基于角色访问控制，并提供全链路审计日志以满足企业合规需求。 ## Notable quotes > [01:35](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=95s) the next shiny thing — Rajesh Gundugollu > [01:35](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=95s) multicloud is the new norm — Rajesh Gundugollu > [04:37](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=277s) hiding all your chaos behind a drywall — Rajesh Gundugollu > [11:12](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=672s) let's go full AI — Rajesh Gundugollu > [30:58](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=1858s) running blind — Rajesh Gundugollu > [30:58](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=1858s) directional pulse check — Rajesh Gundugollu > [35:57](https://www.youtube.com/watch?v=GFSVWYwcuT0&t=2157s) first-class citizen — Rajesh Gundugollu ## Mentioned Rajesh Gundugollu, Ansh Gandhi, Asurion, Azure, GCP, AWS, OpenAI, Gemini, Bedrock, SageMaker, Claude, Anthropic, Llama, Qwen, Kafka, OpenSearch, PostgreSQL, S3 Tables, NMSLIB, Lucene ## Video description Rajesh Gundugollu, Director Data & ML Engineering, Asurion Ansh Gandhi, Senior Software Engineer, Asurion Abstracts How can organizations deploy generative AI at enterprise scale while maintaining security, compliance, and cost-effectiveness? In this talk, I will reveal a comprehensive framework built on three essential pillars that support millions of daily AI transactions: knowledge platforms with automated pipelines, model serving infrastructures with intelligent routing, and evaluation frameworks that optimize for quality and cost. I will share practical implementation patterns for high-availability knowledge flows, abstractions for model lifecycle management, and evaluation strategies that balance performance with business constraints. Join me to discover how these foundational systems work together to create resilient GenAI applications that deliver value while keeping security, compliance, and costs in check. [Source](https://www.youtube.com/watch?v=GFSVWYwcuT0)

[1:35] Rajesh Gundugollu: the next shiny thing [1:35] Rajesh Gundugollu: multicloud is the new norm [4:37] Rajesh Gundugollu: hiding all your chaos behind a drywall [11:12] Rajesh Gundugollu: let's go full AI [30:58] Rajesh Gundugollu: running blind [30:58] Rajesh Gundugollu: directional pulse check [35:57] Rajesh Gundugollu: first-class citizen

[0:04] Rajesh Gundugollu: anything that has a power button. That's how we like to say it. We care for people and their tech wherever, whenever, and however. [1:35] Rajesh Gundugollu: Abstraction is the most important layer that you would want to build because it's like hiding all your chaos behind a drywall. [11:12] Rajesh Gundugollu: If you go back and say it's going to cost us X amount of dollars—let's say, okay, I can run this, but it's going to cost us $10 million—if you put that in numbers, that's when the reality comes into picture. [30:58] running blind

[0:04] Good morning everybody. Um, thanks for [0:06] taking the time to attending this [0:08] session. I'm super excited to present [0:11] all of our learnings to you. I'm Rajesh [0:13] Gundugalu and I have An Gandhi with me. [0:16] Um, and together we're going to walk you [0:18] through today's session which is [0:20] engineering AI platforms for scale. [0:25] So [0:26] this is what we're going to cover who we [0:28] are. What did we do and how did we do [0:31] it? What did we learn from that? What [0:34] should you take away from this? And [0:37] hopefully we'll have like enough time [0:38] for questions. That's where a lot of [0:40] interesting conversations take place. [0:44] So before we get into the actual content [0:47] who we are. So we are from Ashuran. [0:50] Ashirin is a techare company and we work [0:53] with over 300 million customers [0:56] worldwide [0:57] and we offer protection and insurance [1:00] plans on anything that has a power [1:01] button. That's how we like to say it. [1:06] We care for people and their tech tech [1:08] wherever, whenever, and however. [1:12] And we work on a global scale with over [1:14] 18,000 employees and uh 15 countries. [1:19] And we have over 9,000 experts who are [1:22] available uh to help the customers [1:25] virtual in store and on the road and we [1:28] cover over 88% of US population. [1:35] So before heading into the actual [1:37] content for today in the very same [1:39] conference uh uh last year which is 2024 [1:42] we presented actually founding [1:45] principles or foundational elements for [1:48] uh for your AI platforms to be [1:50] successful and one of the core themes [1:52] for that was how how why do you need [1:54] actually a very strong data foundation [1:56] even before we talk about AI. So I just [1:59] wanted to recap because we tend to skip [2:02] too fast and then try to build the next [2:04] shiny thing but actually foundations are [2:06] really really strong uh are very [2:08] important that's what we believe in. So [2:10] uh this is a quick recap from what we [2:12] have presented last year and you can see [2:14] the core theme and you will see actually [2:16] as we move into the next few slides and [2:18] we touch upon various systems you'll see [2:20] that all of them are actually pointing [2:22] back to the same things because we'll be [2:24] using these over and over again but in a [2:27] very advanced way so that it actually [2:29] these core principles help us run faster [2:32] with all of your AI platforms. So the [2:35] first one is investing in a robust and [2:36] reliable data platform which is a very [2:38] given thing and this is where we we [2:41] focused on having a near realtime and a [2:43] very robust uh analytics platform for [2:46] you to be able to run all of your [2:47] analytics because you'll see that all of [2:50] these systems are going to be producing [2:52] massive amounts of events and data and [2:54] you would want to know what's going on [2:55] and whether you are trending towards the [2:57] right direction or not. And the second [3:00] one is uh making technology choices. And [3:03] this is where we talked about [3:05] multicloud. So it's not a no no longer a [3:08] question about should I use GCP or AWS [3:11] because we already know that there are [3:13] best of breeds in each one of those. [3:14] OpenAI is in Azure and then Gemini is in [3:18] GCP and you have like the rest of the [3:20] models in AWS. So it's no longer a [3:22] question of which way do I go? You have [3:24] to be multicloud. So that is the new [3:27] norm. [3:28] The next one is uh applying [3:32] data management principles. So we all [3:35] know that 3 years ago this was not a [3:37] thing that we are talking about but now [3:39] this is like the the most uh the most [3:42] talked about thing and the everybody has [3:44] pivoted and we are all working on it. So [3:46] you can see naturally there is not a lot [3:48] of maturity in this space. So we don't [3:50] need to wait for that maturity and best [3:52] practices to come naturally. We can [3:55] actually take all of those from the [3:57] existing matured side which is like [3:59] data. So data has been here for quite [4:01] some time and then we have a lot of [4:03] lessons that we have learned from data [4:06] uh data systems and data engineering [4:08] right you can take a lot of those and [4:10] apply actually to your LLMs as well. So [4:12] that is the key point there that we [4:14] focused on saying that you you don't [4:16] need to wait for this this uh ecosystem [4:19] to mature. you can actually take a lot [4:20] of lessons learned and apply them as it [4:22] is as common sense principles to the AI [4:25] systems as well. The next one is [4:27] prioritizing security and observability. [4:29] Again, I don't need to go through those, [4:31] right? Those should be your first class [4:32] citizens when you're building data and [4:35] AI systems as well. The last one is [4:37] focusing on abstraction. So this is [4:39] where I uh heavily [4:43] emphasize that abstraction is the most [4:45] important uh layer that you would want [4:47] to build because it's like hiding all [4:50] your chaos behind uh a drywall. So that [4:53] is how I would like to say because these [4:56] systems tend to change and then evolve [4:57] and you don't want your downstream [4:59] systems to be always disrupted by your [5:02] engineering chaos. So you would want to [5:04] start with a a very robust abstraction [5:06] layer. So that's the those are the key [5:08] themes that we presented last year. So [5:11] what did we do in the last one year is [5:14] um obviously we built upon this and we [5:16] scaled our systems to I would consider [5:18] them massive because we are probably [5:20] processing millions of transactions [5:22] every day and handling millions of chat [5:25] transactions and call transactions with [5:26] our customers. So we are doing I would [5:29] say a good job in handling all of that [5:31] while keeping everything else in check. [5:33] Right. The big ones are cost, security, [5:36] compliance and whatnot. We can talk [5:38] about all of those. So how did we do [5:40] that is what we're going to show or uh [5:42] we'll show some key themes that you [5:44] would want to take away. So we boiled [5:46] down all of those into three systems or [5:49] three modules. The first one is [5:51] knowledge representation and retrieval. [5:53] This is where your uh vector databases [5:57] and augmentation and all of that happen. [5:59] The second one is model hosting which is [6:02] where you would want to you talk about [6:04] all the models like anthropic models, [6:07] open AI and also the open source models [6:09] like llamas or quen models, embedding [6:11] models, reeranker models and all of that [6:14] you would you would host. The last one [6:17] is the most important one which is [6:19] evaluations [6:20] and that has two pieces to it. One is [6:22] the quantity and quality. [6:25] So [6:27] in the evaluations even though it's at [6:30] the actually at the bottom of the list [6:32] it made a lot of sense uh in the very [6:34] last moment when we were preparing this [6:36] we saw the theme emerge right the [6:38] quantity evaluations are actually a [6:41] horizontal it's not like a a vertical [6:43] like a model hosting or a knowledge uh [6:46] knowledge uh hosting or knowledge [6:47] management it's actually a horizontal [6:49] that cuts across all these uh frameworks [6:52] so an is going to walk us through the [6:55] quantity evaluations first even though [6:57] it's the last on the list. you will see [6:58] why that connects the dots very well and [7:00] then we'll cover the knowledge [7:02] representation retrieval model hosting [7:03] and then I'll wrap it up with the uh [7:06] quantity sorry quality evaluations and [7:08] then we can open it up for questions [7:11] an o [7:14] >> hello hi my name is an hope everybody's [7:16] enjoying the conference so far um yeah [7:18] I'll start with the u quantity [7:20] evaluations so as Rajes mentioned um [7:23] this is something that is across domains [7:25] it's not just tied to evaluating models [7:28] or uh just the knowledge base. It's [7:31] something that cuts across domains and [7:32] it's horizontal. Uh the first thing [7:35] we're trying to do here with evaluating [7:37] uh the quantity is trying to make the [7:39] system more predictable. As we know LLMs [7:41] are nondeterministic and there are a lot [7:43] of unknowns attached to the LLM itself. [7:46] Um so we start with uh low testing the [7:49] LLM itself that it could be LLM or you [7:52] know something that we're hosting [7:53] ourselves. It could be an embedding [7:55] model. Um what we do is we decide we [7:58] take certain input parameters. Uh one we [8:01] start with what's the request per second [8:02] that we're expecting the load to be [8:04] right that could be something from let's [8:06] say 10 requests per second or it could [8:07] be like 50 requests per second. So we [8:09] take the request per second as the uh [8:12] input parameter. Um next we take in how [8:14] many how many seconds or how many [8:16] minutes do we want to run the load test [8:17] for. Right? Um then we look at the [8:20] concurrency parameter u and that's more [8:23] so if we're using an API to test the [8:25] endpoint. So let's say we're testing an [8:27] open a open openi endpoint or we testing [8:30] um let's say ge gemini endpoint. We [8:32] don't want to overwhelm the system while [8:34] testing the system itself. Right? So we [8:36] have a concurrency parameter there. We [8:38] select the endpoint that we're testing. [8:39] Now the endpoint could be something as [8:40] you mentioned across domain. So it could [8:42] be an embedding model. It could be [8:43] something like an anthropic model. It [8:45] could be something on bedrock or it [8:48] could be something on Gemini, right? So [8:49] or or it could be something that we're [8:50] hosting ourselves like a SageMaker model [8:52] and an open source model. Um and then [8:55] we're looking at um a data set that we [8:57] want to test with, right? So the request [9:00] response structure as we know varies [9:01] across models. So let's say Gemini has a [9:04] different request response structure um [9:06] and Anthropy could have a separate [9:07] response structure and then let's say [9:09] embedding is a completely separate model [9:10] itself. Right? So to help users what [9:14] we've done is we've come up with our own [9:15] uh sample data sets as well. So and we [9:17] followed the rule of 10 is to one. So [9:19] for every 10 input tokens we have some [9:21] standard u like a 100 out so th000 input [9:24] tokens and 100 output tokens all the way [9:25] up to 4,000 input tokens and 400 output [9:27] tokens. So what that does is if a user [9:30] doesn't have a specific data set to test [9:32] on we give them some sample data sets. [9:34] Otherwise users have the option of [9:36] creating their own uh data sets that [9:37] they want to test let's say for an [9:38] embedding model. Um and then of course [9:42] you select your uh your your data set [9:44] and then you select the inference [9:45] parameters. So you could select [9:47] something from you know like the [9:48] temperature, what's the maximum output [9:50] tokens that you want to test for. And [9:52] once we have all these parameters, what [9:53] we do is we go through and run our [9:56] tests, right? And once we run our tests, [9:58] what we do is we within the same [10:00] platform, we show them what the what the [10:02] results for the test were. every dot [10:04] that you see. So the blue line is [10:05] essentially the P95 latency that we [10:08] observed and on the x axis you see uh [10:10] the requests per second. So for three [10:12] requests per second what's the P95 uh [10:14] that you observed at five requests per [10:16] second. What's the P95 and the P50 that [10:18] you observed? And every dot that you see [10:20] there is is essentially a low test that [10:22] we've run. Um and what we generally do [10:25] is we run you can you can choose you can [10:27] choose it as a parameter or you [10:28] essentially can run three low tests for [10:31] each test and uh you kind of average the [10:33] P95. The idea there being since we're [10:36] making calls to the APIs there are [10:37] certain aspects that you you can't [10:39] account for in every run. So there are [10:41] some network latencies that come into [10:42] play and some inconsistencies. So an [10:44] average kind of takes care of all the [10:47] inconsistencies that come into play. Um [10:49] and then again like looking at this [10:52] graph what what essentially this does is [10:53] it'll give you a very pred predictable [10:56] um idea of if you take a certain model [10:58] at a certain input output tokens what is [11:01] the latency that you should expect in [11:02] production and and there are no um there [11:05] there's no we we kind of trying to [11:07] reduce the things that are unknown in [11:09] production right so that's the idea [11:11] behind [11:12] >> if I if I may add so you the the why [11:15] that we should do this right you would [11:17] wonder like why should we even do this [11:19] and to to put it blunt you if you go you [11:22] would have many of this or maybe [11:24] everybody in this room would have heard [11:26] from your top leadership saying that [11:28] okay let's go full AI right let's go do [11:30] it but if you go back and say it's going [11:32] to cost us x amount of dollars let's say [11:35] okay I I can run this but it's going to [11:37] cost us 10 million if you put that in [11:39] numbers that's when the reality comes [11:40] into picture right so what happens what [11:43] we did is that is what we exactly tried [11:45] to do we tried to put as much [11:47] predictability ility into that and say [11:49] boil down into okay for for the type of [11:52] latencies that you're expecting for the [11:54] volume of the requests that you're [11:56] expecting this is what it's going to [11:58] cost and for us to be able to give you [12:00] give a predictable performance we need [12:02] to provision those endpoints so if you [12:04] go and use a pay as you go or an regular [12:07] endpoint in open AAI your SLA is not up [12:11] the uptime of the availability of the [12:13] endpoint is guaranteed but not the [12:15] latency because you are sharing that [12:17] that traffic with the rest of the world. [12:20] So you would want to have predictable [12:22] performance when you're putting a [12:23] chatbot in front of your customer, [12:24] right? So that means your SLA should be [12:26] guaranteed. That means you need to be [12:28] paying upfront for those endpoints. And [12:30] paying upfront is like whether you're [12:31] using it or not, you're probably paying [12:33] $100,000 for provisioning an endpoint [12:36] which is going to be giving you [12:38] milliseconds or probably seconds in [12:40] terms of latencies. And in order to give [12:42] you that kind of uh educated what do you [12:45] call decision making or datadriven [12:47] decisions making saying that okay based [12:49] on my expected volume input output [12:52] tokens and SLAs and everything probably [12:55] a provision endpoint makes sense versus [12:57] no I'm going to use a pigo endpoint. So [13:00] that kind of a decision making you need [13:02] a framework that can you can quickly [13:03] iterate through that is why we had to [13:05] build one like that. Next uh we we'll [13:08] talk about why did we there was a [13:11] question [13:13] your load test that's why you're going [13:15] against the point of strategy here not [13:19] to take charge of [13:21] >> yeah we we will get charged so [13:23] >> yeah yeah yeah [13:26] but the idea is to so so uh your [13:29] duration for the low test right so if [13:30] you're running it for 5 minutes the idea [13:32] is like you would rather know the charge [13:34] at at 5 minute interval rather than [13:35] knowing it in a production scale where [13:37] like you're running it for probably [13:38] hours and hours together, right? So the [13:39] idea as we said like is to bring more [13:41] predictability to the system and it's [13:43] running in production and you know the [13:44] unknowns beforehand as opposed to [13:46] running this in production and then you [13:47] realize the cost are something you know [13:48] like these weren't accounted for and and [13:51] the the other idea of course is to make [13:53] make these repeatable. So uh once you [13:55] start selecting your requests per second [13:56] and you start selecting your endpoints u [13:59] you're able to iterate over these at a [14:01] pretty fast pace once you have the [14:02] framework in place. So you're not [14:04] worried about uh you know like the the [14:06] whatifs kind of are what you're trying [14:08] to solve with the framework. [14:11] Uh yeah, next we'll talk about like the [14:12] consistency the inconsistent uh [14:14] concurrency. So uh as you've seen like [14:16] there are a lot of frameworks uh that [14:18] that'll give you u your load testing [14:20] features. Uh but one thing we noticed uh [14:23] very quickly is when you let's say [14:25] you're load testing for the worst case [14:26] scenario and that's what we're load [14:27] testing for is the worst case scenario [14:29] is where you your endpoint is constantly [14:31] being hit with let's say for simplicity [14:33] I start with four requests per second [14:34] right um some of these frameworks will [14:36] of course say that they're they're [14:38] hitting your endpoint with with four [14:39] requests per second but then if you look [14:40] at time t equals to 5 or time t equals [14:42] to six you you essentially have these [14:43] pockets in between that that that are [14:46] essentially not your endpoint isn't [14:47] being hit with four requests per second [14:49] right so you now these small errors in [14:51] your calculations quickly add up and [14:54] when when you're dealing at scale um the [14:56] latencies could be off or the cost might [14:58] not have been uh calculated correctly uh [15:01] is the reason we decided to come up with [15:02] this framework for us to load test these [15:04] endpoints. So what we ended up doing [15:06] with consistent concurrency is we making [15:07] sure that the endpoint is consistently [15:09] as let's say 4 per second. So there is [15:11] there's no there's no room for the [15:14] endpoint to breathe and we we're trying [15:15] to test the endpoint to its limits where [15:17] we we the idea is to know where would it [15:19] fail and we we're trying to make sure [15:20] that the endpoint fails so that we know [15:22] that beforehand as opposed to it being [15:23] in production and failing there. Next [15:25] I'll go over the model hosting and the [15:27] abstraction layer. Uh so we we we look [15:29] at this in two parts. So the first part [15:31] is on the left where the request comes [15:33] in and we have the A workflow and and [15:35] the second part would be the B workflow [15:37] that's on the top right. Right? So when [15:39] an inference request comes in right we [15:41] we're hitting the abstraction layer and [15:43] the abstraction layer essentially has [15:44] three three problems to solve right one [15:47] is being the abstraction layer itself. [15:48] So it's it kind of takes away the pain [15:50] of uh the request response and the [15:53] vendor uh specific request response [15:55] structures. Um the second is making sure [15:58] the uh the system itself the the [16:01] integrations with the LLMs they're fall [16:03] tolerant. And the third would be we're [16:04] trying to make sure that we rate [16:06] limiting the endpoints and we rate [16:08] limiting the usage. Um, of course we'll [16:10] have different applications using u the [16:13] abstraction layer right so once a [16:15] request comes in it for it it looks at [16:18] two specific parts. It'll look at the [16:19] model metadata that's the A2 and then it [16:22] also look at the uh model usage metrics [16:24] that's the A3. uh using both these [16:27] endpoints the the basic task that it's [16:29] trying to achieve is for a request it's [16:30] trying to check if that specific request [16:33] has access to the model it's trying to [16:35] access is the model healthy right is and [16:38] we'll come back to how the model is [16:39] being marked as healthy or unhealthy but [16:41] the idea is for the model metadata to [16:43] look at uh if the request that it's [16:45] trying to uh the model it's trying to uh [16:47] get access to is that even valid and the [16:49] next thing it's trying to do on A3 is [16:51] trying to make sure that the model [16:53] itself that we're hitting um are we are [16:55] we going to go overboard with the limit [16:56] that the model has for itself, right? So [16:58] it could be the model has uh a rate [17:01] limit at 100 requests per second, right? [17:02] Are are we the 101st requests per [17:04] second, right? Should we follow should [17:06] we fall over to the next model in place, [17:08] right? So that is something that uh we [17:10] calculate the usage metrics for. Now [17:12] both of them serve a little uh different [17:14] purposes. The model metadata is is more [17:17] for an application specific uh decision. [17:20] So you could have three applications [17:21] hitting a bed bedrock endpoint and we're [17:24] trying to make sure that none of those [17:26] three applications kind of uh step on [17:28] each other's stores right so so that [17:29] they're not taking over each other's [17:31] capacity and the model usage metrics is [17:33] more about the endpoint itself so are we [17:36] good with uh the anthropic on US east [17:38] one for example or are we good with open [17:39] AAI on uh in the Canada region for [17:42] example right so we're trying to measure [17:43] both these things once we have both [17:45] these information the abstraction layer [17:47] needs to intelligently route the request [17:49] So it could for example choose to route [17:51] the request to cloud one on the [17:53] provision endpoint and it sees based on [17:55] the usage metrics that you know we're at [17:57] capacity or we the model is unhealthy [17:59] itself we need to move it to a payo on a [18:01] cloud pro cloud provision at uh a pay go [18:03] endpoint at cloud 2 right so the [18:05] abstraction layer decides at runtime on [18:08] where the request should be routed to [18:10] right that makes it that makes sure that [18:12] we intelligently handling the request [18:14] and at the same time you're also making [18:16] the system as a whole fall tolerant to [18:18] the uh a particular vendor go going [18:21] down, right? So that is something that [18:22] that might happen or probably has [18:24] happened already is where a specific [18:26] vendor kind of goes down and then we [18:27] left with with no other uh vendor to [18:29] send the request to, right? So that [18:30] that's that's the job for the [18:32] abstraction layer, right? This completes [18:33] the A part. Now when we look at the B [18:35] part, uh we have the B workflow. What [18:38] what that's doing it's a separate [18:39] workflow and that will essentially keep [18:41] querying these endpoints for health [18:43] metrics, right? And then we have health [18:45] alarms set on those metrics which will [18:46] go back and uh give a a feedback to the [18:49] model metadata itself. So let's say we [18:51] we're looking at the openi endpoint and [18:53] we see a pay go endpoint go down right [18:55] that updates the model metadata saying [18:57] this particular endpoint is is already [18:59] down. So now when a new request comes in [19:02] again for the workflow a we see that [19:04] this particular endpoint isn't available [19:05] right so we again look at what's the [19:07] fallback model should should uh from [19:09] from a provision endp point should we go [19:10] to a cloud provisioning tool uh or [19:12] should we go to a pigo endpoints for [19:14] cloud cloud 2 right so this essentially [19:16] adds a feedback to the system itself and [19:18] this makes it more fault tolerant yeah [19:20] and so we we'll now go about how do you [19:22] handle all of this at scale right so we [19:24] we looking at it in three parts one is [19:26] the pre pre-launch evaluation one is the [19:29] operational resilency itself and and the [19:30] third is the post uh post-launch [19:32] evaluation. Now pre-launch evaluation is [19:34] something we've already discussed that [19:36] that was the load load testing framework [19:37] essentially right we covered what what [19:39] are the input output tokens what are [19:41] what is the P95 latency that the model [19:44] itself is giving us in production what's [19:46] the use case and what's the what's the [19:48] P95 for the use case itself so for [19:50] example if in in u while running the [19:52] load test we see the P95s are at 10 [19:54] seconds right and the use case is for a [19:55] chatbot that essentially wouldn't work [19:57] right so that's where we would probably [19:58] have to tweak either the use case or the [20:01] model itself right so you've got to pay [20:02] the price somewhere there. Um and then [20:04] uh we look at the modalities. Are we [20:06] looking at a text model? Are we looking [20:07] at an image model or like a video model? [20:09] Um are we processing documents for [20:11] example, right? And then we look at the [20:13] infrastructure. U we look at u is this [20:16] something that should probably go on a [20:17] provisioned units endpoint or should we [20:18] look at a piggo endpoint or there could [20:20] be some compliance uh reasons where you [20:22] would probably want to host this on our [20:24] own cloud, right? And we probably [20:25] wouldn't want to use another vendor for [20:27] it, right? Uh so that's that's the [20:28] fourth thing. And the fifth and the most [20:30] important thing is of course the cost, [20:31] right? So we we've got to make a [20:33] decision on u what is all of this going [20:36] to cost and and what's the cost that [20:37] we're okay with right u and coming to [20:40] the operational resiliency once this is [20:42] in production what what do we look at we [20:44] need to make sure that the endpoints [20:45] that we're using they're scaling well we [20:46] have well definfined autoscaling [20:48] policies there uh we rate limiting the [20:50] application and the endpoint itself so [20:51] we've got to handle two rate limitings [20:53] there uh so that the applications using [20:55] the endpoints don't step on each other's [20:57] stores in some sense and also uh the [20:59] endpoint doesn't start sending us 429 [21:01] errors because we just overboard. Um, [21:03] and then when a request comes in, is [21:06] this use case for a real-time use case [21:08] or is there something that we could [21:09] probably go for a batch influence, [21:10] right? Going towards batch inference [21:12] would of course add some delay to the [21:13] request but also bring down cost by [21:16] let's say by 50%. Um, and then adding [21:19] redundancy and failover, right? That's [21:20] that's the part where the intelligent [21:22] routing kind of decides to go from let's [21:25] say a provision throughput to um, you [21:27] know, a payo endpoint or completely [21:29] switch providers itself, right? go from [21:30] vendor one to vendor two u and the [21:33] postlaunch evaluation is something we've [21:34] we've got to constantly run um either [21:37] using synthetic test we need to look at [21:38] accuracy relevance and latency this is [21:41] essentially making sure we haven't [21:42] drifted away from what we started with [21:44] so let's say we're pointing to the [21:46] latest model on Gemini and u Google [21:48] updates the model version right is the [21:51] newer version giving out different [21:52] results has the latency changed right [21:54] has the relevance changed and that is [21:56] something we need to constantly uh [21:58] monitor [21:59] >> so the next module in this is the [22:01] knowledge representation. So this is [22:03] where you take all of your content that [22:05] you are actually augmenting the [22:07] inference requests with. Right? So I'll [22:10] just go over the components involved and [22:13] then we can talk about how this works. [22:14] So if you see on the left side you have [22:16] a content source which is usually your [22:18] content management system and then there [22:20] is a process that is actually either [22:22] being pushed or pulled in terms of what [22:24] are the changes to that system and there [22:26] is a the middle purple box is the [22:28] processor that is actually processing it [22:30] and it's the green box is the one which [22:32] is actually taking the delta or the [22:34] changes coming into a storage inbound [22:36] which is very uh straightforward right [22:39] and the process actually does all your [22:41] pre-processing post-processing [22:43] extracting some keywords or whatever you [22:45] want and generating the embeddings and [22:47] all of that and it is storing. So that [22:49] is the the key here that you I would [22:51] want you guys to take away from this is [22:54] one to have a robust framework but two [22:56] is on the right hand side we have [22:58] actually two destinations. One is the [23:00] analytics destination and two is the [23:02] semantic search destination. So that's [23:04] the key here. So why did we separate [23:06] that? The reason is you would want to [23:09] know how well your knowledge articles [23:12] are being used. Are they actually being [23:14] helpful in your uh inference requests [23:16] that that are being produced? Are they [23:19] actually being used as it is in your [23:21] chat applications or whatever other [23:22] applications? Are they being edited? Are [23:25] they being completely discarded? If they [23:27] are being used, what's the percentage of [23:29] being uh what's the percentage usage? [23:31] That means in a response that has been [23:34] generated which is an augmented [23:35] response, is your knowledge article [23:38] making up 50% of the response or is it [23:40] only making 5% of the response? And if I [23:42] have like four sources in my knowledge, [23:44] right? I have maybe PDF 1, PDF 2, PDF 3. [23:48] Maybe only PDF 3 is the one which is [23:50] being very useful in my final response. [23:53] One and two may be useless. How do I [23:55] know that? Right? So in order to get [23:58] answer to all of these questions, you [24:00] would want to run a ton of analytics, [24:03] right? And you do not want to run those [24:05] analytics on your semantic search, which [24:07] is actually it has one job, which is [24:09] semantic search. And I don't want to [24:12] overwhelm that storage engine with my [24:14] analytics queries. So that is the reason [24:16] we actually split that. That means the [24:18] same process writes the data to two [24:20] different destinations. One goes to the [24:22] analytics endpoint, two goes to our [24:24] semantic search. So this is the reason [24:26] again weaving back into why having a [24:28] robust analytics platform is very uh [24:31] very much required is again the same [24:33] thing here, right? So once the data is [24:36] processed you might there will be tons [24:38] of other processing because you would [24:40] want to do keyword filtering even before [24:42] doing a semantic search or you might [24:44] want to do keyword filtering after the [24:45] semantic search or you would you might [24:47] want to do a hybrid search right or you [24:49] might want to do a keyword boost search. [24:52] So for all of these you are you cannot [24:54] use your knowledge base as it is [24:56] articles you have to do some processing [24:58] before you load the data that is where [25:00] the center purple box uh comes in and it [25:03] handles all of that processing and then [25:05] once your data is in your final semantic [25:07] uh engine it's pretty straightforward [25:09] how you use that [25:11] how do you do this at scale again same [25:13] question right so most of this I would [25:15] draw your attention to why at scale part [25:18] because even in the previous discussion [25:20] what an walked us through about [25:21] knowledge uh sorry a model abstraction [25:23] layer and all I'm I'm sure that the [25:26] question comes like why do I need to do [25:28] this if there is already load balancing [25:30] happening within bedrock across multiple [25:32] inference regions right it's a valid [25:34] question so if you do not have that much [25:36] scale that kind of an abstraction layer [25:38] or load balancing is not needed that is [25:40] why we are saying that at scale this is [25:43] what you would want to do because [25:45] bedrock gives away after the the [25:47] capacity is reached within those two [25:49] cross regions US1 and uh east one east [25:52] two it gives up the next approach is to [25:55] create a new account that's the [25:56] recommended approach and we do not want [25:58] to go through that and what if you run [26:00] out of that and we did run out of that [26:01] and our fallback model from bedrock was [26:04] actually going to clark so that kind of [26:06] a resiliency is what is required for you [26:09] to put a bot in front of a customer [26:11] because you do not want to disrupt the [26:13] or be known for a bad customer [26:15] experience right okay coming back to the [26:18] at scale part how do you do this [26:19] knowledge uh represent presentation and [26:21] retrieval at scale is pre-launch [26:23] evaluations. Again, going back to your [26:25] quantity evaluations. You have a ton of [26:27] uh configurations that you would want to [26:29] run through. What is the engine that I [26:31] choose? Like should I go with NMS lib or [26:33] lucine? Should I uh what is my K value [26:37] should be? Should I go with two or [26:38] should I go with 10? Right? or what [26:41] modalities are actually impacting the [26:43] latency if I put like text search or [26:45] should I just go with image search or [26:49] should I host this in uh in-memory [26:51] database or should I host this on uh [26:53] something like S3 S3 tables or Postgress [26:55] for cost reasons. So all of these [26:57] combinations you would want to run [26:59] through quickly and come up with a [27:00] perfect configuration that actually [27:02] meets your needs for that use case. [27:05] Okay. So that is where the evaluation [27:07] comes in framework and evaluation comes [27:09] into picture. Once you do that, then the [27:11] next part is operational resiliency. Now [27:13] you have done your due diligence and you [27:14] have put the system in front of the [27:17] customers and it's being hit with [27:18] traffic. How do you now still prevent it [27:21] from going from um whatever edge cases [27:24] that you haven't planned for? That's [27:26] where your operational resiliency comes [27:27] into picture which is like you would you [27:29] would do some autoscaling with some [27:31] upper bound right and still you want to [27:33] preserve that system and protect it from [27:36] unwanted spikes and that is where your [27:37] rate limiting comes into picture. So you [27:39] would have some agreements with your uh [27:41] consumers and say this is the rate [27:43] limiting that we agreed to and add some [27:45] 20% buffer and then go with that and [27:48] then you have uh routing these to [27:51] appropriate endpoints because of the [27:52] cost needs. None of this is cheap. So [27:55] you would want to use the best endpoints [27:57] and the best storage mechanisms for ROI. [28:00] So you would want to route the real-time [28:02] ones and batch ones to different [28:03] endpoints and then you would want to add [28:05] redundancy and failover in various cloud [28:08] providers right AWS goes down you don't [28:10] want your system to go down so that's [28:12] where all of this come and the [28:13] post-launch is about all about quality [28:15] evaluation now that you have made all of [28:18] this uh uh decisions and you have [28:21] launched the product how well is it [28:23] actually performing because these are [28:25] all not exact deterministic ones and if [28:28] you're using KN&N search which is like [28:30] hey nearest neighbor you're actually [28:32] most likely going to do approximate [28:33] search because you're trading off [28:35] accuracy for speed. So when you do [28:37] approximate it is approximate right. So [28:39] even if a small change in the input is [28:41] going to have like a major change in [28:44] your output. So the relevance might not [28:46] be what you actually seen in the in the [28:49] beginning. So that drift you would want [28:50] to keep a pulse check on that. So that [28:52] is what accuracy relevancy and latency [28:55] there are a lot of things that are [28:56] changing here. So you would want to keep [28:58] running constant functional checks, [29:00] functional evaluations on this which [29:02] will give you a health check on your [29:03] system. So that is how you would uh run [29:06] this at scale. [29:10] So the last part in this is the quality [29:13] evaluations. So, we talked about [29:15] quantity evaluations at the beginning, [29:17] which is all about if I'm running, for [29:20] example, okay, what's my P95 latency? If [29:24] I have 4,000 tokens as input and 400 as [29:26] an output, I don't need to really care [29:28] about whether the response from the [29:30] model is accurate. All I have to do is [29:32] I'm going to create a synthetic thousand [29:34] uh uh synthetic data with thousand [29:37] transcripts or whatever. Let's say I'm [29:39] summarizing transcript, right? I can [29:40] generate any transcript or use the same [29:43] transcript and then ask the LLM to [29:45] summarize it within 400 tokens and I [29:49] don't need to go and actually verify [29:50] whether the summary is actually correct [29:52] because here I'm not bothered about the [29:54] quality. I'm bothered about the [29:55] quantity. I'm thinking about how fast is [29:57] the response coming back. Okay. Now this [30:00] one is about all about quality. So if [30:03] you want to do quality [30:06] the biggest prohibitor for that [30:08] especially in the cost prohibitive [30:10] aspect of that is if you want to be [30:12] comprehensive that means let's say there [30:14] are a million chat events that are [30:16] happening you cannot run and you do not [30:19] want to run your quality evaluation on [30:21] on all million of those because that is [30:23] going to be super expensive okay and [30:25] timeconuming so the key that I want you [30:28] guys to take away from this one is [30:30] sampling which is a very known thing [30:32] this is not nothing new. So that is what [30:34] the central uh this the middle one on [30:36] the left side is the purple box is your [30:39] event sample. So we actually will go [30:41] with sampling and what we do is there [30:44] are two types of samplings which I will [30:45] go into the next slide. So what that [30:47] does is it basically looks for specific [30:49] events like maybe 5% maybe 10% it's up [30:53] to you. You would want to start with a [30:54] higher number and then once you get [30:56] comfortable you would want to dial it [30:58] down to probably 5% and if you see [31:00] something that is not really working [31:02] well you would want to dial it up and [31:03] down. So but I would rather stay at 5 [31:06] 10%. [31:08] So that sampler is only looking at 5% of [31:10] your incoming events. On the left hand [31:12] side you have all your applications that [31:14] is what your geni applications are and [31:17] they are producing a ton of events. And [31:19] on the right hand side you have your [31:21] analytics. So I'll again go back to the [31:23] same thing saying you need to have a [31:25] very very strong data and analytics [31:27] platform. So again I'm reinforcing that [31:30] concept saying that without which you do [31:32] not you're running blind basically if [31:33] you don't have data and analytics [31:34] platform right. So that applications uh [31:38] those applications are producing events [31:40] and then you are looking for a [31:41] percentage of those and specific events [31:44] not all events. Let's take example if uh [31:47] if there is a customer uh service expert [31:49] or representative who is handling a chat [31:52] with an external facing customer right [31:55] you do not want to evaluate how well [31:57] that chat went through at every moment [32:00] you would want to do it at the end of it [32:02] so I would be focused on the last event [32:05] that says chat complete okay but if I'm [32:09] handling 50 customers in a day I don't [32:11] want to run this for all 50 if I get [32:13] probably five that gives me a [32:14] directional u pulse check on what's [32:17] going on. So that's what the sampler [32:19] does. It samples those events and then [32:21] sends it to a different queue and that [32:23] is where my evaluation processes are [32:24] running. So what will the evaluation [32:26] processes look like? It's you can be as [32:29] creative as you want to be and that is [32:30] where you can also use LLMs to run a [32:33] very detailed scoring saying okay here [32:36] is the chat that happened give me a [32:38] scoring on 0 to 5 on the friendliness of [32:40] this chat. Tell me whether this is um [32:43] issue has been resolved. What's the tone [32:46] of the customer? What's the tone of the [32:48] uh chat uh sorry representative? What's [32:51] the depth of the knowledge? Has the [32:53] knowledge articles been used here or [32:54] not? What's the relevancy? Right? So you [32:57] can actually run a lot of these tests [33:00] and that is all about quality. And those [33:03] evaluation processes are going to be [33:04] running and they're going to emit a ton [33:07] of data and all of the data again is [33:09] going back to your event distributor [33:11] which is like a Kafka in this case and [33:13] they all make their way into analytics [33:14] where you can connect all the dots and [33:16] run your reports to derive insights and [33:18] then you would do some changes based on [33:21] whether it is going well or whether it [33:23] is not going well. Right? [33:26] So all of this is config driven. If you [33:28] see there is a sample uh sampling [33:30] configs. So based on the use case and [33:32] based on my need I can create [33:34] predetermined configs and this process [33:37] is going to be running in parallel with [33:40] your actual applications that are [33:41] deployed in production to keep a pulse [33:43] check. Now how do you do this at scale [33:46] is a question again right so that is [33:48] where I mentioned earlier saying [33:50] sampling there are two types so one is [33:53] the sampling based the second one is [33:55] time bound based so sampling what we did [33:57] uh what we actually realized and learned [34:01] is instead of going even with a 5% 10% [34:04] sampling there will be cases where you [34:07] would want to actually have more data [34:09] but for a very short period of time best [34:12] case is you change that prompt a little [34:15] bit because you are saying that uh the [34:17] prompt that you are guiding the LLM to [34:20] be more helpful and not be very [34:22] elaborate or not be very don't use tech [34:25] words jargon tech jargon in your [34:27] response is what you're you have uh in [34:30] your prompt but you you that's not [34:32] working. So what do you do? So you [34:34] change the prompt but now you want to [34:36] keep a very close eye on how well this [34:38] new prompt is actually uh helping drive [34:42] the customer's uh chat but you don't [34:45] want to run that forever. So you would [34:46] want to keep a pulse check on that for [34:48] the first 24 hours right so that is [34:50] where your time bound comes so what you [34:52] do you actually put a shortlived so that [34:54] is what we we call long lived and [34:56] shortlived. So long lived is a very [34:58] generic type of quality test. They run [35:01] forever as long as your application is [35:03] running and keep generating insights and [35:05] they are doing 5% 10% whatever the dial [35:07] you have set to. The shortlived are time [35:09] bound. I'm releasing this feature on [35:11] January 10th at 4:15 Eastern time. I [35:16] want it to run for the next 72 hours. So [35:19] your uh in the previous slide I showed [35:21] you that sampler. it what it does is it [35:23] looks for the specific events within the [35:25] time bound but everything all the events [35:27] and then it is going to run your [35:29] evaluations which you are um more [35:32] interested in for those durations. So [35:34] that way you are not exhaustively [35:36] checking forever. At the same time you [35:38] are not check uh pulse sampling check [35:40] forever. You're doing a mix of both. And [35:42] then you're you need to have like if you [35:44] want to run it at scale you need to have [35:46] like a comprehensive integrated [35:47] analytics framework. And then it needs [35:49] to be real time and also it needs to be [35:51] a one-stop one which are like pretty [35:53] standard. And um I'll just wrap this one [35:57] up with the securing the stack. So for [36:00] all of these things that you are doing, [36:02] you would want to be very very careful [36:04] with your PII. So you need to have an [36:06] inbuilt PI detection and redaction [36:08] system so that you will leak as as fewer [36:12] of your PII data to your models. That is [36:16] the PI protection. Then you would want [36:18] to use as many guardrails as possible so [36:21] that it is there is no scope for misuse. [36:24] And then you would want to again ru uh [36:27] connecting back to the previous 2024 [36:29] things that I said like foundationally [36:31] you need to have security as your first [36:32] class uh citizen. So your authentication [36:35] and authorization for all of your data [36:37] and the models which is your rollback [36:39] role based access control and visibility [36:42] into usage and so that you can have a uh [36:45] idea about the governance and [36:46] compliance. So that's the uh I'll wrap [36:49] it up and then we can take probably one [36:51] question if you