Keeping Models Honest: AI Observability for Scalable, Reliable Systems

确保模型可靠：面向可扩展与高可靠系统的 AI 可观测性 主持人 Tristan Arbus 与嘉宾 Tammy Dugan、Brooke Hopkins 及 Shahid Quraishi 共同探讨了在生产环境中维护和调试非确定性 AI 系统的运营挑战。专家组强调，要攻克棘手的 20% 长尾边缘案例，必须超越静态离线基准测试，转向类似自动驾驶系统那样的持续仿真与评估飞轮。核心建议包括：在多智能体执行图中开展细粒度的逐步追踪、对上游 RAG 流水线实施模块化错误分析，以及通过影子发布来评估实际运行中的行为保真度。嘉宾们指出，在各类企业部署中，长期可靠性取决于将生成式能力与确定性基线、人机协同把关机制以及创造性的恢复兜底方案相结合。 决策层面的可靠性比单纯的运行时间或延迟更为关键；AI 系统即便运行迅速且统计表现稳定，仍有可能采取错误的业务操作。 (0:04) 人机协同架构使企业能够自动化处理繁琐的多智能体工作流（如供应商联络），同时将高风险的数据库提交置于人工审核门控之后。 (4:02) 生成式 AI 让概念验证的构建快得令人产生错觉，但要在生产环境中攻克剩余 20% 的长尾分布，需要大量的仿真与评估基础设施。 (7:24) 可观测性必须从二元的底层基础设施指标（正常运行时间/延迟）转向细粒度的多智能体执行追踪，记录每个节点的确切提示词、输入和中间状态。 (9:37) 第三方基础模型提供商的静默后台更新，要求系统具备持续回归测试套件和动态滑动窗口评估，而非依赖静态离线基准测试。 (11:52)

Deploying scalable, reliable AI systems requires shifting from conventional uptime monitoring to granular multi-agent tracing, continuous sliding-window evaluations, and bounded human-in-the-loop architectures. The Production Reality and the Long-Tail Gap (7:24) Modern generative AI enables teams to build deceptive proofs of concept that appear complete on day one but fail when confronted with production edge cases. Brooke Hopkins contrasts this with autonomous vehicle engineering at Waymo: achieving an initial 80% baseline capability occurred relatively quickly, but resolving the remaining 20% long-tail distribution required over six years of rigorous simulation and evaluation infrastructure. Shahid Quraishi observes that models frequently earn flawless offline benchmark scores yet fail during rollout because offline tests cannot capture real-world multidimensional complexity. Production datasets are non-stationary and subject to cyclical business shifts, such as holiday return surges in retail or annual enrollment periods in healthcare. Furthermore, third-party foundation model providers frequently release unannounced background updates that alter model outputs without warning. To counter these silent shifts, panelists emphasize replacing static, one-time offline benchmarks with continuous daily regression test suites and dynamic sliding-window evaluations that feed emerging edge cases back into the testing loop. Granular Multi-Agent Tracing and Root Cause Diagnosis (9:37) Traditional software observability relies on binary pass/fail indicators, HTTP error codes, and server metrics like CPU load and latency. In contrast, multi-agent generative systems rarely trigger hard system crashes; instead, composite workflows degrade silently when intermediate reasoning or tool invocations drift. Quraishi notes that diagnosing these failures requires capturing the exact prompts, input states, intermediate payloads, and outputs at every discrete sub-agent node, allowing engineers to trace backward through the execution graph. Tammy Dugan warns against the reflex to immediately swap foundation models or rewrite system prompts when an agent underperforms. Applying Andrew Ng's modular error analysis often reveals that the failure originated upstream within the retrieval-augmented generation (RAG) pipeline—either because the knowledge base lacked necessary ground truth or retrieval surfaced irrelevant context. When debugging complex prompt templates, Hopkins recommends automated prompt fuzzing, sentence deletions, and order permutations across conversational test batches to isolate which instructions actually steer agent behavior. Mitigating Operational Risk with Architectural Guardrails (4:02) Rather than deploying autonomous models as unconstrained, end-to-end automations, enterprise deployments require deliberate risk partitioning. In Cummins' "Smart PO" system, automated agents query ERP records to detect obsolete purchase orders, email suppliers to halt part manufacturing, and parse responses. However, Dugan highlights that Cummins intentionally maintained a human-in-the-loop boundary: demand planners retain exclusive authority to approve ERP updates, eliminating the severe operational risk of erroneous automated write actions. To safely validate candidate models against live conditions, Quraishi and Hopkins advocate shadow deployments (silent launching), replaying live production traffic through candidate models to benchmark latency, cost, and behavioral fidelity without returning outputs to end users. Hopkins also advises replacing brittle all-or-nothing failure escalations with creative recovery paths. Systems can deploy asynchronous batch jobs to clean ambiguous inputs—such as misheard contact numbers—or route low-confidence fields through lightweight human checkpoints before resuming autonomous downstream execution. Managing Trace Explosion and Probabilistic Drift (39:28) Unlike traditional predictive models that track fixed feature counts against definitive ground truth—such as Cummins' 90-day engine failure telemetry—large language models operate across vast, non-deterministic parameter spaces. In production, testing shifts from deterministic assertions to probabilistic risk envelopes, requiring organizations to define acceptable failure thresholds depending on task criticality (such as one error in a thousand for casual interactions versus one in a million for compliance workflows). To prevent engineering teams from being overwhelmed by voluminous trace data, Quraishi recommends segmenting operational scenarios into strict priority tiers, focusing human review on primary business workflows. Hopkins explains that modern observability platforms synthesize raw telemetry by pairing analytical agents with unsupervised machine learning clustering algorithms. This groups disparate log anomalies into actionable bug patterns, such as isolating how often an agent unexpectedly switches languages mid-conversation. The panel agrees on a pragmatic balance: utilizing commercial tools for data ingestion plumbing while retaining custom internal decision logic to govern how systems respond to anomalies. Transitioning to enterprise-grade agentic AI requires moving away from the assumption that models can operate autonomously without oversight. Long-term reliability depends on continuous regression testing, modular root-cause analysis, and keeping autonomous systems bounded by domain-expert human validation.

Devon Energy、康明斯（Cummins）、Coval 与 7-Eleven 的工程专家指出，企业级生成式智能体系统的核心挑战在于非确定性环境下的“决策级可靠性”；工程团队必须摆脱仅关注运行时间和延迟的传统二元监控，转向涵盖细粒度执行追踪、动态评估飞轮、人机协同边界及模块化根因分析的全面可观测性体系。 从二元监控转向多智能体链路追踪 (9:37) Devon Energy 的 Tristan Arbus 强调，生产系统追求的核心是“决策级可靠性”，即模型即便速度快、可用性高且统计指标稳定，仍可能产生错误的业务决策。7-Eleven 研发工程经理 Shahid Quraishi 进一步指出，传统软件监控主要依赖服务是否崩溃的二元判断与 HTTP 状态码，但在多智能体架构中，系统很少直接抛出崩溃异常，复合工作流却可能彻底背离业务目标。这类故障通常源于子智能体在微小的提示词或输入变动下产生不可预知的行为偏离，而非模型本身存在根本性缺陷。 为了解决此类非确定性故障，可观测系统必须完整记录工作流中每个子智能体节点的具体提示词、输入参数、中间状态转换和输出结果。Quraishi 举例说明，若一个多步工作流在第六步中断，工程师无法仅凭最终报错或延迟峰值定位故障，必须依赖执行图追溯能力反向排查上游推理偏差。康明斯（Cummins）的 Tammy Dugan 亦指出，自主智能体在跨平台网络交互中会引发复杂的级联网络效应，建立深度的端到端追踪机制是评估复合智能体决策有效性的技术前提。 克服长尾瓶颈：从演示幻象迈向评估飞轮 (7:24) Coval 首席执行官 Brooke Hopkins 观察到，生成式 AI 颠覆了经典软件从最小可行产品逐步演进的生命周期：团队能在短时间内构建出高度惊艳的单次概念验证原型（POC 幻象），但一旦扩展至数千万次真实交互，系统往往在复杂的长尾边缘场景中失效。Hopkins 援引 Waymo 自动驾驶开发经验指出，自动驾驶早在 2016 年就解决了 80% 的常规路况，但攻克剩余 20% 的长尾极端场景耗费了逾六年仿真与评估基础设施建设；智能体虽免受物理定律约束，同样面临攻克这 20% 边缘长尾的严峻考验。 针对模型上线后面临的数据漂移与环境变迁，Quraishi 指出第三方模型服务商常在无通知的情况下在底层调整权重，导致传统一次性离线基准测试完全失效。Hopkins 强调，高效工程团队需构建闭环的“持续改进飞轮”，摒弃涵盖所有假想场景的静态基准，转而构建动态滑动窗口评估机制。此外，多季节与周期性分布评估至关重要：正如自动驾驶车队必须经受万圣节奇装异服行人的检验，零售业的假期购物峰值与医疗保险行业的年度投保期同样要求评估数据集覆盖完整的业务周期波动，而非单点静态截面。 模块化根因诊断与提示词模糊测试 (13:49) 针对智能体表现欠佳时团队盲目更换底层模型或频繁微调系统提示词的行业反模式，Dugan 倡导采用吴恩达（Andrew Ng）提出的模块化错误分析框架。Dugan 指出，故障往往隐藏在检索增强生成（RAG）管道的上游环节，包括知识库本身缺失客观基准事实（人类专家同样无法作答）或检索机制召回了无关上下文；团队必须分步度量各环节失效率，在修改生成模型前优先修复上游检索与数据质量。Hopkins 补充道，工程团队初期无需被复杂的宏观指标大屏所困扰，深入人工审查 20 至 25 条端到端真实对话日志往往是最直接明晰的诊断路径。 面对提示词中各指令语句相互交织且难以单独量化影响的难题，Hopkins 提出利用自动化模糊测试（fuzzing）、语句排列组合与系统性 A/B 测试来评估指令依从度。通过算法自动剔除或重排单句指令并在大规模测试集上运行合成扰动，团队能精准量化各个提示词分块对最终输出的决定性影响。针对海量追踪数据带来的审查过载，Hopkins 指出 Coval 结合分析型智能体与传统无监督机器学习聚类算法，将零散追踪数据自动归并为具体缺陷主题（例如智能体在对话中意外切换语种），从而摆脱单条日志逐一排查的低效模式。 风险隔离与容错：人机协同与影子部署 (4:02) 在业务落地实践中，盲目追求全自主闭环往往伴随极高的合规与经营风险。Dugan 以康明斯的“智能采购订单（Smart PO）”系统为例：该系统虽能自主查询 ERP、邮件联络供应商并解析交付答复，消除计划员每单 2 至 3 小时的繁琐工作，但系统坚决保留了人机协同（Human-in-the-Loop）防线，必须经由业务人员审核授权后才回写 ERP 记录，并借助 MLflow 将生产异常 trace 持续注入回归测试集。Hopkins 与 Quraishi 进一步提倡通过“静默发布（影子部署）”将实时生产流量并行分流至候选模型，在不对最终用户展示的情况下横向对比延迟、成本与行为保真度，并引入 1% 的小比例人工抽检以弥补自动化评估盲区。 Arbus 将突发生产事故处理比作“犯罪现场”：首要任务是安全封控与风险隔离，随后才是根因分析。Hopkins 提醒团队摒弃“系统必须 100% 成功”的脆弱假设，建立创造性的容错机制，例如对低置信度字段引入轻量级人工核验，或利用异步离线批处理机制清洗听错的电话号码等异常数据，避免系统一遇报错就直接中断。针对非确定性漂移，专家们建议引入独立于主流程的“观察者智能体”，以全局视角捕捉逻辑跳跃或异常行为，同时在架构选型上采取平衡策略：利用成熟商业工具解决遥测管线等基础通道，将自研资源集中于企业专有的动态干预决策逻辑中。 与会专家一致认为，将生成式智能体转化为可靠企业生产力，本质上是工程思维由确定性单元测试向概率化质量边界管制的深刻范式转移；企业不应将大模型视为通用的银弹，而应在明确业务风险包络的前提下，坚持“AI 赋能”而非“超出人类专业审查能力盲目使用”，通过严密的透明度披露与确定性工具兜底，建立人机协同的可信系统。

## Background 本内容由 Ai4 于 2026 年 8 月 22 日发布在 YouTube 平台上。视频简介指出，本次圆桌汇聚了来自戴文能源（Devon Energy）、康明斯（Cummins）、7-Eleven 以及 Coval 的数据科学与工程专家，共同探讨在自主智能体（Agent）和 LLM 复杂化背景下，如何构建面向生产环境的 AI 可观测性与调试监控体系，涵盖漂移检测、决策链路追踪、根因分析及业务闭环反馈等核心议题。 ## TL;DR 圆桌深入剖析了从传统指标监控向以“决策可靠性”和“细粒度子步骤追踪”为核心的 AI 可观测性范式转移。嘉宾们指出，生成式 AI 虽然极易搭建概念验证（PoC），但量产落地的真正挑战在于攻克最后 10% 至 20% 的长尾极端案例，必须依托滑动窗口式的动态评估飞轮与影子模式（Shadow Mode）进行持续验证。面对多智能体流水线的级联失效，团队应采用模块化错误分析、提示词模糊测试（Prompt Fuzzing）以及旁路“观察者 Agent”与多级轻量人机协同（HITL）回退机制。此外，专家们强调必须划定确定性代码与概率模型的边界，保持调用透明度，谨防对 AI 产生超出专业能力的“过度信任”。 ## Atmosphere & style 整场讨论呈现出极其扎实、务实的工业界实战氛围，毫无浮夸的 AI 概念炒作，处处透露着来自产线与业务一线的工程经验。主持人以“犯罪现场封锁与溯源”等生动隐喻巧妙把控节奏，来自康明斯、7-Eleven 及具备 Waymo 自动驾驶背景的嘉宾们围绕真实落地的血泪教训（如制造供应链订单处理、万圣节长尾数据分布）展开直率交流。听众能够沉浸式感受到非确定性系统在规模化部署时的真实挑战，获得兼具方法论深度与落地操作性的架构心智启发。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=r8sSROyPndQ&t=4s) AI 可观测性超越了延迟与可用性等传统指标，其核心挑战在于监控具有非确定性特征的 Agent 在复杂业务与分布式决策层面的正确性。 - [04:02](https://www.youtube.com/watch?v=r8sSROyPndQ&t=242s) 在工业级 Agent 落地中，把握自主性与人机协同（HITL）的平衡至关重要，关键业务变更应保留人工把关，并将异常案例反哺回 Eval 测试集。 - [07:24](https://www.youtube.com/watch?v=r8sSROyPndQ&t=444s) 生成式 AI 的 PoC 演示极其容易，但系统规模化的真正分水岭在于攻克最后 10% 到 20% 的长尾边缘案例（Edge Cases），如同自动驾驶的发展成熟路径。 - [09:37](https://www.youtube.com/watch?v=r8sSROyPndQ&t=577s) 多智能体排障需要范式转变，必须完备记录每个子 Agent 在每一步的 Prompt、上下文交互及输出，以便通过单步逆向回溯（Backtracking）定位根因。 - [11:52](https://www.youtube.com/watch?v=r8sSROyPndQ&t=712s) 面对第三方模型静默更新与线上数据漂移，静态测试已经失效，团队应构建自动化回归套件与动态移动的“滑动窗口”评估飞轮。 - [13:49](https://www.youtube.com/watch?v=r8sSROyPndQ&t=829s) 智能体失效往往源于知识库检索缺失或上下文失真而非模型本身，通过模块化解耦和抽检真实对话样本进行错误分析（Error Analysis），比盲目更换模型更有效。 - [17:36](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1056s) 构建从公开基准粗筛、私有域压测、全拟真仿真到“影子模式”（Shadow Mode）静默上线的评测体系，可在不干扰现网业务的前提下安全验证新模型。 - [23:05](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1385s) 确定性代码应作为系统兜底与业务流转的基石，GenAI 仅充当路由大脑；产品必须向用户明确披露 AI 身份，避免因伪装人类引发信任危机。 - [25:24](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1524s) 警惕对语音与 Agent 系统的“过度信任”，严禁盲目追求一步到位承接 100% 话务量，应优先聚焦单一垂直场景做到极致，并严守专业能力边界。 - [28:51](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1731s) 传统 ML 依赖固定特征阈值和物理 Ground Truth 进行监控，而多智能体系统则需要从二元判定转向基于全链路 Trace 的概率化风险容忍度测试与人机抽检。 - [32:51](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1971s) 评估数据集必须充分覆盖全年度的季节性波动与代表性偏差（如节假日客流、极端装扮或退货潮），防止模型在脉冲场景下发生灾难性失效。 - [34:49](https://www.youtube.com/watch?v=r8sSROyPndQ&t=2089s) 应对非确定性 Agent 需设计多级回退机制（如异步数据清洗、轻量级人工微断点确认），并可引入独立的“观察者 Agent”进行实时旁路异常监测。 - [38:44](https://www.youtube.com/watch?v=r8sSROyPndQ&t=2324s) 面对海量 Trace 调用链数据，可结合传统 ML 聚类算法与评估 Agent 进行异常分群与聚合降噪，从而高效下钻分析多轮行为漂移。 - [41:32](https://www.youtube.com/watch?v=r8sSROyPndQ&t=2492s) 多 Agent 系统的失效通常源于输入上下文退化，采用提示词模糊测试（Prompt Fuzzing）与 A/B 回测可量化单个指令的边际贡献并剔除冗余。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=r8sSROyPndQ&t=4s) 圆桌讨论开场与嘉宾背景介绍 主持人 Tristan Arbus 开场介绍了本次圆桌讨论的主题——"Keeping Models Honest"（让模型保持诚实/可靠）。讨论的议题不仅涵盖可用性（uptime）和延迟（latency）等传统监控指标，还将深入探讨分布式故障分析、在线评估、信任机制等 AI 可观测性（Observability）核心领域。Tristan 首先分享了自己的背景：他此前做过机械工程师和钻井工程师，最近近十年在 Devon Energy 担任数据科学家，致力于在复杂的油气运营环境中构建并扩展数据驱动的解决方案。对他而言，最关心的可观测性挑战是**决策层面的可靠性（decision-level reliability）**——系统哪怕运行飞快、高可用且在统计学上稳定，但作出的决策依然可能是完全错误的。 随后，各位嘉宾依次作自我介绍并分享各自最关注的指标与挑战： - **Tammy Dugan**（Cummins AI 创新与网络风险缓解总监）：最关注 **Agentic AI（智能体 AI）** 的监控，特别是具有非确定性（non-deterministic）特征的软件在不同平台与网络间跳转交互时所产生的网络效应（network effects）。 - **Brooke Hopkins**（Coval 创始人兼 CEO）：其创立的 Coval 平台通过仿真（simulation）、可观测性与人机协同（Human-in-the-loop）测试来帮助企业扩展语音智能体（Voice Agents）。Brooke 曾就职于 Waymo 并领导评估基础设施团队，负责自动驾驶仿真开发工具，目前正将自动驾驶和机器人领域的经验迁移至金融、医疗等注重品牌和风险的 Fortune 500 企业。 - **Shahid Quraishi**（7-Eleven R&D 工程经理）：已在 7-Eleven 工作逾 7 年，目前最关注**多智能体模型（Multi-agent model）中的系统链路追踪指标（system trace metrics）**，即精准定位在每一步是由哪个子智能体（sub-agent）执行了具体操作。 --- ### [04:02](https://www.youtube.com/watch?v=r8sSROyPndQ&t=242s) 工业实战案例：康明斯的 "Smart PO" 智能体落地与人机平衡设计 Tammy 分享了康明斯落地的一个名为 **"Smart PO"（智能采购订单）** 的实际项目。在制造业中，面对庞杂的设备型号和繁复的配置选项，维持供应链一致性极具挑战。当生产线调整或客户取消订单时，需求计划员（demand planners）往往需要从 ERP 系统中排查未完结的采购订单（PO），手动逐一发邮件联络成千上万家供应商进行取消或修改，以避免库存积压；手动处理这些流程通常需要耗费 2 到 3 个小时。 Smart PO 系统通过 Agent 自动读取 ERP 数据、识别可取消订单、直接发邮件与供应商沟通并解析回复，再将提炼后的结果呈现给计划员。谈及如何平衡评估与业务价值，Tammy 强调最关键的架构决策是**把握 Agent 自主性与人机协同（Human-in-the-loop）之间的平衡**： ```mermaid flowchart TD accTitle: 康明斯 Smart PO 业务与评估流 subgraph S1["自动化执行层"] A["ERP 系统扫描"] --> B["识别潜在取消订单"] B --> C["Agent 自动联络供应商并解析回复"] end subgraph S2["人工验证与闭环"] C --> D["呈现建议给需求计划员"] D --> E{"人工审核确认"} E -->|"确认生效"| F["更新 ERP 采购状态"] E -->|"异常/错误"| G["捕获案例并扩充 Eval 评估集"] end ``` - 系统原本可以做成全自动闭环（直接自动改写 ERP），但由于采购变更影响巨大，最终选择保留人工把关，仅自动化最繁琐的联络与回复解析环节。 - 在上线前建立了严格的 Eval 评测集，上线后使用 **MLflow** 追踪记录工作流的各个步骤；一旦发现任何异常，立即将其反哺回评估测试集中以持续迭代。 --- ### [07:24](https://www.youtube.com/watch?v=r8sSROyPndQ&t=444s) 从 Demo 到量产：长尾效应与自动驾驶成熟度曲线的启示 Brooke 指出，当今 AI（尤其是语音 Agent）最大的认知误区在于**搭建一个单次运行成功的 PoC（概念验证）极其容易**。以往传统软件的 MVP（最小可行产品）往往粗糙并逐步完善，而如今的生成式 AI 在第一天就能看起来像一个功能完备的成品。真正的分水岭在于系统扩展时的稳定性——能否承受数千万次对话的冲击，以及能否妥善处理那**最后 10% 到 20% 的长尾边缘案例（edge cases）**。 Brooke 类比了自动驾驶的发展路径：Waymo 在 2016 年就已经有自动驾驶汽车能够上路行驶（完成了前 80% 的能力），但随后又花了整整 6 年时间才真正实现完全自动驾驶的落地。AI Agent 虽然无需面对现实物理世界的摩擦与定律，迭代速度会更快，但攻克长尾 20% 复杂场景的挑战完全一致。Shahid 也对此补充道：模型在离线离线测试（offline mode）中往往表现完美，但上线实测时会面临海量指标的验证；"模型本身没有错"，错在真实商业场景的复杂度和长尾问题需要更长周期去磨合。 --- ### [09:37](https://www.youtube.com/watch?v=r8sSROyPndQ&t=577s) 可观测性范式转移：多智能体系统的细粒度步骤追踪 在探讨传统监控与 Agent 时代可观测性的本质区别时，Shahid 强调了系统维度的根本性转变： | 维度 | 传统软件监控 (Traditional Monitoring) | 多智能体可观测性 (Multi-Agent Observability) | |:--- |:--- |:--- | | **监控目标** | 关注系统存活、延迟、CPU 利用率等基础设施指标 | 关注决策正确性、非确定性行为及多步骤协同逻辑 | | **信号形态** | 明确的二元状态（Pass 或 Fail / 错误码返回） | 复合型上下文（提示词 Prompt、中间推理、子智能体动作） | | **排障方式** | 查看聚合仪表盘（Dashboard）与服务日志 | 单步回溯（Backtracking）与跨 Agent 链路追踪 | 在多智能体流水线中，如果一个包含 6 步的工作流在第 6 步发生失败，仅凭最终结果无法定位根因。可观测性系统必须完备记录每一个子智能体在每一步的**输入 Prompt、上下文交互以及确切输出**，工程师才能据此逆向回溯（backtrack），精准定位到底是哪个环节的子 Agent 出现了认知偏离或执行差错。 ### [11:52](https://www.youtube.com/watch?v=r8sSROyPndQ&t=712s) 智能体评估的飞轮效应与动态测试集 Shahid Quraishi 率先指出了当前生产环境中模型运维面临的一个普遍痛点：在多轮交互或依赖单点输出的场景下，模型的输入指标和 Prompt 提示词会随着时间推移不断发生漂移。更为棘手的是，许多底层调用的是第三方商业模型服务，供应商在后台静默更新模型权重或调整接口时根本不会发出任何预警。这就导致传统的“只在上线前跑一次测试”的静态验证流程彻底失效。Shahid 强调，团队必须建立一套属于自己的自动化回归测试套件（Regression Test Suite），哪怕在日常运行中，每天也必须高频次、持续性地运行这些测试套件，以此捕捉模型输出的变化与关键指标的偏移。 Brooke Hopkins 紧接着对这一观点进行了升华。他指出，顶尖的 AI 团队早已不再将评估视作单一时间点上的静态门禁，而是致力于构建一个自我迭代的“改进飞轮”（Flywheel of Improvement）。在传统软件工程中，“如果你把一个按钮做好了，它基本上就能一直正常工作”，除非底层依赖包发生重大升级；然而智能体模型截然不同——底层数据在变、用户行为在变、模型本身也在动态漂移。最成熟的团队会从生产流量中不断捕捉失败或边界案例，反哺回智能体的迭代循环，持续刷新其测试用例、评估指标和优化目标。 Brooke 形象地将这一机制比作自动驾驶系统的研发逻辑：自动驾驶团队永远在搜寻新的长尾极端工况（Edge Cases）并将其注入训练与测试管道。因此，AI 系统的评估集不应该试图覆盖“全天下所有的用例”，而应该是一个动态移动的“滑动窗口”（Sliding Window），核心在于明确“我们刚刚掌握了哪些能力，目前还有哪些缺陷”，并推动这扇窗口向更高阶的能力边界持续推移。 ```mermaid flowchart TD accTitle: 智能体持续改进飞轮 subgraph Production ["生产运行层"] A["在线生产流量与真实交互"] --> B["捕捉生产中的长尾极端案例"] end subgraph Evaluation ["评估与迭代层"] B --> C["动态更新测试用例库"] C --> D["滑动窗口评估指标"] D --> E["Prompt / 管道迭代优化"] E --> A end ``` --- ### [13:49](https://www.youtube.com/watch?v=r8sSROyPndQ&t=829s) 复合智能体系统的根因分析与错误分析（Error Analysis） 主持人 Tristan Arbus 将话题引向多智能体系统的根因分析（Root Cause Analysis）。随着工作流中串联的子智能体越来越多，决策链路的追踪与状态解耦变得极为错综复杂。当一个深埋在分布式流程底层的系统发生幻觉或崩溃时，团队该如何在实时环境下快速定位根因？ Shahid 结合自身的实战经历说明，故障的罪魁祸首往往不是单纯的“模型不够聪明”，而是系统与数据交互的方式出了差错。由于输入指标或 Prompt 不断漂移，数据维度的细微扰动会导致某个子智能体突然产生反常反应。在这种情况下，排查哪个环节开始失控如同大海捞针。他认为最有效的破局方式是在整个系统层面建立严格的端到端可追溯性机制（Traceability），为每一个子智能体的输入、输出与上下文进行全量追踪，精确监控各个节点在面临输入扰动时的即时反应。 Tammy Dugan 进一步点出了工程师们普遍容易陷入的认知误区：很多团队一看到智能体表现不符合预期，下意识的第一反应就是“是不是我选错模型了？我是不是该换个模型？”但现实往往与模型本身无关。以知识库驱动的智能体为例，模型可能在严格按照指令执行，但知识库底层压根没有包含回答该问题所需的事实材料（甚至连人类专家看着都无法作答），或者 RAG（检索增强生成）检索模块调取了错误的文档片段。 Tammy 分享了自己最初从吴恩达（Andrew Ng）那里学到的经典机器学习方法论——错误分析（Error Analysis）： - **模块化解耦排查**：不要一开始就从头到尾进行黑盒调优，而是把流水线拆解为若干独立模块（如文档检索、上下文注入、模型推理）。 - **逐个环节排查错误分布**：深入观察具体样本是在哪个环节掉链子的。如果发现根因出在检索召回率上，那就集中精力优化检索，而不是盲目地去改写一千遍 Prompt，或者去试用 10 个不同的新模型。 - **由小见大解决瓶颈**：先搞清楚系统在哪里犯错最多，再有针对性地逐步解决。 Brooke Hopkins 对此高度赞同，并给出了一个立竿见影的落地建议：很多团队在项目启动时热衷于搭建宏大、完备的全局评估指标看板，试图一口吃成胖子。但最接地气、最有效的起步动作，往往是团队围坐在一起，老老实实逐条抽检 25 场真实的对话记录（Error Analysis）。直接观察底层原始数据，往往能最快打破认知盲区。 --- ### [17:36](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1056s) 多层级评测体系与静默上线（Shadow Mode） 面对离线基准测试（Offline Benchmarks）与真实在线生产表现严重脱节的行业顽疾，Tristan 询问 Brooke 如何构建持续在线评估闭环，在不干扰现网业务的前提下安全捕捉一致性、鲁棒性与公平性的下滑。 Brooke 阐述了 Coval 采用的多层级评测架构： 1. **公开通用粗筛（Coarse Knob）**：团队维护了开源基准平台（benchmarks.coval.dev），针对市面上所有主流的语音模型（TTS、STT、语音到语音等）进行多维度横评（包括指令遵循能力、响应延迟等）。这一步的作用仅仅是粗粒度筛选候选范围。 2. **私有领域数据本地压测**：从公开基准中选出表现最好的 Top 3 到 Top 5 模型，放入企业自身的专有数据中运行本地基准测试。Brooke 举例道：“一个模型懂不懂法律或医疗知识毫无意义，如果你正在为 7-Eleven 构建客服智能体，或者只是为了回答‘某家门店是否正在营业’，那么它处理特定品牌名称、特定零售场景任务的准确率才是唯一重要的指标。” 3. **端到端管道全仿真**：将候选模型串联进真实的业务流中（挂载实际的 RAG 检索管线），在语音场景下通过全拟真语音仿真环境，测试系统在全链路上的实际表现。 4. **静默发布至生产验证**：最后将模型部署上线，核实前面的层层测试是否与真实生产流量下的表现保持一致。 ```mermaid flowchart LR accTitle: 智能体多层级选型与评估流程 A["公开全网 Benchmark<br/>(粗筛 Top 3~5)"] --> B["私有域数据测试<br/>(品牌/垂直任务验证)"] B --> C["端到端管线仿真<br/>(挂载真实 RAG / 语音)"] C --> D["静默上线 / 影子模式<br/>(生产流量回放与对比)"] D --> E["全量发布与人在回路"] ``` Shahid 补充了一个经过实战检验的最佳实践——“影子模式”（Shadow Mode / 静默上线）：当准备引入新模型时，让新模型实时消费生产环境中的真实输入数据，但不让它直接对客户端产生对外动作，而是将所有预测结果沉淀至后台日志。团队随后可以将新模型在真实流量下的表现与当前线上模型进行全量指标对比。此外，在全量切换前，务必让业务方团队亲自验收：“它是否切实解决了预期的业务痛点？处理效率是否比当前方案更快？” Brooke 对静默上线策略表示“100% 认同”，指出 Coval 在部署新模型与仿真器时也完全采取这种先静默比对、再逐步灰度的方式。同时，他极力主张引入“人在回路”（Human-in-the-Loop）：哪怕算法自动打分再完备，如果不符合人类评审员的主观常识判断，评估体系便形同虚设。系统应当抽取少量真实生产流量（例如 1% 的流量或特定仿真会话）由人工介入复核，以捕捉那些自动化测试集尚未收录的前沿极端用例。Shahid 也表示，当业务方得知全流程中存在人类兜底复核机制时，对整个 AI 系统的信任度和信心都会成倍提升。 --- ### [21:22](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1282s) 犯罪现场隐喻与模型可解释性、可信度的权衡 在进入关于可信度与可解释性的探讨前，Tristan 分享了一个他个人极其推崇的故障处置隐喻：**AI 系统的突发故障在很大程度上就像一个“犯罪现场”（Crime Scene）**。 > “警察赶到犯罪现场时做的第一件事，永远是封锁现场（Secure the Area），随后才会展开调查取证。同理，在复杂系统发生故障时，遏制影响扩散（Containment）与深挖根本原因（Root Cause Analysis）是完全不同的两码事。” 在理清系统止血与根本溯源的区别后，讨论过渡到模型的可解释性与透明度。Tristan 指出，为了让系统具备可解释性往往需要付出巨大的计算开销（Computational Overhead），而在集成专有系统与第三方闭源黑盒模型时，复杂度更是成倍攀升。企业内部该如何在工程团队、业务利益相关者与最终用户之间建立透明度与信任？ Tammy Dugan 在回应时指出了当前行业存在的盲目跟风现象：业界现在对 AI、尤其是生成式 AI（Generative AI）的热情过于高涨，人们倾向于把大模型当成银弹，试图将其生搬硬套到任何业务场景中。然而在她看来，构建可靠系统首先需要的是系统的一致性与稳定性；在很多时候，最高效、最稳健的解决方案恰恰是那些看起来“枯燥无味”的传统方法…… ### [23:05](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1385s) 确定性代码与非确定性 AI 的结合：透明度与信任危机 在构建混合系统时，绝大部分基础架构依然由普通软件代码（Vanilla Software）构成，无论这些代码是由人类编写还是由 AI 生成，其底层运行的都是确定性的常规逻辑。生成式 AI（GenAI）在其中更像是“蛋糕上的糖霜”（the icing on the cake）或是执行工具路由（Tool Routing）的大脑。然而，一旦引入生成式 AI，系统便接入了不可完全控制的非确定性、概率性特征，必须在此之上建立严密的防护栏（Guardrails）与评估机制。 建立系统信任的关键在于两点：其一是审慎评估在何时、何处真正需要引入具备非确定性特征的生成式 AI；其二则是保持极致的透明度（Transparency）。嘉宾分享了一段切身经历：在给当地汽车修理厂打电话预约保养时，电话接通后与客服沟通片刻才突然意识到对方是一个 AI，这种未提前告知的使用方式让人倍感被冒犯与不安，甚至让人怀疑是否还要继续与这家车行合作。嘉宾强调，产品必须向用户坦诚何时在调用 AI，切忌试图“欺骗用户以为对方是真人”。此外，由于第三方模型通常属于黑盒，难以直接验证其可解释性，最有效的验证途径是严密监控模型对具体输入和输出的反应指标，结合产品与业务团队的持续评估，确认模型是在按预期工作还是在大多数场景中出现幻觉。 ```mermaid flowchart TD accTitle: AI 与确定性软件的混合架构与信任机制 subgraph CoreEngine["系统核心架构"] A["确定性底层软件代码<br/>(Vanilla Software Logic)"] --> B["生成式 AI 路由大脑<br/>(GenAI Tool Routing)"] end subgraph GuardAndEval["防护与可解释性验证"] B --> C["防护栏与评估机制<br/>(Guardrails & Evaluations)"] C --> D["输入/输出响应指标追踪<br/>(I/O Behavior Monitoring)"] D --> E["产品与业务团队业务验证<br/>(Business Alignment)"] end subgraph TrustFramework["用户信任构建"] F["场景审慎评估<br/>(Thoughtful AI Adoption)"] G["明确透明度披露<br/>(Explicit AI Disclosure)"] end CoreEngine --> TrustFramework ``` - **确定性与概率性的边界**：常规代码负责兜底与业务流转，GenAI 仅作为决策路由的大脑，不可盲目放权。 - **透明度原则**：严禁在语音或交互界面伪装人类，提前告知 AI 身份有助于用户在系统发生异常行为时建立合理的心理预期。 - **黑盒模型验证路径**：针对不透明的第三方模型，侧重观测输入与输出的端到端反应，量化幻觉率与业务助益程度。 --- ### [25:24](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1524s) 警惕“过度信任”：语音 AI 落地陷阱与专业能力边界 随着智能体（Agentic Systems）与新型大模型越来越逼真地拟合人类行为，人们往往会产生“过度信任”（Over-trusting）。当前的语音 AI 技术已经能够实现极为逼真的人类语调，如果系统在交互中不明确声明自身为 AI，极易误导用户。一个优秀的范例是 Claude Code 等工具，它会在执行交互时清晰透明地展示“当前正在做什么、为什么这么做”，使用户在理解其能力边界的同时建立起健康的信任，而不是将其视为不可测的黑盒。 过度信任的另一个典型陷阱在于企业盲目追求让语音 AI 解决所有问题，试图一步到位承接 100% 的呼叫中心话务量。这种“全面平庸”的策略极易引发灾难性的用户挫败感。更稳健的做法是聚焦于极其具体的垂直场景并做到极致，再逐步横向拓展。与会者进一步指出，AI 聊天机器人和 LLM 的核心价值在于作为“赋能工具”（Enabler），但许多人将其推向了超出自身评估能力（Level of Competency）的领域。当前不少 AI 厂商极力推销 AI 能放大个人能力的理念，导致部分从业者越界使用却无法判断生成结果的优劣。团队必须保持敬畏，尊重数学、科学、编程等垂直领域的深厚专业壁垒，让领域专家深度参与校验；同时需根据用户群体划分可观测性深度——一位物理学研究者对系统可观测性的诉求，显然与普通电商购物者有着本质不同。 - **Claude Code 式透明度**：在执行每一步推理和工具调用时实时呈现意图与行为路径，打破黑盒效应。 - **场景落地策略**：严禁直接上线处理 100% 话务量的通用 Voice AI，应优先聚焦单一特定场景打造高完成度闭环。 - **能力边界法则**：使用者必须具备评估 AI 输出质量的专业能力，避免在缺乏领域知识（如复杂科学计算、底层代码架构）的情况下盲目采纳生成内容。 --- ### [28:51](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1731s) 数据漂移与模型衰退：从传统预测模型到复杂智能体系统的监控策略 在讨论系统漂移（Data Drift）与自动化回滚/重训触发机制时，传统机器学习模型与多智能体大模型展现出截然不同的监控范式。数据漂移可能由用户行为迁移、供应链波动或底层物理硬件变动引发： | 维度 | 传统机器学习模型 (Traditional ML) | 大语言模型与多智能体系统 (LLMs & Agentic Systems) | |:--- |:--- |:--- | | **特征空间与复杂度** | 包含 100 至 200 个具体指标，特征分布清晰 | 拥有数十亿级组合与非线性路径，系统复杂度极高 | | **可观测性实现手段** | 直接监控输入/输出特征漂移，对准确率、敏感度设固定阈值 | 依赖全链路调用追踪（Traceability），定位具体受影响的子智能体 | | **测试与监控范式** | 二元判定（通过 / 失败，Pass/Fail） | 概率化事件分析（如合规故障率是 1/1,000 还是 1/1,000,000） | | **真实标准 (Ground Truth)** | 具有确定性的业务落地结果（如康明斯 90 天发动机故障预测） | 结合 LLM-as-a-Judge 与人机协同（Human-in-the-Loop）抽样校验 | 康明斯（Cummins Inc.）分享了传统工业预测模型的观测实践：在进行发动机 90 天提前故障预测时，系统拥有完全确定的物理结果（发动机最终是否损坏）。团队可以通过追踪灵敏度（Sensitivity）、特异度（Specificity）和准确率（Accuracy）等核心指标，在一个月的数据沉淀周期内精确比对预测与真实表现，从而设定硬性的退化阈值。而在多智能体系统中，监控必须从传统的“二元通过/失败”转向“风险容忍度概率测试”，例如精准评估在数十万次对话轨迹中出现合规失败的概率，并通过人机协同（Human-in-the-Loop）机制持续对异常流量进行抽检。 --- ### [32:51](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1971s) 应对季节性波动与数据集代表性偏差（从 Waymo 万圣节到电商退货） 在系统面临季节性波动（Seasonality）与环境分布偏移时，模型评估指标很难在全周期保持恒定。为此，团队在引入新指标时，必须以正在生产环境中运行的旧模型为基准（Benchmark）进行对比。当工作流依赖子智能体执行关键业务模块时，必须为每个子智能体的行为划定明确的确定性边界（Action Boundaries），并依托训练团队构建的严密回归测试套件（Regression Suite）进行持续验证。 更为关键的是审视数据集的代表性偏差（Representative Bias）。评估数据集绝不能仅仅截取某一特定时间切片，而必须覆盖完整年度的周期性分布。嘉宾举了几个极具代表性的跨行业案例： - **自动驾驶领域（Waymo）**：在万圣节期间，街头会出现大量身穿幽灵、小猫等奇装异服的儿童与行人，自动驾驶感知与决策系统必须在训练与验证集中充分覆盖此类非典型长尾场景。 - **医疗与健康保险**：保险理赔与就医行为存在明显的季度周期性，保单结算高峰期的流量模式与平日迥异。 - **电商零售行业**：节假日促销旺季伴随着订单暴增与节后海量退货潮，数据流分布具有强烈的脉冲特征。 系统设计者必须确保测试集覆盖全年的时间跨度及相关环境参数，防止模型因季节性数据偏移而在特殊节点产生灾难性失效。 ### [34:49](https://www.youtube.com/watch?v=r8sSROyPndQ&t=2089s) 智能体系统的观测演进、多层回退与“以 Agent 监控 Agent” 随着 AI 系统向无需或极少需要人类干预的自主智能体（Agentic Systems）演进，传统的护栏（Guardrails）和遥测方式必须经历根本性变革，以应对分布式自动化工作流中的涌现行为（Emergent Behaviors）与级联失效（Cascading Failures）。很多 AI 项目之所以在立项后惨遭废弃，核心原因在于系统在离线测试的特定子集数据上表现完美，一旦进入充满海量长尾组合的真实生产环境便迅速崩溃。面对这一挑战，工程团队首先需要厘清哪些业务场景可以完全交付给 Agent 自动化流转，哪些核心路径必须嵌入“人机协同”（Human-in-the-Loop, HITL）作为校验节点。 与单纯追求“100% 成功率”相比，更为关键的是构建富有创意的回退机制（Fallback Systems）。业界以往常见的做法往往只是机械地将错误抛出并“上报给人工客服”，但成熟的架构应当具备更多样化的工具箱：例如设立离线批处理任务，专门对前端语音识别容易听错的电话号码进行清洗与纠偏；或者引入极轻量的人工微干预，仅让人类确认姓名或号码无误后，立即将控制权交还给 Agent 继续执行下游任务。 针对生成式 AI 固有的非确定性（Non-deterministic）与随机性（Stochasticity），单靠静态的 Prompt 护栏很难拦截百万分之一极端扰动组合导致的系统“脱轨”。一种前沿且极富创意的思路是部署专门的“观察者 Agent”（Observing Agent）进行实时旁路监测。它就像一个站在系统外的人类专家一样，专门捕捉那些不合常理的异常行为并及时拉响警报。在 AI 可观测性领域的“自研还是外购”（Build vs. Buy）抉择上，标准化的指标采集与数据管道（Plumbing）完全可以借助成熟的第三方产品，而基于自身业务数据和特有反应逻辑的决策模型，则应当由团队内部自主构建。 ```mermaid flowchart TD accTitle: Agentic Monitoring and Creative Fallback Architecture accDescription: Flow showing how Agentic tasks are monitored by a spectator agent and handled through tiered fallback paths. subgraph CoreFlow ["核心业务智能体流"] In["输入数据 / 提示词"] --> Agent["自主执行 Agent"] Agent --> Out["业务输出"] end subgraph ObservabilityLayer ["智能观测层"] Agent -.->|"实时追踪数据 (Traces)"| ObsAgent["旁路观察者 Agent\n(捕捉异常/偏离行为)"] ObsAgent -->|"判定严重异常"| Trigger["触发防护策略"] end subgraph FallbackStrategies ["多级分流与回退机制"] Trigger --> Decision{"故障/风险分类"} Decision -->|"微小格式错误"| BatchJob["离线批处理修复\n(如实体/号码清洗)"] Decision -->|"关键字段确认"| MicroHITL["轻量级人工断点校验\n(确认后交还 Agent)"] Decision -->|"致命级联故障"| HumanEscalation["全面人工接管"] end ``` - **离线与线上的鸿沟**：模型在受限离线数据集上的高表现无法直接推导至生产环境，必须针对不同业务风险等级设计人机协同分流策略。 - **回退机制创新**：避免一刀切的人工接管，善用异步数据修正任务与人机断点确认，维持整体流程的高吞吐。 - **观察者模式（Agent-on-Agent）**：利用独立的监测 Agent 识别极端扰动下产生的异常逻辑，弥补预设硬编码规则的不足。 - **自研与采购边界**：底层数据抓取与遥测管道使用通用工具，核心决策与自愈反应逻辑由自研控制。 --- ### [38:44](https://www.youtube.com/watch?v=r8sSROyPndQ&t=2324s) 非确定性范式转变与海量追踪数据的规模化分析 软件工程正经历从“确定性输入输出”向“评估非确定性系统”的范式转移，这也是当前 AI 可观测性最引人入胜却又充满挑战的未解课题。针对全链路可观测性（Full Traceability）带来的海量追踪数据如何规模化评估、是否需要使用“LLM as a Judge”或对抗性 Agent 的问题，首要策略是对使用场景进行优先级分桶（Bucketing），切忌试图用一套系统一次性解决全量长尾问题。团队应根据核心指标编写确定性的评估测试套件，在引入人机协同流水线验证跑通后，再逐步向边缘用例拓展。 在底层基础设施已能轻松捕获海量 Trace 的今天，真正的痛点在于如何从庞大的数据汪洋中提炼出聚合洞察。在 Coval 的工程实践中，团队采用了评估 Agent 与传统机器学习聚类（Clustering）相结合的方案：利用无监督模型对调用链中的异常模式与失败类型进行自动归类分群，同时提供深度聚合分析报表。这种能力让工程师能够精准下钻排查特定隐蔽缺陷，例如快速定位并量化“多轮对话中 Agent 意外切换语言”的发生频率与根本诱因。 - **优先级分桶演进**：将海量场景按业务影响度排序，优先打通高优先级测试集与 HITL 闭环，再有序覆盖长尾场景。 - **混合分析架构**：结合传统 ML 聚类算法（聚合归类相似 Trace 错误）与 LLM 评估 Agent（细粒度语义分析），实现规模化降噪。 - **聚合下钻报表**：支持对细微行为漂移（如跨语言混淆、指令顺从度微跌）进行宏观量化与微观链路排查。 --- ### [41:32](https://www.youtube.com/watch?v=r8sSROyPndQ&t=2492s) 模型为何“永远没错”与提示词模糊测试（Fuzzing）评估 针对此前讨论中提到的“模型永远没有错”（The model is never wrong），现场嘉宾进行了深度澄清：在多步骤、多智能体协作（Multi-agent Pipelines）系统中，单个模型节点仅是严格根据输入的 Prompt 参数和上下文数据产生对应的计算输出。如果子 Agent 给出了荒谬的结果，其根源往往是上游传递的上下文数据失真或阶段性提示词存在偏差（即典型的 Garbage In, Garbage Out）。因此，全链路追踪的核心价值就在于回溯每一个子步骤接收的具体输入与 Prompt，从而准确判定是哪一个环节导致了上下文污染。 在面对生产级提示词的精细化评估时，工程师常常面临 Prompt 中各语句相互交织、部分语句毫无作用而部分语句影响巨大的困境。针对这种句级交叉影响的评估，业内最有效的实战方法是采用“提示词模糊测试”（Prompt Fuzzing）与 A/B 测试。通过对 Prompt 内部的句子进行排列组合、随机删减、变量注入，并在大规模对话数据集上批量回测，团队可以量化衡量每句话对指令遵循度（Instruction Adherence）以及端到端对话成功率的真实边际贡献，从而剔除冗余修饰，打造高度稳健的生产级 Prompt 架构。 - **溯源定责逻辑**：多 Agent 系统的失效通常源于输入上下文与 Prompt 的退化，通过全链路追踪可精准定位上下文传递链中的断裂点。 - **提示词模糊测试（Prompt Fuzzing）**：通过打乱语序、增删语句等置换测试，系统化量化单个指令对最终结果的边际效应。 - **指令遵循度量化**：利用批量 A/B 回测检验 Prompt 在不同对话流中的指令约束力，消除冗余并提升模型执行鲁棒性。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=r8sSROyPndQ&t=4s) 系统哪怕运行飞快、高可用且在统计学上稳定，但作出的决策依然可能是完全错误的。 — Tristan Arbus > [07:24](https://www.youtube.com/watch?v=r8sSROyPndQ&t=444s) 搭建一个单次运行成功的 PoC（概念验证）极其容易。 — Brooke Hopkins > [11:52](https://www.youtube.com/watch?v=r8sSROyPndQ&t=712s) 如果你把一个按钮做好了，它基本上就能一直正常工作。 — Brooke Hopkins > [13:49](https://www.youtube.com/watch?v=r8sSROyPndQ&t=829s) 是不是我选错模型了？我是不是该换个模型？ — Tammy Dugan > [17:36](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1056s) 一个模型懂不懂法律或医疗知识毫无意义，如果你正在为 7-Eleven 构建客服智能体，或者只是为了回答‘某家门店是否正在营业’，那么它处理特定品牌名称、特定零售场景任务的准确率才是唯一重要的指标。 — Brooke Hopkins > [21:22](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1282s) 警察赶到犯罪现场时做的第一件事，永远是封锁现场（Secure the Area），随后才会展开调查取证。同理，在复杂系统发生故障时，遏制影响扩散（Containment）与深挖根本原因（Root Cause Analysis）是完全不同的两码事。 — Tristan Arbus > [22:25](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1345s) 最稳健的解决方案恰恰是那些看起来“枯燥无味”的传统方法…… — Tammy Dugan > [25:24](https://www.youtube.com/watch?v=r8sSROyPndQ&t=1524s) 在执行每一步推理和工具调用时实时呈现意图与行为路径，打破黑盒效应。 — Brooke Hopkins > [41:32](https://www.youtube.com/watch?v=r8sSROyPndQ&t=2492s) 模型永远没有错。 — Shahid Quraishi ## Mentioned Tristan Arbus, Tammy Dugan, Shahid Quraishi, Brooke Hopkins, Devon Energy Corporation, Cummins Inc., 7-Eleven, Coval, Waymo, MLflow, Claude Code, Andrew Ng, Smart PO ## Video description Tristan Arbus - Senior Data Scientist, Devon Energy Corporation Tammy Dugan - Chief Data Scientist, Cummins Inc. Shahid Quraishi - Engineering Manager, R&D, 7-Eleven Brooke Hopkins - CEO / Founder, Coval As AI systems grow more complex—spanning multimodal models, pipelines, and autonomous agents—the need for observability has never been greater. This panel brings together ML engineers, data scientists, and infrastructure leaders who are building the next generation of AI monitoring and debugging tools. Panelists will explore best practices for tracking model performance in production, detecting drift and bias, tracing decision flows, and correlating signals across distributed systems. Expect candid insights into how observability differs from traditional monitoring, what it takes to achieve real-time visibility into LLMs and agentic systems, and how teams are designing feedback loops that keep models adaptive, transparent, and aligned with business outcomes. [Source](https://www.youtube.com/watch?v=r8sSROyPndQ)

[0:04] Tristan Arbus: 系统哪怕运行飞快、高可用且在统计学上稳定，但作出的决策依然可能是完全错误的。 [7:24] Brooke Hopkins: 搭建一个单次运行成功的 PoC（概念验证）极其容易。 [11:52] Brooke Hopkins: 如果你把一个按钮做好了，它基本上就能一直正常工作。 [13:49] Tammy Dugan: 是不是我选错模型了？我是不是该换个模型？ [17:36] Brooke Hopkins: 一个模型懂不懂法律或医疗知识毫无意义，如果你正在为 7-Eleven 构建客服智能体，或者只是为了回答‘某家门店是否正在营业’，那么它处理特定品牌名称、特定零售场景任务的准确率才是唯一重要的指标。 [21:22] Tristan Arbus: 警察赶到犯罪现场时做的第一件事，永远是封锁现场（Secure the Area），随后才会展开调查取证。同理，在复杂系统发生故障时，遏制影响扩散（Containment）与深挖根本原因（Root Cause Analysis）是完全不同的两码事。 [22:25] Tammy Dugan: 最稳健的解决方案恰恰是那些看起来“枯燥无味”的传统方法…… [25:24] Brooke Hopkins: 在执行每一步推理和工具调用时实时呈现意图与行为路径，打破黑盒效应。 [41:32] Shahid Quraishi: 模型永远没有错。

[0:04] Tristan Arbus: I care a lot more that we're able to make good decisions with our systems. You know, they can be fast, available, statistically stable and still be wrong. [0:04] Tammy Dugan: network effects of non-deterministic software [7:24] Brooke Hopkins: you don't have to deal with physics [11:52] Brooke Hopkins: it will continue working pretty much... until you need some package upgrade. [13:49] Tammy Dugan: Which model should I pick? [17:36] Tristan Arbus: To me it's a lot like a crime scene. The first thing that the police do when they show up to a crime scene... is they secure the area, and then they do their investigation. The point being that containment and root cause analysis are different things. [22:00] Tammy Dugan: the boring traditional [23:05] Tammy Dugan: icing on the cake [25:24] Tammy Dugan: There's a difference between using AI as an enabler and using AI beyond your level of competency to evaluate the quality of the output. [32:51] Brooke Hopkins: How is your data set representative of not just like one day in time but actually the entire year or any other parameters that might be relevant? [37:02] Brooke Hopkins: off the rails [37:02] Brooke Hopkins: Hey, that's weird [41:32] Shahid Quraishi: model is never wrong

[0:04] I'm Tristan Arbus. I'm going to be moderating  today's discussion on keeping models honest. We're [0:10] going to be talking about number of different  topics including traditional signals like uptime [0:16] and latency uh but also uh distributed failure  analysis and online evaluation trust um all sorts [0:25] of stuff. So um after the discussion, we will  take some audience questions near the end if we [0:31] have time. We'll see how the discussion goes. Um  we're going to start the panel with just a quick [0:37] round of introductions. Um I'd like in addition to  your name, role. Uh maybe also we could all share [0:45] a specific model metric or observability challenge  kind of at the top of our list or just that we're [0:53] thinking a lot about these days. Uh me again, I'm  Tristan Arvis. Uh my background kind of sits at [0:59] the intersection of engineering, applied AI, and  enterprise data science initiatives. Um I in a [1:08] past life was a mechanical engineer and a drilling  engineer, but most recently I spent nearly a [1:13] decade as a data scientist at Devon Energy uh  building and scaling datadriven solutions for [1:21] complex oil and gas operational environments. And  I'd say the challenge I care most about would be [1:28] decision level reliability. I care a lot more  that we're able to make good decisions with our [1:34] systems. um you know they can be fast, available,  statistically stable and still be wrong. Um so [1:41] from there let's just go down the line here.  Tammy, let's start with you. Hello everyone. [1:47] My name is Tammy Dugan and I'm the director  of AI innovation and AI cyber risk mitigation [1:52] at Cummins. And I think the thing that I think  about most in terms of observability, especially [2:00] with Agentic AI, is being able to monitor what  the agents are doing. um especially the network [2:05] effects of non-deterministic software and how  it bounces around through different platforms [2:10] and through the network. Amazing. Hey everyone,  I'm Brooke Huffkins. I'm the founder of Koval [2:17] and we build a platform to help you scale voice  agents. So through simulation, observability, [2:22] and then human loop testing, we help to create  self-improving agents uh specifically for voice. [2:28] And my background is from Whimo. So, I led our  evaluation infrastructure team at Whimo and my [2:32] team was responsible for all of our developer  tools for simulation. And now we're taking all [2:37] of the learnings from self-driving and robotics  and using that to help Fortune 500s across the [2:41] board through financial services, healthcare, um,  and anyone that cares about their brand and risk [2:47] to help scale their voice agents. Hey, hi. Uh, my  name is Shahid. Uh, I'm an engineering manager at [2:53] 7-Eleven R&D. Um I've been working at CNLN for  last 7 plus years and one of the thing that I [3:03] in the observability space that I'm really  interested is in uh system trace uh metrics [3:09] when you're working on a system where you have a  multi- aent model to identify like which sub agent [3:15] have taken action at which step. So that's one  of the area that I'm really interested on. Okay, [3:23] great. Uh so before we get into like the real meat  and potatoes, let's just talk a little bit about [3:30] uh what kind of world we're operating in and what  kind of systems we're talking about. Uh so Tammy, [3:37] I'm going to start with you on this one. Um  take a deployed system that you can discuss [3:42] and let's talk about what did you evaluate  before release? Uh what did you instrument [3:47] in production? And maybe most importantly, how did  you connect those signals to business value? Um, [3:54] less on the, you know, exact architecture and  more on the design decisions, if you will. [4:02] So, the project I'll talk about, um, we call  it smart PO. So, one of the challenges in [4:08] manufacturing, especially a company like Cummins  that manufactures lots of different equipment [4:13] with different um choices of configuration, is  how do you have a consistent supply chain? So, [4:19] we have an extraordinary set of demand planners  that have to decide at any given time how many [4:24] engines are we going to manufacture and what  um what parts do we need for those engines. [4:30] This can be very challenging because demand  can change whether it's um the line changes [4:37] or customers cancel orders and we have POS with  thousands of different suppliers that are open [4:44] and that sometimes they need to be cancelled and  it's there's a lot of effort from these demand [4:48] planners to try to reach out and contact these  suppliers to try to to stop these POS and stop [4:54] this uh this these tools or these uh supplies that  they need for the engines and overrun inventory. [5:01] So what this software does is it goes in and it  looks at the ERP system to determine um which [5:10] are potential orders that we can um cancel and  then the agents can go out and directly email [5:16] uh the suppliers get a response back, process that  and then present this to the demand planners and [5:22] then they can decide uh whether um the suppliers  understood and um we can automate some of that [5:29] process to speed things up. So this is a huge  benefit because it takes you know two to three [5:34] hours to do this without this automated system  and it just makes the process a lot smoother for [5:43] them and it's it's hugely valuable and it's also  important because we have human in the loop to [5:48] make sure that um the AI does a lot of stuff on  its own but it's not too autonomous and we have a [5:55] lot of kind of safeguards in place like before we  deployed it in production we made sure we had eval [6:01] s to make sure it was behaving proper properly.  We use ML flow to monitor all the different steps [6:06] in the process so that we can track where it's  been and what it's doing. If we see any issues, [6:11] we can add that to the evaluation set. And all  those pieces have been very valuable to make sure [6:16] that the system is reliable um and works the way  we expect it to. Would you would you say there's [6:22] one single decision in the design that maybe  improved the connection between evaluation and [6:28] business value and sort of helped bridge that gap?  So I think the decision was balancing autonomous [6:37] behavior of the agent to h the amount of human  in the loop. So, one thing that we could have [6:44] done is have the whole process of communicating  with the suppliers be fully automated and then [6:49] update update the ERP systems based on that  input that they provided. But we wanted to be [6:56] able to balance that because as you can imagine  that has huge ramifications if it doesn't work [7:00] the way that it should. So balancing automating  the tedious process of contacting the suppliers [7:07] and trying to figure out what their response is  versus the demand planner still truly being in [7:13] charge of figuring out what needs to happen  in terms of the actual action at the end and [7:19] validating that before the the agent updates  ERP system I think was really important. Yeah. [7:24] Great. So I have a followup for you actually.  Um, I'm sure you've seen a lot of different [7:30] deployments at a lot of different companies  in your position. So, on a similar note, um, [7:36] what evidence to you sort of separates what you  would consider to be a polished evaluation demo, [7:42] if you will, from actual realized business value.  Yeah, I think this is especially true with voice, [7:49] but I think true of many agents today is it's  very, very easy to create a proof of concept [7:54] that works one time. And I think that's what's  really changed with AI is that previously it used [7:59] to be that you would have an MVP that wasn't very  good and then would get consistently better. Um, [8:04] you know, maybe you just build out like a small  part of something, but today it can be really easy [8:08] to make it look like a fullyfledged product on day  one. And really the challenge is how do you scale [8:12] it? How does it operate over tens of millions of  conversations? How does it operate in all the in [8:18] like the last 10% of use cases, all of these  edge cases? And so coming from self-driving, [8:23] I actually think there's a really similar  demand like um evolution and maturity curve [8:28] where you know the first 80% actually happened  pretty quickly. So in 2016 there was already a [8:33] self-driving car that could go down the road but  then it took an additional 6 years to understand [8:38] to actually get it to fully autonomous on the  road self-driving. I think it's going to go a [8:43] lot faster than that with AI because everything  is accelerating and I think also you don't have [8:48] to deal with physics which is great. Um but still  to I think there's this where we are 80% today [8:55] there's still that 20% of what is the long tail  and how do you make sure it's working. Yeah. So [9:01] just to add to what Tammy just said right u I've  seen a scenario where the model works completely [9:07] fine in an offline mode and when we actually do  the production roll out where you're seeing that [9:13] okay uh not each and every metrics work exactly  as you expect because there are billions of [9:20] um different metrics that you need to validate  and one of the reason is the model is never wrong. [9:27] moduh works perfectly but the use case the  business use case that you're trying to solve [9:32] uh is something that uh will take more time  in doing it. Yeah. So so Shahid that actually [9:37] leads me directly to my next question I think if  you want to follow up on that. Um so traditional [9:44] software monitoring right focuses on things  like uptime, latency, even CPU utilization. [9:52] Uh but ML observability introduces totally  different variables. Um especially when we [9:57] start to think about the rise of generative AI  and agentic systems. So how has your thinking [10:04] approach and investment in observability changed  in this new paradigm? Yeah, in a in a traditional [10:11] system basically you just check whether the system  is running or not, right? That's just one metric [10:16] that you evaluate. Uh but in an agentic system  there are it's become pretty complex. You need to [10:22] identify first whether is the system working in  a right way and in a sub aent model when you have [10:29] multiple agent sub agent working did it work in  a right way in a right step. So the things become [10:34] pretty complex compared to our agentic model. It's  not like in your traditional system where you just [10:40] spin an additional dashboard and you can see how  now you'll be able to figure that out. So yeah [10:49] um so the signal shift from um from either pass or  fail. So in a traditional system it will be like [10:57] hey the system is running perfectly. So or it will  return either an error or some uh error responder [11:03] for um but in a multi-step agent you need to see  whether what is the output for each and every step [11:12] uh in a subg like what was the output for for the  first one what was the input what was the prompt [11:17] that you have shared what was the output for  that because if you're not tracking that properly [11:23] um your system will be not able to identify  where exactly in a subation model the issue came [11:29] Um so that's one of the metrics. So you need  in an observability pattern you need to capture [11:34] output of each and every sub agent so that uh if  if if the workflow is failing in step six if you [11:41] have logged each and every step properly you can  then go back and say hey this was my subision who [11:46] reacted on this input it was this was the output  and then you can backtrack and uh correct on [11:52] that part. Um the other step that I've seen is um  most of the um the model that we're running uh is [12:01] interacting on the single output like for example  um the model input metrics keep on changing the [12:09] prompt keep on uh changing over the period of time  um so even when you're doing a production roll out [12:15] u the the third party model will not even alert  you that they are uh updating something in the [12:20] background so you need to have your own regression  test suite whenever you are trying to roll out [12:26] something in production so that you can track  like what was the output what metric change and [12:32] not like your traditional model like you will  only run it once uh but every time uh in a day [12:36] you need to keep on running this regression suite  to identify like what was the output I think the [12:42] best teams are really thinking about how like how  do we not just evaluate the model at one point in [12:48] time but create systems that create this flywheel  of improvement because the difference with agents [12:53] compared to existing software is if you make a  button work one time, it will continue working [12:57] pretty much, you know, until like you need  some package upgrade. But with aic models, [13:02] your data is constantly shifting, your user  behavior is constantly shifting, the models are [13:06] shifting. And so we see the best teams actually  have, you know, cases where they find examples [13:11] in production, feed them back into their agentic  loop and are constantly evolving their test cases, [13:16] their um their metrics, like what they're looking  for and what the use case they're optimizing for. [13:21] Again, I think a lot of this is very similar to  self-driving where you're constantly looking for [13:25] like new edge cases, new edge cases, new um you  know, new cases and feeding that back in. So, your [13:32] evals are really a sliding window more than like  every use case under the sun. You're really trying [13:37] to say what did we just become good at and what  are we not yet good at and how can we constantly [13:41] be shifting this window to greater and greater  capabilities. Yeah, great. So let's talk a little [13:49] bit more about let's say root cause analysis in  agentic workflows right so as models scale and [13:56] pipelines incorporate multiple interconnected  agents uh tracing decision flows become really [14:02] difficult. Um, so when we're talking about a  multi-step tracing and state angle, how would you [14:09] approach sort of the challenge of real-time root  cause analysis when a complex system hallucinates [14:16] or fails somewhere deep in a distributed workflow?  Um, I'll leave that as an open question for [14:21] whoever wants to take that. So yeah, um, I have  usually seen that it's not exactly the model is [14:28] always right as I said earlier, right? So it's  basically depend on uh how you are interacting [14:33] on the data. So bas uh the most of the scenario  that we have seen is uh the model works fine but [14:39] the data sets like how we explained right the prom  keep on changing your some of your input metrics [14:44] change so the data based on the data the reaction  change. So we have seen a issue in this domain [14:52] where uh the subation model will suddenly react in  a different way and it it become very hard to find [14:58] out where which steps or which sub agent started  hallucinating. So the best way was like the one [15:05] that I mentioned earlier right so it's better  to have uh proper traceability defined on for [15:10] each and every sub region input and output and  try to find out like how each mo uh sub sub aent [15:15] was reacting on the changing prom the changing uh  input and that will be the best way to identify [15:22] what what is going wrong um so as a followup to  that I kind of think about it like this so what [15:31] I've seen can happen is people will say, you  know, my agent's not doing what I want. Which [15:38] model should I pick? So, a lot of times people  jump to am I using the right model? And that's not [15:44] really always the problem. So, for example, if you  have an agent that's taking a knowledge base to [15:51] um do an action or answer a question and the  model is doing exactly what it should, but it the [16:00] the knowledge base that it's pulling from either  doesn't contain the content that it actually needs [16:05] to answer the question, like a human couldn't even  answer it, or um the rag pipeline or whatever is [16:11] retrieving um the knowledge documents is not  retrieving the right documents. So in that case [16:17] you need to look at it in terms of um modular  architecture where you have different pieces [16:23] of the pipeline that you can troubleshoot whether  it's pulling the documents in or it's running the [16:29] model or however you do that and and I I love um  and I learned it from Andrew Ing who we're very [16:36] privileged to have him speaking tomorrow morning  when I was first learning ML is error analysis. [16:42] You go through each piece of the pipeline and you  look at the different errors you're you're seeing [16:47] and you see, oh, it's actually how my documents  are being pulled and I'm going to troubleshoot [16:52] that first. I'm not going to jump to the end and  do a million things with my prompt and try 10 [16:58] different models. I'm going to start at the, you  know, the little pieces and see where is are my [17:04] errors happening the most and then troubleshoot  through there. And that's much more effective [17:08] than trying to just, you know, end to end look at  it. Yeah, that's definitely something we advise [17:15] our customers to do first is it's easy to try and  first set up all of your metrics and have holistic [17:20] metrics that help you understand how your agent  is doing everywhere all at once. But really doing [17:25] error analysis of looking at 25 conversations  and looking at what's working, what's not is one [17:31] of the most useful activities. So starting with  looking at your data is a great place to start. [17:36] Yeah, Brooke, let's actually continue to talk a  little bit about uh continuous online evaluation. [17:42] Um I want to so when you're when you're  evaluating model readiness, let's say, uh there's [17:48] often a disconnect between isolated offline  benchmarks and actual online performance. Um, [17:55] so how are you designing these sort of continuous  feedback loops and online evaluation frameworks [18:02] that can safely flag those kinds of drop offs in  consistency, robustness, fairness. Uh, and that's [18:10] ideally without disrupting the live production  environment. Yeah. So we actually the way I think [18:16] about benchmarks is it's a multi-level. Um, so we  actually run benchmarks.co.ai AI and we benchmark [18:23] all the different voice models out there. So TTS,  ST, speechtoech, etc. on instruction following, [18:29] latencies, etc. And this is really useful for  being a coarse knob of what model should I even be [18:35] looking at. Um, so what I typically recommend is  that you then take, you know, the top five or the [18:40] top three models and then experiment with those  on your own data and actually run your own local [18:45] benchmarks to see how those models perform on the  task at hand. it, you know, the model may or may [18:51] not be able to answer legal issues or healthcare  issues, but if you're building, you know, [18:56] a customer service agent for 7-Eleven or if you're  building, you know, are the is the store open, [19:02] it doesn't matter how much legal knowledge it has.  And so really like then focusing on how well does [19:06] it perform on our data for like our brand names  that we represent or our specific tasks um is [19:13] a really important step. And then from there you  can actually do end-to-end evaluations. And that's [19:17] actually running through your system um using  your rag your own rag pipelines um for voice AI [19:23] actually running the voice agent end to end with  a voice um with a voice simulation and seeing how [19:29] you perform for those use cases. And then finally  the last step is you launch it to production and [19:34] see if all of these previous tests represent what  you're seeing in production. So every single step [19:39] of this loop is important in selecting models.  Yeah. Uh even I've seen one thing like uh if [19:44] you have uh if you're planning to roll out a new  model like that's something that I think I was [19:49] always successful in doing it is to have your  uh the newer model feed the actual production [19:56] data. Don't let it react to it but you can have  everything log and see like how it is going to [20:01] behave once you actually roll out into production.  So you can have the complete metrics uh like find [20:07] out like okay how it will work on set of benchmark  on hey this is my model which is in production [20:12] right now and the new model how it will react to  uh the production data and the other thing that [20:18] you can do is like have this model before you  roll out to be evaluated by your business team [20:24] to see hey is it actually solving the business  use case that they wanted and is it doing in a [20:29] faster way than what they are currently doing it.  So all these metrics are something really valuable [20:35] 100%. I think um silently launching is actually  exactly how we roll out new models to all of our [20:40] simulators is we'll first launch the model and  then compare how well it's doing between the two. [20:45] Um and then human in the loop evaluation is really  important because your evals don't matter if they [20:50] don't match how a human would have judged that  conversation or that interaction. And so having [20:55] you know 1% of traffic of like your simulations  or like a very small portion of all conversations [21:00] that are happening with your system go through  human review then you can be looking for this [21:05] you know frontier edge cases that you might have  already missed. Yeah. And when people know that [21:10] okay you have a human in the loop uh in your the  end to end workflow it's increases the confidence [21:16] in the system so they will trust more and that  will have a better impact on it. Yeah. I'm going [21:22] to actually add a little color to that myself just  cuz there's a a metaphor I particularly like when [21:27] it comes to dealing with model failures  or failures of these kinds of systems, [21:32] right? Um to me it's a lot like a crime scene. Um  the first thing that the police do when they show [21:39] up to a crime scene, right, is they they secure  the area uh and then they do their investigation, [21:43] right? The point being that containment and root  cause analysis are different things. Um, and just [21:51] always keep that in mind, right? Um, so let's move  on. Let's let's talk a little bit about uh trust [22:00] and transparency and tooling. Um, Tammy, I'm going  to start with you on this one again, I think. Um [22:09] so interpretability and explanability are critical  for building trustworthy systems but they carry a [22:16] lot of computational overhead uh and often become  more difficult as model complexity and system [22:22] complexity increases. Uh the challenge is even  greater when organizations rely on third party [22:28] or off-the-shelf models combining them with their  own. So with that in mind, uh how do you build [22:33] trust and transparency for internal engineering  teams, business stakeholders and other end users? [22:43] Um so I I think of a couple of things. So one  is being able to have a consistent system. So [22:53] everybody's excited about AI, especially  generative AI, and they jump to using it [22:58] for anything and everything. But sometimes the  right solution is to have the boring traditional [23:05] vanilla software that's code whether AI generated  it or not but the actual code that's running is [23:12] just regular software code and the Gen AI is  the icing on the cake. It's the it's the brains [23:19] that's routing doing the tool routing. Um, but  anytime you incorporate that, there's that kind [23:25] of non-deterministic probabilistic piece that  you can't completely control and you have to [23:29] put extra guard rails and extra evaluations around  it to make sure it's behaving. So, I think you can [23:36] build a lot more trust in a system if you're  really thoughtful about when and where you're [23:40] actually using things like generative AI that have  non-deterministic behavior. And then the other [23:46] piece is being transparent. And you know, we've  talked about traceability and observability to [23:52] make sure that we know what it's doing. And then  a basic piece that I've actually enc encountered [23:58] in my own life is saying when AI is being used  and when it's not. So recently I was making a [24:06] a service appointment with my local um mechanic  and when I called them I I got a customer service [24:14] representative and I quickly realized that it was  an AI and I was really thrown and it it made me [24:20] wonder if I wanted to go through this business  because they didn't admit that that's what they [24:24] were using. So, I think that you need to be really  transparent with people in your products when and [24:29] where you're using AI, too, so that they know um  if it behaves in an odd way, they they can expect [24:36] it and and not trying to trick people in thinking  that um it's a human when it's not. Um so, [24:44] interpretility is pretty hard to validate because  the model is pretty close. the the best thing [24:49] that you can validate is to see how the model is  reacting to your uh input and output. So that's [24:57] one of the metric that can help you to identify  hey is this third party model working exactly how [25:03] it is expected or is it just hallucinating on most  of the use case on that. Um other way that you [25:10] can do it is have this being validated by your um  product team, business team and see like how uh it [25:19] is helping them. So these are some of the metric  that you can do it. If I could flip this question [25:24] on its head a little bit. Um I think again talking  about agentic systems and newer models that almost [25:33] mimic human behavior. It feels like you're  interacting more with a a human than AI. Um, [25:39] have you run into any instances where people are  overrusting of these systems and how do you manage [25:46] and deal with those sorts of issues? Yeah, I think  along those lines of voice AI definitely has kind [25:53] of the potential these voices are good enough now  that you can mistake that it is actually a human [25:59] when it's an AI running the call. And along the  same lines is I think it's really important to [26:05] say when it is an AI and then also providing some  details of like what is it doing and why. Um so [26:11] I think cloud code does a really good job of this  where it will tell you exactly what's happening as [26:16] the as the chat is happening and so you start to  gain trust and understand its capabilities versus [26:22] it being a black box. Um I think a place where  people might overrust AI is actually throwing [26:27] voice AI at all of their problems all at once.  So, we see this a lot where you have a voice [26:32] AI that just handles any sort of inquiry um or  is trying to aim to handle 100% of call volume [26:38] all at once. I think this is the same with agents  in general is you should try to focus on the very [26:44] specific use case and do a really good job with  that one and then move on to the use the next use [26:49] case versus doing a semi okay job or a mediocre  job at everything all at once. Yeah. Uh I think [26:57] what um she just mentioned I think we need to  be really careful on how we are doing that uh [27:03] because it may lead to customer frustration too  if we are not getting the response in one or two [27:10] uh interaction. So yeah, like I do agree with  that. Agree. One thing I would add too, um, [27:18] it doesn't even have to be sophisticated agentic  AI, even AI chat bots and LLM interaction. There's [27:24] a difference between using AI as an enabler  and using AI beyond your level of competency [27:30] to evaluate the quality of the output. And  I think a lot of people struggle with that [27:34] because there's a lot of excitement and a lot of a  lot of vendors and large AI companies are sort of [27:42] selling this idea that it can, you know, magnify  your impact, move move you beyond where you're at, [27:50] but people can push that too far. And I think  people need to be cautious about that. And they [27:55] also need to be respectful of the the expertise  in different areas that they don't directly have. [28:02] and make sure that they continue to work with  people in those different areas of expertise, [28:07] whether it's science, whether it's math, whether  it's coding, these areas that we've made huge [28:13] leaps to bridge the gap between people that  are practitioners of it and not. Um, but we [28:19] don't want to go too far with that because we can  get in some um really concerning areas with that. [28:27] If if there's anything I would add to that  conversation, it's always just to be considering [28:32] uh who is the user, I think a physicist will  probably have uh more observability needs than [28:39] your standard Amazon shopper, so to speak. Um  so that's something that's always important to [28:44] keep in mind as well. Um so I've got a I've got a  couple more questions here and the first one we're [28:51] going to focus a little bit more on traditional  AI, traditional quote unquote. Uh and then yeah, [28:58] exactly. Um but I want to talk I do want to talk  for a second about uh drift triggers and model [29:04] action and those sorts of things. Um so obviously  data drifts can happen for a number of reasons. [29:11] It can happen very quickly. Uh anything from  user behavior and supply chains to just changes [29:18] in physical hardware um all sorts of things,  right? Um, so from the hardware and operating [29:25] envelope side, like what what specific thresholds,  statistical tests, automated triggers do you put [29:33] in place to determine exactly when a production  model needs to be pulled back, retrained, [29:38] fine-tuned, etc. Open question. Um I think it's  depend mainly on like how you are u basically [29:50] interacting on your data data set. Um it basically  loop back to what I mentioned earlier right every [29:57] the the system start becoming complicated when you  have multiple sub agent uh in the field. Uh when [30:03] it is a simple traditional model which you have  custom made it it just have around around 100 [30:10] to 200 metrics. So you can easily evaluate what  is your input and output. But looking at a large [30:16] language model which has more than 1 billion  combination the system start behaving pretty [30:22] complicated. So identifying each and everything  will be pretty um not part of it. So you can [30:29] actually break down into two parts. One basically  having a strong traceability across your complete [30:34] input and output and see exactly what subvision  is getting impacted and secondly you can just see [30:43] uh is there a better uh testing mechanism that  you can add it in place so that uh you can find [30:49] out like hey what exactly is my model behaving or  deviating from the the actual one. Yeah, I think [30:57] a lot of practices don't change significantly  compared to traditional software engineering as [31:02] well where keeping you know live monitoring of  certain metrics that you don't want to regress [31:07] especially around latency or completion rates  or um you know certain trajectories an analyzing [31:14] completion rates of those trajectories. Um, but at  the same time there's a big shift from instead of [31:19] looking at this either passed or failed, you're  now looking at probabilities of certain types [31:23] of events. And so setting your tests up so that  you're really looking at what is the probability [31:27] that a certain, you know, we're looking for like  one in a thousand conversations will have this [31:32] versus one in a million conversations will have a  compliance failure. And so really thinking about [31:37] what is your risk tolerance and what are the  rates that your acceptable rates. Yeah. Uh and [31:44] uh additionally what you can do is you can narrow  down on your testing strategy to saying that hey [31:49] this is what uh you will be focused on and then  you can use a large language model and add a human [31:54] in the loop in middle which will validate those  subset of the uh streams and see like how the the [32:02] model is behaving and that can help you evaluate  on uh where you are. In terms of um traditional [32:09] ML that we've used a lot in Cumins, we do a lot  of predictive modeling and in this case you have [32:15] grounded results, right? So if you're trying  to predict engine failure n 90 days in advance, [32:20] then you can find out did the engine fail or  not. So you have you can really ground metrics [32:25] like sensitivity, specificity, accuracy and say,  you know, these are the thresholds that I expect. [32:31] Um I've got a month worth of data. I can actually  see, you know, I predicted these would fail. did [32:37] they fail or not? And then you can actually  track those. So that's a nice grounded way to [32:40] do it for at least um predictive modeling. Sure.  How do um you account for let's say seasonality [32:51] or obviously not all your metrics are going  to be exactly the same over time when you're [32:57] when you're making these sorts of adjustments.  So how do you plan for those sorts of things? [33:02] So u basically what like I usually focus on  is um see like how the system is behaving on [33:11] your earlier model which is already running  in production right now and then you can set [33:15] a benchmark on the new metrics that you're  introducing. Uh you need to be really careful [33:20] on when you're trying to have a workflow defined  and you're trying to execute some of your very [33:27] critical module through sub agents. So you need  to have a boundary defined on what the action [33:32] will look like. Then you will see that okay  this is a critical steps that I need to get [33:38] uh executed. So you can have your some of  your uh regression suite that some of your [33:45] team has written who has actually trained the  model and then based on that they can say hey [33:50] you will run this sets of regression and see  like the how your system is working on it. [33:56] This is also where having like thinking about  the bias within your data set and how data [34:01] how representative your data set is of production  data. Um for cases like this is actually important [34:06] to have a data set that's distributed over the  last year. Um especially if there's seasonality [34:11] to your data. I think a good example with Whimo  was actually Halloween where you have kids in [34:16] Halloween costumes and people walking around  in like a little ghost costume or a cat. How [34:21] do you make sure that um how well the Whimo is  going to interact in those uh different cases? [34:27] And the same is true with you know healthcare you  have seasonality to you know insurance and claims [34:33] um you know the holiday season you might have more  returns or more orders. So really thinking about [34:39] like how is your data set representative of not  just like one day in time but actually the entire [34:44] year or any other um parameters that might be  relevant. No, I I so I have I have one more [34:49] question I'd like to get to. Um and then hopefully  we'll have a few minutes for questions left at the [34:54] end. So if you're thinking about those, get those  ready. Um but I'd like to take this same topic [35:00] and sort of talk about it when we're talking about  agentic systems now. Um, so when we're looking at [35:06] the next wave of agentic AI systems that operate  with fairly minimal human intervention, how must [35:15] our approach to these kinds of guard rails and  telemetry evolve to catch these kinds of emergent [35:21] behaviors or cascading failures across those kinds  of distributed automated workflows? Yeah. Um, [35:29] so I think that's a really good question. So one  of the main reason why most of the AI initiative [35:34] gets scrapped is because the system work really  well in an offline mode based on subsets of uh [35:41] use cases that like how you mentioned right so it  will work fine but the most of the systems are not [35:47] ready with the real production environment where  there are any number of combinations of data set [35:52] that is feed into so the best way is to guard rail  on what are some of the use case that you wanted [35:59] to actually have control by human in the loop in  between and what are some of the uh use cases that [36:06] you can have an agentic uh flow to to take care of  that. So I think based deciding on which uh some [36:14] of the use case that can go without having a human  validate that is something that I've seen like uh [36:20] it will be really helpful for a a use case to try  and I think also thinking about fallback systems [36:26] um I think people don't think about they think  about how do we make sure that this works 100% of [36:31] the time and not when it does fail what do we do  about it the common answer is you know escalate to [36:37] a human but are there other tools in your toolkit  that you can used to, you know, maybe you sent [36:42] have an offline job that goes and corrects all the  phone numbers that maybe were mis misheard or can [36:47] you have an agent, can you have a human agent  in the loop that will say this was we correctly [36:52] identified their name or their phone number and  then the the agent can move on. Um, so coming [36:56] up with more creative ways of having fallback  systems. And another thing I would add to that too [37:02] in terms of creativity of monitoring the system  is generative AI is inherently non-deterministic. [37:10] There's a lot of things that you can do to try to  control its behavior, but there's always a little [37:14] bit of randomness of um stokisticity into in it.  So, one way to think about it is have kind of a [37:24] creative maybe like agent that's watching its  behavior and sort of looking for behavior that [37:30] doesn't make sense, right? It's just sort of  looking for anomalous weird behavior where you [37:34] just have like the the one in a million different  perturbations that added together get it to do [37:41] something that's really off the rails. And if you  can have a step back and have another agent that's [37:45] monitoring it, it may be able to see that just  like if you had a human looking at it and saying, [37:49] "Hey, that's weird." Right? Instead of having like  prompt guard rails and all these different things, [37:54] having an an agent observing it and looking for a  strange behavior. Yeah. U this is the most common [38:00] thing that I've seen that people are usually  confused on AI observability space where they [38:06] are usually confused on hey shall we build the  whole system or shall we buy it. So one of the [38:12] thing that I usually um like there are a lot of  um products which help you in capturing the data. [38:19] So I usually like do it in this way like uh you  can build the PI plumbing and then you can have [38:25] your own decision model where you can build it.  So you can c there are a lot of tons of product [38:32] which will help you capture all the metrics  and all but all the decision making effort and [38:36] based on how your system will react based on your  data is something that you can build on your own. [38:44] Yeah, I think if I I'll piggyback on the  creativity comment a little bit there too. I [38:49] think this is truly is a new paradigm and it's  not completely solved yet. Um so definitely I [38:56] I'll echo the creativity notion a lot. Um, it's  a fascinating place to be in right now. You know, [39:03] we're all used to for a very long time having  deterministic systems with deterministic answers [39:09] and now we're trying to figure out how to evaluate  non-deterministic systems and it's certainly a [39:14] challenge and it's likely a combination of a  lot of different things. Um, so with that said, [39:21] we do have about four minutes left. I'm  happy to open up the floor for questions. [39:28] Yeah, right here. Yeah. So, with full  traceability, you're going to have a ton [39:32] of data to review. I'm wondering how that scales.  Um, you use like LM as a judge. I like your idea [39:41] of the adversarial agent in real time, but you're  collecting a lot of data. What are the strategies? [39:50] So yeah uh like you will be capturing a lot  of data but you can't have your whole whole [39:55] system target on each and every use case. So like  best thing that you can evaluate is like you can [40:01] have all your use cases put it in a bucket and  then define hey this is my top priority that I [40:06] wanted to do it first and then based on that you  can have um that I I feel is the right thing to [40:12] do it. Yeah, you can have sets of test cases  written by your uh theme and then say that hey [40:18] this is the one of the use case that I want my  theme to to validate and then you will have a [40:23] human in the loop pipeline set up for that to  to see that hey is this being solved with this [40:28] human in the loop metrics and test suites that you  added and then if that is working fine then you [40:34] can slowly start adding more and more use cases  to it but I agree with you if you try to solve [40:38] everything with the large data set that you're  capturing uh it will be really difficult. Yeah, [40:44] I think along those lines, it's really, you know,  I think the infrastructure today to capture all of [40:49] this data is certainly there, but something that  we focus on at Koval is how do we provide this [40:53] layer to help you understand this large  swath of data. So, we provide agents that [40:58] go through all of your traces and provide like  um in incremental data points, but then also [41:04] that aggregate analysis of how do you shift sift  through all of this data at scale. Um we do this [41:10] through a combination of traditional ML models  with clustering. So being able to cluster all [41:14] these different issues and you can create multiple  clustered um types and then also doing more of a [41:20] report style where you can go in and dig into  how often is my agent switching languages. I [41:25] heard about this from one customer. So how do  I dig in and go see how often that's happening? [41:32] Have a couple minutes left. We can do one  more. Yeah, we're here. Hi. I was just [41:37] wondering if you can clarify what you meant  by the models that were wrong. I don't know. [41:47] Maybe you [41:51] Sorry, I didn't get your question. I think  the question was um you mentioned that the [41:55] model is never wrong. Uh what did you mean by  that? Yeah. So when I say model is never wrong, [42:01] what I meant is like model will be able to uh  give it give you the in based on the input you [42:06] will be able to get the output there uh the output  will be based on some of the input parameter that [42:12] you had given. But in your uh multi- aention  model multi-step model you will see that okay [42:17] the only way to identify whether your sub agent  has been working right or wrong will depend on [42:23] what is the input data set that was feed in. So  those are like way to identify hey the model will [42:29] be able to give it but it will the accuracy will  depend on what was the data the input that would [42:34] fe fade in what was the prompt that I actually  sent during that particular traceability. So [42:40] if you're able to trace each and everything  then you'll be able to uh track it properly. [42:48] I think we could do actually one more  if we have a quick one. Yeah right here. [42:52] Oh, thank you. Question only likely touched upon  eval. Um, so I'm working in a space where I have [43:02] to meticulously production rate prompts. And so  one of the things we do for is basically how each [43:10] sentence is in the prompt. I've already came up  with maybe ideas or frameworks to test that more. [43:20] Um since we know that everything kind of  cross influences like each set cross one [43:27] but there are usually some that kind of do  nothing and some that do a lot. So I was [43:33] wondering if you have your routine. Yeah. So  we do this through AB testing and fuzzing. Um, [43:42] so maybe not directly helping you to do like  actually the reorganization of the prompt, [43:47] but what you can do is you can do fuzzing of a  prompt where you either reorganize the prompt, [43:52] you take sentence out, added sentences in, and  then run a bunch of tests to see which ones are [43:58] impacting the the endto-end conversation the  most. Specifically for conversations, what this [44:04] looks like is if you run certain conversations,  how often is that um instruction adherence, [44:09] etc. Um, so yeah, I would say like fuzzing  in permutations is typically what we do, [44:14] but I'm curious actually like what you've seen not  working or what was particularly hard. All right, [44:19] I think unfortunately unfortunately we're we're  out of time. Uh, feel free to follow up this [44:24] conversation afterwards. Um, but thank you to all  of our panelists and thank you to all of you for [44:29] coming. Um, this was a great time. So thanks.  Enjoy the rest of your conference everyone.