Optimizely Replaces AI Eval Benchmarks With Controlled A/B Testing

Optimizely 用受控 A/B 测试替代 AI 评估基准 内部评分标准和基准评估往往只是迎合了开发者的偏见，团队需要在授予自主部署权限之前，通过受控实验来验证 Agent 的输出。 Josh Holt 指出，AI Agent 的表现宛如“盲目自信”的替补球员，在出错时不会停机报错，而是极为确信地交付存在缺陷的输出。为解决该问题，团队必须依据失败代价建立分层验证阶梯，使严谨程度从简单的直觉核验逐步升级为受控实验。由于合成评分标准和以大模型充当裁判（LLM-as-a-judge）的方法存在过度刷榜的风险且仅仅反映了开发者的偏见，因此利用 A/B 测试在真实用户流量中检验才是获取客观事实的唯一途径。通过实证结果数据形成闭环，能让团队安全解锁 Agent 的自主运行能力，并利用现实中的失败案例持续丰富自动化回归测试套件。 AI Agent 的失效方式与传统软件不同：它们不会抛出错误或停机，而是以盲目自信的态度生成有缺陷的输出。 (0:04) 在多步骤 Agent 工作流中，基础数据微小的幻觉（如网站访客数量）会产生级联反应，进而破坏下游的每一个推理环节。 (2:06) Agent 评估分为三个不同的时间尺度：系统级开发评估、实时运行工件验证以及部署后的结果度量。 (3:15) 内部 LLM 裁判和评分规则衡量的只是与开发者质量预设的一致性，而现实世界的实验度量的则是实际的业务结果。 (3:15) 验证的严谨程度应当与出错代价相匹配，循序渐进地从直觉核验走向系统化评估、人机协同关卡和受控实验。 (6:32)

Josh Holt of Optimizely argues that autonomous AI agents fail with convincing confidence rather than halting, making internal proxies like developer vibe checks and static benchmarks unreliable. Because an agent's output is fundamentally a hypothesis, teams must validate agent actions through controlled real-world experimentation before granting them greater operational autonomy. The Trap of Irrationally Confident Agents (0:04) Traditional software halts or throws an explicit error when it breaks, but AI agents routinely deliver flawed outputs with complete conviction. Holt compares this behavior to sports writer Bill Simmons's concept of the 'irrationally confident player'—a bench player who takes impossible, contested shots under the delusion that he is Kobe Bryant. Agents similarly operate under the unearned assumption that they are exceptionally capable, masking severe errors behind persuasive delivery. In an early Optimizely pipeline designed to ideate A/B tests, a foundational traffic-checking step failed unpredictably: across consecutive runs querying the exact same weekly window, the agent reported visitor figures swinging wildly from 58,000 to 13,000 to 24,000. Because downstream steps relied on visitor volume to calculate statistical power and select viable candidate pages, this silent hallucination corrupted the entire reasoning chain. Limits of Vibe Checks and Synthetic Benchmarks (3:15) Agent evaluation spans three distinct operational horizons: development loops using synthetic evals, runtime validation via filters or human reviewers, and post-deployment outcome measurement. Holt cautions that relying on developer 'vibe checks' simply trains an autonomous loop to mirror the personal biases and idiosyncratic tastes of individual engineers. Meanwhile, relying on static rubrics fosters 'benchmaxing,' where frontier models overfit to synthetic benchmarks while deteriorating in real-world utility. As an example of benchmaxing, Holt cites Claude Opus 3.5, which posted record-breaking scores on coding benchmarks but drew user frustration when its conversational tone deteriorated into babble during routine queries. Automated LLM judges only measure how closely an output matches an engineer's internal assumption of quality, not whether that output actually works in production. Scaling Verification Rigor to Earn Autonomy (6:32) Validation mechanisms must scale in proportion to the cost of an agent being wrong. Low-stakes internal tasks require only informal spot-checks, customer-facing capabilities require systematic regression suites, and high-impact actions demand human-in-the-loop review gates. Fully autonomous production actions, where the blast radius of error is highest, necessitate controlled experimentation. These verification tiers complement one another. For example, an informal vibe check first uncovered the root cause of the traffic discrepancy—the pipeline was recomputing visitor metrics multiple times throughout the workflow—allowing engineers to implement a deterministic fix by computing the metric once at the start. To move up the autonomy ladder and allow an agent to deploy changes independently, teams must first provide quantifiable empirical proof that the agent's ideas consistently achieve higher customer acceptance and win rates than human baselines. Establishing Causality Through Controlled A/B Testing (13:13) Observing raw production metrics after deploying an agent update fails to establish causality, as changes in performance may stem from seasonality, marketing promotions, or visitor mix. While software engineering advances rapidly because compilers and unit tests provide near-zero-cost verification within seconds, evaluating human user experience and preference takes days or weeks of live traffic. Validating agent-generated customer flows therefore requires randomized A/B testing against control cohorts. Instrumenting agents with controlled experiments completes an operational flywheel. Real-world statistical outcomes determine deployment decisions, rolling out winning agent variants while halting or reviewing underperforming ones. Furthermore, every empirical outcome provides objective ground truth: if a variation favored by internal rubrics fails in production, that real-world outcome is converted into a regression test case, continuously calibrating synthetic evaluation suites against actual human behavior. Holt concludes that engineers cannot rely on an agent's self-reported certainty or subjective spot-checks. Building rigorous measurement scaffolding and randomized A/B testing frameworks is the only way to verify that autonomous agent actions drive real-world business results rather than silent, persuasive failures.

Josh Holt 指出，AI Agent 往往表现出类似于体育界“非理性自信球员”的盲目笃定；其生成的内容在本质上仅是待检验的假设，必须通过受控 A/B 测试和真实业务结果进行因果验证，而非依赖开发者直觉或合成基准。 Agent 的非理性自信与多步连锁失效 (0:04) 传统软件在遭遇错误时通常会抛出异常或直接中断进程，但自主 AI Agent 却截然不同，往往以极高且具说服力的确信度输出虚假信息。Josh Holt 借用体育作家 Bill Simmons 提出的“非理性自信球员”概念对此进行类比：如同替补席末端登场却自视如科比、面对三人防守盲目抢投超远三分的球员，AI Agent 往往默认自己无所不知，以毫无动摇的姿态交付存在严重缺陷的答案。 在 Optimizely 开发的 A/B 测试构思 Agent 中，这一特性引发了底层推理的连锁崩溃。Agent 在流程第一步检索目标网页前一周的访客流量时，在连续运行中先后给出了 58,000、13,000 和 24,000 等剧烈波动的虚假数字。由于访客体量直接决定实验是否具备统计效力、哪些页面适合作测试候选以及变体能否达到显著性，初始流量数据的幻觉导致整个下游的多步决策链条彻底失效。工程团队最终通过将流量检索改为工作流起始处的单次确定性计算，才解决了这一问题。 验证阶梯与合成评测的跑分陷阱 (10:31) 评估 Agent 系统需要在不同时间尺度上回答问题，其严密程度必须与出错成本严格对齐。评估体系包含四个层次：用于低风险内部任务的人工直觉检查（Vibe Checks）、基于 LLM 裁判与基准测试集的开发期系统化评测（Systematic Evals）、用于高风险操作但受限于人力带宽的人机协同审核（Human-in-the-Loop），以及最终基于真实流量与业务指标的受控实验（Controlled Experimentation）。 如果仅依赖前三层代理指标，系统极易陷入优化偏差。仅靠直觉检查只会引导 Agent 迎合特定工程师的个人偏好；而过度依赖静态评测与合成规则，则会导致模型过度拟合特定指标的“跑分陷阱”（Benchmaxing）。例如 Anthropic 与 OpenAI 频繁发布跑分破纪录的模型，但在实际落地中往往表现脱节；演讲中列举的 Claude Opus 3.5（口语代指 Opus 5）虽在标准化代码基准上得分极高，但在日常对话中语调退化甚至产生无法理解的胡言乱语。静态评测仅代表开发者主观认定的质量代理，唯有真实用户转化结果才能规避认知偏差。 观测数据偏差与 UX 验证的现实成本 (13:13) 单纯在生产环境中监控上线后的转化率或用户采纳率，不足以证明 Agent 改动的有效性。单纯的观测数据充斥着混杂变量：流量激增、季节性变化、营销推广活动或特定时段进入网站的用户群特征偏离，都会扭曲数据表现。为了建立真正的因果关系，必须采用标准的随机双盲 A/B 测试，让一组用户接收旧版 Agent 的输出，另一组接收新版 Agent 的输出，从而隔离外部干扰。 这种验证逻辑也解释了为何 AI 在软件工程领域的迭代速度远快于消费端用户体验（UX）领域。在代码开发中，编译器、语法检查器和自动化 CI/CD 单元测试提供了秒级、近乎零成本的闭环反馈信号；而在涉及人类主观偏好的电商结账漏斗等体验优化中，验证往往需要将真实活体用户分流引入变体，耗费数天乃至数周积累具有统计学意义的样本量。高昂的人类偏好验证成本，决定了 Agent 必须通过严谨的受控实验来闭环。 受控实验驱动的飞轮与自主权授予 (15:29) 通过受控实验评估 Agent，能够构建自给自足的技术演进飞轮。当 Agent 引入新版本并在实验中展现出统计学显著的业务正向增长时，系统会自动全量发布；若表现欠佳，则安全回滚或转入人工审核。与此同时，每一次真实实验无论输赢，都构成了客观的真实基准（Ground Truth）。如果一个在内部评估标准中得分极高但在线上实际测试中惨败的变体出现，该失败案例会被直接转化为回归评测套件中的自动化测试用例，以此校准静态基准并免疫未来退化。 Josh Holt 强调，证明 Agent 生成的建议能在真实生产中持续取得超越人类基线的胜率，是赋予其更高自主权的必要前提。用户绝不会容忍一个在黑盒中单方面执行线上变更的失控系统。Agent 只有在历史上以可量化的客观数据证明其构思的有效性，才能逐步从“提出建议供人类采纳的助手”，跨越到“被允许自主实施并运行实验、仅向人类汇报结果的自主系统”。信任只能由经过验证的实证结果挣得。 Josh Holt 总结认为，AI Agent 绝非传统意义上非对即错的确定性软件，其高确信度的幻觉极具迷惑性。从业者不能寄希望于 Agent 自述的置信度或静态评测分数，构建基于真实业务指标的受控 A/B 实验测量体系，是将不可靠的生成假设转化为可验证商业价值的唯一路径。

## Background 本内容来自 YouTube 频道 Ai4，于 2026 年 8 月 18 日发布。视频由 Optimizely 首席产品负责人 Josh Holt 演讲，简介指出让 AI Agent 行动的技术已基本解决，但建立信任仍是难题；由于 Agent 会极其自信地输出看似正确的错误答案，Josh 提出将其输出视为假设，借助受控 A/B 实验和真实业务指标来完成验证闭环。 ## TL;DR Josh Holt 指出 AI Agent 普遍存在类似板凳球员盲目抢投三分的“过度自信”问题，单靠直觉检查或容易陷入刷榜陷阱的静态评测集无法建立真正的信任。他提出了一个与试错成本相匹配的四级验证阶梯（感官检查、系统评测、人机协同、业务结果实验），强调对于高自主权场景，唯有真实环境下的受控 A/B 测试能排除混杂变量。通过衡量客户采纳率与实验胜率等硬性商业结果，团队不仅能为 Agent 提供扎实的自主权护栏，还能把真实实验的成败反哺为评估集中的新认知，驱动系统在实践中持续进化。 ## Atmosphere & style 演讲风格务实且极具洞见，Josh 以清晰平实的行业工程视角切入，既没有过度炒作 AI 的浮夸辞藻，也没有枯燥空洞的理论堆砌。他巧妙地借用体育领域的比喻打破技术严肃感，使讨论显得生动而富有画面感。整场分享节奏连贯、抽丝剥茧，听感上如同聆听一位身处一线的资深技术领袖推心置腹地拆解系统架构与认知陷阱，传达出一种理性、严谨且深谙真实业务复杂性的清醒力量。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=4s) 智能体普遍存在“盲目自信”（Irrationally Confident）的特点，即便输出看似合理却完全错误的答案也笃定不移，因此必须建立客观的验证机制。 - [02:06](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=126s) 基础数据查询的不确定性（如访客流量统计偏差）会在智能体工作流中逐级放大，导致下游策略构思与推理全面失真。 - [03:15](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=195s) 验证智能体应涵盖三个时间维度：开发循环中的系统评测集（Evals）、运行时的产物护栏（Guardrails），以及最终的真实业务结果衡量（Outcome Measurement）。 - [03:15](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=195s) 智能体的输出本质上只是一个“假设”，不能仅依赖主观规则或“AI 评判 AI”，必须通过受控实验测量实际业务转化来闭环验证。 - [06:32](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=392s) 验证的严谨程度必须与犯错成本挂钩，从感官检查（Vibe Checks）、系统评测、人机协同（HITL）逐级递进至业务结果实验（Outcome Experiments）。 - [09:18](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=558s) 赋予智能体更高自主权的前提是拥有扎实的数据护栏，唯有用可验证的胜率与采纳率证明成效，客户才会信任全自动部署。 - [10:31](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=631s) 过度依赖主观偏好或静态评分容易陷入“基准刷榜”（Benchmaxing）陷阱，导致模型在测试集上表现优异但在真实场景中过度拟合且不可用。 - [13:13](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=793s) 单纯的前后对比无法排除季节性和流量波动等混杂变量的干扰，必须依靠严格设置对照组的 A/B 测试来确立算法改进与业务增长的因果关系。 - [13:13](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=793s) 代码生成因单机验证成本低而进展迅速，但复杂的人类偏好与商业决策必须通过接入真实流量的实验进行度量，验证周期长但必不可少。 - [15:29](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=929s) 将受控实验嵌入开发闭环能形成持续进化的飞轮，实验中的失败案例可推翻固有假设并反哺评估集，促使智能体持续向真实世界收敛。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=4s) 智能体输出即假设：Optimizely 的实验验证之道 演讲者 Josh 来自 Optimizely 的产品团队，专注于实验平台（Experimentation Platform）的建设。Optimizely 广泛为市场营销团队提供 AI 解决方案，其客户的核心诉求是通过实验提升数字资产（如官网、电商平台等）的转化率、客单价和用户留存率。Josh 的工作重点在于利用 AI 智能体帮助客户运行更多高质量的测试，从而推动业务成果。 在构建面向实验场景的智能体过程中，Josh 发现了一个普遍存在的痛点：智能体经常会极其自信地向你“撒谎”，即便给出的答案漏洞百出，它依然表现得笃定不移。Josh 引用了著名体育作家兼播客主持人 Bill Simmons 提出的“盲目自信球员”（Irrationally Confident Players）理论来形容这种现象——就像球队里排名第七的替补球员，一上场就在被三人包夹的情况下盲目抢投高难度三分球，自以为是 Kobe Bryant，但实际上水平平平。AI 智能体同样认为自己是“天底下最聪明的存在”，它们给出的答案往往看似合理却完全错误。因此，必须建立起一套行之有效的验证机制来赢得用户信任。 - **Optimizely 核心目标**：帮助企业在其网站与电商平台上优化转化漏斗、提高订单价值与留存。 - **“盲目自信”的比喻**：智能体如同盲目自信的板凳球员，能力有限却以极高的确定性输出错误结果。 --- ### [02:06](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=126s) 流量查询与智能体运行中的确定性失效 Josh 分享了一个具体的产品工作流案例：利用智能体为客户生成 A/B 测试的灵感与假设（Test Ideation）。进行 A/B 测试的前提是网站必须具备足够的流量，系统需要将真实访客分流到不同的测试版本中并观测其行为变化。因此，查询网站的真实访问量（Visitor Traffic）是整个测试构思链路中的基础输入，它决定了可以在哪些页面进行测试以及能够开展哪种类型的测试。 然而，在开发这一环节时，智能体在查询“上周网站访客数量”这一简单任务上屡次出错：第一次返回约 58,000 人，第二次变成了约 13,000 人，第三次又变成了 24,000 人。这个看似微小的数值查询偏差，会导致整个后续构思流程全面偏离轨道，因为智能体后续的所有决策和建议都是以该数据为基准进行推理的。 ```mermaid flowchart TD accTitle: Agent Ideation Workflow Failure Case subgraph FailureCase ["智能体构思工作流中的偏差传递"] A["查询上周流量 (基础步骤)"] -->|输出不一致: 58k / 13k / 24k| B["智能体推理与假设生成"] B --> C["生成测试方案 (全面失真)"] end ``` - **测试前置条件**：没有流量就无法分配对照组与实验组，流量规模决定了实验的类型与可行性。 - **错误放大效应**：基础数值的随机与不一致性会直接污染整个工作流的下游生成质量。 --- ### [03:15](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=195s) 验证智能体的三个维度与最终结果驱动 面对智能体的错误，常见的验证思路通常分为两个层面：一是在开发迭代阶段通过评测集（Evals）系统化评估整个系统的平均性能，判断修改是否带来了整体提升；二是在运行时（Runtime）验证单次执行的具体输出（例如某次返回的 15,000 或 58,000 是否正确）。Josh 将验证智能体所要回答的问题拆解为跨越不同时间尺度的三个层次： 1. **开发循环（Development Loop）**：在系统级迭代中，利用评测数据集（Eval Sets）和大模型作为裁判（LLM-as-a-judge）来衡量平均表现。 2. **运行时评估（Runtime Artifact Evaluation）**：在智能体每次运行时，通过预设护栏（Guardrails）或人工介入（Human-in-the-loop）来评估单次生成的产物是否合格。 3. **真实业务结果衡量（Outcome Measurement）**：前两层验证本质上只是主观判断（由人类或基于人类主观设定的规则进行裁判）。若评测系统本身对“什么是好的测试想法”的定义存在偏差，评测结果也会失真。 Josh 强调，构思智能体的目的绝非“为了提建议而提建议”，真正的核心指标是：客户是否采纳了这些建议？采纳这些建议后上线的实验，其胜率（Win Rate）是否显著高于人类团队自主构思的方案？智能体的输出本质上只是一个“假设”（Hypothesis），唯有通过受控实验测量真实的业务结果，才能闭环证明智能体的实际价值。 --- ### [06:32](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=392s) 验证阶梯与试错成本的匹配 针对不同的业务影响度和试错成本，Josh 提出了一个四级“验证阶梯”（Verification Ladder）。验证的严谨程度应当与“犯错的代价”（Cost of Being Wrong）严格成正比： | 验证层级 | 适用场景与试错成本 | 典型方法与局限 | |:--- |:--- |:--- | | **1. 感官检查 (Vibe Checks)** | 试错成本极低（如仅供内部阅读的竞品周报抓取智能体） | 开发者肉眼观察输出、凭直觉快速手动调优 | | **2. 系统评测 (Systematic Evals)** | 面向外部客户交付、错误影响中等 | 采用基准数据集、LLM-as-a-judge 进行离线评分 | | **3. 人机协同 (Human in the Loop)** | 涉及实际业务操作、具备直接影响 | 引入人工审核把关；缺点是无法规模化扩展 | | **4. 业务结果实验 (Outcome Experiments)** | 高度自主化（如智能体全自动构思、上线测试并部署） | 通过 A/B 测试直接衡量转化率、胜率等硬指标 | Josh 指出，前三层方法都局限在评估“对单次输出的静态主观判断”，唯有第四层（真实实验）能够度量智能体是否真正达成了业务目标。同时，这些层级并非互斥，而是相互补充并反哺的。 以之前提到的流量查询飘移为例，Josh 正是通过第一阶段的“感官检查”（Vibe Check）一眼看出了数据的荒谬。经排查，原因是工作流中不同节点各自独立且以微小的差异重复计算了访客量。修复方案非常明确：在工作流起点预先准确计算一次数值，并将其作为固定上下文向下传递给所有后续环节使用。通过直观的错误排查，一个底层的逻辑缺陷被迅速修复并沉淀到了系统中。 ### [09:18](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=558s) 自动化评估与度量驱动的自主权护栏 在构建 AI Agent 的过程中，团队每一次进行迭代、实际运行以及用户反馈问题的过程，都可以沉淀为系统化评估集（Eval Suite）中的全新测试用例。日常的自我评估与问题追踪，正是不断丰富评估测试集的源头活水。然而，真正的转折点在于将评估推进到对 Agent 实际目标成效的度量上。以 Optimizely 内部的测试构思 Agent 为例，它为客户生成实验假设与测试方案，但仅仅生成文本是不够的，必须追踪两个核心指标：客户在多大频率上真正采纳并运行了这些 Agent 提出的想法，以及这些想法转化为高胜率实验的比例究竟有没有显著提高。 随着行业从辅助工具转向完全自主的闭环系统，团队渴望赋予 Agent 越来越高的自主性。而这种度量体系正是为自治提供“安全护栏”（Guardrails）与信心的核心支柱。如果产品团队能够拿出可验证的数据表明“客户确实极度认可这些想法，且它们的胜率更高”，团队才有底气跨出下一步，对客户提议：“甚至不需要我们每次把想法推给你逐条审核，你大可信任系统，让我们自动部署并运行这些实验，最后直接向你汇报结果。”想要让企业客户接受这种高度自主的运作模式，唯一可行的路径就是拿出可证伪、可验证的事实验证。 ```mermaid flowchart LR accTitle: Agent 自主权提升的度量闭环 subgraph Iteration["日常迭代与反馈"] A["Agent 生成测试想法"] --> B["用户反馈与问题发现"] B --> C["沉淀为系统化 Eval 测试集"] end subgraph Measurement["核心指标衡量"] C --> D["统计想法采纳率"] D --> E["统计实验胜率提升幅度"] end subgraph Autonomy["构建自主信任"] E --> F["具备数据驱动的护栏"] F --> G["客户信任自治：全自动上线与复盘"] end ``` --- ### [10:31](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=631s) 反馈循环与“基准刷榜”陷阱（Benchmaxing） 当下 AI 领域最热门的话题无疑是自主反馈闭环（Autonomous Loops）——大家都在憧憬 Agent 不仅能执行任务，还能随着使用时间的增加自我进化、越变越强。但残酷的现实是：Agent 的进化轨迹完全取决于你给它喂入的反馈信号。如果团队仅仅依靠直觉检查（Vibe Checks），Agent 获得的唯一反馈是某个工程师个人的主观喜好，它就会盲目迎合该工程师个人的口味，不断产出该人认为“看起来不错”的内容；反之，如果主要依赖静态打分表或系统化评估集，Agent 就会全力去拟合这份人工制定的评分规则。 这种现象正是大语言模型行业中所谓的“基准刷榜”（Benchmaxing）。像 Anthropic、OpenAI 等顶级实验室发布新模型时，总会展示亮眼的编程基准跑分，但在实际工程环境中，用户的真实体验往往大打折扣。模型过度拟合于刷高基准分，在真实复杂的现实场景中反而频频碰壁。演讲者举例提到，比如近期备受关注的某些新模型（如 Opus 系列更新），虽然测试集跑分极高，但实际体验却遭到大量用户吐槽：由于过于向编程任务过拟合，当用户尝试与它正常对话交流时，它吐出的全是晦涩难懂的胡言乱语（“a bunch of babble”）。评估指标归根结底只是人类对“好想法”的主观抽象映射，很可能脱离客观现实。只有将度量锚定在真实商业产出上——例如“这些想法是否真的提升了线上胜率”——系统才能依据真实有效的反馈机制自我进化，摆脱个人偏见与集体假想的束缚。 - **Vibe Checks 的局限**：模型只会学习个别评估者的偏好，而非真实的有效性。 - **Eval Rubric 的风险**：人工评估准则容易沦为象牙塔标准，导致 Agent 出现“刷分但不可用”的现象。 - **以真实成效（Outcomes）为反馈**：让真实业务指标驱动强化与优化，而非依靠直觉与经验推测。 --- ### [13:13](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=793s) 混杂变量与 A/B 测试的不可替代性 仅仅埋点度量业务数据依然不足以闭环，因为很多人容易陷入“前后对比”的误区。团队很容易抱有侥幸心理：更新了 Agent 算法，一周后发现客户的采纳率和胜率指标确实双双上升，便兴高采烈地认定是 Agent 的升级发挥了作用。然而，真实世界充斥着大量混杂变量（Confounding Variables）：这一周流量是否出现自然增长？是否存在季节性周期？是否正好有更具尝鲜意愿的优质客户访问了系统？ 这正是 A/B 测试存在的核心价值。只有设立严格的对照组（Control Group），让部分用户接收新 Agent 生成的策略、另一部分接收旧版本策略，才能排除噪音，在统计学上坚实地证明“是我的算法改动因果导向了我所期望的业务增长”。这也揭示了为什么 AI 目前在代码编写领域进展神速，而在复杂业务决策中推进较慢的深层原因： 1. **代码领域的验证成本极低**：语法有明确的规范，代码能否编译、测试用例是否跑通几秒内便可知晓，因此模型能极其迅速地完成验证迭代。 2. **人类偏好与转化场景验证昂贵**：如果想验证 Agent 生成的全新结账流程（Checkout Flow）到底好不好，无法单机跑脚本验证，必须接入真实流量、等待真实用户互动产生转化数据，验证周期被拉长。但这正是未来让 Agent 适应人类复杂偏好与真实商业转化的必由之路。 --- ### [15:29](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=929s) 实验飞轮：在失败中修正认知，驾驭过度自信的 Agent 只要将以结果为导向的度量与受控实验嵌入开发流程，Agent 系统的自增强飞轮就会正式转动。每次对 Agent 进行算法或 Prompt 更新时，通过灰度与受控实验跟踪核心业务指标。如果实验确证带来了显著正向提升，该版本即可作为全新生产版本推向全量；若指标下滑，则立即自动回滚或交由人工决策介入。 ```mermaid flowchart TD accTitle: 基于受控实验的 Agent 持续进化飞轮 A["对 Agent 逻辑/Prompt 进行改动"] --> B["受控 A/B 实验分流测试"] B --> C{"业务指标是否真正提升？"} C -- "是" --> D["全量发布新版本 Agent"] C -- "否" --> E["自动回滚或提交人工干预"] B --> F["捕获真实测试数据与结果"] F --> G["发现认知偏差：更新评估集 (Evals)"] G --> H["新增反向回归用例，指导下一轮迭代"] H --> A ``` 更深刻的价值在于，这种实验数据能反哺团队的评估基准。团队制定的评估规则仅仅代表当下对业务的“假设”，每次实验的输赢都是对客观世界真实规律的探索。如果一个在评估集中拿下高分、看似绝妙的想法，在真实 A/B 测试中导致转化率骤降，这就推翻了团队原有的盲目假设——这不仅不是坏事，反而是极为宝贵的新认知。团队可以将这次失败案例直接录入评估集中作为防回归用例（Regression Test），倒逼人工准则与 Agent 逻辑向现实世界靠拢。 演讲在尾声给出了振聋发聩的总结：**“AI Agent 永远都会表现得无比自信（Your agents are always going to be confident），因此你的核心职责，是建立起扎实的度量体系与受控实验机制，确保它们真正在为你所期望的业务成果做正确的事。”** ## Notable quotes > Getting AI agents to act is mostly solved but trusting what they did isn't. Agents fail in a way software doesn't: confidently, returning a wrong answer that looks right. — Josh Holt > An agent's output is a hypothesis, and a controlled experiment is how you learn whether it's true. Unlike an AI grading an AI, a measured outcome can't be talked into a wrong answer. — Josh Holt > [00:04](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=4s) 智能体经常会极其自信地向你“撒谎”，即便给出的答案漏洞百出，它依然表现得笃定不移。 — Josh Holt > [03:15](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=195s) 构思智能体的目的绝非“为了提建议而提建议”，真正的核心指标是：客户是否采纳了这些建议？采纳这些建议后上线的实验，其胜率（Win Rate）是否显著高于人类团队自主构思的方案？ — Josh Holt > [03:15](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=195s) 智能体的输出本质上只是一个“假设”（Hypothesis），唯有通过受控实验测量真实的业务结果，才能闭环证明智能体的实际价值。 — Josh Holt > [06:32](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=392s) 验证的严谨程度应当与“犯错的代价”（Cost of Being Wrong）严格成正比。 — Josh Holt > [09:18](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=558s) 想要让企业客户接受这种高度自主的运作模式，唯一可行的路径就是拿出可证伪、可验证的事实验证。 — Josh Holt > [10:31](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=631s) Agent 的进化轨迹完全取决于你给它喂入的反馈信号。 — Josh Holt > [10:31](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=631s) 评估指标归根结底只是人类对“好想法”的主观抽象映射，很可能脱离客观现实。只有将度量锚定在真实商业产出上……系统才能依据真实有效的反馈机制自我进化，摆脱个人偏见与集体假想的束缚。 — Josh Holt > [13:13](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=793s) 只有设立严格的对照组（Control Group），让部分用户接收新 Agent 生成的策略、另一部分接收旧版本策略，才能排除噪音，在统计学上坚实地证明“是我的算法改动因果导向了我所期望的业务增长”。 — Josh Holt > [15:29](https://www.youtube.com/watch?v=sD2eCfJvjTk&t=929s) AI Agent 永远都会表现得无比自信（Your agents are always going to be confident），因此你的核心职责，是建立起扎实的度量体系与受控实验机制，确保它们真正在为你所期望的业务成果做正确的事。 — Josh Holt ## Mentioned Josh Holt, Optimizely, Ai4, Bill Simmons, Kobe Bryant, Anthropic, OpenAI, Claude Opus ## Video description Josh Holt - Principal Product Leader, Optimizely Getting AI agents to act is mostly solved but trusting what they did isn't. Agents fail in a way software doesn't: confidently, returning a wrong answer that looks right. I build AI agents inside an experimentation platform, and the verification method everyone else is building from scratch was already sitting in my product: the A/B test. An agent's output is a hypothesis, and a controlled experiment is how you learn whether it's true. Unlike an AI grading an AI, a measured outcome can't be talked into a wrong answer. This talk is what building trustworthy agents taught me: treat every output as a hypothesis, match verification rigor to the cost of being wrong, and let measured outcomes, not confidence, close the loop. [Source](https://www.youtube.com/watch?v=sD2eCfJvjTk)

Josh Holt: Getting AI agents to act is mostly solved but trusting what they did isn't. Agents fail in a way software doesn't: confidently, returning a wrong answer that looks right. Josh Holt: An agent's output is a hypothesis, and a controlled experiment is how you learn whether it's true. Unlike an AI grading an AI, a measured outcome can't be talked into a wrong answer. [0:04] Josh Holt: 智能体经常会极其自信地向你“撒谎”，即便给出的答案漏洞百出，它依然表现得笃定不移。 [3:15] Josh Holt: 构思智能体的目的绝非“为了提建议而提建议”，真正的核心指标是：客户是否采纳了这些建议？采纳这些建议后上线的实验，其胜率（Win Rate）是否显著高于人类团队自主构思的方案？ [3:15] Josh Holt: 智能体的输出本质上只是一个“假设”（Hypothesis），唯有通过受控实验测量真实的业务结果，才能闭环证明智能体的实际价值。 [6:32] Josh Holt: 验证的严谨程度应当与“犯错的代价”（Cost of Being Wrong）严格成正比。 [9:18] Josh Holt: 想要让企业客户接受这种高度自主的运作模式，唯一可行的路径就是拿出可证伪、可验证的事实验证。 [10:31] Josh Holt: Agent 的进化轨迹完全取决于你给它喂入的反馈信号。 [10:31] Josh Holt: 评估指标归根结底只是人类对“好想法”的主观抽象映射，很可能脱离客观现实。只有将度量锚定在真实商业产出上……系统才能依据真实有效的反馈机制自我进化，摆脱个人偏见与集体假想的束缚。 [13:13] Josh Holt: 只有设立严格的对照组（Control Group），让部分用户接收新 Agent 生成的策略、另一部分接收旧版本策略，才能排除噪音，在统计学上坚实地证明“是我的算法改动因果导向了我所期望的业务增长”。 [15:29] Josh Holt: AI Agent 永远都会表现得无比自信（Your agents are always going to be confident），因此你的核心职责，是建立起扎实的度量体系与受控实验机制，确保它们真正在为你所期望的业务成果做正确的事。

[0:04] Josh Holt: Getting AI agents to act is mostly solved but trusting what they did isn't. Agents fail in a way software doesn't: confidently, returning a wrong answer that looks right. [0:04] Josh Holt: agents fail in ways traditional software does not. Instead of throwing an error or halting, an agent will look you in the eye and repeatedly lie with total conviction. [0:04] Josh Holt: the smartest thing on God's green earth [3:15] Josh Holt: An agent's output is a hypothesis, and a controlled experiment is how you learn whether it's true. Unlike an AI grading an AI, a measured outcome can't be talked into a wrong answer. [6:32] Josh Holt: treat every output as a hypothesis, match verification rigor to the cost of being wrong, and let measured outcomes, not confidence, close the loop. [9:18] Josh Holt: Hey, instead of me even proposing the ideas to you, you should just trust me... let me implement them and run them for you autonomously and I'll just like report back [10:31] Josh Holt: benchmaxing [3:15] Josh Holt: The agent's output is fundamentally a hypothesis, and the experiment is the objective test of that hypothesis.

[0:04] Uh so over the next 20 minutes, I'm hoping  to get you to start thinking about how the [0:08] output of your agents is something that you  can test and hopefully through that you can [0:12] build confidence that it's actually driving the  outcomes that you're hoping for. My name is Josh. [0:17] I'm on the product team at Optimizely.  If you don't know Optimizely, we make [0:22] uh AI solutions for marketing teams broadly. I  specifically am on the experimentation team. And [0:27] so we have experimentation platform. My customers  are generally trying to increase uh conversions on [0:34] their their digital properties. So their website,  maybe their e-commerce, and they're trying to get [0:38] people, you know, higher order value, increase  retention, etc. And so when I think about AI, [0:43] it's a lot of how I can help our customers run  more highquality tests to drive more impact [0:48] to their business for the things that they care  about, the outcomes that they're trying to drive. [0:52] And so I want to talk through some examples of  the things that I have been building agents for [0:57] my experimentation customers and some of the  things that I've learned along the way. I think [1:01] this is probably something most people here are  familiar with at this point, which is having an [1:06] agent repeatedly lie to you and just extremely  confidently and you know it's wrong. And so this [1:12] is something that I've I've encountered and  you really need to beware be aware of these [1:17] irrationally confident agents. I'm a big Bill  Simmons fan. If you don't know who he is, he uh [1:23] was a former sports writer, now turned podcaster,  and he has this idea of irrationally confident [1:28] players. And so, this is the guy that's like the  seventh best on the team, but when he comes in, [1:33] he's shooting all of the threes he has no business  taking that are never going to go in. Uh, and he's [1:38] triple covered. And the problem is that guy thinks  he's Kobe Bryant, one of the best players of all [1:43] time. But in reality, he's probably more like  me and has no business shooting those shots. [1:49] your agents are going to be the same. They all  think they're the smartest thing on God's green [1:54] earth. Uh and they're not. And so they're going to  be very confident when they're giving you answers [1:57] and they're going to be uh wrong sometimes. And we  need to figure out how do we build mechanisms to [2:01] earn the trust and make sure it's actually doing  the thing that we want. I'm going to come back to [2:06] this example a few times uh throughout. And so one  of the one of the workflows that we help enable [2:12] uh on on my team is helping our customers  think of what ideas what ideas they should do [2:17] uh test on. So um one of the inputs of that  is how much traffic you have on your website. [2:22] You need visitors in order to test. So if  you want to test a new checkout experience, [2:26] you need to have users who we can divert to  different variations and measure their behavior [2:30] to see which variation is is working. So it's  important for us to know how many visitors you [2:34] have on your site because that's going to dictate  where you can test, where you can't, the types of [2:38] tests you could do. And so this is just one little  check in this ideation big workflow where it needs [2:43] to look up how many visitors you have. And as I'm  building this, I kept getting the wrong answer [2:49] um from my agent just for how many visitors it  has. First time it said about 58,000 visitors in [2:54] the last week. Second time something like 13,000.  Uh and second third time was 24,000. And so this [3:01] is a big problem. Um this is one little check  in this whole overall workflow that it's doing [3:06] a bunch of things, but the issue is it's grounding  everything it's doing after this in this number. [3:10] And so this is a big problem that that need to be  solved. And so naturally then you're thinking like [3:15] okay well what are the ways that we can validate  our agent. Generally I would think of this in two [3:20] ways. There's sort of like as you're developing  and iterating agents you're trying to validate [3:24] the system. And so this is going to be a lot of  things I'm sure you've heard about building up [3:28] evals. And so generally this is going to help you  as you're making changes to your agents. You're [3:33] understanding like are we improving on on this or  are we uh not improving. Um and it really gives [3:38] you like an idea of the average of your agent  how it's performing. Um, but differently in the [3:43] example I was giving is you also want to be able  to validate like an individual run like this thing [3:48] that this agent just executed is the thing that it  just told me that 15,000 number the 58,000 number. [3:54] Uh, was that thing right and did it do the thing I  wanted it to do. Ultimately, I think you're trying [4:00] to sort of answer three different questions when  it comes to validating what your agent is doing. [4:04] They're sort of on different time scales. And so  the first is what I was just talking about which [4:08] is sort of like in the development loop as you're  iterating on your agent. You're trying to validate [4:12] what it is really the the system in general.  And so this is like going to be using eval sets [4:18] of outputs that you want to compare against LLM  as a judge. And then there's like runtime every [4:22] time you run your agent you're trying to you're  trying to evaluate the artifact it created. So the [4:27] individual output it got is this good enough? And  so you can use similar techniques there. Compare [4:31] it to like guard rails and see if it it's within  that. uh you can also like put it in front of a [4:36] person human in a loop and have them judge. Now  these first two methods though ultimately are just [4:41] some level of judgment from either a person an  individual or some representation that we've put [4:47] into the system. So if I've created an eval for  saying what is a good test idea that eval system [4:54] is just a representation of what I think a good  test idea is and so it's going to be as flawed [4:58] as my representation of what that test idea is.  And so what I want to talk about is how you can [5:05] actually evaluate the outcome that your agents  are trying to drive through. And that's what we [5:11] do in our product. We run experiments. And so our  customers don't know which version of a checkout [5:16] experience users are going to actually convert  on. So they run a test and they actually validate [5:21] that. Now there's downsides to this. So you need  users and time and traffic and so it's going to [5:27] take more time. But this is the actual way you can  tie what your agent is doing to the actual outcome [5:32] you're trying to drive. I'm building agents for  test ideation, giving you ideas to test. But I'm [5:37] not creating that agent because I just want to  come up with test ideas for test ideas sake. I [5:42] want to create agents that my customer or I want  to give them ideas that my customers want to run. [5:46] So I want to see are you accepting these ideas?  And then ultimately I'm hoping that the ideas we [5:50] can come up with are going to win at a higher rate  when you run those tests than what you would come [5:56] up with otherwise. And so ultimately those are the  outcomes I'm trying to drive. And so it's not just [6:00] enough to have an eval that is saying like okay  is this test idea good based on some criteria. I'm [6:06] trying to measure actually what are these ideas  yielding as the ultimate outcome. And so that's [6:12] sort of the thesis of what I'm talking about  today. I think what your out the output of your [6:16] agent is a hypothesis. An experiment is how you  can actually see is that truly actually driving [6:23] the outcome that you want and as you're making  changes, how can you actually measure it to make [6:26] sure you're moving in the right direction and that  the changes you're making are causing that. So I [6:32] would think we've talked about a few different  methods here. Um, and I would think of it sort of [6:36] in this ladder and there's sort of a scale of how  costly it is to do these things and I think it's [6:42] sort of proportional to the cost of being wrong.  And so in the first example, you're simply just [6:48] going to start by like vibe checks. You're going  to build an agent to do something and you're going [6:52] to look at some outputs and then you're going to  be like, "Okay, this isn't quite right." And then [6:56] go make updates and just keep iterating back and  forth. And if the cost of being wrong isn't much, [7:00] let's say it's like I have an agent that is  scanning my competitors and sending me a report [7:04] of what they've done in the last week. Like that's  just for me. It's not really a big deal. So maybe [7:08] just vibe checks are fine. Like that's not really  big. But as you start are you are delivering [7:13] agentic experience to your customers like you need  to be more thorough and you need to make sure that [7:18] the cost of being wrong is higher. This is where  you start building in things like systematic evals [7:22] what we've just been talking about. You're doing  elements of judge you have these old data sets [7:26] etc for something that's like actually maybe  going to be impactful something that's going to [7:30] take action and you need to be really uh careful  about that you start using human in the loop. Um [7:36] and so the issue there is like it doesn't scale.  Um you know you have an individual user doing [7:41] everything. And as we start thinking about these  more autonomous systems we're trying to build. Um [7:45] the things that are the most costly. So let's say  I'm actually trying to autonomously run the test [7:50] ideas I'm coming up with and actually deploy them  automatically on your site and just report back. [7:55] I better be dang sure that the ideas I'm coming  up with in the thing I'm implementing is really [7:59] really well. And so this is where having actual  core measurement of the outcomes you're trying to [8:04] drive is going to be extremely important um  for something that actually has a high cost [8:08] of being wrong. And so something just to keep  in mind through all this is these first three [8:13] rungs and really you're going to be doing a  mix of all these. It's not just you're going [8:16] to do one or the other. The first three rungs are  really still just able to evaluate the judgment [8:22] of like an individual agent output. The fourth  thing is the only way you can really judge and [8:27] measure is the agent actually driving the  outcome that I'm hoping it will achieve. [8:34] So really I would think you need to do each of  these and there's ways to do them well. And the [8:39] good news is that they each sort of can help feed  other parts of the system. And so in my example [8:45] I was getting three different answers for uh how  many visitors our website was getting in the last [8:50] week. And I was just doing the vibe check analysis  looking at it and I just knew it was wrong. And so [8:56] what I came up with was instead of I was having  in this whole workflow it was like computing the [9:01] number different times and it would do it slightly  differently and that's why it was like completely [9:04] going off. And so the fix for that was I just  computed the value up front once and then passed [9:09] it down and everything else just used the value  that I got. And so that's an example of me just [9:13] looking at it and I basically found an error  scenario and I came up with a fix that now can [9:18] become a test case in your systematic eval set. So  every time you're iterating on your agents, you're [9:24] running them, people, you're seeing issues, people  are reporting issues, those can become new test [9:28] case that you can run in your eval suite. So even  your just self assessments of these things help [9:32] feed into that. And then ultimately if you start  going into measuring what that agent is trying to [9:39] do. So in my example again my agent produces test  ideas for my customers but I'm going to measure [9:45] like how often are my customers actually accepting  and running those ideas and then are those ideas [9:50] winning at a higher clip and as we move towards  these more autonomous loops and we want to give [9:55] more autonomy. putting in this measurement is  what gives you the guardrails and the confidence [10:00] to be able to have things taking more action on  their own because I can verifiably say, "Okay, [10:05] well these ideas, customers really like them and  they're winning at a higher rate." And so if I [10:11] want to move to something where I want to propose  to you and be like, "Hey, you should just instead [10:14] of me even proposing the ideas to you, you should  just trust me that when I propose them to you, [10:19] they're good and then let me implement them and  run them for you autonomously and I'll just like [10:23] report back." The only way you can build that  trust for a customer to be acceptable with [10:26] that and move to that level of autonomy is if you  can verifiably prove something like that that it [10:31] actually works. The big topic these days is really  around these loops. And so I've talked about this [10:36] a little bit. Everyone wants this promise of like  your agents aren't just running and doing a thing, [10:41] but how is this improving on itself over time?  And so the the hope is your agent is always as [10:48] you continue to use it getting better and better  and better. But I think what's important to know [10:52] about this is your agent is going to loop and move  in the direction based on the feedback that you're [10:57] giving it. And so if you're only doing vibe checks  if the only feedback the agent is getting when it [11:03] runs is from me and my assessment, it's going to  optimize for test ideas that I like. If I'm like, [11:08] "Oh, that seems like a good idea," it's going to  keep doing more of that thing. And if I say it's [11:12] a bad idea, it's going to keep doing that other  thing. If you have, let's say, systematic evals [11:16] that you're running and that's primarily your  method, it's going to optimize to that rubric [11:21] that you think it represents what a good test  idea is. In in my case, I think of this, you hear [11:27] the term benchmaxing uh with LLM. So, we'll come  out with a new uh the you know the anthropic open [11:34] AAI everyone. They'll come out with their new  models and then they'll show you all the stats [11:37] on all the like coding benchmarks and related  things. And then sometimes people's experience [11:41] in practice doesn't seem to match what you think  the benchmarks are. And there's an issue where [11:47] the models can be overfitted to just be basically  trying to score well on the benchmark. And maybe [11:53] in practice it's not actually doing well. And you  have a similar problem here because my rubric, [11:58] my evals of what I say is good test ideas is just  my representation of what I think that is. But [12:05] ultimately that might not actually match reality.  And so you'll see like Opus 5 came out recently [12:10] and you're seeing people get a lot of push back  where like it scores really well on benchmarks, [12:14] but people really hate talking to it because  you can't understand like what the heck it's [12:17] saying because it's like so fit to work  on coding use cases that when I ask it [12:22] to like tell me something, it it's giving  me a bunch of babble. And so ultimately, [12:27] if you're measuring things and you're actually  then measuring the outcomes you're hoping that [12:31] this agent is trying to achieve. So in my case,  like do these tests win more? the system is going [12:37] to learn based on like what actually works. The  feedback loop that it's going to get is ideas that [12:42] end up yielding more and more uh positive outcomes  that I'm trying to see. And so you can sort of [12:47] get away from this bias of just any individual  judgment or some collective judgment. And this [12:52] is why we do testing in general. This is why my  product exists. It's really hard for anyone when [12:58] you're trying to make a change to your product  or on your website to understand what behavior [13:02] is going to change based on that. And that is why  people test different variations of things and [13:06] see what actually resonates and truly wins. Now  the kick here is just putting in the measurement [13:13] um isn't quite enough because what you'd  be tempted to do is you're like okay so [13:19] instead of just having evals that is like what  I think is good test ideas. I'm going to have uh [13:24] the measurement in place. So I'm going to measure  how often people are accepting those uh how often [13:28] those are winning. And if I just look at like  last, let's say I make a change to my agent and [13:33] then a week later, uh, those numbers are up. And  I'm like, sweet. All right, over the last week, [13:38] um, both of those numbers went up. But the issue  is, uh, there's so many confounding variables in [13:43] real life. And this is why you need to actually  test things. You need to have a control group so [13:47] we can actually understand like, okay, well, maybe  we were just getting more traffic, uh, at that [13:52] time, their seasonality, maybe the type of people  were coming to my website, like maybe were really [13:56] accepting ideas. And so this is where you run AB  tests. So you can have people who either get the [14:01] new thing or the old thing and you can verifiably  prove like did my change cause the outcome I was [14:05] hoping to see. I think this uh concept of how you  can actually like verify it um is interesting when [14:15] we start thinking about like the types of use  cases that AI has been really well suited for [14:18] today and sort of where we think things are going  to go. And clearly like coding is where things [14:23] have really resonated so far. And I think what  you see is because the cost of verifying that [14:29] the agent is doing the thing we want it to do is  super cheap. We have built-in like understanding [14:33] syntax you code has to compile. We have test cases  and so it's very easy to prove that the thing is [14:39] yielding the outcome you want um because it has  built-in tooling for that. It's very easy to to [14:43] loop the agent towards being better at coding  tests. In the real world when you're trying to [14:48] test your new checkout flow like you don't have  a cheap way of being able to do that. That's why [14:52] you have to run a test to actually verify that  the thing you want is yielding the outcome you [14:56] want. This is where you need to test things and  that requires time and traffic because we need [15:00] real people to actually interact with us. And so  it's coming, but the time to verify this is going [15:07] to take longer because the learning cycles are are  a lot longer. But anything that's leaning towards [15:11] more how do we understand and verify how agents  solve for things that are closer to like human [15:16] preference and taste in testing and conversion  whatever. This is where I think uh having more [15:22] like measured experiments is really going to  come into play in the future. The good news is [15:29] um if you start measuring the outcomes that your  agents are driving towards and start doing this [15:34] practice, you're going to help sort of start  kicking this flywheel of your agentic loop. [15:40] It's going to serve a couple purposes. So one is  every time you're maybe making a change, so I make [15:46] an update to my ideation agent and I'm going to  measure it and test it on the metrics that I care [15:50] about. And I'm going to be able to evaluate is the  change that I made to my agent actually improving [15:55] on the outcomes or not that I want to see. And  I can build in systems then to say if yes, okay, [16:00] cool. That's the new version we're going to deploy  to everybody. If not, let's roll it back or let's [16:03] put it in front of someone to be able to decide.  And so that helps me evaluate is is this agent [16:08] working? and how do I is this change actually  improving in the way I want? But the other [16:12] thing is it's feeding into those other mechanisms  I talked about. And so if your evals is just your [16:19] current understanding of what good looks like for  this agent, for this use case, every time you make [16:24] a change and you run a test, if you either wins  or loses, that means you learn something. You [16:30] learn something that is new truth that you didn't  understand before. And so that helps you feed back [16:34] into your evals to say, okay, well, here's what I  thought represented a good idea, but we actually [16:39] made a change and it got worse. And so maybe my  understanding was off and we learned something. [16:44] So now you have a new test case that you can put  in your evals pro protect against regressions and [16:48] help improve your thing further. And so this is  going to help just naturally improve your agent [16:53] just by instrumenting this type of measurement  and doing this type of controlled experimentation. [16:59] That's the the gist of the talk. Uh, I just want  to leave you with this thought is your agents are [17:04] always going to be confident and so your job is  to put in the measurement and experimentation to [17:09] actually make sure that they're doing the right  thing for the outcome that you want. Thank you.