Amazon AI Lead Revamps Metrics as Agent Completion Rates Conceal Churn

Defining Success Metrics across Generative and Agentic Experiences with Amazon — Ai4

Ai4

Traditional completion and engagement metrics conceal user friction in generative software, requiring product managers to govern tail interaction turns, intervention recovery, and lagged business outcomes.

AI-assisted summary of Defining Success Metrics across Generative and Agentic Experiences with Amazon — original source. Check the original for full context.

Source published: 2026-08-19

Detailed notes

## Background Published by Ai4 on YouTube on August 19, 2026, this talk features Yash Chaturvedi, Head of Product at Amazon. According to the description, Chaturvedi draws from his experience shipping AI products across Prime Video and Amazon Retail to illustrate how traditional deterministic metrics fail in agentic systems, sharing a multi-layer evaluation framework and diagnostic pre-ship questions to properly measure accuracy, user trust, and business impact.

## TL;DR Chaturvedi argues that deterministic software metrics like simple accuracy, completion rate, and turn-based engagement hide severe friction in probabilistic AI products. To solve this, he introduces a three-layer evaluation framework dividing measurement across offline Output Quality (acceptance rates and evals as PRDs), real-time Interaction Quality (P90 turns, intervention rate, and error recovery), and lagged Outcome Quality (downstream business KPIs paired with counter-metrics). He advocates for a focused four-metric dashboard over sprawling telemetry and concludes with four diagnostic questions product teams must resolve before deploying AI agents into production.

## Atmosphere & style The presentation carries the authoritative, pragmatic tone of an enterprise product leader deconstructing industry misconceptions with hard-earned operational insights. Chaturvedi speaks candidly about internal telemetry traps—such as high completion rates masking acute user distrust—giving the talk an urgent, problem-solving momentum. The delivery is structured, analytical, and instructive, transforming complex probabilistic failure modes into clean frameworks and actionable product management principles.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=KRFsHDplaEE&t=4s) Traditional deterministic metrics can mask poor AI user experiences; for example, an agent with a 94% task completion rate still suffered from user backlash and severe loss of trust. - [02:10](https://www.youtube.com/watch?v=KRFsHDplaEE&t=130s) Traditional product development breaks down in generative AI due to three flawed assumptions: that there is always a single right answer, that task completion equals success, and that high engagement/reprompting means user value. - [03:36](https://www.youtube.com/watch?v=KRFsHDplaEE&t=216s) Comprehensive AI evaluation requires a three-layer framework operating at different cadences: Output Quality (offline pre-release), Interaction Quality (real-time live sessions), and Outcome Quality (lagged downstream business metrics). - [03:36](https://www.youtube.com/watch?v=KRFsHDplaEE&t=216s) If every metric on an AI product dashboard updates at the exact same frequency or cadence, the team is missing at least one critical evaluation layer. - [04:49](https://www.youtube.com/watch?v=KRFsHDplaEE&t=289s) Evaluations (evals) are the new Product Requirement Documents (PRDs); product managers must define binary rubrics and acceptance rates rather than relinquishing product definition to engineering or legal teams. - [04:49](https://www.youtube.com/watch?v=KRFsHDplaEE&t=289s) While LLM-as-a-judge enables scalable offline evaluation across broad markets, human acceptance criteria drift over time, necessitating continuous recalibration of judge models. - [09:07](https://www.youtube.com/watch?v=KRFsHDplaEE&t=547s) For interaction quality, medians hide extreme failure modes; teams should ignore median turns to success and strictly optimize for P90 or P95 tail metrics. - [09:07](https://www.youtube.com/watch?v=KRFsHDplaEE&t=547s) Using a transition matrix that maps the last successful step to the first failed step allows engineering teams to pinpoint exact operational bottlenecks in agentic workflows. - [09:07](https://www.youtube.com/watch?v=KRFsHDplaEE&t=547s) Teams must track recovery rate and intervention rate to ensure that inevitable in-flight agent errors do not lead to total session abandonment and customer attrition. - [12:34](https://www.youtube.com/watch?v=KRFsHDplaEE&t=754s) To govern AI metrics effectively, teams must focus on a single North Star KPI, pair every optimization metric with a counter-metric, and separate metric ownership from optimization. - [12:34](https://www.youtube.com/watch?v=KRFsHDplaEE&t=754s) The ideal production AI dashboard balances four key metrics: one lagged downstream business outcome KPI and three real-time interaction guardrails (Intervention Rate, Recovery Rate, and P90 Turns to Success). - [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) Before launching AI products, teams must verify acceptance criteria ownership, in-product recovery capabilities, intervention rate trends, and the lag time to downstream business outcomes.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=KRFsHDplaEE&t=4s) The Flaw of Deterministic Metrics in AI Products

When teams transition from deterministic software to generative and agentic systems, standard telemetry often conceals catastrophic user experiences. Yash Chaturvedi, Head of Product Management for Generative and Agentic AI at Prime Video, illustrates this with a recent deployment: his team launched an AI agent that registered a seemingly stellar 94% task completion rate on internal dashboards. However, qualitative customer feedback told a completely different story. Users reported that they hated the experience and explicitly stated, "we don't trust your AI." The metric signaled resounding success, but the actual customer experience was a failure.

Having shipped AI products across Prime Video, Amazon Retail, marketing automation, and advertising—and holding nine or ten approved patents in generative AI—Chaturvedi argues that metrics inherited from deterministic software fail when applied to generative and agentic workflows. Because deterministic assumptions collapse in probabilistic environments, product teams require a fundamentally different mental model and evaluation framework.

- **Speaker Credentials**: Yash Chaturvedi leads product management for GenAI and Agentic AI at Prime Video, with 9–10 approved patents across retail, marketing automation, and entertainment. - **The Telemetry Trap**: A 94% completion rate masked deep customer dissatisfaction and erosion of trust.

---

[02:10](https://www.youtube.com/watch?v=KRFsHDplaEE&t=130s) Three Flawed Assumptions of Traditional Software Metrics

Traditional product development relies on three core assumptions that break down in generative AI:

1. **"There is always a right answer"**: Deterministic software operates on binary correctness. In contrast, generative models sample outputs from a broad probability distribution. Because only a subset of possible outputs is acceptable, simple accuracy is the wrong question to ask. 2. **"Completion equals success"**: An agent might successfully purchase an item after forcing a user through fourteen confusing, repetitive conversational turns. While the task technically reached completion, the user suffered through immense friction. Chaturvedi compares grading an agent solely on task completion to evaluating a surgeon based solely on whether "the patient is able to walk out of the hospital." 3. **"Engagement means value"**: In generative interactions, high turn counts and frequent reprompts are often mistaken for deep user engagement. In reality, five consecutive reprompts represent pure friction and user struggle rather than value creation.

```mermaid flowchart TD accTitle: Three Flawed Assumptions vs Reality in GenAI subgraph Flawed["Flawed Deterministic Assumptions"] A1["There is always one right answer"] A2["Task completion equals success"] A3["High engagement means high value"] end

subgraph Reality["Probabilistic GenAI Realities"] R1["Outputs are samples from a distribution"] R2["14 confusing turns is a failed UX"] R3["Frequent reprompts represent user friction"] end

A1 --> R1 A2 --> R2 A3 --> R3 ```

---

[03:36](https://www.youtube.com/watch?v=KRFsHDplaEE&t=216s) The Three-Layer Evaluation Framework

To accurately capture system performance and user experience, evaluation must be divided into three distinct operational layers: Output Quality, Interaction Quality, and Outcome Quality. Each layer operates at a different temporal cadence. Output quality is evaluated offline prior to every model or product release; interaction quality is tracked live during every user session; and outcome quality is measured downstream as a lagged business metric on a weekly or quarterly basis.

Chaturvedi provides a simple litmus test for product telemetry: if every metric on a product dashboard updates at the same cadence, the team is missing at least one critical layer of measurement.

```mermaid flowchart LR accTitle: Three-Layer Evaluation Framework subgraph L1["Layer 1: Output Quality"] direction TB OQ1["Offline Pre-Release"] OQ2["Acceptance Rate & Evals"] end

subgraph L2["Layer 2: Interaction Quality"] direction TB IQ1["Live Session Telemetry"] IQ2["P90 Turns, Edits, Recovery"] end

subgraph L3["Layer 3: Outcome Quality"] direction TB OC1["Lagged Downstream KPIs"] OC2["Business ROI & Retention"] end

L1 --> L2 --> L3 ```

- **Output Quality Cadence**: Measured offline with every model checkpoint and build release. - **Interaction Quality Cadence**: Measured live with every real-time customer session. - **Outcome Quality Cadence**: Lagged business metrics tracked weekly or quarterly. - **The Dashboard Litmus Test**: Identical update frequencies across all dashboard metrics indicate an incomplete evaluation architecture.

---

[04:49](https://www.youtube.com/watch?v=KRFsHDplaEE&t=289s) Layer 1: Output Quality, Evals as PRDs, and LLM-as-a-Judge

At the output quality layer, teams must abandon exact-match accuracy and instead evaluate the "acceptance rate"—the percentage of generated artifacts that clear a predefined quality bar. In agentic workflows, this requires scoring both the final produced artifact and the intermediate trajectory taken to get there.

Defining this bar has become a fundamental product management responsibility. Across the industry, "evals are the new product requirement documents (PRDs)." Product managers who do not write evaluation suites effectively surrender product definition to whichever internal group sets the guardrails.

At Prime Video, generative pipelines produce marketing trailers and promotional artwork across 200 territories. Quality is treated not as a static specification, but as an acceptable spectrum defined by binary rubrics covering brand safety, legal compliance, and zero-edit publishability. Teams establish golden datasets using human-in-the-loop (HITL) domain experts to define launch gates. Because human review cannot scale across hundreds of territories, evaluation is scaled using LLM-as-a-judge. However, because human tolerance and acceptance drift over time, teams must periodically recalibrate guardrails and feed updated criteria back into the judge models.

- **Evals Ownership**: Product managers must hold the pen on evaluation rubrics to maintain control of product definition rather than relinquishing it to legal or science teams. - **Binary Launch Gates**: Rubrics evaluate whether assets are brand-safe, compliant, and publishable without manual human edits. - **LLM-as-a-Judge Drift**: Fixed LLM guardrails degrade relative to evolving human preferences, requiring regular calibration loops.

---

[09:07](https://www.youtube.com/watch?v=KRFsHDplaEE&t=547s) Layer 2: Interaction Quality and Failure Mapping

Interaction quality measures the operational cost imposed on the user to achieve their goal. For generative editing tools, this is tracked via edit distance—measuring the delta between the model’s initial draft and the final exported artifact. For agentic systems, teams evaluate "turns to success."

Turn counts follow a wide distribution where medians obscure critical failure modes. A workflow might show an encouraging median of 3 turns, but possess a P90 of 11 turns—the exact tail where frustrated users abandon the product. Teams should ignore the median and optimize strictly for P90 or P95 metrics.

```mermaid flowchart TD accTitle: Transition Matrix for Agent Failure Tracing subgraph TransitionMatrix["Transition Error Matrix"] direction TB S1["Step N: Last Successful Step\n(e.g., Generate Correct SQL)"] S2["Step N+1: First Failed Step\n(e.g., Database Execution Timeout)"] S1 -->|Trace Failure| S2 end S2 --> Act["Pinpoint Engineering Hotspot\n(Fix execution timeout, not SQL prompt)"] ```

Interaction telemetry also tracks intervention rate (how frequently humans override, edit, or abandon actions) and recovery rate (the percentage of sessions that recover after an agent encounters an error). Because agents often fail with high confidence, poor recovery results in permanent loss of user trust. To diagnose failures, teams construct a transition matrix mapping the last successful step against the first failed step. This isolates the exact failure hotspot—such as distinguishing between an agent drafting incorrect SQL versus failing at query execution—removing guesswork from engineering resource allocation.

- **P90/P95 over Median**: Medians hide extreme user frustration; product managers should monitor tail latency and tail turn counts. - **Intervention Rate Targets**: The goal is near-zero intervention, but a literal zero target is unrealistic and can signal that users are avoiding complex tasks entirely. - **Recovery Rate**: Determines whether an in-flight error causes total session abandonment and customer attrition. - **Transition Matrices**: Mapping the last successful step to the first failed step pinpoints the exact engineering intervention required.

---

[12:34](https://www.youtube.com/watch?v=KRFsHDplaEE&t=754s) Layer 3: Outcome Quality and the 4-Metric Dashboard Rule

Outcome quality measures lagged downstream business value, such as customer retention, revenue, and sustained repeat usage. In an initiative involving virtual product placement in Prime Video streams, early output metrics were strong: inserted brand posters looked natural, and internal reviewers consistently approved them. The true test of outcome quality, however, was whether advertisers returned to purchase placements repeatedly over subsequent quarters.

To align product and science teams, organizations should enforce three governance rules: 1. **Focus on a single primary North Star KPI** rather than shifting optimization goals between retention and monetization mid-flight. 2. **Pair every optimization metric with a counter-metric** to prevent teams from gaming the user experience. 3. **Separate ownership**: Never permit the team optimizing a metric to define its measurement criteria.

Chaturvedi notes that while product managers typically want a single summary number and data scientists request dashboards with forty granular variables, the optimal operational dashboard consists of four balanced metrics: one lagged outcome KPI and three real-time interaction guardrails.

---

[15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) Four Questions to Ask Before Shipping AI Products

Chaturvedi concludes with four essential diagnostic questions every product team must answer before deploying generative or agentic AI into production:

- **What does acceptance look like, and who is deciding those criteria?** Ensure product managers define the explicit rubrics rather than delegating the product definition. - **When the agent fails, can the customer recover in-product?** Verify that fallback mechanisms and human-in-the-loop workflows allow users to gracefully recover from failures. - **Is the intervention rate trending toward trust or toward abandonment?** Monitor whether human interventions reflect productive collaboration or growing user frustration leading to churn. - **Which downstream business outcome lags your quality metrics, and by how much?** Establish clear proxy relationships between offline output benchmarks, real-time interaction metrics, and long-term business impact.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=KRFsHDplaEE&t=4s) we don't trust your AI. — Yash Chaturvedi

> [02:10](https://www.youtube.com/watch?v=KRFsHDplaEE&t=130s) There is always a right answer — Yash Chaturvedi

> [02:10](https://www.youtube.com/watch?v=KRFsHDplaEE&t=130s) Completion equals success — Yash Chaturvedi

> [02:10](https://www.youtube.com/watch?v=KRFsHDplaEE&t=130s) Engagement means value — Yash Chaturvedi

> [02:10](https://www.youtube.com/watch?v=KRFsHDplaEE&t=130s) the patient is able to walk out of the hospital. — Yash Chaturvedi

> [04:49](https://www.youtube.com/watch?v=KRFsHDplaEE&t=289s) evals are the new product requirement documents (PRDs). — Yash Chaturvedi

> [09:07](https://www.youtube.com/watch?v=KRFsHDplaEE&t=547s) turns to success — Yash Chaturvedi

> [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) What does acceptance look like, and who is deciding those criteria? — Yash Chaturvedi

> [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) When the agent fails, can the customer recover in-product? — Yash Chaturvedi

> [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) Is the intervention rate trending toward trust or toward abandonment? — Yash Chaturvedi

> [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) Which downstream business outcome lags your quality metrics, and by how much? — Yash Chaturvedi

## Mentioned Yash Chaturvedi, Amazon, Prime Video, Amazon Retail, Ai4

## Video description Yash Chaturvedi - Head of Product, Amazon

When an AI agent completes a multi-step task, how do you actually know it succeeded? As generative and agentic experiences move from demos into production, most teams are still applying metrics built for deterministic software, and those metrics quietly miss what these systems get wrong. I’ll draw on my experience shipping AI products at Amazon to show where those metrics break down. I’ll share a practical framework for defining success across generative outputs and multi-step agent workflows, balancing accuracy, user trust, and business impact. You will leave with a concrete set of questions to run against their own product before it ships.

[Source](https://www.youtube.com/watch?v=KRFsHDplaEE)

中文简报

智能体完成率掩盖用户流失，亚马逊AI主管重塑评估指标传统的完成度与参与度指标掩盖了生成式软件中的用户摩擦，要求产品经理必须对长尾交互轮次、干预恢复和滞后业务成果进行管控。 Chaturvedi 指出，单纯的准确率、完成率以及基于轮次的参与度等确定性软件指标，掩盖了概率型 AI 产品中严重的用户摩擦。为解决该问题，他提出了一个三层评估框架，将度量划分为离线“输出质量”（将验收率与评测作为 PRD）、实时“交互质量”（P90 交互轮次、干预率及错误恢复）以及滞后“成果质量”（结合反向指标的下游业务 KPI）。他提倡使用聚焦核心四项指标的仪表盘，而非泛滥庞杂的遥测数据，并总结了产品团队在将 AI 智能体部署至生产环境前必须解决的四个诊断性问题。传统确定性软件指标往往掩盖较差的 AI 体验；例如某 AI Agent 虽达到 94% 的任务完成率，却因极高的交互摩擦遭遇严重的用户反弹与信任崩塌。 (0:04) 生成式 AI 的评估必须分为三个不同周期的层级：离线发布前的输出质量（Output Quality）、实时会话中的交互质量（Interaction Quality），以及滞后的业务结果质量（Outcome Quality）。 (3:36) 评估体系（Evals）已成为新的产品需求文档（PRD），产品经理必须亲自定义二元质量标准与验收率，避免将产品定义权让渡给技术或法务团队。 (4:49) 在交互质量方面，中位数往往掩盖灾难性失败，团队应严格优化 P90 或 P95 尾部交互轮次，并通过映射最后成功步骤与首个失败步骤的状态转移矩阵精准定位工程瓶颈。 (9:07) 推荐采用 4 指标监控面板，包括 1 个滞后的北极星业务 KPI 及 3 个实时交互护栏指标（人工干预率、错误恢复率、P90 成功轮次），并为每个优化指标绑定反向制衡指标。 (12:34)

Reading summary

Yash Chaturvedi, Head of Product Management for Generative and Agentic AI at Prime Video, argues that deterministic software telemetry fails in probabilistic environments by confusing completion and repetitive engagement with actual user value. To prevent misleading dashboards from masking user frustration and distrust, product teams must adopt a three-layer evaluation framework operating across separate operational cadences. The Breakdown of Deterministic Product Telemetry (0:04) Traditional product development relies on three core assumptions: binary correctness, task completion equaling success, and high engagement indicating value. In probabilistic generative systems, these assumptions collapse. Outputs represent samples from a broader distribution rather than single correct answers, while repetitive conversational turns and reprompts typically reflect friction rather than genuine engagement. Chaturvedi illustrates this failure with an internal agent deployment that achieved a 94% task completion rate on dashboards while user feedback revealed deep dissatisfaction and erosion of trust. He compares evaluating an agent solely on task completion to judging a surgeon exclusively on whether a patient walks out of the hospital, noting that technical task completion can still impose an unacceptably frustrating experience. A Three-Tier Evaluation Framework Across Cadences (3:36) To accurately capture AI product performance, evaluation must be divided across three distinct temporal cadences: Output Quality, Interaction Quality, and Outcome Quality. Output quality is evaluated offline prior to every model release using acceptance rates against benchmark golden datasets. Interaction quality tracks live session telemetry to measure user friction and operational effort during execution. Outcome quality monitors lagged downstream business impact on a weekly or quarterly basis. Chaturvedi introduces a dashboard litmus test based on this separation: if every metric on a product dashboard updates at the exact same frequency, the team is missing at least one critical layer of measurement. Output Quality Rubrics and Interaction Failure Tracing (4:49) At the output layer, product managers must define the acceptance bar via explicit rubrics covering brand safety, compliance, and zero-edit publishability, with Chaturvedi asserting that evaluations have become the new product requirement documents. For Prime Video promotional assets spanning 200 territories, golden datasets and LLM-as-a-judge pipelines automate quality gates, though rubrics require regular recalibration as human tolerances drift. For live interaction quality, teams should discard median metrics in favor of P90 or P95 turn counts. A workflow showing an encouraging median of 3 turns can conceal a P90 tail of 11 turns where users abandon the system. Telemetry should also track intervention and recovery rates, using transition matrices that map the last successful step to the first failed step to isolate engineering hotspots such as query execution timeouts rather than prompt drafting errors. Metric Governance and the Four-Metric Dashboard (12:34) Aligning product and science teams requires strict operational governance: focusing on a single North Star KPI, pairing every optimization metric with a counter-metric to prevent gaming, and never allowing the team optimizing a metric to define its measurement criteria. In Prime Video's virtual product placement tests, visually realistic outputs still required long-term validation through repeat quarterly advertiser spend. Rather than relying on single aggregate scores or forty-variable dashboards, Chaturvedi recommends a four-metric dashboard. This architecture pairs one lagged business outcome metric with three real-time interaction guardrails: intervention rate, recovery rate, and P90 turns to success. Chaturvedi outlines four diagnostic questions before shipping AI products: verifying who defines acceptance rubrics, ensuring in-product recovery after agent failure, determining whether intervention trends toward trust or churn, and identifying the lag between quality benchmarks and business impact. A persistent operational challenge remains managing the drift of LLM judges against evolving human standards.

精炼阅读摘要

Prime Video生成式与Agent AI产品负责人Yash Chaturvedi指出，传统确定性软件的遥测指标在概率型AI环境中容易形成严重误导，94%的任务完成率等表面数据往往掩盖了极差的用户信任度。评估生成式与Agent系统必须摒弃单一精度指标，建立由离线输出质量、实时交互质量与滞后结果质量构成的三层分级度量体系。确定性度量的三大误区与遥测陷阱 (0:04) Chaturvedi指出，传统软件评估依赖三项在生成式AI中彻底失效的核心假设。首先，“必有标准答案”的假设崩溃，因为生成式模型是从概率分布中抽样输出，单纯度量二元准确率并不合理，核心在于符合预期的接受率。其次，“完成即成功”的假设存在严重盲区；如果一个Agent迫使用户经历14轮混乱且重复的对话才完成购物，该交互体验实际上已经失败，Chaturvedi将此比作仅凭“病人能否自行走出医院”来评判外科手术的好坏。最后，“高互动等同于高价值”的推论同样不成立，用户连续五次修改提示词重新提问（reprompt）反映的是严重的系统阻力与挫折感，而非深度参与。在Prime Video的早期部署中，一个表面达成94%完成率的Agent，其定性反馈却是用户普遍表示反感且不再信任该AI。输出质量：评测集代行PRD与裁判校准 (4:49) 在离线输出质量层，产品团队需要放弃精确匹配逻辑，转而通过设定质量阈值来评估“接受率”，并在Agent工作流中同步评定最终成果与执行轨迹。业内正在确立“评测集即新产品需求文档（PRD）”的规范，产品经理若不亲自制定评测基准，将丧失对产品标准的定义权。以Prime Video面向200个国家和地区生成营销素材为例，团队借助人机协同（HITL）由领域专家沉淀黄金基准数据集，设定品牌安全、合规与免人工编辑即可直接发布的二元红线。鉴于跨地区人工复核的规模限制，系统依赖大模型担任裁判（LLM-as-a-judge）进行门禁打分；但由于人类对模型表现的容忍度与标准会随时间漂移，团队必须定期校准规则并将新标准重新输入裁判模型。交互质量：P90尾部损耗与故障转移矩阵 (9:07) 交互质量评估的是用户达成目标过程中付出的操作代价。对于编辑类工具，该维度可通过初稿与最终成果间的“编辑距离”来量化；对于Agent系统，核心指标则是“达成成功的交互轮次”。Chaturvedi强调，轮次分布的中位数会掩盖关键失败场景，例如中位数为3轮的良好表象下，P90分位数可能高达11轮，而这正是用户因沮丧而弃用产品的危险区间，因此团队必须针对P90或P95等尾部表现进行优化。此外，实时遥测还需追踪人工干预率和出错后的会话恢复率。为了精确定位Agent在极高自信状态下的隐蔽失败，团队可建立比对“最后成功步骤”与“首个失败步骤”的故障转移矩阵，借此明确区分模型生成错误SQL还是底层查询执行超时，从而消除工程排查的猜测成分。结果质量：制衡机制与四指标看板法则 (12:34) 结果质量度量的是按周或季度滞后显现的下游商业价值，如客户留存率、营收与复购行为。在Prime Video的流媒体虚拟商品植入实践中，尽管初期内部评审对画面的自然度普遍认可，但真正的结果检验在于广告商后续季度是否持续复购。为防范团队刷取指标，组织须遵守三项治理规则：设定单一的北极星核心指标以防中途摇摆、为每个优化指标配对一个反向制衡指标（counter-metric）、以及严格分离指标优化团队与度量标准制定团队的权限。在看板设计上，面对产品经理偏好单一代号与数据科学家堆砌40项变量的矛盾，Chaturvedi提出四指标看板原则：由1个滞后核心结果指标，搭配人工干预率、故障恢复率与P90成功轮次这3个实时交互护栏指标共同组成。 Chaturvedi总结指出，在发布生成式或Agent系统前，团队必须明确验收标准的定义归属、应用内的故障自愈能力、干预率是否体现信任建立，以及线下指标与长期商业结果之间的滞后关系。当前概率型系统面临的持续挑战在于，由于人类用户心智与容忍度呈现动态演进，若缺乏定期的校准循环，既有的自动化评测与裁判机制将逐步与真实体验产生偏离。

Detailed notes — zh

## Background 该视频由 YouTube 频道 Ai4 于 2026 年 8 月 19 日发布，主讲人为亚马逊 Prime Video 生成式与智能体 AI 产品负责人 Yash Chaturvedi。视频简介指出，随着 AI Agent 从 Demo 走向生产环境，传统确定性软件的评估指标已无法准确捕捉系统的隐性失效，演讲结合亚马逊落地经验，分享了一套平衡准确度、用户信任与业务价值的三层度量框架及上线前自检清单。 ## TL;DR 演讲首先指出传统确定性指标（如精确匹配率、单纯的任务完成率和高交互黏性）在生成式与 Agent 场景中的失效，解释了即便完成率高达 94% 也可能因冗余多轮交互而引发用户抵触的现实困境。为此，演讲者提出了包含离线产出质量（Output Quality）、在线实时交互质量（Interaction Quality）以及滞后业务成果（Outcome Quality）的三层度量评估模型。在该框架下，产品经理需以评测集替代传统 PRD，重点追踪 P90/P95 达成轮次、人工介入率与状态转移矩阵错误归因。最终，团队应构建精简的四指标监控看板，并通过上线前的四大核心自问确保系统具备真正的容错自愈能力与商业转化价值。 ## Atmosphere & style 整场演讲呈现出务实、犀利且极具实战沉淀的工业界风格。主讲人以团队亲历的翻车事故开篇，毫不避讳地破除产品与算法团队常见的虚荣指标和认知误区，用词直击要害。叙事节奏层层递进，从尖锐的痛点剖析无缝过渡到系统性的架构工具与落地清单，给听众带来如同置身头部大厂核心产品评审会般的硬核与清醒感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=KRFsHDplaEE&t=4s) 传统确定性软件的核心假设在生成式与 Agent AI 中全面失效，仅盯“任务完成”和“高互动”会掩盖极其糟糕的用户体验。 - [00:04](https://www.youtube.com/watch?v=KRFsHDplaEE&t=4s) 生成式系统输出属于概率分布采样，团队应放弃精确匹配准确率，转向衡量符合及格线的验收率（Acceptance Rate）。 - [03:36](https://www.youtube.com/watch?v=KRFsHDplaEE&t=216s) AI 评估需建立三层度量框架：离线触发的产出质量、在线实时的交互质量，以及具备时间滞后性的成果质量。 - [03:36](https://www.youtube.com/watch?v=KRFsHDplaEE&t=216s) 如果监控看板上的所有指标都以相同的速度和频率刷新，说明该体系必然遗漏了关键层级的指标。 - [04:49](https://www.youtube.com/watch?v=KRFsHDplaEE&t=289s) 评测集即新的 PRD（Evals are the new PRDs），产品经理若不亲自编写评测集，实际上就放弃了对产品的定义权。 - [09:07](https://www.youtube.com/watch?v=KRFsHDplaEE&t=547s) 交互质量中衡量达成轮次切忌使用中位数，必须紧盯 P90 或 P95 指标，避免长尾极端步骤导致用户挫败流失。 - [09:07](https://www.youtube.com/watch?v=KRFsHDplaEE&t=547s) 人工介入率直接映射用户对 AI 的信任度，其优化目标是趋近于零而非等于零，需防范用户因放弃而导致的伪零介入。 - [09:07](https://www.youtube.com/watch?v=KRFsHDplaEE&t=547s) 通过“状态转移矩阵”对比最后成功步骤与首个失败步骤，可精准绘制错误热力图，避免工程优化盲目试错。 - [12:25](https://www.youtube.com/watch?v=KRFsHDplaEE&t=745s) 设计指标大盘应专注单一业务核心 KPI，并为每个优化指标设立反向约束指标，防止团队刷指标伤害全局体验。 - [12:25](https://www.youtube.com/watch?v=KRFsHDplaEE&t=745s) 高效大盘应精简为 4 项关键指标：1 项离线产出质量、3 项在线交互护栏（P90 轮次、介入率、恢复率）及 1 项滞后业务成果。 - [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) 产品交付生产环境前需自检 4 个核心问题：验收裁决权、产品内自愈容错能力、人工介入信任趋势以及滞后指标转化路径。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=KRFsHDplaEE&t=4s) 确定性软件指标的失效与大模型时代的认知误区在人工智能席卷各行各业的当下，如何真正评估生成式与 Agent（智能体）AI 产品的成功成了一个核心难题。演讲者 Yash Chaturvedi 现任亚马逊 Prime Video 生成式与智能体 AI 产品管理负责人，拥有 9 到 10 项获批的生成式 AI 专利，曾主导零售、营销自动化及广告领域的诸多 AI 落地。他以一个月前团队亲身经历的“翻车事故”切入：团队上线了一款 AI Agent，监控大盘显示任务完成率高达 94%，表面上看大获全胜；然而收集到的真实用户原声（Anecdotes）却泼了一盆冷水——用户极其厌恶该功能，直言“我们根本不信任你们的 AI”。指标显示“成功”，但用户却用脚投票，这正是为传统确定性软件构建的指标在生成式与 Agent 体验中全面失效的典型体现。传统软件评估建立在三项已经过时的核心假设之上： 1. **“总有唯一正确答案”（There is always a right answer）**：传统确定性系统中输入与输出是一对一的，但在生成式系统中，模型输出只是整个概率分布中的一次采样，部分输出可以接受，多数可能不可用，因此一味盯住“精确匹配准确率”（Exact Match Accuracy）完全问错了问题。 2. **“完成即成功”（Completion equals success）**：一个 Agent 经过 14 轮极其混乱、令人困惑的对话最终帮用户下单了一件商品。从系统日志上看任务确实完成了，但用户经历了 14 次反复重试与挫败，整个过程体验极差。Yash 打了一个生动的比方：这就像外科医生把“病人只要能活着走出医院”当成手术成功的唯一指标一样荒谬。 3. **“高互动等于高价值”（Engagement means value）**：用户进行了 5 次重新提示（Reprompting），在传统数据分析中看似是极高的“用户黏性/参与度”，但实际上是用户为了完成任务所承受的巨大摩擦力。 ```mermaid flowchart TD accTitle: 传统确定性假设与生成式现实的错位 subgraph Legacy["传统确定性软件假设（失效）"] A1["假设 1：总有唯一正确答案"] A2["假设 2：任务完成 = 业务成功"] A3["假设 3：高交互/重试 = 高用户价值"] end subgraph Reality["生成式与 Agent 现实（新挑战）"] B1["分布采样：准确率指标失效，需看接受率"] B2["过程痛苦：14 轮冗余交互即便完成也是失败"] B3["交互摩擦：频繁 Prompt 重试代表体验摩擦"] end A1 --> B1 A2 --> B2 A3 --> B3 ``` --- ### [03:36](https://www.youtube.com/watch?v=KRFsHDplaEE&t=216s) 生成式与 Agent AI 的三层度量评估框架为了解决上述断层，Yash 提出了一套兼顾生成式内容与多步 Agent 工作流的三层度量框架。这三层指标并非平行更新，而是具备不同的时间尺度与评估场域： | 评估层级 | 核心关注问题 | 评估方式与触发周期 | 典型度量示例 | |:--- |:--- |:--- |:--- | | **第一层：产出质量（Output Quality）** | 生成的产物（Artifact）有多好？ | 离线评测（Offline），伴随每次模型与版本发布 | 验收率（Acceptance Rate）、黄金测试集通过率 | | **第二层：交互质量（Interaction Quality）** | 用户为了完成任务付出了多大代价？ | 在线实时（Live），伴随每次会话与交互产生 | P90/P95 达成轮次、人工介入率、错误恢复率 | | **第三层：成果质量（Outcome Quality）** | 是否真正达成了预期的业务增长结果？ | 滞后观测（Lagged），按周/按季度追踪的业务 KPI | 客户留存率、广告主复购率、ROI | Yash 强调了一个检验团队指标大盘的“试金石”（Litmus Test）：**如果看板上的所有指标都以相同的速度和频率刷新，说明指标体系绝对漏掉了关键层级。** 只有离线、实时与滞后指标相互配合，才能还原产品的真实运行状态。 ```mermaid flowchart LR accTitle: AI 成功评估的三层架构模型 subgraph Layer1["第一层：产出质量 (Output Quality)"] L1A["离线评估 (Offline)"] --> L1B["伴随模型/版本发布触发"] end subgraph Layer2["第二层：交互质量 (Interaction Quality)"] L2A["在线实时 (Live)"] --> L2B["伴随用户会话实时追踪"] end subgraph Layer3["第三层：成果质量 (Outcome Quality)"] L3A["滞后指标 (Lagged)"] --> L3B["周/季度业务 KPI 验证"] end Layer1 --> Layer2 --> Layer3 ``` --- ### [04:49](https://www.youtube.com/watch?v=KRFsHDplaEE&t=289s) 第一层：产出质量（Output Quality）与“评测即 PRD” 在产出质量层，团队必须果断放弃确定性的精确匹配（Exact Match），转向衡量**验收率（Acceptance Rate）**——即生成内容达到预定义及格线的百分比。对于纯生成式场景，重点评估最终产物；对于 Agent 场景，则既要评估最终产物，也要评估其推演路径。当下 AI 产品经理的角色正在发生本质变化：“评测就是新的产品需求文档（Evals are the new PRDs）”。Yash 告诫团队中的产品经理：“如果你不去亲手编写评测集（Evals），你实际上就是把产品的定义权拱手让给了别人。”在亚马逊 Prime Video 跨越全球 200 多个国家和地区规模化生成营销预告片和海报物料的业务中，一张“好图片”不再是一个死板的规格参数，而是一个接受区间。产品团队制定了二元评分准则（Binary Rubric），严格审核内容是否符合品牌安全、合规要求以及是否达到无需人工修改即可直接发布的标准。 - **专家人机协同（HITL）与黄金数据集**：由领域专家在人机协同模式下构建高标准的黄金基准数据集（Golden Data Set），将通过率转化为上线把关标准（Launch Gate），每一次模型失败的模式都会被沉淀为后续版本的评估项。 - **大模型作为裁判（LLM as a Judge）与评估权之争**：法务团队希望掌控合规规则，算法团队希望定义模型缩放指标，但产品经理必须主导评估标准的制定。面对海量吞吐，团队采用 LLM 作为裁判来进行大规模打分。 - **人类接受度衰退与动态校准**：实践发现，当一套安全护栏建立后，人类用户对其接受度和容忍度会随时间推移而退化（Deteriorate）。因此团队需要定期复盘、重塑准则，将最新的用户偏好重新灌入 LLM 裁判中形成闭环。 --- ### [09:07](https://www.youtube.com/watch?v=KRFsHDplaEE&t=547s) 第二层：交互质量（Interaction Quality）与错误根因分析交互质量聚焦于用户在使用产品时的成本与阻力。对于生成式工具，关键指标是**编辑距离（Edit Distance）**，即模型初稿与用户最终导出/保留版本之间的修改幅度；而对于 Agent 体验，核心在于追踪**达成成功的轮次（Turns to Success）**、**人工介入率（Intervention Rate）**以及**错误恢复率（Recovery Rate）**。在衡量达成轮次时，Yash 尖锐地指出：“每当有产品经理跟我谈达成轮次的‘中位数’（Median）时，我都会忍不住翻白眼。”中位数往往会掩盖严重的体验缺陷——例如在某次实际测试中，达成任务的中位数仅为 3 步，但 P90 却高达 11 步。正是这些处于长尾分布（Tail-end）的极端步骤导致用户产生强烈挫败感并彻底流失，因此团队必须紧盯 **P90 或 P95 指标**。此外，人工介入率（用户接管、覆盖、放弃的频率）直接映射了人对 AI 的信任度，其优化目标是“趋近于零”而非绝对的“等于零”——若完全没有人工介入，通常意味着 AI 根本没有在关键场景发挥作用。 ```mermaid flowchart TD accTitle: Agent 状态转移矩阵与错误热力图 subgraph TransitionMatrix["状态转移矩阵 (Transition Matrix)"] S1["上一成功步骤：生成 SQL 语句"] --> F1{"首个失败步骤"} F1 -->|"热点聚集 (Hotspot)"| E1["失败：SQL 运行时执行错误"] F1 -->|"低频"| E2["失败：参数解析错误"] end E1 ==> R1["精准定位工程投入：无需猜测，直接优化执行引擎"] ``` 当 Agent 发生错误时，能否在产品内自愈决定了这单错误只会损失一个会话，还是会彻底摧毁用户信任并损失一个客户。为了精准定位 Agent 的失败节点，团队构建了**状态转移矩阵（Transition Matrix）**：将每次失败调用链的“最后成功步骤”与“首个失败步骤”绘制成二维矩阵。如果数据显示 Agent 总是能正确生成 SQL 语句，但在执行阶段频繁报错，团队就能立即抓出这个热点（Hotspot），研发力量无需盲目猜测，即可直接投入到具体环节攻坚。 --- ### [12:25](https://www.youtube.com/watch?v=KRFsHDplaEE&t=745s) 第三层：成果质量（Outcome Quality）与四指标精简看板成果质量直接衡量业务结果，这在生成式与 Agent AI 之间是通用的，且必然存在滞后效应。Yash 分享了 Prime Video 一年前研发的“虚拟产品植入”（Virtual Product Placement, VPP）功能——利用 AI 在视频流中无缝植入品牌海报或商品元素。在上线第二周，离线模型评估表现完美，植入视觉极为自然，审核人员全票通过；但真正的成果质量度量，是随后数周和数季度内广告主是否真正为该技术买单并持续复购。针对北极星指标与业务仪表盘的设计，Yash 总结了四大原则： 1. **专注单一核心 KPI**：切忌在留存率（Retention）与商业变现（ROI）之间频繁切换摇摆，必须锁定产品当前最核心优化的单一业务目标。 2. **为每个优化指标搭配反向约束指标（Counter-metric）**：防止团队为了刷高某一局部指标而“玩弄系统”（Game the system），破坏全局体验。 3. **指标优化者不得掌控指标定义权**：例如负责算法优化的团队绝不能自行定义其考核指标。 4. **4 项指标构筑的“让所有人都不爽”但最有效的大盘**：产品经理通常想要 1 个全局数字图省事，算法科学家往往列出 40 多个繁杂参数；而最佳实践是将指标精简为 4 个——涵盖离线产出评估、3 项在线交互实时护栏（P90 达成轮次、介入率、恢复率）以及 1 个滞后的北极星业务指标。这种结构虽然让 PM 无法偷懒只盯一个数字，也让算法人员无法堆砌指标，却最真实地反映了产品全貌。 --- ### [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) 生产环境交付前的四大核心自问演讲最后，Yash 给所有正在将 AI 产品推向生产环境的团队留下了 4 个必答问题，作为产品发布前的终极自检清单： - **自问 1：合格与接受（Acceptance）的边界到底是什么，由谁来定义与裁决？** 明确验收红线，确保由产品业务方主导评测标准的制定。 - **自问 2：当 AI 出错时，用户能否在产品内部顺利恢复（Recover）？** 检查产品内是否具备足够健壮的人机协同容错机制，避免让单次错误演变成会话崩溃。 - **自问 3：人工介入率的趋势是在走向“信任”还是走向“放弃”？** 动态观察用户介入的心理动机，把控人机协作的信任拐点。 - **自问 4：滞后的业务成果指标落后于质量指标多久？两者的折损代理变量（Proxy）是什么？** 理清离线质量到真实商业变现之间的转化路径与滞后周期。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=KRFsHDplaEE&t=4s) 我们根本不信任你们的 AI — 用户 > [00:04](https://www.youtube.com/watch?v=KRFsHDplaEE&t=4s) 这就像外科医生把‘病人只要能活着走出医院’当成手术成功的唯一指标一样荒谬。 — Yash Chaturvedi > [03:36](https://www.youtube.com/watch?v=KRFsHDplaEE&t=216s) 如果看板上的所有指标都以相同的速度和频率刷新，说明指标体系绝对漏掉了关键层级。 — Yash Chaturvedi > [04:49](https://www.youtube.com/watch?v=KRFsHDplaEE&t=289s) 评测就是新的产品需求文档（Evals are the new PRDs） — Yash Chaturvedi > [04:49](https://www.youtube.com/watch?v=KRFsHDplaEE&t=289s) 如果你不去亲手编写评测集（Evals），你实际上就是把产品的定义权拱手让给了别人。 — Yash Chaturvedi > [09:07](https://www.youtube.com/watch?v=KRFsHDplaEE&t=547s) 每当有产品经理跟我谈达成轮次的‘中位数’（Median）时，我都会忍不住翻白眼。 — Yash Chaturvedi > [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) 合格与接受（Acceptance）的边界到底是什么，由谁来定义与裁决？ — Yash Chaturvedi > [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) 当 AI 出错时，用户能否在产品内部顺利恢复（Recover）？ — Yash Chaturvedi > [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) 人工介入率的趋势是在走向‘信任’还是走向‘放弃’？ — Yash Chaturvedi > [15:59](https://www.youtube.com/watch?v=KRFsHDplaEE&t=959s) 滞后的业务成果指标落后于质量指标多久？两者的折损代理变量（Proxy）是什么？ — Yash Chaturvedi ## Mentioned Yash Chaturvedi, 亚马逊 (Amazon), Prime Video, Ai4, 虚拟产品植入 (Virtual Product Placement / VPP), LLM as a Judge ## Video description Yash Chaturvedi - Head of Product, Amazon When an AI agent completes a multi-step task, how do you actually know it succeeded? As generative and agentic experiences move from demos into production, most teams are still applying metrics built for deterministic software, and those metrics quietly miss what these systems get wrong. I’ll draw on my experience shipping AI products at Amazon to show where those metrics break down. I’ll share a practical framework for defining success across generative outputs and multi-step agent workflows, balancing accuracy, user trust, and business impact. You will leave with a concrete set of questions to run against their own product before it ships. [Source](https://www.youtube.com/watch?v=KRFsHDplaEE)

Source quotes

[0:04] 用户: 我们根本不信任你们的 AI [0:04] Yash Chaturvedi: 这就像外科医生把‘病人只要能活着走出医院’当成手术成功的唯一指标一样荒谬。 [3:36] Yash Chaturvedi: 如果看板上的所有指标都以相同的速度和频率刷新，说明指标体系绝对漏掉了关键层级。 [4:49] Yash Chaturvedi: 评测就是新的产品需求文档（Evals are the new PRDs） [4:49] Yash Chaturvedi: 如果你不去亲手编写评测集（Evals），你实际上就是把产品的定义权拱手让给了别人。 [9:07] Yash Chaturvedi: 每当有产品经理跟我谈达成轮次的‘中位数’（Median）时，我都会忍不住翻白眼。 [15:59] Yash Chaturvedi: 合格与接受（Acceptance）的边界到底是什么，由谁来定义与裁决？ [15:59] Yash Chaturvedi: 当 AI 出错时，用户能否在产品内部顺利恢复（Recover）？ [15:59] Yash Chaturvedi: 人工介入率的趋势是在走向‘信任’还是走向‘放弃’？ [15:59] Yash Chaturvedi: 滞后的业务成果指标落后于质量指标多久？两者的折损代理变量（Proxy）是什么？

Source quotes

[0:04] Yash Chaturvedi: we don't trust your AI. [2:10] Yash Chaturvedi: There is always a right answer [2:10] Yash Chaturvedi: Completion equals success [2:10] Yash Chaturvedi: Engagement means value [2:10] Yash Chaturvedi: the patient is able to walk out of the hospital. [4:49] Yash Chaturvedi: evals are the new product requirement documents (PRDs). [9:07] Yash Chaturvedi: turns to success [15:59] Yash Chaturvedi: What does acceptance look like, and who is deciding those criteria? [15:59] Yash Chaturvedi: When the agent fails, can the customer recover in-product? [15:59] Yash Chaturvedi: Is the intervention rate trending toward trust or toward abandonment? [15:59] Yash Chaturvedi: Which downstream business outcome lags your quality metrics, and by how much?

Original transcript

[0:04] As I was putting together uh the success matrix for this session, I was like in the [0:09] day of AI that we live in, how can I make sure people get the insights without querying cloud, [0:14] open AI or reading a podcast. So I'll do my best uh to kind of gauge in from my experiences and [0:22] give you a short brief uh framework on how you can evaluate your generative and [0:28] agentic AI experiences. Um I'll start with the situation about a month ago um my team shipped [0:35] an AI agent. Um it did what it was supposed to do. Um the dashboard said that hey 94% of [0:43] uh the task was completed. Um we read the customer anecdote as we always do. Um the customer hated [0:50] it. Um they were like hey we don't trust your AI. Um so um the matrix said success but the customer [0:58] said no we hate it. Um that's kind of the pitch of this uh presentation. Um just as an intro my name [1:06] is Yashadurvedi. Uh I lead product management for generative and agentic AI at prime video. [1:12] Um I've shipped many air products uh across Prime Video, retail, uh marketing, automation [1:19] and advertising. Um I have nine or 10 uh depending upon the day approved patents on generative AI. So [1:29] I can kind of give you a sense of authority that uh I know what I'm talking about. [1:33] Um the lessons have been that the matrix that were built for deterministic uh softwares or AI uh fail [1:43] on generative and agentic experiences and product in this presentation I'll give you the framework [1:49] uh we use instead um there are three layers in matrix uh and kind of the mental model that we [1:56] utilize um it covers both generative and agentic experiences um and it um kind of The presentation [2:03] ends with uh four questions that I want you to ask your teams and yourself when deploying the AI [2:10] uh products into production. Um so there are three u inheritance uh assumptions that fail. The first [2:19] one is there's always a right answer. So in the deterministic environment that we used to live in, [2:24] there used to be one way of doing things. Um there was always a right answer but as we all know [2:31] generative uh systems return one sample out of an entire distribution. Some outputs are acceptable [2:39] most are not. So accuracy is the wrong question to ask. Second one completion equals success. [2:47] An agent can order an item in 14 confusing turns with the user. The task is completed. [2:55] the customer ended up uh ordering the product but the customer was frustrated and suffered uh [3:01] through those 14 prompts and reprompts. The third one is engagement means value. Um five reprompts [3:09] looks like engagement but it is the fiction uh to getting the work done for the customer. So [3:17] gating an um grading an agent on task completion alone is like how you go into an insurgery and the [3:24] surgeon decides okay the the success criteria is that the patient is able to walk out of the [3:28] hospital. Is it the truly a success criteria that we want to kind of measure the surgeon on? Not [3:36] really. Um so here is the framework. Um there are three layers. Uh output quality. What is the pro [3:46] uh how is how good is the produce output? Interaction quality. What did it cost the customer [3:53] to get this done? Third uh outcome quality. Did the business results happen? Uh the output quality [4:02] uh is measured offline with every release uh of your product, every release of your models. [4:09] Interaction is measured live with every session, with every customer engagement. outcome is lagged. [4:15] It e it is either your business matrix that you're trying to drive. So it can be a weekly KPI that [4:19] you're tracking or a quarterly KPI that is driving certain business value. Here is a quick litmus [4:26] test for your own dashboard. If every matrix on the update uh is updating at the same speed, [4:32] you're missing one uh one layer. Uh so kind of re recind of go to this framework. Same framework is [4:40] utilized across generative and agentic AI. Um only your targets uh change per layers. [4:49] Um so the layer one uh quality output um so as I mentioned drop exact match accuracy. This is not [4:59] a good measure for generative and agentic experiences. Measure acceptance rate the [5:06] percentage of output that clears a predefined bar. Generative score the artifact as to how good the [5:13] artifact is based on your experts in the room or like your end users for agentic experiences score [5:21] the final artifacts and the part it took to get there who is setting the bar uh I hope the product [5:29] uh managers in your organizations are um nowadays uh there's in the industry there's a notion of uh [5:37] how eval are the new product requirement documents ments. Now, uh, evaluating is a core PM skill. Uh, [5:46] that wasn't initially. I want to even go further which I tell my product managers on the team is if [5:54] you don't write the evals, you're hand handing your product definition to whomever it is, [5:59] whomever is setting up those uh product guardrails uh and evals. Recently one of the products that I [6:05] was working on um one of our internal users tried to define it and I had to kind of get intervened [6:11] with with the team to kind of stop them. Um so how do we do this? The generative marketing we we [6:19] are generating marketing uh trailers and artwork at scale. Good image uh as to hey look at your [6:27] previous experiences what a good image looks like is no longer a spec. It's a spectrum of like hey [6:32] it's an acceptable out output. So my product team is writing a binary rubric whether it is [6:41] brand safe, it is compliant, it's an acceptable uh uh publishable uh outcome without any edit [6:49] or without any edits. Um we are utilizing HITL human in the loop with an expert. Um we are kind [6:57] of building up on a very good golden data set. uh and then we are looking at pass and fail rate for [7:04] each of them. The result is there's an acceptance rate that becomes a launch gate. So based on this [7:10] golden set that we are creating with the human in the loop we're defining what this acceptance rate [7:16] would look like at the time of launch and then whatever the failure modes are wherever your [7:22] models are failing where wherever the outcome is failing that becomes our evaluation criterias for [7:27] the uh proceeding launches. Um there are a lot of PM artic uh newsletters these days that talk about [7:37] uh error analysis and how to analyze these errors uh for generative AI. Um one thing that they miss [7:43] is like how do we scale these um it can be done at a very kind of one or two sample sets. Uh but at [7:50] at a prime uh prime video level we we kind of deal in 200 territories. We have multiple artifact that [7:56] we're dealing with. So how do we do this at scale? Um and it's not about how do we evaluate but it is [8:04] also about who owns these. Um legal team wants to own setting up these criterias because of the [8:10] compliance rules. Um the science team wants to own is because it's how they are defining how do how [8:16] do the models and the output scale. Um so it's all about who takes the pen and we have kind of this [8:22] push that the product managers are the ones who are kind of pushing this pen to define what these [8:26] uh evaluation criteras are. Um also scoring them at volume. Uh we are using LLM as a judge. Um [8:34] this can't be scaled through and human. So of course like we are kind of using LLM as [8:39] a judge. Um and what we have noticed uh that once we set a guardrail um the humans acceptance kind [8:46] of deteriorates over time. So we at a cadence are kind of going back and refining what these [8:52] guardrails are what is acceptable to the users and kind of feed that back back into our LLMs. [8:58] Um so it's a kind of a good feedback loop of like hey once it degrades you kind of go back and [9:02] refine what these guardrails needs to be feed that into your LLMs and then kind of go through this [9:07] loop. Um the second layer is interaction quality. Um for generative uh we measure the work between [9:18] the models that it was drafted and what the uh customer ended up keeping or kind of exporting [9:23] from our tools. um what was the edit distance uh from the initial model output to what the edits [9:31] with all the hits that they kind of exported it out of the tool. um so in this step human in the [9:37] loop is very critical. Um then turns to success as to when we're looking at agentic AI experiences, [9:46] how many turns does it take for us to get to success? Um and it's a distribution. It's not [9:51] binary. It's like hey two steps, three step, but you can it's a it's a distribution. Um in one of [9:57] the examples that we have seen recently, this is what the curve looks like on the screen. Um so [10:02] the medium was three steps. Um but P 0 uh P90 was 11. Um and the tail end is where the customers are [10:11] kind of getting frustrated and are quitting. Um so we don't usually track the median. We always track [10:18] for either P90 or P95 matrix uh when it comes to interaction uh and turn to success. Um I roll [10:26] my eyes every time a product manager talks about medium when it comes to uh turn to success because [10:33] that's not the right matrix that we're looking at. Um also there is a intervention rate. Um how often [10:40] do humans take over? So how often there is a need for hitl in the loop. Um how many times they're [10:46] making edits, overriding, abandoning uh the action. Um and that kind of directly ties to how [10:54] the humans are trusting the system that you you're building. So the trust factor is essential as we [11:00] driving adoption uh for AI. Um and the target is not always zero. you don't want the uh uh the in [11:08] the the intervention rate to be zero. Uh you're optimizing to getting to near zero, but it if it [11:13] gets to zero, that means your uh AI is either not serving the purpose that the human needs [11:19] um or it's just not working uh that that they want it to kind of work. Then is the recovery rate [11:26] um after the agent errors um what percentage of sessions still succeed? Um because a lot of times [11:33] agents will uh fall with confident uh and they will give a full sentence of like hey why do they [11:41] think they're very confident in this output. Uh so recovery determines whether an error costs you [11:46] a session uh whether they abandoned the session or they kind of just totally abandoned uh uh the [11:52] trust in AI. So it might cost you a customer. Um and to find uh uh where the agents are failing, [12:01] build this uh transition um matrix um map each failure trace last success step by first failed [12:10] step um and then count it. The hotspot tells you uh where to spend engineering time. Um in this [12:18] example, the agent writes the cor current SQL um but fails as execution. So there's no guessing on [12:25] where the engineering time needs to be spent. Now the third layer is uh outcome quality. U this is [12:34] identical for both uh agentic and generative AI. Um this is your business result. What what [12:40] I what is this trying to drive for the business? Um and this is measured downstream. Uh and there's [12:47] a lag. You can't kind of expect an answer right away. Um, one year ago, um, we were working on [12:55] virtual product placement for Prime Video where we're trying to insert a brand or a poster or, [13:00] um, some kind of branding element into the stream. Um, and in week two, all our models outcome matrix [13:07] were very strong. Uh, placements looked natural. Um, the reviewers were approving it. Um and then [13:14] we were tracking as to how this product uh was kind of getting funded and what the uh uh business [13:21] outcomes were. Um and then were the advertisers coming back and buying this product again and [13:28] again. So that was kind of the business outcome or the uh outcome matrix that we were u measuring. So [13:37] there are kind of uh four numbers uh when it when it comes to kind of like looking at [13:42] uh the northstar which is your business outcome what business outcome you're trying to drive [13:49] um the cleanest part to the number uh is to stop offering hands off between like hey I need to [13:56] optimize either for retention and then eventually I'll move to getting more ROI from the users. So [14:02] stick to one KPI uh that this model is trying to optimize or your product is trying to optimize [14:07] for. The second rule is pair every optimization matrix with a c with a counter matrix. Um that [14:14] makes it very difficult for uh us to game the system or game the experience. Um and never let [14:21] the team optimize a matrix own its own definition. So a lot of times we'll see science matrix own [14:28] certain metrics um and they are the ones kind of optimizing those matrix. So you you never let the [14:34] uh person who's optimizing a matrix own its own definition. Um now to the dashboard every [14:41] product manager wants one number uh so that it's like their life is easier. They're kind [14:46] of optimizing that particular matrix. I've seen a lot of instances where the science team is looking [14:51] for 40 different numbers input output matrix all across the board. Um and the correct answer in my [14:59] uh worldview is these four metrics that I highlighted. Um and then all both sides are [15:05] unhappy because product uh folks are not able to optimize one KPI and the scientists are not able [15:10] to get whatever they're looking for. Um but that's how you know it's the right matrix to track. [15:18] um one notch outcome matrix is um is lagged to be honest. So it's not like something you [15:24] launch today you'll get what the customer retention or the ROI from the users are. [15:30] um the two or the three uh interaction guardrails, intervention rate, recovery rate, [15:37] uh P90 turn to uh success. All of these are kind of real-time matrix that we engage. Um and then [15:46] output evaluation stays offline in every release. So like you're kind of out like all your output [15:51] matrix you're kind of doing it before you're launching the uh the models of the product online. [15:59] I'll end with like the four takes takeaway questions that you should ask while building the [16:05] quest uh while while building your AI products. Uh one, what does acceptance look like and who's [16:12] defining and uh deciding those? Second is when it fails, can the customer recover in product? So, [16:20] is there a good enough hit model within your product? Third one is the uh the intervention [16:26] rate trending is it trending towards trust or is it tending towards abandon. So like where [16:31] I spoke about not kind of optimizing it for to be zero all the time. So there should be [16:36] a high level of trust when people are dealing with AI systems and AI products rather than [16:41] kind of optimizing for no human in the loop and no intervention rate. The fourth and the final one is [16:48] uh which outcome lacks your quality matrix and by how much? So what is your proxy that like you can [16:54] look at when you're optimizing for certain out uh output matrix and outcome matrix? Um please [17:02] feel free to reach out to me if you have any questions. Uh thank you so much for your time.