Meta Product Lead Overhauls AI Design as User Demands Shift

应对用户需求转变，Meta产品负责人重构AI设计 生成式AI团队必须手动标注早期输出并制定严格的质量标准，以打造能够端到端解决复杂用户问题的产品。 Radhika Bhargava 阐释了生成式 AI 如何推动用户期望从多步骤构建查询转变为端到端解决问题。她重点指出了“你为什么就不能……”现象，即用户旅程已变为动态的对话路径，而非可预测的固定工作流。为应对非二元的成功判定标准，她介绍了一个通过明确目标、可衡量维度、分级量表、压力测试和常态化评估来定义 AI 质量的四步框架。最后，她建议产品负责人全面负责整个问题、确立明确的质量护栏，并亲自参与早期的手动标注。 Meta 超级智能团队产品负责人 Radhika Bhargava 指出，用户习惯已发生根本转变，不再愿意手动拆解任务，而是期望 AI 产品端到端直接解决完整问题。 (0:04) “为什么你不能直接搞定……”现象反映出用户对 AI 自动理解复杂语境与执行步骤的预期，促使交互流程从传统的静态表单演进为动态自适应对话。 (1:50) 由于生成式 AI 的表现存在多元光谱而非单一对错，产品团队需通过明确目标、拆解质量支柱、设定清晰量表及压力测试，构建结构化的质量评估标准。 (5:52) 针对评估体系的落地，产品经理在初期应亲自完成首轮人工标注打分，切忌过早外包或自动化，以深入捕捉细微边缘案例并校准用户体验。 (11:51)

Radhika Bhargava, Product Lead on Meta's Superintelligence Team, argues that generative AI has shifted user expectations from manual task execution to complete, end-to-end problem resolution. Consequently, product leaders must move beyond binary pass/fail metrics to establish multi-dimensional quality rubrics and hands-on evaluation workflows. From Manual Querying to Dynamic Problem-Solving (1:50) Bhargava outlines a transition in software interaction from rigid, step-by-step query building to open-ended conversational goals. In traditional software, such as legacy search filters on platforms like LinkedIn, users manually configured discrete parameters like title, company, and location. Generative AI alters this dynamic by allowing users to submit natural language prompts with high-level context, expecting the system to infer intent and complete the underlying tasks. This shift creates the "why can't you just..." phenomenon, where users expect models to automatically grasp nuance and navigate execution steps. As a result, product workflows change from predictable sequences into dynamic conversations that can branch unexpectedly. Because outcomes are no longer binary, identifying product failures requires diagnosing a wider range of issues, such as intent misinterpretation, reasoning gaps, technical limits, or tone mismatches. Structuring Non-Binary Quality Definitions (5:52) Because generative AI outputs exist on a spectrum, Bhargava advises product managers to construct formal quality rubrics to serve as team-wide baselines. Using a hypothetical AI language learning app, she illustrates that quality design begins with a single testable hypothesis—such as learners gaining native speaking confidence within 30 days—which is then decomposed into measurable dimensions like grammatical accuracy, conversational fluency, and cultural engagement. Teams must map these pillars to practical scoring scales, such as a 1-to-5 range, where numerical increments correspond to noticeable differences in user experience. Bhargava warns against overly fine-grained scoring systems, noting that evaluators cannot reliably distinguish adjacent scores if the criteria lack discernible boundaries. Stress-Testing and Operationalizing Evaluations (5:52) Before applying rubrics to model outputs, teams must stress-test them through alignment and completeness checks. Alignment checks require multiple team members to independently score sample conversations across easy, medium, and complex tiers to ensure minimal delta between evaluators, while completeness checks verify that no essential dimension of user experience has been omitted. Once calibrated, rubrics are deployed across offline evaluations on historic or synthetic data and online evaluations on live traffic. Bhargava emphasizes that product leaders should perform the initial annotation passes themselves rather than immediately outsourcing or automating the process, arguing that hands-on evaluation is necessary to uncover subtle edge cases and ambiguities in the evaluation criteria. Bhargava emphasizes that product leaders must resist the urge to automate evaluation prematurely, asserting that deeply engaging with manual rubric calibration is necessary to ensure products satisfy rising user expectations for end-to-end execution.

Meta 超级智能团队（Superintelligence Team）产品负责人 Radhika Bhargava 指出，生成式人工智能已使产品开发逻辑发生根本性转变，用户不再愿意自行拆解任务，而是期待系统端到端地解决完整问题。面对结果缺乏单一正确答案的开放式交互，产品经理必须将核心工作聚焦于构建多维度的质量防护栏与严谨的人工评估体系。 从分步查询转向端到端动态交互 (1:50) Bhargava 指出，在传统软件架构下，用户本质上充当着查询构建者的角色。例如在领英等招聘场景中，用户必须手动拆解需求，逐一在职位、公司和地域等既定字段中输入筛选条件。然而在生成式 AI 时代，用户可以直接输入具有丰富上下文的高阶自然语言意图，例如寻找在早期初创公司主导过 AI 产品并希望加入 A 轮公司的产品经理。这种演进促成了“为什么你不能直接搞定”的心理预期，用户期望系统能自动理解微妙语境并补全执行步骤。 这种预期的转变彻底重构了用户旅程，原本可预测的线性流程被能够实时分支的自适应对话所取代。生成式系统会根据情况反问澄清、推荐替代视角或呈现意料之外的候选结果。这导致产品的成功标准不再是非黑即白的二元判断，而诊断系统故障也变得愈发复杂，产品经理需要逐一排查意图理解错误、技术底层限制、表达语气不当以及逻辑推理漏洞等多种潜在成因。 质量量化框架与多维度打分标尺 (5:52) 由于生成式 AI 的表现处于连续的光谱区间而非简单的通过或失败，Bhargava 提出了用于指导工程与产品团队的四步质量定义框架，并以语言学习应用为例进行说明。产品首先必须明确目标（Step 0），即确立一条清晰且可检验的产品假设，如“用户使用该应用 30 天后能自信地使用母语交流”。随后，团队需将主观体验拆解为客观的质量支柱（Step 1），立足真实用户痛点量化为语法词汇的准确性、对话衔接的流畅度以及维持学习动力的文化语境与互动性。 在确立评估维度后，团队需建立直观映射用户感受的评分标尺（Step 2）。Bhargava 建议采用 1 至 5 分的区间体系，并警示团队切勿设计颗粒度过细的量表。如果评估人员无法在实际操作中清晰辨别相邻分值（例如 4 分与 5 分）的本质差异，过于精密的打分细则将失去一致性与实用指导价值。 准则压力测试与亲自人工标注实践 (5:52) 在正式将评估准则投入实际研发前，团队必须对其进行严格的压力测试（Step 3）。压力测试包含两项核心检验：一致性检查要求多名团队成员针对简单、中等和复杂三种对话场景独立打分，验证评分标准是否存在较大分歧；完整性检查则确保所有关键的用户体验维度均未遗漏。Bhargava 强调，未经校准的准则往往会导致纸面看似完备的产品在实际用户体验中出现严重落差。 在评估落地执行环节（Step 4），团队应结合针对历史或合成数据的离线评估，以及针对线上实时流量的在线评估。Bhargava 明确要求产品管理者必须亲手完成第一轮人工标注，而不能立即将该流程外包或全盘自动化。只有亲自深入分析真实边缘样本，产品负责人才能及时捕捉评估准则中的含糊地带，理解用户体验的深层细节并有效应对持续抬高的用户预期。 Bhargava 的核心主张在于产品经理必须转变角色，从线性功能的搭建者转变为端到端系统质量守门人。不过该框架高度依赖前期团队内部的主观假设与充分的人工校准；在产品规模迅速放大后，如何在兼顾深度人工评估与自动化评测效率之间取得长期平衡，依然是留给产品开发团队的实践课题。

## Background 本视频由 Ai4 于 2025 年 8 月 27 日发布在 YouTube 平台，主讲人为 Meta 超级智能团队（Superintelligence Team）的产品负责人 Radhika Bhargava。视频简介指出，本次分享聚焦于生成式 AI 与大语言模型如何重塑产品开发范式，旨在为产品团队提供应对全新交互形态、定义产品质量标准及避开常见落地陷阱的实战指南。 ## TL;DR 演讲深入剖析了 AI 原生时代用户心智的根本性转变：用户不再愿意充当机械拆解步骤的“查询构建者”，而是期望系统能够自主理解意图并端到端交付结果。针对开放式生成结果具有非二元化与连续光谱的特点，Radhika 提出了涵盖“明确假设、维度拆解、量表设计、压力测试与工程化评测”的四步质量评估框架。她强调“定义质量”已成为 AI 产品经理的首要职责，并警示团队切忌过早将标注工作外包或自动化，PM 必须亲自下场标注以洞察真实边界情况。 ## Atmosphere & style 整个分享节奏明快、条理极其严密，洋溢着一线资深 AI 产品负责人务实而深刻的工程化思维。Radhika 的表达冷静透彻且直击痛点，借助生动的对比案例将复杂的评估难题拆解为清晰的战术方法论。聆听过程既能感受到用户交互范式颠覆带来的认知冲击，又能体会到在充满不确定性的生成式技术中构建严谨评价体系的踏实与厚重。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=4s) 用户心智发生根本转变，不再满足于工具仅辅助单个环节，而是期望 AI 系统能够端到端解决整套复杂问题。 - [01:53](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=113s) 用户角色从手动的“查询构建器”转变为“目标对话者”，催生了“为什么你就不能直接帮我搞定”的全新预期。 - [03:35](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=215s) GenAI 打破了确定且线性的用户旅程，交互变成了实时演化、根据语境高度自适应的动态对话系统。 - [03:35](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=215s) AI 产品的成败不再是非黑即白的二元状态，而是一个连续光谱，使得故障排查与归因难度大幅增加。 - [05:52](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=352s) 在缺乏单一标准答案的开放生成场景中，产品经理的首要职责从流程设计转变为“定义质量”。 - [05:52](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=352s) 构建质量框架需从明确单一可测试的产品假设出发，并将主观感受拆解为准确性、流畅度、文化语境等多维指标。 - [05:52](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=352s) 评分量表必须贴合用户能够真实感知的体验梯次，切忌制定评审人员自身都难以区分的过细颗粒度。 - [09:08](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=548s) 评估标准必须经过一致性与完备性的压力测试，跳过团队校准是打造出“纸面完美但实际翻车”产品的最快途径。 - [09:08](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=548s) 在将评估工程化之前，PM 必须亲自完成初轮数据标注，以捕捉边界情况并厘清灰色地带，切忌过早外包或全自动化。 - [11:51](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=711s) 打造真正跨越用户高预期门槛的 AI 产品，核心在于接管端到端任务、树立清晰质量护栏并扎实打磨评测闭环。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=4s) AI 时代产品经理角色的演变与新范式 在机器学习与人工智能领域深耕十年的 Meta 超级智能团队（Superintelligence Team）产品负责人 Radhika Bhargava 开启了本次分享。她回顾了过去产品经理（PM）职责的变迁，强调当前最重要的并非仅仅关注工作战术上的调整，而是要深刻理解用户行为发生的根本性转变。现如今，用户不再满足于产品只解决问题的一个环节，而是期望产品能够直接端到端解决他们的整套问题。这种心智模型的颠覆，彻底重塑了产品负责人在 AI 时代规划功能和构建产品的方式。 本次演讲聚焦于三大核心模块：首先剖析生成式 AI（GenAI）产品的全新范式以及整个行业如何围绕“AI 原生（AI-first）”发生演进；其次探讨随着 AI 技术的深度渗透，用户预期与工作流发生了怎样的剧变；最后深入探讨产品负责人具体的战术落地路径——当系统不再存在单一标准答案时，团队该如何定义成功，以及为什么质量评估体系（Quality Frameworks & Evaluations）会成为 PM 最关键的武器。 - Radhika Bhargava 在 Meta 长期负责将前沿研究成果转化为实际产品落地（Research to Product）。 - 现代产品已从简单的问答、摘要、推荐助手，深度嵌入到代码生成、内容编辑和研究辅助等日常核心工作流中。 --- ### [01:53](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=113s) “为什么你就不能直接帮我搞定？”：从查询构建者到目标对话 AI 原生产品与传统软件最本质的差异在于用户预期的范式转移。在传统时代，用户习惯于将自己的复杂诉求拆解为一个个具体的步骤和过滤条件，本质上是在替系统分担“问题拆解”的工作。以往用户的心态是“我需要先做 A，再做 B，然后做 C”；而如今用户的心理转变为：“这是我的最终目标，你自己去想办法帮我达成”。Radhika 将这种心态称为“为什么你就不能直接帮我搞定”（"Why can't you just..."）现象——用户要求系统能够直接理解其深层意图并端到端解决整个问题。 ```mermaid flowchart TD accTitle: 传统搜索与生成式 AI 搜索工作流对比 subgraph Traditional["传统模式（用户充当查询构建器）"] direction TB A1["明确职位: Product Manager"] --> A2["筛选公司: Early Stage"] A2 --> A3["限定地点: 某城市"] A3 --> A4["机械执行多重条件过滤"] end subgraph GenAI["GenAI 模式（自然语言目标对话）"] direction TB B1["自然语言输入完整诉求"] --> B2["系统理解上下文与潜在动机"] B2 --> B3["自主推断并交付端到端匹配结果"] end ``` 以 LinkedIn 的人脉搜索为例：在 GenAI 出现之前，用户如同“人类查询构建器（Human Query Builders）”，必须在高级搜索中逐一手动填写职位（如 Product Manager）、挑选目标公司、限定地理位置等参数。而在当前环境下，用户的诉求直接变成了自然语言表达：“帮我找一位在早期初创公司主导过 AI 产品上线、并且有兴趣加入 A 轮初创团队的产品经理”。这种转变意义深远——用户不再逐项定义机械细节，而是通过自然对话表达真实目标，系统则必须自主理解其中的细微差别、复杂语境与核心动机。 --- ### [03:35](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=215s) 动态交互路径与非二元化的质量困境 这种交互模式的变革打破了传统产品中可预测的线性用户旅程。传统的用户路径是高度确定且顺序固定的：用户进入高级搜索、填入字段、点击搜索、浏览结果、然后再调整过滤条件，每个用户走的几乎都是同一套流程。但在生成式 AI 产品中，用户旅程演变成了一个实时演化、高度动态的对话系统。用户输入一个开放式请求后，AI 可能会追问以澄清需求，可能会主动建议不同的搜索维度，甚至展示出乎意料但极具价值的人才档案，从而彻底改变交互方向。这种动态自适应能力虽是 AI 的核心威力所在，却给产品设计带来了全新挑战——PM 无法再针对单一、可预测的流程做静态设计。 与交互复杂度相伴而来的是故障诊断难度的急剧上升。在传统软件中，产品成败通常是二元的（Binary）：用户指定了某种过滤规则，系统要么返回正确结果，要么报错，故障点一目了然。但在开放式的 GenAI 输出中，“成功”不再是非黑即白的二元状态，而是一个连续的光谱（Spectrum）： - 当生成结果令人失望时，很难立刻判断究竟是 AI 误解了用户意图、底层模型推理能力不足、语气风格不合适，还是技术本身存在物理局限。 - 面对同一个开放式提示词，往往可能存在多种同样合理且有效的回答。 --- ### [05:52](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=352s) 质量评估框架构建：从产品假设到多维指标拆解 由于不再存在单一的标准答案，产品经理的核心精力必须从传统的流程设计转向“定义质量”（Defining Quality）。PM 必须为团队提供清晰的北极星标准，明确区分“差（Bad）”、“良好（Good）”与“卓越（Great）”。Radhika 提出了一个严密的评估体系构建框架，并以构建一款“AI 驱动的语言学习应用”为例进行了系统阐述： ```mermaid flowchart LR accTitle: AI 产品质量框架落地四步法 S0["Step 0: 明确假设<br/>定义单一可测试的成功目标"] --> S1["Step 1: 维度拆解<br/>准确性/流畅度/文化语境/参与度"] S1 --> S2["Step 2: 划定分阶量表<br/>1-5分映射用户真实可感知差异"] S2 --> S3["Step 3: 压力测试与对齐<br/>团队一致性与完备性检验"] S3 --> S4["Step 4: 评估工程化<br/>离线/在线评测与 PM 自主标注"] ``` #### Step 0：明确核心产品假设（Product Hypothesis） 在动工前，必须用一句话提炼出可测试的成功定义。例如：“用户在使用该应用 30 天后，能够自信地用目标母语进行口语对话。”这为后续的所有质量衡量提供了锚点。 #### Step 1：将主观感受拆解为可度量的质量维度（Dimensionalize Quality） 团队需要将抽象的“体验好坏”拆解为具体支柱指标，以便进行客观测量和针对性迭代： - **准确性（Accuracy）**：会话中的语法与词汇是否准确无误； - **流畅度（Fluency）**：上下文对话是否自然连贯、毫无生硬感； - **文化语境（Cultural Context）**：用词是否符合当地真实文化习惯与语境； - **互动参与度（Engagement）**：内容是否能够激发学习者持续对话的动力。 在制定这些维度时，必须深入挖掘真实的用户摩擦点，观察用户在哪些环节卡壳、在何处流失，并将这些实际痛点转化为具体的评分准则。 #### Step 2：定义贴合感知差异的评分量表（Scoring Range） 量表的设计必须反映用户能实际感知到的体验梯次（例如 1 到 5 分）： - 1 分表示每句话都有严重语法错误； - 5 分表示完全无错误且地道自然。 - **避免过细颗粒度**：如果评审者自己都无法清晰区分 4 分与 5 分的区别，就必须将等级合并，严禁制定脱离用户真实感知的过度精细量表。 --- ### [09:08](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=548s) 评估基准的压力测试与工程化落地 框架中最关键且最容易被忽视的环节是 **Step 3：压力测试（Stress-Testing the Rubric）**。在全面应用评分标准之前，必须进行两项核心检查： 1. **一致性检查（Alignment Check）**：让 5 名团队成员针对同一批 Prompt 和输出进行独立打分，评估团队内部评分是否存在巨大方差（Delta）。 2. **完备性检查（Completeness Check）**：审查当前的评分维度是否遗漏了影响体验的关键因素。 具体实操方法是挑选三种不同难度的典型场景——简单用例、中等难度用例以及极高复杂度的极端用例，让团队成员按照评分标准各自打分。针对分歧点展开辩论并迭代标准指南。Radhika 严肃警告：“跳过这一步，是打造出‘在纸面上看起来完美、但在现实中让用户大失所望’的产品最快的方式。” 通过验证后进入 **Step 4：评估工程化（Operationalization）**，将评分标准持续应用于真实产品输出并长期追踪： - **离线评测（Offline Evals）**：针对历史交互日志或产品未上线前的假设数据进行分析； - **在线评测（Online Evals）**：在生产环境中针对真实流量进行实时表现监测。 - **PM 必须亲自完成初轮标注**：Radhika 极力劝诫 PM 切忌过早将标注工作外包或自动化。只有 PM 亲自下场标注第一批数据，才能发现隐藏在真实数据中的边界情况（Edge Cases）和标准中模糊不清的灰色地带。 --- ### [11:51](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=711s) 总结：构建用户真正想用的 AI 产品 Radhika 在分享尾声重申了驱动 AI 产品成功的三大核心支柱： 1. **拥抱端到端诉求**：用户对 AI 的期望已经从“提供建议”升级为“接管并解决整个复杂任务”，产品必须具备交付完整结果的能力。 2. **定义质量是 PM 的首要职责**：面对没有标准答案的开放生成场景，清晰、客观、多维的质量定义是团队唯一的导航护栏。 3. **切勿仓促跳过评估流程**：过早追求自动化或外包标注往往欲速不达。唯有扎扎实实打磨评分准则、完成人工校验、建立严密评测体系的团队，才能打造出真正跨越用户高预期门槛的卓越 AI 产品。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=4s) 用户不再满足于产品只解决问题的一个环节，而是期望产品能够直接端到端解决他们的整套问题。 — Radhika Bhargava > [01:53](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=113s) 我需要先做 A，再做 B，然后做 C — Radhika Bhargava > [01:53](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=113s) 这是我的最终目标，你自己去想办法帮我达成 — Radhika Bhargava > [01:53](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=113s) 为什么你就不能直接帮我搞定？ — Radhika Bhargava > [01:53](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=113s) 帮我找一位在早期初创公司主导过 AI 产品上线、并且有兴趣加入 A 轮初创团队的产品经理 — Radhika Bhargava > [05:52](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=352s) 用户在使用该应用 30 天后，能够自信地用目标母语进行口语对话。 — Radhika Bhargava > [09:08](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=548s) 跳过这一步，是打造出‘在纸面上看起来完美、但在现实中让用户大失所望’的产品最快的方式。 — Radhika Bhargava > [11:51](https://www.youtube.com/watch?v=qc2WKOsxJ94&t=711s) 用户对 AI 的期望已经从‘提供建议’升级为‘接管并解决整个复杂任务’，产品必须具备交付完整结果的能力。 — Radhika Bhargava ## Mentioned Radhika Bhargava, Meta, Meta 超级智能团队 (Superintelligence Team), LinkedIn, Ai4 ## Video description Radhika Bhargava, Product Lead, Superintelligence Team, Meta Abstracts The rules of product development are changing fast — and this session is your playbook for keeping up. We’ll dive into how genAI and LLMs are reshaping what it means to ship delightful, high quality products, from rethinking your spec to building the right taxonomy for your experience. Expect real-world examples, sharp frameworks, and a few surprising pitfalls even the best teams fall into. [Source](https://www.youtube.com/watch?v=qc2WKOsxJ94)

[0:04] Radhika Bhargava: 用户不再满足于产品只解决问题的一个环节，而是期望产品能够直接端到端解决他们的整套问题。 [1:53] Radhika Bhargava: 我需要先做 A，再做 B，然后做 C [1:53] Radhika Bhargava: 这是我的最终目标，你自己去想办法帮我达成 [1:53] Radhika Bhargava: 为什么你就不能直接帮我搞定？ [1:53] Radhika Bhargava: 帮我找一位在早期初创公司主导过 AI 产品上线、并且有兴趣加入 A 轮初创团队的产品经理 [5:52] Radhika Bhargava: 用户在使用该应用 30 天后，能够自信地用目标母语进行口语对话。 [9:08] Radhika Bhargava: 跳过这一步，是打造出‘在纸面上看起来完美、但在现实中让用户大失所望’的产品最快的方式。 [11:51] Radhika Bhargava: 用户对 AI 的期望已经从‘提供建议’升级为‘接管并解决整个复杂任务’，产品必须具备交付完整结果的能力。

[1:50] Radhika Bhargava: why can't you just... [5:52] Radhika Bhargava: After 30 days of using this app, learners feel confident speaking the native language.

[0:04] All right. Hey everyone, I'm Rodica. Uh, [0:06] I've been working in the machine [0:08] learning and AI space for the past [0:09] decade and most recently I've been [0:11] working in the product team um on our [0:13] researcher product efforts at Meta. As I [0:16] reflect on how the role of a product [0:18] manager has evolved over the years, I [0:20] think it's really important for us to [0:22] focus not just on the tactical job [0:24] changes but on how user behavior itself [0:26] has fundamentally shifted. For example, [0:29] users now expect products to solve their [0:31] entire problem and not just a part of [0:33] it. And that changes everything about [0:35] how we build. This has shaped how we as [0:38] product leaders develop features and [0:41] think about building products in the age [0:42] of AI. And that's exactly what I'll [0:44] focus on today. [0:47] First, we'll talk about the new paradigm [0:49] of Genai products, how the industry has [0:51] evolved with these AI first products. [0:54] Then we'll look at how user expectations [0:56] and workflows have changed as products [0:59] leverage AI. And then finally, we'll [1:02] dive into what this means tactically for [1:04] product leaders, how to define success [1:06] when there's no single right answer, and [1:08] why quality frameworks and evaluations [1:11] become your most important tool. [1:15] So there are a lot of uh popular AI [1:17] first products that have taken off. How [1:20] many of you have tried one of these [1:22] products? [1:23] I don't see a lot of raise of hands, but [1:25] I'm going to assume the venue. [1:27] Everyone's tried at least one of these [1:28] products for asking questions about just [1:31] Q&A, summarizing content, coming up with [1:35] recommendations. AI is becoming part of [1:38] our everyday tasks. It's embedded in a [1:41] lot of apps you use now. Whether you're [1:43] using code generation tools, content [1:45] editing assistants, or research helpers. [1:49] So, what's different about these AI [1:51] first products? [1:53] What really sets these products apart is [1:55] this fundamental shift of user [1:57] expectations. [1:58] Before users broke down their requests [2:01] into detailed steps and filters, they [2:03] were essentially doing a lot of the [2:05] problem breakdown themselves. Instead of [2:07] thinking, I need to do step A then B and [2:10] C, they're now thinking, here's my end [2:13] goal. You figure out how to get me [2:15] there. This has created what I call the [2:19] why can't you just phenomenon. Users are [2:22] now approaching products saying why [2:24] can't you just understand what I'm [2:25] trying to achieve and help me get there. [2:27] They expect products to solve their [2:29] entire problem end to end and not just a [2:31] part of it. [2:34] So here's what the shift look like looks [2:36] like in practice. I'm using this example [2:38] of LinkedIn people search in on the left [2:42] in the pregenai world. Users filled out [2:44] specific fields. They'll enter the [2:46] title. They're looking for a product [2:48] manager. They'll pick the company. [2:50] They'll select the location. They were [2:52] essentially acting like the human query [2:54] builders, breaking down their need into [2:57] exact parameters the system needed to [2:59] understand. [3:00] But now users have a different [3:02] expectation. They want to say something [3:05] like, "Find me a PM who's launched AI [3:07] products at early stage startups and [3:09] would be interested in joining a series [3:12] A startup." The difference here is [3:14] pretty profound. [3:16] Users went from human query builders to [3:19] having natural conversations about their [3:21] actual goals. They're no longer [3:24] specifying every detail themselves. They [3:27] expect the system to figure it out and [3:29] understand the nuance and context behind [3:31] what they're really looking for. [3:35] This shift has fundamentally changed how [3:37] users move through the products as well. [3:39] So, pregenai, user journeys were pretty [3:42] sequential. Users followed a predictable [3:45] step-by-step path. However, if you look [3:48] to the right, user journeys are pretty [3:51] dynamic. They emerge, they adapt based [3:53] on the conversation. [3:55] If I continue down the LinkedIn example, [3:58] traditional approach would require a [4:00] user go into advanced search. They might [4:02] fill in some fields. They'll hit search [4:04] and review some of the results and then [4:06] they'll refine their search. But in this [4:09] path, every user is basically following [4:11] the exact same sequence. [4:13] Now with Gen AI however people just type [4:17] find me a PM who's launched AI products [4:19] in an early stage company and that [4:22] search can lead it anywhere. The AI [4:24] might ask clarifying questions. They [4:26] might suggest different angles of search [4:28] um or surface unexpected profiles that [4:31] change the whole direction. This dynamic [4:33] nature of what makes AI products [4:35] powerful. They can start in real they [4:38] can adapt in real time to what users [4:40] actually need. But it creates this [4:42] fundamental challenge for us as product [4:45] leaders. We can no longer just design [4:47] for a single predictable user journey [4:50] because every interaction can unfold [4:52] completely differently. [4:54] And as a result of this complexity, it's [4:57] even harder to diagnose failures. [5:00] Pregenai, you sort of knew where went [5:02] where things went wrong because success [5:04] was relatively binary. The user is [5:06] telling you exactly what filter they [5:08] want. They want to apply it so you can [5:10] check if it worked or not. However, [5:13] postgenai on the right side, failures [5:16] aren't that easy to detect. There's no [5:18] real right answer. With an open-ended [5:20] request, it's not always clear whether a [5:23] result is disappointing because AI [5:26] misunderstood the intent or there was [5:28] some limitation in the underlying [5:30] technology or is it something else [5:32] entirely? Was it the tone the AI used or [5:36] is it a reasoning problem? And even if [5:39] the AI gives a response, there might be [5:41] multiple valid answers to the exact same [5:43] query. So in this new world, what's [5:46] considered successful is on a spectrum. [5:48] It's no longer binary. [5:52] So what does this mean for you as [5:54] product leaders? It means a majority of [5:56] your time should be spent on defining [5:58] quality. In the traditional world, when [6:00] success was binary, things either worked [6:03] or they didn't. But now there's no [6:05] single right answer. Teams need clear [6:08] guidance on what separates good versus [6:11] bad and good versus great. Your quality [6:14] definition becomes a northstar that [6:16] guides every decision your team is [6:18] making. So if defining quality is the [6:21] most critical job for you as a product [6:24] leader, how do you actually do it? Let's [6:27] walk through a framework and I'll use an [6:28] example along the way to ground us. So [6:31] let's imagine we're building an AI [6:33] powered language learning app. [6:35] Step zero here is making sure you have [6:38] clarity on your goal. You need a very [6:41] strong product hypothesis. You should [6:43] define what success means for your AI [6:45] experience in one clear testable [6:47] statement [6:49] for your our learning language uh [6:51] application. That might mean after 30 [6:54] days of using this app, learners feel [6:56] confident speaking the native language. [6:59] This part doesn't really change much [7:02] from pre or post AI, but again, it's [7:05] really important to make sure what goal [7:06] you're grounding in. [7:10] This next step is where you start to [7:12] dimensionalize quality into key pillars. [7:15] So, you have your goal defined, but [7:17] let's break it down from subjective [7:20] quality dimensions into measurable [7:22] components that are relevant on your [7:24] specific use case. [7:26] So in our example that we were using, [7:29] your pillars might be accuracy. Does it [7:32] actually use correct grammar and [7:34] vocabulary in these conversations? [7:36] Fluency. Does the conversation flow [7:39] naturally? Cultural context and [7:41] engagement. [7:43] Does it keep users motivated to [7:45] continue? When you're what you're really [7:47] doing here is taking something really [7:49] subjective and then breaking it down [7:51] into more objective measurable [7:53] dimensions that your team can actually [7:54] measure and improve upon. Instead of [7:57] debating whether AI is good, you can [8:00] actually have a specific conversation [8:02] about whether it's accurate, it's [8:05] fluent, culturally appropriate, or [8:07] engaging. [8:09] What's really important here though is [8:11] you want to review actual user data when [8:13] you're defining these dimensions. What [8:16] do users get stuck on? Where do they [8:18] disengage? You want to find these common [8:20] friction points and then figure out how [8:22] to translate them into rubric [8:24] dimensions. [8:25] So once you have this initial set of [8:28] quality dimensions, you want to start [8:30] defining a scoring range, you need to [8:33] choose a scale that reflects noticeable [8:35] differences in your experience. So for [8:38] example, you can use a scale from 1 to 5 [8:40] for accuracy where one means major [8:44] grammatical errors in every single [8:45] sentence and then five means error-free [8:48] natural sounding. What's really [8:50] important here is you want to avoid [8:52] scales that are way too fine grained. As [8:55] an example, if you yourself can't really [8:57] tell the difference between a four or a [8:59] five and it feels relatively identical, [9:01] you should collapse them. Your scale [9:03] should map to actual differences your [9:05] users can feel. All right. So, you have [9:08] your quality dimensions, your rubric, [9:11] and you have some scoring attached to [9:12] it. Step three here is, in my mind, the [9:16] most important step. You should debate, [9:18] iterate, and repeat. You need to stress [9:20] test your rubric before you commit to [9:22] it. There's two main checks that I [9:25] typically run. The first one is the [9:27] alignment check. If you ask five team [9:30] members and you say, "How would you [9:32] score this answer based on the prompt [9:34] the user provided?" Is there agreement [9:37] or do people have a high delta in their [9:39] scores? [9:41] The second check is the completeness [9:42] check. Are there critical dimensions [9:45] you're missing in terms of defining [9:47] quality and your rubric? [9:50] To do these two checks practically, you [9:52] can start by giving your rubric and to [9:54] your team and have them score three [9:56] conversations. You can pick an easy [9:58] scenario, a medium difficult scenario, [10:00] and then a super complex scenario that [10:02] you expect users to ask for. And then [10:05] you can examine where scores diverge [10:07] within your team and then refine both [10:08] your guidelines as well as your rubric. [10:12] And key point here is that you should [10:14] not be moving forward until your rubric [10:17] passes these checks. Skipping this step [10:19] is the fastest way to build a product [10:21] that looks done on paper but in reality [10:24] underwhelms your users. [10:28] All right, so now that you've validated [10:30] the rubric, it's really time to put it [10:32] to work. You can now focus on [10:34] operationalizing it by creating an [10:36] evaluation. [10:38] Your core task task is pretty simple. [10:42] Consistently apply your rubric to real [10:44] product output and then track results [10:46] over time. You have two main options [10:49] here. You can use offline evaluations [10:52] um where you're basically analyzing past [10:54] interactions or maybe hypothesized [10:56] interaction if the product doesn't exist [10:58] yet or online evaluations where you're [11:01] actually measuring real live performance [11:04] on real traffic. My main tip here is [11:07] that make sure you do the first pass of [11:09] this evaluation and annotations [11:11] yourself. Don't outsource this [11:13] immediately. I know it's tempting to [11:15] hand this off and um but it's really [11:18] important to understand the nuances of [11:20] your own rubric first and you'll [11:22] discover these edge cases and and [11:23] ambiguities that only become apparent [11:26] when you're actually applying the [11:27] framework to real live data. [11:31] With these four steps we just walked [11:33] through, you've now defined your quality [11:35] framework. You've hopefully built a [11:37] rubric. You've validated it. And now [11:40] you're consistently applying it to real [11:41] data. You're doing the hard work of [11:44] understanding quality looks like and [11:46] you're actually applying it to a [11:47] specific product. [11:51] And that brings us to the heart of why [11:53] this framework matters so much. I want [11:55] to reiterate three main points that I [11:58] just talked about. First, users expect [12:01] AI to take on more of the problem. [12:04] They're not only looking for better [12:06] search results or smarter suggestions, [12:09] but they want you to solve their entire [12:11] challenge. This is why getting your [12:13] quality right from the start is so [12:15] essential. [12:17] Second, defining good is critical in [12:19] this new era of AI. As we've seen [12:22] throughout this framework that we just [12:23] walked through, when AI can do multiple [12:26] directions and there's no single right [12:28] answer, your quality definition becomes [12:30] the guardrail that your product team can [12:33] track and your team needs to align to. [12:36] And then lastly, this connects directly [12:39] to what we just covered. Don't rush the [12:41] eval process. That initial investment in [12:44] doing your own annotations, [12:46] understanding your rubric's nuances, and [12:48] consistently applying it is very worth [12:51] it. Most teams want to skip straight to [12:53] outsourcing or automation. But the [12:56] companies that take the time to get [12:57] their quality frameworks right will [13:00] build AI products that the users [13:01] actually want to use. And this [13:04] systematic approach to quality isn't [13:06] just about building better AI. It's [13:08] about meeting this fundamentally higher [13:10] expectation that users now have for what [13:12] AI products should do for them. [13:16] And with that, thank you for joining [13:18] this