New York Life, WalkMe Executives Tie AI Value to Workflow Changes

纽约人寿与WalkMe高管：AI价值取决于工作流程变革 在人工智能部署中，传统财务KPI依然适用，但企业必须重新设计底层员工工作流程并摒弃虚荣的使用指标，才能实现切实的成本节约。 与会嘉宾探讨了从传统软件指标向兼顾成本、价值与人为影响的现代AI评估框架的转变。他们认为，提示词总数、原始用户注册量或孤立的原型数量等传统虚荣指标无法体现真实的投资回报，甚至可能阻碍绩效。相反，成功的企业级部署需要围绕AI重新设计业务工作流，建立包含拦截控制点（stop-gates）的严格治理体系，并衡量员工留存率、客户满意度以及收入提升等高价值滞后指标。归根结底，弥合企业AI投资与一线员工落地采用之间的差距，取决于规范严谨的产品管理、分阶段的迭代推广以及扎实的变革管理。 企业对 AI 的核心财务期望与传统软件一致，仍然要求通过降低成本、提升收入或提高留存率来体现明确的投资回报率（ROI）。 (2:30) 管理层与一线员工之间存在明显的 AI 信任鸿沟：61% 的高管信任由 AI 执行复杂业务流程，而实际操作的一线员工中仅有 9% 持相同看法。 (2:30) 评估 AI 影响必须超越部署量和总 Prompt 数等虚荣指标，从“成本、价值与人性”三大支柱出发，关注工作流层面的准确性、一致性与长期业务产出。 (4:44) 为防止范围蔓延，企业部署应从小切入，通过将 AI 深度嵌入员工核心工作界面来消除跨系统切换摩擦，并设立明确的终止机制以严格把控财务风险。 (22:03)

While executive expectations for AI remain anchored to conventional financial outcomes—reducing costs, growing revenue, and retaining customers—measuring its real impact requires organizations to discard activity-based vanity metrics in favor of workflow redesign and adoption tracking. Replacing Vanity Metrics with Business Outcomes (13:07) Panelists argue that early AI metrics and legacy efficiency standards fail to capture genuine financial return. Nate Campbell notes that digital transformation costs rose 20% year-over-year, prompting executive demands for clear return on investment; yet tracking raw deployment volume or basic activity provides no accountability. Tali Rosenblum explains that initial indicators, such as counting model on-boarding instances (e.g., Anthropic Claude or OpenAI) or tallying isolated prototypes, quickly lose relevance. Instead, organizations must assess production-grade solutions that consolidate fragmented tools into unified platforms. Adam Gunnett warns that tracking prompt volume is an uninformative vanity metric under flat-rate software pricing, noting that generating ten prompts to receive one usable answer represents inefficiency rather than progress. Jill Hosmer-Jolley adds that traditional call center targets, such as mandatory 30-second wait times or 90-second call caps, become counterproductive once automation handles routine tasks. When AI resolves baseline issues, customer retention depends on allowing representatives the time needed for complex, high-touch problem solving. Workflow Redesign and Frontline Empowerment Adoption metrics reveal a significant operational disconnect: Campbell cites WalkMe survey data indicating that 61% of executives trust AI to execute complex tasks, compared to only 9% of frontline workers. Gunnett observes that applying legacy KPIs to AI-assisted processes yields distorted results, comparing the practice to early factories that merely swapped steam engines for electric motors without reconfiguring pulley systems. True transformation requires rethinking workflows around the technology rather than forcing AI into legacy procedures. Addressing workflow friction directly impacts performance, particularly in enterprise environments where employees navigate an average of 2.88 applications per process. Campbell details a travel company case study where embedding real-time AI guidance directly into booking agents' single-pane interfaces lifted bookings by 15% while closing skill gaps between new hires and tenured staff. Similarly, Busy Beaver utilizes AI in retail aisles to assist inexperienced employees with complex technical queries; stores exhibiting higher active AI usage consistently report superior employee retention, reduced turnover, and higher customer satisfaction scores. Governance via Accelerated Roadmaps and Kill Switches To counter scope creep and prevent budget overruns, Rosenblum outlines New York Life's governance model, which replaces multi-year development cycles with iterative three-month roadmaps. Crucially, the organization integrates formal stop gates and project kill switches. Because product teams often develop emotional attachments to their initiatives, management must foster a culture that actively celebrates terminating underperforming programs to reallocate resources toward more viable capabilities. Campbell emphasizes that financial and operational risks are best contained through phased, diverse pilot groups rather than sweeping enterprise-wide rollouts. Effective pilot testing requires evaluating software across three distinct user profiles: advanced power users who test technical limits, brand-new hires who expose onboarding and workflow friction, and underperforming employees to determine whether the system successfully elevates baseline productivity. Evaluating Bottom-Up Utilities Versus Top-Down Agentic Workflows (42:54) Campbell distinguishes between conversational AI tools used as informal assistants and top-down deterministic workflows. Unstructured, bottom-up implementations—where employees use chatbots as conversational "thinking partners"—operate like workplace utilities or modern employee benefits. Consequently, their impact cannot be tracked through rigid task-level efficiency, but should instead be measured via human capital metrics such as individual job satisfaction, morale, and voluntary retention. In contrast, top-down agentic solutions insert structured AI workflows directly into corporate systems. Because these processes run deterministically, they produce standardized, repeatable performance metrics. This division allows enterprise leadership to apply strict compliance and operational KPIs to automated, high-risk processes while measuring generative assistant tools through broader organizational health indicators. The panel concludes that capturing AI's financial value depends heavily on rigorous change management, continuous workflow audits, and iterative governance rather than raw technical deployment. However, bridging the 52-point trust gap between leadership and frontline personnel remains an ongoing challenge for enterprise-scale adoption.

在Ai4研讨会上，来自零售、保险及数字采用领域的企业管理者指出，人工智能（AI）的财务成功仍取决于成本削减、营收增长和客户留存等传统底线指标，但企业必须抛弃提示词数量等虚荣指标，通过重构工作流与分阶段治理实现真实价值。 财务底线未变与信任鸿沟 (2:30) WalkMe美洲区客户成功副总裁Nate Campbell援引其公司对4000名专业人士的《数字采用现状报告》调查指出，数字化转型成本同比上涨20%，导致管理层对投资回报率（ROI）的期望显著提高。首席财务官关心的核心财务逻辑并未改变，依然聚焦于AI是否能降低成本、增加收入或提升客户留存率。然而，前沿部署中存在严重的信任脱节：61%的高管信任AI能够执行复杂流程，而在实际操作工作流的一线员工中，该信任比例仅为9%。在具体工作流层面，评估重点需要转向用户是否能更快速、更智能地工作，并有效降低合规风险。 摒弃虚荣指标转向成果衡量 (13:07) 与会专家一致认为，企业必须停止追踪过时的活动类指标。Campbell指出，单纯记录部署总量或工具使用活跃度却不挂钩业务成果，属于缺乏问责机制的表现。纽约人寿（New York Life）副总裁Tali Rosenblum强调，半年前流行的初期上线人数、模型测试量（如Anthropic Claude或OpenAI）或零散的原型数量已迅速贬值，当前重点应放在能带来实际业务价值的企业级生产工具上。Busy Beaver业务情报与战略副总裁Adam Gunnett认为，在固定软件收费模式下，总提示词数量纯属虚荣指标，若发出10次提示却得到9个错误答案则毫无意义。主持人Jill Hosmer-Jolley补充道，当AI自动化了80%的日常流程后，客服中心过去强求的30秒等待或90秒通话限制等传统效率考核已适得其反，客户在复杂问题上更看重从容、高质量的服务。 重塑工作流与一线人员赋能 (7:20) 专家强调，评估AI必须重新审视工作流本身，若将新技术直接套入旧流程将导致误导性结果。Gunnett以工厂电气化历史作比：早期工厂仅用电动机替换蒸汽机却保留原有的滑轮传动系统，未能带来根本性变革；只有围绕小型分布式电机彻底重塑车间布局，工业革命的潜力才得以释放。在Busy Beaver的19家区域家装门店中，AI工具被用于协助缺乏经验的年轻员工解答专业维修难题。数据显示，高AI采用率的门店不仅员工留存率更高、离职率降低，且客户满意度评分与营收均优于采用较慢的门店。Campbell分享的旅游预订公司案例也表明，将AI直接嵌入坐席核心工作界面后，新老员工技能差距得以弥合，进而推动预订量增长15%。 项目敏捷治理与终止机制 针对范围蔓延（Scope Creep）和系统复杂性，Campbell指出，跨多系统操作（企业平均流程涉及2.88个应用程序）会带来严重的上下文切换摩擦，团队必须每次集中攻坚一个高优先级工作流。在技术路线上，Busy Beaver选择与外部供应商合作而非自研模型，聚焦于明确业务挑战。Rosenblum指出，现代AI开发周期已从传统的3年缩短至3个月的敏捷迭代路线图，纽约人寿在项目治理中建立了正式的“终止开关”（Kill Switches）与阶段审查门槛。由于叫停投入大量心血的项目在文化上具有挑战性，管理层必须建立一种机制，像庆祝成功发布一样肯定果断叫停低效项目的决策，并通过多项备选能力储备实现平滑转向。 分层测试与自下而上的价值区分 为避免预算失控和上线风险，Campbell建议推行小规模分层试点，测试对象应覆盖三类典型画像：检验技术上限的AI资深用户（Power User）、评估上手门槛与培训需求的新员工，以及验证能否补齐技能短板的低绩效员工。在回答受众关于聊天机器人与确定性工具的区别时，Campbell将AI应用划分为两类：自下而上的“思考伙伴”（如非结构化会话机器人）更像一种现代员工福利或通用基础设施，其价值主要体现在工作满意度、员工留存和个人效能上；而自上而下嵌入工作流的确定性或智能体（Agentic）方案，才能输出可预测、可重复且直接影响企业核心KPI的运营成果。 总体而言，与会者达成共识：AI能否产生财务回报，取决于企业能否建立严谨的产品管理与变革管理机制，敢于重构业务流程并及时止损。然而，面对高度去中心化的组织结构（如纽约人寿旗下1万名独立代理人网络），如何在满足不同业务模式与严格合规要求的同时，持续平衡企业集中管控与一线自主采用，仍是企业规模化落地中需持续探索的现实难题。

## Background 本视频由 Ai4 于 2026 年 8 月 22 日发布在 YouTube 平台上。视频简介指出，本次圆桌研讨会汇集了高校学者及来自纽约人寿、WalkMe 和 Busy Beaver 的多位企业高管，深入探讨如何为企业级 AI 项目量身定制关键绩效指标（KPI），将运营效率改善与财务收益直接挂钩，从而穿透数据噪音、精准验证 AI 带来的真实商业价值。 ## TL;DR 圆桌讨论围绕衡量 AI 财务与业务影响的核心指标展开。专家们指出，单纯追踪提示词调用量或上线数量等虚荣指标已经失效，企业必须回归成本降低与收入增长的商业本质，重点关注采纳率、员工留存与长期客户价值等滞后指标。讨论强调了将 AI 无缝嵌入员工工作流、设立短迭代周期与“熔断/杀手开关”（Kill Switch）以及构建宽容失败文化的重要性。最后，嘉宾们呼吁企业摆脱工业革命初期的“蒸汽机”单点替换思维，借助完善的变革管理机制彻底重塑业务流程。 ## Atmosphere & style 整场研讨会节奏明快、生动风趣且极接地气，主持人 Jill 幽默风趣的穿针引线让现场互动氛围十分热烈。三位来自不同行业背景的嘉宾形成了绝佳的互补视角——既有大型险企的严谨产品管理规范，也有区域零售商深入一线的务实痛点，还有科技厂商对“采纳层”与信任鸿沟的犀利洞察。嘉宾们巧妙运用“水上摩托与航空母舰”、“蒸汽机与分布式电动机”等生动隐喻，将原本枯燥的财务与指标治理议题拆解得既通俗深刻又充满启发性。 ## Key takeaways - [02:38](https://www.youtube.com/watch?v=TAMGiUe-P30&t=158s) 高管（61%）与基层员工（9%）在 AI 信任度上存在巨大鸿沟，向管理层证明 AI 投资合理性的底层逻辑仍未改变，核心依旧是成本是否降低与收入是否增加。 - [04:44](https://www.youtube.com/watch?v=TAMGiUe-P30&t=284s) 衡量 AI 价值需结合成本、价值与人性三个维度；单纯的技术准确性若脱离了工作流的实际利用率便毫无价值，长期客户留存比单次即时销售提升更具战略意义。 - [07:20](https://www.youtube.com/watch?v=TAMGiUe-P30&t=440s) 在零售等高流动性行业，AI 赋能一线员工可显著增强员工的掌控感与工作士气，进而有效降低员工流失率并带动门店销售与服务评分提升。 - [13:31](https://www.youtube.com/watch?v=TAMGiUe-P30&t=811s) 单纯的软件部署量、模型创建数量以及 Prompt 调用次数已沦为无效的虚荣指标，企业应转向评估规模化业务成果与可持续竞争优势。 - [20:32](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1232s) 将 AI 深度嵌入员工的核心工作流中，不仅能减少跨应用切换的摩擦、帮助新手快速达到专家水平，还能直接驱动顶线（Top-line）业绩增长。 - [21:53](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1313s) 防止项目范围膨胀（Scope Creep）的关键是从单一具体工作流切入；中小企业无需自研底层模型，而应以明确的商业结果为导向灵活借力外部服务商。 - [25:47](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1547s) AI 项目落地必须回归产品管理基本功，将传统的长期规划压缩为 3 个月以内的短周期路线图，并设置关键止损点以便灵活转向。 - [27:28](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1648s) AI 的 KPI 会随生命周期演进：研发阶段聚焦时延与准确率等技术指标，生产阶段则必须转向活跃采纳率、销售转化率与客户留存等业务成效指标。 - [32:35](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1955s) 为防止投资套牢，企业需在研发过程中建立‘杀手开关’（Kill Switch）机制与能力备选项矩阵，并在组织内部营造包容且庆祝有价值失败的企业文化。 - [37:59](https://www.youtube.com/watch?v=TAMGiUe-P30&t=2279s) AI 推广应遵循‘从水上摩托到航空母舰’的循序渐进原则，先在代表性小规模群体中完成敏捷验证，再全面推向全组织。 - [39:31](https://www.youtube.com/watch?v=TAMGiUe-P30&t=2371s) 变革管理（Change Management）是 AI 战略落地的生命线，缺乏配套的沟通、系统化培训与持续支持体系会导致先进技术探索付诸东流。 - [41:48](https://www.youtube.com/watch?v=TAMGiUe-P30&t=2508s) 企业必须摆脱‘蒸汽机’式的单点替换思维，将 AI 视为新型分布式电动机，从根本上重塑整个业务流程与生产流线以释放生产力。 - [42:53](https://www.youtube.com/watch?v=TAMGiUe-P30&t=2573s) 评估 AI 需区分场景：作为自底向上‘思考伙伴’的探索型工具侧重衡量员工满意度与赋能感，而嵌入主干工作流的确定性智能体流程则必须绑定刚性业务底线 KPI。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=TAMGiUe-P30&t=0s) 开场致辞与与会嘉嘉宾介绍 研讨会的主持人 Jill Hosmer-Jolley（加州州立大学蒙特雷湾分校教授）热情开场，打趣地向观众透露自己在今年 7 月与这三位专家开研讨会预备会时，就被他们源源不断的新颖想法所震撼。她开玩笑说今天可能要讲上三个小时，劝大家“别离开房间”。此外，她特别感谢了 Busy Beaver 的 Adam Gunnett——因为原定的一名嘉宾在最后时刻临时缺席，Adam 义不容辞地紧急救场。教授还根据现场视野情况，现场要求嘉宾们把椅子向前挪了挪，展现出活泼且充满互动感的现场氛围。 随后，三位嘉宾依次进行了自我介绍并阐述了各自的业务背景： * **Adam Gunnett**：区域性家居改进零售商 Busy Beaver 的业务智能与策略副总裁（VP of Business Intelligence & Strategy）。Busy Beaver 拥有 19 家线下门店，他指出作为中小型区域零售商，他们在 AI 的应用与衡量方式上与大型企业有着显著不同的视角。 * **Tali Rosenblum**：纽约人寿保险（New York Life Insurance Company）企业 AI 引导分析与情报解决方案副总裁（VP of AI, Analytics and Data）。她已在纽约人寿服务了 16 年（自第一个孩子出生后便加入公司），见证了公司通过不断迭代 AI 与数据分析产品来为客户和保险代理人（Agents）创造价值的演进历程。 * **Nate Campbell**：WalkMe（已被 SAP 收购）的美洲区客户成功副总裁（VP of Customer Success Americas）。他的核心关注点在于“应用采纳（Adoption）”，即如何在员工日常工作流程中合适的时间节点引导其采用 AI 技术。作为客户成功负责人，他同时站在客户采纳 AI 以及自家团队采纳 AI 工具的双重视角。 --- ### [02:38](https://www.youtube.com/watch?v=TAMGiUe-P30&t=158s) AI 时代 KPI 的变革：与传统衡量指标的异同 主持人将话题引入核心焦点——“谈谈钱与投资回报（Let's talk money）”，并向 Nate 提问：在 AI 领域衡量成功，与传统的 KPI 相比究竟有何差异？ Nate 结合 WalkMe 每年发布的《数字采纳现状报告》（该报告深度调研了 4,000 名从基层员工到管理层的专业人士）分享了几个关键洞察： ```mermaid flowchart LR accTitle: AI Adoption and Value Friction A["企业数字化转型成本 <br>（同比上升 20%）"] --> B["高管对 AI 处理复杂流程的信任度 <br>（61%）"] A --> C["基层员工对 AI 处理复杂流程的信任度 <br>（9%）"] B --> D["巨大认知认知鸿沟 <br>（阻碍 AI 采纳与价值落地）"] C --> D ``` * **成本与期望剧增**：企业数字化转型的成本同比上升了 20%，这意味着高管层对这些转型项目交付实际效益的期望值被大幅拉高。 * **高管与基层员工的信任鸿沟**：61% 的高管相信 AI 能够胜任复杂的业务流程，但真正需要操作工具、驱动价值落地的基层员工中，这一信任度仅为 9%。这种巨大的认知差距使得“采纳层（Adoption Layer）”成为了决定 AI 最终能否成功的关键。 在谈到 KPI 本身时，Nate 认为**底层逻辑并没有本质区别**。无论技术如何演变，在向 CFO 证明投资合理性时，最终的核心依然是：**成本是否降低？收入是否增加？** 唯一的区别在于细节指标的切入点（如模型消耗成本、架构部署成本等）。在工作流层面，企业关注的依旧是员工是否工作得“更好、更快、更聪明”、合规风险是否降低，以及软件客户的续约留存率是否提升。 --- ### [04:44](https://www.youtube.com/watch?v=TAMGiUe-P30&t=284s) 衡量 AI 价值的三维框架：成本、价值与人性 在纽约人寿深耕 16 年的 Tali 随后分享了她在大型金融保险机构观察到的变化。她总结指出，业务的核心野心（Ambitions）——例如提供更优质的客户服务、赋能代理人、提高生产力与客户留存率——并没有改变，改变的是“实现方式（The How）”。 Tali 提出了衡量 AI 实际价值的三维框架： ```mermaid flowchart TD accTitle: Three Dimensions of AI Measurement Framework A["AI 衡量标准三维框架"] --> B["成本 (Cost) <br> 运营开支与效率优化"] A --> C["价值 (Value) <br> 业务增量与长期留存"] A --> D["人性 (Humanity) <br> 代理人与客户体验的结合"] C --> E["技术维度的测试 <br> 模型准确性、一致性"] C --> F["工作流维度的验证 <br> 步骤落地与实际利用率"] ``` Tali 强调，AI 带来的变革主要体现在**技术**与**工作流**两个层面： 1. **技术层面**：企业现在必须对模型输出的准确性（Accuracy）和一致性（Consistency）进行前所未有的大规模测试，确保将其交付给员工时能够放心落地。 2. **工作流层面**：如果一个 AI 工具即使技术指标再高、预测再准，但在实际工作流中无人问津（Lack of Utilization），那它就无法创造实质效益。 此外，Tali 还指出，对于纽约人寿这样拥有 10,000 多名保险代理人、服务数百万客户的强人际纽带业务，AI 绝不是给予确定性的硬性命令，而是提供智能化建议（Recommendations）。相比于在某一瞬间“多卖出了一美元”的即时销售提升，**客户是否因为更优质的交互体验而被长期留存下来**，往往是更具决定意义的滞后指标（Lagging Indicator）。 --- ### [07:20](https://www.youtube.com/watch?v=TAMGiUe-P30&t=440s) 区域零售企业的实践：以 AI 赋能基层员工与降低流失率 作为小型区域零售商代表的 Adam 对 Nate 和 Tali 的观点表示高度赞同，并从零售业实际痛点出发补充了不同视角。Adam 强调，评估 AI 必须回归到具体的“工作流”中。如果直接沿用旧流程的 KPI 去套用 AI 赋能的新流程，就会产生偏差——“虽然基线和业务责任人没变，但只要聚焦于最终的业务结果（Business Outcome），你就可以重新定义具体的 KPI。” 主持人借此追问 Adam：在开发和验证这些新 KPI 时，他会向谁调取最宝贵的信息？ Adam 表示，他的团队总是直接深入一线，向**一线店员（Frontline Associates）**收集直接反馈。零售业是一个员工流失率（Employee Turnover）极高、招聘挑战极大的行业： * **传统痛点**：家居改进（Home Improvement）行业技术门槛高，以往店里极度依赖懂行的退休水管工等行业专家。然而面对年轻、未经训练的新员工时，培训成本极高，店员常因无法解答顾客问题而产生挫败感。 * **AI 赋能**：通过为店员配备智能 AI 工具，新员工能够随时调取专业知识解答顾客疑难。这不仅没有像传统观念那样削弱员工积极性，反而极大地**增强了员工的掌控感与工作热情（Employee Morale）**。 * **滞后指标的验证**：Busy Beaver 发现，积极采纳 AI 平台的门店，其**员工留存率显著上升、员工流失率下降，同时店面客户服务评分与销售收入也同步增长**。与那些尚未接受新工具的门店相比，形成了非常鲜明的对照。 --- ### [13:31](https://www.youtube.com/watch?v=TAMGiUe-P30&t=811s) 哪些 KPI 已经失效？专家眼中的“无用指标” 主持人 Jill 向三位嘉宾提出了一个现场互动性极强的问题：“在你不得不跟踪的指标中，哪一个是目前看来最无用、纯粹是因为历史原因被沿用至今的？” 三位专家分别给出了深刻的行业剖析： * **Nate Campbell（WalkMe）——单纯的“部署量/上线量（Deployed Numbers）”**： Nate 批评了将“软件是否部署成功”或“是否有人点开”作为核心 KPI 的做法。他认为这代表企业不敢深挖业务本质，不敢对真正的业务结果负责。衡量成功的重点必须从“工具被使用了”延伸至“使用者输出了怎样的质量指标，是否让业务做到了更快、更好”。 * **Tali Rosenblum（New York Life）——早期的“模型数量与活动活跃度（Activity & Adoption）”**： Tali 表示半年前被奉为圭臬的指标，现在可能就已经过时。在企业引入 Anthropic Claude 或 OpenAI 的初期阶段，衡量“有多少人在尝试”、“创建了多少个模型”是有意义的；但当进入规模化生产阶段时，这些活动指标就变得非常低效。企业应该转向评估**“产品是否形成了企业级规模”、“是否避免了重复建设（如公司内部出现了 10 个不同的知识库管理工具）”**以及**“是否创造了可持续的竞争优势”**。 * **Adam Gunnett（Busy Beaver）——“总提示词调用量（Total Prompts）”**： Adam 直言不讳地指出，很多大公司喜欢统计 Prompt 数量，但他认为这是毫无意义的指标。“如果用户输入了 10 次 Prompt，结果收到了 9 次糟糕的答案，这个数字对我有什么价值？”由于公司采用的是平价订阅制，单纯的 Prompt 调用次数并不能直接反应业务成效，必须始终聚焦于最终交付的 Outcomes。 在讨论的最后，主持人 Jill 补充了一个来自呼叫中心（Call Center）流程变革的例子：过去呼叫中心将“等待时间（Wait Time）控制在 30 秒内”或“90 秒内挂断电话”视为硬性 KPI。然而在 AI 承担了 80% 简单重复的工作后，留给人工接线员的都是复杂的 20% 深度问题。此时，用户反而愿意等待更久以获得高质量的解答，通话时间拉长也成为常态——传统以“快”为核心的 KPI 在 AI 重新塑造工作流后已然失效。 ### [18:49](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1129s) 从交易型互动转向长期品牌价值 Panel 讨论了如何将 AI 融入业务流程，从而重塑与客户的沟通模式。传统流程中，客户致电（例如处理账单或缴费）往往只是完成一项交易；而引入 AI 后的重构流程，能够在关键时机为客服代表提供洞察，推动更有价值的互动。 <Image alt="以 AI 驱动的客户服务重构流程示意图" caption="重构客服流程：从交易型互动走向价值创造" src="image_agent_tag_1"/> --- 这种重构不仅提升了客户体验，还能通过体验升级后的转化率变化来精准衡量 ROI。 * **体验升级与等待价值**：正如患者愿意在诊所耐心等待 30 分钟一样，当客户认为互动体验具备高度价值且超越了简单的“交易”属性时，短时间的等待便不再是负面体验。 * **个性化建立长期信任**：纽约人寿（New York Life Insurance Company）的 Tali Rosenblum 强调，AI 带来的个性化能力能够让客服代表实时掌握客户的具体背景与困境，从而改变沟通的语调与关系形态，确立可长期信赖的品牌形象。 --- ### [20:32](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1232s) 在工作流中无缝集成 AI：以旅游预订为例 WalkMe 的 Nate Campbell 分享了一个旅游预订公司的真实客户案例。对于任何需要提供实时服务的客服或代理团队而言，高流动率是普遍挑战。因此，团队成员往往呈现“新老交替”的断层状态——一半是经验丰富的资深员工，另一半则是刚入职或正在接受培训的新人。 为了解决在新老交替中保持服务质量的难题，该公司将原本孤立的 AI 工具直接嵌入到员工的“核心工作流（Flow of Work）”中。 ```mermaid flowchart LR accTitle: AI 工作流集成路径 A["传统模式: 外部 AI 工具"] --> B["频繁切换上下文 / 信息孤岛"] C["集成模式: 嵌入核心工作流"] --> D["实时辅助客服与客户沟通"] D --> E["业务转化率与顶线收入提升 15%"] ``` * **消除“新手感”**：AI 的实时辅助让新员工在与客户沟通时表现得如同资深专家，显著改善了客户体验。 * **驱动业务增长**：除了非财务维度的体验提升外，该项改进还直接推动预订量增长了 15%，实现了客户满意度与顶线（Top-line）业绩的双赢。 --- ### [21:53](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1313s) 应对风险与控制“范围蠕变”（Scope Creep） 针对主持人 Jill Hosmer-Jolley 提出的关于如何管控财务与非财务风险、以及如何防止项目要求无休止膨胀（Scope Creep）的问题，各方代表给出了实操性建议。 Nate Campbell 解释道，客户在签署协议后常常渴望“一口吞下大象”，希望立刻解决所有难题。但企业必须建立自我约束机制： * **从单一工作流切入**：必须专注于一个具体的工作流，深入分析数据并倾听一线用户的反馈，明确日常操作中的真实痛点。 * **应对上下文切换**：现代企业架构极其复杂，员工处理一个典型流程往往需要跨越平均 **2.88** 个应用程序。多系统（如 Google Doc、不同代码库）之间频繁切换极易导致错误。 * **指标设定与循序渐进**：首先解决一个具体难题，验证其对成本降低、合规风险控制以及效率提升的实际效果，建立一套成熟的操作方法论后再逐步扩展。 Busy Beaver 的 Adam Gunnett 则代表了中小型企业的视角： * **灵活借力外部服务商**：作为一家没有内部 AI 开发团队的小型公司，Busy Beaver 并不自研模型，也不必纠结于底层的 Token 消耗速率。 * **用对工具与明确业务目标**：Adam 提出“不同的任务需要不同的载体”——正如去赛道要开跑车、运木材需要皮卡一样。他将具体的业务挑战（如零售定价优化、商品企划等）和期望的商业结果提交给 AI 服务商，由服务商构建对应的平台与解决方案。 --- ### [25:47](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1547s) 回归产品管理基本功与研发迭代周期 Tali Rosenblum 指出，在当前各大企业蜂拥抢占 AI 创新高地的浪潮中，人们容易忽视最基础的**产品管理（Product Management）基本功**。 ```mermaid flowchart TD accTitle: 现代 AI 产品管理迭代机制 A["明确预期业务目标 (Outcomes)"] --> B["设定清晰的项目范围 (Scope)"] B --> C["构建 3 个月短周期 Roadmap"] C --> D["设置关键止损点 (Stop Gates)"] D --> E{"数据与用户反馈评估"} E -- 达到预期 --> F["继续推进与扩展应用"] E -- 未达预期 --> G["执行灵活转向 (Pivot)"] ``` * **明确目标与范围**：从一开始就必须清晰定义预期的业务目标与实施范围，切忌盲目跟风。 * **短周期与止损点**：不同于传统耗时 3 年的路线图，现代 AI 项目的落地周期往往压缩在 3 个月以内。因此，必须在流程中建立清晰的止损点（Stop Gates）和转向点（Pivot Points），给予团队根据实际反馈调整策略的自由。 --- ### [27:28](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1648s) KPI 的演进：从技术指标到业务成果 Adam Gunnett 详述了 AI 项目在不同生命周期阶段所关注的关键绩效指标（KPI）的演进路径： * **初始研发与技术验证阶段**：重点关注平台与工具本身的技术指标，如**时延（Latency）**与**准确率（Accuracy）**。 * **上线与生产应用阶段**：评估重点转向业务成果与一线使用情况。以零售场景为例，关键指标包括门店层面的**活跃采纳率（Active Adoption Rate）**、**客户服务评分**、**销售转化率（Conversion Rate）**以及**客户留存率（Retention）**。 Nate Campbell 补充道，在挑选优先切入的场景时，应优先考虑能产生最大业务回报或解决最严重业务痛点（如重大合规隐患）的流程。 | 评估维度 | 核心关注点与示例 | |:--- |:--- | | **紧急程度评估** | 区分“真正紧急”与“仅是锦上添花”的需求。如万事达卡（Mastercard）通过 AI 拦截了近 100 亿美元的诈骗损失，合规与风控往往具备极高优先级。 | | **基准线与持续优化** | 必须对全工作流进行基准线测试（Benchmarking），上线后持续跟踪评估，避免“一上线就不管”的现象。 | --- ### [30:30](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1830s) 解决大型组织中的目标错位 在拥有上万名一线代理（Agents）的大型机构（如纽约人寿）中，总部统一的战略目标与地方独立经营者的日常需求往往存在差异。 * **多元化需求适配**：代理人的业务模式不尽相同，有的提供端到端的全方位财务规划与咨询服务，有的则侧重于单一产品的销售。系统必须兼顾不同模式的需求。 * **持续倾听与个性化**：借助 AI 技术，公司能够更好地针对不同类型的代理人与客户提供定制化支持，实现总部战略与一线体验的契合。 --- ### [32:35](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1955s) 预算控制、“杀手开关”（Kill Switch）与失败文化 针对项目是否容易超支以及如何在长期实施中控制财务风险的问题，Tali Rosenblum 与 Adam Gunnett 展开了深入探讨。 ```mermaid flowchart TD accTitle: 投资保护与杀手开关机制 A["资本投人 (Investment)"] --> B["设置定期止损节点 (Stop Gates)"] B --> C{"指标是否达标?"} C -- 是 --> D["继续投入并扩展成果"] C -- 否 --> E["触发杀手开关 (Kill Switch)"] E --> F["灵活转向至能力备选项 (Portfolio)"] F --> G["文化层面庆祝经验积累与试错"] ``` * **建立“杀手开关”（Kill Switch）**：将 AI 视为一项投资，在研发过程中设置多个定期检查的止损点。一旦发现项目的 ROI 或产出未达预期，便果断触发“杀手开关”停掉项目，避免资金持续套牢。 * **构建能力备选项（Capabilities Portfolio）**：由于研发与产品团队容易对其倾注心血的“作品”产生情感依赖，企业需要提供成熟的备选方案矩阵，让团队在项目被砍时能迅速转向其他有价值的方向。 * **重构企业文化**：在组织内部营造一种“不仅庆祝成功，也同样认可并庆祝有价值的失败”的文化氛围，消除团队承认项目失败的心理顾虑。 此外，Adam Gunnett 打趣道，这是他参加的第一个将“Agents”一词用来指代“人类员工”而非“AI Agents”的研讨面板。他指出，许多企业过去犯的错误是将“系统上线”当成终点，而实际上这只是持续获取用户反馈、优化流程与探索下一个效率增长点的起点。 Nate Campbell 总结道，控制成本的关键在于“从小规模试点开始（Start Small）”。在将新系统全面推广之前，应当先提供给一小部分代表性群体测试——包括 AI 专家、刚入职的新人以及业绩待提升的员工，在小范围内验证并完善后再行推广。 <FollowUp label="想了解纽约人寿如何具体构建‘杀手开关’与止损评估机制吗？" query="请详细介绍纽约人寿在 AI 项目管理中构建止损点（Stop Gates）和‘杀手开关’（Kill Switch）的具体标准与实施流程。"/> ### [037:59] 敏捷扩展与转折点：从“水上摩托”到“航空母舰” 讨论回顾了 AI 项目上线与迭代的循序渐进过程。在项目小规模推出时，团队能够收集到大量多维度的反馈与分析数据，从而判断各项核心指标是否在朝正确的方向发展、成本是否可控以及投资是否达到了预期效果。唯有在这些验证都得到积极反馈后，才适合将其全面推向拥有成千上万员工的广泛业务场景。 主持人 Jill Hosmer-Jolley 强调了敏捷机制中“熔断开关”（Kill Switch）以及“全面部署并非终点”的理念，并分享了一个生动的比喻：企业不应该一开始就直接去驾驶一艘装载着一万辆水上摩托的庞大航空母舰，而应该先骑好灵活的水上摩托（Jet Ski），随着认知与体系成熟再升级到渔船，最后才过渡到航空母舰。随后，主持人引导三位嘉宾给出各自最希望观众在今晚晚餐时仍能牢记的核心结语与关键心得。 ```mermaid flowchart LR accTitle: AI 项目演进与扩展阶段 A["灵活验证<br/>（水上摩托 / Jet Ski）"] --> B["阶段放大<br/>（渔船 / Fishing Boat）"] B --> C["全面规模化<br/>（航空母舰 / Aircraft Carrier）"] ``` --- ### [039:31] 嘉宾核心金句（一）：变革管理是 AI 落地的生命线 来自 New York Life 的 Tali Rosenblum 指出，业界在探讨技术能力时往往容易忽视一个至关重要的词汇——变革管理（Change Management）。当前市场涌现出层出不穷的技术赋能，赋予了每个人成为创新者和开发者的潜力，但这种能力若要在大型企业中真正发挥价值，必须依赖强有力的组织支撑。 Tali 强调，如果企业内部没有建立起相配套的沟通机制、系统化的培训方案以及持续的运营支持体系，那么再先进的 AI 探索最终都会付诸东流。因此，正如团队必须严格遵守产品管理纪律去圈定核心范围一样，变革管理机制的落地也是决定 AI 战略成败不可或缺的基石。 - **组织赋能的挑战**：人人皆可创新的背后，离不开清晰的自上而下沟通与制度化引导。 - **关键支撑体系**：有效的沟通（Communications）、培训体系（Training）和支持保障（Support System）是避免 AI 举措半途而废的防线。 --- ### [040:44] 嘉宾核心金句（二）：深入业务流程，构建极致的用户采纳层 WalkMe 的 Nate Campbell 提出了他的核心信条：始终聚焦于“采纳层”（Adoption Layer）并勇于尝试大胆突破（Take big swings）。他指出，要设计出一套优秀的落地采纳方案，就必须极其深入地挖掘每一项操作背后的深层原因（The "why"）。 这要求团队必须细致剖析业务流程中的每一个具体环节：员工在哪个步骤会感到困惑？在何处容易犯错？场景切换发生在哪里？系统又该如何在实际工作流（Flow of work）中与用户互动以确保其顺利完成任务。当团队把这些底层细节打磨透彻后，就能向管理层或 CFO 呈现出更加全面、真实的 KPI 组合与业务成效。 - **深挖流程痛点**：精准捕捉员工在工作流中的情绪卡点、混淆点与上下文切换断层。 - **以采纳促指标**：通过前期的严谨调研与用户测试，最终推导出能够切实证明业务价值的多元 KPI。 --- ### [041:48] 嘉宾核心金句（三）：重塑工厂格局——摆脱“蒸汽机”思维定势 Busy Beaver 的 Adam Gunnett 结合前两位嘉宾的观点，分享了关于技术变革与流程再造的历史隐喻。他提到，人们往往会对变革管理产生天然的抵触心理，但历史证明，仅仅做局部替代是无法释放真正生产力的。 在工业革命初期，当电动机刚刚发明时，工厂只是简单地把电动机安装在原来蒸汽机的位置，继续沿用陈旧复杂的滑轮与皮带传动系统。虽然速度略有提升，但并没有带来颠覆性的变革；直到工厂开始将小型电动机分散布置在各个工位，彻底重新设计了整个工厂的生产流线，电力的真正潜力才被全面释放。Adam 强调，AI 正是当代的“新型电动机”，企业绝不能为了规避变革管理而墨守陈规，必须借助 AI 的契机去从根本上重塑整个业务流程。 ```mermaid flowchart TD accTitle: 蒸汽机模式与新型分布式 AI 流程对比 subgraph Old["传统替代思维（蒸汽机模式）"] O1["保持旧业务流与组织架构不变"] --> O2["直接用 AI 单点替换旧工具"] O2 --> O3["仅获得微弱的局部提速"] end subgraph New["流程重塑思维（分布式电机模式）"] N1["以 AI 为核心重新设计业务流程"] --> N2["将智能能力融入各个具体工位"] N2 --> N3["彻底释放业务生产力与潜能"] end ``` --- ### [042:53] 现场问答：如何衡量“思考伙伴”与“确定性工具”的 KPI 在最后的现场提问环节，会议助理 Emily 迅速跑过全场传递麦克风。现场观众提出了一个普遍面临的难题：当员工将聊天机器人（Chatbot）仅仅作为日常启发的“思考伙伴”（Thinking Partner）时，应如何衡量其 KPI？这与那些能产出明确、可见交付物的确定性 AI 工具在指标衡量上有何区别？ Nate Campbell 将这两种场景划分为“自底向上”与“自顶向下”两种模式进行了深度解答： 1. **思考伙伴（自底向上的能力支持）**： - **定位**：类似于“AI 即服务”（AI as a Service）或企业为员工提供的新型福利包（Benefits Package），允许员工按需探索，而非遵循预设的开发路径。 - **衡量方式**：这类工具的价值往往体现在员工的工作满意度、个人赋能以及个体层面的绩效提升上，难以用单一的刚性业务 KPI 进行硬性约束。 2. **确定性方案（自顶向下的智能体流程）**： - **定位**：深度嵌入到日常工作流中、非可选的预设智能体（Agentic）解决方案。 - **衡量方式**：必须与业务主干流程绑定，通过稳定、一致的业务 KPI（如处理时效、差错率、财务收益）来直接评估其对业务底线的实质性影响。 在风趣的道别与掌声中，主持人宣布本场关于 AI 财务影响与 KPI 衡量的圆桌讨论圆满结束。 ## Notable quotes > [07:20](https://www.youtube.com/watch?v=TAMGiUe-P30&t=440s) 虽然基线和业务责任人没变，但只要聚焦于最终的业务结果，你就可以重新定义具体的 KPI。 — Adam Gunnett > [13:31](https://www.youtube.com/watch?v=TAMGiUe-P30&t=811s) 如果用户输入了 10 次 Prompt，结果收到了 9 次糟糕的答案，这个数字对我有什么价值？ — Adam Gunnett > [37:59](https://www.youtube.com/watch?v=TAMGiUe-P30&t=2279s) 企业不应该一开始就直接去驾驶一艘装载着一万辆水上摩托的庞大航空母舰，而应该先骑好灵活的水上摩托，随着认知与体系成熟再升级到渔船，最后才过渡到航空母舰。 — Jill Hosmer-Jolley > [39:31](https://www.youtube.com/watch?v=TAMGiUe-P30&t=2371s) 如果企业内部没有建立起相配套的沟通机制、系统化的培训方案以及持续的运营支持体系，那么再先进的 AI 探索最终都会付诸东流。 — Tali Rosenblum > [40:44](https://www.youtube.com/watch?v=TAMGiUe-P30&t=2444s) 始终聚焦于“采纳层”并勇于尝试大胆突破。 — Nate Campbell > [40:44](https://www.youtube.com/watch?v=TAMGiUe-P30&t=2444s) 要设计出一套优秀的落地采纳方案，就必须极其深入地挖掘每一项操作背后的深层原因。 — Nate Campbell > [21:53](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1313s) 不同的任务需要不同的载体——正如去赛道要开跑车、运木材需要皮卡一样。 — Adam Gunnett > [32:35](https://www.youtube.com/watch?v=TAMGiUe-P30&t=1955s) 从小规模试点开始。在将新系统全面推广之前，应当先提供给一小部分代表性群体测试。 — Nate Campbell ## Mentioned Jill Hosmer-Jolley, Tali Rosenblum, Nate Campbell, Adam Gunnett, California State University Monterey Bay, New York Life Insurance Company, WalkMe, SAP, Busy Beaver, Anthropic Claude, OpenAI, Mastercard, Google Doc ## Video description Jill Hosmer-Jolley — Faculty, California State University Monterey Bay Tali Rosenblum — Vice President, Executive Officer | Enterprise AI-Led Analytics & Intelligence Solutions, New York Life Insurance Company Nate Campbell — VP, Customer Success Americas, WalkMe Adam Gunnett — VP of Business Intelligence & Strategy, Busy Beaver Not all AI metrics are created equal. This panel will explore effective AI use cases through the lens of specific KPIs that were relevant to the project. Panelists will dive into how they select the right KPIs for the right use cases, ensuring that AI initiatives are measured against outcomes that reflect true business value. Panelists will explore how to link operational improvements to financial performance, which leading and lagging indicators matter most across different functions, and how to build dashboards that executives actually use. Expect practical guidance on mapping use cases to meaningful metrics so organizations can separate signal from noise and validate ROI with confidence. [Source](https://www.youtube.com/watch?v=TAMGiUe-P30)

[7:20] Adam Gunnett: 虽然基线和业务责任人没变，但只要聚焦于最终的业务结果，你就可以重新定义具体的 KPI。 [13:31] Adam Gunnett: 如果用户输入了 10 次 Prompt，结果收到了 9 次糟糕的答案，这个数字对我有什么价值？ [37:59] Jill Hosmer-Jolley: 企业不应该一开始就直接去驾驶一艘装载着一万辆水上摩托的庞大航空母舰，而应该先骑好灵活的水上摩托，随着认知与体系成熟再升级到渔船，最后才过渡到航空母舰。 [39:31] Tali Rosenblum: 如果企业内部没有建立起相配套的沟通机制、系统化的培训方案以及持续的运营支持体系，那么再先进的 AI 探索最终都会付诸东流。 [40:44] Nate Campbell: 始终聚焦于“采纳层”并勇于尝试大胆突破。 [40:44] Nate Campbell: 要设计出一套优秀的落地采纳方案，就必须极其深入地挖掘每一项操作背后的深层原因。 [21:53] Adam Gunnett: 不同的任务需要不同的载体——正如去赛道要开跑车、运木材需要皮卡一样。 [32:35] Nate Campbell: 从小规模试点开始。在将新系统全面推广之前，应当先提供给一小部分代表性群体测试。

[13:07] Adam Gunnett: 10 prompts and I got nine bad answers... doesn't make a difference to me. [2:30] Nate Campbell: 61% of executives trust AI to perform complex processes, compared to only 9% of frontline employees who actually execute those workflows. [2:30] Nate Campbell: Cost of Digital Transformation: Up 20% year-over-year, driving significantly higher expectations from executive leadership for a clear return on investment. [7:20] Tali Rosenblum: 10,000 agents serving millions of clients across the U.S. [18:49] Nate Campbell: 15% increase in bookings [22:03] Nate Campbell: 2.88 applications [24:30] Tali Rosenblum: 3-month iterative roadmaps [27:36] Nate Campbell: $10 billion

[0:04] I have to tell you, you are in for a treat with  these three. I was on a conference call with them [0:10] like in July and just idea after idea after idea.  So, we're going to do three hours today. Don't [0:15] leave the room. You don't mind, right? We're all  good. We're all good. Um, and Adam here stepped up [0:21] in the very last minute when we lost a panelist  at the last minute. So, thank you, Adam. Um, [0:26] I'm going to move you guys move your chairs up  a little bit. I'm going to be I'm a professor so [0:32] I change everything no matter if it needs it or  not. And there's people over here can't see us. [0:38] Hi. Okay. So, back to my notes. Tada. Hey. So,  first of all, I'm going to start with you, Adam, [0:46] and then I'm going to go across. Can you just  give everybody like what they're going to get [0:49] from you in the next uh however long we spend  together? Yes. So, uh, I just got pulled in [0:55] off the blackjack table because you needed, uh,  someone to fill in. Uh, no, I'm actually the, uh, [1:01] vice president of business intelligence strategy  for a regional retailer called Busy Beaver. So, [1:06] we have, you know, 19 stores and the way we  approach things is a little bit different than [1:11] some of these bigger companies. That is awesome.  Thank you so much. Uh, Tally, hi guys. Welcome. [1:19] Uh, my name is Tally Rosen Bloom. Um, I'm the vice  president of AI, analytics, and data in New York [1:26] Life. I've been there for about 16 years. And  I know that because right after my first child, [1:32] uh, I came to New [clears throat] York Life  and and it's a company that I keep. Um, [1:37] we have been on an evolution to create amazing AI  and analytics products to create value for clients [1:44] and agents. Can't wait to tell you about it today.  Okay. And Nate. And I'm Nate Campbell. I'm a VP of [1:52] customer success at WalkMe, an SAP company. Um,  today, I mean, at the end of the day, I mean, [1:58] it's been fantastic just learning from so many  different people. So, I feel like my viewpoints in [2:02] the last like 48 hours have also kind of shifted  a little bit as well. But at the end of the day, [2:06] like we we focus on adoption. We're we're in the  application, in the flow of work, helping people [2:10] adopt AI at the right moment in time throughout  their day. And obviously, at the end of the day, [2:16] I'm a VP of customer success. And so obviously  I'm, you know, in a unique position where I'm [2:20] working with customers on adoption of AI. I'm  working with my team on the adoption of our AI. [2:25] So we'll see where the conversation goes and  what point of view I take. [laughter] Sounds [2:30] good. Well, so for two days, everybody's thinking  they've got their shopping list going for I want [2:35] that one and that one and that one and that  one. And so I think we have a nice crowd in [2:38] this room today are like, let's talk money.  So I'm going to start with Nate before I go [2:43] to Tally. And it's like measuring success. How  different is measuring success in the AI space [2:49] from conventional KPIs? Yeah, it's it's a great  question. I mean, where like to kind of start [2:56] with so our state of digital adoption report, it's  something that we put out every single year. It [3:01] uh interviews 4,000 different professionals from  the IC level to the leadership level. And we saw [3:06] a couple of different really key points come  out of it this year. uh we saw that uh cost of [3:13] transformations up 20% year-over-year significant  so the expectations coming out of those expect [3:20] uh coming out of those transformations from  leaders are obviously significantly higher we [3:24] also see the trust in AI to do complex processes  is 61% for executives 9% for employees and so [3:34] the people deciding on whether or not AI is  obviously integral which obviously it should [3:40] to the people who are actually there to adopt it  and drive the value of it. there's a massive gap [3:45] and so that comes back around to the adoption  layer and that's you know primarily obviously [3:49] my key focus and so when it comes to like the  differences in the KPIs I don't see much of a [3:55] difference at the end of the day if you're trying  to prove to your CFO what exactly we're you know [4:01] rationalizing in relation to any investment they  always come back around to is cost getting lower [4:07] is revenue going higher and obviously the KPIs  that come back around to that is going to change [4:11] obviously a little bit different the cost of  consumption versus you know the model and the [4:16] setup of the process itself maybe a little bit  unique in that way but if you look at it at the [4:21] workflow level you're looking at still the same  things in relation to are people working better [4:27] faster and smarter are you decreasing our  compliance risks are you you know helping you [4:31] know drive a true outcome that's actually going to  lead to decrease cost increased revenue or in my [4:37] case obviously increased retention from a software  standpoint that sounds great thank you So Tally, [4:44] I'm going to say so the question is really about  change, right? The difference between before and [4:49] after. And you've been there 16 years at New York  Life. You've seen KPI measurements change so much [4:55] over time. What have you observed in the past,  you know, 5 to seven years where AI is coming into [5:01] it? I I think that's great. There's it's there  are many things that are not changing and many [5:08] things that are changing all at the same time.  I absolutely agree with you. I think it's it's [5:13] um cost, it's sales. I like to break it apart  into three components for a second. Um and I like [5:19] to say it's cost, it's value, and it's humanity.  Right? These are the three factors that we take a [5:26] look at from a measurement perspective. And the  way that we're taking a look at this is first [5:32] and foremost, you talk about this need, the  business, right? The business has particular [5:37] ambitions. Those ambitions generally speaking are  not changing. Whether it be make sure that we have [5:44] better customer service, make sure that our agents  are able to provide that service to our clients, [5:51] right? Those are not changing. Make sure  that they're more productive, more yielding, [5:56] more there's more retention. All these aspects  are key aspects that are unchanging. It's the [6:03] how we're going about it, right? And then when  you think about the how we're going about it, [6:09] I break it apart into two components that are  changing. And the how is the technical and the [6:15] workflow aspect. The technical aspect, we are now  testing things at levels that we've never had to [6:21] test them before. Right? Insights, intelligence,  things of that nature in the past. Have you [6:27] visualized properly? Have you done these sort of  now you're testing the accuracy, the consistency? [6:34] You're testing the workflows and whether the steps  that you anticipated are going to be taking place [6:40] are the steps that are taking place. Because at  the end of the day, you need to sign off. You need [6:45] to feel comfortable that what you're putting into  production in the hands of people that are being [6:51] entrusted to create value. It's generating the  needed value. And so that's a lot of um confidence [6:58] that you have to have as a leader uh to go forward  with something of this nature. So the level of [7:04] accountabilities and the measurements have evolved  profoundly as a result. That's awesome. And it [7:10] doesn't remind me here how big your company is,  but I know it's way larger than what Adam works [7:16] with. Adam, can you give us the smaller company  perspective on what Tally just shared? Yeah. So I [7:22] think you know everything that both Nate and Tally  just shared is you know spot on. you know, cost, [7:28] revenue. But I think what's different about AI and  when you're evaluating it is you need to go back [7:34] and look at the workflow because if you try to use  necessarily the same KPIs on uh the old process, [7:42] then you know um as you work and evolve through  the workflow, you know, it might make sense to [7:50] have a different KPI. So, you know, the baseline's  the same, the owner is the same, but when you [7:56] concentrate on the business outcome, then you  can decide what you want your KPI to be. Who do [8:02] you get the most valuable information from when  you're developing those and I'm going to compare [8:08] your I'm going to start with Tally so you have  a chance to think about it. Where do you go to [8:14] get that information as you're approaching as our  previous speaker told you these very new horizons [8:19] we couldn't compare. we don't have, you know, very  much year-over-year. At least it's not necessarily [8:25] relevant as things change so significantly.  So, how do you develop your KPIs in this new [8:29] environment where every quarter is aggressively  different than the previous one? It's a it's a [8:35] great question. I I think it depends, right? It  depends on what you want to do, the risk level [8:42] that you're taking with what you want to do. It's  it's very different if you're developing something [8:48] um in house uh for productivity purposes. You  can um you can play around with it and hold the [8:56] accuracy bar, you know, pretty low, but the point  in time that you're developing something that is [9:01] production grade. You really want to position  the company for success and you're a steward of [9:07] this company. And from a governance perspective,  um, you're really making sure that it's not just, [9:14] you know, the the classic digital clickth through  rate and whether you know that person clicked [9:19] the thing that yielded the sales thing, right?  Ultimately, are you getting the value? Ultimately, [9:26] are they proceeding with the steps? something  that from a workflow perspective at the end of [9:31] the day doesn't um doesn't have utilization, even  if it's accurate, even if it's consistent, right? [9:40] Doesn't deliver a lot of value. And so often times  it's in the nuances of these things. If you're [9:46] going after creating a better experience, are you  going to calculate the sales? Well, maybe. Okay, [9:54] you want to achieve a sales lift, but you want  to get into the nuances of the conversation. [10:00] And you might even be better off having a lagging  indicator around whether you were able to retain [10:06] that client more so than in that moment were you  able to align an additional dollar and that's the [10:14] that's the emphasis right when you have that  crosscut between AI which is you know by all [10:21] standards right that's a key differentiator  it doesn't give you the deterministic sort of [10:26] element it gives you the recommend recommendations  it takes you to the next level. If you're able to [10:31] optimize the combination of the two of this,  it's very differentiated. And because we're in [10:38] such a a humanbased business where we have 10,000  agents across the US interfacing with millions [10:46] of clients, it becomes very very important.  Thank you. And I came up with that question [10:51] after Adam's previous answer. So I'm I'm eager to  hear where do you collect the information for new [10:59] or validating current KPIs. So you know we're  always reaching out to you know our frontline [11:05] associates because you know the solution that I'm  kind of thinking of in general is you know an AI [11:10] solution we give to and empower our associates  and just as said those lagging indicators. So [11:17] you know retail is a very high turnover industry.  So as we, you know, implement these solutions, [11:23] do we see our retention rate go up and our  turnover go down? Do we see our customer service [11:28] metrics at stores suddenly start to increase? And  you can see a correlation between the stores that [11:35] have a high active adoption rate of our platform  and the stores that still are not bought in, [11:41] the stores that are bought in that are seeing the  increased revenue, better customer service stores, [11:46] better retention. So, it's kind of like some of  those lagging indicators that you pointed out. So, [11:52] I'm going to throw you a new question not on  this list, Adam. Uh, oh, because you mentioned [11:56] something that's really a a key thing for me I'm  always talking about as a professor of management, [12:00] which is turnover, employee turnover. So, when  you talk about retention, you're probably talking [12:05] customers and employees. But you know what do  you see in the employee segment for turnover and [12:14] employee satisfaction employer of choice those  measurements and KPIs once from pre-adoption to [12:19] post adoption so for the stores that adapt quickly  how do those affect the employees yeah so I think [12:24] the employees are more engaged um you know that's  not typically a metric when you know someone with [12:30] a technology background I didn't think that that  was going to you know uh boost the employee morale [12:36] but people feel empowered when they have the tools  to be able to do their jobs. And a lot of time in [12:41] home improvement, you know, I know nothing about  home improvement. I barely can, you know, fix a [12:46] broken toilet. And a lot of times when we're going  out in the workforce, we have to get these young, [12:51] untrained employees. And that used to be a real  challenge because we'd always want, you know, the [12:56] industry professional, the retired plumber. But  now because of this AI tool, we can have people [13:01] that are empowered and they get excited because  they can do their job and help their customer. I [13:07] love that answer. Thank you. Now, this one is a  fast one. Think of and you can all do this too. [13:13] Well, we will have a Q&A at the end. Uh so you'll  be able to participate and actually note to the [13:18] guys on the side. Hey audio visual guys, this says  six minutes, but aren't we a lot longer than that? [13:28] Anyway, just check on that. So, the  three of you and everybody in the room, [13:32] what's your most useless KPI that you seem to  have to keep track just because it's, you know, [13:38] maybe grandfathered into the reports, but it no  longer matters. And I'm going to start with you, [13:42] Nate. Yeah, I mean at the end of the day and it's  same thing within our company, same thing with [13:48] our customers, but anytime that somebody is happy  with deployed numbers, not outcome numbers being [13:54] the key indicator of success, it always means  that we're not digging in far enough. They're [13:58] not holding themselves accountable to something  that is actually going to be meaningful for the [14:04] business. And so that's why we always, you know,  try to push back, dig deeper. anytime it's like [14:10] what's what what's the success of this roll out  it's like well we want to make sure it's used I [14:15] was like okay so it's used and so the moment  it's used well what does that mean what's the [14:20] quality indicators coming out of that in order to  ensure that the right outcomes are being driven [14:25] and it's all tying back to how you're doing your  business better and how you're doing it faster [14:29] and how you're driving the right outcome that  you're going to bring back to your business and [14:34] that outcome connection of the outcome to the uh  to the KPI Hi to the activity that fed it. I mean, [14:41] that is a beautiful triangle. In education, we  use that all the time. So, I love I love you're [14:46] talking my language, Nate. Um, all right, Tally.  What's the most useless KPI you're still forced to [14:52] ne to follow? Yeah, I I'm going to dovetail with  what you shared, Nate, because I have to say the [15:00] things that we thought were useful six months  ago might not be so useful now, right? And I [15:04] think these things evolve and I think that's okay.  um in the journey of many companies these days as [15:11] they're onboarding people into anthropic claw you  know you know open all these sort of capabilities [15:19] right in the beginning you take people on a  journey and you start start to use it start to [15:25] get a feel of it and uh start to have it be part  of and and magic ends up happening right because [15:31] then they're able to deploy their their mind into  other things and and you start to see that um but [15:38] but then that's the beginning of that right at  the end of the day right you start to shift your [15:45] questions profoundly and the questions are around  how do you take the innovation the beauty that has [15:52] been created and create the next level of scaled  products products at the end of the day are going [15:59] to yield tangible value for clients for agents for  all employees at large you start to ask questions [16:06] Like if a knowledge management capability has been  created in the company, maybe like 10 of them, [16:13] right? Should there be 10 knowledge management  capabilities? And you start to measure, right, [16:19] whether you're building up capabilities much more  at an enterprise scale, much more at a production [16:26] scale. So that shift from adoption and and  activity and how many models have been created now [16:34] starts to shift towards those outcomes, the value  proposition, the products, the solutions that are [16:41] going to create a competitive advantage to really  enable um a better experience for our agents [16:49] and clients is what it comes down to. That's  wonderful. And so, Adam, you're primarily in [16:55] the retail uh organization uh the retail industry.  Tell me, what do you think is the most worthless [17:02] KPI that we're people are still tracking? Well,  I'll say this because uh I saw the CEO of the [17:07] solution provider we use in the audience and  it's comes from his CTO and that is, you know, [17:14] total prompts. You know, we're playing a flat  rate. So, you know, the number of prompts really [17:20] doesn't matter to me. you know, uh, 10 prompts  and I got nine bad answers. You know, that doesn't [17:26] make a difference to me. So, I like to stay  focused on the outcomes. I understand technically [17:31] why he's tracking that. Uh, but, you know, I know  a lot of larger organizations will, you know, [17:37] track prompts, you know, registrations, things  like that. But, uh, for that for us, that's not [17:42] an important stat. And I don't generally do this,  but I'm going to add one that I if we had a fourth [17:47] panelist would ask them the question, but earlier  in a panel this morning, um, our workflow changing [17:55] workflow change agent panel, they were talking  about how AI can take away 80% of the tasks that [18:03] are easy and quick and that leaves you that 20%  can now expand. So, one of the things that I think [18:09] about KPIs that are worthless are, for example,  wait times. When I was doing a call center, you [18:15] know, it's like you want to keep your weight times  down to like under 30 seconds. People do you find [18:20] people are willing to wait longer because they're  going to have a more valuable experience once they [18:25] do get their time. They're not going to be rushed,  things like that. or the amount of time even once [18:32] they get to an agent maybe tally when they get to  an agent they're they're fine having that call go [18:37] five minutes so now the KPI of do you get them off  the phone in 90 seconds doesn't matter anymore do [18:43] you see some of those things where we can put the  workflow changes into those KPIs yeah I I I think [18:49] that's so smart I think that's probably the most  challenging aspects that many companies have which [18:56] is you take all these capabilities and you say how  do How do you reimagine? How do you rethink that [19:02] process? How do you take that conversation and  have a much more valuable conversation, right? [19:08] If if that client is calling to make a payment,  right? How do you optimize? Well, have you talked [19:15] to your agent, right? How are you doing? Right?  And and really make that bridge and make that [19:21] experience something that is lasting. I think  that that's the moment to if we can measure the [19:28] lift and that sort of conversion from experience  A to experience B that's really the the value ad [19:36] proposition so so yes waiting for five minutes you  wait for the doctor for five minutes right that's [19:43] more than that point 30 minutes yeah why do you  wait it's a valuable it's a valuable experience [19:49] it's something that you get out of it that is not  um so transactional And I I think that you know we [19:56] talk a lot about AI. I also like to talk about  specifically intelligence and the buildout of [20:01] intelligence as a as a founding member of AI. Um,  I think that personalization aspect to know when [20:10] that customer service representative is getting  that phone call and to know what's going on um, [20:16] in Suz's world and she needs, you know, some  support and help that changes the conversation, [20:21] the tone, the relationship and the brand of  uh, of the company and really establishes [20:28] um, you as somebody that you want to get to know  over the long term. So, I think it's critically [20:32] important. Excellent. It reminds me of a customer  story that we have where you know we had a like [20:38] a it was travel booking company. They have agents  that are meeting with customers. You have you know [20:46] typically you know for any agent that's talking  live you have like a single pane that's on your [20:51] uh computer anytime that you're going through a  process. And obviously it's typically a relatively [20:56] high turnover um industry or you know career  path in which then you're always going to have a [21:02] breakdown between you know half your people being  very tenured other half always on boarding or [21:07] relatively new and so how you support people live  in the flow of work is always a challenge and so [21:13] does that work okay and so that's where like they  basically had AI tool it was outside the core of [21:23] work we bring it into the core of work and or into  the flow of work so that they can interact with it [21:27] like during the conversation and so obviously it's  going to you know lead to a better outcome because [21:33] obviously the end user or the people on the phone  they're going to be meeting with somebody who [21:37] isn't coming off as new they're going to have a  better experience with that person but then also [21:41] hey it also increase their bookings by 15%. And so  it you know it has a you know better user exper or [21:48] better customer experience and it typically will  lead to the right outcome on the top line as well. [21:53] That sounds great and and that customer experience  is definitely one of those non-financial outcomes [21:58] that we still need to control for and  try to encourage. Um one of the questions [22:03] uh we were talking about in July that uh we're  all pretty excited about is two parts really. [22:09] How do you control for the risk, right?  The financial and non-financial risk, [22:14] but additionally, oh, and you all are going  to gasp. How do you control for scope creep, [22:21] right? And we're all like, we want to do all these  things. It's like, you get this contract for this [22:26] and then it's like, but I want, you know, ketchup  and mustard and pickles and horseradish and blah [22:30] blah blah blah blah blah scope creep galore.  How do you control for that? And for this, [22:35] I'm going to go for Nate and then Adam because  you're on the, you know, outside contract side. So [22:40] you guys are on the different side of the table  on this question. Yeah. I mean, scope creep, [22:44] I mean, it's common, you know, obviously I I feel  it. Every Monday I get back from the weekend and [22:49] I want to do a billion things. Um, and so I  don't know why this thing's challenging. Um, [22:56] and so I I struggle with that as well. And then  I always have to rein myself in. But I have the [22:59] same thing with customers. They, you know, they  sign on and they want to, you know, go a million [23:03] miles an hour. or they want to solve everything  and boil the ocean. But at the end of the day, [23:07] you know, if you're going to do things the right  way, you start with that one workflow, right? You [23:10] start with one. Look at the data, talk to your  end users, understand like what are they doing [23:15] on a daily basis that, you know, they either,  you know, don't know how to do it. Ideally goes [23:20] faster. You talk to your operations teams like for  some reason they always get this one field wrong. [23:25] uh and because they don't know the dependency  of the codes that live in this other system and [23:29] lives in this Google doc and blah blah blah blah  and it's too complex for people to keep up with [23:33] especially if the average uh process you have to  span 2.88 88 applications. It's too much context [23:40] switching for to to accomplish. And so when things  are that complex with the today's enterprise [23:45] architecture, you have to start with one.  Figure out, you know, throughout this process, [23:50] how do you accomplish it? What are the pitfalls?  How fast are they going through it? How slow are [23:55] they going through it? Are there certain fields  they don't fill out correctly consistently? What [23:59] are the downstream impacts of that? And it's going  to have several different KPIs that all tie back [24:05] to either, you know, probably saving you cost. You  know, they're probably situations particularly in [24:10] your line of work where it could be compliance  related kind of issues if not done correctly [24:15] or certain outcomes that will lead to people just  working faster and being able to cover, you know, [24:20] more client calls or whatever it might be,  right? But it starts with one, do it really well, [24:25] set a way of working and then go to the next  one and then slowly build that up over time. [24:30] And Adam, you're on the opposite side of that.  You know the customers need a lot of things. You [24:35] actually know they need that creeping scope. So  how do you handle that from your side? Yes. So [24:42] you know uh us being a small organization, we're  not developing anything inhouse. We don't have an [24:47] AI team. So when we look at solutions, you know,  we kind of uh you know, started with the frontline [24:54] employees. So you know we wanted to empower our  employees to be able to answer customers questions [24:59] and then the next step was okay how can we use  AI to bring intelligence to our back office. [25:05] So that's where we look like things at pricing  optimization merchandising things and I do not [25:11] have to worry about you know what model am I going  to use for this what are my token rates going to [25:17] be you know you need different tools for different  jobs and different vehicles for different tasks. [25:22] So, you know, if you want to go real fast, you're  going to get in a sports car. If you want to get [25:28] some wood and pick that up, you're going to need  a pickup truck. Uh, so that's where I kind of hand [25:34] that off to an AI provider where they're I'm  telling them what our business challenges are, [25:39] what my business outcome wants to be, and then  they're kind of building the platform for me. Wow, [25:47] that's pretty cool. Let me let me just ask you and  it like you guys as you were talking it reminded [25:54] me of something that's really foundational but  sometimes we forget in this AI dawn which is [26:00] the fundamentals of product management. Um I  think that one of the things that we have done [26:09] as many companies are doing is they're racing  they're racing to innovate and I I think the [26:15] opportunity is in front of them and and it's  uh it's luxurious and it's tempting and I think [26:22] the the many companies as they're doing this are  finding out that some of the fundamentals that are [26:27] tied to product management which is identifying  the outcomes identifying the scope and really [26:32] being clear around that uh needs to be done from  the get-go. Equally I believe that we're in a dawn [26:42] where many people as they're deploying are also  learning and uh we need to take those learnings [26:49] and we need to give room for people to apply their  learnings and and evolve their points of views [26:56] around maybe it's not so much the the scope but  the opportunity and we need to have an open point [27:04] of view around how do we revisit that particularly  in in today's age where um some of these [27:09] implementations are running not in the course of a  three-year road map. They're running in the course [27:15] of a threemonth road map, right? Making sure  that there's stop gates, there's pivot points, [27:22] and we give people the opportunity to voice that  over is equally important. That's excellent. Thank [27:28] you. All right, Adam, here is a big one for you,  but you are the right guy for this one. what are [27:36] the KPIs that turn up as the critical later in  the project and so sort of walk us through the [27:43] stages initial development the technical KPIs  the production phase just for everybody that's [27:51] starting this and and going down the road tell  them what the road looks like yeah so I think you [27:56] know in the beginning you're kind of getting into  you know the technical details different things [28:02] like you you know the latency the accuracy See,  you know, those are more measures of the platform [28:08] itself and the toll. Um, you know, as you evolve,  you're going to be more on the business outcomes [28:14] and how it actually, you know, works in the field.  So, for us, it's, you know, like I said, active [28:20] adoption at the store rate. So, when we see that  the users at the store are engaged, we see things [28:26] like that, our customer service scores go up, the  conversion rate gets better, um, and like I said, [28:32] retention. So like I believe in the beginning  you need to start with those technical focus on [28:38] making sure that the platform and solutions right  and then from there I kind of pivot towards more [28:45] um looking at the business metrics. That sounds  great. And Nate, this relates to your pick one [28:50] and go first. What uh what would you pick first?  How do you figure that out? [laughter] Yeah, I [28:57] mean it always I I mean I pick on you know what's  typically going to drive best possible outcome [29:04] uh or not the best but the biggest possible  outcome. Uh I mean there's some things that [29:10] obviously just you know drive a lot of people you  know insane within an organization the way it's [29:14] not done correctly. It has a lot of ripple effects  that obviously can you know cause a lot of harm. [29:20] And so obviously that will always kind of break  out between what is truly urgent between what is [29:26] uh more of a nice to have. Um but then obviously  you know depending on the state of the business [29:31] there's also you know different priorities based  on you know you could have come out of a year [29:35] where you know a financial institution had a bad  compliance year and so they're just doubling down [29:40] on that. Yeah. You know I don't what was the you  know the presentation before I think Mastercard [29:45] 10 billion fraud or you know saved or something  like that. I mean that's a big deal. It's a lot [29:49] of money. So obviously they're probably a little  bit less focused on, you know, other parts of the [29:53] business in that kind of situation. So anyway, uh  we try to obviously analyze it. It's important to [29:58] obviously understand like the entire workflow and  all the different ways that you can impact the [30:03] business in a positive way and try to remember  those things and try to benchmark it and then, [30:08] you know, continue to calculate from there and  make sure that you're improving over time and [30:12] don't just go live and come back later and and  and forget about it. benchmark it, understand it, [30:18] and then obviously if you accomplish it, great.  Reiterate, figure out how you can continue to do [30:22] it better. Cool. So, you're talking a lot about  functional area impact and business unit impact [30:30] and that kind of thing, but at New York Life,  you know, 10,000 people out there running their [30:35] own small business and your corporate umbrella  objectives might be very different than the guy at [30:46] the desk on Main Street. And how do you go about  rolling this out? Something that's important to [30:55] you, but maybe he feels or she feels like it's a  a barrier to their workday. How do you handle that [31:03] so that it's rewarding and win-win for everybody  involved? Yeah. Well, it's a great question. We [31:09] we really work hard to try to converge the two  together. Um I think where it becomes challenging [31:16] often times is when you have 10,000 agents, right?  Um they have somewhat different points of views on [31:25] on what good is. And so we um because as you can  foresee from an advisory perspective some of our [31:35] agents are very much um doing a full end to end  holistic advice and guidance a set of capabilities [31:43] uh other agents are just you know providing one of  the products right and that's okay but we need to [31:50] cater to the variety of agents that we have in the  market I think particularly as we go forward with [31:57] some the AI capabilities in the regulatory space,  it becomes um it becomes challenging. Um but it's [32:04] also from a personalization perspective, we need  to be very mindful that what is the solution for [32:10] one might not be applicable to another and that's  where we need to dovetail and be listening to [32:18] our customers continuously in the solutions that  we're providing. I I do think that uh with AI we [32:25] have great opportunities to do that. Um and it's a  major focus. Excellent. So one of the things that [32:35] used to be a much longer scope and it's definitely  not is we we threw all this money into the system, [32:42] right? And so we've we've fully developed it.  We kept our scope under control. Um everybody's [32:48] working together. So um how do you go about  measuring? Did it do what it was supposed to [32:55] do at the cost planned? So tell me a little  bit about how that's turning. Does it tend to [33:01] go over under? How is it working out? And from  both sides for you and to your chosen vendors, [33:08] like is everybody coming out sort of in the right  place? What kind of adjustments do you feel like [33:13] you have to make? Do you have monthly or how do  you handle that? because we know that overspending [33:18] and getting that bill at the end of 12 months at a  full implementmentation can be a problem to us. Um [33:27] I have to say that one of the things that we have  done in our implementations is we've built in a [33:34] lot of stop gates and kill switches along the way.  Um because at the end of the day uh we look at [33:42] this as an investment right the company's making  an investment in creating great client and agent [33:47] experiences in optimizing efficiencies and are we  getting our values worth? Is this proceeding in in [33:56] the way that we thought that it would be? And so  building those stop gates to make sure that from [34:02] a dollars and cents perspective, you're making  those smart investments uh becomes really critical [34:09] to enable you to um have those pivot points that  then uh reshuffles the card. But that means that [34:18] when you create something like this, you have to  create a very good uh portfolio of capabilities [34:25] to say okay if not this then something else  because people become very attached to their [34:31] bodies of work right they invest in this they have  put their ideas oftent times uh their engineering [34:38] skills their product skills all this in order to  make a great outcome happen and it's difficult [34:44] for people to say Hey, this isn't working out.  Then let's pivot to something else, right? And [34:51] we have to make sure that it's legitimate. And  not only is it okay, it's actually celebrated. [34:57] And that's something from a culture perspective  that we really focus very intently on to make sure [35:03] that we are celebrating equally uh the failures  with the with the wins. Um, and it's sometimes not [35:12] intuitive. And I love the kill switch idea. I mean  that's you know especially in financial services [35:17] it's just it's data data data daily weekly monthly  and quarterly and and catching it early kill [35:23] switches like that's that's a phrase that I love.  Thank you Adam. What uh what do you see when they [35:30] you know relative to the full implementation and  did we get what we planned for and how the cost [35:36] comes out at the end. Yeah. So, you know, a couple  things that I'm thinking of as I sit here and one [35:42] of the things I can't get out of my head. I think  this is the first panel when someone is using the [35:46] word agents and they're actually talking about  humans. So, when you said you had 2,000 agents, [35:51] I'm like, "Oh my god, 10,000 AI agents." Wow, you  guys are really ahead over there. Ask me how many [35:58] agent like AI agents. AI agents. Uh, many. How  many do you have? Many. Many. Okay, that's a [36:07] good answer. Um but no it's great to see that you  know humans are still getting uh jobs and you know [36:13] uh you know they have those tools and empowerment  but uh you the mistake that I've made in the past [36:19] is thinking that the implementation uh was the  the goal line was the end and continuing to get [36:27] that continuous feedback and then going uh back  and revisiting the processes the workflows and [36:34] uh for us you know as we've gone more efficient  with some of the AI use cases you know we've seen [36:40] well here's where we can go next and you know that  usually comes at a cost but uh you plan for that [36:47] but uh the whole kind of journey has been you know  how is our business changed how are our customers [36:54] more happy uh so it just continues to evolve and  Nate before we go to our closing question did you [37:01] have some different experiences from these two  yeah I mean I just you know just being obviously [37:06] in you know a little bit of a different line of  work is I think at the end of the day shouldn't [37:13] be afraid to start small right and you have  obviously you know cost shouldn't be a surprise [37:19] if you if you obviously developed it you designed  it you accounted for you know full adoption and [37:26] utilization in the right way in order to drive  the right outcome you should know what that's [37:30] going to cost but like you should be starting with  obviously a small group of people you shouldn't [37:36] design something that is transformative and then  just throw it out there and just be like it's [37:41] going to be great. Turn around and walk away. It's  going to be something that you ideally, you know, [37:46] give it out to a couple people and you send it  out to somebody who's an expert, somebody who's, [37:51] you know, you're AI expert. Everybody has one.  Um, you're going to send it out to somebody who's [37:55] brand new. You're going to send it out to somebody  who's an underperformer. I don't know. like then [37:59] you're going to get a whole host of feedback and  different ways of of analyzing what's going on so [38:05] that you can start to get a little bit of sense  well is this metric starting to move in the right [38:09] way and this metric's doing going the right way  what's the feedback look like but obvious then [38:12] then what's the cost and is it you know obviously  panning out in the way that you want and then [38:17] obviously then you go ahead and release it to  10,000 people whatever it might be so I love this [38:21] we have the kill switch we have the know that it's  not the end of the road at full implementation [38:28] which is really key and yours is really about  pivot points. You know, start so you can pivot and [38:33] I I you know, I always use the aircraft carrier  versus jet ski and you're saying stick in the [38:42] jet ski until you graduate to the fishing boat and  then don't just jump right to the aircraft carrier [38:48] and you're steering an aircraft carrier that's  carrying 10,000 jet skis. So, I really I get that. [38:54] So, one of the things that I always think is super  important uh in these uh conferences is I want you [39:02] to share with the audience that thing that you  want to stay in their head when they're eating [39:08] their dinner tonight and going home and they're  going to say that ROI panel when that person said [39:13] that thing. So, after they tell you, I'm going to  open it up for questions. Uh I don't know who's [39:20] running the microphone around. Do we have someone  running the microphone around for the questions? [39:24] Maybe Emily, can you do that for me? Yeah, that'd  be great. Um, so what's that thing that you want [39:31] to stick in their head? And I'm going to start  with Taylor. Um, we haven't talked about one thing [39:38] uh which is a word that um or a set of words that  we talk about a lot, which is change management. [39:45] uh with all these capabilities that we're putting  out to the market and it's great. Everybody has [39:52] now more of an ability to be an innovator um a  developer, right? Um give themselves and others [40:00] these set of capabilities. How that manifests  in a broad organization? If you don't have the [40:07] communications uh in place, if you don't have the  training in place, if you don't have that support [40:14] system in place, all of your efforts are going to  go by the way wayside. And so I look at this also [40:21] as one of the we talked about product management  discipline and we talked about in in the [40:27] um to the extent that you're identifying your core  sort of scope. I look at your change management [40:34] implementation as also critically important for  success. I love that. Keep in mind it is a change [40:41] management culture organizational everything.  Nate, what's your what's your mantra you want [40:46] in their mind? Yeah, I mean at the end of the  day, I mean obviously for me I always come [40:51] back to the adoption layer and taking big swings.  And obviously in order to design a great adoption [40:58] solution, you got to dig really deep into the why  behind everything that you're doing. You got to [41:03] understand every single step of the process. You  got to understand how people feel in the process, [41:06] where they get confused in the process, where  they just do it wrong, what it changes context, [41:10] and how are you approaching them in that flow  of work in order to ensure that they're the [41:14] most successful that they can possibly be. And  then when you come out of that, you're going to [41:18] understand a lot more KPIs and a lot more ways  that you're impacting the business so that the [41:22] solution that you're putting out to your, you  know, boss or your CFO eventually. That's it. [41:28] That that's going to be a lot of different KPIs,  a lot of different outcomes that you're going [41:33] to be able to drive because you'll have done the  homework. You'll have designed the right solution. [41:37] You'll have tested it with the right people. And  there's, you know, obviously so much potential [41:42] every single week. There's something else that  we can do. So okay now Adam yeah so I what I was [41:50] thinking about what both of them said you know  I think change management people are resistant [41:54] to change management and then Nate has continued  to talk about continuing to tweak and improve the [42:00] processes and get that continual feedback and I  like the analogy you know when they h they used [42:06] to have a steam engine running you know factories  and when they have first invented the electric [42:13] uh engine They put it in the place of the steam  engine with all those different pulley systems [42:18] and yes it made things quicker but it didn't  necessarily fundamentally change it like the [42:23] engine later could. So when they start placing  the smaller engines around the factory they kind [42:28] of redid the whole way the factory was designed.  So I think with AI gives us a unique opportunity [42:34] to kind of you know uh not try to avoid change  management and keep the process the same but then [42:41] not realize the potential you have you know in  your hands with that new engine that is AI. I love [42:49] that. That's a really good one. All right we have  uh just a couple minutes left. Quick questions. [42:54] Anybody have a quick question for the crowd here?  I mean you can corner them in the hallway too. We [42:59] have a question way back here. Sorry, Emily.  Run, run, run. Emily was the co-chair with me [43:06] of the Monday academic conference. I know she's  up just sprinting across the stage. How do you [43:13] measure KPIs for people who are using say your  your chat bots as a thinking partner compared to [43:20] someone who's building something that you know  they can show you that it does something? Oh, [43:26] who wants that one? That's one. Nate. [laughter]  Yeah. I mean, I always break it down into I mean, [43:32] a couple different categories. So, when you take  it as a thinking bot, I think of that as like [43:37] almost like the bottomup solution where you're,  you know, providing almost AI as a service. I [43:42] almost think of it as almost like just a part of  like a benefits package for people nowadays of [43:46] being able to use AI in the way that you need  it. Not based on like some predeterministic [43:51] uh development outcome that somebody else uh  developed, but it's something that you can use [43:56] in your day-to-day in which it's like at the  end of the day if people have the tools that [43:59] they need to be successful, you shouldn't see  that reflected in the satisfaction of of of [44:04] their job and obviously the performance of the  individual to be successful in their own right. [44:10] But then it comes back around to the more of like  the topdown solutions, the more like predetermined [44:15] agentic solutions that are, you know, driving and  that's where I'm always looking at like what are [44:20] those things that are not so much optional  but are naturally part of the flow of work [44:25] because that's what's going to really drive like  a consistent KPI metric that is typically going [44:31] to be what's going to impact the business in a  different way. And so it's it's it's hard to, [44:36] you know, answer in the most clear way, but  that's in general how I think about it. And [44:42] I think he also said he'll be in the hallway  after this. Um, all right. So, we're out of time, [44:47] and I just really want to thank all three of you.  You've been amazing. Thank you so much. [applause]