Fintech Executives Shift AI Agents From Pilots to Core Revenue Engines

金融科技高管推动AI Agent从试点转向核心营收引擎 企业唯有摒弃“概念验证观光”，并在严格的数据治理下将多Agent流水线直接整合至核心创收业务流程，AI部署方能取得成功。 企业与金融科技领袖探讨了信贷、零售银行、咨询及数字商务领域从投机性的“AI观光”向生产级Agent工作流的转变。他们剖析了未经治理的数据孤岛和缺乏基准衡量策略等关键失效点，并将其与严谨的产品主导型部署模式进行了对比。关键技术与运营解决方案包括多Agent仿真环境、跨职能部署小组以及Mastercard Agent Pay等交易协议。讨论最后涉及提示工程的应用、员工变革管理，以及自主自动化与必要的人文同理心之间的关键平衡。 德勤研究显示仅约 30% 的企业 AI 概念验证（PoC）最终进入生产环境，主要受阻于数据孤岛、度量策略缺失以及可选的人机协作模式；同时，60% 至 80% 的严肃企业投入正聚焦于直接创收的核心业务流程重构。 (3:01) 在中小企业（SMB）借贷领域，程序化语音与对话智能体大幅降低了人工跟进与审核的可变成本，使金融机构能够以标准高触达体验盈利性覆盖长尾小额借贷群体。 (7:22) 万事达卡推出 Agent Pay 协议，致力于将生成式 AI 交互的非确定性与金融支付清算的确定性机制连接起来，在用户自定义消费限额与银行级安全标准下实现智能体自主交易。 (11:11) 为防止 Prompt 或模型微调引发意外功能退化，企业部署越来越依赖前置自动化仿真套件，通过对抗性测试智能体在上线前针对历史边缘用例进行全面回归测试。 (18:25)

Deploying production-grade enterprise AI agents requires moving past isolated proofs-of-concept into redesigning core business workflows, backed by rigorous data readiness, specialized testing simulations, and disciplined change management. Escaping Proof-of-Concept Fatigue and Operational Blockers (3:01) Enterprises are experiencing fatigue with pilot projects, with Deloitte benchmarking indicating that only about 30% of proofs-of-concept (PoCs) ever reach production. While software development and customer service scripts show initial wins, 60% to 80% of serious enterprise effort focuses on core revenue-generating operations. For example, wealth management firms use agents to parse client calls, log life events into CRMs, and generate follow-up financial recommendations, while contact centers use agents to consolidate advisor workflows and reduce internal transfers. Deployments fail primarily due to fragmented data repositories, a lack of baseline productivity metrics, and voluntary adoption. In human-in-the-loop environments where using agent assistance is optional, employees often default to manual research, dropping projected return on investment to zero. To navigate unstructured or poor internal documentation, engineering teams can bypass flawed static knowledge bases by transcribing high-performing historical customer calls to generate accurate resolution paths. Workflow Re-engineering and Multi-Agent Orchestration (7:22) Autonomous agents enable organizations to transform sequential manual processes into automated pipelines. In small-to-medium business lending, Lendflow CEO Jon Fry explains that voice agents handle front-line document collection, applicant questions, and scheduling, reducing funding turnaround times. This shift removes variable human labor costs, allowing lenders to profitably service lower-margin loan applicants that were historically abandoned due to poor unit economics. Rather than relying on a single monolithic model to manage an entire workflow, advanced implementations deploy an orchestration layer across specialized, sequential agents. In financial workflows, distinct agents independently collect documentation, categorize and extract structured records, enrich files with external data, and perform underwriting calculations. Passing structured context between modular components improves accuracy and simplifies maintenance. Mitigating Risks with Pre-Production Simulations and Governance (18:25) Because minor prompt or model adjustments can unintentionally break agent behavior across edge cases, production rollouts demand rigorous testing frameworks. Lendflow runs pre-production multi-agent simulation suites where automated agents conduct adversarial tests and replay historical edge cases against target bots. Following deployment, secondary telemetry agents monitor live interactions for anomalies, routing failures to human quality assurance teams whose corrections feed back into testing suites. Operationalizing agents in regulated environments also requires tailored governance architectures. Chime evaluates agent candidate projects through four filters: member value, process scale and repetition, acceptable risk appetite with defined human handoffs, and data grounding. To tackle consumer commerce within conversational interfaces, Mastercard engineered its Agent Pay protocol, which introduces verification checks and user-defined constraints—such as spending caps and approved merchant categories—to bridge non-deterministic model outputs with deterministic payment clearing networks. Culture, Prompt Adoption, and Measuring Financial Impact (22:38) Successful agent rollouts require organizational change management to run in parallel with technical development. Simply providing staff with broad access to chatbots yields limited financial return because many employees struggle to write effective prompts. To overcome this hurdle, Mastercard’s Federico Cohen Freue uses generative models to draft and refine structured prompts for other tools, while Deloitte's Jatin Dave emphasizes showcasing internal power users to demonstrate workflow-specific applications directly to peers. Assessing financial return requires metrics that extend beyond vanity indicators like monthly active users. Organizations must evaluate how extensively workers edit agent-generated text, measure the true delta in task completion time, and track downstream performance differences, such as outreach win rates. Because isolating AI attribution from macroeconomic conditions and marketing promotions becomes difficult quarters after launch, measurement mechanisms must be embedded into the initial system architecture. While routine administrative tasks benefit from rapid automation, enterprise leaders emphasize maintaining human oversight in emotionally charged or high-touch scenarios, such as sensitive customer escalations, where automated agents cannot replicate human empathy and judgment.

来自Sound Ventures、Deloitte、Chime、Mastercard及Lendflow的高管指出，企业正在告别概念验证阶段的“AI观光”，转向将AI智能体融入核心创收与信贷等生产流程；然而，智能体的规模化落地依然受制于数据质量孤岛、衡量指标缺失以及变革管理等瓶颈。 告别概念验证：从边缘试验转向核心业务流程 (3:01) Deloitte的Jatin Dave表示，当前企业普遍存在PoC倦怠，根据Deloitte的基准研究，仅有约30%的企业PoC能最终进入生产环境。尽管IT软件开发与代码重构是公认见效最快的领域，但目前60%至80%的严肃企业投入正聚焦于重塑核心创收流程。例如财富管理机构正利用智能体分析顾问与客户的对话，自动捕获家庭生命周期事件并写入CRM，进而触发具体的理财方案建议；后台中心藉此将坐席所需技能缩减至三分之一，显著减少跨部门转接。 Dave总结了智能体无法进入生产的三大阻碍：碎片化与未整理的数据孤岛（其阻碍程度远超信息安全审查）、缺少基线生产力与商业结果的度量体系，以及人机协作机制下的非强制使用。如果底层数据不互通或员工选择忽略智能体推荐而继续手动操作，项目的实际投资回报率就会归零。 重塑信贷与商业闭环：优化单位经济学并打通支付协议 (7:22) 在信贷领域，Lendflow首席执行官Jon Fry阐明了智能体如何改变中小企业（SMB）贷款的单位经济学。传统模式依赖人工跟进材料、审核报表与挽回流失线索，高昂的人力成本迫使贷款机构只筛选高利润的大额客户。通过将多步骤流程转化为程序化自动管道，智能体充当即时响应的第一道防线，引导借款人完成文件录入与初审，人工团队则退居二线负责质检与边缘案例。这不仅大幅缩短了放款周期并提升转化率，还让机构能以极低的边际成本覆盖长尾小额借款人群，同时在拒贷时提供解释与教育以积累客户信任。 在电商与消费领域，针对Pinterest首席执行官Bill Ready关于自主购物AI尚遥远的看法，Mastercard的Federico Cohen Freue指出，大量消费者已在利用ChatGPT或Claude等对话引擎调研复杂购物需求，但在购买环节因跳转至商家网站输入卡号而产生体验断层。Mastercard推出的Agent Pay旨在连通大模型的非确定性输出与金融清算网络的确定性标准，允许用户预先设定支出上限与商户类别等防护规则，并运用现有的代币化与安全标准，确保智能体代理交易具备等同于物理刷卡的安全与防欺诈保障。 金融级工程规范：模拟红队演练与全流程变革管理 (16:12) 在强监管与高复杂性业务中，不能将智能体视为孤立的技术黑客项目。Chime首席数据、AI与分析官Barkha Saxena强调，Chime采用跨职能敏捷团队模式，在编写代码前便将AI专家、数据工程师、风险官与政策团队汇集在一起，同步推进技术研发与组织变革培训，防止开发出的工具无人问津。Chime在筛选智能体场景时依托四大支柱：会员价值主张、规模与重复性、风险承受度（如密码重置完全自主，敏感交易保留人工），以及数据上下文质量。对于已上线智能体，工程团队会持续监测反馈并迭代模型。 针对智能体拥有一定自主权而可能在微调提示词时引发边缘退化的问题，Jon Fry介绍，Lendflow建立了“智能体对智能体”的多智能体预发布仿真测试环境，通过自动调用数百个历史场景进行对抗性测试与回归演练；上线后则通过实时监测智能体分析交互情绪并捕获失败点，由人工QA团队纠偏并将新案例反哺至测试集。Sound Ventures的Zach Gronfein也提到，市场正在涌现出如Witness AI（运行时权限控制）与Haze Labs（预发布红队测试）等第三方专业治理平台。 兼顾自下而上探索与自上而下量化考核 (22:38) 面对董事会的AI战略要求，企业需要平衡全员普惠化尝试与顶层战略价值。虽然企业可以向员工开放模型工具，但撰写高质量提示词仍是普及瓶颈。Cohen Freue分享其个人实用技巧是直接让AI为自己优化提示词，从而避开复杂的学习门槛；Dave则建议通过挖掘业务部门的“超级用户”向同行演示突破案例，而非单纯依赖自上而下的行政命令。自上而下的场景评估则需严格对照三项准则：用户是否真正需要（渴望度）、能否直接影响利润表（可行性），以及现有数据底座能否支撑（技术可行性）。 针对企业最为棘手的ROI衡量难题，Dave指出月活跃用户（MAU）属于虚荣指标，真正的业务度量必须深入到交互细节：例如跟踪员工对智能体生成草稿的修改保留比例、对比有无AI辅助的任务完成时间差，以及AI生成内容在最终转化率上的表现。由于企业CFO通常在部署四至五个季度后要求看到明确的财务回报，企业必须在立项初期便将细粒度的归因与评估指标植入产品设计中。 自动化的边界：保持人机分工与情感体验 (34:04) 尽管智能体在串联收集、提取、数据丰富与决策计算等多步骤编排方面前景广阔，但高管们一致认同自动化存在明确边界。Saxena结合其过往在Poshmark与McKinsey的经历指出，社交电商、时尚造型以及涉及强烈情感共鸣的敏感客服等高触感领域，依赖人类特有的品味、同理心与信任，无法被算法完全替代。 合理的未来分工应是将繁琐的行政事务交给自治软件，例如全自动整理会议要点并拟定跟进邮件，而将需要深层人际连接的高风险沟通留给人类。在推进具体试点时，Cohen Freue建议企业从易获取数据、低品牌与合规风险、高重复度以及员工极度厌恶的手动行政痛点切入，既能最小化落地阻力，又能为日后构建更复杂的编排系统积累治理经验。 当前企业部署AI智能体正从盲目跟风的概念验证转向追求确定性的组织与工程落地。然而，如何有效剥离宏观经济等外部变量以向财务部门精准证明智能体的直接ROI归因，以及如何在追求全自动化交易与规避幻觉合规风险之间维持动态平衡，仍是行业持续探索的核心课题。

## Background 该内容由行业峰会频道 Ai4 于 2025 年 8 月 26 日发布在 YouTube 平台。视频记录了一场由 Sound Ventures 投资人 Zach Gronfein 主持的圆桌研讨，汇聚了来自 Chime、万事达卡（Mastercard）、Lendflow 以及德勤（Deloitte）的企业高管与 AI 专家，旨在分享跨行业部署 AI Agent 创造降本增效与业务增长的真实落地案例与战略决策经验。 ## TL;DR 圆桌深入探讨了企业如何告别仅有 30% 转化率的“AI 观光”POC 阶段，将 AI Agent 真正部署至生产环境并实现商业价值闭环。嘉宾们结合金融信贷程序化、Agent Pay 消费支付、强监管环境下的护栏构建及多 Agent 编排层演进，分享了各自的实战路径。会议提出了“自下而上工具普惠”与“自上而下财务损益（P&L）对齐”的双轨推进策略，强调优先从数据易得、低风险、高重复且员工反感的高频痛点切入。最后，专家们详细阐述了 Agent-to-Agent 回归仿真测试、业务深度归因度量，以及依托超级用户与 Prompt 闭环优化推动组织级采纳的落地方法论。 ## Atmosphere & style 整场对谈务实、专业且极富实战锐气，嘉宾们毫不避讳当前企业 AI 落地中的数据缺陷、组织阻力与 CFO 归因质询等尖锐痛点，彻底摒弃了浮躁的技术炒作。四位来自不同领域的领袖与主持人配合默契，既有严密的工程架构与决策框架推演，又穿插着诸如“渴望 Agent 代回 Slack 消息”等切中职场真实负担的幽默互动。现场交流节奏紧凑、干货密集，在理性严谨的商业逻辑与对人类情感温度的敬畏之间达成了绝佳平衡，营造出极具启发性与实操指导价值的行业研讨氛围。 ## Key takeaways - [03:01](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=181s) 企业从“AI 观光”转向生产环境面临严峻瓶颈，目前仅约 30% 的 POC 项目能最终投产，核心阻碍在于底层数据匮乏不可靠、缺乏成体系的采用率评估机制以及人机协同中员工自愿使用的门槛。 - [03:01](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=181s) 高价值 AI Agent 落地正从被动记录纪要转向主动驱动核心业务行动（如财富管理场景中根据对话自动提示推荐 529 计划），占企业 60% 至 80% 投入的核心流程能产生最高财务回报。 - [07:22](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=442s) 信贷等传统繁琐业务通过 AI Agent 转变为端到端“程序化”流程，不仅消除了人工等待断点、大幅缩短放款周期，更能以极低边际成本为长尾客户提供高质量服务并沉淀品牌信任。 - [09:47](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=587s) AI 的投资回报率（ROI）不仅体现在直接账面数字上，通过高质量交互带来的客户信任感、品牌归属感与情绪价值等非财务收益，对提升客户留存和达成长期北极星指标同样至关重要。 - [11:09](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=669s) 消费级流量正向 AI Agent 转移，Mastercard Agent Pay 通过在生态关键节点设置防护栏消除大模型非确定性风险，打通从 AI 意图发现到代客安全支付的无缝交易闭环。 - [14:38](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=878s) 在金融强监管环境下，企业应将 Agent 视作完整产品来打造，立项伊始即由数据、算法、风控与合规团队协同制定护栏，并同步推进组织变革管理与技能培训。 - [18:25](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1105s) 打破单体式 IT 系统的二元交付定势，采取最小可行产品（MVP）敏捷迭代策略，逐步增加 Agent 自主性与复杂度，依托一线真实反馈持续调优以加速业务成果落地。 - [19:32](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1172s) 构建严谨的工程化发布与仿真回归测试体系，通过“以 Agent 测试 Agent”的对抗模拟跑通海量历史场景，并在生产监控中结合人工质检将异常案例滚雪球式沉淀为新测试集。 - [21:08](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1268s) 随着 Agent 自主操作能力成熟，周边安全工具链加速演进，包括运行时基于员工权限映射的数据访问治理（如 Witness AI）以及预发布红队对抗演练（如 Haze Labs）。 - [22:38](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1358s) 推进企业 AI 战略需采取双轨制：自下而上推进工具民主化与草根资产共享，自上而下基于期望度、财务可行性（P&L）与技术可行性严苛筛选核心价值场景。 - [25:03](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1503s) Chime 确立了以“用户价值、规模与重复性、风险承受度、数据底座质量”为核心的四维决策框架，并坚持投产后保留专职团队持续追踪反馈闭环而非一次性交付。 - [28:59](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1739s) 企业在寻找首批落地切入点时应聚焦“易获取数据、低风险、高重复、员工极度反感”的任务以消除阻力；面对数据缺陷时，可创新性地将客服通话录音等已有资产转录重构为结构化知识库。 - [32:00](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1920s) 业务流程自动化正在从单一任务 Agent 演进为多 Agent 协同体系，顶层编排层（Orchestrator）通过跨模型组合与实时上下文共享，将系统底层复杂度彻底从终端用户侧屏蔽。 - [34:02](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=2042s) AI 自动化存在明确边界，涉及高上下文、高接触度以及人类复杂情感共情的领域（如情绪困境中的客服或高个性化时尚消费决策），不应推行 100% 的全盘自动化。 - [38:09](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=2289s) 成效度量必须超越浅层的 MAU 统计，深入工作流追踪内容采纳率、改动幅度、单任务耗时缩短以及回复率和赢单率等业务指标，并在立项前置设计指标体系以应对财务归因质询。 - [40:09](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=2409s) 破解非技术员工的提示词工程瓶颈无需追求全员专家化，可从“茶水间吐槽”挖掘痛点，利用 LLM 进行元提示词生成，扶持“超级用户”带动组织扩散，并建立持续的仿真调优运维闭环。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=0s) 圆桌嘉宾介绍与企业级 AI 战略背景 会议主持人 Zach Gronfein 首先开场，对在漫长议程后依然留在现场的与会者表示感谢。他强调，本次圆桌邀请了四位从不同切入点在企业端和客户端落地 AI Agent（人工智能体）的顶尖从业者。Zach 首先介绍了自己及所在机构 Sound Ventures——这是一家由知名演员 Ashton Kutcher 和音乐经纪人 Guy Oseary 共同创立、管理规模超过 10 亿美元的风投基金，高度聚焦于 AI 应用层，不仅早期投资了 OpenAI 和 Anthropic 等底层基础模型公司，也深度布局了大量利用这些模型创造实际价值的应用层企业。 随后，各位圆桌嘉宾依次介绍了各自的业务版图与 AI 推进重点： * **Barkha Saxena**：金融科技公司 Chime 的首席数据、AI 及分析官（CDO）。Chime 是一家以使命为导向的 FinTech 机构，致力于为美国普通大众提供日常金融服务与进步支持。Barkha 领导着支持会员价值交付的整体数据与 AI 战略，目前团队已在风控（Risk）、市场营销（Marketing）、工程研发（Engineering）以及客户支持（Support）等多个核心业务环节构建并部署了 AI Agent。 * **Federico Cohen Freue**：万事达卡（Mastercard）AI 与数据运营高级副总裁。他的团队主要负责 Mastercard 的企业级 AI 与数据战略，重点探索如何面向内部员工规模化部署 AI 工具、如何推进企业 AI 转型旅程、对员工进行系统化的 AI 技能培训，以及执行多项核心战略规划。 * **Jon Fry**：Lendflow 首席执行官。Lendflow 专注于构建嵌入式借贷（Embedded Lending）与 AI 自动化基础设施。Jon 在中小微企业（SMB）借贷领域拥有长达 15 年的连续创业经验，曾创办并运营获客引流、客服与入驻呼叫中心搭建，以及 SMB 借贷审批决策引擎等多家企业，这些实战经验直接为 Lendflow 的产品演进提供了核心洞察。 * **Jatin Dave**：德勤（Deloitte）AI 业务董事总经理。德勤作为全球规模最大的专业服务机构之一，其 AI 与工程团队致力于协助全球企业客户梳理并制定顶层 AI 战略、开发定制化的场景解决方案与技术平台，并全力推动企业内部的实际应用与长期运维。 --- ### [03:01](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=181s) 告别“AI 观光”：从 POC 到生产环境的真实转化与落地阻碍 针对当前企业界普遍存在的“AI 观光（AI Tourism）”现象——即企业短期内因高客单价和强意愿采购大量功能重叠的工具试水，随后逐步整合淘汰——主持人 Zach 向 Jatin Dave 发问：在真正从概念验证（POC）跨越到生产环境的过程中，究竟哪些 AI Agent 场景能够展现出真正的业务粘性？ Jatin 坦言，“AI 观光”一词非常贴切，现场很多企业负责人甚至直言“我已经做够了 POC，再也不想做一个新的 POC 了”。德勤的研究数据显示，**目前仅有约 30% 的 POC 项目能够最终进入生产环境**。更为严峻的是，当 Jatin 现场询问台下有多少人真正建立了“使用采用率（Adoption）衡量机制”时，举手者寥寥无几。 ```mermaid flowchart TD accTitle: AI 从 POC 转化到核心业务落地的全流程与关键障碍 subgraph S1["概念验证与生产转化"] A["企业 POC 探索"] -->|"仅约 30% 转化率"| B["进入生产环境"] end subgraph S2["核心业务价值场景（占 60-80% 投入）"] B --> C["IT 编码与应用开发"] B --> D["客服与联络中心技能整合"] B --> E["财富管理：意图捕获与行动触发<br/>（如识别孙辈出生并自动推荐 529 计划）"] end subgraph S3["三大落地核心阻碍"] F["底层数据匮乏或不可靠"] G["缺乏采用率量化评估策略"] H["人机协同机制下员工采用纯属自愿"] end ``` 在具备高粘性的场景分布上，Jatin 指出： * **IT 与软件研发**：代码编写与应用开发是目前成效最显著、影响最深远的领域。 * **传统通用职能**：客户服务、联络中心、销售与市场营销等。 * **企业核心业务流程（占 60% 至 80% 的工作重心）**：这是产生最高回报的领域。例如在财富管理（Wealth Management）场景中，财务顾问（FA）是带来营收的核心角色。过去 AI 只是类似 Teams 上的 Copilot 帮忙整理会议纪要，而如今高粘性的 Agent 能够“驱动具体行动”——当监听到客户在通话中提到“家里即将迎来孙辈”时，Agent 会自动将孙辈添加到家庭关系图谱中，并在下一次通话前主动提示财务顾问向客户推荐“529 大学教育储蓄计划”。 * **呼叫中心技能整合**：某财富管理机构借助 AI Agent 将联络中心坐席所需的技能类别精简整合至原先的三分之一，坐席得以全能化，大幅减少通话转接，为财务顾问及其终端客户提供了更快速、高质量的服务。 谈及阻碍落地与粘性的绊脚石，Jatin 归纳了三大核心挑战： * **数据基础脆弱**：撇开信息安全（InfoSec）团队的合规限制不谈，企业最致命的瓶颈是“缺乏数据或数据极其不可靠”；没有可靠数据，AI Agent 在实际业务中基本毫无用处。 * **衡量策略缺失**：极少有企业真正建立了成体系的业务成效衡量机制。 * **应用推广（Change Management / Adoption）不足**：目前多数工具仍保留 Human-in-the-loop（人在回路），界面上仅提供一个 AI 生成辅助按钮；如果终端员工选择不点击、依然习惯自行手动检索，企业投资的 AI 效益将完全归零。 --- ### [07:22](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=442s) Lendflow 的实践：将中小企业借贷流程“程序化”与重塑长尾客户体验 针对 Lendflow 如何运用语音 Agent（Voice Agents）协助中小微企业完成贷款申请、资料收集、日程预约及唤醒沉睡线索（Revive cold leads），Zach 向 Jon Fry 询问了这些落地对转化率及放款速度（Speed to funding）的实际提升。 Jon 解释，AI Agent 带来的本质变革在于**将过去高度依赖“人力资本（Human Capital）”驱动的繁琐借贷流程转变为端到端的“程序化（Programmatic）”流程**。在传统借贷模式下，业务推进极易因等待人工发起下一步操作而陷入停滞，这在高速增长和规模化扩张期尤为致命。 ```mermaid flowchart LR accTitle: Lendflow 语音 Agent 与人机协作借贷流程 subgraph ClientStage["客户端全生命周期触达"] A["潜客/沉睡线索"] --> B["语音 Agent 自动化跟进"] B --> C["申请表单填写与资料收集"] C --> D["自动排期与推进放款"] end subgraph HumanStage["后台支撑体系"] E["人工团队（审核/QA/质检）"] F["持续数据反馈与模型迭代"] E <--> F end B -.-> E C -.-> E ``` 将借贷各触点“程序化”后带来的核心价值体现在以下维度： * **显著提升处理速度与转化率**：消除了人工断点，确保申请在各个环节不会无故搁置或丢失，大幅缩短了从申请到最终放款的周期。 * **全流程人机协作闭环**：前端由 AI Agent 作为第一道防线保障 24/7 实时响应，后台配备专业人工团队负责审计、质量保证（QA）以及持续构建数据反馈回路，促使 Agent 随时间推移不断进化。 * **激活长尾客户，重塑品牌声誉**：在传统纯人工模式下，受限于服务产能与投资回报率（ROI），借贷机构往往被迫将精力倾斜给高利润的大客户，忽视金额较小的长尾线索。而 AI 极大地摊薄了边际服务成本，使机构能够为所有客户（无论规模大小）提供一致的“魔法般体验（Magical experience）”。 * **提升用户留存与复借意愿**：即便部分小微企业最终未获得授信，Agent 也在交互中给予了充分的聆听、解答与金融知识普及（Education），加上定期的自动化回访跟进，极大增强了用户粘性与未来回流借款的可能性。 ### [09:47](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=587s) 客户体验中的信任建立与非财务 ROI 的衡量 在金融与借贷等存在严重信息不对称的行业中，用户往往因为无法判断哪个服务商或产品最优，而不断横向比价和尝试。此时，唯一能够与用户建立信任纽带的就是真实的交互体验。如果企业能够通过技术持续与客户保持联系、普及专业知识并稳定输出价值，客户再次回访和转化的概率就会大幅提升。这种体验的改善不仅影响业务的多个终端触点，更直接拉动了底层核心指标。 针对投资回报率（ROI）的衡量，主持人指出人们习惯于紧盯量化财务指标，但通过 AI Agent 提升客户体验所带来的情绪价值、信任感与品牌归属感同样不可忽视。这些定性维度的收获虽然难以直接用数字标价，但对客户留存率和企业的长期健康发展至关重要。与会嘉宾进一步强调，部署技术或进行内部投资时，ROI 绝不仅限于直接的账面回报；只要达成了预设的核心成功指标、切实撬动了业务杠杆，即使短期内未在财务上回本，同样是一笔成功的投资。关键在于明确业务的“北极星指标”（North Star），并以此指导技术的落地。 --- ### [11:09](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=669s) 消费级 Agent 的支付闭环：Mastercard 与 Agent Pay 针对近期 Pinterest 首席执行官 Bill Ready 在财报电话会上提出的观点——“AI Agent 代替用户自主购物的时代仍遥不可及”，主持人向万事达卡（Mastercard）高级副总裁 Federico Cohen Freue 提出了探讨。万事达卡早在当年 4 月便推出了 Agent Pay 功能，允许 AI Agent 代表用户完成安全支付。 ```mermaid flowchart LR accTitle: AI Agent 消费与支付闭环流程 subgraph S1["意图发现阶段"] A["用户向 Agent 提出需求<br/>(如按步态选跑鞋)"] --> B["消费级 LLM / Agent<br/>(ChatGPT / Claude / Gemini)"] end subgraph S2["支付与风控阶段"] B --> C["调用 Mastercard Agent Pay<br/>安全支付能力"] C --> D["内置防护栏与合规校验<br/>(消除非确定性风险)"] D --> E["无缝完成交易闭环"] end ``` Federico 指出，当前越来越多的互联网商品搜索与发现流量正在向消费级 Agent 转移，用户频繁使用 ChatGPT、Claude、Copilot 以及 Gemini 寻找心仪的商品。例如，用户根据自己的步态（Gate）和比赛类型向 Agent 咨询合适的跑鞋： - **断裂的体验闭环**：当用户选定商品后，自然的心理诉求是“直接帮我完成付款”，而不是被迫跳转到商家网站重新手动填写凭证和卡号。 - **无缝与信任**：万事达卡聚焦于解决 Agent 支付的互操作性与安全性，使 AI 交易达到与线下刷卡、手机触碰支付（Tap to Pay）同等级别的安全度和信任感。 - **化除非确定性（Non-determinism）**：Agent Pay 在生态系统的多个关键节点设置了防护栏（Guardrails），确保 Agent 仅在用户授权的明确边界内购买指定商品，消除大模型幻觉带来的交易偏差。 --- ### [14:38](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=878s) 金融强监管环境下的 Agent 护栏与落地：Chime 的全流程产品化策略 针对金融科技（Fintech）等强监管领域中 Agent 的安全与落地问题，Chime 首席数据官 Barkha Saxena 分享了 Chime 的实践路径。她指出，业内普遍存在实验很多但仅有约 30% 能真正投产上线的现象。Chime 正处于一个规模适中的“黄金平衡点”：既有足够的业务体量让 AI Agent 释放规模效应，又不会庞大到因组织冗余导致数据资产割裂、难以管控。 ```mermaid flowchart TD accTitle: Chime 跨职能 AI Agent 研发模式 subgraph Squad["跨职能产品团队 (Cross-Functional Squad)"] D["数据团队 (Data Team)"] A["AI 算法团队 (AI Team)"] R["风控团队 (Risk Team)"] P["合规与政策团队 (Policy Team)"] end Squad --> MVP["定义衡量标准 & 确立护栏"] MVP --> Track1["构建合规 Agent 产品"] MVP --> Track2["组织变革管理 (Change Management)"] MVP --> Track3["员工与用户培训 (Training)"] Track1 & Track2 & Track3 --> Output["高采用率与业务价值交付"] ``` 在资源分配上，Chime 严格区分黑客松式的自由探索与核心业务投资。作为一家以会员为中心（Member-obsessed）的公司，所有重大 AI 投入均从“能否为会员创造价值”出发（例如提升服务满意度、加速会员获取金融产品的流程）： - **全流程跨职能团队介入**：将 Agent 视作完整的产品来打造。从立项第一天起，数据团队、AI 团队、风控团队（Risk Team）与政策法规团队（Policy Team）便协同设计，提前确定合规标准与量化评估体系。 - **双轨并行的落地保障**：在技术开发的同时，同步推进文化转型（Culture Change）与技能培训（Training）。若缺乏变革管理，系统上线后往往会面临无人使用的尴尬局面。 --- ### [18:25](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1105s) 敏捷迭代与 MVP：避免单体式 Agent 系统的落地陷阱 与会嘉宾随后深入讨论了部署 AI Agent 时的交付策略。在传统的企业 IT 认知中，系统上线往往被看作一个非黑即白的“二元节点”（要么完全不上，要么全量部署）。但采用产品化视角部署 Agent 系统，必须打破这种单体式（Monolithic）开发的思维定势。 - **从最小可行产品（MVP）起步**：首先提炼出最核心的场景与能力边界交付给最终用户或内部员工，以较低的复杂度完成第一轮变更管理与习惯培养。 - **动态反馈与复杂度递增**：在后续迭代中逐步增加 Agent 的自主性和系统复杂性，将一线使用者（往往是企业内部员工）的真实反馈持续回流至模型调优中。 - **加速业务成果落地**：相比于闭门研发数月甚至数年、等待整个庞大系统“彻底就绪”再交付的做法，敏捷迭代能以更小试错成本让组织更快捕获实际业务收益。 ### [19:32](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1172s) AI Agent 的发布流程与回归仿真测试体系 部署具有自主性（Agency）的 AI Agent 必须遵循与传统软件工程一样严谨的发布规范。开发团队不能跳过任何既定环节，必须严格执行代码审查（Code Review）、Agent 行为审查以及系统的质量保证（QA）测试。由于 Agent 具备自主决策能力，往往对提示词或逻辑的一个微小改动，就会产生意料之外的连锁反应，破坏系统在过去某些复杂场景下的正常表现。为了解决这一问题，企业必须构建全自动化的历史场景仿真体系。 以语音 AI（Voice AI）为例，团队可以通过“以 Agent 测试 Agent”的方式，让多个模拟 Agent 拨打电话给待上线的目标 Agent，对其在海量历史场景中的交互行为和最终结果进行多维度指标测量。而在系统正式推向生产环境后，监控机制依然在实时运转：监控 Agent 会持续分析通话记录与用户情绪（Sentiment），一旦捕捉到负面情绪或交互失败的异常案例，就会立即将案例转交人工介入进行二次 QA 排查。最关键的是，每一次人工纠偏和复盘产生的案例，都会被即时沉淀并追加到仿真测试集中。随着测试集不断滚雪球式扩充，每次迭代发布都需跑通全部历史用例，从而确保 Agent 在生产环境中的输出精度与可靠性随着时间推移达到极高水准。 ```mermaid flowchart TD accTitle: AI Agent 仿真测试与持续迭代闭环 subgraph PreProd["测试与发布阶段"] A["修改 Agent 逻辑 / 提示词"] --> B["代码审查 & Agent 审查"] B --> C["仿真引擎：Agent-to-Agent 对抗模拟"] C --> D{"历史场景回归测试"} D -- "通过" --> E["发布至生产环境"] D -- "未通过" --> A end subgraph Production["生产监控与反馈闭环"] E --> F["生产实时监控 (情绪分析 / 失败检测)"] F --> G["人工 QA 介入核查与纠偏"] G --> H["转化为新增仿真测试用例"] H --> C end ``` - **全流程工程化规范**：Agent 上线需经历与传统产品功能相同的 Code Review、Agent Review 及 QA 流水线，严禁随意热更新。 - **Agent 对打仿真机制**：利用自动化 Agent 模拟用户端发起呼叫或请求，成规模地对新 Agent 施加回归测试，防范单点逻辑变动引发未知场景溃败。 - **持续反馈与用例沉淀**：通过情绪监控标记异常通话，人工纠偏后立即将新案例固化进回归用例库，驱动系统准度持续提升。 --- ### [21:08](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1268s) 运行时权限与红队测试：生态演进与冷启动困境 从投资人与市场观察者的视角来看，随着 AI Agent 真正代表用户执行自主操作的能力不断成熟，周边安全与测试工具链正在迅速崛起。例如 Witness AI 专注于运行时（Runtime）治理，确保企业员工在调用大模型或部署 Agent 时，Agent 所具备的数据访问权限严格受限于该员工在组织内的实际职权范围；而像 Haze Labs 等初创公司则深耕预发布（Pre-production）阶段，提供红队演练（Red Teaming）与对抗性测试（Adversarial Testing），在模型和 Agent 触达真实用户之前彻底扫除安全漏洞与逻辑缺陷。这也引发了企业在“自研还是采购”（Build vs. Buy）之间的深度权衡。 与此伴随的另一个核心痛点是企业的战略迷茫。在过去几年里，大量大型企业的 C-Suite 高管反复表达过类似的困惑：“我们深知公司必须拥有一套 AI 战略，但我们根本不知道这套战略到底该长什么样。”一年前这种焦虑围绕着大语言模型（LLM）展开，如今则全面转向了 AI Agent。面对“明知必须在组织内部某处落地、却不知从何下手”的局面，企业迫切需要找到最容易见效、门槛最低的切入点（Lowest-Hanging Fruit）。 - **运行时权限管控（Witness AI）**：基于员工角色与数据访问权限进行严格映射，防止 Agent 越权访问敏感企业数据。 - **预发布红队对抗（Haze Labs）**：在投入生产前通过对抗性测试与压力测试识别模型缺陷。 - **战略认知断层**：高管普遍具备拥抱 Agent 的紧迫感，但在落地场景选择与投入路径上存在显著的认知空白。 --- ### [22:38](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1358s) 双轨推进策略：自下而上普惠与自上而下价值闭环 企业在探索 AI 与 Agent 落地时，应当采取价值驱动的“双轨制”（Two-Pronged Strategy）策略： ```mermaid flowchart LR accTitle: AI 战略落地的双轨推进模型 subgraph Track1["自下而上：AI 民主化与草根创新"] T1A["全员普及 ChatGPT / Claude 等工具"] --> T1B["提示词工程技能培训"] T1B --> T1C["共享 Claude Artifacts / Projects"] T1C --> T1D["将高频草根方案封装进标准工作流"] end subgraph Track2["自上而下：价值导向与损益闭环"] T2A["三维评估框架"] --> T2B["用户期望度 (Desirability)"] T2A --> T2C["商业可行性 (Viability / P&L)"] T2A --> T2D["技术可行性 (Feasibility)"] T2B & T2C & T2D --> T2E["聚焦真正带来财务回报的核心用例"] end ``` 第一条轨道是“自下而上的 AI 普惠化（Democratization）”，让全员都能使用类似 ChatGPT、Claude 或企业内部封装的大模型工具。然而，“工具建好了大家自然会用并产生价值”往往是一厢情愿。最大的现实挑战在于员工是否具备编写优质 Prompt 的能力——“任何人都能写 Prompt，但写出好 Prompt 绝非易事”。这不仅需要企业提供提示词工程培训（引导员工明确目标、给出精准指令），更需要建立资产共享机制。例如员工利用 Claude Artifacts 或 Claude Projects 开发出了高效模板，如何将其共享给团队？当某个自发创新的工具在基层获得广泛认可后，企业便应将其提炼并内嵌到标准化业务工作流中。 第二条轨道则是“自上而下（Tops-Down）的聚焦与价值筛选”。盲目挑选一个局部场景建一个 Agent 并插上“成功交付”的旗帜毫无意义，因为如果无法映射到损益表（P&L），就根本没有实际业务价值流转进来。企业必须基于三个核心维度对落地场景进行严苛评估： 1. **期望度（Desirability）**：客户是否真正渴望并喜爱这项功能？ 2. **可行性（Viability）**：它能否为企业创造收入或显著降本，直接在财务损益表上体现影响？ 3. **技术可实现性（Feasibility）**：现有技术与数据条件是否足以支撑稳定交付？ --- ### [25:03](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1503s) Chime 实践：高阶决策框架与人机协同工作流重塑 作为内部提效与外部用户体验双向拥抱大模型的代表，数字银行 Chime 在评估与落地 AI Agent 时建立了一套极为务实的四维决策框架： | 评估维度 | 核心考量与落地标准 | |:--- |:--- | | **1. 用户价值（Member Value）** | 必须明确创造实际价值，无论是直接交付面向客户（Member-facing）的新产品功能，还是赋能内部研发团队大幅提速、快速消化长期技术排期。 | | **2. 规模与重复性（Scale & Repetitiveness）** | 拒绝为单次偶发的大流量场景盲目立项，重点寻找高频次、高度重复的任务场景，确保自动化在长期运行中产生规模效益。 | | **3. 风险承受度（Risk Appetite）** | 根据业务敏感度实施差异化策略。例如密码重置等低风险任务由 Agent 独立闭环；而在高接触度（High-Touch）复杂服务中，Agent 则作为辅助工具赋能人工客服。 | | **4. 数据底座质量（Data Quality & Context）** | 数据是 Agent 智能与上下文的根本来源。“AI Agent 的聪明程度完全取决于输入数据的质量”。如果底层数据出错，将导致决策失误，直接伤害客户体验与信任。 | 在具体落地中，Chime 极为强调工作流的端到端重塑（Workflow Redesign）与人机协作。例如在客服支持场景中，技术团队与一线客服人员紧密并肩作战共同构建解决方案，消除员工的排斥感；在风控（Risk）流程中，技术团队与风险专家逐一梳理工作节点，明确哪些环节数据就绪且风险可控可交由 Agent 给出决策，并重新设计后续人工复核与处置流程。 产品上线只是真正挑战的开端。Chime 坚持在系统投产后维持专职团队，持续追踪线上表现并构建闭环反馈机制（Feedback Loop），不断利用真实业务数据进行迭代优化，而非在交付首个版本后就就地解散团队。这种对数据底座、业务风险和人机协同的深度敬畏，构成了企业长期释放 Agent 价值的坚实壁垒。 ### [28:59](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1739s) 挑选落地场景的核心考量与数据创造力 主持人 Zach Gronfein 抛出一个针对现场观众的核心问题：企业内部如果想要确定第一批可以落地的 AI 智能体场景，应该问自己哪一个关键问题，才能把宝贵的精力、时间与资源集中在最刀刃的地方？ Barkha Saxena（Chime 首席数据官）首先指出，首要任务是寻找能够通过“自动化与重复性”获得规模化效益的场景，紧接着就是数据基础。她强调，根据自己的经验，许多概念验证（PoC）项目之所以无法投入生产环境，核心原因只有两个：**数据**和**企业文化**。 Jatin Dave（德勤 AI 董事总经理）针对数据问题补充道，虽然数据常被视为瓶颈，但很多时候可以通过创新方式“推断（infer）”或重构数据。他分享了一个生动的真实客户案例：该客户的客服团队原本抱怨说：“我们客服环境里的知识库根本没法用，因为所有的知识库文章都是一堆垃圾（a bunch of crap），从来没人维护。” 然而，该企业其实拥有大量经过验证的客服通话录音。通过将这些录音进行语音转录，他们成功提取并自动生成了高质量的标准知识库文章，进而顺利接入 AI 产品中。Jatin 指出，面对数据缺陷时，企业需要发挥创造力去构建正确的数据结构。 ```mermaid flowchart LR accTitle: 从非结构化通话录音重构知识库流程 subgraph RawData["原始多模态数据"] A["高可靠客服通话录音"] end subgraph Transform["数据重构与知识提取"] B["语音转录文本 (Transcription)"] C["知识文章自动生成与校验"] end subgraph Deployment["AI 应用落地"] D["结构化知识库 (Knowledge Base)"] E["AI 客服 Agent / 生产系统"] end A --> B B --> C C --> D D --> E ``` Federico Cohen Freue（万事达卡 AI 与数据运营高级副总裁）进一步总结了落地策略的两种路径： 1. **现成方案（Off-the-shelf）**：选择行业内已经非常成熟、讨论度极高的成熟场景（例如软件工程辅助编码、客服支持等），直接采购现成产品。 2. **内部切入的四大特征法**：如果选择在组织内部自行探索，只需寻找同时具备以下四个特征的任务或流程： - **易于获取数据**（Easy access to data） - **低风险**（Low risk，失败不会导致声誉受损或严重损失） - **高度可重复**（Repeatable） - **员工极度讨厌做**（People really dislike doing） Federico 强调，挑出员工反感且难度不高的任务切入，能够极大地消除组织阻力并推动技术采纳。企业不需要一开始就重构庞大的理论框架，而是应凭借常识迈出第一步，在早期用例中摸索出自己特有的规模化路径、合规标准以及生产环境的可观测性（Observability）体系。 --- ### [32:00](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1920s) 多 Agent 协同与编排层的下一代演进 讨论转向“下一代技术前沿与尚未完全实现的应用场景”。Jon Fry（Lendflow 首席执行官）从信贷借贷（Lending）业务出发，阐述了单一任务 Agent 向多 Agent 协同体系演进的必然趋势。 Jon 表示，企业往往是从低垂的果实（Low-hanging fruit）开始，自动化某项低价值、低效率的任务。但流程是连续的：一个 Agent 触发并完成任务后，需要通知下一个环节。在信贷场景中，一个完整的工作流可以拆解为多个专职 Agent： - **文档收集 Agent**：负责向用户索取必要贷款文件； - **分类与提取 Agent**：识别文件类型并结构化提取关键字段； - **数据丰富 Agent**：调用外部数据源对信息进行补充验证； - **计算与决策 Agent**：运行财务模型并执行信贷运算。 ```mermaid flowchart TD accTitle: 借贷流程中的多 Agent 协同与编排架构 subgraph Orchestration["顶层编排抽象层 (Orchestrator Layer)"] ORCH["工作流编排与上下文共享中枢"] end subgraph SubAgents["独立专业子 Agent 矩阵"] AG1["文档收集 Agent"] AG2["分类与数据提取 Agent"] AG3["外部数据丰富 Agent"] AG4["信贷计算与决策 Agent"] end ORCH <--> AG1 ORCH <--> AG2 ORCH <--> AG3 ORCH <--> AG4 AG1 -->|"输出文件"| AG2 AG2 -->|"提取数据"| AG3 AG3 -->|"增强数据"| AG4 ``` Jon 指出，未来的魔法时刻在于建立在这些子 Agent 之上的**编排层（Orchestration Layer）**。每个基础模型都有其独特的优劣势，编排层能够将不同模型组合为一个有机整体，并在各 Agent 之间实时共享上下文与数据，将底层系统复杂度完全从终端用户（无论是客户还是内部员工）面前抽象剥离，实现极度高效的自动化。 --- ### [34:02](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=2042s) 人性温度与 AI 自动化的边界 Barkha Saxena 则从反向视角探讨了“AI 永远不应该或无法完全替代”的领域。结合此前在 Poshmark（社交电商平台）长达十年的工作经验，她强调，任何涉及**高上下文（High Context）、高接触度（High Touch）以及人类复杂情感**的领域，都不可能也不应该被完全自动化： - **客户支持中的人性交互**：虽然部分客服流程可被自动化，但当用户产生情绪波动或面临复杂困境时，依然需要真实的真人对话与共情。 - **高度个性化的消费决策**：时尚与购物是极其个性化、带有强烈个人色彩的体验。消费者更希望自己做主，或是听取其信任的真人的建议，而非完全托付给算法。 Barkha 坦言，她衷心希望技术团队不要试图在这些带有情感色彩的领域推进 100% 的全盘自动化。 --- ### [35:31](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=2131s) 从会议代写邮件到智能 Slack 自动回复：解放琐事负担 针对 AI Agent 的理想形态，现场展开了一场幽默且切中痛点的讨论。嘉宾们普遍认同，Agent 的核心价值是代表人类去自主执行那些“大家讨厌的日常琐事”。 - **会议纪要与一键跟进**：Zach Gronfein 现场调查观众“有谁真正喜欢在开会时做笔记”，全场仅有一人举手。Zach 指出，目前的会议转录工具仅停留在输出文字记录层面；未来的 Agent 将能结合与对方历史沟通的完整上下文，在会议结束的瞬间自动起草好待办事项（Action Items）与跟进邮件，实现“一键发送”。 - **自动处理 Slack 消息的构想**：Barkha 开玩笑表示如果能有 Agent 每周自动帮她买牛奶她会很开心，但她最渴望的是能有 Agent 帮她自动回复所有 Slack 频道的未读消息。 - **上下文感知路由与真伪博弈**：针对 Barkha 的想法，Jon Fry 和其他嘉宾现场支招，指出这在技术上已经完全可行——只需要建立一个**分流路由层（Routing layer）**，让 Agent 监听所有公开和私聊频道，并针对不同身份（例如回复 CEO、回复团队成员还是对接产品部门）自动切换不同的语气和上下文。现场笑称，真正的博弈在于如何让回复逼真到别人察觉不出是 AI，但如果被同事识破了，也只能算作一场有趣的互动。 --- ### [37:55](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=2275s) 观众问答环节开启 随着全场讨论渐入高潮，主持人 Zach Gronfein 在会议最后阶段将话筒交给现场观众。来自 Ascent 公司的客户体验（CX）AI 负责人 Raheem 起身提问，探讨企业级落地的具体挑战。 ### [38:09](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=2289s) AI 落地成效度量：从表面活跃度到深层业务归因与产品思维 在评估企业级 AI 应用的成效时，单纯统计月活跃用户数（MAU）是最浅显的一步。真正的挑战在于深入交易与工作流内部进行精细化度量：在被支持的业务环节中，AI 的调用频率究竟有多高？其给出的答案质量如何？以生成会议纪要或销售邮件这类常见场景为例，当商务拓展（BizDev）代表发送邮件时，他们实际采用了多少 AI 原生生成的内容，又修改了多少？完成这项任务节省了多少时间？更关键的是，人工撰写的邮件与 AI 生成的邮件在客户回复率和转化率上是否存在显著差异？ ```mermaid flowchart TD accTitle: AI 业务价值度量与归因链路 subgraph S1["浅层采纳度量"] A["平台月活 (MAU)"] --> B["AI 功能调用频次"] end subgraph S2["过程质量与效率度量"] B --> C["AI 内容采纳率 / 修改幅度"] C --> D["单任务耗时缩短"] end subgraph S3["深层业务与财务归因"] D --> E["业务转化提升\n(回复率 / 赢单率)"] E --> F["财务回报 (P&L 影响)"] end ``` 这种度量虽然在技术上并不复杂，但在立项之初就需要大量严密的产品思维，以提前定义能够验证“产品是否足够好”的指标体系。企业常常在推行 AI 项目四到五个季度后遭遇严峻考验——CFO 会直接发问：“我们投入了大笔资金，财务损益表（P&L）上的实际收益在哪里？”此时，想将最终的销售额增长完全归因于 AI 是极为困难的，因为业绩增长可能源于同期市场营销活动的拉动或整体行业行情的向好。但企业可以通过前置的度量指标证明价值，例如证明“邮件回复率得到了显著改善”或“商机赢单率明显上升”。不幸的是，在许多企业中，成效度量往往是“事后诸葛亮”——大家往往在系统上线后为了应付 CFO 的质询才仓促开始设计指标。 - 采纳度与工作流渗透：不仅看平台整体 MAU，还要看具体业务交易中 AI 的渗透率。 - 人机协作精细指标：关注业务人员对 AI 生成草稿的采纳比例、改动幅度、任务处理时长对比。 - 业务转化对比：追踪纯人工撰写与 AI 生成内容的回复率（Response Rate）与转化赢单率（Win Rate）。 - CFO 归因挑战：销售增长难以剥离市场热度与营销活动影响，需依赖扎实的代理指标（Proxy Metrics）证明投资回报率。 --- ### [40:09](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=2409s) 破解重复性任务与提示词工程难题：自研 Prompt、超级用户与持续迭代机制 来自马萨诸塞州一家制药公司的微观数据科学团队负责人提出了一个普遍痛点：在尝试用 AI Agent 替代员工不愿意做的繁重、重复性工作时，究竟有没有“银弹”可以快速教育非技术员工掌握高质量的提示词工程（Prompt Engineering）？ ```mermaid flowchart LR accTitle: 提示词工程闭环优化与生产部署流程 subgraph Init["输入与初稿"] A["业务痛点 / 原始想法"] --> B["LLM 辅助构建结构化 Prompt"] end subgraph Feedback["异常与仿真闭环"] B --> C["实际运行 / 捕获异常"] C --> D["问题归因 & 反馈修正"] D --> E["多场景仿真测试"] end subgraph Deploy["交付生产"] E --> F["更新部署至 Production"] end ``` 对于这一难题，嘉宾们给出了结合组织机制与技术流程的复合解法： 1. **深入业务现场与“茶水间洞察”**：以产品思维和流程重塑视角切入，深入业务一线做发现工作（Discovery Work），分析数据流在何处停滞、哪些环节产生的异常（Exceptions）最多。同时，绝对不能忽视非正式的“茶水间闲聊（Water cooler talk）”——员工平时聚集在一起抱怨最多、公认最痛苦的流程节点，正是部署 Agent 的最佳突破口。 2. **利用大模型实现“以 AI 提示 AI”**：对于非专业开发者，无需强行钻研复杂的提示词技巧。一种行之有效的高效做法是让大模型为自己编写 Prompt——用户只需输入自己粗糙的原始想法，要求模型输出结构优良的提示词框架，再将结果直接复制回工具中，借助 LLM 自身的能力实现自我增强。 3. **推行组织级采纳项目与发掘“超级用户”**：在组织层面推进落地时，必须建立系统的采纳推进机制（Adoption Program）。以财务团队处理发票为例，员工可能需要面对格式混乱的发票或附带的繁杂表格，花两个小时手动复制粘贴到结构化表格中；如果他们从未亲眼见过 AI 的处理能力，就根本不会意识到自动化提取的可行性。通过展示“可能性的艺术（Art of the possible）”，并重点扶持团队中率先跑通流程的一到两名“超级用户（Super Users / Power Users）”，由他们将成功经验转化并在团队内分享推广，能带来远超自上而下说教的学习效果。 4. **建立端到端的 Prompt 持续迭代闭环**：提示词工程绝不仅限于写出第一版 Prompt，更关键的是后续的运维与优化机制。当线上出现错误时，系统需要能够识别具体问题所在、将错误场景反馈回模型、重新调整并生成新 Prompt、运行全量仿真以确保覆盖所有边界场景，最终将经验证的 Prompt 发布至生产环境。这种工程化迭代流程的价值远胜于最初单次编写的提示词。 - 痛点挖掘：从流程阻滞点、高频异常数据以及员工日常“茶水间吐槽”中锁定高价值用例。 - 提示词自我生成：非技术人员可直接委托 LLM 进行元提示词构建（Meta-prompting），快速获得结构化模板。 - 超级用户带动机制：无需全员成为技术专家，通过赋能少数先锋用户建立示范效应，加速组织级扩散。 - 闭环工程运维：涵盖“错误捕获 $\to$ Prompt 重构 $\to$ 场景仿真测试 $\to$ 生产上线”的完整生命周期管理。 ## Notable quotes > [03:01](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=181s) 我已经做够了 POC，再也不想做一个新的 POC 了 — Jatin Dave > [07:22](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=442s) 将过去高度依赖“人力资本”驱动的繁琐借贷流程转变为端到端的“程序化”流程 — Jon Fry > [11:09](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=669s) AI Agent 代替用户自主购物的时代仍遥不可及 — Bill Ready > [21:08](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1268s) 我们深知公司必须拥有一套 AI 战略，但我们根本不知道这套战略到底该长什么样 — Zach Gronfein > [22:38](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1358s) 任何人都能写 Prompt，但写出好 Prompt 绝非易事 — Federico Cohen Freue > [25:03](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1503s) AI Agent 的聪明程度完全取决于输入数据的质量 — Barkha Saxena > [28:59](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=1739s) 我们客服环境里的知识库根本没法用，因为所有的知识库文章都是一堆垃圾，从来没人维护 — Jatin Dave > [38:09](https://www.youtube.com/watch?v=v-6Ewla4JmI&t=2289s) 我们投入了大笔资金，财务损益表（P&L）上的实际收益在哪里？ — Jatin Dave ## Mentioned Zach Gronfein, Sound Ventures, Ashton Kutcher, Guy Oseary, OpenAI, Anthropic, Barkha Saxena, Chime, Federico Cohen Freue, Mastercard, Jon Fry, Lendflow, Jatin Dave, Deloitte, Pinterest, Bill Ready, ChatGPT, Claude, Gemini, Copilot, Agent Pay, Witness AI, Haze Labs, Claude Artifacts, Claude Projects, Poshmark, Slack, Ascent, Raheem ## Video description Zach Gronfein, Investor, Sound Ventures Barkha Saxena, Chief Data Officer, Chime Jon Fry, CEO, Lendflow, Inc. Jatin Dave, Managing Director, AI, Deloitte Federico Cohen Freue, Senior Vice President, AI & Data Operations, Mastercard Abstracts AI agents are already creating measurable business value across industries, from finance and healthcare to retail and manufacturing. In this panel, executives and AI leaders will share real-world success stories of AI agent deployments, detailing the tangible benefits they’ve achieved—including cost savings, increased efficiency, improved customer satisfaction, and new revenue streams. The discussion will highlight key lessons learned, common pitfalls, and the strategic decisions that led to successful implementations. Whether you're considering AI agents for your organization or looking to optimize existing deployments, this panel will offer invaluable insights from those leading the way. [Source](https://www.youtube.com/watch?v=v-6Ewla4JmI)

[3:01] Jatin Dave: 我已经做够了 POC，再也不想做一个新的 POC 了 [7:22] Jon Fry: 将过去高度依赖“人力资本”驱动的繁琐借贷流程转变为端到端的“程序化”流程 [11:09] Bill Ready: AI Agent 代替用户自主购物的时代仍遥不可及 [21:08] Zach Gronfein: 我们深知公司必须拥有一套 AI 战略，但我们根本不知道这套战略到底该长什么样 [22:38] Federico Cohen Freue: 任何人都能写 Prompt，但写出好 Prompt 绝非易事 [25:03] Barkha Saxena: AI Agent 的聪明程度完全取决于输入数据的质量 [28:59] Jatin Dave: 我们客服环境里的知识库根本没法用，因为所有的知识库文章都是一堆垃圾，从来没人维护 [38:09] Jatin Dave: 我们投入了大笔资金，财务损益表（P&L）上的实际收益在哪里？

[3:01] Jatin Dave: I'm done with PoCs. I don't want to do one more. [3:01] Jatin Dave: Recommend Section 529 College Savings Plan [34:04] Barkha Saxena: If there can be an AI agent or multiple AI agents who can respond to my Slack messages in all channels, I will take it... figuring out like how you respond to your CEO versus someone in your team to the product. [40:09] Federico Cohen Freue: What has actually worked for me is I use my AI agent to prompt engineer for me. So, I put in my thoughts and I say, give me a well-structured prompt that I'm going to give this tool, and then I literally copy and just feed it right back in... I haven't had to study prompt engineering because now for one-shot things... I'm good enough just being augmented by the LLM itself. [29:05] Jatin Dave: a bunch of crap

[0:00] All [0:04] right, thank you everyone for joining us [0:06] for this panel today. I know it's [0:07] probably been a long day hearing many [0:10] interesting topics. We have a great [0:12] discussion planned for you today uh for [0:15] you know really bright people attacking [0:17] the problem of deploying agents across [0:20] the the you know enterprise and for [0:22] customers via very different angles. So [0:25] I think it'd be great to start just [0:26] going down the line um introducing [0:28] everyone. I'll start. Uh, my name is [0:30] Zach Ronfine. I'm an investor at Sound [0:32] Ventures. We're a venture firm that is [0:35] very focused on the AI application [0:36] layer. Over a billion dollars in [0:38] management, co-founded by Ashton [0:40] Kutcher, the actor and Gio, the music [0:42] manager, um, investors in OpenAI and [0:46] Anthropic, as well as many of the [0:47] companies that are hopefully leveraging [0:49] those model providers to build [0:51] incredible value for all of us here. [0:54] >> Hey everyone, I'm Bara Sakenna. I am [0:57] chief data AI and analytics officer at [0:59] Chime. Uh Chime is a missiondriven [1:02] fintech company which is focused on uh [1:05] bringing financial progress for everyday [1:08] American. Um as part as in my role I [1:11] lead the entire data and AI strategy [1:14] which is focused on delivering value for [1:16] our members. Uh we have built and [1:18] deployed AI agents across uh risk, [1:21] marketing, engineering and support and [1:23] looking forward to this discussion. [1:27] Hi everybody. Uh my name is Federric [1:28] Goko and Frede. I'm senior vice [1:30] president of uh AI and data operations [1:32] at Mastercard. Uh and there I uh my team [1:35] focuses on our enterprise AI and data [1:38] strategy. How we deploy AI internally [1:41] for the use of our employees. Uh as well [1:44] as how do we actually go through that [1:46] journey? How do we train our employees [1:48] to learn about AI and use it? How do we [1:51] go through it? Uh as well as a number of [1:52] strategic efforts. So looking forward to [1:54] uh today. [1:56] Everyone, I'm uh John Fry. I'm the CEO [1:58] at Lindflow. Uh we have a platform um [2:01] for uh embedded lending infrastructure [2:04] and AI um and automation infrastructure. [2:07] I've uh been in the SMB lending space [2:09] for the past 15 years. I've had a few [2:11] different companies in the space um [2:13] operating across a few different parts [2:14] of the platform that gave us a lot of [2:16] the insights for Lindflow. um the [2:18] customer acquisition and the legion side [2:20] uh setting up call centers for customer [2:21] support and onboarding um as well as [2:23] building out decisioning and decisioning [2:25] infrastructure for uh SMB lenders. So [2:28] excited to dig in and go into the [2:29] details with all of you. Um yeah, great [2:31] to be here and good afternoon everyone. [2:34] My name is Jatin Dav. I'm a managing [2:36] director in the AI practice at Deote. [2:38] Deote is possibly the largest [2:41] professional services company in the [2:42] world. On the AI and engineering side, [2:45] we help our clients, enterprises think [2:48] through what their AI strategy should [2:50] be, building out those use cases and [2:53] solutions and platforms and also helping [2:56] them drive adoption and then maintain [2:58] those platforms. Um, nice to see all [3:00] here. [3:01] >> Great. I'd love to start with what's [3:03] working now and you know Jottton as [3:06] we've looked at the enterprise [3:08] landscape. Uh we've seen a number of [3:10] companies growing rapidly from a revenue [3:13] standpoint primarily driven by high [3:15] contract values and a strong willingness [3:17] to adopt from enterprises themselves. I [3:20] think we're in a period of AI tourism to [3:23] some degree where enterprises are [3:24] willing to adopt many solutions that may [3:26] be overlapping today with the goal of [3:29] cons consolidating and choosing a [3:30] long-term partner moving forward. From [3:33] your vantage point, what you know use [3:37] cases with AI agents are really uh most [3:41] prominent in your [3:43] understanding of of enterprise [3:44] deployments and when you're going from [3:46] PC to production. What use cases are [3:48] actually driving stickiness? [3:51] >> You're very right. You could say it's [3:53] the world of AI tourism. Folks here [3:55] would say, "I'm done with PC's. I don't [3:57] want to do one more." The reality is out [3:59] there our research shows only about 30% [4:02] of the PC's make it to production which [4:04] means people are trying a lot not [4:06] everything is even going into production [4:08] and then after it goes into production I [4:10] want I want to see people raise their [4:12] hands if you are actually measuring [4:14] adoption who here is saying I measure [4:17] adoption okay well good to see a few [4:19] hands up um those are the challenges in [4:22] taking an idea from P to production and [4:26] scale in terms of what's sticking um [4:29] within organizations the most common [4:32] ones I'm sure you've all heard of it's [4:34] about coding and mostly application [4:38] development in the IT space which is [4:40] where we see significant impact followed [4:43] by your common areas of customer service [4:46] contact center sales and marketing those [4:49] being the most common areas but about 60 [4:52] to 80% of the effort the number of use [4:56] cases that organizations are working on [4:58] are what are core to their business [5:00] processes. [5:02] For example, um in the financial [5:04] services space, your finan in the wealth [5:07] management space, your financial advisor [5:10] is a very core member of the firm that [5:13] leads to revenue generation. And in that [5:16] space, we're seeing a lot of use cases [5:18] that are already in production that are [5:21] driving adoption around, hey, help me [5:24] just good old take meeting notes. But [5:26] this isn't about your co-pilot on Teams, [5:28] let me press a button, get an output. [5:30] This is about driving action, which [5:31] means AI helping you identify [5:33] opportunities. AI telling you, hey, on [5:35] the call, I heard there was going to be [5:37] a grandchild. Therefore, I'm going to [5:39] add that grandchild to the family tree [5:41] so that on your next call, you know that [5:43] you have to recommend a 529 plan to this [5:47] person. So, it's really driving those [5:49] business processes that are heart to [5:51] their business. That's where we see most [5:54] impact, most opportunity in terms of [5:56] what that impact could look like for a [5:59] wealth management services firm. They're [6:02] focusing on the contact center, helping [6:04] these advisers [6:05] out there. They are consolidating the [6:09] skills in the contact center down to a [6:11] third, which means their contact center [6:13] reps are going to have a lot more skills [6:15] than they did in the past, which is [6:18] going to translate for them into fewer [6:21] handoffs between calls, which means [6:23] better, faster service going to the [6:26] financial advisors who in turn are going [6:28] to impact their clients. In terms of [6:30] what's driving stickiness, let me flip [6:32] it and say what is challenging in this [6:34] space. the the challenge areas are one I [6:39] don't have data or the data is really [6:42] unreliable that is honestly the biggest [6:45] blocker keep apart the infosc team [6:47] saying you can't go do this or can't do [6:49] that it's just if you don't have data [6:50] the AI is honestly practically useless [6:53] um second is the measurement strategy [6:55] where very few organizations have a [6:58] strategy or actively trying that third [7:00] is around adoption [7:02] do you have an adoption program in place [7:04] you've rolled out AI I honestly it's [7:06] optional today with human in the loop [7:09] cuz you have a button and if the end [7:11] user doesn't click the button and says [7:12] I'm going to do the research myself good [7:14] luck on getting any of those benefits. [7:16] So those are the areas that are honestly [7:18] blockers and what's not letting people [7:21] adopt it. [7:22] >> Super helpful and and John you know [7:24] you've mentioned using voice agents to [7:26] help SMBs complete applications collect [7:29] documents schedule calls you know revive [7:31] cold leads across your live deployments. [7:34] How have those agents impacted either [7:37] conversion rates or speed to funding [7:39] compared to some of your old processes? [7:42] >> Those are definitely a few of the key [7:44] metrics we focus on. Um, you know, it's [7:47] has impact on both. Before it's it's [7:49] really transforming a very, you know, [7:52] human capital um oriented process into a [7:55] programmatic one, right? And so that [7:57] means that you're not waiting for the [7:59] human always to initiate the next step. [8:01] that and lending um that can be really [8:03] challenging when you're operating at a [8:04] lot of scale, especially if you're [8:05] growing very quickly. And so being able [8:07] to transfer that to make that [8:09] programmatic um has a really big impact [8:11] on speed and also conversion things are [8:13] less likely to get lost. And so at each [8:15] of the touch points is just identifying [8:17] what those are through the process and [8:18] then implementing and making sure you [8:20] have those agents in place and you know [8:22] that leads to things getting done [8:24] faster, right? And of course, you always [8:26] have that back line of humans to augment [8:28] and support the process, to audit, to [8:30] QA, to have the feedback loop to make [8:32] sure that's improving over time, but um [8:34] you have that first line of defense that [8:36] you can count on things to happen when [8:38] you like them to because um there's a [8:40] lot of different touch points in the [8:41] process um through the life of that [8:42] customer, not only in in the initial [8:44] application, but all the way to the end. [8:46] And that ultimately also provides a much [8:48] better experience for the end customer. [8:50] Um and it also helps you service [8:52] customers. You know, it's better for [8:53] your brand, too, because it helps you [8:55] service customers that you couldn't [8:56] service well before because it wasn't [8:58] ROA positive or what didn't make as much [9:00] sense because you're are you going to [9:02] focus on the large uh you know uh most [9:05] profitable leads or are you going to [9:06] focus on the ones that are smaller and [9:08] um when you have limited capacity, it [9:10] means you're always focusing on the [9:11] larger ones less on the smaller ones. [9:13] Now, you can provide the same magical [9:15] experience across the board, which is [9:17] very good for your brand, right? It [9:18] means they're more likely to come back [9:19] to you and now you have the touch points [9:21] for them to come back as well. Also, it [9:22] creates great experience for the [9:24] customers which creates higher [9:25] conversion rates as well through the [9:27] process and also means that they're more [9:30] likely to return, right? They're more [9:31] likely to turn because they had a even [9:33] if they had a bad experience, maybe they [9:34] were educated in that process, right? [9:36] And so, they were taken care of. They [9:38] had they were heard. And if they [9:40] weren't, then um they're going to go off [9:42] to the next one. But if they did and [9:44] you're also following up and engaging [9:45] with them, then they're more likely to [9:47] re-engage with you than others they may [9:48] have dealt with, right? because it's [9:50] very likely they'd be, you know, [9:51] shopping around or trying different [9:52] things because of the information [9:54] asymmetry. They're not sure um what the [9:57] best provider or the best product is [9:59] going to be. And so, um the only thing [10:00] that can help them build trust is uh the [10:03] experience that they have. And so, if [10:05] you're staying in touch with them, [10:06] you're educating them, you're providing [10:08] value, you're consistent, um you're [10:10] building that trust, then um it also [10:12] incre creates a likelihood that you're [10:14] going to come back to you. So um many [10:17] different end endpoints are touch back [10:19] uh impacted and uh yeah it it touches [10:22] those key metrics a lot [10:23] >> and I think it's an important note and [10:24] we'll get to a section where we touch on [10:26] ROI shortly but often when we measure [10:29] ROI we're thinking quantitatively but [10:31] you're describing measuring ROI on a [10:34] qualitative basis from a customer [10:35] experience standpoint and that's that's [10:37] emotion that's feeling that's something [10:38] that you can't necessarily put into [10:40] numbers but matters just as much for [10:43] retention and and you know the long-term [10:44] health of the business. Um, [10:47] >> yeah. Yeah. I I would add I would add in [10:49] terms of that ROI doesn't necessarily [10:51] always have to be financial. Just like [10:53] with any other investment you'll make [10:54] internally or when you deploy [10:56] technology, you may have other goals for [10:58] it. And it's okay if it doesn't pay back [11:00] as long as you achieve the success [11:01] metrics. What's what's really important [11:02] is what are you really trying to achieve [11:04] there? Why are you moving the needle and [11:06] actually using that as your north star? [11:09] >> Absolutely. And I guess coming back to [11:11] you, um, interesting timing for this [11:14] panel because just last week, uh, [11:16] Pinterest CEO Bill Ready said on an [11:18] earnings call that, uh, the world where [11:20] AI agents autonomously shop on a user's [11:23] behalf, is still far from the future. [11:26] Mastercard in April launched agent pay, [11:28] which lets agents make secure AI [11:31] payments. I'm sorry, makes lets agents [11:33] make secure payments on behalf of users. [11:35] So, I have two questions for you, Feder [11:37] Rico. one, why is bill ready wrong? And [11:41] uh and two, you know, what are the most [11:44] common types of purchases that you're [11:45] seeing through agent pay today? [11:48] >> Yeah, h happy to happy to answer that. [11:50] Uh let's focus on uh on on a couple of [11:53] data points. What what we're seeing [11:54] already and and it's been it's been [11:56] public in the media is [11:59] a growing amount of internet traffic [12:01] related to product discovery is [12:02] happening more and more uh on the [12:05] consumer agent. So think chat GPT claude [12:07] co-pilot [12:09] Gemini people consumers are going to [12:12] these platforms to search for products [12:15] and as we started looking at at that [12:17] type of experience what we realized was [12:20] the loop isn't closed. what you expect [12:22] after you go in and you ask for, for [12:25] example, a pair of shoes for running [12:27] based on your gate and uh specifics of [12:30] the type of races you run. Was that once [12:33] you actually interact with the agent and [12:34] you select what pair of shoes you want [12:37] to buy, there's a natural tendency to [12:40] say, well, then go ahead and and and [12:43] execute the payment for me. Why do I [12:44] have to manually enter my credentials [12:47] and and go to the the merchants website [12:49] and do that? So we we there's been [12:51] signals um amongst consumer patterns [12:54] that that desire is there but then we [12:56] said well what's the right way to enable [12:59] that now while I won't make a comment on [13:02] on uh on on Bill Readyy's uh comment [13:06] what I will say is just like with any [13:09] other form of payments consumers and [13:11] businesses are going to select what is [13:12] most seamless and trustworthy for them [13:16] and so that's what we focus on we want [13:17] to make sure that if if if consumers [13:19] Consumers and businesses have a desire [13:20] to have AI agents make payments on their [13:24] behalf. Even for small routine types of [13:26] transactions, we have to make sure that [13:29] it's as seamless, as trustworthy, and as [13:32] secure as any transaction you make, [13:34] whether online or whether when you're [13:36] tapping your card in your phone. And so [13:37] that's where we focus on. So the way to [13:39] think about agent pay which is the [13:41] capability that we developed is how do [13:43] we take the capabilities in the network [13:45] and how do we make them so so that a [13:47] they're interoperable with the common [13:49] ways agents work but two that we do so [13:52] in a way that you take some of the [13:55] nondeterminism of AI agents and you say [13:57] how do I make sure that at different [14:00] points in the ecosystem I can make sure [14:02] that the agent is making a transaction [14:04] that you the consumer intended for the [14:07] products or services that you intended [14:09] within a number of of safeguards and [14:11] guardrails that you the consumer or you [14:13] the the you know the originating [14:14] business made and that's kind of the key [14:17] focus and so I I don't think we're ready [14:19] to make a statement on you know all [14:22] payments will be agent made or not but I [14:24] think it's much rather that [14:26] we see a growing a growing consensus [14:28] that AI agents or or consumers desire AI [14:32] agents to make payments on their behalf [14:33] and it's our role in the ecosystem to [14:35] enable that. It makes sense and it's it [14:38] actually brings up a topic that I think [14:40] everyone here could touch on um which is [14:43] guard railing, right? And making sure [14:44] that these agents are operating in ways [14:47] that are compliant and and uh to the [14:49] expectations of both the companies [14:51] deploying them and the users using them. [14:53] Maybe Barka starting with you in a [14:55] regulated environment like fintech, how [14:59] does I guess one, how is Chime, you [15:01] know, deploying agents both externally [15:03] to the users and internally today and [15:05] how do you think about you know ensuring [15:08] that these agents are performing within [15:10] the guardrails of your expectations. [15:12] >> Yep. Um great question. So I would say [15:14] you know first of all when we look at as [15:16] I was hearing everyone and especially [15:18] like Jatin when you were talking about [15:19] like people experiment and uh like only [15:22] 30% of the agents actually go [15:24] deployment. I think one unique thing [15:26] about Chime is uh uh we we are at I [15:30] would say at the sweet spot of the size [15:32] like we are we are big enough that there [15:34] is a scale for AI agents to deliver [15:37] value but at the same time we are not so [15:40] big where lots of things can happen here [15:42] and there and when it comes to [15:43] deployment where is my data uh before [15:46] chime I was at Mckenzie so I'm very [15:48] familiar of the the the problem you were [15:50] talking about uh but within that at [15:53] chime because it really does matter like [15:55] Hey, where are we making an investment? [15:56] Like every dollar counts. We can just [15:58] like yes people can experiment but we [16:00] leave that as more like the innovation [16:02] and the hackathon people trying but [16:04] there are areas where we identify that [16:07] hey here are the places where we really [16:09] want to invest in building AI agents to [16:12] deliver the value and uh the thinking [16:15] about like hey where do we want to [16:16] invest in it started with again Chime is [16:19] a very member obsessed company. So we [16:22] looked at it from the perspective of [16:23] what are the areas where we think we can [16:25] deliver value for our members and uh and [16:28] we are going to measure the success [16:30] based on the the member criteria whether [16:32] it could be uh the satisfaction we [16:34] deliver or uh we can help our member [16:37] access uh the whatever they were looking [16:40] for different financial products faster [16:42] and doing all of that being in a [16:45] regulated environment it's very [16:47] important that the AI agents we are [16:49] building it's data compliant as well as [16:51] the regulatory compliance and and then [16:54] adding to that uh uh you know you you [16:58] were all talking about the adoption [17:00] right you can you can build the product [17:02] but if nobody is going to use it it's [17:04] it's just not useful right so we took a [17:07] more like the the product approach of [17:09] building these AI agents because when [17:11] you build it with that mindset you are [17:14] like when when we think about this is [17:16] the area we're going to invest in there [17:17] is a cross functional team which comes [17:19] into to play for my team there's an AI [17:22] team and there's a data team because we [17:24] start with this is the data we are going [17:25] to we will need to build these agents [17:28] there is a risk team and the policy team [17:30] they're all in the designing of the [17:32] product from the very beginning and we [17:34] decide on how we are going to measure it [17:36] before we start to build the product and [17:38] then at the same time we have as we [17:41] started to invest heavily on the the AI [17:44] agents development and and just the [17:46] broader AI we at the same time we kicked [17:49] off parallel work streams around the [17:52] culture change because it's super [17:54] important and I remember coming from [17:56] McKenzie like there was a lot of thing [17:58] we were advising our clients were was [18:00] around the change management you have to [18:02] go through to get people to adopt the AI [18:04] agents and second the training because [18:06] the two have to go hand in hand in hand [18:09] so again taking the product approach has [18:11] allowed us to stay in the guard rails to [18:15] your question deliver the value and make [18:18] sure there's an adoption because we are [18:20] investing in parallel in terms of the [18:22] culture as well as u the training. [18:25] >> I'd add one more thing. So we've taken a [18:27] similar approach. We we like to talk [18:29] about it as we have a product team [18:30] that's focused on this. One of the other [18:33] benefits to it is it often gets you to [18:36] outcomes faster because one of the [18:39] things I keep hearing is well we either [18:41] deploy the agents or not and it's it's [18:43] not a binary moment right when you take [18:46] a product focused approach to deploying [18:48] AI agents or any technology for that [18:51] matter you start with an MVP right you [18:53] start with the the the lowest common [18:54] denominator you get enhance your people [18:57] you go through the first version of [18:58] change management as you're building the [18:59] next and as you grow in complexity the [19:03] agendic systems you're building, you're [19:05] also reflecting feedback from from your [19:07] your customers who in many cases are [19:10] your own people. And so that tends to [19:13] get you to outcomes faster than just [19:15] saying, I'm going to monolithically [19:17] build an AI AI agent system, wait until [19:20] I have it ready, and then deploy it. [19:22] Rather, working iteratively helps you [19:24] get get to those points. [19:27] Yeah, I'll I'll jump into I mean it's [19:29] it's analogous just to you know shipping [19:32] uh code um as well and you have to [19:35] follow the same processes as if you're [19:37] you know releasing a new feature uh and [19:39] this you know the same queue right [19:41] you're going to do a code review right [19:43] you're going to do a agent review you're [19:45] going to put it into QA you're going to [19:46] QA it um and then you're going to [19:48] implement maybe some different types of [19:50] processes that are the same um but just [19:53] can do those better so for example um [19:55] creating simulations of past scenarios [19:57] uh and having agents um call other [19:59] agents to test all the scenarios because [20:00] many times with agent uh with agents um [20:03] that have agency uh you can make one [20:05] change and it can impact another [20:07] scenario from the past or a number of [20:09] different scenarios that you didn't [20:10] anticipate and so you need to have [20:12] simulations for those to make sure that [20:14] when you ship them right you have for [20:16] example for voice AI you have many [20:18] different agents calling the new agent [20:20] that you've made the changes to and [20:21] measuring the outcomes and making sure [20:23] that um you know all the past scenarios [20:26] have been completed and then when you [20:28] ship it and you iterate to your point um [20:30] you have an agent is monitoring the [20:32] calls and looking for things like [20:34] sentiment of the call and and measuring [20:36] that and then looking for scenarios [20:38] where there was a bad sentiment or [20:39] something failed and then augmenting [20:41] that with humans who are manually going [20:42] through and QAing and each time there's [20:44] a correction that made it also needs to [20:45] be added into the simulations so that [20:47] you can make sure um now from that point [20:49] forward every time um you ship a change [20:52] to an agent right it's going through all [20:54] of the past scenarios and that list is [20:55] going grow and grow and so that you can [20:57] ensure over time with more and more um [20:59] precision that the agents are going to [21:01] be quality when they're in production [21:03] and after a certain amount of time then [21:05] you get to a very high level of [21:06] accuracy. [21:08] >> Absolutely. I mean and from our vantage [21:10] point, we're also seeing this market [21:12] emerge as the capabilities of agents to [21:14] truly act autonomously on users behalfs [21:17] emerge as well, right? You have [21:19] companies like Witness AI who are [21:20] operating at runtime, ensuring that [21:23] enterprises and and their employees can [21:25] use LLMs and based on their role within [21:27] the company um deploy agents that have [21:30] the same permissions that they do with [21:32] regard to data access. You have [21:34] companies like Haze Labs who are all [21:36] over the pre-production side and red [21:38] teaming adversarial testing make sure [21:40] making sure that models and agents are [21:42] ready to deploy before they hit users [21:44] hands. So um I think the buy versus [21:46] build dynamic is something that [21:48] companies I'm sure a lot of people in [21:49] this audience have faced themselves is [21:51] do I buy this? Do I build this with my [21:53] resources internally? I think we'll [21:55] continue to see that evolve. Um, you [21:58] know, John, I I I have one question for [22:00] you and and it relates to another thread [22:03] that we've heard over the last few years [22:05] where we've spoken to many seuitees at [22:07] large enterprises where they've said, [22:09] you know, we we need an AI strategy. We [22:12] know we need an AI strategy, but we [22:13] don't exactly know what that strategy [22:15] is. A year ago, that may have been as it [22:18] relates to LLMs. Today, it's as it [22:20] relates to agents. when you hear that [22:22] and you hear, hey, you know, we know [22:24] that we have to start implementing this [22:25] somewhere within the org, but we don't [22:27] know exactly where to start. What are [22:29] some of the kind of, you know, lowest [22:31] hanging fruit that you tend to recognize [22:33] or recommend as it relates to where [22:35] folks can deploy their first agent use [22:37] case? [22:38] >> Um, when it comes to deploying use cases [22:41] or deploying AI at a at a company, we [22:43] take a a valuedriven approach to helping [22:46] identify this broadly. When you when it [22:48] comes to strategies, it's a two-prong [22:50] strategy which most organizations [22:52] follow. One is let's democratize AI. [22:53] Everyone should have access to a chat [22:56] GPT like product whether it's claude, [22:59] whether it's your homegrown tooling [23:00] which in the background calls an LLM. So [23:03] that's one. And then second is a more [23:05] tops down focused [23:08] um effort around identifying value. on [23:11] the first one it's about well people [23:14] have access to to this chatbot that I've [23:17] rolled out now people will use it you [23:20] know I've built it they'll come they'll [23:22] use there'll be value coming out the [23:24] challenges on that one tend to be more [23:25] around the around do people even know [23:28] how to write a good prompt it's hard to [23:31] write good prompts anyone can write a [23:33] prompt but good prompts that's not easy [23:35] so have you trained your people in terms [23:37] of prompt engineering and I don't mean [23:39] everyone needs to write a two-page [23:41] prompt, but do do they know what they're [23:44] after? Are they being very specific in [23:46] those prompts? And then it's about [23:48] sharing within them, which means I've [23:51] built a prompt, a claude artifact or a [23:54] claude project. Now, how am I sharing [23:57] that with the rest of my team members? [23:59] Cuz it could be useful for for a lot of [24:00] people in the grassroots kind of world [24:03] where I wrote a problem for myself. I [24:05] might not know how to share it. And then [24:07] as you see more acceptance on on the [24:09] sharing front, how are you building it [24:11] within the workflow of the user? So that [24:14] means you're elevating something that is [24:16] done at the grassroots level up into [24:18] something that can be used in the [24:20] enterprise. And on the second one, which [24:22] is a more tops down approach, the the [24:25] evaluation criteria around not just ROI, [24:29] but around is this a desirable use case? [24:32] Would my customers love it if this were [24:34] part of the product? Is it viable in [24:37] terms of am I going to make money off of [24:39] it? Will it have a P&L impact? And then [24:41] is this feasible or not? That results in [24:44] us boiling down to the areas that matter [24:47] for the business. Cuz it's very easy to [24:50] go pick a use case, pick an area and [24:52] say, I'm going to build a piece of this [24:55] puzzle and then you can, you know, plant [24:57] a flag of success, but there's no value [25:00] flowing through at that point into the [25:01] P&L, [25:03] >> right? You know, Barka Chime was an [25:05] early adopter of LLMs internally to [25:09] improve efficiency amongst the team's [25:11] workflows. Uh, also an adopter of AI as [25:13] it relates to the user experience. I'm [25:16] curious, how do you decide which [25:18] internal workflows are best to augment [25:20] via AI via agents to drive efficiency [25:23] and what approach to integrating agents [25:26] has also uh maximized the user [25:28] experience as you've continued to [25:29] explore this landscape? [25:30] >> Yep. So I would say it's uh when we when [25:33] we think about the AI strategy we look [25:35] at the the four uh uh the four things as [25:38] we decide like hey do we really want to [25:40] put an AI agent there. So first one is [25:42] again it has to create value for members [25:44] sometimes it comes from the member [25:46] facing uh products we are rolling out [25:49] and sometimes it comes through like [25:51] internally enabling engineering teams to [25:53] build faster. So all the features which [25:55] we wanted to build for a longer time [25:57] horizon can be done faster. So first [25:59] that value proposition has to be clear. [26:01] Second uh the problem needs to have the [26:03] scale so that the automation actually [26:05] makes sense and within that uh it's not [26:08] the the problem of just like the hey the [26:10] one-time problem of the large volume but [26:12] is it a repetitive task which needs to [26:14] be done. So we look for those signals to [26:16] see hey if the agents are going to [26:17] deliver value in the long term. Third is [26:20] just the risk appetite right. So even [26:22] when we build the agents for the [26:24] support, we didn't automate the whole uh [26:27] workflow by design because there was [26:29] certain uh things which we wanted the [26:32] agents like the password reset. Of [26:34] course agents can do it but anything [26:36] which was hightouch we enabled a [26:38] different type of agents to actually [26:40] empower our agents uh serve our members [26:43] better. So that and then the fourth one [26:46] is is I think you called out the data. I [26:48] mean it's it's almost interesting that [26:50] everybody talks about it but they kind [26:52] of just brush it off like they don't [26:54] really very few people I guess the [26:57] people who are in the in the depth of [26:59] the data understand truly like how big [27:01] that problem is I mean at the end AI [27:04] agents are as intelligent as the data [27:06] you are putting in like beyond the LLM [27:08] your context comes from your data and if [27:10] that data is wrong it can create a risk [27:14] and if we end up serving our members [27:16] wrong it matters to us right when when a [27:18] member calls we need to have the right [27:20] information and the right decision for [27:22] the member. So again we look at it from [27:24] the the the member value uh the risk [27:27] appetite of the the decision the [27:29] automation and and is it a scalable [27:32] problem where we will derive the value [27:34] and and when it comes to the integration [27:36] again it's the workflow integration with [27:39] humans right even like for example when [27:41] we launched this for the support we [27:43] worked with the agents hand in hand to [27:46] make sure that we are building the [27:47] solution with them so that they feel [27:50] like fully integrated into the process [27:53] um in in some other places like when we [27:55] use these AI models for our risk uh [27:58] flows, we work with the risk teams to [28:00] make sure hey here are the parts of the [28:02] workflow which we believe that we are we [28:05] are comfortable the data is ready and we [28:07] want AI agents to be able to give us a [28:09] decision but then what is the workflow [28:11] after that so we are redesigning the [28:13] entire workflows to make sure uh we are [28:16] we are looking at the things end to end [28:18] and and once it's we we put it on and [28:21] start using it in production [28:23] We are monitoring it to make sure that [28:26] no matter how good product you build [28:28] like everybody who has built a product [28:29] you know that the real thing comes when [28:31] you when it goes live building that [28:33] feedback loop to make sure we are [28:34] learning and continuing to iteate on it [28:36] so the team doesn't dis assemble as soon [28:38] as the product gets delivered it [28:40] continues there is a and again uh having [28:43] been to the companies of the various [28:45] sizes like I do know that what we have [28:47] is kind of a luxury because a lot of [28:48] time you don't have it but I think we [28:50] just have this right scale and the and [28:52] the mindset and again having some uh [28:54] early start has allowed us to just [28:56] refine these things that as LLMs are [28:59] becoming more and more powerful we are [29:00] ready to be able to just continue to uh [29:03] invest and get value from it. [29:05] >> What is that one for people in the [29:07] audience? you know, if they're thinking [29:09] about those first few use cases they [29:12] could start to knock out inside of their [29:14] organizations, what's the one question [29:17] that they should be asking themselves to [29:18] help distill where they should dedicate [29:20] their energy, time, and resources today [29:22] as it relates to that use case. [29:25] >> Again, I'm going to so first of course [29:27] identifying the the use case where you [29:29] are going to get the scale through the [29:31] automation repetition and then data. I [29:33] cannot overemphasize uh I mean Jatin you [29:36] can speak to it but a lot of PC's don't [29:38] go into production for two reasons based [29:40] on my experience one is the data second [29:42] is the culture in the organization [29:45] >> on go ahead [29:46] >> on the data front I would say that [29:48] sometimes sometimes it is possible to [29:51] infer information we're working with a [29:54] client where they were like yeah we [29:56] can't use any of these knowledge bases [29:58] in our support environment because you [30:00] know our knowledge articles are a bunch [30:03] of crap. Nobody's maintaining them. [30:06] Well, but you have call recordings [30:09] which are reliable because you know that [30:11] solve the problem. You can transcribe [30:14] those and if you transcribe those, you [30:15] can come up with the right knowledge [30:18] articles which could then feed into AI. [30:20] In their case, they were buying a [30:22] product so they had to feed it in. So, [30:23] it was possible in their case to build [30:26] out the data structure in the right way. [30:28] But it's not as I said it's only [30:30] sometimes when you can do it but you [30:31] have to get creative with on the data [30:33] side. [30:34] >> I would I would mention I I think [30:36] there's two approaches you can take. One [30:38] is there's a lot of talk already ex uh [30:41] you know in the media about there's a [30:42] number of use cases that the industry [30:45] has seemed to got in into a good rhythm [30:47] in like software engineering customer [30:50] support etc. there's a reason why [30:52] there's a lot of talk about it and so [30:53] that's one approach like you can buy [30:55] offtheshelf solutions for that uh more [30:57] and more honestly if you just want to [30:59] look internally just find tasks or [31:02] processes that have very easy access to [31:04] data are low risk are repeatable and [31:08] that your people really dislike doing [31:10] you find things with those four [31:12] characteristics you can start there [31:14] because it'll drive adoption because [31:16] it's things where you're not going to [31:17] have a lot of resistance people want to [31:18] stop doing that type of work but also So [31:21] you cherrypick things that are easy, [31:22] maybe not very complex data, maybe not [31:24] something that if it goes wrong, you [31:26] know, it's going to cause reputational [31:28] or even worse worse type of damages. You [31:31] find things that are repeatable where [31:32] once you deploy it, you have confidence [31:34] that you can get it moving that way. [31:36] Just does you don't have to kind of [31:38] reinvent every framework. Just find what [31:40] is common sense. Start there, get you [31:42] testing because you go through those [31:44] first use cases, you'll realize, okay, [31:46] what is going to be our flavor to scale [31:48] things? what is going to AR flavor for [31:50] how we ensure that there's compliance [31:52] and that we do have observability once [31:54] we're in production and all of those [31:56] things. [31:57] >> Absolutely. Um I would love to move on [32:00] to a section of kind of you know what's [32:03] next, what is not possible yet that you [32:05] all are you all are excited about. John, [32:07] I'd love to start with you. as you [32:09] continue to follow where this space is [32:11] going, what use cases are are emerging [32:14] that have not been able to come to [32:18] fruition today, but that you are excited [32:20] about being able to implement into lend [32:22] flow moving forward. [32:25] >> Yeah, for sure. So, I think um as you go [32:27] along and you you have the first one, [32:29] right, the the lowhanging fruit, right? [32:30] Some repetitive low value inefficient [32:34] task and and you start there, right? Um [32:36] there's always going to be a natural [32:37] progression from that point, right? you [32:39] need, you know, need the trigger of [32:40] course to trigger that, but um when it's [32:42] finished, you also need to be able to [32:44] notify the next step of the process, [32:45] right? And then um maybe there's another [32:47] agent that can come in for that part of [32:49] the process and then um the next part, [32:52] right? So um it's an agent for maybe um [32:54] collecting a document in lending, right? [32:56] To extracting categorizing that [32:58] document, extracting data from that [32:59] document, then going and enriching with [33:01] new information and then acting on that [33:03] information, running calculations, [33:04] right? And each of those is independent [33:06] agent, right? Right? And as you go along [33:08] and as you progress then um the agents [33:10] can build off of each other. Right? And [33:12] so the context from each individual [33:14] agent can feed into the other agents so [33:16] they can all do a better job, right? And [33:18] you can have this um orchestration and [33:20] abstract more and more of the complexity [33:21] by having the orchestration layer on top [33:23] of these each of these individual [33:25] components, right? Think of that is like [33:26] a leg and an arm and you know a certain [33:28] activity. And if you have an [33:30] orchestrator on top, then it extracts [33:32] more and more of the complexity away [33:33] from the end users, whether that's a [33:35] customerf facing user or an internal [33:37] team member, right? And so I think we're [33:38] just really getting to the point where [33:40] you can have all these different types [33:41] of models and each have their strengths [33:43] and weaknesses. Um, but they can, um, [33:45] come together as a, you know, a larger [33:46] whole. And, uh, I think that's really [33:48] exciting. Um, and when they can share [33:50] context and the data that's been [33:51] mentioned several times, it's so [33:52] important. I think that's where you're [33:53] going to have the really magical [33:54] experiences happen. um and be able to [33:57] you know make these processes [33:59] hyperefficient um and great. [34:02] >> Absolutely. Yeah, Parker. [34:04] >> Yeah, I would say um like I think the [34:07] question was what one use case AI will [34:08] not be able to solve, right? That's [34:10] >> uh well, we can hit on that for sure. I [34:12] I the question that I had asked John was [34:15] more so what's one use case that's not [34:18] quite possible today given the [34:19] capabilities of the models but given [34:21] your understanding of the trajectory of [34:23] where things are going you are excited [34:25] about coming to fruition but [34:28] >> oh I was just going to answer the first [34:29] part of the question [34:30] >> yeah feel free to answer any part [34:33] >> yeah I would say that's where I am more [34:35] in the camp of the the the bill's [34:37] comment you mentioned in uh at Pinterest [34:40] so before chime and McKenzie I was at [34:43] Poshmark which is in the in the social [34:45] commerce space and and really saw the [34:48] fashion and what it means to people and [34:50] and I feel that anything where where any [34:54] there's any aspect of the the high [34:56] context and high touch and the and the [34:58] human emotions matter I don't think we [35:00] will be able to completely fully [35:02] automate it and that includes the [35:04] support because there are some part [35:06] which can be automated some does require [35:09] a human to talk so I would say that's [35:11] the place I have sincere hope that we [35:12] never try to do the full automation. [35:14] Similarly, like the shopping is such an [35:16] individualistic experience for some [35:18] people will be okay but there's a lot of [35:20] people like having spent 10 years at [35:22] Poshmark fashion is really really [35:25] personal like you want to make that [35:26] decision yourself or through the other [35:29] people people you trust. [35:31] >> Absolutely. I think the the the ideal [35:34] world of agents is one where you know on [35:37] the note of what Feder Rico mentioned [35:39] these pieces of software are acting [35:41] autonomously on our behalf for the tasks [35:44] that we do not like doing. [35:45] >> I I would be very happy if an AI agent [35:48] can just buy me milk every week so I [35:50] don't have to remember. [35:51] >> Well, I'm curious. Raise your hand if [35:52] you very much enjoy taking notes on [35:55] calls. [35:57] >> We we got one taker. [35:59] >> Not many. and and and we're seeing a lot [36:01] of platforms that are, you know, joining [36:03] meetings and transcribing meetings. And [36:04] today those transcripts kind of live [36:07] within that end output, but I think soon [36:10] we'll be in a world where given the [36:12] context of the multiple conversations [36:13] you've had with the individual on the [36:15] other line. You said you do X, Y, and Z [36:18] action item or follow-up and before that [36:19] meeting's over, you have that email [36:21] drafted ready for a one-click send. So [36:24] that's something that I'm personally [36:25] excited about where it understands all [36:27] of the data that I've accumulated in my [36:29] life and my conversations and now it's [36:31] actually taking on the burden of doing [36:33] some of the work that feels repetitive [36:35] more mundane and and not necessar not [36:37] necessarily related to uh my my brain [36:41] power and more so just the the energy I [36:42] have to expend on manual tasks. Actually [36:45] I would say as you are saying is there [36:47] is one example where AI agents today are [36:50] not able to do the task because I tried [36:52] to build something but I hope someday [36:54] they do. If there can be an AI agent or [36:58] multiple AI agents who can respond to my [37:01] Slack messages in all channels I will [37:04] take it. Going back to what you said, [37:06] right? The things you don't want to do. [37:08] >> Absolutely. [37:09] >> Uh that that can be done. That's done. [37:12] That's being done. [37:14] >> So [37:15] >> I will talk to you different types of [37:18] people and figuring out like how you [37:20] respond to your CEO versus someone in [37:23] your team to the product. I would love [37:25] to see [37:26] >> Yeah, you need a routing layer for each [37:27] each one and then you um route through [37:29] and then you just have it monitor all [37:31] the all the channels and all the [37:32] conversations you have. You can do the [37:33] private ones and the public ones as [37:35] well. So that's Yeah, [37:37] >> you are getting an email. [37:38] >> Yeah. [37:40] >> Guess who's going to reply? [37:43] >> Yeah, the trick is to make sure that [37:45] people can't um tell that it's an AI. So [37:48] I, you know, it's kind of good if they [37:49] know so that way they can call you out [37:51] if they if they recognize. Uh but, you [37:53] know, it's part of the game, so it's [37:54] fun. [37:55] >> Well, I I want to make sure um we get [37:58] maybe a couple questions from the [37:59] audience in the last five minutes that [38:01] we have here. Uh yeah, go go go for it. [38:05] >> Thank you. Uh my name is Raheem by the [38:07] way. I lead AI for CX for Ascent which [38:09] is a large Canadian company. Uh [38:11] Jonathan, you mentioned blockers to AI [38:13] was adoption, data and measurement. [38:16] Could you expand on the measurement [38:18] please for us? [38:19] >> Um so measurement is around it's around [38:23] one what does adoption look like? That's [38:26] easy. Everybody's looking at their [38:27] monthly active users on whatever [38:29] platform they use. But then within there [38:31] for the transactions that are being [38:33] enabled, how often is AI being used? And [38:36] then to what degree is it actually [38:39] giving the right answers for example [38:42] generating meeting minutes, generating [38:44] emails, those are very common use cases. [38:48] But when a rep or a bisdev rep actually [38:52] sends an email out, how much of the AI [38:55] originated answer did they use and how [38:57] much did they change it and how much how [39:00] much time did it take to do that work [39:02] and was their risk was the bisdev the [39:06] human writing the email what was the [39:08] response rate on that like versus the AI [39:10] generated ones right these are hard [39:12] these are easy things to measure but it [39:15] requires a lot of initial product [39:16] thinking to think about what am I going [39:18] to measure that is going to tell me if [39:20] what I built is good enough or not cuz [39:24] essentially what happens is four five [39:28] quarters out the CFO in an enterprise [39:30] says I spent a lot of money where's my [39:33] P&L impact and you know it's hard to [39:36] drive attribution down just to AI cuz [39:39] hey I built out a bisdev agent it's [39:42] sending out emails I'm selling more but [39:45] was that because there was a promot [39:47] motion that was running. Was that [39:48] because you're just in a hot market or [39:50] was that because of AI? It's hard to [39:52] attribute at that level. But you can say [39:55] my response rates were better, my win [39:57] rates were better. So that's the [39:59] measurement part that I'm talking about [40:01] and it's almost always an afterthought [40:03] because people are like, I rolled this [40:05] out and now I have to explain to the CFO [40:07] so I'm going to start measuring. [40:09] >> Hi, I'm Swapnil. I lead a data science [40:12] team at a pharma company in [40:13] Massachusetts. Uh I have a question [40:16] follow-up question for Fed Rico and [40:17] Jatin. Federico you mentioned build [40:19] agents which replace people's repetitive [40:22] tasks they don't like to do. Jin you [40:24] mentioned prompt engineering is very [40:27] very hard. I am working on something [40:30] similar where I'm trying to replace [40:31] those repetitive tasks. Is there any [40:34] magic silver bullet that can help me tie [40:36] people's need to replace these clunky [40:39] repetitive tasks they don't want to do [40:41] and kind of bring them uh how do I [40:44] educate them with better prompt [40:46] engineering because I have I've seen [40:48] that problem I have few read a few [40:52] directions on how that could be improved [40:53] but anything else you can expand on [40:56] >> yeah so I I'd share two things so going [40:58] back to the product thinking the way [40:59] that we we strategically have selected [41:02] did a number of areas around the company [41:04] where we've gone very deep and we said [41:06] we're really going to transform here. So [41:07] with that same product thinking we've [41:09] did we've done discovery work what do [41:11] people work on where in data do you see [41:14] processes stalling or where are the most [41:16] exceptions and that's where we started [41:18] like it is not revolutionary it's just [41:20] process re-engineering thinking and even [41:23] a simple version of that gets you to [41:25] start seeing that I would not also [41:28] discount just the water cooler talk like [41:31] where people in the company already [41:33] acknowledge there's an issue or there's [41:35] an opportunity you can start there. And [41:37] maybe to to pass over to Jatton, in [41:39] terms of prompt engineering, I'm [41:41] personally not an expert, but what has [41:43] actually worked for me is I use my AI [41:45] agent to prompt engineer for me. So, I [41:47] put in my thoughts and I say, give me a [41:49] well structured prompt that I'm going to [41:50] give this tool and then I literally copy [41:52] and just feed it right back in. It saves [41:56] me, you know, in in the in the spirit of [41:57] saving repetitive task. I I haven't had [42:00] to study prompt engineering because now [42:02] for you know like oneshot things or [42:04] whatnot. [42:06] I'm good enough just being augmented by [42:08] the LLM itself. I agree on the water [42:11] cooler part that you said. Absolutely. [42:13] That is truly the way to make this [42:15] happen. As you're thinking of doing that [42:18] at an organizational level, you do need [42:20] a program of some sort that you would [42:22] run. Let's just call it an adoption [42:25] program and then prompt engineering [42:27] training is a part of it. Truly what the [42:30] the way people improve upon it is one [42:33] they need to write a prompt. If you [42:34] think of finance organization where [42:36] people's job is processing invoices. [42:39] They may not be very keen on figuring [42:42] out what's the best way to extract [42:44] information from a really crappy invoice [42:46] or a spreadsheet they received along [42:48] with it that they'll spend two hours [42:50] trying to just copy paste over in a [42:51] structured format. Can AI do it for them [42:54] faster? Yes. But if they've never seen [42:56] it, they won't know that it's even [42:58] possible. So you you do need to run a [43:02] program that helps people see what's [43:04] called art of the possible. And it's not [43:05] always you showing it. It's more of the [43:09] team members. Let's just call them the [43:10] super users or the power users. The two [43:13] people in the team that have figured [43:14] something out, enabling them to share [43:16] with the larger team is what will drive [43:18] more. Call it prompt learning. [43:22] That's c certainly true, you know, [43:25] because you don't have to everyone build [43:26] their own agent for the prompt. You can [43:27] have someone do it and then make sure [43:29] it's translating it into a structure [43:31] that's been effective on on other agents [43:33] um in a language, you know, transferring [43:34] to maybe markdown or JSON or whatever [43:36] that most of the other team won't know. [43:38] And just pulling in some uh some other [43:40] uh pieces that people brought in from um [43:42] previous in the conversation. It's not [43:44] always it remember it's also this it's [43:46] not about the first prompt. You also [43:48] need the process of it went wrong, [43:50] identifying what went wrong, feeding it [43:52] back in and telling it rebuild the [43:54] prompt based on this and then running [43:56] the simulation to make sure it covered [43:58] all the scenarios and then pushing the [44:00] new prompt to production. So that is [44:02] equally if not more important part of [44:04] the process than just getting that [44:05] initial prompt.